natcap.invest (c)logging

DaveDave Member, NatCap Staff
Hey Team,

I'm reporting some strange -- but not necessarily unwanted -- behavior related to natcap.invest, logging, jupyter notebooks, and fiona.

I'm doing some pre/post-processing of invest results in a jupyter notebook where I've imported the following:


In my workflow, whenever I make a call to read/write a shapefile using geopandas (which then calls fiona), fiona spews out endless DEBUG messages to the console. It's especially problematic in Jupyter because all those printouts bog down the browser and eat up GB of memory. I noticed this only happens if I have natcap.invest imported.

I've got workarounds, so not looking for a fix, just thought this might reveal some unintended side-effects of how natcap.invest does it's logging. And I know how @James get's excited about logging!

Comments

  • DaveDave Member, NatCap Staff
    Oops, I posted too soon and left out the part where I say which packages I'm using:

    natcap.invest.coastal_vulnerability.coastal_vulnerability_core (3.3.3)
    geopandas (0.2.1)
    fiona (1.7.0)

    Thanks!
  • RichRich Administrator, NatCap Staff
    Oh I bet that's somewhere where we set the logging level.  Offhand I bet it's because the logging level is something global.  You might try a `logging.basicConfig(level=logging.INFO)` or `WARN` at the top of your code but after the invest imports.  I'm sure James will have something more interesting to suggest too.
  • jdouglassjdouglass Administrator, NatCap Staff
    Hey guys, you're both right on the money ... there's an issue with the current version of natcap.invest where importing (almost) any of the models or the UI will set up a logging handler that sends ALL logging across all packages to stdout, b ut only if such a handler hasn't already been set.  This will be fixed with the next release of InVEST.

    Until then, the best thing to do would be to do this in your script *before* importing natcap.invest:

    import logging
    logging.basicConfig(level=logging.ERROR)

    This way, only log messages with an ERROR level or higher would print to stdout, and the buggy logging stuff in natcap.invest will be ignored.  The full list of logging levels is here: https://docs.python.org/2/library/logging.html#logging-levels, if you'd rather tweak how much logging is allowed through, and basicConfig has some more parameters as well if you'd like to format the logging messags that do make it through: 
  • DaveDave Member, NatCap Staff
    Thanks guys, that did the trick!
This discussion has been closed.