SDR model failing in ls factor stage

Hi - I am trying to script an analysis in the SDR model through the Python API and when I test on a a sample dataset I get the following error (the full output is given in the attached file). It also gives the same error running on the much larger dataset for my full study region.
...
05/02/2017 11:37:09  pygeoprocessing.routing.routing_core WARNING  no flow direction found for 8 2
05/02/2017 11:37:15  pygeoprocessing.routing.routing_core INFO     calculate transport cells_to_process.size() = 52324
05/02/2017 11:37:15  natcap.invest.sdr    INFO     calculate ls term
/usr/local/lib/python2.7/dist-packages/natcap/invest/sdr.py:358: RuntimeWarning: invalid value encountered in less
  percent_slope[valid_mask] < 9.0,
/usr/local/lib/python2.7/dist-packages/natcap/invest/sdr.py:372: RuntimeWarning: invalid value encountered in greater
  big_slope_mask = percent_slope[valid_mask] > slope_table[-1]
Traceback (most recent call last):
  File "sdr_parameterstest2.py", line 28, in <module>
    natcap.invest.sdr.execute(args)
  File "/usr/local/lib/python2.7/dist-packages/natcap/invest/sdr.py", line 178, in execute
    f_reg['flow_direction_path'], f_reg['ls_path'])
  File "/usr/local/lib/python2.7/dist-packages/natcap/invest/sdr.py", line 397, in _calculate_ls_factor
    "intersection", dataset_to_align_index=0, vectorize_op=False)
  File "/usr/local/lib/python2.7/dist-packages/pygeoprocessing/geoprocessing.py", line 2389, in vectorize_datasets
    out_block = dataset_pixel_op(*dataset_blocks)
  File "/usr/local/lib/python2.7/dist-packages/natcap/invest/sdr.py", line 379, in ls_factor_function
    m_exp[~big_slope_mask] = m_table[m_indexes]
IndexError: index 4 is out of bounds for axis 1 with size 4

I am running this using Python 2.7.12 inside a freshly-installed virtual machine of Mint Linux 18..1 I have installed the InVEST API through the pip command sudo pip install natcap.invest. The version of natcap.invest is 3.3.3. I have previously successfully run the InVEST SDR model through the GUI using an older binary Linux release of InVEST (the version was 3.3.0a1.post174+nce22418c1746, dating
from early in 2016), but this is my first time trying to run this module directly from Python. Does anybody have an idea what this error means?

Thanks,

Allan Hollander
Tagged:

Comments

  • RichRich Administrator, NatCap Staff
    Hi Allan, we did some non-backwards compatible overhaul of PyGeoprocessing in the 3.3.3 release toward a 1.0 release of that library.  Offhand I'd guess you still have the old version globally installed?  Can you try to `pip uninstall pygeoprocessing` as many times as it takes, then `pip install pygeoprocessing==0.4.0a2` to see if that fixes this issue?  Also might make sense to `pip uninstall natcap.invest` as many times as it takes too. At least this is what I do when I have issues like this.  :)

    Please post again if this doesn't help and we'll follow up further.
  • adhollanderadhollander Member
    edited May 9
    Hi Rich, I've now tried testing with uninstalling and reinstalling natcap.invest and uninstalling and then installing pygeoprocessing==0.4.0a2. This fails immediately with the error:

    Traceback (most recent call last):
      File "sdr_parameterstest2C.py", line 28, in <module>
        natcap.invest.sdr.execute(args)
      File "/usr/local/lib/python2.7/dist-packages/natcap/invest/sdr.py", line 113, in execute
        biophysical_table = pygeoprocessing.get_lookup_from_csv(
    AttributeError: 'module' object has no attribute 'get_lookup_from_csv'

    Trying again with the current natcap.invest and pygeoprocessing (installed via pip install natcap.invest and pip install pygeoprocessing) I get the original error with the ls factor, the last bit of the traceback being

    05/09/2017 16:45:10  pygeoprocessing.routing.routing_core INFO     calculate transport cells_to_process.size() = 243424
    05/09/2017 16:45:10  natcap.invest.sdr    INFO     calculate ls term
    /usr/local/lib/python2.7/dist-packages/natcap/invest/sdr.py:358: RuntimeWarning: invalid value encountered in less
      percent_slope[valid_mask] < 9.0,
    /usr/local/lib/python2.7/dist-packages/natcap/invest/sdr.py:372: RuntimeWarning: invalid value encountered in greater
      big_slope_mask = percent_slope[valid_mask] > slope_table[-1]
    Traceback (most recent call last):
      File "sdr_parameterstest2D.py", line 28, in <module>
        natcap.invest.sdr.execute(args)
      File "/usr/local/lib/python2.7/dist-packages/natcap/invest/sdr.py", line 178, in execute
        f_reg['flow_direction_path'], f_reg['ls_path'])
      File "/usr/local/lib/python2.7/dist-packages/natcap/invest/sdr.py", line 397, in _calculate_ls_factor
        "intersection", dataset_to_align_index=0, vectorize_op=False)
      File "/usr/local/lib/python2.7/dist-packages/pygeoprocessing/geoprocessing.py", line 2389, in vectorize_datasets
        out_block = dataset_pixel_op(*dataset_blocks)
      File "/usr/local/lib/python2.7/dist-packages/natcap/invest/sdr.py", line 379, in ls_factor_function
        m_exp[~big_slope_mask] = m_table[m_indexes]
    IndexError: index 4 is out of bounds for axis 1 with size 4


    -- Allan
    Post edited by adhollander on
  • RichRich Administrator, NatCap Staff
    Ahh okay, sorry about that.  I hadn't read carefully, somehow I thought you were doing a dev version of InVEST.  But yes, 3.3.3 is dependent on PyGeoprocessing 0.3.2 which it sounds like you have.  That exception that's raising is a numpy issue deep in a routine that shouldn't be exposed to any user data.  It's odd...

    Any chance you could dropbox me your script & data and I can run on my end to see if I can recreate that error and debug on my end?
  • Thanks for wanting to take a look at my test dataset! I've uploaded the files to a box.com folder at https://ucdavis.box.com/s/q8owsmbpsijnbvh3apko5ospkgqhhdtz .(Let me know if you have trouble accessing these). The parameters are in the file sdr_parameterstest2D.py; adjust pathnames to fit. I can successfully run this test dataset using an old binary linux InVEST distribution (InVEST version 3.3.0a1.post174+nce22418c1746), but not with the current Python code.

    -- Allan
  • RichRich Administrator, NatCap Staff
    Hi Allan, I'm still not sure what's going on there.  But running through your data once, I can see the DEM has hydrological pits in it.  I also saw some errors fly by during the slope calculation. I sort of wonder if there's some invalid DEM values in there somehow and the LS term was failing because it was processing one of those.

    At any rate, I filled your DEM (Wang & Liu in SAGA GIS) and was able to run the SDR model all the way through with the clean DEM.  Here's a link to the filled DEM in case you wanted to access it directly: https://www.dropbox.com/s/oldvk84zommhabd/dem30_no_sinks.tif?dl=0

    I hope that fixes the problem on your end too.  Please post again if not!
  • Hi Rich - thanks for your help! Using the fill routine in SAGA GIS I made a clean DEM for my entire study region (the Southern California National Forests and surrounding areas) and ran the SDR routine successfully as a script using the latest InVEST version. I guess something in the way the current PyGeoprocessing is interacting with NumPy in the SDR calculation demands having really clean DEMs.

    -- Allan
  • RichRich Administrator, NatCap Staff
    That's exactly it.  PyGeoprocessing has an implementation of a D-infinity flow algorithm (Taraboton-style), but that algorithm in itself doesn't resolve non-trivial hydrological pits.  Easy to think of those as craters, though most are simpler.  By the time we get to 1.0 we'll have an Wang&Liu implementation built in so this won't be an issue anymore, but that takes a couple of quiet weeks for me to get through something like that.  We 
Sign In or Register to comment.