SDR Crash

Hello there,

I am running Invest 3.3.1 SDR module in Windows 10 Enterprise.
The computer is a Mac, but running Windows through Bootcamp.
The model runs for about 30 minutes, then crashes right after "draining away from higher".

I've corrected the DEM (300 MB) for NoData and negative values, and it all seems to be projected properly. Is the file size too large?

Any insights?
Thanks,
NH

Comments

  • Hunt0416Hunt0416 Member
    edited November 2016
    I've attached a screen grab of what happens when the model crashes.  

    Thank you,
    NH
    Post edited by Hunt0416 on
  • Attached.
    1125 x 640 - 730K
  • RichRich Administrator, NatCap Staff
    Hi, looks like a segmentation fault which is a Bad Thing.  It would be strange if this were data related, but just to be sure, can you dropbox me your datastack to richsharp@stanford.edu?  

    Also there was a point where I was compiling a back-end library of InVEST with a sub-par compiler and the symptoms of that were occasional segmentation faults.  I've fixed that, but unfortunately I can't remember what version of InVEST I did that in.  I see you're running 3.3.1, would you be willing to try InVEST 3.3.2? http://data.naturalcapitalproject.org/invest-releases/InVEST_3_3_2_x86_Setup.exe

    Also, we have a Mac version of InVEST that you could try if you like http://data.naturalcapitalproject.org/invest-releases/InVEST 3.3.2.dmg

    So lets try all that and go from there?
  • Hi Rich,

    Thank you for your feedback, I have downloaded the 3.3.2 version and got the same crash results - at the same point in the progression.  I'll pass along the datastack shortly.

    Thanks again!
    Natalie
  • RichRich Administrator, NatCap Staff
    Hi Natalie, I'd emailed you this on the side, but also wanted to post here for any other users that might encounter this issue in any of the routed models:

    Your DEM is very large in terms of pixel dimensions (approximately 60k X 40k).  There is an outstanding bug in PyGeoprocessing related to very very large raster sizes that I have yet to get to, but I suspect is your issue here.  Is there any chance you could try downsampling your DEM?  Perrine recently submitted (if not already published) a paper that suggests that the accuracy of the SDR and NDR models is not strongly correlated with cell size up to a point.  5km pixels might be too large, but since you're asking a question about the entire US Midwest, 90m pixels would not add any significant discretization error to the final result and would also take your DEM size down by a factor of 9.  Would that be an acceptable approach for the moment?  I can't otherwise guarantee when I'll be able to address the routing bug in PyGeoprocessing.

  • PerrinePerrine Moderator, NatCap Staff
    Hey there, 

    Just to confirm that downsampling the DEM to 90m resolution is reasonable for most applications.
     
    The paper that Rich mentioned isn't published yet, but we published another one last year where we used 90m  DEMs (mostly because we had large study sites). See ref below if you want to read further.

    Cheers
    Perrine

    Ref: Chaplin-Kramer, B., Hamel, P., Sharp, R., Kowal, V., Wolny, S., Sim, S., Mueller, C., 2016. Landscape configuration is the primary driver of impacts on water quality associated with agricultural expansion. Environ. Res. Lett. 11. http://iopscience.iop.org/article/10.1088/1748-9326/11/7/074012

     
  • Excellent, thank you for this feedback!

    The newly 300m aggregated DEM seems to be working properly within SDR.  Now I'm encountering other errors that I'll try troubleshooting - something about bad allocations  (shapely.geos      ERROR    bad allocation).  

    Thanks again,
    Natalie
  • RichRich Administrator, NatCap Staff
    Hi Natalie, a GEOS bad allocation sounds Very Bad too.  Can you also dropbox me your new 300m DEM?  Hopefully I can get that error on my end and work from there?
  • RichRich Administrator, NatCap Staff
    Hi Natalie, I got your subsampled DEM.  On my end everything routed okay, but I got an issue that there wasn't an entry for a landcover "0" code in your biophysical table.  That appears to be the case when I look at your data and is an easy fix by adding an appropriate row.  Is that maybe what's happening on your end too?  

    I'm otherwise not able to recreate a bad allocation error.

    I hope that helps something.  Post again if not!
  • Hi Rich,

    I added a "0" to the biophysical table, and am still getting the same results. I've attached a screen grab of some of the log that doesn't seem to appear on the full log file (both are attached).

    Any thoughts?
    Thanks again!


  • After breaking up the DEM, NLCD, and other datasets into HUC2 -sized portions, the model runs!

    I think the errors were associated with the size, as you suggested earlier.

    Thank you again for all your help,
    Natalie
  • RichRich Administrator, NatCap Staff
    Ahh sorry I snoozed on this Natalie.  I think you got it.  We're aware of the crashyness of the InVEST routing and are generally working on it sooner rather than later.






  • p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica}
    p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px}

    Hi Rich,


    After having split up a larger DEM into smaller pieces, I am having difficulties with one of the sub watersheds (WBD10).  For the watershed_results_sdr.shp I get all -9999 values, and no rasters load when I click on the sed_export.tif, sed_retention.tif, or sed_retention_index.tif files.  


    I ran the DEM through all pit filling operations, checked that all data are in the same projection, all eff_n and eff_p values are less than 1.0.


    The WBD10 DEM comes from the same larger DEM as all other sub watersheds, and I am getting reasonable results from the others.  Do you have any insights?


    Thanks,

    Natalie Hunt


    Here's a link to the log file and datastack:

  • RichRich Administrator, NatCap Staff
    Hi Natalie, I ran your data on my end, and I do see some -9999s on the aggregate output, but in all subwatershed I see seem to correspond to subwatersheds that lie outside the extent of the raster data.  That would make sense in this case since we use -9999 as a placeholder nodata garbage value.

    You can see below where I highlighted the subwatersheds with -9999 in green and valid in red and overlaid a blue shaded DEM so you can see the extent of your data: http://i.imgur.com/pf4DYHU.jpg

    Also note some of those red watersheds extend beyond your data boundaries, but aren't invalid because they're half aggregating... In any case might be okay or not depending on your context!


  • Hi Rich,

    Thank you for your patience in all this, I really appreciate it.  I see that the -9999 values correspond to the watersheds outside of the DEM extent. The issue I’m having is that the for the watersheds within the extent, I’m getting sed_retent and sed_export values of 0 in the watershed_results_sdr.shp, and all raster outputs (sed_export.tif, sed_retention.tif) do not appear when I add them to my Map.

    This contrasts with the results I’m getting from the other parts of the larger Midwestern modeling area (which constitutes 10 modeling units).  All of the modeling units (including the troublesome one) come from the same larger DEM, LULC, erosivity, etc… datasets, so I’m at a loss as to what is making this particular unit so problematic. 

    Thanks,
    Natalie Hunt

  • RichRich Administrator, NatCap Staff
    Oh crap sorry, I don't know how I missed that.  It's very obvious when I bother to look. Could have saved me a lot of worthless visualization. :)

    There's a couple issues that are compounding your SDR export issue. The direct cause is because no streams are classified... BUT streams aren't classified because the flow accumulation raster is fragmented (take a look 'flow_accumulation.tif' in the intermediate output).  In turn, no pixel exceeds your threshold flow accumulation limit to make a stream in the first place.  SDR does a lot of calculations about distance downstream, but if none exist that factor turns into a 0/0 -> NaN and everything barfs.

    But easy fix: Fundamentally your flow accumulation raster is fragmented because the DEM has hydrological pits and downstream runs are terminated early.  Can you pit fill your DEM in your favorite pit filler (archydro or SAGA?) and give it a run again?


  • Thanks so much, the pitted-DEM was the problem.  Turns out, you have to run the sink filler multiple times on those larger DEMs.

    Thanks again!
    Natalie
Sign In or Register to comment.