Seasonal water Yield: invalid value encountered in double_scalars

Hello guys,

I come back to you because of an error I can't solve.
Everything went well for almost an hour, but the running crashed because of this "double_scalars" problem.

I don't really know what it means.
I searched across the topics but no one talks about it...

Thanks for any help !

Here is the log (I joined the full detailed log if it can help):


Traceback (most recent call last):
  File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\natcap.invest.iui.executor", line 560, in runModel
  File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\natcap.invest.seasonal_water_yield.seasonal_water_yield", line 158, in execute
  File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\natcap.invest.seasonal_water_yield.seasonal_water_yield", line 456, in _execute
  File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\natcap.invest.seasonal_water_yield.seasonal_water_yield", line 804, in _aggregate_recharge
RuntimeWarning: invalid value encountered in double_scalars


06/16/2016 16:19:53  root               ERROR    Exiting due to failures
06/16/2016 16:19:53  root               INFO     Elapsed time: 57m 49.79s

Comments

  • RichRich Administrator, NatCap Staff
    Hi Emile, it looks like it might be a case where there's an input watershed that's not covering any valid pixels.  But I can't be sure without seeing it myself.  Also not quite sure what to do in that case.  Can you dropbox me your datastack to richsharp@stanford.edu and I'll try to run on my end and see if I can make a reasonable fix.

  • RichRich Administrator, NatCap Staff
    Hi Emile, it looks like there might be a few problems with your data.  First, your soil raster has some nodata holes that invalidate the entire pixel stack of any other data that overlap with it.  (attached see black and white image with the green of the DEM showing through).  Also I ran the model with your data and much of the flow direction raster is full of holes which often happens with a DEM full of hydrological pits.

    What's ultimately causing your error is that the model is running as far as it can go and then aggregating the result over your watershed polygons.  Unfortunately many of the polys cover swaths of nodata which result in a divide by zero error when calculating the mean of whatever biophysical property is being summarized.  Also the stream layer is poorly defined since the flow accumulation algorithm can't walk upstream very far before encountering another pit.  I attached a picture below of the calculated flow accumulation raster overlaid on your watersheds; you can see the gaps of your polygons showing through the holes.

    In the end, the model shouldn't crash on a divide by zero error, but you're also still going to need to fix these other issues with your data before you can get a reliable result. Once you do get your data fixed you shouldn't have this error anymore, but nevertheless I've patched this issue that'll now simply report a warning on the output but complete all the way through.  If you find that helpful you can download the development version here that will be later merged into the next release of InVEST: http://data.naturalcapitalproject.org/nightly-build/invest-forks/richpsharp/InVEST_richpsharp3.3.1.post14+n88060995cc02_x86_Setup.exe
    578 x 466 - 36K
    567 x 445 - 353K
  • EmileEmile Member
    Hello Rich,

    Thank you again for your help, it is much much appreciated !

    I have filled the nulled values in the DEM and was confronted to a new error.

    I tried to make it through the DEM_Route but this new error kept coming :

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\pygeoprocessing.geoprocessing", line 98, in get_nodata_from_uri
    ValueError: cannot convert float NaN to integer

    I reviewed all my DEM and lulc and can't find any NaN. What's weird is that the error comes pretty early in the run, and I didn't change anything in my data but the DEM.

    I suspect the Lulc raster to be responsible for this because the watershed shapefile is a bit larger than the Lulc in some places, which can produce some Non-Valued pixels on the Lulc, but none of my trials to fix this succeeded.

    Sorry to come asking your help that much often =/
  • RichRich Administrator, NatCap Staff
    This is a little tricky, but looks like one of the input rasters has a "NaN" as its nodata value.  I've never seen that before but could see how it might happen if the raster were converted from one type to another.  Offhand I might guess that it's your DEM since you just did a lot of work on it, but it could be any of your input raster really.

    Can you take a look at the nodata values of your input rasters?  LULC, DEM, precips, soil group, and so on?

    If you can't find anything can you Dropbox me your datastack to richsharp@stanford.edu and I'll see if I can recreate the error on my end?
  • EmileEmile Member
    Hey Rich,

    The model completed today !

    Apparently it was the soil group raster's fault.
    I resolved it by taking my original raster, clipping it with my watershed shp and setting the zeros 0 as non-values.

    Thanks a lot for your constant help !

    Have I a problem if my Local recharge is positive on a river pixel ? I guess it's because my Lulc does not make any difference between waterways and lakes/ponds.
  • RichRich Administrator, NatCap Staff
    Depends on how the river pixel is defined.  The model creates a stream layer from the DEM that you can find in the intermediate outputs.   Local recharge should be zeros along all those pixels.  Otherwise that's the only way the model knows a pixel is supposed to be a river.
  • EmileEmile Member
    Well I Included rivers into my lulc now, setting CN_A and CN_B on 99 and 99, but my Stream.tif still comes all black (set on 0).

    I guess it's still my DEM working against me.

    I'll try with another DEM from STRM, hope it'll work.
    Do you have something else in mind if it doesn't ?

    Cheers
  • PerrinePerrine Moderator, NatCap Staff
    Hi Emile, 

    The rivers need not (and should probably not) be included in the LULC. The model will create the stream network based on the DEM.

    So I'd recommend checking your DEM and the threshold flow accumulation (a smaller value should result in longer stream network).

    You can test it with the RouteDEM model also in the InVEST suite.

    Cheers
  • EmileEmile Member
    Hi Perrine,

    Thanks for your reply.
    Actually I tried quite everything haha.
    Getting the rivers defined through my lulc helps the model to create the stream network as some of my water bodies get 99 as CN_A and CN_B. When I get a stream.tif full black without defining the rivers myself, it gets a little better with it.

    Secondly, I tried to reduce the number of watersheds too (from 700 to around 90). It helped a bit.

    Thirdly I put my DEM through the RouteDEM model but nothing changed. I also tried to find others DEM of my zone (ASTER, then STRM, then ...) but they all look the same (except the resolution differences).

    Fourthly (I don't know if we can really say fourthly haha), I tried different TFA from 50 to 1500 but when I get almost nothing with 1500, I just receive too many smallish non-existent streams with 50. Besides, medium values don't approach my stream network as much as your demonstrative default model does.

    I guess I don't have that many choices left and will take the averagely best looking model, but I doubt it will give greats results when I'll run SDR on it.

    Thanks again for your helpful comments
  • PerrinePerrine Moderator, NatCap Staff
    Hi Emile, 

    Just a few follow-ups:

    1) If you have the streams in your LULC, the model will treat these pixels correctly but will probably not represent the hydrologic routing correctly (because this routing is based on the streams delineated from the DEM)

    2) That's a lot of watersheds! Not a big issue in theory, but we usually work with a much smaller number of watersheds, which makes things more manageable (but maybe your decision context necessitates all these subunits)

    3) RouteDEM uses the same algorithm as SDR so you should get the same results. I was suggesting this to work on your DEM and avoid all the other steps involved with SDR (if the DEM was the issue)

    4) If you get small bits of streams everywhere with smaller TFA values, it may be that you have an issue with your DEM indeed. One option is that there are a lot of pits and the flow accumulation "stops" too often along a flow path.
    Have you tried filling your DEM? I've had some great results with the "Wang and Liu filling" algorithm in Qgis (from the SAGA library). Maybe something to try?

    Cheers

  • EmileEmile Member
    The 4) is the good one !

    I didn't know about this Wang and Liu Filling algorithm, it is super efficient !

    I used the output Wang and Liu DEM and the vectorized Wang and Liu watershed bassins raster given by the Algorithm and got a nice looking stream.tif.

    Thank you so much Perrine, I owe you a big one here ! :)
Sign In or Register to comment.