This forum is shutting down! Please post new discussions at community.naturalcapitalproject.org

KeyError: 3

yonseiyonsei Member
edited March 2017 in Freshwater Models

Hello! I was trying to run Nutrient Delivery Ratio model and encountered an error. I looked through the forum for similar discussion and it seems like it refers to watershed ID in biophysical table. Could You please explain how to fix it?     


Initializing...

03/23/2017 00:25:52 root DEBUG Loading <module 'natcap.invest.ndr.ndr' from 'C:\INVEST~1.3_X\INVEST~1\natcap\invest\ndr\ndr.pyc'> in frozen environment

Arguments:

biophysical_table_path D:/new/biotable.csv

calc_n True

calc_p False

dem_path D:/new/routedem/pit_filled_dem.tif

k_param 2

lulc_path D:/new/lulc/lulc.tif

runoff_proxy_path D:/new/precipitation/precipitation.tif

subsurface_critical_length_n 150

subsurface_eff_n 0.8

threshold_flow_accumulation 1000

watersheds_path D:/new/delin/watersheds.shx

workspace_dir D:\new



03/23/2017 00:25:52 root INFO Logging will be saved to natcap.invest.ndr.ndr-log-2017-03-23--00_25_52.txt

03/23/2017 00:25:52 root DEBUG Loaded the model from natcap.invest.ndr.ndr

03/23/2017 00:25:52 root INFO Executing the loaded model

03/23/2017 00:25:52 root INFO Running InVEST version "3.3.3"

03/23/2017 00:25:52 root INFO Python architecture: ('32bit', 'WindowsPE')

03/23/2017 00:25:52 root INFO Disk space remaining for workspace: 255.57 GB

03/23/2017 00:25:52 root INFO Pointing temporary directory at the workspace at D:\new

03/23/2017 00:25:52 root INFO Starting natcap.invest.ndr.ndr

03/23/2017 00:25:52 root INFO Setting tempfile.tempdir to D:\new\tmp

03/23/2017 00:25:53 natcap.invest.ndr.ndr INFO calculating flow accumulation

03/23/2017 00:25:54 pygeoprocessing.routing.routing_core INFO labeling flats

03/23/2017 00:25:54 pygeoprocessing.routing.routing_core INFO draining away from higher

03/23/2017 00:25:54 pygeoprocessing.routing.routing_core INFO draining towards lower

03/23/2017 00:25:54 pygeoprocessing.routing DEBUG flats exist, calculating flow direction for them

03/23/2017 00:25:55 pygeoprocessing.routing.routing_core INFO labeling flats

03/23/2017 00:25:55 pygeoprocessing.routing.routing_core INFO draining away from higher

03/23/2017 00:25:55 pygeoprocessing.routing.routing_core INFO draining towards lower

03/23/2017 00:25:55 pygeoprocessing.routing INFO flats exist on second pass, must be flat areas that abut the raster edge

03/23/2017 00:25:56 pygeoprocessing.routing DEBUG starting flow accumulation

03/23/2017 00:25:58 natcap.invest.ndr.ndr INFO Calculating slope

03/23/2017 00:26:04 natcap.invest.ndr.ndr INFO Classifying streams from flow accumulation raster

03/23/2017 00:26:04 root ERROR ---------------------------------------------------

03/23/2017 00:26:04 root ERROR ---------------------- ERROR ----------------------

03/23/2017 00:26:04 root ERROR ---------------------------------------------------

03/23/2017 00:26:04 root ERROR Error: exception found while running natcap.invest.ndr.ndr

03/23/2017 00:26:04 root DEBUG

03/23/2017 00:26:04 root DEBUG Build details

03/23/2017 00:26:04 root DEBUG Interpreter

03/23/2017 00:26:04 root DEBUG Current temp dir: D:\new\tmp

03/23/2017 00:26:04 root DEBUG tempfile.tempdir: D:\new\tmp

03/23/2017 00:26:04 root DEBUG

03/23/2017 00:26:04 root DEBUG System

03/23/2017 00:26:04 root DEBUG OS : Windows-7-6.1.7601-SP1

03/23/2017 00:26:04 root DEBUG Processor architecture: AMD64

03/23/2017 00:26:04 root DEBUG FS encoding : mbcs

03/23/2017 00:26:04 root DEBUG Preferred encoding: cp949

03/23/2017 00:26:04 root DEBUG

03/23/2017 00:26:04 root DEBUG Python

03/23/2017 00:26:04 root DEBUG Version : 2.7.9

03/23/2017 00:26:04 root DEBUG Build : ('default', 'Dec 10 2014 12:24:55')

03/23/2017 00:26:04 root DEBUG Compiler : MSC v.1500 32 bit (Intel)

03/23/2017 00:26:04 root DEBUG Implementation : CPython

03/23/2017 00:26:04 root DEBUG Architecture : 32bit

03/23/2017 00:26:04 root DEBUG Linkage format : WindowsPE

03/23/2017 00:26:04 root DEBUG

03/23/2017 00:26:04 root DEBUG Packages

03/23/2017 00:26:04 root DEBUG Cython : ?

03/23/2017 00:26:04 root DEBUG Numpy : 1.11.2

03/23/2017 00:26:04 root DEBUG Scipy : 0.16.1

03/23/2017 00:26:04 root DEBUG OSGEO : 1.11.3

03/23/2017 00:26:04 root DEBUG Shapely : 1.5.5

03/23/2017 00:26:04 root DEBUG InVEST : 3.3.3

03/23/2017 00:26:04 root DEBUG

03/23/2017 00:26:04 root DEBUG

03/23/2017 00:26:04 root DEBUG Exception not environment-related

03/23/2017 00:26:04 root DEBUG Printing traceback

Traceback (most recent call last):

File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\natcap.invest.iui.executor", line 620, in runModel

File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\natcap.invest.ndr.ndr", line 385, in execute

File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\pygeoprocessing.geoprocessing", line 2389, in vectorize_datasets

File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\natcap.invest.ndr.ndr", line 297, in map_load

KeyError: 3



03/23/2017 00:26:04 root ERROR Exiting due to failures



https://www.dropbox.com/sh/1wxkg6tu0ag2km2/AACm3qPIQw7hK0ytvR3LroJRa?dl=0 ;


Thank You in advance!




Post edited by yonsei on

Comments

  • swolnyswolny Member, NatCap Staff
    Hi yonsei -

    Looking at your land cover map and biophysical table, and it looks like you have land use code values 1 through 24 in your lulc.tif (and aligned_lulc.tif), but your biophysical table (biotabledot.csv) does not contain these lucode values. Please update your biophysical table so that the lucodes in lulc.tif match the 'lucode' field in biotabledot.csv.

    ~ Stacie

  • Thank You, Stacie! I made corrections in biophysical table and it runs well.


    However looking at the results, I get very low values for both  N and P load and export in output watershed attribute table. I checked biophysical table, it seems to be correct.  


    Thank You, NatCap team, for your guidance! Your help is greatly appreciated! 


    1600 x 900 - 247K
    1283 x 862 - 191K
    1600 x 900 - 441K
    1600 x 900 - 358K
  • swolnyswolny Member, NatCap Staff
    Hi yonsei -

    Is the Shape_Area field in the table correct? If so, you're using some very small polygons for aggregation, so I'm not surprised that the corresponding N/P values are also very small. Similarly, what is your pixel size? Another possibility: double-check that the units for your inputs are what the model requires - the User Guide lists what each one should be.

    ~ Stacie

  • Thank You, Stacie. Cell size (X,Y) is 0.0083333333, 0.0083333333. I guess, that might be a reason...

    Also, is it possible that Nitrogen export value might be negative (as it  is shown in one of the images above)? 

  • Dear NatCap team, is there any way to correct the total P, N load and export values from watershed results other than running the model again? When I try to reproject input maps (cell size is no longer 0.0083... but 839.691.....   ) it says the intersection is empty. 


    Thank You !
  • swolnyswolny Member, NatCap Staff
    Hi yonsei -

    I'm not sure what you mean by "correcting" the results. But if you're changing any of the inputs to the model, you need to run the model again. Is there a particular reason why you are trying not to run it again?

    If the intersection is empty, look at the coordinate systems of all of your inputs, they should all be in the same projected (not geographic)  coordinate system (such as UTM, but others work too.) If any are in a different coordinate system, reproject them all to match and try running the model again.

    ~ Stacie

  • Thank You, Stacie. 

    The reason is that I had a very nice-looking Watersheds shapefile derived from DEM, and when I changed the properties (cell size) of raster files, and run the model, I got an empty intersection. Even though projection was the same, they didn't overlap with shapefile in ArcMap, probably because their extents were different... When I tried to derive watershed again from a "modified" DEM using Delineateit tool, I got small watersheds here and there, but bigger watersheds covering whole AOI were important for me. 

    However, using waterheds shapefile from hydrosheds.cr.usgs.gov worked well. I am new to ArcGIS and learning with You..Thank You, NatCap team, for your extesnive replies! 
  • swolnyswolny Member, NatCap Staff
    Hi yonsei -

    One thing to consider is whether the watershed that you're using from HydroSHEDS matches the basin on the DEM that you're using in the model. The HydroSHEDS watersheds were produced by using software to delineate basins from an SRTM DEM. If you are not using the same DEM data in the Nutrient model, it is possible that their watershed boundaries won't match up correctly with your topography, since each DEM is a bit different. If they don't match up correctly, then you'll be producing results that aren't completely hydrologically correct.

    You might have gotten smaller watersheds than expected using DelineateIt if your Outlet Points are further away from the stream network than the Snap Outlet Points value that you used as input to the model. Or, if the Threshold Flow Accumulation value was small, and created a lot of small streams, so the tool snapped the Outlet Points to the small streams, instead of bigger ones like you expected. You can play with the Snap Outlet Points and Threshold Flow Accumulation values and see if that improves the results.

    ~ Stacie
  • Thank You, Stacie.Your help is very much appreciated:)

    I downloaded HydroSHEDS Conditioned DEM (30arc sec) to match with HydroSHEDS Watersheds shapefile. 

    I have a very basic question about cell size of maps. The source maps before reprojection  have 0.0083 cell size. Could You provide some guidance on "right" cell size? 

    Thank You in advance! 
    568 x 473 - 37K
  • swolnyswolny Member, NatCap Staff
    Hi yonsei -

    There is no "right" cell size for the model. The right cell size is the one that lets you create a good topography for your area, provides the level of detail that you need for your study, and doesn't cause you to run out of memory while running the tools. 

    Data is generally collected at a particular resolution, and the data providers feel good about the data accuracy at that resolution. In this case, you have the 30 arc second resolution DEM, and if you look at the Resolution section of this HydroSheds page, that equates to around a 1km cell size, so that's the cell size that I would use. They also provide finer-resolution versions, if 1km is too coarse for your application. 

    DEMs in particular are tricky because when you change resolution, you can alter the topography, and thus slope, stream location, stream paths, etc, and really change the results of a hydrologic model. That's why our models resample all other inputs to be the same as the DEM, instead of resampling the DEM to match something else.

    ~ Stacie


Sign In or Register to comment.