Pre-processing toolbox for ArcGIS 10.4?

GuilleGuille Member
Hello!

I am working with RIOS and using the ArcGIS 10.4. I am getting an error that I already saw in some discussions on this forum:

Error processing Erosion Control
objective:  Failed to execute. Parameters
are not valid.

ERROR 001000: Lookup field: Field
Sed_Exp does not exist

Failed to execute (Lookup).


Is it a problem with the Arc version?


Thanks in advance!

Guille.

 

Comments

  • swolnyswolny Member, NatCap Staff
    Yes, the issue is that I haven't been keeping up with supporting new Arc versions, and every new version breaks scripts. Sorry about that.

    @LonBar has been great in rolling a RIOS preprocessor that doesn't require Arc - you can see the posting with information here. To run it, you'll need to install python packages and run the script, with input data as arguments. 

    I am the process of checking it out right now, and actually am stuck in a broken Python dependency, but I'll post back with my findings when I get it all running and compared with the Arc version. Looking through the code so far, it seems that it should perform well.

    ~ Stacie
  • GuilleGuille Member
    Thanks a lot Stacie!!

    I will also try it, I appreciate your help.

    Guille.
  • swolnyswolny Member, NatCap Staff
    Howdy -

    I just updated the ArcGIS version of the RIOS pre-processor for ArcGIS 10.4.1. It would be great if others have time to test it before we make it official. Attached is the zipfile with toolbox and script. 

    Nothing has changed about the interface (and hardly anything has changed about the script!) so if the toolbox somehow doesn't work for you, you can just point your old RIOS preprocessing toolbox at the .py script and it should work.

    I'll also post this on some of the other forum postings that have raised this issue. Please let me know if it works for you!

    ~ Stacie
  • GuilleGuille Member
    Hi Stacie!

    Thank you so much for this. I will try it and let you know ASAP.

    Guillermo.
  • Hi Stacie,

    I've also run into some problems running the RIOS preprocessor. The first problem was fixed by using the new version for AcrGIS 10.4.1, however now I run into a new one. The toolbox is able to calculate and produce the downslope and upslope retention rasters, but fails to compute the connectivity index - it also takes 4h to run (DEM = 10m over 630km2). I've gone through it twice and it keeps exiting at the same point.  

    Here is the log - please let me know if you have any tips!

    Sylvie

    Executing: RIOS-Pre-Processing true true true # # C:\Users\Labo_JD\Documents\Sylvia\RIOS\preprocessing2 C:\Users\Labo_JD\Documents\Sylvia\RIOS\Composite_CDL_MDDELCC.tif C:\Users\Labo_JD\Documents\Sylvia\RIOS\Biophysical_coefficients_reduced.csv C:\Users\Labo_JD\Documents\Sylvia\RIOS\NED_DEM_burnt.tif "C:\Users\Labo_JD\Documents\Sylvia\Models\Baie Missiquoi\SDR\GlobalR_UTM.tif" "C:\Users\Labo_JD\Documents\Sylvia\Models\Baie Missiquoi\SDR\Erodibility_clip_UTM.tif" "C:\Users\Labo_JD\Documents\Sylvia\Models\Water yield\Root_UTM.tif" # # # # 10 30 "C:\Users\Labo_JD\Documents\Sylvia\Models\Baie Missiquoi\BM_watershed_UTM_adj.shp" #
    Start Time: Wed Nov 15 09:39:19 2017
    Running script RIOS-Pre-Processing...

    Validating arguments...

    Erosion Control selected, checking required inputs:
    DEM
    Erosivity
    Erodibility
    Soil depth
    Land use/land cover
    RIOS general coefficient table
    Threshold flow accumulation
    Riparian buffer distance

    Phosphorus Retention selected, checking required inputs:
    DEM
    Erosivity
    Erodibility
    Soil depth
    Land use/land cover
    RIOS general coefficient table
    Threshold flow accumulation
    Riparian buffer distance

    Nitrogen Retention selected, checking required inputs:
    DEM
    Soil depth
    Land use/land cover
    RIOS general coefficient table
    Threshold flow accumulation
    Riparian buffer distance

    Checking input raster projections...

    Creating hydrology layers...


    Processing Erosion Control objective...

    Mapping coefficients to landcover...

    Creating downslope retention index...

    Defining flow direction channels...

    Created Erosion downslope retention index: C:\Users\Labo_JD\Documents\Sylvia\RIOS\preprocessing2\Output\erosion_downslope_retention_index.tif

    Creating upslope source...

    Created Erosion upslope source: C:\Users\Labo_JD\Documents\Sylvia\RIOS\preprocessing2\Output\erosion_upslope_source.tif

    Creating riparian continuity index...

    Created stream layer C:\Users\Labo_JD\Documents\Sylvia\RIOS\preprocessing2\Output\streams_10.shp

    Error creating riparian continuity index:  ERROR 999999: Error executing function.
    Failed to execute (Buffer).


    Error processing Erosion Control objective:  ERROR 999999: Error executing function.
    Failed to execute (Buffer).


    Error running script
    Failed script RIOS-Pre-Processing...

    Traceback (most recent call last):
      File "C:\Users\Labo_JD\Documents\Sylvia\RIOS\RIOS_Pre_Processing_ArcGIS_10_4_1\RIOS_Pre_Processing.py", line 1280, in <module>
        raise Exception
    Exception
  • swolnyswolny Member, NatCap Staff
    Hi Sylvie -

    One place I've seen this before is when there's a very complex/detailed stream network, such that Arc has trouble making buffers around all of the streams. Do you happen to have a stream network like this? It looks like you might be using a threshold flow accumulation of 10, with a 10m DEM, which I think should produce a ton of tributaries to buffer. Let me know if this is the case. 

    ~ Stacie
  • Hi Stacie,

    Thanks for getting back to me. Yes this is the case. I am using a flow accumulation threshold of 10 because in my runs of the SDR and NDR models I found that if I used anything bigger I got large holes in my results dataset. Perhaps this is not the best approach. Do you recommend re-running the preprocessor with a larger flow accumulation threshold?

    Best
    Sylvie
  • swolnyswolny Member, NatCap Staff
    Hi Sylvie -

    The flow accumulation threshold should be set so that the resulting stream output most closely matches real world streams. This is really important for the SDR and NDR models, since the amount and properties of the landscape between each grid cell and the stream has a big impact on the result - if there's a large distance between the cell and the stream, it's much less likely that any sediment/nutrient generated on that cell will make it to the stream (especially if there's good natural veg there.) So the export output will be very different if you set the TFA to 10 (which will generate lots of tributaries) vs, say, 1000 (medium amount of tributaries), vs 10,000 (far fewer tributaries.)

    If you're getting holes in the SDR/NDR output, the first thing to look at is your DEM, as these models are also sensitive to any hydrologic pits. If you haven't already, look at the stream layer, if there are discontinuous streams then you have pits. Try filling the DEM with QGIS's Wang and Liu algorithm and see if that helps. 

    ~ Stacie
  •  Hi Stacie,

    Thanks for these tips. I ended up using a different DEM but with a TFA of 100 which seemed to give me decent results without lots of holes. If I upped the TFA to 1000, it started to look like swiss cheese. 

    I'm now running into an error when I run RIOS which says 'NoneType' object has no attribute GetRasterBand'. Looking on other forum posts this came up with the Coastal vulnerability model as well, but that seemed to get fixed with a patch to the model. Have others run into this error before using RIOS? Below is my log output. Happy to share the data inputs as well. 

    Thanks Sylvie

    Initializing...

    12/04/2017 10:47:35 root DEBUG Loading <module 'natcap.rios.rios' from 'C:\PROGRA~1\RIOS11~1.16X\natcap\rios\rios.pyc'> in frozen environment

    Arguments:

    activities {u'Culture_Couverture': {u'measurement_unit': u'area', u'measurement_value': 10000.0, u'unit_cost': 6000.0}, u'Bandes_5m': {u'measurement_unit': u'area', u'measurement_value': 10000.0, u'unit_cost': 3000.0}, u'Drainage_controllee': {u'measurement_unit': u'area', u'measurement_value': 10000.0, u'unit_cost': 1000.0}, u'Bande_10m': {u'measurement_unit': u'area', u'measurement_value': 10000.0, u'unit_cost': 5000.0}}

    activity_shapefiles [u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/Buffer_prevent.shp', u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/ControlledDrainage_prevent.shp']

    budget_config {u'years_to_spend': 1, u'activity_budget': {u'Culture_Couverture': {u'budget_amount': 25000.0}, u'Bandes_5m': {u'budget_amount': 25000.0}, u'Drainage_controllee': {u'budget_amount': 25000.0}, u'Bande_10m': {u'budget_amount': 25000.0}}, u'if_left_over': u'Report remainder', u'floating_budget': 0}

    lulc_activity_potential_map {0: [], 5120: [u'Culture_Couverture'], 2180: [u'Culture_Couverture'], 2181: [u'Culture_Couverture'], 15110: [], 10120: [u'Culture_Couverture'], 5131: [u'Culture_Couverture'], 5134: [u'Culture_Couverture'], 15120: [u'Culture_Couverture'], 10131: [u'Culture_Couverture'], 20: [], 5141: [u'Culture_Couverture'], 10134: [u'Culture_Couverture'], 20120: [u'Culture_Couverture'], 5147: [u'Culture_Couverture'], 10140: [u'Culture_Couverture'], 10141: [u'Culture_Couverture'], 15134: [u'Culture_Couverture'], 10181: [u'Culture_Couverture'], 5168: [u'Culture_Couverture'], 15131: [u'Culture_Couverture'], 15140: [u'Culture_Couverture'], 5030: [u'Culture_Couverture'], 10158: [u'Culture_Couverture'], 5160: [u'Culture_Couverture'], 2140: [u'Culture_Couverture'], 15147: [u'Culture_Couverture'], 20140: [u'Culture_Couverture'], 2158: [u'Culture_Couverture'], 10030: [u'Culture_Couverture'], 2141: [u'Culture_Couverture'], 10160: [u'Culture_Couverture'], 2120: [u'Culture_Couverture'], 50: [], 20147: [u'Culture_Couverture'], 15158: [u'Culture_Couverture'], 10168: [u'Culture_Couverture'], 20030: [u'Culture_Couverture'], 5180: [u'Culture_Couverture'], 5181: [u'Culture_Couverture'], 2110: [], 15168: [u'Culture_Couverture'], 10180: [u'Culture_Couverture'], 15030: [u'Culture_Couverture'], 20168: [u'Culture_Couverture'], 15180: [u'Culture_Couverture'], 210: [], 2131: [u'Culture_Couverture'], 20180: [u'Culture_Couverture'], 20110: [], 220: [], 10147: [u'Culture_Couverture'], 351: [], 2147: [u'Culture_Couverture'], 5158: [u'Culture_Couverture'], 230: [], 1000: [], 2153: [u'Culture_Couverture'], 2030: [u'Culture_Couverture'], 20134: [u'Culture_Couverture'], 2160: [u'Culture_Couverture'], 10110: [], 302: [], 20158: [u'Culture_Couverture'], 5110: [], 2168: [u'Culture_Couverture'], 2132: [u'Culture_Couverture'], 250: [], 255: [], 20131: [u'Culture_Couverture'], 5140: [u'Culture_Couverture'], 341: []}

    lulc_coefficients_table_uri C:/Users/Labo_JD/Documents/Sylvia/RIOS/Slope_specific/Biophysical_coefficients_byslope.csv

    lulc_uri C:/Users/Labo_JD/Documents/Sylvia/RIOS/Slope_specific/LULC_Slope_aggregated.tif

    objectives {u'erosion_drinking_control': {u'rios_model_type': u'rios_tier_0', u'priorities': {u'revegetation_assisted': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': u'~0.5', u'Beneficiaries': 1.0, u'On-pixel source': 0.25, u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}, u'agricultural_vegetation_managment': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': u'~0.5', u'Beneficiaries': 1.0, u'On-pixel source': 0.25, u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}, u'fertilizer_management': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': u'~0.5', u'Beneficiaries': 1.0, u'On-pixel source': 0.25, u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}, u'keep_native_vegetation': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': 0.5, u'Beneficiaries': 1.0, u'On-pixel source': u'~0.25', u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}, u'ditching': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': u'~0.5', u'Beneficiaries': 1.0, u'On-pixel source': 0.25, u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}, u'revegetation_unassisted': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': u'~0.5', u'Beneficiaries': 1.0, u'On-pixel source': 0.25, u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}, u'pasture_management': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': u'~0.5', u'Beneficiaries': 1.0, u'On-pixel source': 0.25, u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}}, u'factors': {u'Soil depth': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/Models/Baie Missiquoi/Water yield/Root_UTM.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}, u'Soil erodibility': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/Models/Baie Missiquoi/SDR/Erodibility_clip_UTM.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}, u'On-pixel retention': {u'bins': {u'value_field': u'Sed_Ret', u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/Slope_specific/LULC_slope_regroup_forest_urban.tif', u'uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/Slope_specific/Biophysical_coefficients_byslope.csv', u'key_field': u'lulc_general'}}, u'Beneficiaries': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/Population_density_UTM_clip.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}, u'On-pixel source': {u'bins': {u'value_field': u'Sed_Exp', u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/Slope_specific/LULC_slope_regroup_forest_urban.tif', u'uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/Slope_specific/Biophysical_coefficients_byslope.csv', u'key_field': u'lulc_general'}}, u'Rainfall erosivity': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/Models/Baie Missiquoi/SDR/GlobalR_UTM.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}, u'Downslope retention index': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/preprocessing5_ned30/Output/erosion_downslope_retention_index.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}, u'Upslope source': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/preprocessing5_ned30/Output/erosion_upslope_source.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}, u'Riparian continuity': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/preprocessing5_ned30/Output/erosion_riparian_index.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}}}}

    open_html_on_completion True

    priorities {u'revegetation_assisted': {u'erosion_drinking_control': 0.8}, u'agricultural_vegetation_managment': {u'erosion_drinking_control': 1.0}, u'fertilizer_management': {u'erosion_drinking_control': 0.0}, u'keep_native_vegetation': {u'erosion_drinking_control': 0.0}, u'ditching': {u'erosion_drinking_control': 0.0}, u'revegetation_unassisted': {u'erosion_drinking_control': 0.0}, u'pasture_management': {u'erosion_drinking_control': 0.0}}

    results_suffix _test1

    transition_map {u'revegetation_assisted': {u'Culture_Couverture': 0.0, u'Bandes_5m': 1.0, u'Drainage_controllee': 0.0, u'Bande_10m': 1.0}, u'agricultural_vegetation_managment': {u'Culture_Couverture': 1.0, u'Bandes_5m': 0.0, u'Drainage_controllee': 0.0, u'Bande_10m': 0.0}, u'fertilizer_management': {u'Culture_Couverture': 0.0, u'Bandes_5m': 0.0, u'Drainage_controllee': 1.0, u'Bande_10m': 0.0}, u'keep_native_vegetation': {u'Culture_Couverture': 0.0, u'Bandes_5m': 0.0, u'Drainage_controllee': 0.0, u'Bande_10m': 0.0}, u'ditching': {u'Culture_Couverture': 0.0, u'Bandes_5m': 0.0, u'Drainage_controllee': 0.0, u'Bande_10m': 0.0}, u'revegetation_unassisted': {u'Culture_Couverture': 0.0, u'Bandes_5m': 0.0, u'Drainage_controllee': 0.0, u'Bande_10m': 0.0}, u'pasture_management': {u'Culture_Couverture': 0.0, u'Bandes_5m': 0.0, u'Drainage_controllee': 0.0, u'Bande_10m': 0.0}}

    transition_types [{u'file_name': u'keep_native_vegetation', u'transition_type': u'protection', u'id': u'trans1', u'label': u'Keep native \nvegetation'}, {u'file_name': u'revegetation_unassisted', u'transition_type': u'restoration', u'id': u'trans2', u'label': u'Revegetation \n(unassisted)'}, {u'file_name': u'revegetation_assisted', u'transition_type': u'restoration', u'id': u'trans3', u'label': u'Revegetation \n(assisted)'}, {u'file_name': u'agricultural_vegetation_managment', u'transition_type': u'agriculture', u'id': u'trans4', u'label': u'Agricultural \nvegetation \nmanagement'}, {u'file_name': u'ditching', u'transition_type': u'agriculture', u'id': u'trans5', u'label': u'Ditching'}, {u'file_name': u'fertilizer_management', u'transition_type': u'agriculture', u'id': u'trans6', u'label': u'Fertilizer \nmanagement'}, {u'file_name': u'pasture_management', u'transition_type': u'agriculture', u'id': u'trans7', u'label': u'Pasture \nmanagement'}]

    workspace_dir C:\Users\Labo_JD\Documents\Sylvia\RIOS\Slope_specific



    12/04/2017 10:47:35 root INFO Logging will be saved to natcap.rios-log-2017-12-04--10_47_35.txt

    12/04/2017 10:47:35 root DEBUG Loaded the model from natcap.rios.rios

    12/04/2017 10:47:35 root INFO Executing the loaded model

    12/04/2017 10:47:35 root INFO Python architecture: ('32bit', 'WindowsPE')

    12/04/2017 10:47:35 root INFO Disk space remaining for workspace: 994.26 GB

    an exception encountered when logging

    12/04/2017 10:47:35 root INFO Pointing temporary directory at the workspace at C:\Users\Labo_JD\Documents\Sylvia\RIOS\Slope_specific

    12/04/2017 10:47:35 root INFO Updating os.environ["TMP"]=C:\Users\Labo_JD\Documents\Sylvia\RIOS\Slope_specific\tmp to C:\Users\Labo_JD\Documents\Sylvia\RIOS\Slope_specific

    12/04/2017 10:47:35 root INFO Updating os.environ["TEMP"]=C:\Users\Labo_JD\Documents\Sylvia\RIOS\Slope_specific\tmp to C:\Users\Labo_JD\Documents\Sylvia\RIOS\Slope_specific

    12/04/2017 10:47:35 root INFO Updating os.environ["TMPDIR"]=C:\Users\Labo_JD\Documents\Sylvia\RIOS\Slope_specific\tmp to C:\Users\Labo_JD\Documents\Sylvia\RIOS\Slope_specific

    12/04/2017 10:47:35 root INFO Setting tempfile.tempdir to C:\Users\Labo_JD\Documents\Sylvia\RIOS\Slope_specific\tmp

    12/04/2017 10:47:35 root INFO Starting natcap.rios

    12/04/2017 10:47:35 natcap.rios.ipa INFO ensuring that the optional objectives have non-empty factors

    12/04/2017 10:47:35 natcap.rios.ipa DEBUG {u'erosion_drinking_control': {u'rios_model_type': u'rios_tier_0', u'priorities': {u'revegetation_assisted': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': u'~0.5', u'Beneficiaries': 1.0, u'On-pixel source': 0.25, u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}, u'agricultural_vegetation_managment': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': u'~0.5', u'Beneficiaries': 1.0, u'On-pixel source': 0.25, u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}, u'fertilizer_management': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': u'~0.5', u'Beneficiaries': 1.0, u'On-pixel source': 0.25, u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}, u'keep_native_vegetation': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': 0.5, u'Beneficiaries': 1.0, u'On-pixel source': u'~0.25', u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}, u'ditching': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': u'~0.5', u'Beneficiaries': 1.0, u'On-pixel source': 0.25, u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}, u'revegetation_unassisted': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': u'~0.5', u'Beneficiaries': 1.0, u'On-pixel source': 0.25, u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}, u'pasture_management': {u'Soil depth': 0.25, u'Soil erodibility': 0.25, u'On-pixel retention': u'~0.5', u'Beneficiaries': 1.0, u'On-pixel source': 0.25, u'Rainfall erosivity': 0.25, u'Downslope retention index': u'~1', u'Riparian continuity': 0.5, u'Upslope source': 1.0}}, u'factors': {u'Soil depth': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/Models/Baie Missiquoi/Water yield/Root_UTM.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}, u'Soil erodibility': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/Models/Baie Missiquoi/SDR/Erodibility_clip_UTM.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}, u'On-pixel retention': {u'bins': {u'value_field': u'Sed_Ret', u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/Slope_specific/LULC_slope_regroup_forest_urban.tif', u'uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/Slope_specific/Biophysical_coefficients_byslope.csv', u'key_field': u'lulc_general'}}, u'Beneficiaries': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/Population_density_UTM_clip.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}, u'On-pixel source': {u'bins': {u'value_field': u'Sed_Exp', u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/Slope_specific/LULC_slope_regroup_forest_urban.tif', u'uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/Slope_specific/Biophysical_coefficients_byslope.csv', u'key_field': u'lulc_general'}}, u'Rainfall erosivity': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/Models/Baie Missiquoi/SDR/GlobalR_UTM.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}, u'Downslope retention index': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/preprocessing5_ned30/Output/erosion_downslope_retention_index.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}, u'Upslope source': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/preprocessing5_ned30/Output/erosion_upslope_source.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}, u'Riparian continuity': {u'raster_uri': u'C:/Users/Labo_JD/Documents/Sylvia/RIOS/preprocessing5_ned30/Output/erosion_riparian_index.tif', u'bins': {'inverted': False, 'type': 'interpolated', 'interpolation': 'linear'}}}}}

    12/04/2017 10:47:35 natcap.rios.ipa INFO Updating references to the general LULC coefficients table

    12/04/2017 10:47:35 natcap.rios.ipa INFO Starting Water Funds Prioritizer

    12/04/2017 10:47:35 natcap.rios.ipa INFO Looping through objectives to sort and prioritize

    12/04/2017 10:47:35 natcap.rios.ipa INFO We're doing normalization because of interpolated

    12/04/2017 10:47:37 natcap.rios.ipa INFO We're doing normalization because of interpolated

    12/04/2017 10:47:38 natcap.rios.ipa DEBUG it was defined, must need a LULC

    12/04/2017 10:47:38 natcap.rios.ipa DEBUG {0.0: 0.0, 5120.0: 0.81, 2180.0: 0.84, 2181.0: 0.84, 15110.0: 0.83, 10120.0: 0.81, 5131.0: 0.83, 5134.0: 0.84, 15120.0: 0.81, 10131.0: 0.83, 20.0: 0.2, 5141.0: 0.83, 10134.0: 0.84, 20120.0: 0.81, 15131.0: 0.83, 10140.0: 0.84, 10141.0: 0.83, 15134.0: 0.84, 341.0: 0.2, 10160.0: 0.84, 5147.0: 0.84, 15140.0: 0.84, 5030.0: 0.24, 5160.0: 0.84, 220.0: 0.75, 15147.0: 0.84, 20140.0: 0.84, 10158.0: 0.84, 15030.0: 0.24, 5168.0: 0.84, 20168.0: 0.84, 50.0: 0.505, 20147.0: 0.84, 15158.0: 0.84, 10168.0: 0.84, 10030.0: 0.24, 5180.0: 0.84, 5181.0: 0.84, 20030.0: 0.24, 20158.0: 0.84, 15168.0: 0.84, 10180.0: 0.84, 10181.0: 0.84, 2120.0: 0.81, 15180.0: 0.84, 2030.0: 0.24, 210.0: 0.68, 2131.0: 0.83, 20180.0: 0.84, 20110.0: 0.83, 2134.0: 0.84, 10147.0: 0.84, 2140.0: 0.84, 2141.0: 0.83, 351.0: 0.83, 2147.0: 0.84, 5158.0: 0.84, 230.0: 0.73, 1000.0: 0.94, 2153.0: 0.84, 2158.0: 0.84, 20134.0: 0.84, 2160.0: 0.84, 302.0: 0.2, 2110.0: 0.83, 5110.0: 0.83, 2168.0: 0.84, 5140.0: 0.84, 250.0: 0.73, 20131.0: 0.83, 10110.0: 0.83, 255.0: 0.75}

    12/04/2017 10:47:38 root ERROR ---------------------------------------------------

    12/04/2017 10:47:38 root ERROR ---------------------- ERROR ----------------------

    12/04/2017 10:47:38 root ERROR ---------------------------------------------------

    12/04/2017 10:47:38 root ERROR Error: exception found while running natcap.rios

    12/04/2017 10:47:38 root DEBUG

    12/04/2017 10:47:38 root DEBUG Build details

    12/04/2017 10:47:38 root DEBUG Interpreter

    12/04/2017 10:47:38 root DEBUG Current temp dir: C:\Users\Labo_JD\Documents\Sylvia\RIOS\Slope_specific\tmp

    12/04/2017 10:47:38 root DEBUG tempfile.tempdir: C:\Users\Labo_JD\Documents\Sylvia\RIOS\Slope_specific\tmp

    12/04/2017 10:47:38 root DEBUG

    12/04/2017 10:47:38 root DEBUG System

    12/04/2017 10:47:38 root DEBUG OS : Windows-8-6.2.9200

    12/04/2017 10:47:38 root DEBUG Processor architecture: AMD64

    12/04/2017 10:47:38 root DEBUG FS encoding : mbcs

    12/04/2017 10:47:38 root DEBUG Preferred encoding: cp1252

    12/04/2017 10:47:38 root DEBUG

    12/04/2017 10:47:38 root DEBUG Python

    12/04/2017 10:47:38 root DEBUG Version : 2.7.9

    12/04/2017 10:47:38 root DEBUG Build : ('default', 'Dec 10 2014 12:24:55')

    12/04/2017 10:47:38 root DEBUG Compiler : MSC v.1500 32 bit (Intel)

    12/04/2017 10:47:38 root DEBUG Implementation : CPython

    12/04/2017 10:47:38 root DEBUG Architecture : 32bit

    12/04/2017 10:47:38 root DEBUG Linkage format : WindowsPE

    12/04/2017 10:47:38 root DEBUG

    12/04/2017 10:47:38 root DEBUG Packages

    12/04/2017 10:47:38 root DEBUG Cython : ?

    12/04/2017 10:47:38 root DEBUG Numpy : 1.10.2rc1

    12/04/2017 10:47:38 root DEBUG Scipy : 0.16.0b2

    12/04/2017 10:47:38 root DEBUG OSGEO : 1.11.3

    12/04/2017 10:47:38 root DEBUG Shapely : 1.5.5

    12/04/2017 10:47:38 root DEBUG

    12/04/2017 10:47:38 root DEBUG

    12/04/2017 10:47:38 root DEBUG Exception not environment-related

    12/04/2017 10:47:38 root DEBUG Printing traceback

    Traceback (most recent call last):

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.rios\build\rios\out00-PYZ.pyz\natcap.rios.rui.executor", line 611, in runModel

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.rios\build\rios\out00-PYZ.pyz\natcap.rios.rios", line 24, in execute

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.rios\build\rios\out00-PYZ.pyz\natcap.rios.rios", line 174, in execute_30

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.rios\build\rios\out00-PYZ.pyz\natcap.rios.rios", line 1227, in _normalize_rasters

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.rios\build\rios\out00-PYZ.pyz\natcap.rios.rios", line 1374, in _map_raster_to_table

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.rios\build\rios\out00-PYZ.pyz\pygeoprocessing.geoprocessing", line 95, in get_nodata_from_uri

    AttributeError: 'NoneType' object has no attribute 'GetRasterBand'

    None


  • swolnyswolny Member, NatCap Staff
    Hi Sylvie -

    When I've gotten that error with any of the NatCap tools in the past, it's often been because I've run out of disk space. But it looks like you have almost 1TB to work with, so that isn't the problem. Given the log output, one thing I would check is your LULC raster and table. Maybe I'm misreading the logfile, but it looks like your lucodes in the coefficient table might be floating point instead of integer? If so, try changing them to integers. Or, is it possible that you have an lucode of 255? 255 can also be a NoData value, and there have been issues around this in other models.

    Otherwise, I'm not sure what to advise. Rich might have a better idea of what's happening, but if it's not obvious to him, I can take a look at your input data. 

    ~ Stacie
  • Thank you very much Stacie, I do have a class called 255 which I'll try changing!

    Sylvie
  • Hey Stacie,

    Sorry to bother again. But I had a quick question about the pre-set transition types (i.e Keep Vegetation, Unassisted Revegetation, Assisted Vegetation, Agricultural Mgt Practices, Ditching..etc). When RIOS is calculating the effectiveness of activities assigned to these transitions, does the algorithm treat them differently depending on whether an activity is classed uner Unassisted or Assisted Revegation etc? 

    Is there a complete description of these transition types and how they are used in the computations? In the guide book I found one passage where it mentions that the rank order would be reversed between Keep Native Vegetation to that of Revegetation or Agricultural Vegetation Management. However I can't find any more detail about how these transition types influence the calculation. 

    I am curious as I would like to include two riparian buffer widths and wondered if they could be classed under to different transition types so that I can specify different weights? The same goes for two activities under agricultural management. Otherwise how would I specify that the two activities under a single class contribute differently to the objectives?

    Thanks!
    Sylvie


  • shaedenshaeden Member
    Good Day

    I am having difficulties with running the RIOS preprocessor in ArcGIS. I have attempted to undertake the preprocessing for multiple objectives and the script runs to a point and freezes. I have cancelled this and then tried performing it for just one objective to see if that makes any change. However, a different error comes up as displayed below. Please can you assist

    Executing: RIOS-Pre-Processing true # # # # E:\Shaeden "F:\Shaeden_Post Doc\RIOS_Input Data sets\RIOS_simulations\uMngeni_Simulations\Input Data\Land Cover\lc_utm" "F:\Shaeden_Post Doc\RIOS_Input Data sets\RIOS_simulations\uMngeni_Simulations\Input Data\RIOS_lulc_biophysical_table_uMngeni_nutrients.csv" "F:\Shaeden_Post Doc\RIOS_Input Data sets\RIOS_simulations\uMngeni_Simulations\Input Data\Elevation\dem_utm" "F:\Shaeden_Post Doc\RIOS_Input Data sets\RIOS_simulations\uMngeni_Simulations\Input Data\Global Rainfall Erosivity\eros_utm" "F:\Shaeden_Post Doc\RIOS_Input Data sets\RIOS_simulations\uMngeni_Simulations\Input Data\Soils\erod_utm" "F:\Shaeden_Post Doc\RIOS_Input Data sets\RIOS_simulations\uMngeni_Simulations\Input Data\Soils\depth_utm" # # # # 1000 15 "F:\Shaeden_Post Doc\RIOS_Input Data sets\RIOS_simulations\uMngeni_Simulations\Input Data\mngeni_utm.shp" #
    Start Time: Tue Mar 27 13:33:05 2018
    Running script RIOS-Pre-Processing...

    Validating arguments...

    Erosion Control selected, checking required inputs:
    DEM
    Erosivity
    Erodibility
    Soil depth
    Land use/land cover
    RIOS general coefficient table
    Threshold flow accumulation
    Riparian buffer distance

    Checking input raster projections...

    Error processing hydrology layers: ERROR 010429: Error in GRID IO: WindowSetLyr: Window cell size does not match layer cell size. name: e:\shaeden\intermediate\ras035, adepth: 32, type: 1, iomode: 6, version: 3, compression: 0, eval case: 1 .
    ERROR 010244: Could not set the analysis window.
    ERROR 010067: Error in executing grid expression.
    Failed to execute (FlowDirection).


    Error running script
    Failed script RIOS-Pre-Processing...

    Traceback (most recent call last):
      File "F:\Shaeden_Post Doc\RIOS_Input Data sets\RIOS_simulations\uMngeni_Simulations\Input Data\RIOS_Pre_Processing_ArcGIS_10_4_1\RIOS_Pre_Processing.py", line 1280, in <module>
        raise Exception
    Exception

    Failed to execute (RIOS-Pre-Processing).
    Failed at Tue Mar 27 13:33:05 2018 (Elapsed Time: 0,54 seconds)

  • swolnyswolny Member, NatCap Staff
    Hi @shaeden -

    That's an unusual error to get. Looking at the online help for this error message, it indicates that there is no extent or cell size set for a layer (probably the DEM, since it's trying to calculate flow direction), or the extent doesn't overlap with other inputs. So the first thing to do is make sure that all of your spatial inputs are in the same projected coordinate system. That way they should have valid cell sizes, extents, and overlap each other.

    Also, just checking, are your rasters in ESRI GRID format? It looks like it from the input list. That's fine in general, but TIFFs are often easier to work with (and you can give them much longer file names!)

    ~ Stacie
  • shaedenshaeden Member
    Hi Stacie

    All the inputs are in the same projected coordinate system and are in a tiff format. I ran the processor again and no longer receive the same error but it does not run beyond the point of determining flow direction channels. I am not sure what the issue could be, I have tried using the sample data provided with RIOS in the processor as well and run into similar difficulties.

    I will try to manually derive the requisite inputs for RIOS now but any assistance on this issue would be much appreciated for future work.  

    Kind Regards

    Shaeden
  • swolnyswolny Member, NatCap Staff
    Hi Shaeden -

    Would you please post the latest set of error messages, related to flow direction channels? Thanks.

    ~ Stacie
  • shaedenshaeden Member
    Hi Stacie

    Thank you for the quick response. Please see attached the log of the run for processing the erosion control objective. Some outputs are created such within the intermediate and output folders such as "lulc_coeff, flow acc, flow dir, slope etc. However during the "defining of flow channel directions", the processor does not move pass "updating flow accumulation".

    I have also attached two zip files containing the input data that I am using to run the pre-processor, in the event that there may be an error within these files that I am unaware off.

    Thank you once again for your assistance

    Kind Regards

    Shaeden
  • shaedenshaeden Member
    This is the land cover tiff raster along with the requisite files to open it, I had to attach these as two separate zip folders due to their size. My apologies for any inconvenience that this may cause.
  • swolnyswolny Member, NatCap Staff
    edited March 31
    Hi @shaden -

    Thanks for sending your inputs, and sorry about the file limit size. You can always put them on Dropbox or Google Drive or anywhere else that's convenient and accessible if you need to provide data in the future.

    I've run into this problem before, and have never found out why it happens occasionally. Arc just refuses to work properly sometimes, it's very frustrating. The way I've worked around this before is to hack the script, so it runs to the point where it hangs, then I do manually the command that hangs (because the exact same FlowLength command works fine manually), then let the script do the rest. Which is not at all optimal, of course.

    If you haven't already manually derived the inputs, I'll try to do the same workaround process on your inputs. This doesn't help with future work, but at least it will get you going now and hopefully the next set of data that you try will work. 

    ~ Stacie
    Post edited by swolny on
  • swolnyswolny Member, NatCap Staff
    Just in case it's useful, I've run your inputs through the preprocessor, working around the issue. You can download the outputs here. Sorry for the trouble.

    ~ Stacie

  • shaedenshaeden Member
    Hi Stacie

    I am still attempting to manually derive the necessary inputs, although I am quite familiar with ArcGIS, this task is still proving to be a steep learning curve. Thank you for your assistance and providing these inputs this is extremely helpful. 

    Now that you were able to derive these inputs for the erosion control objective, if I were to run the processor for the other objectives, is there a chance I would run into a similar situation as described previously?
  • swolnyswolny Member, NatCap Staff
    I'm afraid that you're likely to have the same problem with the other objectives too. The issue is something mysterious with Flow Length and Flow Accumulation, where they go off into LaLaLand in the script, but the exact same command and inputs work fine outside of the script. I've spent a lot of time trying to figure it out, haven't, and unfortunately it's not on the top of our priority list.

    If you do any Python coding, here is the same 10.4.1 geoprocessing script, but with the dysfunctional things commented out and (in the Erosion section of code) notes about how to work around it. You could do this workaround instead of doing everything by hand. I can also provide more detailed notes, if those in the script don't make sense.

    ~ Stacie
  • shaedenshaeden Member
    Thank you once again for all your assistance, I will give it a try manually and if I have no luck I will get assistance with running the processor in python.

    On a separate note, out of curiosity, is there plans to add the pre-processor to RIOS to circumvent the use of a GIS software, if so when will this be available?
  • swolnyswolny Member, NatCap Staff
    Personally, I would love it if the pre-processor was added to RIOS. Originally that was the plan, but it turned out to be harder than expected to re-create all of the functionality that's in the Arc version (since our software team basically re-writes all of those geoprocessing tools from scratch), and given other priorities it just hasn't happened. Since an increasing number of people have been using RIOS, perhaps it's time to revisit whether it can be moved up on the To Do list.

    ~ Stacie

  • shaedenshaeden Member
    Hi Stacie

    I have been experiencing similar issues when manually using the flow accumulation function to generate the upstream source index, I suspect that because the flow direction grid and the weighted sediment export raster are expressed at different spatial resolutions ArcGIS is getting stuck in an infinite loop during the generation of flow accumulation.

    I have managed to gain access to ArcGIS 10.2 and wanted to give the preprocessor a try here to see if that makes any difference before seeking assistance with the preprocessing in python. Could you please provide the preprocessor for ARCGIS 10.2?

    Kind Regards

    Shaeden
  • swolnyswolny Member, NatCap Staff
    I believe that the preprocessor that's provided on our website is still the version for 10.2. 

    Interesting that flow accumulation isn't working for you outside of the script either. Every time it doesn't work for me within the script, it always has worked fine done manually. Have you tried resampling all of your other inputs to be the same resolution as your DEM?

    ~ Stacie
  • shaedenshaeden Member
    I will give that a try and let you know if I have any success.
  • shaedenshaeden Member
    Hi Stacie

    I am pleased to let you know that by resampling all my input data sets to a 1km spatial resolution, i have managed to get the pre-processor to run in ArcGIS 10.4. I have generated several output including the downlsope retention index. However, my success was short lived and I have run into a new error, please see attached.

    Any idea what this could be in relation to?
  • swolnyswolny Member, NatCap Staff
    Glad to know that resampling helped! Too bad there's another error though. Which version of Arc and the preprocessor are you now using?

    I've never had that error before, and it's surprising. The script is just trying to take (1.0 - soil depth), and it's complaining that the 1.0 doesn't exist. One thing I wonder is if it's related to a difference in using periods versus commas as decimals. I see in the logfile that other decimal values have commas, but I'm using a period in this equation. When we've had similar problems before, we've found it useful for you to try setting your Windows Language setting to English, where you can also specify that decimals are given as periods, not commas. Can you try this and let me know if it changes anything?

    If not, can you send me your inputs and I'll try with the 10.4 version of the preprocessor and see if it happens for me? I'd be surprised, but you never know.

    ~ Stacie

  • shaedenshaeden Member
    Hi Stacie

    I am pleased to inform you that your suggestion has worked and the pre-processor has run successfully for the erosion control objective. I will now attempt running the pre-processor for the remaining objectives and let you know the outcome.

    Kind Regards

    Shaeden
  • shaedenshaeden Member
    Hi Stacie

    It runs fine for all other objectives as well. Thank you once again for all your assistance, if possible please can you provide me with your full name and organizational affiliation so that I can acknowledge the role that you have played in assisting me with my research. 

    Kind Regards

    Shaeden
Sign In or Register to comment.