This forum is shutting down! Please post new discussions at community.naturalcapitalproject.org

# Water scarcity model in Water yield model did not run, and different outcomes for Mac&Windows

Member
Dear InVEST developers,

I am trying to run a water yield model, particularly taking into account water demand by high pop density urban areas. When i ran the model in my mac, the output indicated the model had run successfully, but when I checked the output raster for wyield, i realised that it did not reflect/ take into account the demand by urban areas, and some agricultural areas (for example, my urban areas had the highest water yield compared to an area of evergreen forest). When i went to check the log, it mentioned that my data sets for the watershed vector and the tables (biophysical table and demand table) was not recognised as a supported file format. I have attached the entire message from my mac below.

When i tried running the water yield model with the same input files in Windows 10, I got a different output message which gives ValueError encountered: The rasters' and vectors' intersection is empty (not all rasters and vectors touch each other).
I have ensured that all my rasters and vector files are in the same projection (Projected coordinate system: WGS_1984_World_Mercator) and I am using the most updated InVEST version (InVEST 3.5) on both the mac and Windows. The entire message from my Windows is also attached below. Could you kindly advise me on what I can do, or if I am missing anything?

FROM MAC:

p, li { white-space: pre-wrap; }

Showing messages with level INFO and higher

10/04/2018 11:03:26 natcap.invest.utils INFO Writing log messages to /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/output_4Oct/InVEST-Hydropower-Water-Yield-log-2018-10-04--11_03_26.txt

10/04/2018 11:03:26 natcap.invest.ui.model Level 100 Starting model with parameters:

Arguments for InVEST natcap.invest.hydropower.hydropower_water_yield 3.4.2:

biophysical_table_uri /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/Biophysical_table_2010_2.csv

calculate_valuation True

calculate_water_scarcity True

demand_table_uri /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/Demandtable_2010_2.csv

depth_to_root_rest_layer_uri /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/Abs_depth_mm_pcs.tif

eto_uri /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/ET0_test1.tif

lulc_uri /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/lutest6.tif

pawc_uri /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/PAWC_test1.tif

precipitation_uri /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/Prec_test1.tif

results_suffix

seasonality_constant 22

sub_watersheds_uri /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/watershed_test4_q.shp

valuation_table_uri /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/Hydropower_Valuation_table_2010.csv

watersheds_uri /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/watershed_test4_q.shp

workspace_dir /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/output_4Oct

10/04/2018 11:03:26 osgeo ERROR [errno 4] /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/Demandtable_2010_2.csv' not recognised as a supported file format.

10/04/2018 11:03:26 natcap.invest.hydropower.hydropower_water_yield INFO Starting Water Yield Core Calculations

10/04/2018 11:03:26 osgeo ERROR [errno 4] /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/Hydropower_Valuation_table_2010.csv' not recognised as a supported file format.

10/04/2018 11:03:26 osgeo ERROR [errno 4] /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/watershed_test4_q.shp' not recognised as a supported file format.

10/04/2018 11:03:26 osgeo ERROR [errno 4] /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/Biophysical_table_2010_2.csv' not recognised as a supported file format.

10/04/2018 11:03:26 osgeo ERROR [errno 4] /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/output_4Oct' not recognised as a supported file format.

10/04/2018 11:03:26 osgeo ERROR [errno 4] /Users/Theo/Desktop/my stuff/NUS SCHOOL /FYP/Theo_InVEST_test4/watershed_test4_q.shp' not recognised as a supported file format.

10/04/2018 11:03:26 natcap.invest.ui.usage._log_model WARNING an exception encountered when logging URLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),)

10/04/2018 11:03:26 natcap.invest.hydropower.hydropower_water_yield INFO Reclassifying temp_Kc raster

10/04/2018 11:03:26 natcap.invest.hydropower.hydropower_water_yield INFO Reclassifying tmp_root raster

10/04/2018 11:03:26 natcap.invest.hydropower.hydropower_water_yield INFO Reclassifying tmp_veg raster

10/04/2018 11:03:30 natcap.invest.ui.model INFO Execution finished

10/04/2018 11:03:30 natcap.invest.utils INFO Elapsed time: 4.66s

10/04/2018 11:03:30 natcap.invest.ui.execution INFO Execution finished

10/04/2018 11:03:30 natcap.invest.ui.usage._log_exit_status WARNING an exception encountered when _log_exit_status <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

From WINDOWS:

Showing messages with level INFO and higher

10/04/2018 13:11:22 natcap.invest.utils INFO Writing log messages to C:\Users\E0008806\Desktop\Lee Hui Yian Theodora\Theo_InVEST_test4\InVEST-Hydropower-Water-Yield-log-2018-10-04--13_11_22.txt

10/04/2018 13:11:22 natcap.invest.ui.model Level 100 Starting model with parameters:

Arguments for InVEST natcap.invest.hydropower.hydropower_water_yield 3.5.0:

biophysical_table_uri C:/Users/E0008806/Desktop/Lee Hui Yian Theodora/Theo_InVEST_test4/Biophysical_table_2010_2.csv

calculate_valuation False

calculate_water_scarcity True

demand_table_uri C:/Users/E0008806/Desktop/Lee Hui Yian Theodora/Theo_InVEST_test4/Demandtable_2010.csv

depth_to_root_rest_layer_uri C:/Users/E0008806/Desktop/Lee Hui Yian Theodora/Theo_InVEST_test4/Abs_depth_mm.tif

eto_uri C:/Users/E0008806/Desktop/Lee Hui Yian Theodora/Theo_InVEST_test4/ET0_test1.tif

lulc_uri C:/Users/E0008806/Desktop/Lee Hui Yian Theodora/Theo_InVEST_test4/lutest6.tif

pawc_uri C:/Users/E0008806/Desktop/Lee Hui Yian Theodora/Theo_InVEST_test4/PAWC_test1.tif

precipitation_uri C:/Users/E0008806/Desktop/Lee Hui Yian Theodora/Theo_InVEST_test4/Prec_test1.tif

results_suffix

seasonality_constant 22

sub_watersheds_uri

watersheds_uri C:/Users/E0008806/Desktop/Lee Hui Yian Theodora/Theo_InVEST_test4/watershed_CPCStest4.shp

(the watershed file is different for this output message because i tried clipping the watershed boundary to the other raster's extent, but i got the same error message anyway)

workspace_dir C:\Users\E0008806\Desktop\Lee Hui Yian Theodora\Theo_InVEST_test4

10/04/2018 13:11:22 natcap.invest.hydropower.hydropower_water_yield INFO Validating arguments

10/04/2018 13:11:22 osgeo ERROR [errno 4] Unable to open C:\Users\E0008806\Desktop\Lee Hui Yian Theodora\Theo_InVEST_test4\watershed_4.shx or C:\Users\E0008806\Desktop\Lee Hui Yian Theodora\Theo_InVEST_test4\watershed_4.SHX.Try --config SHAPE_RESTORE_SHX true to restore or create it

10/04/2018 13:11:22 osgeo ERROR [errno 4] Failed to open file C:\Users\E0008806\Desktop\Lee Hui Yian Theodora\Theo_InVEST_test4\watershed_4.shp.It may be corrupt or read-only file accessed in update mode.

10/04/2018 13:11:22 natcap.invest.ui.model ERROR Exception while executing <function execute at 0x1161D4F0>

Traceback (most recent call last):

File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\env\lib\site-packages\natcap\invest\ui\model.py", line 1541, in _logged_target

File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\env\lib\site-packages\natcap\invest\hydropower\hydropower_water_yield.py", line 137, in execute

File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\env\lib\site-packages\pygeoprocessing\geoprocessing.py", line 606, in align_and_resize_raster_stack

ValueError: The rasters' and vectors' intersection is empty (not all rasters and vectors touch each other).

10/04/2018 13:11:22 natcap.invest.ui.model INFO Execution finished

10/04/2018 13:11:22 natcap.invest.utils INFO Elapsed time: 0.12s

10/04/2018 13:11:22 natcap.invest.ui.execution ERROR Target <function _logged_target at 0x117DED30> failed with exception

Traceback (most recent call last):

File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\env\lib\site-packages\natcap\invest\ui\execution.py", line 68, in run

File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\env\lib\site-packages\natcap\invest\ui\model.py", line 1541, in _logged_target

File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\env\lib\site-packages\natcap\invest\hydropower\hydropower_water_yield.py", line 137, in execute

File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\env\lib\site-packages\pygeoprocessing\geoprocessing.py", line 606, in align_and_resize_raster_stack

ValueError: The rasters' and vectors' intersection is empty (not all rasters and vectors touch each other).

10/04/2018 13:11:22 natcap.invest.ui.execution INFO Execution finished

Lastly, I would like to ask if the watershed vector must be the exact same extent as the other rasters? This is because my study area is a state, but the watershed boundaries for 1-2 watersheds overlaps into the neighbouring state. In this case would you advise to or against clipping the watershed boundary to the other raster's extent?

Thank you so much for taking the time to read my message! I truly appreciate it if you can kindly share some of your advice!

Have a blessed day!

Warmest regards,

Theo

• Member, NatCap Staff
Hi Theo -

When you're running on Mac, are you certain that the water demand isn't being taken into account? Urban areas will often have highest runoff, even with water demand, if it's mostly impermeable surface. Related, forest often evapotranspires more than other land cover types, so you're likely to see lower runoff there. Check your Kc values to see what the difference is between these land cover types. And look at the watershed results shapefile to see if consum_* and rsupply_* values are given per watershed.

You don't need to worry about the osgeo errors about a file not being recognised as a supported file format, that doesn't impact the results.

The error on Windows says,
osgeo ERROR [errno 4] Unable to open C:\Users\E0008806\Desktop\Lee Hui Yian Theodora\Theo_InVEST_test4\watershed_4.shx or C:\Users\E0008806\Desktop\Lee Hui Yian Theodora\Theo_InVEST_test4\watershed_4.SHX.Try --config SHAPE_RESTORE_SHX true to restore or create it
10/04/2018 13:11:22 osgeo ERROR [errno 4] Failed to open file C:\Users\E0008806\Desktop\Lee Hui Yian Theodora\Theo_InVEST_test4\watershed_4.shp.It may be corrupt or read-only file accessed in update mode.

I'm not sure what's happening here, but are you sure that the shapefile isn't corrupt, is accessible, has all of the necessary files, is not being used by another process, etc? You could try exporting it to a new shapefile and see if that helps.

No, the vector does not need to be exactly the same as the rasters. Often we recommend making your rasters a bit bigger than the vector so you don't lose data at the edges. For water models, we also recommend using the whole watershed, so you have a complete picture of the hydrology, although for Annual Water Yield it's not as important, since water is not routed around the landscape.

~ Stacie
• Member
Hi Stacie!

Thank you so much for your reply!

I went to check the output from my mac, and there were no values for consum_* and rsupply_* values given in the watershed results shapefile, only precip_mn, PET_mn, AET_mn, wyield_mn, num_pixels, and wyield_vol.

For Windows, I tried exporting the watershed vector file as a new shapefile but I got the same error (I've made sure that they are in the same projection). One possible problem I can think of at the moment was because I transferred the shapefile from Qgis in my mac, to ArcGis in the windows - I'm not sure if that might have sparked the corrupt file issue, but i did the same for the other rasters (moving back and forth qgis and arcgis via tiffs) and those don't have an issue.

Thank you once again! I'll continue to try and see what I can do for both mac and windows!

Warm regards,
Theo
• Member
edited October 2018
Post edited by E0008806 on
• Administrator, NatCap Staff
Hi Theodora, I don't know what's going on with the virus stuff. But I notice you're running a different version of InVEST on your Mac machine (3.4.2 vs. 3.5.0 on Windows). 3.4.2 did have a bug where the water yield scarcity model was broken, but we fixed it on 3.5.0. So hopefully that's helpful on that point.
• Member
Hi Rich!

Thank you for replying!

I've updated InVEST in my mac to 3.5.0 and it worked! Thank you so much!

Warmest regards,
Theo
• Member
Hi Rich,
I'm having the same problem with McAfee detecting InVEST as a virus and deleting it.  I had the 3.6.0 version installed.   I'll see if I can figure out how to tell McAfee to make an exception but in the meantime just wanted to ask if there is any possibility that the program has been infected.

Thanks,
Neile
• Administrator, NatCap Staff
Hi @NeileK, it's unlikely (though I suppose it's possible) that InVEST has been infected with malware.  Historically, a couple of InVEST users have had issues with McAffee and Symantec quarantining the InVEST application.  Until we cryptographically sign our binaries, there's not a whole lot to do about this other than to set an exception for your antivirus software.

To the best of my knowledge, other antivirus tools (including sophos and windows defender with updated definitions) have not identified any malware in InVEST, so I'm hoping that this is just a one-off issue with McAffee.