This forum is shutting down! Please post new discussions at community.naturalcapitalproject.org

delineate it watersheds

Hello,

I am trying to use DelineateIT to create watersheds for many pour-points that represent groups of buildings at flood risk. When I do so, the model appears to work well for many pour-points, but for others the polygon delineated is the opposite of what I would expect: it is the entire extent of the layer that is NOT a part of the watershed (I have attached a screen shot of an example, the watershed polygon is green and partially transparent)

Has anyone else had this happen, or have any ideas about what might be causing this to occur?

Thank you,
Keri
635 x 824 - 65K

Comments

  • swolnyswolny Member, NatCap Staff
    Yes, Keri, I've seen that. Happens to me every time, actually. You end up getting one polygon that defines the watershed, and another that defines what's *not* the watershed, for every watershed.

    What I've done in the past is just select out the not-watersheds and delete them. You can do this manually if there are a few watersheds, as the watershed and not-watershed are grouped together in the shapefile. If there are a lot, I've done this (in ArcGIS, should be able to do this in QGIS too): make a buffer around the whole area of interest and use Select by Location on the DelineateIt output shapefile to select the polygons that are inside the buffered AOI polygon. Export the selection to a new file and you'll have just the actual watersheds.

    ~ Stacie
  • RichRich Administrator, NatCap Staff
    Hi Keri, would you be willing to dropbox me your data?  (richsharp@stanford.edu) That's something I should fix.
  • Hi Rich,

    I am happy to dropbox my data to you, but the files are a bit ridiculous (Im working across VT at 30m resolution). I'll clip my data to a smaller extent and get those to you soon, sorry Ive been away from the computer with these datasets for a bit. 

    Keri
  • RichRich Administrator, NatCap Staff
    (I posted this to the wrong thread earlier... reposting here)

    No need Keri!  We patched that issue today.  Here's a development build of InVEST you can use that only generates the real watershed polygons if you need to get to it right now: http://data.naturalcapitalproject.org/nightly-build/invest-forks/richpsharp/InVEST_richpsharp3.3.2.post61+n72e8e5a02b2d_x86_Setup.exe

    Also in debugging this issue we found a selection of polygons by "MOD("FID",2)=1" and delete was sufficient for cleaning up the output.

    Please let me know if something it doesn't work there or I can help further.
  • Hi,

    I am trying to run DelineateIt again, for a very similar problem as described above, but it is consistently crashing at some point in the process of working through the flow direction/accumulation. Any ideas why this may be the case? I have attached an example log, but this is not always the exact point where it stops working.

    Thanks!
    Keri
  • RichRich Administrator, NatCap Staff
    Sorry I'm not sure Keri, hopefully not a memory error... Can you dropbox me your dataset and I'll see if I can recreate the issue on my end? richsharp@stanford.edu
  • Hi Rich,

    Thank you! I actually tried to run it again and the model made it a lot further, but a ran into an error that actually resulted in an error message. I've attached the log here, and am dropboxing you my datasets now.

    Thanks again!
    Keri
  • here is the dropbox link. They are in the delineateIt folder. 


    Thanks!
    Keri
  • swolnyswolny Member, NatCap Staff
    Hi Keri -

    This doesn't answer your question, but one thing I see is that you're trying to delineate over 6000 points at a time. When I had to do 30,000 watersheds once, I did the processing in batches of no more than around 3000-4000, as I tended to run into trouble with much more than that. In this case, I see that DelineateIt isn't making it too far into your points before it stops, but just to be aware that you might want to break the point dataset into two and merge the results at the end.

    Also, I just looked at the log you posted, and don't see an error message, it just ends with a line about processing one of the outlet points. Could you check that out and post the logfile with the whole error message? Thanks.

    ~ Stacie
  • Hi Stacie and Rich,

    Cutting my dataset in half worked like a charm! Thank you. I always have a hard time knowing when I am asking my computer to do something ridiculous. 

    Thanks again,
    Keri
  • actually I am getting empty datasets, I was not looking closely enough. I get the error: Encountered a point that is outside of the dem" This seems like a projection issue, but both of my datasets are in NAD_1983_StatePlane_Vermont_FIPS_4400_Feet. 
  • RichRich Administrator, NatCap Staff
    Hi Keri, I'm running through your data now, but I notice your points are projected in the GRS80 datum, not NAD83.  I'm not a projection expert, but might explain if some points are a little outside the DEM?

    Hopefully more soon as this crashes or completes.
  • RichRich Administrator, NatCap Staff
    Hi Keri, I ran through your dataset without a problem.  The resulting watersheds look super cool btw.  And here's the dropbox link if all you care about are the results: https://www.dropbox.com/sh/cmx1wqpgars2rmg/AAANrzXFuv0r2yndkoyrOvO7a?dl=0

    Otherwise I'm not sure what the issue could be on your end.  The log you posted doesn't have an error in it, just seems to terminate early.  Just want to check that the model wasn't still running?  Also for what it's worth I see you're running a development version of InVEST (3.3.2postxxx).  I don't think we changed anything from that version to the 3.3.3 release, but you could also try that if you're needing to try another run for some reason.  You can download it here: http://data.naturalcapitalproject.org/invest-releases/3.3.3/InVEST_3.3.3_x86_Setup.exe


  • Hello, I am trying to delineate watersheds using Delineate tool, but it results in error:

    Initializing...

    03/08/2017 22:25:56 root DEBUG Loading <module 'natcap.invest.routing.delineateit' from 'C:\INVEST~1.2_X\INVEST~1\natcap\invest\routing\delineateit.pyc'> in frozen environment

    Arguments:

    dem_uri D:/final/demutmwgs.tif.ovr

    flow_threshold 1000

    outlet_shapefile_uri D:/final/Export_Output.shp

    snap_distance 10

    workspace_dir D:\final



    03/08/2017 22:25:56 root INFO Logging will be saved to natcap.invest.routing.delineateit-log-2017-03-08--22_25_56.txt

    03/08/2017 22:25:56 root DEBUG Loaded the model from natcap.invest.routing.delineateit

    03/08/2017 22:25:56 root INFO Executing the loaded model

    03/08/2017 22:25:56 root INFO Running InVEST version "3.3.2"

    03/08/2017 22:25:56 root INFO Python architecture: ('32bit', 'WindowsPE')

    03/08/2017 22:25:56 root INFO Disk space remaining for workspace: 258.76 GB

    03/08/2017 22:25:56 root INFO Pointing temporary directory at the workspace at D:\final

    03/08/2017 22:25:56 root INFO Updating os.environ["TMP"]=D:\final\tmp to D:\final

    03/08/2017 22:25:56 root INFO Updating os.environ["TEMP"]=D:\final\tmp to D:\final

    03/08/2017 22:25:56 root INFO Updating os.environ["TMPDIR"]=D:\final\tmp to D:\final

    03/08/2017 22:25:56 root INFO Setting tempfile.tempdir to D:\final\tmp

    03/08/2017 22:25:56 root INFO Starting natcap.invest.routing.delineateit

    03/08/2017 22:25:56 natcap.invest.routing.delineateit INFO creating directory D:\final

    03/08/2017 22:25:56 root ERROR ---------------------------------------------------

    03/08/2017 22:25:56 root ERROR ---------------------- ERROR ----------------------

    03/08/2017 22:25:56 root ERROR ---------------------------------------------------

    03/08/2017 22:25:56 root ERROR Error: exception found while running natcap.invest.routing.delineateit

    03/08/2017 22:25:56 root DEBUG

    03/08/2017 22:25:56 root DEBUG Build details

    03/08/2017 22:25:56 root DEBUG Interpreter

    03/08/2017 22:25:56 root DEBUG Current temp dir: D:\final\tmp

    03/08/2017 22:25:56 root DEBUG tempfile.tempdir: D:\final\tmp

    03/08/2017 22:25:56 root DEBUG

    03/08/2017 22:25:56 root DEBUG System

    03/08/2017 22:25:56 root DEBUG OS : Windows-7-6.1.7601-SP1

    03/08/2017 22:25:56 root DEBUG Processor architecture: AMD64

    03/08/2017 22:25:56 root DEBUG FS encoding : mbcs

    03/08/2017 22:25:56 root DEBUG Preferred encoding: cp949

    03/08/2017 22:25:56 root DEBUG

    03/08/2017 22:25:56 root DEBUG Python

    03/08/2017 22:25:56 root DEBUG Version : 2.7.9

    03/08/2017 22:25:56 root DEBUG Build : ('default', 'Dec 10 2014 12:24:55')

    03/08/2017 22:25:56 root DEBUG Compiler : MSC v.1500 32 bit (Intel)

    03/08/2017 22:25:56 root DEBUG Implementation : CPython

    03/08/2017 22:25:56 root DEBUG Architecture : 32bit

    03/08/2017 22:25:56 root DEBUG Linkage format : WindowsPE

    03/08/2017 22:25:56 root DEBUG

    03/08/2017 22:25:56 root DEBUG Packages

    03/08/2017 22:25:56 root DEBUG Cython : ?

    03/08/2017 22:25:56 root DEBUG Numpy : 1.11.2rc1

    03/08/2017 22:25:56 root DEBUG Scipy : 0.16.1

    03/08/2017 22:25:56 root DEBUG OSGEO : 1.11.3

    03/08/2017 22:25:56 root DEBUG Shapely : 1.5.5

    03/08/2017 22:25:56 root DEBUG InVEST : 3.3.2

    03/08/2017 22:25:56 root DEBUG

    03/08/2017 22:25:56 root DEBUG

    03/08/2017 22:25:56 root DEBUG Exception not environment-related

    03/08/2017 22:25:56 root DEBUG Printing traceback

    Traceback (most recent call last):

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\natcap.invest.iui.executor", line 560, in runModel

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\natcap.invest.routing.delineateit", line 75, in execute

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\pygeoprocessing.routing.routing", line 401, in delineate_watershed

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\pygeoprocessing.geoprocessing", line 3258, in tile_dataset_uri

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\pygeoprocessing.geoprocessing", line 2305, in vectorize_datasets

    File "C:\Users\natcap-servers\jenkins-home\workspace\natcap.invest\label\GCE-windows-1\exe\build\invest\out00-PYZ.pyz\pygeoprocessing.geoprocessing", line 2057, in align_dataset_list

    Exception: The datasets' intersection is empty (i.e., not all the datasets touch each other).



    03/08/2017 22:25:56 root ERROR Exiting due to failures



    What might be the reason?


    Here is the dropbox link: https://www.dropbox.com/sh/a5afx46cs8xjozi/AAD1kvyl6Ifc04r6HpMnSvDBa?dl=0 ;


    Thank you !

  • swolnyswolny Member, NatCap Staff
    Hi Gohalya -

    I'm looking at your inputs, and am a little confused by the metadata. On one hand it indicates that the layers have a projected coordinate system of WGS_1984_UTM_Zone_46N. On the other, it says that the cell size of the DEM is 0.0083333333, which would mean that each cell is only .0083 meters x .0083 meters in size, which would be an extremely high resolution DEM, and would likely produce a DEM file size much larger than the 8MB that it is. 

    By any chance, did your original files start out in a geographic coordinate system (where .0083 of a degree is a more likely value)? I'm thinking that instead of "reprojecting" the geographic coordinate system to a projected one, something like "set projection" was used instead. If you use a reproject tool (like Project Raster in ArcGIS, or Warp (Reproject) in QGIS), the coordinate system will be transformed correctly from geographic to projected. But if you just manually change the coordinate system, through, say, editing the raster's Properties in ArcGIS, you aren't actually reprojecting, and the result could be this kind of strange mix of values, which is hard for the tool to interpret properly. 

    Check this out and let us know what you find.

    ~ Stacie
  • Hello, Stacie.

    Yes, you were right about the projection, previously I have used ArcGIS Define Projection, probably it doesn't actually reproject the data, but "set projection" as You've mentioned (I wasn't able to run ArcGIS Project tool, it said "One or more dropped items were invalid and will not be added to the control”). But  QGIS worked well for reprojecting. 

    So I succeeded in completing Delineate and Route DEM models, but resulted Watershed map is rather a failure, there is one small watershed outside the area of my interest (it's very upstream of the river basin).

    Could You please recommend me to solve it?



    Thank You! 
  • swolnyswolny Member, NatCap Staff
    I'm looking at your DelineateIt results, and they actually look mostly good to me. There is a nice-looking watershed created for each point, which you can see by looking at the attribute table of the watersheds.shp, or re-symbolizing your output so there is no fill color, only outline color.

    However, you're also running into a known bug in the tool, where two polygons are created for each watershed - one that is the watershed itself, and one that contains all of the area outside of the watershed. Which version of InVEST are you using? If it's not the latest (3.3.3), try downloading and using 3.3.3 and see what happens. If you are using 3.3.3, then we've still got a bug. But it's easy enough to work around, I've described how in this thread.

    ~ Stacie

  • RichRich Administrator, NatCap Staff
    P.S., we DID fix that bug in InVEST 3.3.3 Stacie!  
  • Hello, Stacie and Rich! 

    Yes, I got much better results in InVEST 3.3.3 ! Thank You so much for your help :) 

    Kind regards,
    Gauhar
  • Hi Rich,

    Thank you very much! I am not sure what I am running into on my end because I've been running the tool pretty frequently and successfully, but luckily for this particular project I can take the output and run with it. 

    Keri


  • RichRich Administrator, NatCap Staff
    Great, I love a bargain basement solution to a technical problem.  Ask away if we can help further.
Sign In or Register to comment.