Recreation/Visitation Model errors

Hi All,
I'm running into a set of errors when Itry to run the recreation model. I have a large, but simple, AOI. I am attempting to do a base run with neither gridding nor a predictor table included yet. The run is set to be solely for the year 2011. I suspect my AOI is too large. I received the following errors in my printout:

...
10/27/2017 06:40:09 osgeo.gdal ERROR [errno 1] C:\Users\Michael\Desktop\FMEC\Recreation\rec11\aoi.shp is not a directory. 
... 
Note: this is the AOI file created by InVEST

...
10/27/2017 06:54:45 root ERROR Error: exception found while running natcap.invest.recreation.recmodel_client
...

...
OSError: [Errno 28] No space left on device
...
This is also the error on the final printout.


Any guidance or suggestions is greatly appreciated! 

Thanks,
Michael



Comments

  • DaveDave Member, NatCap Staff
    Hi Michael,

    This error is suggesting there is not enough space for the results on your local drive. Can you check on the amount of disk space left on your C drive?

    Posting the entire logfile as an attachment might be helpful too. Thanks!
  • Hi Dave,
    Thanks for the quick reply! 

    Attached is an entire log file. I have 1.2 TB available on my computer, so I would guess that is not the problem, but maybe if the AOI is too big? 

  • DaveDave Member, NatCap Staff
    Okay, that's clearly not the problem then!

    Looks like your first instinct was right, that the problem is on our server. @Rich can hopefully help us out!
  • jdouglassjdouglass Administrator, NatCap Staff
    Hey guys,  I'm not 100% sure why it wasn't responding in the first place (and with disk space errors, no less), and it looks like the server has been effectively offline for the last few hours and definitely having some issues.  The recreation server is back online, and should be responding as expected.

    I'm so sorry about the service interruption!
    James
  • I just tried again and it ran successfully. So, thanks! The elapsed time was ~22min. Do you guys have any idea about how bogged down this will get when I introduce a predictor table and try gridding the AOI? It seems like this might bump up the processing time significantly.

    Thanks again,
    Michael
  • DaveDave Member, NatCap Staff
    Thanks James!

    Yes, it will surely take longer with more grid cells and predictors. It's hard to predict, so I would suggest doing some tests with quite large grid cells at first, then reducing the size as needed. What is the extent of your AOI?

    Once you settle on a cell size, you can always use the resulting gridded shapefile as the input AOI for any future model runs. This will let you skip the "grid aoi" option.

    When the model processes predictors it prints out its progress as it's going, so you can see how far along it is.
  • Hi guys,
    Well, my study area is basically the Western conterminous US. As a test over the weekend, I tried running the whole AOI with predictors and the default gridding setting (4km, right?). This ran for 4 hours and then booted me. I'm now working to split up my AOI into more digestible chunks. I was thinking of the following AOIs: OR-WA-ID; CA-NV; UT-AZ-CO-NM; MT-WY. The last two groups also have little pieces of the neighboring states to the east. Do you think that these AOIs are an appropriate size or should I go down to, say, individual states? 

    A related question: Do I need to clip my predictor layers to fit the extent of each AOI? 
  • DaveDave Member, NatCap Staff
    Hey Michael,

    Splitting up the AOI is definitely the way to go, this model just isn't optimized for those continental-scale runs.

    I don't know the details of your application or your research question, but consider if you can prototype the entire analysis from A-Z with just a small portion of your AOI. And then if everything is working and you are getting meaningful results, then tackle the technical challenge of including the entire area. Depending on your research question, a reasonable prototype area may even be smaller than one state.

    The value in the cell size parameter will have units that match the units of your AOI's coordinate system; I'm not sure if there is a default value. Start very large so that you get a complete run in a reasonable amount of time.

    You do not need to clip predictors ahead of time, the model takes care of that, just make sure all your predictor layers use the same coordinate system as your AOI.

    Good luck! Keep the questions coming if you have them!
  • Hi Dave,
    Thanks for the suggestions. This makes a lot sense. When you say, "then tackle the technical challenge of including the entire area" I just want to make sure I know what steps are being recommended here. This is my interpretation: 1) Automate the division of the study area into an appropriately-sized set of AOIs , 2) get one of these AOIs working in InVEST soup-to-nuts, 3) automate a process to run all the AOIs through InVEST one at a time (they may all have the same predictor table and predictor layers, correct?), 4) automate a process to compile all the AOI-specific outputs into a single set of outputs.... Am I interpreting this correctly?

    On a slightly different note, I think I'm confused about what the difference is between manually submitting a large set of AOIs individually like I've described above and gridding the AOI. Is it just the drain on computing power? 

    Michael
  • Hey Guys.
    So I'm a lot closer to having a functional run, My AOI is a handful of counties in Northern Idaho. I ran the model without the regression (and with and without gridding) and it worked fine.

    When I incorporated the predictor table I encountered an error, printout attached. The layer mentioned immediately before the ---ERROR--- notification is a 15m buffer around roads that is a percent-of-polygon variable. This same error appears both with and without AOI gridding.

    Suggestions?

    Thanks, Michael 
  • DaveDave Member, NatCap Staff
    @michael_weinerman, Sorry for the slow response, I was offline for a few days.

    For this latest error with the predictor, please double-check your predictors CSV input table. The model is looking for a column header named 'type', so be sure you have one, all lower-case. You can see an example table in the user's guide here: http://data.naturalcapitalproject.org/nightly-build/invest-users-guide/html/recreation.html#data-needs

    This is a great example of why it's useful to test on a smaller area first, so you can catch errors like this before scaling up.

    One reason to split up the AOI before feeding into the model is because of the way the model queries the global dataset of photos. It first finds photos in the bounding box of the entire AOI, so if that bbox is very large, then subsequent queries by grid cell/polygon will take longer.

    The workflow you outlined sounds right to me. Each individual model run will return the PUD results and process the predictors for that area, as well as run a regression. If you are interested in the regression model results for your entire large AOI (e.g. coefficient values per predictor), then you will need to run that final regression outside of InVEST, with other software, after you combine the PUDs and predictor values from each individual run into one big table.

    On a side-note, it would be interesting to see if the coefficient values for the predictors are relatively stable across the different subregions, or if there are geographic patterns.
  • Ahhhh, yes, I had written "types" instead of type. I thought I had triple-checked all those details....

    Ok, the regressions appear to work mostly fine now. Some of the layers were doing odd things in the regression when gridding was set to 40,000m, but these all seem to have been evened out when I dropped this down to 4000m. I'm sure I'll be coming back with more questions, but I just wanted to write thanks again! 

  • Hey all,
    I realize this question is better suited to a GIS forum, but perhaps somebody has tackled this particular problem in this setting before. I'm having trouble projecting a dem .tif file onto the same coordinate system as the AOI. I've used the same coordinate system under Project Raster, but it's not lining up with the AOI. Any suggestions on how to fix this? Do I just need to clip the dem down to an appropriate size? 

    Michael 
  • swolnyswolny Member, NatCap Staff
    Hi Michael -

    What do you mean exactly by "it's not lining up with the AOI"? Is it in a completely different place, or just shifted slightly? 

    And to "Do I just need to clip the dem down to an appropriate size?" - more detail here would be useful. Is the DEM significantly larger than your AOI? Could it be large enough that it cannot be projected into the same coordinate system? (Like, if it's the whole US in WGS84, you couldn't fit that into UTM Zone 10N.) In these situations, I usually clip the DEM to an area a bit larger than the AOI, do the reprojection, then do the final clip to the AOI. 

    With a little more information, I might be able to throw out more ideas.

    ~ Stacie
  • Hi Stacie,
    Thanks for the reply. The InVEST log (see above attachment) reports that the DEM layer is not the same projection as the AOI. I used the Raster - Project tool to project to the same coordinate system as the AOI and then also reprojected after I clipped the DEM to be slightly larger than the AOI. My AOI is a clump of about 5 counties in Idaho - is this just too large an area to have a raster as a predictor? 

    If I'm understanding correctly, after clipping the DEM to be a little larger than the AOI, then reprojecting, I should then clip the DEM again to the extent of the AOI? I followed this process precisely and I still receive the message that the projection does not match the base vector. In the log file the base vector projection matches that of the DEM, but then some of the details are different (eg, some 'parameter' inputs). Maybe I need to select a different coordinate system? 

    I can run the model with no errors if I omit the DEM layer. 

    Thanks again,
    Michael
  • DaveDave Member, NatCap Staff
    Hi Michael,

    I think you're probably correct that some of the very subtle differences in the specifications of those coordinate systems are causing the problem. Based on the log, it looks like the systems are the same, but specified in a slightly different way for some reason, and maybe fooling the model's function that checks to be sure the data are in the same system before proceeding.

    It might be a pain, but I would try choosing a different system for all the inputs. I know this one to be reliable. http://spatialreference.org/ref/epsg/nad83-utm-zone-11n/

    The process of clipping to the AOI isn't strictly necessary since the model will do that for you, though it's not a bad idea to clip ahead of time if you end up running the model many times with that raster dataset.
  • Hi Dave,
    I reprojected everything to the coordinate system you suggested and it seems to be running fine now, thanks!

    Unrelated to the projection issue, I'm seeing a series of notifications in a row similar to the following, but with different coordinates. Do you know what this means and should I be concerned? 

    11/21/2017 06:44:09 shapely.geos WARNING Ring Self-intersection at or near point -89366.005740922992 4668974.5538578024
  • DaveDave Member, NatCap Staff
    Excellent!

    No need to be concerned with those warnings unless an error pops up. It's telling you that some geometries in the input data are technically invalid according to the GEOS specification. Nevertheless, the data is still usable and results of overlay operations will not be affected (as far as I know).
  • FulviaFulvia Member
    Hello,
    with reference to the last problem raised by michael_weinerman, I first started getting the message:
    07/11/2018 09:15:35 shapely.geos WARNING Ring Self-intersection at or near point 372114.35749999998 4583223.2072999999
    and finally what you see in the attachment.
    What can it be related with?
    Thanks,
    Fulvia
    993 x 966 - 376K
  • DaveDave Member, NatCap Staff
    Hi Fulvia,

    It is safe to ignore those WARNINGS, though in this case it's followed by an error. It looks like there's a problem in the Oak.shp file. You could try opening that file in GIS and using a "repair/check geometry" tool. That may pinpoint specific problems in that layer. It is possible that a single feature/polygon in that file is causing the problem. Dissolving polygons may help, buffering by 0 may also help. Improperly formed data can be quite a pain!
Sign In or Register to comment.