Sediment Static Map Generator

LauraLaura Member
Hello,

I am starting to use OPAL and need some help understanding the inputs. I'm unclear about the relationship between the LULC raster, offset scenario land use/land cover raster, paved landcover code, and bare landcover code.

Are the inputs for the static map generators available? I do not see them with the sample data.

I think that the offset scenario land use/land cover raster is created from the LULC raster, but I am not clear how that should be done. How should I indicate changes? Should I reclass what will change to a new LULC code, replace what will change with no data values, or something else?

Do the paved landcover code and bare landcover code apply to the LULC raster or the offset scenario land use/land cover raster? Should the paved landcover code and bare landcover code be existing values in the raster or should they be new values (integers) that are not in the raster?

Additionally, I am wondering about the runtime of OPAL. In particular I am seeing a big difference between running the InVEST Sediment Delivery Ratio, which takes about 2 hours, and running the OPAL Sediment Static Map Generator, which is taking several days. I am analyzing the Canton of Geneva at 2 meters resolution (15287 by 13777 pixels) with 1,327 subwatersheds. It seems that it is generating the report that is taking so much time. Can you help me understand why that is the case?

Thank you,
Laura

Comments

  • jdouglassjdouglass Administrator, NatCap Staff
    Hi @Laura,

    When using the static map generator, the paved and bare scenarios are generated automatically.  Depending on the options you select, a third static map (either for a restoration or protection scenario) may be created.  Each sediment static map is the marginal value between two runs of the InVEST SDR model.  One of these runs is the baseline (this is your "LULC Raster" input), and the other is a scenario LULC.  In the case of the paved and bare scenarios, OPAL creates the LULC raster for you, by converting all LULC pixels to the bare or paved landcover code.  Restoration and Protection scenarios tend to have more interesting spatial variation of landcover codes, so OPAL requires that as the "Offset Scenario Land Use/Land Cover raster", and also requires that you specify which scenario type it represents (restoration or protection).  That's the relationship between the inputs you mention.

    We don't explicitly distribute sample inputs for the static map generators, but you could make use of the InVEST sample inputs instead, as the OPAL static map generators merely automate a process that calls the appropriate InVEST model.

    Your protection or restoration raster could certainly be derived from your current LULC, and it's up to you to decide what that scenario might look like.  Oftentimes, these scenario rasters are created with the assistance of land-use change modeling tools or scenario generators to help predict what a future scenario might look like.  InVEST has two such scenario generator tools, but there are many out there.

    The paved and bare landcover codes must have entries in your biophysical tables.

    With regard to runtime, calculating all three of these static maps will require 4 runs of SDR, plus some minimal additional time for raster calculations ... I'd expect the calculations alone to take 8-9 hours on your computer based on the 2-hour runtime of SDR that you mentioned.  There's one other thing, however, that will dominate the runtime:  did you happen to check the option to 'calculate generated static map quality?'  If so, the static map generator will run SDR the specified number of times (default is 25) *per watershed* to verify the accuracy of the static map approach on your inputs.  This is effectively the same verification process we went through while preparing the results for the OPAL paper (see below).  So if you have 1,327 watersheds and you're testing the static maps with 25 simulations per watershed, you'll have 33,175 runs of SDR to get through.

    For more detailed information about OPAL, I'd recommend taking a gander at the user's guide (http://data.naturalcapitalproject.org/opal-releases/1.1.0/OPAL_1.1.0_manual_en.pdf) and the paper that introduced the tool (http://www.sciencedirect.com/science/article/pii/S1364815216302110).
  • jdouglassjdouglass Administrator, NatCap Staff
    PS: If the runtime isn't due to the static map testing option I mentioned, could you attach your logfile from the static map generation run to this issue so I can take a look?
  • LauraLaura Member
    edited July 24
    Hi @jdouglass,

    Thank you for your quick and detailed reply. I spent the last week going over your answers, rereading the OPAL user’s guide, the MAFE-T user‘s guide, and the OPAL paper, and trying outing new static generator and OPAL runs on the sample data and my own data. I am having many of the same problems as before, but hopefully I have more information now to help isolate the problem.

    I think I am clear now about the paved and bare landcover codes. I have now added paved and bare landcover codes to my biophysical table and I am using them in the corresponding fields in the sediment static generator.

    I think I am also clear about the Offset Scenario Land Use Land Cover Raster. I am now using a restoration Offset Scenario Type and made the raster by converting all agriculture land in the LULC raster to natural environments, for example crops to grasslands and pasture to brushlands.

    I tried a variety of things, and while in the best case the Sediment Static Map Generator does finish running and produces a sediment_restore_static_map.tif all the values are 0, and both sediment_paved_static_map.tif and sediment_bare_static_map.tif are missing.

    The run I described in my first post did not finish after 3 days, so I canceled it and changed the number impact simulations to 1 and changed the watersheds to a less detailed 22 watersheds for the same extent and resolution, but after 4 days it is similarly waiting on the report to finish generating. The following with some changes in numbers is continuously printed in the command prompt window:

    (2498658.3460000027, 2505689.8227239395, 1118059.4734999954, 1135728.430000007)
    2503194.90208 2505140.15588 1122513.08866 1124458.34246
    False

    In another run I selected 1 watershed from the 22 and resampled the data from 2m to 500m, and while the model finished in 37 seconds it was missing the outputs I mentioned.

    In another run I selected 1 watershed from the 22 and used the original 2m rasters and it completed in 28 minutes, but was missing the outputs I mentioned. Here are the outputs from that run:
    https://drive.google.com/open?id=0BxTXC9ADemLKR2hsWDU5eU1yckk

    In another run I selected 2 watersheds from the 22 and used the original 2m rasters and it completed in 46 minutes, but was missing the outputs I mentioned.

    I have attached the log files from my runs.

    Thank you,
    Laura
    Post edited by Laura on
  • LauraLaura Member
    Here are inputs for the run using 1 of 22 watersheds at 2m:

    https://drive.google.com/open?id=0BxTXC9ADemLKeTBFQ0F0MXhhV1E

    Here are inputs for the run using 22 of 22 watersheds at 2m, which is the complete run I am trying to do now:

    https://drive.google.com/open?id=0BxTXC9ADemLKSm1aQzFFRXZSMm8

    I will no longer try a run with the 1,327 fractional watersheds.

    Thank you!

  • jdouglassjdouglass Administrator, NatCap Staff
    Thanks for your patience with this, Laura.  Would you mind trying out this development build of OPAL and seeing if that at least produces the bare and paved static maps?  I've also taken the liberty of removing all of the extra printing to the console, as this is probably slowing things down a ton.


    Thanks!
  • LauraLaura Member
    Hi James,

    Thank you for the development build of OPAL. I get the error message below when I try to run it.

    C:\Program Files\OPAL_9_1.1.0_e3d381cad5de>"C:\Program Files\OPAL_9_1.1.0_e3d381
    cad5de\run_sediment_sm.bat"

    C:\Program Files\OPAL_9_1.1.0_e3d381cad5de>opal_exe_cli.exe opal_sediment_sm.jso
    n
    Traceback (most recent call last):
      File "<string>", line 8, in <module>
      File "C:\Users\natcap-servers\jenkins-home\workspace\Permitting_win32\src\pyin
    staller\PyInstaller\loader\pyi_importers.py", line 276, in load_module
      File "C:\Users\natcap-servers\jenkins-home\workspace\Permitting_win32\build\ru
    n_opal\out00-PYZ.pyz\natcap.opal", line 7, in <module>
      File "C:\Users\natcap-servers\jenkins-home\workspace\Permitting_win32\src\pyin
    staller\PyInstaller\loader\pyi_importers.py", line 276, in load_module
      File "C:\Users\natcap-servers\jenkins-home\workspace\Permitting_win32\build\ru
    n_opal\out00-PYZ.pyz\natcap.versioner", line 154, in <module>
      File "C:\Users\natcap-servers\jenkins-home\workspace\Permitting_win32\build\ru
    n_opal\out00-PYZ.pyz\natcap.versioner", line 76, in get_version
      File "C:\Users\natcap-servers\jenkins-home\workspace\Permitting_win32\build\ru
    n_opal\out00-PYZ.pyz\natcap.versioner", line 149, in vcs_version
    natcap.versioner.VersionNotFound: A version could not be loaded from scm in C:\P
    rogram Files\OPAL_9_1.1.0_e3d381cad5de�


    Thank you,
    Laura
                                           
  • jdouglassjdouglass Administrator, NatCap Staff
    Hi @Laura, thanks for your patience with this.  I've put together another build that I've verified will launch, and should fixthe issue where the static map generator is skipping the bare and paved scenarios.  Could you try this out and see if it takes care of the issue?


    Thanks,
    James
  • LauraLaura Member
    Hi @jdouglass,

    I was able to run this new development version of OPAL, although it would only work on a computer that already had OPAL 1.1.0 installed. I ran the Sediment Static Map Generator, but I got a strange error message. See the attached screenshots and log file.

    I would prefer to use Sediment Static Map Generator, but I am also alternatively trying to use the Custom Static Map Generator with manual runs of the InVEST SDR model. In that case when I run OPAL I get warnings for invalid polygons, which ArcGIS confirms when I run Check Geometry, but after running Repair Geometry ArcGIS no longer finds invalid polygons, but OPAL still finds invalid polygons. Do you have any advice regarding the polygons? While OPAL is finding invalid polygons it becomes unresponsive and quits. See the attached log.


    Thank you,
    Laura
  • jdouglassjdouglass Administrator, NatCap Staff
    Hi Laura,  thanks for taking a look at the build.  I've fixed the error you captured in the screenshot and assembled another dev build (download link here: http://data.naturalcapitalproject.org/nightly-build/adept/installer/OPAL_30_1.1.0_937dc5f476d5_x86_Setup.exe) that resolves the issue.  I was able to run the sediment static map generator through to completion with the data linked to in a previous post after unchecking the "Calculate Generated Static Map Quality" option.

    It's hard to say exactly what's up with the polygons ... I'd hope that Arc would have fixed them like it claimed to!  Is it possible that the file had been edited by Arc but not saved to disk?  If not, could you link to the vector or send it to jdouglass@stanford.edu so I can take a look?

    Oh, and for the computer that isn't able to run OPAL, could you take a look at this post (https://forums.naturalcapitalproject.org/index.php?p=/discussion/1198/error-on-first-run-of-opal-1-1-0/p1) and see if installing the MSVC redistributable allows OPAL to launch?

    Thanks for your help with this!
  • LauraLaura Member
    Hi @jdouglass,

    Thank you I was able to run the Sediment Static Map Generator and got results. I am not sure about the values, so I will need to take a closer look and think about how to interpret them. OPAL is still crashing after finding invalid polygons. I have attached the log file. Here is a ZIP of the polygons:

    https://drive.google.com/open?id=0BxTXC9ADemLKQjF0V1o3Zzh0U1U

    The computer that could not run the development version of OPAL without OPAL 1.1.0 can run the newest development version perfectly. I always need to do what that post says once on each new computer, but the error message (see the attached file) I got before was about something different. I will let you know if I get that message again.

    Thank you,
    Laura
  • jdouglassjdouglass Administrator, NatCap Staff
    Hi Laura,

    Interesting ... it looks like GDAL (which InVEST uses to handle reading and writing of raster and vector formats) is saying that there are a lot of ring self-intersections in the vector's geometry.  Fortunately, these are pretty easy to fix, and I've gone ahead and included a fix for this in the OPAL source code.  Could you try out this development build of OPAL and see if that causes it to no longer crash?


    I'm glad that there's a workaround for getting OPAL to run on new computers, and feel free to post the message if the other crash does happen.

    James
  • LauraLaura Member

    The new version seems improved in that it is no longer skipping polygons, but it is still becoming nonresponsive and crashing when processing mn_erase_sce_merge_v4.shp. See the attached log file and screenshot.

    Thank you,
    Laura
Sign In or Register to comment.