This forum is shutting down! Please post new discussions at community.naturalcapitalproject.org

Memory error in Coastal Blue Carbon model

How many unique habitat types can the blue carbon model pre-processor handle? My LULC table has 83,700 unique habitat types (all mangrove, marsh, and seagrass with slightly different carbon stock measurements). I get a memory error when I try to run the pre-processor. I'm wondering if there are too many values for the model to handle?

I've attached the log, the LULC input table, and the raster. Thanks for your help!

Comments

  • jdouglassjdouglass Administrator, NatCap Staff
    Hi Monica, I just wrote back to you over email, but I'd like to write back here for posterity as well.  The real challenge for the CBC model is in the mapping of transitions between landcover codes where we end up with n^2 possible transitions, all of which need to be represented in the transient table.

    There's no great solution for this at the moment, since the right way to represent this sort of spatial variation is probably not with tables anyways.  A cheapo workaround for this might be to slice up the landcover raster into more manageable chunks (I bet tiles of 256x256 would probably do the trick).

    The other thing to consider is whether there are any transitions at all!  From the logfile, I see that the same landcover raster is represented twice in the snapshot list.  If an evaluation of the current carbon stocks is all you need, then the analysis would be much, much simpler.
  • MonMon Member
    Hi James,
    Thanks for your explanation of the memory error. This was our test run with no transitions, but we plan on using transitions in other scenarios. I'll try the slicing trick and see if that makes it easier to handle.
  • MonMon Member
    Hi James,

    I did some rounding and was able to reduce the number of unique habitat values to ~350 without slicing them up. The pre-preprocessor ran smoothly, and the core model was able to show the green success box, but the carbon stock raster outputs don't seem to have the correct values. They range between -Inf and 0. The rest of the raster outputs appear to be reasonable numbers so I'm not sure why the carbon stock ones are being problematic.
  • jdouglassjdouglass Administrator, NatCap Staff
    Hi Monica,

    I'm so sorry about the delay on this!

    So, the main issue here is that I was completely wrong about how the model indexes its landcover codes, and the values of the lulc-class columns *do* indeed need to be unique.  The carbon stock rasters were the clearest numerical consequence of this and almost all of the landcover codes identified by the model were defaulting to -inf because it couldn't find the correct carbon types during reclassification.

    Hope this helps, and I'm so sorry for my previous misinformation!
    James
  • MonMon Member
    Hi James,

    I made unique names for the LULC class columns, and I think I'm getting closer. All the output rasters have real positive numbers as values now, but something is still not right with the carbon storage rasters. The carbon storage raster for the start year in the model has a maximum value (1930.5) that is less than the maximum value of C storage values in the C Initial table (2010.4), which makes me think something is off.
  • MonMon Member
    Hi James,

    I took a closer look at the carbon storage output rasters, and it turns out the visualization method in ArcGIS was just not showing the maximum values of the rasters, causing us to interpret them as artificially low. The rasters are correct and the model is working. Thanks for all your help! 

    Monica
  • MonMon Member
    Hi James,

    I had one more question, more general this time. We're trying to estimate value of carbon sequestration at all of the snapshot and transition years, not just the final analysis year. In the past, I've run the model multiple times and switched out the end years to accomplish this. This took a lot of time, so I was wondering if there's a more efficient way of doing it. Is it possible to get the net present value rasters from the snapshot and transition years? Are they calculated during the modeling run and stored somewhere separately from the outputs?

    Monica
  • jdouglassjdouglass Administrator, NatCap Staff
    Hi Monica,

    NPV is indeed calculated for each timestep throughout the model, but it isn't written out anywhere until the final analysis year.  I don't see an immediate way to calculate these values from the files that are written out to disk, but it also doesn't seem like it'd be that challenging to add to the model.

    So I understand what you're looking to do, if we have a CBC run with a starting year of 2010, transition years at 2020 and 2030 and an analysis year at 2040, do you currently re-run the model a total of 40 times to get NPV calculations for 2011, 2012 .... 2039, 2040 in order to use the NPV raster from each run?
  • MonMon Member
    Hi James,

    I haven't been re-running it every single year, but have been doing it at every transition year and the final analysis year. So in the example that would be 2020, 2030, and 2040. We only had a few transition years in my last project, so re-running the model was manageable though not ideal. However, for my current project we're aiming to create some maps of gradual change due to sea level rise at intervals of 5-10 years apart and model blue carbon for those, which will increase the number of transition years and time points where we're interested in NPV. 
  • MonMon Member
    Hi James,

    I wanted to follow up on the NPV rasters for transition years in addition to the final analysis years. Do you think it would be feasible to add that to the model?
  • jdouglassjdouglass Administrator, NatCap Staff
    Hey Monica, sorry about the delay here!  This should be feasible ... we're already calculating these valuations, so makes sense to write them out partway and is minor to implement.  I've added this feature in the development build below ... could you take a look and check that it's doing what you would expect?

    Also, it's worth noting that the NPV rasters sum everything up to and including the target year.  So, the NPV raster for, say 2050, includes the year 2050.  Let me know if should only include 2049!  It's an easy change to implement.

    James

  • MonMon Member
    Hi James, including the target year is fine. The net present value layers from the start year, transition years, and final analysis years are included. There's one raster that's titled "net_present_value" with no year in the name. The values are the same as the values for the final year. It matches, but it's a bit confusing.

    One other thing I notice from the output for this build is that almost every raster has -Inf as the low value when run with certain inputs. Based on the locations of the -Inf cells, it seems like -Inf is in the position of NoData cells, but the Properties say the NoData value is -16777216 for all output rasters. All the actual data values seem to be in the right spots though. I tried it with a different set of inputs and everything seems fine with the output, so maybe it's something with the input data, but I'm not sure. The aligned LULC rasters that went into these model runs came from the preprocessor fix you sent earlier today.
  • jdouglassjdouglass Administrator, NatCap Staff
    Hi Monica, thanks for the feedback!  I've got an in-progress dev version of InVEST that removes the 'net_present_value.tif' raster in favor of named rasters at the start year, transition year and final analysis year (if provided).

    The -inf values sound suspicious.  Internally, the model does some masking using NaN in place of nodata, and it's quite possible that something is not being un-masked/remasked before being written out.  Would you be willing to send me your inputs where you can see these values?  Or does this happen with the rasters you linked to earlier in this thread?  The sample data doesn't seem to reproduce this issue.

    Thanks!
    James
  • MonMon Member
    Hi James,

    Here are my inputs. The pre-processor works fine and the aligned rasters look normal, but the end outputs of the Erosion scenario have -Inf in some of the rasters. I've also attached the inputs for the other two scenarios, Restoration and No Net Change, which have normal outputs. The start year rasters (2020) for each scenario are supposed to be identical, though since that was the raster that was creating trouble before you gave me that new pre-processor build, I'm not sure if something happened between the times I copied and pasted it into the different folders for the respective scenarios. Still, it comes out of the pre-processor for each scenario looking identical in everything except for file size.
  • jdouglassjdouglass Administrator, NatCap Staff
    Thanks for attaching your inputs, Monica, and I'm very sorry for the delay!  I'm hoping to be able to take a look at these very soon.

    James
  • jdouglassjdouglass Administrator, NatCap Staff
    Hi Monica,

    Thanks again for including your inputs, and I'm so sorry about the delay on this!  The issue came down to how the CBC model was handling masking of nodata values in its internal reclassification, an issue that will arise when there is a mismatch between the nodata value defined in the lulc and the range of values allowed by the lulc's datatype.  I've added a few lines to the model to better handle this, and I'm no longer seeing -inf in the output rasters.


    Thanks, and again, I'm so sorry about the delay!
    James
Sign In or Register to comment.