This forum is shutting down! Please post new discussions at community.naturalcapitalproject.org

Habitat Quality Overflow Error

dgrafiusdgrafius Member
edited July 2014 in Terrestrial Models
I'm trying to run the standalone Biodiversity/Habitat Quality model, and keep encountering an error; "OverflowError: cannot fit 'long' into an index-sized integer." I keep thinking all my input data have been formatted correctly but the error seems to suggest that somewhere, something is the wrong data type. I have four threat rasters that are the same extent and only have 0 or 1 values, and I can't see how the problem would be with my LULC raster since I've already used it without difficulty in the Carbon Storage model. I'll attach my most recent run log to this post in case it's of any help.

I'd appreciate any insight anyone might have about where there problem might be occurring, since the log doesn't do much to point me at which file is causing the problem. I unfortunately do not have much experience with Python coding, so there might be clues I'm missing.

Thanks much!
Darren

Comments

  • DougDoug Administrator, NatCap Staff
    Hi Darren,

    Our lead engineer is currently working on an upgraded algorithm for that functionality. This is good news for the next release and the long term.

    For the short term, hopefully we can come up with a workaround or some guidance.
    We initially believe it to be a casting issue in the function, but we're going to take a closer look.

    Can you tell me about how big your input datasets are? The threat rasters, LULC, etc... ?

    Thanks!

    Doug
  • Hi Doug, thanks a lot for the response. The analysis I'm doing is a general, overall habitat quality/disturbance risk study, so I've got four threats; major roads (~1GB), minor roads (~1GB), railways (~500MB), and buildings (~4GB). I'm running it all on a 25m resolution LULC map covering an area roughly 30km by 30km (~90KB, 8 classes). I've also got a more detailed 0.5m resolution LULC raster that I'll run the model on later once I sort this issue out to compare scales (hence the higher resolution of my threat maps), but for now I'm just trying to get it working with the smaller dataset.

    I'm attaching a .zip file with my threat and habitat sensitivity tables, the 25m LULC raster, and the railway raster to act as an example for my threat maps, which all should have the same format and extent.

    I've already used the LULC map successfully (as well as the higher resolution one I mentioned) in the Carbon Sequestration model, so it doesn't seem like that should be causing the problem if it is something with my data. As I said though, I'm not the best with coding so it's hard to tell if it's one of my data formats causing the problem or something with the code somewhere.

    It is perhaps also worth noting that I first tried running this with InVEST 2.6.0 and got the error, then when I saw 3.0.1 was out I upgraded and still get the same error.

    Let me know if I can provide any additional information that might help!

    Thanks,
    Darren
  • DougDoug Administrator, NatCap Staff
    Hi Darren,

    Is it possible you could try running the model one threat at a time. That is, for your threats.csv file, start with only:

    THREAT,MAX_DIST,WEIGHT
    bldg,1,1

    Then add back
    THREAT,MAX_DIST,WEIGHT
    bldg,1,1
    majrd,5,1

    Be curious to see if it happens with just one threat and if so, it'd be great to get a zipped up file of your inputs either on dropbox or some other way with however many of the threat rasters are needed to reproduce.

    Thanks,
    Doug
  • Good thought. I'll give that a go next chance I get (probably Monday) and report back!
  • Right, I've given it a try with just a single threat input and I still get the same error. I'm attaching a zip file containing all the inputs; LULC map, major road threat raster, habitat and threat input tables referencing just that raster, and the run log. Should be everything you (or anyone else) needs to run it and presumably get the same error I've been getting. Although if you don't I'll be rather curious!

    Darren
  • DougDoug Administrator, NatCap Staff
    Hi Darren,

    I just did some debugging with my supervisor and we think that your current issue is related to your threat raster. The raster is quite large in that it has a large row x col dimension which unfortunately causes an issue in Python with the way we currently handle those files.

    As I mentioned above, my supervisor is working on a fix for this functionality and will be available in the next release.

    A possible work around is to sample down that threat raster, if possible.

    Sorry for the unfortunate news, hopefully we can get that release and fix out sooner than later.

    Doug
  • Doug,

    That appears to have done it! Thanks so much for your help, I don't think that's the sort of thing I would've ever figured out on my own. I resampled the threat rasters to 50m resolution, which for the sort of work I'm doing here isn't really a problem since I'm using several models and getting a very general overview rather than needing a really exact picture at this point. I imagine if I need to get more precise, one option could be slicing up my study area into smaller regions and running the model separately on each.

    Again I really appreciate all your help!

    Cheers,
    Darren
This discussion has been closed.