Shape matching of raster files

Hi everyone,

I have a question related to the shapes of my GIS data. My goal is to quantify ecosystem services (Carbon, SDR, NDR, and Water Yield models) in a watershed, but as the InVEST manual suggests (page 140), some data, such as DEM, must extend beyond the watershed of interest, rather than being clipped to the watershed boundaries. I'm clipping my raster files to match with counties boundaries, which are beyond the watershed I'm working on. Is this a correct approach? 
Please find attached a figure where you can see the counties boundaries and the watershed boundaries.

Thank you!
Fabio
510 x 711 - 13K

Comments

  • swolnyswolny Member, NatCap Staff
    Hi Fabio -

    A few thoughts about this. For the hydrology models, it's important to include the whole watershed in the analysis, otherwise you won't be modelling all of the landscape that's contributing to your point of interest, and there will likely be places where the hydrology is cut off artificially (along a country border, for example), leading to inaccurate results. This is not necessary for the Carbon model, since there is no flow path or other sort of interdependence across the area of interest.

    Some of the inputs to the model might have high resolution (often land use), and others low resolution (often climate.) For those with low resolution, i.e. large cell size, you will want to clip them beyond the boundary of the watershed, so that the whole watershed is covered in data. If you clip rasters of large cell size to watershed boundaries, you generally end up with big holes of NoData around the edge of the watershed, which produces NoData in the model results. 

    If I'm reading your figure correctly, the watershed is in pink, within the county boundaries. I'm not sure of the scale, but I would say that if clipping to county boundaries allows all of your input data to completely cover your watershed, then it's fine to do that. Otherwise, what I usually do is create a buffer around the watershed of 1km or whatever size is useful, given the resolution of your inputs, and clip my inputs to that buffered watershed.

    ~ Stacie


  • Stacie,

    Thank you very much for this insight. It certainly helps! 
    I'm making sure that all my watershed is covered by data. However, I keep getting this error message saying "Python int too large to convert to C long" (NDR Model). I've learned from other discussions on this forum that the reason for this error may be a no data value. 

    Therefore, I have a follow up question: Should ALL my raster files be clipped to the exact same shape? For example my precipitation file is clipped to the counties boundaries and my DEM file is a rectangle bigger than the counties boundaries. Is this a problem? 

    Please find attached a figure showing my watershed, a precipitation layer (counties boundaries) and the DEM, which is bigger than the other layers (LULC is the same shape of precipitation and it's not visible on the figure).

    Thank you very much,
    Fabio

    495 x 741 - 175K
  • swolnyswolny Member, NatCap Staff
    Hi Fabio -

    Your raster files do not need to be clipped to the exact same shape, that shouldn't be causing your error.

    I think I'll punt addressing the specific error that you're getting to our software team. 

    ~ Stacie


  • Stacie,

    Thank you for your help.
    I'll keep trying and see if I can properly run the model.

    Best,
    Fabio
  • RichRich Administrator, NatCap Staff
    Hi Fabio, could you Dropbox me your datastack to richsharp@gmail.com and I can take a look!
  • Hi Rich,

    I'll give a last try to run the model tomorrow in my office, if I can't run the model again, I'll send you the files. I just want to avoid you to waste time on this if the problem is too primary.

    I very appreciate your attention.
    Best,
    Fabio
  • Rich,

    I just shared my files with you.

    Thank you again,
    Fabio
  • RichRich Administrator, NatCap Staff
    Oh @fabiosecanho, I'm sorry, my dropbox account is richsharp@stanford.edu (not @gmail.com). Would you mind resending?
  • Hi Rich, thank you very much for your attention. 

    I just shared my data with you.
    I could figure out the error by fixing my LULC raster file. However, the output of my NDR model for Nitrogen looks weird. There's some straight lines on the north section of the watershed that don't look "natural". Do you know why I'm getting those lines? (I'm not running for Phosphorus yet).
    I've attached my NDR output that I generated using the files I shared with you.

    Thanks!! 
    571 x 863 - 65K
    501 x 729 - 107K
  • RichRich Administrator, NatCap Staff
    Hi Fabio, I should have asked first what version of InVEST you're using. I just ran your data through the development version of InVEST on my machine with no problems. Would you be willing to download this version and try it for yourself? http://data.naturalcapitalproject.org/invest-releases/3.4.0rc1.post112+nc9ba4d091970/InVEST_3.4.0rc1.post112+nc9ba4d091970_x86_Setup.exe

    And please post back here if it's not working for you.
  • Hi Rich,

    I was running InVEST 3.3.3, but now I installed this version you suggested (3.4). I like the results better on this version. However, I still have the same problem, these straight lines. I'm attaching some screenshots that show these lines very clear.

    Thank you!
    521 x 804 - 336K
    1023 x 853 - 887K
    978 x 814 - 281K
  • RichRich Administrator, NatCap Staff
    Ahh I see. Sorry I misunderstood at first.

    Those long diagonal jaggies are caused by the underlying routing mechanism. When I look at your input DEM that entire area seems to be at a height of 255 meters. Is that what you expect? If so, that's just a giant flat plane and the only thing the routing algorithm can do is find the shortest distance to the first downhill pixel. In your case that's a long way to travel and you get a long diagonal line. Does that make sense?

    Offhand, I'd guess your DEM got clamped somehow in your preprocessing or wherever you got it from in the first place. 255 would be exactly the cutoff for a raster with 8 bit pixels even though the data type of your dem now is a 16 bit int.

    Does that make sense?
  • Rich,

    No, this is not what I expect for sure.  And yes, it makes sense! Thanks for the explanation. However, I checked some random points on this raster files and I didn't get 255 for all of them, as you can see on the figures I've attached.
    Are you looking at "pixel value"?
    Should I download the original DEM file again and extract again?
    Which source would you suggest to me to download this data?

    Thanks!
    1014 x 575 - 252K
    731 x 750 - 113K
  • Rich,

    I see what you meant now. Only that area on top has the same height of 255 meters. That's true.
    I'll try to fix that!!

    Thank you
  • RichRich Administrator, NatCap Staff
    Yep that's what I meant. I'm so vested at this point, if you figure out what happened, please let me know!
  • Rich,

    Thanks for the help. Although I couldn't fix this problem yet, at least I know what is the problem! 
    I have six DEM quadrants that cover my study area, and their data are correct. However, when I use "mosaic to new raster" (ArcMap), my final raster loses data, as you identified. I think I'll try to download DEM data from somewhere else and try again (my data is from NRCS).
    Please find attached a figure with the six original DEM, and the new raster.

    Thank you!!
    606 x 853 - 649K
    597 x 844 - 465K
  • swolnyswolny Member, NatCap Staff
    When you mosaic the DEM tiles, what are you choosing for the "Pixel type" input to the tool? By default, it's set to "8 bit unsigned" (with maximum value of 255), but you can change it to something like "16 bit signed" (in case you have negative elevation values.) I've used this tool a lot and never lost data, but I always change the Pixel type to 16 bit.

    Also a reminder that if your next step is to reproject the mosaicked DEM, be sure to choose "Bilinear" or "Cubic" as your Resampling Technique, or you'll probably end up with a checkerboard pattern when you run hydrology tools on the resulting raster.

    ~ Stacie
  • Stacie,

    Thank you for your help.

    I got a better DEM map now using your suggestions. However, I'm still not satisfied with the final output yet. I think there are too many "NoData" points on some places that shouldn't be any. I was investigating the reason for that, and I believe that my DEM is still defective. I looked at the intermediate outputs, specifically to "stream", and I think InVEST is not modelling them properly. I've attached a figure showing the output of my NDR, where all of this dark green areas are assigned zero, and the white areas have "NoData". This big white area almost matches with a reservoir on the area, as we can see on the land cover figure. But some of these "NoData" areas lies on top of agriculture, forest or other lands that should have some value assigned. 

    Looking at the DEM figure, I can't see anything wrong, but the reason I believe there must be something wrong is because of the intermediate output "stream". The lines are not continuous, and they don't match with the actual streams on this area. 

    I compared this "stream" output to the one from template, and the template one has continuous lines.

    Any clue?

    Thank you very much
    1101 x 873 - 232K
    1102 x 876 - 259K
    1079 x 873 - 273K
    1096 x 875 - 12K
  • RichRich Administrator, NatCap Staff
    Hi @fabiosecanho, those outputs look like a classic case of a DEM that has hydrological pits. I use the wang and Liu algorithm in SAGA GIS but I've heard Archydro has an equivalent. Can you try filling those pits and giving it one more try?
  • Rich, @Rich 

    Thank you for your suggestion. I used ArcHydro to fill in the sinks and my results are much better. There are only two questions left:

    1) As you can see on my NDR output, there are some areas with NoData (white pixels). However, these areas coincide exactly with the intermediate output "stream". In other words, these white pixels on the output are the streams generated by InVEST. Is that normal?

    2) I still have some straight lines on my output, even after fill in the sinks. However, these straight lines lie on top of a reservoir (all straight lines lie on top of water). Is that normal as well?

    Other than that, the output looks great! I'm getting there!

    Thank you!    
    1091 x 869 - 633K
    1100 x 867 - 43K
    1102 x 874 - 345K
  • swolnyswolny Member, NatCap Staff
    Yes, it's normal for there to be NoData values along the stream network. The models route nutrient until it reaches the stream, then stops, there is no in-stream processing modeled.

    And it's also usual for the streams to show up as straight lines across water bodies. It's similar to your earlier issue with the areas that were all flat with values of 255 - where places are flat, the flow direction is ambiguous, in reservoirs too.

    ~ Stacie
  • Stacie,

    This is what I thought for both questions. Well, looks like my model is ready. I'll start working on my other models and post again when I have questions.
    I very appreciate your help! Both of you guys!!!

    Thank you!!!!
Sign In or Register to comment.