Seasonal Water Yield error help

Hi - I am using the seasonal water yield model on a fairly large region. After about 2 hours running, I receive a Set Projection, get raster band error. It may be to me running out of disk space but I also see a lot of errors in the log file. I have attached 2 log files with this issue (should be approximately the same). Could you help me figure out why this is failing?

Thank you

Comments

  • swolnyswolny Member, NatCap Staff
    Hi @atutterow -

    Yes, it is due to running out of disk space. You can see a lot of "no space left on device" messages in the log file. These models, especially Seasonal Water Yield, can use a lot of disk space in the intermediate processing, especially over a large region or high-resolution dataset.

    ~ Stacie
  • Thanks, @swolny - I ran it again with > 500 GB of space. It failed again with the same message and I still have room on my drive. Could it possibly take up more space than this? I attached my log file again.
  • swolnyswolny Member, NatCap Staff
    @atutterow, I'm not sure what's going on. From the very beginning it is saying that there's not enough disk space on E:\Tulare2015, which shouldn't be if you have 500GB available. SWY can take up a lot of space, especially if your inputs are large. What is the size of your DEM? Whatever resolution that is, all of the intermediate and final layers will be as well. Still, I doubt that the DEM is that huge. 

    The only other thing I can think of offhand is if E:\Tulare2015 is some sort of network drive that might cause problems. Is this a local disk or something else?

    ~ Stacie
  • H @swolny - Thanks. My DEM has cell size 30 x 30 meters. It's "uncompressed size" is 6.20 GB. 

    E:\Tulare2015 is a 1 TB external hard drive (400 GB of it is taken up with other info).
  • swolnyswolny Member, NatCap Staff
    Wow, that's a really big DEM, and that means that each intermediate and final raster from the model will also be 6.2GB, which will definitely take up a lot of space. I usually try to keep mine under 1GB, both for disk space and memory. Are you sure you're not running out of memory instead of disk?

    If you're sure you're not running out of disk or memory, then I'll have to turn this over to our software team. They do a lot of work to make the models better able to handle large datasets, and they might have some ideas.

    ~ Stacie
  • @swolny - yes, it is big! I believe I shouldn't be running out of memory or disk space.
  • swolnyswolny Member, NatCap Staff
    Ok then, @Rich and @jdouglass, I'm punting to you!

    ~ Stacie

  • RichRich Administrator, NatCap Staff
    Hi @atutterow, those errors are coming directly from the underlying GDAL library and are consistent with an out-of-disk-space error (or a scary hardware error!).

    But assuming the disk isn't failing, that error is coming from the step that aligns the raster stack including the monthly precipitation rasters. The model resizes all rasters to be the same pixel size as the DEM and then clips everything to the raster stack's intersecting bounding box. The very first disk error seems to be happening on the very first precipitation reprojection. Is it possible that your precipitation rasters are already rather large but their pixel size is larger than the dem? If so, the model will be making the precipitation rasters EVEN LARGER by effectively increasing their pixel count. How big are your base precipitation rasters compared to your DEM?

    And even though it looks like nothing finished, could you see if you could inspect "F:\Tulare2016\prcp_a0.tif" or any of the other rasters generated there to see what their sizes are?
  • Hi @Rich - Thank you.

    The base precip rasters are 319 KB while the DEM is 6 GB. The precip rasters also encompass a larger area than the DEM does, but I have had success with even smaller DEMs with the same precip input.

    I am unable to check the size of the rasters generated - says it is invalid.

    Thanks!
  • RichRich Administrator, NatCap Staff
    Hi @atutterow, in that case it doesn't sound like the issue is a finer scaling... These are really large rasters, is there any way you could share them with me via dropbox or google docs? Something that would let me try it on my end to see what's going on?
  • Hi @Rich


    I am using the climate zone advanced settings as well - I included my raster and rain events for that.

    Thanks for your help!
  • Hello @Rich - any update?

    Thanks!
  • RichRich Administrator, NatCap Staff
    Hi @atutterow, I'm running it now and observing that it's making pretty big intermediate files when aligning the rasters. I'll watch it and let you know.

    p.s. I notice that your soil raster is mostly nodata except for some regions in southern California. If you were to clip your stack to the bounding box of the nodata region things might run a little faster because you'd have less data to process.

  • RichRich Administrator, NatCap Staff
    Hi @atutterow, I think I see what might have been happening on your end. This version of the seasonal water yield model uses an older version of our geoprocessing pipeline that makes temporary files via an operating system call. Those files are created in the system temp file folder which is likely *not* your workspace. It's likely you were running out of space on your system drive, not your workspace drive, then when the model crashed it cleaned up all those temporary files.

    This is unfortunate because we're aware of this issue and we're in the middle of upgrading seasonal water yield to the newer pipeline, but it is not trivial work, it may take me a 2-4 more weeks. In the meantime, if you want to continue with this, is there any chance you might be able to pare down your dataset a little if possible? Like I mentioned in the last post your soil dataset is mostly nodata and you might be able to clip everything down to a tighter bounding box, you'll otherwise get nodata value results everywhere you have a nodata soil pixel. If that's not possible, could you try browsing to your system's temporary file directory and seeing if you can remove anything there? (If you're on windows this is likely in C:\Windows\Temp) or just in general see if there's anything you can do to make a few gigabytes of space on your system drive?

    If none of that is possible, I'll post back here when we get the SWY model upgrade complete.
  • Hi @Rich - Thanks for the troubleshooting. I'll try to free up some more space but I believe I have already gone through and cleaned up my drive as much as possible. I'll also crop my inputs.

    Let me know when the upgrade is complete! That will help significantly. 

    Thanks again!
Sign In or Register to comment.