Pollination - Error due to memory ?

I am using InVEST pollination model. Everything runs fine until the software displays an error message : 
"NoneType'
object has no attribute '___getitem___
'"
However, the rasters (abundance, supply) have been correctly created (at least apparently). But I notice that, since they are quite big (7 Go), my computer has run out of memory, so I guess the error message could be due to a lack of free space.
How confident can I be in the rasters that have been created ? Should I re-run the model with more space ? 

Comments

  • My log : 

    05/03/2018 11:47:01  natcap.invest.utils INFO     Writing log messages to C:\Users\Manou\Desktop\Etudes\2-AgroParisTech\3e année - M2 BEE\Stage\Simulations\4-Pollination\1-MOS\2012\Output\InVEST-Crop-Pollination-log-2018-05-03--11_47_01.txt
    05/03/2018 11:47:01  natcap.invest.ui.model Level 100 Starting model with parameters: 
    Arguments for InVEST natcap.invest.pollination 3.4.2:
    farm_vector_path                 
    guild_table_path                 C:/Users/Manou/Desktop/Etudes/2-AgroParisTech/3e année - M2 BEE/Stage/Simulations/4-Pollination/1-MOS/2012/guild_table.csv
    landcover_biophysical_table_path C:/Users/Manou/Desktop/Etudes/2-AgroParisTech/3e année - M2 BEE/Stage/Simulations/4-Pollination/1-MOS/2012/landcover_biophysical_table.csv
    landcover_raster_path            C:/Users/Manou/Desktop/Etudes/2-AgroParisTech/3e année - M2 BEE/Stage/Simulations/4-Pollination/1-MOS/2012/lulc_final.tif
    results_suffix                   
    workspace_dir                    C:\Users\Manou\Desktop\Etudes\2-AgroParisTech\3e année - M2 BEE\Stage\Simulations\4-Pollination\1-MOS\2012\Output

    05/03/2018 11:47:01  natcap.invest.pollination INFO     Checking to make sure guild table has all expected headers
    05/03/2018 11:47:07  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 6.19% complete
    05/03/2018 11:47:12  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 33.01% complete
    05/03/2018 11:47:18  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 61.89% complete
    05/03/2018 11:47:23  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 94.90% complete
    05/03/2018 11:47:32  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 47.45% complete
    05/03/2018 11:47:38  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 96.96% complete
    05/03/2018 11:47:47  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 49.51% complete
    05/03/2018 11:48:02  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 52.40% complete
    05/03/2018 11:48:16  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 52.41% complete
    05/03/2018 11:48:21  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 92.72% complete
    05/03/2018 11:48:33  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 52.41% complete
    05/03/2018 11:48:54  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 43.32% complete
    05/03/2018 11:48:59  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 88.71% complete
    05/03/2018 11:49:12  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 45.39% complete
    05/03/2018 11:49:17  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 94.90% complete
    05/03/2018 11:50:23  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 72.57% complete
    05/03/2018 11:51:08  pygeoprocessing.geoprocessing INFO     starting convolve
    05/03/2018 11:51:14  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 256)
    05/03/2018 11:51:19  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 1024)
    05/03/2018 11:51:25  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 1536)
    05/03/2018 11:51:30  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 2304)
    05/03/2018 11:51:36  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 2816)
    05/03/2018 11:51:41  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 3584)
    05/03/2018 11:51:47  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 4352)
    05/03/2018 11:51:52  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 4864)
    05/03/2018 11:51:58  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 5632)
    05/03/2018 11:52:27  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 4.02% complete
    05/03/2018 11:52:32  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 24.19% complete
    05/03/2018 11:52:38  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 60.46% complete
    05/03/2018 11:52:43  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 88.69% complete
    05/03/2018 11:53:17  pygeoprocessing.geoprocessing INFO     starting convolve
    05/03/2018 11:53:23  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 512)
    05/03/2018 11:53:28  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 1280)
    05/03/2018 11:53:34  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 1792)
    05/03/2018 11:53:39  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 2560)
    05/03/2018 11:53:45  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 3072)
    05/03/2018 11:53:50  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 3840)
    05/03/2018 11:53:55  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 4352)
    05/03/2018 11:54:01  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 5120)
    05/03/2018 11:54:06  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 5632)
    05/03/2018 11:54:46  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 4.03% complete
    05/03/2018 11:54:54  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 8.06% complete
    05/03/2018 11:55:02  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 12.09% complete
    05/03/2018 11:55:08  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 16.13% complete
    05/03/2018 11:55:14  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 24.19% complete
    05/03/2018 11:55:20  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 32.25% complete
    05/03/2018 11:55:25  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 56.44% complete
    05/03/2018 11:56:07  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 4.03% complete
    05/03/2018 11:56:13  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 12.09% complete
    05/03/2018 11:56:20  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 20.16% complete
    05/03/2018 11:56:25  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 28.22% complete
    05/03/2018 11:56:30  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 36.28% complete
    05/03/2018 11:56:35  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 44.34% complete
    05/03/2018 11:56:41  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 60.47% complete
    05/03/2018 11:56:48  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 72.57% complete
    05/03/2018 11:56:54  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 80.63% complete
    05/03/2018 11:56:59  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 88.69% complete
    05/03/2018 11:57:04  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 96.75% complete
    05/03/2018 11:57:20  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 4.03% complete
    05/03/2018 11:57:27  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 12.09% complete
    05/03/2018 11:57:33  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 16.12% complete
    05/03/2018 11:57:40  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 32.25% complete
    05/03/2018 11:57:45  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 40.31% complete
    05/03/2018 11:57:51  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 72.57% complete
    05/03/2018 11:57:56  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 96.75% complete
    05/03/2018 11:58:00  py.warnings        WARNING  C:\INVEST~1.2_X\INVEST~1\natcap\invest\utils.py:329: RuntimeWarning: invalid value encountered in divide

    05/03/2018 11:58:51  pygeoprocessing.geoprocessing INFO     starting convolve
    05/03/2018 11:58:56  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 768)
    05/03/2018 11:59:01  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 1536)
    05/03/2018 11:59:07  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 2304)
    05/03/2018 11:59:13  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 3328)
    05/03/2018 11:59:19  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 4096)
    05/03/2018 11:59:24  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 5120)
    05/03/2018 11:59:30  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 5888)
    05/03/2018 12:00:03  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. -0.01% complete
    05/03/2018 12:00:12  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 4.02% complete
    05/03/2018 12:00:20  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 8.06% complete
    05/03/2018 12:00:25  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 16.13% complete
    05/03/2018 12:00:31  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 40.31% complete
    05/03/2018 12:00:36  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 80.63% complete
    05/03/2018 12:01:17  pygeoprocessing.geoprocessing INFO     starting convolve
    05/03/2018 12:01:23  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 512)
    05/03/2018 12:01:28  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 768)
    05/03/2018 12:01:33  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 1280)
    05/03/2018 12:01:39  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 2048)
    05/03/2018 12:01:44  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 2816)
    05/03/2018 12:01:49  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 3584)
    05/03/2018 12:01:57  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 4096)
    05/03/2018 12:02:02  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 4608)
    05/03/2018 12:02:08  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 5632)
    05/03/2018 12:02:13  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 6144)
    05/03/2018 12:02:59  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 44.34% complete
    05/03/2018 12:03:04  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 88.68% complete
    05/03/2018 12:03:13  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 44.34% complete
    05/03/2018 12:03:42  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 8.05% complete
    05/03/2018 12:03:47  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 16.12% complete
    05/03/2018 12:03:53  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 24.18% complete
    05/03/2018 12:03:58  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 44.34% complete
    05/03/2018 12:04:04  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 64.50% complete
    05/03/2018 12:04:09  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 80.62% complete
    05/03/2018 12:04:14  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 92.72% complete
    05/03/2018 12:04:33  pygeoprocessing.geoprocessing INFO     starting convolve
    05/03/2018 12:04:38  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 512)
    05/03/2018 12:04:43  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 1280)
    05/03/2018 12:04:49  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 1792)
    05/03/2018 12:04:55  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 2560)
    05/03/2018 12:05:00  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 3328)
    05/03/2018 12:05:06  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 3840)
    05/03/2018 12:05:11  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 4608)
    05/03/2018 12:05:16  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 5120)
    05/03/2018 12:05:22  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (4096, 5888)
    05/03/2018 12:05:42  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 8.05% complete
    05/03/2018 12:05:47  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 40.31% complete
    05/03/2018 12:05:52  pygeoprocessing.geoprocessing INFO     raster stack calculation approx. 72.57% complete
    05/03/2018 12:06:12  osgeo              ERROR    [errno 1] _tiffWriteProc:No space left on device
    05/03/2018 12:06:12  osgeo              ERROR    [errno 1] TIFFAppendToStrip:Write error at scanline 3584
    05/03/2018 12:06:12  osgeo              ERROR    [errno 1] _tiffWriteProc:No space left on device
    05/03/2018 12:06:12  osgeo              ERROR    [errno 1] TIFFAppendToStrip:Write error at scanline 3584
    05/03/2018 12:06:12  osgeo              ERROR    [errno 1] c:\users\manou\appdata\local\temp\tmpc42h0l\convolved_mask.tif, band 1: An error occured while writing a dirty block
    05/03/2018 12:06:13  osgeo              ERROR    [errno 1] _tiffWriteProc:No space left on device
    05/03/2018 12:06:13  osgeo              ERROR    [errno 1] TIFFAppendToStrip:Write error at scanline 3840
    05/03/2018 12:06:13  osgeo              ERROR    [errno 1] _tiffWriteProc:No space left on device
    05/03/2018 12:06:13  osgeo              ERROR    [errno 1] TIFFAppendToStrip:Write error at scanline 3840
    05/03/2018 12:06:13  pygeoprocessing.geoprocessing INFO     starting convolve
    05/03/2018 12:06:19  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 768)
    05/03/2018 12:06:25  pygeoprocessing.geoprocessing INFO     convolution operating on signal pixel (0, 1536)
    05/03/2018 12:06:30  osgeo              ERROR    [errno 1] TIFFFillTile:Read error at row 256, col 3584; got 7066 bytes, expected 262144
    05/03/2018 12:06:30  osgeo              ERROR    [errno 1] TIFFReadEncodedTile() failed. 
    05/03/2018 12:06:30  osgeo              ERROR    [errno 1] c:\users\manou\appdata\local\temp\tmpc42h0l\convolved_mask.tif, band 1: IReadBlock failed at X offset 2, Y offset 9
    05/03/2018 12:06:30  osgeo              ERROR    [errno 1] GetBlockRef failed at X block offset 2, Y block offset 9
    05/03/2018 12:06:30  natcap.invest.ui.model ERROR    Exception while executing <function execute at 0x0DCD56F0>
    Traceback (most recent call last):
      File "c:\users\natcap-servers\jenkins-home\workspace\natcap.invest\label\gce-windows-1\release_env\lib\site-packages\natcap\invest\ui\model.py", line 1504, in _logged_target
      File "c:\users\natcap-servers\jenkins-home\workspace\natcap.invest\label\gce-windows-1\release_env\lib\site-packages\natcap\invest\pollination.py", line 502, in execute
      File "site-packages\taskgraph\Task.py", line 215, in add_task
      File "site-packages\taskgraph\Task.py", line 419, in __call__
      File "site-packages\pygeoprocessing\geoprocessing.py", line 1825, in convolve_2d
    TypeError: 'NoneType' object has no attribute '__getitem__'
    05/03/2018 12:06:30  natcap.invest.ui.model INFO     Execution finished
    05/03/2018 12:06:30  natcap.invest.utils INFO     Elapsed time: 19m 28.51s

  • jdouglassjdouglass Administrator, NatCap Staff
    Based on the logfile, pollination definitely crashed because it ran out of disk space:

    05/03/2018 12:06:12  osgeo              ERROR    [errno 1] _tiffWriteProc:No space left on device

    While the rasters that were created *before* this crash should have been created properly, the model run was interrupted.  Unless the outputs you're hoping to get from the model are some intermediate rasters, I'd re-run the model with your workspace on a disk with more available space.

    Hope this helps!
  • Thank you, it helps a lot ! I just re-ran it directly on a flashdrive, and it ended correctly. 
  • wenhaowenhao Member
    Hi everyone,

    I was running the InVEST model today and encountered the same error as well.
    "No space left in the device"
    We've tried to switch to run on different drives but the error still comes up... even the sample dataset. Is there any quick way around?

    Thanks in advance.
  • jdouglassjdouglass Administrator, NatCap Staff
    Hi @wenhao, could you attach your logfile so we can take a look at the specific error that's causing the model to fail?
Sign In or Register to comment.