Repository navigation
Error when downloading large slope file #3
Description
Describe the bug
Running the workflow for whole Europe runs into an error when downloading slope. We suspect that this will be fixed by pointing to the new version of the wrapper here, as soon as it is released.
To Reproduce
snakemake results/module_area_potentials/results/NUTS0/area_potential_pv_rooftop.tif
Expected behavior
Slope data should be download without any trouble.
Screenshots
Job 7: Download slope data covering the bounds of the input shapefile.
Reason: Missing output files: results/module_area_potentials/resources/automatic/cutout/NUTS0/slope.tif; Input files updated by another job: results/module_area_potentials/resources/user/shapes/NUTS0.parquet
Activating conda environment: .snakemake/conda/530cc284f69c59c16a5f8deb4e32965f_
Activating conda environment: .snakemake/conda/530cc284f69c59c16a5f8deb4e32965f_
rasterio._err.CPLE_AppDefinedError: _TIFFPartialReadStripArray:Cannot read offset/size for strile around ~304
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "rasterio/_io.pyx", line 969, in rasterio._io.DatasetReaderBase._read
File "rasterio/_io.pyx", line 199, in rasterio._io.io_multi_band
File "rasterio/_io.pyx", line 205, in rasterio._io.io_multi_band
File "rasterio/_err.pyx", line 325, in rasterio._err.StackChecker.exc_wrap_int
rasterio._err.CPLE_AppDefinedError: slope.in.degree_edtm_m_240m_s_20000101_20221231_go_epsg.4326_v20241230.tif, band 1: IReadBlock failed at X offset 40, Y offset 3: _TIFFPartialReadStripArray:Cannot read offset/size for stril
e around ~304
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/data/user/jlauner/wind-value-map/.snakemake/scripts/tmpy40_zbsk.wrapper.py", line 100, in <module>
data = src.read(window=window, boundless=True)
File "rasterio/_io.pyx", line 687, in rasterio._io.DatasetReaderBase.read
File "rasterio/_io.pyx", line 688, in rasterio._io.DatasetReaderBase.read
File "rasterio/_io.pyx", line 972, in rasterio._io.DatasetReaderBase._read
rasterio.errors.RasterioIOError: Read failed. See previous exception for details.
Desktop (please complete the following information):
- OS: Linux
- Version: head of branch feature-full-rules
Activity
sjpfenninger commentedon Jul 25, 2025
This is now resolved in practice, but leaving the issue open as a reminder to point to the newest wrapper version once released. For now, by default full TIF files are downloaded, and the on-demand download which triggers the above issue can be optionally configured.
irm-codebase commentedon Jul 29, 2025
The general issue is Zenodo's rate limits.
We are limited to max 60 requests per minute or 2000 per hour. This can happen when requesting many small files in tandem, or a large chunked file. Rasterio is hard-coded to a 512x512 read/write window, with each being a request.
https://rasterio.groups.io/g/main/topic/how_to_politely_clip_large/114209356
The new version of the
rio-clipwrapper is already merged, but we need to wait for the next snakemake-wrappers release. It lets you modify the GDAL environment during the request to hopefully avoid this failure.These settings were suggested by GDAL devs: