2020.10.13 19:25 "Re: [Tiff] faster Zip/Deflate compression/decompression with libdeflate", by Roger Leigh
Although we have better alternative than Deflate now with ZSTD, it is still heavily used as being far more supported than ZSTD or other alternatives. As an example of this, very recently, in the geospatial domain, Sentinel2 L2A scenes have been reprocessed as Deflate- compressed TIFF and published in https://registry.opendata.aws/sentinel-2-l2a-cogs/. This represents a volume of 200 million TIFF files for 10.5 petabytes.
The libdeflate work looks really great. I’ll try to find time to review it in more detail tomorrow.
One thought: GitLab supports Git LFS for storing “large files”. It might we worth using it for the TIFF sample data; same would apply to the other image data already in the repository. Doesn’t need doing for this merge request, but might be worth considering migrating over to. It would permit adding more test data to the repository in the future as well without affecting repository clone time (but it would download the branch-specific large file content on demand when you check out a branch, if you didn’t already pull it down explicitly).
Regarding ZSTD, what’s the current situation with regard to the Adobe tag registry? Do we have a number for ZSTD yet?