2020.10.13 17:41 "[Tiff] faster Zip/Deflate compression/decompression with libdeflate", by Even Rouault

Hi,

Just wanted to bring to attention merge request https://gitlab.com/libtiff/libtiff/-/merge_requests/158 which I've submitted:

"""

Add support for optional building against libdeflate for faster Zip/Deflate compression/ decompression

So we can have 2 kind of builds with the Zip/Deflate codec:

- zlib only
- zlib + libdeflate

Speed improvements in the 35%-50% range can be expected when libdeflate is used. Compression level up to 12 is now supported (capped to 9 when zlib is used). Still requires zlib for situations where libdeflate cannot be used (that is for scanline access, since libdeflate has no streaming mode)

Pseudo-tag TIFFTAG_DEFLATE_SUBCODEC=DEFLATE_SUBCODEC_ZLIB/ DEFLATE_SUBCODEC_LIBDEFLATE is added to control which subcodec (zlib or libdeflate) should be used (it defaults of course to libdeflate, when it is available). This is mostly aimed at being used on the writing side, to be able to reproduce output of previous libtiff versions at a binary level, in situations where this would be really needed. Or as a safety belt in case there would be unforeseen issues with using libdeflate. It can be used to know when libdeflate is available at runtime (DEFLATE_SUBCODEC_LIBDEFLATE will be the default value in that situation).

Of course, deflate codestreams produced by libdeflate can be read by zlib, and vice-versa.

"""

I've tested the integration of this with GDAL in particular (its test suite helped to identify a corner case, which I've added to libtiff's own test suite)

Although we have better alternative than Deflate now with ZSTD, it is still heavily used as being far more supported than ZSTD or other alternatives. As an example of this, very recently, in the geospatial domain, Sentinel2 L2A scenes have been reprocessed as Deflate- compressed TIFF and published in https://registry.opendata.aws/sentinel-2-l2a-cogs/. This represents a volume of 200 million TIFF files for 10.5 petabytes.

Even

Spatialys - Geospatial professional services
http://www.spatialys.com