2020.02.05 17:35 "Re: [Tiff] Inhaling an image all at once in scanline format", by Bob Friesenhahn

Slightly confused here (yes that's not hard).

The tiff_conf.h file which I copied from tiffconf.vc.h has unconditional

#define STRIPCHOP_DEFAULT TIFF_STRIPCHOP

I can't see how anything in Makefile.vc or nmake.opt could override that - should I comment that line out to disable the chopping?

It seems like you would need to comment out the line. This smells like another nmake-build bug.

With the default strip chopping enabled, I see that libtiff reports 17 rows per strip for the image you posted earlier

With it enabled I get ONE row per strip:

I am using the Unixish configure script on a Unix system. It may use different defaults.

Have you perhaps specified a custom value for STRIP_SIZE_DEFAULT on your system of around 256kB as that's the only way I can see that allows 17 rows to be inhaled as one strip. Compiling with /DSTRIP_SIZE_DEFAULT=262144, I got:

256kB (or maybe something less like 128kB) is a reasonable number since the L2 cache size on many Intel CPUs is 256kB (allocated in units of the 64-byte cache line size). If you intend to immediately access the data then your own code is likely to need some of its L2 cache of its own, so 256kB might be slightly too large.

Interestingly this was slower than either of the above by a noticeable amount (about 1.9 times slower than the first case and 4.6 times slower than the second case above)!

This does not surprise me at all.

So I have left strip chopping enabled and have defined STRIP_SIZE_DEFAULT to 256kB as this seems to work best!

Even with strip chopping, you can assemble your own full image in memory by using a stride to offset each collected strip into your larger memory buffer. This requires a simple loop.

Finally if an image has been created with one row per strip, is there a good way to over-ride that programmatically in a safe manner (no I don't mean tinkering with tif->tif_dir) so that it will read as much as STRIP_SIZE_DEFAULT bytes instead of the single row?

No. The strip size is a physical property of the image file. There are higher level APIs which will do this for you but you don't want to use them.

PS I see that no-one ever adopted my proposal to change TIFF_IO_MAX in tif_unix.c to be 16777216UL (16MB) The current value of 2GB seems a tad silly.

I do remember that. The I/O is unlikely to be larger than the strip or tile size.

Something to consider is that it is common for the input TIFF file to be memory mapped. This is very useful for TIFF in that it avoids the cost of many seek and read operations. However, if your intention is to read a great many large input files in a sequence then memory-mapped I/O can be very harmful since the operating system may cache large amounts of data and not be very efficient or smart about expelling data which is no longer needed. This is observed by fast re-reads of the same input file, but slow performance with many input files.

Bob
--
Bob Friesenhahn
bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt