2018.01.16 19:52 "Re: [Tiff] Strategies for multi-core speedups", by Kemp Watson
Also Joseph, you might find yourself limited by network bandwidth or BDP long before you are limited by application threading, good idea to think about network simulation (if relevant) before detailed design.
On Tue, Jan 16, 2018 at 2:34 PM, Paavo Helde <email@example.com> wrote:
On 16.01.2018 20:36, Joe.Maniaci@ricoh-usa.com wrote:
> I am a complete newbie to digital imagery and TIFFs in general, so please
> forgive any ignorance.
> I'm an automated test tool developer and I am actually going to start
> working on a viewer for the TIFFs that my company generates, to potentially
> include relying on the bigtiff format and this is the thing I've spent most
> of my time thinking about. I've come to the conclusion that for the images
> we generate, up to 10GB uncompressed(soon to be 40GB), that I will need to
> preprocess a single TIFF into multiple TIFFs containing intelligently
> placed tiles just to get around the I/O issue. By intelligently placed, I
> mean that if a user were to scroll their view window to the right, and five
> tiles(vertically aligned) had to be loaded into memory, I could
> theoretically have those files intelligently located in different TIFF
> files(subTIFFs?) that would allow multiple threads to operate on. So each
> thread only had to find one tile. So for my tool it would be a staggering
> amount of pre-processing, but real time manipulation is extremely fluid for
> the user which is basically my number one requirement.
> So for a case like mine, thread overhead is nothing to worry about. That
> and the more I read about the C++11 thread library, the less I worry. I can
> just leave it to the library to determine how many threads should operate
> in my 4-core environments, or my 16-core environments. At least that's the
> impression I've garnered.
If all you want is a viewer then you are limited by the screen sizes which are typically not more than ca 2000x2000 pixels. And this is pretty small image nowadays, there is no need to multithread anything, you could just read the needed tiles from the TIFF file in real time and maybe add a small cache of the read tiles so you don't need to read them again if the user pans left 1 pixel.
For zooming out the TIFF file should have different frames of the same image stored in decreasing resolutions so that a suitable resolution can be chosen for display - these are called pyramidal tiffs. The TIFF standard suggests that such varying resolution images should be stored as sub-IFD-s but as many viewers do not support sub-IFD-s they are typically stored just in standard IFD-s, and some XML piece in the first IFD provides the details. I am sure there are multiple variants of such pyramidal tiff formats, one from our company as well, and of course there are multiple readers supporting such tiff files.