1999.06.04 03:45 "Re: Large File Support", by Frank Warmerdam

I am investigating for our project what file formats support large images (ie > 2GB). I am not a TIFF expert but noticed that the TIFF 6.0 spec has a field in the Directory Entry structure called "count". Which is 4 bytes unsigned. This would seem to limit TIFF to an image size to something less than 4 GB. Is this correct?

For our UNIX machines (HP, SGI, SUN) we are compiling 32 bit applications and using the transitional API for accessing files > 2GB defined by the Large File Summit. Most of these are just the same calls with 64 added on (ie open is open64). I noticed that the tiff library does not use these. Will tiff be modified in the future for this or is 2GB the current limit of tiff for 32 bit compiles?

I am new to this mail list so if this has been discussed before I am sorry. I tried to retrieve an archive from the mail list but it never responded.

Bruce, and the list,

My understanding is that the 4GB limit is primarily imposed by the use of 32bit offsets for the beginning of tile/strip data chunks within the file itself. As I see it, the fix for this is 64 bit offsets and this will be a major compatibility break.

Is there even an Adobe (or defacto) approved approach to this?

In my opinion:

Agreement should be reached on a way of handling 64bit offsets in TIFF files.
libtiff should be modified to support reading of 64bit files.
libtiff should be modified to be able to generate 64bit files. Of course it should not do that by default.

Even with this approach it will be ages between 64bit files are widely supported, but at least if we can get a 64bit toolkit out there the process would begin.

Bruce, I presume you are with the photogrammetry group at Marconi? Within the remote sensing, and photogrammetry community the 4GB limit is becoming significant already. I might be able to get funding from a government agency, or software vendor to extend libtiff if there was general agreement on an approach.

I can think of a few approachs. One is to add a TIFF_LONGLONG type to the list of data types, and this would be a 64bit integer. The TIFFTAG_STRIPOFFSETS and TIFFTAG_TILEOFFSETS could optionally be of this type.

A full 64 bit solution would also allow any other file offset or data block size to be 64bit, but I think that is overkill. On the other hand, it would avoid problems with treating some offset specially, and if we are taking the 64bit plunge maybe it should be done comprehensively.

A third approach would be to add new TIFFTAG_TILEOFFSET32 and TIFFTAG_STRIPOFFSET32. If found, the additional 32bit number would be a 4GB page on which the chunk starts. Put together with the normal offset it gives a 64bit address. Files smaller than 4GB, but generated with 64bit offset handling would still be compatible with older applications since they would just ignore the extra tags.

Any opinion on the approaches? Is this a solved problem, but just not incorporated into libtiff?

Best regards,

---------------------------------------+--------------------------------------
I set the clouds in motion - turned up | Frank Warmerdam, warmerda@home.com
light and sound - activate the windows | http://members.home.com/warmerda
and watch the world go round - Rush | Geospatial Programmer for Rent