2006.04.05 00:17 "Re: [Tiff] BigTIFF Suggestion", by Joris Van Damme

Larry,

I appreciate your help. When I referred to a CRC, I meant that I save a CRC of the four-byte offset only. It may be overkill, but it, along with my little signature, allows me to be quite certain that what I read at the end of the file is what I wrote there and nothing else. That way, there is very little danger of seeking to the wrong offset for appending if I read a file written by a different app or one which was truncated.

Consider following scenario:

You write n IFDs to a file, indexed 0 to n-1
You append offset of IFD n-1
An app comes along, and unlinks last page, IFD n-1, by zeroing next-IFD offset in IFD n-2. It is usual for page-delete operations to work that way, merely unlinking stuff and not cleaning up garbadge (which would be very difficult and costly), not even if the 'stuff' to clean up is at the end of file and truncation would be sufficient (which is very difficult and costly to detect).
Next your app comes along, reads offset of IFD n-1, sees CRC on this offset is OK, and thus appends IFD n to IFD n-1... but IFD n-1 is 'deleted', unlinked from list, and your appending operation is unknowningly in vain.
It is my experience that it is generally dangerous to try and outsmart the simple linked list. I've often seen some subsequent operations are not anticipated, resulting in unexpected things later in the pipeline... Even if you can think of a way to exclude above scenario, who's to say there are no others we're not thinking of right now? I've learned from experience to be extremely carefull in these matters.

I do not want to put the information in a separate file because users may move and rename their files.

I can understand that. I honestly fear that may not be a real good solution to your problem.

We have a number of requirements for image sequence file, some of which conflict with each other. The two biggest are the ability to hold very large numbers of very large images, and to be able to be written quickly. Our application does not yet support files larger than 4GB, but I have been told to add the support, so I have been comparing various options, one of which might be BigTIFF. I think that PDF is probably too complicated for our purposes. I may implement a proprietary format optimized for our needs and also allow users to save their files as BigTIFF for compatibility with other applications.

I see... Possibly, a clearly non-TIFF but likely TIFF-based proprietary format may be the best solution, if you're ready to put in that kind of effort. Even if your only modifications aside from appended last-IFD-offset are only by pre-pending some stray bytes (additional signature?) and changing some important ones (TIFF signature?), that'll be sufficient in keeping other apps from interfering with your files, solving your problem but excluding a lot of bad scenario's from happening.

Joris Van Damme
info@awaresystems.be
http://www.awaresystems.be/
Download your free TIFF tag viewer for windows here:
http://www.awaresystems.be/imaging/tiff/astifftagviewer.html