2021.04.23 13:46 "[Tiff] "TIFFAdvanceDirectory": Error fetching directory count .....getting fewer "frames" but still working..", by Pablo Conesa

2021.04.24 14:52 "Re: [Tiff] "TIFFAdvanceDirectory": Error fetching directory count .....getting fewer "frames" but still working..", by Bob Friesenhahn

This software is used under a stressed system where huge tiff files are written in a network share and processed as soon as they are ready from another machine.

I've checked now (system is relaxed), and all the files have 50 frames (reported by python tifffile package) but my wild guess is that during the processing (stressed status) frames detected are less but still software "Works"?

Does this make any sense?

It seems like the files are being read while they are still being updated. It may be that the client thinks it is done (because the operating system told it so) but the operating system used by the client is still sending data to the server. Meanwhile, the reading client is depending on the implementation of the network share to access the file and the client likely uses caching, which may cause it to not know the correct answer.

What type of network share is being used? What are the server and client operating systems involved?

If this is Linux and using something like inotify(7) to notice when files are available, then it is possible to be notified about a new file available while the file is still being updated or even "settling" in the OS (e.g. file is "there" but OS has not yet actually done everything to make it completely so).

The rename() system call within the same filesystem is normally expected to be atomic. This still does not mean that two clients of a server can not have a differing opinion of the current state.

Some sort of locking beyond whether a file appears to exist or not is likely to be required.

Bob

Bob Friesenhahn
bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt