2005.01.14 04:52 "Re: [Tiff] JPEG (T.30-E, fax) in TIFF", by Lee Howard

Following your recommendations below (I've also commented below) I have produced:

http://gateway.howardsilvan.com/fax000003471.tif

You'll notice that StripOffsets and StripByteCounts still have three values, and that seems to be a libtiff bug that is influenced (strangely enough) by the SamplesPerPixel tag. Most of libtiff's utilities seem to compensate, however:

fax000003471.tif: Warning, incorrect count for field "StripOffsets" (3, expecting 1); tag trimmed.
fax000003471.tif: Warning, incorrect count for field "StripByteCounts" (3, expecting 1); tag trimmed.

As I've tried your recommended YCbCrSubsampling values before, it is not surprising, then, to see tiffcp produce these messages:

JPEGPreDecode: Warning, Improper JPEG sampling factors 2,2
Apparently should be 1,1..
JPEGPreDecode: Warning, Decompressor will try reading with sampling 2,2..
JPEGLib: Warning, Application transferred too many scanlines.

In a nutshell, I think this is a tale-tale sign that libtiff cannot decode this data and that libtiff must be enchanced in order to support it.

It took a while, but I was finally able to decode the TIFF with code that does not depend on LibTiff, for crosstesting. I'll privatly send you a reduced resolution ordinary YCbCr jpeg of the result in the copy to you, for checking.

Thank you, I did receive this image, and indeed, it looks quite perfect - better, in fact, than I've ever seen it, as I've no JPEG decoders that are capable of decoding the ITULAB JPEG data itself.

** StripOffsets/StripByteCounts **

The count of these tags is wrong, but you already knew.

Yes. I'm quite certain that this is not an effect of the procedure that I follow, but rather, the problem seems entirely a result of the SamplesPerPixel tag. I will, however, mention that the total TIFF file size is nearly twice what it should be due to the procedure that I follow. This appears to be because I've misunderstood the TIFFSetWriteOffset function in that it doesn't actually cause previously "written" data to be "overwritten", but rather it seems to just move directory pointer to the current position, leaving the old data still in the file, but "unused". I'd rather have the data overwritten.

** FillOrder **

I've removed this tag.

** YCbCrSubsampling **

I've not explicitly set this tag, so it is produced as 2,2.

** Decode **

I haven't (yet) been able to check whether the source JPEG contains the APP marker used to specify Decode parameters other than the ITULAB default, because I seem to have misplaced the spec that explained that APP marker and can't find any old code and can't remember, either. But, using the default to decode your image yields visually pleasing results, and I know from past experience that non-default Decode parameters are rare at best. So I guess we now know reasonably sure that you can probably ignore the Decode tag, too.

I've left this tag unset.

As for the JPEG markers, as best as I've been able to see, the source JPEG only contains the SOI, DHT, DQT, SOF, SOS, and DNL markers. I do not find an APP marker in the source JPEG.

** More general **

Unless, of course, you are set out to build yourself a solution that works for *any* ITULAB color-fax-jpeg, as opposed to the ones from the same single source, in that case you'll probably have to detect proper values for the YCbCrSubsampling and Decode tags from your JPEG stream. Let me know if that is the case, and I shall try and find time to help out if I may, this is all very interesting...

Once libtiff can decode the ITULAB JPEG compressed TIFF file then I will begin testing how much my single test case is the norm. I'll have various color fax senders start sending me test color faxes to see how well or poorly it fares. If I do need to calculate the values of the YCbCrSubsampling and Decode tags, then we can address that at that point. However, at this point I am content to assume that all color JPEG faxes will function as this one does (which, according to experience, will no doubt prove to be an incorrect assumption).

Plus, I'm still keen on the general JPEG to JPEG-in-TIFF converter... I'm thinking of a general scheme, something like

• sequentially add blocks of JPEG data to the single strip (so that the scheme fits a slow JPEG reception without temp buffer, like you want it)
• while doing so, also sequentially feed these blocks to LibJpeg, which should be linked in LibTiff anyway, so as to read all relevant properties like color space, subsampling, etc, from the JPEG stream. At this stage, also especially note DNL, and if so, remember position to write fix-up and count lines
• when that's done, fill in the tags with the info read from the JPEG stream.
• also do the DNL fix-up now, if necessary

I'm open to ideas on how to better handle the SOF fixup with the information from the DNL marker.

I'm unsure, however, how much of tag setting LibTiff allows to be postponed untill after the strip writing stage.

From experience with traditional fax encoding (CCITT G3, G4), only the compression tag needs to be set beforehand.

I you or anyone on this list wishes to join in to explore this further, now would be a good time to say so.

I'm always excited to do new and interesing things, but I'm not a libtiff coder, so I don't know of how much benefit my participation is.

In any case, please keep us informed whatever happens next. It's not the first time ITULAB related stuff comes up in this list, so it would be nice to see and archive where this is going.

The way it looks now, is that I have to wait for ITULAB progress with libtiff. So I'm looking forward to progress on this bug report:

http://bugzilla.remotesensing.org/show_bug.cgi?id=736

Once libtiff supports ITULAB enough, then I will be able to enable HylaFAX to support at least *receiving* color faxes (using JPEG rather than JBIG, JBIG would come later). *Sending* faxes requires that HylaFAX place a "tagline" or "header" on each page of an outgoing fax, and currently HylaFAX uses its own built-in G3/G4 decoder/encoder (very similar to libtiff's) to do that. I don't want to have to build-in a JPEG decoder/encoder into HylaFAX, so I'll have to learn how to link-in libjpeg or get libtiff to do that work for HylaFAX.

Thanks.

Lee.