1998.09.29 20:57 "libtiff 3.4 beta037 bug", by Christopher Tracy

I believe I have found a problem with libtiff 3.4 beta 037 in how it handles fax3 decoding. I am using the libtiff library on a Solaris 2.5.1 SPARC system.

The problem is one of not enough memory being requested for allocation. This occurs in the file tif_fax3.c in the Fax3SetupState() function. Here's the code snippet:

>       if (tif->tif_mode == O_RDONLY) {        /* 1d/2d decoding */
>               Fax3DecodeState* dsp = DecoderState(tif);
>               uint32 nruns = needsRefLine ?
>                    2*TIFFroundup(rowpixels,32) : (rowpixels);
>
>               dsp->runs = (uint16*) _TIFFmalloc(nruns*sizeof (uint16));
>               if (dsp->runs == NULL) {
>                       TIFFError("Fax3SetupState",
>                           "%s: No space for Group 3/4 run arrays",
>                           tif->tif_name);
>                       return (0);
>               }
>               <snip>
>       }

When determining the number of runs (nruns) for the decode buffer (I'm only dealing with 1D but there may be a similar issue for 2D), the value of rowpixels is used. However, there is one case where this is not enough. A scanline of the form 1b 1w 1b 1w 1b 1w... 1b 1w (i.e., alternating black and white pixels, starting with black and ending with white) will actually require rowpixels + 1 runs. This is because of the 0w (white terminator) run length required at the beginning of the scan line. So the run buffer is acutally filled like this (with n==rowpixels):

runs[0] = 0 // 0 whites
runs[1] = 1 // 1 black
runs[2] = 1 // 1 white
runs[3] = 1 // 1 black
runs[4] = 1 // 1 white
  ...
runs[n-1] = 1 // 1 black
runs[n] = 1 // 1 white!!! Buffer overflow!!!

My solution was to change the allocation request to 'rowpixels + 2'. The additional run is required because the _TIFFFax3fillruns() function requires the run length buffer to contain an even number of runs and will extend the buffer pointer by 1 if not, zero filling that run length. Here's the code snippet for that:

>        if ((erun-runs)&1)
>           *erun++ = 0;

where erun points to the end of the run length buffer.

Any comments? Please let me know when and how this will actually be addressed in the libtiff source. Any plans for the next beta or full release of the library?

Thank you,

Christopher Tracy
Xpedite Systems