2008.11.14 22:03 "Re: [Tiff] Memory leak (TIFFOpen, TIFFReadTile)?", by Phillip Crews

Ilkka,

For Windows:

Don't use Task manager to monitor a process's memory use. Get Process Explorer from www.sysitnernals.com, which has been bought by Microsoft and redirects there now. Add a column for and watch the "Private bytes" for the process. This is the committed, writeable, non-shared pages used by a process and is a quick indicator to watch for when either memory allocation patterns go wonky or memory leaks occur.

For large memory allocations, you may want to switch to VirtualAlloc() and VirtualFree() rather than malloc/free. VirtualFree() actually returns the pages to the O/S rather than leave them hanging around for future mallocs. (I believe some newer Windows C runtimes will automatically use VirtualAlloc/Free under some conditions, but I've had my own memory management library for so long I'm not sure how it works. realloc() can interfere with this too, since pages have to be wholly free before they can be released.) Note that there is no VirtualRealloc(); but realloc() itself is sometimes an easy way to cause runaway page allocations and unusable holes in the process address space.

The often unpredictable distribution of sizes and lifetimes of memory allocations contribute to heap fragmentation; there's also the low fragmentation heap in XP and later, but that's more geared to programs that allocate memory in smaller sizes. It attempts to minimize unusable holes in the heap. See: http://msdn.microsoft.com/en-us/library/aa366750(VS.85).aspx for more information.

Sorry if that's too rambling. Been a long week.

- Phillip

I have some huge TIFFs, size upto 2 gB, 1, 3 or 4 bands, 2 bytes per band, tiled. Huge aerial image-scanner mats with the ads80 instrument,

I'm trying to make untiled greyscale/rgb pixel-interleaved raw-image versions, I am working with MS Vis studio 6 and C in an XP machine.

I believe I am using the 3.6.1 version of libbtiff. libtiff.lib is from Feb 2004.

  I now have a working solution for the task of untiling and writing
a bip-image, but there is a memory leak somewhere? I am reading
and converting a 1517454784 bytes file. XP's task manager tells me
  the process is gradually consuming nearly 1.3 Gbytes at the end.

I tried with _TIFFmalloc() and malloc() but it does not make a difference - memory consumption goes up. Any suggestions out there? Here is the simple code (include statements not there),:

ilkka

struct NRGB16 {
  uint16 N;
  uint16 R;
  uint16 G;
  uint16 B;
};

int main(int argc, char* argv[])
{

// Open the 1-2 GByte block file (ads40-imagery)
TIFF* tif = TIFFOpen("c:\\temp\\ads40\\O-08230753NRGBN00AL2_0_0.tif", "r");
FILE *fOut;
  uint32 i;
         if (tif) {
         uint32 imageWidth, imageLength;
         uint32 tileWidth, tileLength;
         uint32 x, y;
           NRGB16* buf;
         unsigned char Red, Green, Nir;
           int Column, Row, OldRow;
          // Query the Image & Tile dimensions

         TIFFGetField(tif, TIFFTAG_IMAGEWIDTH, &imageWidth);
         TIFFGetField(tif, TIFFTAG_IMAGELENGTH, &imageLength);
         TIFFGetField(tif, TIFFTAG_TILEWIDTH, &tileWidth);
         TIFFGetField(tif, TIFFTAG_TILELENGTH, &tileLength);

// Open the output file, 3-channel viewable version of the aerial image
fOut=fopen("c:\\temp\\ads40\\O-08230753NRGBN00AL2_0_0_CIR_uint8.raw","wb");
OldRow = -1;
// Allocate storage for the 4-channel, 2 bytes per channel image TILE
                  buf = (NRGB16*)malloc(tileLength*tileWidth*4*2);
// Start untiling the image
          for (y = 0; y < imageLength; y += tileLength)
           for (x = 0; x < imageWidth; x += tileWidth)
          {
           i=TIFFReadTile(tif, &buf[0], x, y, 0, 0);
           for (i=0;i < tileLength*tileWidth;i=i+1)
           { // Carry out simple 16-8 bit reduction
           Nir = 255; Green = 255; Red = 255;
          if ( double(buf[i].N)/20 <= 255) Nir = unsigned char(buf[i].N/20);
          if ( double(buf[i].R)/15 <= 255) Red = unsigned char(buf[i].R/15);
          if ( double(buf[i].G)/15 <= 255) Green = unsigned char(buf[i].G/15);
            // Compute the global col,row coordinates in the untiled image
            Column = x + (i % tileWidth);
            Row = y + (i / tileWidth);
         // Check if we started a new row, set the file pointer accordingly
    if (Row != OldRow) {fseek(fOut,Row*imageWidth*3+Column*3, SEEK_SET);}
    OldRow = Row;
    // Write the RGB-bytes
    fwrite(&Nir, 1, 1, fOut);
    fwrite(&Green, 1, 1, fOut);
    fwrite(&Red, 1, 1, fOut);
    }
    // Memory leak? Try reallocating the buf
    buf = (NRGB16*)realloc(buf,tileLength*tileWidth*4*2);
// Watch for memory sizes of these blocks (showing no leak here)
    printf("%d %d\n",_msize(tif),_msize(buf));
    }
    // Finally free everything
    free(buf);
    fclose(fOut);
    TIFFClose(tif);
  }
  return 0;
}