2004.01.14 12:01 "[Tiff] COLORMAP and byte padding", by Stephan Assmus

2004.01.14 19:36 "Re: [Tiff] COLORMAP and byte padding", by Marti Maria

Hi,

Here is a small program to check the error by using both methods, shifting 8, (same as *, / by 256) and the "good" method, which is by 257. But this should be not so surprising. If you are scaling a value with range 0.. 255 to a range 0..65535, the factor would be 65535 / 255, and that equals to 257 :-)

--- Cut ---

#include <stdio.h>
#include <stdlib.h>

int main()
{
   int i;
   unsigned char a8_by256, a8_by257;
   unsigned short a16_by256, a16_by257;
   int error256, error257;
   int max256 = 0, max257 = 0;
   double sum256=0., sum257=0.;

 for (i=0; i < 0xFFFF; i++) {

      a8_by256 = (unsigned char) (i >> 8);
      a8_by257 = (unsigned char) (double) floor(i / 257. + .5);

      a16_by256 = ((unsigned short) a8_by256 << 8);
      a16_by257 = ((unsigned short) a8_by257 << 8) | a8_by257;

      error256 = abs(a16_by256 - i);
      error257 = abs(a16_by257 - i);

      if (error256 > max256) max256 = error256;
      if (error257 > max257) max257 = error257;

      sum256 += error256;
      sum257 += error257;
   }

   printf("Max error:  by 256 = %d, by 257 = %d\n", max256, max257);
   printf("Mean error: by 256 = %g, by 257 = %g\n", sum256 / 65535., sum257 / 65535.);

   return 0;
}

--- Cut --