Re: image formats - compression and loading

[TischlerJZ@ornl.gov (Jon Tischler)]

Tim wrote:

>Ok, I realize this is heresy and I expect to get torched for it, but is
>it really essential that compression of scientific data be absolutely
>lossless in order that the information contained in that data be
>preserved without loss?  For example, if you know that statistical
>errors in your data are larger than some quantity 'e', can't you
>randomly diddle numbers by less than 'e' without losing information?
>
>Another example: In many cases, we use 32-bit numbers systematically
>because we need a large dynamic range, but we never have 32-bit
>precision, and so we could usually drop eight, or even 16, bits if only
>we didn't introduce any statistically significant artifacts in the process.

People have done exactly what you describe.  I know someone at the NIST
reactor that does this.  The problem is figuring out how much precision is
needed and then only saving that much.  This requires making decisions
(correct decisions) about the data on the fly.  That is very hard to do
reliably.  Much easier to just use a lossless compression scheme.

Jon



Jon Tischler, UNI-CAT                               Solid State Division, ORNL
Bldg. 438D Argonne National Lab                     email: TischlerJZ@ornl.gov
Argonne National Laboratory                         TEL:   (708) 252-0861
9700 S. Cass Ave.                                   FAX:   (708) 252-0862
Argonne, IL  60439