Re: image formats - compression and loading

[bunker@biocat1.iit.edu (grant bunker)]

In regard to compression, I think a standard file format should be lossless
and plain vanilla, and if compression needs to be done, that it be after
the fact. Realistically, good compression needs to be depend on the nature of
the data - for example the differences between small angle scattering data
and crystallographic data can be exploited to give optimal compression in
most cases. Probably a more general method can be developed to handle both
cases but I think it's better to allow the flexibility to use different
methods.

Rather than throwing out the least significant bits as in Tim's modest proposal
I would favor encoding the significant features in the data in some economical
form and recording the residual in a separate place, to be discarded, or
analyzed in more detail as necessary. If the encoding of the significant feature
s
of the data is done well, the residuals should have a limited dynamic range and
would fit into say, 8 or 12 or 16 bits. Factoring the data set into significant
features plus mud would also facilitate beamline procedures such as crystal
alignment for which idealized 'cartoons' of the data may be adequate because the
entire data set needn't be manipulated - it's already partially digested, but
reversibly, so the original dataset can be constructed.

As you may have guessed I have some ideas on how to go about this but there is
a lot to be done. So it's best to keep it open-ended and let it develop as
time permits.

grant bunker