image formats - compression and loading

[Peregrine McGehee <mcgehee@cfht.hawaii.edu>]

Our Data Acquisition group is looking at ways of effectively moving
and displaying large CCD images. Although, like many astronomical
sites, we are using FITS as the basic image format we are interested
in new formats as our detectors become larger and more complex.

Is anyone else who is working with large format CCDs who is at all 
concerned with the issues outlined below?

Are there solutions to these problems that are based around the HDF
image format?

One of our staff programmers, Sidik Isani, is actively working on
this and writes:

------- start --------
In the interest of making sure that the next generation 'joe's image format'
has a chance of being usable for telescope images, I'm thinking of sending
some sample fits files, and a wish list of what we'd like to see in an
image storage scheme to various hackers/researchers.  On my list to send
this stuff to,  is a research project at a university in Canada, and the
creator of the PNG standard.  (The latter may already be talking to astros--
his format supports lossless compression of 16-bit grey-scale images.)

I need a few samples of the different types of images that we come across
here (bias, flats, darks, object whatever) and also examples of anything
else that happens with any sort of frequency in these images (large areas
of image with no light?? Bad CCD rows or columns? Anything that might be
important to an image compression algorithm.)  A range of fits files sizes
and image dimensions will be useful as well. (esp the BIG ones!!)


Also, let me know about similar efforts on this topic.

Thanks,

- - Sidik <isani@cfht.hawaii.edu>
- ---------------------------------------------------------------------------
Here's a draft of a wish list -- comments welcome!

1. Lossless, 16-bit grey (We won't have more than 16-bits ever, will we?)
2. Progressive loading
3. Obviously the initial stages of loading will have loss ... we would
   like this loss to be as artifact-free as possible
4. Uncompressed header information
5. 'Unlimited' image size and unlimited amount of header information

Here are some hard numbers of compression ratios for /users/observer/M57.fits:

from a 2 Mb, 15-bit image (=100%)

1.4 Mb lossLESS PNG file (70%)
0.7 Mb lossLESS jpeg (35%) ... too bad l-jpeg isn't really well supported :-(
0.038 Mb lossY p-jpeg (2%) ... except the image goes from having about 12,000
                  distinct grey-levels to around 250.  Fine for most viewing,
                  but no good for data analysis.
- ----------------------------------------------------------------------------
------- end -------


Aloha,
	Peregrine
-------------------------------------------------------------------------------
Peregrine M. McGehee			http://www.cfht.hawaii.edu/~mcgehee
Telescope Control Systems Group 	Canada-France-Hawaii Telescope
(808) 885-3178				P.O. Box 1597, Kamuela, Hawaii 96743