Re: netCDF format

[zzt@solid.ssd.ornl.gov (Jon Tischler)]

Mark Rivers correctly points out that:

> 'A netCDF file can contain an unlimited number of variables, each of
> potentially different data types and sizes.'

However, the only thing that differentiates these variables is their name.
There is no offically recognized way of separating one from another in a
netCDF file.  It is like a completly flat file system.  You could group
files in a flat file system by making rules for filenames (all system files
start with an 's', all user1 files start with a 'u1',...) but the
nomenclature quickly becomes cumbersome, and it is difficult for a large
group of people to consistenly use such a scheme.

In the HDF standard the Vset interface provides a clear way to group the
data into a scan, and you can name things anything you want.  Likewise when
you read in the data, the organization is obvious and does not require a
large manual for interpretation.

He further continues:

>At this point my recommendation would be to use netCDF for everything
>it CAN be used for and to go to HDF when one outgrows netCDF. It is
>nice that the NCSA HDF code can read/write both file formats.

Using netCDF now and then changing to HDF later seems to defeat the purpose
of choosing a standard.  Standards are very hard to change, better one too
roomy than one too small.  Besides, if we choose too limited a standard,
then we just encourage people to break it.  Let's choose a standard general
enough to store the data we have now, and extensible enough to store the
data we will want tomorrow.  A data file that can do all that is no harder
to write or understand than a limited standard.

Besides, a minimal HDF file would look just like (and could be interpreted
in the same way) as a netCDF file.  Your netCDF based programs should not
even see the Vset grouping unless they looked for it.

>I think there are some real limitations in the use of SDS for even very simple
>data which I will illustrate at the committee meeting next week.

I think that the way I am proceeding is general enough to happily
accomodate your examples (this is a challenge, right?).

>I will be building an IDL GUI file reader for netCDF data to allow P+C
>specification of which variables to read and how.

If you use the new HDF 3.3r3 libraries, then your GUI file reader should be
able to properly interpret an HDF file using SDS's, even if it only uses
netCDF calls.

>Different groups have different philosphies on how much information should go
>in a single data file. Does a 'stream of consciousness' for an entire shift go
>in one file, or are there separate files for each scan?

I agree completely, and I think that HDF is the only supported standard
that can support both viewpoints while still allowing the data to be
understood by one method.

>I am glad to see the discussion has started!

and I hope we reach a conclusion.

Jon



Jon Tischler                                        Solid State Division
ORNL, Bldg 3025, MS-6030    internet  zzt@ornl.gov  	TEL  (615) 574-6505
Oak Ridge, TN 37831-6030    bitnet    zzt@ornlstc   	FAX  (615) 574-4143