data-catcher brainstorm

[mooney@aps.anl.gov (Tim Mooney)]

Dear folks,

I'd like to hold a series of brainstorming sessions, with both users
and developers, to sketch out the direction(s) in which further
development of the EPICS Data Catcher should go (or not go, I
suppose).  The first session will be held this Friday 6/30 at 10:00 in
Bldg. 360, Rm. E356.  I realize this is impossibly short notice for
many of you, but don't worry if you can't make it because nothing
happens at first sessions of this kind anyway, other than a general
sort of clearing away of the underbrush.

Background:

Currently, 'The Data Catcher' is a catch-all term (<- JOKE, heh, heh)
for the ideas behind several grossly similar IDL programs that read,
display, and store arrays of data acquired by EPICS scan, MCA, and
waveform records.  'catch1d', a program that can passively support
1-dimensional scans acquired by the scan record, is probably the most
widely used instance at the moment.  A prototype 2-dimensional 'data
catcher' also exists, and there is a prototype MCA 'data catcher' as
well as CARS' polished MCA application.

We have requests lined up for support of scan types that would require
many more distinct 'data catchers', if the current ad-hoc mode
of development were to continue.  I believe nobody wants this nightmare,
but a general killer application that can acquire, display, and store data
any way an experimenter might want is a significant undertaking, which
yells for input from every relevant source.  Here are some of the ways
experimenters have already told us they want to acquire data:

1) straight 1-D scan, currently supported by catch1d (up to four positioners
   and four detectors)

2) straight 1-D scan with an arbitrary number of positioners and (scalar)
   detectors

3) 1-D scan, each of whose 'data points' is a spectrum

4) 1-D scan, each of whose 'data points' is a 1-D scan.  User might want
   to examine 1-D scans #43 through #57 as a 2-D plot, before acquiring
   further data.  (But throw away scan #52, because the detector was
   saturated during part of it.)

5) 1-D scan, each of whose 'data points' is an image (e.g., a movie, or
   tomography data)

6) straight 2-D scan, different from (4) in that a single set of values
   of uninvolved motors, temperatures, etc. applies to all component
   1-D arrays.  (I.e., the user really wants an image.)

7) an image

8) 2-D scan, each of whose 'data points' is a spectrum.

...and so on...


Now is the perfect time to step back and plan: catch1d has reached a
fork in the road; the APS data-file format is well chilled, but not
frozen; ditto the scan and MCA records; some of the EPICS system
developers are fresh (whaddaya wamee to say--wilted?) from a code
review of cDev; bulk-data transfer is transferring; IDL 4.0 is
out; and IDL's EPICS underwear is still under discussion.

I should mention some points that *I* think are non-negotiable, so we'll
be sure to get at least some irate feedback:

- we support the APS data-file format

- we don't in every case require the client that initiates a scan to
  also be the data catcher

- we do things right or we do things ad-hoc; we don't do both. 


...and some points that are up in the air:


- do we wait for CA to support large arrays?

- do we take data exclusively over the network?

- what belongs in hardware, what belongs in EPICS/HiDEOS, and what
  belongs in the client?

- where does this project lie relative to other priorities?

- where does data acquisition end and data analysis begin?


If you have comments and don't expect to attend, please email me.  I'll
convey everything I get.  The development group will probably have to
be whittled down to Ben-chin and one or two others before any actual
code begins to emerge; we should hammer out a detailed functional
description before that happens.

Tim