Chandra CalDB4 is a new calibration database structure and
interface, that is being introduced with the release
of CIAO 4.1 on December 15, 2008.
CalDB4 conforms to the general FITS paradigm established in the
pre-existing HEASARC CalDB system,
and can be maintained in the same directory space. However, CALDB4 is not compatible with Chandra CalDB 3.5.0
or any earlier versions of the Chandra CalDB, and may not be stored in the same directory tree as those earlier versions.
The specific version of CalDB4 that will be released to the
public is "4.1.0,"
to coincide with CIAO "4.1," and hopefully prevent confusion that would result if users were to believe that
CALDB "4.0.0" is compatible with CIAO "4.0" (released December 2007), which it definitely is not.
The new CalDB will only work with CIAO 4.1 or later versions.
"Introduction to CalDB4"
The CALDB4 software and directory/index structure has been defined
to address these additional general requirements:
1) To provide an index and directory structure that is applicable to an arbitrary mission configuration, and controllable by CalDB managers.
2) To provide the necessary automation of calibration parameter identification and evaluation.
3) To eliminate hard-coding of parameter names and/or values wherever possible in CIAO 4.1, and later in the DS.
4) To accommodate mission-independent tooling in the long term.
CalDB 4.1.0 (so designated to correspond to CIAO 4.1 to prevent
confusion) implements a new, flexible index system,
with keyword parameter/index column control set by the CalDB manager. In addition to the flexible indexing, the
software interface employs a hand-shaking implementation with the CalDB to identify necessary parameters for the
requested calibration data, and to supply the necessary values as available from the input data file headers, or as supplied
by the user when necessary. All that is required of the tool (or the user) to begin selection of calibration data is to know
the mission (CHANDRA) and the name of the particular calibration data needed (such as DET_GAIN or QE). The context
of the search determines the other required parameter values to get the right data.
The developer or the user do not need to know which parameters are
required to select the data; the interface will request
the appropriate information automatically, usually from an events, pha, pha2, or pi data file.
A snapshot copy of the CALDB4 requirements specifications is
This document currently specifies the new flexible index system,
CalDB configuration file "caldb.config," the CalDB query interface and algorithm, and the index builder software.
CIAO 4.1 incorporates the new caldb4 software interface and
associated library (See ECR-2008-026, concurrent),
and supports the index builder tool "calindex".
CIAO 4.1 and CalDB 4.1.0 are mutually required for full functionality of both.
CALDB4 and the specific implementation of CALDB 4.1.0 have been
built to address Chandra CalDB requirements
and to address the above general specifications, with the ongoing tacit approval of HEASARC CalDB personel.
HEASARC has been involved in the early specification and investigation of new requirements. Our general progress
has been included in annual reviews of the HEASARC CXC support contract. The CALDB4
requirements and implementations are to serve as a possible new HEASARC CalDB standard, with incumbent updates
to the OGIP standard taken as needed. We have built CALDB4 for Chandra, but upon sufficient internal review and
approval, intend to deliver the software and tools for CALDB4 to HEASARC pending verification of all specified
requirements and possibly some interfacing additions to be specified later.
The CalDB configuration file locates the individual CalDB trees and
index files for specific compliant
missions. The file is located at $CALDB/software/tools, and must contain the specs for all missions included
in the $CALDB directory tree for all the data to be accessible through the software interface. "caldb.config" is an ASCII
In the old specification CHANDRA's caldb configuration appears as:
CHANDRA ACIS CALDB data/chandra/acis caldb.indx CALDB data/chandra/acis
CHANDRA EPHIN CALDB data/chandra/ephin caldb.indx CALDB data/chandra/ephin
CHANDRA HRC CALDB data/chandra/hrc caldb.indx CALDB data/chandra/hrc
CHANDRA PCAD CALDB data/chandra/pcad caldb.indx CALDB data/chandra/pcad
CHANDRA SIM CALDB data/chandra/sim caldb.indx CALDB data/chandra/sim
CHANDRA TEL CALDB data/chandra/tel caldb.indx CALDB data/chandra/tel
The above information specifies the MISSION: CHANDRA and the INSTRUMENTs available to Chandra
under the old paradigm, where all calibration data are separated by the INSTRUME (keyword) to which they apply.
ACIS, EPHIN, HRC, PCAD, and SIM all correspond to legitimate instruments. TEL had to be added to
subsume the calibration data for PIXLIB, HRMA, and the GRATINGs.
In the new caldb.config for Chandra CalDB 4.1.0:
CHANDRA ACIS CALDB data/chandra/acis caldb.indx CALDB data/chandra/acis CALDB data/chandra/acis key.config
CHANDRA EPHIN CALDB data/chandra/ephin caldb.indx CALDB data/chandra/ephin CALDB data/chandra/ephin key.config
CHANDRA HRC CALDB data/chandra/hrc caldb.indx CALDB data/chandra/hrc CALDB data/chandra/hrc key.config
CHANDRA PCAD CALDB data/chandra/pcad caldb.indx CALDB data/chandra/pcad CALDB data/chandra/pcad key.config
CHANDRA SIM CALDB data/chandra/sim caldb.indx CALDB data/chandra/sim CALDB data/chandra/sim key.config
CHANDRA DEFAULT CALDB data/chandra/default caldb.indx CALDB data/chandra/default CALDB data/chandra/default key.config
CHANDRA DEFAULT CALDB data/chandra/pimms caldb.indx CALDB data/chandra/pimms CALDB data/chandra/pimms key.config
Note that each row contains three more elements, corresponding to
the location and name of the "key.config" file, which gives the keyword
configuration of the new flexible INDEX files that are built by "calindex".
Also, the TEL branch, which
was not entirely appropriate for storing HRMA, PIXLIB, and GRATING
files, has been eliminated. Instead,
there is the designation of DEFAULT to replace (and break) the now-outdated INSTRUME separation paradigm for CalDB files. When
calibration data for mission elements do not correspond to any particular INSTRUME value, these elements may be indexed and stored under
one or more directories designated as DEFAULT in caldb.config. So, it is no longer necessary to address HRMA or GRATING files
in the code, by hardcoding "TEL" as the instrument associated with HRMA or HETG, for example.
2. Directory structure: (Full table listing is included here
All branches have relatively predictable structures, with one
exception: the "pimms/" branch, which
is a special case of files used only to set up the PIMMS effective area files during the software build. Under
pimms/, the files are divided between those for ACIS configurations, and those for HRC.
|"caldb.indx" file for ACIS
|"key.config" file for ACIS
|index/ (Stores versioned index
files linked to "caldb.indx"
|"caldb.indx" file for DEFAULT
|"key.config" file for DEFAULT
|"caldb.indx" file for EPHIN
|"key.config" file for EPHIN
|"caldb.indx" file for HRC
|"key.config" file for HRC
|"caldb.indx" file for PCAD
|"key.config" file for PCAD
|"caldb.indx" file for PIMMS|
|"key.config" file for PIMMS caldb.indx|
|"caldb.indx" file for SIM
|"key.config" file for SIM
3. Mandatory Index Columns:
The following columns are required in all CalDB index files. In some
cases, such as the CAL_DIR and
the CAL_FILE, the requirement is obvious, since the location, filename, extension number, and calibration
code name are primary to a CalDB structure. Some designations are required for backward compatibility
with previously existing HEASARC standards, which the requirements document above specifies and
acknowledges must be maintained. However, despite being mandatory, they do not impinge on the
flexibility in building a CalDB for an arbitrary mission.
The mandatory columns and associated keywords are given in the
Requirements Document, Table 1, pp. 4 and 5.
I list the columns here for completeness, with definitions:
file Column Name
Keyword, if any
||”ONLINE” for calibration files
present in the
CALDB directory tree. ”OFFLINE” or
”<device name>” for calibration files not
present in the CALDB tree. See below for
information on constructing the fully qualified
path to the calibration file from CAL_DEV,
CAL_DIR, and CAL_FILE.
||The path to the directory
containing the calibration
file specified by CAL_FILE.
||The name of the calibration file.
||The name of the type of
calibration data. Used as
the basis for identifying the type of data being
||CBD* (varies CBD10001
|The calibration boundary
conditions. These are
additional query constraints that are either
sufficiently infrequent that including a query
column in the index file is inappropriate, or that
are not readily represented as query columns in the
index file (examples would be numeric ranges).
||The FITS extension number that
calibration data. May be zero if the calibration
data are stored in the primary HDU (for example,
an image). <This is 1 less than the data model
||The “quality” of the
calibration, defined as
follows: 0 = “Good. No known issues.”; 1 =
“Some calibration issues known. May be default
for processing or analysis.”; 2 = “Calibration
issues identified and replacement needed or
anticipated. May be used, but will be superseded
in the near future.”; 3 = “Former operational
mode, good enough for processing or analysis of
archival data taken in that mode. Significant
issues known; however recalibration of this mode
is unlikely.”; 4 = “Strongly deprecated or
superseded. Issue a warning to this effect if
used. Reprocessing needed if these data were used
in earlier processing; same for earlier analysis.
Kept to enable selection of these data as related
files.”; 5 = “ Bad. Never to be used again. To be
retired. A better option is available in the CalDB,
including related files data.”; 9
= “Dataset no
||The UTC date when this entry was
added to the
index file, in yyyy-mm-dd format.
CALDB4 has adopted the above from the previous HEASARC CalDB
specification, and has added none to this list.
The calindex software has added the designation for the CAL_QUAL Equivalent header keyword as "CAL_QUAL",
which it now reads and puts the value into the index file.
4. Predefined Index File Optional
Certain columns have been established by the accumulated CalDB history to contain certain types of data and are
interpreted by software (or not) in particular ways. CALDB4 has defined these as "optional", but having a particular
designation and type. In so doing, we have added a few of our own new columns in CalDB4, some of which are in
analogous to the pre-existing ones. These are in Table 2 of the requirements document, pp. 6 and 7.
||The calibration file class, one
of “PCF”, ”BCF”,
”CPF”, or “USR”. Not interpreted by software.
||The type of the calibration
file, one of ”DATA”,
”TASK”, or “FEF”. Not interpreted by software.
||Short string description of the
Not interpreted by software.
||The beginning valid date of the
yyyy-mm-dd format. This column is interpreted
||The beginning UTC valid time of
in hh:mm:ss format. This column is interpreted
||The MJD corresponding to
This column is interpreted by software.
||The ending UTC valid date of the
yyyy-mm-dd format. This column is interpreted
||The ending UTC valid time of the
hh:mm:ss format. This column is interpreted by
||The MJD corresponding to
This column is interpreted by software.
||Numeric “fidelity” of
calibration data. May be
used to select amongst different calibrations of the
same quality that satisfy the query requirements.
The ordering of the FIDELITY is defined such
that numerically greater values correspond to
higher fidelity calibration data. This column is
interpreted by software.
NOTES: CAL_CLAS, CAL_DTYP, AND CAL_DESC are all
mandatory columns and keywords in the existing
HEASARC standard. They are not mandatory in CALDB4.
CAL_VED, CAL_VET and END_TIME are analogous to the CAL_VSD, CAL_VST, and REF_TIME counterparts.
These are not supported in the existing HEASARC standard, but are added in CALDB4.
5. Keyword Configurations and "key.config":
Each branch of CalDB 4.1.0 has a particular specification for the
index file in "key.config" as listed in
the table above. "key.config," like "caldb.config," is always an ASCII file. In CalDB 4.1.0, the key.config
files reside just along side of the "caldb.indx" links. A key.config file looks like the following:
# Time-stamp: <2008-11-10 10:38:02 caldbmgr>
# CHANDRA key.config table
# for INSTRUME = <Manager specifies this>
#colName dataType format hdrKey queryCol nullVal
TELESCOP string a10 TELESCOP yes ""
INSTRUME string a10 INSTRUME yes "NONE"
CAL_DESC string a70 CDES* no "NONE"
CAL_VSD string a10 CVSD* no ""
CAL_VST string a8 CVST* no ""
REF_TIME double d "" no 0.0
CAL_VED string a10 CVED* no ""
CAL_VET string a8 CVET* no ""
END_TIME double d "" no 0.0
FIDELITY double d FDLT* no 0.0
The pound sign designates a comment, ignored by calindex.
The columns are the index file column name, data type, format, the
corresponding header keyword in the CalDB files,
the specification of whether the column is a query column, and null value. The column names are added to the index by
calindex, in addition to the mandatory columns described above. When a column is a "query" column, the value corresponding
to the "header keyword" designated thereto, is searched for in the input file headers whenever a CalDB call is made to
the index in question. This then is part of the built-in hand-shaking between the interface and the CalDB index structure,
to determine what is important in the selection of particular CalDB data.
As indicated in the caldb.config specification above for CHANDRA,
there are seven distinct branches now in CalDB 4.1.0,
with corresponding caldb index and key.config files. The specific key.config and resulting caldb.indx files will appear at
the following locations, once CALDB4 has been installed locally, and $CALDB has been set:
6. CHANGES to CalDB files in CALDB 4.1.0:
The data in CALDB 4.1.0 are largely the same as the data in CalDB version 3.5.1; in fact only one file has been added
and it contains no new data, but simply accommodates a special RMF building case (See ECR_2008_028). Pre-existing
CalDB filenames are all preserved, so that the transition in analysis may be seamless.
The only changes made to CalDB files from 3.5.1 to 4.1.0 are to the various HDU headers. Such changes include:
1) Eliminate unnecessary boundary conditions from the CalDB index
files (such as ENERGY ranges, CCD_ID
specifications with they are not needed, or redundant specifications
for GRATING or GRATTYPE, which are now handled as index columns in CalDB 4.1.0.) These unnecessary
boundary conditions are moved from the CBD* keyword series to a new series CBO* for Optional boundary
conditions. They remain in the file headers, but are not interpreted by calindex.
2) Add calibration validity end date and time keywords CVED0001 and
3) Add a CAL_QUAL specification if it is to be non-zero.
4) Add a FIDELITY specification if needed (FDLT0001).
"calindex" uses all of
these keyword values in building the new index files.