Chandra CalDB4 is a new calibration database structure and
interface, that is being introduced with the release
of CIAO 4.1 on December 15, 2008.
CalDB4 conforms to the general FITS paradigm established in the
pre-existing HEASARC CalDB system,
and can be maintained in the same directory space. However, CALDB4 is
not compatible with Chandra CalDB 3.5.0
or any earlier versions of the Chandra CalDB, and may not be stored in
the same directory tree as those earlier versions.
The specific version of CalDB4 that will be released to the
public is "4.1.0,"
to coincide with CIAO "4.1," and hopefully prevent confusion that would
result if users were to believe that
CALDB "4.0.0" is compatible with CIAO "4.0" (released December 2007),
which it definitely is not.
The new CalDB will only work with CIAO 4.1 or later versions.
User Documentation:
"Introduction to CalDB4"
http://cxc.harvard.edu/caldb4/about_CALDB4/Introduction.html
The CALDB4 software and directory/index structure has been defined
to address these additional general requirements:
1) To provide an index and directory structure that is applicable to an
arbitrary mission configuration, and controllable by CalDB managers.
2) To provide the necessary automation of calibration parameter
identification and evaluation.
3) To eliminate hard-coding of parameter names and/or values wherever
possible in CIAO 4.1, and later in the DS.
4) To accommodate mission-independent tooling in the long term.
CalDB 4.1.0 (so designated to correspond to CIAO 4.1 to prevent
confusion) implements a new, flexible index system,
with keyword parameter/index column control set by the CalDB manager.
In addition to the flexible indexing, the
software interface employs a hand-shaking implementation with the CalDB
to identify necessary parameters for the
requested calibration data, and to supply the necessary values as
available from the input data file headers, or as supplied
by the user when necessary. All that is required of the tool (or the
user) to begin selection of calibration data is to know
the mission (CHANDRA) and the name of the particular calibration data
needed (such as DET_GAIN or QE). The context
of the search determines the other required parameter values to get the
right data.
The developer or the user do not need to know which parameters are
required to select the data; the interface will request
the appropriate information automatically, usually from an events, pha, pha2, or pi data file.
A snapshot copy of the CALDB4 requirements specifications is
available here.
This document currently specifies the new flexible index system,
CalDB configuration file "caldb.config," the CalDB query interface and
algorithm, and the index builder software.
CIAO 4.1 incorporates the new caldb4 software interface and
associated library (See ECR-2008-026, concurrent),
and supports the index builder tool "calindex".
CIAO 4.1 and CalDB 4.1.0 are mutually required for full functionality
of both.
CALDB4 and the specific implementation of CALDB 4.1.0 have been
built to address Chandra CalDB requirements
and to address the above general specifications, with the ongoing tacit
approval of HEASARC CalDB personel.
HEASARC has been involved in the early specification and investigation
of new requirements. Our general progress
has been included in annual reviews of the HEASARC CXC support
contract. The CALDB4
requirements and implementations are to serve as a possible new HEASARC
CalDB standard, with incumbent updates
to the OGIP standard taken as needed. We have built CALDB4 for Chandra,
but upon sufficient internal review and
approval, intend to deliver the software and tools for CALDB4 to
HEASARC pending verification of all specified
requirements and possibly some interfacing additions to be specified
later.
The CalDB configuration file locates the individual CalDB trees and
index files for specific compliant
missions. The file is located at $CALDB/software/tools, and must
contain the specs for all missions included
in the $CALDB directory tree for all the data to be accessible through
the software interface. "caldb.config" is an ASCII
file.
In the old specification CHANDRA's caldb configuration appears as:
#CHANDRA
CHANDRA ACIS CALDB data/chandra/acis caldb.indx CALDB data/chandra/acis
CHANDRA EPHIN CALDB data/chandra/ephin caldb.indx CALDB
data/chandra/ephin
CHANDRA HRC CALDB data/chandra/hrc caldb.indx CALDB data/chandra/hrc
CHANDRA PCAD CALDB data/chandra/pcad caldb.indx CALDB data/chandra/pcad
CHANDRA SIM CALDB data/chandra/sim caldb.indx CALDB data/chandra/sim
CHANDRA TEL CALDB data/chandra/tel caldb.indx CALDB data/chandra/tel
The above information specifies the MISSION: CHANDRA and the
INSTRUMENTs available to Chandra
under the old paradigm, where all calibration data are separated by the
INSTRUME (keyword) to which they apply.
ACIS, EPHIN, HRC, PCAD, and SIM all correspond to legitimate
instruments. TEL had to be added to
subsume the calibration data for PIXLIB, HRMA, and the GRATINGs.
In the new caldb.config for Chandra CalDB 4.1.0:
#CHANDRA
CHANDRA ACIS CALDB data/chandra/acis caldb.indx CALDB data/chandra/acis
CALDB data/chandra/acis key.config
CHANDRA EPHIN CALDB data/chandra/ephin caldb.indx CALDB
data/chandra/ephin CALDB data/chandra/ephin key.config
CHANDRA HRC CALDB data/chandra/hrc caldb.indx CALDB data/chandra/hrc
CALDB data/chandra/hrc key.config
CHANDRA PCAD CALDB data/chandra/pcad caldb.indx CALDB data/chandra/pcad
CALDB data/chandra/pcad key.config
CHANDRA SIM CALDB data/chandra/sim caldb.indx CALDB data/chandra/sim
CALDB data/chandra/sim key.config
CHANDRA DEFAULT CALDB data/chandra/default caldb.indx CALDB
data/chandra/default CALDB data/chandra/default key.config
CHANDRA DEFAULT CALDB data/chandra/pimms caldb.indx CALDB
data/chandra/pimms CALDB data/chandra/pimms key.config
Note that each row contains three more elements, corresponding to
the location and name of the "key.config" file, which gives the keyword
configuration of the new flexible INDEX files that are built by "calindex".
Also, the TEL branch, which
was not entirely appropriate for storing HRMA, PIXLIB, and GRATING
files, has been eliminated. Instead,
there is the designation of DEFAULT to replace (and break) the
now-outdated INSTRUME separation paradigm for CalDB files. When
calibration data for mission elements do not correspond to any
particular INSTRUME value, these elements may be indexed and stored
under
one or more directories designated as DEFAULT in caldb.config. So, it
is no longer necessary to address HRMA or GRATING files
in the code, by hardcoding "TEL" as the instrument associated with HRMA
or HETG, for example.
2. Directory structure: (Full table listing is included here
only
for completeness.)
All branches have relatively predictable structures, with one
exception: the "pimms/" branch, which
is a special case of files used only to set up the PIMMS effective area
files during the software build. Under
pimms/, the files are divided between those for ACIS configurations,
and those for HRC.
| $CALDB/data/chandra/ |
||
| acis/ |
||
| "caldb.indx" file for ACIS |
||
| "key.config" file for ACIS
caldb.indx |
||
| index/ (Stores versioned index
files linked to "caldb.indx" |
||
| 2d_psf/ |
||
| badpix/ |
||
| bkgrnd/ |
||
| contam/ |
||
| cti/ |
||
| dead_area/ |
||
| det_gain/ |
||
| disp_reg/ |
||
| evtsplt/ |
||
| fef_pha/ |
||
| grade/ |
||
| gti_lim/ |
||
| lsfparm/ |
||
| osip/ |
||
| p2_resp/ |
||
| qe/ |
||
| qeu/ |
||
| t_gain/ |
||
| default/ |
||
| "caldb.indx" file for DEFAULT |
||
| "key.config" file for DEFAULT
caldb.indx |
||
| index/ |
||
| aimpts/ |
||
| axeffa/ |
||
| geom/ |
||
| greff/ |
||
| obi_tol/ |
||
| reef/ |
||
| sgeom/ |
||
| sky/ |
||
| tdet/ |
||
| vignet/ |
||
| wpsf/ |
||
| ephin |
||
| "caldb.indx" file for EPHIN |
||
| "key.config" file for EPHIN
caldb.indx |
||
| index/ |
||
| geom/ |
||
| hrc/ |
||
| "caldb.indx" file for HRC |
||
| "key.config" file for HRC
caldb.indx |
||
| 2d_psf/ |
||
| amp_sf_cor/ |
||
| badpix/ |
||
| eftest/ |
||
| fptest/ |
||
| gaplookup/ |
||
| gmap/ |
||
| gti_lim/ |
||
| lsfparm/ |
||
| qe/ |
||
| qeu/ |
||
| rmf/ |
||
| sattest/ |
||
| tapringtest/ |
||
| tgmask2/ |
||
| tgpimask2/ |
||
| pcad/ |
||
| "caldb.indx" file for PCAD |
||
| "key.config" file for PCAD
caldb.indx |
||
| index/ |
||
| align/ |
||
| ccd_char/ |
||
| ccd_resp/ |
||
| dark_curr/ |
||
| fdc/ |
||
| gyro_sfma/ |
||
| iru_char/ |
||
| rwa_bspd/ |
||
| pimms/ |
||
| "caldb.indx" file for PIMMS | ||
| "key.config" file for PIMMS caldb.indx | ||
| index/ |
||
| acis/ |
||
| hrc/ |
||
| sim/ |
||
| "caldb.indx" file for SIM |
||
| "key.config" file for SIM
caldb.indx |
||
| index/ |
||
| det_pos/ |
||
| det_poscorr/ |
3. Mandatory Index Columns:
The following columns are required in all CalDB index files. In some
cases, such as the CAL_DIR and
the CAL_FILE, the requirement is obvious, since the location, filename,
extension number, and calibration
code name are primary to a CalDB structure. Some designations are
required for backward compatibility
with previously existing HEASARC standards, which the requirements
document above specifies and
acknowledges must be maintained. However, despite being mandatory, they
do not impinge on the
flexibility in building a CalDB for an arbitrary mission.
The mandatory columns and associated keywords are given in the
Requirements Document, Table 1, pp. 4 and 5.
I list the columns here for completeness, with definitions:
| Index
file Column Name |
Equivalent
Header Keyword, if any |
Description |
| CAL_DEV |
”ONLINE” for calibration files
present in the CALDB directory tree. ”OFFLINE” or ”<device name>” for calibration files not present in the CALDB tree. See below for information on constructing the fully qualified path to the calibration file from CAL_DEV, CAL_DIR, and CAL_FILE. |
|
| CAL_DIR |
The path to the directory
containing the calibration file specified by CAL_FILE. |
|
| CAL_FILE |
The name of the calibration file. |
|
| CAL_CNAM |
CCNM* (CCNM0001) |
The name of the type of
calibration data. Used as the basis for identifying the type of data being queried. |
| CAL_CBD |
CBD* (varies CBD10001 to CBD90001) |
The calibration boundary
conditions. These are additional query constraints that are either sufficiently infrequent that including a query column in the index file is inappropriate, or that are not readily represented as query columns in the index file (examples would be numeric ranges). |
| CAL_XNO |
The FITS extension number that
contains the calibration data. May be zero if the calibration data are stored in the primary HDU (for example, an image). <This is 1 less than the data model block number.> |
|
| CAL_QUAL |
CAL_QUAL |
The “quality” of the
calibration, defined as follows: 0 = “Good. No known issues.”; 1 = “Some calibration issues known. May be default for processing or analysis.”; 2 = “Calibration issues identified and replacement needed or anticipated. May be used, but will be superseded in the near future.”; 3 = “Former operational mode, good enough for processing or analysis of archival data taken in that mode. Significant issues known; however recalibration of this mode is unlikely.”; 4 = “Strongly deprecated or superseded. Issue a warning to this effect if used. Reprocessing needed if these data were used in earlier processing; same for earlier analysis. Kept to enable selection of these data as related files.”; 5 = “ Bad. Never to be used again. To be retired. A better option is available in the CalDB, including related files data.”; 9 = “Dataset no longer exists.”. |
| CAL_DATE |
The UTC date when this entry was
added to the index file, in yyyy-mm-dd format. |
CALDB4 has adopted the above from the previous HEASARC CalDB
specification, and has added none to this list.
The calindex software has
added the designation for the CAL_QUAL Equivalent header keyword as
"CAL_QUAL",
which it now reads and puts the value into the index file.
4. Predefined Index File Optional
Columns
Certain columns have been established by the accumulated CalDB history
to contain certain types of data and are
interpreted by software (or not) in particular ways. CALDB4 has defined
these as "optional", but having a particular
designation and type. In so doing, we have added a few of our own new
columns in CalDB4, some of which are in
analogous to the pre-existing ones. These are in Table 2 of the
requirements document,
pp. 6 and 7.
| INDEX
FILE COLUMN NAME |
EQUIVALENT HEADER KEYWORD |
DESCRIPTION |
| CAL_CLAS |
CCLS* |
The calibration file class, one
of “PCF”, ”BCF”, ”CPF”, or “USR”. Not interpreted by software. |
| CAL_DTYP |
CDTP* |
The type of the calibration
file, one of ”DATA”, ”TASK”, or “FEF”. Not interpreted by software. |
| CAL_DESC |
CDES* |
Short string description of the
calibration data. Not interpreted by software. |
| CAL_VSD |
CVSD* |
The beginning valid date of the
calibration, in yyyy-mm-dd format. This column is interpreted by software. |
| CAL_VST |
CVST* |
The beginning UTC valid time of
the calibration, in hh:mm:ss format. This column is interpreted by software. |
| REF_TIME |
The MJD corresponding to
CAL_VSD/CAL_VST. This column is interpreted by software. |
|
| CAL_VED |
CVED* |
The ending UTC valid date of the
calibration, in yyyy-mm-dd format. This column is interpreted by software. |
| CAL_VET |
CVET* |
The ending UTC valid time of the
calibration, in hh:mm:ss format. This column is interpreted by software. |
| END_TIME |
The MJD corresponding to
CAL_VED/CAL_VET. This column is interpreted by software. |
|
| FIDELITY |
FDLT* |
Numeric “fidelity” of
calibration data. May be used to select amongst different calibrations of the same quality that satisfy the query requirements. The ordering of the FIDELITY is defined such that numerically greater values correspond to higher fidelity calibration data. This column is interpreted by software. |
NOTES: CAL_CLAS, CAL_DTYP, AND CAL_DESC are all
mandatory columns and keywords in the existing
HEASARC standard. They are not mandatory in CALDB4.
CAL_VED, CAL_VET and END_TIME are analogous to the CAL_VSD, CAL_VST,
and REF_TIME counterparts.
These are not supported in the existing HEASARC standard, but are added
in CALDB4.
5. Keyword Configurations and "key.config":
Each branch of CalDB 4.1.0 has a particular specification for the
index file in "key.config" as listed in
the table above. "key.config," like "caldb.config," is always an ASCII
file. In CalDB 4.1.0, the key.config
files reside just along side of the "caldb.indx" links. A key.config
file looks like the following:
# Time-stamp: <2008-11-10 10:38:02 caldbmgr>
# CHANDRA key.config table
# for INSTRUME = <Manager specifies this>
#colName dataType format
hdrKey queryCol nullVal
# ---------------------------------------------------------
TELESCOP string
a10 TELESCOP
yes ""
INSTRUME string
a10 INSTRUME
yes "NONE"
CAL_DESC string
a70 CDES*
no "NONE"
CAL_VSD
string a10
CVSD*
no ""
CAL_VST
string
a8
CVST*
no ""
REF_TIME double
d
""
no 0.0
CAL_VED string
a10 CVED*
no ""
CAL_VET string
a8
CVET*
no ""
END_TIME double
d
""
no 0.0
FIDELITY double
d
FDLT*
no 0.0
The pound sign designates a comment, ignored by calindex.
The columns are the index file column name, data type, format, the
corresponding header keyword in the CalDB files,
the specification of whether the column is a query column, and
null value. The column names are added to the index by
calindex, in addition to the
mandatory columns described above. When a column is a "query" column,
the value corresponding
to the "header keyword" designated thereto, is searched for in the
input file headers whenever a CalDB call is made to
the index in question. This then is part of the built-in hand-shaking
between the interface and the CalDB index structure,
to determine what is important in the selection of particular CalDB
data.
As indicated in the caldb.config specification above for CHANDRA,
there are seven distinct branches now in CalDB 4.1.0,
with corresponding caldb index and key.config files. The specific
key.config and resulting caldb.indx files will appear at
the following locations, once CALDB4 has been installed locally, and
$CALDB has been set:
ACIS:
$CALDB/data/chandra/acis/key.config
$CALDB/data/chandra/acis/caldb.indx
DEFAULT:
$CALDB/data/chandra/default/key.config
$CALDB/data/chandra/default/caldb.indx
EPHIN:
$CALDB/data/chandra/ephin/key.config
$CALDB/data/chandra/ephin/caldb.indx
HRC:
$CALDB/data/chandra/hrc/key.config
$CALDB/data/chandra/hrc/caldb.indx
PCAD:
$CALDB/data/chandra/pcad/key.config
$CALDB/data/chandra/pcad/caldb.indx
PIMMS:
$CALDB/data/chandra/pimms/key.config
$CALDB/data/chandra/pimms/caldb.indx
SIM:
$CALDB/data/chandra/sim/key.config
$CALDB/data/chandra/sim/caldb.indx
6. CHANGES to CalDB files in CALDB 4.1.0:
The data in CALDB 4.1.0 are largely the same as the data in CalDB
version 3.5.1; in fact only one file has been added
and it contains no new data, but simply accommodates a special RMF
building case (See ECR_2008_028). Pre-existing
CalDB filenames are all preserved, so that the transition in analysis
may be seamless.
The only changes made to CalDB files from 3.5.1 to 4.1.0 are to the various HDU headers. Such changes include:
1) Eliminate unnecessary boundary conditions from the CalDB index
files (such as ENERGY ranges, CCD_ID
specifications with they are not needed, or redundant specifications
for GRATING or GRATTYPE, which are now handled as index columns in
CalDB 4.1.0.) These unnecessary
boundary conditions are moved from the CBD* keyword series to a new
series CBO* for Optional boundary
conditions. They remain in the file headers, but are not interpreted by
calindex.
2) Add calibration validity end date and time keywords CVED0001 and
CVET0001
3) Add a CAL_QUAL specification if it is to be non-zero.
4) Add a FIDELITY specification if needed (FDLT0001).
"calindex" uses all of
these keyword values in building the new index files.