Data Analysis: Do I Need to Reprocess My Data?

Previous   Contents   Next



Data Analysis: Do I Need to Reprocess My Data?

This is the question that most users of the "Chandra Interactive Analysis of Observations" (CIAO) software are confronted with every time a new release of CIAO or the Chandra Calibration Database (CALDB) appears on the horizon. It is an even more pressing issue if a paper is about to be submitted with data analysed with an "old" version of the software or "old" calibration files.

Relax! The answer is often: "No, you do not need to reprocess!" Depending on the nature of the release, however, the response could be: "If you reprocess, you will gain X amount of accuracy." In rare cases we may also say: "All users should reprocess to take advantage of this improvement."

In order to determine whether or not to reprocess your particular dataset, there are two pieces of information needed:

  1. The version of the software used to process the data in the first place, and the version of the CALDB used to create the data.
    The software and CALDB version information are stored in the file header in the ASCDSVER and CALDBVER keywords, respectively. The software version listed will either be the standard data processing (SDP, aka "the pipeline") version number (e.g. "6.6.0") or the version of CIAO used to create the data product (e.g. "CIAO 3.2 Thursday, December 2, 2004"). The "Note on Processing Versions"[http://cxc.harvard.edu/ciao/threads/reprocessed/] has more information.
  2. A list of the changes in the new CIAO and CALDB releases. The CIAO and CALDB release notes contain a complete listing of all changes in each update. As of CIAO 3.2, the CIAO release notes also contain a special section called "How CIAO 3.2 and CALDB 3.0.0 Affect Your Analysis". This section of the release notes describes the tool and calibration changes that affect analyses in progress, i.e. any processing steps which should be run again for greater accuracy.

In the future, this information will be also organized by science category and instrument, such as "point source, ACIS-S, no grating". The page will be updated with relevant information at every CIAO and CALDB release, so users have a one-stop location to determine if an analysis task needs to be redone.

CIAO release notes: http://cxc.harvard.edu/ciao/releasenotes/CALDB release notes: http://cxc.harvard.edu/caldb/downloads/releasenotes.html

When reprocessing is the best or worst decision

There are two extremes when discussing reprocessing: brand new and very old data. In this case, "new" and "old" are relative to the current version of SDP.

Data that a user has just received from the Chandra X-Ray Center - e.g. General Observing (GO) projects - has been processed with the most recent version of SDP. These data have been created with the newest tools and calibration. In some cases, it is better than what can be achieved in CIAO, as updates often get added to SDP sooner than to a CIAO release.
The only reason a user would reprocess this data is to change any of the default parameter settings used in SDP. For instance, an ACIS dataset could be reprocessed to look at the difference between CTI-corrected (the pipeline default) and CTI-uncorrected data.

At the other extreme is very old data. If a user is working with a dataset that was last processed a few years ago, it may be a better use of time to simply reprocess the data rather than work out what calibration files or tools have been changed since then.
References on how to reprocess data in CIAO:
Analysis Guides http://cxc.harvard.edu/ciao/guides/
Data Preparation threads http://cxc.harvard.edu/ciao/threads/data.html
For the most general case, refer to the "Create a New Level=2 Event File" http://cxc.harvard.edu/ciao/threads/createL2/

Recent CIAO and CALDB Releases

Please note that although they were released at the same time, the CIAO 3.2.1 patch does NOT require the CALDB 3.0.1 upgrade. The two releases function independently. See the release notes for complete details:
CIAO: http://cxc.harvard.edu/ciao/releasenotes/
CALDB: http://cxc.harvard.edu/caldb/downloads/releasenotes.html
CIAO and CALDB are available for download from: http://cxc.harvard.edu/ciao/download/
They are also available from the European mirror site:
http://ledas-cxc.star.le.ac.uk/ciao/download/

Elizabeth Galle and Antonella Fruscione, on behalf of the CIAO Development team




Previous   Contents   Next




cxchelp@head.cfa.harvard.edu