Data Analysis: Do I Need to Reprocess My Data?

Relax! The answer is often: "No, you do not need to reprocess!" Depending on the nature of the release, however, the response could be: "If you reprocess, you will gain X amount of accuracy." In rare cases we may also say: "All users should reprocess to take advantage of this improvement."

In order to determine whether or not to reprocess your particular dataset, there are two pieces of information needed:

1. The version of the software used to process the data in the first place, and the version of the CALDB used to create the data.
   The software and CALDB version information are stored in the file header in the ASCDSVER and CALDBVER keywords, respectively. The software version listed will either be the standard data processing (SDP, aka "the pipeline") version number (e.g. "6.6.0") or the version of CIAO used to create the data product (e.g. "CIAO 3.2 Thursday, December 2, 2004"). The "Note on Processing Versions"[http://cxc.harvard.edu/ciao/threads/reprocessed/] has more information.
2. A list of the changes in the new CIAO and CALDB releases. The CIAO and CALDB release notes contain a complete listing of all changes in each update. As of CIAO 3.2, the CIAO release notes also contain a special section called "How CIAO 3.2 and CALDB 3.0.0 Affect Your Analysis". This section of the release notes describes the tool and calibration changes that affect analyses in progress, i.e. any processing steps which should be run again for greater accuracy.

In the future, this information will be also organized by science category and instrument, such as "point source, ACIS-S, no grating". The page will be updated with relevant information at every CIAO and CALDB release, so users have a one-stop location to determine if an analysis task needs to be redone.

CIAO release notes: http://cxc.harvard.edu/ciao/releasenotes/CALDB release notes: http://cxc.harvard.edu/caldb/downloads/releasenotes.html

There are two extremes when discussing reprocessing: brand new and very old data. In this case, "new" and "old" are relative to the current version of SDP.

Data that a user has just received from the Chandra X-Ray Center - e.g. General Observing (GO) projects - has been processed with the most recent version of SDP. These data have been created with the newest tools and calibration. In some cases, it is better than what can be achieved in CIAO, as updates often get added to SDP sooner than to a CIAO release.
The only reason a user would reprocess this data is to change any of the default parameter settings used in SDP. For instance, an ACIS dataset could be reprocessed to look at the difference between CTI-corrected (the pipeline default) and CTI-uncorrected data.

At the other extreme is very old data. If a user is working with a dataset that was last processed a few years ago, it may be a better use of time to simply reprocess the data rather than work out what calibration files or tools have been changed since then.
References on how to reprocess data in CIAO:
Analysis Guides http://cxc.harvard.edu/ciao/guides/
Data Preparation threads http://cxc.harvard.edu/ciao/threads/data.html
For the most general case, refer to the "Create a New Level=2 Event File" http://cxc.harvard.edu/ciao/threads/createL2/

• CIAO 3.2 and CALDB 3.0.0: 15 December 2004
  The CIAO 3.2 software release contains a new tool for creating ACIS response matrices (mkacisrmf), four new ACIS hot pixel tools, and an enhanced ARDLIB to handle ACIS spatial contamination. It also includes the latest proposal planning tools for Chandra Cycle 7. CALDB 3.0.0 contains essential elements for CIAO 3.2 to work properly: new ACIS response, gain, and spatially-dependent CONTAM files for making RMFs, ARFs, GARFs, and exposure maps.
  
• CIAO 3.2.1 and CALDB 3.0.1: 10 February 2005
  The CIAO 3.2.1 patch contains updated versions of the acis_build_badpix and acis_run_hotpix tools. CALDB 3.0.1 includes new HRC-S QE and QE Uniformity files, as well as improved ACIS-S chip corner positions.

Please note that although they were released at the same time, the CIAO 3.2.1 patch does NOT require the CALDB 3.0.1 upgrade. The two releases function independently. See the release notes for complete details:
CIAO: http://cxc.harvard.edu/ciao/releasenotes/
CALDB: http://cxc.harvard.edu/caldb/downloads/releasenotes.html
CIAO and CALDB are available for download from: http://cxc.harvard.edu/ciao/download/
They are also available from the European mirror site:
http://ledas-cxc.star.le.ac.uk/ciao/download/

Elizabeth Galle and Antonella Fruscione, on behalf of the CIAO Development team