Search https://cxc.harvard.edu/ciao/
Contact the CXC HelpDesk
Last modified: 7 Feb 2022

URL: https://cxc.cfa.harvard.edu/ciao/threads/reprocessed/
/ ciao / threads / reprocessed /
Return to Threads Page: Top | All | Intro

A Note on Processing Versions

CIAO 4.16 Science Threads

Overview

Synopsis:

The CIAO Science Threads are designed to work with Chandra data that has been created using recent calibration data. In the past we have referred to this as "reprocessed" data, since early datasets have had to be reprocessed using the new calibration files.

There are two versions of "reprocessed" data that we talk about;

  1. data that has been reprocessed by the Chandra Data Archive, because new or improved data and algorithms have been released since the data was originally processed,

  2. data that has been reprocessed by a user, either with the chandra_repro script or manually using the acis_process_events or hrc_process_events tools, to apply the latest Calibration Database files.

Last Update: 7 Feb 2022 - Review for CIAO 4.14. Updated for Repro-5 and CALDB 4.9.6.

Contents

Standard Data Processing (SDP) Pipeline Versions

Checking the pipeline version

Download the sample data: 1843 (ACIS-I, G21.5-0.9)

unix% download_chandra_obsid 1843 evt2

The version naming convention of the standard data processing pipeline changed after version R4CU5UPD14 to the "DS" system, starting with DS 6.0.0. Any data with a processing version of 6.0.0 or higher is more recent than R4CU5UPD14 (the "DS" term is not included in the version string).

The processing version information is stored in the ASCDSVER keyword: 

unix% dmkeypar acisf01843N004_evt2.fits ASCDSVER echo+
10.9.2

For an older version of the same dataset: 

unix% dmkeypar acisf01843N001_evt2.fits ASCDSVER echo+
R4CU5UPD8.2

Checking the CALDB version

It is also useful to check the version of the CALDB used in the SDP pipeline, since the threads reference the CALDBVER header keyword when discussing processing changes: 

unix% dmkeypar acisf01843N004_evt2.fits CALDBVER echo+ 
4.9.4
[WARNING]
Warning

The CALDBVER header keyword is not updated by the CIAO tools. The value always reflects the version of the CALDB applied in standard data processing (SDP).

To find out which version of the CALDB you are running on your local system, use the check_ciao_caldb script: 

unix% check_ciao_caldb --latest
CALDB environment variable = /export/CIAOT/ciaot_install/20211019/ciao-4.14/CALDB
        which is a link to = /data/chandra_caldb/ciao4.12
             CALDB version = 4.9.6
             release date  = 2021-09-21T16:00:00 UTC
CALDB query completed successfully.
The CALDB installation is up to date.

The --latest flag checks the CALDB web site to see if your installation is up to date.

The information is also stored in the caldb_version.fits file: 

unix% dmlist "$CALDB/docs/chandra/caldb_version/caldb_version.fits[cols caldb_ver]" data
 
--------------------------------------------------------------------------------
Data for Table Block CALDBVER
--------------------------------------------------------------------------------

ROW    CALDB_VER

     1 4.1.0
     2 4.1.1
     3 4.1.2
     4 4.1.3
     5 4.1.4
     6 4.2.0
...
    75 4.9.2.1  
    76 4.9.3    
    77 4.9.4    
    78 4.9.5    
    79 4.9.6

This file is automatically updated each time the CALDB is upgraded on your system, so the final row always indicates the current version.

chandra_repro versions

After running chandra_repro 

unix% chandra_repro 1843 out=""

users will find that the ASCDSVER keyword has been updated to the current CIAO version: 

unix% dmkeypar acisf01843_repro_evt2.fits ASCDSVER echo+
CIAO 4.14

However as noted above, the CALDBVER keyword is not updated 

% dmkeypar acisf01843_repro_evt2.fits CALDBVER echo+
4.9.4

The version of the CALDB and the version of the chandra_repro script itself are both written into the output level2 event file header as COMMENT records: 

unix% dmlist acisf01843_repro_evt2.fits header,clean | egrep "COMMENT.*chandra_repro"
 --  COMMENT              chandra_repro version 01 December 2021
 --  COMMENT               chandra_repro was run with CALDB 4.9.6

What next?

Once you have checked the processing and CALDB versions used to create the archive data, checked the V&V report that is produced for each observation, and reviewed the Data Caveats section, you should reprocess your data to ensure that the latest calibration products have been applied.

History

22 Dec 2004 updated for CIAO 3.2/CALDB 3.0.0
01 Dec 2005 updated for CIAO 3.3/CALDB 3.2.0: screen output of caldb_version.fits
01 Dec 2006 updated for CIAO 3.4 and CALDB 3.3.0: screen output of caldb_version.fits
09 Jan 2008 updated for CIAO 4.0 and CALDB 3.4.2: screen output of caldb_version.fits; ObsID 1843 is now version N002
30 Dec 2008 updated for CIAO 4.1 and CALDB 4.1.0: screen output of caldb_version.fits
25 Jan 2010 updated for CIAO 4.2 and CALDB 4.2.0: example of using check_ciao_caldb; screen output of caldb_version.fits
10 Jan 2011 updated for CIAO 4.3 and CALDB 4.4.1: updates to screen output
03 Jan 2012 reviewed for CIAO 4.4 and CALDB 4.4.7: updates to screen output
03 Dec 2012 Review for CIAO 4.5; update file version and screen output
31 May 2013 Minor updates to distinguish what type of reprocessing is being discussed; added a What next? section.
10 Dec 2013 Review for CIAO 4.6. No changes.
22 Dec 2014 Review for CIAO 4.7; updated screen output
09 Feb 2021 Updated for CIAO 4.13; added note on getting CALDB version for chandra_repro outputs.
07 Feb 2022 Review for CIAO 4.14. Updated for Repro-5 and CALDB 4.9.6.
Return to Threads Page: Top | All | Intro
/ ciao / threads / reprocessed /