Last modified: 22 Feb 2022

URL: https://cxc.cfa.harvard.edu/ciao/threads/intro_data/

Introduction to the Data Products

CIAO 4.16 Science Threads


Overview

Synopsis:

The CXC performs standard data processing (SDP) on all Chandra science data; this procedure is commonly referred to as "the pipeline." The processing runs in several stages or "levels", each of which is built on the results of the preceding level. A well-defined set of data products are the result of each level. For most cases, the level 2 (L2) - or L1.5 for grating data - data files should be used. The level 3 (L3) products are available from the Chandra Source Catalog (CSC).

This thread examines the primary and secondary Chandra data products which were obtained by following the How to Download Chandra Data from the Archive thread, which uses ObsID 1843 (ACIS-I, G21.5-0.9) as the sample data. Filenames given in brackets - e.g. [hrc_dtf1.fits] - indicate a data product that is not created for ACIS imaging observations; this makes the thread applicable to any standard Chandra data distribution.

The data returned by the Chandra archive is usable, but we strongly suggest that for each observation you have downloaded you

  1. review the V&V report to see if anything untoward happened during the observation,

  2. and then reprocess the data (e.g. with chandra_repro), to ensure that the latest calibration products have been used.

This steps should be done before you start your analysis.

Related Links:

Last Update: 22 Feb 2022 - Review for CIAO 4.14. Updated for Repro5/CALDB 4.9.6.


Contents


Archive versions

When a Chandra observation is taken, it is processed through the Standard Data Processing (SDP) pipeline, and the results stored in the Chandra archive. Occasionally the Chandra X-ray Center determines that there have been enough improvements to the software, or changes in the calibration products, that the archive - or a significant fraction of it - has to be reprocessed. This means that new version of the files will be created; so, if you downloaded an observation a long time ago, it is worth checking the archive to see if a new version is available. At the time of writing (February 22, 2022), the last reprocessing run was Repro V.

The current processing status, and history, of an observation can be found from the Chandra Observations Processing Status page. For ObsId 1843, the page looks like Figure 1 as of February 2022.

Figure 1: Archive processing status of ObsId 1843

[Thumbnail image: The observation has been processed three times.]

[Version: full-size]

[Print media version: The observation has been processed three times.]

Figure 1: Archive processing status of ObsId 1843

The final table on the page shows that the data has been processed four times, the last time being in March 2021. This is why the "version number" for many of the files is N004 - e.g. acisf01843N004_evt2.fits.

The note on processing versions thread explains the version names used for the SDP pipeline, and how to check what version was used to create a particular dataset.


ObsID Directory

The ObsID directory is the top level in the directory tree created when the tarfile is unpacked. It contains two files and two data directories:

unix% pwd
/intro_data/1843

unix% ls
00README
axaff01843N003_VV001_vv2.pdf
oif.fits
primary/
secondary/
V&V report
axaff01843N003_VV001_vv2.pdf

All standard data products are checked by a CXC scientist before release to ensure data quality and to investigate the cause of any exposure losses or other anomalies. This process is known as "verification and validation", or V&V for short. The V&V report is distributed in PDF format with the data files.

This file contains a summary report of the V&V report. The full report is in a second PDF file (axaff01843N003_VV001_vvref2.pdf), which is packaged in the secondary data directory.

Users should review this information before beginning the analysis to ensure that there aren't any caveats from the V&V scientist.

Observation index file
oif.fits

The observation index file contains a summary of the data products associated with an observation. It is equivalent to a structured directory listing with descriptive information about the observation.

README
00README

Each observation includes an ASCII README file that provides information on how to cite Chandra data in publications, how to acknowledge usage of CIAO and Sherpa, and how to use the Chandra X-ray Center as a long term repository for aggregated high-level science data products.

In particular, the CXC encourages users to include Dataset Identifiers in their publications.

In the following sections, we describe the contents of the primary and secondary data directories.


Primary Directory

The data products are arranged such that all products necessary for most analysis (e.g. the CIAO threads) are in the primary directory.

unix% cd primary/
unix% pwd
/intro_data/1843/primary

unix% ls
acisf01843N004_cntr_img2.fits
acisf01843N004_cntr_img2.jpg
acisf01843N004_evt2.fits
acisf01843N004_full_img2.fits
acisf01843N004_full_img2.jpg
acisf01843_000N004_bpix1.fits
acisf01843_000N004_fov1.fits
orbitf084197100N001_eph1.fits
pcadf01843_000N001_asol1.fits
[hrc_dtf1.fits]
[acis_plt2.jpg]
[acis_pha2.fits]
[NOTE]
Note

For a small number of observations the output will look different to above, because the observation was taken in interleaved mode - in which case there are two versions of certain files, labeled with e1 and e2, - or it is a multiple observation interval observation - when the level 1 and 1.5 data products can have multiple versions.

The majority of these files can be grouped into three categories:

Images

Two images are produced for every dataset: a full-field sky image (full_img2.jpg) and a high-resolution image of the center of the field (cntr_img2.jpg). FITS images are only created for imaging observation (no gratings).

Images are not created for ACIS operating in continuous clocking mode.

Imaging observations also have a full field image with the source candidates overlaid (src_img2.jpg). [The source images are no longer created as of SDP version DS 8.4 (June 2011).]

Grating observations include an image of each spectral component plotted as wavlength vs counts (heg_plt2.jpg, meg_plt2.jpg, and/or leg_plt2.jpg). [The grating images are no longer created as of SDP version DS 8.0 (July 2009).]

Remove those three sets of files from the directory listing and you have the primary data products necessary for most types of analysis:

acisf01843N004_evt2.fits
acisf01843_000N004_bpix1.fits
acisf01843_000N004_fov1.fits
orbitf084197100N001_eph1.fits
pcadf01843_000N001_asol1.fits
[hrc_dtf1.fits]
[acis_pha2.fits]
Level 2 event file
evt2.fits

The level 2 event file is the most important data product you receive. This file is created from the level 1 event list by filtering on the good time intervals (GTI) and status bits. The result is a list of events that is suitable for use in data analysis.

When new calibration is released, it is often necessary to remake the level 2 event file in order to apply the changes. This process is explained in the Create a New Level 2 Event File thread.

Source list
src2.fits

A source candidate list. These detections are obtained from a single run of celldetect with S/N threshold equal to 3 and should not be used directly in your analysis. Users interested in spatial analysis should perform their own detection runs with one of the three Chandra Detect tools: celldetect, wavdetect, and vtpdetect. The Detect Manual describes which tool is suited to a particular analysis.

For instructions on displaying the source list over the data, refer to the Using the Output of Detect Tools thread.

This file is only created for imaging observations.

[These files are no longer created as of SDP version DS 8.4 (June 2011).]

Bad pixels
bpix1.fits

A list of pixels identified as "bad"; criteria for flagging a pixel are listed in the Bad Pixels dictionary entry. Any tool that reads this file will exclude the bad pixels from its calculations.

The Reprocessing Data to Create a New Level=2 Event File thread contains more information, including instructions on how to make a new ACIS bad pixel file, if necessary. The New Observation-Specific HRC Bad Pixel File thread explains when and how to make a new HRC bad pixel file.

Sky field of view
fov1.fits

The field-of-view file contains regions which describe the edges of each instrument chip (one polygon per chip). This region file can be used to filter event files or images to contain specific chips.

This file has been created since SDP version 6.9.0. The skyfov tool may be used to create an fov1.fits file for observations with an earlier processing version.

These files are used in the Chandra FootPrint service for searching the archive of publicly-available observations.

Orbit ephemeris
orbit_eph1.fits

The definitive orbit ephemeris information for the observation. This file is required if you wish to barycenter-correct the observation times in order to account for the difference in photon arrival times as the Earth and Chandra move around the Sun.

The orbit ephemeris is also needed if you are analyzing an object which was moving with respect to the observer when observed, e.g. a planet or comet.

Aspect solution
pcad_asol1.fits

The aspect solution describes the orientation of the telescope as a function of time. The detected position of an event and the corresponding telescope aspect are combined for an accurate determination of the celestial position of that event.

Since an aspect solution file is created for each stable aspect interval, there may be more than one pcad_asol1.fits file for an observation. All the files must be used whenever a tool requires the aspect solution as input.

Detailed information on the aspect solution files is available from the the Aspect Solution why topic.

Dead time factor
dtf1.fits

The dead time correction factors as a function of time. For ACIS observations, the dead time correction is recorded in the DTCOR header keyword. Since the HRC dead time correction factor varies during the observation, however, the value of DTCOR represents an average and the individual values are stored in the dtf1.fits file.

The Computing Average HRC Dead Time Corrections thread explains when you might need to recompute the deadtime statistics for your observation.

This file is only created for HRC observations.

Type II PHA / Spectra
pha2.fits

The PHA file contains a spectrum for one grating part and order in each row of the file. The number of rows varies depending on the detector and grating combination: e.g. ACIS/HETG data has twelve rows while HRC/LETG only has two.

Detailed information on the pha2.fits file is available from the Examining Grating Spectra and Regions thread.

This file is only created for grating observations.


Secondary Directory

If you are interested in reprocessing your data, you will also need the level 1 files, which are in the secondary directory.

unix% cd ../secondary/
unix% pwd
/intro_data/1843/secondary

unix% ls
acisf01843_000N004_evt1.fits
acisf01843_000N004_flt1.fits
acisf01843_000N004_msk1.fits
acisf01843_000N004_mtl1.fits
acisf01843_000N004_stat1.fits
acisf084271178N004_0_bias0.fits
acisf084271178N004_1_bias0.fits
acisf084271178N004_2_bias0.fits
acisf084271178N004_3_bias0.fits
acisf084271178N004_4_bias0.fits
acisf084271178N004_5_bias0.fits
acisf084272477N004_pbk0.fits
axaff01843N003_VV001_vvref2.pdf
[acis_evt1a.fits  -or-  hrc_evt1a.fits]
[hrc_std_flt1.fits]
[hrc_std_dtfstat1.fits]
aspect/
ephem/

The V&V report - axaff01843N003_VV001_vvref2.pdf - is explained in the ObsID Directory section.

Aspect offsets
aoff1.fits

A set of offsets versus time (e.g. ra_off=(ra-ra_nom)) . The one piece of remaining useful info is information about gaps in the aspect records. These are derived from the aspect solution files (pcad_asol1.fits, primary directory) for the observation.

Most users will never need this file in their analysis, instead using the aspect solution files directly.

[This file will no longer created as of SDP version DS 8.4.1 (October 2011).]

Level 1 event file
evt1.fits

The level 1 event file contains all the events recorded for the observation. While many of these events have a status bit set to flags them as "bad", none of the information has been removed. This file is filtered on GTIs and status bits to create the level 2 event file.

The evt1.fits file is also the starting point for reprocessing your data, as explained in the Create a New Level 2 Event File thread.

Level 1.5 event file (optional)
evt1a.fits

The ACIS or HRC Level 1.5 event file is created for observations that used the gratings (LETG or HETG). The file contains additional columns related to the gratings including the grating order , which arm (eg HEG vs MEG), and the resolved grating wavelength. All these values are based on the zeroth order location used in the processing.

The zeroth order location the regions assigned grating coordinates is contained in the REGION extension which is appended to this file.

Good time intervals
flt1.fits / std_flt1.fits

The GTI information for the observation, e.g. the start and stop times of all accepted time intervals over the observation. The major contributor to creating GTIs is information about when there is aspect data and when that aspect data is good. When the event file is filtered, the GTIs are stored as extensions of the data file, creating a dynamic record of the time filters applied to the data.

Note that this file is named slightly differently for ACIS (flt1.fits) and HRC (std_flt1.fits) observations.

Mask
msk1.fits

The mask file records the valid part of the detector element - ACIS CCD or HRC plate - used for the observation (i.e. the portion for which events can be telemetered). The active portion of an element may be smaller than the default regions if an observation was performed using subarrays or custom windows. This information is used when creating response files, such as ARFs.

Mission timeline
mtl1.fits

The mission timeline (MTL) consists of all time series data that affects the quality of the data, such as aspect and ephemeris data. This information is used to construct the GTI limits in the flt1.fits file. The MTL ahelp file has more information on the mission timeline.

This file is required by acis_process_events to apply the temperature-dependent CTI calibration to ACIS data.

Alignment offsets
soff1.fits

The science instrument module offsets versus time. These are derived from the aspect solution files (pcad_asol1.fits, primary directory) for the observation. The soff1.fits file is binned, so there's not a one-to-one match to the rows in the pcad_asol1.fits files.

Most users will never need this file in their analysis, instead using the aspect solution files directly.

[These files are no longer created as of SDP version DS 8.4 (June 2011).]

Exposure statistics
stat1.fits

The exposure statistics file contains up to seven extensions, each pertaining to the sequence of ACIS CCD exposures processed and recorded into telemetry.

Most users will never need this file in their analysis.

This file is only created for ACIS observations.

Dead time factor statistics
std_dtfstat1.fits

The dead time factor statistics information is derived from the HRC deadtime factor file (dtf1.fits, primary directory) for the observation.

Most users will never need this file in their analysis.

This file is only created for HRC observations.

Bias map(s)
bias0.fits

When the bad pixel list (bpix1.fits) is created, each bias map is searched for pixels whose bias values are either too low or too high. There is one bias map for each ACIS chip that was used for the observation.

These files are only created for ACIS observations.

Parameter block
pbk0.fits

The parameter block file is needed in conjunction with the bias maps when creating a new bad pixel list. It is used to determine observational parameters, such as which CCDs are active, the READMODE and DATAMODE, etc.

This file is only created for ACIS observations.

Aspect Subdirectory

The secondary/aspect subdirectory contains the aspect quality file (aqual1.fits) and aspect solution files which are based on aspect determined by the on-board computer (osol1.fits):

unix% cd aspect/
unix% pwd
/intro_data/1843/secondary/aspect

unix% ls -1
pcadf084265646N004_osol1.fits
pcadf084271087N004_aqual1.fits
pcadf084272206N004_osol1.fits
pcadf084278602N004_osol1.fits

Most users will never need these files in their analysis. Ground-based aspect analysis, which produces the asol1.fits files (primary directory), gives a better product because more information is available.

There is a special case when the osol1.fits files may be used. The pcad_asol1.fits files are only produced when Chandra is in Kalman lock and on the target. The osol1.fits, however, are always produced. Since HRC is "always on" when it's the chosen detector, it records events while it is slewing between targets; these events are included in the evt1.fits file. There is no Kalman lock aspect solution for these events,which is why sources in HRC evt1.fits files often have "tails". If you really want to use those extra events that were taken as the telescope was moving into position, you can in principle use the osol1.fits files for the analysis.

Observations that requested the aspect camera also acquire optical monitor data of the target source will also contain adat71.fits files which are used in the ACA Montior Data thread.


Ephem Subdirectory

The secondary/ephem subdirectory contains the solar (solar_eph1.fits) and lunar (lunar_eph1.fits) ephemeris files, as well as the viewing angles file (angles_eph1.fits):

unix% cd ../ephem/
unix% pwd
/intro_data/1843/secondary/ephem

unix% ls -1
anglesf01843_000N004_eph1.fits
lunarf084197100N001_eph1.fits
solarf084197100N001_eph1.fits

Most users will never need these files in their analysis. When a tool requires the ephermeris information, it is referring to the orbit ephemeris (orbit_eph1.fits) from the primary directory.


What now?

Once you have checked the V&V report for the observation, you should reprocess the data. This ensures that the latest calibration products have been used, and is often one of the first things we ask users of the Chandra HelpDesk.


History

05 Jul 2005 original version, new for CIAO 3.2
01 Dec 2005 reviewed for CIAO 3.3: no changes
01 Dec 2006 updated for CIAO 3.4: SDP logs no longer included in tarfile
09 Jan 2008 updated for CIAO 4.0: the mkoif tool and firstlook GUI have been removed; filenames updated since ObsID 1843 went through Repro III
30 Dec 2008 reviewed for CIAO 4.1: no changes
20 Jul 2009 summary pages not created as of DS 8.0; added information on the sky field of view file (fov1.fits)
25 Jan 2010 reviewed for CIAO 4.2: no changes
10 Jan 2011 updated for CIAO 4.3: mtl file is used for temperature-dependent CTI correction of ACIS data
28 Jun 2011 soff1.fits, src2.fits, and src_img2.fits files not created as of DS 8.4
26 Aug 2011 aoff1.fits file not created as of DS 8.4.1
03 Jan 2012 reviewed for CIAO 4.4: no changes
03 Dec 2012 Review for CIAO 4.5; minor edits
28 May 2013 Added more text to the synopsis and new sections: Archive versions and What next?.
13 Aug 2013 Added note on new 00README files
03 Dec 2013 Review for CIAO 4.6
16 Jul 2014 Updated for evt1a files in secondary directory.
18 Dec 2014 Reviewed for CIAO4.7; fixed typos.
22 Feb 2022 Review for CIAO 4.14. Updated for Repro5/CALDB 4.9.6.