Skip to the navigation links
Last modified: 25 Jan 2010
Where are the PDFs?

Introduction to the Data Products

CIAO 4.2 Science Threads



Overview

Last Update: 25 Jan 2010 - reviewed for CIAO 4.2: no changes

Synopsis:

The CXC performs standard data processing (SDP) on all Chandra science data; this procedure is commonly referred to as "the pipeline." The processing runs in several stages or "levels", each of which is built on the results of the preceding level. A well-defined set of data products are the result of each level.

This thread examines the primary and secondary Chandra data products where were obtained by following the How to Download Chandra Data from the Archive thread, which uses ObsID 1843 (ACIS-I, G21.5-0.9) as the sample data. Filenames given in brackets - e.g. [hrc_dtf1.fits] - indicate a data product that is not created for ACIS imaging observations; this makes the thread applicable to any standard Chandra data distribution.

Related Links:




Contents



ObsID Directory

The ObsID directory is the top level in the directory tree created when the tarfile is unpacked. It contains two files and two data directories:

unix% pwd
/intro_data/1843

unix% ls -1
axaff01843N001_VV001_vv2.pdf
oif.fits
primary/
secondary/
Observation index file
oif.fits

The observation index file contains a summary of the data products associated with an observation. It is equivalent to a structured directory listing with descriptive information about the observation.

V&V report
axaff01843N001_VV001_vv2.pdf

All standard data products are checked by a CXC scientist before release to ensure data quality and to investigate the cause of any exposure losses or other anomalies. This process is known as "verification and validation", or V&V for short. The V&V report is distributed in PDF format with the data files.

This file contains a summary report of the V&V report. The full report is in a second PDF file (axaff01843N001_VV001_vvref2.pdf), which is packaged in the secondary data directory.

Users should review this information before beginning the analysis to ensure that there aren't any caveats from the V&V scientist.

In the following sections, we describe the contents of the primary and secondary data directories.



Primary Directory

The data products are arranged such that all products necessary for most analyses (e.g. the CIAO threads) are in the primary directory.

unix% cd primary/
unix% pwd
/intro_data/1843/primary

unix% ls -1
acisf01843_000N002_bpix1.fits
acisf01843_000N002_fov1.fits
acisf01843N002_1_sum2.html
acisf01843N002_2_sum2.html
acisf01843N002_3_sum2.html
acisf01843N002_cntr_img2.fits
acisf01843N002_cntr_img2.jpg
acisf01843N002_evt2.fits
acisf01843N002_full_img2.fits
acisf01843N002_full_img2.jpg
acisf01843N002_src2.fits
acisf01843N002_src_img2.jpg
orbitf084197100N001_eph1.fits
pcadf084271087N002_asol1.fits
[hrc_dtf1.fits]
[acis_plt2.jpg]
[acis_pha2.fits]

The majority of these files can be grouped into three categories:

HTML summary files
sum2.html

The HTML files contain a summary of the observation parameters, such as the instrument, detector, and grating that were used. The images from SDP are included in these pages as well.

[These files are no longer created as of SDP version DS 8.0 (or higher).]

Images

Two images are produced for every dataset: a full-field sky image (full_img2.jpg) and a high-resolution image of the center of the field (cntr_img2.jpg).

Imaging observations also have a full field image with the source candidates overlaid (src_img2.jpg).

Grating observations include an image of each spectral component plotted as wavlength vs counts (heg_plt2.jpg, meg_plt2.jpg, and/or leg_plt2.jpg). [The grating images are no longer created as of SDP version DS 8.0 (or higher).]

Remove those three sets of files from the directory listing and you have the primary data products necessary for most analyses:

acisf01843N002_evt2.fits
acisf01843N002_src2.fits
acisf01843_000N002_bpix1.fits
acisf01843_000N002_fov1.fits
orbitf084197100N001_eph1.fits
pcadf084271087N002_asol1.fits
[hrc_dtf1.fits]
[acis_pha2.fits]
Level 2 event file
evt2.fits

The level 2 event file is the most important data product you receive. This file is created from the level 1 event list by filtering on the good time intervals (GTI) and status bits. The result is a list of events that is suitable for use in data analysis.

When new calibration is released, it is often necessary to remake the level 2 event file in order to apply the changes. This process is explained in the Create a New Level 2 Event File thread.

Source list
src2.fits

A source candidate list. These detections are obtained from a single run of celldetect with S/N threshold equal to 3 and should not be used directly in your analysis. Users interested in spatial analysis should perform their own detection runs with one of the three Chandra Detect tools: celldetect, wavdetect, and vtpdetect. The Detect Manual describes which tool is suited to a particular analysis.

For instructions on displaying the source list over the data (S-Lang or Python), refer to the Using the Output of Detect Tools thread (S-Lang or Python).

This file is only created for imaging observations.

Bad pixels
bpix1.fits

A list of pixels identified as "bad"; criteria for flagging a pixel are listed in the Bad Pixels dictionary entry. Any tool that reads this file will exclude the bad pixels from its calculations.

The New ACIS Bad Pixel File: Identify ACIS Hot Pixels and Cosmic Ray Afterglows thread contains more information, including instructions on how to make a new ACIS bad pixel file, if necessary. The New Observation-Specific HRC Bad Pixel File thread explains when and how to make a new HRC bad pixel file.

Sky field of view
fov1.fits

The field-of-view file contains regions which describe the edges of each instrument chip (one polygon per chip). This region file can be used to filter event files or images to contain specific chips.

This file has been created since SDP version 6.9.0. The skyfov tool may be used to create an fov1.fits file for observations with an earlier processing version.

Orbit ephemeris
orbit_eph1.fits

The definitive orbit ephemeris information for the observation. This file is required if you wish to barycenter-correct the observation times in order to account for the difference in photon arrival times as the Earth and Chandra move around the Sun.

The orbit ephemeris is also needed if you are analyzing an object which was moving with respect to the observer when observed, e.g. a planet or comet.

Aspect solution
pcad_asol1.fits

The aspect solution describes the orientation of the telescope as a function of time. The detected position of an event and the corresponding telescope aspect are combined for an accurate determination of the celestial position of that event.

Since an aspect solution file is created for each stable aspect interval, there is often more than one pcad_asol1.fits file for an observation. All the files must be used whenever a tool requires the aspect solution as input.

Detailed information on the aspect solution files is available from the the Aspect Solution why topic.

Dead time factor
dtf1.fits

The dead time correction factors as a function of time. For ACIS observations, the dead time correction is recorded in the DTCOR header keyword. Since the HRC dead time correction factor varies during the observation, however, the value of DTCOR represents an average and the individual values are stored in the dtf1.fits file.

The Computing Average HRC Dead Time Corrections thread (S-Lang or Python) explains when you might need to recompute the deadtime statistics for your observation.

This file is only created for HRC observations.

Type II PHA / Spectra
pha2.fits

The PHA file contains a spectrum for one grating part and order in each row of the file. The number of rows varies depending on the detector and grating combination: e.g. ACIS/HETG data has twelve rows while HRC/LETG only has two.

Detailed information on the pha2.fits file is available from the Examining Grating Spectra and Regions thread.

This file is only created for grating observations.



Secondary Directory

If you are interested in reprocessing your data, you will also need the level 1 files, which are in the secondary directory.

unix% cd ../secondary/
unix% pwd
/intro_data/1843/secondary

unix% ls -1
acisf01843_000N002_aoff1.fits
acisf01843_000N002_evt1.fits
acisf01843_000N002_flt1.fits
[hrc_std_flt1.fits]
acisf01843_000N002_msk1.fits
acisf01843_000N002_mtl1.fits
acisf01843_000N002_soff1.fits
[hrc_std_dtfstat1.fits]
acisf01843_000N002_stat1.fits
acisf084271178N002_0_bias0.fits
acisf084271178N002_1_bias0.fits
acisf084271178N002_2_bias0.fits
acisf084271178N002_3_bias0.fits
acisf084271178N002_4_bias0.fits
acisf084271178N002_5_bias0.fits
acisf084272477N002_pbk0.fits
aspect/
ephem/
axaff01843N001_VV001_vvref2.pdf

The V&V report - axaff01843N001_VV001_vvref2.pdf - is explained in the ObsID Directory section.

Aspect offsets
aoff1.fits

A set of offsets versus time (e.g. ra_off=(ra-ra_nom)) . The one piece of remaining useful info is information about gaps in the aspect records. These are derived from the aspect solution files (pcad_asol1.fits, primary directory) for the observation.

Most users will never need this file in their analysis, instead using the aspect solution files directly.

Level 1 event file
evt1.fits

The level 1 event file contains all the events recorded for the observation. While many of these events have a status bit set to flags them as "bad", none of the information has been removed. This file is filtered on GTIs and status bits to create the level 2 event file.

The evt1.fits file is also the starting point for reprocessing your data, as explained in the Create a New Level 2 Event File thread.

Good time intervals
flt1.fits / std_flt1.fits

The GTI information for the observation, e.g. the start and stop times of all accepted time intervals over the observation. The major contributor to creating GTIs is information about when there is aspect data and when that aspect data is good. When the event file is filtered, the GTIs are stored as extensions of the data file, creating a dynamic record of the time filters applied to the data.

Note that this file is named slightly differently for ACIS (flt1.fits) and HRC (std_flt1.fits) observations.

Mask
msk1.fits

The mask file records the valid part of the detector element - ACIS CCD or HRC plate - used for the observation (i.e. the portion for which events can be telemetered). The active portion of an element may be smaller than the default regions if an observation was performed using subarrays or custom windows. This information is used when creating response files, such as ARFs.

Mission timeline
mtl1.fits

The mission timeline (MTL) consists of all time series data that affects the quality of the data, such as aspect and ephemeris data. This information is used to construct the GTI limits in the flt1.fits file. The MTL ahelp file has more information on the mission timeline.

Most users will never need this file in their analysis.

Alignment offsets
soff1.fits

The science instrument module offsets versus time. These are derived from the aspect solution files (pcad_asol1.fits, primary directory) for the observation. The soff1.fits file is binned, so there's not a one-to-one match to the rows in the pcad_asol1.fits files.

Most users will never need this file in their analysis, instead using the aspect solution files directly.

Exposure statistics
stat1.fits

The exposure statistics file contains up to seven extensions, each pertaining to the sequence of ACIS CCD exposures processed and recorded into telemetry.

Most users will never need this file in their analysis.

This file is only created for ACIS observations.

Dead time factor statistics
std_dtfstat1.fits

The dead time factor statistics information is derived from the HRC deadtime factor file (dtf1.fits, primary directory) for the observation.

Most users will never need this file in their analysis.

This file is only created for HRC observations.

Bias map(s)
bias0.fits

When the bad pixel list (bpix1.fits) is created, each bias map is searched for pixels whose bias values are either too low or too high. There is one bias map for each ACIS chip that was used for the observation.

These files are only created for ACIS observations.

Parameter block
pbk0.fits

The parameter block file is needed in conjunction with the bias maps when creating a new bad pixel list. It is used to determine observational parameters, such as which CCDs are active, the READMODE and DATAMODE, etc.

This file is only created for ACIS observations.

Aspect Subdirectory

The secondary/aspect subdirectory contains the aspect quality file (aqual1.fits) and aspect solution files which are based on aspect determined by the on-board computer (osol1.fits):

unix% cd aspect/
unix% pwd
/intro_data/1843/secondary/aspect

unix% ls -1
pcadf084265646N002_osol1.fits
pcadf084271087N002_aqual1.fits
pcadf084272206N002_osol1.fits
pcadf084278602N002_osol1.fits

Most users will never need these files in their analysis. Ground-based aspect analysis, which produces the asol1.fits files (primary directory), gives a better product because more information is available.

There is a special case when the osol1.fits files may be used. The pcad_asol1.fits files are only produced when Chandra is in Kalman lock and on the target. The osol1.fits, however, are always produced. Since HRC is "always on" when it's the chosen detector, it records events while it is slewing between targets; these events are included in the evt1.fits file. There is no Kalman lock aspect solution for these events,which is why sources in HRC evt1.fits files often have "tails". If you really want to use those extra events that were taken as the telescope was moving into position, you can in principle use the osol1.fits files for the analysis.


Ephem Subdirectory

The secondary/ephem subdirectory contains the solar (solar_eph1.fits) and lunar (lunar_eph1.fits) ephemeris files, as well as the viewing angles file (angles_eph1.fits):

unix% cd ../ephem/
unix% pwd
/intro_data/1843/secondary/ephem

unix% ls -1
anglesf01843_000N002_eph1.fits
lunarf084197100N001_eph1.fits
solarf084197100N001_eph1.fits

Most users will never need these files in their analysis. When a tool requires the ephermeris information, it is referring to the orbit ephemeris (orbit_eph1.fits) from the primary directory.



History

05 Jul 2005 original version, new for CIAO 3.2
01 Dec 2005 reviewed for CIAO 3.3: no changes
01 Dec 2006 updated for CIAO 3.4: SDP logs no longer included in tarfile
09 Jan 2008 updated for CIAO 4.0: the mkoif tool and firstlook GUI have been removed; filenames updated since ObsID 1843 went through Repro III
30 Dec 2008 reviewed for CIAO 4.1: no changes
20 Jul 2009 summary pages not created as of DS 8.0; added information on the sky field of view file (fov1.fits)
25 Jan 2010 reviewed for CIAO 4.2: no changes

Return to Threads Page: Top | All | Intro

Where are the PDFs?
Last modified: 25 Jan 2010