Reprocessing Data to Create a New Level=2 Event File

Use the chandra_repro script (Recommended)

The CXC strongly recommends you recalibrate your event data so that it is consistent with the other calibrations you will need. The easiest way is to use the chandra_repro reprocessing script, which automates the recommended data processing steps presented in the CIAO analysis threads. The script reads data from the standard data distribution (e.g. primary and secondary directories) and creates a new bad pixel file, a new level=2 event file, and a new level=2 Type II PHA file (grating data only) along with the appropriate per-order and per-arm ARF and RMF files.

To run the script:

unix% cd <obsid>

unix% ls
primary/  secondary/

unix% chandra_repro

After chandra_repro is finished, this thread is complete and you may continue with your analysis. You should not follow the rest of this thread.

If you want to learn more about the individual processing steps, the options available, and the custom calibrations that can be applied you may continue reading this thread.

Do I have to run this thread?

Our recommendation is that all users reprocess their data in CIAO to ensure that the newest software and consistent calibration updates are applied to the dataset before you begin your analysis. Not all data need to be reprocessed, but any data can be reprocessed without any negative effects.

Proceed to the next section to get started.

Software and Calibration Updates

Updates to ACIS TGAIN calibration for -120 C data

The ACIS time-dependent gain (TGAIN) calibration files for the -120 C focal plane temperature are frequently updated in the calibration database (CALDB). If your data were taken recently - even if the ASCDSVER is DS 7.6.7 or higher - a newer TGAIN file may be available for your dataset.

Newly acquired data processed in standard data processing use a TGAIN file from a previous epoch. It may be several months until the final TGAIN calibration is available. Typically the differences between epochs are small. However, if your science requires high fidelity energy calibrations (large number of counts and a line dominated spectrum), then you will want to use the definitive TGAIN file when it is available.

The Improvements in Calibration section of the TGAIN why topic lists the updates by date and CALDB version. The observation start date and focal plane temperature (in K) are recorded in the header of the event file:

unix% dmkeypar acis_evt1.fits DATE-OBS echo+
2008-01-19T20:41:39

unix% dmkeypar acis_evt1.fits FP_TEMP echo+
153.60722351

Most data in the Chandra Archive has been updated as part of the bulk reprocessing effort known as Repro-4 effort. However, calibration and software changes have been introduced since the start of reprocessing. Users are therefore still encouraged to reprocess their data to ensure they have a consistent dataset.

Proceed to the next section to begin.

Get Started

Since this thread is meant to represent the most generic case, it is not written with a specific ObsID. Files are referred to by the detector and type - acisf01843_000N001_evt1.fits becomes acis_evt1.fits, hrcf01557_000N001_std_flt1.fits becomes hrc_std_flt1.fits, and so on.

Furthermore, it is assumed that all relevant files are in the same working directory.

Proceed to the next section.

Update aspect solution file

Chandra data processed with software version DS10.8.4 and later has the boresight corrections applied directly to the aspect solution. This provides users a better estimate of the off-axis angles. CIAO 4.13 includes the asp_offaxis_corr tool which applies this correction to the aspect solution.

The easiest way to check if this part of the thread is relevant is to check the CONTENT keyword in the aspect solution file: pcad*_asol1.fits.

unix% dmkeypar pcad_asol1.fits CONTENT echo+
ASPSOLOBI

If CONTENT=ASPSOLOBI, then the boresight correction is already applied and you can proceed to the next section.

If you have multiple aspect solution files, then you should check them all. If there is one with CONTENT=ASPSOLOBI, then you should use the file and omit the other files and proceed to the next section.

If CONTENT=ASPSOL, then you should continue with this section. We will assume that the time-ordered list of the aspect solution file(s) are stored in a stack list, eg

unix% /bin/ls pcadf*_asol1.fits > pcad_asol.lis

First we need to combine all the aspect solution files. Because the aspect solution files may span a different time range then the current observation, we need to be sure we also apply a time filter to the files. We will take the time range from the TSTART and TSTOP values in the event file header.

unix% set tstart=`dmkeypar acis_evt1.fits tstart echo+`
unix% set tstop=`dmkeypar acis_evt1.fits tstop echo+`
unix% dmmerge "@pcad_asol.lis[time=${tstart}:${tstop}]" pcad_repro_asol1.fits

The time filter is applied to each file name listed in the stack.

The next step is to apply the boresight correction using the asp_offaxis_corr tool. This tool modifies the input file in place. We also need to update the CONTENT keyword to reflect the updated content

unix% asp_offaxis_corr pcad_repro_asol1.fits acis
unix% dmhedit pcad_repro_asol1.fits file="" op=add key=CONTENT value=ASPSOLOBI

This section is now complete. Proceed to the next section.

ACIS or HRC?

Question: which detector was used for your observation? (It does not matter yet whether or not a transmission grating was also used.)

• ACIS
• HRC

ACIS Observations

Question: was the observation taken in continuous-clocking mode (i.e. the read mode is "continuous")?

unix% dmkeypar acis_evt1.fits readmode echo+
CONTINUOUS

• yes: check the accuracy of the source coordinates
• no: proceed to the "reset ACIS status bits" step

Check Accuracy of Source Coordinates (CC-mode data only)

It is important to verify that the coordinates of the source are accurate and precise to less than 0.5 arcsec (i.e. one pixel). The source coordinates are used to determine the position-dependent absolute times of arrival and pulse heights for continuous-clocking mode data. If they are imprecise, then the output times of arrival and pulse heights will also be inaccurate.

The coordinates that should be used are the best-known right ascension (RA_TARG) and declination (DEC_TARG) coordinates for the source:

unix% dmlist acis_evt1.fits header | egrep '(RA|DEC)_TARG'
0057 RA_TARG                     83.6333330              Real8        Observer's specified target RA
0058 DEC_TARG                    22.0144720              Real8        Observer's specified target Dec

These coordinates have the correct level of accuracy precision.

If the coordinates need to be changed, then here are some suggestions about how to determine the values:

• use previously-published source coordinate values,
• if the coordinates are not well known, then try obtaining them from a Chandra observation of the source which was taken in TE mode,
• if neither of the above methods are available, then it may be possible to use two, separate continuous-clocking mode observations to find the source coordinates, provided that the observations did not have the same roll angle.

It is not possible to fix the coordinates in the dataset if the position is not known from some other dataset or publication (i.e. you cannot determine accurate source coordinates from the dataset you are attempting to correct).

Once the new source coordinates are determined, modify the file header with dmhedit:

unix% dmhedit acis_evt1.fits filelist="" operation=add key=RA_TARG value='newvalue'
unix% dmhedit acis_evt1.fits filelist="" operation=add key=DEC_TARG value='newvalue'

Note that the ability to reconstruct the TIMEs of arrival is limited by

• the uncertainties in the coordinates RA_TARG and DEC_TARG,
• the uncertainties in the aspect reconstruction, and
• the PSF "blur."

If RA_TARG and DEC_TARG are accurate to much less than a pixel, then the times of arrival should be accurate to about 4 ms for most observations (i.e. those without any unusual aspect problems).

Proceed to the next step.

Reset ACIS status bits

Status bits 1-5, 14-20 and 23 in the input event file are reset so that they can be updated by destreak and acis_process_events. This reset includes removing the effects of acis_run_hotpix and (previously) acis_detect_afterglow. More information is available in the cosmic-ray afterglow why topic. Details about what each status bit represents are available from the "STATUS bits in event files" memo.

The acis_clear_status_bits script clears the status bits in place; it does not create a new output file. The file must have write permission, and you may wish to make a copy before modifying it.

unix% acis_clear_status_bits acis_evt1.fits

The CLSTBITS header keyword is added to record which bits were cleared.

unix% dmkeypar acis_evt1.fits  CLSTBITS echo+
11111111011000000011111111000001

Proceed to the next step.

Remove streak events (Optional)

Running destreak on the ACIS-S4 chip (ccd_id=8) can improve the quality of your data. Removing the streaks before creating a new bad pixel file improves the streak detection efficiency and prevents misidentifying pixels with many streak events as hot pixels. For details on how the tool works, see the Destreaking ACIS Data why topic and "ahelp destreak".

Note that there is some evidence that for bright, continuum-dominated sources observed with the gratings, applying destreak removes more source events than streak events; see the destreak why topic for details. Furthermore, spectral order sorting alone will greatly reduce the amount of streak data present in any extracted spectrum. Therefore destreaking should be considered optional for this case.

The mask parameter is set to "None" so that all events in the input file are evaluated as possible streak events. The parameter setting filter=no flags the streak events in the output file, but does not remove them.

unix% punlearn destreak
unix% pset destreak infile=acis_evt1.fits
unix% pset destreak outfile=acis_dstrk_evt1.fits
unix% pset destreak mask=None filter=no
unix% destreak
Input dataset/block specification (acis_evt2.fits): 
Output dataset/block specification (acis_dstrk_evt2.fits): 

Question: was the observation taken in timed mode (i.e. the read mode is "TIMED", not "CONTINUOUS")?

unix% dmkeypar acis_dstrk_evt1.fits readmode echo+
TIMED

• yes (timed mode data): make a new bad pixel file
• no (CC-mode data): proceed to the "Run acis_process_events to make a new level=1 event file" step

Create a new ACIS bad pixel file

This section of the thread does not apply when the input data were taken in ACIS continuous-clocking mode, as the ACIS afterglow tools cannot be used with this data mode.

There are three steps in creating a new ACIS bad pixel file:

1. identify known bad pixels and columns from the calibration database and pixels with bad bias values (acis_build_badpix with a custom bitflag)

   

   New bitflag to include frame-store shadow

   Badpixel bit 17 has been added to the CALDB badpixel file in CALDB 4.9.0 and support for it was added in CIAO 4.12. The default acis_build_badpix bitflag value has changed to include bit 17.

2. search for afterglows and hot pixels (acis_find_afterglow)

3. mark pixels adjacent to afterglows and hot pixels and sort the list of bad pixels and afterglows (a second run of acis_build_badpix)

In order to run acis_build_badpix, you need an obs.par file for the observation, created by running dmmakepar

unix% dmmakepar acis_dstrk_evt1.fits acis_obs.par

You will also need a list of the bias files.

unix% cat bias.lis
acis_2_bias0.fits
acis_4_bias0.fits
acis_5_bias0.fits
acis_3_bias0.fits
acis_1_bias0.fits

Now run the tools:

unix% punlearn acis_build_badpix 
unix% pset acis_build_badpix obsfile=acis_obs.par
unix% pset acis_build_badpix pbkfile=acis_pbk0.fits  
unix% pset acis_build_badpix biasfile=@bias.lis 
unix% pset acis_build_badpix outfile=acis_abb1_bpix1.fits 
unix% pset acis_build_badpix bitflag=00000000000000120021100020022222
unix% pset acis_build_badpix calibfile=CALDB 
unix% acis_build_badpix mode=h verbose=0

unix% punlearn acis_find_afterglow
unix% pset acis_find_afterglow infile=acis_dstrk_evt1.fits
unix% pset acis_find_afterglow outfile=acis_aglow_bpix1.fits
unix% pset acis_find_afterglow badpixfile=acis_abb1_bpix1.fits 
unix% pset acis_find_afterglow maskfile=acis_msk1.fits
unix% pset acis_find_afterglow statfile=acis_stat1.fits
unix% acis_find_afterglow mode=h verbose=0

unix% punlearn acis_build_badpix 
unix% pset acis_build_badpix obsfile=acis_obs.par
unix% pset acis_build_badpix pbkfile=acis_pbk0.fits  
unix% pset acis_build_badpix biasfile=None
unix% pset acis_build_badpix outfile=acis_repro_bpix.fits
unix% pset acis_build_badpix calibfile=acis_aglow_bpix1.fits
unix% pset acis_build_badpix procbias=no
unix% acis_build_badpix mode=h verbose=0

The new bad pixel file is called acis_repro_bpix.fits.

Proceed to the next step.

Run acis_process_events to make a new level=1 event file

Running acis_process_events produces a new level=1 event file that has the latest calibration applied:

• the latest temperature-dependent charge transfer inefficiency (CTI) correction
• the latest time-dependent gain adjustment
• the latest gain map
• a new bad pixel file
• PHA randomization
• a sub-pixel adjustment (for timed mode observations)
• background cleaning for very-faint mode data (optional)
• continuous clocking mode times of arrival and pulse heights (be sure to read the data caveats)

Determine the eventdef

The eventdef parameter specifies the names and data types of the columns in the output event data file. Six definitions are included in the parameter file for acis_process_events:

The event mode of an observation can be found in the DATAMODE and CONTENT values stored in the file header:

unix% dmkeypar acis_evt1.fits DATAMODE echo+
FAINT

unix% dmkeypar acis_evt1.fits CONTENT echo+
EVT1

This dataset is for a timed exposure faint mode observation without grating data. Therefore, the proper eventdef parameter is "stdlev1."

Set the parameters

• If you created a new bad pixel file when running the Reprocessing Data to Create a New Level=2 Event File thread or the Customizing an ACIS Bad Pixel File thread, use that file in this analysis. Otherwise, use the bpix1.fits file from the Archive.

• If the asp_offaxis_corr tool was used, as shown above, to update the aspect solution then you will need to also create a separate observation parameter file with updated pointing information. This is a different file then the file used to create the badpixel file.

  Start by downloading the make_par script .

  Then run the script as shown below to create an observation parameter file with the updated values for the boresight.

  unix% python make_par acis_dstrk_evt1.fits pcad_repro_asol1.fits pcad_obs.par
  

  If the asp_offaxis_corr tool was not necessary, then you should have a single aspect solution file, pcad_asol1.fits

• The parameter for VFAINT background cleaning (check_vf_pha) may be set. The tool will ignore this parameter if your observation was not taken in this mode.

  The check_vf_pha algorithm should be used very carefully. As discussed in the why topic, it can misidentify valid source events as background events. The default for chandra_repro is check_vf_pha=no.

• The parameter for a sub-pixel adjustment (pix_adj) is set to EDSER by default. However, since the sub-pixel adjustments affect not only the coordinates, but also the times of arrival for continuous clocking mode datasets, and since such adjustments can introduce non-astrophysical features in the times of arrival, the recommended setting for continuous clocking mode observations is pix_adj=NONE (i.e. no adjustments).

unix% punlearn acis_process_events
unix% pset acis_process_events infile=acis_dstrk_evt1.fits
unix% pset acis_process_events outfile=acis_new_evt1.fits
unix% pset acis_process_events badpixfile=acis_new_bpix1.fits
unix% pset acis_process_events acaofffile=pcad_asol1.fits
unix% pset acis_process_events mtlfile=acis_mtl1.fits
unix% pset acis_process_events eventdef=")stdlev1"
unix% pset acis_process_events check_vf_pha=yes

If the asp_offaxis_corr tool was used then also set these parameters:

unix% pset acis_process_events acaofffile=pcad_repro_asol1.fits
unix% pset acis_process_events obsfile=pcad_obs.par

If the DATAMODE is CC33_FAINT or CC33_GRADED, then also use the following.

unix% pset acis_process_events pix_adj=NONE

Run the tool

unix% acis_process_events
Input event file or stack (acis_evt1.fits): 
Output event file name (acis_new_evt1.fits): 
aspect offset file ( NONE | none | <filename>) (pcad_asol1.fits):

A number of possible warning messages are explained in the Common acis_process_events Warnings section.

% dmkeypar acis_new_evt1.fits SUM_2X2 echo+
# dmkeypar (CIAO): ERROR: Keyword 'SUM_2X2' was not found in file 'acis_new_evt1.fits'.

unix% dmkeypar acis_new_evt1.fits DY_AVG echo+
# dmkeypar (CIAO): ERROR: Keyword 'DY_AVG' was not found in file 'acis_new_evt1.fits'.

unix% r4_header_update acis_new_evt1.fits
Missing keywords 'OCLKPAIR,ORC_MODE,SUM_2X2,FEP_CCD' from event file 'acis_new_evt1.fits' header.
Missing keywords 'DY_AVG,DZ_AVG,DTH_AVG' from event file 'acis_new_evt1.fits' header.

unix% dmkeypar acis_new_evt1.fits SUM_2X2 echo+
0
unix% dmkeypar acis_new_evt1.fits DY_AVG echo+
0.045452421825

Question: is your observation imaging or grating data?

• Imaging
• Grating

unix% dmcopy "acis_repro_evt2.fits[EVENTS][x=:,y=:]" acis_filter_xy_repro_evt2.fits

HRC Observations

All HRC-S (imaging and grating) observations taken after 01 January 2012, which were processed with a CalDB version released prior to the 4.5.1 release (24 July 2012), should be reprocessed to apply the updated HRC-S time-dependent gain map calibration released with CalDB 4.5.1. The High Voltage settings of the HRC-S microchannel plates were increased in March 2012, resulting in a corresponding gain increase and required change to HRC-S T_GAIN calibration.

Running hrc_process_events produces a new level=1 event file that has the latest calibration applied:

• the new time-dependent gain correction (available for both HRC-S and HRC-I)
• AMP_SF correction
• degap correction
• Dead Time correction (DTCOR)

HRC-S: Determine the 'gainfile' parameter

On 29 March 2012, the High Voltage (HV) settings of the HRC-S microchannel plates were increased, causing a corresponding rise in the detector gain, and a required update to the HRC-S time-dependent gain map (T_GMAP) calibration. As a result, all observations taken after 01 January 2012 must be reprocessed using one of the two new calibration T_GMAP files available in CalDB 4.9.2:

        hrcsD1999-07-22t_gmapN0003.fits
        hrcsD2012-03-29t_gmapN0003.fits

If you are analyzing one of the ObsIDs listed below, you must manually set the 'gainfile' parameter of hrc_process_events to the appropriate CalDB path and name, as shown below.

Old High-Voltage Setting:
unix% pset hrc_process_events gainfile ${CALDB}/data/chandra/hrc/t_gmap/hrcsD1999-07-22t_gmapN0003.fits

New High-Voltage Setting:
unix% pset hrc_process_events gainfile ${CALDB}/data/chandra/hrc/t_gmap/hrcsD2012-03-29t_gmapN0003.fits

Otherwise, you may run hrc_process_events with 'gainfile' set to the default 'CALDB', and the correct time-dependent gain map will automatically be applied.

Set the parameters

• If the asp_offaxis_corr tool was used, as shown above, to update the aspect solution then you will need to also create a separate observation parameter file with updated pointing information. This is a different file then the file used to create the badpixel file.

  Start by downloading the make_par script .

  Then run the script as shown below to create an observation parameter file with the updated values for the boresight.

  unix% python make_par hrc_evt1.fits pcad_repro_asol1.fits pcad_obs.par
  

  If the asp_offaxis_corr tool was not necessary, then you should have a single aspect solution file, pcad_asol1.fits

unix% punlearn hrc_process_events
unix% pset hrc_process_events infile=hrc_evt1.fits
unix% pset hrc_process_events outfile=hrc_new_evt1.fits
unix% pset hrc_process_events badpixfile=hrc_bpix1.fits
unix% pset hrc_process_events acaofffile=pcad_asol1.fits
unix% pset hrc_process_events badfile=NONE
unix% pset hrc_process_events do_amp_sf_cor=yes

If the asp_offaxis_corr tool was used then also set these parameters:

unix% pset hrc_process_events acaofffile=pcad_repro_asol1.fits
unix% pset hrc_process_events obsfile=pcad_obs.par

Run the tool

unix% hrc_process_events
input level 0 event file/stack (hrc_evt1.fits): 
output level 1 file (hrc_new_evt1.fits): 
bad pixel file ( NONE | none | <filename>) (hrc_bpix1.fits): 

A number of possible warning messages are explained in the Common hrc_process_events Warnings section.

unix% dmkeypar hrc_new_evt1.fits DY_AVG echo+
# dmkeypar (CIAO): ERROR: Keyword 'DY_AVG' was not found in file 'hrc_new_evt1.fits'.

unix% r4_header_update hrc_new_evt1.fits
Missing keywords 'DY_AVG,DZ_AVG,DTH_AVG' from event file 'hrc_new_evt1.fits' header.

unix% dmkeypar hrc_new_evt1.fits DY_AVG echo+
0.045452421825

Question: is your observation HRC-S imaging, HRC-I imaging, or grating data?

• HRC-S Imaging
• HRC-I Imaging
• Grating

unix% punlearn hrc_dtfstats
unix% pset hrc_dtfstats infile=hrc_dtf1.fits
unix% pset hrc_dtfstats outfile=hrc_dtfstats_new.fits
unix% pset hrc_dtfstats gtifile=hrc_repro_evt2.fits"[gti]"
unix% hrc_dtfstats
Input file (hrc_dtf1.fits): 
Output file (hrc_dtfstats_new.fits): 
File containing GTI to filter on (<filename>|NONE) (hrc_repro_evt2.fits[gti]): 

unix% dmlist hrc_dtfstats_new.fits"[cols DTCOR]" data,clean
#  DTCOR
     0.35535140943668

unix% dmkeypar hrc_repro_evt2.fits ONTIME echo+

ONTIME * NEWDTCOR = NEWTIME

unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=LIVETIME value=NEWTIME
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=EXPOSURE value=NEWTIME
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=DTCOR value=NEWDTCOR

unix% dmkeypar hrc_repro_evt2.fits ONTIME echo+
2189.40009910

2189.40009910*0.35535140943668 = 778.00641103599186717498

unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=LIVETIME value=778.0064110
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=EXPOSURE value=778.0064110
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=DTCOR value=0.35535140943668

unix% punlearn hrc_dtfstats
unix% pset hrc_dtfstats infile=hrc_dtf1.fits
unix% pset hrc_dtfstats outfile=hrc_dtfstats_new.fits
unix% pset hrc_dtfstats gtifile=hrc_repro_evt2.fits"[gti]"
unix% hrc_dtfstats
Input file (hrc_dtf1.fits): 
Output file (hrc_dtfstats_new.fits): 
File containing GTI to filter on (<filename>|NONE) (hrc_repro_evt2.fits[gti]): 

unix% dmlist hrc_dtfstats_new.fits"[cols DTCOR]" data,clean
#  DTCOR
     0.35535140943668

unix% dmkeypar hrc_repro_evt2.fits ONTIME echo+

ONTIME * NEWDTCOR = NEWTIME

unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=LIVETIME value=NEWTIME
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=EXPOSURE value=NEWTIME
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=DTCOR value=NEWDTCOR

unix% dmkeypar hrc_repro_evt2.fits ONTIME echo+
2189.40009910

2189.40009910*0.35535140943668 = 778.00641103599186717498

unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=LIVETIME value=778.0064110
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=EXPOSURE value=778.0064110
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=DTCOR value=0.35535140943668