Last modified: 12 Dec 2024

URL: https://cxc.cfa.harvard.edu/ciao/threads/createL2/

Reprocessing Data to Create a New Level=2 Event File

CIAO 4.17 Science Threads


Overview

Synopsis:

The structure of this thread is presented as a series of questions and answers, the latter of which are links that take you further down the analysis tree. Following this thread from the top down will not produce correct results.

The chandra_repro script runs all the commands in this thread automatically. If you run chandra_repro, you have completed all steps in this thread and you are done.

Purpose:

To generate a new level=2 event file (evt2.fits) for any Chandra observation.

Related Links:

Last Update: 12 Dec 2024 - Added a new step for HRC processing to check for secondary science corruption.


Contents


Use the chandra_repro script (Recommended)

The CXC strongly recommends you recalibrate your event data so that it is consistent with the other calibrations you will need. The easiest way is to use the chandra_repro reprocessing script, which automates the recommended data processing steps presented in the CIAO analysis threads. The script reads data from the standard data distribution (e.g. primary and secondary directories) and creates a new bad pixel file, a new level=2 event file, and a new level=2 Type II PHA file (grating data only) along with the appropriate per-order and per-arm ARF and RMF files.

To run the script:

unix% cd <obsid>

unix% ls
primary/  secondary/

unix% chandra_repro

After chandra_repro is finished, this thread is complete and you may continue with your analysis. You should not follow the rest of this thread.

If you want to learn more about the individual processing steps, the options available, and the custom calibrations that can be applied you may continue reading this thread.


Do I have to run this thread?

Our recommendation is that all users reprocess their data in CIAO to ensure that the newest software and consistent calibration updates are applied to the dataset before you begin your analysis. Not all data need to be reprocessed, but any data can be reprocessed without any negative effects.

Proceed to the next section to get started.

Software and Calibration Updates

Updates to ACIS TGAIN calibration for -120 C data

The ACIS time-dependent gain (TGAIN) calibration files for the -120 C focal plane temperature are frequently updated in the calibration database (CALDB). If your data were taken recently - even if the ASCDSVER is DS 7.6.7 or higher - a newer TGAIN file may be available for your dataset.

Newly acquired data processed in standard data processing use a TGAIN file from a previous epoch. It may be several months until the final TGAIN calibration is available. Typically the differences between epochs are small. However, if your science requires high fidelity energy calibrations (large number of counts and a line dominated spectrum), then you will want to use the definitive TGAIN file when it is available.

The Improvements in Calibration section of the TGAIN why topic lists the updates by date and CALDB version. The observation start date and focal plane temperature (in K) are recorded in the header of the event file:

unix% dmkeypar acis_evt1.fits DATE-OBS echo+
2008-01-19T20:41:39

unix% dmkeypar acis_evt1.fits FP_TEMP echo+
153.60722351

Most data in the Chandra Archive has been updated as part of the bulk reprocessing effort known as Repro-4 effort. However, calibration and software changes have been introduced since the start of reprocessing. Users are therefore still encouraged to reprocess their data to ensure they have a consistent dataset.

Proceed to the next section to begin.


Get Started

Since this thread is meant to represent the most generic case, it is not written with a specific ObsID. Files are referred to by the detector and type - acisf01843_000N001_evt1.fits becomes acis_evt1.fits, hrcf01557_000N001_std_flt1.fits becomes hrc_std_flt1.fits, and so on.

Furthermore, it is assumed that all relevant files are in the same working directory.

Proceed to the next section.


Update aspect solution file

Chandra data processed with software version DS10.8.4 and later has the boresight corrections applied directly to the aspect solution. This provides users a better estimate of the off-axis angles. CIAO 4.13 includes the asp_offaxis_corr tool which applies this correction to the aspect solution.

The easiest way to check if this part of the thread is relevant is to check the CONTENT keyword in the aspect solution file: pcad*_asol1.fits.

unix% dmkeypar pcad_asol1.fits CONTENT echo+
ASPSOLOBI
[TIP]
Missing "*asol1.fits" files?

Chandra observations of the Moon and the Earth do not have standard aspect solution files, *_asol1.fits files; the Earth and Moon block the aspect camera from being able to track any stars. For these observations, processing relies on the On Board Computer (OBC) aspect solution computed using data from the gyros used to maneuver the spacecraft. The ASPTYPE keyword in the event file indicates if the OBC aspect solution is needed.

% dmkeypar 5311/secondary/hrcf05311_000N006_evt1.fits.gz ASPTYPE echo+
OBC

Observations other than the Moon an Earth use the standard ASPTYPE=KALMAN aspect solution files, ie the *_asol1.fits files.

The OBC aspect solution is stored in one or more *_osol1.fits files which are located in the secondary/aspect/ directory. Users can provide the list (stack) of OBC aspect solution files to the tools the in the same way as the standard Kalman aspect solution files. For example:

% /bin/ls secondary/aspect/pcadf*_osol1.fits > pcad_asol1.lis

Note: all observations include the OBC solution; however it is less accurate than standard Kalman filtered aspect solution and should only be used when the Kalman filtered aspect solution is not available.

There is no need to apply the boresight correction to the OBC aspect solution file(s). Users can proceed to the next section.

If CONTENT=ASPSOLOBI, then the boresight correction is already applied and you can proceed to the next section.

If you have multiple aspect solution files, then you should check them all. If there is one with CONTENT=ASPSOLOBI, then you should use the file and omit the other files and proceed to the next section.

If CONTENT=ASPSOL, then you should continue with this section. We will assume that the time-ordered list of the aspect solution file(s) are stored in a stack list, eg

unix% /bin/ls pcadf*_asol1.fits > pcad_asol.lis

First we need to combine all the aspect solution files. Because the aspect solution files may span a different time range then the current observation, we need to be sure we also apply a time filter to the files. We will take the time range from the TSTART and TSTOP values in the event file header.

unix% tstart=`dmkeypar acis_evt1.fits tstart echo+`  # -or- if using tcsh then use: set tstart=...stuff...
unix% tstop=`dmkeypar acis_evt1.fits tstop echo+`               # -or- if using tcsh then use: set tstart=...stuff...
unix% dmmerge "@pcad_asol.lis[time=${tstart}:${tstop}]" pcad_repro_asol1.fits

The time filter is applied to each file name listed in the stack.

The next step is to apply the boresight correction using the asp_offaxis_corr tool. This tool modifies the input file in place. We also need to update the CONTENT keyword to reflect the updated content

unix% asp_offaxis_corr pcad_repro_asol1.fits acis
unix% dmhedit pcad_repro_asol1.fits file="" op=add key=CONTENT value=ASPSOLOBI

This section is now complete. Proceed to the next section.


ACIS or HRC?

Question: which detector was used for your observation? (It does not matter yet whether or not a transmission grating was also used.)


ACIS Observations

Question: was the observation taken in continuous-clocking mode (i.e. the read mode is "continuous")?

unix% dmkeypar acis_evt1.fits readmode echo+
CONTINUOUS

Check Accuracy of Source Coordinates (CC-mode data only)

It is important to verify that the coordinates of the source are accurate and precise to less than 0.5 arcsec (i.e. one pixel). The source coordinates are used to determine the position-dependent absolute times of arrival and pulse heights for continuous-clocking mode data. If they are imprecise, then the output times of arrival and pulse heights will also be inaccurate.

The coordinates that should be used are the best-known right ascension (RA_TARG) and declination (DEC_TARG) coordinates for the source:

unix% dmlist acis_evt1.fits header | egrep '(RA|DEC)_TARG'
0057 RA_TARG                     83.6333330              Real8        Observer's specified target RA
0058 DEC_TARG                    22.0144720              Real8        Observer's specified target Dec

These coordinates have the correct level of accuracy precision.

If the coordinates need to be changed, then here are some suggestions about how to determine the values:

  • use previously-published source coordinate values,
  • if the coordinates are not well known, then try obtaining them from a Chandra observation of the source which was taken in TE mode,
  • if neither of the above methods are available, then it may be possible to use two, separate continuous-clocking mode observations to find the source coordinates, provided that the observations did not have the same roll angle.

It is not possible to fix the coordinates in the dataset if the position is not known from some other dataset or publication (i.e. you cannot determine accurate source coordinates from the dataset you are attempting to correct).

Once the new source coordinates are determined, modify the file header with dmhedit:

unix% dmhedit acis_evt1.fits filelist="" operation=add key=RA_TARG value='newvalue'
unix% dmhedit acis_evt1.fits filelist="" operation=add key=DEC_TARG value='newvalue'

Note that the ability to reconstruct the TIMEs of arrival is limited by

  • the uncertainties in the coordinates RA_TARG and DEC_TARG,
  • the uncertainties in the aspect reconstruction, and
  • the PSF "blur."

If RA_TARG and DEC_TARG are accurate to much less than a pixel, then the times of arrival should be accurate to about 4 ms for most observations (i.e. those without any unusual aspect problems).

Proceed to the next step.


Reset ACIS status bits

Status bits 1-5, 14-20 and 23 in the input event file are reset so that they can be updated by destreak and acis_process_events. This reset includes removing the effects of acis_run_hotpix and (previously) acis_detect_afterglow. More information is available in the cosmic-ray afterglow why topic. Details about what each status bit represents are available from the "STATUS bits in event files" memo.

The acis_clear_status_bits script clears the status bits in place; it does not create a new output file. The file must have write permission, and you may wish to make a copy before modifying it.

unix% acis_clear_status_bits acis_evt1.fits

The CLSTBITS header keyword is added to record which bits were cleared.

unix% dmkeypar acis_evt1.fits  CLSTBITS echo+
11111111011000000011111111000001

Proceed to the next step.


Remove streak events (Optional)

Running destreak on the ACIS-S4 chip (ccd_id=8) can improve the quality of your data. Removing the streaks before creating a new bad pixel file improves the streak detection efficiency and prevents misidentifying pixels with many streak events as hot pixels. For details on how the tool works, see the Destreaking ACIS Data why topic and "ahelp destreak".

Note that there is some evidence that for bright, continuum-dominated sources observed with the gratings, applying destreak removes more source events than streak events; see the destreak why topic for details. Furthermore, spectral order sorting alone will greatly reduce the amount of streak data present in any extracted spectrum. Therefore destreaking should be considered optional for this case.

The mask parameter is set to "None" so that all events in the input file are evaluated as possible streak events. The parameter setting filter=no flags the streak events in the output file, but does not remove them.

unix% punlearn destreak
unix% pset destreak infile=acis_evt1.fits
unix% pset destreak outfile=acis_dstrk_evt1.fits
unix% pset destreak mask=None filter=no
unix% destreak
Input dataset/block specification (acis_evt2.fits): 
Output dataset/block specification (acis_dstrk_evt2.fits): 

Question: was the observation taken in timed mode (i.e. the read mode is "TIMED", not "CONTINUOUS")?

unix% dmkeypar acis_dstrk_evt1.fits readmode echo+
TIMED

Create a new ACIS bad pixel file

This section of the thread does not apply when the input data were taken in ACIS continuous-clocking mode, as the ACIS afterglow tools cannot be used with this data mode.

There are three steps in creating a new ACIS bad pixel file:

  1. identify known bad pixels and columns from the calibration database and pixels with bad bias values (acis_build_badpix with a custom bitflag)

    [NOTE]
    New bitflag to include frame-store shadow

    Badpixel bit 17 has been added to the CALDB badpixel file in CALDB 4.9.0 and support for it was added in CIAO 4.12. The default acis_build_badpix bitflag value has changed to include bit 17.

  2. search for afterglows and hot pixels (acis_find_afterglow)

  3. mark pixels adjacent to afterglows and hot pixels and sort the list of bad pixels and afterglows (a second run of acis_build_badpix)

In order to run acis_build_badpix, you need an obs.par file for the observation, created by running dmmakepar

unix% dmmakepar acis_dstrk_evt1.fits acis_obs.par

You will also need a list of the bias files.

unix% cat bias.lis
acis_2_bias0.fits
acis_4_bias0.fits
acis_5_bias0.fits
acis_3_bias0.fits
acis_1_bias0.fits

Now run the tools:

unix% punlearn acis_build_badpix 
unix% pset acis_build_badpix obsfile=acis_obs.par
unix% pset acis_build_badpix pbkfile=acis_pbk0.fits  
unix% pset acis_build_badpix biasfile=@bias.lis 
unix% pset acis_build_badpix outfile=acis_abb1_bpix1.fits 
unix% pset acis_build_badpix bitflag=00000000000000120021100020022222
unix% pset acis_build_badpix calibfile=CALDB 
unix% acis_build_badpix mode=h verbose=0

unix% punlearn acis_find_afterglow
unix% pset acis_find_afterglow infile=acis_dstrk_evt1.fits
unix% pset acis_find_afterglow outfile=acis_aglow_bpix1.fits
unix% pset acis_find_afterglow badpixfile=acis_abb1_bpix1.fits 
unix% pset acis_find_afterglow maskfile=acis_msk1.fits
unix% pset acis_find_afterglow statfile=acis_stat1.fits
unix% acis_find_afterglow mode=h verbose=0

unix% punlearn acis_build_badpix 
unix% pset acis_build_badpix obsfile=acis_obs.par
unix% pset acis_build_badpix pbkfile=acis_pbk0.fits  
unix% pset acis_build_badpix biasfile=None
unix% pset acis_build_badpix outfile=acis_repro_bpix.fits
unix% pset acis_build_badpix calibfile=acis_aglow_bpix1.fits
unix% pset acis_build_badpix procbias=no
unix% acis_build_badpix mode=h verbose=0

The new bad pixel file is called acis_repro_bpix.fits.

Proceed to the next step.


Run acis_process_events to make a new level=1 event file

Running acis_process_events produces a new level=1 event file that has the latest calibration applied:

Determine the eventdef

The eventdef parameter specifies the names and data types of the columns in the output event data file. Six definitions are included in the parameter file for acis_process_events:

DATAMODE CONTENT event mode eventdef string
(V)FAINT EVT1 timed exposure (very) faint stdlev1
GRADED EVT1 timed exposure graded grdlev1
CC33_FAINT EVT1 continuous clocking faint cclev1
CC33_FAINT TGEVT1 continuous clocking faint with gratings cclev1a
CC33_GRADED EVT1 continuous clocking graded ccgrdlev1
CC33_GRADED TGEVT1 continuous clocking graded with gratings ccgrdlev1a

The event mode of an observation can be found in the DATAMODE and CONTENT values stored in the file header:

unix% dmkeypar acis_evt1.fits DATAMODE echo+
FAINT

unix% dmkeypar acis_evt1.fits CONTENT echo+
EVT1

This dataset is for a timed exposure faint mode observation without grating data. Therefore, the proper eventdef parameter is "stdlev1."

Set the parameters

  • If you created a new bad pixel file when running the Reprocessing Data to Create a New Level=2 Event File thread or the Customizing an ACIS Bad Pixel File thread, use that file in this analysis. Otherwise, use the bpix1.fits file from the Archive.

  • If the asp_offaxis_corr tool was used, as shown above, to update the aspect solution then you will need to also create a separate observation parameter file with updated pointing information. This is a different file then the file used to create the badpixel file.

    unix% dmmakepar acis_dstrk_evt1.fits"[events]" pcad_obs.par
    unix% ra=`dmkeypar pcad_repro_asol1.fits ra_pnt echo+`  # or for tcsh shell use: set ra=`dmkeypar pcad_repro_asol1.fits ra_pnt echo+`
    unix% dec=`dmkeypar pcad_repro_asol1.fits dec_pnt echo+`  # or for tcsh shell use: set dec=`dmkeypar pcad_repro_asol1.fits dec_pnt echo+`
    unix% roll=`dmkeypar pcad_repro_asol1.fits roll_pnt echo+`  # or for tcsh shell use: set roll=`dmkeypar pcad_repro_asol1.fits roll_pnt echo+`
    unix% pset pcad_obs.par ra_pnt=$ra dec_pnt=$dec roll_pnt=$roll
    unix% pset pcad_obs.par ra_nom=$ra dec_nom=$dec roll_nom=$roll
    unix% pset pcad_obs.par dy_avg=0.0 dz_avg=0.0 dth_avg=0.0
    

    If the asp_offaxis_corr tool was not necessary, then you should have a single aspect solution file, pcad_asol1.fits

  • The parameter for VFAINT background cleaning (check_vf_pha) may be set. The tool will ignore this parameter if your observation was not taken in this mode.

    The check_vf_pha algorithm should be used very carefully. As discussed in the why topic, it can misidentify valid source events as background events. The default for chandra_repro is check_vf_pha=no.

  • The parameter for a sub-pixel adjustment (pix_adj) is set to EDSER by default. However, since the sub-pixel adjustments affect not only the coordinates, but also the times of arrival for continuous clocking mode datasets, and since such adjustments can introduce non-astrophysical features in the times of arrival, the recommended setting for continuous clocking mode observations is pix_adj=NONE (i.e. no adjustments).

unix% punlearn acis_process_events
unix% pset acis_process_events infile=acis_dstrk_evt1.fits
unix% pset acis_process_events outfile=acis_new_evt1.fits
unix% pset acis_process_events badpixfile=acis_new_bpix1.fits
unix% pset acis_process_events acaofffile=pcad_asol1.fits
unix% pset acis_process_events mtlfile=acis_mtl1.fits
unix% pset acis_process_events eventdef=")stdlev1"
unix% pset acis_process_events check_vf_pha=yes

If the asp_offaxis_corr tool was used then also set these parameters:

unix% pset acis_process_events acaofffile=pcad_repro_asol1.fits
unix% pset acis_process_events obsfile=pcad_obs.par

If the DATAMODE is CC33_FAINT or CC33_GRADED, then also use the following.

unix% pset acis_process_events pix_adj=NONE

Run the tool

unix% acis_process_events
Input event file or stack (acis_evt1.fits): 
Output event file name (acis_new_evt1.fits): 
aspect offset file ( NONE | none | <filename>) (pcad_asol1.fits):

A number of possible warning messages are explained in the Common acis_process_events Warnings section.


Update Header Keywords

For some data, taken early in the Chandra mission, it will be necessary to update the event file headers with some keywords that were added during the last Chandra reprocessing (known as "Repro-4").

The r4_header_update script is used to add several keywords. If the keywords already exist then the tool will simply ignore them; so it is always safe to run it. The file is modified in place.

% dmkeypar acis_new_evt1.fits SUM_2X2 echo+
# dmkeypar (CIAO): ERROR: Keyword 'SUM_2X2' was not found in file 'acis_new_evt1.fits'.

unix% dmkeypar acis_new_evt1.fits DY_AVG echo+
# dmkeypar (CIAO): ERROR: Keyword 'DY_AVG' was not found in file 'acis_new_evt1.fits'.

unix% r4_header_update acis_new_evt1.fits
Missing keywords 'OCLKPAIR,ORC_MODE,SUM_2X2,FEP_CCD' from event file 'acis_new_evt1.fits' header.
Missing keywords 'DY_AVG,DZ_AVG,DTH_AVG' from event file 'acis_new_evt1.fits' header.

unix% dmkeypar acis_new_evt1.fits SUM_2X2 echo+
0
unix% dmkeypar acis_new_evt1.fits DY_AVG echo+
0.045452421825

If very old data is being reprocessed, it may be necessary to also specify the pbkfile and the asolfile parameters.


Question: is your observation imaging or grating data?


Imaging Observations

Filter on Grade, Status, and Good Time

There are two filtering steps for ACIS imaging observations:

  1. Filter for bad grades and for a "clean" status column (i.e. all bits set to 0):

    unix% punlearn dmcopy
    unix% dmcopy "acis_new_evt1.fits[EVENTS][grade=0,2,3,4,6,status=0]" \
          acis_flt_evt1.fits
    
  2. The Good Time Intervals (GTIs) supplied by the pipeline now need to be applied; these are stored in the acis_flt1.fits file.

    unix% punlearn dmcopy
    unix% dmcopy "acis_flt_evt1.fits[EVENTS][@acis_flt1.fits]" \
          acis_repro_evt2.fits
    

    Be sure to include the @ symbol in the filter expression; the command will not be executed properly if it is omitted.

    [TIP]
    Zero Exposure Time

    See the Zero-exposure Event File caveat for tips on how to deal with observations with no good time.

There are a few observations where ACIS begins to collect event data before the aspect solution is available. Under some rare circumstances these events are not removed by the GTI filter which results in the first few events in the Level 2 event file to contain invalid sky coords (beyond the 1 to 8192 range). These events can be removed with the following dmcopy command:

unix% dmcopy "acis_repro_evt2.fits[EVENTS][x=:,y=:]" acis_filter_xy_repro_evt2.fits

Users do not generally need to worry about these invalid event unless they are merging observations.

The thread is now complete. acis_repro_evt2.fits is a level=2 event file with the latest calibration products applied.


Grating Observations

There are CIAO threads that deal specifically with generating a level=2 event file for an ACIS grating observation. Follow the appropriate one, depending on which transmission grating was used:

Single source:

Multiple sources:

Once you leave this thread, you do not need to return to complete any further steps.


Common acis_process_events Warnings

It is important to remember that the following messages are warnings, not errors. The tool may print one of these messages, but will still create a valid event file.

WARNING: problem reading ctifile, cti adjustment will not be applied.
# acis_process_events (CIAO 4.5): WARNING: problem reading ctifile, cti adjustment will not be applied. 
                                  Changing apply_cti=yes to apply_cti=no.

This warning may also be triggered by product=T_GAIN.

The most common cause for this warning is that the observation does not contain -120 C data. Since the CTI and TGAIN are only calibrated for this focal plane temperature, it is not possible to use it on other observations.

Since no file is found, acis_process_events continues running as if apply_cti=no or apply_tgain=no and produces a valid output file that has not been CTI-corrected or had the TGAIN applied.

The parameter check_vf_pha=yes, but the DATAMODE key does not indicate a very-faint mode observation.
# acis_process_events (CIAO 4.5): WARNING: The parameter check_vf_pha=yes, but the DATAMODE key does not indicate
                                  a very-faint mode observation.  Changing to check_vf_pha=no.

ACIS very-faint background cleaning can only be performed on observations that were taken in VFAINT mode. If the check_vf_pha parameter is set to "yes" but the data has a DATAMODE other than VFAINT, acis_process_events continues to reprocess the data as if check_vf_pha=no and produces a valid output file that does not have the background cleaning applied.

The ra_targ, dec_targ, or roll_nom specified by pcad_obs.par does not match the values in the event file- using the obs.par values.
# acis_process_events (CIAO): WARNING: The ra_targ, dec_targ, or roll_nom specified by pcad_obs.par does not match the values in the event file- 
using the obs.par values.

This warning is expected when users have run the asp_offaxis_corr tool to apply the boresight correction to their aspect solution file. The updated pointing information in the aspect solution file (RA_PNT, DEC_PNT, and ROLL_PNT) now being used which differs from the old values. The PNT values in the updated aspect solution are used to update both the PNT values in the event file (mean optical axis location) and the NOM values in the event file (the tangent point location).


HRC Observations

All HRC-S (imaging and grating) observations taken after 01 January 2012, which were processed with a CalDB version released prior to the 4.5.1 release (24 July 2012), should be reprocessed to apply the updated HRC-S time-dependent gain map calibration released with CalDB 4.5.1. The High Voltage settings of the HRC-S microchannel plates were increased in March 2012, resulting in a corresponding gain increase and required change to HRC-S T_GAIN calibration.

Running hrc_process_events produces a new level=1 event file that has the latest calibration applied:

HRC-S: Determine the 'gainfile' parameter

On 29 March 2012, the High Voltage (HV) settings of the HRC-S microchannel plates were increased, causing a corresponding rise in the detector gain, and a required update to the HRC-S time-dependent gain map (T_GMAP) calibration. As a result, all observations taken after 01 January 2012 must be reprocessed using one of the two new calibration T_GMAP files available in CalDB 4.9.2:

        hrcsD1999-07-22t_gmapN0003.fits
        hrcsD2012-03-29t_gmapN0003.fits

If you are analyzing one of the ObsIDs listed below, you must manually set the 'gainfile' parameter of hrc_process_events to the appropriate CalDB path and name, as shown below.

ObsID HV setting Object Grating Start Date
14324 old Mkn421 LETG 2012-07-04 04:09:45
14396 old Mkn421 LETG 2012-07-03 19:09:24
14397 old Mkn421 LETG 2012-07-04 13:10:06
14238 new HZ43 LETG 2012-03-18 05:16:00
Old High-Voltage Setting:
unix% pset hrc_process_events gainfile ${CALDB}/data/chandra/hrc/t_gmap/hrcsD1999-07-22t_gmapN0003.fits

New High-Voltage Setting:
unix% pset hrc_process_events gainfile ${CALDB}/data/chandra/hrc/t_gmap/hrcsD2012-03-29t_gmapN0003.fits

Otherwise, you may run hrc_process_events with 'gainfile' set to the default 'CALDB', and the correct time-dependent gain map will automatically be applied.

Set the parameters

  • If the asp_offaxis_corr tool was used, as shown above, to update the aspect solution then you will need to also create a separate observation parameter file with updated pointing information. This is a different file then the file used to create the badpixel file.

    unix% dmmakepar hrc_evt1.fits"[events]" pcad_obs.par
    unix% ra=`dmkeypar pcad_repro_asol1.fits ra_pnt echo+`  # or for tcsh shell use: set ra=`dmkeypar pcad_repro_asol1.fits ra_pnt echo+`
    unix% dec=`dmkeypar pcad_repro_asol1.fits dec_pnt echo+`  # or for tcsh shell use: set dec=`dmkeypar pcad_repro_asol1.fits dec_pnt echo+`
    unix% roll=`dmkeypar pcad_repro_asol1.fits roll_pnt echo+`  # or for tcsh shell use: set roll=`dmkeypar pcad_repro_asol1.fits roll_pnt echo+`
    unix% pset pcad_obs.par ra_pnt=$ra dec_pnt=$dec roll_pnt=$roll
    unix% pset pcad_obs.par ra_nom=$ra dec_nom=$dec roll_nom=$roll
    unix% pset pcad_obs.par dy_avg=0.0 dz_avg=0.0 dth_avg=0.0
    

    If the asp_offaxis_corr tool was not necessary, then you should have a single aspect solution file, pcad_asol1.fits

unix% punlearn hrc_process_events
unix% pset hrc_process_events infile=hrc_evt1.fits
unix% pset hrc_process_events outfile=hrc_new_evt1.fits
unix% pset hrc_process_events badpixfile=hrc_bpix1.fits
unix% pset hrc_process_events acaofffile=pcad_asol1.fits
unix% pset hrc_process_events badfile=NONE
unix% pset hrc_process_events do_amp_sf_cor=yes

If the asp_offaxis_corr tool was used then also set these parameters:

unix% pset hrc_process_events acaofffile=pcad_repro_asol1.fits
unix% pset hrc_process_events obsfile=pcad_obs.par

Run the tool

unix% hrc_process_events
input level 0 event file/stack (hrc_evt1.fits): 
output level 1 file (hrc_new_evt1.fits): 
bad pixel file ( NONE | none | <filename>) (hrc_bpix1.fits): 

A number of possible warning messages are explained in the Common hrc_process_events Warnings section.


Update Header Keywords

For some data, taken early in the Chandra mission, it will be necessary to update the event file headers with some keywords that were added during the last Chandra reprocessing (known as "Repro-4").

The r4_header_update script is used to add several keywords. If the keywords already exist then the tool will simply ignore them; so it is always safe to run it. The file is modified in place.

unix% dmkeypar hrc_new_evt1.fits DY_AVG echo+
# dmkeypar (CIAO): ERROR: Keyword 'DY_AVG' was not found in file 'hrc_new_evt1.fits'.

unix% r4_header_update hrc_new_evt1.fits
Missing keywords 'DY_AVG,DZ_AVG,DTH_AVG' from event file 'hrc_new_evt1.fits' header.

unix% dmkeypar hrc_new_evt1.fits DY_AVG echo+
0.045452421825

If very old data is being reprocessed, it may be necessary to also specify the asolfile parameter.

Note: The set of keywords that are updated are different for ACIS and HRC data.

[New]Identify and Address HRC Secondary Science Corruption (optional)

Roughly 1 in 3 HRC observations have occurrences of Secondary Science Corruption (SSC). For most observations this results in a slight decrease in the total amount of good time (on the order of less than 1%) and a slight decrease in the estimated dead-time correction for the observation (again, on the order of less than 1%).

Users can run the patch_hrc_ssc script to check to see if the observation has any SSC occurrences and replace the suspect dead time factors values during the SSC with the median value. The script also computes new Good Time Intervals. The script can be run using the following parameters:

unix% pset patch_hrc_ssc dtf_infile=hrc_dtf1.fits 
unix% pset patch_hrc_ssc mtl_infile=hrc_mtl1.fits 
unix% pset patch_hrc_ssc evt_infile=hrc_new_evt1.fits
unix% pset patch_hrc_ssc evt_outfile=hrc_new_ssc_evt1.fits
unix% pset patch_hrc_ssc gti_outfile=hrc_new_flt1.fits 
unix% pset patch_hrc_ssc dtf_outfile=hrc_new_dtf1.fits 
unix% patch_hrc_ssc mode=h clob+

The output dead time factors file: hrc_new_dtf1.fits replaces the archive hrc_dtf1.fits in all later processing and data analysis.

The files hrc_new_ssc_evt1.fits and hrc_new_flt1.fits should be used in place of the original files in the next steps in this thread.


Question: is your observation HRC-S imaging, HRC-I imaging, or grating data?


HRC-S Imaging Observations

There are two filtering steps for HRC-S imaging observations:

  1. Apply the status filter that is specific to HRC-S observations; a value of 0 demands that the bit be flagged as "good", a value of x indicates that either status (0/1) is acceptable. The PI filter removes about 20% of the background with no X-ray losses:

    unix% punlearn dmcopy
    unix% dmcopy \
          "hrc_new_evt1.fits[status=xxxxxx00xxxx0xxx0000x000x00000xx]" \
          hrc_flt_evt1.fits
    
  2. The Good Time Intervals (GTIs) supplied by the pipeline now need to be applied; these are stored in the hrc_std_flt1.fits file.

    unix% punlearn dmcopy
    unix% dmcopy \
          "hrc_flt_evt1.fits[EVENTS][@hrc_std_flt1.fits]" \
          hrc_repro_evt2.fits
    

    Be sure to include the @ symbol in the filter expression; the command will not be executed properly if it is omitted.

    [TIP]
    Zero Exposure Time

    See the Zero-exposure Event File caveat for tips on how to deal with observations with no good time.

Computing Average HRC Dead Time Corrections

The average HRC deadtime correction is computed using the tool hrc_dtfstats to ensure the accuracy of the deadtime statistics. (The deadtime should be recomputed after any time filtering is applied to an HRC dataset, e.g. applying GTIs or filtering background flares.)

unix% punlearn hrc_dtfstats
unix% pset hrc_dtfstats infile=hrc_dtf1.fits
unix% pset hrc_dtfstats outfile=hrc_dtfstats_new.fits
unix% pset hrc_dtfstats gtifile=hrc_repro_evt2.fits"[gti]"
unix% hrc_dtfstats
Input file (hrc_dtf1.fits): 
Output file (hrc_dtfstats_new.fits): 
File containing GTI to filter on (<filename>|NONE) (hrc_repro_evt2.fits[gti]): 

In this example, the new dtfstats file has a value of DTCOR=0.35535:

unix% dmlist hrc_dtfstats_new.fits"[cols DTCOR]" data,clean
#  DTCOR
     0.35535140943668

The ONTIME header value is multiplied by the new DTCOR value, and the LIVETIME and EXPOSURE header keyword values are updated with dmhedit. The commands in general are:

unix% dmkeypar hrc_repro_evt2.fits ONTIME echo+

ONTIME * NEWDTCOR = NEWTIME

unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=LIVETIME value=NEWTIME
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=EXPOSURE value=NEWTIME
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=DTCOR value=NEWDTCOR

The following commands use example values to illustrate:

unix% dmkeypar hrc_repro_evt2.fits ONTIME echo+
2189.40009910

2189.40009910*0.35535140943668 = 778.00641103599186717498

unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=LIVETIME value=778.0064110
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=EXPOSURE value=778.0064110
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=DTCOR value=0.35535140943668

The thread is now complete; hrc_repro_evt2.fits is a level=2 event file with the latest calibration products applied.

User may optionally want to apply a 'pi' based filter to their data as discussed in the HRC PI filtering thread.


HRC-I Imaging Observations

There are two filtering steps for HRC-I imaging observations:

  1. Apply the status filter that is specific to HRC-I observations; a value of 0 demands that the bit be flagged as "good", a value of x indicates that either status (0/1) is acceptable:

    unix% punlearn dmcopy
    unix% dmcopy \
          "hrc_new_evt1.fits[status=xxxxxx00xxxx0xxx00000000x0000000]"  \
          hrc_flt_evt1.fits
    
  2. The Good Time Intervals (GTIs) supplied by the pipeline now need to be applied; these are stored in the hrc_std_flt1.fits file.

    unix% punlearn dmcopy
    unix% dmcopy \
          "hrc_flt_evt1.fits[EVENTS][@hrc_std_flt1.fits]" \
          hrc_repro_evt2.fits
    

    Be sure to include the @ symbol in the filter expression; the command will not be executed properly if it is omitted.

    [TIP]
    Zero Exposure Time

    See the Zero-exposure Event File caveat for tips on how to deal with observations with no good time.

Computing Average HRC Dead Time Corrections

The average HRC deadtime correction is computed using the tool hrc_dtfstats to ensure the accuracy of the deadtime statistics. (The deadtime should be recomputed after any time filtering is applied to an HRC dataset, e.g. applying GTIs or filtering background flares.)

unix% punlearn hrc_dtfstats
unix% pset hrc_dtfstats infile=hrc_dtf1.fits
unix% pset hrc_dtfstats outfile=hrc_dtfstats_new.fits
unix% pset hrc_dtfstats gtifile=hrc_repro_evt2.fits"[gti]"
unix% hrc_dtfstats
Input file (hrc_dtf1.fits): 
Output file (hrc_dtfstats_new.fits): 
File containing GTI to filter on (<filename>|NONE) (hrc_repro_evt2.fits[gti]): 

In this example, the new dtfstats file has a value of DTCOR=0.35535:

unix% dmlist hrc_dtfstats_new.fits"[cols DTCOR]" data,clean
#  DTCOR
     0.35535140943668

The ONTIME header value is multiplied by the new DTCOR value, and the LIVETIME and EXPOSURE header keyword values are updated with dmhedit. The commands in general are:

unix% dmkeypar hrc_repro_evt2.fits ONTIME echo+

ONTIME * NEWDTCOR = NEWTIME

unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=LIVETIME value=NEWTIME
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=EXPOSURE value=NEWTIME
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=DTCOR value=NEWDTCOR

The following commands use example values to illustrate:

unix% dmkeypar hrc_repro_evt2.fits ONTIME echo+
2189.40009910

2189.40009910*0.35535140943668 = 778.00641103599186717498

unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=LIVETIME value=778.0064110
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=EXPOSURE value=778.0064110
unix% dmhedit hrc_repro_evt2.fits filelist="" op=add key=DTCOR value=0.35535140943668

The thread is now complete; hrc_repro_evt2.fits is a level=2 event file with the latest calibration products applied.

User may optionally want to apply a 'pi' based filter to their data as discussed in the HRC PI filtering thread.


Grating Observations

There are CIAO threads that deal specifically with generating a level=2 event file for an HRC grating observation. Follow the appropriate one, depending on which detector was used:

Single source:

Multiple sources:

Once you leave this thread, you do not need to return to complete any further steps.


Common hrc_process_events Warnings

It is important to remember that the following messages are warnings, not errors. The tool may print one of these messages, but will still create a valid event file.

mjd_obs keyword is missing in infile. value from computation.
# hrc_process_events (CIAO 4.5): WARNING: mjd_obs keyword is missing in infile. value from computation.

Some older event files do not have the MJD_OBS header keyword. In this case, the tool uses the DATE-OBS and MJDREF values to compute the correct MJD_OBS.

Out of sequence events discovered in hrc_evt1.fits.
# hrc_process_events (CIAO 4.5): The following error occurred 2482 times: dsHPEEVENTSEQERR - 
                                 WARNING: Out of sequence events discovered in hrc_evt1.fits.

There are occasional events that appear in the telemetry stream with time tags that make them seem to have occurred out-of-sequence. One special case where this occurs has been documented (see the Out-of-sequence HRC Time Tags memo); other occurrences are most likely to be caused by hiccups in the HRC or by double-bit errors in telemetry (single-bit errors are corrected).

This warning may be ignored in most cases, as long as the number of events flagged as out-of-sequence is a small fraction of the total number of events in the file; see the hrc_process_events bug page for more information.

The ra_targ, dec_targ, or roll_nom specified by pcad_obs.par does not match the values in the event file- using the obs.par values.
# hrc_process_events (CIAO): WARNING: The ra_targ, dec_targ, or roll_nom specified by pcad_obs.par does not match the values in the event file- 
using the obs.par values.

This warning is expected when users have run the asp_offaxis_corr tool to apply the boresight correction to their aspect solution file. The updated pointing information in the aspect solution file (RA_PNT, DEC_PNT, and ROLL_PNT) now being used which differs from the old values. The PNT values in the updated aspect solution are used to update both the PNT values in the event file (mean optical axis location) and the NOM values in the event file (the tangent point location).


History

22 Dec 2004 updated for CIAO 3.2: minor changes to parameter files; use ACIS bad pixel file (badpixfile parameter)
12 Jan 2005 grating sections now include links to multiple-source analysis threads
01 Feb 2005 added note about "Event island contains 1 or more bad pixels" warning to acis_process_events section
20 Jun 2005 CIAO 3.2.2 patch: minor acis_process_events parameter change (default value of threshfile is CALDB instead of NONE)
12 Dec 2005 updated for CIAO 3.3: the CTI and time-dependent gain corrections may be applied to CC-mode data (see the Continuous Clocking Mode why topic for details); parameter file updates for destreak
01 Dec 2006 updated for CIAO 3.4: CIAO version in errors and warnings
08 Feb 2008 updated for CIAO 4.0: links point to why topics and dictionary entries instead of other ACIS and HRC threads, which have been deprecated
31 Mar 2008 updated for CALDB 3.4.3: new -120 C TGAIN files and first -110 C TGAIN file: both changes made in Do I have to run this thread? section
16 Jun 2008 added CTI correction for back-illuminated chips to Do I have to run this thread? section
07 Jan 2009 updated for CIAO 4.1: acis_process_events no longer prints the "Event island contains 1 or more bad pixels." warning when verbose=0; other warnings were updated
18 Dec 2009 updated for CIAO 4.2: new ACIS TGAIN, HRC-I TGAIN, and HRC-S TGAIN released in CALDB 4.2.0
28 Dec 2009 additional updates for CIAO 4.2: HRC imaging - "pha=0:254" filter is replaced by "pi=0:300"
20 Jan 2010 linked to Customizing an ACIS Bad Pixel File thread; include "[cols -status]" when running acis_process_events
18 Feb 2010 added the MJD_OBS hrc_process_events warning
01 Jul 2010 the chandra_repro reprocessing script automates data processing for ACIS imaging data
02 Sep 2010 incorporated ACIS very-faint background cleaning into this thread; destreak mask parameter updated for VFAINT data
27 Sep 2010 added "Computing Average Dead Time Corrections" section for HRC data
27 Oct 2010 added information on DS 8.3.3 in "do I have to run this thread?" section
15 Dec 2010 updated for CIAO 4.3: new ACIS sub-pixel event repositioning; new ACIS calibration; new HRC-I gain map; removed DS 8.3.3 warning from "do I have to run this thread?" section since the calibration now matches CIAO 4.3 and CALDB 4.4.1.
25 Feb 2011 the 25 Feb release of the chandra_repro reprocessing script supports data processing for ACIS and HRC grating data
04 Mar 2011 added Check Accuracy of Source Coordinates section for continuous-clocking mode data
26 Jul 2011 required software updates are listed in Synopsis
15 Dec 2011 reviewed for CIAO 4.4: reset status bits before running destreak, run destreak before creating a new bad pixel file, use acis_find_afterglow in place of acis_run_hotpix
27 Feb 2012 added link to HRC-S Event Position Errors Near the Aim Point webpage in the Synopsis.
24 Jul 2012 added notice of HRC-S T_GAIN calibration changes to the Synopsis
13 Dec 2012 Review for CIAO 4.5. Many of the details in software and calibration changes were related to repro-3; repro-4 is now underway with additional changes. Added additional clarification that check_vf_pha=no is the default in chandra_repro due to the possibility of excluding true source events. Added note about predictive TGAIN in SDP.
24 Apr 2013 Added info about scripts release 4.5.2 that includes creation of ARF and RMF for grating datasets. Also make clear that after chandra_repro is run, the thread is complete.
27 Jun 2013 Removed the pi=0:300 filter from HRC-S/No Grating section based on the recommendation from the calibration team in connection with the CALDB 4.5.7 release.
14 Jul 2014 Reviewed for CIAO 4.6; added r4_header_update steps, if needed after acis_process_event or hrc_process_events.
13 Nov 2014 Clarified HRC-S t-gain file pick for observation's high-voltage setting.
02 Dec 2014 Updated for removal of acis_process_events calc_cc_times parameter from CIAO 4.7.
13 Apr 2015 Added a note about optional extra dmcopy command to remove rare instance invalid sky coordinates.
10 Dec 2015 Updated for CC mode changes in CIAO 4.8.
31 Aug 2016 Updated acis_process_events pix_adj parameter for CC mode datasets.
05 Nov 2019 Updated acis_build_badpix bitflag for CIAO 4.12 changes to mark the bottom of each CCD as "bad" due to the shadow cast by the framestore shield.
04 Dec 2020 Added new asp_offaxis_corr tool to apply boresight corrections to the aspect solution, in necessary.
13 Jan 2022 Reviewed for CIAO 4.14. No changes.
18 Jan 2024 Updated to remove make_par script since dmhedit now correctly positions the units string in the comments.
25 Mar 2024 Added a note about "OBC" (On Board Computer) mode for observations of the Moon and Earth and added a link to the 0 exposure caveat.
12 Dec 2024 Added a new step for HRC processing to check for secondary science corruption.