Search https://cxc.harvard.edu/ciao/
Contact the CXC HelpDesk
Last modified: 24 January 2022

URL: https://cxc.cfa.harvard.edu/ciao/threads/hrc_dtfstats/
/ ciao / threads / hrc_dtfstats /
Return to Threads Page: Top | All | Data Prep

Computing Average HRC Dead Time Corrections

CIAO 4.16 Science Threads

Overview

Synopsis:

HRC deadtime corrections are determined as a function of time from detector total event and valid event counters (usually sampled every 2.05 seconds but may differ, depending on telemetry format) and written to a deadtime factor (dtf1) file. The average deadtime correction (DTCOR) for an observation is computed from the dtf1 file, filtered by the relevant good time intervals, and is applied to the corresponding ONTIME to compute the LIVETIME (and EXPOSURE) of the observation. If the user applies different time filters then the DTCOR, LIVETIME, and EXPOSURE keyword values should be updated to match.

Purpose:

To recompute deadtime statistics from an HRC deadtime factor file (dtf1.fits) with different time filters from those used in standard processing, and to use them to update the DTCOR, LIVETIME, and EXPOSURE of the observation.

[NOTE]
Note

This thread is needed to correct the count rates (counts/sec) when doing photometry. Light curves and exposure maps take the dtf1.fits file in as a separate input and do not require the DTCOR keyword value be updated in this way.

Run this thread if:

The event file was filtered on time, changing the good time intervals (GTIs), e.g. as a result of application of a background. Time filters may also be applied if the deadtime factors have been flagged as variable in the standard deadtime statistics (std_dtfstat1) file.

This thread is also required if your data was processed with software version 7.6.4 through 7.6.7: a bug in HRC standard processing led to the use of incorrect good time intervals (GTIs) in the calculation of DTCOR in the dtfstats file, and hence the LIVETIME and EXPOSURE. Get Started shows how to determine what version of the software has been applied to your data.

Related Links:

Last Update: 24 Jan 2022 - Review for CIAO 4.14. Updated for Repro-5 and CALDB 4.9.6

Contents

Get Started

Download the sample data: 3698 (HRC-I, CAS A)

unix% download_chandra_obsid 3698 evt2,dtf

This thread should be done after any necessary reprocessing is completed (e.g. running chandra_repro).

In this thread, we assume that all relevant files are in the same working directory.

The software version is stored in the ASCDSVER header keyword: 

unix% dmkeypar hrcf03698_000N006_dtf1.fits ASCDSVER echo+
10.10

This example will use a time filter on the event file. 

unix% dmcopy "hrcf03698N006_evt2.fits[time=164259750:164261850]" filter_evt2.fits

The time range corresponds to a period of quiescent background. We can check that the time filter updated the ONTIME and related keywords; however, the DTCOR value is not automatically updated 

unix% dmlist hrcf03698N006_evt2.fits  header,clean | egrep 'DTCOR|ONTIME|LIVETIME|EXPOSURE'
ONTIME                    5171.3814679682 [s]       
DTCOR                    0.755343479530             Dead time correction
LIVETIME                  3906.1692719921           Ontime multiplied by DTCOR
EXPOSURE                  3906.1692719921           Total exposure time, with all known corr. appl.

unix% dmlist filter_evt2.fits header,clean | egrep 'DTCOR|ONTIME|LIVETIME|EXPOSURE'
ONTIME                    2100.0 [s]                
DTCOR                    0.755343479530             Dead time correction
LIVETIME                  1586.2213070130           Ontime multiplied by DTCOR
EXPOSURE                  1586.2213070130           Total exposure time, with all known corr. appl.

Examine the deadtime corrections

We can find the average, mean, deadtime correction in the dtf1.fits file using dmstat: 

unix% dmstat "hrcf03698_000N006_dtf1.fits[cols dtf]"
DTF
    min:    0         @:    1 
    max:    1         @:    2 
   mean:    0.76564416062 
  sigma:    0.270005164 
    sum:    2081.0208286 
   good:    2718 
   null:    0

This value, 0.766, is slightly different from the value in the header of the event file 

unix% dmkeypar filter_evt2.fits DTCOR echo+
0.75534347953

The value in the event file is computed with hrc_dtfstats which uses data during the good time intervals and is computed using a good-time weighted average.

The other thing to note is the large standard deviation (sigma) reported by dmstat. This indicates that the DTF values are highly variable. It can therefore be useful to examine the files in ChIPS: 

unix% python
>>> import matplotlib.pylab as plt
>>> from pycrates import read_file
>>> 
>>> infile="hrcf03698_000N006_dtf1.fits"
>>> tab = read_file(infile)
>>> tt = tab.get_column("time")
>>> dd = tab.get_column("dtf")
>>> 
>>> xlab = tt.name + (" ({})".format(tt.unit) if len(tt.unit) else "")
>>> ylab = dd.name + (" ({})".format(dd.unit) if len(dd.unit) else "")
>>> 
>>> plt.plot(tt.values, dd.values)
>>> plt.xlabel(xlab)
>>> plt.ylabel(ylab)
>>> plt.title(infile)
>>> plt.axvline(164259750, color="red")
>>> plt.axvline(164261850, color="red")
>>> 
>>> plt.savefig("plot.png")

The plot is shown Figure 1. The dead time factors are highly variable due to a combination of the bright extended source and mildly varying background.

Figure 1: Plot of deadtime correction value vs. time

[Thumbnail image: The plot of time (s) vs DTF is highly variable.]

[Version: full-size]

[Print media version: The plot of time (s) vs DTF is highly variable.]

Figure 1: Plot of deadtime correction value vs. time

The dead time factors are highly variable due to a combination of the bright extended source and mildly varying background.

The red lines indicate the time range that is being used in this example. The individual DTF values in this range are much closer to 1 and therefore the DTCOR value needs to be recomputed.

Since a time filter was applied over a time range where the DTF values are different from the mean, the DTCOR value must be recomputed.

Recompute the deadtime statistics

The new GTI created by the time filter needs to be used to recompute the deadtime statistics using the hrc_dtfstats tool. This tool takes the existing dtf1.fits file and creates a new file with the event GTI information applied. 

unix% punlearn hrc_dtfstats
unix% pset hrc_dtfstats infile=hrcf03698_000N006_dtf1.fits
unix% pset hrc_dtfstats outfile=dtf.stats
unix% pset hrc_dtfstats gtifile=filter_evt2.fits
unix% hrc_dtfstats
Input file (hrcf03698_000N006_dtf1.fits): 
Output file (dtf.stats): 
File containing GTI to filter on (<filename>|NONE) (filter_evt2.fits):

The contents of the parameter file may be checked using plist hrc_dtfstats.

The new dtfstats file has a value of DTCOR=0.964 

unix% dmlist dtf.stats"[cols DTCOR]" data,clean
#  DTCOR
     0.96380777655308

which agrees with the range of values plotted in Figure 1.

Update the Event File

Finally, we update the event list header with dmhedit. 

unix% dmhedit filter_evt2.fits file= op=add key=DTCOR value=0.963807

The LIVETIME and ONTIME also need to be corrected for the change in DTCOR. They will automatically get updated whenever any time filter is applied to the file. We can use an empty filter to trigger the update 

unix% dmcopy filter_evt2.fits"[time=:]" dtcorr_evt2.fits

We can check the results using dmdiff 

unix% dmdiff filter_evt2.fits dtcorr_evt2.fits data=no comments=no
Infile 1:  filter_evt2.fits
Infile 2:  dtcorr_evt2.fits

----------------------------------------------------------------------
Compare Headers
----------------------------------------------------------------------
Compare Key Lists:
Compare Keyword Details:
Compare Keyword Values:
Keyword: Message:                                                        Value(s):                    Diff:
-------- -------------------------------------- ---------------------------------- ------------------------
CHECKSUM Values are not equal                   Y6eha3cfZ3cfa3cf  7e1R7b1Q7b1Q7b1Q
HISTNUM  Values are not equal                                485  492                        +7 (+1.44%)
LIVETIME Values are not equal                     1586.221307013  2023.9963307615      +437.775 (+27.6%)
EXPOSURE Values are not equal                     1586.221307013  2023.9963307615      +437.775 (+27.6%)
...

Which shows the expected change in the LIVETIME and the EXPOSURE keywords.

The corrected event list, dtcorr_evt2.fits, should be used for all further analysis. 


Parameters for /home/username/cxcds_param/hrc_dtfstats.par


        infile = hrcf03698_000N006_dtf1.fits Input file
       outfile = dtf.stats        Output file
       gtifile = filter_evt2.fits File containing GTI to filter on (<filename>|NONE)
    (lookupTab = ${ASCDS_CALIB}/dmmerge_header_lookup.txt -> /soft/ciao/data/dmmerge_header_lookup.txt) lookup table
      (maincol = DTF)             Name of the deadtime factor column
       (errcol = DTF_ERR)         Name of the deadtime factor error column
     (chisqlim = 5)               Limit for the variability test
      (clobber = no)              Clobber the output file if it exists
      (verbose = 0)               Verbose level.
         (mode = ql)

History

01 Dec 2006 new for CIAO 3.4
25 Jan 2008 updated for CIAO 4.0: updated ChIPS plotting; added dmstat command; new lookupTab parameter for header merging rules
24 Jun 2008 updated image display to place figures inline with text
06 Feb 2009 updated for CIAO 4.1: Python and S-Lang syntax included for ChIPS plotting
05 Feb 2010 updated for CIAO 4.2: ChIPS version
13 Jan 2011 reviewed for CIAO 4.3: no changes
15 Feb 2011 added link to the HRC Deadtime Correction with Telemetry Saturation caveat
28 Jun 2011 The CIAO 4.3 version of hrc_dtfstats should not be run on data processed with SDP version DS 8.4 (released 28 June 2011) or higher
15 Dec 2011 reviewed for CIAO 4.4: the hrc_dtfstats tool computes a simple average instead of an error-weighted average; minor updates to screen output. This software update makes the tool consistent with standard data processing (SDP); running this thread on recently-processed datasets will no longer introduce a regression into the data processing.
03 Dec 2012 Review for CIAO 4.5
02 Dec 2013 Review for CIAO 4.6
15 Jul 2014 Updated to use different OBS_ID that shows time variation within an observation.
18 Dec 2014 Reviewed for CIAO 4.7; no changes.
02 Apr 2019 Update to use matplotlib plotting.
24 Jan 2022 Review for CIAO 4.14. Updated for Repro-5 and CALDB 4.9.6
Return to Threads Page: Top | All | Data Prep
/ ciao / threads / hrc_dtfstats /