Filtering Data

Time Filtering in General

Applying time filters to an event file involves four primary steps:

1. Define the filter

   Determine a filtering criterion that can be expressed as a valid dmgti userlimit parameter.

2. Create a Good Time Interval

   A good time interval (GTI) indicates which data are "good", based on the chosen filter definition. The tool dmgti is used to create a GTI table:

   unix% punlearn dmgti
   unix% pset dmgti infile=filename_mtl2.fits
   unix% pset dmgti outfile=myfilter_gti.fits
   unix% pset dmgti userlimit="<filtering criterion>"
   unix% dmgti
   

3. Apply the GTI table

   Apply the GTI table to the event fits file with dmcopy:

   unix% punlearn dmcopy
   unix% dmcopy "filename_evt2.fits[@myfilter_gti.fits]" filename_evt2_flt.fits 
   

4. Verify filtering

   Looking at the EXPOSURE keyword(s) is one way to see the results of filtering; this will show the amount of time that was eliminated in creating the filtered event file:

   unix% dmlist event_filename.fits header | grep EXPO
   unix% dmlist event_filename_flt.fits header | grep EXPO
   

   The tool dmkeypar may also be used to examine specific header keywords:

   unix% dmkeypar event_filename.fits EXPOSUR7 echo+
   

Eliminate High Background Times

The Chandra Calibration team notes the following in its study of the ACIS background available from the Calibration page: "A phenomenon not anticipated prior to launch that can seriously affect the scientific value of an observation is background flares, when the count rate can increase by a factor of up to 100. Such flares have been observed anywhere in the orbit, including near the apogee." Therefore, eliminating high-background periods is another type of filtering that might be required for your data:

Here is one example of how to eliminate periods of high background. The ACIS background and Filtering lightcurves threads also discuss ways of filtering out flares.

1. View the image

   An event file may be viewed directly by ds9:

   unix% ds9 acis_1843_evt2_0.3-10.fits &
   

   A log scale (Scale → log) and some adjustment to the color table show the object and the background. It may also be necessary to adjust the blocking factor to view the whole image at once; try "Bin → block 8" for a reasonable view.

2. Select the background area

   From this image, regions may be created to define the background area. In this example, the background will be defined as the entired dataset minus the selected source region.

   Select "Region → Shape → Ellipse" from the ds9 toolbar, then hold down the left mouse button at the center of the field, and drag until the ellipse covers the desired area. Clicking on the resulting shape will select the region, allowing you to reposition or resize it. This region (Figure 1) is used for the example.

   Figure 1: Selecting a region of interest

   

   [Version: full-size]

   

   Figure 1: Selecting a region of interest

   This source region will be excluded from the data in order to create a background lightcurve.

   The defined region is saved using the following steps:

   1. Region → Save Regions... → Save As "obj.reg".
   2. After choosing "OK" in the region filename dialog, a format dialog is opened. Set the format to "CIAO" and the coordinate system to "Physical".

   The resulting file is:

   unix% cat obj.reg 
   # Region file format: CIAO version 1.0
   ellipse(1628,4116,100,140,0)
   

3. Extract a background-only lightcurve

   Now it is possible to create a background lightcurve by using the "virtual file" syntax. We do this by specifying the whole field, minus the source object(s). In this case, the ellipse that is defined in obj.reg is omitted; note that we could have also used the exclude syntax to ignore the source (i.e. "exclude sky=region(obj.reg)" in the infile parameter:

   unix% punlearn dmextract
   unix% pset dmextract \
         infile="acis_1843_evt2_0.3-10.fits[sky=field()-ellipse(1628,4116,100,140,0)][bin time=::3.24104]"
   unix% pset dmextract outfile=background_lc.fits
   unix% pset dmextract opt=ltc1
   unix% dmextract
   Input event file  (acis_1843_evt2_0.3-10.fits[sky=field()-ellipse(1628,4116,100,140,0)][bin time=::3.24104]): 
   Enter output file name (background_lc.fits): 
   

   You can check the resulting parameter file with plist dmextract. Here the shorthand "sky" was used to specify x,y coordinates. Since "sky" is an alias of "(x,y)", both work in this context.

4. Examine the lightcurve and define a filter

   The lightcurve that was just created can be plotted in python

   unix% python
   
   >>> from pycrates import read_file
   >>> import matplotlib.pylab as plt
   >>> 
   >>> tab = read_file("background_lc.fits")
   >>> xx = tab.get_column("time").values
   >>> yy = tab.get_column("count_rate").values
   >>> 
   >>> plt.plot(xx,yy,marker="o",linestyle="")
   >>> plt.xlabel("Time (s)")
   >>> plt.ylabel("Count Rate (c/s)")
   >>> plt.title("G21.5-0.9 [Chip S3, T=120, Offsets=-1,20,-4]")
   >>> plt.show()
   

   The plot is shown in Figure 2.

   Figure 2: Lightcurve of the background

   

   

   Figure 2: Lightcurve of the background

   The lightcurve may also be examined with dmstat:

   unix% dmstat "background_lc.fits[cols count_rate]" 
   COUNT_RATE[count/s]
       min:        0             @:        1
       max:        120.31249977          @:        1037
      mean:        28.919604201
     sigma:        19.953609176
       sum:        86961.249832
      good:        3007
      null:        0
   

   Use dmlist to locate the region of maximum count rate:

   unix% dmlist "background_lc.fits[cols time,count_rate]" data
   ...
     1028  84274232.3370999992        78.7499998474
     1029  84274235.5781400204        97.4999998111
     1030  84274238.8191800117        91.8749998220
     1031  84274242.0602200031        91.8749998220
     1032  84274245.3012599945       109.6874997875
     1033  84274248.5423000157       104.6874997972
     1034  84274251.7833400071        91.8749998220
     1035  84274255.0243799984        95.3124998153
     1036  84274258.2654200196       100.3124998056
     1037  84274261.5064600110       120.3124997669
     1038  84274264.7475000024       107.4999997917
     1039  84274267.9885400236        66.5624998710
     1040  84274271.2295800149        69.3749998656
     1041  84274274.4706200063        67.4999998692
   ...
   

   The filter could also include an estimate for the lower limit of the count rate column:

   unix% dmlist "background_lc.fits[cols time,count_rate][count_rate>65]" data
   
   --------------------------------------------------------------------------------
   Data for Table Block LIGHTCURVE
   --------------------------------------------------------------------------------
   
   ROW    TIME                 COUNT_RATE
   
        1  84273308.6407000124        77.1874998504
        2  84273742.9400600195        70.3124998638
        3  84273746.1811000109        68.7499998668
        4  84274229.0960600078        70.6249998632
        5  84274232.3370999992        78.7499998474
        6  84274235.5781400204        97.4999998111
        7  84274238.8191800117        91.8749998220
        8  84274242.0602200031        91.8749998220
        9  84274245.3012599945       109.6874997875
       10  84274248.5423000157       104.6874997972
       11  84274251.7833400071        91.8749998220
       12  84274255.0243799984        95.3124998153
       13  84274258.2654200196       100.3124998056
       14  84274261.5064600110       120.3124997669
       15  84274264.7475000024       107.4999997917
       16  84274267.9885400236        66.5624998710
       17  84274271.2295800149        69.3749998656
       18  84274274.4706200063        67.4999998692
       19  84274277.7116599977        76.8749998510
   ...
   

   The spike occurs in the lightcurve between 84274232 and 84274267. To remove this feature from the dataset, we need to define a filter that excludes rates greater than ~80.0 counts/sec.

5. Create a Good Time Interval table

   As mentioned before, the Good Time Interval (GTI) table contains the times during which the data are good, based on the filter definition. For this example, it will be the times when the count rate is below 80.0 counts/sec, as determined in the previous step:

   unix% punlearn dmgti
   unix% pset dmgti infile=background_lc.fits
   unix% pset dmgti outfile=bkg_gti.fits
   unix% pset dmgti userlimit="count_rate<=80.0"
   unix% dmgti
   Input MTL file (background_lc.fits): 
   Output GTI file (bkg_gti.fits): 
   User defined limit string (count_rate<=80.0):
   

   A more complex filter could be applied instead, such as "count_rate>= 5.5&&count_rate<=80.0"; see ahelp dmfiltering for other examples of filtering syntax.

   You can check the resulting parameter file with plist dmgti.

6. Apply the GTI table

   To apply the GTI table to the dataset, dmcopy is run:

   unix% dmcopy "acis_1843_evt2_0.3-10.fits[@bkg_gti.fits]" acis_1843_evt2_0.3-10_bkgflt.fits 
   

7. Verify filtering

   Examination of the EXPO keywords shows the amount of filtering that was done in the creation of the new event file:

   unix% dmlist acis_1843_evt2_0.3-10.fits header | grep EXPO
   0117 EXPOSURE                  7854.4664748687 [s]       Real8        Exposure time
   0118 EXPOSUR7                  7854.4664748687 [s]       Real8        Exposure time
   0119 EXPOSUR6                  7854.4664748687 [s]       Real8        Exposure time
   0120 EXPOSUR3                  7851.2665536184 [s]       Real8        Exposure time
   0121 EXPOSUR2                  7854.4664748687 [s]       Real8        Exposure time
   0122 EXPOSUR1                  7854.4664748687 [s]       Real8        Exposure time
   0123 EXPOSUR0                  7854.4664748687 [s]       Real8        Exposure time
   
   unix% dmlist acis_1843_evt2_0.3-10_bkgflt.fits header | grep EXPO
   0117 EXPOSURE                  7812.8664748911 [s]       Real8        Exposure time
   0118 EXPOSUR7                  7812.8664748911 [s]       Real8        Exposure time
   0119 EXPOSUR6                  7812.8664748911 [s]       Real8        Exposure time
   0120 EXPOSUR3                  7809.6665536407 [s]       Real8        Exposure time
   0121 EXPOSUR2                  7812.8664748911 [s]       Real8        Exposure time
   0122 EXPOSUR1                  7812.8664748911 [s]       Real8        Exposure time
   0123 EXPOSUR0                  7812.8664748911 [s]       Real8        Exposure time