lc_clean - remove periods of high (or low) count rates using the same technique as the ACIS background files
lc_clean(filename, [outfile=None], [mean=None], [clip=3.0], [sigma=None], [scale=1.2], [minfrac=0.1], [plot=True], [rateaxis="y"], [pattern="solid"], [gcol="green"], [pcol="red"], [verbose=1])
The lc_clean() routine removes flares (and periods of anomalously-low count rates) from lightcurves using the same technique as used on the ACIS backgroud files. It is based on the lc_clean program created by Maxim Markevitch. For more information see the Using the ACIS "Blank-Sky" Background Files thread. The deflare tool can be used to run this algorithm from the command line; see "ahelp deflare" for more information.
Loading the routine
The routine can be loaded into a Python interpreter - such as Sherpa or ipython - by saying:
from lightcurves import *
See "ahelp lightcurves" for more information on the lightcurves module.
The lc_sigma_clip() and lc_clean() routines share a number of common arguments and options. The filename argument - which is the only required argument for either call - is used to specify the name of the light curve to analyze. The required columns for this file are discussed in the "Format of Lightcurves" section below.
Common options between the three routines (i.e. those with the same meaning)
|outfile||None||If set, this gives the name of the GTI file to create.|
|verbose||1||Do we display screen output (1) or not (0)?|
|plot||True||Should the light curve be displayed as a plot?|
|pattern||solid||If a GTI file is created as well as a plot, then we indicate the excluded times using regions filled with this pattern. A value of "none" means that the excluded regions will not be shown, otherwise it should be one of: solid, nofill, updiagonal, downdiagonal, horizontal, vertical, crisscross, grid, or polkadot.|
|rateaxis||"y"||Should the light curve (the top plot) be drawn with the count rate along the y axis ("y") or x axis ("x")?|
|gcol||"green"||The color used to display "good" points when drawing the light curve.|
|pcol||"red"||The color used to color regions indicating excluded times (see the pattern argument).|
Although all routines contain an optional sigma argument, the meaning of this parameter is not exactly the same.
The lc_clean algorithm was designed to work with the ACIS blank-sky datasets. The routine calculates - or is given - a mean rate from which it deduces a minimum and maximum valid count rate. Since the calculations assume Poisson statistics all the time bins are assumed to have the same width (i.e. exposure value). For further information see the documentaion linked to in the thread.
The "mean", "clip", "sigma", "scale", and "minfrac" options control the behavior of the routine. If mean is set then the value is taken as the initial guess, otherwise the level is calculated from the data, with outliers clipped using the clip parameter (which defaults to 3.0). This is not an iterative clipping scheme - as used in lc_sigma_clip() - since the variance of the lightcurve is assumed to be
var = mean-rate / bin-width
Once the mean level has been calculated, the limits are given by (if sigma is changed from its default value of None)
mean - sigma * sqrt(var)
mean + sigma * sqrt(var)
otherwise they are
mean / scale
mean * scale
where scale defaults to 1.2. The minfrac option determines the minimum fraction of bins that must remain "good" for the routine to finish.
How do we use the output GTI file?
If the outfile argument is set, then a GTI file will be created listing those time periods which are considered to be "good" by the algorithm. This file can be used to filter an event file using the following syntax (we assume here that the event file to be filtered is evt2.fits and the output of the routine is called evt.gti):
unix% dmcopy "evt2.fits[@evt.gti]" evt2_filt.fits
A warning about low count-rate lightcurves
The lc_clean algorithm excludes bins which have a zero count rate, whether it is because the exposure time of the bin is zero or because no counts were detected in this bin. This is because the algorithm is not designed for use with low count-rate data. Therefore the routines should be used with extreme care when applied to data for which zero-rate bins are expected (for example a very-faint source).
A note on times
When run with verbose set to 1, durations are displayed alongside time filters. These values are instructive, but should not be taken as the actual exposure-length of each interval because they may not account for
- the dead-time correction of the observation (DTCOR header keyword);
- exposure variations within the bin due to existing time filters (i.e. the time filters already applied to the observation).
The best way to find out what the final exposure time will be is to create a GTI file, use it to filter the event file, and then look at the EXPOSURE value of the filtered file.
The default parameters are used to analyze the light curve in the file bg.lc. A plot will be created and information about the calculation, including the final filter range, is printed to the screen.
The plot can be printed using the Matplotlib savefig command: for example
>>> import matplotlib.pyplot as plt >>> plt.savefig("bg.lc.ps")
will create a postscript plot.
>>> lc_clean("bg.lc", outfile="clean.gti")
Since the outfile parameter is set, the routine creates a GTI file (clip.gti) that represents the "good" times calculated by the algorithm. After writing out the file the filtered-out times will be displayed on the plot using solid red regions.
The output file (clean.gti) can be used to filter an event file (say evt2.fits) like so:
unix% dmcopy "evt2.fits[@clean.gti]" evt2_filt.fits
>>> lc_clean("bg.lc", outfile="bg.gti", pattern="none")
Create a GTI file (bg.gti) but do not use regions to indicate the excluded times in the count-rate plot.
>>> lc_clean("bg.lc", outfile="bg.gti", pattern="grid", pcol="orange")
Create a GTI file (bg.gti) and use an orange grid fill to indicate the excluded times in the count-rate plot.
>>> lc_clean("bg.lc", outfile="bg.gti", verbose=0, plot=False)
Create a GTI file (bg.gti) but do not create any screen output or a plot.
Format of Lightcurves
The lc_clean routine is designed to work with light curves that were created using the CIAO dmextract tool, run with the opt parameter set to ltc1. They should however also work with any file which has the following columns:
- TIME_MIN and TIME_MAX (both columns are optional)
- COUNT_RATE or, if not present, RATE
If the file contains OBJECT and OBS_ID keywords then these will be used to label the plot.
Changes in the 4.11.4 (2019) release
Labels with underscores in them are now properly displayed by Matplotlib.
Changes in the 4.11.2 (April 2019) release
Switch to Matplotlib
The plots are now created with Matplotlib rather than ChIPS.
About Contributed Software
This script is not an official part of the CIAO release but is made available as "contributed" software via the CIAO scripts page. Please see this page for installation instructions.
See the bugs page for this script on the CIAO website for an up-to-date listing of known bugs.
Refer to the CIAO bug pages for an up-to-date listing of known issues.