Skip to the navigation links
Last modified: February 2010

URL: http://cxc-newtest.cfa.harvard.edu/ciao4.2/lc_sigma_clip.sl.html
AHELP for CIAO 4.2

lc_sigma_clip

Context: sl.contrib

Synopsis

lc_sigma_clip - remove flares from a light curve using an iterative sigma-clipping algorithm (used to be called analyze_ltcrv)

Syntax

lc_sigma_clip(filename; [outfile=NULL], [sigma=3.0], [minlength=3],
[plot=1], [rateaxis="y"], [pattern="solid"], [gcol="green"],
[pcol="red"], [verbose=1]);

Description

The lc_sigma_clip() routine - which used to be called analyze_ltcrv() prior to CIAO 4.1 - uses a simple iterative sigma-clipping routine to detect and remove flares from a lightcurve. For more information see the Filtering Lightcurves thread.

Loading the routine

The routine can be loaded into a S-Lang interpreter - such as chips, sherpa, or slsh - by saying:

require ("lightcurves");

See "ahelp lightcurves" for more information on the lightcurves module.

Common arguments

The lc_sigma_clip() and lc_clean() routines share a number of common arguments and options. The filename argument - which is the only required argument for either call - is used to specify the name of the light curve to analyze. The required columns for this file are discussed in the "FORMAT OF LIGHT CURVES" section below.

Common options between the two routines (i.e. those with the same meaning)

Option Default Meaning
outfile NULL If set, this gives the name of the GTI file to create.
verbose 1 Do we display screen output (1) or not (0)?
plot 1 Should the light curve be displayed as a plot?
pattern solid If a GTI file is created as well as a plot, then we indicate the excluded times using regions filled with this pattern. Valid values are the same as those for region and histograms - see the "Fill Patterns" section of "ahelp chipsopt". A value of "none" means no regions will be drawn.
rateaxis "y" Should the light curve (the top plot) be drawn with the count rate along the y axis ("y") or x axis ("x")?
gcol "green" The color used to display "good" points when drawing the light curve.
pcol "red" The color used to color regions indicating excluded times (see the pattern argument).

Although both routines contain an optional sigma argument, the meaning of this parameter is not exactly the same for the two routines.

Using lc_sigma_clip

The lc_sigma_clip routine performs an iterative sigma-clipping algorithm, removing those points that fall outside the range

mean - n * sigma

to

mean + n * sigma

at each iteration until all data points are within this range. The default value for n is 3.0 but this can be changed using the sigma parameter as described below. This algorithm is robust but not perfect; it can easily "overclean" a noisy lightcurve, or fail to reject any points, and so should not be used blindly.

The two options for lc_sigma_clip are "sigma" and "minlength". The sigma argument determines the value used to clip the count rate data at each iteration, and defaults to 3.0. The minlength option, which defaults to 3, is the minimum number of consecutive bins that must all lie within the final rate limits for that range to be considered good.

How do we use the output GTI file?

If the outfile argument is set, then a GTI file will be created listing those time periods which are considered to be "good" by the algorithm. This file can be used to filter an event file using the following syntax (we assume here that the event file to be filtered is evt2.fits and the output of the routine is called evt.gti):

unix% dmcopy "evt2.fits[@evt.gti]" evt2_filt.fits

A warning about low count-rate lightcurves

The lc_sigma_clip algorithm excludes bins which have a zero count rate, whether it is because the exposure time of the bin is zero or because no counts were detected in this bin. This is because the algorithm is not designed for use with low count-rate data. Therefore the routines should be used with extreme care when applied to data for which zero-rate bins are expected (for example a very-faint source).

A note on times

When run with verbose set to 1, durations are displayed alongside time filters. These values are instructive, but should not be taken as the actual exposure-length of each interval because they may not account for

  • the dead-time correction of the observation (DTCOR header keyword);
  • exposure variations within the bin due to existing time filters (i.e. the time filters already applied to the observation).

The best way to find out what the final exposure time will be is to create a GTI file, use it to filter the event file, and then look at the EXPOSURE value of the filtered file.

Example 1

chips> lc_sigma_clip("bg.lc");

The default parameters are used to analyze the light curve in the file bg.lc. A plot will be created and information about the calculation, including the final filter range, is printed to the screen.

The plot can be printed using the ChIPS print_window command, for example

chips> print_window("bg.lc", {"fittopage", 1, "orientation",
"landscape"});

will create a postscript plot (bg.lc.ps) which is drawn in landscape format and fills the page.

Example 2

chips> lc_sigma_clip("bg.lc"; outfile="clip.gti");

Since the outfile parameter is set, the routine creates a GTI file (clip.gti) that represents the "good" times calculated by the algorithm. After writing out the file the filtered-out times will be displayed on the plot using red cross-hatched regions.

The output file can be used to filter an event file (say evt2.fits) like so:

unix% dmcopy "evt2.fits[@clip.gti]" evt2_filt.fits

Example 3

chips> lc_sigma_clip("bg.lc"; outfile="bg.gti", pattern="none");

Create a GTI file (bg.gti) but do not use regions to indicate the excluded times in the count-rate plot.

Example 4

chips> lc_sigma_clip("bg.lc"; outfile="bg.gti", pattern="crosshatch",
pcol="orange");

Create a GTI file (bg.gti) and use an orange cross-hatch fill to indicate the excluded times in the count-rate plot.

Example 5

chips> lc_sigma_clip("bg.lc"; outfile="bg.gti", verbose=0, plot=0);

Create a GTI file (bg.gti), but do not create any screen output or a plot.

FORMAT OF LIGHT CURVES

The lc_sigma_clip routine is designed to work with light curves that were created using the CIAO dmextract tool, run with the opt parameter set to ltc1. They should however also work with any file which has the following columns:

  • TIME
  • TIME_MIN and TIME_MAX (both columns are optional)
  • COUNT_RATE or, if not present, RATE

If the file contains OBJECT and OBS_ID keywords then these will be used to label the plot.

CHANGES IN CIAO 4.2 - 1.1

The ObsID label was missing from the plot.

CHANGES IN CIAO 4.2 - 1.0

The routine was updated to work with CIAO 4.2.

CHANGES IN CIAO 4.1 - 1.1

The lc_sigma_clip routine has been enhanced to better handle "extreme" lightcurves:

  • large variations in the light curve which result in the algorithm failing to find any good times are now recognized;
  • lightcurves which result in a large number of "good" time intervals can now be processed (in earlier versions they would cause dmgti or mtl_build_gti to fail)

The screen output - when verbose is 1 - includes more information, including accounting for the DTCOR value (dead-time correction factor) if present in the lightcurve.

CHANGES IN CIAO 4.1 - 1.0

Combined the lc_clean and analyze_ltcrv scripts

The lightcurves script is a combination of the lc_clean.sl and analyze_ltcrv.sl scripts from earlier releases of CIAO.

Users who load analyze_ltcrv.sl or lc_clean.sl will find that a warning message is displayed, but that they should be able to use the commands as before. This is a temporary change, intended to avoid immediate disruption to scripts and workflow, but users are urged to update to using lightcurves.sl directly since the lc_clean() and lc_sigma_clip() routines it contains provide more options than the old versions.

Renamed analyze_ltcrv to lc_sigma_clip

The analyze_ltcrv routine has been renamed to lc_sigma_clip. This routine can now create a GTI file rather than just printing the good time intervals to the screen.

Improved (and common) options

The method for specifying optional arguments - such as the name of an output file - has been re-worked to use the qualifier support in S-Lang v2.1. Common options for the two routines now have the same name and behave the same way.

Plot suppport

Plots of the lightcurves, together with a histogram of the cleaned (and original) data are displayed by default (they can be turned off). If a GTI file is created then the time ranges excluded by this file are added to the plot (this can also be turned off).

NOTES

This script is not an official part of the CIAO release but is made available as "contributed" software via the CIAO scripts page. Please see the installation instructions page for help on installing the package.

Bugs

See the bugs page for this script on the CIAO website for an up-to-date listing of known bugs.

See Also

py.contrib
lc_clean, lc_sigma_clip, lightcurves
sl.contrib
lc_clean, lightcurves
tools
acis_detect_afterglow, acis_find_hotpix, acis_streak_map, acisreadcorr, axbary, deflare, destreak, dmcopy, glvary

Last modified: February 2010