Running wavdetect
CIAO 4.17 Science Threads
Overview
Synopsis:
wavdetect operates on the input in two stages. First it detects possible source pixels in a dataset by repeatedly correlating it with "Mexican Hat" wavelet functions with different scale sizes. Pixels with sufficiently large positive correlation values are removed from the image as assumed sources, and subsequent correlations are performed at the same scale. The second stage generates a source list from information from the first stage at each wavelet scale. This tool separates closely spaced point sources and finds extended sources so long as wavelet scales are chosen appropriately.
Purpose:
To illustrate several ways to use the detect tool: wavdetect.
Related Links:
- Analysis Guide: Extended Sources
-
Detect manual: other examples of using this tool are available in the "Running wavdetect" chapter.
Last Update: 17 Feb 2022 - Review for CIAO 4.14. Updated to use new obsid.
Contents
- Getting Started
- Create a PSF Map
- Running wavdetect
- Using wavdetect on non-Chandra data
- Caveats
- Parameter files:
- History
- Images
Getting Started
Download the sample data: 2254 (ACIS-S, 3C 295)
unix% download_chandra_obsid 2254
unix% chandra_repro 2254 2254/repro
wavdetect uses a large amount of memory. For images larger than 2048x2048 users will need to be running on a machine with at least 2Gb of memory installed. Larger images may be used with increased memory.
To run this thread, we want a source image and a congruent exposure map (for the "Running wavdetect with an exposure map" section). We chose to create both by running the fluximage script. The image and exposure map may also be created by following the Multiple Chip ACIS Exposure Map and Exposure-corrected Image thread.
In the following examples, a full resolution image of the ACIS-I array is used:
unix% dmcopy "acisf002254_000N003_fov1.fits[ccd_id=0:3]" acisf002254.fov unix% dmcopy "acisf002254_repro_evt2.fits[ccd_id=0:3,sky=region(acisf002254.fov)]" 2254_evt2_filtered.fits unix% fluximage 2254_evt2_filtered.fits binsize=1 bands=broad outroot=acisf002254 psfecf=0.393 unix% dmlist acisf002254_broad_thresh.img blocks -------------------------------------------------------------------------------- Dataset: acisf02254_broad_thresh.img -------------------------------------------------------------------------------- Block Name Type Dimensions -------------------------------------------------------------------------------- Block 1: EVENTS_IMAGE Image Int4(2733x2735)
To view the file, load it into ds9:
unix% ds9 acisf002254_broad_thresh.img &
which produces Figure 1.
[Version: full-size]
Figure 1: Image of the cluster core
Create a PSF Map
The PSF map is an observation specific file that provides information on the size of the PSF at each pixel in the image. It is an optional file, but wavdetect cannot compute point-spread-function sizes without it. The tool will still run but the source characteristics will not be reliable.
The examples in this thread use eband=1.4967 and ecf=0.393 values.
unix% mkpsfmap acisf002254_broad_thresh.img outfile=step_by_step_acisf002254_psfmap.fits energy=1.4967 ecf=0.393
The eband value should be the same effective energy used to create the exposure map and the ecf value is the percentage of the source counts at the given effective energy that is desired to be encompassed by the PSF. This value determines the size of the PSF map that is produced; an ecf=0.393 corresponds to the 1σ integrated volume of a 2D Gaussian. The contents of the parameter file may be checked using plist mkpsfmap.
If you are using a narrow energy band then you should adjust the energy parameter to match your energy band. Also, if you find you are detecting many small spurious sources, you may want to increase the eband (encircled energy) parameter.
mkpsfmap spectrum's parameter allows the PSF map to be spectrally weighted. Users can use the same thread to create spectral weights as used by mkinsmap.
Running wavdetect
A Simple Example
To run wavdetect with the basic parameters set, it is necessary to specify the input and output filenames only. Note that it is not required to create an ASCII version of the region file; if the regfile parameter is left blank, it will not be used.
unix% punlearn wavdetect unix% pset wavdetect infile=acisf002254_broad_thresh.img unix% pset wavdetect psffile=acisf002254_broad_thresh.psfmap unix% pset wavdetect outfile=acisf002254_src.fits unix% pset wavdetect scellfile=acisf002254_scell.fits unix% pset wavdetect imagefile=acisf002254_imgfile.fits unix% pset wavdetect defnbkgfile=acisf002254_nbgd.fits unix% pset wavdetect regfile=acisf002254_src.reg unix% wavdetect Input file name (acisf002254_broad_thresh.img): Output source list file name (acisf002254_src.fits): Output source cell image file name (acisf002254_scell.fits): Output reconstructed image file name (acisf002254_imgfile.fits): Output normalized background file name (acisf002254_nbgd.fits): wavelet scales (pixels) (2.0 4.0): Image of the size of the PSF (acisf002254_broad_thresh.psfmap):
The contents of the parameter file may be checked using plist wavdetect.
There are many outputs from the tool:
-
outfile: the FITS source list; contains same sources as regfile (see below)
-
scellfile: the source cell map shows the image pixels which were used to estimate source properties.
-
imagefile: the reconstructed source image is the combined correlation maxima regions, which correspond to probable sources. It is essentially what the data would look like if there were no background and noise fluctuations.
-
defnbkgfile: the normalized background derived from the data after the identified sources have been taken out.
-
regfile: the ASCII region file; contains same sources as outfile (see above).
Full descriptions of these files are given in the "Wavdetect Input Parameters & Data Products Reference" chapter of the Detect manual.
To display the image with the detections overlaid:
unix% ds9 acisf002254_broad_thresh.img &
Load the source list from either file (Region → Load Regions... → acisf002254_src.fits OR acisf002254_src.reg). The detections are shown in Figure 2.
[Version: full-size]
Figure 2: Default detections
Key parameters that you may wish to vary are scales and sigthresh. The scales parameter is a list of radii (in image pixels) of Mexican Hat wavelet functions; a wavelet transform is performed for each scale in the list. The wavelet scales used should be chosen to match the size of the sources, given the PSF size, that are of interest to the user. Generally scales are input such that succeeding value is a factor of 2 or SQRT(2) larger than the previous one. The primary concern is to match the first scale size to the PSF. The choice of how many scaled wavelets to use depends somewhat on the available computing resources.
The sigthresh parameter is the significance threshold for source detection. Specifying a sigthresh that is proportional to the inverse of the number of pixels in the image allows ~1 false source per field (i.e. the default value of 10-6 corresponds to one spurious source in a 1000x1000 pixel map).
The result of running wavdetect on the same image with scales="1.0 2.0 4.0 8.0 16.0" and sigthresh=4e-06 are shown in Figure 3 (zooming in reveals the core and both hotspots (Figure 4) of this radio galaxy at the center of the cluster). The contents of the parameter file for this run may be checked using plist wavdetect.
[Version: full-size]
Figure 3: Obsid 2254 detections: different scales and sigthresh
[Version: full-size]
Figure 4: Galaxy core and hotspots
With an Exposure Map
In regions with substantial exposure variations, wavdetect may inaccurately estimate detection significance. Using an exposure map can suppress false positives (erroneous detections) and can help refine source property estimates. On account of this, the number of sources detected with an exposure map may be different than the number detected without one.
The exposure map was created in the Get Started section by running the fluximage script with the broad energy band definition. The exposure map may also be created by following the Multiple Chip ACIS Exposure Map and Exposure-corrected Image thread.
unix% punlearn wavdetect unix% pset wavdetect infile=acisf002254_broad_thresh.img unix% pset wavdetect psffile=acisf002254_broad_thresh.psfmap unix% pset wavdetect expfile=acisf002254_broad_thresh.expmap unix% pset wavdetect outfile=acisf002254_expmap_src.fits unix% pset wavdetect scellfile=acisf002254_expmap_scell.fits unix% pset wavdetect imagefile=acisf002254_expmap_imgfile.fits unix% pset wavdetect defnbkgfile=acisf002254_expmap_nbgd.fits unix% pset wavdetect regfile=acisf002254_expmap_src.reg unix% wavdetect Input file name (acisf002254_broad_thresh.img): Output source list file name (acisf002254_expmap_src.fits): Output source cell image file name (acisf002254_expmap_scell.fits): Output reconstructed image file name (acisf002254_expmap_imgfile.fits): Output normalized background file name (acisf002254_expmap_nbgd.fits): wavelet scales (pixels) (2.0 4.0): Image of the size of the PSF (acisf002254_broad_thresh.psfmap):
The contents of the parameter file may be checked using plist wavdetect.
To display the image with an overlay of the source detections:
unix% ds9 acisf002254_broad_thresh.img &
Load the source list from either file (Region → Load Regions... → acisf002254_expmap_src.fits OR acisf002254_expmap_src.reg). The results are shown in Figure 5.
[Version: full-size]
Figure 5: Obsid 2254 detections: with an exposure map
There isn't any significant exposure variation across the field in this example, so the source properties do not change in the exposure map run.
To examine the source properties, use either dmlist with the column names of interest:
unix% dmlist "acisf002254_expmap_src.fits[cols RA,DEC,PSF_SIZE]" data -------------------------------------------------------------------------------- Data for Table Block SRCLIST -------------------------------------------------------------------------------- ROW RA DEC PSF_SIZE 1 212.7564572165 52.0672758583 8.5082626343 2 212.9904537636 52.1067162749 8.0642518997 3 212.8099536903 52.1081606779 4.2496347427 4 212.7450178603 52.1200116958 4.3459811211 5 212.6347701839 52.1293555975 8.0236558914 6 212.8562454429 52.1523497928 1.6900422573 7 212.9888631721 52.1537083760 5.3986010551 8 212.7953056866 52.1632417214 1.2381882668 9 212.9007241916 52.1664415390 1.8939571381 10 212.8973928110 52.1693932690 1.7141796350 ...
More information on working with source lists is available in the Using the Output of Detect Tools thread.
Using wavdetect on non-Chandra data
This example shows how to run wavdetect on a non-Chandra dataset. In this case, an XMM pn image, P0551690201PNS001IMAGE_8000.FTZ, is used.
Data from other missions is supported if the user can construct a psfmap file. For example, a psfmap file for an arbitrary instrument with a PSF that does not vary significantly over the field-of-view, can be generated with the dmimgcalc tool:
unix% dmimgcalc P0551690201PNS001IMAGE_8000.FTZ none xmm_psf.fits op="imgout=((img1*0)+9.0)" unix% dmhedit xmm_psf.fits none add BUNIT "arcsec"
which generates an XMM psfmap with a constant size of 9". This is based on the estimate that at 1.5 keV, XMM 0.393 ecf ranges from 7" on-axis to ~11" at edge of the field-of-view. If the size is wrong, then the wrong scales may be used for the determination of source parameters, and thus the source properties may be wrongly estimated. However, the detection process will be unaffected.
Just as with Chandra data, an exposure map should be used to provide wavdetect with the information it need to determine the edge of the detector. The pipeline produced exposure map will be used in this thread.
Now, run wavdetect:
unix% punlearn wavdetect unix% pset wavdetect infile=P0551690201PNS001IMAGE_8000.FTZ unix% pset wavdetect expfile=P0551690201PNS001EXPMAP8000.FTZ unix% pset wavdetect psffile=xmm_psf.fits unix% pset wavdetect scales="2 4 8 16 32" unix% pset wavdetect outfile=xmm_src.fits unix% pset wavdetect scellfile=xmm_scell.fits unix% pset wavdetect imagefile=xmm_imgfile.fits unix% pset wavdetect defnbkgfile=xmm_nbgd.fits unix% pset wavdetect regfile=xmm_src.reg unix% wavdetect Input file name (P0551690201PNS001IMAGE_8000.FTZ): Output source list file name (xmm_src.fits): Output source cell image file name (xmm_scell.fits): Output reconstructed image file name (xmm_imgfile.fits): Output normalized background file name (xmm_nbgd.fits): wavelet scales (pixels) (2 4 8 16 32): Image of the size of the PSF (xmm_psf.fits):
The contents of the parameter file may be checked using plist wavdetect.
[Version: full-size]
Figure 6: XMM detections
Caveats
Source detection and cosmic-ray afterglows
CIAO users who are trying to detect sources of about 10 counts or less should be aware of potential confusion with cosmic-ray afterglows that have not been removed from the data.
The Cosmic-Ray Afterglows why topic explains the current processing options and the limitations of the available tools used to detect afterglows.
A detection tool, not a flux tool
This tool is designed to be used as a source detection algorithm, and only secondarily as a source flux measurement tool. Flux estimation with wavdetect is generally quite reliable, but there is a small fraction of cases where the algorithm will fail to return useful source parameter estimates. This is inevitable, as there is no perfect source detection algorithm.
In section 3.2.2 of "A Wavelet-Based Algorithm for the Spatial Analysis of Poisson Data" (P. E. Freeman, et al. 2002, ApJS, 138, 185) it says:
"We note two situations where care must be exercised in interpreting results. First, the source cell for an [apparently] extended source may be too small if the smoothing scale is ~ [the PSF scale]."
Parameters for /home/username/cxcds_param/mkpsfmap.par infile = acisf002254_broad_thresh.img Input image file name outfile = acisf002254_broad_thresh.psfmap Output image file name energy = 1.4967 Energy of PSF to lookup [keV] ecf = 0.393 ECF of PSF to lookup (psffile = CALDB) PSF Calibration file (units = arcsec) Units of output image (geompar = geom.par) Pixlib geometry file (clobber = no) Clobber files? (mode = ql)
Parameters for /home/username/cxcds_param/wavdetect.par # # parameter file for wavdetect # # # input # infile = acisf02254_0.5-7.0_bin1.img Input file name # # output # outfile = acisf02254_src.fits Output source list file name scellfile = acisf02254_scell.fits Output source cell image file name imagefile = acisf02254_imgfile.fits Output reconstructed image file name defnbkgfile = acisf02254_nbgd.fits Output normalized background file name # # scales # scales = 2.0 4.0 wavelet scales (pixels) # # end of wtransform parameters # ######################################################################## ######################################################################## # # wrecon parameters # # # PSF size parameters # psffile = acisf02254_psfmap.fits Image of the size of the PSF (regfile = acisf02254_src.reg) ASCII regions output file # # output options # (clobber = no) Overwrite existing outputs? (ellsigma = 3.0) Size of output source ellipses (in sigmas) (interdir = ${ASCDS_WORK_PATH} -> /tmp) Directory for intermediate outputs # ######################################################################### # # wtransform parameters # # # optional input # (bkginput = ) Input background file name (bkgerrinput = no) Use bkginput[2] for background error # # output info # (outputinfix = ) Output filename infix # # output content options # (sigthresh = 1e-06) Threshold significance for output source pixel list (bkgsigthresh = 0.001) Threshold significance when estimating bkgd only (falsesrc = -1.0) Allowed number of false sources per image (sigcalfile = ${ASCDS_CALIB}/wtsimresult.fits -> /soft/ciao/data/wtsimresult.fits) Significance calibration file # # exposure info # (exptime = 0) Exposure time (if zero, estimate from map itself (expfile = ) Exposure map file name (blank=none) (expthresh = 0.1) Minimum relative exposure needed in pixel to analyze it # # background # (bkgtime = 0) Exposure time for input background file # # iteration info # (maxiter = 2) Maximum number of source-cleansing iterations (iterstop = 0.0001) Min frac of pix that must be cleansed to continue # # end of wrecon parameters # ######################################################################## # # run log verbosity and content # (log = no) Make a log file? (verbose = 0) Log verbosity # # mode # (mode = ql)
Parameters for /home/username/cxcds_param/wavdetect.par # # parameter file for wavdetect # # # input # infile = acisf02254_0.5-7.0_bin1.img Input file name # # output # outfile = acisf02254_src_2.fits Output source list file name scellfile = acisf02254_scell_2.fits Output source cell image file name imagefile = acisf02254_imgfile_2.fits Output reconstructed image file name defnbkgfile = acisf02254_nbgd_2.fits Output normalized background file name # # scales # scales = 1.0 2.0 4.0 8.0 16.0 wavelet scales (pixels) # # end of wtransform parameters # ######################################################################## ######################################################################## # # wrecon parameters # # # PSF size parameters # psffile = acisf02254_psfmap.fits Image of the size of the PSF (regfile = acisf02254_src_2.reg) ASCII regions output file # # output options # (clobber = no) Overwrite existing outputs? (ellsigma = 3) Size of output source ellipses (in sigmas) (interdir = ${ASCDS_WORK_PATH} -> /tmp) Directory for intermediate outputs # ######################################################################### # # wtransform parameters # # # optional input # (bkginput = ) Input background file name (bkgerrinput = no) Use bkginput[2] for background error # # output info # (outputinfix = ) Output filename infix # # output content options # (sigthresh = 4e-06) Threshold significance for output source pixel list (bkgsigthresh = 0.001) Threshold significance when estimating bkgd only (falsesrc = -1) Allowed number of false sources per image (sigcalfile = ${ASCDS_CALIB}/wtsimresult.fits -> /soft/ciao/data/wtsimresult.fits) Significance calibration file # # exposure info # (exptime = 0) Exposure time (if zero, estimate from map itself (expfile = ) Exposure map file name (blank=none) (expthresh = 0.1) Minimum relative exposure needed in pixel to analyze it # # background # (bkgtime = 0) Exposure time for input background file # # iteration info # (maxiter = 2) Maximum number of source-cleansing iterations (iterstop = 0.0001) Min frac of pix that must be cleansed to continue # # end of wrecon parameters # ######################################################################## # # run log verbosity and content # (log = no) Make a log file? (verbose = 0) Log verbosity # # mode # (mode = ql)
Parameters for /home/username/cxcds_param/wavdetect.par # # parameter file for wavdetect # # # input # infile = acisf02254_0.5-7.0_bin1.img Input file name # # output # outfile = acisf02254_expmap_src.fits Output source list file name scellfile = acisf02254_expmap_scell.fits Output source cell image file name imagefile = acisf02254_expmap_imgfile.fits Output reconstructed image file name defnbkgfile = acisf02254_expmap_nbgd.fits Output normalized background file name # # scales # scales = 2.0 4.0 wavelet scales (pixels) # # end of wtransform parameters # ######################################################################## ######################################################################## # # wrecon parameters # # # PSF size parameters # psffile = acisf02254_psfmap.fits Image of the size of the PSF (regfile = acisf02254_expmap_src.reg) ASCII regions output file # # output options # (clobber = no) Overwrite existing outputs? (ellsigma = 3.0) Size of output source ellipses (in sigmas) (interdir = ${ASCDS_WORK_PATH} -> /tmp) Directory for intermediate outputs # ######################################################################### # # wtransform parameters # # # optional input # (bkginput = ) Input background file name (bkgerrinput = no) Use bkginput[2] for background error # # output info # (outputinfix = ) Output filename infix # # output content options # (sigthresh = 1e-06) Threshold significance for output source pixel list (bkgsigthresh = 0.001) Threshold significance when estimating bkgd only (falsesrc = -1.0) Allowed number of false sources per image (sigcalfile = ${ASCDS_CALIB}/wtsimresult.fits -> /soft/ciao/data/wtsimresult.fits) Significance calibration file # # exposure info # (exptime = 0) Exposure time (if zero, estimate from map itself (expfile = acisf02254_2.3_bin1.expmap) Exposure map file name (blank=none) (expthresh = 0.1) Minimum relative exposure needed in pixel to analyze it # # background # (bkgtime = 0) Exposure time for input background file # # iteration info # (maxiter = 2) Maximum number of source-cleansing iterations (iterstop = 0.0001) Min frac of pix that must be cleansed to continue # # end of wrecon parameters # ######################################################################## # # run log verbosity and content # (log = no) Make a log file? (verbose = 0) Log verbosity # # mode # (mode = ql)
Parameters for /home/username/cxcds_param/wavdetect.par # # parameter file for wavdetect # # # input # infile = P0551690201PNS001IMAGE_8000.FTZ Input file name # # output # outfile = xmm_src.fits Output source list file name scellfile = xmm_scell.fits Output source cell image file name imagefile = xmm_imgfile.fits Output reconstructed image file name defnbkgfile = xmm_nbgd.fits Output normalized background file name # # scales # scales = 2 4 8 16 32 wavelet scales (pixels) # # end of wtransform parameters # ######################################################################## ######################################################################## # # wrecon parameters # # # PSF size parameters # psffile = xmm_psf.fits Image of the size of the PSF (regfile = xmm_src.reg) ASCII regions output file # # output options # (clobber = no) Overwrite existing outputs? (ellsigma = 3.0) Size of output source ellipses (in sigmas) (interdir = ${ASCDS_WORK_PATH} -> /tmp) Directory for intermediate outputs # ######################################################################### # # wtransform parameters # # # optional input # (bkginput = ) Input background file name (bkgerrinput = no) Use bkginput[2] for background error # # output info # (outputinfix = ) Output filename infix # # output content options # (sigthresh = 1e-06) Threshold significance for output source pixel list (bkgsigthresh = 0.001) Threshold significance when estimating bkgd only (falsesrc = -1.0) Allowed number of false sources per image (sigcalfile = ${ASCDS_CALIB}/wtsimresult.fits -> /soft/ciao/data/wtsimresult.fits) Significance calibration file # # exposure info # (exptime = 0) Exposure time (if zero, estimate from map itself (expfile = P0551690201PNS001EXPMAP8000.FTZ) Exposure map file name (blank=none) (expthresh = 0.1) Minimum relative exposure needed in pixel to analyze it # # background # (bkgtime = 0) Exposure time for input background file # # iteration info # (maxiter = 2) Maximum number of source-cleansing iterations (iterstop = 0.0001) Min frac of pix that must be cleansed to continue # # end of wrecon parameters # ######################################################################## # # run log verbosity and content # (log = no) Make a log file? (verbose = 0) Log verbosity # # mode # (mode = ql)
Parameters for /home/username/cxcds_param/wavdetect.par
History
03 Jan 2005 | reviewed for CIAO 3.2: no changes |
03 Jun 2005 | updated links for CIAO 3.2 version of Detect manual |
19 Dec 2005 | updated for CIAO 3.3: updated files in detect data tarfile |
01 Dec 2006 | reviewed for CIAO 3.4: no changes |
16 Jan 2008 | updated for CIAO 4.0: added information about new wavdetect algorithm; updated parameter file listings to include falsesrc and sigcalfile, removed kernel; ds9 now automatically looks for the "[SRCLIST]" extension in the region file, so it doesn't have to be specified |
16 Apr 2008 | added Source Detection and Cosmic-Ray Afterglows caveat |
12 Jan 2009 | updated for CIAO 4.1: wavdetect algorithm was improved in this release; images are inline |
27 Jan 2010 | reviewed for CIAO 4.2: no changes |
15 Dec 2010 | updated for CIAO 4.3: the wavdetect "interdir" parameter uses ${ASCDS_WORK_PATH} instead of the working directory (".") for writing intermediate files |
15 Dec 2011 | reviewed for CIAO 4.4: wavdetect takes an observation-specific PSF map file as input instead of using the Chandra PSF table data or the tool's internal parameterization of the ROSAT PSF; minor updates to screen output and images; use fluximage to create the image and exposure map; added an example of non-Chandra data |
25 Jul 2012 | updated getting started example for necessary files |
03 Dec 2012 | Review for CIAO 4.5 |
28 Jun 2013 | Add a note about mkpsfmap parameters |
13 Dec 2013 | Review for CIAO 4.6. Updated to make sure chandra_repro is run due to early mission data being used. |
23 Dec 2014 | Review for CIAO 4.7. Minor edits only. |
12 Jun 2015 | Added information for mkpsfmap eband and ecf values. |
22 Dec 2015 | Added exposure map for XMM dataset and corrected dmhedit command. |
07 Dec 2018 | Updated for changes to fluximage which now can automatically create the PSF map. |
17 Feb 2022 | Review for CIAO 4.14. Updated to use new obsid. |