Search https://cxc.harvard.edu/ciao/
Contact the CXC HelpDesk
Last modified: 31 Jan 2022

URL: https://cxc.cfa.harvard.edu/ciao/threads/merge_all/
/ ciao / threads / merge_all /
Return to Threads Page: Top | All | Imag

Using merge_obs to combine observations and create exposure-corrected images

CIAO 4.17 Science Threads

Overview

Synopsis:

In order to create an exposure-corrected image from a set of observations, an exposure map has to be created for each observation (the exposure map is essentially an image of the effective area at each sky position, accounting for the effects of the telescope dither pattern which are especially important near the edges of the detector), and then combined. The merge_obs, reproject_obs, and flux_obs scripts wrap up all the CIAO tools needed to perform this task (they replace the merge_all tool, removed in CIAO 4.6).

This thread is aimed at users who have multiple observations covering the same area of sky. A related thread is Making an Exposure-corrected Mosaic thread.

For users with a single observation, the following threads should be consulted:

Purpose:

To combine observations and create exposure-corrected image, or images.

Related Links:

Last Update: 31 Jan 2022 - Reviewed for CIAO 4.14. Updated for Repro5, CALDB 4.9.6, and new FOV files.

Contents

Getting Started

Download the sample data: 11823 (ACIS-I, RCW 103); 12224 (ACIS-I, RCW 103)

unix% download_chandra_obsid 11823,12224

In this thread we assume that the observations have been reprocessed by chandra_repro and can be found in obsid/repro/; e.g. 

unix% chandra_repro 11823,12224 outdir=

Using merge_obs

Calibration products

The merge_obs script uses all the necessary information available to create properly-calibrated exposure maps for the observations. That is, it includes the per-observation aspect-solution, bad-pixel, mask, FOV and DTF files (where relevent).

In the following we supply the minimal information to merge_obs, namely the location of the event files (it searches for file names that contain the string evt in <dirname>/repro/ then <dirname>/primary/ then <dirname>/) and the output root. For ACIS data the default binning is 8 and choice of bands is broad (that is 0.5 to 7 keV with the exposure map evaluated at 2.3 keV). Note that the observations do not need to be listed in time order and that the script finds the locations of the ancillary files - e.g. aspect solution and bad-pixel files - it needs from the header keywords in the event files. 

unix% punlearn merge_obs
unix% merge_obs 11823,12224 rcw103/
Running merge_obs
Version: 05 November 2021

Found 11823/repro/acisf11823_repro_evt2.fits
Found 12224/repro/acisf12224_repro_evt2.fits
Verifying 2 observations.
Using CSC ACIS broad science energy band.
Calculating new tangent point.
New tangent point: RA=16h 17m 38.237s Dec=-51d 1' 24.66"

Observations to be reprojected:

  Obsid  Obs Date   Exp    DETNAM     SIM_Z    FP   Sepn   PA  
                   (ks)                (mm)    (K)   (')  (deg)
---------------------------------------------------------------
1 11823 2010-06-01  62.5 ACIS-0123   -233.587 153.4   0.1   +56
2 12224 2010-06-27  17.8 ACIS-0123   -233.587 153.4   0.1  -124

Running tasks in parallel with 4 processors.
Reprojecting 2 event files to a common tangent point.
Merging reprojected events files to rcw103/merged_evt.fits

Calculating the output grid

The merged images will have 368 by 367 pixels, a pixel size of 3.936 arcsec,
    and cover x=2648.5:5592.5:8, y=2560.5:5496.5:8.

Creating the fluxed images for 2 observations in parallel.
Creating 4 aspect histograms for obsid 11823
Creating 4 aspect histograms for obsid 12224
Creating 4 instrument maps for obsid 11823
Creating 4 instrument maps for obsid 12224
Creating 4 exposure maps for obsid 11823
Creating 4 exposure maps for obsid 12224
Combining 4 exposure maps for obsid 11823
Combining 4 exposure maps for obsid 12224
Thresholding data for obsid 11823
Exposure-correcting image for obsid 11823
Thresholding data for obsid 12224
Exposure-correcting image for obsid 12224

Combining 2 observations.

The following files were created:

The co-added clipped counts image is:
     rcw103/broad_thresh.img

The co-added clipped exposure map is:
     rcw103/broad_thresh.expmap

The combined FOV is:
     rcw103/merged.fov

The co-added exposure-corrected image is:
     rcw103/broad_flux.img

Warning: the merged event file rcw103/merged_evt.fits
   should not be used to create ARF/RMF/exposure maps because
      the RA_NOM keyword varies by 0.0032 (limit is 0.0003)
      the DEC_NOM keyword varies by 0.0013 (limit is 0.0003)
      the ROLL_NOM keyword varies by 44.4 (limit is 1.0)

The exposure-corrected image is shown in Figure 1.

Figure 1: Broad-band exposure-corrected image of RCW 103 created by merge_obs

[Thumbnail image: The grid of ACIS-I data has been combined to show the RCW 103 remnant.]

[Version: full-size]

[Print media version: The grid of ACIS-I data has been combined to show the RCW 103 remnant.]

Figure 1: Broad-band exposure-corrected image of RCW 103 created by merge_obs

unix% ds9 -scale log -zoom 2 -cmap b rcw103/broad_flux.img

The exposure-corrected 0.5 to 7 keV image is displayed. Since the two observations had different roll angles the image is no longer a square. It can be compared to the "raw" counts image shown in Figure 2.

Figure 2: Broad-band image and exposure map of RCW 103 created by merge_obs

[Thumbnail image: Instrumental effects, such as chip edges, bad columns, and the grid pattern used, can be seen.]

[Version: full-size]

[Print media version: Instrumental effects, such as chip edges, bad columns, and the grid pattern used, can be seen.]

Figure 2: Broad-band image and exposure map of RCW 103 created by merge_obs

unix% ds9 -scale log -smooth -cmap b rcw103/broad_thresh.img rcw103/broad_thresh.expmap -scale linear

The image on the left - shown with a logarithmic scale - is the counts image in the 0.5 to 7 keV band. The image on the right - shown with a linear scale - is the exposure map for the data, and was evaluated at 2.3 keV. Both images have been "thresholded" to remove pixels where the exposure map has dropped to a small value (in this case, 1.5 percent of the maximum value per observation).

If we look in the directory created by merge_obs we find a number of files:

File name Description
11823_reproj_evt.fits
12224_reproj_evt.fits 		The reprojected event files.
merged_evt.fits The merged event file (this contains only those columns that are common to all the reprojected event files).
11823_broad_thresh.img
12224_broad_thresh.img 		The counts image for each reprojected event file. If the expmapthresh parameter is set then these images have been thresholded to remove pixels where the corresponding exposure value is low.
11823_broad_thresh.expmap
12224_broad_thresh.expmap 		The exposure map for each reprojected event file. If the expmapthresh parameter is set then these images have been thresholded to remove pixels where the exposure is low.
11823_broad_flux.img
12224_broad_flux.img 		The exposure-corrected image for each reprojected event file.
11823.fov
12224.fov 		The Field of View (FOV) files matching the new reprojected event files.
broad_flux.img
broad_thresh.expmap
broad_thresh.img
broad.fov 		The exposure-corrected image, exposure map, FOV file, and counts image for the combined data.

There are a number of optional parameters to merge_obs that allow you control over the process, such as:

  • the bands parameter lets you chose the energy range (or ranges) used to create the images and the energy (or spectral weights file) used to create the exposure map(s);
  • the image size, location, or bin size can be varied by setting one of the binsize, maxsize, or xygrid parameters;
  • the location used to reproject the observations can be given (rather than being calculated by the script) by setting the refcoord parameter;
  • the units of the exposure map(s) - and hence the exposure-corrected image(s) - can be changed using the units parameter;
  • the ancillary files to use - e.g. aspect solution or bad-pixel files - can be set, rather than read from the event file headers, using the asolfiles, badpixfiles, maskfiles, and dtffiles parameters;
  • and the number of processors used by the script can be changed with the nproc parameter (by default, all processors are used), or even forced to use only one (by setting parallel to no).

Using reproject_obs and flux_obs

The merge_obs script allows you to create exposure-corrected images easily, but if you want to try different parameter settings (e.g. choice of bin size, location, or bands), then some of the work it does each time is wasted. For these occasions it can be useful to split the process into a reprojection stage - done by reproject_obs - and the calculation of the exposure-corrected images - done by flux_obs.

So, the previous section can also be done by saying: 

unix% punlearn reproject_obs
unix% reproject_obs 11823,12224 rcw103_manual/
Running reproject_obs
Running reproject_obs
Version: 15 November 2021

Found 11823/repro/acisf11823_repro_evt2.fits
Found 12224/repro/acisf12224_repro_evt2.fits
Verifying 2 observations.
Calculating new tangent point.
New tangent point: RA=16h 17m 38.237s Dec=-51d 1' 24.66"

Observations to be reprojected:

  Obsid  Obs Date   Exp    DETNAM     SIM_Z    FP   Sepn   PA  
                   (ks)                (mm)    (K)   (')  (deg)
---------------------------------------------------------------
1 11823 2010-06-01  62.5 ACIS-0123   -233.587 153.4   0.1   +56
2 12224 2010-06-27  17.8 ACIS-0123   -233.587 153.4   0.1  -124

Running tasks in parallel with 4 processors.
Reprojecting 2 event files to a common tangent point.
Merging reprojected events files to rcw103_manual/merged_evt.fits

The following files were created:

The reprojected FOV files:
     rcw103_manual/11823.fov
     rcw103_manual/12224.fov

The combined FOV file:
     rcw103_manual/merged.fov

The reprojected event files:
     rcw103_manual/11823_reproj_evt.fits
     rcw103_manual/12224_reproj_evt.fits

The merged event file:
     rcw103_manual/merged_evt.fits

Warning: the merged event file rcw103_manual/merged_evt.fits
   should not be used to create ARF/RMF/exposure maps because
      the RA_NOM keyword varies by 0.0032 (limit is 0.0003)
      the DEC_NOM keyword varies by 0.0013 (limit is 0.0003)
      the ROLL_NOM keyword varies by 44.4 (limit is 1.0)

to create the reprojected event files, then these can be used to create the exposure-corrected image: 

unix% punlearn flux_obs
unix% flux_obs "rcw103_manual/*_reproj_evt.fits" rcw103_manual/
Running flux_obs
Version: 05 November 2021

Verifying 2 observations.
Using CSC ACIS broad science energy band.
Calculating the output grid

The merged images will have 368 by 367 pixels, a pixel size of 3.936 arcsec,
    and cover x=2648.5:5592.5:8, y=2560.5:5496.5:8.

Creating the fluxed images for 2 observations in parallel.
Creating 4 aspect histograms for obsid 11823
Creating 4 aspect histograms for obsid 12224
Creating 4 instrument maps for obsid 11823
Creating 4 instrument maps for obsid 12224
Creating 4 exposure maps for obsid 11823
Creating 4 exposure maps for obsid 12224
Combining 4 exposure maps for obsid 11823
Thresholding data for obsid 11823
Exposure-correcting image for obsid 11823
Combining 4 exposure maps for obsid 12224
Thresholding data for obsid 12224
Exposure-correcting image for obsid 12224

Combining 2 observations.

The following files were created:

The co-added clipped counts image is:
     rcw103_manual/broad_thresh.img

The co-added clipped exposure map is:
     rcw103_manual/broad_thresh.expmap

The combined FOV is:
     rcw103_manual/merged.fov

The co-added exposure-corrected image is:
     rcw103_manual/broad_flux.img

and the output is the same as Figure 1 and Figure 2.

We can now use the reprojected event files to create a three-color image, using a smaller pixel scale, and zooming in on the core, by saying: 

unix% flux_obs "rcw103_manual/*_reproj_evt.fits" rcw103_manual/ \
   xygrid=3425.5:4896.5:2,3328.5:4576.5:2 \
   bands=csc
Running flux_obs
Version: 05 November 2021

Verifying 2 observations.
Using CSC ACIS soft science energy band.
Using CSC ACIS medium science energy band.
Using CSC ACIS hard science energy band.
Finding which observations overlap the output grid

The merged images will have 735 by 624 pixels, a pixel size of 0.984 arcsec,
    and cover x=3424.5:4896.5:2, y=3328.5:4576.5:2.

Creating the fluxed images for 2 observations in parallel.
Creating 4 aspect histograms for obsid 11823
Creating 4 aspect histograms for obsid 12224
Creating 12 instrument maps for obsid 11823
Creating 12 instrument maps for obsid 12224
Creating 12 exposure maps for obsid 11823
Creating 12 exposure maps for obsid 12224
Combining 4 exposure maps for 3 bands (obsid 11823)
Thresholding data for obsid 11823
Exposure-correcting 3 images for obsid 11823
Combining 4 exposure maps for 3 bands (obsid 12224)
Thresholding data for obsid 12224
Exposure-correcting 3 images for obsid 12224

Combining 2 observations.

The following files were created:

The co-added clipped counts images are:
     rcw103_manual/soft_thresh.img
     rcw103_manual/medium_thresh.img
     rcw103_manual/hard_thresh.img

The co-added clipped exposure maps are:
     rcw103_manual/soft_thresh.expmap
     rcw103_manual/medium_thresh.expmap
     rcw103_manual/hard_thresh.expmap

The combined FOVs are:
     rcw103_manual/merged.fov

The co-added exposure-corrected images are:
     rcw103_manual/soft_flux.img
     rcw103_manual/medium_flux.img
     rcw103_manual/hard_flux.img

where the result is Figure 3.

Figure 3: Three color, exposure-corrected image of RCW 103

[Thumbnail image: The edge of the remnant is a different color to the central emission.]

[Version: full-size]

[Print media version: The edge of the remnant is a different color to the central emission.]

Figure 3: Three color, exposure-corrected image of RCW 103

unix% cd rcw103_manual/
unix% ds9 -scale mode 99.5 -smooth -rgb -red soft_flux.img -green medium_flux.img -blue hard_flux.img -zoom to fit

The soft (red) channel is for the 0.5 to 1.2 keV band, the medium (green) channel is for the 1.2 to 2.0 keV band, and the hard (blue) channel is for the 2.0 to 7.0 keV band. More information can be found at the merge_obs parameter.

The image can be compared to that obtained using just one observation.

Converting from merge_all to merge_obs

Most of the parameters that merge_all required do not need to be set for merge_obs, since the script will calculate sensible values based on the input data. For instance, the command 

unix% merge_all @evt.all.lis @asol.lis chip=2,3,5,6,7 energy=2.3 \
       refcoord="123.45 -23.78" xygrid="2000.5:6278.5:8,3168.5:5824.5:8" \
       merged=out/merged.evt expmap=out/merged.emap expcorr=out/merged.flux

where the entries in evt.all.lis include the energy filter [energy=500:7000] (or have previously been filtered with this) can be replaced by either of 

unix% merge_obs @evt.lis out/

or 

unix% reproject_obs @evt.lis out/

depending on whether or not you want the exposure map and exposure-corrected image. Note that the files in evt.lis should not include any energy filter (the default band for ACIS data is 0.5 to 7.0 keV). If you wish to over-ride the script, e.g. to specify a tangent point or output grid, then you can.

The following table lists the merge_all parameters and their equivalents in merge_obs; it does not describe those parameters that have no equivalent in merge_all.

merge_all merge_obs Description
evtfile infiles As the energy filtering is now specified by the bands parameter, so no energy filter should be applied to the individual observations. You can now also include additional filters, so infiles="@evt.lis[ccd_id=0:3]" or infiles="*/repro/*evt*[ccd_id=7]" are both valid. The event files no longer need to be sorted by time.
asol asolfiles This parameter is no longer required since, if not given, the script uses the ASOLFILE keyword in each event file to find the aspect solution file (or files). This parameter only needs to be given if the aspect solution files have been renamed (e.g. because you have updated them to match WCS solutions between observations) or moved, and have not updated the ASOLFILE keyword in the header. The files no longer need to be sorted by time.
dtffile dtffiles This parameter is no longer required for HRC data since it is found from the DTFFILE keyword in each event file. It only needs to be specified if you have moved or renamed these files and have not updated the DTFFILE keyword in the header.
chip There is no equivalent parameter in merge_obs. If you wish to restrict the chips used then apply a filter to the input event files - e.g. infiles="@evt.lis[ccd_id=0:3]".
refcoord refcoord This parameter is no longer required. If not given, a reference position will be calculated given the tangent points of all the input event files.
xygrid xygrid This parameter is no longer required. If not given, the extents of the reprojected event files will be calculated and the union of them used to chose a grid that covers all the data. In this case the binning size is chosen by either the maxsize or binsize parameter.
energy bands The choice of energy range is now given by the bands parameter, which specifies both the energy range and the energy at which to evaluate the energy map or the weighting file to use. This means that energy filters should no longer be included on the event files sent in to the infiles parameter. It is also possible to specify multiple bands in one run of merge_obs.
merged The name of the merged event file is either <outroot>/merged_evt.fits or <outroot>_merged_evt.fits, depending on whether the outroot parameter ends in a '/' or not. The merge_obs script will create this directory if it does not exist.
expmap The name of the combined exposure map is either <outroot>/<band>_thresh.expmap or <outroot>_<band>_thresh.expmap, depending on whether the outroot parameter ends in a '/' or not. If you do not want to create the combined images and exposure maps then use the reproject_obs script instead, which just does the reprojection and merging of the event files.
expcorr The name of the combined fluxed image is either <outroot>/<band>_flux.img or <outroot>_<band>_flux.img, depending on whether the outroot parameter ends in a '/' or not. If you do not want to create the combined images and exposure maps then use the reproject_obs script instead, which just does the reprojection and merging of the event files.
intdir tmpdir The merge_obs script is more careful about naming temporary files and cleaning up after itself when an error occurs.

History

09 Jan 2012 reviewed for CIAO 4.4: no changes
15 Oct 2012 The thread has been converted to use the new merge_obs script, part of the 15 October 2012 scripts package release, rather than the deprecated merge_all script. The observations used in the thread have been changed.
03 Dec 2012 Review for CIAO 4.5; mkexpmap chatter removed
03 Dec 2013 Review for CIAO 4.6; the pbkfiles parameter has been removed from merge_obs; merge_all has been removed from the contributed package.
22 Dec 2014 Review for CIAO 4.7. Added a link to the fine astrometric correction thread.
31 Jan 2022 Reviewed for CIAO 4.14. Updated for Repro5, CALDB 4.9.6, and new FOV files.
Return to Threads Page: Top | All | Imag
/ ciao / threads / merge_all /