merge_obs

Synopsis

Reproject and combine multiple observations to create a merged event file and exposure-corrected images.

merge_obs  infiles outroot [bands] [xygrid] [maxsize] [binsize]
[asolfiles] [badpixfiles] [maskfiles] [dtffiles] [refcoord] [units]
[expmapthresh] [background] [bkgparams] [psfecf] [psfmerge] [random]
[parallel] [nproc] [tmpdir] [cleanup] [clobber] [verbose]

Description

The merge_obs script takes a stack of event files and reprojects them to the same tangent point, merges them, creates exposure maps for each observation, and divides the resulting images to produce a coadded, exposure-corrected image. This process can be done for one or more energy bands.

For recently-processed data the script can be run with only two parameters: for example

unix% merge_obs "*/repro/" out/

will store the merged data products in the directory out/, creating this directory if necessary. The output files include:

• out/merged_evt.fits, the merged events file, and
• out/broad_flux.img, the exposure-corrected broad-band image.

We took advantage of the stack support to process all the directories that match "*/repro/" - the quotes are needed to ensure that the shell does not do the matching, which would result in a parameter file error - and then looking for event files in these directories (see the description of the infile parameter for a full discussion of what file names are searched for). The examples below show other ways to use the stack support for specifying files.

How is merge_obs better than merge_all?

The merge_all script combines the reprojected event files, then creates a single multi-chip exposure map for this file. The merge_obs script creates exposure maps for each of the individual (reprojected) observations and then sums the resulting counts and exposure map images. The method merge_all uses does not correctly handle the case when observations were not taken with the same settings, such as differences in the SIM_X and SIM_Z values. This can lead to discrepencies between where the software calculates that chip edges and bad columns should be (when creating response functions like the exposure map or ARF) and where they actually are. The merge_all script does not use the observation-specific bad-pixel, mask, and DTF files, which means that certain instrumental effects (e.g. bad pixels, ACIS subarrays, cosmic-ray dead-time correction) are not included in the exposure map it creates. It also can not include variations in the response over time - e.g. when combining observations separated by multiple years, time dependent quantities, such as the amount of contamination on the ACIS optical-blocking filters, are not handled correctly.

The extra steps taken by merge_obs means it can take longer to run, but the script can do many of these in parallel if your machine has multiple processors, which can reduce this overhead.

If you only wish to reproject the event files and merge them - i.e. you do not want the images or exposure maps - then you can use the reproject_obs script.

The merge_obs, reproject_obs and flux_obs scripts

The merge_obs script combines the reproject_obs and flux_obs script, so if you need more control, or wish to create exposure-corrected images with multiple bin sizes or grids, then it is suggested that you use reproject_obs and flux_obs instead.

WCS alignment

The individual event files are reprojected to the same tangent point, but no attempt is made to align the observations. If you find that the WCS of the individual observations are not aligned - e.g. you can see "double" sources in the reprojected, merged data - then you need to update the WCS before running merge_obs. An example is the Correcting Absolute Astrometry thread.

Examples

Example 1

unix% download_chandra_obsid 5826,5827
unix% chandra_repro 5826,5827 outdir=""
unix% punlearn merge_obs
unix% merge_obs 5826/,5827/ m87
unix% ds9 m87_broad_flux.img -region m87_merged.fov

The two ACIS observations in the directories 5826/ and 5827/ are reprojected and combined to create m87_broad_flux.img, which is the broad-band (0.5 to 7 keV) image divided by an exposure map created at 2.3 keV. As these are ACIS observations the pixel size is set to 8 (i.e. 3.936 arc seconds).

The location of the necessary support files - e.g. aspect solution, bad-pixel file, mask file - are taken from the headers of the event files.

The processing steps are run in parallel when possible to reduce the run time of the script; this can be turned off by setting the parallel flag to "no", or the number of processors used changed with the nproc parameter.

What event files are used?

When given a directory name, the script looks for files that match one of the following (the first pattern to match is used):

• dirname/repro/*evt*
• dirname/primary/*evt*
• dirname/*evt*

Example 2

unix% punlearn merge_obs
unix% merge_obs
5826/repro/acisf05826_repro_evt2.fits,5827/repro/acisf05827_repro_evt2.f
its m87

Here we explicitly list the event files to use as a comma-separated list of names.

Example 3

unix% merge_obs "*/repro/*evt2.fits" merged/
unix% ds9 merged/broad_flux.img -region merged/merged.fov

Here we use the stack support which lets CIAO tools expand out patterns to list the input event files; in this case all files that end in evt2.fits and are in the repro/ sub-directory are used.

Since the outroot is set to a directory, the exposure-corrected image will be called "merged/broad_flux.img". The directory will be created if it does not already exist.

If the quotes are not used around the pattern then the shell will find all the matching files and send them as a space-separated list to the script, which will lead to an error from the CIAO parameter library.

Example 4

unix% find . -name \*repro_evt2.fits > evt2.lis
unix% punlearn merge_obs
unix% merge_obs @evt2.lis merged/ clobber+

Here we take advantage of the stack support (see ahelp stack) to combine all the files ending in "repro_evt2.fits" that are in the current directory or any of its children (this is the pattern used by chandra_repro to name its output).

Example 5

unix% merge_obs "@evt2.lis[ccd_id=0:3]" merged/acisi

A DataModel filter is used to restrict the combination to only those chips with a ccd_id of 0 to 3 (i.e. the ACIS-I array). This filter is applied to each line of the stack file (evt2.lis). Those event file which contain no data after the filter will be ignored by the script.

The outroot has been changed so that the files do not clash with those from the previous run. If you wish to overwrite them then you would set outroot to merged/ and add clobber=yes to the command line call.

Example 6

unix% merge_obs @evt2.lis merged/ bands=csc

The csc band is a short form for the soft, medium, and hard bands defined by the Chandra Source Catalog. This means that three exposure-corrected images will be created: merged/soft_flux.img, merged/medium_flux.img, and merged/hard_flux.img.

Example 7

unix% merge_obs @evt2.lis merged/ psfecf=0.9

Create per-observation PSF maps using an encircled-energy fraction of 0.9 (that is, the pixels in the PSF map give the radius, in arcsec, of the PSF size that contains 90 per cent of the PSF at that location). These maps are combined to create a single version for the field, using the default psfmerge setting, which uses the minimum ECF value at each pixel.

The combined PSF map is called merged/broad_thresh.psfmap.

Example 8

unix% merge_obs @evt2.lis merged/ psfecf=0.9 psfmerge=exptime

The combined PSF map is created by weighting the ECF values by the observation exposure times rather than taking the minimum value (which is the default psfmerge setting).

Example 9

unix% merge_obs @evt2.lis hires bin=2

The binsize parameter has been changed from its default value of 8 to 2. This may lead to large images if there is not significant overlap of all the observations.

Example 10

unix% merge_obs @evt2.lis hires maxsize=2048

In this example we force the longest dimension of the output images to have 2048 pixels.

Example 11

unix% merge_obs @hrc.lis merged/

In this example a set of HRC-I observations are being merged, which means that the wide band is used by default. The exposure-corrected image will be called merged/wide_flux.img.

The default setting for the background parameter is "default", which means that an estimate of the HRC-I particle background will be subtracted from each observation if the HRC CALDB package is installed and a match is found (as determined by the hrc_bkgrnd_lookup script). Observations for which no matching background file can be found will be skipped; to include these observations set background=none.

Example 12

unix% merge_obs @hrc.lis merged/ background=none

In this case the HRC-I data is processed without any background subtraction (which will be significantly quicker). Please see the caveat about "rings" seen in exposure-corrected HRC-I data for more information on why the particle background is removed by default.

Example 13

unix% merge_obs @hrc.lis merged/ bands=30:200:1.1 binsize=16

In this case the images will be filtered to only include events with PI=30:200, the exposure map is evaluated at 1.1 keV, and a bin size of 16 (rather than the default 32) is used. The final output file will be called merged/30-200_flux.img.

Example 14

unix% punlearn merge_obs
unix% pset merge_obs xygrid=2000.5:6000.5:8,2048.5:6128.5:8
unix% merge_obs @evt2.lis hires

Here we give the explicit grid for the output image (a 500 by 510 pixel image with a bin size of 8). All observations are reprojected, but only those that cover the grid will be used to create the combined image.

Example 15

unix% merge_obs @evt2.lis hires xygrid=merged.img

The grid for the images is taken to match that used to create the image merged.img (i.e. it is the output of get_sky_limits when run on merged.img).

Example 16

unix% merge_obs @evt2.lis combined/ bands=broad,wgt.dat
refcoord="123.45 -12.47"

The reprojection is done so that the center - i.e. (4096.5,4096.5) for ACIS data - is positioned at a Right Ascension of 123.45 degrees and Declination of -12.23 degrees. Note that there is no check that this location is close to any of the observations.

The position can also be given in sexagesimal notation using colon-separated values - e.g.

refcoord="8:13:48 -12:28:12"

Two energy ranges are used: the broad band and the range defined in the spectral weights file "wgt.dat" (the output for this is labelled band1). The weights file.dat can be created manually or with the make_instmap_weights script.

Example 17

unix% merge_obs @evt2.lis combined2/ refcoord=combined/merged_evt.fits

Here we reproject the data using the same reference location as the file combined/merged_evt.files (e.g. created by a previous run of merge_obs).

Example 18

unix% merge_obs @evt2.lis combined/ bands=plwgt.dat,bbwgt.dat nproc=-1

There will be two exposure-corrected images created in combined/, both using spectral weights files: band1_flux.img for plwgt.dat and band2_flux.img for bbwgt.dat.

The default parameter settings for parallel (yes) and nproc (INDEF) mean that all processors on your machine are used. In this example we change nproc to -1, which means that it uses all-but-one of your processors; so on a four-core machine the script will use three cores.

Example 19

unix% punlearn merge_obs
unix% pset merge_obs asolfiles=obs1.asol,obs2a.asol,obs2b.asol
unix% pset merge_obs badpixfiles=obs1.bpix,obs2.bpix
unix% pset merge_obs maskfiles=NONE
unix% merge_obs obs1.evt2,obs2.evt2 out/ bands=0.5:4:2

Here the two event files to be combined are called obs1.evt2 and obs2.evt2 and the ancillary files - aspect solution and badpixel files - are explicitly given. No mask files will be used since maskfiles is set to NONE. In this example the second observation has two asol files - here called obs2a.asol and obs2b.asol.

Rather than use one of the pre-canned energy bands we use the 0.5 to 4 keV band with the exposure map evaluated as 2 keV.

Stack files

We could have also used stack files - e.g.

unix% ls -1 *asol > asol.lis
unix% ls -1 *bpix > bpix.lis
unix% ls -1 *evt2 > evt2.lis
unix% merge_obs @evt2.lis out/ bands=0.5:4:2 asol=@asol.lis
badpix=@bpix.lis mask=NONE

Example 20

unix% merge_obs @evt2.lis merged/ units=time

Here the exposure maps have units of seconds, rather than cm^2 s count / photon, and so the exposure-corrected images have units of count / s. The energy value of the band parameter - in this case 2.3 keV - is ignored and only the low and high energy limits are used.

Parameters

Detailed Parameter Descriptions

Parameter=infiles (file required stacks=yes)

Input event files

Multiple event files are provided as a comma-separated list or as a stack; see "ahelp stack" for more information. The observations do not need to be given in time order.

Listing just a directory name

Rather than list the full path to an event file, you can just give a directory name; in such a case the script searches for any files that match the pattern

• dirname/repro/*evt*
• dirname/primary/*evt*
• dirname/*evt*

where the search is done in the order given above, and stops as soon as a match (or matches) is found.

Skipping files

Any observations that do not match the instrument of the first observation are skipped, as are CC mode observations or those with no data (e.g. because of a ccd_id filter). For HRC data, the first observation determines whether HRC-I or HRC-S are being combined, with observations from the other instrument being skipped. Any file with an OBS_ID keyword of "Merged" is skipped.

If multiple files with the same OBS_ID keyword are found, then all but the first are skipped, unless they are part of an ACIS interleaved-mode observation (i.e. they have CYCLE=P or CYCLE=S).

The input files must contain at least the following columns, otherwise they are skipped:

File headers

The header of the event file(s) is used to determine the names of the supporting data files - aspect solution, bad pixel, parameter block, mask, and dead-time factor files - if they are not explicitly set in the parameter file.

Parameter=outroot (string required)

Root of output files

If the parameter ends in "/" then the output files are placed in this directory (which will be created if necessary), otherwise it is prepended to the file names, with a "_" added. So, with bands=broad and outroot=fi/ the exposure-corrected image is called fi/broad_flux.img whereas with outroot=fi the file is fi_broad_flux.img.

An empty outroot value, or one set to ".", will cause the output files to be written to the current working directory with no prefix.

Parameter=bands (string default=default)

What energy or PI band, or bands, should be used?

The script will create an exposure-corrected image for each band. A band can be given using a name (which will use the appropriate definitions from the Chandra Source Catalog), by explicit limits, or by giving the name of a file containing the spectral weights (see ahelp make_instmap_weights).

For the first two options, the names for the output files are created using the minimum and maximum limits for each band. When using a weights file, the output files are labelled using "band1" to "bandn", where n is the number of weights file given to the parameter.

Note that the bands parameter is not used to filter the merged event file, it is only used to create the images and exposure maps.

The default setting

The value of 'default' is converted to 'broad' for ACIS data and 'wide' for HRC data.

Band names

The following names - based on the definitions from the Chandra Source Catalog - can be used; energies are given in keV and the effective energy is the monochromatic energy used to calculate the exposure map for the band:

CSC

The value "csc" can be used as a short form for the combination of "soft,medium,hard".

Explicit ranges

If you want to use different values to those of the CSC then you can use the format lo:hi:eff where

Spectral weights

If a filename is given then it is assumed to be an ASCII file containing spectral weights; two colums giving the energy in keV and the weight value for that energy. This file can be created using the make_instmap_weights script, the save_instmap_weights() routine from the sherpa_contrib.utils module, or manually; see the description of the spectrumfile parameter of mkinstmap for more information on the format of this file.

The energy limits to use for the images are calculated from this file. If the file contains two lines (before the data) with the format

# ENERG_LO = elo
# ENERG_HI = ehi

then the elo and ehi values are taken for the limits (they are assumed to be in keV). This format is created by the make_instmap_weights script and the save_instmap_weights() routine. If these lines do not exist, or the elo/ehi values are not valid numbers, then the limits will be calculated from the first column using the formulae (x1 is the first value, xn is the last, and xm the last-but-one value):

elo = (3 * x1 - x2) / 2

ehi = (3 * xn - xm) / 2

Multiple bands

Multiple bands can be given either by using the name "csc" or by using a comma-separated list. The different schemes can be combined, so the following are all valid (where wgt1.dat and wgt2.dat are weights files):

bands="csc,broad"
bands="0.5:7:2.5,ultrasoft"
bands="0.5:7:2.5,csc,wgt1.dat,wgt2.dat"

At present multiple bands can not be used with HRC data.

Parameter=xygrid (string default=)

xygrid for output or filename

When set, this parameter overrides both the maxsize and bin parameters, and defines the SKY coordinate range and binning used to create the output count images, exposure maps, and exposure-corrected images.

xygrid is set

When set, it can either be:

• a filename,
• a grid specification of the form "xlo:xhi:#nx or dx,ylo:yhi:#ny or dy".

If a filename is given then its sky range is used to determine the output; this is useful if you want to create an exposure-corrected image that matches an existing image. Otherwise the grid range is given explicitly; examples include "3824.5:5112.5:#161,4008.5:5296.5:#161" and "3824.5:5112.5:8,4008.5:5296.5:8".

xygrid is not set

When left at the default (empty) value, the script calculates the xygrid for the output image based on the observational data; the skyfov tool is used to determine the maximum area of the sky that is covered by the observations. In this mode the bin size is determined by either the maxsize or binsize parameters.

Parameter=maxsize (real default=INDEF min=1)

Maximum image width or height in pixels

This parameter is only used if the xygrid is empty (set to "").

If set (i.e. changed from INDEF) then the maximum dimension of the output image is set to this value. This can be useful if you quickly want to look at data and want to make sure the output image is not too large. Note that due to the way the ranges are calculated the actual image dimension is only guaranteed to be within a few pixels of the requested value.

The SKY ranges for the images are determined automatically, based on the ranges of the FOV files for each observation (see "ahelp skyfov").

Parameter=binsize (real default=INDEF min=0)

Image binning factor

This parameter is only used if the xygrid is empty (set to "") and maxsize is set to INDEF.

When set to INDEF a default value is used: 8 for ACIS data and 32 for HRC data. The binning factor has units of the SKY pixel (0.492" for ACIS and 0.1318" for HRC), so binsize=4 for an ACIS observation will create images with 1.968 arcsecond pixels. The value need not be an integer, and can be less than 1 if you want to look at small-scale structure near the aim point of the observation; in this case it is strongly suggested that a spatial filter is applied to ensure manageable file sizes.

The SKY ranges for the images are determined automatically, based on the ranges of the FOV files for each observation (see "ahelp skyfov").

Parameter=asolfiles (file stacks=yes)

Input aspect solutions

If the value is empty then the ASOLFILE keyword from the events files will be used to find the files to use. Unlike the other ancillary products, an aspect solution is required for the script to run.

The aspect solution files have names like pcadf*_asol1.fits and are included in the output directory of the chandra_repro script. To explicitly specify the asol files use the stack syntax (e.g. a comma, or space, separated string or an external file as described in "ahelp stack" for more information). So to use asol1.fits, asol2.fits, and asol3.fits you could say

asolfile="asol1.fits,asol2.fits,asol3.fits"

or

asolfile="asol1.fits asol2.fits asol3.fits"

or

asolfile=@asol.lis

where asol.lis contains the names of each file, one per line. The files do not have to be given in time order.

Since there may be multiple asol files for an observation there may be more entries in this parameter than there are in the infiles parameter.

Aspect histograms - the output of the asphist tool - can not be used here.

Parameter=badpixfiles (file stacks=yes)

Input bad pixel files

If the value is empty then the BPIXFILE keyword from the events files will be used to find the files to use.

A value of "CALDB" will use the CalDB bad-pixel file and "NONE" will use no bad-pixel file (the value is case insensitive). The bad-pixel file is not required to run the script, but care should be taken when analyzing the output if it is not used.

The bad-pixel file has a name like acisf*bpix1.fits and is included in the output directory of the chandra_repro script. If you have used or created a custom bad-pixel file for the observations then these should be used.

Use the stack syntax - see 'ahelp stack' - to specify multiple files either as a comma (or space) separated list of file names or read from a file using the '@' syntax.

The order of the bad-pixel files does not need to match that of the infiles parameter.

Parameter=maskfiles (file stacks=yes)

Input mask files

If the parameter is empty then the MASKFILE keyword in the events files will be used. A value of "NONE" will use no mask file in the analysis; in this case care should be taken when analyzing the results since the exposure map will not be correct if the data was taken using a sub array (i.e. not all the chips or ccds were read out).

The mask file has a name like acisf*msk1.fits and is included in the output directory of the chandra_repro script.

Use the stack syntax - see 'ahelp stack' - to specify multiple files either as a comma (or space) separated list of file names or read from a file using the '@' syntax.

The order of the mask files does not need to match that of the infiles parameter.

Parameter=dtffiles (file stacks=yes)

Input dtf files for HRC observations

This parameter is only used with HRC observations.

If the parameter is empty then the DTFFILE keyword in the events files will be used. A value of "NONE" will mean that only a mean DTF value will be used.

The file has a name like hrcf*dtf1.fits and is included in the output directory of the chandra_repro script.

Use the stack syntax - see 'ahelp stack' - to specify multiple files either as a comma (or space) separated list of file names or read from a file using the '@' syntax.

The order of the dtf files does not need to match that of the infiles parameter.

Parameter=refcoord (string default=)

Reference coordinates or evt2 file

This parameter defines the central coordinate of the reprojected data; that is the Right Ascension and Declination that maps to SKY=(4096.5,4096.5) for ACIS data, (16384.5,16384.5) for HRC-I data, or (32768.5,32768.5) for HRC-S data.

The default setting ("") means that this value is calculated from the tangent points of the observations. When set it can be either:

• the name of a file, so that the tangent position of the file is used;
• or a space-separated value taken to be the RA and Declination to use (ICRS) in decimal degrees or colon-separated sexagesimal formats.

Examples include:

refcoord=fluxed/broad_flux.img
refcoord="123.45 -12.34"
refcoord="8:13:48 -12:20:24"

Parameter=units (string default=default)

Units for the exposure map

This parameter determines the units of the instrument and exposure maps.

This parameter combines the normalize parameter of mkexpmap and the modifiers for the detsubsys and mirror parameters of mkinstmap (see 'ahelp ardlib').

What is units=time equivalent to?

When units=time is used, instrument maps are created with the additional settings:

• ";IDEAL" is appended to the detsubsys parameter to remove any correction for the detector quantum efficiency (QE), quantum efficiency uniformities (QEU), or the ACIS contamination model;
• the mirror parameter is set to "HRMA;area=1" rather than "HRMA" to ignore the effective area of the Chandra mirrors;
• and the dafile parameter is set to "NONE" to remove the ACIS "dead area" correction, which accounts for the (small) decrease in detector efficiency due to the background cosmic ray flux.

The result is that the instrument map pixels have values of 0 or 1.

Parameter=expmapthresh (string default=1.5%)

Remove low-exposure regions?

If set to "" then no clipping of the counts images or exposure maps is performed before creating the exposure-corrected image.

When set, the parameter determines how the per-observation counts image and exposure maps are clipped before being combined together to create the exposure-corrected image. This can be used to exclude those pixels at the edge of a chip which, due to the strong gradient in the exposure map, end up dominating the final image.

For a full list of supported options, see the description of the cut parameter of dmimgthresh. Here we discuss the two most common options:

• expmapthresh="2%", which excludes those pixels where the exposure map falls below 2% of its peak value,
• and expmapthresh="100", which excludes those pixels where the exposure map value is less than 100 (in this case the value of the units parameter is important).

Note that both the counts image and the exposure map for each observation are thresholded before being combined together rather than the thresholding being applied to the final, coadded, image and exposure map.

Parameter=background (string default=default)

Method for background removal (HRC-I)

This parameter is only used for HRC-I data. The reason for subtracting the particle background component is to help remove the "rings" seen in exposure-corrected HRC-I data, but does significantly increase the time taken to run the script.

If set to "none", or no HRC-I background CALDB files can not be found, then the data is processed with no background subtraction. Observations for which a HRC-I background file can not be found - when background is not set to "none" - will be removed from the analysis.

method=default

This is the same as method=time.

method=time

This setting scales the background file by the ratio of the observation to background exposure times. It is the method used in the thread discussed above.

method=particle

This option scales the background by the ratio of the counts in the range determined by the bkgparams parameter, rather than by exposure time.

Header keywords

The full path to the selected background event file is added to the header file of the background-subtracted image and fluxed image using the BEVTFILE keyword. The scaling method and factor are added to the relevant files using the BKGMETH and BKGSCALE keywords, respectively.

Parameter=bkgparams (string default=[pi=300:500])

Optional argument for background subtraction

This is only used when background subtraction occurs and the method being used is set to "particle". The parameter is taken to be a Data Model filter to apply to both the observation and the particle background file, and is expected to be of the form "[pi=lo:hi]". It is used to select a subset of the observation that represents the particle background - i.e. it is not strongly "contaminated" by cosmic emission - but is close enough (in PI) to the source emission to minimise differences in the spectral shape of the particle emission between the observation and that seen in the background observations.

Parameter=psfecf (real default=INDEF min=0 max=1)

If set, create PSF map with this ECF

If set, create a "PSF map" image, which contains the radius of the PSF which contains this ECF for each pixel, in arcseconds. The calculation is done by the mkpsfmap tool, and the energy is taken to be the same value used to create the exposure maps (the effective energy described in the the bands parameter or the weight file).

The output files are identified by the label "psfmap". The combined version is controlled by the psfmerge parameter.

Note

The PSF sizes are calculated using a number of approximations and so should not be used for detailed analysis of sources, where the PSF should be modelled as discussed in the CIAO PSF Central documentation.

Parameter=psfmerge (string default=min)

How are the PSF maps combined?

There is no one "correct" way to combine the per-observation PSF maps, so this script provides several options, which are controlled by the psfmerge parameter. The output file is identified by the label "psfmap".

The available options for psfmerge are: exptime, expmap, min, max, mean, median, and mid.

psfmerge=exptime

Weight each pixel by the exposure time of the observation, dividing by the total exposure time observed by the pixel.

psfmerge=expmap

Weight each pixel by the exposure map of the observation, dividing by the sum of the exposure map values for the pixel. This result should be similar to using the "exptime" method, but will include features seen in the exposure maps, such as bad pixels and columns.

psfmerge=min

Use the minimum PSF size for each pixel.

psfmerge=max

Use the maximum PSF size for each pixel.

psfmerge=mean

Use the average PSF size for each pixel.

psfmerge=median

Use the median of the PSF sizes for each pixel. If there are an even number (n) values to chose from, then the median is taken to be the n/2-th value, rather than averaging the (n/2) and (n/2)+1 values.

psfmerge=mid

Use the mid-point of the PSF sizes for each pixel. This finds the minimum (min) and maximum (max) PSF size for a pixel and uses (min + max) / 2.

Parameter=random (integer default=0 min=0)

Random seed for randomization.

The random parameter is sent to reproject_events when processing the HRC-I background files to match the observation. A value of 0 uses the system time to seed the random number generator.

Parameter=parallel (boolean default=yes)

Run processes in parallel?

When run on a multi-processor system, many of the tasks can be run in parallel to reduce the execution time of the script. Since the tasks are likely to be memory or I/O-bound, the reduction in run time will be less than the number of cores on a machine. When parallel=yes, the default behaviour is to use all the CPU processors, but this can be changed with the nproc parameter.

This option can be ignored when run on a single-processor system.

Parameter=nproc (integer default=INDEF)

Number of processors to use

This parameter is only used when parallel=yes. It determines the number of processors to use. If maxproc is the actual number of processors on your machine, then a value of INDEF - the default value - means that all maxproc processors will be used. A positive value means to use that number of processors (any value larger than maxproc will be set to maxproc). A negative value is added to maxproc (and any value less than one is set to one).

Parameter=tmpdir (string default=${ASCDS_WORK_PATH})

Directory for temporary files.

Directory for storing temporary files that require further processing before becoming useful. If the directory does not exist then it will be created for use by the script, and then deleted.

Parameter=cleanup (boolean default=yes)

Cleanup intermediary files on exit.

If set to "yes", the intermediate data products are deleted when the script ends, leaving only: the reprojected event files; merged event file; per-observation images, exposure maps, and exposure-corrected images; and the combined images, exposure maps, and exposure-corrected images.

Parameter=clobber (boolean default=no)

Overwrite existing files?

Parameter=verbose (integer default=1 min=0 max=5)

Output verbosity.

The default verbosity value of 1 prints status messages as the script runs. Higher verbosity settings print the commands that are being run. Setting verbose=0 turns off most of the screen output (some output is currently unavoidable).

Energy bands

The energy bands - when given by name - are taken from the Chandra Source Catalog definitions. The default bands setting (i.e. "default") means to use the "broad" band setting for ACIS data and the "wide" band setting for HRC data. Another common choice (for ACIS data) is "csc", which creates the "soft", "medium", and "hard" bands.

The examples and description of the band parameter describe how explicit energy ranges and effective energies can be specified if required. A spectral weights file can also be used if required.

Examples of specifying the bands are:

unix% merge_obs "*evt2.fits" out/
unix% merge_obs "*evt2.fits" out/ bands=csc
unix% merge_obs @evt2.lis out/ bands=0.5:2:1.7
unix% merge_obs @evt2.lis out/ bands=csc,broad
unix% merge_obs @evt2.lis out/ bands=wgt.dat

which use

• the default band (so broad for ACIS, wide for HRC),
• the CSC setting (soft, medium, and hard bands),
• an explicit range of 0.5 to 2 keV with an exposure map evaluated at 1.7 keV,
• four bands (soft, medium, hard, and broad),
• and a spectral-weights file (wgt.dat), created either manually or with the make_instmap_weights script.

ACIS low-energy bands and the ACIS contamination

Please review the Monochromatic Energy for Low Energy Bands document for the latest information on how the ACIS contamination layer can affect the analysis when using a mono-chromatic energy for calculating exposure maps, such as setting the bands parameter to one of the CSC bands (in particular the soft band).

Parallel processing

The execution time can be reduced by running some steps in parallel. This is controlled by the parallel flag, which defaults to yes. The nproc parameter controls the number of processors that will be used; the default is to use all of then. If run on a single-processor machine then both parameters are ignored.

Output files

The primary output files are named using the following schemes:

Combined data

Per-observation data

where blabel is either the band name (e.g. soft, broad, ...), "lo-hi" where lo and hi are the limits of the range (keV for ACIS, PI for HRC), or band1 to bandn (when using spectral weight files). The above is when outroot does not end in a "/"; if it does then remove "outroot_" and the files will be created in the directory named by outroot. Examples of the exposure-corrected image name for a combination of bands and outroot settings are given in the following table:

Names of the exposure-corrected coadded images

For HRC data, using bands=::eval results in a label of "all" being used.

Interleaved mode

For interleaved-mode data, "e1" or "e2" is appended to the ObsId value when creating file names.

Multi-OBI datasets

There are 30 ObsIds which contain multiple OBI segments. If the data has been split using splitobs then they can be combined using merge_obs, but the ObsId labels will contain the suffix _00n (where n is the OBI_NUM value from the header).

Output image area and size

The default behavior - when xygrid="" - is to calculate the output image size so that it covers all the data: a FOV file is created for the observation and the limits of this file are used to determine the X and Y range of the image. In this mode the binning is set by either the maxsize or binzise parameters. The maxsize parameter - if set - determines the maximum number of pixels used along the longest edge of the output, otherwise the binsize parameter is used. This parameter is in units of the native pixel size (i.e. 0.492" for ACIS and 0.1318" for HRC), and can be set to sub-pixel values (e.g. 0.5).

The tool can also be given an explicit grid for the output image size using the xygrid parameter (in this mode the binsize parameter is ignored). The grid can either be:

• a filename,
• or a grid specification, of the form "xlo:xhi:#nx or dx,ylo:yhi:#ny or dy".

When given a file, the sky range of the file is used for the output (e.g. the get_sky_limits tool is run on the file and its xygrid parameter value is used). Otherwise, the grid is given using the same format as the xygrid parameter of mkexpmap; examples include:

3824.5:5112.5:8,4008.5:5296.5:8

and

3824.5:5112.5:#161,4008.5:5296.5:#161

If the requested grid results in the last row or column being partially filled - in other words the width or height divided by the bin size is not a whole number - then the exposure map values in the column or row will be over estimated (since they are calculated assuming that the pixel is "full").

Handling filter or grids that do not overlap the observation

The tool will error out when there is no overlap between the observation and the xygrid (if given) or any spatial filter applied to the input file.

Required data files

For observations that has been reprocessed by chandra_repro, then all that is needed is to give the events file since the required ancillary files can be found using the header of the events file (the chandra_repro script ensures that all the files needed for this script are copied to the same directory as the events file). These ancillary files can be located in one of the following directories, relative to the events file:

• .
• repro/
• primary/
• secondary/
• ../repro/
• ../primary/
• ../secondary/

If the files are not in one of these directories, they have been renamed, or new versions have been created during your analysis (e.g. a new bad-pixel file or a reprojected aspect solution created by reproject_aspect) then they can be explicitly set using the following parameters:

Ancillary data products

The only required information is that provided by the aspect solution file(s); any other file can be ignored, (either explicitly, by setting the parameter to NONE, or implicitly because it can not be found), in which case the appropriate correction or information is ignored.

Processing steps

An overview of how merge_obs works:

• The event files are checked to ensure they are all for the same instrument (ACIS-I and ACIS-S, HRC-I, or HRC-S). Event files with no data - perhaps because of a ccd_id filter - or that are for CC-mode observations are ignored.
• The ancillary files for these observations are found, if not given explicitly given, and matched up with the observations.
• The reference position is either calculated from the data or taken from the refcoord parameter.
• A summary of the observations is displayed; e.g. time taken, observation length, and several other useful details such as the focal-plane temperature (for ACIS), SIM position, and separation from the reference position.
• The event files are reprojected to the new reference position, or copied if no reprojection is needed.
• A FOV file for each observation is created using the reprojected event files. These regions are combined to create the merged FOV file.
• The new events files are merged together to create a single events file, where only those columns common to all the input files are used. This file can be used for further analysis but in most cases it should not be used for spectral analysis; the individual files should be used instead.
• For each new event, file images, exposure maps, exposure-corrected images, and an optional PSF map are created for the requested list of bands. An optional thresholding step is applied to both the image and exposure map when the expmapthresh parameter is set. This step is similar to running fluximage on each observation.
• For each band, the counts images and exposure maps are summed, and used to create exposure-corrected images, one per band. Any observation PSF maps are combined together, using the psfmerge-defined scheme, to create the output PSF map per band.

How do merge_obs, reproject_obs, and flux_obs work together

The merge_obs script can be considered to be a run of reproject_obs followed by flux_obs on all the reprojected events files. That is

unix% merge_obs "*/repro/" out/

is equivalent to

unix% reproject_obs "*/repro/" out/
unix% flux_obs out/ out/

although the latter also creates links (or copies over) the ancillary files to out/. The advantage to using reproject_obs and flux_obs rather than merge_obs is that you have more control: e.g. if you do not want the exposure-corrected images you can just use reproject_obs, or you can do the reprojection once and then run flux_obs multiple times with different bin sizes or grids.

Converting merge_all to merge_obs

Most of the parameters used in merge_all do not need to be specified when running merge_obs, since it calculates sensible values for them given the input data. For instance the command

unix% merge_all @evt.all.lis @asol.lis chip=2,3,5,6,7 energy=2.3 \
       refcoord="123.45 -23.78" xygrid="2000.5:6728.5:8,3168.5:5824.5:8" \
       merged=out/merged.evt expmap=out/merged.emap expcorr=out/merged.fluxed

- where the entries in evt.all.lis include the energy filter "[energy=500:7000]" - can be replaced by

unix% merge_obs @evt.lis out/

or

unix% reproject_obs @evt.lis out/

depending on whether or not you want the exposure map and exposure-corrected image. Note that the files in evt.lis should not include any energy filter (the default band for ACIS data is 0.5 to 7.0 keV). If you wish to over-ride the script - e.g. to specify a tangent point or output grid - then you can.

The following table lists the merge_all parameters and their equivalents in merge_obs. It does not describe the new parameters.

Comparison of parameters

Changes in the scripts 4.14.0 (December 2021) release

When using the maxsize parameter, the calculated pixel size is now adjusted so that it does not have excessive precision: for instance a pixel size of 4.2 is now used rather than 4.19140625. This should allow the use of both the maxsize and the psfecf parameters (prior to this the combination could cause the script to fail).

The script now generates the FOV file for each reprojected event file and a combined region file (merged.fov) that includes all the individual observation FOV files.

There have been improvements to how the script handles a large number of observations.

Changes in the scripts 4.13.1 (March 2021) release

Ignore the frame-store shadow region

The frame-store shadow is now included when calculating the instrument map for ACIS observations. This means that a small number of rows of the instrument and exposure maps are now excluded, to match the data processing. Please see the ACIS frame-store caveat for more information.

Changes in the scripts 4.13.0 (December 2020) release

The warning message about not using the merged event file for calculating instrument responses now restricts the message so that the value does not include excess precision, but is limited to the limit value. The tangent point in the screen output has also seen its accuracy reduced (this only affects the screen output).

Changes in the scripts 4.12.2 (April 2020) release

The script has been updated to support aspect solutions produced in the DS 10.8.3 release (or later). These files have a CONTENT keyword of ASPSOLOBI (earlier releases use ASPSOL), and can be used with skyfov when method=convexhull.

Changes in the scripts 4.11.1 (December 2018) release

The script will now create PSF maps for the observations if the psfecf parameter is set. The combined PSF map can be created using a variety of merging rules, such as weighting by the exposure time or using the minimum value.

Changes in the scripts 4.8.2 (January 2016) release

The random parameter has been added. It is used to control the random parameter of reproject_events when background subtracting HRC-I data.

Changes in the scripts 4.8.1 (December 2015) release

The code has been updated to avoid warning messages from NumPy version 1.9. There is no difference to how the script behaves.

Changes in the scripts 4.7.3 (June 2015) release

Order of DTF files

It is no longer necessary, when giving the DTF files explicitly (rather than the suggested approach of using the DTFFILE keyword in the event file), to give the files in the same order as the event files are given to the infiles parameter.

Improved support for multi-obi datasets

The script is now able to work with multi-obi datasets that have been processed by splitobs and then chandra_repro.

Changes in the scripts 4.6.6 (September 2014) release

Reprojecting observations

Observations will only be reprojected if the tangent point shift is larger than 0.05 arcseconds, otherwise the original event file is copied.

Warning about different tangent points

The script will now only warn if the tangent points of two files differ by more than 0.05 arcsec.

Changes in the scripts 4.6.5 (June 2014) release

Filtering the aspect solution by the event file GTI

The skyfov is run in order to calculate the sky range covered by each observation; e.g. see ahelp get_fov_limits. The aspect solutions are now filtered by the good-time intervals (GTI) of the event file - that is, the script sets

aspect=asol1.fits[@evt.fits]

(assuming the aspect solution is called asol1.fits and the event file evt.fits) when calling skyfov. This should not change most results but may produce slightly-smaller output images in cases where the GTI of the observation covers only a small fraction of the aspect solution.

HRC-I analysis

It is now possible to apply a PI filter range for the images by specifying PI limits in the bands parameter, that is bands=30:200:1.2 will use a PI range of 30 to 200 when creating all the imaging products. The monochromatic energy used for the instrument map is still given in keV, so 1.2 keV in this example.

A new subtraction method, which scales the background to match the data using the ratio of the counts in the band PI=300:500 (this can be changed with the new bkgparams parameter), can be selected by setting method=particle. The default method uses the exposure times of the observation and particle background files for the scaling. The scaling method and factor are added to the relevant files using the BKGMETH and BKGSCALE keywords, respectively.

The status filter used to filter the HRC-I particle background file is now included in the screen output at the default verbosity level and added to the particle background files (retained when cleanup=no) using the STATFILT keyword. The background-unsubtracted files retained when cleanup=no now match the documentation (rather than include the typo 'undsubtracted').

Changes in the scripts 4.6.2 (February 2014) release

There have been some internal changes to try and avoid incompatibilities between CIAO and FTOOLS parameter files.

Setting maskfile to NONE

The bug which meant that maskfile could not be set to "NONE" has been fixed.

Changes in the scripts 4.6.1 (December 2013) release

Removal of the pbkfiles parameter

The pbkfiles parameter has been removed as these files are no longer needed to run mkinstmap on ACIS observations. This means that it is no longer possible to turn off the ACIS dead-area correction for the exposure map; users who need this flexibility will have to generate the exposure maps manually (e.g. the ACIS single-chip thread).

For old data sets the removal of the pbkfile parameter can lead to error messages about missing keywords; in this case please download the latest archive version of your observation with download_chandra_obsid and then reprocess with chandra_repro.

Changes in the scripts 4.5.4 (August 2013) release

The description of the bands parameter has been updated to point out that when using the explicit format - e.g. a:b:c - for an energy band then the a and b values are required for ACIS data but should not be set for HRC data.

The handling of files with a DataModel filter has been improved.

Changes in the scripts 4.5.3 (June 2013) release

Supplying ancillary files

This release fixes a bug introduced in the scripts 4.5.2 release which meant the script could fail with an obscure error message ('ObsInfo' object has no attribute 'cycle') when given a stack of files for the ancillary files, such as the badpixfile parameter.

Changes in the scripts 4.5.2 (April 2013) release

Handling HRC data

When merging HRC files, the sub-spaces of several columns - including CLKTICKS, AV1, and AU1 - have been removed to avoid the possibility of creating multiple GTI blocks. See the HRC subspace caveat for more information.

The script will now warn you if the HRC observations contain old and new data, in which the defnition of the PI column has changed. If you see this warning we strongly advise you to reprocess the old dataset(s) with chandra_repro and re-do your analysis.

Event file selection

The script now skips event files which have an OBS_ID keyword of "Merged".

Interleaved mode data

The script will now include both the primary and secondary cycles (i.e. the "e1" and "e2" forms) of ACIS interleaved mode data. Previously it would just select one of the files (generally the "e1", which is the shorter of the two). If you only want to include one of the cycles then use an explicit list of observations. When processing interleaved-mode data, file names are labelled as "<obsid>e1" or "<obsid>e2".

Aspect solutions

The program now checks that the aspect-solution files are not aspect histograms.

Improved warnings and errors

The handling of missing ancillary files has been improved; if none of a particular type - such as the mask file - can be found then the program runs as if you has set that parameter to 'NONE', otherwise the error message now tells you what observation was missing a file.

Changes in the scripts 4.5.1 (December 2012) release

The script has been updated to work in CIAO 4.5.

About Contributed Software

This script is not an official part of the CIAO release but is made available as "contributed" software via the CIAO scripts page. Please see this page for installation instructions - such as how to ensure that the parameter file is available.

Bugs

There are no known bugs for this tool.

See Also

calibration

ardlib

psf

psf

tools::aspect

asphist, dither_region

tools::background

acis_bkgrnd_lookup, hrc_bkgrnd_lookup, readout_bkg

tools::composite

combine_grating_spectra, combine_spectra, flux_obs, fluximage, reproject_obs, specextract

tools::coordinates

sky2tdet

tools::core

dmextract

tools::response

acis_fef_lookup, acis_set_ardlib, addresp, dmarfadd, eff2evt, find_mono_energy, fullgarf, make_instmap_weights, mean_energy_map, mkacisrmf, mkarf, mkexpmap, mkgarf, mkgrmf, mkinstmap, mkosip, mkpsfmap, mkrmf, mkrprm, mkwarf, psf_project_ray, rmfimg

tools::statistics

aprates