Merging Data from Multiple Imaging Observations
![[CXC Logo]](../../imgs/cxc-logo.gif){kind=logo}
CIAO 4.14 Science Threads
Overview
Synopsis:
When merging event files from observations with similar, but not necessarily identical, target coordinates, the event sky coordinates (X,Y in the event list) must be recomputed. This is often referred to as "regridding" or "reprojecting events to a different tangent point." It is necessary because the sky coordinates are computed relative to the nominal R.A. and Dec. for each individual observation, which will likely differ between observations of the same target.
Purpose:
To combine ACIS or HRC imaging data from multiple observations, e.g. to search for faint sources or low-surface-brightness extended features.
The merged event list should not be used for spectral analysis, since it does not contain sufficient information to generate correct response files. The recommended technique for the spectral analysis case is to generate separate PHA, RMF, and ARF files for each observation and to analyze them simultaneously.
Related Links:
Last Update: 13 Jan 2022 - Reviewed for CIAO 4.14. No changes.
Contents
- Getting Started
- Merging about a Common Tangent Point
- Correcting Aspect Prior to Merging
- History
- Images
Getting Started
Download the sample data: 1842 (ACIS-I, G21.5-09); 1843 (ACIS-I, G21.5-09)
unix% download_chandra_obsid 1842,1843
unix% chandra_repro 1842,1843 outdir=
Merging about a Common Tangent Point
Each event list contains all the World Coordinate System (WCS) information required to transform between sky coordinates and celestial coordinates in its header. If the WCSs are accurate, one may merge event lists simply by converting sky coordinates in one event list to celestial, using that list's WCS, and then converting those celestial coordinates to sky coordinates in the second event list using the second WCS (see the ds9 thread coordinate definitions for more information).
The level=2 event files may be reprojected to a common tangent point in one of two ways:
- Using WCS Information to Merge: all the files are reprojected to match one file.
- Merging about a Specific R.A. and Dec.: the files are reprojected to a new tangent point that is different than any of the input observations.
Using WCS Information to Merge
In this case we choose to map ObsId 1842 onto 1843, so we set the refcoord parameter of reproject_obs to 1843/repro/acisf01843_repro_evt2.fits:
unix% punlearn reproject_obs unix% reproject_obs 1842,1843 merged1/ refcoord=1843/repro/acisf01843_repro_evt2.fits Running reproject_obs Version: 05 November 2020 Found 1842/repro/acisf01842_repro_evt2.fits Found 1843/repro/acisf01843_repro_evt2.fits Verifying 2 observations. Tangent point is taken from the file 1843/repro/acisf01843_repro_evt2.fits New tangent point: RA=18h 32m 11.140s Dec=-10d 34' 9.85" Observations to be reprojected: Obsid Obs Date Exp DETNAM SIM_Z FP Sepn PA (ks) (mm) (K) (') (deg) --------------------------------------------------------------- 1 1843 2000-09-02 7.9 ACIS-012367 -245.857 153.4 0.0 +0 2 1842 2000-09-02 7.4 ACIS-012367 -231.239 153.4 5.1 +101 Running tasks in parallel with 4 processors. Reprojecting 2 event files to a common tangent point. Merging reprojected events files to merged1/merged_evt.fits The following files were created: The reprojected event files: merged1/1843_reproj_evt.fits merged1/1842_reproj_evt.fits The merged event file: merged1/merged_evt.fits Warning: the merged event file merged1/merged_evt.fits should not be used to create ARF/RMF/exposure maps because the RA_NOM keyword varies by 0.0851 (limit is 0.0003) the DEC_NOM keyword varies by 0.0159 (limit is 0.0003) the ROLL_NOM keyword varies by 2.5 (limit is 1.0) the SIM_Z keyword varies by 14.6 (limit is 0.1)
The contents of the parameter file may be checked using dmhistory:
unix% dmhistory merged1/merged_evt.fits reproject_obs reproject_obs infiles="1842,1843" outroot="merged1/" asolfiles="" merge="yes" refcoord="1843/repro/acisf01843_repro_evt2.fits" parallel="yes" nproc="INDEF" linkfiles="yes" tmpdir="./" clobber="yes" verbose="1"
and the script created the following files:
unix% ls -1 merged1/ 1842.asol@ 1842.bpix@ 1842.mask@ 1842_reproj_evt.fits 1843.asol@ 1843.bpix@ 1843.mask@ 1843_reproj_evt.fits merged_evt.fits
The merged event file can be displayed in ds9:
unix% ds9 -cmap b merged1/merged_evt.fits -bin filter 'energy=500:7000' -bin about 3815 3975 -zoom 8
as shown in Figure 1.
Figure 1: Data merged by WCS information
![[Thumbnail image: The source photon distribution in the merged event list appears as slightly double-peaked.]](wcs.png)
[Version: full-size]
Figure 1: Data merged by WCS information
The appearance of the merged source - which has two peaks - suggests that there is a small systematic offset in the two WCSs.
The source photon distribution in the merged event list appears as slightly double-peaked, indicating that there is in fact a small (2-3 pixels) systematic offset in the two WCSs. This may be minimized by first correcting the aspect, as described in the Correcting Aspect Prior to Merging section.
Merging about a Specific R.A. and Dec.
It is also possible to reproject the events to a new tangent point that is different than any of the input observations. In this example, both event lists are reprojected to the arbitrarily chosen point (R.A.,Dec) = (278.086,-10.58625) prior to merging.
unix% punlearn reproject_obs unix% reproject_obs 1842,1843 merged2/ refcoord="278.086,-10.58625" Running reproject_obs Version: 05 November 2020 Found 1842/repro/acisf01842_repro_evt2.fits Found 1843/repro/acisf01843_repro_evt2.fits Verifying 2 observations. New tangent point: RA=18h 32m 20.640s Dec=-10d 35' 10.50" Observations to be reprojected: Obsid Obs Date Exp DETNAM SIM_Z FP Sepn PA (ks) (mm) (K) (') (deg) --------------------------------------------------------------- 1 1843 2000-09-02 7.9 ACIS-012367 -245.857 153.4 2.5 -67 2 1842 2000-09-02 7.4 ACIS-012367 -231.239 153.4 2.7 +89 Running tasks in parallel with 4 processors. Reprojecting 2 event files to a common tangent point. Merging reprojected events files to merged2/merged_evt.fits The following files were created: The reprojected event files: merged2/1843_reproj_evt.fits merged2/1842_reproj_evt.fits The merged event file: merged2/merged_evt.fits Warning: the merged event file merged2/merged_evt.fits should not be used to create ARF/RMF/exposure maps because the RA_NOM keyword varies by 0.0851 (limit is 0.0003) the DEC_NOM keyword varies by 0.0159 (limit is 0.0003) the ROLL_NOM keyword varies by 2.5 (limit is 1.0) the SIM_Z keyword varies by 14.6 (limit is 0.1)
The contents of the parameter file may be checked using dmhistory:
unix% dmhistory merged2/merged_evt.fits reproject_obs reproject_obs infiles="1842,1843" outroot="merged2/" asolfiles="" merge="yes" refcoord="278.086,-10.58625" parallel="yes" nproc="INDEF" linkfiles="yes" tmpdir="./" clobber="yes" verbose="1"
The data looks the same as Figure 1 apart from a different SKY to Ra,Dec mapping; the choice of tangent point means that the source is close to 4090,4095 rather than 3815,3975 above:
unix% ds9 -cmap b merged2/merged_evt.fits -bin filter 'energy=500:7000' -zoom 8
Correcting Aspect Prior to Merging
In order to improve on the results of the previous examples, we must address the problem of the systematic shift in the coordinate systems of the two event lists. To do this, we will shift the aspect solution of one to make it consistent with the other.
![[WARNING]](../../imgs/warning.png)
If you choose to use this method, the shifted aspect solution should then be used throughout the rest of the analysis in order to obtain correct results. This is why we update the ASOLFILE keyword in the reprojected event file.
First we determine what shift to apply to the data. In this case we use dmstat to calculate the centroid of the source in the two observations, but other approaches, such as using the positions calculated by wavdetect are also possible.
There is a bug in the datamodel where the virtual EQPOS column cannot accessed in the same command where other regular column filters are applied. Therefore we filter the file first with dmcopy and then use dmstat to compute the mean RA and DEC coordinates.
unix% punlearn dmstat
unix% dmcopy "1842/repro/acisf01842_repro_evt2.fits[energy=500:7000][sky=circle(4418,4087,5)]" - | dmstat "-[cols eqpos]"
EQPOS(RA, DEC)[deg]
min: ( 278.08616151 -10.587086206 ) @: ( 63 70 )
max: ( 278.08718637 -10.586138674 ) @: ( 34 12 )
mean: ( 278.08680354 -10.586491986 )
sigma: ( 0.00022748737552 0.00016481664255 )
sum: ( 22246.944283 -846.91935884 )
good: ( 80 80 )
null: ( 0 0 )
unix% set p1 = `pget dmstat out_mean`
unix% dmcopy "1843/repro/acisf01843_repro_evt2.fits[energy=500:7000][sky=circle(3805,3974,5)]" - | dmstat "-[cols eqpos]"
EQPOS(RA, DEC)[deg]
min: ( 278.08649298 -10.58658084 ) @: ( 14 42 )
max: ( 278.08732554 -10.585526826 ) @: ( 26 97 )
mean: ( 278.08692147 -10.586098349 )
sigma: ( 0.000140929028 0.00020055219192 )
sum: ( 31423.822127 -1196.2291135 )
good: ( 113 113 )
null: ( 0 0 )
unix% set p2 = `pget dmstat out_mean`
unix% echo $p1
278.08680354,-10.586491986
unix% echo $p2
278.08692147,-10.586098349
We decide to shift ObsId 1842 onto 1843 since it is the shorter observation (although in this case the difference is not large). To find the shift:
-
calculate the difference between the two centroids:
unix% python >>> ra1 = 278.08680354 >>> ra2 = 278.08692147 >>> dec1 = -10.586491986 >>> dec2 = -10.586098349 >>> dra = ra2 - ra1 >>> ddec = dec2 - dec1 >>> dra 0.00011792999998760934 >>> ddec 0.00039363700000016877 -
correct the delta RA by "cos(Dec)" to get a reasonable approximation of the distance between the two points.
>>> mean_dec = ( dec1 + dec2 )/2.0 >>> dra2 = dra * np.cos(mean_dec * np.pi / 180.0) >>> dra2 0.00011592275133369907 -
convert the values to sky pixels. The delta x conversion is done with -1.3667E-4 deg/pixel because the x value is opposite in sign from RA. Dec and y have the same sign, so 1.3667E-4 deg/pixel is used for that calculation. (note that 0.492/3600 = 1.3667e-4, where 0.492 is the ACIS pixel size)
>>> conv = 0.492 / 3600 >>> dx = dra2 * -1 / conv >>> dy = ddec / conv >>> dx -0.8482152536612126 >>> dy 2.880270731708552
The tool wcs_update is used to apply the offsets. The shifts are input to the deltax and deltay parameters in units of sky pixels. Additionally, the WCS information from the event file for the match observation (1843/repro/acisf01843_repro_evt2.fits) is supplied in the wcsfile parameter. Note that the transformfile parameter is left empty.
unix% punlearn wcs_update unix% pset wcs_update infile=1842/repro/pcadf01842_000N001_asol1.fits unix% pset wcs_update outfile=1842/repro/new_asol1.fits unix% pset wcs_update wcsfile=1843/repro/acisf01843_repro_evt2.fits unix% pset wcs_update deltax=-0.85 deltay=2.88 unix% wcs_update Input coordinate transform file (): Either input asol file, or file with WCS to be updated (1842/repro/pcadf01842_000N001_asol1.fits): Output asol file (1842/repro/new_asol1.fits):
We then use the new aspect solution to re-calculate the WCS positions of events in the original event file with the reproject_events tool:
unix% punlearn reproject_events unix% pset reproject_events infile=1842/repro/acisf01842_repro_evt2.fits unix% pset reproject_events outfile=1842/repro/new_evt2.fits unix% pset reproject_events aspect=1842/repro/new_asol1.fits unix% reproject_events Input dataset/block specification (1842/repro/acisf01842_repro_evt2.fits): Output dataset/block specification (1842/repro/new_evt2.fits): Match file (none):
and adjust the ASOLFILE keyword in the new events file to point to the new aspect solution:
unix% dmhedit 1842/repro/new_evt2.fits filelist= op=add key=ASOLFILE value=new_asol1.fits
![[IMPORTANT]](../../imgs/important.png)
reproject_events does NOT preserve the sub-pixel improvements made by the EDSER algorithm in acis_process_events when the aspect parameter is not empty, as above.
The acis_process_events tool can be used to preserve the EDSER improvements, but care needs to be taken if you have used non-standard settings previously with this tool to create the level-2 event file.
unix% punlearn acis_process_events unix% pset acis_process_events apply_cti- apply_tgain- doevtgrade- unix% pset acis_process_events calculate_pi- rand_pha- unix% acis_process_events 1842/repro/acisf01842_repro_evt2.fits 1842/repro/ape_evt2.fits 1842/repro/new_asol1.fits # acis_process_events (CIAO): WARNING: The values of PHA, ENERGY, PI, FLTGRADE and GRADE may be inaccurate because doevtgrade=no. # acis_process_events (CIAO): WARNING: The values of ENERGY and PI may be inaccurate because calculate_pi=no.
Manually reprojecting the data
We can now reproject ObsId 1842 onto 1843 before merging them (making sure to list the inputs to dmmerge in time order):
unix% punlearn reproject_events unix% reproject_events 1842/repro/new_evt2.fits 1842/repro/reproj_evt2.fits 1843/repro/acisf01843_repro_evt2.fits unix% punlearn dmmerge unix% dmmerge 1843/repro/acisf01843_repro_evt2.fits,1842/repro/reproj_evt2.fits merged.fits omit - DEC_PNT values different more than 0.000300 warning: DS_IDENT has different value...Merged... warning: OBJECT has different value...Merged... warning: OBS_ID has different value...Merged... omit - RA_PNT values different more than 0.000300 omit - ROLL_NOM values different more than 1.000000 omit - ROLL_PNT values different more than 1.000000 warning: SEQ_NUM has different value...Merged... omit - SIM_Z values different more than 0.100000
The merged event file can be displayed in ds9:
unix% ds9 -cmap b merged.fits -bin filter 'energy=500:7000' -bin about 3815 3975 -zoom 8
(as shown in Figure 2).
Figure 2: Aspect solution corrected before reprojection
![[Thumbnail image: The source now has a single peak, rather than the double peak visible in previous versions.]](corrasol.png)
[Version: full-size]
Figure 2: Aspect solution corrected before reprojection
The shift in the peak emission visible in earlier versions - such as Figure 1 - is no longer seen.
Using reproject_obs
When combining data from multiple observations - or you wish to reproject the data to a different position - it is easier to run reproject_obs on all the observations, rather than running reproject_events and dmmerge manually. For example, we could have said
unix% punlearn reproject_obs
unix% reproject_obs 1842/repro/new_evt2.fits,1843/repro/acisf01843_repro_evt2.fits merged3/
Running reproject_obs
Version: 05 November 2020
Verifying 2 observations.
Calculating new tangent point.
New tangent point: RA=18h 32m 21.347s Dec=-10d 34' 38.44"
Observations to be reprojected:
Obsid Obs Date Exp DETNAM SIM_Z FP Sepn PA
(ks) (mm) (K) (') (deg)
---------------------------------------------------------------
1 1843 2000-09-02 7.9 ACIS-012367 -245.857 153.4 2.6 -79
2 1842 2000-09-02 7.4 ACIS-012367 -231.239 153.4 2.6 +101
Running tasks in parallel with 4 processors.
Reprojecting 2 event files to a common tangent point.
Merging reprojected events files to merged3/merged_evt.fits
The following files were created:
The reprojected event files:
merged3/1843_reproj_evt.fits
merged3/1842_reproj_evt.fits
The merged event file:
merged3/merged_evt.fits
Warning: the merged event file merged3/merged_evt.fits
should not be used to create ARF/RMF/exposure maps because
the RA_NOM keyword varies by 0.0851 (limit is 0.0003)
the DEC_NOM keyword varies by 0.0159 (limit is 0.0003)
the ROLL_NOM keyword varies by 2.5 (limit is 1.0)
the SIM_Z keyword varies by 14.6 (limit is 0.1)
unix% ds9 -cmap b merged3/merged_evt.fits -bin filter 'energy=500:7000' -bin about 4121 4038 -zoom 8
which creates an image similar to Figure 2, although with different mapping between SKY and WORLD coordinates, because of the different tangent point.
History
| 10 Jan 2012 | reviewed for CIAO 4.4: clarified the order of the steps in the Correcting Aspect Prior to Merging section |
| 15 Oct 2012 | Updated to use reproject_obs rather than merge_all, which is now deprecated. The Correcting Aspect Prior to Merging section has been updated. |
| 03 Dec 2012 | Review for CIAO 4.5 |
| 15 Mar 2013 | Fixed links in manually merging section. |
| 27 Jun 2013 | Tweaked information in the distance equations. |
| 02 Dec 2013 | Review for CIAO 4.6. Noted that the PBK file for ACIS observations is no-longer copied/linked by reproject_obs. |
| 16 Dec 2014 | Reviewed for CIAO 4.7. Added a link to the merged datasets spectrum caveat. |
| 27 Nov 2017 | Added link to bug in DM that affects [cols eqpos] filtering. |
| 13 May 2019 | Updated to use python to compute offsets. |
| 18 Oct 2021 | Updated for Repro-5 changes and CIAO 4.13/CALDB4.9.6. |
| 13 Jan 2022 | Reviewed for CIAO 4.14. No changes. |