Return to Threads Page: Top | All | Imag

Merging Data from Multiple Imaging Observations

CIAO 4.16 Science Threads

Overview

Synopsis:

When merging event files from observations with similar, but not necessarily identical, target coordinates, the event sky coordinates (X,Y in the event list) must be recomputed. This is often referred to as "regridding" or "reprojecting events to a different tangent point." It is necessary because the sky coordinates are computed relative to the nominal R.A. and Dec. for each individual observation, which will likely differ between observations of the same target.

Purpose:

To combine ACIS or HRC imaging data from multiple observations, e.g. to search for faint sources or low-surface-brightness extended features.

The merged event list should not be used for spectral analysis, since it does not contain sufficient information to generate correct response files. The recommended technique for the spectral analysis case is to generate separate PHA, RMF, and ARF files for each observation and to analyze them simultaneously.

Related Links:

Last Update: 26 Jul 2023 - Corrected and renamed the Correcting Astrometry Prior to Merging section. Specifically added the missing step to update the event file and to remove the inaccurate information about preserving the EDSER subpixel event position.

Getting Started

Download the sample data: 1842 (ACIS-I, G21.5-09); 1843 (ACIS-I, G21.5-09)

unix% download_chandra_obsid 1842,1843

unix% chandra_repro 1842,1843 outdir=

Merging about a Common Tangent Point

Each event list contains all the World Coordinate System (WCS) information required to transform between sky coordinates and celestial coordinates in its header. If the WCSs are accurate, one may merge event lists simply by converting sky coordinates in one event list to celestial, using that list's WCS, and then converting those celestial coordinates to sky coordinates in the second event list using the second WCS (see the ds9 thread coordinate definitions for more information).

The level=2 event files may be reprojected to a common tangent point in one of two ways:

Using WCS Information to Merge: all the files are reprojected to match one file.
Merging about a Specific R.A. and Dec.: the files are reprojected to a new tangent point that is different than any of the input observations.

Using WCS Information to Merge

In this case we choose to map ObsId 1842 onto 1843, so we set the refcoord parameter of reproject_obs to 1843/repro/acisf01843_repro_evt2.fits:

unix% punlearn reproject_obs
unix% reproject_obs 1842,1843 merged1/ refcoord=1843/repro/acisf01843_repro_evt2.fits
Running reproject_obs
Version: 05 November 2020

Found 1842/repro/acisf01842_repro_evt2.fits
Found 1843/repro/acisf01843_repro_evt2.fits
Verifying 2 observations.
Tangent point is taken from the file 1843/repro/acisf01843_repro_evt2.fits
New tangent point: RA=18h 32m 11.140s Dec=-10d 34' 9.85"

Observations to be reprojected:

  Obsid  Obs Date   Exp    DETNAM     SIM_Z    FP   Sepn   PA  
                   (ks)                (mm)    (K)   (')  (deg)
---------------------------------------------------------------
1 1843  2000-09-02   7.9 ACIS-012367 -245.857 153.4   0.0    +0
2 1842  2000-09-02   7.4 ACIS-012367 -231.239 153.4   5.1  +101

Running tasks in parallel with 4 processors.
Reprojecting 2 event files to a common tangent point.
Merging reprojected events files to merged1/merged_evt.fits

The following files were created:

The reprojected event files:
     merged1/1843_reproj_evt.fits
     merged1/1842_reproj_evt.fits

The merged event file:
     merged1/merged_evt.fits

Warning: the merged event file merged1/merged_evt.fits
   should not be used to create ARF/RMF/exposure maps because
      the RA_NOM keyword varies by 0.0851 (limit is 0.0003)
      the DEC_NOM keyword varies by 0.0159 (limit is 0.0003)
      the ROLL_NOM keyword varies by 2.5 (limit is 1.0)
      the SIM_Z keyword varies by 14.6 (limit is 0.1)

The contents of the parameter file may be checked using dmhistory:

unix% dmhistory merged1/merged_evt.fits reproject_obs 
reproject_obs infiles="1842,1843" outroot="merged1/" asolfiles="" 
  merge="yes" refcoord="1843/repro/acisf01843_repro_evt2.fits" 
  parallel="yes" nproc="INDEF" linkfiles="yes" tmpdir="./" clobber="yes" verbose="1"

and the script created the following files:

unix% ls -1 merged1/
1842.asol@
1842.bpix@
1842.mask@
1842_reproj_evt.fits
1843.asol@
1843.bpix@
1843.mask@
1843_reproj_evt.fits
merged_evt.fits

The merged event file can be displayed in ds9:

unix% ds9 -cmap b merged1/merged_evt.fits -bin filter 'energy=500:7000' -bin about 3815 3975 -zoom 8

as shown in Figure 1.

Figure 1: Data merged by WCS information

[Thumbnail image: The source photon distribution in the merged event list appears as slightly double-peaked.]

[Version: full-size]

[Print media version: The source photon distribution in the merged event list appears as slightly double-peaked.]

Figure 1: Data merged by WCS information

The appearance of the merged source - which has two peaks - suggests that there is a small systematic offset in the two WCSs.

The source photon distribution in the merged event list appears as slightly double-peaked, indicating that there is in fact a small (2-3 pixels) systematic offset in the two WCSs. This may be minimized by first correcting the aspect, as described in the Correcting Aspect Prior to Merging section.

Merging about a Specific R.A. and Dec.

It is also possible to reproject the events to a new tangent point that is different than any of the input observations. In this example, both event lists are reprojected to the arbitrarily chosen point (R.A.,Dec) = (278.086,-10.58625) prior to merging.

unix% punlearn reproject_obs
unix% reproject_obs 1842,1843 merged2/ refcoord="278.086,-10.58625"
Running reproject_obs
Version: 05 November 2020

Found 1842/repro/acisf01842_repro_evt2.fits
Found 1843/repro/acisf01843_repro_evt2.fits
Verifying 2 observations.
New tangent point: RA=18h 32m 20.640s Dec=-10d 35' 10.50"

Observations to be reprojected:

  Obsid  Obs Date   Exp    DETNAM     SIM_Z    FP   Sepn   PA  
                   (ks)                (mm)    (K)   (')  (deg)
---------------------------------------------------------------
1 1843  2000-09-02   7.9 ACIS-012367 -245.857 153.4   2.5   -67
2 1842  2000-09-02   7.4 ACIS-012367 -231.239 153.4   2.7   +89

Running tasks in parallel with 4 processors.
Reprojecting 2 event files to a common tangent point.
Merging reprojected events files to merged2/merged_evt.fits

The following files were created:

The reprojected event files:
     merged2/1843_reproj_evt.fits
     merged2/1842_reproj_evt.fits

The merged event file:
     merged2/merged_evt.fits

Warning: the merged event file merged2/merged_evt.fits
   should not be used to create ARF/RMF/exposure maps because
      the RA_NOM keyword varies by 0.0851 (limit is 0.0003)
      the DEC_NOM keyword varies by 0.0159 (limit is 0.0003)
      the ROLL_NOM keyword varies by 2.5 (limit is 1.0)
      the SIM_Z keyword varies by 14.6 (limit is 0.1)

The contents of the parameter file may be checked using dmhistory:

unix% dmhistory merged2/merged_evt.fits reproject_obs 
reproject_obs infiles="1842,1843" outroot="merged2/" asolfiles="" merge="yes" 
  refcoord="278.086,-10.58625" parallel="yes" nproc="INDEF" linkfiles="yes" 
  tmpdir="./" clobber="yes" verbose="1"

The data looks the same as Figure 1 apart from a different SKY to Ra,Dec mapping; the choice of tangent point means that the source is close to 4090,4095 rather than 3815,3975 above:

unix% ds9 -cmap b merged2/merged_evt.fits -bin filter 'energy=500:7000' -zoom 8

Correcting Astrometry Prior to Merging

In order to improve on the results of the previous examples, we must address the problem of the systematic shift in the coordinate systems of the two event lists. To do this, we will correct the astrometry of the data to make them consistent with each the other.

Warning

If you choose to use this method, the shifted aspect solution should then be used throughout the rest of the analysis in order to obtain correct results. This is why we update the ASOLFILE keyword in the reprojected event file.

For the details of how to compute and apply astrometric corrections, users should read the Correcting Absolute Astrometry thread. Below is an abbreviated and lightly annotated example of one approach to do this.

First we determine what shift to apply to the data. In this case we use dmstat to calculate the centroid of the source in the two observations, but other approaches, such as using the positions calculated by wavdetect are also possible.

There is a bug in the data model where the virtual EQPOS column cannot be accessed in the same command where other regular column filters are applied. Therefore we filter the file first with dmcopy to the region of interest (and in this example, the energy band, though this is not necessary for correcting astrometry), then use dmstat to compute the mean RA and DEC coordinates. In this example, these two tools are run in one line to skip the creation of an extraneous intermediate file; with this syntax, the output of dmcopy ingested by dmstat is not saved to the directory, and only the output of dmstat is returned.

unix% punlearn dmstat
unix% dmcopy "1842/repro/acisf01842_repro_evt2.fits[energy=500:7000][sky=circle(4418,4087,5)]" - | dmstat "-[cols eqpos]"
EQPOS(RA, DEC)[deg]
    min:	( 278.08616151 -10.587086206 )	      @:	( 63 70 )
    max:	( 278.08718637 -10.586138674 )	      @:	( 34 12 )
  mean:	( 278.08680354 -10.586491986 )
  sigma:	( 0.00022748737552 0.00016481664255 )
    sum:	( 22246.944283 -846.91935884 )
   good:	( 80 80 )
   null:	( 0 0 )

unix% dmcopy "1843/repro/acisf01843_repro_evt2.fits[energy=500:7000][sky=circle(3805,3974,5)]" - | dmstat "-[cols eqpos]"
EQPOS(RA, DEC)[deg]
    min:	( 278.08649298 -10.58658084 )	      @:	( 14 42 )
    max:	( 278.08732554 -10.585526826 )	      @:	( 26 97 )
   mean:	( 278.08692147 -10.586098349 )
  sigma:	( 0.000140929028 0.00020055219192 )
    sum:	( 31423.822127 -1196.2291135 )
   good:	( 113 113 )
   null:	( 0 0 )

We decide to shift ObsId 1842 onto 1843 since it is the shorter observation (although in this case the difference is not large). ObsID 1843 will be the reference dataset. To find the shift:

calculate the difference between the two centroids:

unix% python
>>> ra1 = 278.08680354
>>> ra_ref = 278.08692147
>>> dec1 = -10.586491986
>>> dec_ref = -10.586098349
>>> dra = ra_ref - ra1
>>> ddec = dec_ref - dec1
>>> dra
         0.00011792999998760934
>>> ddec
         0.00039363700000016877

correct the delta RA by "cos(Dec)" to get a reasonable approximation of the distance between the two points.

>>> mean_dec = ( dec1 + dec_ref )/2.0
>>> dra_ref = dra * np.cos(mean_dec * np.pi / 180.0)
>>> dra_ref
          0.00011592275133369907

convert the values to sky pixels. The delta x conversion is done with -1.3667E-4 deg/pixel because the x value is opposite in sign from RA. Dec and y have the same sign, so 1.3667E-4 deg/pixel is used for that calculation. (note that 0.492/3600 = 1.3667e-4, where 0.492 is the ACIS pixel size)
```
>>> conv = 0.492 / 3600
>>> dx = dra_ref * -1 / conv
>>> dy = ddec / conv
>>> dx
          -0.8482152536612126
>>> dy
          2.880270731708552
```

The tool wcs_update is used to apply the offsets. The shifts are input to the deltax and deltay parameters in units of sky pixels. Additionally, the WCS information from the event file for the match observation (1843/repro/acisf01843_repro_evt2.fits) is supplied in the wcsfile parameter. Note that the transformfile parameter is left empty.

unix% punlearn wcs_update
unix% pset wcs_update infile=1842/repro/pcadf01842_000N001_asol1.fits
unix% pset wcs_update outfile=1842/repro/new_asol1.fits
unix% pset wcs_update wcsfile=1843/repro/acisf01843_repro_evt2.fits
unix% pset wcs_update deltax=-0.85 deltay=2.88
unix% wcs_update
Input coordinate transform file (): 
Either input asol file, or file with WCS to be updated (1842/repro/pcadf01842_000N001_asol1.fits): 
Output asol file (1842/repro/new_asol1.fits):

We then also need to update the event file. Since the event file is modified in place we first make a copy and set the wcs_update outfile to blank, "".

unix% dmcopy 1842/repro/acisf01842_repro_evt2.fits 1842/repro/new_evt2.fits clob+
unix% wcs_update infile=1842/repro/new_evt2.fits outfile="" mode=h

Earlier version of this thread

Earlier versions of this thread had a discussion about preserving the EDSER subpixel event position information and how it can get lost when using reproject_events. However, the information that was presented was not entirely accurate and the suggestion to run acis_process_events was incorrect.

EDSER subpixel event position is preserved when running reproject_events without specifying the aspect solution file, ie setting aspect="" (blank).

Since the fine astrometric correction is applied to both the event file and the aspect solution file the physical sky coordinates are unchanged.

Finally we update the ASOLFILE keyword in the new events file to point to the new aspect solution:

unix% dmhedit 1842/repro/new_evt2.fits filelist= op=add key=ASOLFILE value=new_asol1.fits

Manually reprojecting the data

We can now reproject ObsId 1842 onto 1843 before merging them (making sure to list the inputs to dmmerge in time order):

unix% punlearn reproject_events
unix% reproject_events 1842/repro/new_evt2.fits 1842/repro/reproj_evt2.fits 1843/repro/acisf01843_repro_evt2.fits
unix% punlearn dmmerge
unix% dmmerge 1843/repro/acisf01843_repro_evt2.fits,1842/repro/reproj_evt2.fits merged.fits
omit - DEC_PNT values different more than 0.000300
warning: DS_IDENT has different value...Merged...
warning: OBJECT has different value...Merged...
warning: OBS_ID has different value...Merged...
omit - RA_PNT values different more than 0.000300
omit - ROLL_NOM values different more than 1.000000
omit - ROLL_PNT values different more than 1.000000
warning: SEQ_NUM has different value...Merged...
omit - SIM_Z values different more than 0.100000

The merged event file can be displayed in ds9:

unix% ds9 -cmap b merged.fits -bin filter 'energy=500:7000' -bin about 3815 3975 -zoom 8

(as shown in Figure 2).

Figure 2: Aspect solution corrected before reprojection

[Thumbnail image: The source now has a single peak, rather than the double peak visible in previous versions.]

[Version: full-size]

[Print media version: The source now has a single peak, rather than the double peak visible in previous versions.]

Figure 2: Aspect solution corrected before reprojection

The shift in the peak emission visible in earlier versions - such as Figure 1 - is no longer seen.

Using reproject_obs

When combining data from multiple observations - or you wish to reproject the data to a different position - it is easier to run reproject_obs on all the observations, rather than running reproject_events and dmmerge manually. For example, we could have said

unix% punlearn reproject_obs
unix% reproject_obs 1842/repro/new_evt2.fits,1843/repro/acisf01843_repro_evt2.fits merged3/
Running reproject_obs
Version: 05 November 2020

Verifying 2 observations.
Calculating new tangent point.
New tangent point: RA=18h 32m 21.347s Dec=-10d 34' 38.44"

Observations to be reprojected:

  Obsid  Obs Date   Exp    DETNAM     SIM_Z    FP   Sepn   PA  
                   (ks)                (mm)    (K)   (')  (deg)
---------------------------------------------------------------
1 1843  2000-09-02   7.9 ACIS-012367 -245.857 153.4   2.6   -79
2 1842  2000-09-02   7.4 ACIS-012367 -231.239 153.4   2.6  +101

Running tasks in parallel with 4 processors.
Reprojecting 2 event files to a common tangent point.
Merging reprojected events files to merged3/merged_evt.fits

The following files were created:

The reprojected event files:
     merged3/1843_reproj_evt.fits
     merged3/1842_reproj_evt.fits

The merged event file:
     merged3/merged_evt.fits

Warning: the merged event file merged3/merged_evt.fits
   should not be used to create ARF/RMF/exposure maps because
      the RA_NOM keyword varies by 0.0851 (limit is 0.0003)
      the DEC_NOM keyword varies by 0.0159 (limit is 0.0003)
      the ROLL_NOM keyword varies by 2.5 (limit is 1.0)
      the SIM_Z keyword varies by 14.6 (limit is 0.1)
      
unix% ds9 -cmap b merged3/merged_evt.fits -bin filter 'energy=500:7000' -bin about 4121 4038 -zoom 8

which creates an image similar to Figure 2, although with different mapping between SKY and WORLD coordinates, because of the different tangent point.

History

10 Jan 2012	reviewed for CIAO 4.4: clarified the order of the steps in the Correcting Aspect Prior to Merging section
15 Oct 2012	Updated to use reproject_obs rather than merge_all, which is now deprecated. The Correcting Aspect Prior to Merging section has been updated.
03 Dec 2012	Review for CIAO 4.5
15 Mar 2013	Fixed links in manually merging section.
27 Jun 2013	Tweaked information in the distance equations.
02 Dec 2013	Review for CIAO 4.6. Noted that the PBK file for ACIS observations is no-longer copied/linked by reproject_obs.
16 Dec 2014	Reviewed for CIAO 4.7. Added a link to the merged datasets spectrum caveat.
27 Nov 2017	Added link to bug in DM that affects [cols eqpos] filtering.
13 May 2019	Updated to use python to compute offsets.
18 Oct 2021	Updated for Repro-5 changes and CIAO 4.13/CALDB4.9.6.
13 Jan 2022	Reviewed for CIAO 4.14. No changes.
26 Jul 2023	Corrected and renamed the Correcting Astrometry Prior to Merging section. Specifically added the missing step to update the event file and to remove the inaccurate information about preserving the EDSER subpixel event position.

Return to Threads Page: Top | All | Imag

/ ciao / threads / combine /

Merging Data from Multiple Imaging Observations

Overview

Synopsis:

Purpose:

Contents

Getting Started

Merging about a Common Tangent Point

Using WCS Information to Merge

Figure 1: Data merged by WCS information

Figure 1: Data merged by WCS information

Merging about a Specific R.A. and Dec.

Correcting Astrometry Prior to Merging

Manually reprojecting the data

Figure 2: Aspect solution corrected before reprojection

Figure 2: Aspect solution corrected before reprojection

Using reproject_obs

History