HRC Subspace Caveat
Caveat: Merging some HRC observations
A change was made to HRC Level 0 data in standard data processing version DS8.5 (December 5, 2012) which corrected the allowed range values of some engineering columns in the Level 0 event file. These ranges are part of the file's data subspace and are propagated from Level 0 through Level 2.
This can cause problems when files created with different releases are merged. If any part of the subspace is different, the Good Time Intervals will not merge together which can result in the ONTIME (and related keywords such as LIVETIME and EXPOSURE) being incorrectly computed. This affects all CIAO tools and scripts when applied to merging datasets. The merge_obs and reproject_obs scripts will have a fix coming soon.
Users can work around the issue by removing the subspace associated with the columns as in the example below:
unix% dmcopy "hrc_evt2.fits[subspace -av1,-au1,-mjf,-mnf,-endmnf,-sub_mjf,-clkticks]" evt2.fits
The columns listed above are not typically used when analyzing Chandra data so removing the subspace associated with them is acceptable. Removing the subspace of other arbitrary columns should not be done unless the user fully understands the consequences of doing so.
Are my data affected?
The easiest way to illustrate the problem is with an example. Below we are going to attempt to merge two observation of M31.
unix% download_chandra_obsid 6202,13231 evt2
unix% mv {6202,13231}/primary/*evt2.fits.gz ./
unix% dmlist "*evt2.fits.gz" header,clean | egrep "ASCDSVER"
ASCDSVER 8.5.1 Processing system revision
ASCDSVER 8.4.5 Processing system revisionThis shows that indeed the two observations have been processed with two different processing versions: one before the change and one after. We can check for other difference using the dmdiff tool.
unix% dmdiff *evt2.fits.gz data- key- Infile 1: hrcf06202N004_evt2.fits.gz Infile 2: hrcf13231N003_evt2.fits.gz -------------------------- SUBSPACE VALUE DIFFERENCES -------------------------- Message: Column: Row Value(s): Diff: -------- ------- --- --------- ----- Range maxs are not equal CLKTICKS 1 262143 1091567608 +1.09131e+09 (+4.16e+05 %) # dmdiff (CIAO 4.5): WARNING: subspace column "status" type unknown or not supported. # dmdiff (CIAO 4.5): WARNING: subspace column "SUB_MJF" type unknown or not supported.
This shows us that the CLKTICKS (clock ticks) column has a different range. Note that SUB_MJF is not checked since dmdiff is currently unable to check BYTE subspace data types. Looking at the raw data (see dmlist subspace example below), we can see though that it too is different.
Despite these differences, we continue by reprojecting the observations to a common tangent point and using dmmerge to merge the files together.
unix% reproject_events hrcf06202N004_evt2.fits.gz 6202_reprj.fits match=hrcf13231N003_evt2.fits.gz clob+ unix% dmmerge 6202_reprj.fits,hrcf13231N003_evt2.fits.gz merge.fits clob+ BTIMDRFT values are different...FAIL... BTIMNULL values are different...FAIL... BTIMRATE values are different...FAIL... warning: DS_IDENT has different value...Merged... warning: OBJECT has different value...Merged... warning: OBSERVER has different value...Merged... warning: OBS_ID has different value...Merged... omit - RA_PNT values different more than 0.000300 omit - ROLL_NOM values different more than 1.000000 omit - ROLL_PNT values different more than 1.000000 warning: SEQ_NUM has different value...Merged... warning: TITLE has different value...Merged...
The various keyword warnings are normal and expected when merging different observations, see merging_rules. If we now look the structure of the merged file we will see an anomaly
unix% dmlist merge.fits blocks
--------------------------------------------------------------------------------
Dataset: merge.fits
--------------------------------------------------------------------------------
Block Name Type Dimensions
--------------------------------------------------------------------------------
Block 1: PRIMARY Null
Block 2: EVENTS Table 9 cols x 2071847 rows
Block 3: GTI Table 2 cols x 1 rows
Block 4: GTI_CPT2 Table 2 cols x 1 rowsFor an HRC event file there is usually only 1 GTI block that describes the good time. If things merged cleanly this should still be the case but as we suspected, we are now seeing two. If we ask the datamodel to display the subspace information we see
% dmlist merge.fits subspace
--------------------------------------------------------------------------------
Data subspace for block EVENTS: Components: 2 Descriptors: 26
--------------------------------------------------------------------------------
--- Component 1 ---
1 time Real8 TABLE GTI
223264170.3893871307:223282340.8214586079
...
25 SUB_MJF Byte 0:7
26 CLKTICKS Int4 0:262143
--- Component 2 ---
1 time Real8 TABLE GTI_CPT2
440755355.7957056761:440775036.8217450380
25 SUB_MJF Byte 0:64
26 CLKTICKS Int4 0:1091567608 Since SUB_MJF and CLKTICKS are different, the GTIs cannot be combined.
Now, what effect does this have on the data? Initially, not much. If we check the ONTIME keywords in the event files we see
unix% dmlist "*evt2.fits.gz,merge.fits" header,clean | grep ONTIME ONTIME 18170.4320714770 [s] ONTIME 19681.0260393620 [s] ONTIME 37851.4581108390 [s]
Where the last value listed above is associated with the merge.fits and looks to be as expected: the sum of the earlier two. The dmmerge keyword merging rule, calcGTI correctly unions the two GTIs to compute the ONTIME value.
However, this file is no longer self consistent. Any further filtering will trigger the ONTIME to be recomputed and following the CXC conventions will be computed from the first GTI.
unix% dmcopy "merge.fits[sky=circle(16135,16544,25)]" mysrc.fits unix% dmlist mysrc.fits header,clean | grep ONTIME ONTIME 18170.4320714770 [s]
The example above shows that applying a simple spatial filter has resulted in the output file's ONTIME value being incorrectly recomputed. If one were to use this ONTIME value to compute fluxes it would be incorrect.