Calculating Spectral Weights for mkinstmap
CIAO 4.13 Science Threads
When a spectral weights file is provided to mkinstmap, the tool computes a weighted instrument map instead of a monochromatic instrument map. Essentially, a weighted instrument map is a linear combination of monochromatic maps calculated for several energy-weight pairs. More details are given in the Why are weighted instrument maps important? section of this thread.
Last Update: 21 Nov 2019 - Updated for CIAO 4.12 to use matplotlib rather than ChIPS for plotting, and to update the figures to use more-recent calibration products.
- Get Started
- Why are weighted instrument maps important?
- Using the make_instmap_weights script
- Step-by-Step Guide
- Notes and Caveats
- Figure 1: Source spectrum
- Figure 2: Weights using a bin width of 0.25 keV
- Figure 3: A comparison of bin widths of 0.25 and 0.5 keV
- Figure 4: Using a variable bin width
- Figure 5: Comparing a set of weights to an ARF
- Figure 6: Ratio of a weighted instrument map to a mono-energetic one
- Figure 7: Ratio of 0.5 keV to 0.25 keV bin-width instrument maps
- Figure 8: Ratio of variable to 0.25 keV bin-width instrument maps
- Figure 9: Ratio of 0.25 keV bin-width to mono-chromatic exposure maps
- Figure 10: Histograms of the ratio images
Download the sample data: 1838 (ACIS-S, G21.5-0.9)
No data file is actually needed to create the weights file. As we create instrument- and exposure- maps to see what difference adding a weights file makes, we require the same data products for these steps as the Single Chip Exposure Map thread.
Why are weighted instrument maps important?
In dividing a counts image by an exposure map, the primary aim is to obtain an image with units of surface brightness in photons/s/cm2/arcsec2 or, equivalently, in flux/pixel. It is important to understand that this can yield an accurate answer only if the number of counts in each pixel is actually proportional to the integral of the incident source spectrum over the relevant energy band.
Due to the strong energy dependence of the effective area, this is not always true. In particular, the number of detected counts is proportional to the incident flux in a broad band only when the effective area is constant within the chosen band. Large systematic errors may result if one blindly applies a monochromatic exposure map to a broad band image which encompasses a large range of variation in the effective area.
Nevertheless, it is often desirable both to examine broader energy bands and to examine energy bands where the response is not particularly flat. In such cases, if the broad features of the incident spectrum are known (e.g. thermal emission from a cluster) and if the spectral variations within the image are not too large, one can reduce the systematic errors by computing an exposure map which is weighted according to a specific model for the incident spectrum. An Introduction to Exposure Maps (PS, 12pp) gives a comparison of MARX simulated data analyzed using both weighted and monochromatic exposure maps.
Using the make_instmap_weights script
The make_instmap_weights script takes a Sherpa model and an optional energy range and produces a weighted spectrum file.
For this example we are going to use a power law, with a gamma of 1.7, which is absorbed by a line-of-sight column density of 1.3 x 1022 atoms cm-2. The models and parameter values should be changed to match the emission from the source of interest. The model is evaluated over the energy range of 0.5-8.0 keV with bins that are 0.25 keV wide:
unix% punlearn make_instmap_weights unix% make_instmap_weights weights.txt "xsphabs.gal*powlaw1d.p1" \ paramvals="gal.nh=1.3;p1.gamma=1.7" emin=0.5 emax=8 ewidth=0.25 Model used: (xsphabs.gal * powlaw1d.p1) Param Type Value Min Max Units ----- ---- ----- --- --- ----- gal.nH thawed 1.3 0 100000 10^22 atoms / cm^2 p1.gamma thawed 1.7 -10 10 p1.ref frozen 1 -3.40282e+38 3.40282e+38 p1.ampl thawed 1 0 3.40282e+38 Created: weights.txt
The output file, weights.txt, contains the energy bin center in keV in the first column and normalized weighting in the second column:
unix% cat weights.txt #TEXT/SIMPLE # X WEIGHT # # DATE = 2019-11-20T16:18:20 / Date and time of file creation # MODELEXPR = (xsphabs.gal * powlaw1d.pl) # ENERG_LO = 0.5 / [keV] Minimum energy # ENERG_HI = 8.0 / [keV] Maximum energy # 0.625 0.0005505499378475431 0.875 0.005133913053133582 1.125 0.02779381690871387 1.375 0.056084831800127403 1.625 0.07066623980281715 1.875 0.07413742820878573 2.125 0.0712275747324698 2.375 0.06669921234265919 2.625 0.06102870878761724 2.875 0.05580786480786807 3.125 0.05067728967895906 3.375 0.04597129491973783 3.625 0.04181074649300642 3.875 0.03810108003743937 4.125 0.03478392863101238 4.375 0.031852303498792825 4.625 0.02928280475960336 4.875 0.02700223451672559 5.125 0.024973234527294007 5.375 0.023162919654288874 5.625 0.021542753530535842 5.875 0.020085760678698657 6.125 0.018773803619361276 6.375 0.017590825350819506 6.625 0.016518718543815914 6.875 0.01554423141902547 7.125 0.014513042710459261 7.375 0.013586416882823383 7.625 0.012875827543709225 7.875 0.012220642621851999
To set the file for use in the spectrumfile parameter of mkinstmap:
unix% pset mkinstmap spectrumfile=weights.txt
This section explains how to manually run the analysis steps and examine the changes that are made with different weight files.
This section of the thread uses the sherpa_contrib utilities package, which can be loaded into Sherpa using the following command:
sherpa> from sherpa_contrib.utils import *
Summary of the Syntax
An example set of commands used to create the weight file follows; please remember to change the data range, binning, spectral model and parameters to those appropriate for your data!
sherpa> from sherpa_contrib.utils import * sherpa> dataspace1d(0.5, 8, 0.25) sherpa> set_source(xsphabs.gal * powlaw1d.pl) sherpa> gal.nh = 1.3 sherpa> pl.gamma = 1.7 sherpa> save_instmap_weights("weights.txt") Created: weights.txt
The output file - here called weights.txt - can then be used as the input for the spectrumfile parameter of mkinstmap.
Create a model in Sherpa
Since the weights are largely independent of the responses, it is not necessary to use the PHA and ARF files to define a model. Instead, we can use the Sherpa dataspace1d command to create an energy grid from 0.5 to 8 keV with a bin width of 0.25 keV:
sherpa> dataspace1d(0.5, 8, 0.25)
Now we want to define a model that represents the source emission: we do not need to worry about the overall normalization of the model, just that the relative amplitudes of any emission components is correct.
For this example we are going to use a power law, with a gamma of 1.7, which is absorbed by a line-of-sight column density of 1.3 x 1022 atoms cm-2. The models and parameter values should be changed to match the emission from the source of interest.
sherpa> set_source(xsphabs.gal * powlaw1d.pl) sherpa> gal.nh = 1.3 sherpa> pl.gamma = 1.7 sherpa> plot_source(xlog=True, ylog=True)
The plot of the spectrum is shown in Figure 1:
Figure 1: Source spectrum
Plot the weights
sherpa> plot_instmap_weights(xlog=True, ylog=True)
Figure 2: Weights using a bin width of 0.25 keV
Save the weights
sherpa> save_instmap_weights("weights.txt") Created: weights.txt
unix% head -10 weights.txt #TEXT/SIMPLE # X WEIGHT # # DATE = 2019-11-20T16:18:20 / Date and time of file creation # MODELEXPR = (xsphabs.gal * powlaw1d.pl) # ENERG_LO = 0.5 / [keV] Minimum energy # ENERG_HI = 8.0 / [keV] Maximum energy # 0.625 0.0005505499378475431 0.875 0.005133913053133582 unix% tail -5 weights.txt 6.875 0.01554423141902547 7.125 0.014513042710459261 7.375 0.013586416882823383 7.625 0.012875827543709225 7.875 0.012220642621851999
unix% dmstat weights.txt sigma- X min: 0.625 @: 1 max: 7.875 @: 30 mean: 4.25 sum: 127.5 good: 30 null: 0 WEIGHT min: 0.00055054993785 @: 1 max: 0.074137428209 @: 6 mean: 0.033333333333 sum: 1 good: 30 null: 0
The output table gives the energy at the center of each band (in keV, X) and the fraction of the incident flux falling in that band (the spectral weighting, WEIGHT). This file may now be used to create a weighted instrument map; e.g. see the Compute an ACIS Exposure Map (Single Chip) and Build Fluxed Image thread.
In the following sections of the thread we show how you can change the binning used to create the weights and visualize the differences. Note that using a bin width of 0.25 keV is generally excessive, in that it over-resolves the energy grid, and so will lead to longer run times of mkinstmap than are strictly necessary.
If you wish to use a different energy range then you can call dataspace1d to create a new grid, or filter the grid by using the ignore and notice routines. In the following we create new datasets with different binning characteristics to investigate the differences these choices can make.
First we use the same energy range but double the bin width to 0.5 keV to create Figure 3:
sherpa> dataspace1d(0.5, 8, 0.5, id=2) sherpa> set_source(2, get_source()) sherpa> plot_instmap_weights(xlog=True, ylog=True) sherpa> plot_instmap_weights(2, overplot=True) sherpa> save_instmap_weights(2, "weights2.txt") Created: weights2.txt
Figure 3: A comparison of bin widths of 0.25 and 0.5 keV
We now decide to use variable-width bins, manually choosing energy ranges over which we consider the effective area to be approximately flat. These values are for demonstration purposes only and should not be considered to be valid choices for your analysis!
As the dataspace1d command only creates regularly-spaced grids, we use the load_arrays command to create the grid: the xlo*0 statement is used to create a fake data array, since we ignore the data values for this calculation. The result is shown in Figure 4.
sherpa> edges = [0.5, 1, 1.5, 2, 2.25, 2.5, 2.75, 3, 3.5, 4, 6, 8] sherpa> xe = np.asarray(edges) sherpa> xlo = xe[:-1] sherpa> xhi = xe[1:] sherpa> load_arrays(3, xlo, xhi, xlo*0, Data1DInt) sherpa> set_source(3, get_source()) sherpa> plot_instmap_weights(xlog=True, ylog=True) sherpa> plot_instmap_weights(2, overplot=True) sherpa> plot_instmap_weights(3, overplot=True) sherpa> save_instmap_weights(3, "weights3.txt") Created: weights3.txt
Figure 4: Using a variable bin width
It may be useful to plot the ARF for your source along with the weights, to check that the binning you have used is sensible. In the following we use a combination of "low-level" Sherpa methids and Matplotlib commands to plot the data from the file src.arf on top of the weights, creating Figure 5:
sherpa> import matplotlib.pyplot as plt sherpa> from sherpa.astro.plot import ARFPlot sherpa> arf = unpack_arf('src.arf') sherpa> aplot = ARFPlot() sherpa> splot.prepare(arf) sherpa> plot_instmap_weights(3) sherpa> ax2 = plt.twinx() sherpa> aplot.plot(overplot=True, color='orange') sherpa> ax2.set_ylabel('ARF cm$^2$') sherpa> ax2.set_yscale('log') sherpa> ax2.set_ylim(30, None) sherpa> plt.xlim(0, 9)
Figure 5: Comparing a set of weights to an ARF
How do the resulting instrument maps differ?
In this section we create instrument maps for the weights created above, and compare them to a mono-energetic version. All the instrument maps were created using the following set of parameters (taken from the single chip exposure map thread):
unix% punlearn mkinstmap unix% pset mkinstmap pixelgrid="1:1024:#1024,1:1024:#1024" unix% pset mkinstmap obsfile=repro/acisf01838_repro_evt2.fits unix% pset mkinstmap detsubsys=ACIS-7 unix% pset mkinstmap maskfile=repro/acisf01838_000N003_msk1.fits
In this example, since we are only interested in how the effective area varies with the weighting, we have not used acis_set_ardlib to ensure that the observation-specific bad-pixel file has been used.
The maps that were created are given below:
- Mono-energetic: imap.e1.7.fits
unix% mkinstmap imap.e1.7.fits NONE 1.7 mode=h
- 0.25 keV bin width: imap.weights.fits
unix% mkinstmap imap.weights.fits weights.txt 1 mode=h
- 0.5 keV bin width: imap.weights2.fits
unix% mkinstmap imap.weights2.fits weights2.txt 1 mode=h
- Variable bin width: imap.weights3.fits
unix% mkinstmap imap.weights3.fits weights3.txt 1 mode=h
On a reasonably modern Linux machine the run times for mkinstmap were approximately 3 s, 9 s, 6 s, and 5 s for these four maps. The actual run times depend strongly on operating system and disk access, so these numbers should only be taken as rough guides.
A quick comparison using dmstat shows that the three weighted maps are similar, but significantly different to the mono-energetic case:
unix% dmstat imap\*fits cen- med+ sig- File=imap.e1.7.fits IMAP[cm**2] min: 0 @: ( 1 1 ) max: 634.42694092 @: ( 212 432 ) mean: 593.81272372 median: 602.91555786 sum: 622657770.59 good: 1048576 null: 0 File=imap.weights.fits IMAP[cm**2] min: 0 @: ( 1 1 ) max: 415.99172974 @: ( 212 425 ) mean: 382.92987149 median: 389.11810303 sum: 401531072.93 good: 1048576 null: 0 File=imap.weights2.fits IMAP[cm**2] min: 0 @: ( 1 1 ) max: 407.64544678 @: ( 212 425 ) mean: 375.16585608 median: 381.27789307 sum: 393389912.71 good: 1048576 null: 0 File=imap.weights3.fits IMAP[cm**2] min: 0 @: ( 1 1 ) max: 407.20336914 @: ( 212 425 ) mean: 374.24775966 median: 380.35220337 sum: 392427218.84 good: 1048576 null: 0
Comparing the median values - shown in bold above - shows that the weighted maps are roughly 65 % of the mono-energetic value. Dividing imap.weights.fits by imap.e1.7.fits produces the image shown in Figure 6.
unix% punlearn dmimgcalc unix% dmimgcalc imap.weights.fits,imap.e1.7.fits out=ratio1.fits op=imgout=img1/img2 mode=h unix% dmstat ratio1.fits cen- sig- med+ ratio1.fits min: 0.60567945242 @: ( 594 624 ) max: 0.65803092718 @: ( 220 2 ) mean: 0.64471042245 median: 0.64539122581 sum: 666638.95804 good: 1034013 null: 14563
The use of imgout=img1/img2 for the operation parameter of dmimgcalc ensures that 0-pixel values get replaced by NaN values, and so will be ignored by dmstat and ds9. If we had used op=div and set both infile and infile2 then these pixels would have been set to 0 instead.
unix% sherpa -n sherpa> import matplotlib.pyplot as plt sherpa> img = read_file('ratio1.fits').get_image().values sherpa> fig = plt.figure(figsize=(8, 8)) sherpa> plt.imshow(img, origin='lower') sherpa> plt.axis('off') sherpa> plt.subplots_adjust(left=0, right=1, top=1, bottom=0) sherpa> cax = plt.axes([0.1, 0.875, 0.6, 0.05]) sherpa> cbar = plt.colorbar(cax=cax, orientation='horizontal')
Figure 6: Ratio of a weighted instrument map to a mono-energetic one
Figure 7: Ratio of 0.5 keV to 0.25 keV bin-width instrument maps
Figure 8: Ratio of variable to 0.25 keV bin-width instrument maps
How do the resulting exposure maps differ?
unix% punlearn mkexpmap unix% pset mkexpmap asphistf=7.asphist unix% pset mkexpmap xygrid="3643.1:4723.1:#1080,3853.4:4933.4:#1080" unix% pset mkexpmap normalize- unix% mkexpmap instmap=imap.e1.7.fits outf=emap.e1.7.fits mode=h unix% mkexpmap instmap=imap.weights.fits outf=emap.weights.fits mode=h unix% dmimgcalc emap.weights.fits,emap.e1.7.fits outf=emap.ratio.fits op=imgout=img1/img2 mode=h
Figure 9: Ratio of 0.25 keV bin-width to mono-chromatic exposure maps
An alternative view is the histogram of the pixel values in the various ratio images, as shown in Figure 10:
Figure 10: Histograms of the ratio images
To start with, we read the image data using the Crates routine read_file, and access the pixel-data directly. We then filter the image values to exclude those pixels with values of 0, and use the filtered set of pixels in the call to simple_hist.
import numpy as np import matplotlib.pyplot as plt from pycrates import read_file from ciao_contrib.utils import simple_hist i1 = read_file('ratio1.fits').get_image().values i2 = read_file('ratio2.fits').get_image().values i3 = read_file('ratio3.fits').get_image().values ie = read_file('emap.ratio.fits').get_image().values ix1 = np.where(i1 > 0) ix2 = np.where(i2 > 0) ix3 = np.where(i3 > 0) ixe = np.where(ie > 0) h1 = simple_hist(i1[ix1], nbins=20) h2 = simple_hist(i2[ix2], nbins=20) h3 = simple_hist(i3[ix3], nbins=20) he = simple_hist(ie[ixe], nbins=20)
The data can then be plotted (using the mid-point of each bin, and normalized by the total number of points), in two horizontally-aligned plots:
def hplot(ax, h, lbl): ax.plot((h.xlow + h.xhigh) / 2, h.y / h.y.sum(), label=lbl) fig, (ax1, ax2) = plt.subplots(1, 2, sharey='row', figsize=(6, 4)) plt.subplots_adjust(wspace=0.05) hplot(ax1, h1, 'ratio1.fits') hplot(ax1, he, 'emap.ratio.fits') ax1.legend(loc='upper left') ax1.set_ylim(-0.005, 0.17) ax1.set_ylabel('Fraction of pixels') hplot(ax2, h2, 'ratio2.fits') hplot(ax2, h3, 'ratio3.fits') ax2.legend(loc='upper left') plt.text(0.5, 0.02, 'Pixel Value', horizontalalignment='center', transform=fig.transFigure)
The comparisons above were all made with the on-axis chip; larger differences may be seen between different weighting schemes for ACIS chips that are further off axis, and so are subject to stronger, energy-dependent, vignetting.
Notes and Caveats
The An Introduction to Exposure Maps document provides background reading on why weighted exposure maps are important, and under what conditions they produce meaningful results.
If you wish to make several exposure maps for different bands (e.g. 1-2 keV, 2-4 keV, and 4-10 keV), rerun the script for each of these bands. Any time you use the weighting scheme, the sum of the weights should be equal to one. If one (incorrectly) makes a single set of weights with this script and then subdivides them to weight different bands, the weights of each instrument map would not sum to one. Thus, the instrument and exposure maps that are derived would have effective areas that are much too small.
To compute an image which gives the integrated flux over the full energy range, it may be best to first compute flux-corrected images in several narrow energy bands (where the ARF is nearly flat) and then sum those fluxed images together. Weighted exposure maps work well for an energy band where the ARF variation isn't very large, but for a full-band 0.5-10 keV image, it may not be a good idea to compute the flux by dividing the counts image by a single number. This is especially true for cases where the source spectrum varies significantly within the image; in that case, there is no general way to compute a single set of weights which will be sensible for every part of the image.
The same spectral weights used by mkinstmap can be used with mkpsfmap to produce a spectrally weighted PSF map.
|03 Aug 2009||Updated for CIAO 4.1: spectrum.sl has been replaced by the sherpa_contrib.utils module; see the Changes from CIAO 3.4 section; a comparison of weighted to mono-energetic instrument maps has been added.|
|09 Feb 2010||updated for CIAO 4.2: the "get_imagevals" command has been renamed to get_piximgvals|
|12 Mar 2010||Added an example of the essential commands to the synopsis section at the start of the thread, for those users who just need reminding of the syntax; updated some of the figures to use ChIPS for display rather than ds9.|
|19 Jul 2010||the S-Lang syntax has been removed from this thread as it is not supported in CIAO 4.2 Sherpa v2. Contact the CXC Helpdesk if you need assistance transitioning from S-Lang to Python.|
|13 Jan 2011||reviewed for CIAO 4.3: no changes|
|21 Nov 2011||updated the module name in the import statement from sherpa_contrib to sherpa_contrib.utils|
|24 Jan 2012||reviewed for CIAO 4.4: added a Using the make_instmap_weights script section; separated the in-depth analysis into a "step-by-step" section|
|03 Dec 2012||Review for CIAO 4.5; same spectral weights can be used by mkpsfmap as part of the wavdetect thread.|
|02 Dec 2013||Review for CIAO 4.6: the pbkfile parameter of mkinstmap is no-longer used - with the obsfile parameter being set to the event file rather than the aspect histogram - and the examples have been updated to use CALDB 4.5.9.|
|22 Dec 2014||Review for CIAO 4.7; no changes.|
|21 Nov 2019||Updated for CIAO 4.12 to use matplotlib rather than ChIPS for plotting, and to update the figures to use more-recent calibration products.|