Why is there data being plotted or fit outside the energy range I noticed?

In the following example, we notice energies in the 0.5 to 8.0 keV range, but when the we plot the data, there is a data point above 10 keV. Why?

In CIAO 4.17 the commands like group_counts now report the selected data range. This follows the changes in CIAO 4.16 to report the selected data range with commands like ignore and notice.

sherpa> load_pha("9774.pi")
read ARF file 9774.arf
read RMF file 9774.rmf
read background file 9774_bg.pi

sherpa> group_counts(20)
dataset 1: 0.00146:14.9504 Energy (keV) (unchanged)

sherpa> notice(0.4, 8.0)
dataset 1: 0.00146:14.9504 -> 0.3796:14.9504 Energy (keV)

For this example we shall change the plot defaults to add a line to show the bin edges (the "linestyle" attribute) and to hide the Y error bars ("yerrorbars") using the new-to-CIAO 4.16 get_plot_prefs commmand:

sherpa> prefs = get_plot_prefs("data")
sherpa> prefs["linestyle"] = "solid"
sherpa> prefs["yerrorbars"] = False
sherpa> plot_data(xlog=True)

Figure 1: unexpected data

Figure 1: unexpected data

The position of the start and end points (0.4 and 8 keV) have been marked on the plot as vertical black (partially opaque) lines with the matplotlib commands

sherpa> plt.axvline(0.4, c='k', alpha=0.5)
sherpa> plt.axvline(8, c='k', alpha=0.5)

These two lines fall within the first and last bins.

As shown in the plot, the issue here is that the last grouped bin starts below 8 keV but ends at ~ 15 keV. This also happens at the low end too, since the group that contains 0.4 keV starts at 0.38 keV (unless the source is obscurred or very hard then the low-energy groups will likely be narrower, and hence less of an issue).

It is suggested that any filtering is done before the data is grouped, since (as of CIAO 4.16) the group_counts, group_snr, and other variants will now default to only grouping the noticed data range. This means that the suggested approach is

sherpa> load_pha("9774.pi")
read ARF file 9774.arf
read RMF file 9774.rmf
read background file 9774_bg.pi

sherpa> notice(0.4, 8.0)
dataset 1: 0.00146:14.9504 -> 0.3942:8.0008 Energy (keV)

sherpa> group_counts(20)
dataset 1: 0.3942:8.0008 Energy (keV) (unchanged)

Note

Prior to CIAO 4.16 we suggested that users of the group_xxx commands like group_counts should use the tabStops argument of these routines. This is now done automatically for you. Setting tabStops="nofilter" will ignore the noticed data range, re-creating the pre-4.16 behaviour.

The screen output already shows that the noticed data range is now closer to the requested range, and we can see it in Figure 2.

Figure 2: Controlling the grouped data range

Figure 2: Controlling the grouped data range

Warning

There is no guarantee that each group will pass the grouping requirements. In this case, the last group is unlikely to contain at least 20 counts, but the location of problematic groups depends on the particular method used and the quality of the data.

Unfortunately in CIAO 4.16 there is no simple way to show the quality values. The get_quality routine will return the quality values of all the channels in the PHA data set (this returns the ungrouped data) and values of 2 indicate those channels where the grouping did not meet the requirements. The ignore_bad call will exclude those "bad" channels but unfortunately has the side-effect of removing any existing filter and causing problems with other parts of CIAO.

More information

A full discussion of spectral file formats and conventions is beyond the scope of this FAQ. Interested readers can review the documents "The OGIP Spectral File Format" and "The Calibration Requirements for Spectral Analysis"