Chandra X-Ray Observatory
	(CXC)
Skip to the navigation links
Last modified: 16 December 2010

URL: http://cxc.harvard.edu/sherpa/updates.html

Latest Updates


This document highlights important changes and additions to Sherpa functionality in the CIAO 4.3 Sherpa release.

To compare the features and functionality of Sherpa in CIAO 4.3 to Sherpa in CIAO 3.4, refer to the pages About Sherpa and Sherpa 3.4 to 4.3 Conversion.

CIAO 4.3 Sherpa Updates to Sherpa Functionality


Startup Options

Removal of S-lang Support

In CIAO 4.3, the S-Lang interface to Sherpa is not supported. There is no longer any means of importing Sherpa as a S-Lang module, as the S-Lang version of the Sherpa high-level UI has been removed.

  • It is no longer possible to do 'require("sherpa");' in slsh, or any other S-Lang application.
  • Sherpa no longer reads the environment variables CIAO_SCRIPT_LANG or SHERPA_SCRIPT_LANG, and will issue error messages if these variables are set to "slang".
  • There is no longer an option to use slsh as the interpreter in the Sherpa application script. The "-l" command-line option has been removed. The following are the remaining command-line options which may be specified when Sherpa is started:
% sherpa -h
-- sherpa
  usage: sherpa [-x] [-n] [-b] [-rcfile file] [-norcfile] <file>
     -x       launch sherpa shell in separate display terminal
     -n       do not print banner
     -b       batch mode
     -rcfile  load a user specifiable preference file
     -norcfile do not load any user specifiable preference file (overrides -rcfile)
     <file>   appropriate python command file to execute
.sherpa.rc
  • The option "shell" is no longer needed in the user's .sherpa.rc file. If this is set to "slang", the application will exit with an error:
  
Sherpa ERROR: Sherpa no longer offers a S-Lang interface.
The 'shell' option in /home/username/.sherpa.rc is no longer used. 
  • The new section [parallel] in the .sherpa.rc file includes a switch "numcores"; more information is available in the "Parallelization" section, below. The default value for "numcores" is None. To specify the number of cores to use for all Sherpa concurrent functions, change the default value:
     [parallel]
     numcores : 3
   
Parallelization

Several computationally intensive functions, such as projection and conf, were parallelized in the CIAO 4.2 release, to make use of multi-core systems (i.e., laptops or desktops with 2 or 4 cores). In Sherpa in CIAO 4.3, a new option, "numcores" has been added to the following functions to specify the number of cores to use in parallelization. The default is to use all cores available.


Loading, Filtering, and Viewing Data

The new function get_bkg_scale returns the coefficient which is used to scale background counts during background subtraction of a source spectrum, or a simultaneous fit of source and background spectra. The complete scaling factor used to scale the background counts in these cases consists of the product of the source-to-background exposure and extraction region area (backscal) ratios (the backscal value may be retrieved with the get_backscal function). The complete background scale is returned by the show_data and show_all functions.

The load_filter function has been updated to include the ability to read FITS images that hold filter information, e.g., an image file containing 1s and 0s to mark which pixels should be filtered. There is a new keyword argument 'ignore' that indicates whether the filter information should be used to notice or ignore data points. "ignore" is False by default.

sherpa> load_filter(id=1, filename, bkg_id=None, ignore=False, ncols=2)
sherpa> set_filter(id=1, val, bkg_id=None, ignore=False)

The plot_source command now supports the 'factor' setting of set_analysis. This means that calling plot_source while the set_analysis 'factor' setting is '1' will plot E F(E) versus E in keV (XSPEC plot eufspec), or λ f(λ) versus λ in Angstroms, depending on which units are set for the spectral analysis. A set_analysis 'factor=2' setting will plot E2 F(E) versus E, or λ2 f(λ) versus λ (XSPEC plot eeufspec).

There are new plotting functions to plot individual model components contributing to the fit of a multi-component model to data. Users can pass one or more Sherpa model components to the plotting function to quickly view the contribution to the model. Similarly, users can view 2D model components in DS9.

The following new functions set the x and y axis scales for Sherpa plots:

These functions accept arguments similar to the generic plot command: "data", "model", "source", "delchi", etc.


Models

Sherpa models now have a "cache" attribute for users to control behavior. The default value for "cache" is a non-zero value indicating that caching will be turned on only if all parameters in the model are frozen. Indicating zero will turn off caching even if all model parameters are frozen.

Model caching is now available for XSPEC and 1D analytic models; caching is turned on by default. 2D Sherpa analytical models are not cached by default due to the potential impact on memory usage. Compared to fit results in previous releases, there should be no change to any calculated value. The only difference that may be seen is in reduced program execution time.

sherpa> set_source( xsphabs.abs1 * powlaw1d.p1 )
sherpa> thaw(abs1)
sherpa> abs1.cache=0

The new 1D integration helper function, "integrate1d", can be used to define 1D numerical integration on a particular arbitrary Sherpa model expression. This function is used inside a Sherpa model definition as an explicit indication that Sherpa should numerically integrate the expression in parenthesis. The helper function numerically integrates the expression as a whole, correctly, instead of each component individually. Note that the model expression should not include any XSPEC additive models, as those models perform integration on themselves.

sherpa> set_model(integrate1d.int1(beta1d.b1*gauss1d.g1))
sherpa> print int1
    integrate1d.int1
    Param Type Value Min Max Units
    ----- ---- ----- --- --- -----
    int1.epsabs frozen 2.22045e-16 -3.40282e+38 3.40282e+38
    int1.epsrel frozen 0 -3.40282e+38 3.40282e+38
    int1.maxeval frozen 10000 -3.40282e+38 3.40282e+38

This helper function is used in lieu of the "integrate" flag found on Sherpa models.

New Models

The following models are now available in Sherpa:

The scale1d model is simply the const1d model with the integrate flag turned off by default. If scale1d is set as a model for fitting an integrated data set, it will behave as a simple constant by default.

The logparabola model is defined by a log-parabolic function of the following form:

f(x) = A (x/x_ref)[-gamma - beta*log10(x/x_ref)]
   

New XSpec models included in Sherpa are:

  • xssirf

The new XSPEC functions set_xsxset and get_xsxset() are associated with the XSPEC function XSET for defining model environment variables.

sherpa> set_xsxset("NEIVERS", "2.0")
sherpa> set_xsxset("NEIAPECROOT", "/home/brefsdal/apec")
sherpa> set_model(xsvnei.nn)

sherpa> get_xsxset("NEIAPECROOT")
sherpa> '/home/user/apec'
PSF Instrument Models

2D PSF models are no longer required to have equal (x,y) size parameters; rectangular PSFs are now supported. It is also now possible to simultaneously fit multiple data sets with each independent source model convolved by a different PSF model.

Table Models

Additive and multiplicative XSPEC-style table models are now supported by the Sherpa table model using load_table_model, e.g.:

sherpa> load_table_model("tab1", "bhmodel.fits")
sherpa> set_model(tab1)

A table model contained in a file written in the format required by the XSpec table model, i.e. a file containing an N-dimensional grid of model spectra, can be loaded and set as a source model in Sherpa.

The Sherpa table model has also been updated to support linear interpolation of data points on the data set grid from the model grid supplied from file. The grids need not be of constant or comparable bin size, and it is no longer necessary to match the length of the table model column to that of the data. If the table model grid is not sorted, Sherpa will sort it in ascending order.

User Models

The new function add_model is available as an alternative to using load_user_model together with add_user_pars for explicitly defining user models as Python classes for use in Sherpa. The add_model command registers a user-defined Sherpa model class as a Sherpa model type, to allow users to create instances of models that are recognized automatically in Sherpa model expressions.

      
sherpa> from sherpa.models import PowLaw1D
sherpa> class MyPowLaw1D(PowLaw1D): pass
sherpa> add_model(MyPowLaw1D)
sherpa> set_model(xswabs.abs1*mypowlaw1d.p1)
Definining Model Expressions

Sherpa now allows the user to define model expressions that apply response matrices, or PSFs, to some models, while not applying the response or the PSF to the rest of the model expression. An example of this kind of model is an expression where a spectral model is defined in energy space, and folded through a response matrix; then, a background model defined in counts, which is not folded through the response, is added to the model expression.

The new functions set_full_model() and set_bkg_full_model() allow users to explicitly define instrument responses and convolutions that are applied to specified model components. PSF and table models may be used in model expressions defined with these functions.

Legacy functionality is still supported with set_source() and set_model(); CIAO 4.3 scripts using these functions will continue to work in the current Sherpa.

   Automatic            Manual Definition
   =============        =================
   set_source()         set_full_model()
   set_model()

   set_bkg_source()     set_bkg_full_model()
   set_bkg_model()
   
Instrument Responses

The new function get_response returns the instrument response model associated with a PHA data set, which may be used together with the set_full_model() and set_bkg_full_model() functions for explicitly defining complex convolved model expressions.

The get_arf and get_rmf functions have been updated with the ability to return the 'instrument' version of an ARF and RMF data set, allowing the output of these functions to be used like that of the get_response function for explicitly defining convolved model expressions. The multiplication with the exposure time is implicit in such a model expression. The following two sets of commands produce equivalent results.

sherpa> arf = get_arf()
sherpa> rmf = get_rmf()
sherpa> set_full_model(rmf(arf(xsphabs.abs1*powlaw1d.p1)))

sherpa> rsp = get_response()
sherpa> set_full_model(rsp(xsphabs.abs1*powlaw1d.p1))
Fitting

The Primini and sigma-rejection iterative fitting methods which were available in Sherpa 3.4 are now available in Sherpa 4. Several new functions are available to allow users to set the iterative fitting method, to find out what the current iterative fitting method is, and to get and set options for this method. These functions are:

If the iterative fitting method is "none" (the default value), then no iterative fitting is done - when "fit()" is called, the optimization method is called once, and Sherpa otherwise operates as expected. The fit statistic and optimization methods are selected independently of the iterative fitting method

Primini's method is used for re-calculating statistical errors using the best-fit model parameters from the previous fit, until the fit can no longer be improved.

The sigma-rejection method is based on the IRAF SFIT function. In successive fits, data points for which ((data - model) / error) exceeds some threshold are added to the filter, and automatically excluded from the next fit.

Primini's method and sigma-rejection can only be called when the fit statistic is a χ2 statistic; it cannot be used with least-squares, Cash or C-statistic.

The new functions calc_stat_info and get_stat_info implement functionality similar to the GOODNESS command from Sherpa 3.4. Goodness-of-fit information usually found in a Sherpa fit result is now also calculated separately. It is no longer necessary to perform a new fit just to return goodness-of-fit information.


Additional Enhancements & Bug Fixes

Parallelization

In Sherpa in CIAO 4.3, users will now see that when Ctrl-C is issued while proj() or conf() is running, Sherpa will correctly kill off slave processes from any parallel evaluation. Also, parallel evaluation of proj() and conf() is now automatically turned off on single core machines.

Models

New arguments have been added to the list_models command; valid arguments are "all", "xspec", "1d", and "2d".

The Sherpa powlaw1d power law model no longer suffers from a numerical instability with a gamma setting close to 1, while the integrate flag is switched on.

XSPEC model 'refflag' parameters are now always frozen.

The XSPEC function set_xsabund is updated to accept user-defined files containing solar abundances.

The atten model now correctly ignores the integrate setting.

The error messages which are printed during model evaluation when arrays are different sizes, have been improved.

Data Display

The plot functions int_proj and int_unc include dashed lines indicating the best fit statistic and best fit parameter value.

The plot_bkg_unconvolved command has been removed; the identical plot_bkg_source command should be used, instead.

The show_all command now correctly displays information on background data sets loaded into the Sherpa session.

Sherpa now uses a private DS9 window for communication, with ID 'sherpa' for session imaging.

Axis units for plots created with the contour_* functions are now correctly returned in pixels when the coordinate setting for the session is 'physical'.

When the statistical errors resulting from a fit are calculated by the chosen fit statistic and the systematic errors are set by the user, the error bars are now correctly displayed on plots of the fit.

The get_arf function no longer erroneously returns the ARF for data set 1, when a data set other than 1 is explicitly entered as the argument.

The show_bkg_source command now correctly recognizes a specified bkg_id value

Data Properties

For grating data sets, set_analysis("wave") now correctly sets both the source and background to wavelength units.


Saving Data

The save function now includes XSPEC module settings in the saved binary file. These settings are the abundance, chatter level, cosmology, electron cross-section, and model variables set via the XSPEC "xsxset" function.

The restore function reads these settings back in and calls the appropriate XSPEC module functions to reset the values.

The save_all function, which saves attributes of a Sherpa session to a human-readable text file, has been modified. The text file created by save_all now includes:

  • The iterative fitting method, if that has been set;
  • Source model assignment via either set_source or set_full_model, as appropriate;
  • XSPEC module settings for abundances, chatter level, cosmology, electron cross-section, and model settings via the "xsxset" function.

Text files created with the CIAO 4.2 version of save_all can still be used in the CIAO 4.3 version of Sherpa.


Modules

The utils module include a new utility function, parallel_map(), which is a parallelized version of the native Python map which evaluates a function over a list of arguments.

sherpa> from sherpa.utils import parallel_map
sherpa> parallel_map(lambda x: x*x, [1,2,3]) -> [1,4,9]

The follow statistical PDF functions for multivariate normal and multivariate student T distributions have also been added to Sherpa utils module.

sherpa> dof = get_fit_results().dof
sherpa> sigma = get_covar_results().extra_output
sherpa> mu = get_model().thawedpars
sherpa> x = numpy.random.multivariate_normal(mu, sigma)
sherpa> from sherpa.utils import multinormal_pdf, multit_pdf
sherpa> t_pdf = multit_pdf(x, mu, sigma, dof)
sherpa> normal_pdf = multinormal_pdf(x, mu, sigma)

The new vectorized function "interpolate" may be used as follows:

      sherpa> from sherpa.utils import interpolate
      sherpa> yout = interpolate(xout, xin, yin, method='linear'|'nearest')

Last modified: 16 December 2010
Smithsonian Institute Smithsonian Institute

The Chandra X-Ray Center (CXC) is operated for NASA by the Smithsonian Astrophysical Observatory. 60 Garden Street, Cambridge, MA 02138 USA.   Email:   cxchelp@head.cfa.harvard.edu Smithsonian Institution, Copyright © 1998-2019. All rights reserved.