_images/sherpa_logo.gif

Sherpa: Modeling and Fitting in Python

Sherpa is a modeling and fitting application for Python. It contains a powerful language for combining simple models into complex expressions that can be fit to the data using a variety of statistics and optimization methods. It is easily extensible to include user models, statistics and optimization methods.

What can you do with Sherpa?

  • fit 1D (multiple) data including: spectra, surface brightness profiles, light curves, general ASCII arrays
  • fit 2D images/surfaces in Poisson/Gaussian regime
  • build complex model expressions
  • import and use your own models
  • use appropriate statistics for modeling Poisson or Gaussian data
  • import the new statistics, with priors if required by analysis
  • visualize a parameter space with simulations or using 1D/2D cuts of the parameter space
  • calculate confidence levels on the best fit model parameters
  • choose a robust optimization method for the fit: Levenberg-Marquardt, Nelder-Mead Simplex or Monte Carlo/Differential Evolution.

To install Sherpa see Sections Source and Binary Installation. For an example of a Sherpa session check Run Sherpa section. More examples are given in a few Sherpa IPython Notebooks:

Sherpa Quick Start

Image Fitting

Template Fitting

A Simple Bayes Analysis of an X-ray Spectrum

For detailed documentation see: http://cxc.harvard.edu/sherpa

Install Sherpa

Sherpa can be installed from a binary distribution or built from sources.

The binary distribution is suited for people wanting to have Sherpa up and running as soon as possible in its standard form. The binaries are built and tested on Linux 32, Linux 64 and Mac OSX (>=10.8)

Source installation is available for platforms incompatible with the binary builds, or for users wanting to customize the way Sherpa is built and installed.

Sherpa 4.8.0 is known not to work with matplotlib version 1.5, but does with earlier versions.

Binary Install

The binary installation of Sherpa 4.8.0 was released on January 26, 2016. It has been tested on Linux 32, Linux 64 and Mac OSX (versions > 10.8)

Sherpa binaries can be seamlessly installed into Anaconda Python. You need to add Sherpa channel to your configuration, and then install Sherpa:

$ conda config --add channels https://conda.binstar.org/sherpa
$ conda install sherpa

If matplotlib is to be used with Sherpa, the constraint ‘matplotlib < 1.5’ must be used: for example:

$ conda install sherpa 'matplotlib < 1.5'

To test that your installation works type:

$ pip install pytest-cov
$ sherpa_test

To update Sherpa:

$ conda update sherpa

If you do not have Anaconda Python and want to create the minimum environment for trying Sherpa check the README on GitHub.

Configuration Files

Sherpa expects a user configuration file sherpa-standalone.rc to be in the $HOME directory. If this file is not present Sherpa will use the default internal configuration file with the IO and Plotting back-ends set to pyfits and pylab.

matplotlib comes with a configuration file matplotlibrc. For smooth behavior with Sherpa, be sure to indicate interactive=True in ~/.matplotlib/matplotlibrc.

Source Installation

It takes only a few simple steps to build and install Sherpa in any Python 2.7 environment on Linux or Mac OSX. For example Anaconda Python Distribution contains many scientific software components needed for the analysis and Sherpa fits seamlessly into that environment.

The Sherpa source code is available on GitHub: https://github.com/sherpa/sherpa

Download the Sherpa source tar or zip file: sherpa-4.8.0.zip or sherpa-4.8.0.tar.gz

Unpack the files:

$ unzip sherpa-4.8.0.zip
or
$ tar xvf sherpa-4.8.0.tar.gz

To install:

$ cd sherpa
$ python setup.py install

The Sherpa code will be built and installed in the directory ${prefix}/lib/python2.7/site-packages/sherpa, where ${prefix} can be determined with:

$ python -c 'import sys; print(sys.prefix)'

Sherpa requires the standard Python packages and system compilers for the build:

Python: setuptools, numpy
System: gcc, g++, gfortran, make, flex, bison

In addition I/O requires ‘astropy’ (replaces pyfits) for reading FITS files and matplotlib for standard plotting, ds9 for imaging.

After the installation you can run the test to check if the installation was successful. This requires to install ‘pytest-cov’ first:

$ pip install pytest-cov
$ sherpa_test

Note that the ${prefix}/bin should be in the $PATH to run the test. The tests should succeed, but there could be two expected warnings if ‘ds9’ or/and ‘XSPEC’ models are not found. These are not necessary for the use of Sherpa and only needed if you plan to perform X-ray analysis or use ‘ds9’ to display images.

Copy the source code repository

If you want to look at the code, use the code and maybe develop your own code in Git you can ‘clone’ the full repository structure with the default ‘git clone’:

$ git clone https://github.com/sherpa/sherpa.git

This would also be useful if you want to contribute your own code to Sherpa via ‘pull request’. The contributions are welcome. We advise to follow the astropy guidelines for developing your contributions. We require the code to have a description of the code, unit and integration test and a user documentation.

Custom Source Build

The costumization of the source build could be made by updating updating the setup.cfg file in ‘sherpa’ directory. For example to use the fftw local library one needs to set the following configuration options:

fftw=local
fftw-include-dirs=/usr/local/include
fftw-lib-dirs=/usr/local/lib
fftw-libraries=fftw3

where /usr/local path may need to be changed to the local directories with the header (.h) files and the libfftw3.so shared object file.

It can also allow you to build XSPEC models.

XSPEC Model Library

Sherpa does not support XSPEC models for X-ray spectral analysis by default. However, it is possible to instruct Sherpa to build its XSPEC extension module by changing the build configuration options.

You may need to build XSPEC yourself, and in any case to point Sherpa to existing binary libraries for XSPEC, cfitsio, and CCfits. Additionaly, you will need to point Sherpa to the libgfortran shared library. These dependencies are required to build XSPEC itself, so they are assumed to be present on the system where you want to build Sherpa with XSPEC support. Here we assume that XSPEC has been already build.:

$ git clone https://github.com/sherpa/sherpa.git
$ cd sherpa

Edit the setup.cfg file. Find the XSPEC configuration section in the file, uncomment the relative options and make sure they point to the location of the XSPEC, cfitsio, CCfits, and gfortran libraries. For instance:

with-xspec=True
xspec_lib_dirs=/opt/xspec/lib
cfitsio_lib_dirs=/usr/local/lib
ccfits_lib_dirs=/usr/local/lib
gfortran_lib_dirs=/usr/local/lib

You may need to change the values in the above example to reflect the actual directories where the libraries are to be found on your system.

Then, build Sherpa in the standard way:

$ python setup.py install

Note that XSPEC needs a set of libraries to run:

libCCfits.0.0.0.dylib
libCCfits.la.gin
libXSModel.dylib
libcfitsio.a
libCCfits.0.dylib
libCCfits.la.org
libXSUtil.dylib
libcfitsio.dylib
libCCfits.dylib
libXS.a
libcfitsio.2.3.37.dylib
libCCfits.la
libXSFunctions.dylib
libcfitsio.2.dylib

Their location can be specified in DYLD_LIRBARY_PATH (on Mac OSX) or LD_LIBRARY_PATH (on Linux). For example::

$ export DYLD_LIBRARY_PATH=$HOME/xspeclib

where $HOME/xspeclib is the directory with all the required libraries.

Run Sherpa

You can import Sherpa into your ipython session:

(conda)$  ipython --pylab
Python 2.7.9 |Anaconda 2.2.0 (x86_64)| (default, Dec 15 2014, 10:37:34)
Type "copyright", "credits" or "license" for more information.

IPython 3.0.0 -- An enhanced Interactive Python.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
Using matplotlib backend: MacOSX

In [1]: from sherpa.astro.ui import *
WARNING: imaging routines will not be available,
failed to import sherpa.image.ds9_backend due to
'RuntimeErr: DS9Win unusable: Could not find ds9 on your PATH'

The standard warnings are issued if you do not have ds9 models in your path. The image with ds9 will not be available. See the Dependencies section below.

Now to simulate a simple shape (a parabola with errors):

In [2]: x = np.arange(-5, 5.1)

In [3]: y = x*x + 23.2 + np.random.normal(size=x.size)

In [4]: e = np.ones(x.size)

The data can now be loaded into Sherpa:

In [5]: load_arrays(1, x, y, e)

In [6]: plot_data()
_images/data.png

For this example we know what model to use, so pick a polynomial and free-up some of the parameters:

In [7]: set_source(polynom1d.poly)

In [8]: print(poly)
polynom1d.poly
   Param        Type          Value          Min          Max      Units
   -----        ----          -----          ---          ---      -----
   poly.c0      thawed            1 -3.40282e+38  3.40282e+38
   poly.c1      frozen            0 -3.40282e+38  3.40282e+38
   poly.c2      frozen            0 -3.40282e+38  3.40282e+38
   poly.c3      frozen            0 -3.40282e+38  3.40282e+38
   poly.c4      frozen            0 -3.40282e+38  3.40282e+38
   poly.c5      frozen            0 -3.40282e+38  3.40282e+38
   poly.c6      frozen            0 -3.40282e+38  3.40282e+38
   poly.c7      frozen            0 -3.40282e+38  3.40282e+38
   poly.c8      frozen            0 -3.40282e+38  3.40282e+38
   poly.offset  frozen            0 -3.40282e+38  3.40282e+38

In [9]: thaw(poly.c1, poly.c2)

With everything set up, the data can be fit using the standard optimization method levmar and chi2 statistics:

In [10]: fit()
Dataset               = 1
Method                = levmar
Statistic             = chi2
Initial fit statistic = 12190
Final fit statistic   = 5.40663 at function evaluation 8
Data points           = 11
Degrees of freedom    = 8
Probability [Q-value] = 0.713361
Reduced statistic     = 0.675829
Change in statistic   = 12184.6
   poly.c0        22.2341
   poly.c1        0.109262
   poly.c2        1.06812

In [11]: plot_fit_resid()
_images/fit.png

and an estimate of 1 sigma parameters uncertainties is:

In [12]: conf()
poly.c0 lower bound:        -0.455477
poly.c1 lower bound:        -0.0953463
poly.c0 upper bound:        0.455477
poly.c2 lower bound:        -0.0341394
poly.c1 upper bound:        0.0953463
poly.c2 upper bound:        0.0341394
Dataset               = 1
Confidence Method     = confidence
Iterative Fit Method  = None
Fitting Method        = levmar
Statistic             = chi2gehrels
confidence 1-sigma (68.2689%) bounds:
   Param            Best-Fit  Lower Bound  Upper Bound
   -----            --------  -----------  -----------
   poly.c0           22.2341    -0.455477     0.455477
   poly.c1          0.109262   -0.0953463    0.0953463
   poly.c2           1.06812   -0.0341394    0.0341394

Dependencies

Data I/O support and plotting need astropy (replaces pyFITS) and matplotlib. Imaging requires ds9/XPA.

[astropy]http://www.astropy.org/
[mpl]Hunter, JD (2007). Matplotlib: A 2D graphics environment. Computing in Science and Engineering. 9: 90-95. http://matplotlib.sourceforge.net.
[ds9]http://hea-www.harvard.edu/RD/ds9/
[xpa]http://hea-www.harvard.edu/saord/xpa/

Release Notes Sherpa 4.8.0 for Python

January 26, 2016

The Sherpa 4.8.0 code released on January 26, 2016 is the same as the CIAO 4.8 version. There are, however, some updates required by the standalone Sherpa build. The listing below includes some major updates and specific pull requests with the numbering from GitHub:

  • introduces wstat statistics which is an extension of cstat with Poisson background data. It also provides the ability to include background data with the user statistics.
  • XSpec models upgrade to support version 12.9.0d. The XSpec extension has been redesigned to be more robust.
  • Some classes from the template model which were not exposed by __all__ are now are accessible.
  • clean up of deprecation warnings from Numpy version 1.9.
  • compiler warning clean up
  • updates to be compliant with Python’s PEP-8 style guide
  • various pull requests including:

#32: Fix SegFault from CRATES update in 4.8b1. For CIAO4.8 CRATES has been modified to return variable length arrays rather than the zero-padded fixed length arrays. Sherpa continues to utilize the zero padded fixed length arrays.

#44: save_quality now correctly outputs QUALITY as the column name instead of GROUPS.

#48: Fix up several issues seen in plot labels - titles and Y-axis labels for commands such as sherpa.ui.plot_data, sherpa.ui.plot_fit_resid, and sherpa.ui.plot_chisqr.

#59: Fix bug #38 (grouping twice gives an IndexError exception). An unhandled corner case in one of the Sherpa internal methods (utils.create_expr) was triggering an IndexError when two group_counts operations were performed back to back. The fix handles the case so that applying group_counts twice does not result in an Exception.

#77: Replace == and != comparisons to None with checks using is and is not.

#78: OutOfBoundErr exceptions in some sherpa.utils functions are properly caught. There were several places where screen output used either print or sys.stderr.write.

#81: Ensure that XSpec models which fail (ie. due to missing a needed data file) return 0’s for all bins, rather than random values.

#82: The XSpec “spectrum number” value is now set to 1 rather than 0.

#83: Removed S-Lang scripts, files, and references in the code.

#84: Clarified error messages in Xspec extension. Also, changed the class of the exception from RuntimeError to more appropriate exception types, in particular LookupError, ValueError, KeyError. This is a backwards incompatible change- code which previously caught theRuntimeError will not catch the new error.

#87: Some methods in sherpa/fit.py assigned mutable objects to default arguments. This has now been fixed. More instances of this issue have been identified (Bug #95) and will removed in the future.

#90: Added background data to the UserStat class.

#94: Implement wstat statistic as described at the following url: https://heasarc.gsfc.nasa.gov/xanadu/xspec/manual/XSappendixStatistics.html

#96: Remove the unused myoptfct module.

#99: Correct the documentation for the set_exposure function.

#100: Fix bug #97, whereby the ‘save_all’ function would not create the necessary set_source() line. This does not fix all cases, but it does the simple PHA case such as:

load_pha('src.pi')
set_source(xsphabs.gal * powlaw1d.pl)
save_all('test.out).

It also ensures that files created by save_all can be run using IPython’s ‘%run’ directive, by explicitly importing the sherpa.astro.ui module.

#101: Fix handling of non-contiguous energy grid in XSpec models.

#110: Update the sherpa.astro.datastack module documentation to include information from the CIAO ahelp documentation and to match the style used by the sherpa.astro.ui module.

#111: Update the documentation to include more information about pyBLoCXS (MCMC).

Caveats

Incompatibility with matplotlib 1.5: Sherpa 4.8.0 is not compatible with matplotlib 1.5. Unfortunately, this version is currently the default package installedby conda. Users should install sherpa with` matplotlib=1.4` numpy=1.9.

Test requirements are not installed automatically: sherpa_test does not work out of the box. Users should issue “pip install pytest-cov” in order for sherpa_test to run.

SAO DS9 issue on Ubuntu 14.04: the ds9 binaries shipped with Ubuntu and installed through apt-get install do not seem to work as expected. Binaries downloaded directly from the SAO ds9 page seem to work instead.