Table Of Contents

This Page

_images/sherpa_logo.gif

Sherpa: Modeling and Fitting in Python

Sherpa is a modeling and fitting application for Python. It contains a powerful language for combining simple models into complex expressions that can be fit to the data using a variety of statistics and optimization methods. It is easily extensible to include user models, statistics and optimization methods.

What can you do with Sherpa?

  • fit 1D (multiple) data including: spectra, surface brightness profiles, light curves, general ASCII arrays
  • fit 2D images/surfaces in Poisson/Gaussian regime
  • build complex model expressions
  • import and use your own models
  • use appropriate statistics for modeling Poisson or Gaussian data
  • import the new statistics, with priors if required by analysis
  • visualize a parameter space with simulations or using 1D/2D cuts of the parameter space
  • calculate confidence levels on the best fit model parameters
  • choose a robust optimization method for the fit: Levenberg-Marquardt, Nelder-Mead Simplex or Monte Carlo/Differential Evolution.

To install Sherpa see Sections Source and Binary Installation. For an example of a Sherpa session check Run Sherpa section. More examples are given in a few Sherpa IPython Notebooks:

Sherpa Quick Start

Image Fitting

Template Fitting

A Simple Bayes Analysis of an X-ray Spectrum

For detailed documentation see: http://cxc.harvard.edu/sherpa

Install Sherpa

Sherpa can be installed from a binary distribution or built from sources.

The binary distribution is suited for people wanting to have Sherpa up and running as soon as possible in its standard form. The binaries are built and tested on Linux 32, Linux 64 and Mac OSX (10.8 and 10.9)

Source installation is available for platforms incompatible with the binary builds, or for users wanting to customize the way Sherpa is built and installed.

Source Installation

It takes only a few simple steps to build and install Sherpa in any Python 2.7 environment on Linux or Mac OSX. For example Anaconda Python Distribution contains many scientific software components needed for the analysis and Sherpa fits seamlessly into that environment.

The Sherpa source code is available on GitHub: https://github.com/sherpa/sherpa

Download the Sherpa source tar or zip file: sherpa-4.7.zip or sherpa-4.7.tar.gz

Unpack the files:

$ unzip sherpa-4.7.zip
or
$ tar xvf sherpa-4.7.tar.gz

To install:

$ cd sherpa
$ python setup.py install

The Sherpa code will be built and installed in the directory ${prefix}/lib/python2.7/site-packages/sherpa, where ${prefix} can be determined with:

$ python -c 'import sys; print(sys.prefix)'

Sherpa requires the standard Python packages and system compilers for the build:

Python: setuptools, numpy
System: gcc, g++, gfortran, make, flex, bison

In addition I/O requires pyfits for reading FITS files and matplotlib for standard plotting, ds9 for imaging.

After the installation you can run the test to check if the installation was successful:

$ sherpa_test

Note that the ${prefix}/bin should be in the $PATH to run the test. The tests should succeed, but there could be two expected failures if ‘ds9’ or/and ‘XSPEC’ models are not found. These are not necessary for the use of Sherpa and only needed if you plan to perform X-ray analysis or use ‘ds9’ to display images.

Copy the source code repository

If you want to look at the code, use the code and maybe develop your own code in Git you can ‘clone’ the full repository structure with the default ‘git clone’:

This would also be useful if you want to contribute your own code to Sherpa via ‘pull request’. The contributions are welcome. We advise to follow the astropy guidlines for developing your contributions. We require the code to have a description of the code, unit and integration test and a user documentation.

Custom Source Build

The costumization of the source build could be made by updating updating the setup.cfg file in ‘sherpa’ directory. For example to use the fftw local library one needs to set the following configuration options:

fftw=local
fftw-include-dirs=/usr/local/include
fftw-lib-dirs=/usr/local/lib
fftw-libraries=fftw3

where /usr/local path may need to be changed to the local directories with the header (.h) files and the libfftw3.so shared object file.

It can also allow you to build XSPEC models.

XSPEC Models

Sherpa does not support XSPEC models for X-ray spectral analysis by default. However, it is possible to instruct Sherpa to build its XSPEC extension module by changing the build configuration options.

You may need to build XSPEC yourself, and in any case to point Sherpa to existing binary libraries for XSPEC, cfitsio, and CCfits. Additionaly, you will need to point Sherpa to the libgfortran shared library. These dependencies are required to build XSPEC itself, so they are assumed to be present on the system where you want to build Sherpa with XSPEC support. Here we assume that XSPEC has been already build.:

$ git clone https://github.com/sherpa/sherpa.git
$ cd sherpa

Edit the setup.cfg file. Find the XSPEC configuration section in the file, uncomment the relative options and make sure they point to the location of the XSPEC, cfitsio, CCfits, and gfortran libraries. For instance:

with-xspec=True
xspec_lib_dirs=/opt/xspec/lib
cfitsio_lib_dirs=/usr/local/lib
ccfits_lib_dirs=/usr/local/lib
gfortran_lib_dirs=/usr/local/lib

You may need to change the values in the above example to reflect the actual directories where the libraries are to be found on your system.

Then, build Sherpa in the standard way:

$ python setup.py install

Binary Install

The binary installation of Sherpa 4.7 was released on April 17, 2015. It has been tested on Linux 32, Linux 64 and Mac OSX (10.8 and 10.9).

Sherpa binaries can be seamlessly installed into Anaconda Python. You need to add Sherpa channel to your configuration, and then install Sherpa:

$ conda config --add channels https://conda.binstar.org/sherpa
$ conda install sherpa

To test that your installation works type:

$ sherpa_test

To update Sherpa:

$ conda update sherpa

If you do not have Anaconda Python and want to create the minimum environment for trying Sherpa check the README on GitHub.

Configuration Files

Sherpa expects a user configuration file sherpa-standalone.rc to be in the $HOME directory. If this file is not present Sherpa will use the default internal configuration file with the IO and Plotting back-ends set to pyfits and pylab.

matplotlib comes with a configuration file matplotlibrc. For smooth behavior with Sherpa, be sure to indicate interactive=True in ~/.matplotlib/matplotlibrc.

Run Sherpa

You can import Sherpa into your ipython session:

(conda)$  ipython --pylab
Python 2.7.9 |Anaconda 1.9.2 (x86_64)| (default, Dec 15 2014, 10:37:34)
Type "copyright", "credits" or "license" for more information.

IPython 2.3.1 -- An enhanced Interactive Python.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
Using matplotlib backend: MacOSX

In [1]: from sherpa.astro.ui import *
WARNING: imaging routines will not be available,
failed to import sherpa.image.ds9_backend due to
'RuntimeErr: DS9Win unusable: Could not find ds9 on your PATH'

The standard warnings are issued if you do not have ds9 models in your path. The image with ds9 will not be available. See the Dependencies section below.

Now to simulate a simple shape (a parabola with errors):

In [2]: x = np.arange(-5, 5.1)

In [3]: y = x*x + 23.2 + np.random.normal(size=x.size)

In [4]: e = np.ones(x.size)

The data can now be loaded into Sherpa:

In [5]: load_arrays(1, x, y, e)

In [6]: plot_data()
_images/data.png

For this example we know what model to use, so pick a polynomial and free-up some of the parameters:

In [7]: set_source(polynom1d.poly)

In [8]: print(poly)
polynom1d.poly
   Param        Type          Value          Min          Max      Units
   -----        ----          -----          ---          ---      -----
   poly.c0      thawed            1 -3.40282e+38  3.40282e+38
   poly.c1      frozen            0 -3.40282e+38  3.40282e+38
   poly.c2      frozen            0 -3.40282e+38  3.40282e+38
   poly.c3      frozen            0 -3.40282e+38  3.40282e+38
   poly.c4      frozen            0 -3.40282e+38  3.40282e+38
   poly.c5      frozen            0 -3.40282e+38  3.40282e+38
   poly.c6      frozen            0 -3.40282e+38  3.40282e+38
   poly.c7      frozen            0 -3.40282e+38  3.40282e+38
   poly.c8      frozen            0 -3.40282e+38  3.40282e+38
   poly.offset  frozen            0 -3.40282e+38  3.40282e+38

In [9]: thaw(poly.c1, poly.c2)

With everything set up, the data can be fit using the standard optimization method levmar and chi2 statistics:

In [10]: fit()
Dataset               = 1
Method                = levmar
Statistic             = chi2
Initial fit statistic = 12190
Final fit statistic   = 5.40663 at function evaluation 8
Data points           = 11
Degrees of freedom    = 8
Probability [Q-value] = 0.713361
Reduced statistic     = 0.675829
Change in statistic   = 12184.6
   poly.c0        22.2341
   poly.c1        0.109262
   poly.c2        1.06812

In [11]: plot_fit_resid()
_images/fit.png

and an estimate of 1 sigma parameters uncertainties is:

In [12]: conf()
poly.c0 lower bound:        -0.455477
poly.c1 lower bound:        -0.0953463
poly.c0 upper bound:        0.455477
poly.c2 lower bound:        -0.0341394
poly.c1 upper bound:        0.0953463
poly.c2 upper bound:        0.0341394
Dataset               = 1
Confidence Method     = confidence
Iterative Fit Method  = None
Fitting Method        = levmar
Statistic             = chi2gehrels
confidence 1-sigma (68.2689%) bounds:
   Param            Best-Fit  Lower Bound  Upper Bound
   -----            --------  -----------  -----------
   poly.c0           22.2341    -0.455477     0.455477
   poly.c1          0.109262   -0.0953463    0.0953463
   poly.c2           1.06812   -0.0341394    0.0341394

Dependencies

Data I/O support and plotting can be supplemented using PyFITS and “matplotlib”. Imaging requires ds9/XPA.

[pyfits]http://www.stsci.edu/resources/software_hardware/pyfits
[mpl]Hunter, JD (2007). Matplotlib: A 2D graphics environment. Computing in Science and Engineering. 9: 90-95. http://matplotlib.sourceforge.net.
[ds9]http://hea-www.harvard.edu/RD/ds9/
[xpa]http://hea-www.harvard.edu/saord/xpa/

Release Notes Sherpa 4.7 for Python

April 17, 2015

The Sherpa 4.7 code released on April 17, 2015 is the same as the CIAO 4.7 version. There are, however, small differences required for the standalone build.

Differences between Standalone 4.7 and CIAO 4.7

  • Standalone Sherpa Python and CIAO Sherpa now look for different configuration files in the $HOME directory. For the standalone this is $HOME/.sherpa-standalone.rc. If this file is not present, Sherpa falls back to the internal configuration file. The default IO and plotting backends are set to pyfits and pylab in the Standalone Sherpa, and he stack trace is not silenced. Users can update the settings and override the configuration file location by exporting the SHERPARC environment variable, as supported by the previous versions of Sherpa and CIAO.
  • All source files now have copyright and licensing information. A summary is included upfront in the repository main directory.
  • Orphan code that was not actually used has been removed.
  • Fixed code triggering deprecation warnings from Numpy 1.9.
  • Fixed code triggering compiler warnings.
  • Added documentation (README, ipython notebook).
  • The version string (sherpa.__version__) depends on the git commit/tag, unlike in CIAO where it is fixed to 40701.
  • Some classes from the template module were not exposed by __all__, and template models were not imported in the sherpa session.
  • The standalone repository contains some code originated in CIAO and also includes other OTS: the code coming from CIAO has been updated to 4.7.
There is a caveat:
  • the datastack module is not imported if no plotting packages are available. matplotlib has to be installed to use that module.