Compile a XSPEC user model for use in Sherpa *experimental*
convert_xspec_user_model name modelfile [extrafile [extrafile [..]] Unlike most CIAO contributed scripts, there is no parameter file. The supported command-line flags can be found using -h/--help: --no-xspec do not include XSPEC libraries when building --cfitsio include the CFITSIO library when building -c / --clobber will overwrite existing files. -p / --prefix flags are used to control the user model names. -k / --keep flags keep the .pyf file used in compilation. -v / --verbose changes the amount of screen output. -f / --f2py passes through arguments to the f2py call. --version prints the script version
Not supported in CIAO 4.11
The script has not been updated to work in CIAO 4.11. Please contact the CXC HelpDesk if you find this script useful.
The convert_xspec_user_model tool will compile a XSPEC user model into a form usable by Sherpa; it is therefore similar to the "initpackage" command in XSPEC. Unlike the initpackage command, this script does not require that the XSPEC source package is installed, and is not run from Sherpa but the command line.
This script is *experimental* and will not work for all models; please see the "Known Problems" section below for more information and contact the CXC HelpDesk if you find problems or models that it will not compile. At present any model requiring access to the udmget XSPEC routine is not supported.
The script takes the model file - often called model.dat or lmodel.dat - and the source code in the working directory, using them to create a Python module consisting of the compiled model and some supporting Python code that creates the Sherpa model. The output is a Python module, whose name is the first argument to the script. This can be loaded into Sherpa by saying (assuming that you ran 'convert_xspec_user_model relxill ...' and that you are in the same directory):
sherpa> import relxill
or, by using the lmod rouine, which allows you to load in models from other directories, displays a list of the models in the module, and imports all the convolition routines into the current namespace:
sherpa> from sherpa_contrib.xspec.local import lmod sherpa> lmod('relxill', '/data/models/relxill')
At this point the models can then be used; the default behavior of the script is to add the prefix "xs" to the model name (in a similar manner to how Sherpa uses the prefix "xs" for the XSPEC models), but this can be changed using the --prefix flag, as shown below in the examples.
The interface to the XSPEC convolution models has not been thoroughly tested, so take care when using them! Note that there is currently no support in Sherpa for extending the grid to use in a computation, which means that some of the convolution models are not going to give the desired results. Models like CFLUX should be okay.
The convolution models require a slightly different interface, which is designed to follow the Sherpa load_conv and load_psf commands. For each convolution model there will be a corresponding load_XXX command; e.g. if the model file contains a convolution model called specconv then there will be a load_specconv command (note that this naming scheme ignores the value of the --prefix flag). If you have used the "import name" form, shown above, then you have to use "name.load_specconv" to call the function. This can be avoided by importing the symbols directly - e.g.
sherpa> from relxill import load_specconv sherpa> from relxill import *
- or by using the lmod routine discussed earlier (see ahelp lmod).
The load routines take a single argument, a string giving the model name, which can then be used as a model component and any parameters changed. For example,
sherpa> load_specconv("convmdl") sherpa> print(convmdl) ... sherpa> convmdl.spin = 3.2 sherpa> set_source(convmdl(powlaw1d.pl))
where this assumes that the specconv model has a spin parameter.
Sherpa already supports additive and multiplicative table models using the load_xstable_model command; this was added in CIAO 4.9 and replaces the support for XSPEC table models in the load_table_model command.
unix% ls *.f mdl.f unix% convert_xspec_user_model mymodel lmodel.dat
The mymodel module is created, which contains the model(s) defined in the file lmodel.dat and with code in mdl.f. If the script completes successfully then it will have created the directory mymodel/, and you can then load it into Sherpa with the command:
sherpa> import mymodel
At this point any additive or multiplicative model will be available using the XSPEC name with the prefix "xs". So, if lmodel.dat contained models mdl1 and mdl2 then instances of these models can be created and inspected as any other Sherpa model; for example (assuming mdl2 has an alpha component):
sherpa> set_source(xsphabs.galabs * xsmdl1.mdl) sherpa> xsmdl2.gas sherpa> print(mdl) sherpa> gas.alpha = 2
unix% convert_xspec_user_model mymodel2 lmodel.dat --prefix
In this case the model names are not preceeded by "xs", as in the example above, because the --prefix argument has been given. This means that instead of saying xsmdl1.mdl you would just say mdl1.mdl.
Note that there is no check that the model names do not match existing Sherpa models or function names, so there is a chance for confusion.
To use a different prefix than "xs" supply an argument to the --prefix flag, but it must start with a capital letter - e.g.
would make the model name be xsummdl1 rather than xsmdl1.mdl. The requirement for a capital letter is because the name is also used to create a Python class (this class is not used in normal use of Sherpa).
unix% ls *.c polar.c polarutils.c unix% convert_xspec_user_model polar lmodel_polar.dat --clobber
Here the models in the file lmodel_polar.dat - with C code automatically picked up from the polar.c and polarutils.c files - are compiled to create a polar module. The --clobber argument is used to delete any existing version of the module.
unix% convert_xspec_user_model foo model.dat --no-xspec --cfitsio
In this case the model uses CFITSIO to read in files, but not other XSPEC libraries. This may cause problems when trying to load the module, due to differences in the FORTRAN compiler used to build CIAO and the model code. If this happens then convert_xspec_user_model will fail, so the --no-xspec and --cfitsio flags are used to try to avoid this problem.
Loading the module
Loading a model from a different directory
Since the code generates a Python module, you can load in modules from different directories by adjusting your PYTHONPATH environment variable or the os.sys.path array. We provide a simple wrapper routine, lmod, in the sherpa_contrib.xspec.local package, which can be used in a similar manner to the XSPEC lmod command. For example, after
unix% cd /data/models unix% convert_xspec_user_model mymod model.dat
you can load in the model package with the following:
sherpa> from sherpa_contrib.xspec.local import lmod sherpa> lmod("mymod", "/data/models")
which will also display the models contained in the module; see ahelp lmod for more information.
Automatically loading the module into Sherpa
Models can automatically be loaded into Sherpa by taking advantage of the IPython startup directory. Any Python file in $HOME/.ipython-ciao/profile_sherpa/startup/ will be loaded when Sherpa is started up, so add a file to this directory - e.g. 90-xspec-models.py - with commands to set up the models. An example file is shown below; adjust as appropriate:
import os from sherpa_contrib.xspec.local import lmod # This assumes that convert_xspec_user_model was run in the two directories # given in the lmod command: lmod("carbatm", "/data/models/carbatm", verbose=False) lmod("relxill", "/data/models/relxill", verbose=False) # The carbatm model needs to know where the data files are # via XSPEC settings; relxill via an environment variable: set_xsxset("CARBATM", "/data/models/carbatm") os.environ["RELLINE_TABLES"] = "/data/models/relxill" # display a screen message to remind me what I have loaded print("XSPEC models: carbatm and relxill")
Starting Sherpa will display the message before the first prompt:
unix% sherpa -n XSPEC models: carbatm and relxill sherpa-1>
The verbose flag is set to False when calling lmod() to avoid the model names always being displayed when Sherpa is started.
This only works for the Sherpa application. If you have loaded up Sherpa into ChIPS, an IPython session, or a script then you wil have to manually load in the models you need.
Change in the IPython configuration directory
The location of the IPython configuration directory changed in CIAO 4.8 to $HOME/.ipython-ciao/. In CIAO 4.5 to 4.7, inclusive, it was $HOME/.ipython-ciao45/.
Please contact the CXC HelpDesk if you have a problem or need to use an unsupported feature, such as platform or language.
The OS-X support is limited and there has been no testing with the Python 3.5 variants of CIAO 4.9.
All the languages supported by XSPEC user models - i.e. C, C++, and Fortran - should be supported, as long as the compiler is compatible with that used to compile the XSPEC models in CIAO (see below).
No support for udmget
Those models that require the --udmget flag to the XSPEC initpackage are not supported at this time.
Sherpa only supports the additive, multiplicative, and convolution models; other models will be ignored. The convolution models have only been lightly tested.
Models that need to be re-evaluated per spectrum or that calculate model variances are converted but there has been no testing to see if they work correctly (note that Sherpa ignores the error array that is returned by these models).
When using convolution models, one possible problem is if you have ignored an energy - or wavelength - range within the data; if this range is larger than the RMF at this energy then it may be possible that the convolution will not properly account for edge effects.
Choice of module name
The first argument to convert_xspec_user_model is used to create a Python module of the same name that can then be loaded into Sherpa or a Python script. It can not match the name of one of the models (once the prefix value has been added), and it should not match other Python modules or common symbols, otherwise it could confuse users. There is only limited checking to see whether there are possible name clashes for the module and model names.
Periodic parameters are not supported.
Accessing XSPEC routines
The default behavior is to try and link against the XSPEC libraries, but the results have not been well tested. One known problem is if the gfortran version is significantly different to that used to build the XSPEC models provided as part of CIAO, which can lead to errors such as
undefined symbol: _gfortran_copy_string
If the model does not make use of the XSPEC libraries then the --no-xspec flag can be used to avoid linking against the XSPEC code, which may help in this situation.
If the model does not use the XSPEC libraries but it does use CFITSIO then try both the --no-xspec and --cfitsio flags; an example of when this is needed is seeing messages such as
undefined symbol: ftgmsg_
The --f2py option can be used to fine-tune the compile and link options used by f2py if the --no-xspec and --cfitsio options are insufficient. The --keep argument is useful here, since it displays the f2py command line used by the script/
External files or set up
The equivalent location of the spectral/xspec/ directory of XSPEC is the $ASCDS_INSTALL/ots/spectral/modelData/ directory. The set_xsxset command can be used to set an XSPEC "xset" variable, if the model uses these to define alternative locations for data files or other configuration information. The os.environ directionary can be used to set an environment variable if these are used by the model; for example
sherpa> set_xsxset('CARBATM', '/data/models/data/carbatm') sherpa> os.environ['TBLLOC'] = '/data/models/data'
will set the CARBATM XSPEC variable and the TBLLOC environment variable.
Not all parameter names are usable in Sherpa, due to the Python grammar. Below is a list of the conversion rules used, but if you are in doubt then just print out an instance of a model to see what the names are.
The conversion rules are:
- The @ character is replaced with the string At.
- The name break is converted to break_, lambda to lambda_, type to type_; any parameter surrounded by < and > has these characters removed and _ave added to the end (so "<kT>" would be changed to "kT_ave"); and a name ending with "(...)" is replaced by "_..." (so "Rin(G)" becomes "Rin_G").
- Any invalid character for a Python symbol is replaced by _.
What does the script do?
The compilation of the code is handled by the f2py script (despite the name it can also handle C code; see the F2PY home page for more information).
There are two parts to the process:
- create a module of the compiled code
- create the Python interface to this compiled code.
The model definition file is parsed to extract all the models, which provides details on the model name, its type (such as additive or multiplicative), the arguments, the name of the routine that evaluates the model, and the language used for this routine. The parsing follows the definition given in the XSPEC appendix on local models.
Creating the compiled code
This information is then used to create a f2py signature file which defines the interface we want to have in Python to the compiled code. The signature file is named <module>.pyf in the working directory and is deleted by default but can be retained using the --keep flag, which will also display the coommand-line arguments used to run f2py, in case it needs changing. The signature file then processed by f2py to create the compiled module, which is moved to the location <module>/_<module>.so.
If the f2py call fails, then the command-line used to run it will be displayed and the signature file will not be deleted. Extra flags can be passed through to f2py using the --f2py (or -f) flag; for instance
unix% convert_xspec_user_model mdl lmodel.dat -f="-DDBG" -f="-L/opt/local/lib" -f="-lmutils"
will add the options -DDBG, -L/opt/local/lib, and -lmutils to the f2py call. Use
unix% $ASCDS_INSTALL/ots/bin/f2py -h
for more information on the flags supported by f2py.
What files are compiled?
The following files in the current working directory are compiled: *.f, *.f03, *.f90, *.c, *.cxx, *.C, and *.cc. Note that if C++ models - i.e. those with a C_ prefix in the model.dat file - cause the script to create a file called wrap_<modulename>.cxx, which is also compiled by f2py and then deleted (unless --keep is specified).
Creating the python code
The script creates one Python file, __init__py, in the directory, which contains the classes for the models along with the code needed to "install" them into Sherpa (e.g. so that they are included in the output of list_models). The model class names are created by appending the model name from the definition file to the prefix value (which defaults to "XS"); since Python class names have to start with a capital letter the prefix must be capitalised (or, if blank, then the first character of the model name is capitalized).
Changes in the scripts 4.11.1 (December 2018) release
The script has not been updated to work with XSPEC 12.10.0e (which is distributed as part of CIAO 4.11). Please contact the CXC HelpDesk if you find this script useful.
Changes in the scripts 4.9.2 (April 2017) release
The script should now work with XSPEC convolution models.
Changes in the scripts 4.9.1 (December 2016) release
The script has been updated to work with CIAO 4.9 but has only seen very-limited testing. There is still no support for convolution models.
Changes in the scripts 4.8.2 (January 2016) release
Support for models with an initialization string
XSPEC models which use an initialization string, such as the snapec model can now be converted.
Changes in the scripts 4.8.1 (December 2015) release
This script has not been updated to reflect changes made in Sherpa in this release. Please contact the CXC HelpDesk if you need to use this script.
Changes in the scripts 4.6.6 (September 2014) release
The script is new in this release.
This script is not an official part of the CIAO release but is made available as "contributed" software via the CIAO scripts page. Please see the installation instructions page for help on installing the package.
No support for convolution models
The support for XSPEC convolution models was broken with the release of CIAO 4.8 and the code has not yet been updated to address this.
Support on El-Capitan and later
The security changes in OS-X El-Capitan mean that an additional step may be required, after running convert_xspec_user_model. If the tool reports a failure when trying to import the module (which it does to check that the build was a success) then try the following command (replacing lname with the first argument to the convert_xspec_user_model call):
unix% install_name_tool `ls -1 name/*so` -change libcfitsio.2.dylib @rpath/../ots/lib/libcfitsio.2.dylib
Refer to the CIAO bug pages for an up-to-date listing of known issues.