Skip to the navigation links
Last modified: November 2010

URL: http://cxc-newtest.cfa.harvard.edu/ciao4.2/ciao_runtool.html
AHELP for CIAO 4.2

ciao_runtool

Context: py.ciao_contrib

Synopsis

Run CIAO tools from Python as if they were functions (CIAO contributed package).

Syntax

from ciao_contrib.runtool import *

Run the dmcopy tool:
dmcopy("evt2.fits[bin sky=::8]", "img.fits")

Run dmstat:
dmstat.punlearn()
dmstat.median = True
dmstat("src.dat[cols z,fx]", verbose=0)
print("The medians are {0}".format(dmstat.out_median))

Description

The ciao_contrib.runtool module allows CIAO tools to be run as if they were Python functions, and supports a pset-like mode where tool parameters can be set before the tool is run, named parameters when calling the tool, and easy access to the parameter values after the tool has completed. Parameter values are given using native Python types - e.g. True and False for boolean parameters.

The ciao_contrib.runtool module is provided as part of the CIAO contributed scripts package.

Loading the routines

The module can be loaded into ChIPS or Sherpa by saying:

from ciao_contrib.runtool import *

Note that it is not provided by the ciao_contrib module itself, unlike some of the other ciao_contrib modules.

Tools and parameter files

The runtool module provides a function for each major tool in the CIAO bin and contrib/bin directories that has a parameter file. It also provides functions to set and get settings for those parameter files without an associated tool, such as ardlib, lut and colors.

Extra routines

The following routines are also provided:

  • make_tool() - create a routine that can be used to call a CIAO tool
  • set_pfiles() - easily change the user portion of the PFILES environment variable
  • list_tools() - returns a list of the tools and parameter files that are covered by this module

Running a tool

A tool can be run by calling the routine with all the arguments it needs; required arguments are given either by positional order or as a named argument whereas optional arguments always have to be given as a named argument. As an example

chips> dmcopy("in.fits[sky=region(src.reg)][bin sky=::1]", "img.fits")
chips> dmstat(infile="img.fits", median=True, centroid=False)

Parameters can also be set using a parameter-style interface of <toolname>.<parname> before calling the tool as shown below:

chips> dmcopy.outfile = "img.fits"
chips> dmcopy.infile = "in.fits[sky=region(src.reg)][bin sky=::1]"
chips> dmcopy()
chips> dmstat.median = True
chips> dmstat.centroid = False
chips> dmstat(infile="img.fits")

You can mix both approaches, as shown above in the last dmstat call.

Clearing, setting, and viewing parameter settings

Each tool has a punlearn() method which will reset the values to the CIAO defaults. As shown above, parameter values can be set either when calling the tool or via the <toolname>.<parname> approach, and the current settings can be displayed using print. As an example

chips> dmjoin.punlearn()
chips> dmjoin.interpolate = "closest"
chips> print(dmjoin)
Parameters for dmjoin:

Required parameters:
              infile =                  Input file
            joinfile =                  Input join file
             outfile =                  Output file
                join =                  Join col

Optional parameters:
         interpolate = closest          Interpolate mode
             verbose = 0                Debug Level(0-5)
             clobber = False            Clobber

There are also write_params() and read_params() methods that can be used to create a .par file from the current settings or read values in from a given .par file. See "help <toolname>.read_params" or "help <toolname>.write_params" for more information on these methods.

Support for unique parameter prefixes

As with the command line, parameter names can be shortened to any unique prefix. This means that dmjoin.inf can be used to read or write the infile parameter of dmjoin, but dmstat.joi will fail since it matches both the join and joinfile parameters.

This support also holds when calling a tool, so

chips> dmcopy(infile, outfile, cl=True)

calls dmcopy with the clobber parameter set to yes.

An error will be raised if the prefix is not unique; for instance

chips> dmjoin.joi
AttributeError: Multiple matches for dmjoin parameter 'joi', choose from:
  joinfile join

A SyntaxError will occur if a Python reserved word - such as "in" or "for" - is used as the parameter name.

Parameter constraints

If a parameter can only be one of a set of values then the parameter value can be set to a unique substring, just as on the command line. As an example, the interpolate parameter of dmjoin can be one of "linear", "first", "last", "closest", "furthest", "minimum", or "maximum", so the following are all valid:

chips> dmjoin.interp = "min"
chips> dmjoin("a.dat","b.dat","c.dat","x",interp="lin")

An error occurs if a parameter is set to an invalid value; e.g. either using a number outside the valid range or if there are a limited set of choices, as shown below:

chips> dmjoin.verbose = 10
ValueError: dmjoin.verbose must be <= 5 but set to 10
chips> dmjoin.interp = "foo"
ValueError: The parameter interpolate was set to foo when it must be one of:
  linear first last closest furthest minimum maximum

The return value

If there is no screen output from the tool then the tool returns None, otherwise it returns a single string containing both the STDOUT and STDERR channels. This output can either be stored in a variable or - if used from an interactive environment such as ChIPS and Sherpa - it will be automatically displayed. In the following, the dmcopy call creates no screen output, so it returns None which is not displayed by ChIPS, whereas the dmstat output is displayed

chips> dmcopy(infile, outfile)
chips> dmstat(outfile)

y
    min:	10 	      @:	1 
    max:	87.3 	      @:	5 
   mean:	35.3 
  sigma:	28.55548984 
    sum:	176.5 
   good:	5 
   null:	0
chips> out = dmstat(outfile)
chips> lines = out.split("\n")
chips> print(lines[1])
    min:	10 	      @:	1 

What happens if the tool fails?

An IOError is raised if the tool returns a non-zero exit status. The error contains the screen output of the tool, as shown below where I used an invalid file:

chips> dmstat("simple.dat[cols z]")
IOError: An error occurred while running 'dmstat':
  # DMSTAT (CIAO 4.2): could not open infile simple.dat[cols z].

The return value can be retrieved using the get_runtime_details() method of the tool, which is described below.

Unfortunately not all CIAO tools set the exit status correctly on error, so these will appear to have run successfully even on failure.

How long did the tool run for?

The get_runtime_details() method returns a dictionary which lists information on the last time the tool was run using this interface. This information is cleared when the punlearn() method is used. The keyword and values returned are listed below. Note that no keyword is guaranteed to be present and that the timings are not intended for precise bench-marking so should not be considered to be accurate to more than about one second.

Keyword Value
code The return value for the tool (0 indicates success, otherwise there was a failure).
start The start time, as returned by the time.localtime() routine.
end The end time, as returned by the time.localtime() routine.
delta A string representing the run time. It is useful for a simple display, but use the start and end fields for a more accurate representation.
args An array of the command-line arguments used to run the tool, stored as (name,value) pairs.
output The screen output for the tool (includes both STDOUT and STDERR in the same string). This is not cleaned up to remove trailing white space as is done for the return value when calling the tool.

See the documentation for the Python time module for more information on how to use the start and end values.

Example 1

Providing all the arguments in the call

In this example we run the dmcopy tool and provide all the necessary arguments when the routine is called. The dmcopy.punlearn routine is used to ensure the settings are at their default values.

chips> dmcopy.punlearn()
chips> dmcopy("evt2.fits[ccd_id=7]", "ccd7.fits", clobber=True)

We adapt this to loop through the ccd_id values 0, 1, 2, 3, and 6 and create the files "ccd<n>.fits". We also take advantage of the support for using parameter prefixes to shorten "clobber" to "cl".

chips> for ccd in [0,1,2,3,6]:
           infile = "evt2.fits[ccd_id={0}]".format(ccd)
           outfile = "ccd{0}.fits".format(ccd)
           dmcopy(infile, outfile, cl=True)
           print("Created: {0}".format(outfile))


Created: ccd0.fits
Created: ccd1.fits
Created: ccd2.fits
Created: ccd3.fits
Created: ccd6.fits

Example 2

Setting the arguments before the call

The "pset" approach can also be used to set the arguments before the call. The original example above can be re-written as:

chips> dmcopy.punlearn()
chips> dmcopy.infile = "evt2.fits[ccd_id=7]"
chips> dmcopy.outfile = "ccd7.fits"
chips> dmcopy.cl = True
chips> dmcopy()

Example 3

Mixing both approaches

Both approaches can be combined. For instance, if you wish to call a tool multiple times with some parameterds fixed you may wish to use the pset approach for those parameters and call the tool with those arguments that changes.

Here we set the clobber argument and then call dmcopy with the infile and outfile arguments.

chips> dmcopy.punlearn()
chips> dmcopy.cl = True
chips> dmcopy(infile, outfile)

Example 4

Accessing parameter values after a tool runs

Some tool set parameter values after being run. These values can be inspected and used as shown below, which includes setting the parameter value for the mkexpmap tool based on the output of get_sky_limits.

chips> get_sky_limits("img.fits", verbose=0)
chips> dmf = get_sky_limits.dmfilter
chips> print("Filter: {0}".format(get_sky_limits.dmf))
Filter: X=4047.5:4098.5:#51,Y=3920.5:3971.5:#51
chips> mkexpmap.xygrid = get_sky_limits.xygrid

The whole set of parameters can also be displayed:

chips> print(get_sky_limits)
Parameters for get_sky_limits:

Required parameters:
               image = img.fits         Image for which you want to know the binning

Optional parameters:
           precision = 1                Precision [# decimal points] for output numbers
            dmfilter = X=4047.5:4098.5:#51,Y=3920.5:3971.5:#51  DM filter syntax to match image
              xygrid = 4047.5:4098.5:#51,3920.5:3971.5:#52  xygrid parameter for mkexpmap to match image
             verbose = 0                Debug Level (0-5)

Example 5

Multiple copies of a tool

Multiple versions of a tool can be created, each with its own set of parameters. As a simple case, if you want to run dmstat with the median parameter set to True or False you could try the following:

chips> dms1 = make_tool("dmstat")
chips> dms1.median = False
chips> dms2 = make_tool("dmstat")
chips> dms2.median = True

Then using dms1 or dms2 will run dmstat with the median parameter set to False and True repsectively:

chips> dms1("lc.fits[cols count_rate]")
chips> dms2("lc.fits[cols count_rate]")

ROUTINES WITH NO CORRESPONDING TOOL

There are a number of CIAO parameter files which have no corresponding tool - for instance ardlib, geom, colors and lut. These are provided in this module for reading and setting parameters, but you can not call them. Also, as discussed below in the "PARAMETER FILES" section, setting these parameters does not change the on-disk parameter file unless you call the write_params method.

Here is an example of accessing the lut parameter file:

chips> print(lut)
Parameters for lut:
                   a = ${ASCDS_CALIB}/a.lut  Color lookup table
                aips = ${ASCDS_CALIB}/aips.lut  Color lookup table
                   b = ${ASCDS_CALIB}/b.lut  Color lookup table
                  bb = ${ASCDS_CALIB}/bb.lut  Color lookup table
                blue = ${ASCDS_CALIB}/blue.lut  Color lookup table
               color = ${ASCDS_CALIB}/color.lut  Color lookup table
               green = ${ASCDS_CALIB}/green.lut  Color lookup table
                grey = ${ASCDS_CALIB}/grey.lut  Color lookup table
              halley = ${ASCDS_CALIB}/halley.lut  Color lookup table
               heaob = ${ASCDS_CALIB}/heaob.lut  Color lookup table
                heat = ${ASCDS_CALIB}/heat.lut  Color lookup table
                 hsv = ${ASCDS_CALIB}/hsv.lut  Color lookup table
                  i8 = ${ASCDS_CALIB}/i8.lut  Color lookup table
            rainbow1 = ${ASCDS_CALIB}/rainbow1.lut  Color lookup table
            rainbow2 = ${ASCDS_CALIB}/rainbow2.lut  Color lookup table
                ramp = ${ASCDS_CALIB}/ramp.lut  Color lookup table
                 red = ${ASCDS_CALIB}/red.lut  Color lookup table
                 sls = ${ASCDS_CALIB}/sls.lut  Color lookup table
            standard = ${ASCDS_CALIB}/standard.lut  Color lookup table
chips> hl = lut.halley
chips> print(os.path.expandvars(hl))
/soft/ciao-4.2/data/halley.lut

(assuming CIAO is installed in /soft/ciao-4.2/) but it can not be called

chips> lut(b="tst.lut")
TypeError: 'CIAOParameter' object is not callable

PARAMETER FILES

Using these routines does not change any parameter file on disk, unless you use one of the routines listed below. This means it is safe to use this module to run tools from a script whilst using CIAO tools from the command line; the parameter files used by the command-line tools will not be over-written or changed by the Python script.

Any routine that is an instance of the CIAOToolParFile class will not change the on-disk parameter file.

Routines that do change the on-disk parameter file

Running the acis_run_hotpix, axbary, create_bkg_map, dmgti, evalpos, mkarf, mkexpmap, mkgarf, mkgrmf, mkinstmap, modelflux, pileup_map, srcextent, tgdetect or wavdetect routines will change the on-disk parameter file for the tool. All these routines are instances of the CIAOToolDirect class.

How to check whether a parameter file will be changed?

The class of the routine - which can be returned by the type() call or checked using isinstance() - determines whether a parameter file will be changed ("CIAOToolDirect") or not ("CIAOToolParFile"). For instance,

chips> type(wavdetect)
       <class 'ciao_contrib.runtool.CIAOToolDirect'>
chips> type(dmcopy)
       <class 'ciao_contrib.runtool.CIAOToolParFile'>

These class names are rather confusing and may be changed in a later release.

WARNING: use of ardlib and other such parameter files

Many instrument-specific tools use parameter files like ardlib and geom to find certain files. When the tool is run, they still use the on-disk version of the file and not the values stored in the Python object of the same name. So the setting

chips> ardlib.AXAF_ACIS0_BADPIX = "bpix.fits[ccd_id=0]"

will not be picked up when a tool such as mkinstmap is run. One solution is to use the write_params method, which is described further below, to write out the ardlib parameter file to disk after making changes.

chips> ardlib.AXAF_ACIS0_BADPIX = "bpix.fits[ccd_id=0]"
chips> ardlib.write_params()

Updating a parameter file

If you want to write the current set of parameters to disk, you can use the "write_params" method. As an example, where e1, e2 and e3 are numbers that have been calculated elsewhere

chips> expr = "imgout=(img1*{0}+img2*{2})+{3}".format(e1,e2,e3)
chips> dmimgcalc.operation = expr
chips> dmimgcalc.write_params()

will write the expression out to the dmimgcalc parameter file (eg try "!plist dmimgcalc" to see the result).

To write to a specific file, give a file name as the argument to write_params. A ".par" suffix will be appended if the filename does not contain one.

Updating to match a parameter file

The default values - and those used when the punlearn() method is used - are taken from the CIAO defaults. To update the values to match those from an on-disk file use the "read_params" method:

chips> dmimgcalc.read_params()

To read from a specific file, give a file name as the argument to write_params. A ".par" suffix will be appended if the filename does not contain one.

Changing the PFILES environment variable

The set_pfiles() routine changes the user portion of the PFILES environment variable. This controls where the tools look for ancillary parameter files (e.g. ardlib) and where the read_params and write_params methods default to when no explicit filename is given. See 'help set_pfiles' for more information.

DIFFERENCES TO CIAO TOOLS

No mode parameter

All tools are run with the mode parameter set to "hl". This is to avoid the tool trying to prompt you for missing parameter values, which would lead to it appear to hang as the prompt would not be seen by the user.

Missing parameters

As well as the mode parameters, there are a few tools which have a parameter that can only be set to a single value. These parameters are excluded from the interface.

Parameter values

Parameters should be set using the appropriate Python type, namely booleans, integers, floats or strings. For floating-point values, None is used to indicate the "INDEF" setting. None can be used for filenames and strings to indicate the value "". File names are given as strings, but you do not need to add extra quotes if it contains a space - so the following is valid

chips> dmstat("evt2.fits[sky=region(src.reg)][cols energy]")

Redirected values

There is limited support for the use of parameter redirects - the use of ")<parname>" to use the value from another parameter - but it only supports redirects to the same parameter file.

CHANGES IN THE NOVEMBER 2010 RELEASE

Updated tool list

The module now includes routines to run the chandra_repro, combine_spectra and make_psf_asymmetry_region tools.

Floating-point values

A bug has been fixed that could cause problems when parameter values were set to floating-point numbers.

Bugs

INDEF values

The use of None as a replacement for INDEF for numeric parameters has not been well tested.

See the bugs pages for an up-to-date listing of known bugs.


Last modified: November 2010