Last modified: 1 November 2022

URL: https://cxc.cfa.harvard.edu/ciao/releasenotes/ciao_1.1_release.html

CIAO 1.1 Release Notes


Contents:

  1. Introduction
  2. CIAO 1.1 tool list
  3. List of Tools with modified parameter files
  4. Tool/Application Notes
  5. Known Bugs/Issues

  1. Introduction

    Detailed specifications and implementation information on the UI software contributed to CIAO1.1 can be found on the team web page at http://hea-www.harvard.edu/uinfra/uimain.html.

    A well-configured CXCDS environment should be all that is required to use the analysis applications. Please check the notes from previous releases for information on features not explicitly covered below.

    Supported Platforms:
    Solaris 2.6, 7
    Redhat 5.2, 6.0
    Slackware 3.5

    >> we are working on Slackware 4.0

  2. CIAO 1.1 tool list

    The following is a list of applications and software tools included in CIAO1.1:

            Applications            sherpa
    				filterwindow
    				firstlook
    				chips
     				prism                   
            
            ACIS                    acis_format_events
    				acis_process_events
    				acis_sort_time
    
            HRC 			hrc_process_events
    
            Gratings  		tg_resolve_events
    				tg_create_mask
    				tgextract
    
            Datamodel               dmcopy		dmlist
    				dmsort		dmextract
    				dmappend	dmmerge
            			dmregrid	dmwritefef
    				dmhedit		dmmakepar
    				dmreadpar 	dmrega2fits
    				dmstat		dmimgcalc
    				dmarfadd	dmgti
    
            Detect                  celldetect
    				vtpdetect
    				wavdetect
    				tgdetect
    
            TCD                     csmooth
    				aconvolve
    
            Response                mkrmf
    				mkpsf
    				mkexpmap
    				mkinstmap
    				mkarf
    				mkgarf
    	        		asphist
    				asp_apply_sim
            
            Timing                  lightcurve
    
  3. Tools with modified Parameter files

    The following tools have had their parameter files modified in CIAO1.1. This means that when using CIAO1.1 the parameter files a user may have around from CIAO1.0 must be deleted in order for the new version to be accesed by the software.

    • dmlist - added verbose parameter
  4. Tool/Application Notes

    Data Model 
    ==========
    
    o Enhancements to the DM interface:
    
       - Handles ASCII region files produced by SAOTNG
      
         dmcopy evt.fits[sky=region(saoimage.reg)]  out.fits
    
       - Handles physical coordinates in FITS images created by dmcopy.
    
         If im.fits is a blocked IMAGE made by
           dmcopy "evt.fits[bin x=500:1080:4,y=400:1080:2]" im.fits
         then
           "im.fits[#1=100:120,#2=200:240]"
         is a filtered image with LOGICAL (blocked) pixels 100-120, 200-240, and
         "im.fits[x=900:1080,y=800:880]" is approximately the same filtered image, 
         since the use of "x" and "y" is taken to mean PHYSICAL (unblocked) pixels.
    
         Recall that in tables (eg event lists), there is no blocking: logical
         and physical coordinates are the same.
    
       - Improved recording of region filters in the output file's data subspace.
    
    o Bug fixes:
    
       - Bug fixed which caused wrong data subspace (GTI) for evt2 files
    
       - CONTINUE keyword was not handled correctly
      
       - Block numbering corrected for binned blocks
      
       - Fixed bug which caused keywords to be copied out of order.
      
       - Fixed bug with limited precision in subspace keywords
      
       - Fixed bug with intersecting data subspaces when ranges overlapping   
      
       - Amended order of new FITS CTYPE keywords to work around IRAF bug
      
       - Fixed bug which caused dmcopy to add multiple cases of
         the FITS literature reference and the long keyword warning
      
       - The default name of a binned image block has "_IMAGE" appended
         to the source table name.
      
       - Units added to GTIs.
      
       - TDIM handled correctly
      
       - External file filtering improved
      
       - Column selection on vector cols "[cols sky]" now works
      
       - Fixed bug in checksums for GTIs
      
       - Fixed bug losing coordinate info on binning event lists.
      
      
    User Interface - M Noble
    ========================
    
    o IMAGING
      Either SAOtng or DS9 may be used for imaging ($HOME/.CXCdefaults)
      Pending further sherpa testing, DS9 will become the default as of 
      this release.
    
    o CHIPS
      - Batch mode plotting now supported in CHiPS (no X display needed)
      - There is no longer any command/token overlap between the sherpa 
        and chips parsers.
      - All chips commands are now accessible via XPA, through the
        addition of an XPA server thread within the chips application.
      - New "type" command allows user to inspect data that has been loaded
        into CHiPS plots.
    
    o FIRSTLOOK
      - 1.0 beta version available w/in CIAO1.1, supporting most of
        of grating- and non-grating analysis threads.
      - Looks for unzipped versions of files when .gz cannot be found
      - Integrated tooltips (aka "help bubbles")
      - New "Help->About" functionality, with connection to README
    
    o FILTWIN
      - 1.0 beta version available w/in CIAO1.1, supporting a good deal
        of iterative filter recording/saving/updating.
      - Can communicate directly with ChIPS and imager, as well as
        receive commands from FirstLook.
      - New "Help->About" functionality, with connection to README
    
    o PRISM
      - Separate vector columns for plotting individual components
        against other scalar columns in table
      - Integrate tooltips (aka "help bubbles") and eliminate quickhelp.
    
    o TOOLAGENT    
      - Ability to stop tools after launching (from w/in GUI)
      - Updated for CIAO1.1 list of tools
      - Feedback to user when tool completes (from w/in GUI)
      - Context sensitive help, so that when tool is loaded into GUI
        param editor, clicking Help gives help specific to that tool
    
    o Online Help
      - Expanded online-help by inclusion of more documentation for
        sherpa, CHiPS, and various tools
      - Added support for cross-referencing and/or grouping related 
        CXCDS tools and/or application commands.
      - Look on local machine for browser before launching another to
        display help.
    
    o Portability
      - Extended portability to Redhat6.0 and Solaris2.7.  We now support
        five platforms in all, the entire suite of which are:
    
    		Solaris 2.6, 2.7
    		Redhat Linux 5.2, 6.0
    		Slackware Linux 3.5
    
      - More scripting has been implemented to support automated porting
        and packaging, including an experimental obsvis "subpackage".
    
    o Regression Testing
      The UI regression tests continue to improve, and may now be used 
      to test portable releases, FITS- and IRAF-file compatibility, several
      new visualization features, AND portions of sherpa functionality.
    
    
    Data Analysis Tools - KJG
    =========================
    
    o General (i.e in libraries):
    
      - header library 
        Updated to get TSTART and TSTOP from the GTI subspace
        and recompute ONTIME, LIVETIME and EXPOSURE when merging 
        multiple files.
    
      - Error Library
        This has had a fix applied to a bug that shows up in one of
        the Linux builds.  In addition many new error codes have been
        defined, and there is some new functionality for looking for
        errors in a list without copying them.
    
    o New development:
    
      - Exposure Map Software Suite
    
        The exposure map code suite consists of the following tools:
    
         1. mkinstmap   -- generate an instrument map
         2. mkexpmap    -- generate an exposure map
         3. mkarf       -- create an imaging ARF
         4. mkgarf      -- create a grating ARF
    
        Each of these tools require access to calibration data such as
        effective areas, grating efficiencies, detector QEs, bad pixel lists,
        etc.  These four programs access calibration data via ARDLIB, the
        analysis reference data library.
    
        With the exception of mkinstmap, these programs also require as input
        an aspect histogram (generated by asphist).  The histogram provides a
        condensed history of the telescope dither.
    
        mkinstmap is used by mkexpmap to generate an exposure map.  It is
        essentially the off-axis effective area function of the mirror
        projected onto a specified detector and multiplied by the detector's
        QE and bad pixel list.  It may be computed at a single energy or summed
        across a range of energies with user specified weights.
    
        mkexpmap transforms the instrument map to the sky and then moves it
        around on the sky in accordance with the aspect histogram.  The
        resulting object is called an exposure map.  Each cell of the exposure
        map represents the effective exposure given to that region of the sky.
      
        mkarf produces a type-I OGIP compatible ARF for a specified off-axis
        angle and detector/sky region.  (Regions will appear in the next
        release.  Until then, the user should selected events in a region big
        enough ensure that the PSF fraction is 1).
    
        mkgarf produces a type-I OGIP compatible ARF for grating analysis.
        The grating ARF is computed for a specified order, detector, and
        grating arm.  The dmaddarf program could be used to combine grating
        ARFS for different detector chips.
    
      - asp_apply_sim 
        This new tool applies the sim factors to the raw aspect offsets, then 
        writes out the results to an aspoff table.  The infile, which contains 
        the raw aspect offsets and the sim factors, was produced by the tool 
        'asp_calc_offsets'.
     
        The output columns include 'time x_offsets y_offsets roll_offsets stf_y 
        stf_z stf_roll', where the columns of 'time stf_y stf_z stf_roll' are 
        copied directly from the input file and the columns of 'x_offsets y_offsets 
        roll_offsets' are the aspect offsets with the sim factors. 
       
        Some header keywords, such as the detector name, the instrument name, 
        the roll_nom and the sim values, are required for the pixel library 
        to convert the chips coordinates to the focal plane coordinates or vice versa.
        
      - asphist
        This tool takes aspect offsets files for an observation and will create a 
        binned histogram of the aspect motion during the observation using the octree 
        algorithm. The output is a table of the duration of the bin and position of 
        the spacecraft during that duration. The GTIs are also appended to the output
        table as a subspace. 
      
      - dmarfadd
        Combine ARFs (Auxiliary Response Files) and will scale the spectral response 
        when energy bins between two files overlap. The scaling factor is based on the 
        exposure time. The user can only combine ARF of the same order and Grating. If
        there is no grating in place, there is no order restriction. The ARF must be on 
        the same energy grid. Positive and negative orders of the same magnitude can 
        be added.
      
          KNOWN LIMITATIONS: The maximum number of files that can be added is 8. 
      
      - dmimgcalc 
        This tool performs elementary arithmetic on pairs of images (or binned
        event lists).  Pixel-by-pixel addition, subtraction, multiplication,
        division, and comparison are supported.  This tool is presently very
        basic and is not as richly functional as other dm tools.
      
      - dmregrid:
        This new tool regrids each input image within the input dataset based on the 
        binning specification and the other parameters, which together specify the 
        offsets, minimum and maximum channels, bin scales, rotation angle, and center 
        of rotation.  Each of these parameters can either be a stack or a single 
        value; if a single value is given, it is applied to each input file.  For a 
        stack of input files, the sum of the regridded images will be output to the 
        output file.
      
          KNOWN BUG : If the binning specification is given by a stack in which the 
     	          second (or later)  item results in a greater number of output 
    	          pixels than the first binning specification, a segmentation fault 
    	          results.  The fix for this bug will appear in the next release/patch.
      
      - lightcurve 
        Bin source and background events into specified time intervals and 
        computes the net count rate for each bin. 
      
        The purpose of the lightcurve tool is to bin source and background events 
        into specified time intervals and computes the net count rate for each bin.  
        The results computed by lightcurve can then be used to plot light curve of 
        the source.  lightcurve has the same functionalities as that of ltcurv in 
        PROS.  However, lightcurve has one new binning option besides letting the 
        user select the total number of bins, or the bin length.  This new binning 
        option allows the user to set the average number of counts per bin, and then 
        the program will set the total number of bins and bin length accordingly.  
        
          LIMITATIONS:
            o The input file must be a time-sorted event file.
            o The output file is in fits format only.
      
      - mkpsf
        The mkpsf program is one of the Point Spread Function (PSF) tools.
      
        The program retrieves an image from a FITS format library
        file based on information entered by the user and contained in
        the input file.
      
        The program retrieves an image based on energy, x and y
        coordinates specified by the user, bins it according to the input file,
        rotates it based on roll provided in the input file, and saves it
        in a FITS format image file.
      
      - mkrmf 
        Generate 2-D (at present) response matrix file (RMF) based on  
        a FITS embedded redistribution function file (i.e, ECD table) and with 
        inputs of MATRIX grid specification (i.e. intrinsic photon energy grids 
        and PHA channels). The output MATRIX/EBOUND FITS file is
        encoded on-disk in format of "legacy" or CXC.
      
        1> Changed the tool name from rmfcalc to mkrmf
        2> Made low_thresh parameter (in mkrmf.par) visible to user
        3> Added an offset header keyword in EBOUND extension which is
           extracted from the input ecd-table FITS file
        4> Updated pha (or pi) bin method in linear space for a bin step
           step = (max-min+1)/num_bin
        5> Retained energy bin method the same, or
           step = (max-min)/(num_bin-1)
        6> min/max/num_bin in pha axis must make the bin step an integer,
             otherwise mkrmf exists with error
        7> the pha grids must be binned linearly otherwise mkrmf exits with
           error
    
    o Updates to Existing Tools:
      
      - dmlist, dmcopy 
        small changes have been made for CIAO 1.1, although the internals have been 
        substantially improved and there have been a lot of minor bug fixes.
      
      - dmreadpar
        Fixed a bug in the template code that caused an infinite loop to 
        be entered.
      
      - celldetect 
        Cell placement algorithm has changed slightly so that results of
        celldetect will not vary with different region filters.  The new
        results are likely to be (slightly) different from earlier results.
        Bug 1562 has been fixed, which caused celldetect to exit with no error
        message when the block specifier (eg: "[EVENTS]") for an event list
        was omitted.
      
      - mtl_build_gti
        The limits syntax has been updated to be more user friendly, both in 
        referencing columns and defining new columns.  A new parameter, userlimit, 
        was added, and the output MTL is optional.
      
      - tgdetect 
        Because tgdetect uses celldetect, it is, in principle affected by the
        celldetect change.  However, no differences in results have yet been
        noted.
      
      - tgextract
        1> Fixed the checksum error bug.
        2> Changed the binning order when the bin unit is Angstrom, eV or keV.  
           Now, the output is in desending order of wavelength if bin unit
           is Angstrom, and the output is in ascending order of energy if the 
           bin unit is in eV or keV.
        3> Added X, Y columns (these are the x,y coordinates of the center of 
           the sources) in the output spectrum.  As a result, the input files to
           tgextract must have a region extension that provides the x,
           y coordinate of the center of the sources.
        4> Replaced the output BACKGROUND column with BACKGROUND_UP and 
           BACKGROUND_DOWN.
        5> Redefined the equation for calculating the value of header keword 
           BACKSCAL.
        Note that there are no changes in the interfaces.
    
    o SHERPA New Functionality and Bug Fixes
    
      1> PSFFROMFILE function can be used now as a 2D source model (i.e. the
         ability to fit psf profiles to sources to determine if they are
         point-like or extended) or an instrument model.  The choice of sliding
         cell convolve or FFT convolve is given to the user.
    
      2> Fakeit has been added to allow simulations of 1D spectra.
    
      3> Powell algorithm have been greatly revised and improved.
    
      4> Grid method now automatically determines grid points in log space
         if the linear space grid spacing is too large, and issues a warning to
         the user that this is being done.
    
      5> Guessing an initial parameters ranges was improved to avoid working
         within too small parameter space. We still advice to always use at
         least two optimization methods to confirm the final results.  (see
         Optimization section of the Manual for advice on fitting methods).
    
      6> A bug in the uncertainty algorithm has been fixed.  Also the user
         will see warning if the uncertainty algorithm finds a better minimum.
    
      7> The confidence ranges can be displayed.
    
      8> A problem with automatic replacement of the instrument model files
         when the new pha file is read into Sherpa  has now been fixed.
    
      9> Some plotting commands have been changed to differentiate between
         CHIPS and Sherpa commands (check the Manual for details). Note "lplot"
         replaces "plot" command.
    
  5. Known Bugs/Know Limitations

      
    1. aconvolve:
    
       The center parameter doesn't work properly. 
       When the user selects the option center=yes, the fft of the data 
       (as well as the kernel) shows the maximum response in the corner, 
       not at the center.
    
    2. csmooth and aconvolve:
    
       The FFT code (for convolution) in some cases for axes with odd number 
       runs into a segmentation fault.
       The work around is to crop the image to a slightly smaller 
       size.
       
       For example for an image infile.fits with 726x751 pixels size that produced a  
       segmentation fault the user can crop the image to 725x750 pixels and then rubn
       csmooth, or he/she could run csmooth on a cropped image directly:
    
    	csmooth infile="infile.fits[1:725,1:750]"
    
    3. dmarfadd
    
       The maximum number of files that can be added is 8.
     
    4. dmregrid
    
       There is a known bug in dmregrid.  If the binning specification is given by 
       a stack in which the second (or later)  item results in a greater number of 
       output pixels than the first binning specification, a segmentation fault 
       results.  The fix for this bug will appear in the next release/patch.
    
    5. lightcurve
    
          LIMITATIONS:
            o The input file must be a time-sorted event file.
            o The output file is in fits format only.
    
    6. Data Model filtering
    
       Numbers in filters can't end with a decimal point
       [x=15.]  fails, use [x=15.0] or [x=15] instead.
    
    =========================================
      SHERPA  known bugs
    =========================================
    
    1> Model definition:
    --------------------
    
      a> Two dimensional models used in fitting binned data use the model
         value from the central position on the bin. These models are not
         integrated over the bin. Therefore for rapidly changing model over the
         bin the result can be incorrect. Note also that the result will
         strongly depend on the binning applied to the data. (Bug 1559)
    
      b> If there is no data or the filter ignores the entire data set (ignore all)
         then this data set is empty and any source definition causes Floating
         exception error and crash. (Bug 1733)
    
    
    2> PSFFROMFILE:
    ---------------
    
      For PSFFROMFILE to work on two-dimensional images, one cannot apply region
      filters within Sherpa.  Such filters should be applied outside of Sherpa
      (e.g. with DataModel) before Sherpa analysis. (Bug 1677)
    
    
    3> Known problems with the Image display in Sherpa:
    ---------------------------------------------------
    
      a> The displayed images from Sherpa have the Y axis flipped (Bug 1603)
    
      b> Composite filters (adding or subtracting regions) entered
         interactively using the Imager are not applied correctly to the
         data. The user should use the command line input if the aplied filter
         is composed of 2 or more filter regions. (Bug 1604)
      
      c> "Image source" when the source model is PSFFROMFILE causes
         Segmentation Fault. (Bug 1732)
    
    
    4> Plotting problems:
    ---------------------
    
      a> Updated plotting definitions are not remembered after the new type
         of plot is being introduce. For example "lplot data" "lplot data"
         remembers the modification made to the first plot, but "lplot data"
         "lplot fit" does not remember the modifications to the data plot.  In
         the order: "lplot data" "lplot fit" "lplot data" the last "lplot data"
         command also does not remember any modification to the first "lplot
         data", due to fit plot made in between.  (Bug 1040)
         
      b> The axes of the multiple plots are not aligned automatically. (Bug 1612)
    
    ===========================================================================
    UI Known bugs  12/16/99
    ------------------------
    
    GENERAL
    -------
    1597: There are colormap issues with GUI applications that may lead to
    the error message 
    
    1629: ExitAll does not exit sherpa 
    
    PRISM
    -----
    1530: PRISM shows header keywords in a different format than dmlist 
    
    1540: In PRISM, the TWM window maneger does not support Motif window
    titlebar
    
    Example: expand a column data in PRISM, and note that when
    running under TWM the absence of a title bar prevents the user from
    knowing just what column is being viewed (because the window titlebar
    is being used to display the column name, as well as its
    vectorness/arrayness).
    
    1541: the GUI status/message area cannot ben cleaned out from old messages
    
    1574: PRISM cannot generate multiple plot draw areas if multiple rows
    are selected for plotting
    
    1616: PRISM does not have "editing" capability
    
    
    1656: In PRISM the expand window "seems" to allow edits of data matrix
    but in fact does not
    
    1721: Prism crashes if a block is reopened after expanding vector cols
    
    PRISM crashes if you reopen a block while a vector column is expanded.
    Occassional prism gets into a state where when you try to expand the
    SRC_POS column you get both vector column dialog box and the "column
    is not expandable" warning dialog.
    
    1731: When running sherpa, 1) launching imager, 2) suspending sherpa
    to get a unix prompt, 3) running prism, prism will not come up.
    
    1757: The TSTART/TSTOP keyword are rounded in the header of files seen
    through PRISM. Workaround: use >dmlist file header
    
    FIRSTLOOK
    ---------
    1709 OIF tree window is sometimes hidden
    
    The OIF tree window sometimes opens up BEHIND the firstlook window (at
    least under CDE).
    
    1754: The "STOP" button in First look is not working
    
    FILTWIN
    -------
    1751 : filtwin: File menu could be clearer
    
    The File->Restore menu option in FiltWin sometimes confuses users into
    thinking they can use it to load arbitrary data files into FiltWin.
    This is not so, as the menu is intended to be used to restore
    previously saved "saved filter state," ie the filters described w/in
    the filtwin display during a previous "File->Save filter description"
    operation.
    
    CHIPS
    -----
    1528: CHIPS window can be killed by process that didn't create it.
    
    Example:
    Prism  -> plot two columns -> Chips window comes up.
    Then run filter window, and then immediately exit it. 
    When filter window dies, it takes CHiPs window with it.
    
    1613: log scale ticks in chips
    
    While using sherpa with the x scale set to log, the command: "ticks x
    0.0002" generated the following error: Small tick spacing must be <=
    big spacing When the scale was changed to linear linear, the command
    worked but the results were poor. The curve disappeared from the
    screen and 3 sets of overplotted tickmarks appeared.
    
    Further comments on the bug: In the majority of situations, log ticks
    are OK.  It is only in specific situations where possibly zero or only
    one tick is displayed.  Axes with range/domain like (1,10) or
    (1,10000), etc work fine.  Odd situations like axes (logarithmic
    obviously) with a range/domain of (1.7, 2.9) or (90000,90010) often
    display few or no ticks.  This is a difficult problem to solve in SM.
    Unless I can get help with an algorithm to figure out ticksize values
    to feed into SM, this bug cannot be fixed.  A query to the smusers
    mail group went unanswered.
    
    1744: Exiting chips kills ds9, even when ds9 was launched as a
    separate application
    
    TOOLAGENT
    ---------
    1754 : INDEF not supported by parameter interface gui
    
    Bug Description : The gui parameter interface does not support INDEF
    as a value, as a result tools like csmooth do not produce correct
    results when launched from Toolagent (the INDEF parameters get viewed
    and passed to cxcwrapper as zeros instead of indefs).  Workaround: run
    the tool from the command line
    
    IMAGER
    -------
    1610: Interrution of Sherpa by CTRL-C kills the imager as well