Converting SED Data to a Supported Format

The SED Importer tool which is bundled with the Iris software package allows you to convert your SED data written in an unsupported format into an IVOA-compliant and Iris-compatibile FITS or VOTable format. You may upload your SED data from a binary or text-based file on your local disk; from a URL address (http or ftp); from the NASA Extragalactic Database (NED), based on a query on target name; or even transmit data from a remotely connected application, such as TOPCAT. Some examples of data file formats which are considered non-standard within the context of Iris, are CSV, ASCII, and certain instances of FITS and VOTable (those which do not conform to the IVOA standard and are not compatible with common VO tools).

When you load SED data into the SED Importer from a file written in a format which is not supported by Iris, you are prompted to enter a few key pieces of information about the data format so that the SED Importer is able to convert it into a form which is recognized by VO tools. The resulting configuration information may be saved to file, separately from the converted SED file, to be used later in the non-interactive command-line interface of the tool (e.g., to quickly convert a batch of files). If you happen to load a file into the SED Importer which is already in the supported FITS or VOTable format - e.g., to build an aggregate SED from a mix of segments in unsupported and supported formats - this extra step of entering configuration information is unnecessary.

Supported Input File Formats

The file formats supported by the SED Importer, listed below with brief descriptions, are those which are supported by common VO tools. For each, a simple, generic assumption is made: data is arranged in a tabular format, with all rows having the same number of columns, and possibly with a header where metadata is stored. The way in which both data and metadata are stored depends on the specific format.

For detailed information on these formats, refer to the the "Extended Information" section of this document.

• ASCII - text file with columns separated by spaces and/or tabs

• CSV - text file with columns separated by commas (the first row may contain the name of the columns)

• FITS - consists of a series of Header Data Units (HDUs), each containing two components: an ASCII text header and the binary data. The header contains a series of header keywords that describe the data in a particular HDU and the data component immediately follows the header.

• VOTABLE - (text or binary) XML standard for the interchange of data represented as a set of tables. Consists of an unordered set of rows, each of a uniform structure, as specified in the table metadata. Each row in a table is a sequence of table cells, and each of these contains either a primitive data type, or an array of such primitives.

• IPAC - a custom bar-separated text format by IPAC

• TST - Tab Separated Table (comments are ignored, metadata is in key, value pairs)

Interoperability with SAMP

In the lower-left corner of the SED Importer desktop there is an icon that shows the status of the Simple Application Messaging Protocol (SAMP) connection, which the tool uses to communicate with Iris.

SAMP is a Virtual Observatory protocol that allows remote desktop applications to communicate with each other. If you use other SAMP-enabled Virtual Observatory applications that manipulate tables of astronomical data, such as Topcat or Aladin, you can transmit tables of data from these external applications to the SED Importer.

The main mode of interaction with the SED Importer is through the Graphical User Interface (GUI), but a non-interactive, command-line interface (CLI) is also available for advanced users. In order to use the SED Importer CLI, what is referred to in this document as a 'setup' file is required input, along with the file to be converted. The setup file stores the configuration information needed to convert SED data written in an unsupported format into one of the supported FITS or VOTable formats; it may be created by using the SED Importer in interactive mode, first.

To open the SED Importer GUI, simply start the tool on the Unix command line by typing the full path to the SED Importer script in the Iris installation directory, as shown below:

% <basedir>/iris-beta-2.5-<plat>-<arch>/SedImporter &

This opens the desktop interface which is available for using the tool in interactive mode. The desktop contains links to help documentation for both the Iris and SED Importer GUIs - which run as separate applications in the current release of Iris - as well as icons that launch the respective applications.

Most of the windows launched by the SED Importer will be confined within this desktop. Each window can be iconified and its icon will stay in the SED Importer desktop itself, so that it will not take up space in your applications bar.

The SED Importer desktop takes on the same look and feel as your native desktop, and the behaviour is consistent with your native environment. For example, on Mac OS X, iconified windows appear as miniatures at the bottom-center of the desktop, while on Linux, they are draggable buttons placed on the bottom-left of the desktop. The SED Importer desktop can itself be resized, maximized and iconified in your native desktop.

The "Load SED" icon on the SED Importer desktop is the entry point into the tool.

Upon selecting this option, you are prompted to provide an ID for the new SED which you will create, then a SED buider window opens in which you may load a spectroscopic segment using the "New Segment(s)" option. Data may be loaded from a file on your local disk; an http or ftp URL address; from the NED SED service (internet connection required for this option); or from a remotely connected application. A "segment" refers to a data spectrum, a photometric point, or a series of photometric points; see the Iris FAQ entry "Which SED data types are supported by Iris?" for more details.

An arbitrary number of data segments may be loaded into the SED Importer using the "New Segment" function. You are provided with three options for loading data: browsing for a file on your local disk, entering a URL file location, or searching NED for SED data associated with the entered target name.

To transmit data to the SED Importer from a remotely connected application, such as Topcat, simply broadcast the table from the external application (or send it directly to the SED Importer), and you will be provided with the same options as if you had loaded data using one of the standard options listed above. Note that currently, you are not able to transmit data from the SED Importer to a remote application, as the "Broadcast SED" option in the this SED builder window is disabled; it will be available in a future release of the software, for broadcasting the SED to all SAMP-enabled applications that are currently running on the user desktop.)

In the SED builder window, you have the option of entering a target name and coordinates into the Target Info field, to be associated with the new SED and recorded in the metadata of the saved SED file. If this information is provided and an internet connection is available, the name is resolved and the associated coordinate fields are automatically populated. (Note that if you have entered a target name in to the Target Info field, this name will be offered as a suggestion whenever you choose to upload a data segment using the NED SED web service option.)

A SED builder window with example target information entered is shown below.

After loading a data segment into the SED Importer from a file in an unsupported format - e.g., using the URL option with an HTTP file location entered as "http://cxc.cfa.harvard.edu/csc/temp/sed/threads/importer/3c273_hut.ascii" - another interactive window opens, labeled "Import Setup Frame". In this window, you must enter various pieces of information which will be used to define the configuration for the format conversion for this particular data segment (i.e., so that the tool knows how to convert the data from the unsupported format to a supported one). The "Save Setup" button beneath the "Setup Help" window is available for those who wish to write this configuration to file for use with the SED Importer CLI, described in the "Advanced Usage" section below.

The fields in the Import Setup Frame window which require entries are the X Axis, Y Axis, and Y Error fields. Until these fields are populated, the Setup Help window will contain warning messages indicating that the form is incomplete, and you will not be able to save the current setup to a file or import new segments into the SED.

In the X Axis section, you must characterize the column in your file which corresponds to the spectral coordinate axis, e.g., wavelength in Angstrom units, frequency in Hz, or energy in eV. In the Column drop-down menu, you will find the name of the columns as they are in your file. If no column names can be found, then "colN" will be used, where N is the number of the column as it appears in the file.

In the Y Axis section, you are to characterize the column in your file which corresponds to the flux density axis, e.g., energy flux density in ergs/s/cm2/Hz, photon flux density in photons/s/cm2/Hz, or the AB magnitude equivalent of the flux density.

In the Y Error section, you can characterize the error for the Y Axis using several different options:

In the remaining fields of the Import Setup Frame window, you can view the name you assigned to the SED along with the name and path of the associated file; as well as view and edit the target information and data publisher (e.g., "NED") which was optionally entered in the SED builder window, or add this information here if it was not done previously.

After entering the required configuration information into the setup window, the "Add Segment to SED" and "Save Setup" buttons become active.

At this point you can optionally save the configuration to a text file for later use of the tool in non-interactive mode, as well as add the configured segment to the new SED. Adding the segment closes the "Import Setup Frame" window and brings you back to the interactive window which is labeled with your SED ID and contains general information about all SED segments loaded in the session thus far.

Here, you may continue loading data segments from various locations, and repeat the process outlined above until you are finished building your SED. The list of loaded segments is shown with associated coordinates, publisher information, and the number of points in the segment, if this information is available. You can select one or more segments in this list and the relevant buttons in the Segments Operations section to perform tasks on the selected segments.

As the SED Importer has the flexibility to read data files in a variety of formats and from different locations, you are able to use the tool to gather multiple SED data segments and save them together as an aggregate SED, in a single session, and save to a single FITS or VOTable format file.

In order to build a multi-segment SED - where each segment may be converted using a different configuration - you can load a second segment using the "New Segment" option in the SED builder window, using any of the file upload options described in the "Importing Data" section, above. For example, if the first segment had been loaded using the "URL" option, the second segment could be loaded from the NED SED Service by entering the object name "3c273" into the appropiate field of the data import window.

In this case, the segment would be directly added to the SED, without prompting you to enter a separate configuration for the segment in the Import Setup Frame window, because this step is not required for data loaded from NED. However, you are prompted to do so when loading data from file, from URL, or from a remotely connected client.

When you have finished importing and configuring data segments within the SED Importer GUI, and optionally saving the associated setup file(s) for later use in the SED Importer CLI, you may write the new SED to an Iris-compatible FITS or VOTable format file by selecting "Save SED". The converted SED data file is now in an Iris-compatible format and therefore may be loaded into the Iris GUI for SED analysis.

Once you have used the SED Importer to convert your SED data into an IVAO-compliant FITS or VOTable format, you may load the converted file into Iris for analysis. The "Launch Iris" icon on the SED Importer desktop is available as a shortcut to the Iris GUI, which runs separately from the SED Importer GUI in the current release.

This is identical to launching Iris from the command line in the usual way:

% <basedir>/iris-beta-2.5-<plat>-<arch>/Iris &

Once open, the 'Read from File' option in the Iris File menu may be used to load the converted SED data file.

Two lifebuoy shaped icons on the SED Importer desktop will point your default browser to the documentation pages for both the SED Importer (this page) and Iris (the other components of the Iris How-to Guide). If the program cannot open these links in your default browser, a simple browser should appear in the SED Importer Desktop itself (note that formatting errors may result, in this case).

Creating a Setup File
Using a Setup File
Editing a Setup File

The SED Importer may be run non-interactively from the Unix command line, using both the data file to be converted, and a setup file containing the configuration information for the conversion. The setup file must be created using the SED Importer interactively, first, according to the procedure described in the section "Entering the Conversion Configuration", above.

Creating a Setup File for the CLI

A SED Importer setup file is a text file which records the configuration information used by the tool to convert from the user-input unsupported format to a supported VOTable or FITS data format. Its intended use is to automate the file conversion procedure in scripting. It may be created and saved within the "Import Setup Frame" window of the SED Importer GUI.

The full set of instructions for creating a setup file is provided in the "Enter Conversion Configuration" section, above; the basic steps are:

• Start a new SED building sesssion by clicking on the "Load SED" icon in the SED Importer desktop interface.

• Assign it a label when prompted and then click the "New Segment(s)" button in the SED builder window to load data from a disk, URL, the NED SED web service, or a remotely connected application.

• In the "Import Setup Frame" window of the SED Importer GUI, enter the spectral coordinate and flux density characterization of the loaded data so that the tool can make the conversion to a supported data format.

• Save the configuration to a setup file using the "Save Setup" option.

Using a Setup File

To run the SED Importer from the command line using a newly created setup file as input, the following arguments must be provided, in the order shown:

• config_file - The setup file which contains the importing configuration(s), which was created using the SED Importer in interactive mode
• output_file - The output file that will contain the new SED.
• format - The format in which the new SED file must be written. Choose between 'vot' and 'fits'. If omitted, the default will be 'vot'.

For example:

% <basedir>/iris-beta-2.5-<plat>-<arch>/SedImporter config_file.ini outputfile.vot vot

The setup file created in the SED Importer GUI allows you to quickly convert data in a given unsupported format into one of those supported by Iris. This is particularly useful when you have a long list of files in the same unsupported format which you need to convert and analyze in Iris: instead of loading each file into the GUI and unnecessarily re-creating the same conversion configuration, you can simply use the one setup file you already created to convert a batch of files on the command line.

Note that a setup file output by the GUI may also be loaded into the GUI, using the "Load from Setup" option in the File menu. This is useful when you are building a multi-segment, aggregate SED in the GUI, and would like to contribute a segment for which you have already created a conversion configuration, in a previous session.

Editing a SED Importer Setup File

The contents and format of a SED Importer setup file are described in detail in this section, so that you may learn how to edit the file and customize the configuration to suit your needs, independently of the GUI.

Setup File Format

The output setup file looks like a Windows ini file or a MySQL configuration file. It is organized into sections, with each section representing a separate data segment. This means that in a single setup file you can include many segments from many different files.

Each section has a title between squared brackets, e.g. "[Segment1]". Titles are not used when the file is processed, but different titles mark different segments; this means that two sections with the same title would refer to the same segment. While it is allowed to fragment information in different sections it is not wise to do so because if you include the same information more than once in different subsections, the result may become unpredictable.

Beside the title, all the information is expressed in key/value pairs: the key and the value are on the same line and they are separated by the character "=", e.g., "XAxisColumnNumber = 5"; the order does not matter.

Decimal numbers may be represented in scientific notation, e.g., 5.5E-7.

Setup File Contents

The contents of an SED Importer setup file is shown below, where there is a field to specify the location of the input file to be converted, as well as various other fields for specifying the configuration of this input file.

[Segment0]
XAxisColumnNumber = 5
XAxisQuantity = FREQUENCY
XAxisUnit = HERTZ
YAxisColumnNumber = 6
YAxisQuantity = SPVFLUXDENSITY
YAxisUnit = FLUXDENSITYFREQ0
constantErrorValue = 2.0
errorType = ConstantValue
fileLocation = file:/Users/data/3c273.csv
formatName = CSV
publisher = UNKNOWN
targetDec = 2.05238729
targetName = 3c273
targetRa = 187.27791798

The fields of the setup file are defined below.

targetName - A string representing the name of the object this segment
             belongs to, e.g. 3c273

targetRa - The Right Ascension of this segment in decimal degrees
           (double), e.g. 187.27791798

targetDec - The Declination of this segment in decimal degrees
            (double), e.g. 2.05238729

publisher - A string representing the data curator of this segment.

fileLocation - A URL pointing to the actual location of the file. If
               it is a local file, the absolute path of the file must
               be preceded by the protocol file:, e.g.,
               file:/User/data/3c273.csv

formatName - the name of the file format which has to be used for
             reading the file. The string must be chosen among these ones:

             VOTABLE
             CSV
             FITS
             ASCIITABLE
             IPAC
             TST

XAxisColumnNumber - an integer representing the column position in the
                    file, where 0 represents the first column.

XAxisQuantity - a string representing the spectral quantity of this
                segment, among:
  
                FREQUENCY
                WAVELENGTH
                ENERGY

XAxisUnit - a string representing the X Axis units. The units have to
            be consistent with the Axis quantity, i.e.:
           
            FREQUENCY: HERTZ, KHZ, MHZ, GHZ.
            WAVELENGTH: ANGSTROM, CM, M, MICRON, NM.
            ENERGY: EV, KEV, MEV.

YAxisColumnNumber - an integer representing the column position in the
                    file, where 0 represents the first column.

YAxisQuantity - a string representing the spectral quantity of this
                segment, among:

                SPVFLUXDENSITY (Flux Density)
                SPVMAGNITUDE (Magnitude)
                SPVPHOTONFLUXDENSITY (Photon Flux Density)

YAxisUnit - a string representing the X Axis units. The units have to
            be consistent with the Axis quantity. In the following
            table you will find the strings of the units that are
            supported and consistent with each quantity; where
            applicable, the unit string is indicated. Notice that you
            do not have to include the unit string but the corresponding
            label (e.g. FLUXDENSITYFREQ0):
   
 SPVFLUXDENSITY:
    FLUXDENSITYFREQ0: erg/s/cm2/Hz
    FLUXDENSITYFREQ1: Jy
    FLUXDENSITYFREQ2: Watt/m2/Hz
    FLUXDENSITYWL0: erg/s/cm2/Angstrom
    FLUXDENSITYWL1: Watt/m2/um (micron)
    FLUXDENSITYWL2: Jy-Hz
 
 SPVMAGNITUDE:
    ABMAG
    STMAG
    OBMAG
 
 SPVPHOTONFLUXDENSITY:
    PHOTONFLUXDENSITY0: photon/s/cm2/Hz 
    PHOTONFLUXDENSITY1: photon/s/cm2/Angstrom

errorType: the type of the error that characterize the Y Axis, among
           the following:

  ConstantValue
  SymmetricColumn
  SymmetricParameter
  AsymmetricColumn
  AsymmetricParameter

 ConstantValue:
   constantErrorValue: a decimal number representing the value of the
   error, e.g. 0.2

 
 SymmetricColumn:
   symmetricErrorColumnNumber: an integer representing the position of
   the error column in the file, where the first column is in position 0.

 
 AsymmetricColumn:
  - lowerErrorColumnNumber: an integer representing the position of
    the lower error column in the file, where the first column is in position 0
 
  - upperErrorColumnNumber: an integer representing the position of
    the lower error column in the file, where the first column is in position 0

 
 SymmetricParameter:
  - symmetricErrorParameter: the name of the parameter in the file
    header that contains the value of the error for all the points in the file.

 
 AsymmetricParameter:
  - lowerErrorParameter: the name of the parameter in the file header
    that contains the value of the lower error for all the points in the file.
 
  - upperErrorParameter: the name of the parameter in the file header
    that contains the value of the upper error for all the points in the file.

Since the SedImporter uses the STIL library to read files, STIL being the library on the top of which TOPCAT is implements, the following documentation about the detailed specification of the supported formats is derived from the TOPCAT documentation (http://www.star.bris.ac.uk/~mbt/topcat/sun253/inFormats.html)

FITS

FITS binary and ASCII table extensions can be read. Currently, only the first extension (first HDU after the primary HDU) of the file is imported. For normal FITS files, header cards in the table's HDU header will be made available as table parameters. Only header cards which are not used to specify the table format itself are visible as parameters (e.g. NAXIS, TTYPE* etc cards are not). HISTORY and COMMENT cards are run together as one multi-line value.

VOTable

VOTable is an XML-based format for tabular data endorsed by the International Virtual Observatory Alliance; while the tabular data which can be encoded is by design close to what FITS allows, it provides for much richer encoding of structure and metadata. SedImporter should be able to read any table which conforms to the VOTable 1.0, 1.1 or 1.2 specifications. This includes tables in which the cell data are included in-line as XML elements (VOTable/TABLEDATA format), or included/referenced as a FITS table (VOTable/FITS) or included/referenced as a raw binary stream (VOTable/BINARY). STIL does not attempt to be fussy about input VOTable documents, and it will have a good go at reading VOTables which violate the standards in various ways. If the VOTable contains more than one table, the current version of the SedImporter will read only the first one.

ASCII Table

In many cases tables are stored in some sort of unstructured plain text format, with cells separated by spaces or some other delimiters. There is a wide variety of such formats depending on what delimiters are used, how columns are identified, whether blank values are permitted and so on. It is impossible to cope with them all, but TOPCAT attempts to make a good guess about how to interpret a given ASCII file as a table, which in many cases is successful. In particular, if you just have columns of numbers separated by something that looks like spaces, you should be just fine.

Here are the detailed rules for how the ASCII-format tables are interpreted:

 - Bytes in the file are interpreted as ASCII characters.

 - Each table row is represented by a single line of text.

 - Lines are terminated by one or more contiguous line termination
   characters: line feed (0x0A) or carriage return (0x0D).

 - Within a line, fields are separated by one or more whitespace
   characters: space (" ") or tab (0x09).

 - A field is either an unquoted sequence of non-whitespace
   characters, or a sequence of non-newline characters between matching
   single (') or double ('') quote characters - spaces are therefore
   allowed in quoted fields.

 - Within a quoted field, whitespace characters are permitted and are
   treated literally.

 - Within a quoted field, any character preceded by a backslash
   character ("\") is treated literally. This allows quote characters
   to appear within a quoted string.

 - An empty quoted string (two adjacent quotes) or the string "null" 
   (unquoted) represents the null value

 - All data lines must contain the same number of fields (this is the
   number of columns in the table)

 - The data type of a column is guessed according to the fields that
   appear in the table. If all the fields in one column can be parsed as 
   integers (or null values), then that column will turn into an
   integer-type column. The types that are tried, in order of
   preference, are: Boolean, Short Integer, Long, Float, Double, String

 - Empty lines are ignored.

 - Anything after a hash character "#" (except one in a quoted string)
   on a line is ignored as far as table data goes; any line which
   starts with a "!" is also ignored. However, lines which start with
   a "#" or "!" at the start of the table (before any data lines) will
   be interpreted as metadata as follows:

   The last "#"/"!"-starting line before the first data line may
   contain the column names. If it has the same number of fields as
   there are columns in the table, each field will be taken to be the
   title of the corresponding column. Otherwise, it will be taken as a
   normal comment line.

   Any comment lines before the first data line not covered by the
   above will be concatenated to form the "description" parameter of
   the table.

    #
    # Here is a list of some animals.
    #
    # RECNO  SPECIES         NAME         LEGS   HEIGHT/m
      1      pig             "Pigling Bland"  4  0.8
      2      cow             Daisy        4      2
      3      goldfish        Dobbin       ""     0.05
      4      ant             ""           6      0.001
      5      ant             ""           6      0.001
      6      ant             ''           6      0.001
      7      "queen ant"     'Ma\'am'     6      2e-3
      8      human           "Mark"       2      1.8

In this case it will identify the following columns:

    Name       Type
   
    RECNO      Short
    SPECIES    String
    NAME       String
    LEGS       Short
    HEIGHT/m   Float

It will also use the text "Here is a list of some animals" as the Description parameter of the table. Without any of the comment lines, it would still interpret the table, but the columns would be given the names col1..col5.

If you understand the format of your files but they do not exactly match the criteria above, a good approach to take is to write a simple, free-standing program or script which will convert them into the format described here; you may find Perl or awk suitable languages to use for this purpose.

IPAC

The Infrared Processing and Analysis Center at CalTech uses a text-based format for storage of tabular data, defined here. Tables can store column name, type, units and null values, as well as table parameters. They typically have a filename extension ".tbl" and are used for Spitzer data amongst other things. An example looks like this:

   \title='Animals'
   \ This is a table with some animals in it.
   |   RECNO |    SPECIES |          NAME |    LEGS | HEIGHT   |
   |    char |       char |          char |     int | double   |
   |         |            |               |         | m        |
   |         |            |          null |         |          |
            1          pig   Pigling Bland         4        0.8
            2          cow           Daisy         4          2
            3     goldfish          Dobbin         0       0.05
            4          ant            null         6      0.001

Comma-Separated Values

Comma-separated value ("CSV") format is a common semi-standard text-based format in which fields are delimited by commas. Spreadsheets and databases are often able to export data in some variant of it. The intention is that TOPCAT can read tables in the version of the format spoken by MS Excel amongst other applications, though the documentation on which it was based was not obtained directly from Microsoft.

The rules for data which it understands are as follows:

 - Each row must have the same number of comma-separated fields.

 - Whitespace (space or tab) adjacent to a comma is ignored.

 - Adjacent commas, or a comma at the start or end of a line
  (whitespace apart) indicates a null field.

 - Lines are terminated by any sequence of carriage-return or newline 
   characters ('\r' or '\n') (a corollary of this is that blank lines
   are ignored).

 - Cells may be enclosed in double quotes; quoted values may contain
   linebreaks (or any other character); a double quote character
   within a quoted value is represented by two adjacent double quotes.

 - The first line may be a header line containing column names rather
   than a row of data. Exactly the same syntactic rules are followed for
   such a row as for data rows.

TST

Tab-Separated Table, or TST, is a text-based table format used by a number of astronomical tools including Starlink's GAIA and ESO's SkyCat on which it is based. A definition of the format can be found in Starlink Software Note 75. The implementation here ignores all comment lines: special comments such as the "#column-units:" are not processed.

An example looks like this:

    Simple TST example; stellar photometry catalogue.

    A.C. Davenhall (Edinburgh) 26/7/00.

    Catalogue of U,B,V colours.
    UBV photometry from Mount Pumpkin Observatory, 
    see Sage, Rosemary and Thyme (1988). 

    # Start of parameter definitions. 
    EQUINOX: J2000.0 
    EPOCH: J1996.35

    id_col: -1
    ra_col: 0
    dec_col: 1

    # End of parameter definitions.
    ra<tab>dec<tab>V<tab>B_V<tab>U_B
     <tab> <tab> <tab> <tab>
    5:09:08.7<tab> -8:45:15<tab>  4.27<tab>  -0.19<tab>  -0.90
    5:07:50.9<tab> -5:05:11<tab>  2.79<tab>  +0.13<tab>  +0.10
    5:01:26.3<tab> -7:10:26<tab>  4.81<tab>  -0.19<tab>  -0.74
    5:17:36.3<tab> -6:50:40<tab>  3.60<tab>  -0.11<tab>  -0.47
    [EOD]