Fetch documentation

Synopsis

The basic syntax for calling fetch is:

fetch [options] table_col_spec1 [table_col_spec2 ...]

Table_col_spec is a table column specifier that specifies one or more columns (MSIDs) within the telemetry archive tables. The allowed formats are:

<table>:<col1>,[<col2>,...]   # col1,... in specified table
<col1>,..                     # col1,... within any table (must be unique)

Options

The available command line options are:

-h, --help               Show this help message and exit
--obsid=OBSID            Return data for OBSID
--start=START            Start date of processing
--stop=STOP              Stop date of processing
--dt=DT                  Sampling interval (sec)
--outfile=OUTFILE        File for fetch output (default = stdout)
--statusfile=STATUSFILE  Write out fetch status each status-interval seconds
--status-interval=INTRVL Time interval between statusfile update and file size check (sec)
--max-size=MAX_SIZE      Approximate output file size limit (default = None)
--ignore-quality         Output bad quality rows (suppressed by default)
--mind-the-gaps          Abort if data gap detected (instead of just setting quality=1)
--file-format=FILE_FMT   Stop date of processing
--time-format=TIME_FMT   Format for output time stamp

In more detail the options are:

obsid

Process over the kalman interval for the given ObsId. This overrides any --start and --stop values.

--start, --stop

Specify the time range of the query. Both options require a value in one of the available DateTime Formats. The default values are 2007:001:23:00:00 and 2007:002:01:00:00, respectively.

--dt

Specify the time sampling for output data (in seconds). The default value is 32.8 seconds. Enhancing fetch to automatically determine the time sampling based on the first output column is a high priority. Note that fetch never averages, so if the output sampling period is longer than the natural sampling of the data column, then data values will be missed.

--outfile

Write the output table to the specified file. Default is writing to STDOUT.

--statusfile

Write a status file with the given name during processing. This gives a way of tracking the progress during long requests. This is written out once each status-interval seconds (approximately). An example output is:

current_row: 82400
total_rows: 787609
percent_complete: 10.5
process_start: Tue Jan 27 12:43:24 2009
current_time: Tue Jan 27 12:43:36 2009
datestart: 2008:001:12:00:00.000
datestop: 2008:300:12:00:00.000
columns: date 5ephint 5eiot quality
status: processing

--status-interval

Write the status file at the given time interval (seconds).

--max-size

Stop processing if the output file grows beyond the given file size (bytes). This is approximate since the file size is checked only each status-interval seconds. If this limit is exceeded then the processing status will be something like:

status: File size limit 1000000 bytes exceeded

--ignore-quality

The fetch default is to suppress output of any rows where any of the output column values have bad quality in the input telemetry (e.g. missing minor frame). With this command line option included then bad quality rows will be output with a quality column value of 1. The bad column values will be set to None.

--mind-the-gaps

Gaps in telemetry are normally just skipped so that the output data file has similar gaps. With this flag set the program will abort if a data gap is detected.

--file-format=FILE_FORMAT

Provide output table data in the specified FILE_FORMAT. The available options are csv (comma-separated),``tab`` (tab-separated),``rdb`` (currently same as tab, to be fixed), fits (FITS; not yet implemented). The default is csv.

--time-format=TIME_FORMAT

Specify the format for the date column in the output table data. Available options are given in the DateTime Formats section. The default is date.

DateTime Formats

Fetch (and in general the SKA telemetry archive toolsuite) supports a wide range of formats for representing date-time stamps. Note that within this and other documents for this tool suite, the words ‘time’ and ‘date’ are used interchangably to mean a date-time stamp.

The available formats are listed in the table below:

Format	Description	System
secs	Elapsed seconds since 1998-01-01T00:00:00	tt
numday	DDDD:hh:mm:ss.ss... Elapsed days and time	utc
jd*	Julian Day	utc
mjd*	Modified Julian Day = JD - 2400000.5	utc
date	YYYY:DDD:hh:mm:ss.ss..	utc
caldate	YYYYMonDD at hh:mm:ss.ss..	utc
fits	FITS date/time format YYYY-MM-DDThh:mm:ss.ss..	tt
unix*	Unix time (since 1970.0)	utc
greta	YYYYDDD.hhmmss[sss]	utc

* Ambiguous for input parsing and only available as output formats.

The default time “System” for the different formats is either tt (Terrestrial Time) or utc (UTC). Since TT differs from UTC by around 64 seconds it is important to be consistent in specifying the time format. Utilities to convert between the formats exist and can be made available if there is interest.

In general when supplying a date as an input (e.g. for the --start option), the format will be be automatically detected. If you enter a date in a format that is not recognized the tool will spit out a full code traceback (not too pretty), but the key will be to notice the final line which indicates a problem with the input time format:

Traceback (most recent call last):
  File "/proj/sot/ska/share/telem_archive/fetch.py", line 222, in ?
    main()
  File "/proj/sot/ska/share/telem_archive/fetch.py", line 48, in main
    for date in get_date_stamps(opt.start, opt.stop, opt.dt):
  File "/proj/sot/ska/share/telem_archive/fetch.py", line 98, in get_date_stamps
    datestop  = Chandra.Time.DateTime(stop).mxDateTime
  File "build/bdist.linux-i686/egg/Chandra/Time.py", line 298, in __getattr__
  File "build/bdist.linux-i686/egg/Chandra/Time.py", line 251, in convert
Chandra.Time.ChandraTimeError: Invalid input format 'None'

Module API

Fetch values from the SKA telemetry archive.

Functions

Ska.TelemArchive.fetch.fetch(obsid=None, outfile=None, statusfile=None, status_interval=5, max_size=None, ignore_quality=False, mind_the_gaps=False, debug=False, start=None, stop=None, dt=32.8, out_format=None, time_format='secs', colspecs=['ephin2eng:'])

Fetch data from the telemetry archive.

Parameters:

• obsid – Return data for obsid
• outfile – File for fetch output (default = stdout)
• statusfile – Write out fetch status each status-interval seconds
• status_interval – Time interval between statusfile update and file size check (sec)
• max_size – Approximate output file size limit (default = None)
• ignore_quality – Output bad quality rows (suppressed by default)
• mind_the_gaps – Abort if data gap detected (instead of just setting quality=1)
• start – Start date of processing
• stop – Stop date of processing
• dt – Sampling interval (sec)
• out_format – Format for output (‘csv’, ‘space’, ‘dmascii’, ‘tab’) (default=None => list)
• time_format – Format for output time stamp
• colspecs – List of column specifiers

Return type:

headers, values = tuple, list of tuples