mica.archive.aca_dark

mica.archive.aca_dark.dark_cal

mica.archive.aca_dark.dark_cal.dark_id_to_date(dark_id)

Convert dark_id (YYYYDOY) to the corresponding DateTime ‘date’ format.

Parameters:date – dark id (YYYYDOY) Returns:str in DateTime ‘date’ format

mica.archive.aca_dark.dark_cal.dark_temp_scale(t_ccd, t_ccd_ref=-19.0)

Return the multiplicative scale factor to convert a CCD dark map from the actual temperature t_ccd to the reference temperature t_ccd_ref.

Based on best global fit for dark current model in plot_predicted_warmpix.py. Previous value was 0.62 instead of 0.70. This represents the change in dark current for each 4 degC decrease:

>>> from mica.archive.aca_dark import temp_scalefac
>>> print temp_scalefac(t_ccd=-15, t_ccd_ref=-19)
0.7

Parameters:

• t_ccd – actual temperature (degC)
• t_ccd_ref – reference temperature (degC, default=-19.0)

Returns:

scale factor

mica.archive.aca_dark.dark_cal.date_to_dark_id(date)

Convert date to the corresponding YYYYDOY format for a dark cal identifiers.

Parameters:date – any DateTime compatible format Returns:dark id (YYYYDOY)

mica.archive.aca_dark.dark_cal.get_dark_cal_dirs(*args, **kwds)

Get an ordered dict of directory paths containing dark current calibration files, where the key is the dark cal identifier (YYYYDOY) and the value is the path.

Parameters:source – source of dark cal directories (‘mica’|’ska’) Returns:ordered dict of absolute directory paths

mica.archive.aca_dark.dark_cal.get_dark_cal_id(date, select='before')

Return the dark calibration id corresponding to date.

If select is 'before' (default) then use the first calibration which occurs before date. Other valid options are 'after' and 'nearest'.

Parameters:

• date – date in any DateTime format
• select – method to select dark cal (before|nearest|after)

Returns:

dark cal id string (YYYYDOY)

mica.archive.aca_dark.dark_cal.get_dark_cal_image(*args, **kwargs)

Return the dark calibration image (e-/s) nearest to date.

If select is 'before' (default) then use the first calibration which occurs before date. Other valid options are 'after' and 'nearest'.

Parameters:

• date – date in any DateTime format
• select – method to select dark cal (before|nearest|after)
• t_ccd_ref – rescale dark map to temperature (degC, default=no scaling)

Returns:

1024 x 1024 ndarray with dark cal image in e-/s

mica.archive.aca_dark.dark_cal.get_dark_cal_props(*args, **kwargs)

Return a dark calibration properties structure for date

If select is 'before' (default) then use the first calibration which occurs before date. Other valid options are 'after' and 'nearest'.

If include_image is True then an additional column or key image is defined which contains the corresponding 1024x1024 dark cal image.

Parameters:

• date – date in any DateTime format
• select – method to select dark cal (before|nearest|after)
• include_image – include the dark cal images in output (default=False)

Returns:

astropy Table or list of dark calibration properties

mica.archive.aca_dark.dark_cal.get_dark_cal_props_table(start=None, stop=None, include_image=False, as_table=True)

Return a table of dark calibration properties between start and stop.

If include_image is True then an additional column or key image is defined which contains the corresponding 1024x1024 dark cal image.

If as_table is True (default) then the result is an astropy Table object. If False then a list of dicts is returned. In this case the full contents of the properties file including replica properties is available.

Parameters:

• start – start time (default=beginning of mission)
• stop – stop time (default=now)
• include_image – include the dark cal images in output (default=False)
• as_table – return a Table instead of a list (default=True)

Returns:

astropy Table or list of dark calibration properties

mica.archive.aca_dark.dark_model

Routines related to the dark current model and guide / acq success prediction.

mica.archive.aca_dark.dark_model.acq_success_prob(mag, warm_frac, prob_offset=0)

Calculate probability of acquisition success for a star with mag magnitude and a CCD warm fraction warm_frac. Uses the empirical relation:

P_acq_success = offset(mag) + scale(mag) * warm_frac

In ../guide_acq_success/plot_acq_success.py we find the best fit relation:

log10(scale) = 0.185 + 0.990 * (mag - 10) + -0.491 * (mag - 10)**2 log10(offset) = -1.489 + 0.888 * (mag - 10) + 0.280 * (mag - 10)**2

mica.archive.aca_dark.dark_model.get_acq_success(date, t_ccd, mag)

Return probability of acquisition success for given date, temperature and mag.

Any of the inputs can be scalars or arrays, with the output being the result of the broadcasted dimension of the inputs.

This is based on the dark model and acquisition success fitting presented in the State of the ACA 2013 (sot/state_of_aca/guide_acq_stats)

Parameters:

• date – Date(s) (scalar or np.ndarray)
• t_ccd – CD temperature(s) (degC, scalar or np.ndarray)
• mag – Star magnitude(s) (scalar or np.ndarray)

Returns:

Acquisition success probability(s)

mica.archive.aca_dark.dark_model.get_dark_hist(date, t_ccd)

Return the dark current histogram corresponding to date and t_ccd.

Parameters:

• date – date in any DateTime format
• t_ccd – CCD temperature (deg C)

Returns:

bin_centers, bins, darkhist

mica.archive.aca_dark.dark_model.get_dark_model(date, t_ccd)

Return the dark current model corresponding to date and t_ccd.

Parameters:

• date – date in any DateTime format
• t_ccd – CCD temperature (deg C)

Returns:

TBD

mica.archive.aca_dark.dark_model.get_guide_success(date, t_ccd, mag)

Return probability of guide (bad_trak) success for given date, temperature and mag.

Any of the inputs can be scalars or arrays, with the output being the result of the broadcasted dimension of the inputs.

This is based on the dark model and guide success fitting presented in the State of the ACA 2013 (sot/state_of_aca/guide_acq_stats)

Parameters:

• date – Date(s) (scalar or np.ndarray)
• t_ccd – CD temperature(s) (degC, scalar or np.ndarray)
• mag – Star magnitude(s) (scalar or np.ndarray)

Returns:

Guide success probability(s)

mica.archive.aca_dark.dark_model.get_sbp_pars(dates)

Return smooth broken powerlaw parameters at date. This is based on the sbp fits for the darkhist_peaknorm histograms, with parameters derived from by-hand inspection of fit trending. See NOTES.

mica.archive.aca_dark.dark_model.smooth_broken_pow(pars, x)

Smoothed broken power-law. Pars are same as bpl1d (NOT + gaussian sigma): 1: gamma1 2: gamma2 3: x_b (break point) 4: x_r (normalization reference point) 5: ampl1 # NOT 6: sigma (bins)

mica.archive.aca_dark.dark_model.temp_scalefac(T_ccd)

Return the multiplicative scale factor to convert a CCD dark map from the nominal -19C temperature to the temperature T. Based on best global fit for dark current model in plot_predicted_warmpix.py. Previous value was 0.62 instead of 0.70.

mica.archive.aca_dark.update_aca_dark

mica.archive.aca_dark.update_aca_dark.get_ccd_temp(tstart, tstop)

Get the best estimate of CCD temperature between tstart and tstop

Parameters:

• tstart – start time in CXC seconds
• tstop – stop time in CXC seconds

mica.archive.aca_dark.update_aca_dark.get_dark_cal_id(date)

Get the dark cal ID in the Ska dark current files corresponding to date. It is assumed the dark cal dirs are labeled by YYYYDOY within root.

This routine allows for plus/minus one day of slop.

Parameters:date – Date of dark cal Returns:str in yeardoy format

mica.archive.aca_dark.update_aca_dark.get_zodi_props(dark_id)

Get zodiacal light information for given dark_id.

Parameters are: (‘date’, ‘ra’, ‘dec’, ‘el’, ‘eb’, ‘sun_el’, ‘l_l0’, ‘zodib’).

Parameters:dark_id – dark cal ID as YYYYDOY Returns:table Row object

mica.archive.aca_dark.update_aca_dark.main()

Update all the Mica dark cal directories.

mica.archive.aca_hdr3

Experimental/alpha code to work with ACA L0 Header 3 data

class mica.archive.aca_hdr3.MSID(msid, start, stop, msid_data=None, filter_bad=False)

ACA header 3 data object to work with header 3 data from available 8x8 ACA L0 telemetry:

>>> from mica.archive import aca_hdr3
>>> ccd_temp = aca_hdr3.MSID('ccd_temp', '2012:001', '2012:020')
>>> type(ccd_temp.vals)
'numpy.ma.core.MaskedArray'

When given an msid and start and stop range, the object will query the ACA L0 archive to populate the object, which includes the MSID values (vals) at the given times (times).

The parameter msid_data is used to create an MSID object from the data of another MSID object.

When filter_bad is supplied then only valid data values are stored and the vals and times attributes are np.ndarray instead of ma.MaskedArray.

Parameters:

• msid – MSID name
• start – Chandra.Time compatible start time
• stop – Chandra.Time compatible stop time
• msid_data – data dictionary or object from another MSID object
• filter_bad – remove missing values

filter_bad(copy=False)

Filter out any missing values.

After applying this method the vals attributes will be a plain np.ndarray object instead of a masked array.

Parameters:copy – return a copy of MSID object with bad values filtered

class mica.archive.aca_hdr3.MSIDset(msids, start, stop)

ACA header 3 data object to work with header 3 data from available 8x8 ACA L0 telemetry. An MSIDset works with multiple MSIDs simultaneously.

>>> from mica.archive import aca_hdr3
>>> perigee_data = aca_hdr3.MSIDset(['ccd_temp', 'aca_temp', 'dac'],
...                                 '2012:001', '2012:030')

Parameters:

• msids – list of MSIDs
• start – Chandra.Time compatible start time
• stop – Chandra.Time compatible stop time

mica.archive.aca_l0

Functions

mica.archive.aca_l0.get_slot_data(start, stop, slot, imgsize=None, db=None, data_root=None, columns=None)

For a the given parameters, retrieve telemetry and construct a masked array of the MSIDs available in that telemetry.

>>> from mica.archive import aca_l0
>>> slot_data = aca_l0.get_slot_data('2012:001', '2012:002', slot=7)
>>> temp_ccd_8x8 = aca_l0.get_slot_data('2005:001', '2005:010',
...                                     slot=6, imgsize=[8],
...                                     columns=['TIME', 'TEMPCCD'])

Parameters:

• start – start time of requested interval
• stop – stop time of requested interval
• slot – slot number integer (in the range 0 -> 7)
• imgsize – list of integers of desired image sizes (defaults to all -> [4, 6, 8])
• db – handle to archive lookup table
• data_root – parent directory that contains archfiles.db3 (for use when db handle not available)
• columns – list of desired columns in the ACA0 telemetry (defaults to all in 8x8 telemetry)

Returns:

data structure for slot

Return type:

numpy masked recarray

mica.archive.aca_l0.get_files(obsid=None, start=None, stop=None, slots=None, imgsize=None, db=None, data_root=None)

Retrieve list of files from ACA0 archive lookup table that match arguments. The database query returns files with

tstart < stop and tstop > start

which returns all files that contain any part of the interval between start and stop. If the obsid argument is provided, the archived obspar tstart/tstop (sybase aca.obspar table) are used.

>>> from mica.archive import aca_l0
>>> obsid_files = aca_l0.get_files(obsid=5438)
>>> time_files = aca_l0.get_files(start='2012:001', stop='2012:002')
>>> time_8x8 = aca_l0.get_files(start='2011:001', stop='2011:010',
...                             imgsize=[8])

Parameters:

• obsid – obsid
• start – start time of requested interval
• stop – stop time of requested interval
• slots – list of integers of desired image slots to retrieve (defaults to all -> [0, 1, 2, 3, 4, 5, 6, 7, 8])
• imgsize – list of integers of desired image sizes (defaults to all -> [4, 6, 8])
• db – handle to archive lookup table
• data_root – parent directory of Ska aca l0 archive

Returns:

interval files

Return type:

list

mica.archive.asp_l1

Script to update Ska file archive aspect L1 products. Module also provides methods to retrieve the directory (or directories) for an obsid.

This uses the obsid_archive module with a configuration specific to the aspect L1 products.

Functions

mica.archive.asp_l1.get_files(obsid=None, start=None, stop=None, revision=None, content=None)

List asp_l1 files for an obsid or a time range.

>>> from mica.archive import asp_l1
>>> obs_files = asp_l1.get_files(6000)
>>> obs_gspr = asp_l1.get_files(6000, content=['GSPROPS'])
>>> range_fidpr = asp_l1.get_files(start='2012:001',
...                                stop='2012:030',
...                                content=['FIDPROPS'])

Parameters:

• obsid – obsid
• start – time range start (Chandra.Time compatible)
• stop – time range stop (Chandra.Time compatible)
• revision – revision integer or ‘last’ defaults to current released version
• content – archive CONTENT type defaults to all available ASP1 types

Returns:

full path of files matching query

mica.archive.asp_l1.get_dir(obsid)

Get ASP L1 directory for default/released products for an obsid.

>>> from mica.archive import asp_l1
>>> asp_l1.get_dir(2121)
'/data/aca/archive/asp1/02/02121'

Parameters:obsid – obsid Returns:directory Return type:string

mica.archive.asp_l1.get_obs_dirs(obsid)

Get all ASP L1 directories for an obsid in the Ska file archive.

>>> from mica.archive import asp_l1
>>> obsdirs = asp_l1.get_obs_dirs(6000)

obsdirs will look something like:

{'default': '/data/aca/archive/asp1/06/06000',
2: '/data/aca/archive/asp1/06/06000_v02',
3: '/data/aca/archive/asp1/06/06000_v03',
'last': '/data/aca/archive/asp1/06/06000',
'revisions': [2, 3]}

Parameters:obsid – obsid Returns:map of obsid version to directories Return type:dictionary

mica.archive.obsid_archive

Base Class used in mica.archive.asp_l1 and mica.archive.obspar

Generalized module for fetching and archiving obsid-organized telemetry such as asp_l1 and obspar products.

class mica.archive.obsid_archive.ObsArchive(config)

Object to store configuration, logging, and processing tasks to fetch obsid telemetry from the CXC archive and store in a Ska file archive, while logging the archive files to a file lookup database.

The configuration dictionary config may have these key/values:

• data_root: directory for products

  (example /data/aca/archive/asp1)

• temp_root: directory for temporary storage of fetched

  telemetry

• bad_obsids: file containing list of obsids that should be

  ignored when in regular update mode

• cols: headers that will be included in file lookup table

• sql_def: sql file to build file lookup archfiles table

• apstat_table: axafapstat database table from which to find

  new processing (by id)

• apstat_id: field in apstat_table to use as unique CXCDS

  processing id

• label: label of product type for log messages

• small: arc5gl keyword/filetype for small file from products

  (example asp1{fidprops}). This will be retrieved with “get %s” % config[‘small’] and the retrieved files will be used to determine product version.

• small_glob: glob to match files retrieved by

  “get %s” % config[small] (example ‘fidpr‘)

• small_ver_regex: regular expression to search for version from

  retrieved files (example ‘pacdfd+N(d{3})_’)

• full: arc5gl keyword for products (example ‘asp1’)

• rebuild: If True/set, allow update mode to rebuild the database

  from obsid 1.

Parameters:config – configuration dictionary Returns:ObsArchive instance

get_all_obspar_info(i, f, archfiles)

Read obspar and add ‘obsid’ and ‘filename’ keys to the dictionary i and archfiles are just passed to make the logging prettier.

get_arch(obsid, version='last')

Retrieve telemetry for an observation from the CXC archive and store in the Ska file archive.

Parameters:

• obsid – obsid
• version – ‘default’, ‘last’, or revision/version number

Returns:

obsid directory in Ska file archive

Return type:

directory string

get_arch_info(i, f, archfiles)

Get information for a file for the file lookup table/database. For obspars, call the get_obspar_info() method. For FITS files, call get_fits_info() method.

get_dir(obsid)

Return the latest released directory for an obsid Return None if there are no ‘default’ / released products.

static get_file_ver(tempdir, fileglob, ver_regex)

Determine the version/revision of a set of archived files from their file names.

Parameters:

• tempdir – directory containing files
• fileglob – glob to match files in question
• ver_regex – regular expression to pull out version from the set of files

Returns:

version number

Return type:

integer

get_fits_info(i, f, archfiles)

Read FITS file f with index i (position within list of filenames archfiles) and get dictionary of values to store in file lookup database. This values include all header key/value pairs with keys in config[cols] plus the header checksum, the filename, the year, and day-of-year.

Parameters:

• i – index of file f within list of files archfiles
• f – filename
• archfiles – list of filenames for this batch

Returns:

info for a file

Return type:

dictionary

get_obs_dirs(obsid)

Return a dictionary of the directories available for an obsid. This is just done with a glob in the data directories.

get_obspar_info(i, f, archfiles)

Wrap get_all_obspar_info() and just include columns in config[‘cols’]

get_todo_from_links(archive_dir)

Return a list of all of the *_last directories in the file archive (and specify revision=default to attempt to get new released products for them).

get_ver_num(obsid, version='default')

Determine the version number associated with the current released products or with the products referenced by “version=last”.

Parameters:

• obsid – obsid
• version – version string (‘default’|’last’)

Returns:

version

Return type:

integer

set_env()

Set environment included an arc5gl handle and and a handle to the axafapstat database

set_read_env()

Set environment included an arc5gl handle and and a handle to the axafapstat database

update()

Run the update process using the config already passed to the object.

update_link(obsid)

Create links in the obsid data directories to make it easy to find the current ‘default’/released data, all versions that have been archived, and the ‘last’/unreleased/provisional data if available.

This is designed so that if obsid 5 has released data in version 1 and provisional data in version 2, that the directories and links will look like:

directory 00005_v01 directory 00005_v02 link 00005 -> 00005_v01 link 00005_last -> 00005_v02

mica.archive.obspar

Script to update Ska file archive obspars. Module also provides methods to retrieve the directory (or directories) for an obsid.

This uses the obsid_archive module with a configuration specific to the obspar products.

Functions

mica.archive.obspar.get_dir(obsid)

Get obspar directory for default/released products for an obsid.

>>> from mica.archive import obspar
>>> obspar.get_dir(2121)
'/data/aca/archive/obspar/02/02121'

Parameters:obsid – obsid Returns:directory Return type:string

mica.archive.obspar.get_obs_dirs(obsid)

Get all obspar directories for an obsid in the Ska file archive.

>>> from mica.archive import obspar
>>> obsdirs = obspar.get_obs_dirs(6000)

obsdirs will look something like:

{'default': '/data/aca/archive/obspar/06/06000',
2: '/data/aca/archive/obspar/06/06000_v02',
3: '/data/aca/archive/obspar/06/06000_v03',
'last': '/data/aca/archive/obspar/06/06000',
'revisions': [2, 3]}

Parameters:obsid – obsid Returns:map of obsid version to directories Return type:dictionary

mica.archive.obspar.get_obspar(obsid, version='default')

Get the obspar for obsid. Return as a dict.

>>> from mica.archive import obspar
>>> obspar.get_obspar(7001)['detector']
'ACIS-I'

Parameters:

• obsid – obsid
• version – processing version/revision

Returns:

dictionary of obspar

mica.vv

Functions

mica.vv.get_vv(obsid, version='default')

Retrieve V&V data for an obsid/version. This reads the saved JSON and returns the previously- calculated V&V data.

Parameters:

• obsid – obsid
• version – ‘last’, ‘default’, or version number

Returns:

dict of V&V data

mica.vv.get_vv_dir(obsid, version='default')

Get directory containing V&V products for a requested obsid/version, including plots and json.

Parameters:

• obsid – obsid
• version – ‘last’, ‘default’ or version number

Returns:

directory name for obsid/version

mica.vv.get_vv_files(obsid, version='default')

Get list of V&V files available for a requested obsid/version.

Parameters:

• obsid – obsid
• version – ‘default’, ‘last’ or version number

Returns:

list of files

mica.vv.get_rms_data()

Retrieve/return all data from RMS trending H5 archive

Returns:numpy array of RMS data for each star/obsid/version

mica.vv.get_arch_vv(obsid, version='last')

Given obsid and version, find archived ASP1 and obspar products and run V&V. Effort is made to find the obspar that was actually used during creation of the ASP1 products.

Parameters:

• obsid – obsid
• version – ‘last’, ‘default’, or revision number of ASP1 products

Returns:

mica.vv.Obi V&V object