Running ACORN

Acorn is a Data Systems tool which allows one to decommutate telemetry data both by accessing the real time telemetry feeds or by processing telemetry data dump files.

This document describes how to use Acorn to decom telemetry data, the format of the resultant files called "tracelog files", and demontrates how to read and process tracelog files.

Acorn Operations Topics:

Introduction
Where to find dump data
Running Acorn on the OPS LAN
Running Acorn on the HEAD LAN
Format of a Tracelog file
Getting stats on a Tracelog file
Plotting data from a Tracelog File

Introduction

Acorn takes, as input, telemetry data files and produces a flat ASCII file known as a "Tracelog" file. You can then process this Tracelog file in any way that you like, using any tools that you like. This document will describe how to obtain a data dump file; how to process it through Acorn; describe the format of the tracelog file; show how to obtain stats on the tracelog file; and how to plot data from the file.

The instaantiation of Acorn that you will use in this process, extracts MSID's based upon the data entered in the file called:

    acis-acorn-msid-script_plotters

If an MSID is not listed in that file, it is not extracted. On the HEAD LAN, the file is located in:

    /home/acisdude/ACORN/acis-acorn-msid-script_plotters

On the OPS lan, it is located in:

    /export/acis-flight/acis/bin/PLOTTERS/ENGPLOTTER/CODE/acis-acorn-msid-script_plotters

WARNING: Do not edit this file unless you have solid knowledge of how Acorn uses it. Incorrect editing of the file will disable the execution of Acorn.

The reason the files are located in different directories, on different LANs is so that, in the case of Colossus-v, maintenance of the R/T Web page Acorn automatically gets you up to date when you try to run it stand alone.

On the other hand, on the HEAD LAN you want to be able to run this from any account on any machine. Using the aciscdp-v or acis60-v instantiations of Acorn would force you to log on to those systems as acisweb. Instead, I've created a separate instantiation and will take the appropriate maintenance steps.

Existing Tracelog files

R/T programs already process dump data. One process creates a Tracelog file of the most recent 10 days worth of data; another processes the Real Time UDP telemetry feeds to the R/T web pages. You are free to use those tracelog files.

See

Getting stats on a Tracelog file

and

Plotting data from a Tracelog File

To learn how to use the data.

10 Day Tracelog files

R2D2-V creates a tracelog file covering the last 10 days, from the available GOT data dumps. This program runs every hour, so whenever a new data dump appears, it is incorporated in the 10 day file in about an hour. The oldest dump in the 10-day long file falls off the back end.

On the HEAD lan, the file is located at:

    /data/acis/eng_plots/acis_eng_10day.tl

On the OPS lan the file is located at:

    /data/acis-bak/eng_plots/acis_eng_10day.tl

If the data you want are within the last 10 days, the simplest way to get it is to use the 10 day file.

The MSIDs available in these files are:

TIME 1CBAT 1CBBT 1DACTBT 1PIN1AT 1PDEAAT 1PDEABT 1CRAT 1CRBT 1WRAT 1WRBT

1DPAMYT 1DPAMZT 1DEAMZT 1SSPYT 1SSMYT 1DP28AVO 1DP28BVO 1DPICACU 1DPICBCU 1DPP0AVO

1DPP0BVO 1DE28AVO 1DEP3AVO 1DEP2AVO 1DEP1AVO 1DEP0AVO 1DEN0AVO 1DEN0BVO 1DEN1AVO 1DEN1BVO

1DEP3BVO 1DEP2BVO 1DEP1BVO 1DEP0BVO 1DEICACU 1DEICBCU 1DE28BVO

3TSMYDPT 3TSPYFET 3TSPZDET 3TSPZSPT 3TSMXSPT 3TSMXCET 3RCTUBPT 3TTACS1T 3TTACS2T 3TTACS3T

1DAHHAVO 1DAHHBVO 1DAHAVO 1DAHBVO 1DAHACU 1DAHBCU 1DAHAT 1DAHBT 1OAHAT 1OAHBT

2DETART 2DETBRT 2SHLDART 2SHLDBRT

Tracelog Files Updated During Real Time Contacts.

If you are the midst of a Real Time Comm, all 4 R/T Web pages are running their own instantiation of Acorn
and creating Tracelog files. You can find those files on each of the 3 ACIS Virtual machines, and Colossus-v, at:

    /export/acis-flight/acis/bin/*.tl

They are constantly being added-to during the R/T pass. Different classes of MSIDs appear in different files.
For example, the Hardware and Software LEDs can be found in:

    acisHWLEDs_00574661335.25.tl

and

    acisSWLEDs_00574661335.25.tl

Running Acorn on a Data Dump File

If you cannot wait for the hourly R2D2-V 10-day run, or want to decom some data dump file prior to 10 days ago, then a stand-alone run of Acorn is the answer. You have two options here:

Run ACORN on Colossus-v
Run ACORN on any machine on the HEAD LAN

No matter where you want to run Acorn, you must get the data dumps off the Ops LAN. And of course, there are 3 places to get those:

GOT data dumps

    - NOTE: Data from previous years can be found at: 

            /dsops/critical/GOT/input/old

            ...which has various subdirectories for previous years:
  
            colossus-v-121: ls input/old
            1999  2001  2003  2005  2007  2009  2015  2017  2019
            2000  2002  2004  2006  2008  2010  2016  2018

GOT raw dumps

GRETA dumps (to be described later)

NOTES:

The exit line that you see when Acorn is done:

You can, if you wish, concatenate two or more gzip files into one big gzip file, gunzip that big file and run that through Acorn. The files should be contiguous in time, given that you end up with one tracelog file. If you want separate tracelog files for separate dumps, run Acorn on each dump separately.

Log onto Colossus-v as acisdude
Copy (a simple cp command) the data dump you wish to decom into your working directory
gunzip the file.
Run the Acorn Setup script.
Run Acorn on that file.
SFTP the resultant tracelog (.tl) file to the HEAD lan if that's where you want to work with it, or MOVE the file to a working directory.

If you want to run Acorn on the HEAD LAN, the steps are:

Log into any HEAD machine as yourself
SFTP the data dump you wish to decom from the Ops LAN into your working HEAD LAN directory
gunzip the file.
Run the Acorn Setup script.
Run Acorn on that file.

Steps in Detail - OPS LAN

Log onto Colossus-v as acisdude

Set up Acorn by executing the following command:

CD into a working directory where you would like the tracelog files from the Acorn run to appear.

Copy the data dump you wish to decom into your working directory.

gunzip the file.

Run Acorn on that file.

SFTP the resultant tracelog file onto a HEAD machine if you prefer to do your analysis there.

NOTES:

The exit line that you see when Acorn is done:

Steps in Detail - HEAD LAN

Log onto any HEAD machine as anyone you want to be.

Set up Acorn by executing the following command:

CD into a working directory where you would like the tracelog files from the Acorn run to appear.

SFTP the data dump you wish to decom into your working directory.

gunzip the file.

Run Acorn on that file.

NOTES:

The exit line that you see when Acorn is done:

Tracelog File Format

Tracelog files are flat ASCII files organized in columns of MSIDs where the first column is always TIME in seconds from 1985.

Tracelog files have a .tl extension

The file looks like this:

TIME	1CBAT	1CBBT	1DACTBT	1PIN1AT	1PDEAAT	1PDEABT	1CRAT......
00986401945.30 -39.769363 7.620178 255.000000 00986401953.50 19.124655 29.727272 -39.769363	
    .
    .
    .
    .

Acorn puts out data time stamps as counted from Jan. 1, 1985 Chandra Time starts from Jan. 1, 1998. We need a time constant to
convert Acorn times to Chandra times. The following value is the number of seconds in 13 years (1998 - 1985) plus 3 leap days

DIFF_1985_1998 = 410227200

GOT Dump File Overlap

GOT Telemetry Data Dump files overlap by approximately a couple of minutes. Looking at two consecutive dump files in /dsops/critical/GOT/input/:

    2016_095_2114_096_0416_Dump_EM_01017

    2016_096_0414_096_1154_Dump_EM_01273

Notice that the earlier file ends at 096_0416 and the later file begins at 096_0414. I extracted those two files, ran Acorn on them and got some stats:

2016_095_2114_096_0416_Dump_EM_01017:

    Tracelog file start date:  2016:095:16:09:16.916

     Tracelog file stop date:  2016:095:21:15:24.916

2016_096_0414_096_1154_Dump_EM_01273:

    Tracelog file start date:  2016:095:21:13:46.516

     Tracelog file stop date:  2016:096:04:15:07.116

So there's an ~2 minute overlap in the Dump files. I don't think this is going to make a major difference in the software we run, but it is an artifact of the way they store the dump data and we all need to be aware of it.

Number of lines/repeated values

Suppose you have Acorn extracting two MSID's:

1PERMJF and 3PERMJF

The first occurs once per MJF and the second occurs 4 times per MJF, Acorn's resultant tracelog file will have 4 lines per MJF. And the MSID that is sampled once per MJF will have it's value repeated 4 times. It will look like this:

    1PERMJF         3PERMJF
     42.0            71.0
     42.0            72.0
     42.0            73.0
     42.0            74.0

Getting Stats on a Tracelog File

Once you have a tracelog file (acisENG10d_00986401945.30.tl), perhaps you'd like to know something about it. This Python function:

ReadTracelogFile(filespec)

Will read in the Tracelog file pointed to by "filespec", create a data structure (numpy rec array) and return that structure to the calling program.

The function also adjust the times from 1985 time to Chandra (1998) time.

The function definition can be found in:

/data/acis/LoadReviews/script/PYTHON/ReadTracelog.py

The following Python script will tell you some things about the tracelog file:

    /data/acis/LoadReviews/script/PYTHON/TracelogStats.py

The output will look something like this:

Tracelog file start date:  2016:095:16:31:41.716
 Tracelog file stop date:  2016:095:21:39:44.516
Dump time is:  18482.8000001  seconds or:  5.13411111113  hours
Number of data rows is:  342

Plotting Tracelog File Data

Once you have the numpy array available (after executing ReadTracelog.py) you can plot from one to 5 MSIDs of your choice by calling the function:

FivePlots(msid_list, tracelog_data, tracelog_file_path)

This function lives in:

/data/acis/LoadReviews/script/PYTHON/5plots.py

Using the Python Functions

To use these functions, you first have to enable the Ska environment, and then instantiate Python.

Open a clean terminal.

Instantiate the Ska Environment by typing:

source /proj/sot/ska/bin/ska_envs.csh

Instantiate the Python plotting environment by typing at the prompt:

Execute the Python file you created:

Here is an example of a program that uses the functions to generate Panel Plots of whatever MSID's you like
(so long as they exist in the Tracelog file).

import Chandra.Time

# Bring in supporting programs and tools
execfile('/data/acis/LoadReviews/script/PYTHON/PlotTools.py')
execfile("/data/acis/LoadReviews/script/PYTHON/ReadTracelog.py")
execfile("/data/acis/LoadReviews/script/PYTHON/5plots.py")
#
# Read in a Tracelog file
#
# Specify the path to the tracelog file
filespec =
  "/home/gregg/acorntest/acisENG10d_00985114479.70.tl"

# Read in the tracelog file and load the data structure tl_data
tl_data = ReadTracelogFile(filespec)

# Now put up a 5 plot panel plot of a few MSIDs
mymsids = ["1DPAMZT", "1DPAMYT", "1DEAMZT", "1DPICACU", "1DP28AVO"]
FivePlots(mymsids, tl_data, filespec)


# Now put up a 3 plot panel plot of some other MSIDs
mymsids = ["1PDEAAT", "1CRAT", "1WRAT"]
FivePlots(mymsids, tl_data, filespec)

The inputs are:

Path to your tracelog file:

filespec = "/home/gregg/acorntest/acisENG10d_00985114479.70.tl"

A list of MSIDs you'd like to have plotted:

mymsids = ["1DPAMZT", "1DPAMYT", "1DEAMZT", "1DPICACU", "1DP28AVO"]

You can plot any number of MSIDs in the panel plots - but I recommend you keep it between 1 and 5. More than 5 and they get hard to read. The example above shows you how to create one plot with 5 and another plot with 3. the number of panels is driven by the number of MSID's you place in the list.

Back To Top