| Wayside OCC and Real-Time Activities |
The Control Room at the Wayside OCC has six computers for use by the ACIS team. These are used during real-time activities and to install ACIS tables into the Chandra ODB (Operations Data Base). All are currently running CentOS Linux.
From left to right, these computers are:
All the computers besides the EHS can communicate beyond the OCC firewall to the the rest of the world in some fashion. In addition you can use the occguest Wi-Fi network to connect your laptop or mobile device to the internet.
Each set of hardware are maintained by a different team. Problems with Ishmael or Acisway should go to SysHelp (syshelpAThead.cfa.harvard.edu). Problems with EHS should be reported to the GOT (gotATipa.harvard.edu, 617-592-6010 or cross the control room and knock). Problems with Emily or Malang are the responsibility of the MIT team. Start with Catherine, then try Peter Ford (pfordATmit.edu) and Jim Francis (francisjATmit.edu).
As the ACIS stations can sit idle for long periods, sometimes the monitors are powered off. Always check for this, before despairing that a particular machine is dead.
Ishmael and Acisway: Log in using the acisweb account.
The acisweb account on these machines has our acis_docs git
depository, perhaps out of date, but not much
else. You can use ssh to log in to your HEAD account or
anywhere else from there. Log in remotely through pogow. Meteor, in the ACIS/HRC office, has the same setup.
EHS: Log in using your individual EHS password, then log into the EHS network using either sot0axaf or sotmaxaf, depending on whether you are displaying telemetry data during a real-time activity (sot0axaf) or installing new ACIS tables (sotmaxaf). The EHS passwords, both individual and sot0axaf/sotmaxaf, change quarterly, so make sure you have the most recent ones. Check with the GOT for new individual passwords, with Paul for sot0axaf, and with Catherine for sotmaxaf.
EGSE: In normal operation, these stay up and are always running our
telemetry display and processing software -- use the Esc key to wake
up the screen. The account is jennyg
and you can get the password from Catherine. Remote access is through
your individual pogow account, then on to emily or malang with the
jennyg account. Instructions for screen-sharing remotely are in
Section 10. The psci output files are in
/home/jennyg/Data/ in subdirectories labeled by the the date and time
when psci was last restarted.
Wireless: The wireless network is occguest. Anyone in the OCC or the acisdudes can give you the password.
Always try to arrive early, at least 30 minutes, before a real-time activity to ensure the necessary machines and processes are running correctly. Touching base with the OC/CC on-duty is also a good idea.
Typical ACIS activities in the control room during real-time contact are:Any real-time activity led by the ACIS team should produce an ACIS SOT shift report which is emailed to sot_shift@cfa.harvard.edu shortly after the activity is completed. MTA will then ensure that it gets posted to the SOT shift report web page. Talk to Scott Wolk if it doesn't appear there within a few days. A recent example that can be used as a template is here. A google archive of SOT shift emails is here
The filename convention is Mon_DD_YYYY_X####.txt where the #### is the day of mission and X is the "shift", as defined in 1999 when SOT was doing three shift operation. The SOT "shifts" are L (6am to 2pm), M (2pm to 10pm), and H (10pm to 6am). The daily SOT shift report uses "L", so you may need to add an additional signifier for a morning activity. You can get the day of mission by looking at recent SOT shift reports in your email or on the shift report web page.
The shift report should include the start and stop of the activity (probably the start and stop of the DSN comm), the names of the participating ACIS personnel, the plan, a summary of what happened, and a detailed timeline. You may also want to paste in the Chandra Snapshot at the start and/or stop of the activity. Make sure there's enough detail that some future ACIS ops person, who wasn't around for the activity, could still figure out everything that went on.
Note: This writeup assumes you are at the Wayside OCC in person.
Emily and Malang can also used remotely, through This is often where all the action is for an ACIS-centered
real-time activity. The MIT EGSE, Emily and Malang, are
identical Linux CentOS 7 machines used to run Note: This writeup assumes you are at the Wayside OCC in person.
Emily and malang can also used remotely, through
Once a month or so, an ACIS operations team member should check that
all the machines are functioning properly and, if necessary, restart
the processes running on the MIT machines. Anyone with OCC access,
the acisweb password, a HEAD account, an EHS account, and the
sot0axaf/sotmaxaf passwords can do it. It generally doesn't take more
than 5-10 minutes.
The steps are: If anything doesn't behave as it should, inform the proper authorities. Then send an email to acisdude with the OCC ACIS status.
New ACIS tables need to be installed into the Chandra ODB (Operations Database), which gives the FOT all the ACIS commands and the mapping of commands for each SI mode. New ACIS tables also need to be copied to Emily and Malang, so their processes will recognize the new ACIS modes. This is described in the SACGS short form, steps 7.5 - 7.11.
UNDER CONSTRUCTION! Talk to Catherine first before trying to
screenshare. Avoid multiple connections to the same machine and
coordinate carefully with other users. Emily and Malang can be accessed remotely through screensharing. It
is best to coordinate screensharing to prevent multiple users at the
same time. You
will need a pogow or SAO VPN account and the jennyg password. These instructions
are specific to connecting from a Mac, but connecting from a Linux
machine is not dissimilar.4. Real-time Activities: Emily and Malang
ssh and
screensharing.psci and
acisCtl, which are able to access ACIS
software housekeeping and science telemetry that is not visible to
the OC/CC. Screenlock is disabled so hit the Esc key to
wake up the screen if it isn't already.
Running psci from the command line produces output files,
including eventlists, bias maps, and logs of telemetry packets,
commands, exposure records, and science reports. acisCtl
is GUI-based and has a selection of windows which display various
software and hardware telemetry values. A hardcopy of the
acisCtl manual is in the ACIS office and in the Control Room and includes
screenshots of the various menus and windows. The manual is also
available on-line here and
by clicking "About acisCtl 2.0..." in the acisCtl primary window.
psci before the real-time activity starts. This will give you a clean directory and make it easier to find necessary output files. You may want to restart acisCtl as well, but if it appears to have been receiving data at the last scheduled comm and you are short on time, you can also leave it alone. You do not have to quit acisCtl to restart psci.
acisCtl, (maybe you just
logged in), open a terminal window and do "acisCtl -f". Click on
"Start ACIS Interface", "Start Packet Logging", "Start Raw Input
Logging". Then open as many displays as you want.
psci is outlined Section 6 in the Monthly OCC Check
acisCtl in a nutshell. More detail can be found in the manual including screenshots and discussions of the individual windows.
acisCtl
windows populated with data. acisCtl remembers where
you place its windows, updating the ~/.acisctlgeom file
as you do so.
acisCtl from the command line ("acisCtl -f"),
this is the only window that opens automatically. The "Quit" button
will clean up all windows and exit acisCtl entirely.
This window is shown in the upper left of the screenshot. The
options in the acisCtl 2.0 window are described below.
acisCtl,
you should start both types of logging. Logging will continue
indefinitely, moving to new files as the file size gets large or when
there is a long gap in telemetry between contacts.
acisCtl parameters. Casual
users should not need to change anything here.
psci output directory before starting ACIS
commanding.
acisCtl verifiers:
psci
output directory. This is a text file, so you can scroll through to find
what you need to see.
psci verifiers:
psci process that is started from
the command line. If files are not updating,
see Section 6 for how to
restart psci.
psci output directory. They are numbered
sequentially as they arrive. Restarting the processes in a new
directory on Emily and Malang before a real-time procedure will
ensure that the first file written is #1,
i.e. *.dumpedPatches.1.dat.
ds9.
psci output directory.
You compare this file to the one listed in the SOP by
using diff. They should be identical.
psci output directory. Assuming you
restarted psci for the activity, a "grep READ_BEP
*.packet.log | wc" should tell you how many packets were
received. If the directory is not new, your count may
include extra bepReadReply results, such as a WBTXINGBLL
parameter dump during a
previous belt passage. You can differentiate
these by time.
psci for this activity, the # will be 1. The flight software dump
procedure only requires counting the packets in
real-time, but you can compare this against a copy of a previous
dump
(/home/jennyg/Data/Reference/FSWDumps). Alternatively, a
shell script is available that can do the comparison for
you. "comp-eeprom.sh -v (filename of the bepReadReply)"
If the bepReadReply file is not available, you can also
go to the Packet Logging Directory (listed in
acisCtl Paramters), find the Packet Log
file (*.pkts.gz) and do "comp-eeprom.sh -v -n1 (filename
of packet log)", where the number is the same as the
bepReadReply file.
ltxings which will output ASCII
text from the original binary (you can redirect the
output to a file, if desired). The parameter
dump will appear as a single *.bepReadReply.#.dat file
and the tool is run as "ltxings (filename of the
bepReadReply)". The parameter dump includes three
versions of the parameters (TXinit.TX, TXnext.TX, txings.TX)
and txings.tx which reports the accumulated threshold
crossing calculation. For verifying uplink of new
parameters, confirm that TXinit.TX and TXnext.TX are
the same as in the uplink command. If this dump
was preceded by a warmboot, the contents of txings.TX
and txings.tx should be all zeros.
5. How can I tell Emily and Malang are receiving data?
Outside of a comm pass:
During a DSN contact:
acisCtl has lost its connection to COG (Chandra
Operations Gateway) and is not receiving telemetry.
acisCtl. (Note: This appears to not be true
at Wayside. At last check, these times were also the time of last
data.) Check the terminal window in which acisCtl was
launched and note down any error messages found there. Report them to
Catherine, Peter and Jim.
psci output directory and do an "ls -l". (The
output directory is in /home/jennyg/Data/occ.YrMonDy.HHMM/, where the
date and time are the last time psci was restarted). If the most recent file times are not from the most
recent comm, psci has died or is not receiving
telemetry.
If you suspect emily and/or malang are not receiving data, follow the instructions below for restarting.
psci output directory and check that
files are being updated.
6. Monthly OCC Check
ssh and
screensharing.
psci process in a new directory (very
useful if you're doing complicated real-time verification,
otherwise, if it ain't broke leave it alone):
psci output directory, like the one labelled
"jennyg@emily:~/Data/occ.15Jan20.1251" in the screenshot) (#8) . Or just open a new terminal window.
psci processes, for
example with "ps aux | grep psci", then "kill -9" on the result.
[jennyg@malang occ.26Apr19.1432]$ ps aux | grep psci
jennyg 8058 0.0 0.0 5192 3008 ? S Apr26 0:08 psci -m -u -B -T -h /home/devop/acis-centos7/tools/lib/huff.dat -t current.dat -l occ.26Apr19.1432
jennyg 16413 0.0 0.0 112664 976 pts/3 S+ 14:26 0:00 grep --color=auto psci
In this case, the process ID you want to kill is 8058.
psci output directory
for you, then instruct you to on how to stop and restart psci. The script
will:
psci processes (you just did that
step already but it doesn't hurt to double-check).
acisCtl ACIS
Interface and both logging processes. (If the ACIS Interface
window is open (#2), the
Interface is already running. All three buttons will say "Start" if they are not running and "Stop" if
they are already running.)
psci Data
directory and (2) start psci
[jennyg@malang Data]$ startPacketlog
Find any running psci (ps aux | grep psci) and kill it
acisCtl 2.0 window => Start ACIS interface (if not already started)
acisCtl 2.0 window => Start Packet Logging (if not already started)
acisCtl 2.0 window => Start Raw Input Logging (if not already started)
Then cut and paste the following to start psci:
cd /home/jennyg/Data/occ.07May19.1438
(filterClient -h malang| psci -m -u -B -T -h /home/devop/acis-centos7/tools/lib/huff.dat -t current.dat -l occ.07May19.1438 >> psci.log.07May19.1438 ) >& error.psci.07May19.1438 &
psci is writing files to your directory.
7. Installing new ACIS tables
8. General setup information
acisCtl manual is available online here,
or by clicking "About acisCtl 2.0..." in the acisCtl primary
window, and a
hardcopy of the acisCtl manual is in the Control Room
and the ACIS office. The most up-to-date version can be found on
the MIT LAN at $ACISTOOLSDIR/src/GSEtest/acisCtl/acisCtl.pdf. At
the OCC, acisCtl is run in -f mode (flight)
rather than -e which is used on the ACIS Engineering Unit.
For under-the-hood details, see the memo "The Architecture of
acisCtl 2.0".
acisCtl and psci. Do
not change the shell.
acisCtl graphical windows cannot be resized. The
text windows can be lengthened beyond their 24 line minimum but
their widths cannot be changed. You can move acisCtl windows and
the new locations will be saved in ~/.acisctlgeom, so they will
appear there next time you ask for them.
psci will go into
/home/jennyg/Data/occ.YrMonDy.HHMM/ where the date/time is set
when psci is started. Older directories may be moved
into /home/jennyg/Data/OldData/ to keep the listing managable.
/home/jennyg/Data/Reference/ contains the ACIS table files
(current.dat and current.cfg) and dumps of the FSW (EEPROM) and patches.
9. Troubleshooting Emily and Malang
gedit window (#9). Please record any issues
and send an email to the proper authorities.
sudo service gdm restart
acisCtl (press Quit in the primary "AcisCtl 2.0" window)
and restart with "acisCtl -f". Then restart psci as above.
acisCtl (press Quit in the primary "AcisCtl 2.0" window) and restart acisCtl with "acisCtl -f".
Then restart logging and psci as above. If that doesn't work, call the
GOT (got@ipa.harvard.edu or 617-592-6010 or walk across the Control
Room and knock) and ask them to recycle the COG to Emily
(or Malang). Look for the message "opening telemetry connection from
COG"
in the ACIS Interface window which effectively means you have connected. There will be no other indication of success other than data arriving during the next (or current) contact.
10. Screensharing with Emily and Malang
ssh -L 59000:localhost:5900 -J
cegrant@pogow.cfa.harvard.edu jennyg@malang.cfa.harvard.edu
Substitute your pogow username for "cegrant" and change malang to
emily if desired. You will be asked for your pogow and jennyg
passwords. If you are connected through the SAO VPN, you can remove
the "-J" to skip the jump through pogow.
x0vncserver is
running (it should always be):ps -ef | grep x0vncserver
who should show just the one connection from pogow or vpn.
vnc://localhost:59000
Periodically the windows on Emily and Malang become unresponsive. Logging out and back in again seems to solve the problem. Auto-login is enabled, so you can do this through Screensharing.
If you can interact with the windows and you are logging out proactively, you should first stop logging and the ACIS interface, then quit acisCtl. Log off in the upper right of the CentOS window. You may need to reconnect your Screenshare. Then restart acisCtl and psci normally.
If windows are unresponsive, you will have to log off without quitting acisCtl. Before restarting acisCtl, look for a process called "tlmGet" and kill it. Then restart acisCtl and psci normally.
If you cannot log off through the screenshare (i.e. you can't
interact with the display at all), restart the window manager in
your ssh command line:
sudo service gdm restart
If you cannot start screensharing, check in your ssh
session whether x0vncserver is running. If it is not,
restart with "x0vnc &".
Either way, reopen the "NOTES" file and add an entry:
gedit NOTES &
Can be compared to new dumps of flight software or patches using a diff. Make sure to "gunzip" the old files first if necessary.
Previous flight software dumps:
Last updated: 3 Oct 2022 by C. Grant