Because of new operating modes by Flight Ops, lr has been modified to handle new contingencies. These are:
- Continuity loads with the same name as Review load or with a different load letter than is pointed to by .../ofls.
- Full load (science + vehicle commands) where only the vehicle commands are enabled.
1. Load Letter for the continuity load
Typically when we issue the lr command we give the load letter for the Review load only:
lr JAN3122A JAN2422
lr would then use the directory pointed to by the .../ofls soft link to get information about the JAN2422 continuity load.
However with the SEP2021B load, the continuity load specified by mission planning was the SEP2021A load. This
confused the construction of the ACIS-History.txt file and the thermal models. The thermal models went into an infinite
loop because it kept using the review load as the continuity load and therefore the Time of First Command of the assembled history never moved back.
To fix these problems you now MAY apply a load letter to the continuity load. You would do this under these two circumstances:
- Continuity load has the same name as the review load, but is an earlier load letter
- Continuity load has a differnt name but for some reason the continuity load used by MP is not the one pointed to by the .../ofls soft link
For the first possibility, we can now issue the following lr command: lr SEP2021B SEP2021A
The second possibility has not happened yet but it may. Here is a possible scenario:
There is an OCT0122 A, B and C load
OCT0122B was uplinked to the spacecraft and run
OCT0822A load comes out for review and the Continuity load is therefore OCT0122B
Therfore the command you should issue is: lr OCT0822A OCT0122B
You don't need to specify the continuity load letter under normal circumstances.
2. Full loads (science + vehicle commands) where only the vehicle commands are enabled
Typically if a weekly review load contains both science and vehicle
loads, both will be enabled after uplink. However in late 2021 and
2022, a new paradigm has emerged:
Full loads sent out for review but only the vehicle command will be enabled after uplink.
This situation has occurred twice so far in the mission:
- JAN1722 vehicle-only-enabled
- OCT2821 vehicle-only-enabled
....with more to come.
Handling this situation requires a new switch to the lr command: --VO (Vehicle Only).
There are two acceptable values for the switch:
- "VOR" - Vehicle Only Review
- "VOC" - Vehicle Only Continuity.
VOR - Vehicle Only Review
If you are about to review a weekly load, and:
the load is a full load (science + vehicle commands), and
you know, before you begin your review, that only the vehicle
commands will be enabled after uplink, then
you need to add the --VO VOR switch to your lr command line:
lr --VO VOR Review-load Continuity-load
All other lr switchology remains the same (e.g. -T, -B, --nolucky, --nomodels etc).
Under the hood, what happens is that lr will save the CR*.backstop
file to ORIG_CR*.backstop and then copy the ....vehicle/VR*.backstop
into CR*.backstop and the review done by lr is on the "new" CR file. The
benefit of this is that when the review load becomes the continuity
load the next week, lr and the thermal models will automatically
process the correct commands.
VOC - Vehicle Only Continuity
It may happen that when you review the weekly load, there is every
intention of enabling both the science and vehicle commands. So you
ran lr without any --VO switch. But later on, after approval,
only the vehicle commands were enabled. There's not much you can do
about that with regard to that load. But when the subsequent weekly
load comes out you need to inform LR that the continuity load should
be read as Vehicle Only. That's where the --VO VOC switch comes in.
The LR command would be:
lr --VO VOC Review-load Continuity-load
This will cause lr to use the .../vehicle/VR*.backstop file in the
Continuity directory for assembling a history used by lr and the
thermal models.
All other lr switchology remains the same (e.g. -T, -B, --nolucky, --nomodels etc).
Special Cases
Things can get complicated during shutdowns, which complicate the
switches you must give to LR. One such example almost occurred
during
the Shutdown on Friday, February 9, 2024. DOY date
2024:041:03:23:00.00. And it has occurred in the past.
The FEB0724 load was interrupted with an SCS-107 due to high Solar
Radiation. There was some talk of postponing the Resumption to Science
load until after Sunday. The approved FEB1224 load had already
been uplinked. If the resumption load wasn't going to begin until
after FEB1224 began, only the FEB1224 vehicle load would
have
been activated. This did not subsequently happen but if it did, the handling
of the LR switches for the following Science Resumption Load reviews takes some
care.
This discussion informs you how to handle that situation if it did occur.
Assuming it did happen, the situation, graphically, would have looked
like this:
The FEB0724 load was interrupted at 2024:041:03:23:00.00, via an
SCS-107. This means that the Vehicle-Only portion of the FEB0724 load
continues to execute, but the Science Load stops (as indicated by the vertical red line
through the FEB0724 load).
The FEB1224 load was reviewed and approved prior to the shutdown. It
was uplinked to the satellite. If the start time of the Science
Resumption Load occurs
AFTER the start of the of the FEB1224 load,
some commands in the the vehicle-only portion of the FEB1224 load will
execute.
This means that the Science Resumption Load is interrupting the
FEB1224 load just like a TOO load interrupts an executing load. The
command line should look like this (assuming the Science Reumption
Load is FEB1024A):
lr --VO VOC -b FEB1024A FEB1224
Your responses to the LR questions will inform LR that this should be
treated as a TOO (i.e. HRC-S was not inserted into the focal plane).
The VOC switch must be included because only the Vehicle-Only commands
of the FEB1224 load were activated (in this scenario) so only the
vehicle commands should be include in the assembled history.
Scripts Used
1. lr (calls)
a. acis-backstop.pl - Does the majority of the checks.
b. saa_checker.pl - reports the solar array angle for forward sun and
tail sun observations.
c. LRHS - python script to just test health and safety.
d. run_PSMC.pl - executes the python PSMC model.
e. mk_pscm_html.pl - adds the PSMC output to our PSMC webpage.
f. bright-source - looks for bright X-ray sources too close to our FOV.
g. earth-acis-FOV - reports the solid angle of the earth in the
radiator's FOV.
h. plot_FOV_data.pl - plots the data from earth-acis-FOV.
i. backstop_comp.sh - compares the vehicle command backstop with the
combined command backstop to check for descrepancies.
j. show_web_lr.pl - Creates a colorized html version of the load
review text, and copies it to
https://cxc.cfa.harvard.edu/acis/lr_texts/yyyy/MMMddyyV_lr.html
2. ACE-update.pl
Script Descriptions
1. lr:
usage: lr {current load} {previous week} |& tee {directory of your choice}{current load}.log
example: lr JAN1302A JAN0602 |& tee /home/acisdude/JAN1302A.log
NOTE: The purpose of the additional string: "|& tee {directory of your choice}{current load}.log" to
the basic LR command "lr {current load} {previous week}" is to
capture the screen output resulting in the execution of the lr script. By
team agreement we've decided to use the "current load" name as the
name of the resultant .log file.
We copy the resultant file: "{directory of your choice}{current load}.log" into the relevant ofls
directory (oflsa, oflsb etc) once the execution of lr
completes.
NOTE: If lucky is unavailable, for example if loads have been created
at CDP, Use the --nolucky switch. The lr scrpt will direct you
how to get your desired
tarball file in place for further processing.
NOTE: If lr fails to run to completion, the load directories on both
/data/acis/LoadReviews and /data/acis-bak/LoadReviews must be deleted
before the lr run can be retried. You can accomplish this with the
command line:
lr --clean {current load} {previous week}
If you suspect the failure is due to a recent change to the lr script,
you can revert to the previous version of lr by cd'ing to the
directory /data/acis/LoadReviews/script, inspecting the soft link
from lr to the current version of the script, and replacing it with a
softlink to the preceding version -- e.g., from
lr -> lr-v1.50 to
lr -> lr-v1.49
The lr script is a PERL script that speeds up the load
review process. It gathers needed files from the Offline System
(OFLS), creates necessary directories and calls on other scripts that
utilize those files while consolidating the actual files to be
reviewed. For more info on acis-backstop.pl click here. This script
performs the following functions:
- For the "A" version of the load, it creates the new directory structure:
/data/acis/LoadReviews/{year}/{load date}/oflsa, for all other versions it only
creates the next "ofls" directory.
- Executes FTP lucky to obtain the FOT backstop file. This file is actually a zipped
tarball containing many files, most importantly the backstop file used
for the load review. lr then unzips the tar file, finds the backstop
files and all needed files, and extracts them from the tar file.
- The script then uses the combined backstop file together with the
previous load to call another script:
/data/acis/LoadReviews/acis-backstop.pl /data/acis/LoadReviews/{year}/{load date}/oflsa/{backstop-file}
/data/acis/LoadReviews/{year}/{previous load date}/ofls/ACIS-History.txt
The history file provides continuity between weeks and is generated by
the acis-backstop.pl script.
- Once acis-backstop.pl has finished running, lr zips the tar file sets up a symbolic link between the
current version "oflsx" load and the final version "ofls." This is
done with each new version of the loads to ensure that the latest
version is linked to the final version.
- The script also locates the version of the ACIS Tables that was
used to generate the command loads and displays this on the screen to
the user. This is important if there is commanding in the load that
requires a recently created SI Mode, ie: the most recent version of
the ACIS Tables needs to be used.
- lr calls saa_checker.pl and displays observations in which
the Solar Array Angle (SAA) is between 45-60 degrees and 130-160 degrees. It has been
determined that this attitude can cause thermal heating of the PSMC or
the ACIS DEA/DPA components. This gives us an early warning of the possiblility for heating.
- run_PSMC.pl is executed. This is a wrapper around the python
psmc_check.py code written and maintained by Tom Aldcroft to predict
the PSMC temperatures. After execution, lr calls mk_psmc_html.pl to
add an entry to our PSMC webpage table and make the proper links.
- Bright-sources is executed. This C++ code uses the manuever
summary file to determine if any bright sources cross the ACIS FOV
during the week. Any sources in the FOV for more than 2 minutes are
reported. This will be less conservative than the FOT code, but covers
many more sources.
- The earth on the ACIS cold radiator code, earth-acis-FOV.pl, is
executed to calculate the expected solid angle of the Earth
illuminating the cold radiator. Following this, plot_FOV_data.pl
creates a plot of these data for visual inspection.
- SOSA changes required a check between the vehicle backstop file
VR_.backstop and the combined vehicle and observation
backstop file, CR_.backstop. This is a sanity check.
2. ACE-update.pl:
usage: ACE-update.pl
example: ACE-update.pl
ACE-update.pl is a simple perl script which effectively does a
bunch of mv's, cat's, and cp's. When a load has been approved, an
ACIS Ops member will run this script to copy the ACIS Focal Plane
history
files to their primary and backup directories. It also copies the
approved load to a backup directory.
The ACIS Load Review
Once lr has completed running, a number of files should have been
created, they include:
1. ACIS-LoadReview.txt - Reviewed by ACIS Operations Team
2. saa_check.txt - Reviewed by ACIS Operations Team
4. ACIS-Tables.txt - Reviewed by ACIS Operations Team
5. ACIS-History.txt - To be used in the next load
6. ACIS-FPHIST.dat - Used in ACIS orbital fluence monitor
7. ACIS-GRATHIST.dat - Used in ACIS orbital fluence monitor
8. ACIS-OBSHIST.txt - Used in ACIS Real-Time Engineering Web Page
9. ACIS-TLMHIST.txt - Used in ACIS Real-Time Engineering Web Page
10. ACIS-TSCHIST.txt - Used in ACIS Real-Time Engineering Web
11. ACIS-STORED-HIST.txt - Used for next load
12. ACIS-LoadReview_HandS.txt - Health and Safety check load review
13. ACIS-Perigee.txt - information on last perigee
14. output_psmc - directory containing the output for the PSMC review
15. _angles.txt - output earth solid angles versus time
16. _angles_idl.ps - output earth solid angle plot
17. violations.txt - output bright source violations
Page
The ACIS Operations scientist on duty will then meticulously perform the
following manual checks on the ACIS-LoadReview.txt file:
- the SI mode in OCAT is consistent with the SI mode in the load
- the ACIS observing parameters stipulated in the OCAT are consistent with that of the parameter block being used in the observation
This consists of checking:
- that the chips selected in OCAT are the ones being selected in the parameter block
- that any sub-array defined in OCAT (start row, number of rows, alternating exposures, frame time) is consistent with the parameter block
- that the ACIS exposure mode and event telemetry format OCAT entries are consistent with the parameter block
- that the ACIS CCD "power-on" (ie, WSPOW****) is appropriate for the chip selection specified in OCAT
- that if a z-sim offset is specified, that it is specified in the load
- that the command load translates the SIM to the appropriate instrument (ACIS-I or ACIS-S) for that observation
- that for every ACIS observation there is "power-down" of all the video boards, a dump of the system configuration and Huffman tables, prior to the ACIS start science command
- that the appropriate gratings are inserted (or retracted) for that observation
- that the correct window block is used if one is specified
- that any unique observing parameters such as on-chip summing, an event filter range, changes to dither, or forced bias, are also defined in the parameter block that is used in the command load
- that the integration time is consistent with the integration time
remaining according to OCAT
- the commanding for an ACIS diagnostic measurement is correct and as requested
- prior to perigee, that all video boards are powered down, and the
system configuration and Huffman tables are dumped
- for HRC observations greater than 5 ks in duration, that ACIS is obtaining useful calibration data
- there is no significant "dead time" between stop of science observations and onset of calibration measurements on ingress and similarly egress of perigee
- that an OBSID change occurs prior to the start of the next
science run
- ensure that the automatic tests have been done correctly
- examine the thermal check output html files for violations (listed as "NOT OK" at the top of the page, and investigate any issues flagged.
These checks are performed in addition to the automatic tests that are
done while acis-backstop.pl is running. These tests can be broadly
categorized into the 3 groups listed below. Under each group is a
brief description of the test performed. If a particular test fails a
comment to that effect is printed immediately at that time location in
the output file.
1. Automatic SIM and radiation zone tests
- check that there are ORBITAL EVENTS in the load (apogee/perigee, predicted radiation belt entry and exit times)
- ensure that there is a 10 kilo second (ks) pad time on entry to the rad zone (defined as pad = {time of radiation belt entry} - {time of RADMON disable})
- ensure that no SIM translation occurs during the entry leg pad time.
- ensure that the RADMON disable command is issued at least 10 ks before entry into the radiation belt but no earlier than 15 ks
- ensure that no SIM translation occurs during the radiation zone transit
- ensure that no SIM translation occurs during the exit leg pad time
- ensure that there is no SIM translation during an ACIS science run
- ensure that the RADMON enable command is issued at least 10 ks after exit from the radiation belt but no later than 15 ks
- ensure that there is a 10 ks pad time on exit form the radiation zone (defined as pad = {time of RADMON enable} - {time of radiation zone exit})
- Coupled with 2 of the above tests, the script ensures that the RADMON disable occurs before radiation zone entry and that RADMON enable occurs after radiation zone exit.
2. Automatic ACIS tests
- ensure an OBSID change occurs at least 3 minutes after a stop science
- check that there are at least as many occurrences of OBSID changes as there are ACIS start science commands
- check that there is at least one WSVIDALLDN for every start science
- check that there is at least one RS_0000001 for every start science
- check that there is at least one RH_0000001 for every start science
- check that there is at least one ACIS stop science command for every ACIS start science command
- check to see if the instrument is in format (FMT) 2 if ACIS is in the focal plane. (the observatory runs in format 2 when ACIS is in the FP and format 1 when HRC is in the FP)
- check to see that we are in FMT 2 when doing a CTI measurement
- check to see that we are in the HRC-S position prior to radiation belt entry
3. Gratings tests
- ensure that there is a low energy transmission grating (LETG) retraction for every LETG insertion
- ensure that there is a high energy transmission grating (HETG) retraction for every HETG insertion
Once a load has been approved for upload to the spacecraft, the ACIS
Ops scientist on duty will run ACE-update.pl to update the ACIS Focal
Plane history files.
Please refer to the following documents for further insight into the
ACIS Load Review. Keep in mind that the methods outlined in these
documents may be out of date and you should refer to this website for
the most recent methods. The information in these documents does,
however, detail the steps involved in manually performing a load
review.
- Shanil's email
explaining the operation of acis-backstop.pl
- HOWTO_do_loadreviews-text
file from April 2001 describing how to manually perform load review
- Shanil's memo
(PS)
explaining in more detail the operation of acis-backstop.pl and the manual
load review process