Monitoring and alerts.
ACIS Ops Response to SCS107.
When to call a telecon.
How to call a telecon.
The ACIS Ops team learned in the first weeks of the Chandra mission that the
camera's front illuminated (FI) chips are highly vulnerable to damage
from low energy protons at roughly 100 keV. Such protons cause
displacement damage in the Si lattice producing charge traps which
significantly increase the CTI. A single belt passage increased
the FWHM by 20 eV in FI chips at -120C. As detailed in
MIT's report, the expected damage from belt passages
varies markedly as Chandra's orbit changes. 1999-2001, 2010-2015, and 2020-2026
are the worst periods.
ACIS Ops team continuously monitors space weather
in an attempt to gauge Chandra's present and near future radiation
environment. Radiation levels of concern can arise from two sources:
Trapped radiation belts.
The ACIS instrument must be protected through each perigee passage
through the radiation belts:
- The SIM must be translated to the HRC-S position while the
Chandra Radiation Model (CRM) predicts proton fluxes above
threshold. The CRM determines the radiation belt entry and exit
times, which often but not always align with the EEF1000 and
XEF1000 orbital events, as determined by NSDDC's AE-8 electron
model. Translations to and from HRC-S regularly occur at pad times before
entry and after exit. CTI measurements are taken during these pad
times. More recently, since 2020, pad times may be omitted in effort to extend
the science orbit up to the modeled proton threshold in the orbit.
- The "backstop pad time" runs from Radmon disable to EEF 1000,
and from XEF 1000 to Radmon enable. The "CRM pad time" runs from
the time of soft proton entry of the belts according to the CRM
model (lengthened up to a maximum of 10 ks if ACIS Ops consents) to
the AE8 predicted time of electron radiation belt entry.
- At all points in the orbit, the radiation monitor must be enabled
whenever ACIS is in the focal plane. During radiation belt
crossings, the radiation monitor should be disabled once the SIM
translation to HRC-S is complete.
- Video boards must be powered down during belt passages.
There should be a minimum of around 2 hours between powerdown
This requirement is enforced by diligent review of perigee passages in
weekly command loads.
- High solar background.
In order to limit increase in the full width half max at the top of an
FI chip to an annual
0.1%, the ACIS operations team has established a radiation exposure budget for
for ACIS: an annual accumulated
fluence of 2e10 protons/cm2-ster-MeV.
As a rule of thumb, expecting 8 to 10 major events a year, we allow
ourselves to accumulate a fluence of about 2e+9 for any one
When unsafe radiations are anticipated or detected, either on-board radiation monitors
or ground-based commanding must issue an SCS-107 command. Crucially, this safing
action translates the SIM to HRC-S position and shuts down the ACIS vid-boards,
keeping ACIS as safe as possible from radiation damage.
Warning of a high radiation environment may come from several sources,
most of them handily gathered at CXC's
MTA Radiation Central page (older site version here may still have some useful links: OLD Radiation Central).
The following summary moves from data with the longest to the shortest
Our best long look-ahead comes from tracking sunspots
on the near and far faces of the Sun. Spots take about 26 days to
make an (apparent, from the Earth's viewpoint) rotation across the
Sun; a couple days less at the equator and more near the poles.
Solar X-rays and CMEs
In the event of an X class solar flare, NOAA's Space Weather Prediction
Center will send an email alert to acisdude with
the subject line "SUMMARY: X-Ray Event exceeded X1". That's our cue to
jump to Today's Space Weather,
which displays the last three days of solar X-ray flux in a logarithmic
graph. Each horizontal line represents a factor of 10 increase in
energy. Flares below M-7 or so are unlikely to give ACIS any
grief. If they're above that level, we need to start keeping an eye on
ACE and GOES rates. The most energetic particles may arrive in 0.5 to
3 days, depending in part on where the flare and associated CME were directed.
Another useful page to monitor is the The Space Weather Dashboard Page.
The first plot near the top displays the last couple days of GOES X-ray data; to the right is a 3-day forecast, and further down are useful "windmill" models.
In the past, Shanil received email notices of Coronal Mass Ejections (CMEs)
from firstname.lastname@example.org, which he forwarded to the rest
of us. More recently, SpaceWeatherAlerts is used to provide helpful
and rapid notifications. Jack is subscribed and automatically forwards their alerts to the acisdude email list.
velocity of CMEs is variable, but if one is
Earth directed, it typically takes two to five days for the ejected
mass to arrive, and several hours for it to sweep past Earth. Note that the arrival of the bulk CME ejecta is
generally distinct and delayed from the most energetic
particles described in the paragraph above.
The warnings most indicative of current Chandra conditions are the
alerts from MTA based on ACE fluence monitoring. The ACE
instrument orbits at Earth-Sun L1, so that it experiences solar winds
and storms about an hour in advance of the Earth. The ACE page
(SWPC ACE page here)
and charts in realtime of electron
and proton fluxes at a variety of energies. We are interested in the
P3 proton channel (115 - 195 keV). So as not to rely on a single
detector, the two derived P3 numbers, conservatively scaled from the P5 and P6
channels can also trigger messages. Alerts types are as follows:
- ACIS flumon alerts, when fluence for
the orbit exceeds 1.0e9 and 2.0e9. Notice had been to acisdude only, but recently (Jan 2021) has been expanded to sot_ace_alert.
- MTA high-flux alerts when the fluence integrated over two-hours exceeds 3.6e8. This more
urgent condition will send text and email alerts to the
The GOES satellites are in geosynchronous orbits about 38K
kilometers up, with GOES-R (aka GOES-16) monitoring
near-Earth proton levels, bearing
in mind that Chandra's inclination exposes us to a different cross
section of the magnetosphere.
The GOES-R alert condition is being investigated with values TBD.
Previously, MTA sent a yellow alert when GOES-11 P2 > 30 and P5 > 0.25; and
red alerts for P2 > 90.9 and P5 > 0.70. The red limits are designed to
indicate a probable trip of the old P4GM (P2) and P41GM (P5) EPHIN
channels. (NB: GOES definitions of P2 and P5 channels differ from the
ACE definitions of P3, etc.)
Kp is a measure of the magnetic
field at the surface of the Earth, averaged from data at various
ground stations around the globe. Increases in the Kp indicates that
the particle density in the radiation belts is changing.
In the past, MTA sent notification through sot_yellow_alert when Earth's
near-term Kp index as projected by the
Kp index model was high. This is not currently being done.
EPHIN EPHIN was the primary radiation monitor on Chandra but is no longer functional, having been decommissioned on day 2018:250.
HRC With EPHIN now defunct, we
rely on the High Resolution Camera's anticoincidence
shield as our on-board radiation detector. This has severe drawbacks:
the HRC can detect only high energy protons, and it saturates before
it reaches levels corresponding to the EPHIN's red limits. The prospects were
discussed in the 2005 Flight
Note 443, with a breakdown of results of adopting the
highest possible threshold value in a 2006 memo
from Mike Juda.
Upon an HRC anti-co shield radiation limit-crossing, HRC automatically triggers an on-board SCS-107 command.
As of Jan. 2021, it has initiated 9-10 such safing actions.
ACIS TXINGs ACIS on-board TXINGs are used
as a tertiary radiation monitor and can also be used to trigger autonomous radiation shutdowns. This
monitor backs up the HRC anticoincidence shield, but it should be noted that threshold
crossings are far from ideal as a radiation damage indicator, and also that this monitor is only active
when ACIS is running. Thresholds crossings reveal
significant elevation in particle background, which is not a direct indicator of
the presence of CTI-inducing particles. More details are available in these
SPIE publications led by Peter Ford and Catherine Grant: here and here.
As of Jan. 2021, TXINGs has twice autonomously safed Chandra.
SCS 107 notification
When an SCS-107 is commanded, or RADMON executes one spontaneously and
the fact becomes apparent on the next contact, MTA sends an email to
sot_red_alert indicating the time at which the SCS107 DISA status was
noted in realtime telemetry. (This will not ordinarily be the same as
the time at which the SCS-107 occurred. The OC will usually promulgate
that actual time before the contact finishes.)
As a cross-check that the SCS-107 is real, our Realtime page will show
DPA-A and DPA-B currents of about 0.43 and 0.28 respectively, and the
SIM at HRC-S.
Events that had been scheduled in the daily load before the SCS-107
will continue to show up on the Replan Central page,
but greyed out.
Confirm that ACIS is safe via our realtime page:
If the SIM fails to translate, run the
SOP_UNSAFE_ACIS PHASE 1.
SIM is at the HRC-S position -99616
DPA input currents indicate that 3 FEPs are powered on:
1DPICACU ~ 0.74 A, 1DPICBCU ~ 0.92 A
DEA time-averaged power is less than but near 50 W
Status bits indicate that a science run is not active, the last boot
was a normal boot and the flight SW is still running:
1STAT1ST = 1(green) science not active
1STAT2ST = 1(green) normal boot
1STAT0ST = 0 or 1 toggles every 64s to indicate SW is running
If other checks fail, consult the anomaly pages. There is a known class of anomalies in which the
1STAT1ST bit fails to set. This is expected to have been fully fixed by the updated buscrash patch included with FSW Std G Opt I (v58). But for completeness, these are documented under
ACIS Science Run Termination Failure Anomaly.
Check the Fluence monitors (on the RT pages) to determine how much ACIS has accumulated
already. These are based on the ACE Flux.
Run history-files.pl -s107 with the SCS 107 time.
webpage instructions for radiation replan reviews are less
detailed but more up-to-date.)
Be ready for the replan. Review the CTI RTS and be ready to write one
up if there is enough time.
In the event of an SCS-107, ACIS is safe, and the Engineering Manager
or the Flight Director will call a telecon.
A radiation alert has come through and P3 channel rates have been
rising. ACIS should assess the radiation levels to determine whether or not to call a telecon.
Reply to the same list that was notified if an update is warranted (such as alerting that all is safe an no telecon will be held).
Do the following calculations:
- Determine exposureTimeToRadzone (i.e., to Radmon Disable). (If we're already
in the radzone, this is the RadmonDisable for the next orbit.)
Subtract from these times any period spent at HRC-S or HRC-I, and
divide by 2 or 5 respectively when the LETG or HETG is in place.
- Determine exposureTimeToContact, and
exposureTimeTo2ndContact (i.e., the next
commanding opportunity, and the one following that.) Make the same
adjustments for SIM position and gratings as in (1); namely, subtract
from these times any period spent at HRC-S or HRC-I, and divide by 2
or 5 respectively when the LETG or HETG is in place.
- Is the ACE flux clearly falling?
If so, set "CalcFlux" to the
current reading. Use the real P3 channel.
Otherwise, extrapolate the flux to the next UTimeToRadzone or
UTimeToContact (whichever is first) and set "CalcFlux" to that value.
RadZFlu = current orbital fluence + CalcFlux * timeToRadzone
ContFlu = current orbital fluence + CalcFlux * timeToContact
NxtContFlu = current orbital fluence + CalcFlux *
Gregg has a GUI interface that can help with the computations: see this memo page.
If there's a significant radiation risk, the default is to call a telecon via
sot_red_alert before the next contact. However, if the alert comes
outside of reasonable (~10pm to 7am) hours and
then email sot_yellow_alert and the alert email list (e.g., sot_ace_alert), and call for the telecon the next
morning, between 7 am and the first contact opportunity.
- ContFlu < 1.0e9 and the first contact occurs after 7 am or
- NxtContFlu < 1.0e9 and the following contact occurs after 7 am
Finally, if ACE flux is clearly falling, and both RadZFlu and NxtContFlu are
below 1.0e9, you may defer a decision on calling a telecon and continue to
monitor, and should send a notification to this effect. A template for
the radiation notice is available here.
The purpose of calling this telecon is for ACIS to brief the rest of operations
and argue the necessity of a shutdown. If time allows, first notify acisdude-text
to organize a quick cross-check with the team.
Upon agreement, send an email to sot_red_alert. Radiation telecons will ordinarily be
on the contingency number (844) 467-6272 (111165#) or at the corresponding Google-meet.
Required personnel to make a decision on whether to shut down are a
flight director, and representatives from FOT engineering and FOT
MP. Desirable are people from PCAD (either Eric or Tom), SOT MP, SOT
lead, and Steve O'Dell from NASA.
SCS-107 EXCUTION AND NON-LOAD TRACKING
If an SCS-107 occured - either manually or automatically - you want
to record that Non-Load
Event. When execution of the SCS-107 begins, be sure to note the time.
The instructions for running the Non-Load Event Tracker can be found
Instructions for writing CAP for CTI using an RTS
Template for Radiation Alert Response.
Joe's Summary on radiation response doc,
Rad response Ex. 1 doc,
Rad response Ex. 2 doc,
Rad response Ex. 3 txt,
There are more examples of radiation response, and memos about the decision process to be found off the ACIS memos page, under "Radiation Shutdown Decisions," which leads you to this page.
Here is a link to the MTA archive of old radiation shutdown plots and information. It's also off the SOT Homepage under Radiation, and Radiation Event Archive. The archive of Radiation Events.
There's a tutorial on space weather on the ACIS twiki, here.
Royce Buehler, last update Sep 2017
Richard Edgar, last update Apr 2019
Jack Steiner, last update Aug 2022