Introduction.
Monitoring and alerts.
ACIS Ops Response to SCS107.
When to call a telecon.
How to call a telecon.
More links

Introduction

The ACIS Ops team learned in the first weeks of the Chandra mission that the camera's front illuminated (FI) chips are highly vulnerable to damage from low energy protons at roughly 100 keV. Such protons cause displacement damage in the Si lattice producing charge traps which significantly increase the CTI. A single belt passage increased the FWHM by 20 eV in FI chips at -120C. As detailed in MIT's report, the expected damage from belt passages varies markedly as Chandra's orbit changes. 1999-2001, 2010-2015, and 2020-2026 are the worst periods.

ACIS Ops team continuously monitors space weather in an attempt to gauge Chandra's present and near future radiation environment. Radiation levels of concern can arise from two sources:
  1. Trapped radiation belts. The ACIS instrument must be protected through each perigee passage through the radiation belts:
    1. The SIM must be translated to the HRC-S position while the Chandra Radiation Model (CRM) predicts proton fluxes above threshold. The CRM determines the radiation belt entry and exit times, which often but not always align with the EEF1000 and XEF1000 orbital events, as determined by NSDDC's AE-8 electron model. Translations to and from HRC-S regularly occur at pad times before entry and after exit. CTI measurements are taken during these pad times. More recently, since 2020, pad times may be omitted in effort to extend the science orbit up to the modeled proton threshold in the orbit.
    2. The "backstop pad time" runs from Radmon disable to EEF 1000, and from XEF 1000 to Radmon enable. The "CRM pad time" runs from the time of soft proton entry of the belts according to the CRM model (lengthened up to a maximum of 10 ks if ACIS Ops consents) to the AE8 predicted time of electron radiation belt entry.
    3. At all points in the orbit, the radiation monitor must be enabled whenever ACIS is in the focal plane. During radiation belt crossings, the radiation monitor should be disabled once the SIM translation to HRC-S is complete.
    4. Video boards must be powered down during belt passages. There should be a minimum of around 2 hours between powerdown and perigee.

    This requirement is enforced by diligent review of perigee passages in weekly command loads.

  2. High solar background. In order to limit increase in the full width half max at the top of an FI chip to an annual 0.1%, the ACIS operations team has established a radiation exposure budget for for ACIS: an annual accumulated fluence of 2e10 protons/cm2-ster-MeV.

    As a rule of thumb, expecting 8 to 10 major events a year, we allow ourselves to accumulate a fluence of about 2e+9 for any one radiation event.

When unsafe radiations are anticipated or detected, either on-board radiation monitors or ground-based commanding must issue an SCS-107 command. Crucially, this safing action translates the SIM to HRC-S position and shuts down the ACIS vid-boards, keeping ACIS as safe as possible from radiation damage.

Monitoring and Alerts

Warning of a high radiation environment may come from several sources, most of them handily gathered at CXC's MTA Radiation Central page (older site version here may still have some useful links: OLD Radiation Central). The following summary moves from data with the longest to the shortest predictive range.

Sunspot tracking

Our best long look-ahead comes from tracking sunspots on the near and far faces of the Sun. Spots take about 26 days to make an (apparent, from the Earth's viewpoint) rotation across the Sun; a couple days less at the equator and more near the poles.

Solar X-rays and CMEs

In the event of an X class solar flare, NOAA's Space Weather Prediction Center will send an email alert to acisdude with the subject line "SUMMARY: X-Ray Event exceeded X1". That's our cue to jump to Today's Space Weather, which displays the last three days of solar X-ray flux in a logarithmic graph. Each horizontal line represents a factor of 10 increase in energy. Flares below M-7 or so are unlikely to give ACIS any grief. If they're above that level, we need to start keeping an eye on ACE and GOES rates. The most energetic particles may arrive in 0.5 to 3 days, depending in part on where the flare and associated CME were directed.

Another useful page to monitor is the The Space Weather Dashboard Page. The first plot near the top displays the last couple days of GOES X-ray data; to the right is a 3-day forecast, and further down are useful "windmill" models.

In the past, Shanil received email notices of Coronal Mass Ejections (CMEs) from stenborg@kreutz.nascom.nasa.gov, which he forwarded to the rest of us. More recently, SpaceWeatherAlerts is used to provide helpful and rapid notifications. Jack is subscribed and automatically forwards their alerts to the acisdude email list. The velocity of CMEs is variable, but if one is Earth directed, it typically takes two to five days for the ejected mass to arrive, and several hours for it to sweep past Earth. Note that the arrival of the bulk CME ejecta is generally distinct and delayed from the most energetic particles described in the paragraph above.

ACE

The warnings most indicative of current Chandra conditions are the alerts from MTA based on ACE fluence monitoring. The ACE instrument orbits at Earth-Sun L1, so that it experiences solar winds and storms about an hour in advance of the Earth. The ACE page (SWPC ACE page here) provides tables and charts in realtime of electron and proton fluxes at a variety of energies. We are interested in the P3 proton channel (115 - 195 keV). So as not to rely on a single detector, the two derived P3 numbers, conservatively scaled from the P5 and P6 channels can also trigger messages. Alerts types are as follows:
  1. ACIS flumon alerts, when fluence for the orbit exceeds 1.0e9 and 2.0e9. Notice had been to acisdude only, but recently (Jan 2021) has been expanded to sot_ace_alert.
  2. MTA high-flux alerts when the fluence integrated over two-hours exceeds 3.6e8. This more urgent condition will send text and email alerts to the sot_ace_alert list.

GOES

The GOES satellites are in geosynchronous orbits about 38K kilometers up, with GOES-R (aka GOES-16) monitoring near-Earth proton levels, bearing in mind that Chandra's inclination exposes us to a different cross section of the magnetosphere.

The GOES-R alert condition is being investigated with values TBD. Previously, MTA sent a yellow alert when GOES-11 P2 > 30 and P5 > 0.25; and red alerts for P2 > 90.9 and P5 > 0.70. The red limits are designed to indicate a probable trip of the old P4GM (P2) and P41GM (P5) EPHIN channels. (NB: GOES definitions of P2 and P5 channels differ from the ACE definitions of P3, etc.)

Kp index

Kp is a measure of the magnetic field at the surface of the Earth, averaged from data at various ground stations around the globe. Increases in the Kp indicates that the particle density in the radiation belts is changing.

In the past, MTA sent notification through sot_yellow_alert when Earth's near-term Kp index as projected by the Kp index model was high.

This is not currently being done.

EPHIN

EPHIN was the primary radiation monitor on Chandra but is no longer functional, having been decommissioned on day 2018:250.

HRC

With EPHIN now defunct, we rely on the High Resolution Camera's anticoincidence shield as our on-board radiation detector. This has severe drawbacks: the HRC can detect only high energy protons, and it saturates before it reaches levels corresponding to the EPHIN's red limits. The prospects were discussed in the 2005 Flight Note 443, with a breakdown of results of adopting the highest possible threshold value in a 2006 memo from Mike Juda. Upon an HRC anti-co shield radiation limit-crossing, HRC automatically triggers an on-board SCS-107 command. As of Jan. 2021, it has initiated 9-10 such safing actions.

ACIS TXINGs

ACIS on-board TXINGs are used as a tertiary radiation monitor and can also be used to trigger autonomous radiation shutdowns. This monitor backs up the HRC anticoincidence shield, but it should be noted that threshold crossings are far from ideal as a radiation damage indicator, and also that this monitor is only active when ACIS is running. Thresholds crossings reveal significant elevation in particle background, which is not a direct indicator of the presence of CTI-inducing particles. More details are available in these SPIE publications led by Peter Ford and Catherine Grant: here and here. As of Jan. 2021, TXINGs has twice autonomously safed Chandra.

SCS 107 notification

When an SCS-107 is commanded, or RADMON executes one spontaneously and the fact becomes apparent on the next contact, MTA sends an email to sot_red_alert indicating the time at which the SCS107 DISA status was noted in realtime telemetry. (This will not ordinarily be the same as the time at which the SCS-107 occurred. The OC will usually promulgate that actual time before the contact finishes.) As a cross-check that the SCS-107 is real, our Realtime page will show DPA-A and DPA-B currents of about 0.43 and 0.28 respectively, and the SIM at HRC-S.

Events that had been scheduled in the daily load before the SCS-107 will continue to show up on the Replan Central page, but greyed out.

Response to SCS 107

Confirm that ACIS is safe via our realtime page: If the SIM fails to translate, run the SOP_UNSAFE_ACIS PHASE 1.

If other checks fail, consult the anomaly pages. There is a known class of anomalies in which the 1STAT1ST bit fails to set. This is expected to have been fully fixed by the updated buscrash patch included with FSW Std G Opt I (v58). But for completeness, these are documented under ACIS Science Run Termination Failure Anomaly.

Check the Fluence monitors (on the RT pages) to determine how much ACIS has accumulated already. These are based on the ACE Flux.
Run history-files.pl -s107 with the SCS 107 time. (The memos webpage instructions for radiation replan reviews are less detailed but more up-to-date.)
Be ready for the replan. Review the CTI RTS and be ready to write one up if there is enough time.

When to Call a Radiation Telecon

In the event of an SCS-107, ACIS is safe, and the Engineering Manager or the Flight Director will call a telecon. A radiation alert has come through and P3 channel rates have been rising. ACIS should assess the radiation levels to determine whether or not to call a telecon. Reply to the same list that was notified if an update is warranted (such as alerting that all is safe an no telecon will be held). Do the following calculations:

  1. Determine exposureTimeToRadzone (i.e., to Radmon Disable). (If we're already in the radzone, this is the RadmonDisable for the next orbit.) Subtract from these times any period spent at HRC-S or HRC-I, and divide by 2 or 5 respectively when the LETG or HETG is in place.
  2. Determine exposureTimeToContact, and exposureTimeTo2ndContact (i.e., the next commanding opportunity, and the one following that.) Make the same adjustments for SIM position and gratings as in (1); namely, subtract from these times any period spent at HRC-S or HRC-I, and divide by 2 or 5 respectively when the LETG or HETG is in place.
  3. Is the ACE flux clearly falling? If so, set "CalcFlux" to the current reading. Use the real P3 channel.
    Otherwise, extrapolate the flux to the next UTimeToRadzone or UTimeToContact (whichever is first) and set "CalcFlux" to that value.
  4. Calculate
    RadZFlu = current orbital fluence + CalcFlux * timeToRadzone
    ContFlu = current orbital fluence + CalcFlux * timeToContact
    NxtContFlu = current orbital fluence + CalcFlux * timeTo2ndContact.

Gregg has a GUI interface that can help with the computations: see this memo page.

If there's a significant radiation risk, the default is to call a telecon via sot_red_alert before the next contact. However, if the alert comes outside of reasonable (~10pm to 7am) hours and
  1. ContFlu < 1.0e9 and the first contact occurs after 7 am or
  2. NxtContFlu < 1.0e9 and the following contact occurs after 7 am
then email sot_yellow_alert and the alert email list (e.g., sot_ace_alert), and call for the telecon the next morning, between 7 am and the first contact opportunity.

Finally, if ACE flux is clearly falling, and both RadZFlu and NxtContFlu are below 1.0e9, you may defer a decision on calling a telecon and continue to monitor, and should send a notification to this effect. A template for the radiation notice is available here.

How to Call a Radiation Telecon

The purpose of calling this telecon is for ACIS to brief the rest of operations and argue the necessity of a shutdown. If time allows, first notify acisdude-text to organize a quick cross-check with the team. Upon agreement, send an email to sot_red_alert. Radiation telecons will ordinarily be on the contingency number (844) 467-6272 (111165#) or at the corresponding Google-meet.

Required personnel to make a decision on whether to shut down are a flight director, and representatives from FOT engineering and FOT MP. Desirable are people from PCAD (either Eric or Tom), SOT MP, SOT lead, and Steve O'Dell from NASA.

SCS-107 EXCUTION AND NON-LOAD TRACKING

If an SCS-107 occured - either manually or automatically - you want to record that Non-Load Event. When execution of the SCS-107 begins, be sure to note the time.

The instructions for running the Non-Load Event Tracker can be found here.

Linkfest

Instructions for writing CAP for CTI using an RTS
Template for Radiation Alert Response.
Joe's Summary on radiation response doc, pdf
Rad response Ex. 1 doc, pdf
Rad response Ex. 2 doc, pdf
Rad response Ex. 3 txt, pdf,

There are more examples of radiation response, and memos about the decision process to be found off the ACIS memos page, under "Radiation Shutdown Decisions," which leads you to this page.

Here is a link to the MTA archive of old radiation shutdown plots and information. It's also off the SOT Homepage under Radiation, and Radiation Event Archive. The archive of Radiation Events.

There's a tutorial on space weather on the ACIS twiki, here.

Document history:
Royce Buehler, last update Sep 2017
Richard Edgar, last update Apr 2019
Jack Steiner, last update Aug 2022