ACIS Update: Twenty-Five Years of Operations

Paul Plucinsky for the ACIS Operations Team

The Advanced CCD Imaging Spectrometer (ACIS) continues to function superbly, producing spectacular results over the past year. All ten CCDs are fully functional and all electronics are nominal, operating on the primary units. The ACIS flight software (SW) was updated on 19 September, 2023, to correct a bug that could lead to an infinite loop during the bias map computation in alternating exposure mode, to accurately report the state of the software in all cases after a safing action, to report more diagnostic information in the case of an unexpected condition, and to force a restart and reloading of the software if a front-end processor powered off unexpectedly. These updates will make ACIS more operationally robust and will help minimize any downtime for some anomalous conditions.

The space weather environment has become more active as the Sun approaches the peak of this Solar cycle. Chandra has interrupted the observing schedule six times from November 2023 through June 2024 due to high radiation levels. Two of these radiation safing events were triggered by the monitor in the ACIS flight SW called “txings.” The first of these txings triggers represents the first such event since the algorithm was modified in August 2022.

As of this writing, ACIS has performed over 29,000 science runs in the mission. Typically there is one science run per observation, so this is approximately the number of observations ACIS has executed, including calibration and diagnostic runs. As most Chandra users know, ACIS is an incredibly flexible instrument with many parameters that define the instrument configuration. Chandra users can tailor these parameters to select the active CCDs, clocking mode, event telemetry format, spatial region, energy range, and time resolution most appropriate for their science goals. To date 1,552 unique ACIS configurations have been created over the mission, most to optimize the science of the General Observers (GOs). All indications are that ACIS is capable of producing more spectacular results for many years.

Of course, these impressive accomplishments do not happen by accident. They are the result of the diligent efforts of a dedicated group of people who work within the Chandra Science Operations Team (SOT) and Flight Operations Team (FOT) at SAO, MIT, and Northrop Grumman (NG). The ACIS Operations Team (AOT) has been responsible for ensuring the health and safety and proper configuration and operation of the instrument since launch. The makeup of the team has changed over the mission, but the dedication to the successful operation of ACIS has remained constant. As we celebrate twenty-five years of discovery with Chandra, it is appropriate to recognize the efforts of the engineers and scientists who have made ACIS the tremendous success that is today by reminiscing about a few memorable moments in the mission.

The ACIS Door Anomaly and Opening in Orbit

There were many exciting “once in a mission” moments in the Summer of 1999, starting with the launch of Chandra on the Space Shuttle Columbia. The first such event for ACIS occurred while Chandra was still in the shuttle cargo bay before the satellite had been deployed. We needed to power-on ACIS to open the “little vent valve” in order to prevent water from accumulating on the focal plane. After deployment, a major milestone was the first diagnostic runs with all 10 CCDs demonstrating that they had survived launch. These crucial, first-time events are too numerous to describe in this article, but one activity deserves more reflection: the opening of the ACIS door. The ACIS door opening was already a critical event on the Chandra orbital activation timeline, but it became an even higher profile event after the door mechanism experienced a failure during ground testing. This failure occurred on the last day of the spacecraft thermal vacuum testing at TRW (now NG), and, with Chandra nominally four months from launch, it represented a major crisis for the Chandra project. The Science Instrument Module (SIM) had to be removed from the spacecraft, ACIS removed from the SIM, the door mechanism replaced, tested, and demonstrated to work, ACIS then had to be installed on the SIM, and the SIM installed on the spacecraft again. After this work had been completed, the spacecraft functional tests had to be repeated. Remarkably, all of this work was done within six months and Chandra was ready to ship to the Cape for launch thanks to the extraordinary efforts of the technicians and engineers at Martin Marietta (MM), Ball Aerospace, and TRW.

Unfortunately, the cause of the door mechanism failure was never determined (Podgorski, Tice, & Plucinsky 2000 and Kahan et al. 2001); there were several plausible ideas, but none could be proven with any certainty. Thus, there was significant anxiety in the project as the day approached to open the door, knowing that the cause had never been identified and the success of the Chandra mission depended on the door opening. Neil Tice was the engineer at MM who was responsible for the design of the door mechanism. Neil was confident that the door would open as planned and wanted to be with us in the control room for this critical activity. We developed a conservative procedure for opening the door that would operate the mechanism in stages, as opposed to opening the door in one step. This approach would allow the behavior of the mechanism to be verified after each step, offering us the possibility to pause or stop the procedure if unexpected behavior was observed. Our plan was to operate the mechanism five times, and only after the fifth activation should the door have been completely open.

The door opening activity commenced on 8 August, 1999, during the day shift (Chandra operations were 24-hours-a-day so soon after launch). On that day, there was a significant crowd of interested observers in the ACIS operations room—which we affectionately referred to as the “ACIS Chapel,” since the room was a converted coat closet attached to the Technical Support Team room in the Hampshire Street Operations Control Center. The operations team for this activity included Neil Tice (MM), Ed Bougahn (the MIT engineer tasked with the responsibility for the flight operation of the door mechanism), Andy Northrup (the FOT engineer tasked with developing the flight procedure and products), and myself as the SOT ACIS lead, in addition to the FOT & SOT members who were controlling and monitoring the spacecraft. The key component of the door mechanism is a paraffin actuator, in which the paraffin is heated and expands, thereby doing work on a linkage that rotates a shaft to open the door. It was this paraffin actuator that was discovered to have failed in the ground testing.

The first operation of the mechanism on-orbit was designed to power on the heater to begin heating the wax, but to then turn the heater off before any motion of the door would be initiated. The heater was turned on for 90 seconds in this first step and the maximum temperature of the paraffin matched pre-flight expectations, but, as expected, no motion of the door was observed. This first step ensured that the commands worked, that the electrical connections to the actuator were still intact after launch, and that the paraffin was behaving as it had in ground tests. After the wax had cooled back to its original temperature, the next step was to turn the heater on for 110 seconds. During this heating cycle, the wax reached a temperature at which it was expected to begin to rotate the shaft to open the door. We observed that the angle of the door with respect to the ACIS camera body had increased from zero to ~11 degrees. This was a crucial result as we now knew that the mechanism was working and had overcome any stiction between the O-ring and the top of the camera body. But the door was only open 11 degrees—we had more work to do!

The next cycle lasted 125 seconds and the door angle increased to 19.5 degrees: an encouraging result, but still far from what we needed for a successful mission. The fourth cycle lasted 140 seconds, and the door angle increased to 36.5 degrees. At this point, we were confident the mechanism was working, but we did not have a fully open door that would allow X-rays from the High-Resolution Mirror Assembly (HRMA) to reach the ACIS focal plane (FP). We were so close, but not there yet. The fifth cycle lasted 200 seconds, and when it finished the door angle had reached the fully open position. After two intense hours, the operations team could relax and the entire Chandra team could celebrate: the ACIS door was now open. Neil was right, the door mechanism worked as expected. Chandra and ACIS would see “first light” a few days later on 12 August, 1999, when the HRMA Sunshade door was opened and we discovered Leon X-1.

Commissioning and Discovery of Soft Proton Damage

The ACIS activation and checkout activities proceeded rapidly after the door was opened. The flight SW was updated to the latest patch version, all instrument modes were verified, and all functional tests were completed. The instrument was performing flawlessly, and the quality of the data was excellent. Now the exciting work of using ACIS with the finest X-ray telescope ever constructed could begin. As the calibration program to characterize the performance of the HRMA+ACIS system started, the team was delighted that all of the hard work of the previous years had culminated in a successful activation on-orbit.

However, on Labor Day weekend of 1999, it was recognized that the performance of the frontside-illuminated (FI) CCDs was significantly worse than both what it had been just a few weeks earlier when the door was closed and what had been measured before launch. The backside-illuminated CCDs did not exhibit any degradation, a result that was puzzling at the time—but that provided a clue as to the cause. This discovery sparked a massive effort by the entire Chandra team to understand the origin of the degradation. Within a matter of weeks, it was determined that protons with energies around 100 keV were reflecting off of the HRMA surfaces with sufficient efficiency and propagating to the FP (Kolodziejczak et al. 2000). These protons penetrated deep enough in the FI CCDs to damage the buried channel region near the FI surface (Prigozhin et al. 2000a,2000b), resulting in an increase in the charge-transfer inefficiency (CTI).

The intensity of these soft protons was highest when Chandra transited the radiation belts and during Solar storms. It was determined that the best course of action was to translate the SIM every radiation belt passage and during severe Solar storms to move ACIS out of the focus of the HRMA (O’Dell et al. 2003). These operational changes were implemented in early October 1999 and have been executed every orbit since then. The AOT worked long hours from September to December to characterize the current performance of the CCDs, to experiment with alternate clocking sequences to mitigate the effect of CTI, and to understand the structure of the radiation belts to better protect ACIS. Once these operational changes were implemented, the rate of CTI increase reduced to the low value expected before launch. By the holiday season of 1999, the crisis had abated and operations returned to a more normal schedule. We were all relieved that the instrument performance had stabilized and looked forward to the nominal science phase of the mission.

A prime responsibility of the AOT has been to ensure that every command load for Chandra properly safes ACIS for the radiation belt transit (Virani et al. 2002 and DePasquale et al. 2002). In addition, the AOT and SOT carefully monitor the Solar weather and take appropriate actions to safe ACIS during severe storms (such as the historic storm in May 2024). These efforts continue to this day and have minimized the CTI increase to tolerable levels, enabling ACIS to continue to produce valuable data after twenty-five years in orbit.

Contamination

The next major challenge for ACIS operations and the calibration team was the discovery of the decrease in the low energy sensitivity in early 2002 (Plucinsky et al. 2003). This led to another large effort by the Chandra project to understand the cause of the degradation. It was determined that a layer of a mostly hydrocarbon contaminant with some oxygen and fluorine was accumulating on the ACIS optical-blocking filter (OBF; Marshall et al. 2003). The ACIS instrument had been designed to “bakeout” the instrument to remove such a contamination layer. The bakeout conceived before launch would raise the FP temperature from −120 °C to +30 °C and raise the camera body temperature from −60 °C to +25 °C. The MIT ACIS team conducted ground experiments that showed that the CTI of the FI CCDs would increase significantly if the detectors were to be warmed from −120 °C to +30 °C (Grant et al. 2005), which were consistent with the flight experience in which the CTI of the FI CCDs increased when the FP temperature was increased to +30 °C. Therefore, a new bakeout scenario needed to be developed which kept the FP temperature below −80 °C while also making the OBF as warm as possible to drive off the contaminants.

Such a scenario was developed (Plucinsky et al. 2004) with detailed thermal modeling from Neil Tice. The efficacy of such a bakeout was modeled (O’Dell et al. 2005, 2017) and found to depend sensitively on the volatility of the contaminant. If the contaminant has a high volatility, a bakeout to these temperatures would remove most or all of the contaminant, but if the volatility is low, little or none of the contaminant would be removed. Given the uncertainty on the outcome of the bakeout and the potential risk to the OBF and the CCDs, the Chandra project decided not to proceed with a bakeout. The AOT spent considerable effort in studying a bakeout and developing an operational scenario to achieve the desired conditions for the bakeout, but once the cause was understood and the additional absorption could be modeled accurately, the crisis had ended and the AOT could return to nominal operations.

The radiation damage early in the mission and the contamination issue represented the most serious challenges to ACIS operations; however as the mission has progressed, the ever-increasing temperatures of the ACIS electronics and FP have required the most attention from the AOT. A multi-pronged approach has been adopted that allows ACIS to continue to deliver breakthrough science while ensuring the instrument remains within safe operating temperatures. The steps that have been adopted are limiting the length of a single observation at a given solar pitch angle, reducing the number of operational CCDs, and specifying different temperature limits depending on the science objectives of a particular observation. These strategies have been highly successful at maintaining the high observing efficiency of Chandra and ACIS and providing the highest quality data to the Chandra observers. The AOT will continue to employ these strategies in the coming years to maintain the spectacular results from ACIS.

A Change of Venue

A major event in the history of Chandra was the move of the Operations Control Center (OCC) from the Hampshire St. facility in Cambridge to the Wayside facility in Burlington. This transition took several years to plan and was executed in 2019. This move produced a tremendous amount of work for the Chandra Ground Operations Team (GOT) to first configure, populate, and test the new OCC, and then to maintain two OCCs until operations could transition to the Wayside facility for good. The Hampshire St. facility had the unique advantage for the AOT that most of the ACIS engineering team was located in the same building, one floor below the OCC. This proximity made meetings and the support of real-time activities especially convenient for the MIT staff. The AOT is of course dependent upon the Chandra GOT for all of the infrastructure that makes the OCC work, but the AOT team is responsible for maintaining machines in the OCC that are used to support real-time activities with ACIS, such as recovery from anomalies and flight SW patches. We were quite comfortable in our home at Hampshire St. for the first 19 years of the mission, so the move was undertaken with some sadness. Nevertheless, new ACIS machines were purchased, configured, and installed in the new Chandra OCC at Wayside. The AOT had full functionality at the new OCC from the first day of operations at that facility thanks to the hard work of many people on the Chandra project. The facility has supported several ACIS realtime activities with the same level of support as the old OCC, most notable among these being the last two ACIS flight SW patch activities in August 2022 and September 2023. The Wayside OCC stands prepared to support any ACIS realtime activities on short notice.

ACIS has explored the X-ray universe for twenty-five years, fundamentally changing our understanding of astrophysics in many areas. It has been a great pleasure working on ACIS, but what has truly made the experience wonderful are all the people who have dedicated themselves to make ACIS the success that it is. It has been an honor and a privilege to work with this exceptional group of individuals. We look forward to continuing ACIS operations for many years in the future!

References

• DePasquale, J. M., Virani, S. N., & Plucinsky, P. P. 2002, Proc. SPIE, 4844, 454.
• Grant, C. E., LaMarr, B., Prigozhin, G. Y., et al. 2008, Proc. SPIE, 7021, 702119.
• Kahan, M. A., Tice, N. L., Podgorski, W. A., Plucinsky, P. P., & Doyle, K. B. 2001, Proc. SPIE, 4198, 186.
• Kolodziejczak, J. J., Elsner, R. F., Austin, R. A., & O'Dell, S. L. 2000, Proc. SPIE, 4140, 135.
• Marshall, H. L., Tennant, A., Grant, C. E., et al. 2004, Proc. SPIE, 5165, 497.
• O'Dell, S. L., Blackwell, W. C., Cameron, R. A., et al. 2003, Proc. SPIE, 4851, 77.
• O'Dell, S. L., Swartz, D. A., Plucinsky, P. P., et al. 2005, Proc. SPIE, 5898, 313.
• O'Dell, S. L., Swartz, D. A., Tice, N.W., et al. 2017. Proc. SPIE. 10397.
• Podgorski, W.P., Tice, N.W., & Plucinsky, P.P. 2000, AMS, 34, 15.
• Plucinsky, P. P., O'Dell, S. L., Tice, N. W., et al. 2004, Proc. SPIE, 5488, 251.
• Plucinsky, P. P., Schulz, N. S., Marshall, H. L., et al. 2003, Proc. SPIE, 4851, 89.
• Prigozhin, G. Y., Kissel, S. E., Bautz, M. W., et al. 2000a, Proc. SPIE, 4012, 720.
• Prigozhin, G. Y., Kissel, S. E., Bautz, M. W., et al. 2000b, Proc. SPIE, 4140, 123.
• Virani, S. N., Ford, P. G., DePasquale, J. M., & Plucinsky, P. P. 2002, Proc. SPIE, 4844, 464.