Anomalies in the Psyche Mission Fault Protection Performance CL25_0754.pdf

Anomalies in the Psyche Mission Fault Protection Performance

1. This document has been reviewed and determined not to contain export controlled technical data.
Anomalies on the Psyche
Mission: Fault Protection
Performance and Lessons
Learned
Presenter: Virginia
Sereno
Authors: Virginia
Sereno, Jonathan
Summer, Alexander
Lumnah, Swapnil Pujari,
Travis Imken, Steve
Snyder
2025 IEEE Aerospace Conference
March 3
rd
, 2025
Big Sky, MT
Jet Propulsion Laboratory,
California Institute of
Technology
© 2024 California Institute of Technology. Government sponsorship
acknowledged.

2. March 3
rd
, 2025 Psyche - 2This document has been reviewed and determined not to contain export controlled technical data. Friday October 13
th
,
2023
Credit:
SpaceX
This document has been reviewed and determined not to contain export controlled technical data.
Psyche Launch
You Are
Here

3. March 3
rd
, 2025 Psyche - 3This document has been reviewed and determined not to contain export controlled technical data.This document has been reviewed and determined not to contain export controlled technical data.
Psyche Mission Overview
March 6, 2024 Psyche - 3This document has been reviewed and determined not to contain export controlled technical data.
Science
Understand a previously unexplored
component of the early building blocks
of planets: iron cores
1.Determine whether Psyche is a core, or if it is
unmelted material
2.Determine the relative ages of Psyche’s
surface regions
3.Determine whether small metal bodies
incorporate the same light elements as are
expected in the Earth’s high-pressure core (Si,
K, S, C, O)
4.Determine whether Psyche was formed under
more oxidizing or more reducing conditions
than Earth’s core
5.Characterize Psyche’s topography
Salient Features
Category: 2
Risk Class:B
PP Classification: III
Target: (16) Psyche – Large metal asteroid
Instruments: Imager (ASU), Gamma Ray &
Neutron Spectrometer (APL),
Magnetometer (DTU)
Propulsion: Xenon Solar Electric, Nitrogen
Cold Gas
Spacecraft Partner: Maxar - Palo Alto
Tech Demo: Deep Space Optical
Communications (DSOC)
Launch: October 2023
Capture: August 2029

4. March 3
rd
, 2025 Psyche - 4This document has been reviewed and determined not to contain export controlled technical data.
Fault Protection and System Modes Overview
•Fault Protection (FP) provides the capability to
detect errors & respond to them, ensuring the
spacecraft remains in a safe state
•Safe mode provides a power positive, thermally
safe, commandable state in the presence of faults
–Sys_Standby_RWA – “light touch” safing utilizing HGA
comm and RWA control in 3J
–Sys_Safe_RWA – secondary level of safing utilizing
LGA comm and RWA control in 2S
–Sys_Safe_CGS – lowest level of safing utilizing LGA
comm and CGS control in 2S

5. March 3
rd
, 2025 Psyche - 5This document has been reviewed and determined not to contain export controlled technical data.
Preparation for Launch and Operations
Nominal and Off-Nominal Readiness Tests
Test: Nominal Readiness Test
Assessment: Pass
Description: team practiced applying the launch certified set of parameters,
transitioning the spacecraft to internal power supplies and assessing telemetry to
give a “GO” for transition to Launch mode.
Test: Off-Nominal Readiness Test
Assessment: Pass
Description: abbreviated version of the launch procedure, primarily focused on
recovering from the expected entry into cold gas safe mode and transitioning up
to the nominal operating mode with the intervention of “gremlins” on the ground
injecting faults to stress the team and the vehicle.
Fault Protection Parameter Management
The Psyche Spacecraft launched with 9108 system parameters in flight software
•Fault protection team oversaw definition of 3400 parameters to manage device health, primeness, and behavior of fault
monitors and responses
•Parameters to define monitor threshold, persistence, latch state and mask state; response timeout, tier, and mask
state.
•Fault protection parameters continue to change as the spacecraft is in flight and the team observes the behavior of the
system
•The spacecraft flight software was updated to version R6.0 in mid-2024 and added 42 new fault protection parameters to
support new monitors and responses
•FSW update provided the team had the opportunity to update the hard-coded values of nearly 400 other fault protection
parameters.
•While the spacecraft nominally operates with parameter values that are stored in dedicated parameter memory, flight
software will reference a parameter’s hard-coded value in the event this memory cannot be read on a flight software boot. 

6. March 3
rd
, 2025 Psyche - 6This document has been reviewed and determined not to contain export controlled technical data.
Operational Performance & Current Mission Status
•Fault Protection has performed nominally throughout Launch, ICO, and early
cruise.
–FP statistics are HEALTHY; All monitors are GREEN/BLACK.
–Expected device health and primness configuration persists throughout all nominal activities.
•Psyche spacecraft has successfully completed all the planned activities during launch and
initial checkout and is cruise thrusting on the way to (16) Psyche.
•The fault protection engine has recorded:
–Nominal conditions that do not persist to pose a threat to the spacecraft health and
safety.
–Off-nominal conditions that trigger fault protection.
•Cases where fault protection was invoked are categorized by root cause:
–Anomalies related to conservative threshold settings.
–Anomalies related to limitations in testing fidelity.
Findings from all anomalies to date conclude the union of FSW and the FP design executed per
expectations,
given the performance of the hardware and operating environment.

7. March 3
rd
, 2025 Psyche - 7This document has been reviewed and determined not to contain export controlled technical data.
Anomalies Related to Conservative Threshold
Settings
Fault Protection rightly triggered on a real overtemperature event, 25C above predicts. Based
on analysis of temperature trends by thermal experts a threshold increase was deemed
appropriate. 
•Background: Over temp.
event detected the day
after launch, temperature
sensors on the cover of the
DSOC docking mechanism.
•Problem: Sun-facing
orientation, warmer than
thermal model predicted.
•Corrective Action:
–Fault protection invoked two
tiers of response actions to
power off DSOC, mark it
SICK, and demote system
modes.
–Ground increased thresholds.
•Root Cause: overly loose
threshold settings, based
on thermal model predicts
that were violated in flight.
•Lessons Learned:
–Thermal modeling is
inherently difficult,
thresholds sizing should
account for sufficient margin
to prevent AFT violation.

8. March 3
rd
, 2025 Psyche - 8This document has been reviewed and determined not to contain export controlled technical data.
Anomalies Related to Conservative Threshold
Settings
The assessment of the timeout was based on thermal model predicts to protect against errant
thermal conditions causing issues. Determining true safety limits would have incurred a
significant cost, therefore thresholds were set with an abundance of caution.
•Background: Two weeks
into the mission,
unexpected condition
occurred during an activity
to release DSOC launch
locks.
•Problem: DSOC raised a
safety flag indicating the
temp. of a component did
not decrease in the
expected amount of time.
•Corrective Action:
–Fault protection invoked a
response to power off DSOC,
mark it SICK, with minimal
impact to operations.
–Ground adjusted internal
timeout settings on heating
and cooling cycles.
•Root Cause: thermal
predicts biased to protect
against errant thermal
conditions, abundance of
caution chosen.
•Lessons Learned:
–Thermal modeling is
inherently difficult,
thresholds sizing should
account for sufficient margin
to prevent AFT violation.

9. March 3
rd
, 2025 Psyche - 9This document has been reviewed and determined not to contain export controlled technical data.
Anomalies due to Limitations in Testing Fidelity
Analysis of the GNC Flood DP from January 3, 2024 showed repeated trips of the monitor during
commanded motion, coincident with pot samples that occur during rollover from the minimum
to the maximum value.
•Background: Two variables used to orient the
thrusters
1.Spacecraft attitude.
2.Dual Axis Positioning Mechanisms (DAPMs) – two motors, act
as elbow and shoulder joints to position thrusters.
•Each motor has a coarse potentiometer and 2 fine potentiometer.
•Problem: Dual Axis Positioning Mechanism (DAPM)
Fault during thruster check-out activity, ~2 months
after launch.
•Corrective Action:
–Fault protection invoked a response to mark the DAPM sick,
without interrupting ongoing spacecraft sequencing.
–Ground commanded generation of high-rate data product to
view DAPM motion in greater detail.
–Permanently masked monitors.
•Root Cause: Measurements taken when crossing the
pot null zone caused a spurious reading high enough to
trip the monitor.
–Monitor error test is not robust to null zone crossings.
–Null zones were not simulated in the testbed, so this behavior
could not be seen in system level testing.
–Unable to run full GNC algorithms on the vehicle during
prelaunch testing, so this behavior went unnoticed in that
venue as well.
•Lessons Learned:
–Limitations in testing fidelity can lead to unexpected behavior
in flight.
–Document differences between test environments to predict
vulnerabilities.
Monitor
trips when
null zone
reading is
within this
window

10. March 3
rd
, 2025 Psyche - 10This document has been reviewed and determined not to contain export controlled technical data.
Anomalies due to Limitations in Testing Fidelity
Findings indicate the root cause of the anomaly was due to performance characteristics of the
PPU at low power levels – Flight Software and the FP design executed per expectation given the
performance of the hardware.
•Background: Thruster startup process:
1.Initial Propellant flow rate supplied to the thruster.
2.Plasma discharge ignites.
3.Power electronics take over the flow control and maintain the
anode current setpoint.
•Problem: Anode current fault during characterization
activity of electric propulsion subsystem.
–Feb. 16, 2024, command to start the thruster at the lowest
throttle level.
–Initial propellant flow was less than expected due to
variabilities in the flow system.
–Anode current took more than 10s to reach the 3A threshold,
tripping fault protection.
•Corrective Action:
–Fault protection invoked a “light touch” safe mode entry
–Ground recovered in less than 8 hours
–Adjustment made to throttle table settings after reviewing
propellant flows.
•Root Cause: Incomplete understanding of
performance characteristics at low power levels.
•Lessons Learned:
–Differences between test environments and flight can result
in a misunderstanding of performance characteristics.
Nominal
startup
Monitor trips
after
meeting the
persistence
(10s)
without
reaching the
undercurrent
threshold

11. March 3
rd
, 2025 Psyche - 11This document has been reviewed and determined not to contain export controlled technical data.
Operational Processes
•To avoid unexpectedly tripping fault protection, and to respond to cases where fault
protection is invoked autonomously the operations team follows a data analysis and
trending process upon the receipt of downlinked from the spacecraft to drive down residual
risk.
•This review process is summarized into 3 subcategories:
1.State Changes: A review of all health, primeness, monitor statistics, response
statistics, and mask state channels to ensure the spacecraft has not changed its state
autonomously.
2.Error Count Trending: A data product which records the minimum and maximum value
for all telemetry channels is downlinked and reviewed daily - all nonzero monitor error
counts in these data products are dispositioned.
3.Threshold Trending: A comparison between the spacecraft telemetry samples and the
fault protection threshold settings is trended over the course of the mission to
determine if a telemetry sample is slowly approaching the fault protection limit over
time.

12. March 3
rd
, 2025 Psyche - 12This document has been reviewed and determined not to contain export controlled technical data.
Conclusion and Lessons Learned
•The Fault Protection engine, FSW, and Spacecraft operations team has performed
well throughout launch, initial checkout, and early cruise thrusting activities.
–Appropriately reacting to unexpected off-nominal conditions.
–Recorded and did not react to conditions below defined threshold and persistence settings
– protecting Psyche’s key resource - missed thrust time.
•Fortunate not to have experienced any “real” hardware failures in flight.
•Analysis of the cases where fault protection did trip concluded all root causes
could be grouped into the following categories:
–Overly conservative threshold settings.
–Limitations in testing fidelity that resulted in an incomplete understanding of the system
behavior.
•Findings indicate the importance of:
–Developing accurate models that fit within budgeting constraints.
–Identifying parameter settings where model predicts may have higher uncertainty.
–Review of testing data for expected as well as unexpected performance.
–Implementation of high-fidelity testing environments to closely match spacecraft
operating conditions.
Psyche Mission is unique in many ways – however lessons learned can be applied universally to
future deep-space missions, and should be considered early in the design process to drive
down risk.

13. March 3
rd
, 2025 Psyche - 13This document has been reviewed and determined not to contain export controlled technical data.
Psyche Fault Protection Launch Operations TeamPsyche Fault Protection Launch Operations Team

14. March 3
rd
, 2025 Psyche - 14This document has been reviewed and determined not to contain export controlled technical data.
Psyche Mission Overview
Science
Understand a previously unexplored
component of the early building blocks
of planets: iron cores
1.Determine whether Psyche is a core, or if it is
unmelted material
2.Determine the relative ages of Psyche’s
surface regions
3.Determine whether small metal bodies
incorporate the same light elements as are
expected in the Earth’s high-pressure core (Si,
K, S, C, O)
4.Determine whether Psyche was formed under
more oxidizing or more reducing conditions
than Earth’s core
5.Characterize Psyche’s topography
Salient Features
Category: 2
Risk Class:B
PP Classification: III
Target: (16) Psyche – Large metal asteroid
Instruments: Imager (ASU), Gamma Ray &
Neutron Spectrometer (APL),
Magnetometer (DTU)
Propulsion: Xenon Solar Electric, Nitrogen
Cold Gas
Spacecraft Partner: Maxar - Palo Alto
Tech Demo: Deep Space Optical
Communications (DSOC)
Launch: October 2023
Capture: August 2029
You Are
Here