Repairing Learning-Enabled Controllers While Preserving What Works

Presented at the 15th ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS 2024).

Abstract: Learning-enabled controllers have been adopted in various cyber-physical systems (CPS). When a learning-enabled controller fails to accomplish its task from a set of initial states, researchers leverage repair algorithms to fine-tune the controller's parameters. However, existing repair techniques do not preserve previously correct behaviors. Specifically, when modifying the parameters to repair trajectories from a subset of initial states, another subset may be compromised. Therefore, the repair may break previously correct scenarios, introducing new risks that may not be accounted for. Due to this issue, repairing the entire initial state space may be hard or even infeasible. As a response, we formulate the Repair with Preservation (RwP) problem, which calls for preserving the already-correct scenarios during repair. To tackle this problem, we design the Incremental Simulated Annealing Repair (ISAR) algorithm, which leverages simulated annealing on a barriered energy function to safeguard the already-correct initial states while repairing as many additional ones as possible. Moreover, formal verification is utilized to guarantee the repair results. Case studies on an Unmanned Underwater Vehicle (UUV) and OpenAI Gym Mountain Car (MC) show that ISAR not only preserves correct behaviors from previously verified initial state regions, but also repairs 81.4% and 23.5% of broken state spaces in the two benchmarks. Moreover, the average signal temporal logic (STL) robustnesses of the ISAR repaired controllers are larger than those of the controllers repaired using baseline methods.

1. Repairing Learning-Enabled Controllers
While Preserving What Works
Pengyuan Eric Lu
1
, Matthew Cleaveland
1
, Oleg Sokolsky
1
, Insup Lee
1
, Ivan Ruchkin
2

1
University of Pennsylvania, United States

2
University of Florida, United States


15
th
ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS 2024)
May 13, 2024

2. Outline
1.Motivation: What is NN control policy repair, and why?
2.Problem: Repair with Preservation (RwP)
3.Solution: Incremental Simulated Annealing Repair (ISAR)
4.Evaluation: Mountain car and unmanned underwater vehicle
2

3. 1. Motivation:
What is NN control policy repair, and why?
3

4. 4
•Safety-critical cyber-physical systems (CPS) had a market cap of $86 billion in
2022, with a 7.6% annual growth rate
•Increasingly, safety-critical CPS use learned control policies, implemented with
neural networks (NN)
Learning-Enabled Control Policies in CPS

5. Example: Unmanned Underwater Vehicle
5

6. Example: Unmanned Underwater Vehicle
6
•Objective: stay outside the danger zone for 30 sec
•Signal temporal logic (STL) [Maler 2004] specifies this as:
•What if the NN policy fails from some initial states s
0
?
•How do we repair the policy to fix those states?

7. Two Requirements of NN Repair
•Correctness – fulfill a formal specification
•Provably correct repair [Sotoudeh 2021, Cruz 2021, Fu 2022]: require strict assumptions,
e.g. limited to finite & known inputs, or specs like a bound on the NN’s Lipschitz constant
•What if no need to always guarantee repair? Perform best-effort repair instead
•Preservation of knowledge – do not unlearn useful behaviors
•Applicable when a NN only fails on a small subset of inputs/specifications
•Cannot simply retrain with a different loss function
•Not explicitly considered in most literature (details later)
7

8. Example: Non-Preserving and Preserving Repair
8
A repair algorithm with knowledge preservationA repair algorithm without knowledge preservation

9. Challenge of Repairing NN Control Policies
9
•Challenge: complex relationship between NN parameters and performance
•Hard to improve performance by modifying parameters; no true action labels
•Hard to extend input-output NN repair to NN control policy repair
•Hard to preserve previously correct behaviors when modifying parameters

10. Related Work 1: Input-Output NN Repair
•Sound and Complete Neural Network Repair with Minimality and Locality Guarantees
[Fu and Li, 2021]
•Fixes only the neighborhood of a failed input
•Assumptions: input-output specs, finite failed inputs, piecewise linear (e.g., ReLU) NNs

•Local Repair of Neural Networks Using Optimization
[Majd, Zhou, Amor, Fainekos, and Sankaranarayanan, 2021]
•Fine-tunes a single layer to fit the output into the safe set, while minimizing the original loss
•Assumptions: the goal is specifiable in the output space and achievable within one layer
10

11. •Runtime-Safety-Guided Policy Repair [Zhou, Gao, Kim, Kang and Li, 2020]
•Trains a NN to imitate a safe MPC, while minimizing the delta of the old and new parameters
•Assumption: small delta in parameters ⇒ small delta in performance (Lipschitz continuity)
•Does not always hold when performance is measured by temporal logic
11
•Summary: limitations of the state of the art
•Goal specs on NN outputs, not closed-loop trajectories
•Finite sets of failed inputs
•Specific to certain NN architectures
•Need for an existing safe controller
•Lipschitz continuity of performance metrics w/r/t NN parameters
Related Work 2: Closed-Loop NN Repair

12. 2. Problem: Repair with Preservation (RwP)
12

13. Problem Statement (Informal)
13
Design a repair algorithm for NN control policies such that:
-The repair fixes as many previously failed initial states as possible
-The system does not fail on the previously successful initial states

14. Our Notation and Assumptions

14

15. STL Robustness (a.k.a. Quantitative Semantics)
•STL robustness [Fainekos & Pappas 2009, Donze and Maler 2010]: a real-valued score
that measures how well a trajectory satisfies an STL specification
15
•Notation: consider different initial states given dynamics and control policy :

Target set

16. Partitioning the Initial States by Policy
•A control policy partitions the initial state space into successful and failed subsets:
16
✅
×



2212
10
30
failed
initial states
successful
initial states

17. Repair with Preservation (RwP) Problem
17
Find an alternative policy that maximizes the quantity of repaired initial states,
while preserving the correctness of successful initial states

18. 3. Solution:
Incremental Simulated Annealing Repair (ISAR)
18

19. Divide and Conquer: Incremental Repair
19

2212
10
30



•If a solution is found, add the repaired
region to the successful set.
•Whether a solution is found or not,
repeat for the next failed region.

20. Tackling the Incremental Repair Challenges
2. In what order to select failed regions for repair?
•Greedy: pick the region with the highest min STL robustness on sampled initial states
3. How to enforce the preservation constraint on successful regions?
•First, add a logarithmic barrier to the objective function
•Second, reject policies that break the constraint on sampled initial states
4. How to avoid getting stuck in local optima?
•Simulated annealing: perturb parameters with increasing randomness [Kirkpatrick 1983]

20

1. How do we know if all initial states in a region satisfy the specification?
•First, estimate robustness on a finite sample of initial states
•Second, verify safety on an infinite set (incomplete but sound) [Ivanov et al. 2019]

21. 21
Refinement of the RwP Problem
Find an alternative policy that
•Maximizes the number of repaired sampled initial states
•Subject to keeping the previously verified initial states still verified

22. Repairing One Region
22

Objective function to maximize:

23. Repairing One Region
23




Objective function to maximize:
// perturb NN weights
// measure improvement
// flip a biased coin
// save new NN weights
// increate the temperature

24. Incremental Simulated Annealing Repair (ISAR)
24
Sort all failed regions by STL
robustness on current policy
Run safeguarded simulated
annealing on the failed region
Append the repaired region
to the successful set
Discard the selected region
if repaired
if not repaired
Initialize:
successful set := verified set
pick a failed region
with highest min
STL robustness
Terminate
if empty

25. 4. Evaluation:
Unmanned underwater vehicle (UUV) and mountain car (MC)
25

26. Case Studies
26
Unmanned Underwater Vehicle (UUV) Mountain Car (MC)
●Staying between and
for 30 seconds

●Reaching within 110 seconds

27. 27

28. UUV
MC
Quantitative Results
28

29. 29
UUV
verification
per region
UUV
sim annealing
per iter
UUV
STL rob check
for all regions
MC
verification
per region
MC
sim annealing
per iter
MC
STL rob check
for all regions
Computation Time Ranges

30. Conclusion
•We resolve conflicts between initial states when repairing NN control policies for STL tasks
•We formulate the Repair with Preservation (RwP) problem and refine it into a solvable version
•We propose Incremental Simulated Annealing Repair (ISAR) to tackle the RwP problem
•We evaluate ISAR on two standard NN control benchmarks
30

31. Limitations and Future Work
• ISAR has computational inefficiencies:
-It randomly perturbs the NN until fixing a failed region while preserving successful regions
-Verification is done from scratch instead of incrementally
• Future work directions:
-More efficient search: interpolate along the trade-off between repair and preservation
-Resolving conflicts between multiple safety specifications, instead of initial states
-More ablation studies: various selection orders, architectures, hyperparameters, systems
31

32. References
•[Cohen 2022] Cohen, Dor, and Ofer Strichman. "Automated repair of neural networks." arXiv preprint arXiv:2207.08157 (2022).
•[Fainekos & Pappas 2009] Fainekos, Georgios E., and George J. Pappas. "Robustness of temporal logic specifications for continuous-time
signals." Theoretical Computer Science (2009).
•[Fu 2021] Fu, Feisi, and Wenchao Li. "Sound and complete neural network repair with minimality and locality guarantees." arXiv preprint
arXiv:2110.07682 (2021).
•[Fu 2022] Fu, Feisi, et al. "Reglo: Provable neural network repair for global robustness properties." Workshop on Trustworthy and Socially
Responsible Machine Learning, NeurIPS (2022).
•[Majd et al. 2021] “Local repair of neural networks using optimization”. arXiv preprint arXiv:2109.14041.
•[Donze & Maler 2010] "Robust satisfaction of temporal logic over real-valued signals." International Conference on Formal Modeling and
Analysis of Timed Systems. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010.
•[Ivanov 2019] Ivanov, Radoslav, James Weimer, Rajeev Alur, George J. Pappas, and Insup Lee. "Verisig: verifying safety properties of hybrid
systems with neural network controllers.“ HSCC (2019).
•[Kirkpatrick 1983] Kirkpatrick, Scott, C. Daniel Gelatt Jr, and Mario P. Vecchi. "Optimization by simulated annealing." Science (1983).
•[Levine and Abbeel 2014] Levine, Sergey, and Pieter Abbeel. "Learning neural network policies with guided policy search under unknown
dynamics." Advances in neural information processing systems 27 (2014).
•[Maler 2004] Maler, Oded, and Dejan Nickovic. "Monitoring temporal properties of continuous signals." FTRTFT (2004).
•[Sotoudeh 2021] Sotoudeh, Matthew, and Aditya V. Thakur. "Provable repair of deep neural networks." SIPLAN (2021).
•[Zhou 2020] Zhou, Weichao, Ruihan Gao, BaekGyu Kim, Eunsuk Kang, and Wenchao Li. "Runtime-safety-guided policy repair." RV (2020).
32

33. Neural Network (NN) Repair
33
NN Repair
With ground-truth outputs:
adversarial learning
Without ground-truth outputs:
direct param modification
Not modifying the NN
architecture – only
search for alternative
parameters
Modifying the NN
architecture
Majority of
literature focus
on this
Ground-truth
outputs available
Ground-truth outputs
unavailable
•Ultimate goal: NNs that fulfill
formal specifications from all
inputs, e.g.
•NN repair aims to modify NN
parameters to make the
counterexamples satisfy the
spec. [Cohen 2022]

34. Repairing NN-based Control Policies
•General-purpose NNs vs. NN-based control policies
•The former is usually evaluated on a single pass output

34

•The latter is usually evaluated on a state trajectory of multiple passes,
•E.g.: UUV must maintain its y position between 10 and 50 for 30 seconds.
•E.g.: Robot must first grab object O at position A, and then go to position B.

35. STL Robustness (a.k.a. Quantitative
Semantics)

35

36. Incremental Simulated Annealing Repair
(ISAR)

36