RL Anomaly Detection

1. Müller, R., Illium, S., Phan, T., Haider, T., and Linnhoff-Popien, C. 2022. Towards Anomaly Detection in Reinforcement Learning. Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, 1799–1803.

Anomaly Detection (AD) is crucial for the safe deployment of Reinforcement Learning (RL) agents, especially in safety-critical applications where encountering unexpected or out-of-distribution situations can lead to catastrophic failures. This work provides a perspective on the state and future directions of AD research specifically tailored for the complexities inherent in RL.

The paper argues that current AD research within RL often relies on overly simplified scenarios that do not fully capture the challenges of sequential decision-making under uncertainty. It establishes important conceptual connections between AD and other critical areas of RL research:

• Lifelong Reinforcement Learning: AD is framed as a necessary component for agents that must continually adapt to changing environments and tasks. Detecting anomalies signals the need for adaptation or learning updates.
• Generalization: The ability to detect anomalies is closely related to an agent’s generalization capabilities; anomalies often represent situations outside the agent’s learned experience manifold.

The study highlights non-stationarity (i.e., changes in the environment dynamics or reward structure over time) as a particularly critical and under-explored challenge for AD in RL. To address this formally, the paper proposes utilizing the framework of block contextual Markov decision processes (BCMDPs) as a suitable model for defining and analyzing non-stationary anomalies.

Finally, it outlines practical requirements and desiderata for future research in this area, advocating for more rigorous evaluation protocols and benchmark environments to advance the development of robust and reliable AD methods for RL agents. [Müller et al. 2022]