AI-Fusion Safety

In collaboration with Fraunhofer IKS, the AI-Fusion project addressed the critical challenge of understanding and ensuring safety in multi-agent reinforcement learning (MARL) systems.

Emergence, defined as the arising of complex, often unpredictable, system-level dynamics from local interactions between agents and their environment, was a central focus due to its implications for system safety and reliability.

The project's objective was to investigate the detection and mitigation of potentially unsafe emergent behaviors in complex systems composed of multiple interacting AI agents, particularly in scenarios involving heterogeneous agents (e.g., mixed-vendor autonomous systems).

To facilitate research into these phenomena, key contributions included the development of specialized simulation tools:

1. High-Performance MARL Simulation Environment:

• A flexible and efficient simulation environment was developed in Python, adhering to the Gymnasium (formerly Gym) API specification.
• Purpose: Designed specifically for training and evaluating reinforcement learning algorithms in multi-agent contexts prone to emergent behaviors.
• Features:
  • Modularity: Supports diverse scenarios through configurable modules and configurations.
  • Observation/Action Spaces: Handles complex agent interactions, including per-agent observations and sequential/multi-agent action coordination.
  • Performance: Optimized for efficient simulation runs, enabling extensive experimentation.

2. Unity-Based Demonstrator Unit:

• A complementary visualization tool was created using the Unity engine.
• Purpose: Allows for the replay, inspection, and detailed analysis of specific simulation scenarios and agent interactions.
• Utility: Aids researchers in identifying and understanding the mechanisms behind observed emergent dynamics.
• View Demonstrator on GitHub

This project involved close collaboration with industry-focused researchers, software development adhering to modern standards, and deep investigation into the theoretical underpinnings of emergence and safety in MARL systems.

The developed tools provide a valuable platform for continued research in this critical area.