A Systematic Approach to Fault Injection Test Case Generation in Practice

abstract

The same failure mode, injected into the same component of the same Cyber-Physical System (CPS) may or may not reveal a residual fault depending only on the parametrization of the tested failure scenario: This is shown in Part 10, Clause 8 of the ISO 26262 standard. Hence, when performing Fault Injection (FI) testing to assess the functional correctness, accuracy, and timing of safety mechanisms, it is essential to determine the “worst-case” scenarios in a typically high-dimensional, unbounded, and continuous parameter space. These scenarios are commonly influenced by intricate interactions among system components and between the system and its environment, and are thus known to change with each modification to the system under test. As a result, manually created FI test suites tend to be error-prone, sub-optimal, and expensive to maintain. In contrast, automated FI Test Case Generation (TCG) approaches have already proven to be effective in identifying potentially unsafe behavior of CPS while significantly reducing manual effort. However, given the lack of systematic methodologies for the application of such techniques in real-world development environments, we set to understand how FI TCG can be applied in continuous development, integration, and testing processes. Based on our findings, we propose a methodology for encoding the generation of “worst-case” FI test cases as an optimization problem to which generic optimization approaches can be applied. We apply this methodology to assess two of the safety mechanisms safeguarding the TTTech Auto MotionWise automotive middleware. Using search-based TCG in a Hardware-in-the-Loop (HiL) setup, we demonstrate how it supports the identification of highly challenging FI test cases and increases confidence in the absence of residual faults with minimal manual effort.