M
"

Directory

High-Level Overview

Glossary

Input

Data Framework

Performance Assurance

Scenario

Create

Format

Store

Environment

Query/Concretise

Allocate

Execute

Safety Argument

Coverage

Test/Evaluate

Safety Case

Decide

Audit

Case Studies

Submit Feedback

SUNRISE Safety Assurance Framework

Coverage

Input Scenario Create Format Store Environment Query & Concretise Allocate Execute Safety Argument Coverage Test Evaluate Safety Case Decide Audit

When referring to Coverage, it contains two aspects, for example ‘X is covering Y’, with this in mind bullets below provide a summary of the types of coverage considerations discussed in the SUNRISE project, followed by more detailed explanation.

  • At the scenario set level: 1) Operational Design Domain (ODD) coverage by scenario set, 2) Requirements coverage by scenario set, 3) Behaviour coverage by scenario sets
  • At individual scenario level: logical parameter covered by concrete parameter values

After execution, the data is analysed as part of the safety assessment of the system. There are two main blocks in this analysis: Coverage and Test Evaluate. The Coverage block examines the test from multiple perspectives and derives a combined coverage outcome. These perspectives might include analysing coverage of the parameter value ranges in the logical scenario using individual concrete parameter values. Based on the analysis outcome, an iterative process derives new concrete scenarios within given logical scenario’s parameter ranges to identify failure conditions (e.g., parameter combinations that lead to system failure). This iterative feedback loop is indicated by the red feedback arrow.

Another aspect of Coverage Analysis is ODD analysis. Here, the accumulated ODD features covered by the set of test scenarios are examined to determine if enough of the system’s ODD is covered. ODD is the focus because it defines the operational boundary within which the system is expected to operate safely. Thus, it is essential to thoroughly explore this claimed boundary using a diverse set of scenarios. These are just two examples that form the coverage concept, and the SUNRISE project will continue to explore this further in later stages.

Audit instructions for 'Coverage'

By following the steps indicated below, a user of the SUNRISE SAF can audit the contents of the Coverage block, thereby ensuring that the combined set of test case results sufficiently covers the ODD and the parameter value ranges:

  1. Verify Scenario Coverage (D5.3 Section 4.2.3):
    1. Check if the coverage analysis includes all four types of coverage metrics described by de Gelder et al. (D5.3 Reference [59]):
      • Tag-based coverage: Verify that scenarios cover all relevant ODD aspects through specific tags
      • Time-based coverage: Confirm all timestamps in driving data are represented
      • Actor-based coverage: Ensure all relevant actors are included in at least one scenario
      • Actor-over-time-based coverage: Check that relevant actors are included throughout their period of importance
  2. Verify Parameter Coverage (D5.3 Section 4.2.3):
    1. According to Laurent et al. (D5.3 Reference [60]), verify that:
      • Parameters influencing ADS decision-making are adequately tested
      • Multiple simulations with different parameter values have been run
      • Changes in parameters lead to statistically significant differences in outcomes (if they should)
      • Key metrics like path deviation and safety (minimum distance to objects) are considered