Allocate
The SUNRISE Safety Assurance Framework is test environment-agnostic, allowing scenarios to be executed in a range of test environments, from fully virtual to hybrid environments (such as Hardware-in-the-Loop) to controlled physical environments (such as proving grounds). Within the Allocate block, a test environment allocation workflow is applied (as detailed below), which will be implemented in case studies within the broader Safety Assurance Framework. Once a scenario is allocated to a test environment (also called test instance), the scenario is executed and the corresponding data is recorded.
The initial allocation process involves two key inputs: test case information and test instance capabilities. Test cases include scenario descriptions, expected behavior of the system under test (SUT), and pass/fail criteria. Specific requirements are extracted from these test cases. The second input consists of available test instances, such as virtual testing, X-in-the-Loop (XiL), and proving ground testing, with field testing excluded due to its uncontrollable nature.
The process begins with comparing test case requirements to the capabilities of test instances using a structured approach. This includes analyzing aspects like scenery elements, environmental conditions, dynamic elements, and test criteria. Once a suitable test instance is identified, the test case is allocated to it.
A virtual simulation-first approach is prioritized for safety and efficiency. Test cases suitable for virtual simulation are executed using the lowest-fidelity simulation capable of meeting requirements, to maximize throughput. After execution, results are reviewed to determine if further testing on higher-fidelity test instances is required. This iterative process may include reallocations to ensure the necessary test coverage and accuracy.
The process includes provisions for external influence, such as road authorities overriding allocations or proving ground operators refusing tests due to safety concerns. These decisions must be documented for the final assessment. Additionally, the Allocate block includes initial reallocation steps, resulting from the Analyse block.
Audit instructions for 'Allocate'
- Review the comparison of test case requirements with test instance capabilities:
- Ensure that the structure outlined in D3.3 Section 3 was followed, which includes scenery elements, environment conditions, dynamic elements, and test criteria (D3.3 Section 3.3).
- Verify that the metrics described in D3.3 Section 4.3 were applied for the comparison.
- Check the decision-making process for test case allocation:
- Confirm that the process outlined in D3.3 Section 4.5 and D3.3 Figure 27 was followed.
- Verify that the “virtual simulation first” approach was applied as described in D3.3 Section 4.2.
- Examine the documentation of the allocation results:
- Review the allocation matrix or table as described in D3.3 Section 4.6 and exemplified in D3.3 Figure 28.
- Ensure that scenarios that could not be allocated or were not sufficiently tested are properly flagged and reported to the “Coverage” block of the “Analyse” part of the SAF.
- Verify the consideration of various metrics:
- Check that both functional and non-functional metrics were considered, as described in D3.3 Sections 4.3 and 4.4.
- Confirm that safety was prioritized in the decision-making process (D3.3 Section 4.5).
- Review the reallocation process:
- Ensure that the iterative allocation to higher-fidelity test instances, when necessary, was performed as described in D3.3 Section 4.5.
- Verify that the reasons for reallocation decisions were properly documented (D3.3 Section 4.6).
- Check for special circumstances:
- Review if any deviations from the general methodology were made due to special circumstances, and if so, ensure they were properly justified (D3.3 Section 4.5).
- Verify the completeness of documentation:
- Ensure that all steps of the decision-making process, including reasons for decisions, were documented and returned to the SAF (D3.3 Section 4.6).
- Check for the presence of a tree structure containing all metrics and results of the comparison to all test instances (D3.3 Section 4.6).
- Review the consideration of safety standards:
- Verify that safety standards such as SOTIF were considered in the allocation process, particularly for identifying potentially triggering conditions or functional insufficiencies of the System under Test (SUT) (D3.3 Section 4.5).