If simulation and analysis are so precise, why is there still such a need for physical testing? As industry requires ever faster design-cycles, while maintaining control of both budgets and accuracy, qualification by analysis becomes more of a reality. Simulation and test must work together. Making simulation as reliable as traditional testing is a challenge that we must meet head-on.
NAFEMS is leading the culture of reliable guidance, professional competency, and rigorous validation that is needed for continued progress towards certification by simulation, with the requisite authority and experience.
On the run-up to our World Congress in Stockholm, Sweden we ran a survey checking the simulation community's attitude to "Certification by Simulation". We wanted to know how far we were from having simulation replace test in the area of certification. It was a simple, blunt question designed to show the disparity of viewpoints we hear on this subject (Figure 1).
Here at NAFEMS, we get the chance to talk to organisations from across industry, and feel it is safe to say that in some areas simulation has already replaced test. Want an example? I recently talked to one regulator from the automotive industry who described how ray tracing was an acceptable way of clearing the passenger car rear
view mirror check. It is a pretty basic analysis. In this instance, numerical simulation can replace test as the technology is mature and the variability in the inputs that drive the simulation are low. The obvious reason for using simulation instead of test is that it reduces cost and time; by using simulation the fiddly, tedious, manual and time
consuming process of setting up a dummy and installing a camera to check the view field is eliminated. You can go on to make the case that the simulation is more useful than the test as the height of the driver can be adjusted easily without having to haul a dummy out of the car and spend time setting up a new one. Sure, errors can creep into a simulation but the same is also true with the test… was the dummy set up at exactly the right position? was the camera perfectly aligned? etc. There is variability on both sides of the simulation vs. test debate. There is some low hanging fruit, in some areas we can eliminate the need to test, and in these areas, I expect OEMS to be making the case to the regulator that simulation can replace test.
The aero industry is taking a long hard look at how Certification and Qualification by Analysis (CQbA) can be
put into practice. The European Union Aviation Safety Agency (EASA) recently closed public consultation on a
proposed certification memorandum that focused on providing guidance on the main certification aspects to
be considered when using modelling and simulation . The EASA Certification Policy section includes the text;
"The use of Modelling & Simulation (M&S) techniques such as those based on computational Finite Methods, to support the showing of compliance with CS-25 structural certification specifications is becoming more and more widespread in the aerospace industry. In some cases these analytical techniques are proposed to replace (some of) the physical testing that otherwise would have taken place to support the showing of compliance. These M&S techniques have become more versatile and powerful in the last decade and have been accompanied by a significant increase in computational capabilities, to the extent that these techniques are now basically available to all applicants, to be applied to a range of physical phenomena at an affordable cost and effort."
But let us not confuse a promotion of certification by simulation with the retirement of testing. One of the
pillars on which CQBA stands is that the model needs to be validated! Testing is still required, but the focus is
shifting to what the testing is used for. Head over to the medical industry and in the US, you will see that the FDA
has been making great strides in addressing this topic. The evidence for medical device regulatory decision making comes from four different types of model; animal, bench (in essence a controlled setting), clinical trial and computer modelling. As with other industries there is the usual time and cost saving drivers, but here in the medical industry, there is a strong case to be made that simulation can allow situations to be assessed that simply are not possible with a clinical trial. Morrison et al  set out the FDA's perspective in the article "Advancing Regulatory Science with Computational Modeling for Medical Devices at the FDA's Office of Science and Engineering Laboratories". The article flags up the need for detailed reporting of the computational modelling as the regulatory staff are not
involved in the running of the models, and I'm sure the regulators in other industries will be in a similar position. Helpfully the FDA have addressed this issue by developing guidance on what information should be included in a report that documents a computational modelling study .
Waymo, the self-driving car project from Google, celebrated 10 Million miles of road testing in 2019. Hiding
in the shadows of this headline is the stat that 15 Billion virtual miles have also been logged via simulation. The
case for using simulation in the regulatory process for autonomous vehicles is simple; without simulation it isn't
possible to assess if the Automated Driver System (ADS) can operate effectively across all the potential scenarios it
will encounter. There is debate about the exact number of miles that need to be logged by a driverless car in the real
world before it can be considered safe. The RAND Corporation report "Driving to Safety – How Many Miles of
Driving Would it Take to Demonstrate Autonomous Vehicle Reliability?" concludes that in order to demonstrate a
fatality rate of <1.09 fatalities per 100 million miles, it would take a fleet of 100 autonomous vehicles (driving for
24 hours a day, 365 days a year at an average speed of 25 miles per hour) 12.5 years! The 1.09 fatalities per 100
million miles is significant as this is the human performance equivalent in the US. The relationship between simulation and test isn't as clear cut here. We can't simply use simulation to identify the worst case scenario that should be tested. There are many situations which will be challenging to the ADS system and the
reason they are challenging, may at first glance not be obvious. It isn't just the multitude of road layouts and
potential traffic /pedestrian scenarios that need to be considered, all these different situations need to be
considered in conjunction with the infinitely variable combination of light and visibility (Rain, snow, dust etc.).
Developers and regulators are going to have to work together and in the US, the National Highway Traffic
Safety Administration has voluntary guidance that is intended to support the organisations working in this
While it is fun to rehash the "when will simulation replace test" debate, the more productive conversation to have is
to look at how simulation and test can best work together. I see a lot of nice jargon for this, "Fusion of Analysis and
Test" , "Smarter Testing" [6,7,8]. In essence it comes down to:
So, here's my take on it all:
- Ian Symington, NAFEMS
EASA, "Modelling & Simulation – CS-25 Structural Certification Specifications," 2020. [Online]. Available:
"Advancing Regulatory Science With Computational Modeling for Medical Devices at the FDA's Office of
USFDA, "Reporting of Computational Modeling Studies in Medical Device Submissions - Guidance for Industry and Food and Drug Administration Staff," 2016. [Online].
N. Kalra and S. M. Paddock, "Driving to Safety - How Many Miles of Driving Would It Take to Demonstrate
NAFEMS UK Steering Committee, "Fusion of Test and Analysis in Structural Dynamics," 15 November 2018.
S. Chisholm, "Smarter Testing Through Simulation for Efficient Design and Attainment of Regulatory
M. Fouinneteau, "Overview of Smarter Testing and Simulations for Airframe," 14 November 2018. [Online].
G. Grewal, T. De Pauw and H. Kabir, "Smarter Testing Through Dynamic Simulation - Boeing Seat System
"Uniform provisions concerning the approval of vehicles with regard to the protection of the occupants of the cab of a commercial vehicle," United Nations Economic Commission for Europe (UNECE), 2012. [Online].