Simulation Supporting Certification

The Analysis Agenda - What Next for Engineering Simulation?

Simulation Supporting Certification

Relying Less on Test

If simulation and analysis are so precise, why is there still such a need for physical testing? As industry requires ever faster design-cycles, while maintaining control of both budgets and accuracy, qualification by analysis becomes more of a reality. Simulation and test must work together. Making simulation as reliable as traditional testing is a challenge that we must meet head-on.

NAFEMS is leading the culture of reliable guidance, professional competency, and rigorous validation that is needed for continued progress towards certification by simulation, with the requisite authority and experience.

On the run-up to our World Congress in Stockholm, Sweden we ran a survey checking the simulation community's attitude to "Certification by Simulation". We wanted to know how far we were from having simulation replace test in the area of certification. It was a simple, blunt question designed to show the disparity of viewpoints we hear on this subject (Figure 1).

Here at NAFEMS, we get the chance to talk to organisations from across industry, and feel it is safe to say that in some areas simulation has already replaced test. Want an example? I recently talked to one regulator from the automotive industry who described how ray tracing was an acceptable way of clearing the passenger car rear
view mirror check. It is a pretty basic analysis. In this instance, numerical simulation can replace test as the technology is mature and the variability in the inputs that drive the simulation are low. The obvious reason for using simulation instead of test is that it reduces cost and time; by using simulation the fiddly, tedious, manual and time
consuming process of setting up a dummy and installing a camera to check the view field is eliminated. You can go on to make the case that the simulation is more useful than the test as the height of the driver can be adjusted easily without having to haul a dummy out of the car and spend time setting up a new one. Sure, errors can creep into a simulation but the same is also true with the test… was the dummy set up at exactly the right position? was the camera perfectly aligned? etc. There is variability on both sides of the simulation vs. test debate. There is some low hanging fruit, in some areas we can eliminate the need to test, and in these areas, I expect OEMS to be making the case to the regulator that simulation can replace test.

Figure 1 – Response to survey on the theme of certification by simulation posted in support of the NAFEMS Stockholm World Congress.

The aero industry is taking a long hard look at how Certification and Qualification by Analysis (CQbA) can be
put into practice. The European Union Aviation Safety Agency (EASA) recently closed public consultation on a
proposed certification memorandum that focused on providing guidance on the main certification aspects to
be considered when using modelling and simulation [1]. The EASA Certification Policy section includes the text;

"The use of Modelling & Simulation (M&S) techniques such as those based on computational Finite Methods, to support the showing of compliance with CS-25 structural certification specifications is becoming more and more widespread in the aerospace industry. In some cases these analytical techniques are proposed to replace (some of) the physical testing that otherwise would have taken place to support the showing of compliance. These M&S techniques have become more versatile and powerful in the last decade and have been accompanied by a significant increase in computational capabilities, to the extent that these techniques are now basically available to all applicants, to be applied to a range of physical phenomena at an affordable cost and effort."

But let us not confuse a promotion of certification by simulation with the retirement of testing. One of the
pillars on which CQBA stands is that the model needs to be validated! Testing is still required, but the focus is
shifting to what the testing is used for. Head over to the medical industry and in the US, you will see that the FDA
has been making great strides in addressing this topic. The evidence for medical device regulatory decision making comes from four different types of model; animal, bench (in essence a controlled setting), clinical trial and computer modelling. As with other industries there is the usual time and cost saving drivers, but here in the medical industry, there is a strong case to be made that simulation can allow situations to be assessed that simply are not possible with a clinical trial. Morrison et al [2] set out the FDA's perspective in the article "Advancing Regulatory Science with Computational Modeling for Medical Devices at the FDA's Office of Science and Engineering Laboratories". The article flags up the need for detailed reporting of the computational modelling as the regulatory staff are not
involved in the running of the models, and I'm sure the regulators in other industries will be in a similar position. Helpfully the FDA have addressed this issue by developing guidance on what information should be included in a report that documents a computational modelling study [3].

Waymo, the self-driving car project from Google, celebrated 10 Million miles of road testing in 2019. Hiding
in the shadows of this headline is the stat that 15 Billion virtual miles have also been logged via simulation. The
case for using simulation in the regulatory process for autonomous vehicles is simple; without simulation it isn't
possible to assess if the Automated Driver System (ADS) can operate effectively across all the potential scenarios it
will encounter. There is debate about the exact number of miles that need to be logged by a driverless car in the real
world before it can be considered safe. The RAND Corporation report "Driving to Safety – How Many Miles of
Driving Would it Take to Demonstrate Autonomous Vehicle Reliability?"[4] concludes that in order to demonstrate a
fatality rate of <1.09 fatalities per 100 million miles, it would take a fleet of 100 autonomous vehicles (driving for
24 hours a day, 365 days a year at an average speed of 25 miles per hour) 12.5 years! The 1.09 fatalities per 100
million miles is significant as this is the human performance equivalent in the US. The relationship between simulation and test isn't as clear cut here. We can't simply use simulation to identify the worst case scenario that should be tested. There are many situations which will be challenging to the ADS system and the
reason they are challenging, may at first glance not be obvious. It isn't just the multitude of road layouts and
potential traffic /pedestrian scenarios that need to be considered, all these different situations need to be
considered in conjunction with the infinitely variable combination of light and visibility (Rain, snow, dust etc.).
Developers and regulators are going to have to work together and in the US, the National Highway Traffic
Safety Administration has voluntary guidance that is intended to support the organisations working in this
area.

While it is fun to rehash the "when will simulation replace test" debate, the more productive conversation to have is
to look at how simulation and test can best work together. I see a lot of nice jargon for this, "Fusion of Analysis and
Test" [5], "Smarter Testing" [6,7,8]. In essence it comes down to:

  1. Reducing the amount of testing that needs to be carried out. Do we need to test all the anchorage locations where the seat connects with the chassis of a bus or can we look, using simulation, to investigate how the load sheds throughout the structure and select the worst case scenario?
  2. Getting more information, getting more value from your testing by performing a better test. One example of this is to use simulation to plan where the sensors should be placed to best capture the response.

So, here's my take on it all:

  • We don't need simulation to replace test in the certification process. If simulation doesn't provide a robust alternative that can allow products to be developed more quickly and cheaply, that is great, we keep testing. In safety critical areas the big signoff test isn't going away. I can watch blade off signoff tests all day, who doesn't love the tension of seeing millions of pounds of kit deliberately written off (see the embedded video.
  • At the most basic level, simulation supports certification by de-risking this final gate where a failure can cripple a company. For domains where the simulation is mature it is already replacing test. I know this comment will raise eyebrows but take a look at UN ECE Regulation 29 (Truck Cab Strength) [9]. In this document it states that certain tests can be eliminated if " .the manufacturer can show by computer simulation or calculations of the strength of the component parts of the cab or by other means to the satisfaction of the Technical Service that the cab will not undergo deformation dangerous to the occupants (penetration into the survival space) if subjected to the conditions of the tests."
  • Across most industries I expect simulation to support certification by helping to identify the worst case which should be tested. This cuts down on the amount of costly testing and also improves the situation for the regulator by providing confidence that we are looking at the most onerous scenario.
  • And finally I believe there are areas where simulation is required to support certification as there is no other game in town.

- Ian Symington, NAFEMS

References

[1]

EASA, "Modelling & Simulation – CS-25 Structural Certification Specifications," 2020. [Online]. Available:
https://www.easa.europa.eu/sites/default/files/dfu/proposed_cm-s-014_modelling_simulation_-_for_consultation.pdf. [Accessed 14 September 2020].

[2]

"Advancing Regulatory Science With Computational Modeling for Medical Devices at the FDA's Office of
Science and Engineering Laboratories," Frontiers Media, 25 September 2018. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fmed.2018.00241/full. [Accessed 14 September 2020].

[3]

USFDA, "Reporting of Computational Modeling Studies in Medical Device Submissions - Guidance for Industry and Food and Drug Administration Staff," 2016. [Online].
Available: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/reporting-computational-modeling-studies-medical-device-submissions [Accessed 18 September 2020].

[4]

N. Kalra and S. M. Paddock, "Driving to Safety - How Many Miles of Driving Would It Take to Demonstrate
Autonomous Vehicle Reliability?" 2016. [Online]. Available: https://www.rand.org/pubs/research_reports/RR1478.html. [Accessed 14 September 2020]

[5]

NAFEMS UK Steering Committee, "Fusion of Test and Analysis in Structural Dynamics," 15 November 2018.
[Online]. Available: https://www.nafems.org/events/nafems/2018/fusion-of-test-and-analysis-in-structural-dynamics/. [Accessed 14 September 2020].

[6]

S. Chisholm, "Smarter Testing Through Simulation for Efficient Design and Attainment of Regulatory
Compliance," 12 June 2017. [Online]. Available: https://www.nafems.org/publications/resource_center/
nwc17_kn_1/
. [Accessed 14 September 2020].

[7]

M. Fouinneteau, "Overview of Smarter Testing and Simulations for Airframe," 14 November 2018. [Online].
Available:https://www.nafems.org/publications/resource_center/c_nov_18_fra_46/. [Accessed 14 September 2020].

[8]

G. Grewal, T. De Pauw and H. Kabir, "Smarter Testing Through Dynamic Simulation - Boeing Seat System
Certification by Analysis Initiative," 1 April 2017. [Online]. Available: https://www.nafems.org/publications/resource_center/bm_apr_17_4/. [Accessed 14 September 2020].

[9]

"Uniform provisions concerning the approval of vehicles with regard to the protection of the occupants of the cab of a commercial vehicle," United Nations Economic Commission for Europe (UNECE), 2012. [Online].
Available:https://www.unece.org/fileadmin/DAM/trans/main/wp29/wp29regs/R029r2e.pdf. [Accessed 14 September 2020]

Simulation Supporting Certification at the US FDA

Tina Morrison, deputy director division for applied mechanics at the US Food and Drug Administration gave us the regulators view of the role that simulation plays in the certification process at the 2018 NAFEMS Americas conference.
Tina showed how data from virtual patients is integrated with the results of real world clinical trials.

Boeing Seat System Certification by Analysis Initiative

In this article Boeing describe their building block approach to FEA and Dynamic Simulation which are laying the groundwork for certification by analysis.

The Business Impact of Modeling and Simulation in Regulated Industries

In this presentation recording Mark Palmer, Distinguished Scientist and Global Simulation Lead at Medtronic shows the value that simulation can add in a highly regulated industry.
Mark looks at the factors that contribute to the decision to pursue a modelling and simulation solution in comparison to empirical forms of evidence.

Virtual Testing for Certification - Opinion

In this article Dr. Mahmood Tabaddor, Global Leader for Modeling, Simulation & Machine Learning at Underwriters Laboratories gives his view on how simulation supports the certification process and how essentially, it all comes down to trust.
This article was was published in the October 2020 issue of the NAFEMS benchmark magazine which focussed on the role simulation plays in the certification process.

Want to Learn More About How Simulation Supports Certification?

Visit the NAFEMS Resource Centre for numerous technical materials produced by NAFEMS and its members over the years.