This training course discusses methods available to support decision-making using combinations of numerical simulations, experimental observations and expert judgments. Irrespective of the source(s) of the information, a common need is to manage uncertainty in the decision-making process. The various types of uncertainty originating in simulation models are discussed and techniques are presented to quantify their effects on the decision. Uncertainty comes in three broad categories: numerical, parametric and model-form. Methods to assess uncertainty in experimental testing and diagnose bias during expert elicitation are also overviewed.
This short-course on the Verification and Validation (V&V) of computational models teaches techniques to quantify prediction uncertainty which includes the broad classes of, first, numerical uncertainty caused by truncation effects in the discretization of partial differential equations and, second, parametric uncertainty caused by the variability of model parameters. It focuses on applications in structural mechanics and structural dynamics. The quantification includes the propagation and assessment of how much uncertainty is present in the simulation of an application of interest ( “what are the sources, how much uncertainty is present?”). It includes understanding which effects control the uncertainty (“is it predominantly the mesh discretization, parameter variability, or other phenomena?”) and what can be done to reduce the overall uncertainty (“should the mesh be refined, should small-scale experiments be performed, should model parameters be calibrated and how?”).
Technical topics addressed include:
1) code and solution verification,
2) numerical uncertainty,
3) the design of computer experiments,
4) sensitivity analysis and variance decomposition,
6) sampling and the propagation of parametric uncertainty,
7) metrics for test-analysis correlation, and
8) model calibration and the assessment of predictive capability.
Definitions and concepts of V&V are not discussed in detail; the short-course focuses, instead, on the implementation and applications of well-established techniques. A pre-requisite is a basic knowledge of the finite element method, or another computational technique, and familiarity with the types of uncertainty that numerical simulations introduce. The illustrations emphasize solid mechanics and structural dynamics even though the techniques presented are general-purpose and can be applied to any simulation. Applications include simulations for nonlinear vibrations, transient dynamics and wind turbine blade vibrations.
The short-course has been taught over 20 times since 2001 at private companies, government institutions, or in conjunction with technical conferences in Europe and the United States.
- The course is completely code independent, attendees are welcome to bring laptops to take notes, but they are not required.
- A full set of printed and bound notes will be issued to every attendee.
Upon completion of this course, attendees will be able to:
- Understand the objectives of code verification, model validation, uncertainty quantification
- Develop procedures for practical code verification and solution verification
- Select a particular mesh size, or time step, to discretize the equations-of-motion
- Quantify the effects of truncation error in numerical simulations
- Assess the trade-offs between more computing resources and more small-scale testing
- Describe the validation paradigm of sensitivity analysis, correlation, uncertainty analysis
- Describe the process to select and compute appropriate features from simulation outputs
- Understand techniques for global sensitivity analysis and effect screening
- Explain the role of designs-of-experiments and analysis-of-variance in model validation
- Define appropriate test-analysis correlation metrics for model revision and calibration
- Discuss when model calibration might, or not, be needed
Who Should Attend?
The short-course is intended for graduate students, researchers, practicing engineers and project managers seeking to understand, or implement, V&V techniques for their applications. Even though key techniques, such as sensitivity analysis and the propagation of uncertainty, are introduced, they are not discussed in depth. Their usefulness is motivated, instead, through the presentation of application examples. The emphasis is placed on explaining how methods can be organized into a process to verify and validate computational models.
The short-course contents are not designed to produce V&V experts. The goal is to provide a sufficient understanding of key techniques such that attendees are able to implement and apply them to their applications, discuss them with their peers and read the pertinent literature.
Graduate students and researchers will be pointed towards essential techniques without having to endure months of literature review. Practicing engineers will understand how to integrate them into a logical process for their applications. Project managers will be exposed to way to define quality controls for the numerical simulations that their projects and customers rely on.
The contents are presented in 12 lectures (one hour each), tentatively organized as shown. The two-day schedule allows for ample discussion and interaction with attendees. The instructors reserve the right to modify the contents to address the audience’s needs and preferences.
Lecture 1- Overview of Verification and Validation
- High-level comments on modeling, simulation and “predictability”
- Overview of Verification and Validation (V&V)
- Definitions, organization of V&V activities
- Which questions does V&V address? What can be learned from V&V?
- Examples of typical studies in solid mechanics and structural dynamics
Lecture 2 - Application of V&V to Wind Turbine Simulations
- Code verification of the finite element software
- Simulation of blade vibration with bounds of numerical uncertainty
- Sensitivity analysis of the numerical simulation
- Calibration of the model using statistical emulators
- Test-analysis correlation and validation assessment
Lecture 3 - Code Verification
- Definition of code verification, typical code verification activities
- How to define benchmark code verification problems?
- The Method of Manufactured Solutions (MMS)
- Examples of code verification studies in structural dynamics
Lecture 4 - Solution Verification
- Definition of solution verification, typical solution verification activities
- The concepts of consistency, stability and convergence
- Modified Equation Analysis (MEA) and its implication to quantify truncation effects
- Richardson’s extrapolation applied to numerical solutions
- The Grid Convergence Index (GCI) and bounds of truncation error
- Examples of solution verification in structural dynamics
Lecture 5 - Feature Extraction for Structural Dynamics
- What makes a good feature of the response analyzed?
- Features for linear, stationary dynamics Features for arbitrary time-series analysis
- Temporal moments and other features for fast, transient dynamics
- Application of Principal Component Analysis (PCA)
Lecture 6 - Testing for Structural Dynamics
- What makes a useful measurement?
- Overview of excitation, sensing, data transmission in structural dynamics
- Overview of signal processing
- The coherence function as diagnostics of measurement quality
(End of first day.)
Lecture 7 - Design of Computer Experiments
- Principles of the design of (physical or computer) experiments
- Full-factorial, fractional factorial designs, orthogonal arrays, central composite design
- Formulation of 2^(n-k) designs
- The concept of statistical aliasing
- Examples of designs-of-experiments applied to structural dynamics simulations
Lecture 8 - Sensitivity Analysis and Effect Screening
- Rationale for effect screening (“where is an observed variability coming from?”)
- Simple, linear approaches to effect screening
- Analysis-of-variance (ANOVA) using a design-of-experiments
- Main-effect and linear interaction screening
- Application to structural dynamics simulations: what is gained?
Lecture 9 - Development of Surrogate Models
- Surrogate modeling using a design-of-experiments
- Diagnostics of quality of an emulator
- Low-order, polynomial emulators
- Kernel regression, Gaussian process modeling
Lecture 10 - Sampling and Propagation of Parametric Uncertainty
- Sampling methods for the forward propagation of (parametric) uncertainty
- Monte Carlo, stratified sampling, Latin Hypercube Sampling (LHS)
- Convergence of statistical estimates
- The concept of a confidence interval
- Application of statistical sampling to simulations in solid mechanics
Lecture 11 - Test-analysis Correlation and Validation Metrics
- Concepts of response features and validation metrics
- Metrics for structural dynamics and general-purpose test-analysis correlation
- Metrics based on Principal Component Analysis (PCA)
- Statistical tests that account for probabilistic uncertainty
- Model calibration and inference uncertainty quantification
Lecture 12 - An End-to-end Example of Verification and Validation
- Engineering example of transient dynamics finite element simulations
- Verification of the finite element software Design and execution of computer experiments (predictions)
- Design of physical experiments (measurements)
- Effect screening and identification of statistically most-significant inputs
- Small-scale validation experiments: what is gained?
- Uncertainty propagation and final validation assessment
Lecture 13 - Concluding Remarks
- Summary of main points made during the short-course
- Aspects of V&V not covered, other sources of information
- Closing comments, discussion with attendees, exit survey
(End of second day.)