Escolar Documentos
Profissional Documentos
Cultura Documentos
8:50am INTRODUCTION OF PARTICIPANTS – All (13 members present - insuffient for a Quorum) -
See Attendance List Appendix B
9:05am AGENDA ADJUSTMENTS – (add brief introductions of working groups, to provide first-time
attendees background and context for deciding best way to participate in breakout groups)
Notes on Discussion:
• Need to have the V&V standards committee provide an overview of V&V standards, how
they interrelate, and context for V&V 50 document development.
• Types of draft documents – Guide and Technical Report seem to be most suited to the work
we are doing.
Notes on Discussion:
• Use cases illustrate the process; not the technology or the example. Methodology of how to
apply V&V standards in Advanced Manufacturing is the point of the use cases.
• Specifically, in 1) physics-based models, 2) data driven and hybrid models.
• Currently no template we’ve agreed upon. Ronay and Rumi using one format; Sudarsan
shared a couple (see May 2016 meeting minutes Appendix H, Appendix I and Appendix K,
posted on
https://cstools.asme.org/csconnect/CommitteePages.cfm?Committee=101978604&Action=
48062&ChooseTemplate=10288)
• V&V10.1 - An Illustration of the Concepts of Verification and Validation in Computational
Solid Mechanics (see Appendix D for Executive Summary, and available on C&S Connect at
https://cstools.asme.org/csconnect/CommitteePublications.cfm?PublicationID=2508&Com
mittee=100079931&NoToolbar=yes, login first)
• V&V10 working on a revision to their standard, which will address UQ more thoroughly.
• 10.1 is a narrative use case; could also have a code-based one, implementable.
• CAUTION – introducing use cases prematurly can cramp the process; don’t build the process
of applying V&V standards to advanced manufacturing around use cases. First the process,
then the use case illustrating it. (i.e don’t build a standard by examples)
• Don’t include use cases in the body of a guidance document; rather as an appendix or
supplement.
~3:00pm FOCUS AREA TEAM BREAKOUT SESSIONS (revised from draft agenda)
• 3-4 pm Terminology, Concepts, Relationships and Taxonomy for VVUQ in Advanced
Manufacturing. Room CR5 – See Appendix E, also posted in C&S Connect for presentation
o Requests – 1) suggest any additional standard references that need to be included;
and 2) review the definitions
• 4-5 V&V Interactions with the Model Life Cycle. – See Appendix F, available on C&S Connect
at
https://cstools.asme.org/csconnect/CommitteePublications.cfm?PublicationID=2508&Com
mittee=100079931&NoToolbar=yes, login first)
• 5-6 pm
o V&V for Additive Manufacturing. See Appendix G for example document shared.
o V&V UQ for Data-centric/driven Modeling in Advanced Manufacturing. – see
Appendix H for draft document shared. 2 outputs identified. 1) Technical Report. 2)
analysis of adopting CRISP DM for advanced Manufacturing.
Notes on Discussion
Restructure V&V50 Working Groups, and have new members join individual working groups, rather than
the main subcommittee (allows for quorums at meetings, and for focused emails and communications
within each working group)
Sending a note to all V&V50 members about the formation of the following Working Groups, and ask for
their interest in participating on them.
After informing entire subcommittee membership, initiate a ballot in C&S Connect to form the following
Working Groups under V&V50, and to add the following members to those groups.
V&V50 Working groups (5) could potentially have a session at V&V Symposium.
1. Terminology, Concepts, Relationships and Taxonomy for VVUQ in Advanced Manufacturing.
o Chair – Sankaran (Maha) Mahadevan
o Interested members: Gaurav Ameta, Tina Lee, Sanjay Jain, Sudarsan Rachuri
• V&V Interactions with the Model Life Cycle
o Chair – Joe Hightower
o Gordon Shao, Eric Sawyer, Rumi Ghosh, Aaron Bernreuther, Mark Benedict, Jeffrey
Fong, Bill Schindel
• VVUQ Challenges and Methods in Systems of Models
o Chair / Vice-Chair– Barron Bichon / Ade Makinde
o Eric Sawyer, Mahesh Mani, Rish Sant, Huijuan Dai, Matteo Bellucci, Maha, Suresh
Sundarraj (confirm?)
• VVUQ Methods in Data-driven and Hybrid models
o Chair / Vice-Chair – Rumi Ghosh / Ronay Ak
o Heather Reed, Gaurav Ameta, Tina Lee, Gordon Shao, Vincent Paquit, Joe Hightower,
Jeffrey Fong, Sagar Kamarthi
• VVUQ Applications in Process Technologies
o Chair / Vice-Chair - Huijuan/Ade
V&V 50 Draft Minutes – November 7-8, 2016 Page 4
o Rish Sant, Gaurav Ameta, Mahesh Mani, Sudarsan Rachuri, Nathan Andrews (confirm),
Matteo Bellucci, Sagar Kamarthi (confirm), Maha Mahadevan, Mark Benedict
Respectfully submitted,
Marian Heller
Staff Secretary, V&V 50
212-591-8514
hellerme@asme.org
V&V 50 Draft Minutes – November 7-8, 2016 Page 5
Appendices:
• Appendix A –
• Appendix B - Attendance List
• Appendix C –
• Appendix D –
• Appendix E –
• Appendix F –
• Appendix G –
• Appendix H –
• Appendix –
• Appendix A –
• Appendix A –
• Appendix A –
• Appendix A –
•
Members Present
Ronay Ak, NIST Sanjay Jain (WebEx)
Matteo Bellucci Tina Lee (present 11/7, WebEx 11/8)
Mark Benedict, Vice Chair Sankaran Mahadevan
Barron Bichon Ade Makinde
Huijuan Dai Sudarsan Rachuri, Chair
Rumi Ghosh (WebEx) Heather Reed
Johnathan Goodsell (WebEx) Gordon Shao
Marian Heller, Secretary Simin Zhou (representing Rafi Haludeen)
Joe Hightower
Members Absent
Ravi Aglave Rafi Haludeen (represented by Simin Zhou)
Nathan Andrews Yucheng Liu
Hessam Babaee John Matlik
Ilias Bilionis Aditya Newalkar
Jeffrey Bischoff Soundar Srinivasan
James Coburn Suresh Sundarraj
Bala Deshpande Ben Thacker
Chris Freitas Cumali Semetay
Ghanshyam Gaudani Sean Whitsitt, Cont. Member
Appendix C – References for Types of Documents and How Standards Committees Work (available for
download on C&S Connect, V&V50 committee page Presentation and Resource Documents
November 2016 – Shared Documents.)
• ASME Codes & Standards Development Committee Procedures with S&T Supplemental (see
PDF inserts below)
o Sections S6.4-S6.6 (Technical Reports, Guides and Draft standards for Trial Use
o Appendix IV: Draft Preparation, Dating of Referenced Standards, and Incorporation
of Copyrighted Material
o Appendix IX: Draft Standards for Trial Use
o Appendix X: Procedure for Development of ASME Technical Reports
• Committee Handbook –
o Section 6 – How Does the Work Get Done? (see PDF insert below)
• S&C Volunteer Training Modules - These cover the general categories of Administrative,
Process, and Legal. They are accessible at the Codes and Standards Web site at
http://cstools.asme.org/TrainingModules.cfm
ASME Codes & Standards Development Committee Procedures with S&T Supplemental Sections S6.4-
S6.6 (Technical Reports, Guides and Draft standards for Trial Use)
V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 7
Minutes – November 7-8, 2016
V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 8
Minutes – November 7-8, 2016
ASME Codes & Standards Development Committee Procedures with S&T Supplemental, Appendix IV:
Draft Preparation, Dating of Referenced Standards, and Incorporation of Copyrighted
Material
V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 9
Minutes – November 7-8, 2016
ASME Codes & Standards Development Committee Procedures with S&T Supplemental, Appendix IX:
Draft Standards for Trial Use and Appendix X: Procedure for Development of ASME Technical Reports
V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 10
Minutes – November 7-8, 2016
Appendix D – V&V10.1 Executive Summary (this is not in the same format as the actual document)
1 EXECUTIVE SUMMARY
This Standard describes a simple example of verification and validation (V&V) to illustrate some of the
key concepts and procedures presented in V&V 10. The example is an elastic, tapered, cantilever, box
beam under nonuniform static loading. The validation problem entails a uniform loading over half the
length of the beam. The response of interest is the tip deflection. The validation test plan and the
metrics and accuracy requirements for comparing the calculated responses with measurements are
specified in the V&V Plan, which is developed in the first phase of the V&V program. In setting validation
requirements and establishing a budget for the V&V program, the V&V Plan considers the level of risk in
using the model for its intended purpose. Successfully meeting the V&V requirements means that the
computational model for the tapered beam has been validated for the intended use discussed in this
document, viz., predicting the response of a tapered beam tested in the laboratory.
To encompass as much of the general V&V process as possible in this example, a computational model
was developed specifically for the tapered beam problem, even though it is more likely that a general-
purpose finite-element code would be used in practice. The conceptual model is a Bernoulli–Euler beam
for which the governing equations are solved with the finite-element method. The computational model
was verified (checked for proper programming of the mathematical model and the solution procedure)
by comparing computed values of tip displacement with an analytical solution to a relevant but simpler
problem. A mesh refinement study initially revealed that the model did not converge at the expected
theoretical rate. Further diagnosis revealed a programming error, correction of which led to the proper
convergence rate. Knowing the allowable error due to lack of convergence then allowed an appropriate
level of mesh refinement to be selected.
For validation (comparing with experimental results), 10 virtual trials of the same test were performed
to quantify the distribution of results due to unintended variations in material properties, construction
of the test specimens, and test execution. Other virtual tests were conducted to characterize
uncertainties in selected model input parameters, namely rotational support stiffness and elastic
modulus. 1
(a) a case where uncertainty data were not available and obtained instead from subject matter
experts.
(b) a case where uncertainty data were available from repeat tests and calculations.
In both cases the same metric was employed to demonstrate the use of uncertainty information in the
model test comparison. The validation metric is a measure of the relative error between the calculated
and measured tip deflection of the beam.
V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 12
Minutes – November 7-8, 2016
The same model was used in both validation cases. In each case, the metric was compared to an
accuracy requirement of 10%. In both cases the model was validated successfully. Had the validation
been unsuccessful, it would have been necessary to correct any model deficiencies, collect additional or
improved experimental data, or relax the validation requirement.
Appendix E: V&V 50 Task Group on Terminology, Concepts, Relationships and Taxonomy for VVUQ in
Advanced Manufacturing
Group Members: Sankaran Mahadevan (lead), Johnathan Goodsell, Sanjay Jain, Tina Lee, Gaurav Ameta
Outline of Topics
1. Verification
• Definition
• Code verification
• Solution verification
• Error estimation
• Accuracy requirements
2. Validation
• Definition
• System response quantities (SRQ)
• Validation domain
• Application domain
• Accuracy requirements
• Validation metrics
• Validation hierarchy
3. Calibration
• Model inputs, model parameters, model discrepancy
• Physics-based models
• Data-driven models
• Data types (precise vs. imprecise; ordered vs. unordered; sparse data; big data;
qualitative data; text, voice and image data)
• Fusion of heterogeneous data
4. Uncertainty quantification
• Aleatory sources
• Epistemic sources data, models
• Uncertainty aggregation (roll-up)
• Predictive capability assessment
• Quantification of Margins and Uncertainty (QMU)
5. Additional topics?
V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 13
Minutes – November 7-8, 2016
Terminology, Concepts, Relationships and Taxonomy for VVUQ in Advanced Manufacturing. For
topics in VVUQ in advanced Manufacturing, the approach will be as follows:
(1) survey the definitions in existing V&V standards (e.g., ASME, IEEE, NASA, AIAA, ISO, DoD etc.)
(2) explore applicability of existing definitions to advanced manufacturing
(3) suggest adaptations/extensions of existing definitions to advanced manufacturing
(4) suggest definitions of new concepts unique to advanced manufacturing
V&V for Additive Manufacturing is a focused topic area to explore V&V issues in a context. It is on a
scale that this nascent group can engage with and make progress on. There is real opportunity to
potentially engage with the software industry that is creating the first AM commercial models and help
define best practices based on lessons learned in prior ASME V&V efforts.
V&V Interactions with the Model Life Cycle. Treating the model as a product consumed in operational
use, the work will focus on understanding a variety of use cases, model types, and the model life cycle.
They will develop a document to provide guidance and methods across the life cycle, accounting for the
variety of use cases and variety of models.