U.S. Senate on Model Validation

This link is to report 116-48, The National Defense Authorization Act for Fiscal Year 2020 by the Committee on Armed Services, United States Senate, dated June 11, 2019: https://www.congress.gov/116/crpt/srpt48/CRPT-116srpt48.pdf

While this 609 page report is probably worth reading from cover to cover, that I have not done. On the other hand, pages 253-254 are worth quoting (the bolding in the text is mine):

Evaluation of modeling and simulation used for force planning and theater operational requirements

The committee notes that the Department of Defense uses a large number and variety of computer models and simulations to support decision-making about force structure, resource allocation, war gaming, and priority weapons platforms and technologies to develop and deploy in support of likely operational scenarios. These models are used to develop information to brief decision makers, including the Congress, about, for example, the current state of the balance of forces in the Pacific and European theaters, the outcomes of likely war scenarios, and the need for investments in advanced technologies and new warfighting capabilities.

The committee is concerned that the quality, accuracy, and dependability of these models, given their important role in decision making processes, has not been adequately validated. The committee notes that technical and engineering models used to develop systems such as body armor and missiles are rigorously verified and validated for veracity of assumptions and technical accuracy using real world data. Other models, such as those used in the financial sector, are developed using expertise from a variety of disciplines, including economics, sociology, and advanced mathematics. The committee is concerned that the models used by organizations including the Joint Staff, Office of Net Assessment, war colleges, and service-level planning entities are simplistic by comparison and not subject to the same level of scrutiny.

Therefore, the committee directs the Secretary of Defense to establish an independent team of academic and industry modeling and subject matter experts to review the quality of modeling and simulation used for force planning, war gaming, resource allocation, and other senior leader decision-making associated with implementation of the National Defense Strategy. The team shall review the technical quality of models currently in use, including their ability to simulate as required by application; physics and engineering, socio-economic impact, readiness, global financial markets, politics, and other relevant inputs and outputs. The team shall assess the quality of these models and make recommendations for investments or policy changes needed to enhance and continuously validate current and future modeling and simulation tools employed to enable senior-level decision-making.

The committee directs the Secretary to support the team with expertise as needed from the Joint Staff, Office of Net Assessment, Under Secretary of Defense for Research and Engineering, and other relevant organizations. The Secretary shall ensure that the team has sufficient resources and access to all data and records necessary to perform its analysis. The committee directs the Secretary to deliver a report on the independent team’s assessments and recommendations with any additional comments, and a specific concurrence or non-concurrence for each recommendation, to the Committees on Armed Services of the Senate and House of Representatives no later than December 31, 2020.

A big thank you to the person who brought this to my attention.