No Action on Validation In the 2020 National Defense Act Authorization

Well, I got my hopes up that the Department of Defense modeling and simulation community was finally going to be forced, kicking-and-screaming, to move forward; ensuring that their models were properly validated, not build upon a “base of sand” and making sure they are not assembled like some “house of cards.” This was to come about through four paragraphs in the Senate’s initial markup of the National Defense Authorization Defense Act (NDAA) of 2020 that instructed DOD to assemble a team “..to assess the quality of these models and make recommendations…not later than December 31, 2020.”

The original four paragraphs are here:

U.S. Senate on Model Validation

Well, it looks like this is not going to happen !!!

According to a little research done by Shawn Woodford, it turns out that that modeling and simulation validation proposal in the original Senate Armed Services Committee report for the 2020 NDAA dated 11 June 2019 did not make it to the final Senate 2020 NDAA bill, passed on 2 July 2020. The proposal was also not included in the House version. The House and Senate versions are now being reconciled in committee and the final 2020 NDAA will probably be approved soon now that there is a general bi-partisan overall budget agreement. There will be a defense budget, but, it appears that it won’t address validation. There is a slim possibility this could change if it is added back in by committee at the last minute.

The 2020 NDAA SASC Report, 11 June 2019:
https://www.congress.gov/116/crpt/srpt48/CRPT-116srpt48.pdf

The 2020 NDAA S. 1790 SASC final markup, 2 July 2019:
https://www.congress.gov/116/bills/s1790/BILLS-116s1790es.pdf

The 2020 NDAA HR 2500 HASC final markup, 12 July 2019:
https://www.congress.gov/bill/116th-congress/house-bill/2500/text

We would love to know who got those four paragraphs placed into the original Senate NDAA mark-up to start with; and afterwards, why they were then stripped out of the final bill. Clearly someone felt it was important enough to be put in there (as do we). We do not know who those “someone” are. And….who was it that stepped in from wherever and made sure those four paragraphs were removed?

If anyone knows anything further about this, please let us know.

P.S.

Source for 1991 RAND “Base of Sand” paper:                                                                 . https://www.rand.org/pubs/notes/N3148.html

The phrase “house of cards” we used in a report we did on casualty estimation methodologies (Casualty Estimation Methodologies Studies, 25 July 2005, The Dupuy Institute, page 32). To quote:

In 1991, Paul Davis and Donald Blumenthal employed the term “base of sand” to describe the essential modeling problem of the day. They described one of the core problems as a lack of a vigorous military science.

Unfortunately, this was the responsibility of the operations research community. Understanding military science was part of what ORO [Operations Research Office] was attempting to do in its early days. It was the operations research community who proposed the models, felt they could develop models, sold them to the military, spent the budget and finally produced the models built upon a “base of sand.” As such, they are the community that needs to correct the problem and produce this theory of combat. They are the scientists.

Yet, the problem is bigger than a “base of sand.” This phrase implies that there is a shortfall of data to start with. Yet every complex model (and most of these models are complex) is built from a number of interrelationships within the model. This is even more so for hierarchy of model structures. Each of these inter-relationships, which are often model unique constructs, is often built upon “expert judgment.” Therefore, the “base of sand” does not just start at the bottom, but carries through to each individual function within a model. As such, what has been built upon this “base of sand” is a “house of cards.”