Category Modeling, Simulation & Wargaming

U.S. Senate and Model Validation – Comments

This is a follow-up to our blog post:

Have They Been Reading Our Blog?

This rather significant effort came out of the blue for us, and I gather a whole lot of others in the industry. The actual proposed law is here:

U.S. Senate on Model Validation

Some people, we gather, are a little nervous about this effort. On the other hand, Clinton Reilly, an occasional commenter on this blog and the Managing Director of Computer Strategies Pty Ltd, Sydney Australia, nicely responds to these concerns with the following post:

I would not be too concerned by the prospect of more rigorous validation. I am sure it represents a major opportunity to improve modelling practices and obtain the increased funding that will be required to support the effort.

One of the first steps will be the development of a set of methodologies that will be tailored to testing the types of models required. I am sure that there will be no straight jacketing or enforced uniformity as it is obvious the needs served by the models are many and varied and cannot be met by a “one size fits all” approach.

Provided modellers prepare themselves by developing an approach to validation that is required by their user community they will be in a good position to work with the investigating committee and secure the support and funding needed.

In the end, validation is not a “pass-fail” test to be feared, it is a methodology to improve the model and improve confidence in the model results, and to fully understand the strengths and weaknesses of a model. This is essential if you are going to be using the model for analytical uses, and practically essential even if you are using it for training.

So this is an opportunity not a threat. It is a much needed leap forward.

Let us begin work on developing an approach to validation that suits our individual modelling requirements so that we can present them to the review committee when it asks for input.

Now, my experience on this subject, which dates back to managing the Ardennes Campaign Simulation Data Base (ACSDB) in 1987, is that many of the U.S. Military Operations Research community will not see it as “…an opportunity, not a threat.” We shall see.

The Elements of Trevor Dupuy’s Theory of Combat

Trevor Dupuy’s combat models (the Quantified Judgement Model (QJM) and the Tactical Numerical Deterministic Model (TNDM)) are formal expressions of his theory of combat. Dupuy provided an extensive discussion of the basis for his theory in his books, particularly Understanding War: History and Theory of Combat (NOVA Publications, 1987). While many are familiar with his models, fewer are aware of the concepts that he based it upon. This will be the first in a series of posts looking at those elements.

As Dupuy explained,

As a starting point for an explanation of a scientific theory, it is useful to define fundamental terms, to state and explain critical assumptions, and to establish—or limit—the scope of the discussion that follows. The definitions and explanations that follow are generally consistent with usage in the military and analytical communities, and with definitions that have been formulated for its work by The Military Conflict Institute. However, I have in some instances modified or restated these to conform to my own ideas and usage. [Dupuy, Understanding Combat, 63]

The basic elements of his theory of combat are:

Definition of Military Combat
The Hierarchy of Combat
The Conceptual Components of Combat
The Scope of Theory
Definition of a Theory of Combat

These will each be discussed in future posts.

Reminder – Speaking at Historicon in Lancaster, PA., Friday 12 July

I will be speaking at Historicon in Lancaster, PA., Friday 12 July. Historicon is one of the three major annual wargaming conventions run by the Historical Miniatures Gaming Society (HMGS). It will be run from 10 July-14 July, 2019. Their website is here: https://www.hmgs.org/general/custom.asp?page=HconHome

As part of this large convention, they have organized a “War College.” This is an impressive effort that includes 18 lectures on Thursday, Friday and Saturday. I have the last lecture on Friday, from 6 – 7 PM. The speakers for this series include published authors Paul Westermeyer, Pete Panzeri, Steve R. Waddell and John Prados, among others. Lecture descriptions are here:                                                                               . https://cdn.ymaws.com/www.hmgs.org/resource/resmgr/historicon/hcon_19/pels/19_war_college_pel_6-19-2019.pdf

I will be doing a presentation similar to the one I did at the New York Military Affairs Symposium (NYMAS). It is based upon part of my book War by Numbers: Understanding Conventional Combat.

How Common are Casualty Estimates?

I do note that several of the talking heads on TV have mentioned that providing a casualty estimate is standard practice when briefing for a military action. This may be the case, I have never been to one of those briefings. But….while casualty estimates for a single proposed operation may be common, my experience is that they are not that common when it comes to evaluating wars.

There were certainly multiple estimates done in 1990 for the 1991 Gulf War both inside and outside the Government. A number of estimates were high and many have never been publicly released. I have yet to have seen any systematic analysis of the casualty estimates done for the Gulf War. Most of these were based upon combat models, so as such, serve as a validation test for these combat models. Wikipedia used to have a discussion on this issue, but it has since disappeared. Needless to say, the estimate derived from a combat model made before a war is probably something that would be very telling. The fact that DOD never conducted an analysis after the war of the various estimates is also very telling.

Now, we did do an estimate in fall of 1995 for a Bosnia peacekeeping operation (the Dayton conference ended with a peace agreement in November 1995). This was the only estimate done for this operation. According to rumor, the Joint Chiefs of Staff asked the Deputy Under Secretary of the Army for Operations Research (DUSA-OR) if they could do such an estimate, and were told that there was no methodology to estimate casualties in an Operation Other Than War (OOTW) (see page 309, America’s Modern Wars). So The Dupuy Institute ended up doing such an estimate, and we did it based upon two different methodologies (see Appendix II, America’s Modern Wars).

As far as I know, we were also the only people who did an estimate in 2004/2005 for the cost and duration of an extended guerilla war in Iraq. Some one else may have done some work on duration, as I kept hearing the statement made that guerilla wars lasted an average of ten years. I could never tell if that figure was pulled from our work or if someone else did some similar work. Regardless, I think we were the only people who did an analytically based casualty estimate of the war. See Chapter 1, America’s Modern Wars.

Now, when I was writing America’s Modern Wars, I decided to look into what estimates had been made in the first half of the 1960s for the war in Vietnam. I really could not find anything analytical, although my search was not exhaustive (meaning I am still waiting for someone to prove me wrong). But, it does not appear that any analytically based estimate was made during the 1960s for casualties and duration for what turned into America’s third bloodiest and second longest war. See pages 4, 29-30, America’s Modern Wars.

So, there appears to have been an estimate done for the casualties for a strike on Iraq (although I doubt it will be made public). On the other hand, it is doubtful if anyone has done an estimate for the full range of options, or addressing the Iranian counter-options, or examining an extended conflict with Iran. This would certainly have to address the range of counter-strikes and other options available to Iran if such a conflict further developed. For various reasons, the DOD seems hesitant to do these types of estimates.

Now, I happen to think our decision makers would be better served if they indeed did have some estimates of casualties and duration of these conflicts when they were contemplating their options. They were not done for the Vietnam War, only one estimate was done for Bosnia and I think only one estimate was done for the continuing war in Iraq. I am guessing, based upon that track record, no such estimate has been made for an extended conflict with Iran.

Speaking at Historicon in Lancaster, PA., Friday 12 July

I will be speaking at Historicon in Lancaster, PA., Friday 12 July. Historicon is one of the three major annual wargaming conventions run by the Historical Miniatures Gaming Society (HMGS). It will be run from 10 July-14 July, 2019. Their website is here: https://www.hmgs.org/general/custom.asp?page=HconHome

As part of this large convention, they have organized a “War College.” This is an impressive effort that includes 18 lectures on Thursday, Friday and Saturday. I have the last lecture on Friday, from 6 – 7 PM. The speakers for this series include published authors Paul Westermeyer, Pete Panzeri, Steve R. Waddell and John Prados, among others. Lecture descriptions are here:                                                                               . https://cdn.ymaws.com/www.hmgs.org/resource/resmgr/historicon/hcon_19/pels/19_war_college_pel_6-19-2019.pdf

I will be doing a presentation similar to the one I did at  the New York Military Affairs Symposium (NYMAS). It is on War by Numbers: Understanding Conventional Combat.

Have They Been Reading Our Blog?

From January through April we ended up doing more than two dozen blog posts on the issue of validation. I also addressed the subject in my book War by Numbers (Chapter 18: Modeling Warfare). It is, of course, an issue I have writing about since 1997 with my article in Volume I, No. 4 of our modeling newsletter (here: http://www.dupuyinstitute.org/tdipub4.htm). We also did a number of efforts inside of the DOD to promote validation, the most significant being our Casualty Estimation Methodologies Study (May 2005) (see: http://www.dupuyinstitute.org/tdipub3.htm). This report resulted in at least one manager at one agency recommending that we be terminated from all contracts with them because we criticized their models. The response to all this discussion was underwhelming to say the least, until now:

U.S. Senate on Model Validation

And then out of the blue, here comes the Senate Armed Services Committee report. We have no idea where this came from, who did it, or why? Were they reading our blog?

Summation of our Validation Posts

U.S. Senate on Model Validation

This link is to report 116-48, The National Defense Authorization Act for Fiscal Year 2020 by the Committee on Armed Services, United States Senate, dated June 11, 2019: https://www.congress.gov/116/crpt/srpt48/CRPT-116srpt48.pdf

While this 609 page report is probably worth reading from cover to cover, that I have not done. On the other hand, pages 253-254 are worth quoting (the bolding in the text is mine):

Evaluation of modeling and simulation used for force planning and theater operational requirements

The committee notes that the Department of Defense uses a large number and variety of computer models and simulations to support decision-making about force structure, resource allocation, war gaming, and priority weapons platforms and technologies to develop and deploy in support of likely operational scenarios. These models are used to develop information to brief decision makers, including the Congress, about, for example, the current state of the balance of forces in the Pacific and European theaters, the outcomes of likely war scenarios, and the need for investments in advanced technologies and new warfighting capabilities.

The committee is concerned that the quality, accuracy, and dependability of these models, given their important role in decision making processes, has not been adequately validated. The committee notes that technical and engineering models used to develop systems such as body armor and missiles are rigorously verified and validated for veracity of assumptions and technical accuracy using real world data. Other models, such as those used in the financial sector, are developed using expertise from a variety of disciplines, including economics, sociology, and advanced mathematics. The committee is concerned that the models used by organizations including the Joint Staff, Office of Net Assessment, war colleges, and service-level planning entities are simplistic by comparison and not subject to the same level of scrutiny.

Therefore, the committee directs the Secretary of Defense to establish an independent team of academic and industry modeling and subject matter experts to review the quality of modeling and simulation used for force planning, war gaming, resource allocation, and other senior leader decision-making associated with implementation of the National Defense Strategy. The team shall review the technical quality of models currently in use, including their ability to simulate as required by application; physics and engineering, socio-economic impact, readiness, global financial markets, politics, and other relevant inputs and outputs. The team shall assess the quality of these models and make recommendations for investments or policy changes needed to enhance and continuously validate current and future modeling and simulation tools employed to enable senior-level decision-making.

The committee directs the Secretary to support the team with expertise as needed from the Joint Staff, Office of Net Assessment, Under Secretary of Defense for Research and Engineering, and other relevant organizations. The Secretary shall ensure that the team has sufficient resources and access to all data and records necessary to perform its analysis. The committee directs the Secretary to deliver a report on the independent team’s assessments and recommendations with any additional comments, and a specific concurrence or non-concurrence for each recommendation, to the Committees on Armed Services of the Senate and House of Representatives no later than December 31, 2020.

A big thank you to the person who brought this to my attention.

Signal Multi-player Game

This was just flagged to me by one of our readers in the UK: https://phys.org/news/2019-05-science-wargames.html

It is a multi-player game developed by researchers at University of California, Lawrence Livermore and Sandia. It was done for the Carnegie Corporation, a non-profit: https://www.carnegie.org/ and https://en.wikipedia.org/wiki/Carnegie_Corporation_of_New_York

They have an open play window every Wednesday and Thursday 1 to 5 PM Pacific Time (4 – 8 EST). The link is here: https://www.signalvideogame.com/

I know nothing about this effort. An image of it is at the top of this blog post. Looks like a fairly typical hex game.

Breakpoints

A couple of our posts on Breakpoints (forced changes in posture) are getting a lot of hits lately. Not sure why or by who. Let me list all of our posts addressing the issue of breakpoints:

What Is A Breakpoint?

Response 3 (Breakpoints)

Breakpoints in U.S. Army Doctrine

C-WAM 4 (Breakpoints)

Diddlysquat

Engaging the Phalanx (part 7 of 7)

It is also discussed in my book War by Numbers, pages 287-289 and briefly mentioned on page 291.

Oh…and here also (forgot about this one as I only did a search on the word “breakpoint”):

Battle Outcomes: Casualty Rates As a Measure of Defeat

Throwing the Dice

This is a follow-up to the blog post:

China and Russia Defeats the USA

One of readers commented on this post and posted the following link: Paper Wargames and Policy Making

In case you are not reading the comments to our blog post, wanted to make sure this was brought to your attention. The article is worth taking a look at just for the pictures. A few highlights:

  1. “In my lifetime, computer-based simulation have largely taken over analytical gaming, sometimes bringing new levels of investigative power, but often just providing the illusion of it as the details of the models and their simplifying assumptions become invisible to players and to the policymakers whose decisions the games are supposed to inform.”
  2. However, it gradually became clear [with the Baltic states wargame] — rather disconcertingly–that we were out in front of most of the official planning, not following in its wake.”
  3. “The resulting game….resolves combat using 12-hour turns and 10-km hexes; units are battalions of ground forces, SAM batteries, and half-squadrons of aircraft (12 fighters or six bombers).”
  4. “…and they like throwing the dice.”