Category Ardennes Campaign Simulation Data Base

Battles versus Campaigns (for Validation)

So we created three campaign databases. One of the strangest arguments I have heard against doing validations or testing combat models to historical data, is that this is only one outcome from history. So you don’t know if model is in error or if this was a unusual outcome to the historical event. Someone described it as the N=1 argument. There are lots of reasons why I am not too impressed with this argument that I may enumerate in a later blog post. It certainly might apply to testing the model to just one battle (like the Battle of 73 Easting in 1991), but these are weeks-long campaign databases with hundreds of battles. One can test the model to these hundreds of points in particular in addition to testing it to the overall result.

In the case of the Kursk Data Base (KDB), we have actually gone through the data base and created from it 192 division-level engagements. This covers every single combat action by every single division during the two week offensive around Belgorod. Furthermore, I have listed each and every one of these as an “engagement sheet’ in my book on Kursk. The 192 engagement sheets are a half-page or page-long tabulation of the strengths and losses for each engagement for all units involved. Most sheets cover one day of battle. It took considerable work to assemble these. First one had to figure out who was opposing whom (especially as unit boundaries never match) and then work from there. So, if someone wants to test a model or model combat or do historical analysis, one could simply assemble a database from these 192 engagements. If one wanted more details on the engagements, there are detailed breakdowns of the equipment in the Kursk Data Base and detailed descriptions of the engagements in my Kursk book. My new Prokhorovka book (release date 1 June), which only covers the part of the southern offensive around Prokhorovka from the 9th of July, has 76 of those engagements sheets. Needless to say, these Kursk engagements also make up 192 of the 752 engagements in our DLEDB (Division Level Engagement Data Base).  A picture of that database is shown at the top of this post.

So, if you are conducting a validation to the campaign, take a moment and check the results to each division to each day. In the KDB there were 17 divisions on the German side, and 37 rifle divisions and 10 tank and mechanized corps (a division-sized unit) on the Soviet side. The data base covers 15 days of fighting. So….there are around 900 points of daily division level results to check the results to. I drawn your attention to this graph:

There are a number of these charts in Chapter 19 of my book War by Numbers. Also see:

Validating Attrition

The Ardennes database is even bigger. There was one validation done by CAA (Center for Army Analysis) of its CEM model (Concepts Evaluation Model) using the Ardennes Campaign Simulation Data Bases (ACSDB). They did this as an overall comparison to the campaign. So they tracked the front line trace at the end of the battle, and the total tank losses during the battle, ammunition consumption and other events like that. They got a fairly good result. What they did not do was go into the weeds and compare the results of the engagements. CEM relies on inputs from ATCAL (Attrition Calculator) which are created from COSAGE model runs. So while they tested the overall top-level model, they really did not test ATCAL or COSAGE, the models that feed into it. ATCAL and COSAGE I gather are still in use. In the case of Ardennes you have 36 U.S. and UK divisions and 32 German divisions and brigades over 32 days, so over 2,000 division days of combat. That is a lot of data points to test to.

Now we have not systematically gone through the ACSDB and assembled a record for every single engagement there. There would probably be more than 400 such engagements. We have assembled 57 engagements from the Battle of the Bulge for our division-level database (DLEDB). More could be done.

Finally, during our Battle of Britain Data Base effort, we recommended developing an air combat engagement database of 120 air-to-air engagements from the Battle of Britain. We did examine some additional mission specific data for the British side derived from the “Form F” Combat Reports for the period 8-12 August 1940. This was to demonstrate the viability of developing an engagement database from the dataset. So we wanted to do something similar for the air combat that we had done with division-level combat. An air-to-air engagement database would be very useful if you are developing any air campaign wargame. This unfortunately was never done by us as the project (read: funding) ended.

As it is we actually have three air campaign databases to work from, the Battle of Britain data base, the air component of the Kursk Data Base, and the air component of the Ardennes Campaign Simulation Data Base. There is a lot of material to work from. All it takes it a little time and effort.

I will discuss the division-level data base in more depth in my next post.

The Use of the Two Campaign Data Bases

The two large campaign data bases, the Ardennes Campaign Simulation Data Base (ACSDB) and the Kursk Data Base (KDB) were designed to use for validation. Some of the data requirements, like mix of personnel in each division and the types of ammunition used, were set up to match exactly the categories used in the Center for Army Analysis’s (CAA) FORCEM campaign combat model. Dr. Ralph E. Johnson, the program manager for FORCEM was also the initial contract manager for the ACSDB.

FORCEM was never completed. It was intended to be an improvement to CAA’s Concepts Evaluation Model (CEM) which dated back to the early 1970s. So far back that my father had worked with it. CAA ended up reverting back to CEM in the 1990s.

They did validate the CEM using the ACSDB. Some of their reports are here (I do not have the link to the initial report by the industrious Walt Bauman):

https://apps.dtic.mil/dtic/tr/fulltext/u2/a320463.pdf

https://apps.dtic.mil/dtic/tr/fulltext/u2/a489349.pdf

It is one of the few actual validations ever done, outside of TDI’s (The Dupuy Institute) work. CEM is no longer used by CAA. The Kursk Data Base has never used for validation. Instead they tested Lanchester equations to the ACSDB and KDB. They failed.

Lanchester equations have been weighed….

But the KDB became the darling for people working on their master’s thesis for the Naval Post-Graduate School. Much of this was under the direction of Dr. Tom Lucas. Some of their reports are listed here:

http://www.dupuyinstitute.org/links.htm

Both the ACSDB and KDB had a significant air component. The air battle over the just the German offensive around Belgorod to the south of Kursk was larger than the Battle of Britain. The Ardennes data base had 1,705 air files. The Kursk data base had 753. One record, from the old Dbase IV version of the Kursk data base, is the picture that starts this blog post. These files basically track every mission for every day, to whatever level of detail the unit records allowed (which were lacking). The air campaign part of these data bases have never been used for any analytical purpose except our preliminary work on creating the Dupuy Air Campaign Model (DACM).

The Dupuy Air Campaign Model (DACM)

This, of course, leads into our next blog post on the Battle of Britain data base.

Validation Data Bases Available (Ardennes)

We never seem to stop discussing validation at The Dupuy Institute even though it seems like most of the military operations research community only pays lip service to it. Still, it is something we encourage, even though it has only been very rarely done. A major part of our work over the years has been creation of historical databases for use in combat model validation, combat model testing, and combat model development. Over a series of posts, let me describe what databases are available.

First there are two big campaign databases. These are fairly well known. It is the Ardennes Campaign Simulation Data Base (ACSDB) and the Kursk Data Base (KDB). I was the program manager for both of them.

The ACSDB is a database tracking the actions of all divisions on both sides for every day of the Battle of the Bulge, from 16 December 1944 to 16 January 1945. That was 36 U.S. and British Divisions and 32 German Divisions and Brigades. It tracked the strength, equipment, losses, ammunition and oil for each of these units. The stats on the database are here: http://www.dupuyinstitute.org/data/ardennes.htm

The ACSDB was done from 1987 to 1990 at Trevor Dupuy’s old company, DMSI. There was around 16 or so people involved with it, including consultants like Hugh Cole and Charles MacDonald. We pulled up the units records for all the units on both sides from the U.S. Archives, UK Public Records Office, and the German achives in Freiburg. It was the largest historical database ever created (I do seem to be involved in creating some large things, like my Kursk book).

The database was programmed in Dbase III and is DOS based. The data base was delivered to CAA (Center for Army Analysis). It is publicly available through NTIS (National Technical Information Service). The pictures in this blog post are screen shots from the DBase III version. We do have our own corporate proprietary version re-programmed into Access and with some updates done by Richard Anderson (coauthor of Hitler’s Last Gamble).

Historical Demonstrations?

Photo from the 1941 Louisiana Maneuvers

Continuing my comments on the article in the December 2018 issue of the Phalanx by Alt, Morey and Larimer (this is part 5 of 7; see Part 1, Part 2, Part 3, Part 4).

The authors of the Phalanx article then make the snarky statement that:

Combat simulations have been successfully used to replicate historical battles as a demonstration, but this is not a requirement or their primary intended use.

So, they say in three sentences that combat models using human factors are difficult to validate, they then say that physics-based models are validated, and then they say that running a battle through a model is a demonstration. Really?

Does such a demonstration show that the model works or does not work? Does such a demonstration show that they can get a reasonable outcome when using real-world data? The definition of validation that they gave on the first page of their article is:

The process of determining the degree to which a model or simulation with its associated data is an accurate representation of the real world from the perspective of its intended use is referred to as validation.

This is a perfectly good definition of validation. So where does one get that real-world data? If you are using the model to measure combat effects (as opposed to physical affects) then you probably need to validate it to real-world combat data. This means historical combat data, whether it is from 3,400 years ago or 1 second ago. You need to assemble the data from a (preferably recent) combat situation and run it through the model.

This has been done. The Dupuy Institute does not exist in a vacuum. We have assembled four sets of combat data bases for use in validation. They are:

  1. The Ardennes Campaign Simulation Data Base
  2.  The Kursk Data Base
  3. The Battle of Britain Data Base
  4. Our various division-level, battalion-level and company-level engagement database bases.

Now, the reason we have mostly used World War II data is that you can get detailed data from the unit records of both sides. To date….this is not possible for almost any war since 1945. But, if your high-tech model cannot predict lower-tech combat….then you probably also have a problem modeling high-tech combat. So, it is certainly a good starting point.

More to the point, this was work that was funded in part by the Center for Army Analysis, the Deputy Secretary of the Army (Operations Research) and Office of Secretary of Defense, Planning, Analysis and Evaluation. Hundreds of thousands of dollars were spent developing some of these databases. This was not done just for “demonstration.” This was not done as a hobby. If their sentence was meant to be-little the work of TDI, which is how I do interpret that sentence, then is also belittles the work of CAA, DUSA(OR) and OSD PA&E. I am not sure that is the three author’s intent.

Validating Attrition

Continuing to comment on the article in the December 2018 issue of the Phalanx by Alt, Morey and Larimer (this is part 3 of 7; see Part 1, Part 2)

On the first page (page 28) in the third column they make the statement that:

Models of complex systems, especially those that incorporate human behavior, such as that demonstrated in combat, do not often lend themselves to empirical validation of output measures, such as attrition.

Really? Why can’t you? If fact, isn’t that exactly the model you should be validating?

More to the point, people have validated attrition models. Let me list a few cases (this list is not exhaustive):

1. Done by Center for Army Analysis (CAA) for the CEM (Concepts Evaluation Model) using Ardennes Campaign Simulation Study (ARCAS) data. Take a look at this study done for Stochastic CEM (STOCEM): https://apps.dtic.mil/dtic/tr/fulltext/u2/a489349.pdf

2. Done in 2005 by The Dupuy Institute for six different casualty estimation methodologies as part of Casualty Estimation Methodologies Studies. This was work done for the Army Medical Department and funded by DUSA (OR). It is listed here as report CE-1: http://www.dupuyinstitute.org/tdipub3.htm

3. Done in 2006 by The Dupuy Institute for the TNDM (Tactical Numerical Deterministic Model) using Corps and Division-level data. This effort was funded by Boeing, not the U.S. government. This is discussed in depth in Chapter 19 of my book War by Numbers (pages 299-324) where we show 20 charts from such an effort. Let me show you one from page 315:

 

So, this is something that multiple people have done on multiple occasions. It is not so difficult that The Dupuy Institute was not able to do it. TRADOC is an organization with around 38,000 military and civilian employees, plus who knows how many contractors. I think this is something they could also do if they had the desire.

 

Validation

Continuing to comment on the article in the December 2018 issue of the Phalanx by Jonathan Alt, Christopher Morey and Larry Larimer (this is part 2 of 7; see part 1 here).

On the first page (page 28) top of the third column they make the rather declarative statement that:

The combat simulations used by military operations research and analysis agencies adhere to strict standards established by the DoD regarding verification, validation and accreditation (Department of Defense, 2009).

Now, I have not reviewed what has been done on verification, validation and accreditation since 2009, but I did do a few fairly exhaustive reviews before then. One such review is written up in depth in The International TNDM Newsletter. It is Volume 1, No. 4 (February 1997). You can find it here:

http://www.dupuyinstitute.org/tdipub4.htm

The newsletter includes a letter dated 21 January 1997 from the Scientific Advisor to the CG (Commanding General)  at TRADOC (Training and Doctrine Command). This is the same organization that the three gentlemen who wrote the article in the Phalanx work for. The Scientific Advisor sent a letter out to multiple commands to try to flag the issue of validation (letter is on page 6 of the newsletter). My understanding is that he received few responses (I saw only one, it was from Leavenworth). After that, I gather there was no further action taken. This was a while back, so maybe everything has changed, as I gather they are claiming with that declarative statement. I doubt it.

This issue to me is validation. Verification is often done. Actual validations are a lot rarer. In 1997, this was my list of combat models in the industry that had been validated (the list is on page 7 of the newsletter):

1. Atlas (using 1940 Campaign in the West)

2. Vector (using undocumented turning runs)

3. QJM (by HERO using WWII and Middle-East data)

4. CEM (by CAA using Ardennes Data Base)

5. SIMNET/JANUS (by IDA using 73 Easting data)

 

Now, in 2005 we did a report on Casualty Estimation Methodologies (it is report CE-1 list here: http://www.dupuyinstitute.org/tdipub3.htm). We reviewed the listing of validation efforts, and from 1997 to 2005…nothing new had been done (except for a battalion-level validation we had done for the TNDM). So am I now to believe that since 2009, they have actively and aggressively pursued validation? Especially as most of this time was in a period of severely declining budgets, I doubt it. One of the arguments against validation made in meetings I attended in 1987 was that they did not have the time or budget to spend on validating. The budget during the Cold War was luxurious by today’s standards.

If there have been meaningful validations done, I would love to see the validation reports. The proof is in the pudding…..send me the validation reports that will resolve all doubts.

Wargaming Thread on Combat Results Tables

Thanks to a comment made on one of our posts, I recently became aware of a 17 page discussion thread on combat results tables (CRT) that is worth reading. It is here:

https://www.boardgamegeek.com/thread/1344914/crts-101/page/1

By default, much of their discussion of data centers around analysis based upon Trevor Dupuy’s writing, the CBD90 database, the Ardennes Campaign Simulation Data Base (ACSDB), the Kursk Data Base (KDB)  and my book War by Numbers. I was not aware of this discussion until yesterday even though the thread was started in 2015 and continues to this year (War by Numbers was published in 2017 so does not appear until the end of page 5 of the thread).

The CBD90 was developed from a Dupuy research effort in the 1980s eventually codified as the Land Warfare Data Base (LWDB). Dupuy’s research was programmed with errors by the government to create the CBD90. A lot of the analysis in my book was based upon a greatly expanded and corrected version of the LWDB. I was the program manager for both the ACSDB and the KDB, and of course, the updated versions of our DuWar suite of combat databases.

http://www.dupuyinstitute.org/dbases.htm

There are about a hundred comments I could make to this thread, some in agreement and some in disagreement, but then I would not get my next book finished, so I will refrain. This does not stop me from posting a link:

Lanchester equations have been weighed….

 

Cost of Creating a Data Base

Invariably, especially with a new book coming out (War by Numbers), I expected to get requests for copies of our data bases. In fact, I already have.

Back around 1987 or so, a very wise man (Curt Johnson, VP of HERO) estimated that for the LWDB (Land Warfare Data Base) that it took 3 man-days to create an engagement. The LWDB was the basis for creating many of our later data bases, including the DLEDB (Division Level Engagement Data Base). My experience over time is that this estimate is low, especially if your are working with primary sources (unit records) for both sides. I think it may average more like 6 man-days an engagement if based upon unit records (this includes the time to conduct research).

But going with Curt’s estimate, let’s take the DLEDB of 752 cases and re-create it. This would take 3 man-days times 752 engagements = 2,256 man-days. This is 9 man-years of effort. Now 9 man-years times a loaded professional rate. A loaded man-year is the cost of a person’s labor times indirect costs (vacation, SS and Medicare contributions, health insurance, illness, office space, etc.), general and administrative costs (corporate expenses not included in the indirect costs, including senior management and marketing), and any fee or profit. Loaded rate is invariably at least 60% of the direct costs and usually closer to 100% of direct costs (and I worked at one company where it was 200% of direct costs). So a loaded man-year may be as low at $120,000 a year but for people like RAND or CNA, it is certainly much higher. Nine man-years times $120,000 = $1,080,000.

Would it really cost more than a million dollars to re-created the DLEDB? If one started from scratch, certainly. Probably (much) more, because of all the research into the Ardennes and Kursk that we did as part of those database projects. The data bases were created incrementally over the course of more than 30 years as part of various on-going contracts and efforts. We also had a core group of very experienced personnel who were doing this.

Needless to say, if any part of the data base is given away, loaned out, or otherwise not protected, we loose control of the “proprietary” aspect of these data bases. This includes the programming and formatting. Right now, they are unique to The Dupuy Institute, and for obvious business reasons, need to remain so unless proper compensation is arranged.

Sorry.

 

P.S. The image used is from the old Dbase IV version of the Kursk Data Base. We have re-programmed it in Access.

 

Dupuy Institute Data Bases

Yes, I still use data base as two words, much to the annoyance of Jay Karamales.

Anyhow, War by Numbers does rely extensively on a group of combat data bases that were developed over several decades. The earliest versions were developed in the 1970s and they were assembled into a large data base of around 600 cases in the 1980s. They were then computerized (they were originally a paper data base), re-organized, re-programed in Access, and greatly expanded. The data bases we currently have include:

Conventional Combat Data Bases:

LADB = Large Action Data Bases of 55 cases

DLEDB = Division Level Engagement Data Base of 752 cases

BLODB = Battalion Level Operations Data Base of 127 cases

CLEDB = Company Level Engagement Data Base of 98 cases

SADB = Small Action Data Base of 5 cases

BaDB = Battles Data Base of 243 cases from 1600-1900

 

We also have:

CaDB = Campaign Data Base of 196 cases. While the other data bases address battles, or engagements of no more than a few days in length, this one summarizes campaigns, often extending for months.

Finally we have three databases tracking campaigns from day-to-day. They are all programmed in Access:

ACSDB = Ardennes Campaign Simulation Data Base (meaning Battle of the Bulge)

KDB = Kursk Data Base

Battle of Britain Data Base

These were primarily intended for model validation efforts.

We also have three insurgency/peaceeping/intervention/OOTW (Operations Other than War) data bases. They are:

WACCO = Warfare and Armed Conflict Data Base of 793 cases

SSCO = Small Scale Operations Data Base of 203 cases

DISS = Dupuy Insurgency Spread Sheets of 109 cases.

 

The DISS data base was the one that America’s Modern Wars is based upon. The other two were earlier efforts.

These links provides some snap shots of the data base content: http://www.dupuyinstitute.org/dbases.htm

These are all company proprietary, although some have been released publicly in earlier forms or different forms (including the CHASE data base of 599 cases, the ACSDB in Dbase III and the KDB in Dbase IV). Our versions have been updated, including revisions to content.

Economics of Warfare 15-1

Moving onto the fifteenth lecture from Professor Michael Spagat’s Economics of Warfare course that he gives at Royal Holloway University. It is posted on his blog Wars, Numbers and Human Losses at: https://mikespagat.wordpress.com/

This lecture addresses the impact of global warming on armed conflict over time. If you happen to be one of those who believes that global warming is pseudo-science/a scientific hoax/an excuse to troll for research $$$/an invention of Al Gore/a liberal plot/a religion/a Chinese plot/obviously false because it was cold yesterday/and so forth…..then probably best to stop reading. On the other hand, we have done some proposals on measuring the impact of climate change on violence and consider this a legitimate area of study. Our attention was drawn to the subject over a decade ago when a CNA (Center for Naval Analysis) paper came out that postulated that global warming could result in more violence. This conclusion does not appear to have been based upon any analysis of data, just the assumption that as things get worse (in the environment) then things are going to get worse (with armed conflict). Of course, going back to Feierabend & Feierabend (and I do go back to them a lot)….poorer counties had less political violence than developing countries. Therefore, it does not necessarily follow that worse environmental and economics conditions results in more violence. The effect may be the reverse, which is that declining conditions may actually result in a reduction of violence. We really don’t know. Trying to examine these effects analytically was the gist of my proposals on the subject, but sequestration happened and budget for anything seemed to disappear.

So….first two sentences of Dr. Spagat’s slides are

“There is a strong scientific consensus that the Earth is getting warmer over time.”

“It is reasonable to imagine that a side effect of global warming could be an increase in armed conflict over time.”

Slide 2 looks at possible channels that could lead to conflict

  1. Dwindling food supply
  2. Dwindling water supply
  3. Sea Level changes causing migration.

On slide 3 he then addresses a study by Burke and others that attempt to address these concerns using a cross-country regression approach and linear probability model.

On slide 5 the results are summarized as “…an increase of 1 degree centigrade for a  particularly country in a particular year is associated with a 0.0447 increase in the probability of there being an ongoing civil war….”

and on slide 11 as: “This means that Burke et al. predict that 15.8-17.1% of the countries in Sub-Saharan Africa in 2030 will suffer from big civil wars rather than the 11% that would occur without the warming climate.”

and on slide 16 as: “Burke et al. go on to predict 393,000 excess battle deaths caused by climate change…” (my bolding). Dr. Spagat then examines this number in the next two slides. It doesn’t sound like he fully accepts it.

Now, Burke based his study on the period from 1981-2002. One of Dr. Spagat’s TAs then used the model to make predictions from the period 2003-2013. There is nothing like trying to use a model to predict the past. It sort of shows whether it really works or not. This was the reasoning because the Ardennes Campaign Simulation Data Base and the Kursk Data Base that we did (they were model validation data bases). It is related in concept to what I did in America’s Modern War, pages 65-68 when I tested my logistics probability model back to the 68 cases used to create the model and tried to figure out for each case why the model was predicting wrong. Once you have a model, there are lots of things to test it to in the past. If you can’t predict the past, you may not be able to predict the future.

Anyhow, the results are on slide 19 and summarized in slide 20 as

  1. “There are 414 “no war” predictions…A war actually happens in 11 out of these 414 cases.”
  2. “There are 37 predictions of “war”. War actually happens in 7 out of these 37 cases.”

Not sure I am any smarter at this point, but I am certainly amused.

His final point is “The Burke et al. model seems to be of some use in predicting wars although it seems have a general tendency to predict war too often.”

And then Dr. Spagat TA test how important the temperature variable is for making these predictions, so takes the temperature variable out of the model !!! This produces a table (slide 21) that is almost identical to his original table. The impact of removing the temperature from the model is that it produced five more false positives (predicted wars that did not happen). I am even more amused.

Spagat’s conclusion (slide 22) is “…They mean that temperature is not very useful for predicting civil war….”

This is a good point to stop…I will pick up the rest of this lecture in another post. The link to the lecture is here: http://personal.rhul.ac.uk/uhte/014/Economics%20of%20Warfare/Lecture%2015.pdf