Category Combat Databases

Korean War Records

Not much to say about captured records in the Korea War as I have never checked on them. I assume there must be some taken from North Korean and Chinese units and they are files away somewhere. My father did capture a Chinese soldier during the Korean War.

Oddly enough, there not been much done in the world of quantitative analysis on the Korea War outside of the work that ORO (Operations Research Office) did in the 1950s. We have never done any significant work on the Korean War. In the late 1980s we did explore conducting some analysis of Korean War battalion-level combat. As part of that effort Trevor Dupuy and I went over to the National Archives at Suitland and pulled up some U.S. Army Korea War records. They appeared to be quite complete. There were a couple of French infantry battalions attached to the U.S. Division and we appear to have good strength and loss data for them also.

Later, in 1989, Trevor Dupuy arranged with China to conduct a joint research project. It was funded by OSD Net Assessment (Andy Marshall). Trevor Dupuy really wanted to do some two-sided analysis of combat with the Chinese Army in Korea, but apparently getting access to the Chinese Army records was still too sensitive at that point. So, instead, they arranged to do a joint research contract on a more general and less sensitive theme like perceptions of each sides intentions during the Korean War. But then in June 1989 the Chinese government rolled over the student protestors in Tiananmen Square with tanks. That ended all joint research projects for many years.

We never got back to trying to conduct a joint research project on combat with China. Instead in 1993, we started a research project on Kursk using Russia records.

Trevor Dupuy did mention that the Chinese informally told him that the United States often overestimated the size of the Chinese forces they were facing, and often underestimated the casualties the Chinese took. I have no idea how valid that is.

 

Anyhow, this is an extended discussion of captured records originally inspired by this post:

The Sad Story Of The Captured Iraqi DESERT STORM Documents

Captured Records: WWII

At the end of World War I, the United States made sure they had access to the German military records in their treaty and sent a research team over there to review and copy the pertinent records. At the end of World War II, we just took everything. The allies gathered together all the material, in part because of concerns over war crimes, and eventually the entire collection was shipped back to the United States.

In the 1960s, the United States decided to repatriate the records back to Germany. Before they shipped them out they decided to copy the entire collection of German World War II records and place them on microfilm. This was a massive effort done by government contractors that took several years. Like all good government contracts, towards the end, this one was behind schedule, so they were having to cut corners by choosing not to copy things that they felt were not particularly relevant. So, the originals records are now in the archives in Freiburg, Germany (a nice little town about as far away from the old East German border as you can get). There are copies on microfilm of most of the German records, sometimes in disorder, in the U.S. Archives II in College Park, Maryland (near Washington DC). The British also have microfilm copies of portions of the German records collection that they captured over at the Public Records Office in Kew (London). The UK collection is a fraction the size of the U.S. microfilm collection, and as far as I know has nothing additional in it.

But, there were some records that were not copied by the U.S., but it is not much. For example, for the Kursk Data Base, I did the German research from the U.S. record collection. There were 17 German divisions in the offensive in the south in July 1943. I made a detailed listing of the records I have reviewed and sent that list to Dr. Arthur Volz over in Germany. He then went to Frieburg and tried to locate additional material on strength and losses from those files. About the only additional material he located was the panzer regiment files from the 11th Panzer Division, which were either not in the U.S. archives or I overlooked when I did my research. That was it. Overall, the original copying effort was pretty exhaustive.

There was one major gap for a long time. For a couple of decades, many of the original German situation maps were in the U.S., but no longer accessible. There were supposed to be copied and sent to Germany, but there was a budget issue. Meanwhile one researcher was handling them so poorly that they canceled access so as to protect them. They have finally copied them and sent the originals back to Germany.

New WWII German Maps At The National Archives

There are also no real Luftwaffe files. Most of the Luftwaffe files were placed on a train and when the order came down from Hitler to destroy everything….these weenies actually obeyed the order and burned all their records. There are also major gaps in the German records after July 1944. Every six months, the German army units wrapped up their records and sent them back to their central archives. Because the war ended in May 1945, many of the records for July-December 1944 never made it back to be filed. Same for the 1945 records. This is why the QJM (Quantified Judgment Model) was originally developed from Italian Campaign Data from 1943 through June 1944.

Anyhow, this is an extended discussion of captured records originally inspired by this post and started with the discussions below.

The Sad Story Of The Captured Iraqi DESERT STORM Documents

Captured Records: World War I

Survey of German WWI Records

 

Survey of German WWI Records

At one point, we did a survey of German records from World War I. This was for an exploratory effort to look at measuring the impact of chemical weapons in an urban environment. As World War I was one of the few wars with extensive use of chemical weapons, then it was natural to look there for operational use of chemical weapons. Specifically we were looking at the use of chemical weapons in villages and such, as there was little urban combat in World War I.

As discussed in my last post on this subject, there is a two-sided collection of records in the U.S. Archives for those German units that fought the Americans in 1918. As our customer was British, they wanted to work with British units. They conducted the British research, but, they needed records from the German side. Ironically, the German World War I records were destroyed by the British bombing of Potsdam in April 1945. So where to find good opposing force data for units facing the British during World War I?

Germany did not form into a nation until 1871. During World War I, there were still several independent states, and duchies inside of the Germany and some of these maintained their own armies. The kingdoms of Bavaria, Wurttemberg and Saxony, along with the Grand Duchy of Baden fielded their own armies. They raised their own units and maintained their own records. So, if they maintained their records from World War I then we could developed a two-sided database of combat between the British and Germans in those cases where the British units opposed German units from those states.

So….for practical purposes, we ended up making a “research trip” to Freiburg (German archives), Stuttgart (Wurttemberg) and then Munich (Bavaria). Sure enough, Wurttemberg had an nice collection of records for its units (a total of seven divisions during the war) and Bavaria still had a complete collection of records for its many divisions. The Bavarian Army fielded over a dozen divisions during the war.

So we ended up in Munich for several days going through their records. Their archives were located near Munich’s Olympic Park, the place of the tragic 1972 Olympics. It was in the old Bavarian Army headquarters that had been converted to an archives. After World War II, it was occupied by the Americans, and on the doors of many of the offices was still the name tags of the American NCOs and officers who last occupied those offices. The records were in great shape. The German Army just before WWII had done a complete inventory of the Bavarian records and made sure that there were complete. It was clear that when we looked into that, that many of these files had not been opened since then. Many of the files had sixty years of dust on them. The exception was the Sixth Bavarian Reserve Division, which clearly had been accessed several times recently. Adolf Hitler had served in that division in WWI.

The staff was extremely helpful. I did bring them gifts of candy for their efforts. They were  neatly wrapped in the box with plastic mice attached to the packaging. Later, they sent me this:

So we were able to establish that good German data could be assembled for those Wurttemberg and Bavarian units that the face the British. The British company that hired us determine that the British records were good for their research efforts. So the exploratory research effort was a success, but main effort was never funded because of changing priorities among their sponsors. This research was occurring while the Iraq War (2003-2011) was going on, so sometimes budget priorities would change rather suddenly.

The Iran-Iraq War (1980-1988) also made extensive used of chemical weapons. This is discussed in depth in our newsletters. See: http://www.dupuyinstitute.org/tdipub4.htm  (issues Vol 2, No. 3; Vol 2, No. 4, Vol. 3, No 1, and Vol 3, No 2). Specifically see: http://www.dupuyinstitute.org/pdf/Issue11.pdf, page 21. To date, I am not aware of any significant work done on chemical warfare based upon their records of the war.

This post is the follow-up to these two posts:

Captured Records: World War I

The Sad Story Of The Captured Iraqi DESERT STORM Documents

Captured Records: World War I

When Shawn Woodford sent me that article on the Captured Iraqi Records, it reminded me of a half-dozen examples I had dealt with over the years. This will keep me blogging for a week or more. Let me start a hundred years ago, with World War I.

The United States signed a separate treaty with Germany after the end of World War I. We were not part of the much maligned Versailles Treaty (although we had to come back to Europe to help clean up the mess they made of the peace). As part of that agreement, we required access to the German military archives.

We used this access well. We put together a research team that included Lt. Colonel Walter Kruger. The United States plopped this team of researchers in Germany for a while and carefully copied all the records of German units that the United States faced. This really covered only the last year of a war that lasted over four years (28 July 1914 – 11 November 1918). Krueger was in Germany with the team in 1922. This was a pretty significant effort back in the days before Xerox machines, microfilm, scanners, and other such tools.

These records were later saved to the U.S. National Archives. So one can access those records now, and will find the records of U.S. units involved, the German units involved (much of it translated), along with maps and descriptions of the fighting. It is all nicely assembled for researchers. It is a very meticulous and nicely done collection.

Just to add to the importance of this record collection, the German archives in Potsdam were bombed by the RAF in April 1945, destroying most of their World War I records (this raid on 14/15 April: apr45 and also: bundesarchiv). So, one cannot now go back and look up these German records. The only surviving primary source combat records for many German units from World War I is the records in the U.S. Archives copied (translated and typed from the original) done by the U.S. researchers.

Lt. Colonel Krueger was fluent in German because of his family background (he was born in Prussia). During World War II, he rose to be a full general (four-star general) in command of the Sixth Army in the Pacific: https://en.wikipedia.org/wiki/Walter_Krueger

This effort sets that standard almost a hundred years ago of what could/should be done with captured records. A later post will discuss the World War II effort.

The Sad Story Of The Captured Iraqi DESERT STORM Documents

 

The Sad Story Of The Captured Iraqi DESERT STORM Documents

The fundamental building blocks of history are primary sources, i.e artifacts, documents, diaries and memoirs, manuscripts, or other contemporaneous sources of information. It has been the availability and accessibility of primary source documentation that allowed Trevor Dupuy and The Dupuy Institute to build the large historical combat databases that much of their analyses have drawn upon. It took uncounted man-hours of time-consuming, pain-staking research to collect and assemble two-sided data sufficiently detailed to analyze the complex phenomena of combat.

Going back to the Civil War, the United States has done a commendable job collecting and organizing captured military documentation and making that material available for historians, scholars, and professional military educators. TDI has made extensive use of captured German documentation from World War I and World War II held by the U.S. National Archives in its research, for example.

Unfortunately, that dedication faltered when it came to preserving documentation recovered from the battlefield during the 1990-1991 Gulf War. As related by Douglas Cox, an attorney and Law Library Professor at the City University of New York School of Law, millions of pages of Iraqi military paper documents collected during Operation DESERT STORM were destroyed by the Defense Intelligence Agency (DIA) in 2002 after they were contaminated by mold.

As described by the National Archives,

The documents date from 1978 up until Operation Desert Storm (1991). The collection includes Iraq operations plans and orders; maps and overlays; unit rosters (including photographs); manuals covering tactics, camouflage, equipment, and doctrine; equipment maintenance logs; ammunition inventories; unit punishment records; unit pay and leave records; handling of prisoners of war; detainee lists; lists of captured vehicles; and other military records. The collection also includes some manuals of foreign, non-Iraqi weapons systems. Some of Saddam Hussein’s Revolutionary Command Council records are in the captured material.

According to Cox, DIA began making digital copies of the documents shortly after the Gulf War ended. After the State Department requested copies, DIA subsequently determined that only 60% of the digital tapes the original scans had been stored on could be read. It was during an effort to rescan the lost 40% of the documents that it was discovered that the entire paper collection had been contaminated by mold.

DIA created a library of the scanned documents stored on 43 compact discs, which remain classified. It is not clear if DIA still has all of the CDs; none had been transferred to the National Archives as of 2012. A set of 725,000 declassifed pages was made available for a research effort at Harvard in 2000. That effort ended, however, and the declassified collection was sent to the Hoover Institution at Stanford University. The collection is closed to researchers, although Hoover has indicated it hopes to make it publicly available sometime in the future.

While the failure to preserve the original paper documents is bad enough, the possibility that any or all of the DIA’s digital collection might be permanently lost would constitute a grievous and baffling blunder. It also makes little sense for this collection to remain classified a quarter of a century after end of the Gulf War. Yet, it appears that failures to adequately collect and preserve U.S. military documents and records is becoming more common in the Information Age.

Cost of Creating a Data Base

Invariably, especially with a new book coming out (War by Numbers), I expected to get requests for copies of our data bases. In fact, I already have.

Back around 1987 or so, a very wise man (Curt Johnson, VP of HERO) estimated that for the LWDB (Land Warfare Data Base) that it took 3 man-days to create an engagement. The LWDB was the basis for creating many of our later data bases, including the DLEDB (Division Level Engagement Data Base). My experience over time is that this estimate is low, especially if your are working with primary sources (unit records) for both sides. I think it may average more like 6 man-days an engagement if based upon unit records (this includes the time to conduct research).

But going with Curt’s estimate, let’s take the DLEDB of 752 cases and re-create it. This would take 3 man-days times 752 engagements = 2,256 man-days. This is 9 man-years of effort. Now 9 man-years times a loaded professional rate. A loaded man-year is the cost of a person’s labor times indirect costs (vacation, SS and Medicare contributions, health insurance, illness, office space, etc.), general and administrative costs (corporate expenses not included in the indirect costs, including senior management and marketing), and any fee or profit. Loaded rate is invariably at least 60% of the direct costs and usually closer to 100% of direct costs (and I worked at one company where it was 200% of direct costs). So a loaded man-year may be as low at $120,000 a year but for people like RAND or CNA, it is certainly much higher. Nine man-years times $120,000 = $1,080,000.

Would it really cost more than a million dollars to re-created the DLEDB? If one started from scratch, certainly. Probably (much) more, because of all the research into the Ardennes and Kursk that we did as part of those database projects. The data bases were created incrementally over the course of more than 30 years as part of various on-going contracts and efforts. We also had a core group of very experienced personnel who were doing this.

Needless to say, if any part of the data base is given away, loaned out, or otherwise not protected, we loose control of the “proprietary” aspect of these data bases. This includes the programming and formatting. Right now, they are unique to The Dupuy Institute, and for obvious business reasons, need to remain so unless proper compensation is arranged.

Sorry.

 

P.S. The image used is from the old Dbase IV version of the Kursk Data Base. We have re-programmed it in Access.

 

Dupuy Institute Data Bases

Yes, I still use data base as two words, much to the annoyance of Jay Karamales.

Anyhow, War by Numbers does rely extensively on a group of combat data bases that were developed over several decades. The earliest versions were developed in the 1970s and they were assembled into a large data base of around 600 cases in the 1980s. They were then computerized (they were originally a paper data base), re-organized, re-programed in Access, and greatly expanded. The data bases we currently have include:

Conventional Combat Data Bases:

LADB = Large Action Data Bases of 55 cases

DLEDB = Division Level Engagement Data Base of 752 cases

BLODB = Battalion Level Operations Data Base of 127 cases

CLEDB = Company Level Engagement Data Base of 98 cases

SADB = Small Action Data Base of 5 cases

BaDB = Battles Data Base of 243 cases from 1600-1900

 

We also have:

CaDB = Campaign Data Base of 196 cases. While the other data bases address battles, or engagements of no more than a few days in length, this one summarizes campaigns, often extending for months.

Finally we have three databases tracking campaigns from day-to-day. They are all programmed in Access:

ACSDB = Ardennes Campaign Simulation Data Base (meaning Battle of the Bulge)

KDB = Kursk Data Base

Battle of Britain Data Base

These were primarily intended for model validation efforts.

We also have three insurgency/peaceeping/intervention/OOTW (Operations Other than War) data bases. They are:

WACCO = Warfare and Armed Conflict Data Base of 793 cases

SSCO = Small Scale Operations Data Base of 203 cases

DISS = Dupuy Insurgency Spread Sheets of 109 cases.

 

The DISS data base was the one that America’s Modern Wars is based upon. The other two were earlier efforts.

These links provides some snap shots of the data base content: http://www.dupuyinstitute.org/dbases.htm

These are all company proprietary, although some have been released publicly in earlier forms or different forms (including the CHASE data base of 599 cases, the ACSDB in Dbase III and the KDB in Dbase IV). Our versions have been updated, including revisions to content.

Aussie OR

Over the years I have run across a number of Australian Operations Research and Historical Analysis efforts. Overall, I have been impressed with what I have seen. Below is one of their papers written by Nigel Perry. He is not otherwise known to me. It is dated December 2011: Applications of Historical Analyses in Combat Modeling

It does address the value of Lanchester equations in force-on-force combat models, which in my mind is already a settled argument (see: Lanchester Equations Have Been Weighed). His is the latest argument that I gather reinforces this point.

The author of this paper references the work of Robert Helmbold and Dean Hartley (see page 14). He does favorably reference the work of Trevor Dupuy but does not seem to be completely aware of the extent or full nature of it (pages 14, 16, 17, 24 and 53). He does not seem to aware that the work of Helmbold and Hartley was both built from a database that was created by Trevor Dupuy’s companies HERO & DMSI. Without Dupuy, Helmbold and Hartley would not have had data to work from.

Specifically, Helmbold was using the Chase database, which was programmed by the government from the original paper version provided by Dupuy. I think it consisted of 597-599 battles (working from memory here). It also included a number of coding errors when they programmed it and did not include the battle narratives. Hartley had Oakridge National Laboratories purchase a computerized copy from Dupuy of what was now called the Land Warfare Data Base (LWDB). It consisted of 603 or 605 engagements (and did not have the coding errors but still did not include the narratives). As such, they both worked from almost the same databases.

Dr. Perrty does take a copy of Hartley’s  database and expands it to create more engagements. He says he expanded it from 750 battles (except the database we sold to Harley had 603 or 605 cases) to around 1600. It was estimated in the 1980s by Curt Johnson (Director and VP of HERO) to take three man-days to create a battle. If this estimate is valid (actually I think it is low), then to get to 1600 engagements the Australian researchers either invested something like 10 man-years of research, or relied heavily on secondary sources without any systematic research, or only partly developed each engagement (for example, only who won and lost). I suspect the latter.

Dr. Perry shows on page 25:

Data-segment……..Start…….End……Number of……Attacker…….Defender

Epoch…………………Year…….Year……..Battles………Victories……Victories

Ancient………………- 490…….1598………….63………………36……………..27

17th Century……….1600…….1692………….93………………67……………..26

18th Century……….1700…….1798………..147…………….100……………..47

Revolution…………..1792……1800…………238…………….168…………….70

Empire……………….1805……1815…………327……………..203…………..124

ACW………………….1861……1865…………143……………….75…………….68

19th Century……….1803…….1905…………126……………….81…………….45

WWI………………….1914…….1918…………129……………….83…………….46

WWII…………………1920…….1945…………233……………..165…………….68

Korea………………..1950…….1950…………..20……………….20………………0

Post WWII………….1950……..2008…………118……………….86…………….32

 

We, of course, did something very similar. We took the Land Warfare Data Base (the 605 engagement version), expanded in considerably with WWII and post-WWII data, proofed and revised a number of engagements using more primarily source data, divided it into levels of combat (army-level, division-level, battalion-level, company-level) and conducted analysis with the 1280 or so engagements we had. This was a much more powerful and better organized tool. We also looked at winner and loser, but used the 605 engagement version (as we did the analysis in 1996). An example of this, from pages 16 and 17 of my manuscript for War by Numbers shows:

Attacker Won:

 

                        Force Ratio                Force Ratio    Percent Attack Wins:

                        Greater than or         less than          Force Ratio Greater Than

                        equal to 1-to-1            1-to1                or equal to 1-to-1

1600-1699        16                              18                         47%

1700-1799        25                              16                         61%

1800-1899        47                              17                         73%

1900-1920        69                              13                         84%

1937-1945      104                                8                         93%

1967-1973        17                              17                         50%

Total               278                              89                         76%

 

Defender Won:

 

                        Force Ratio                Force Ratio    Percent Defense Wins:

                        Greater than or         less than          Force Ratio Greater Than

                        equal to 1-to-1            1-to1                or equal to 1-to-1

1600-1699           7                                6                       54%

1700-1799         11                              13                       46%

1800-1899         38                              20                       66%

1900-1920         30                              13                       70%

1937-1945         33                              10                       77%

1967-1973         11                                5                       69%

Total                130                              67                       66%

 

Anyhow, from there (pages 26-59) the report heads into an extended discussion of the analysis done by Helmbold and Hartley (which I am not that enamored with). My book heads in a different direction: War by Numbers III (Table of Contents)

 

 

Economics of Warfare 15-1

Moving onto the fifteenth lecture from Professor Michael Spagat’s Economics of Warfare course that he gives at Royal Holloway University. It is posted on his blog Wars, Numbers and Human Losses at: https://mikespagat.wordpress.com/

This lecture addresses the impact of global warming on armed conflict over time. If you happen to be one of those who believes that global warming is pseudo-science/a scientific hoax/an excuse to troll for research $$$/an invention of Al Gore/a liberal plot/a religion/a Chinese plot/obviously false because it was cold yesterday/and so forth…..then probably best to stop reading. On the other hand, we have done some proposals on measuring the impact of climate change on violence and consider this a legitimate area of study. Our attention was drawn to the subject over a decade ago when a CNA (Center for Naval Analysis) paper came out that postulated that global warming could result in more violence. This conclusion does not appear to have been based upon any analysis of data, just the assumption that as things get worse (in the environment) then things are going to get worse (with armed conflict). Of course, going back to Feierabend & Feierabend (and I do go back to them a lot)….poorer counties had less political violence than developing countries. Therefore, it does not necessarily follow that worse environmental and economics conditions results in more violence. The effect may be the reverse, which is that declining conditions may actually result in a reduction of violence. We really don’t know. Trying to examine these effects analytically was the gist of my proposals on the subject, but sequestration happened and budget for anything seemed to disappear.

So….first two sentences of Dr. Spagat’s slides are

“There is a strong scientific consensus that the Earth is getting warmer over time.”

“It is reasonable to imagine that a side effect of global warming could be an increase in armed conflict over time.”

Slide 2 looks at possible channels that could lead to conflict

  1. Dwindling food supply
  2. Dwindling water supply
  3. Sea Level changes causing migration.

On slide 3 he then addresses a study by Burke and others that attempt to address these concerns using a cross-country regression approach and linear probability model.

On slide 5 the results are summarized as “…an increase of 1 degree centigrade for a  particularly country in a particular year is associated with a 0.0447 increase in the probability of there being an ongoing civil war….”

and on slide 11 as: “This means that Burke et al. predict that 15.8-17.1% of the countries in Sub-Saharan Africa in 2030 will suffer from big civil wars rather than the 11% that would occur without the warming climate.”

and on slide 16 as: “Burke et al. go on to predict 393,000 excess battle deaths caused by climate change…” (my bolding). Dr. Spagat then examines this number in the next two slides. It doesn’t sound like he fully accepts it.

Now, Burke based his study on the period from 1981-2002. One of Dr. Spagat’s TAs then used the model to make predictions from the period 2003-2013. There is nothing like trying to use a model to predict the past. It sort of shows whether it really works or not. This was the reasoning because the Ardennes Campaign Simulation Data Base and the Kursk Data Base that we did (they were model validation data bases). It is related in concept to what I did in America’s Modern War, pages 65-68 when I tested my logistics probability model back to the 68 cases used to create the model and tried to figure out for each case why the model was predicting wrong. Once you have a model, there are lots of things to test it to in the past. If you can’t predict the past, you may not be able to predict the future.

Anyhow, the results are on slide 19 and summarized in slide 20 as

  1. “There are 414 “no war” predictions…A war actually happens in 11 out of these 414 cases.”
  2. “There are 37 predictions of “war”. War actually happens in 7 out of these 37 cases.”

Not sure I am any smarter at this point, but I am certainly amused.

His final point is “The Burke et al. model seems to be of some use in predicting wars although it seems have a general tendency to predict war too often.”

And then Dr. Spagat TA test how important the temperature variable is for making these predictions, so takes the temperature variable out of the model !!! This produces a table (slide 21) that is almost identical to his original table. The impact of removing the temperature from the model is that it produced five more false positives (predicted wars that did not happen). I am even more amused.

Spagat’s conclusion (slide 22) is “…They mean that temperature is not very useful for predicting civil war….”

This is a good point to stop…I will pick up the rest of this lecture in another post. The link to the lecture is here: http://personal.rhul.ac.uk/uhte/014/Economics%20of%20Warfare/Lecture%2015.pdf

War by Numbers is on Amazon

lawrencefinal

My new book, with a release date of 1 August, is now available on Amazon.com for pre-order: War by Numbers (Amazon)

It is still listed at 498 pages, and I am pretty sure I only wrote 342. I will receive the proofs next month for review, so will have a chance to see how they got there. My Kursk book was over 2,500 pages in Microsoft Word, and we got it down to a mere 1,662 pages in print form. Not sure how this one is heading the other way.

Unlike the Kursk book, there will be a kindle version.

It is already available for pre-order from University of Nebraska Press here: War by Numbers (US)

It is available for pre-order in the UK through Casemate: War by Numbers (UK)

The table of contents for the book is here: War by Numbers: Table of Contents

Like the cover? I did not have a lot to do with it.