The December 1994 issue of Proceedings contains an article by Lieutenant Colonel Mark Cancian entitled “Are You Telling the Truth?” In it he addresses the question, “How optimistic can an estimate be before it is, in effect, a lie?” Program managers are the primary target. Having been the recipient of one of Senator Proxmire’s Golden Fleece Awards— and having wrestled with the fallout from this problem for four years as program manager of the Deep Submergence Systems Program (DSSP) (1966-1971)—I believe some comments may be of interest to those who will be facing this problem, which will not go away.
Colonel Cancian states that “using the low estimate constituted a lie,” but in the course of a truly new development, there are many estimates—and they are used for different purposes. Whether any given estimate is a “lie” depends on how, when, and by whom it is used.
In the case of the nuclear-powered research submarine NR-I, the original “estimate” was $30 million. This number was given to Congress by Admiral Hyman Rickover based only on the availability of that sum in the Polaris Management Fund and hence his request for approval— with no need to go back to Congress for more appropriations.
Congress approved this project without the usual Navy program, and the Chief of Naval Operations found out about it in the press. It fell to my lot to go back and seek additional money; the final bill (identified by audit) was $96 million. The original estimate was simply a guess based on the availability of money and Admiral Rickover’s desire to build the submarine.
In the case of the deep-submergence rescue vehicles (DSRVs), the trail of estimates was longer—beginning with the suggestion by the Deep Submergence Review Group that a submarine- rescue vehicle could be developed from Alvin (the then most successful deep-diving research submersible) and using the cost of that vehicle as a basis for discussion. They estimated $2-$3 million for a new vehicle—detailed characteristics to be developed. This was a “starting” or “discussion” number used by the group looking at needs in light of recent submarine disasters. It did not come from an engineering estimate. It was, however, used by Senator Proxmire via the Golden Fleece Award to criticize Department of Defense management.
By the time the program was approved and preliminary engineering studies were complete, the estimated cost of the vehicle had grown to about $12 million—a reasonable starting point comparable to a commercial development program approval by a chief executive officer or board of directors. The final cost per vehicle was closer to $36 million (a difficult number to extract because the program included many other facets, but a reasonable one for this comparison).
More or less concurrent with the Deep Submergence Systems Program were five civilian programs that were developing their own deep submersibles. The deep ocean was being touted as the next frontier, and the race to capitalize on this led to real competition in the development of the necessary technology. Each concern was developing a vehicle independently, to its own plans and specifications, with no Department of Defense input aside from the normal exchange of technical information available to all.
Because of the intense scrutiny and criticism of DoD management, I explored the problem with all of my civilian counterparts—the managers of each company-funded civilian program. Each was facing the same problems in this totally new and undeveloped field.
The question I posed was, What was the cost growth from the time you went to your board of directors and requested approval to build your submersible at X dollars until vehicle completion of trials? The average was a little more than three times the original estimate. This was consistent with our experience on both NR-I and the deep-submergence rescue vehicle. It also was consistent with a similar review of ten DoD weapon development programs, which experienced a growth factor of 3.1—somewhat lightly referred to as “Pi” by some of my contemporaries.
I would note that today, according to press reports, this cost growth is reasonably comparable to that experienced by the Denver airport, the Senate office building, the Miami airport expansion, the Super Collider, and other civil works that presumably do not suffer the problems inherent in the management of Department of Defense projects. These major civil projects are not reaching into the unknown—they are large but use proven technology—and they all have had the problem of what to believe in the development phases.
In the Deep Submergence Program (as in the space program and many emerging Defense development programs), we needed to develop and prove hardware that had never been built. Today, we have 30 years of developmental experience in these arenas to aid in estimating, but every step forward involves unknowns. Engineers may have high confidence, based on this experience, that new problems can be solved, but the dollar costs of solving these problems with little hard-data background remain very uncertain. The problem for the program managers is how to handle this uncertainty, once they have been assigned to get the job done.
Experience would seem to indicate that a factor of three would be a prudent allowance, but what executive or board would approve a plan in which the proponent starts with a 66% cost uncertainty? Thus, we find intensive pressure to go with hard, tight estimates—“best estimates” if you will—but with the expectation that as the feasibility becomes apparent, additional funds can be justified on a case-by-case basis. This essentially is why such programs usually are handled under the research-and-development umbrella, where developmental costs can be reviewed on a case-by-case basis. The entire Deep Submergence Program was so handled, with the result that there were many estimates and reviews—none of which were advanced with intent to deceive. The question nearly always was, What is the least cost to complete? The answer usually was, “Based on everything we’ve done to date, and if all goes as planned, it will be. . . .”
This problem underlies many of the widely publicized critiques, and being among the leaders in trying to develop totally new systems, the Defense Department is particularly vulnerable to such uncertainties. It is of great importance that such re search-and-development programs be evaluated with recognition of the uncertainties inherent in exploring the unknown. Such estimates are not lies; they are the best compromise possible, giving a reasonable balance between needs, knowledge, and availability of funds.
There is a vast difference between the numbers kicked around the table in early feasibility discussions and the budget number approved at the official start of any program. The latter ought to be based on the best available engineering analysis—it is the number to which the program manager must work—but it inevitably will include areas of uncertainty, and the extent of these should be recognized. Such uncertainty hardly constitutes lying if it is stated up front. We have had enough experience with this problem to alert all of the participants to reality.
The search for an acceptable set of numbers for a new development, a new production contract, or a contract extension involves at least three parties: the party in need (in our case, the Chief of Naval Operations), the party expected to do the work (the program manager or contractor), and the provider of funds (Congress). Negotiations between and among these parties are not unlike Cancian’s bazaar, and, like his example, none of our participants should feel deceived in the bargaining process.
Perhaps the factor of three should be recognized at the start—a sort of “Murphy’s Law” for new programs—to be applied to the budget whenever the task to be undertaken has no history of returned actual costs available for reliable cost estimating. This budget reservation in the out years would not relieve the program manager of cost-control responsibilities, but it would clear the air for a realistic discussion of expected problems. If the groundwork for such an approach could be laid in Congress, it would relieve the program managers of the nearly constant need to fight for more funds and might even make it possible to release reserved funds when the job can be done for less—a truly novel notion. This was the broad intent of the Polaris Management Fund; it allowed the manager to concentrate on getting the job done. It also allowed the diversion of uncommitted funds to start the NR-I's construction.
Colonel Cancian condemns the service as “being less than honest, indeed deceptive. Its estimates are systematically wrong in a way that benefits the service’s self-interest—parochial and self-serving.” I agree with his conclusion that “Truth is a matter of perspective,” but I would add that Truth, like Beauty, is in the eye of the beholder; and there are at least three players in this high-stakes game in which the program managers are the servants, not the masters.