My last piece reviewed several recent criticisms of the Social Security trustees' projection history by Konstantin Kashin, Gary King and Samir Soneji. This piece reviews other criticisms by the same authors, this time focusing on process and presentation. As I indicated in my last piece, their factual observations regarding the projections strike me as essentially correct, though I find myself in disagreement with many of their interpretative conclusions. The following summarizes my perspectives on some of their process criticisms, also restated in my own words.

Criticism: The reports are insufficiently transparent about methodologies and prior projection errors. This undercuts the ability of outside experts to replicate, analyze and offer improvements to the projection methods, as well as the trustees' ability to learn from past mistakes.

Transparency is a longstanding concern of the trustees, and we expend significant time and effort to increase it. The SSA actuary's office now posts a wealth of methodological specifications online going a great deal of the way toward enabling others to replicate their projection methods. The trustees' report itself contains ample sections detailing how projections have been revised over the last year in light of updated information. It also contains a long-running table showing the history of prior actuarial balance estimates. Another table compares actual prior-year operations (both income and expenditures) within each separate trust fund, to projections in the five previous trustees' reports. And there's much more. (Note to readers: to get some of these links to work, the hyperlink address may need to be copied and pasted directly into a browser address window).

Considerable discussion occurs each year between the trustees' offices about how to best explain deviations from prior projections (still more time is likely spent on this for the Medicare report, where the methodological issues are more complex). As one example of a trustee initiative to expand such information, a footnote was recently added to the projection history table, directing readers to an online actuarial note breaking down the changes by source.

The trustees constantly face a trade-off between thoroughness and clarity. As it is, the trustees' report is 250 pages long and its density challenges the most dedicated of readers. In this context the pursuit of transparency is always ongoing. More can be done to explain past forecasting errors accessibly and clearly. A persistent challenge, especially when writing by committee, is to do it without making the reports still longer, more redundant or inaccessible to non-specialists.

Criticism: The reports do not describe sources and magnitudes of projection uncertainty appropriately.

Explaining projection uncertainty is also a work in progress; there is a long way to go but huge strides have been made. In 2003 the trustees added a stochastic (probabilistic) analysis to the report quantifying uncertainty ranges surrounding the projections. The uncertainty analysis and presentation are no doubt imperfect, but I respectfully disagree with some of Kashin et al's characterizations of it. Specifically, they write:

"The Trustees describe the uncertainty intervals in qualitative terms that one would typically use to discuss something stronger than a 95 percent confidence level. For example, in the 2011, 2012, and 2013 Trustees Reports, the report repeated the same definition: 'In the future, the costs of OASI, DI, and the combined OASDI programs as a percentage of taxable payroll are unlikely to fall outside the range encompassed by alternatives I [low cost] and III [high cost] because alternatives I and III define a wide range of demographic and economic conditions.'"

Kashin et al continue:

"The post-2000 forecasts all indicated that both men and women would have lived shorter lives than they did and also offered uncertainty ranges implying that the Trust Funds were on firmer financial ground than turned out to be warranted."

There are some problems with these criticisms. First, the quoted trustees' statement refers to comprehensive hypothetical scenarios involving many variables, whereas the Kashin et al finding about exceeding uncertainty ranges only pertains to one specific variable (life expectancy). It is far more likely that one variable will fall outside the uncertainty ranges than that every variable will. Thus even with the Great Recession being unforeseen, reality has still generally arrived between the trustees' previous low-cost and high-cost scenarios.

Second, the trustees were not asserting the illustrative scenarios fell outside the 95% confidence band; instead, figures in those reports show the alternative scenarios within it. And third, far from expressing excessive confidence in the numbers, the trustees opined that uncertainty is likely greater than shown:

"The future will bring with it the likelihood of substantial shifts, as predicted by many experts and as seen in prior centuries, that are not fully reflected in the current model. As a result, readers should understand that the true range of uncertainty is larger than indicated in this appendix."

Fourth, the trustees would be justified in expressing increasing confidence in aspects of the projections as time passes. The closer we get to a trust fund's projected depletion date, the more certain it becomes. For example, in the 2003 trustees' report, the 80% confidence band on the OASDI depletion date ran from 2034-2057. The range has since narrowed to 2029-2038.

Perhaps most importantly, however, the trustees' language about projection uncertainty is not fairly interpreted as conveying a false sense of security, but quite the opposite. The language was written when some public commentary (see Paul Krugman and Robert Reich) had been wrongly suggesting there was a significant chance the Social Security shortfall might never materialize.

The basis of that confusion was the presence of the hypothetical scenario in the trustees' report in which the trust fund never depleted. That scenario was purely illustrative; it was not des igned to be realistic. It assumed that US fertility permanently rebounded to high levels not seen in any single year during the last forty-five. It also assumed real wage growth at nearly double its rate over the last five business cycles, among other similarly aggressive assumptions.

The language cited by Kashin et al was included to reinforce the point that despite such hypothetical illustrations, corrective legislation would almost certainly be needed. Other trustee language accentuates this point:

"The results of the stochastic simulations, discussed in more detail in appendix E, suggest that trust fund reserve depletion (i.e., the point at which the trust fund ratio reaches zero) is very likely by mid-century."

Criticism: The process is poorly constructed to identify and eliminate sources of bias. Consistency bias is but one form for which there is strong evidence. Increased transparency and openness to the views of outside technical panels would likely reduce projection errors.

There are undoubtedly sources of bias in the trustees' (or any) projections, and one is a bias toward projection consistency with some cost to accuracy. That is to say, a particular assumption is much more likely to be made in the annual trustees' report if it was made in the previous year's report.

Such consistency bias likely has several sources. For one, it is a natural human tendency -- we're all more likely to believe something today if we believed it before. Second, it is an outgrowth of having multiple trustees: they always bring a range of views to the table between them, and must work to turn these distinct views into a common presentation. Where a new agreement on a particular issue cannot be forged in time, it sometimes happens that the trustees agree to continue for the time being with the last agreement reached.

A third source of consistency is self-restraint. Few political appointees associated with the trustees' process want to be seen as pursuing a policy agenda or furthering an ideological viewpoint. At the same time, career staff involved with preparing the reports similarly do not want to be seen as yielding to such. Thus, in instances of disagreement I have observed political staff tending to defer to career staff rather than the other way around. This tendency, coupled with the fact that the reports are processed primarily by career staff with a tenure long exceeding those of the trustees themselves, tends to reinforce consistency from report to report.

The public trustee positions are not full-time jobs, nor are we given the staff and resources to generate new reports from scratch. Nor do the ex officio trustees' offices produce the documents. The vast majority of the production work falls on the shoulders of the SSA actuary's office. The hardworking staff there strive to do the most unbiased work they can, but it's virtually inevitable that there would be a tendency toward consistency in work centralized within any single office.

Conclusion: The trustees' process is undoubtedly imperfect, and it's undeniably true that the trustees failed to project the Great Recession. My biggest interpretative difference with the Kashin et al papers is with the suggestion that the process imperfections are behind recent projection errors. The totality of the evidence does not seem to support that. For example, see this graph showing how three key entities have estimated Social Security's actuarial imbalance in recent years.

Everyone seems to acknowledge that S ocial Security's projected shortfall has grown worse since the Great Recession (see the general trend on this graph showing larger imbalances in later reports). But so far the trustees' projection record doesn't look worse than its counterparts, it actually looks somewhat better. Irrespective of whether the trustees' or CBO's current picture of program finances is more correct, the trustees were much closer to it throughout the last decade.

Perhaps various process reforms can further improve the accuracy of trustees' Social Security projections going forward. To date, however, it's difficult to directly connect process imperfections to bad results. To the contrary, the trustees' projections have been an indispensable public resource, remaining at least as reliable as any other available.

Charles Blahous is a senior research fellow for the Mercatus Center, a research fellow for the Hoover Institution, a public trustee for Social Security and Medicare, and a contributor to e21.