Traffic demand forecasting units, as many planners know, can be the target of intense scrutiny when forecasts are significantly different from observed traffic trends. Although these forecasts only gain attention in a few high profile cases, their questionable outputs may be more of the rule than an exception to it. Bent Flyvberg, Professor of Planning at Aalborg University in Denmark, attests that explanations for these forecast inaccuracies can be explained by the following three phenomena.
- Technical capacity, characterized by inadequate data or models
- Psychological constructs such as optimism bias
- Political-economic factors such as strategic misrepresentation (also known as lying)
Flyvberg tends to emphasize the latter two explanations as the root of most problems. He claims that technical explanations can be statistically disproven, meaning data is not usually the issue.¹ However, I would contend, albeit from limited experience in a state-level forecasting unit, that limited technical capacity is a major hindrance to many forecasts. I would also agree with Flyvberg that reference class forecasting (RCF) may be a potential cure.
At the time of my arrival at this particular state department, the forecasting unit was in legal hot water over questionable forecasts. An advocacy organization had evaluated various large project traffic forecasts and compared them with historical and present traffic trends. In their findings, forecasts were 73% higher on average than the observed trends on those routes.
With limited knowledge of the situation and the validity of the accusations, I was able to identify a few potential sources of faulty forecasts. The traffic forecasting process is a complex one and it takes a very knowledgeable task force to maintain sound methods. In the midst of the controversy, this particular unit was comprised of two employees who maintained the models, two or three that handled various inputs to the models, and one or two that regularly researched alternative methods. These responsibilities were not mutually exclusive and all employees were responsible for carrying out smaller forecasts in different geographic regions of the state. There was no single person that fully understood every detail of the model, as is presumably the case in many other states. Employee turnover in the unit regularly leaves the group with a mess of files and individual procedures to sort through and make sense of. It was commonplace to hear of people trying to track down former employees at different firms or in different units who had maintained certain model inputs at one time or another. Because of the complexity, everyone was absorbed in the details of their specific tasks, leaving little time to coordinate efforts.
Some of my work revealed that the forecasting methods were leading to trends very different from the observed counts of two other datasets, but these patterns were by no means a revelation to the employees on the unit. Within the current system, it was unclear how to better calibrate the forecasts and integrate better data, but it was seen as necessary. It seemed that the unit needed a statistician, an IT specialist, a planner, and at least one person with exceptional organization skills to get back on track.
In addressing problems such as these, RCFs can be very useful. RCFs first require the identification of a reference class of past projects that are comparable to the one in question. Then, one must establish a probability distribution showing how the traffic outcomes of these projects are scattered. The last step is to simply compare your project to the RCF distribution and see where it ranks. This gives you a statistical representation of how similar projects have performed in the past and where you can reasonably expect your project to perform. This data may also hint at reasons for differences between the projects. For instance, one may have overestimated the intensity of future land use, while another may have been using a linear regression where a curvilinear regression was a better fit. In principle, RCFs force the analysts to step back from their limited inside view of the forecast, and consider perspectives and data from the outside. Not only would it address these more obvious limitations, it would help control for any underlying optimism bias and/or strategic misrepresentation. By becoming a standard at the highest levels of forecasting, RCFs that can change the culture of competition for funding and better ground the process in rationality. Reference class forecasting essentially reinforces the role of planners and the benefits of a broader approach in a traditionally compartmentalized process.
In order to maximize the effectiveness of forecasting changes, increased collaboration between states and improved independent peer review efforts are needed. Although the variation in forecasting methods between states has been identified and the rationale behind the differences has been explained, little has been done in developing a set of best practices. Through peer review of forecasters by forecasters, a wealth of knowledge can be shared, bringing fresh perspective to a discipline that is not particularly responsive to change.
Much of what has been discussed thus far is not possible without adequate funding mechanisms. Removing the earmark requirements on federal transportation funding is an important step.² Although a forecasting unit may have a clear vision of how to improve efficiency, the time and capacity to do so often does not fit into the scope of day-to-day activities. Federal funding is often earmarked for a narrow range of project categories. If more funding operated with the flexibility of, say, the Community Development Block Grant (CDBG) program, technical improvements could become a less daunting prospect. This would provide more opportunity and leeway for agencies to experiment with more innovative forecasting methods without sacrificing daily responsibilities. It could allow enough extra capacity to support activities as simple as cross-departmental cooperation or foster other partnerships that often seem infeasible within the status quo.
Even though RCFs seem like a fundamental change to traffic forecasting, they can really be boiled down to an improved method of calibration. They can bolster the planner’s ability to think beyond the spreadsheets and consider the qualitative factors that are difficult to account for within the current system. Additionally, they can create a workplace environment and mindset that prove useful outside of forecasting. RCFs may seem like more effort than they are worth, but when lawsuits ensue and methods must be justified, it is best to be on the right side of progress.
¹Flyvberg, Bent. “Follies of Infrastructure: Why the Worst Projects Get Built, and How to Avoid It.” Lecture, Harvard University, Cambridge, August 8, 2013.
²Flyvberg, Bent, Mette Skamris Holm, and Soren Buhl. “How (In)Accurate Are Demand Forecasts in Public Works Projected.” Journal of the American Planning Association, 2005.
mobility, urbanity, ecology 