Abstract

We study the properties of macroeconomic survey forecast response averages as the number of survey respondents grows. Such averages are “portfolios” of forecasts. We characterize the speed and pattern of the gains from diversification and their eventual decrease with portfolio size (the number of survey respondents) in both (1) the key real-world data-based environment of the U.S. Survey of Professional Forecasters (SPF), and (2) the theoretical model-based environment of equicorrelated forecast errors. We proceed by proposing and comparing various direct and model-based “crowd size signature plots,” which summarize the forecasting performance of k-average forecasts as a function of k, where k is the number of forecasts in the average. We then estimate the equicorrelation model for growth and inflation forecast errors by choosing model parameters to minimize the divergence between direct and model-based signature plots. The results indicate near-perfect equicorrelation model fit for both growth and inflation, which we explicate by showing analytically that, under conditions, the direct and fitted equicorrelation model-based signature plots are identical at a particular model parameter configuration, which we characterize. We find that the gains from diversification are greater for inflation forecasts than for growth forecasts, but that both gains nevertheless decrease quite quickly, so that fewer SPF respondents than currently used may be adequate.

This paper studies how forecast accuracy changes as more economists’ predictions are combined. In particular, it examines macroeconomic survey forecasts—like predictions about economic growth and inflation—from the U.S. Survey of Professional Forecasters (SPF). The authors treat the collection of forecasts like a “portfolio” (similar to a mix of assets in finance) and ask: How much do we gain by combining many different forecasts, and at what point do extra forecasts stop improving the overall prediction?

1. Abstract Overview

Focus:
The study looks at survey forecasts as “portfolios” where each individual forecast is like an asset. The authors explore how the mean squared error (MSE)—a measure of forecast inaccuracy—changes as the number of forecasts in an average increases.
Method:
They introduce “crowd size signature plots,” which are graphs showing how performance (in terms of MSE) changes as more forecasts are averaged together. Both direct (data-driven) and model-based (theoretical) signature plots are developed.
Key Findings:
- The benefits of averaging many forecasts increase quickly at first but then taper off.
- Inflation forecasts seem to benefit more from averaging than growth forecasts.
- An equicorrelation model (a model where every pair of forecast errors has the same correlation) fits the data very well for both types of forecasts.
- These results suggest that using fewer respondents in the survey might be nearly as effective as using the full panel.

2. Introduction and Basic Framework

The Wisdom of Crowds Concept:
Traditionally, “the wisdom of crowds” suggests that averaging the judgments of many individuals can lead to a very accurate result. In economics and finance, this idea is used to combine different experts’ forecasts.
Survey of Professional Forecasters (SPF):
The SPF has been gathering forecasts on key economic variables (such as economic growth and inflation) since 1968. These forecasts are crucial for both academic research and real-time decision-making in policy and business.
Research Questions:
The paper focuses on how forecast error decreases when more forecasts are combined. Specific questions include:
- How fast does accuracy improve with each additional forecast?
- Does the benefit of adding more forecasts vary between different economic variables (e.g., growth versus inflation)?
- What does this mean for designing surveys—could a smaller panel be just as effective?
Crowd Size Signature Plots:
To answer these questions, the authors develop several types of “signature plots”:
- MSE Signature Plots: These show the average error (MSE) as a function of the number of forecasts included.
- Difference Plots (DMSE): These show how much the error decreases when adding one more forecast.
- Ratio Plots (Rbavg): These compare the performance of an averaged forecast to that of a single forecast.
- Distributional Plots (fbavg): These illustrate how the entire distribution of forecast errors changes with averaging.
Technical Framework:
The forecasts are treated as random errors with zero average (meaning there is no systematic bias in any direction). The analysis uses concepts from probability (like the central limit theorem) and portfolio theory (diversification reduces risk).

Example:
Imagine you ask 40 weather forecasters about tomorrow’s temperature. Early on, adding a few more opinions greatly improves the accuracy because individual mistakes cancel each other out. However, after a certain point, adding more opinions gives diminishing improvements. The paper shows similar behavior for economic forecasts.

3. Methodology and Model-Based Analysis

Direct Empirical Analysis

Using SPF Data:
The authors analyze decades of forecast data from the SPF. They plot how the average error (MSE) changes as the number of combined forecasts increases.
- They note that short-term forecasts (for example, predictions about the very near future) are generally more accurate.
- The improvement in accuracy is rapid when moving from one forecast to around five forecasts. Beyond that, the improvement levels off.
Observations on Errors:
The paper also discusses patterns in forecast errors:
- Economic downturns: During recessions, forecasts for growth tend to be too optimistic (over-predicted), while during recoveries, they might be under-predicted.
- Inflation Forecasts: These tend to have less systematic bias but still show variability depending on economic shocks.

Equicorrelation Model

What Is Equicorrelation?
In this model, every pair of forecast errors is assumed to have the same correlation. This simplifies the analysis while capturing the idea that all forecasters might be influenced by similar information.
Technical note: Equicorrelation implies that if one forecast error moves in one direction, all other errors tend to move similarly.
Findings from the Model:
The paper derives analytical formulas showing:
MSE decreases: When forecasts are averaged, the MSE decreases in a predictable way given the constant correlation.
Optimality of Equal Weights: Under equicorrelation, simply averaging forecasts (giving each equal weight) is optimal.
Differences Between Variables: The benefits from diversification (reducing MSE) are greater for inflation forecasts than for growth forecasts because the errors in inflation forecasts are less correlated with each other.

Example:
If every forecast error is moderately related (say, a correlation of 0.5) then averaging 10 forecasts will reduce the overall error more for inflation than for growth if the growth forecasts tend to be more correlated (say, 0.8).

4. Key Findings and Results

Diminishing Returns:
Both empirical data and the equicorrelation model show that while averaging forecasts initially provides significant error reduction, the benefits quickly diminish after a small number of forecasts (around 5 out of 40).
Variable Differences:
- Inflation Forecasts: Show a more substantial benefit from diversification. The reduction in error is more pronounced, and the average error ratio (compared to a single forecast) drops to about 60%.
- Growth Forecasts: Although still benefiting from averaging, the error ratio only drops to about 80%, indicating less benefit from additional forecasts.
Model Fit:
The equicorrelation model fits the SPF data almost perfectly when the parameters are chosen correctly. This means that the simple assumption of equal pairwise correlations is a very good approximation for how forecast errors behave in real-world surveys.
Implications for Survey Design:
Since the improvement in forecast accuracy tapers off quickly, it may not be necessary to use as many forecasters as is currently done. A smaller panel could be nearly as effective, which could have implications for the design and cost of such surveys.

5. Conclusion and Future Directions

Conclusions:
The paper concludes that averaging forecasts is beneficial, but with diminishing returns as more forecasters are added. The equicorrelation model explains the pattern very well, confirming that the simple average is optimal under realistic conditions where forecast errors are similarly related.
Practical Implications:
- For Policy Makers and Businesses: Knowing that a small number of well-selected forecasts can yield most of the benefits may help streamline decision-making processes.
- For Survey Administrators: The findings suggest rethinking the design of forecasting surveys like the SPF. Fewer respondents might be needed, reducing costs without significantly sacrificing accuracy.
Future Research Directions:
The authors suggest exploring more complex models of forecast error correlations, and looking into how these findings might apply to other types of forecasting (such as in finance or even machine learning ensemble methods).

Final Thoughts

In simple terms, this paper tells us that while “the wisdom of crowds” does work for economic forecasts, there is a point of diminishing returns. The initial few forecasts bring large improvements in accuracy, but adding more forecasts beyond that point yields only small gains. The equicorrelation model not only provides a clear explanation for this phenomenon but also confirms that averaging forecasts equally is nearly the best approach. This work has important implications for how economic forecasting surveys might be organized in the future.

Minchul Shin

Understanding the Wisdom of Crowds in Economic Forecasting

On the Wisdom of Crowds (of Economists)