Bin-Conditional Conformal Prediction of Fatalities from Armed Conflict

By David Randahl, Jonathan P. Williams, Håvard Hegre

DOI https://doi.org/10.48550/arXiv.2410.14507

Abstract

Forecasting armed conflicts is a critical area of research with the potential to save lives and mitigate suffering. While existing forecasting models offer valuable point predictions, they often lack individual-level uncertainty estimates, limiting their usefulness for decision-making. Several approaches exist to estimate uncertainty, such as parametric and Bayesian prediction intervals, bootstrapping, quantile regression, but these methods often rely on restrictive assumptions, struggle to provide well-calibrated intervals across the full range of outcomes, or are computationally intensive. Conformal prediction offers a model-agnostic alternative that guarantees a user-specified level of coverage but typically provides only marginal coverage, potentially resulting in non-uniform coverage across different regions of the outcome space.In this paper, we introduce a novel extension called bin-conditional conformal prediction (BCCP), which enhances standard conformal prediction by ensuring consistent coverage rates across user-defined subsets (bins) of the outcome variable. We apply BCCP to simulated data as well as the forecasting of fatalities from armed conflicts, and demonstrate that it provides well-calibrated uncertainty estimates across various ranges of the outcome.Compared to standard conformal prediction, BCCP offers improved local coverage,though this comes at the cost of slightly wider prediction intervals.

Overview

The paper addresses a critical challenge in forecasting armed conflict fatalities: providing not just a single (point) prediction but also a reliable range that reflects the uncertainty in each individual forecast. This is important because decision-makers, such as policymakers and humanitarian organizations, need to know not only what might happen but also how confident we can be in these predictions.

Key Concepts

Forecasting Fatalities: Predicting the number of deaths that may occur in armed conflicts. Since extreme events (like very high fatalities) are rare but highly significant, knowing the range of possible outcomes is crucial.
Point Predictions vs. Uncertainty: Traditional models usually offer a best-guess number (point prediction) but do not show how much this number could vary. For example, predicting “0 deaths” might hide the possibility that there is also a small chance of very high fatalities.
Prediction Intervals: These are ranges (intervals) around a point prediction within which the actual outcome is expected to fall with a certain probability (for example, 90%). They communicate uncertainty by saying, “We are 90% sure that the true number will be within this range.”
Non-Conformity Scores: A key technical term here. Think of it as a measure of error that tells you how different a new prediction is from what the model has seen before. In simple terms, it is like checking “how unusual” the prediction error is compared to past errors.
Exchangeability: A statistical assumption meaning that the order of the data points does not matter. This assumption is necessary for many methods (including conformal prediction) to work reliably.
Conformal Prediction (CP): A general, model-agnostic method that wraps around any prediction algorithm to produce prediction intervals. Standard conformal prediction (SCP) guarantees that, on average, the true outcome will fall within the prediction interval (e.g., 90% of the time) but does not ensure that this level of certainty holds uniformly across different ranges of outcomes.

Methodology: Introducing Bin-Conditional Conformal Prediction (BCCP)

Why BCCP?

Limitation of SCP: Standard CP ensures that, overall, a certain percentage of outcomes fall within the predicted range. However, if you look at different parts of the outcome space (for instance, low fatalities versus high fatalities), the level of uncertainty (coverage) might not be the same. In the context of armed conflict, this is particularly important because while many predictions are near zero (due to the rarity of high-fatality events), the rare high-fatality cases are crucial for planning and response.
BCCP’s Improvement: BCCP extends SCP by dividing the outcome space into “bins” (subgroups or ranges) and calibrating the prediction intervals within each bin. This means that for every range of fatalities, the method aims to ensure that the interval is correctly calibrated—i.e., if you say you are 90% confident, then roughly 90% of the time the actual fatalities in that bin will fall within the interval.

How Does BCCP Work?

Data Splitting:
- Training Set: Used to build the prediction model.
- Calibration Set: Used to measure the errors (non-conformity scores) of the predictions.
- Calculating Non-Conformity Scores:
  For each case in the calibration set, the method computes how “off” the model’s prediction is compared to what actually happened.

Binning the Outcome:
- The range of fatalities (the outcome) is divided into bins. For example, one could split the data at the 25th, 50th, and 75th percentiles.
- This allows the method to look at smaller, more homogeneous groups of outcomes rather than treating all predictions the same.

Creating Prediction Intervals in Each Bin:
- For each bin, BCCP uses the non-conformity scores specific to that range to determine the prediction interval.
- There are two approaches:
  - Discontiguous Intervals: The final prediction interval may be a union of separate ranges.
  - Contiguous Intervals (Contiguized): These separate ranges are combined into one continuous interval, which may cover a bit more than needed in some areas.

Trade-Offs:
- More Bins: Better local calibration (i.e., more accurate uncertainty estimates for each range) but can lead to wider overall intervals.
- Fewer Bins: Narrower intervals overall, but may miss local variations in uncertainty.

Simulation Study and Findings

Simulation Setup

Data Generation:
The authors use simulated data where two features are drawn from a uniform distribution, and the fatality outcome is generated from a log-normal distribution. This distribution is chosen because it is heavily skewed—many values are close to zero, but there can also be very high values.
Modeling Approach:
A simple linear regression is applied (using the logarithm of the outcome) to predict fatalities. Predictions are then transformed back to the original scale.
Evaluation:
The performance of BCCP is compared with SCP and other methods (such as bootstrapping, quantile regression, and parametric prediction intervals) by looking at:
Overall Coverage: Whether the true outcomes fall within the predicted intervals the expected percentage of times (e.g., 90%).
Local (Bin-Specific) Coverage: Whether this coverage is consistent across different ranges (bins) of the outcome.
Interval Width: How wide the prediction intervals are.

Key Findings

Improved Local Calibration:
BCCP consistently provides prediction intervals that are well-calibrated within each bin. This means that for every subgroup (e.g., low, medium, high fatalities), the prediction intervals are more reliable.
Wider Intervals:
One trade-off of the BCCP method is that the intervals may be slightly wider than those produced by standard conformal prediction. Wider intervals indicate a more conservative (safer) estimate of uncertainty, which is often desirable in high-stakes situations like armed conflict forecasting.
Comparison to Alternatives:
When compared to other methods (like bootstrapping and quantile regression), BCCP tends to perform better in maintaining the correct level of uncertainty across all parts of the outcome range. This robustness is particularly valuable when dealing with skewed data.

Conclusion and Implications

The paper concludes that:

BCCP is a Promising Extension:
By partitioning the outcome space into bins and calibrating prediction intervals within each bin, BCCP offers a more nuanced way to quantify uncertainty. This is especially important in areas such as armed conflict forecasting, where understanding the full range of possible outcomes can help prevent underestimation of severe events.
Broader Applications:
Although the study focuses on forecasting fatalities from armed conflict, the method is general and can be applied to any field where uncertainty quantification is needed. For example, it could be used in economics, healthcare, or any predictive modeling scenario with skewed data.
Future Directions:
The authors suggest that future work could explore:
- Optimizing how bins are chosen.
- Extending BCCP to handle other data structures.
- Further comparisons with other uncertainty estimation techniques.

Simplified Example

Imagine you are forecasting the number of injuries in different regions after a natural disaster. Traditional models might predict “0 injuries” for most regions but miss the possibility of a few regions experiencing hundreds of injuries. With BCCP, you could divide the regions into groups (bins) based on factors like population density or previous injury counts. Then, for each group, you calculate a range that reliably includes the true injury count 90% of the time. This way, decision-makers can see not just an average prediction but a reliable range tailored to each specific group.

Final Thoughts

The paper offers an innovative method to address a real-world problem: how to communicate and manage uncertainty in high-impact forecasts. BCCP improves upon traditional methods by ensuring that the uncertainty estimates are valid not only overall but also within specific subgroups of data, thereby providing more reliable and actionable information for policymakers and other stakeholders.

Enhancing Armed Conflict Fatality Forecasts: Bin-Conditional Conformal Prediction