Calibrating Climate Risk Probabilities: Reliability Diagrams in Python for Extreme Events

Climate-risk systems increasingly output probabilities rather than simple alerts. A district may receive a $0.72$ heatwave risk for tomorrow, or a catchment may receive a $0.18$ flood-exceedance probability for the next 24 hours. Those probabilities become operationally valuable only when they are calibrated.

That distinction matters because climate decisions are rarely binary. Public-health teams decide whether to open cooling centers, reservoir operators decide whether to pre-release water, and insurers decide whether to stress-test exposure. In each case, the decision depends not just on ranking risk, but on trusting the number attached to the forecast.

This article is for Python users, ML practitioners, climate analysts, and career-switchers moving into climate tech who want a rigorous but practical understanding of forecast calibration. The goal is to connect machine learning evaluation with climate risk management in a way that supports real work, not just benchmark scores.

By the end, you should understand why calibration is different from discrimination, how to read a reliability diagram, how to compute Brier scores and Brier Skill Scores, and how to build a leakage-safe Python workflow for calibrating heatwave or flood probabilities.

Background and prerequisites

You should already be comfortable with basic Python, pandas, NumPy, and standard supervised learning workflows. A basic understanding of binary classification, probability, and train-validation-test splits will make the implementation section much easier to follow.

On the climate side, the most important prerequisite is conceptual rather than mathematical. Extreme-event labels are not just technical artifacts. A heatwave, flood, or drought episode needs a domain definition that reflects local climatology, operational thresholds, and the decisions real users need to make.

For heatwaves, a local percentile threshold is often more meaningful than a single global temperature cutoff. A day above $35^\circ C$ can be routine in one region and dangerous in another. For floods, intense rainfall may matter less than river stage, upstream flow, soil saturation, drainage capacity, or snowmelt conditions.

You also need to keep nonstationarity in mind. Climate risk is not generated from a perfectly stable historical process. The baseline is shifting, the tails are changing, and the relationship between environmental variables and extreme outcomes can drift across years or decades. That means a model can perform well in historical backtests and still become poorly calibrated under newer conditions.

On the technical side, calibration sits at the intersection of machine learning and forecast verification. Many classifiers can produce probabilities, but those probabilities are not automatically trustworthy. Some models are overconfident, some underconfident, and some issue compressed probabilities that are too close to the base rate to support useful operational decisions.

Why calibration matters in climate and environmental decision-making

A common mistake in climate-risk modelling is to rely too heavily on ranking metrics such as ROC-AUC. Those metrics tell you whether the model tends to place dangerous cases above safer ones, which is useful, but they do not tell you whether a predicted probability of $0.7$ should be interpreted as roughly a $70\%$ chance.

That gap is not academic. Suppose a heat-risk system systematically predicts $0.7$ for situations that verify only $0.4$ of the time. The system may still look impressive on ranking metrics, but a city government using that threshold for action will overspend, misallocate resources, and eventually lose trust in the system.

The same issue appears in flood management. A catchment model may correctly rank wetter basins above drier ones, yet still overstate absolute risk in moderate conditions and understate it in the upper tail. A reservoir operator making storage or release decisions needs reliable probabilities, not just good ordering.

In climate adaptation, calibration is therefore a form of decision support quality control. It connects computational output to domain action. That makes it a software problem, an ML problem, and a climate-services problem at the same time.

Core theory: discrimination, calibration, and reliability diagrams

Discrimination versus calibration

Let $Y \in \{0, 1\}$ denote whether an extreme event occurs, and let $\hat{p}(X)$ be the model's predicted event probability based on features $X$.

A model has good discrimination if it tends to assign higher scores to true events than to non-events. That property is often summarized by metrics such as ROC-AUC or average precision. Discrimination answers the question, "Did the model rank risky situations above safer ones?"

Calibration answers a different question. A model is well calibrated when:

$$P(Y = 1 \mid \hat{p}(X) = p) = p$$

for values of $p$ across the probability range.

In words, among all cases where the model predicts $0.3$, the event should occur about $30\%$ of the time. Among all cases where it predicts $0.8$, the event should occur about $80\%$ of the time. That is what makes the probabilities interpretable.

A model can be highly discriminative and poorly calibrated at the same time. That happens often in tabular ML, in weather and hydrological models, and in ensemble-based hazard systems where the raw score is informative but not yet numerically trustworthy.

Reliability diagrams

A reliability diagram is the most direct visual tool for checking calibration. The predicted probabilities are grouped into bins, and for each bin you compute two values: the mean predicted probability and the observed event frequency.

If the model is perfectly calibrated, the points lie near the diagonal line:

$$y = x$$

Points below the diagonal indicate overforecasting. The model is predicting probabilities that are too high relative to what actually happens. Points above the diagonal indicate underforecasting. The event is occurring more often than the predicted probabilities suggest.

In climate-risk settings, reliability diagrams are useful because they reveal where the model fails. A system may be well behaved in low-risk conditions but become too aggressive in the $0.4$ to $0.7$ range. That is exactly the region where many operational thresholds live, so a small visual deviation can have large practical consequences.

Brier score

The Brier score is a proper scoring rule for binary probabilistic forecasts. It is defined as:

$$BS = \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{p}_i)^2$$

where $y_i$ is the observed outcome and $\hat{p}_i$ is the predicted probability.

The score is low when probabilities are close to the observed outcomes and high when they are far away. Because it is a proper score, it rewards honest probabilities. A forecaster cannot improve it in the long run by systematically exaggerating confidence.

For climate applications, this is useful because the Brier score respects the fact that a forecast of $0.2$ can be better than a forecast of $0$ if the event truly happens around $20\%$ of the time in comparable situations. That is a major advantage over threshold-based classification metrics.

Brier Skill Score

The Brier score is more informative when compared to a reference forecast. In climate work, the most natural reference is often climatology, meaning the long-run event frequency.

The Brier Skill Score is:

$$BSS = 1 - \frac{BS_{\text{model}}}{BS_{\text{ref}}}$$

A positive value means the model improves on the reference. A value near zero means it behaves little better than predicting the base rate all the time. A negative value means the model is worse than climatology, which is usually a warning sign that the model is not ready for deployment.

In practice, the Brier score and the reliability diagram should be read together. The score summarizes overall probabilistic quality, while the diagram shows where the model is miscalibrated.

Rare events, binning, and sample size

Extreme events are often rare, which makes calibration analysis more fragile than it appears in clean textbook examples. If you use too many bins, the highest-probability bins may contain very few observations, and the resulting points can move around dramatically from one evaluation window to another.

That is why quantile binning is often preferable to equal-width binning in rare-event work. Quantile bins aim to place a similar number of samples in each bin, which makes the reliability curve more stable and easier to interpret. It does not solve the small-sample problem completely, but it usually improves diagnostic value.

For operational climate systems, you should always inspect the sample count behind each bin, especially in the upper tail. A reliability point based on ten samples should not carry the same weight as one based on ten thousand.

Hands-on implementation in Python

Problem setup

We will build a simple district-level heatwave-risk model. Each row in the dataset represents one region on one day. The target is whether a three-day heatwave begins tomorrow.

This is a useful formulation because it matches a decision-making timeline. Public-health and emergency-management teams often care about next-day onset rather than today's condition, since the intervention window is small and resources must be staged in advance.

The same design pattern can be adapted to flood forecasting. You would replace the heatwave label with a runoff, river-stage, or threshold-exceedance label, then swap the feature set toward precipitation, antecedent wetness, upstream flow, and topography.

Step 1: load region-day data and define the event label

The label must be created carefully. One common mistake is to let the last few rows in each time series silently become negative examples just because future observations are missing. The code below avoids that by marking rows with incomplete future horizons as missing labels rather than false events.

import numpy as np
import pandas as pd

# Example schema:
# region_id, date, tmax_c, tmin_c, rh_pct, precip_mm, soil_moisture_pct,
# tmax_95p_c, tmax_climatology_c, rh_climatology_pct,
# enso_index, elevation_m, latitude, longitude, urban_fraction

df = pd.read_parquet("region_day_weather.parquet")
df["date"] = pd.to_datetime(df["date"])
df = df.sort_values(["region_id", "date"]).reset_index(drop=True)

# A locally meaningful "hot day" threshold based on climatology
df["hot_day"] = (df["tmax_c"] > df["tmax_95p_c"]).astype(int)

g = df.groupby("region_id", group_keys=False)

future_hot_1 = g["hot_day"].shift(-1)
future_hot_2 = g["hot_day"].shift(-2)
future_hot_3 = g["hot_day"].shift(-3)

valid_future_horizon = (
    future_hot_1.notna() &
    future_hot_2.notna() &
    future_hot_3.notna()
)

df["heatwave_next_3d"] = np.where(
    valid_future_horizon,
    ((future_hot_1 == 1) & (future_hot_2 == 1) & (future_hot_3 == 1)).astype(int),
    np.nan
)

This definition is deliberately domain-aware. It uses a local 95th-percentile threshold rather than a single global temperature cutoff. That matters because the same absolute temperature does not imply the same physical or social stress everywhere.

It also aligns the label with an operational question. Decision-makers usually want to know whether a dangerous episode is about to begin, not whether today happens to be above threshold.

Step 2: engineer features from recent climate history

The next step is feature engineering. In extreme-event work, short-memory persistence is often crucial. Recent temperature, rainfall, humidity, and soil moisture patterns can strongly affect tomorrow’s risk, especially when extremes build over several days.

# Rolling summaries using present and past information only
df["tmax_3d_mean"] = g["tmax_c"].transform(lambda s: s.rolling(3, min_periods=1).mean())
df["tmax_7d_mean"] = g["tmax_c"].transform(lambda s: s.rolling(7, min_periods=1).mean())
df["precip_7d_sum"] = g["precip_mm"].transform(lambda s: s.rolling(7, min_periods=1).sum())
df["soil_moisture_7d_mean"] = g["soil_moisture_pct"].transform(
    lambda s: s.rolling(7, min_periods=1).mean()
)

# Seasonal anomalies relative to climatology
df["tmax_anom_c"] = df["tmax_c"] - df["tmax_climatology_c"]
df["rh_anom_pct"] = df["rh_pct"] - df["rh_climatology_pct"]

# Encode seasonality smoothly
doy = df["date"].dt.dayofyear
df["doy_sin"] = np.sin(2 * np.pi * doy / 365.25)
df["doy_cos"] = np.cos(2 * np.pi * doy / 365.25)

feature_cols = [
    "tmax_c", "tmin_c", "rh_pct", "precip_mm", "soil_moisture_pct",
    "tmax_3d_mean", "tmax_7d_mean", "precip_7d_sum", "soil_moisture_7d_mean",
    "tmax_anom_c", "rh_anom_pct", "enso_index",
    "elevation_m", "latitude", "longitude", "urban_fraction",
    "doy_sin", "doy_cos"
]

model_df = df.dropna(subset=feature_cols + ["heatwave_next_3d"]).copy()
model_df["heatwave_next_3d"] = model_df["heatwave_next_3d"].astype(int)

These features combine physics and ML pragmatism. Temperature anomalies and rolling means help the model distinguish transient warm days from sustained build-up toward heatwaves. Soil moisture and rainfall can matter because dry conditions often reinforce heat. Seasonal encoding helps the model learn different baseline regimes without treating time of year as an arbitrary integer.

If you were modelling floods instead, the same structure would still work. You would replace the heatwave label with a hydrological threshold and emphasize antecedent rainfall, basin wetness, snowmelt, upstream flow, and forecast precipitation.

Step 3: Use a time-aware split and train a base model

Calibration for climate risk should respect time. Random splits can leak seasonal and interannual structure into training, which makes probability quality look better than it really is. A chronological split is more honest.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier

train_mask = model_df["date"] < "2020-01-01"
val_mask = (model_df["date"] >= "2020-01-01") & (model_df["date"] < "2022-01-01")
test_mask = model_df["date"] >= "2022-01-01"

X_train = model_df.loc[train_mask, feature_cols]
y_train = model_df.loc[train_mask, "heatwave_next_3d"]

X_val = model_df.loc[val_mask, feature_cols]
y_val = model_df.loc[val_mask, "heatwave_next_3d"]

X_test = model_df.loc[test_mask, feature_cols]
y_test = model_df.loc[test_mask, "heatwave_next_3d"]

base_model = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("rf", RandomForestClassifier(
        n_estimators=500,
        min_samples_leaf=20,
        class_weight="balanced_subsample",
        n_jobs=-1,
        random_state=42
    )),
])

base_model.fit(X_train, y_train)

val_raw = base_model.predict_proba(X_val)[:, 1]
test_raw = base_model.predict_proba(X_test)[:, 1]

A random forest is a sensible baseline for this kind of tabular climate problem. It can capture nonlinear interactions among anomalies, persistence, humidity, geography, and seasonality. It is also a model family that often benefits from calibration, which makes it useful for teaching and for real operational baselines. scikit-learn notes that different classifiers show different characteristic calibration behaviors, which is exactly why calibration should be checked rather than assumed.

Step 4: fit calibration mappings on a separate validation window

Now we learn the probability correction. The critical rule is separation: calibration must be learned on data that was not used to train the base model. scikit-learn’s guidance is explicit about that.

We will use two post-hoc approaches. The first is a logistic recalibration, often called sigmoid-style calibration. The second is isotonic regression, which is more flexible but can overfit if the calibration set is small.

from sklearn.isotonic import IsotonicRegression
from sklearn.linear_model import LogisticRegression

# Sigmoid-style recalibration
sigmoid_calibrator = LogisticRegression()
sigmoid_calibrator.fit(val_raw.reshape(-1, 1), y_val)
test_sigmoid = sigmoid_calibrator.predict_proba(test_raw.reshape(-1, 1))[:, 1]

# Isotonic recalibration
isotonic_calibrator = IsotonicRegression(out_of_bounds="clip")
isotonic_calibrator.fit(val_raw, y_val)
test_isotonic = isotonic_calibrator.transform(test_raw)

This step is conceptually simple. The base model gives you a raw probability-like score. The calibrator learns how that score should be bent or stretched so that it better matches observed frequencies. In operational terms, this is the moment when you stop trusting the classifier’s native probability at face value.

Step 5: evaluate with Brier score, Brier Skill Score, and reliability diagrams

We now compare the uncalibrated and calibrated outputs on a future test period. Because the event is rare, it is useful to inspect both ranking and calibration. Average precision helps with the rare-event ranking side, while the Brier score and the reliability diagram focus on probability quality.

import matplotlib.pyplot as plt
from sklearn.calibration import calibration_curve
from sklearn.metrics import (
    average_precision_score,
    brier_score_loss,
    roc_auc_score
)

def brier_skill_score(y_true, y_prob, reference_rate):
    reference_prob = np.full(len(y_true), reference_rate, dtype=float)
    model_bs = brier_score_loss(y_true, y_prob)
    reference_bs = brier_score_loss(y_true, reference_prob)
    return 1.0 - (model_bs / reference_bs)

predictions = {
    "raw": test_raw,
    "sigmoid": test_sigmoid,
    "isotonic": test_isotonic
}

# Climatology reference from the training window: climatology = y_train.mean()

for name, prob in predictions.items():
    print(
        f"{name:8s} "
        f"ROC-AUC={roc_auc_score(y_test, prob):.3f} "
        f"AP={average_precision_score(y_test, prob):.3f} "
        f"BS={brier_score_loss(y_test, prob):.3f} "
        f"BSS={brier_skill_score(y_test, prob, climatology):.3f}"
    )

plt.figure(figsize=(7, 6))
plt.plot([0, 1], [0, 1], "--", label="perfect calibration")

for name, prob in predictions.items():
    frac_pos, mean_pred = calibration_curve(
        y_test,
        prob,
        n_bins=10,
        strategy="quantile"
    )
    plt.plot(mean_pred, frac_pos, marker="o", label=name)

plt.title("Reliability diagram for heatwave probabilities")
plt.xlabel("Mean predicted probability")
plt.ylabel("Observed event frequency")
plt.legend()
plt.tight_layout()
plt.show()

The diagonal line is the target. If the raw model curve sits below the diagonal in medium and high bins, it is overpredicting heatwave risk. If isotonic or sigmoid calibration moves the curve closer to the line and improves the Brier score, then the calibrated output is more trustworthy as a decision variable.

You should not expect every metric to move in the same direction. Calibration often improves the Brier score more than ROC-AUC because it changes the meaning of the probabilities rather than radically changing the ranking. That is normal. Think of AUC as “who is riskier than whom,” and Brier plus reliability as “how much should I believe the number.”

Why the pipeline is structured this way

The most important design choice is the chronological split. In climate and environmental forecasting, random shuffling across years can leak temporal structure and make evaluation look better than it really is. A time-aware split is a better proxy for how the model will behave in production.

The second key choice is the separate calibration window. The base model is trained on one historical block, while the calibrator is learned on a later but still disjoint block. That matters because post-hoc calibration can overfit if it is learned on the same cases used to fit the underlying model.

The third important detail is the label definition. The example uses a local percentile threshold for hot days before building the three-day event label. That turns the label into a climate-aware target rather than a generic classification artifact.

Interpreting the output

Start with the Brier score and Brier Skill Score. If the calibrated models reduce the Brier score and improve the skill score relative to climatology, that is evidence that the probabilities have become more useful.

Then inspect the reliability diagram. If the raw model sits below the diagonal in the middle bins, it is overforecasting risk. If sigmoid or isotonic calibration moves those points closer to the diagonal, the post-hoc step is doing what it should.

Finally, check the probability histograms. A model can become more calibrated by shrinking everything toward the mean, which may improve reliability but reduce operational sharpness. In practice, you want calibrated probabilities that still spread enough to support meaningful threshold-based action.

Extending the workflow to flood probabilities

The same logic applies to flood forecasting. The main changes are the label definition and the feature space.

For a flood application, the label might be whether the river stage exceeds a warning threshold within the next day or two. The features might include antecedent rainfall, upstream flow, basin wetness indices, forecast precipitation, snowmelt proxies, terrain slope, impervious surface fraction, and catchment response time.

Calibration is often even more important in flood systems because the tail is sparse and the cost of false confidence is high. An underforecasted high-risk event can lead to delayed action, while persistent overforecasting can erode institutional trust and make warning fatigue worse.

Systems, production, and operational deployment

In production, climate-risk systems are usually batch pipelines. Meteorological or hydrological inputs arrive on a daily or sub-daily schedule, features are assembled for each region and lead time, the base model scores the cases, and a calibrator converts the raw outputs into published probabilities.

That architecture may sound simple, but the surrounding systems work is substantial. Climate data often comes from multiple sources with different spatial grids, time conventions, update latencies, and missingness patterns. The calibration layer only works well if the upstream feature pipeline is reproducible and well monitored.

It is good practice to version three artifacts separately: the feature pipeline, the base model, and the calibrator. If reliability degrades after deployment, that separation makes diagnosis much easier. Otherwise, a data shift, feature bug, and calibration bug can all look like the same downstream symptom.

Performance and cost depend more on data engineering than on the calibrator itself. Post-hoc calibration is computationally cheap. The expensive parts are often geospatial joins, temporal aggregation, gridded-data extraction, and model scoring at large spatial scales. For tabular workflows, CPU inference is often enough. GPU cost becomes relevant once you move toward deep spatiotemporal models on raster data.

Monitoring should be climate-specific, not just generic MLOps. A useful operational dashboard should track calibration by season, climate zone, and lead time. It should also monitor event-based rate, missing data, late-arriving feeds, the share of forecasts in high-risk bins, and drift in the probability distribution.

Recalibration may happen more frequently than full retraining. If the score distribution drifts because the base climate or upstream forecast behavior changes, a new calibration mapping may restore probability quality without immediately retraining the core model. That said, recalibration is not a substitute for deeper model maintenance when the underlying relationship between features and events changes materially.

Risk, ethics, safety, and governance

The first risk is representativeness. Climate observations and impact records are uneven across space. Some regions have dense station networks and long histories. Others have sparse monitoring and noisier labels. A model can look globally calibrated while being poorly calibrated in exactly the communities that are most vulnerable.

The second risk is over-interpretation. Domain users may treat a well-calibrated probability as certainty if the product is presented without context. A forecast of $0.7$ does not mean the event must happen. It means that situations like this have historically occurred around $70\%$ of the time. That distinction needs to be visible in product documentation and training.

Privacy and security also matter when hazard probabilities are combined with exposure, vulnerability, or infrastructure data. A district-level environmental forecast may be relatively benign on its own, but risk becomes more sensitive when those outputs are joined with household, medical, mobility, or critical-asset data. That can trigger governance requirements under privacy and security frameworks such as GDPR-style data minimization, access control, auditability, and formal risk assessment.

Robustness matters as much as fairness. Sensor outages, station moves, bias-corrected forecast changes, missing grids, and label-quality drift can all degrade calibration without immediately breaking the pipeline. A system that always returns a number is not necessarily a reliable system.

A strong governance approach, therefore,e combines technical and organizational controls. Evaluate calibration by subgroup, keep humans in the loop for high-impact decisions, limit access to sensitive overlays, version the full pipeline, and communicate both uncertainty and known failure modes in plain language.

Domain-specific case study: heat-health early warning for public health planning

Imagine a public-health agency running a district-level heat-health early warning service. Every afternoon, it receives weather forecasts and must decide where to stage cooling resources, extend clinic hours, and push targeted communication before dangerous heat builds.

The system uses district-day features such as maximum temperature, minimum temperature, humidity, recent rainfall, soil moisture, urban fraction, and local climatology. It predicts whether a three-day heatwave will begin the next day.

Now consider two districts. District A receives a forecast probability of $0.68$. District B receives a forecast probability of $0.24$. If the system is well calibrated, planners can use those probabilities as meaningful inputs into response thresholds and staffing decisions.

Suppose reliability analysis shows that the raw model overstates the middle range and that post-hoc calibration brings the $0.6$ to $0.7$ bin much closer to observed frequency. That improvement is not just a metric win. It changes whether a planner should trust the model enough to open cooling centers or hold resources back.

This is the interdisciplinary value of calibration. The climate side defines a meaningful event and recognizes a shifting hazard baseline. The software side builds reproducible pipelines and monitoring. The ML side turns model scores into trustworthy probabilities. All three are necessary for operational impact.

Skills mapping and learning path

For learners in a bootcamp-style path, this topic develops practical Python and data skills first. You work with time-aware tabular data, rolling features, label construction, metrics, and plots. Those are transferable skills, whether you later work in climate tech, finance, healthcare, or other data-heavy domains.

It also deepens your ML evaluation mindset. Many beginners learn how to train models before they learn how to judge probabilistic outputs properly. Calibration introduces proper scoring rules, model diagnostics, and the idea that a model can be useful in one sense and misleading in another.

On the systems side, the article introduces production thinking. You are not just training a notebook model. You are separating artifacts, monitoring drift, and asking how the probabilities will be used by a real domain team with costs, constraints, and accountability requirements.

On the domain side, you build judgment about event definitions, nonstationarity, operational thresholds, and the consequences of miscalibration. That domain literacy is what turns technical competence into useful climate-risk practice.

A sensible next step is to reproduce the workflow on a public dataset, then compare logistic regression, random forests, and gradient boosting, add spatial holdouts, and finally experiment with region-specific or lead-time-specific calibration. Once that baseline is stable, it becomes much easier to evaluate whether more complex deep-learning approaches are actually worth the extra cost.

Conclusion

Calibration is one of the clearest examples of where machine learning quality and domain usefulness meet. A model that merely ranks risk is not enough for climate adaptation, early warning, or public-sector planning. Users need probabilities that are interpretable, stable, and honest about uncertainty.

Reliability diagrams provide a direct visual check of whether predicted probabilities line up with observed frequencies. Brier scores summarize probabilistic quality in a way that respects uncertainty rather than collapsing it into hard classifications. Together, they offer a stronger foundation for climate-risk systems than ranking metrics alone.

The broader lesson is that climate AI is not just about fitting a model. It is about defining good targets, respecting temporal structure, calibrating outputs, monitoring drift, and communicating uncertainty in a way domain experts can act on responsibly.

If you build calibration into your workflow from the start, you do more than improve a metric. You build a forecasting system that is better aligned with how climate-risk decisions are actually made.