Fairness in Machine Learning

Every time a predictive algorithm sorts people — flagging loan applicants, recommending parole, screening résumés — someone asks: is it fair? The question sounds simple. The mathematics turns out to be brutal.

There are several natural ways to define "fair." A model might be calibrated: when it says "70% risk," roughly 70% of those flagged should actually be high-risk. Or it might equalise false-positive rates across groups: the fraction of innocent people wrongly flagged should be the same in every demographic. Or it might equalise false-negative rates: the fraction of actually-risky people who slip through undetected should be the same everywhere.

All three sound reasonable. All three are demanded by real institutions and legal frameworks. And in 2016–2017, two independent research teams — Chouldechova and Kleinberg, Mullainathan & Raghavan — proved that you generally cannot have all three at once. When two groups differ in their base rates (the true underlying prevalence of the outcome), satisfying any two of these criteria forces a violation of the third. This is not a limitation of today's algorithms. It is a mathematical theorem.

The two groups below have different base rates: Group A has a 30% true positive rate and Group B has a 60% true positive rate. Move the shared decision threshold and watch what happens to calibration, false-positive rate (FPR), and false-negative rate (FNR) for each group.

<!-- {{c_html_intro}} -->
<div class="demo-wrap">
  <p class="desc">{{desc_text}}</p>
  <div class="threshold-row">
    <label for="thresh">{{label_threshold}}: <span id="thresh-val">0.50</span></label>
    <input type="range" id="thresh" min="0" max="100" value="50" step="1">
  </div>
  <div class="groups">
    <div class="group" id="group-a">
      <h3>{{group_a_title}}</h3>
      <p class="base-rate">{{label_base_rate}}: <strong>30%</strong></p>
      <div class="metrics" id="metrics-a"></div>
    </div>
    <div class="group" id="group-b">
      <h3>{{group_b_title}}</h3>
      <p class="base-rate">{{label_base_rate}}: <strong>60%</strong></p>
      <div class="metrics" id="metrics-b"></div>
    </div>
  </div>
  <div class="verdict" id="verdict"></div>
  <canvas id="chart" width="520" height="160" aria-label="{{chart_aria}}"></canvas>
  <p class="chart-caption">{{chart_caption}}</p>
</div>

/* {{c_css_intro}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; padding: .5rem; }
.demo-wrap { max-width: 560px; margin: 0 auto; }
.desc { font-size: .88rem; color: #444; margin: 0 0 .8rem; line-height: 1.5; }
.threshold-row { display: flex; flex-direction: column; gap: .25rem; margin-bottom: .8rem; }
.threshold-row label { font-weight: 600; font-size: .9rem; }
#thresh { width: 100%; accent-color: #1d3557; }
.groups { display: grid; grid-template-columns: 1fr 1fr; gap: .8rem; margin-bottom: .7rem; }
.group { background: #f0f4f8; border-radius: 10px; padding: .7rem .8rem; }
.group h3 { margin: 0 0 .25rem; font-size: .95rem; color: #1d3557; }
.base-rate { margin: 0 0 .4rem; font-size: .82rem; color: #555; }
.metrics { display: flex; flex-direction: column; gap: .3rem; }
.metric-row { display: flex; justify-content: space-between; align-items: center; font-size: .83rem; }
.metric-label { color: #444; }
.metric-val { font-weight: 700; font-size: .9rem; }
.ok { color: #0a7d33; }
.warn { color: #c92f3c; }
.neutral { color: #1d3557; }
.verdict { font-size: .88rem; font-weight: 600; min-height: 1.6em; padding: .4rem .6rem;
           border-radius: 8px; margin-bottom: .6rem; background: #fff3cd; color: #6b4c00; }
canvas { display: block; width: 100%; border-radius: 8px; background: #f9fafb; border: 1px solid #dde3ea; }
.chart-caption { font-size: .78rem; color: #666; margin: .3rem 0 0; text-align: center; }

// Code not found

No matter where you place the threshold, at least one fairness criterion breaks. Lower the threshold and you equalise FNR — but FPR diverges. Raise it and FPR equalises — but FNR diverges. The groups are calibrated by construction, but that calibration coexists with persistent gaps in error rates. The impossibility is not a bug in the slider — it is a theorem.

Let two groups have prevalences (base rates) $p_A$ and $p_B$ with $p_A \ne p_B$. A binary classifier assigns a score; we apply a threshold to get a binary decision. Define for each group:

• PPV (positive predictive value / calibration): among those flagged positive, the fraction that are truly positive.
• FPR (false-positive rate): among truly negatives, the fraction wrongly flagged positive.
• FNR (false-negative rate): among truly positives, the fraction wrongly missed.

These three quantities are linked by a simple identity. If PPV is the same across groups and FPR is the same across groups, then FNR must differ — and vice-versa. The algebra is exact:

$$\text{PPV} = \frac{p \cdot (1 - \text{FNR})}{p \cdot (1 - \text{FNR}) + (1 - p) \cdot \text{FPR}}$$

Fix PPV and FPR to be equal across groups with $p_A \ne p_B$; solve for FNR — you get different values. There is no escape. Chouldechova (2017) derived this identity and showed it means that calibration and equalised error rates are jointly impossible when base rates differ. Kleinberg, Mullainathan & Raghavan (2016) proved the same constraint from a different angle using score distributions.

This is a proved impossibility result — a cousin of impossibility theorems in computational complexity. No amount of clever engineering lifts it. Society must choose which notion of fairness to prioritise for any given deployment, and that choice is inherently political.

The impossibility theorem is not abstract. Every high-stakes classifier lives inside it:

• Criminal justice: the COMPAS recidivism tool was shown by ProPublica (2016) to have higher FPR for Black defendants. Northpointe countered that COMPAS is calibrated. Both were right — the impossibility explains why both findings coexist.
• Hiring: résumé-screening algorithms trained on historical hiring data inherit historical base-rate differences. Equalising callback rates by group (equalising FPR) conflicts with calibrated score interpretation.
• Medical triage: a sepsis-prediction model calibrated on a diverse population can still have unequal FNR across demographic groups if prevalence differs — missing higher-risk patients in lower-prevalence groups.
• Credit scoring: lenders regulated to have equal approval rates (equalised FPR) will necessarily have different default prediction accuracy (PPV) across groups with different default rates.

Understanding the theorem tells practitioners what to expect and forces a conscious, auditable choice among fairness criteria rather than an accidental one. It connects directly to ideas in PAC learning: what a model can and cannot guarantee from data.

Fairness in machine learning is not a dial you can simply turn up. Chouldechova and Kleinberg et al. showed that calibration, equal false-positive rates, and equal false-negative rates form an impossible trinity whenever the groups being compared have different base rates — which is nearly always.

This does not mean we should give up on fairness. It means we must be precise about which fairness we are pursuing, transparent about what we are sacrificing, and humble about the limits of what any algorithm can deliver. The theorem turns a vague aspiration into an exact constraint — and that is the first step toward genuine accountability.

Like P vs NP, the fairness impossibility draws a hard line around what is achievable. Unlike P vs NP, we already have the proof.