Rényi Differential Privacy

Every time a machine-learning model touches your data during training, it leaks a tiny sliver of information. Differential privacy (DP) gives us a mathematical guarantee: no matter what an adversary does, training on your data versus leaving it out produces outputs that are nearly indistinguishable.

The core parameter $\varepsilon$ measures how much the outputs can differ in the worst case. Add a small dose of random noise to each gradient update and you get a DP-SGD algorithm. The trouble is accounting: training runs for thousands of steps, and the privacy cost accumulates with every one of them.

The naive approach — basic composition — simply adds $\varepsilon$ values step by step. It works, but it is pessimistic: it assumes every step hits the worst case simultaneously, which almost never happens in practice. You end up with an enormous stated $\varepsilon$ that makes results look far less private than they actually are.

Rényi differential privacy (RDP), introduced by Ilya Mironov in 2017, threads a better ledger through the problem. By measuring divergence not with a single worst-case ratio but with a Rényi divergence — a whole family of divergences parameterized by an order $\alpha$ — it tracks privacy loss in a way that composes almost perfectly across steps. The result is a tighter, more honest $\varepsilon$ estimate for the same noise level, unlocking models that would otherwise be declared too leaky to release.

Each training step applies a Gaussian mechanism with noise multiplier $\sigma$ to a batch of fraction $q$ of the dataset. The demo computes the privacy cost over $T$ steps under both basic composition (naive $\varepsilon$ addition) and RDP composition (summing the Rényi divergence at each order $\alpha$ then converting).

<!-- {{c_demo_intro}} -->
<div class="controls">
  <label>
    <span class="label-text">{{lbl_sigma}} &sigma;</span>
    <input type="range" id="sigma" min="0.5" max="4" step="0.1" value="1.2">
    <span class="val" id="sigma-val">1.2</span>
  </label>
  <label>
    <span class="label-text">{{lbl_q}} q</span>
    <input type="range" id="q" min="0.001" max="0.1" step="0.001" value="0.01">
    <span class="val" id="q-val">0.010</span>
  </label>
  <label>
    <span class="label-text">{{lbl_steps}} T</span>
    <input type="range" id="steps" min="100" max="10000" step="100" value="1000">
    <span class="val" id="steps-val">1000</span>
  </label>
  <label>
    <span class="label-text">{{lbl_delta}} &delta;</span>
    <input type="range" id="delta-exp" min="-8" max="-4" step="1" value="-5">
    <span class="val" id="delta-val">10<sup>-5</sup></span>
  </label>
</div>
<div class="chart-wrap">
  <canvas id="chart" width="460" height="200"></canvas>
  <div class="y-label">{{lbl_y_axis}}</div>
</div>
<div class="results" id="results">
  <div class="result-box basic">
    <div class="result-label">{{lbl_basic}}</div>
    <div class="result-val" id="basic-val">—</div>
  </div>
  <div class="result-box rdp">
    <div class="result-label">{{lbl_rdp}}</div>
    <div class="result-val" id="rdp-val">—</div>
  </div>
  <div class="result-box ratio">
    <div class="result-label">{{lbl_ratio}}</div>
    <div class="result-val" id="ratio-val">—</div>
  </div>
</div>
<p class="note">{{note_para}}</p>

/* {{c_style_reset}} */
* { box-sizing: border-box; margin: 0; padding: 0; }
body { font-family: system-ui, sans-serif; color: #222; padding: .5rem .6rem; font-size: .9rem; }
.controls { display: grid; grid-template-columns: 1fr 1fr; gap: .5rem .8rem; margin-bottom: .7rem; }
label { display: flex; flex-direction: column; gap: .15rem; }
.label-text { font-size: .78rem; font-weight: 600; color: #555; }
input[type=range] { width: 100%; accent-color: #1d3557; cursor: pointer; }
.val { font-size: .8rem; font-weight: 700; color: #1d3557; }
.chart-wrap { position: relative; margin-bottom: .6rem; }
canvas { display: block; width: 100%; border: 1px solid #dde; border-radius: 6px; background: #fafbfc; }
.y-label { position: absolute; left: -1px; top: 50%; transform: rotate(-90deg) translateX(50%); font-size: .65rem; color: #888; white-space: nowrap; transform-origin: left center; }
.results { display: flex; gap: .5rem; margin-bottom: .5rem; }
.result-box { flex: 1; border-radius: 8px; padding: .4rem .5rem; text-align: center; }
.result-box.basic { background: #fde8e8; border: 1px solid #f5b8b8; }
.result-box.rdp { background: #e8f4e8; border: 1px solid #a8d5a8; }
.result-box.ratio { background: #e8eef6; border: 1px solid #b0c4de; }
.result-label { font-size: .7rem; font-weight: 600; color: #555; margin-bottom: .15rem; }
.result-val { font-size: 1.05rem; font-weight: 700; }
.result-box.basic .result-val { color: #c92f3c; }
.result-box.rdp .result-val { color: #0a7d33; }
.result-box.ratio .result-val { color: #1d3557; }
.note { font-size: .78rem; color: #666; line-height: 1.4; margin-top: .2rem; }

// Code not found

Slide $\sigma$ or $T$ and watch the gap grow. Basic composition reports a budget consumed that is many times larger than what RDP accounting reveals — even though both describe the same algorithm. The difference is not free noise: it is the cost of a pessimistic ledger versus a tight one.

Rényi divergence of order $\alpha > 1$ between distributions $P$ and $Q$ is:

$$D_\alpha(P \| Q) = \frac{1}{\alpha - 1} \log \mathbb{E}_{x \sim Q}\!\left[\left(\frac{P(x)}{Q(x)}\right)^{\!\alpha}\right]$$

A mechanism $\mathcal{M}$ satisfies $(\alpha, \varepsilon_\alpha)$-RDP if for any two neighboring datasets $D, D'$:

$$D_\alpha\!\bigl(\mathcal{M}(D) \| \mathcal{M}(D')\bigr) \le \varepsilon_\alpha$$

Three properties make this definition powerful:

• Exact composition. Apply $k$ independent $(\alpha, \varepsilon_\alpha)$-RDP mechanisms and the result is $(\alpha, k\,\varepsilon_\alpha)$-RDP. No slack, no worst-case penalty per step.
• Gaussian mechanism closed form. The Gaussian mechanism with sensitivity $\Delta$ and noise $\sigma$ satisfies $(\alpha, \alpha\Delta^2/(2\sigma^2))$-RDP for every $\alpha$. That single formula covers all orders at once.
• Conversion to $(\varepsilon, \delta)$-DP. From an RDP guarantee at order $\alpha$ one recovers: $\varepsilon = \varepsilon_\alpha + \frac{\log(1/\delta)}{\alpha - 1}$. Optimize over all $\alpha$ to get the tightest $(\varepsilon, \delta)$ pair.

For DP-SGD with subsampling (each step samples a fraction $q$ of the data), the RDP of the subsampled Gaussian mechanism is:

$$\varepsilon_\alpha^{\text{subsampled}} \approx \frac{\alpha \, q^2}{2\sigma^2}$$

for small $q$ and moderately large $\sigma$. Compare that to basic composition, which would add $q/\sigma^2$ per step without the $q$ factor improvement — the tighter accounting comes precisely from the subsampling amplification captured inside the Rényi bound.

The moment accountant (Abadi et al., 2016) was an earlier form of the same idea; Mironov's 2017 formalization unified it under the RDP framework, making analysis modular and applicable to arbitrary DP mechanisms beyond the Gaussian.

Rényi DP is not just a cleaner theory — it is the accounting engine behind most real deployments of private machine learning:

• Private language models. Google, Apple, and Meta have reported training or fine-tuning large models with DP-SGD and RDP accounting. Without tighter composition, the stated privacy budgets would be so large as to be meaningless.
• Federated learning. When millions of phones train a shared model without sending raw data to a server, each local update must be private. RDP lets engineers set a meaningful per-round noise level and convert to an annual budget the legal team can audit.
• Medical data analysis. Training diagnostic models on patient records requires a demonstrated privacy guarantee. RDP accounting gives researchers a credible $\varepsilon$ to publish alongside results.
• Census and surveys. The US Census Bureau adopted differential privacy for the 2020 Census. Rényi-based tools let statisticians compose many queries while keeping $\varepsilon$ within a policy-set threshold.

Related ideas from differential privacy and PAC learning meet here: the question is not just whether a model is accurate, but whether it leaks. Rényi DP is the tool that makes the tradeoff honest enough to deploy.

The gap between what differential privacy promises and what practitioners can actually afford to deploy has always been an accounting problem. Basic composition treats every step as worst-case; reality is far kinder.

Rényi differential privacy closes that gap by switching to a divergence measure that composes exactly and converts cleanly to the $(\varepsilon, \delta)$ language that policies and papers speak. Ilya Mironov's 2017 paper gave the field a single, modular framework that subsumed earlier moment accountants and made it straightforward to analyze new mechanisms.

The result is that private training of large neural networks is now a real engineering discipline, not just a theoretical aspiration. Every time you read that a model was trained with differential privacy and a stated $\varepsilon$, there is almost certainly an RDP accountant running the ledger behind the scenes.