The Real Complexity of Logistic Regression

Introduction

You check your email and one message is already in the spam folder. A doctor enters a patient's test results and a risk score appears. A bank approves your loan in seconds. Behind every one of these instant decisions sits a surprisingly simple equation — often logistic regression.

The idea is elegant: take a handful of numbers (words in an email, blood pressure, income), multiply each by a learned weight, add them up, and pass the result through an S-shaped curve called the sigmoid. Out comes a number between 0 and 1 — a probability. Above 0.5? Spam. Below? Safe.

What makes logistic regression remarkable is not just its simplicity, but what it guarantees: the algorithm has a single valley to descend, a unique best answer (given the data), and a clear probabilistic meaning. In a field full of black boxes, it is one of the most transparent tools we have.

Try It

Click on the canvas to place blue (class 0) or red (class 1) points. Switch between classes with the toggle, then press Train to fit a logistic regression model using gradient descent. The green curve shows the sigmoid decision boundary — the line where the predicted probability is exactly 0.5.

<div class="controls">
  <label class="toggle-label">
    <span>Placing:</span>
    <button id="classToggle" type="button" class="class-btn class0">Blue (0)</button>
  </label>
  <button id="trainBtn" type="button">Train</button>
  <button id="resetBtn" type="button" class="ghost">Reset</button>
</div>
<canvas id="canvas" width="420" height="320"></canvas>
<div id="info" class="info">Add points, then click Train.</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
.controls { display: flex; align-items: center; gap: .5rem; flex-wrap: wrap; margin-bottom: .5rem; }
.toggle-label { display: flex; align-items: center; gap: .4rem; font-size: .9rem; }
button { font: 600 13px system-ui; padding: .38rem .8rem; border-radius: 7px; cursor: pointer;
         border: 1px solid #1d3557; background: #1d3557; color: #fff; }
button.ghost { background: #fff; color: #1d3557; }
.class-btn.class0 { background: #3a86ff; border-color: #2a6fd0; }
.class-btn.class1 { background: #e63946; border-color: #c92f3c; }
#canvas { border: 1px solid #cdd9e3; border-radius: 8px; cursor: crosshair; display: block;
          max-width: 100%; }
.info { font-size: .88rem; color: #444; margin-top: .45rem; min-height: 1.3em; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Logistic regression demo: place 2 classes of 2-D points, train with gradient descent on log-loss (binary cross-entropy), draw the sigmoid decision boundary.
const canvas = document.getElementById('canvas');
const ctx    = canvas.getContext('2d');
const W = canvas.width, H = canvas.height;

let currentClass = 0;
let points = []; // {x, y, cls}  x,y in [0,1]

// ── sigmoid ──────────────────────────────────────────────────────────────────
function sigmoid(z) { return 1 / (1 + Math.exp(-z)); }

// ── model params ─────────────────────────────────────────────────────────────
let w1 = 0, w2 = 0, b = 0;  // w = [w1, w2], b = bias

function predict(x, y) { return sigmoid(w1 * x + w2 * y + b); }

// ── training: gradient descent on binary cross-entropy ───────────────────────────────────────────────────────────
function train() {
  if (points.length < 2) { setInfo('Add at least 2 points first.'); return; }
  // Check we have both classes
  const cls0 = points.filter(p => p.cls === 0);
  const cls1 = points.filter(p => p.cls === 1);
  if (!cls0.length || !cls1.length) { setInfo('Add points of BOTH classes.'); return; }

  w1 = 0; w2 = 0; b = 0;
  const lr = 0.5, epochs = 3000;
  for (let e = 0; e < epochs; e++) {
    let g1 = 0, g2 = 0, gb = 0;
    for (const p of points) {
      const pred = predict(p.x, p.y);
      const err  = pred - p.cls;   // gradient of log-loss w.r.t. logit
      g1 += err * p.x;
      g2 += err * p.y;
      gb += err;
    }
    const n = points.length;
    w1 -= lr * g1 / n;
    w2 -= lr * g2 / n;
    b  -= lr * gb / n;
  }

  // ── compute final log-loss ────────────────────────────────────────────────────────
  let loss = 0;
  for (const p of points) {
    const pred = Math.max(1e-12, Math.min(1-1e-12, predict(p.x, p.y)));
    loss -= p.cls * Math.log(pred) + (1 - p.cls) * Math.log(1 - pred);
  }
  loss /= points.length;

  draw();
  setInfo(fmt('Trained %epochs% steps · log-loss = %loss% · w=[%w1%, %w2%] b=%b%', { epochs: epochs, loss: loss.toFixed(4), w1: w1.toFixed(2), w2: w2.toFixed(2), b: b.toFixed(2) }));
  console.log(fmt('logistic-regression: w1=%w1% w2=%w2% b=%b% loss=%loss%', { w1: w1, w2: w2, b: b, loss: loss }));
}

// ── draw ─────────────────────────────────────────────────────────────────────
function draw() {
  ctx.clearRect(0, 0, W, H);

  // background: shade by predicted probability if model is non-trivial
  if (w1 !== 0 || w2 !== 0 || b !== 0) {
    const imgData = ctx.createImageData(W, H);
    for (let px = 0; px < W; px++) {
      for (let py = 0; py < H; py++) {
        const nx = px / W, ny = py / H;
        const p  = predict(nx, ny);
        const i  = (py * W + px) * 4;
        // blue tint for p<0.5, red tint for p>0.5
        if (p < 0.5) {
          const t = 1 - p * 2;  // 0..1
          imgData.data[i]   = Math.round(200 + t * 30);
          imgData.data[i+1] = Math.round(215 + t * 15);
          imgData.data[i+2] = Math.round(240 + t * 15);
          imgData.data[i+3] = 200;
        } else {
          const t = (p - 0.5) * 2;
          imgData.data[i]   = Math.round(240 + t * 15);
          imgData.data[i+1] = Math.round(210 - t * 30);
          imgData.data[i+2] = Math.round(215 - t * 40);
          imgData.data[i+3] = 200;
        }
      }
    }
    ctx.putImageData(imgData, 0, 0);

    // decision boundary: p=0.5 ↔ w1*x + w2*y + b = 0
    ctx.beginPath();
    ctx.strokeStyle = '#0a7d33';
    ctx.lineWidth = 2.5;
    if (Math.abs(w2) > 1e-6) {
      // y = -(w1*x + b) / w2 for x in [0,1]
      const x0 = 0, y0 = -(w1 * x0 + b) / w2;
      const x1 = 1, y1 = -(w1 * x1 + b) / w2;
      ctx.moveTo(x0 * W, y0 * H);
      ctx.lineTo(x1 * W, y1 * H);
    } else if (Math.abs(w1) > 1e-6) {
      const xb = -b / w1;
      ctx.moveTo(xb * W, 0);
      ctx.lineTo(xb * W, H);
    }
    ctx.stroke();
  }

  // points
  for (const p of points) {
    ctx.beginPath();
    ctx.arc(p.x * W, p.y * H, 7, 0, 2 * Math.PI);
    ctx.fillStyle = p.cls === 0 ? '#3a86ff' : '#e63946';
    ctx.fill();
    ctx.strokeStyle = '#fff';
    ctx.lineWidth = 1.5;
    ctx.stroke();
  }
}

function setInfo(t) { document.getElementById('info').textContent = t; }

// ── events ───────────────────────────────────────────────────────────────────
canvas.addEventListener('click', e => {
  const rect = canvas.getBoundingClientRect();
  const scaleX = W / rect.width;
  const scaleY = H / rect.height;
  const px = (e.clientX - rect.left) * scaleX / W;
  const py = (e.clientY - rect.top)  * scaleY / H;
  points.push({ x: px, y: py, cls: currentClass });
  draw();
  setInfo(fmt('%length% point(s) placed. Click Train to fit the model.', { length: points.length }));
});

document.getElementById('classToggle').addEventListener('click', () => {
  currentClass = 1 - currentClass;
  const btn = document.getElementById('classToggle');
  btn.textContent = currentClass === 0 ? 'Blue (0)' : 'Red (1)';
  btn.className = 'class-btn class' + currentClass;
});

document.getElementById('trainBtn').addEventListener('click', train);

document.getElementById('resetBtn').addEventListener('click', () => {
  points = []; w1 = 0; w2 = 0; b = 0;
  draw();
  setInfo('Add points, then click Train.');
});

draw();

<div class="controls">
  <label class="toggle-label">
    <span>Placing:</span>
    <button id="classToggle" type="button" class="class-btn class0">Blue (0)</button>
  </label>
  <button id="trainBtn" type="button">Train</button>
  <button id="resetBtn" type="button" class="ghost">Reset</button>
</div>
<canvas id="canvas" width="420" height="320"></canvas>
<div id="info" class="info">Add points, then click Train.</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
.controls { display: flex; align-items: center; gap: .5rem; flex-wrap: wrap; margin-bottom: .5rem; }
.toggle-label { display: flex; align-items: center; gap: .4rem; font-size: .9rem; }
button { font: 600 13px system-ui; padding: .38rem .8rem; border-radius: 7px; cursor: pointer;
         border: 1px solid #1d3557; background: #1d3557; color: #fff; }
button.ghost { background: #fff; color: #1d3557; }
.class-btn.class0 { background: #3a86ff; border-color: #2a6fd0; }
.class-btn.class1 { background: #e63946; border-color: #c92f3c; }
#canvas { border: 1px solid #cdd9e3; border-radius: 8px; cursor: crosshair; display: block;
          max-width: 100%; }
.info { font-size: .88rem; color: #444; margin-top: .45rem; min-height: 1.3em; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// Logistic regression demo: place 2 classes of 2-D points, train with gradient descent on log-loss (binary cross-entropy), draw the sigmoid decision boundary.
const canvas = document.getElementById('canvas');
const ctx    = canvas.getContext('2d');
const W = canvas.width, H = canvas.height;

let currentClass = 0;
let points = []; // {x, y, cls}  x,y in [0,1]

// ── sigmoid ──────────────────────────────────────────────────────────────────
function sigmoid(z) { return 1 / (1 + Math.exp(-z)); }

// ── model params ─────────────────────────────────────────────────────────────
let w1 = 0, w2 = 0, b = 0;  // w = [w1, w2], b = bias

function predict(x, y) { return sigmoid(w1 * x + w2 * y + b); }

// ── training: gradient descent on binary cross-entropy ───────────────────────────────────────────────────────────
function train() {
  if (points.length < 2) { setInfo('Add at least 2 points first.'); return; }
  // Check we have both classes
  const cls0 = points.filter(p => p.cls === 0);
  const cls1 = points.filter(p => p.cls === 1);
  if (!cls0.length || !cls1.length) { setInfo('Add points of BOTH classes.'); return; }

  w1 = 0; w2 = 0; b = 0;
  const lr = 0.5, epochs = 3000;
  for (let e = 0; e < epochs; e++) {
    let g1 = 0, g2 = 0, gb = 0;
    for (const p of points) {
      const pred = predict(p.x, p.y);
      const err  = pred - p.cls;   // gradient of log-loss w.r.t. logit
      g1 += err * p.x;
      g2 += err * p.y;
      gb += err;
    }
    const n = points.length;
    w1 -= lr * g1 / n;
    w2 -= lr * g2 / n;
    b  -= lr * gb / n;
  }

  // ── compute final log-loss ────────────────────────────────────────────────────────
  let loss = 0;
  for (const p of points) {
    const pred = Math.max(1e-12, Math.min(1-1e-12, predict(p.x, p.y)));
    loss -= p.cls * Math.log(pred) + (1 - p.cls) * Math.log(1 - pred);
  }
  loss /= points.length;

  draw();
  setInfo(fmt('Trained %epochs% steps · log-loss = %loss% · w=[%w1%, %w2%] b=%b%', { epochs: epochs, loss: loss.toFixed(4), w1: w1.toFixed(2), w2: w2.toFixed(2), b: b.toFixed(2) }));
  console.log(fmt('logistic-regression: w1=%w1% w2=%w2% b=%b% loss=%loss%', { w1: w1, w2: w2, b: b, loss: loss }));
}

// ── draw ─────────────────────────────────────────────────────────────────────
function draw() {
  ctx.clearRect(0, 0, W, H);

  // background: shade by predicted probability if model is non-trivial
  if (w1 !== 0 || w2 !== 0 || b !== 0) {
    const imgData = ctx.createImageData(W, H);
    for (let px = 0; px < W; px++) {
      for (let py = 0; py < H; py++) {
        const nx = px / W, ny = py / H;
        const p  = predict(nx, ny);
        const i  = (py * W + px) * 4;
        // blue tint for p<0.5, red tint for p>0.5
        if (p < 0.5) {
          const t = 1 - p * 2;  // 0..1
          imgData.data[i]   = Math.round(200 + t * 30);
          imgData.data[i+1] = Math.round(215 + t * 15);
          imgData.data[i+2] = Math.round(240 + t * 15);
          imgData.data[i+3] = 200;
        } else {
          const t = (p - 0.5) * 2;
          imgData.data[i]   = Math.round(240 + t * 15);
          imgData.data[i+1] = Math.round(210 - t * 30);
          imgData.data[i+2] = Math.round(215 - t * 40);
          imgData.data[i+3] = 200;
        }
      }
    }
    ctx.putImageData(imgData, 0, 0);

    // decision boundary: p=0.5 ↔ w1*x + w2*y + b = 0
    ctx.beginPath();
    ctx.strokeStyle = '#0a7d33';
    ctx.lineWidth = 2.5;
    if (Math.abs(w2) > 1e-6) {
      // y = -(w1*x + b) / w2 for x in [0,1]
      const x0 = 0, y0 = -(w1 * x0 + b) / w2;
      const x1 = 1, y1 = -(w1 * x1 + b) / w2;
      ctx.moveTo(x0 * W, y0 * H);
      ctx.lineTo(x1 * W, y1 * H);
    } else if (Math.abs(w1) > 1e-6) {
      const xb = -b / w1;
      ctx.moveTo(xb * W, 0);
      ctx.lineTo(xb * W, H);
    }
    ctx.stroke();
  }

  // points
  for (const p of points) {
    ctx.beginPath();
    ctx.arc(p.x * W, p.y * H, 7, 0, 2 * Math.PI);
    ctx.fillStyle = p.cls === 0 ? '#3a86ff' : '#e63946';
    ctx.fill();
    ctx.strokeStyle = '#fff';
    ctx.lineWidth = 1.5;
    ctx.stroke();
  }
}

function setInfo(t) { document.getElementById('info').textContent = t; }

// ── events ───────────────────────────────────────────────────────────────────
canvas.addEventListener('click', e => {
  const rect = canvas.getBoundingClientRect();
  const scaleX = W / rect.width;
  const scaleY = H / rect.height;
  const px = (e.clientX - rect.left) * scaleX / W;
  const py = (e.clientY - rect.top)  * scaleY / H;
  points.push({ x: px, y: py, cls: currentClass });
  draw();
  setInfo(fmt('%length% point(s) placed. Click Train to fit the model.', { length: points.length }));
});

document.getElementById('classToggle').addEventListener('click', () => {
  currentClass = 1 - currentClass;
  const btn = document.getElementById('classToggle');
  btn.textContent = currentClass === 0 ? 'Blue (0)' : 'Red (1)';
  btn.className = 'class-btn class' + currentClass;
});

document.getElementById('trainBtn').addEventListener('click', train);

document.getElementById('resetBtn').addEventListener('click', () => {
  points = []; w1 = 0; w2 = 0; b = 0;
  draw();
  setInfo('Add points, then click Train.');
});

draw();

Result

Console

Notice that with well-separated points the boundary sits cleanly between them. Overlap the classes and the boundary settles somewhere in between — logistic regression never claims certainty, it gives you a probability. Click Reset to start fresh.

The Real Complexity

Under the hood, logistic regression is an optimization problem — and a particularly well-behaved one.

The model. For each data point x, the model predicts $ $P(y=1 \mid \mathbf{x}) = \sigma(\mathbf{w} \cdot \mathbf{x} + b)$ $ where $\sigma(z) = \frac{1}{1+e^{-z}}$ is the sigmoid. The parameters w (weights) and b (bias) are what we learn.
The loss. Training minimizes the log-loss (also called binary cross-entropy): $ $-[y \log p + (1-y) \log(1-p)]$ $ summed over all points. This is equivalent to maximum likelihood estimation — finding the parameters most consistent with the data.
Convexity. The log-loss surface is convex — it has no local minima, only a single global minimum. Gradient descent is therefore guaranteed to converge to the best solution, no matter where it starts.
Contrast with neural networks. Deep networks optimize non-convex surfaces riddled with saddle points and local minima. Logistic regression's convexity is what makes it so dependable — and why it is still taught first in every ML course.
Limitations. The sigmoid boundary is always a hyperplane in feature space. Problems with curved or non-linear decision boundaries require either feature engineering (polynomials, interactions) or a more expressive model. Compare with k-means for unsupervised settings, or PAC learning for the theoretical framework behind what can be learned at all.

Where It Matters

Few algorithms have touched as many lives as logistic regression. Its interpretability and reliability make it the default choice whenever a probability and a clear explanation are both needed:

Spam detection: early email filters scored each word's weight and fed the sum through a sigmoid. Many production systems still start here before layering on deep models.
Medical diagnosis: logistic models predict stroke risk, ICU readmission, and cancer recurrence. The coefficients are directly interpretable as odds ratios — critical when a doctor must explain a prediction to a patient.
Credit scoring: banks assign loan risk scores using logistic regression trained on payment history and income. Regulators require models they can audit, which keeps logistic regression in wide use.
Click-through prediction: online advertising systems score millions of impressions per second. A simple logistic model is fast, calibrated, and easy to update — qualities that matter more than raw accuracy at that scale.
A/B testing and causal inference: the log-odds parameterization makes it natural to control for confounders and estimate the effect of a treatment, bridging statistics and ML.

Logistic regression is also the single-layer special case of a neural network — every deep classifier is, at its output layer, doing exactly what logistic regression does. Understanding it is understanding the foundation of modern machine learning.

Conclusion

Logistic regression is a story about the power of the right abstraction. A single S-shaped curve — the sigmoid — turns any weighted sum into a probability. A single loss function — the log-loss — turns that probability into a convex surface with one global optimum. And gradient descent finds it, every time.

In a field that often chases complexity, logistic regression stands as proof that elegant math beats brute force. It is still the first model deployed in hospitals, banks, and courtrooms, because a coefficient you can explain is worth more than a black-box score you cannot. And when you look inside any deep learning classifier, you will find logistic regression sitting quietly at the very last layer — still doing the job it was designed to do.

Comments

Loading comments...

https://www.kipuhub.com/en/article/logistic-regression/Content licensed under CC BY-NC 4.0.

Cookies & privacy

Logistic Regression

How a single S-curve learned to classify almost everything