Logistic Regression

You check your email and one message is already in the spam folder. A doctor enters a patient's test results and a risk score appears. A bank approves your loan in seconds. Behind every one of these instant decisions sits a surprisingly simple equation — often logistic regression.

The idea is elegant: take a handful of numbers (words in an email, blood pressure, income), multiply each by a learned weight, add them up, and pass the result through an S-shaped curve called the sigmoid. Out comes a number between 0 and 1 — a probability. Above 0.5? Spam. Below? Safe.

What makes logistic regression remarkable is not just its simplicity, but what it guarantees: the algorithm has a single valley to descend, a unique best answer (given the data), and a clear probabilistic meaning. In a field full of black boxes, it is one of the most transparent tools we have.

Click on the canvas to place blue (class 0) or red (class 1) points. Switch between classes with the toggle, then press Train to fit a logistic regression model using gradient descent. The green curve shows the sigmoid decision boundary — the line where the predicted probability is exactly 0.5.

<div class="controls">
  <label class="toggle-label">
    <span>{{placing}}</span>
    <button id="classToggle" type="button" class="class-btn class0">{{class_blue}}</button>
  </label>
  <button id="trainBtn" type="button">{{train}}</button>
  <button id="resetBtn" type="button" class="ghost">{{reset}}</button>
</div>
<canvas id="canvas" width="420" height="320"></canvas>
<div id="info" class="info">{{info_initial}}</div>

* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; margin: 0; color: #222; }
.controls { display: flex; align-items: center; gap: .5rem; flex-wrap: wrap; margin-bottom: .5rem; }
.toggle-label { display: flex; align-items: center; gap: .4rem; font-size: .9rem; }
button { font: 600 13px system-ui; padding: .38rem .8rem; border-radius: 7px; cursor: pointer;
         border: 1px solid #1d3557; background: #1d3557; color: #fff; }
button.ghost { background: #fff; color: #1d3557; }
.class-btn.class0 { background: #3a86ff; border-color: #2a6fd0; }
.class-btn.class1 { background: #e63946; border-color: #c92f3c; }
#canvas { border: 1px solid #cdd9e3; border-radius: 8px; cursor: crosshair; display: block;
          max-width: 100%; }
.info { font-size: .88rem; color: #444; margin-top: .45rem; min-height: 1.3em; }

// Code not found

Notice that with well-separated points the boundary sits cleanly between them. Overlap the classes and the boundary settles somewhere in between — logistic regression never claims certainty, it gives you a probability. Click Reset to start fresh.

Under the hood, logistic regression is an optimization problem — and a particularly well-behaved one.

• The model. For each data point x, the model predicts $$P(y=1 \mid \mathbf{x}) = \sigma(\mathbf{w} \cdot \mathbf{x} + b)$$ where $\sigma(z) = \frac{1}{1+e^{-z}}$ is the sigmoid. The parameters w (weights) and b (bias) are what we learn.
• The loss. Training minimizes the log-loss (also called binary cross-entropy): $$-[y \log p + (1-y) \log(1-p)]$$ summed over all points. This is equivalent to maximum likelihood estimation — finding the parameters most consistent with the data.
• Convexity. The log-loss surface is convex — it has no local minima, only a single global minimum. Gradient descent is therefore guaranteed to converge to the best solution, no matter where it starts.
• Contrast with neural networks. Deep networks optimize non-convex surfaces riddled with saddle points and local minima. Logistic regression's convexity is what makes it so dependable — and why it is still taught first in every ML course.
• Limitations. The sigmoid boundary is always a hyperplane in feature space. Problems with curved or non-linear decision boundaries require either feature engineering (polynomials, interactions) or a more expressive model. Compare with k-means for unsupervised settings, or PAC learning for the theoretical framework behind what can be learned at all.

Few algorithms have touched as many lives as logistic regression. Its interpretability and reliability make it the default choice whenever a probability and a clear explanation are both needed:

• Spam detection: early email filters scored each word's weight and fed the sum through a sigmoid. Many production systems still start here before layering on deep models.
• Medical diagnosis: logistic models predict stroke risk, ICU readmission, and cancer recurrence. The coefficients are directly interpretable as odds ratios — critical when a doctor must explain a prediction to a patient.
• Credit scoring: banks assign loan risk scores using logistic regression trained on payment history and income. Regulators require models they can audit, which keeps logistic regression in wide use.
• Click-through prediction: online advertising systems score millions of impressions per second. A simple logistic model is fast, calibrated, and easy to update — qualities that matter more than raw accuracy at that scale.
• A/B testing and causal inference: the log-odds parameterization makes it natural to control for confounders and estimate the effect of a treatment, bridging statistics and ML.

Logistic regression is also the single-layer special case of a neural network — every deep classifier is, at its output layer, doing exactly what logistic regression does. Understanding it is understanding the foundation of modern machine learning.

Logistic regression is a story about the power of the right abstraction. A single S-shaped curve — the sigmoid — turns any weighted sum into a probability. A single loss function — the log-loss — turns that probability into a convex surface with one global optimum. And gradient descent finds it, every time.

In a field that often chases complexity, logistic regression stands as proof that elegant math beats brute force. It is still the first model deployed in hospitals, banks, and courtrooms, because a coefficient you can explain is worth more than a black-box score you cannot. And when you look inside any deep learning classifier, you will find logistic regression sitting quietly at the very last layer — still doing the job it was designed to do.