Logistic Regression and Classification Metrics
How logistic regression turns a score into a probability, why accuracy can look great while the model finds almost nothing, and how precision, recall, F1 and the decision threshold really behave.
Logistic regression predicts the probability of a class by passing a weighted sum of features through the sigmoid function, 1 / (1 + e^(-z)), then applies a threshold, 0.5 by default, to pick the label. The detail interviewers expect: on imbalanced data accuracy misleads, so report precision, recall and F1 and choose the threshold by the cost of each error.
Logistic regression predicts the probability of a class by passing a linear score through the sigmoid function, then applies a threshold to decide the label. Accuracy is misleading when classes are imbalanced, so use the confusion matrix, precision, recall and F1, and choose the threshold to suit the cost of each kind of mistake.
- Logistic regression passes a linear score z through the sigmoid 1 / (1 + e^(-z)), mapping it to a probability between 0 and 1, and is trained by minimising log loss.
- Precision is TP / (TP + FP), the share of positive predictions that were right, and recall is TP / (TP + FN), the share of real positives found.
- On a test set with 3% positives, a logistic regression model scored 96.87% accuracy while finding only 1 of 45 real positives, a recall of 2.2%.
- Lowering a logistic regression threshold from 0.5 to 0.1 raised recall from 2.2% to 46.7% on an imbalanced test set, while precision fell to 0.21.
- F1 is the harmonic mean of precision and recall, so it is high only when both precision and recall are high.
01From a line to a probability
Linear regression outputs any real number, which is no use for "will this customer churn, yes or no". Logistic regression computes the same kind of weighted sum, z = w1 x1 + w2 x2 + ... + b, and passes it through the sigmoid function: probability = 1 / (1 + e^(-z)).
The sigmoid maps any z to a number between 0 and 1. Large positive z gives a probability near 1, large negative gives near 0, and z = 0 gives exactly 0.5. The model is trained by minimising log loss (cross-entropy), which punishes a confident wrong answer very heavily.
Despite the name, it is a classification method. To get a label you apply a threshold, 0.5 by default: predict class 1 if the probability is at least the threshold.
02The confusion matrix
For a yes/no problem, comparing predictions with the truth gives four outcomes. A true positive (TP) is a correct yes. A true negative (TN) is a correct no. A false positive (FP) is a wrong yes, such as a spam filter blocking a real email. A false negative (FN) is a wrong no, such as a disease screening missing a sick patient.
From these:
Accuracy = (TP + TN) / everything.
Precision = TP / (TP + FP). Of the items the model called positive, how many were right.
Recall = TP / (TP + FN). Of the real positives, how many the model found.
F1 = the harmonic mean of precision and recall, which is high only when both are.
03The accuracy trap, with real numbers
I generated 5,000 samples where only 3% are positive (think fraud), trained logistic regression, and evaluated on a held-out 1,500:
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import (confusion_matrix, accuracy_score,
precision_score, recall_score, f1_score)
X, y = make_classification(n_samples=5000, n_features=10, n_informative=4,
weights=[0.97, 0.03], flip_y=0, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
clf = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
pred = clf.predict(Xte)
print(accuracy_score(yte, pred)) # 0.9687
print(recall_score(yte, pred)) # 0.022
print(confusion_matrix(yte, pred)) # [[1452 3]
# [ 44 1]]The test set has 45 real positives. Accuracy is 96.87%, which sounds excellent, but a model that always says "negative" scores 97% here. Look at the matrix: of 45 real positives, the model found exactly one (recall 2.2%). Precision is 1 right out of 4 positive calls, 25%. The accuracy number was hiding a model that is nearly useless for the thing you care about.
(The exact figures depend on the scikit-learn version, here 1.9, but the lesson does not.) Whenever the classes are imbalanced, report precision, recall or F1 for the minority class, and always compare with the trivial baseline.
04Moving the threshold
proba = clf.predict_proba(Xte)[:, 1]
for th in (0.5, 0.3, 0.1):
p = (proba >= th).astype(int)
print(th, round(precision_score(yte, p), 3), round(recall_score(yte, p), 3))
# 0.5 precision 0.25 recall 0.022
# 0.3 precision 0.375 recall 0.2
# 0.1 precision 0.21 recall 0.467Lowering the threshold makes the model say "positive" more readily. Recall climbs from 2% to 47% at 0.1. Precision is noisier: at 0.5 the model made only 4 positive calls (1 right), so 0.25 rests on almost nothing. It rises to 0.375 at 0.3, then falls to 0.21 at 0.1 as false alarms pile up (79 false positives against 21 true positives). That is the precision-recall trade-off, and no setting wins both.
Choose by cost. For cancer screening, a missed case is far worse than a false alarm, so favour recall. For a system that automatically blocks accounts, a false alarm hurts a real user, so favour precision. Plot a precision-recall curve over all thresholds rather than guessing, and use ROC-AUC or average precision to compare models without fixing one threshold.
05Other ways to handle imbalance
Set class_weight='balanced' so mistakes on the rare class cost more during training. Resample: oversample the minority class (with care, such as SMOTE, which creates synthetic examples) or undersample the majority, but only inside the training split so no copies leak into the test set. Collect more of the rare class if you can. And evaluate with a metric that matches the goal, not plain accuracy.
A second warning: a probability from logistic regression is only meaningful if the model is calibrated. Check with a calibration curve before telling a business that a case has a "70% chance".
06Common interview questions
Q: Why is accuracy a poor metric for imbalanced data?
A: Predicting the majority class always scores high. 97% accuracy can come with 2% recall on the class that matters.
Q: Precision versus recall?
A: Precision asks how many predicted positives are real. Recall asks how many real positives were found. Raising one usually lowers the other.
Q: When would you prefer recall?
A: When a missed positive is costly, such as disease screening or fraud detection.
Q: What is the sigmoid for?
A: It maps any real-valued score to a probability between 0 and 1.
Q: What is ROC-AUC?
A: The area under the curve of true positive rate against false positive rate across all thresholds. It is the probability that the model ranks a random positive above a random negative.
Q: Is logistic regression a linear model?
A: Yes, the decision boundary is a straight line (or plane). The sigmoid only changes how the score is read.