ROC Curve ve AUC Nedir? Sınıflandırma Modellerini Karşılaştırma

What does a ROC curve show?

A binary classifier does not really produce "yes/no" directly. For every observation it gives a score or probability for the positive class. The "yes" decision appears only when you compare that score against a threshold. Lower the threshold from 0.5 to 0.3 and the model becomes more generous: it catches more true positives, but it also raises more false alarms.

The ROC (Receiver Operating Characteristic) curve is the map of that trade-off. For every possible threshold, two rates are computed and plotted as a point.

Keep the threshold very high and almost nothing is selected as positive; both rates approach zero and the curve starts in the bottom-left corner. Lower it all the way and everything counts as positive; both rates reach 1 and the curve ends in the top-right corner. The path in between shows how well the model ranks positives above negatives.

Working one threshold out by hand

To keep it concrete: suppose we have 100 true positives and 900 true negatives. At a particular threshold the model produces this.

From this, TPR = 80 / 100 = 0.80 and FPR = 90 / 900 = 0.10. That threshold is the point (0.10, 0.80) on the ROC curve. Drop the threshold a little further and the point moves right and up: you catch more sick patients and you needlessly alarm more healthy ones.

One thing worth noticing: 90 false positives looks like a mere 10%. But in absolute terms that is 90 people — 90 extra phone calls, 90 extra reviews in a real operation. A small rate does not mean a small workload.

What does AUC mean?

AUC (Area Under the Curve) is the area beneath the ROC curve, collapsed into one number. Its most useful reading is this: how often does a randomly chosen positive example score higher than a randomly chosen negative one? Ties count as a half.

So AUC = 0.85 does not mean "the model is 85% accurate". It means "in a random positive–negative pair, the positive one scores higher 85% of the time". That distinction matters, because AUC is not an accuracy measure — it is a measure of ranking quality.

I want to underline that last row. In a real project, an AUC above 0.99 should make you suspicious rather than pleased. The usual explanation is that a derivative of the target leaked into the features, or that the same record appears in both the training and test sets.

These values are also not universal quality labels. "Good AUC" is a different number in medical screening than in ad click prediction; the domain, the data quality and the cost of errors decide.

ROC and AUC in Python

The example below trains a logistic regression on scikit-learn's breast cancer dataset. In that dataset class 1 corresponds to the benign label; stating the positive class explicitly prevents readings that come out backwards later.

import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.25,
    stratify=data.target, random_state=42
)

model = make_pipeline(
    StandardScaler(),
    LogisticRegression(max_iter=2000)
).fit(X_train, y_train)

# Column 1 belongs to model.classes_[1]
y_score = model.predict_proba(X_test)[:, 1]

fpr, tpr, thresholds = roc_curve(y_test, y_score, pos_label=1)
auc = roc_auc_score(y_test, y_score)
print(f"AUC: {auc:.3f}")    # AUC: 0.998

plt.plot(fpr, tpr, label=f"Logistic regression (AUC={auc:.3f})")
plt.plot([0, 1], [0, 1], "--", color="gray", label="Random")
plt.xlabel("False positive rate (FPR)")
plt.ylabel("True positive rate (TPR)")
plt.legend(); plt.grid(alpha=0.25); plt.show()

On this dataset the AUC comes out around 0.998 — precisely in the region I just warned about. Here the explanation is innocent: this set was chosen as a textbook example and its classes really do separate cleanly. Do not accept the same number as readily in your own project.

Comparing two models

The rule for model comparison is simple but often broken: same test set, same positive-class definition, same preprocessing. Change any one of those and the comparison is void.

from sklearn.ensemble import RandomForestClassifier

models = {
    "Logistic regression": model,
    "Random Forest": RandomForestClassifier(
        n_estimators=300, random_state=42
    ).fit(X_train, y_train),
}

for name, clf in models.items():
    score = clf.predict_proba(X_test)[:, 1]
    print(f"{name}: AUC={roc_auc_score(y_test, score):.3f}")

# Logistic regression: AUC=0.998
# Random Forest:       AUC=0.995

The gap between 0.998 and 0.995 here comes from a single test split of 143 observations. Reporting a difference that small as "logistic regression is better" would be misleading. On small datasets, use cross-validation instead of one split and report both the mean and the spread.

Another subtlety: two ROC curves can cross. The model with the larger overall area may be worse in the low-FPR region where you actually plan to operate. Plot both curves on the same axes and look at the range you care about.

Threshold selection is a separate decision

A high AUC does not make a model work in the field; you still have to pick the right threshold. And that choice is not statistical but a business decision: what does each kind of error cost?

That last row is very practical. If your call centre can make 200 calls a day, the answer to "what should the threshold be?" is really "take the 200 highest-scoring customers". The threshold becomes a consequence of capacity.

With class imbalance, ROC can mislead

The denominator of FPR contains the true negatives. When the negative class is huge, that denominator inflates and FPR looks smaller than it is. A concrete example: 1,000,000 transactions contain 1,000 frauds. Your model runs at FPR = 0.01, which sounds excellent. But that is 1% of 999,000 clean transactions — roughly 9,990 false alarms. Even if you catch every single fraud, nine out of ten items on your review list will be empty.

That is why, with a rare positive class, the Precision–Recall curve and Average Precision (AP) are more informative. The PR curve directly exposes the question "of the things I called positive, how many really were?" and is not distorted by the size of the negative class the way ROC is.

What AUC does not tell you

Calibration is the one most often overlooked. If you plan to feed a score straight into an expected-value calculation (say, "this customer's churn probability × lifetime value"), ranking is not enough; the probability has to be a real probability. Use a calibration curve or the Brier score for that.

A practical evaluation workflow

1. Define the positive class in business terms and write down the cost of each error type.
2. Separate training, validation and test splits; if there is a time dimension, split chronologically.
3. Take the probability or decision score, not the labels.
4. Plot the ROC curve and compute AUC on held-out data.
5. Choose the threshold on validation data according to the business goal.
6. On the test set, report the confusion matrix, precision, recall and absolute error counts.

Conclusion

A ROC curve shows the balance between the positives a classifier catches and the false alarms it raises at different thresholds. AUC reduces that curve to a single number and summarises the model's ranking quality — not its accuracy.

In practice, the most useful attitude is to treat AUC as a starting point. Use it while selecting a model; when you deploy, talk about the threshold, the absolute error counts and the confusion matrix at that threshold. If the classes are imbalanced, add the PR curve. A single number should never be the whole decision.