Müşteri Kaybı (Churn) Tahmini: Modelden Arama Listesine

Building a churn model is not the hard part. The hard part is turning the probability it produces into an answer to "who do we call this week?" In this post I build a churn prediction end to end on 7,043 telecom customers — and show with numbers why the model's accuracy is a misleading measure of success.

The data is IBM's Telco Customer Churn set, a standard in churn work: 21 columns, one row per customer, and a target of whether the customer left.

“A churn model's value does not show up in how many customers it classifies correctly, but in how many departing customers it can reach with a limited number of calls.”

1. The Problem and the Base Rate

One number has to come first: 1,869 of the 7,043 customers churned, a rate of 26.5%. That number sets the floor for every evaluation that follows.

2. Where Does Churn Concentrate?

Looking at the data before modelling already reveals most of what the model will later find. Four breakdowns are enough.

Month-to-month customers churn at 15 times the rate of two-year contracts. A split that sharp in a single variable is rare; we can already expect the model to lean on this column.

All three tables point the same way: the riskiest profile is a loosely committed, month-to-month customer without support. That customers paying by electronic check churn at three times the rate of those on automatic payment is especially striking — payment method behaves here like a proxy for intent.

The first six months are critical: more than half of new customers leave. Risk falls monotonically with tenure, down to 9.5% past 49 months. That alone says where a retention budget should go.

3. Is "Missing" Data Really Missing?

The TotalCharges column has 11 blank cells. The reflex is to fill them with the median or drop those rows. The better first move is to ask why they are empty.

df["TotalCharges"] = pd.to_numeric(
    df["TotalCharges"].replace(r"^\s*$", pd.NA, regex=True),
    errors="coerce")

df.loc[df["TotalCharges"].isna(), "tenure"].unique()
# array([0])  -> every one of them has zero tenure

All eleven customers have a tenure of zero. These are brand-new subscriptions that have not been billed yet; their total charges are not missing, they do not exist yet. Filling them with the median would invent a history these customers never had.

4. Three Models, One Test Set

Numeric variables are standardised, categoricals one-hot encoded, and three models run through the same pipeline. For the class imbalance I use class_weight="balanced", which makes the model treat the minority class as more expensive to miss.

preprocess = ColumnTransformer([
    ("num", Pipeline([("i", SimpleImputer(strategy="median")),
                      ("s", StandardScaler())]), numeric),
    ("cat", Pipeline([("i", SimpleImputer(strategy="most_frequent")),
                      ("o", OneHotEncoder(handle_unknown="ignore"))]), categorical),
])

X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

for name, model in models.items():
    p = Pipeline([("prep", preprocess), ("model", model)]).fit(X_tr, y_tr)
    probability = p.predict_proba(X_te)[:, 1]

The most telling row is the last one. Logistic regression's 73.8% accuracy is just 0.3 points above the 73.5% of a model that does nothing. Anyone judging this model by accuracy would conclude it does not work. Yet its ROC-AUC is 0.841 and its recall 0.783 — the model catches more than three quarters of the customers who actually leave.

5. The Threshold Is an Assumption

Every metric above rests on one decision: call it churn when the probability passes 0.5. That cut-off comes from habit, not from the mathematics. Changing it does not change the model, only how its probabilities become decisions.

What picks the threshold is not statistics but team capacity and cost. On a list of 1,409 customers, a team able to call 800 a month covers 92.8% of the churn at a threshold of 0.3. With capacity for 400 calls, 0.7 is the realistic choice: 60.2% of churners are caught, but roughly every second call lands on a real one.

6. What Does the Model Say?

The best thing about logistic regression is that it can be read. Converting its coefficients to odds ratios makes each variable's direction and size visible directly.

Permutation importance gives the same ordering: tenure alone costs 0.152 of ROC-AUC when shuffled, followed by InternetService (0.043), Contract (0.040) and MonthlyCharges (0.038). Three quarters of what the model knows sits in four variables.

7. Turning the Model Into Business Value

The original question was: who do we call this week? Accuracy does not answer it; lift does. Sort customers by risk score, take the top slice, and look at how many of them actually churned.

This table is far more useful than the accuracy one. Call the riskiest 10% and three out of every four customers are genuinely about to leave. Calling at random would hit 26.5%; the model makes each call 2.85 times more likely to land. With capacity to call half the base, you reach 87% of the churn.

Three of the four steps are technical and one is not. Scoring and ranking are the model's job; where the list gets cut is set by the team's weekly capacity. The fourth step is purely a matter of design: leaving a randomly chosen share of flagged customers untouched.

8. Common Mistakes

Conclusion

All three models landed close together on this data: ROC-AUC between 0.826 and 0.841. Model selection is not the hard part of this problem. The hard part is attaching the resulting probability to the right question.

Someone looking at accuracy would throw the model away — 73.8%, a mere 0.3 points above doing nothing. Someone looking at the lift table sees this instead: in the riskiest 10%, the churn rate is 75.7%, 2.85 times better than calling at random. Same model, same numbers; the only difference is the question being asked.

“The model tells you who is going to leave. What tells you who to call is the decision that joins the model's output to the team's capacity.”