R ile Sınıflandırma: Lojistik Regresyon, SVM, CART ve Rastgele Orman

Four classification algorithms, one data set, one test set. The aim of this post is not to explain each algorithm — I covered them individually elsewhere on the site. The question here is: on the same problem, what does each one do, and why?

In this third post of the series I fit logistic regression, a support vector machine, a decision tree and a random forest on the Carseats data, and compare all four on the same test set with the same metrics.

“Comparing models takes more than training them on the same data; they also have to be measured on the same data.”

1. The Problem and the Base Rate

Sales in Carseats is continuous. To turn this into a classification problem we split it at 8 thousand units: stores above 8 are High, the rest are Low.

df <- Carseats
df$Sales <- factor(ifelse(df$Sales <= 8, "Low", "High"),
                   levels = c("Low", "High"))

table(df$Sales)        # Low 236  High 164

set.seed(5364)
idx <- unlist(lapply(levels(df$Sales), function(lv) {
  rows <- which(df$Sales == lv)
  sample(rows, size = floor(0.8 * length(rows)))
}))

train <- df[idx, ]    # 319 rows: Low 188, High 131
test  <- df[-idx, ]   # 81 rows:  Low  48, High  33

2. Logistic Regression

We start with the most interpretable option. Logistic regression passes a linear combination through the sigmoid function to produce a probability.

set.seed(86451)
glm_fit <- glm(Sales ~ ., data = train, family = "binomial")

# "High" is the second level, so the model predicts P(High)
prob <- predict(glm_fit, newdata = test, type = "response")
pred <- factor(ifelse(prob > 0.5, "High", "Low"),
               levels = c("Low", "High"))

table(Prediction = pred, Actual = test$Sales)

Coefficients:
                 Estimate Std. Error z value Pr(>|z|)
(Intercept)     -5.282068   2.933818  -1.800   0.0718 .
CompPrice        0.192240   0.029544   6.507 7.68e-11 ***
Income           0.030561   0.009737   3.139   0.0017 **
Advertising      0.343855   0.064202   5.356 8.52e-08 ***
Price           -0.188692   0.026197  -7.203 5.90e-13 ***
ShelveLocGood    9.076140   1.282961   7.074 1.50e-12 ***
ShelveLocMedium  4.229255   0.805378   5.251 1.51e-07 ***
Age             -0.088403   0.018244  -4.846 1.26e-06 ***

    Null deviance: 431.99  on 318  degrees of freedom
Residual deviance: 126.34  on 307  degrees of freedom
AIC: 150.34

Deviance falls from 431.99 to 126.34: the model gains substantial explanatory power over the null model. The coefficients tell the same story as the regression post — a Good shelving location raises the log-odds of high sales by 9.08.

Accuracy 0.864 · Sensitivity 0.849 · Specificity 0.875 · F1 0.836 · AUC 0.929

Where does the 0.5 threshold come from?

The confusion matrix above rests on a single decision: call it "High" when the probability passes 0.5. That threshold does not come from the mathematics — it is an assumption. Changing it does not change the model, only how you turn the model's probabilities into decisions.

p <- predict(glm_fit, newdata = test, type = "response")

for (cut in c(0.3, 0.4, 0.5, 0.6, 0.7)) {
  pred <- factor(ifelse(p > cut, "High", "Low"),
                 levels = c("Low", "High"))
  print(table(pred, test$Sales))
}

# Best threshold by Youden's J:
coords(roc(test$Sales, p), "best", best.method = "youden")
# threshold = 0.5924

A threshold of 0.6 lifts accuracy from 0.864 to 0.889 — three more stores classified correctly, with no loss in sensitivity. Youden's J on the ROC curve also puts the optimum at 0.592 rather than 0.5.

Where you put it is ultimately a question of cost. If missing a high-potential store costs more than investing in the wrong one, the threshold comes down; if the reverse holds, it goes up.

3. Support Vector Machines

An SVM looks for the boundary that separates the two classes with the widest margin. We start with a linear kernel.

library(e1071)

set.seed(144)
svm_fit <- svm(Sales ~ ., data = train,
               kernel = "linear", cost = 1,
               scale = TRUE, type = "C-classification")

pred_svm <- predict(svm_fit, newdata = test)
table(Prediction = pred_svm, Actual = test$Sales)

svm_fit$tot.nSV   # 89 support vectors

Accuracy 0.877 · Sensitivity 0.849 · Specificity 0.896 · F1 0.849 — the highest accuracy of the four.

cost and kernel: should we keep the defaults?

The model above was fitted with cost = 1 and a linear kernel — both default choices. cost controls how much margin violation the SVM tolerates: a small value buys a wide but forgiving margin, a large value a narrow and strict boundary. The right value depends on the data, so it has to be searched for.

set.seed(144)
t1 <- tune(svm, Sales ~ ., data = train, kernel = "linear",
          ranges = list(cost = c(0.01, 0.1, 1, 5, 10)))
t1$performances

t2 <- tune(svm, Sales ~ ., data = train, kernel = "radial",
          ranges = list(cost = c(0.1, 1, 5, 10),
                        gamma = c(0.01, 0.05, 0.1)))
t2$best.parameters   # cost = 5, gamma = 0.05

The best cost is 0.1 rather than 1 — but the gap is only 0.006 against a fold-to-fold standard deviation of 0.057. So the search is not telling us to change cost; it is telling us that cost barely matters across a wide range in this dataset. The one exception is 0.01: too loose a margin clearly degrades the model.

The radial kernel can draw non-linear boundaries and is usually treated as the "stronger" option. Here its cross-validated error is all but identical (0.1066 against 0.1065) and its test accuracy is five points lower. It also leans on 128 support vectors instead of 89 — doing the same job with a more complicated model.

4. Decision Tree (CART)

CART splits the data through successive binary partitions. Its biggest advantage is readability: you can see the model as a series of if–then rules.

library(rpart)
library(rpart.plot)

set.seed(144)
cart_fit <- rpart(Sales ~ ., data = train, method = "class")

# The cp that minimizes cross-validated error
printcp(cart_fit)
best_cp <- cart_fit$cptable[which.min(cart_fit$cptable[, "xerror"]), "CP"]

cart_pruned <- prune(cart_fit, cp = best_cp)
rpart.plot(cart_pruned)

          CP nsplit rel error    xerror       xstd
1 0.32061069      0 1.0000000 1.0000000 0.06707303
2 0.07633588      1 0.6793893 0.6793893 0.06114941
3 0.05343511      2 0.6030534 0.7557252 0.06307573
4 0.03816794      4 0.4961832 0.6335878 0.05981752
5 0.02798982      6 0.4198473 0.5877863 0.05834268
6 0.02290076      9 0.3358779 0.5648855 0.05754827   <- lowest xerror
7 0.01000000     11 0.2900763 0.5725191 0.05781745

Two columns tell the story: rel error is the training error and keeps falling as the tree grows (1.00 → 0.29). xerror is the cross-validated error, and after nine splits it starts rising again. That is where overfitting begins. We take the cp = 0.0229 with the lowest xerror and prune the tree to 10 leaves.

Accuracy 0.741 · Sensitivity 0.667 · Specificity 0.792 · F1 0.677 · AUC 0.769 — the weakest of the four.

5. Random Forest

A random forest trains hundreds of decision trees on different samples and variable subsets, then combines their votes. It largely removes the instability of a single tree.

library(randomForest)

set.seed(144)
rf_fit <- randomForest(Sales ~ ., data = train, importance = TRUE)
rf_fit

# OOB estimate of error rate: 17.55%
# Confusion matrix (OOB):
#      Low High class.error
# Low  171   17   0.0904
# High  39   92   0.2977

pred_rf <- predict(rf_fit, newdata = test)
importance(rf_fit)[, "MeanDecreaseGini"]

The out-of-bag (OOB) error is a random forest's built-in cross-validation: each tree is tested on the observations left out while it was trained. The 17.55% OOB error here is an estimate obtained without setting aside a separate validation set.

The importance ranking matches what the regression model said: price and shelving location are far ahead. Three different methods highlighting the same two variables suggests the finding belongs to the data, not to the method.

Accuracy 0.778 · Sensitivity 0.606 · Specificity 0.896 · F1 0.690 · AUC 0.878

6. Four Models, One Table

Every model was trained on the same training set and measured on the same 81-observation test set. The positive class is "High".

7. Why Did the Simple Models Win?

The expectation usually runs the other way: a random forest is supposed to beat logistic regression. Here the opposite happened, and the reason lies in the data.

The random forest's profile is also notable: its specificity of 0.896 matches the SVM, but its sensitivity is 0.606. It finds low-selling stores well and misses a third of the high-selling ones. If the goal is "identify high-potential stores", this model may be the worst business choice despite ranking second on accuracy.

8. Three Pitfalls in Model Comparison

The third applies to this study too: the 0.013 gap between the SVM and logistic regression amounts to a single store in an 81-observation test set. Declaring "the SVM is better" on that basis would be wrong. A real comparison needs repeated cross-validation.

library(caret)

ctrl <- trainControl(method = "repeatedcv", number = 10, repeats = 5,
                     classProbs = TRUE, summaryFunction = twoClassSummary)

models <- lapply(c("glm", "svmLinear", "rpart", "rf"), function(m) {
  set.seed(144)
  train(Sales ~ ., data = train, method = m,
        metric = "ROC", trControl = ctrl)
})

summary(resamples(setNames(models, c("glm", "svm", "cart", "rf"))))

9. Is the Difference Real? Repeated Cross-Validation

The previous section closed on a question: is the 0.013 gap between SVM and logistic regression real, or noise? There are two ways to answer it, and both point the same way.

First answer: McNemar's test

When two models are measured on the same test set, comparing their accuracies as if they were independent samples is wrong — both are predicting the same 81 stores. The right tool is McNemar's test, built for paired data. The question it asks is: are the stores one model gets right and the other gets wrong balanced between them?

right_svm <- pred_svm == test$Sales
right_glm <- pred_glm == test$Sales

mcnemar.test(table(right_svm, right_glm))
# McNemar's chi-squared = 0, df = 1, p-value = 1

The numbers settle the table: SVM and logistic regression disagree on exactly one of the 81 stores. What puts SVM at the top of the accuracy ranking is a single store. There is no evidence to stand behind the sentence "SVM is better". The gap between SVM and CART (p = 0.029), by contrast, is real.

Second answer: 5 × 10-fold cross-validation

A single split is a single photograph. Splitting the data into 10 folds and repeating that 5 times with different seeds gives 50 measurements per model — enough to see how stable the ranking actually is.

# stratified 10 folds per repeat; 4 models x 50 folds = 200 fits
for (rep in 1:5) {
  set.seed(1000 + rep)
  # ... split into 10 folds preserving class ratios ...
  for (k in 1:10) {
    tr <- df[fold != k, ]; va <- df[fold == k, ]
    # glm / svm / rpart / randomForest are fitted and scored
  }
}

The ranking flips. SVM was first on the single split; averaged over 50 folds, logistic regression leads and wins 40 of them. A paired t-test finds the difference significant: mean gap 0.0071, p = 0.024. Small, but consistent.

Further down the table the picture is sharper: the 0.071 gap between SVM and random forest is significant at p < 0.00001. So the claim that linear models beat the ensembles on this data is not the whim of one split — it holds across 50 folds.

Conclusion

Four algorithms were trained on the same data and measured on the same test set, and the result ran against expectation: the linear models beat the ensembles. That is why the assumption "a random forest is always better" has to be tested rather than believed.

But the real lesson is not inside a single table — it is in questioning that table. The test-set ranking put SVM first; McNemar's test showed that lead amounted to one store, and 50-fold cross-validation put logistic regression ahead on average. The first table was not wrong. It was a photograph of one split.

What all the models share is more valuable still: price and shelving location are the two decisive variables whichever algorithm you use. When a finding shows up in three different methods, it belongs to the data rather than the model.

“The best model is not the one at the top of the accuracy table; it is the one that best answers the question the business is asking.”