Support Vector Machines in R: Theory, Implementation, and Parameter Tuning

Support Vector Machines (SVM) are a modern machine‑learning technique originally designed for binary classification but now widely used for multi‑class non‑linear classification and regression. This guide shows how SVMs work and demonstrates their implementation in R.

How SVM Works

An SVM seeks the optimal hyperplane that maximizes the margin between the closest points of each class (support vectors). When data are not linearly separable, points can be projected into a higher‑dimensional space where a separating hyperplane may exist.

Dealing with Non‑Linear and Multi‑Class Data

If a linear hyperplane cannot separate the data, the algorithm maps the data to a high‑dimensional space and solves a constrained optimisation problem to maximise the margin. For more than two classes, multiple binary SVMs are trained in a one‑vs‑one fashion until all class pairs are distinguished.

Practical Example – Binary Classification with the cats dataset

We use the cats data from the MASS package to predict cat gender from body weight and heart weight. Twenty percent of the data are held out for testing.

library(e1071)

data(cats, package="MASS")

inputData <- data.frame(cats[, c(2,3)], response = as.factor(cats$Sex))

Linear SVM

The key parameters for svm() are kernel , cost , and gamma . For a linear kernel we set:

svmfit <- svm(response ~ ., data = inputData, kernel = "linear", cost = 10, scale = FALSE)

print(svmfit)

plot(svmfit, inputData)

compareTable <- table(inputData$response, predict(svmfit))

mean(inputData$response != predict(svmfit)) # 19.44% misclassification error

Radial (RBF) SVM

The radial basis function kernel allows non‑linear decision boundaries:

svmfit <- svm(response ~ ., data = inputData, kernel = "radial", cost = 10, scale = FALSE)

print(svmfit)

plot(svmfit, inputData)

compareTable <- table(inputData$response, predict(svmfit))

mean(inputData$response != predict(svmfit)) # 18.75% misclassification error

Finding Optimal Parameters

We use tune.svm() to search for the best cost and gamma values via 10‑fold cross‑validation.

set.seed(100)

rowIndices <- 1:nrow(inputData)

sampleSize <- 0.8 * length(rowIndices)

trainingRows <- sample(rowIndices, sampleSize)

trainingData <- inputData[trainingRows, ]

testData <- inputData[-trainingRows, ]

tuned <- tune.svm(response ~ ., data = trainingData, gamma = 10^(-6:-1), cost = 10^(1:2))

summary(tuned)

The best result is cost = 100 and gamma = 0.001 , giving the lowest error rate.

svmfit <- svm(response ~ ., data = trainingData, kernel = "radial", cost = 100, gamma = 0.001, scale = FALSE)

print(svmfit)

plot(svmfit, trainingData)

compareTable <- table(testData$response, predict(svmfit, testData))

mean(testData$response != predict(svmfit, testData)) # 13.79% misclassification error

Grid Plot for Visualisation

A two‑colour grid plot shows the decision regions and highlights support vectors.

n_points_in_grid <- 60

x_axis_range <- range(inputData[, 2])

y_axis_range <- range(inputData[, 1])

X_grid_points <- seq(from = x_axis_range[1], to = x_axis_range[2], length = n_points_in_grid)

Y_grid_points <- seq(from = y_axis_range[1], to = y_axis_range[2], length = n_points_in_grid)

all_grid_points <- expand.grid(X_grid_points, Y_grid_points)

names(all_grid_points) <- c("Hwt", "Bwt")

all_points_predicted <- predict(svmfit, all_grid_points)

color_array <- c("red", "blue")[as.numeric(all_points_predicted)]

plot(all_grid_points, col = color_array, pch = 20, cex = 0.25)

points(x = trainingData$Hwt, y = trainingData$Bwt, col = c("red", "blue")[as.numeric(trainingData$response)], pch = 19)

points(trainingData[svmfit$index, c(2, 1)], pch = 5, cex = 2)