Support Vector Machines (SVMs) using Julia Packages

Support Vector Machines (SVMs) are a versatile and powerful class of supervised learning algorithms used for classification and regression tasks, though they are most commonly associated with classification. The fundamental idea behind SVMs is to find an optimal hyperplane that best separates data points belonging to different classes in the feature space. This "optimality" is achieved by maximizing the margin, which is the distance between the hyperplane and the nearest data points from any class. These nearest points are called support vectors, as they are the critical elements that support or define the hyperplane.

SVMs are particularly effective in high-dimensional spaces, and they can perform well even when the number of dimensions is greater than the number of samples. They also offer flexibility through the use of different kernel functions, allowing them to model non-linear decision boundaries.

Core Ideas of SVMs

Before exploring Julia implementations, let's solidify a few central ideas:

• Hyperplane: In a $p$-dimensional space, a hyperplane is a flat affine subspace of dimension $p-1$. For a two-class classification task in two dimensions ($p=2$), the hyperplane is a line. In three dimensions ($p=3$), it's a plane. For $p > 3$, it's a hyperplane. SVMs seek the hyperplane that creates the best separation.
• Support Vectors: These are the data points that lie closest to the decision boundary (the hyperplane). They are the most difficult points to classify and directly influence the position and orientation of the hyperplane. If these points were moved, the hyperplane would also move.
• Margin: The margin is the distance between the hyperplane and the closest support vectors on either side. SVMs aim to maximize this margin. A larger margin is generally associated with better generalization performance on unseen data.
• Kernels: For data that is not linearly separable in its original feature space, SVMs employ the "kernel trick." Kernels are functions that take the input data and transform it into a higher-dimensional space where a linear separation might be possible. Common kernels include:
  • Linear: For linearly separable data. $K(x_i, x_j) = x_i^T x_j$.
  • Polynomial: $K(x_i, x_j) = (\gamma x_i^T x_j + r)^d$.
  • Radial Basis Function (RBF): $K(x_i, x_j) = \exp(-\gamma ||x_i - x_j||^2)$. This is a very popular and flexible kernel.
  • Sigmoid: $K(x_i, x_j) = \tanh(\gamma x_i^T x_j + r)$.
• Regularization (C parameter): The parameter $C$ (often called the cost or penalty parameter) controls the trade-off between maximizing the margin and minimizing the classification error on the training data.
  • A small $C$ value encourages a larger margin, even if it means misclassifying some training points. This can lead to a simpler decision boundary and potentially better generalization.
  • A large $C$ value penalizes misclassifications more heavily, leading to a smaller margin and a decision boundary that tries to classify all training points correctly. This can sometimes lead to overfitting.

The following diagram illustrates the main components involved in an SVM model:

The diagram above shows the relationship between input data, the SVM's goal, and main components like the hyperplane, support vectors, margin, and kernels.

SVMs in Julia with MLJ.jl

In Julia, SVM implementations are available through external packages, and MLJ.jl provides a consistent interface to use them. The most common package providing SVMs is LIBSVM.jl, which is a wrapper around the widely-used LIBSVM C++ library. You'll typically interact with it via MLJLIBSVMInterface.jl.

Let's walk through implementing an SVM for a classification task.

First, ensure you have the necessary packages. If MLJLIBSVMInterface is not already in your Julia environment, you'll need to add it:

# In the Julia REPL, press ']' to enter Pkg mode
# pkg> add MLJLIBSVMInterface
# Press Backspace to exit Pkg mode

Now, let's set up our Julia script:

using MLJ
using DataFrames, Random, StableRNGs

# Load the Support Vector Classifier (SVC) model type from MLJLIBSVMInterface
# modest=false returns the model type itself, not an instance.
SVC = @load SVC pkg=LIBSVM modest=false
# For a dedicated linear SVM, which can be faster if data is linearly separable:
LinearSVC = @load LinearSVC pkg=LIBSVM modest=false

# For reproducibility, use a stable RNG
rng = StableRNG(123)

# Generate synthetic 2D data for classification
# X will be features, y will be categorical labels
X_raw, y = make_blobs(150, 2; centers=2, cluster_std=0.9, rng=rng, as_table=false)
X = DataFrame(X_raw, :auto) # Convert to DataFrame for MLJ

The make_blobs function generates isotropic Gaussian blobs for clustering or classification. Here, we're creating two distinct groups of points in a 2D space. The plot below shows an example of such generated data.

A scatter plot of synthetic 2D data with two distinct classes, suitable for training an SVM classifier.

Training an SVM with a Linear Kernel

If you suspect your data is linearly separable or want a baseline, a linear kernel is a good start. LinearSVC is optimized for this, but you can also use SVC with the linear kernel.

# Instantiate a linear SVM model
# LinearSVC is often faster for linear problems.
# It internally uses LIBSVM's linear solver.
linear_svm_model = LinearSVC(cost=1.0) # 'cost' is the C parameter

# Alternatively, using SVC:
# linear_svm_model = SVC(kernel=LIBSVM.Kernel.LINEAR, cost=1.0)

# Create an MLJ machine
mach_linear_svm = machine(linear_svm_model, X, y)

# Train the machine
fit!(mach_linear_svm, verbosity=0)

# Make predictions
y_pred_linear = predict_mode(mach_linear_svm, X)

# Evaluate (evaluation metrics are covered in detail in another section)
# For example, calculate misclassification rate:
accuracy_linear = accuracy(y_pred_linear, y)
println("Linear SVM Accuracy: $(round(accuracy_linear, digits=3))")

The cost parameter here is the regularization parameter $C$. A cost=1.0 is a common default.

Training an SVM with an RBF Kernel

For non-linearly separable data, the RBF kernel is a popular choice due to its flexibility.

# Instantiate an SVM model with an RBF kernel
rbf_svm_model = SVC(kernel=LIBSVM.Kernel.RADIAL, # RADIAL is RBF
                    cost=1.0,      # Regularization parameter C
                    gamma=0.5)     # Kernel coefficient for RBF

# `LIBSVM.Kernel` provides access to kernel types:
# LIBSVM.Kernel.LINEAR, LIBSVM.Kernel.POLY, LIBSVM.Kernel.RADIAL, LIBSVM.Kernel.SIGMOID

# Create an MLJ machine
mach_rbf_svm = machine(rbf_svm_model, X, y)

# Train the machine
fit!(mach_rbf_svm, verbosity=0)

# Make predictions
y_pred_rbf = predict_mode(mach_rbf_svm, X)

accuracy_rbf = accuracy(y_pred_rbf, y)
println("RBF Kernel SVM Accuracy: $(round(accuracy_rbf, digits=3))")

In this SVC model:

• kernel=LIBSVM.Kernel.RADIAL specifies the RBF kernel.
• cost=1.0 is the regularization parameter $C$.
• gamma=0.5 is specific to kernels like RBF and Polynomial. For the RBF kernel, $K(x_i, x_j) = \exp(-\gamma ||x_i - x_j||^2)$, gamma defines how much influence a single training example has.
  • A low gamma means a larger similarity radius, so points further apart are considered similar. This leads to a smoother decision boundary.
  • A high gamma means a smaller similarity radius, so only points close to each other are considered similar. This can lead to a more complex, "wiggly" boundary that might overfit the training data.

You can inspect all tunable hyperparameters of a model using params(model_name):

# println(params(rbf_svm_model))

The choice of kernel and the values for hyperparameters like $C$ and $\gamma$ are critical for SVM performance. These are typically determined using techniques like cross-validation and hyperparameter tuning, which are discussed in detail later in this chapter.

Advantages and Considerations for SVMs

Advantages:

• Effective in high-dimensional spaces: SVMs work well even when the number of features is large.
• Memory efficient: They use a subset of training points (the support vectors) in the decision function.
• Versatile: Different kernel functions can be specified for the decision function. Common kernels provide flexibility for various data types.
• Good generalization: The margin maximization objective often leads to good generalization on unseen data, especially when well-tuned.

Considerations:

• Computational cost: Training can be slow for very large datasets, especially with non-linear kernels. The complexity can be between $O(n^2 p)$ and $O(n^3 p)$ for some implementations, where $n$ is the number of samples and $p$ is the number of features.
• Kernel and hyperparameter selection: Performance is highly dependent on the choice of kernel and its parameters (e.g., $C$, $\gamma$). This often requires careful tuning.
• Interpretability: SVMs, especially with non-linear kernels, produce models that are not as easily interpretable as, for example, decision trees.
• Scaling: SVMs are sensitive to feature scaling. It's generally recommended to scale your data (e.g., to zero mean and unit variance) before training an SVM.

When to Consider Using SVMs

SVMs are a good choice for:

• Classification tasks where a clear margin of separation between classes is desired.
• Problems with high-dimensional data, such as text classification or bioinformatics.
• Situations where the data might not be linearly separable, and non-linear kernels can be beneficial.
• When you need a model that can provide good generalization performance, assuming proper tuning.

While SVMs might not always be the fastest algorithm to train, their ability to find complex decision boundaries and their strong theoretical underpinnings make them a valuable tool in the machine learning practitioner's toolkit. As with any model, their performance should be rigorously evaluated using appropriate metrics and validation strategies, as covered in subsequent sections of this chapter.