ENID
Lensa ML

Support Vector Machines

Step 1 of 6: Maximum Margin

Finding the widest street

-3-2-1123-3-2-1123green side
Angle
45°
Margin
0.49
Misclassified
0
Support vectors
1

An SVM picks the hyperplane with the widest margin. Current margin: 0.49. Try other angles to find the widest street.

Drag the hyperplane angle and watch the margin card. In this demo the line always passes through the origin, and the widest line that still separates the classes sits near 22°, with a margin of about 0.61. A full SVM also learns an offset b and solves an optimization problem: minimize ½‖w‖² subject to y_i(w·x_i + b) ≥ 1 for every point. The distance between the two dashed lines is then 2/‖w‖, so a smaller weight vector means a wider street. The problem is convex, so it has one best solution.

Step 2 of 6: Support Vectors

The critical few

-3-2-1123-3-2-1123drag me
Support vectors
2
Margin width
1.23
Point offset
0.0

Only the points on the margin edge, the support vectors, determine the boundary. The line is re-solved for the widest margin every time you move the point. The dragged point is away from the margin, so moving it leaves the boundary and margin unchanged.

Drag the point offset slider to move the highlighted point toward or away from the boundary. In a solved SVM the weight vector is a weighted sum of training points, w = Σ α_i y_i x_i, and α_i is zero for every point outside the margin. Those points could be deleted without changing the model. The points with α_i > 0 are the support vectors. Their count also bounds the leave-one-out error, because leaving out a non-support vector leaves the solution unchanged, so it is still classified correctly.

Step 3 of 6: Soft Margin

Tolerating some errors

-3-2-1123-3-2-1123
C
1.00
Margin
1.40
Violations
13
Accuracy
100%

A sketch of the trend: the boundary here stays fixed and the margin width follows a simple formula in C, not a full soft-margin solve. Moderate C (1.0): a balance between margin width and classification errors.

Drag the C parameter across its log scale from 0.01 to 100. Real data rarely separates cleanly, so the soft-margin SVM gives each point a slack ξ_i and minimizes ½‖w‖² + C·Σ ξ_i. A point with ξ_i between 0 and 1 sits inside the margin, and ξ_i > 1 puts it on the wrong side. The same objective can be written as the hinge loss max(0, 1 − y_i·f(x_i)) plus an L2 penalty on w. Cortes and Vapnik introduced this soft-margin form in 1995. C is usually tuned by cross-validation over a log grid.

Step 4 of 6: The Kernel Trick

Mapping to higher dimensions

-3-2-1123-3-2-1123
Kernel
Polynomial
Accuracy
100%

Polynomial kernel: a linear boundary over products of the features, which appears curved in the original space. Simplification: the shading is a kernel-weighted vote of every training point with equal weight. A trained SVM would keep only the support vectors and learn a weight for each.

Drag the kernel slider from Linear through Polynomial to RBF. The SVM's dual problem uses the data only through dot products x_i·x_j, so each one can be replaced by a kernel k(x_i, x_j) that equals a dot product in a larger feature space without ever building that space. The degree-2 kernel (x·z + 1)² covers all products of up to two features. The RBF kernel exp(−γ‖x − z‖²) corresponds to an infinite-dimensional space. Kernel SVMs need an n × n kernel matrix, which limits them to roughly tens of thousands of points.

Step 5 of 6: RBF Gamma

Controlling boundary flexibility

-3-2-1123-3-2-1123
Gamma
0.40
Train acc
85%
Test acc
87%

Moderate γ (0.40): flexible enough to follow the data without wrapping individual points. As in the previous step, the shading is an equal-weight RBF vote of all training points, a stand-in for a trained SVM; γ affects it the same way.

Drag γ across its log scale from 0.1 to 10 and watch how far each point's influence spreads. The two clusters overlap and 6 of the 40 labels are flipped, so a very local boundary pays a price: near γ = 1 the model scores 83% on its training points and 89% on 300 fresh ones, while at γ = 10 it reaches 95% on training and falls to 79% on the fresh sample. In the RBF kernel exp(−γ‖x − z‖²), γ equals 1/(2σ²) for a Gaussian of width σ, so a large γ means a narrow bump around each training point. γ and C work together: C limits how hard the model fits individual points, and γ sets how local each point's influence is. They are usually tuned jointly on a grid. scikit-learn's default, gamma="scale", sets γ to 1/(n_features · Var(X)).

Step 6 of 6: Multi-class SVM

One-vs-Rest strategy

-3-2-1123-3-2-1123
Classes
3
OvR classifiers
3
Accuracy
100%

For 3 classes, one-vs-rest trains 3 classifiers and picks the one with the highest score. Each dashed line is one classifier's boundary between its class and all the others.

Drag the number of classes from 2 to 5. An SVM separates two classes, so multi-class problems are split into binary ones. One-vs-rest trains K classifiers, each one class against all others. One-vs-one trains a classifier for every pair, K(K − 1)/2 in total, so 10 for 5 classes, and predicts by majority vote. Each pairwise model sees only two classes' data, which keeps training fast. scikit-learn's SVC uses one-vs-one internally, and LinearSVC uses one-vs-rest. The demo trains one linear SVM per class by subgradient descent on the hinge loss, draws each one's boundary as a dashed line, and colors every region by the classifier with the highest score.

What is a support vector machine (SVM)?

A support vector machine is a classifier that separates two classes with the hyperplane that leaves the widest possible margin to the nearest training points. Those nearest points are the support vectors, and they alone determine the boundary. A soft-margin version tolerates some errors, and kernels let the same method draw curved boundaries.

SVMs were among the most accurate general-purpose classifiers from the late 1990s through the 2000s, used for text categorization, handwritten digit recognition, and bioinformatics tasks such as protein classification. They still work well on small and medium datasets with many features. The ideas they introduced, margins, hinge loss, and kernels, appear across machine learning, from kernel ridge regression to the analysis of neural networks.

Key ideas

  • An SVM chooses the separating hyperplane with the widest margin to the nearest points of each class.
  • Only the support vectors, the points on or inside the margin, determine the boundary.
  • The C parameter trades margin width against training errors in the soft-margin SVM.
  • Kernels let an SVM learn nonlinear boundaries by replacing dot products with a similarity function.
  • In the RBF kernel, γ sets how far each training point's influence reaches.
  • Multi-class SVMs combine several binary classifiers, one-vs-rest or one-vs-one.

Common questions

Is an SVM better than logistic regression?

Neither wins in general. Both learn a linear boundary, but the hinge loss ignores points far from the boundary, while logistic loss uses every point and produces probabilities. With a kernel, an SVM can fit nonlinear boundaries that plain logistic regression cannot. On large tabular datasets, gradient-boosted trees often beat both.

Can an SVM output probabilities?

Not directly. The decision function gives a signed score proportional to the distance from the boundary. Platt scaling fits a sigmoid to those scores on held-out data to turn them into probabilities, and scikit-learn does this when probability=True, at extra training cost.

Do SVMs need feature scaling?

Yes. Both the margin and the RBF kernel depend on distances, so a feature with a large range dominates the rest. Standardizing every feature to zero mean and unit variance is a standard first step.

Can SVMs be used for regression?

Yes. Support vector regression (SVR) fits a function that keeps most points inside a tube of width ε around it and ignores errors smaller than ε. Points on or outside the tube become the support vectors, and the same kernels apply.

Related explainers

References

  1. Cortes & Vapnik (1995), "Support-Vector Networks", Machine Learning
  2. Boser, Guyon & Vapnik (1992), "A Training Algorithm for Optimal Margin Classifiers"
  3. James, Witten, Hastie & Tibshirani, "An Introduction to Statistical Learning", chapter 9: Support Vector Machines

Last updated