Support Vector Machine (SVM)

Lesson 18 of 19 in Coddy's Introduction to Machine Learning course.

Support Vector Machine (SVM) is a powerful and versatile supervised machine learning algorithm used for both classification and regression tasks. However, it is more commonly used in classification problems. SVMs offer high accuracy and perform well with a high-dimensional feature space, making them suitable for applications ranging from image recognition to bioinformatics.

Source: wikipedia

The main objective of the SVM algorithm is to find the optimal hyperplane (in two-dimensional space, this is a line) that best separates the classes in the feature space. The SVM model represents the examples as points in space, mapped in such a way that the examples of separate categories are divided by a clear gap that is as wide as possible.

Hyperplane: This is basically a decision boundary that separates the classes. In 2D, this hyperplane is a line, but in higher dimensions, it can be more complex.
Support Vectors: These are the data points nearest to the hyperplane, which are critical in defining the hyperplane position. The SVM classifier is highly influenced by these points, hence the name Support Vector Machine.
Margin: It is the gap between two lines on the closest data points of different classes. It can be considered as a separate measure between the classes. Ensuring the margin is as wide as possible is a key objective in training an SVM.

Types of SVM:

Linear SVM: When the data is linearly separable, we can use a single straight line to separate the classes. This scenario uses linear SVM.
Non-linear SVM: When the data isn't linearly separable, SVM uses what's called the kernel trick to transform the input space into a higher-dimensional space where a hyperplane can be used to do the separation. The kernel trick involves transforming the data into another dimension that has a clear margin between classes of data. It allows SVM to solve non-linear problems. Common kernels include:
- Linear Kernel: Used when the data is linearly separable.
- Polynomial Kernel: Transforms the original feature space into a polynomial feature space of a specified degree.
- Gaussian Kernel: Useful for non-linear data that clusters into groups. It maps the feature space into higher dimensions.

Test yourself

This lesson includes a short quiz. Start the lesson to answer it and track your progress.

Test yourself

This lesson includes a short quiz. Start the lesson to answer it and track your progress.

Test yourself

This lesson includes a short quiz. Start the lesson to answer it and track your progress.

Challenge

Easy

Support Vector Machines (SVMs) utilize kernel functions to transform data into a higher-dimensional space where it is easier to classify. This process is known as the "kernel trick." It allows SVMs to solve non-linear classification and regression problems efficiently.

Here, we'll explore the polynomial kernel allowing curved boundaries in the data. It raises the data to a specified degree d. This kernel is particularly useful when the relationship between the class labels and the attributes is polynomial.
File:Svm 8 polinomial.JPG

Source: wikipedia

Create a function named polynomial_kernel that receives two lists (x, y), a degree (d), and a coef (c) and returns the polynomial kernel.

$K(x,y)=(x^{\mathsf {T}}y+c)^{d}$ → (x transpose dot product y plus c ) in the power of d.

Try it yourself

import numpy as np
def polynomial_kernel(x, y, d, c):
    x_arr = np.array(x)
    y_arr = np.array(y)
    # Write your code here

All lessons in Introduction to Machine Learning

1The Beginning of ML Journey

ML Introduction Data to feed a model What is a ML model

4K-Means Algorithm

K-Means Introduction fit method predict method

2Machine Learning Overview

Supervised learning Unsupervised learning

5Naive Bayes

Naive Bayes Introduction fit method predict method

3K-Nearest-Neighbors (KNN)

KNN Introduction fit method predict method

6Other Models

Logistic Regression Linear Regression Decision TreeSupport Vector Machine (SVM)Models quiz