[ML L3] SVM Intro

A support vector machine (SVM) is a supervised machine learning model that uses classification algorithms for two-group classification problems. After giving an SVM model sets of labeled training data for each  category, they’re able to categorize new text.

So you’re working on a text classification problem. You’re refining your training data, and maybe you’ve even tried stuff out using Naive Bayes. But now you’re feeling confident in your dataset, and want to take it one step further. Enter Support Vector Machines (SVM): a fast and dependable classification algorithm that performs very well with a limited amount of data.

How it works?

The basics of Support Vector Machines and how it works are best understood with a simple example. Let’s imagine we have two tags: red and blue, and our data has two features: x and y. We want a classifier that, given a pair of (x,y) coordinates, outputs if it’s either red or blue. We plot our already labeled training data on a plane:

A support vector machine takes these data points and outputs the hyperplane (which in two dimensions it’s simply a line) that best separates the tags. This line is the decision boundary: anything that falls to one side of it we will classify as blue, and anything that falls to the other as red.

But, what exactly is the best hyperplane? For SVM, it’s the one that maximizes the margins from both tags. In other words: the hyperplane (remember it’s a line in this case) whose distance to the nearest element of each tag is the largest.

None linear data?

Now this example was easy, since clearly the data was linearly separable — we could draw a straight line to separate red and blue. Sadly, usually things aren’t that simple. Take a look at this case:

We can introduce a new linear input param:

z = x^2 + y^2 

That’s great! Note that since we are in three dimensions now, the hyperplane is a plane parallel to the x axis at a certain z (let’s say z = 1).

What’s left is mapping it back to two dimensions:

In other words, we convert a none linear dataset by introduct a new dimensions

[Ref]: https://monkeylearn.com/blog/introduction-to-support-vector-machines-svm/#:~:text=A%20support%20vector%20machine%20(SVM,on%20a%20text%20classification%20problem.

from sklearn.svm import SVC
clf = SVC(gamma='auto', kernel="rbf", C=10000.0)
clf.fit(features_train, labels_train)
accuracy = clf.score(features_test, labels_test)

## 1% data
##kernal="linear" accuracy=0.88
##kernal="rbf" accuracy=0.61
##kernal="rbf" C=10.0 accuracy=0.61
##kernal="rbf" C=100.0 accuracy=0.61
##kernal="rbf" C=1000.0 accuracy=0.82
##kernal="rbf" C=10000.0 accuracy=0.89

## 35% data
##kernal="rbf" C=10000.0 accuracy=0.96

## 50% data
##kernal="rbf" C=10000.0 accuracy=0.987

## 100% data
##kernal="rbf" C=10000.0 accuracy>0.99