Classification and representation
1. Classification
To attempt classification, one method is to use linear regression and map all predictions greater than 0.5 as a 1 and all less than 0.5 as a 0. However, this method doesn't work well because classification is not actually a linear function.
The classification problem is just like the regression problem, except that the values y we now want to predict take on only a small number of discrete values. For now, we will focus on the binary classification problem in which y can take on only two values, 0 and 1. (Most of what we say here will also generalize to the multiple-class case.) For instance, if we are trying to build a spam classifier for email, then$$x^{(i)}$$may be some features of a piece of email, and y may be 1 if it is a piece of spam mail, and 0 otherwise. Hence, y∈{0,1}. 0 is also called the negative class, and 1 the positive class, and they are sometimes also denoted by the symbols “-” and “+.” Given $$x^{(i)}$$, the corresponding $$y^{(i)}$$ is also called the label for the training example.
2. Hypothesis Representation
We could approach the classification problem ignoring the fact that y isdiscrete-valued, and use our old linear regression algorithm to try to predicty given x. However, it is easy to construct examples where this methodperforms very poorly. Intuitively, it also doesn’t make sense for$$h_{ heta}(x)$$ to takevalues larger than 1 or smaller than 0 when we know that y ∈ {0, 1}.To fix this, let’s change the form for our hypotheses hθ(x) to satisfy $$0 leq h_ heta (x) leq 1$$. This is accomplished by plugging$$ heta^{T}x$$ into the Logistic Function.
Our new form uses the "Sigmoid Function," also called the "Logistic Function":
$$
egin{align}& h_ heta (x) = g ( heta^T x )
ewline
ewline& z = heta^T x
ewline& g(z) = dfrac{1}{1 + e^{-z}}end{align}
$$
The following image shows us what the sigmoid function looks like:
The function g(z), shown here, maps any real number to the (0, 1) interval, making it useful for transforming an arbitrary-valued function into a function better suited for classification.
hθ(x) will give us the probability that our output is 1. For example, hθ(x)=0.7 gives us a probability of 70% that our output is 1. Our probability that our prediction is 0 is just the complement of our probability that it is 1 (e.g. if probability that it is 1 is 70%, then the probability that it is 0 is 30%).
$$
egin{align}& h_ heta(x) = P(y=1 | x ; heta) = 1 - P(y=0 | x ; heta)
ewline& P(y = 0 | x; heta) + P(y = 1 | x ; heta) = 1end{align}
$$
3. Decision Boundary
In order to get our discrete 0 or 1 classification, we can translate the output of the hypothesis function as follows:
$$
egin{align}& h_ heta(x) geq 0.5
ightarrow y = 1
ewline& h_ heta(x) < 0.5
ightarrow y = 0
ewlineend{align}
$$
The way our logistic function g behaves is that when its input is greater than or equal to zero, its output is greater than or equal to 0.5:
$$
egin{align}& g(z) geq 0.5
ewline& when ; z geq 0end{align}
$$
Remember.
$$
egin{align}z=0, e^{0}=1 Rightarrow g(z)=1/2
ewline z o infty, e^{-infty} o 0 Rightarrow g(z)=1
ewline z o -infty, e^{infty} o infty Rightarrow g(z)=0 end{align}
$$
So if our input to g is θTX, then that means:
$$
egin{align}& h_ heta(x) = g( heta^T x) geq 0.5
ewline& when ; heta^T x geq 0end{align}
$$
From these statements we can now say:
$$
egin{align}& heta^T x geq 0 Rightarrow y = 1
ewline& heta^T x < 0 Rightarrow y = 0
ewlineend{align}
$$
The decision boundary is the line that separates the area where y = 0 and where y = 1. It is created by our hypothesis function.
Example:
$$
egin{align}& heta = egin{bmatrix}5
ewline -1
ewline 0end{bmatrix}
ewline & y = 1 ; if ; 5 + (-1) x_1 + 0 x_2 geq 0
ewline & 5 - x_1 geq 0
ewline & - x_1 geq -5
ewline& x_1 leq 5
ewline end{align}
$$
In this case, our decision boundary is a straight vertical line placed on the graph where $$x_1 = 5$$, and everything to the left of that denotes y = 1, while everything to the right denotes y = 0.
Again, the input to the sigmoid function g(z) (e.g. $$ heta^{T}X$$ doesn't need to be linear, and could be a function that describes a circle (e.g. $$z = heta_0 + heta_1 x_1^2 + heta_2 x_2^2$$) or any shape to fit our data.