In this post, we are going to compare the two types of machine learning models-generative model and discriminative model-, whose underlying ideas are quite different. Also, a typical generative classification algorithm called Gaussian Discriminant Analysis will be introduced.
Discriminative Model
The most basic discriminative classifier in Machine Learning is Logistic Regression, and a decision boundary is learned to segregate points from two classes in training set. When a new point comes, the algorithm classifies it by checking which side of the boundary it falls. Logistic Regression do classification by calculating p(Y|X,θ), which is modeled by:
g is the sigmoid function. So this is the idea of Discriminative Algorithm: directly calculate p(Y|X,θ).
Generative Model
Generative Algorithm tries to achieve the same goal p(Y|X,θ) using another way: Bayes Rule, which is in the form of:
The key to solve this equation is to get p(X|Y), this need building seperate models for different Y.
When the input X is continuous random variable, we can then use the Gaussian Discriminant Analysis (GDA) model, which models p(x|y) using a multivariate normal distribution.
If we write out them:
then we can calculate the probability for a specific x using Bayes Rule. When X is discrete, we turn to Naive Bayes, which is also a Generative Algorithm.
In conclusion, the idea for Generative Model is to build a model for each class, and then use Bayes Rule to back up getting P(Y|X).
This picture shows the difference ideas of the two groups of algorithms.