# Characteristics of Machine Learning Algorithms

Discriminative / Generative algorithms

Many machine learning algorithms are based on probabilistic models. Discriminative algorithms learn the conditional probability distribution of y given x p(y|x). Generative algorithms learn the joint probability distribution p(y,x) = p(y|x) * p(x), and therefore take in consideration the distribution of x.

Parametric & Non-Parametric algorithms

Neural Network, SVM,… are non-parametric algorithms because they have no fixed parameter vector; depending on the richness of the data we can choose the size of the parameter vector that we need (ex. we can add hidden units, change SVM kernels,…)

Capacity

The capacity is controlled by parameters called Hyperparameters (ex. degree p of a polynomial predictor, kernel choice in SVM, number of layers in a neural network).

# Summary of Machine Learning Algorithms

The table below describes briefly each machine learning algorithm.

 Algorithm Description Characteristics Linear regression To use when Y is normally-distributed Discriminative Parametric Logistic regression To use when Y is Bernoulli-distributed Discriminative Parametric Multinomial logistic regression (softmax regression) To use when Y is multinomially-distributed There are two versions of the algorithm, one based on maximum likelihood maximization, and one based on cross-entropy minimization. Discriminative Parametric Gaussian Discriminant Analysis Supervised classification algorithm To use when Y is Bernoulli distributed and the conditional distribution of X given Y is multivariate Gaussian Generative Parametric Naive Bayes Algorithm Supervised classification algorithm To use when Y is Bernoulli, and the conditional distribution of X given Y is Bernoulli and X features are conditionally independent Generative Parametric EM Unsupervised soft-clustering algorithm Generative Parametric Principal Component Analysis Reduce the dimensionality of X. Calculate eigenvectors for $$XX^T$$. Use eigenvectors with higher eigenvalues to transform data $$x^{(i)} := (u_1^T x^{(i)}, u_2^T x^{(i)},…, , u_k^T x^{(i)})$$ Factor Analysis Reduce the dimensionality of X To use when X is Gaussian. Transform X to Z matrix with lower dimensionality (x ? ?+?z). Neural Network Non-linear classifier Discriminative Non-Parametric K-Nearest Neighbor Regression Predict y as the average of values y1,y2,…,yk of k nearest neighbors of x Discriminative Non-Parametric K-Nearest Neighbor Classifier Predict y as the most common class among the k nearest neighbors of x Discriminative Non-Parametric Support Vector Machine Find a separator that maximizes the margin between two classes Discriminative Non-Parametric K-means Unsupervised hard-clustering algorithm Discriminative Non-Parametric