The Manifold Tangent Classifier (MTC) Putting it all together, here is the high level summary of how we build and train a deep network:
1. Train (unsupervised) a stack of K CAE+H layers (Eq. 4). Each is trained in turn on the representation learned by the previous layer.
2. For each xi ∈ D compute the Jacobian of the last layer representation J (K) (xi) = ∂h(K) ∂x (xi) and its SVD1 . Store the leading dM singular vectors in set Bxi .
3. On top of the K pre-trained layers, stack an output layer of size the number of classes. Finetune the whole network for supervised classification2 with an added tangent propagation penalty (Eq. 6), using for each xi , tangent directions Bxi .
We call this deep learning algorithm the Manifold Tangent Classifier (MTC). Alternatively, instead of step 3, one can use the tangent vectors in Bxi in a tangent distance nearest neighbors classifier.
--written by Salah Rifai