Nearest-neighbor methods use those observations in the training set T closest in input space to x form Y-hat.
Specifically, the k-nearest neighbor fit for Y-hat is difined as follows: Y(x)=1/kΣyi,xi belong to Nk(x).
where Nk(x) is the neighborhood of x defined by the k closest points xi in the traing sample.
Closeness implies a metric, which for the moment we assume is Euclidean distance. So, in words, we find the k observations with xi closest to x in input space, and average their responses.
1.Kernel methods use weights that decrease smoothly to zero with distance from the target point, rather than the effective 0/1 weights used by k-nearest neighbors.
2.In high-dimensional spaces the distance kernels are modified to emphasize some variable more than others.
3.Local regression fits linear models by locally weighted least squares, rather than fitting constants locally.
4.Local models fit to a basis expansion of the original inputs allow arbitrarily complex models.
5.Projection pursuit and neural network models consist of sums of nonlinearly transformed linear models.