The *K*-nearest neighbours classifier (KNN) is a type of instance-based learning, or lazy learning, where the function is only approximated locally and all computation is deferred until classification. Thus, the KNN approach is among the simplest of all discriminative approaches, but this classifier is still especially effective for low-dimensional feature spaces.

The input for the KNN algorithm consists of the *K* closest training samples in the feature space and the output is a class label *l*. An observation (or testing sample) **f**(** y**) is classified by a majority vote of its neighbours, with the observation being labelled by the class most common among its

assigned to the observation

In order to estimate the potentials we consider the class of every neighbour as a vote for the most likely class of the observation. If the number of neighbours, having class *l* is *Kl* we can define the probability of the association potentials as:

\(*\alpha = x_1*\)

(see Figure)

As we can observe from the Figure above, the generative models (Bayes and Gaussian mixtures) try to reproduce the original distributions. In order to do this precisely, a methods need to remember all the samples from the Green Field dataset – 160’000 parameters. Or, in general, restricting ourself to the 8-bit features, a method needs to remember *k·256^m* values, where *k* is the number of categories and *m* is the number of features. The main idea of the generative models is to rebuild the original distribution using much less parameters and therefore generalize the model for samples, that were not observed during training. Bayes model approximates the distribution using only *k·256·m* parameters, and the Gaussian mixture model — *k·G·m·(m +1)* parameters, where *G* is the number of Gaussians in the mixture.

As opposed to the generative models, the discriminative models (Neural Networks, Random Forests, Support Vector Machines and k-Nearest neighbors) do not approximate the original distributions, but provide direct predictions for all testing samples. This grants the discriminative models more generalization power: In the areas, where hardly any training sample was met (left bottom and right top corners of the initial distribution image on Figure) all the generative models show black areas with almost zero potentials, while all the discriminative models how a high confidence about the class labels for these areas.

We have made one performance test, using the task of the sparse coding dictionary learning. The bottleneck of this function is the algorithm, calculating the matrix product. Using the parallel implementation of this algorithm, we may achieve the sparse coding dictionary learning function with 90%-95% sequential processing. Our implementation of the matrix product may be found here.

Our test was performed using two systems:

- Intel® Core™ i7-4820K @ 3.70 GHz + NVIDIA GeForce GTX 780
- Intel® Xeon® X5450 @ 3.00 GHz + NVIDIA GeForce 210

The speed-up, when using PPL and AMP libraries for the first system may be seen at the following figure:

here, **1,00** corresponds to the time needed for 2 iterations of the training algorithm (2 180 seconds). Accordingly, for the AMP | GPU this time is **15,21** times smaller (142 seconds)

The speed-up, when using PPL and AMP libraries for the second system may be seen at the figure:

Here, the AMP library decided to run the code on CPU instead of GPU. This lead to significant performance drop. Again, **1,00** corresponds to the time needed for 2 iterations of the training algorithm (4 437 seconds on X5450). The parallel computing on CPU with the PPL library took **3,65** times less time and parallel computing on CPU with the AMP library took even more time – 4 819 seconds.

On the systems with powerful graphics cards, please build the DGM library with the option **ENABLE_AMP** in CMake. Otherwise, please use only **ENABLE_PPL** option.

Click here for more details… http://research.project-10.de/hcell/

Hannover in Summer Motion from Sergey Kosov on Vimeo.