Nonlinear dimensionality reduction methods often include the construction of a kernel for embedding the high-dimensional data points. Standard methods for extending
the embedding coordinates (such as the Nystr\{o}m method) also rely on spectral decompositions of kernels. It is desirable that the kernels used for embedding and extensions of data capture most of the data sets’ information using a few leading modes of the spectrum.
In this work we propose high-order kernels, which are constructed as multi-scale combinations of Gaussian kernels, to be used within kernel-based embedding and extension frameworks. We review their spectral properties and show that their first few modes capture more information compared to the standard Gaussian kernel.