The Gaussian Process Latent Variable Model (GPLVM)
11 likes15,981 views
This document provides an outline for a talk on Gaussian Process Latent Variable Models (GPLVM). It begins with an introduction to why latent variable models are useful for dimensionality reduction. It then defines latent variable models and shows their graphical model representation. The document reviews PCA and introduces probabilistic versions like Probabilistic PCA (PPCA) and Dual PPCA. It describes how GPLVM generalizes these approaches using Gaussian processes. Examples applying GPLVM to face and motion data are provided, along with practical tips and an overview of GPLVM variants.
![nition
I Consider a set of centered data of n observations and d
dimensions: Y = [y1; : : : ; yn]T .
I We assume this data has a linear relationship with some
embedded latent space data xn. Where Y 2 RND and
x 2 RNq.
I yn = Wxn + n, where xn is the q-dimensional latent variable
associated with each observation, and W 2 RDq is the
transformation matrix relating the observed and latent space.
I We assume a spherical Gaussian distribution for the noise with
a mean of zero and a covariance of](https://image.slidesharecdn.com/gplvmpres-141023123617-conversion-gate01/85/The-Gaussian-Process-Latent-Variable-Model-GPLVM-18-320.jpg)
The Gaussian Process Latent Variable Model (GPLVM)
- 1. Gaussian Process Latent Variable Model (GPLVM) James McMurray PhD Student Department of Empirical Inference 11/02/2014
- 2. Outline of talk Introduction Why are latent variable models useful? De
- 3. nition of a latent variable model Graphical Model representation PCA recap Principal Components Analysis (PCA) Probabilistic PCA based methods Probabilistic PCA (PPCA) Dual PPCA GPLVM Examples Practical points Variants Conclusion Conclusion References
- 4. Why are latent variable models useful? I Data has structure.
- 5. Why are latent variable models useful? I Observed high-dimensional data often lies on a lower-dimensional manifold. Example Swiss Roll" dataset
- 6. Why are latent variable models useful? I The structure in the data means that we don't need such high dimensionality to describe it. I The lower dimensional space is often easier to work with. I Allows for interpolation between observed data points.
- 7. De
- 8. nition of a latent variable model I Assumptions: I Assume that the observed variables actually result from a smaller set of latent variables. I Assume that the observed variables are independent given the latent variables. I Diers slightly from dimensionality reduction paradigm which wishes to
- 9. nd a lower-dimensional embedding in the high-dimensional space. I With the latent variable model we specify the functional form of the mapping: y = g(x) + where x are the latent variables, y are the observed variables and is noise. I Obtain dierent latent variable models for dierent assumptions on g(x) and
- 10. Graphical model representation Graphical Model example of Latent Variable Model Taken from Neil Lawrence: http: // ml. dcs. shef. ac. uk/ gpss/ gpws14/ gp_ gpws14_ session3. pdf
- 11. Plan Introduction Why are latent variable models useful? De
- 12. nition of a latent variable model Graphical Model representation PCA recap Principal Components Analysis (PCA) Probabilistic PCA based methods Probabilistic PCA (PPCA) Dual PPCA GPLVM Examples Practical points Variants Conclusion Conclusion References
- 13. Principal Components Analysis (PCA) I Returns orthogonal dimensions of maximum variance. I Works well if data lies on a plane in the higher dimensional space. I Linear method (although variants allow non-linear application, e.g. kernel PCA). Example application of PCA. Taken from http: // www. nlpca. org/ pca_ principal_ component_ analysis. html
- 14. Plan Introduction Why are latent variable models useful? De
- 15. nition of a latent variable model Graphical Model representation PCA recap Principal Components Analysis (PCA) Probabilistic PCA based methods Probabilistic PCA (PPCA) Dual PPCA GPLVM Examples Practical points Variants Conclusion Conclusion References
- 16. Probabilistic PCA (PPCA) I A probabilistic version of PCA. I Probabilistic formulation is useful for many reasons: I Allows comparison with other techniques via likelihood measure. I Facilitates statistical testing. I Allows application of Bayesian methods. I Provides a principled way of handling missing values - via Expectation Maximization.
- 17. PPCA De
- 18. nition I Consider a set of centered data of n observations and d dimensions: Y = [y1; : : : ; yn]T . I We assume this data has a linear relationship with some embedded latent space data xn. Where Y 2 RND and x 2 RNq. I yn = Wxn + n, where xn is the q-dimensional latent variable associated with each observation, and W 2 RDq is the transformation matrix relating the observed and latent space. I We assume a spherical Gaussian distribution for the noise with a mean of zero and a covariance of
- 19. 1I I Likelihood for an observation yn is: p (ynjxn;W;
- 20. ) = N ynjWxn;
- 21. 1I
- 22. PPCA Derivation I Marginalise latent variables xn, put a Gaussian prior on W and solve using maximum likelihood. I The prior used for xn in the integration is a zero mean, unit covariance Gaussian distribution: p(xn) = N(xnj0; I) p(ynjW;
- 23. ) = Z p(ynjxn;W;
- 25. ) = Z N ynjWxn;
- 26. 1I N(xnj0; I)dxn p(ynjW;
- 27. ) = N(ynj0;WWT +
- 28. 1I) I Assuming i.i.d. data, the likelihood of the full set is the product of the individual probabilities: p(Y jW;
- 29. ) = YN n=1 p(ynjW;
- 30. )
- 31. PPCA Derivation I To calculate that marginalisation step we use the summation and scaling properties of Gaussians. I Sum of Gaussian variables is Gaussian. Xn i=1 N(i ; 2 i ) N Xn i=1 i ; Xn i=1 2 i ! I Scaling a Gaussian leads to a Gaussian: wN(; 2) N(w;w22) I So: y = Wx + ; x N(0; I) ; N(0; 2I) Wx N(0;WWT) Wx + N(0;WWT + 2I)
- 32. PPCA Derivation I Can
- 33. nd a solution for W by maximising the likelihood. I Results in an eigenvalue problem. I Turns out that the closed-form solution for W is achieved when W spans the principal sub-space of the data1. I Same solution as PCA: Probabilistic PCA I Can it be extended to capture non-linear features? 1Michael E. Tipping and Christopher M. Bishop. Probabilistic principal component analysis. (1997).
- 34. Dual PPCA I Similar to previous derivation of PPCA. I But marginalise W and optimise xn. I Same linear-Gaussian relationship between latent variables and data: p(YjX;W;
- 35. ) = YD d=1 N(yd;:jWxd;:;
- 36. 1I) I Place a conjugate prior on W: P(W) = YD d=1 N(wd;:j0; I) I Resulting marginal likelihood is: P(Y jX;
- 37. ) = YD d=1 N(y:;d j0;XXT +
- 38. 1I)
- 39. Dual PPCA I Results in equivalent eigenvalue problem to PPCA. I So what is the bene
- 40. t? I The eigendecomposition is now done on an N q instead of a d q matrix. I Recall marginal likelihood: P(Y jX;
- 41. ) = YD d=1 N(y:;d j0;XXT +
- 42. 1I) I The covariance matrix is a covariance function: K = XXT +
- 43. 1I I This linear kernel can be replaced by other covariance functions for non-linearity. I This is the GPLVM.
- 44. GPLVM I Each dimension of the marginal distribution can be interpreted as an independent Gaussian Process2. I Dual PPCA is the special case where the output dimensions are assumed to be linear, independent and identically distributed. I GPLVM removes assumption of linearity. I Gaussian prior over the function space. I Choice of covariance function changes family of functions considered. I Popular kernels: I Exponentiated Quadratic (RBF) kernel I Matern kernels I Periodic kernels I Many more... 2Neil Lawrence: Probabilistic non-linear principal component analysis with Gaussian process latent variable models. JMLR (2005)
- 45. Plan Introduction Why are latent variable models useful? De
- 46. nition of a latent variable model Graphical Model representation PCA recap Principal Components Analysis (PCA) Probabilistic PCA based methods Probabilistic PCA (PPCA) Dual PPCA GPLVM Examples Practical points Variants Conclusion Conclusion References
- 47. Example: Frey Face data Example in GPMat
- 48. Example: Motion Capture data Taken from Keith Grochow, et al. Style-based inverse kinematics. ACM Transactions on Graphics (TOG). Vol. 23. No. 3. ACM, 2004.
- 49. Plan Introduction Why are latent variable models useful? De
- 50. nition of a latent variable model Graphical Model representation PCA recap Principal Components Analysis (PCA) Probabilistic PCA based methods Probabilistic PCA (PPCA) Dual PPCA GPLVM Examples Practical points Variants Conclusion Conclusion References
- 51. Practical points I Need to optimise over non-convex objective function. I Achieved using gradient-based methods (scaled conjugate gradients). I Several restarts to attempt to avoid local optima. I Cannot guarantee global optimum. I High computational cost for large datasets. I May need to optimise over most-informative subset of data, the active set for sparsi
- 52. cation.
- 53. Practical points Initialisation can have a large eect on the
- 54. nal results. Eect of poor initialisation on Swiss Roll dataset. PCA left, Isomap right. Taken from Probabilistic non-linear principal component analysis with Gaussian process latent variable models., Neil Lawrence, JMLR (2005).
- 55. Plan Introduction Why are latent variable models useful? De
- 56. nition of a latent variable model Graphical Model representation PCA recap Principal Components Analysis (PCA) Probabilistic PCA based methods Probabilistic PCA (PPCA) Dual PPCA GPLVM Examples Practical points Variants Conclusion Conclusion References
- 57. Variants I There are a number of variants of the GPLVM. I For example, the GPLVM uses the same covariance function for each output dimension. I This can be changed, for example the Scaled GPLVM which introduces a scaling parameter for each output dimension3. I The Gaussian Process Dynamic Model (GPDM) adds another Gaussian process for dynamical mappings4. I The Bayesian GPLVM approximates integrating over both the latent variables and the mapping function5. 3Keith Grochow, et al. Style-based inverse kinematics. ACM Transactions on Graphics (TOG). Vol. 23. No. 3. ACM, 2004. 4Wang, Jack, Aaron Hertzmann, and David M. Blei. Gaussian process dynamical models. Advances in neural information processing systems. 2005. 5Titsias, Michalis, and Neil Lawrence. Bayesian Gaussian process latent variable model. (2010).
- 58. Variants I The Shared GPLVM learns mappings from a shared latent space to two separate observational spaces. I Used by Disney Research in their paper Animating Non-Humanoid Characters with Human Motion Data for generating animations for non-human characters from human motion capture data. Shared GPLVM mappings as used by Disney Research I Video
- 59. Variants I Can also put a Gaussian Process prior on X to produce Deep Gaussian Processes6. I Zhang et al. developed Invariant GPLVM7 - permits interpretation of causal relations between observed variables, by allowing arbitrary noise correlations between the latent variables. I Currently attempting to implement IGPLVM in GPy. 6Damianou, Andreas C., and Neil D. Lawrence. Deep Gaussian Processes. arXiv preprint arXiv:1211.0358 (2012). 7Zhang, K., Scholkopf, B., and Janzing, D. (2010). Invariant Gaussian Process Latent Variable Models and Application in Causal Discovery. UAI 2010.
- 60. Plan Introduction Why are latent variable models useful? De
- 61. nition of a latent variable model Graphical Model representation PCA recap Principal Components Analysis (PCA) Probabilistic PCA based methods Probabilistic PCA (PPCA) Dual PPCA GPLVM Examples Practical points Variants Conclusion Conclusion References
- 62. Conclusion I Implemented in GPy (Python) and GPMat (MATLAB). I Many practical applications - pose modelling, tweening I Especially if smooth interpolation is desireable. I Modeling of confounders. Thanks for your time Questions?
- 63. References I Neil Lawrence, Gaussian process latent variable models for visualisation of high dimensional data. Advances in neural information processing systems 16.329-336 (2004): 3. I Neil Lawrence, Probabilistic non-linear principal component analysis wit Gaussian process latent variable models. JMLR (2005) I Gaussian Process Winter School, Sheeld 2013: http://ml.dcs.shef.ac.uk/gpss/gpws14/ I WikiCourseNote: http://wikicoursenote.com/wiki/ Probabilistic_PCA_with_GPLVM