Applied Mathematical Modeling with Apache Solr - Joel Bernstein, Lucidworks
1 like226 views
This document discusses using Apache Solr for mathematical modeling. It covers using Solr to train regression models on data, assess the models by analyzing residuals, and use the models for prediction and anomaly detection. Specific regression techniques discussed include linear regression, polynomial curve fitting, and K-nearest neighbors regression. Probability distributions are also covered as a way to model risk and detect outliers. The document walks through an example of using simple linear regression to model network response times and detect anomalies.
Applied Mathematical Modeling with Apache Solr - Joel Bernstein, Lucidworks
- 1. Mathematical Modeling with Apache Solr Joel Bernstein Senior Data Engineer, Lucidworks Lucene/Solr Committer & PMC @jbernste2 #Activate18 #ActivateSearch
- 2. Agenda • Math Expressions Introduction • Model Types • Training Regression Models • Assessing Regression Models • Prediction and Anomaly Detection
- 3. • Mathematics language that runs inside of Solr Cloud. • Integration of Apache Commons Math with Solr. • Designed for fast quantitative analysis of result sets. • Any Streaming Expression can be used to create the data set. Math Expressions Vector/Matrix Math Text Analysis Statistics Probability Monte Carlo Simulations Linear Regression Curve Fitting Time Series Interpolation, Derivatives, Integrals Digital Signal Processing Machine Learning Computational Geometry
- 4. Statistical Modeling Two of the most commonly used statistical models are: • Regression Models: Predicting Numeric Values • Probability Distributions: Models of probability
- 5. Regression Support in Math Expressions Linear • Simple Linear Regression • Multivariate Linear Regression Non-linear • Loess Regression: Bivariate, robust, often used for time series modeling. • Polynomial Curve Fitting: Bivariate, general purpose modeling of curves. • Harmonic Curve Fitting: Bivariate, sine wave modeling. • Gaussian Curve Fitting: Bivariate, modeling a Gaussian peak. • KNN Regression: Multivariate, robust, distance based, very flexible.
- 6. Probability Distributions • Statistical models of probability • Used to model risk • Perform simulations • Natural outlier detectors • Math Expressions supports many of the commonly used probability distributions • There is an important relationship between the Normal Distribution and Regression Models
- 8. Use Case • Detect unusual slowness in the network. • Use Simple Linear Regression to model the linear relationship of file sizes and response times. • Use the regression model to detect higher than expected response times.
- 10. Response
- 11. Set the Samples to a Variable
- 12. Response
- 13. Vectorize the File Sizes
- 14. Response
- 16. Response
- 17. Plotting the Responses (Sunplot by: Michael Suzuki)
- 18. Response Times
- 20. Response
- 21. Cache the Model
- 23. Statistical Analysis of the Residuals • Residuals – Calculate the Residuals – Describe – Normality Test – Residual Plot (Homoscedasticity) – Model the Residuals
- 24. Residuals • The difference between the actual value and the predicted value is called the residual. • Residuals represent the error of a regression model. • Residuals can be analyzed and modeled as a probability distribution. • In an ideal scenario the residuals well be normally distributed and homoscedastic.
- 25. Predictions
- 26. Response
- 28. Response
- 29. Describe
- 30. Response
- 32. Response
- 33. Cache the Residuals Distribution
- 34. Residual Plot
- 35. Scatter Plot of Residuals
- 36. Prediction
- 37. Getting the Cached Model
- 38. Response
- 40. Response
- 42. Response
- 45. Response
- 46. Thank you! Joel Bernstein Senior Data Engineer, Lucidworks @jbernste2 #Activate18 #ActivateSearch