Predicting Moscow Real Estate Prices with Azure Machine Learning
4 likes516 views
The document summarizes experiments conducted by Team D-Hawks to predict real estate prices in Moscow using data from Kaggle competitions. It describes five different experiments that varied the features used, data cleaning techniques, and machine learning models. The winning experiment used parallel data cleaning paths and multiple boosted decision tree models to achieve the lowest root mean squared error. The team's work demonstrated that feature selection, additional data sources, and testing multiple approaches can improve price prediction accuracy.
Predicting Moscow Real Estate Prices with Azure Machine Learning
- 1. Predicting Real Estate Prices in Moscow A Kaggle Competition University of Washington Professional & Continuing Education BIG DATA 220B SPRING 2017 FINAL PROJECT Team D-Hawks Leo Salemann, Karunakar Kotha, Shiva Vuppala, John Bever, Wenfan Xu Keywords: Big Data, Kaggle, Machine Learning, Azure ML Studio, Boosted Decision Tree, Neural Network, Regression, Tableau
- 2. Problem Description & Datasets Input Data Description Features Observations Housing Data Property, neighborhood, sales date & price 292 30,473 Macroeconomics Daily commodity prices, indicators like GDP 100 2,485 Data Dictionary Feature Definitions Shapefiles Spatial data for maps
- 3. ML Studio Flow 1. Load data; select columns 2. Edit Metadata (set datatype) 3. Clean Missing Data 4. Clip, Normalize, Split 5. Train & Evaluate Boosted Decision Tree, Neural Network
- 4. Azure ML Studio Experiments - Variations Name Strategy Experiment Characteristics Cols Rows Root Mean Squared Error RMSE / STDEV(price) Wenfan Baseline ● Basic 12 real estate features ● Tried 4 regression models, kept 2 13 27,909 2,505,749.58 0.524203184 Leo Incremental add ● Incrementally add more real estate features ● Omit macroeconomic features ● Detailed Human-in-The-Loop process 64 15,693 2,573,721.30 0.538422878 Shiva Feature Selection Pre-processor ● Separate Experiment for Feature Selection (Permutation Feature Importance) ● Joined Macro Data ● Added Retail-specific Features ● Added Decision Forest Regression Module 21 30,471 2,425,862.34 0.507490762 Karunakar Filter Based Feature Selection ● Filter Based Feature Selection ● Boosted Decision tree ● Decision forest regression 38 14,853 3,054,675.32 0.639038531 John Parallel Cleansing Paths ● Joined Macro Data ● Start with all fields, gradually remove ● Parallel cleansing paths (set to zero; set to 391 30,471 2,263,084.20 0.473437552
- 5. The Winning Experiment 2. Clean Missing Data - Try three Modes a. Custom Value Substitution (a fixed value i.e. 0) b. Replace with Mean c. Replace using Probabilistic PCA 3. Clip, Normalize Split (same for all 3 paths) - Handling Categorical & Continuous Variables - Outlier clipping (per-value; not via SQL) - Data Normalization or Feature Scaling 4. Train & Evaluate - Compare Three different models a. Poisson Regression b. Neural Network Regression c. Boosted Decision Tree Regression BA C 1. Collecting Data
- 7. Predictions Based on Normalized Inputs 389 input columns
- 8. Visualization Predicted price VS. Real price ● Actual “waveform” tracks quite well (peaks and valleys line up) ● Fairly consistent delta - always undershooting by about 500K Rubles
- 9. VisualizationVisualization Error analysis based on House property: square meters
- 10. VisualizationVisualization Error analysis based on Geometry property: districts
- 11. Conclusion & Further Work TWL (Today We Learned) ● Azure ML Studio is great for trying multiple techniques in parallel (try that in python!) ● Many ways to approach the problem. ○ Effort required varies a lot … ○ So does the quality of the results. Next time … ● Watch those row counts … did you lose any? ● Deploy Web Service earlier and more often. Someday/Oneday … ● Use different models for different subclasses of real estate.
- 12. THANKYOU!
- 13. Appendix Experiment Variation Details & Results More experiment screenshots
- 14. Azure ML Studio Experiments - Variations Name Strategy Experiment Characteristics Regression Models Notes Wenfan Baseline ● Basic 12 real estate features ● Boosted Decision Tree ● Neural Network ● Bayesian Linear ● Linear Kept Boosted Decision Tree and Neural Network; dropped the others. Leo Incremental add ● Incrementally add more real estate features ● Omit macroeconomic features ● Boosted Decision Tree ● Neural Network Detailed Human-In-The-Loop (HITL) process. Shiva Feature Selection Pre-processor, add Macro & Retail ● Joined Macro Data ● Added Retail-specific Features ● Boosted Decision Tree ● Decision Forest Regression Separate Experiment for Feature Selection (Permutation Feature Importance) Karunakar Filter Based Feature Selection ● Filter Based Feature Selection ● Remove features that aren’t helping ● Boosted Decision Tree ● Forest Regression Kept Filter Based Feature,Boosted Decision tree and Forest regression John Parallel Cleansing Paths - set to 0 vs. median vs. Probabilistic PCA ● Joined Macro Data ● Start with all fields, gradually remove ● Parallel cleansing paths ● Multiple Boosted Decision Tree Models ● Poisson ● Neural Network Multiple simultaneous parallel paths
- 15. Evaluation Metrics Name Strategy Cols Rows Mean Absolute Error Root Mean Squared Error RMSE / STDEV(price) Relative Absolute Error Relative Squared Error Coefficient of Determination Wenfan Baseline 13 27,909 1,448,475.24 2,505,749.58 0.524203184 0.535641 0.386980 0.6130200 Leo Incremental add 64 15,693 1,577,436.18 2,573,721.30 0.538422878 0.507266 0.284116 0.7158840 Shiva Feature Selection Pre-processor 21 30,471 1,390,695.31 2,425,862.34 0.507490762 0.521245 0.352367 0.6476330 Karunakar Filter Based Feature Selection 38 14,853 1,874,864.85 3,054,675.32 0.639038531 0.626830 0.439601 0.5603993 John Parallel Cleansing Paths 391 30,471 1,358,929.12 2,263,084.20 0.473437552 0.487444 0.315758 0.6842420
- 16. Shiva’s Pre-Processor Experiment Permutation Feature Importance algorithm to compute importance scores for each of the feature variables of dataset. 1.Load Housing and macro data; Join data 2. Select ALL columns Edit Metadata (set datatype) 3. Split Data 4. Add Permutation Feature Importance Model. Conn: L: Train Model, R: Dataset Works only for Regression or Classification. 5. Execute Permutation Feature Importance (40 mins). 6. Result lists top most scored features in the dataset.
- 17. Karunakar Pre-Processor Experiment Boosted decision tree algorithm in a decision tree ensemble tends to improve accuracy with some small risk of less coverage. 1.Load Housing data 2. Select columns, Edit Metadata (set datatype) 3. Apply SQL transformations. 4. Filter based feature selection ,normalize data and split data. 5.choosed Boosted decision tree and decision tree regression to choose the best predictive. 6 Apply train and score model for each decision algorithm . 7. Evaluate the data model .
- 18. Karunakar Variation 1. Filter Based Feature Selection (remove features that aren’t helping) 2. Decision Forest Filter Based Feature Selection: 1. Feature selection is the process of selecting those attributes(Columns) in dataset that are most relevant to the predictive modeling. 2. By choosing the right features, it can potentially improve the accuracy and efficiency of classification. 3. Filter Based Feature Selection module to identify the columns in your input dataset that have the greatest predictive power. Pearson Correlation: 1. Pearson’s correlation statistics or Pearson’s correlation coefficient is also known in statistical models as the r value. For any two variables, it returns a value that indicates the strength of the correlation. 2. Pearson's correlation coefficient is computed by taking the covariance of two variables and dividing by the product of their standard deviations. The coefficient is not affected by changes of scale in the two variables.
- 19. Karunakar Variation Decision Forest Regression Model: Decision trees are nonparametric models that perform a sequence of simple tests for each instance, traversing a binary tree data structure until a leaf node (decision) is reached. Decision trees have these advantages: 1. They are efficient in both computation and memory usage during training and prediction. 2. They can represent non-linear decision boundaries. 3. They perform integrated feature selection and classification and are resilient in the presence of noisy features. This regression model consists of an ensemble of decision trees. Each tree in a regression decision forest outputs a Gaussian distribution by way of prediction. An aggregation is performed over the ensemble of trees to find a Gaussian distribution closest to the combined distribution for all trees in the model.