Detecting Mobile Malware with Apache Spark with David Pryce
1 like•377 views
The document summarizes David Pryce's presentation on detecting mobile malware with Apache Spark. It discusses how mobile malware is a growing problem, the features extracted from mobile apps to train a machine learning model, and how Apache Spark was used to build a scalable production-ready model that classifies apps in real-time. It also briefly discusses Wandera's data science team and their focus on solving real-world mobile security problems.
Detecting Mobile Malware with Apache Spark with David Pryce
- 1. David Pryce PhD, Wandera Detecting Mobile Malware with Apache Spark #SAISDev9
- 2. #SAISDev9 Summary • The problem: Mobile-first malware detection • The data and features • The Machine Learning (ML) model • Why Apache Spark? • Making it production ready • Data Science @ Wandera !2
- 3. #SAISDev9 The power of enterprise mobility !3 Devices are prone to security threats Concerns around appropriate usage Data usage costs are opaque and spiraling Potentially exposing sensitive data Seamless internal communication Added flexibility to working hours Access to more apps and productivity tools E-mail and other services available anywhere
- 4. #SAISDev9 Happy hunting ground for attackers !4 “Mobile threats can no longer be ignored” - AUGUST 2017 - GARTNER MARKET GUIDE TO MOBILE THREAT DEFENSE 100% Mobile malware growth in 2016 435% High severity threats (CVSS) growth in 2016 80% of organizations experienced mobile phishing attack 38% of hackers bypass endpoint defense using social engineering
- 6. #SAISDev9 The Rise of Mobile Malware !6 Credit: GData 2017
- 7. Our objectives: Identify and Classify !7 SMS MALWARE TYPES Ransomware Spyware Banker Trojan Rooting Adware
- 8. #SAISDev9 Why is this a novel problem? • Mobile malware is on the rise • Signature based detection is no longer scalable or effective • We needed a solution that could • work across both known and unknown threats; • effectively protect our customers; and • enable threat research to quickly identify new outbreaks • First solution = signatures and lists • Our solution = machine learning! !8
- 9. #SAISDev9 The data … !9 Good and bad apps • Source 1: official app stores • Source 2: seen in our devices • Source 3: seen by our gateway 3rd-party threat intelligence External input verified for labels (supervised learning) Currently storing: ~2 million labelled apps +
- 10. #SAISDev9 … the features … !10 Baidu 2016
- 11. #SAISDev9 … how we extract them !11 Direct metadata extraction • Total unique fields for all apps ~ 500,000 • A typical app ~ 10+ fields • SPARSE VECTOR Solution: • Hashing function (vector to indices) • Allows for fast retrieval • With big enough map (2^20) to avoid clashes • DENSE VECTOR
- 12. #SAISDev9 … and how the Machine Learns !12 • Selected model = Logistic Regression ◦ Models tried = (LogReg, SVM, Decision Tree) • K-fold cross validation to select best parameters • Accuracy: 0.96 

- 13. #SAISDev9 Why Apache Spark? !13 Model persistence PMML paradigm already integrated Truly big data Millions of data points, millions of fields Ease of use Fast, easy and iterative. From EDA to app in days. Scala and python API. Deployment and Scale From local to cluster is easy!
- 14. #SAISDev9 !14 Wandera 2018 Production Ready?
- 15. #SAISDev9 P.M.M.L • Predictive Model Markup Language • Industry standard • Pro: Language agnostic, REST API, good algo coverage • Con: large file size !15
- 16. #SAISDev9 Production Ready? !16 • Saving to PMML (ML vs MLlib / DF vs RDD) • DataFrame API - doesn’t have PMML functionality (yet) • Hacked PMML to get probabilities for predictions • Size of model ~ 20Mb (compressed) • Overall time to train: less than 2 hours on a big enough cluster F
- 17. #SAISDev9 Live Scoring !17 Extracts features & scores app User installs new app 1 2 If score > 0.9 INVESTIGATE / NOTIFY 3
- 18. #SAISDev9 Data Science @ Wandera !18 • Cross-disciplinary team of scientists, analysts & developers • Focus on solving real-world problems in a real-time, distributed network • Global team with presence in USA, London, UK and Czech Republic = Innovative Research + Scalable Architecture + Efficient Feature Delivery
- 20. #SAISDev9 Appendix 1: model testing results !20 Wandera 2018