"You can't just turn the crank": Machine learning for fighting abuse on the consumer web
1 like•833 views
The document discusses the challenges and methodologies of employing machine learning to combat various forms of online abuse, including fraud, scams, and misinformation. It details the machine learning workflow, emphasizing the importance of accurate labeling, the need for adaptive models, and the pitfalls of A/B testing in dynamically changing adversarial environments. Recommendations include focusing on bad behavior rather than content, utilizing anomaly detection, and maintaining a robust defense strategy throughout the machine learning process.
"You can't just turn the crank": Machine learning for fighting abuse on the consumer web
- 1. "You can't just turn the crank" Machine learning for fighting abuse on the consumer web David Freeman Research Scientist/Engineer, Facebook Inc. ScAINet 2018 Atlanta, GA USA, 11 May 2018
- 3. What do they try to do? Malware Payment Fraud Scraping Click Fraud Phishing Spam Social Engineering Fake Products Scams "Like" FraudPromotion Fraud Identity Theft What do we see? Fake Reviews Misinfor- mation Financial Theft Account Resale Fundamental question: Which requests are bad? • Perfect for machine learning!
- 4. What could possibly go wrong? Machine learning workflow Label Train Validate Launch Measure Profit!Lots!
- 5. How do we obtain labeled data? (hint: not from your users) Machine learning workflow Label
- 6. • Human labeling of random samples. • Labelers don't always know what they're looking for • Labelers are inconsistent (with themselves and each other) • Labelers get tired (esp. if most samples are good) • Apply crowdsourcing best practices: • Precise definitions, multiple labeling, ML-assisted sampling • But will it scale? Labeling: Gold standard Objective measurement
- 7. • Find high-precision signals of badness • Examples: unusual user-agent, malformed header • DO NOT BLOCK ON THESE SIGNALS • They are controlled by the adversary • When the adversary adapts you will lose visibility • Automatically generate signals using anomaly detection. Labeling: Silver standard Automatic labeling
- 8. • Use whatever you have! • CS data, rules, other models • Mitigate risks of blindness and feedback loops: • Oversample manually labeled examples • Oversample false positives and false negatives when retraining. • Undersample positive examples from previous iterations of this model. • Sample and label examples near the decision boundary Labeling: Bronze standard Be scrappy
- 9. • Users are terrible at reporting. • Product flows bias reporting. • Reports can be gamed. • Reports can serve as an directional measure. Labeling: Iron standard Have your users do the work
- 10. • Segment the problem • e.g. status with link from country X • Downsample intelligently • if your distribution is lumpy, sample from all the lumps • Learning the prior vs. focusing on the bad stuff • no golden rule here -- you have to experiment Assembling a training set Labeling is just the beginning
- 11. {Training set 2 { Model v2 Refreshing your data Don't forget the past! {Training set 1 { Model v1 Mitigation: • Keep old attacks around (exponential decay?) • Keep old models around (raise thresholds?) { Training set N { Model vN
- 12. How do you know your model is ready to go? Machine learning workflow Train Validate
- 13. • Labels aren't perfect • Often miss on recall • Models interact with each other • Use offline P-R and ROC to stack-rank model candidates Validating Performance Don't trust offline replay Model B FP Model A
- 14. • Fundamental A/B testing assumption: Experiment effects are independent of the cohorts chosen The Perils of A/B Testing
- 15. The Perils of A/B Testing A B X • Looks good so far.... Start with a small experiment
- 16. The Perils of A/B Testing A B X • Did the adversary give up or iterate? Roll it out to (almost) everyone — Option 1
- 17. The Perils of A/B Testing A B • Now your experiment is a vulnerability Roll it out to (almost) everyone — Option 2
- 18. • Run new model online in "log-only" mode • Evaluate performance where the new model disagrees with the old one. • ideally via sampling & labeling • Push based on FP/FN tradeoff Using Shadow Mode Prod model FP New model
- 19. How do you figure out if it worked? Machine learning workflow Launch Measure
- 20. True Positives Don't Matter What's happening here? Time Precision
- 21. • Really want # of good users affected • Solution: use one minus specificity (aka FPR) True Positives Don't Matter What's happening here? Time TP Time FP vs. Time FP Time TP 1 TN FP + TN<latexit sha1_base64="FfpNUyvhzZ48h+ueQ2Hy3cgzX50=">AAACDXicbZDLSgMxFIYz9VbrbdSlm2AVBLHMiKDLoiCupEJv0BlKJs20oUlmSDJiGeYF3Pgqblwo4ta9O9/GtJ2Ftv4Q+PKfc0jOH8SMKu0431ZhYXFpeaW4Wlpb39jcsrd3mipKJCYNHLFItgOkCKOCNDTVjLRjSRAPGGkFw6txvXVPpKKRqOtRTHyO+oKGFCNtrK594MIT6IUS4dTjQfSQ1m+zLMfrGjyG43vXLjsVZyI4D24OZZCr1rW/vF6EE06Exgwp1XGdWPspkppiRrKSlygSIzxEfdIxKBAnyk8n22Tw0Dg9GEbSHKHhxP09kSKu1IgHppMjPVCztbH5X62T6PDCT6mIE00Enj4UJgzqCI6jgT0qCdZsZABhSc1fIR4gE402AZZMCO7syvPQPK24TsW9OytXL/M4imAP7IMj4IJzUAU3oAYaAINH8AxewZv1ZL1Y79bHtLVg5TO74I+szx8ZwZry</latexit><latexit sha1_base64="FfpNUyvhzZ48h+ueQ2Hy3cgzX50=">AAACDXicbZDLSgMxFIYz9VbrbdSlm2AVBLHMiKDLoiCupEJv0BlKJs20oUlmSDJiGeYF3Pgqblwo4ta9O9/GtJ2Ftv4Q+PKfc0jOH8SMKu0431ZhYXFpeaW4Wlpb39jcsrd3mipKJCYNHLFItgOkCKOCNDTVjLRjSRAPGGkFw6txvXVPpKKRqOtRTHyO+oKGFCNtrK594MIT6IUS4dTjQfSQ1m+zLMfrGjyG43vXLjsVZyI4D24OZZCr1rW/vF6EE06Exgwp1XGdWPspkppiRrKSlygSIzxEfdIxKBAnyk8n22Tw0Dg9GEbSHKHhxP09kSKu1IgHppMjPVCztbH5X62T6PDCT6mIE00Enj4UJgzqCI6jgT0qCdZsZABhSc1fIR4gE402AZZMCO7syvPQPK24TsW9OytXL/M4imAP7IMj4IJzUAU3oAYaAINH8AxewZv1ZL1Y79bHtLVg5TO74I+szx8ZwZry</latexit><latexit sha1_base64="FfpNUyvhzZ48h+ueQ2Hy3cgzX50=">AAACDXicbZDLSgMxFIYz9VbrbdSlm2AVBLHMiKDLoiCupEJv0BlKJs20oUlmSDJiGeYF3Pgqblwo4ta9O9/GtJ2Ftv4Q+PKfc0jOH8SMKu0431ZhYXFpeaW4Wlpb39jcsrd3mipKJCYNHLFItgOkCKOCNDTVjLRjSRAPGGkFw6txvXVPpKKRqOtRTHyO+oKGFCNtrK594MIT6IUS4dTjQfSQ1m+zLMfrGjyG43vXLjsVZyI4D24OZZCr1rW/vF6EE06Exgwp1XGdWPspkppiRrKSlygSIzxEfdIxKBAnyk8n22Tw0Dg9GEbSHKHhxP09kSKu1IgHppMjPVCztbH5X62T6PDCT6mIE00Enj4UJgzqCI6jgT0qCdZsZABhSc1fIR4gE402AZZMCO7syvPQPK24TsW9OytXL/M4imAP7IMj4IJzUAU3oAYaAINH8AxewZv1ZL1Y79bHtLVg5TO74I+szx8ZwZry</latexit><latexit sha1_base64="FfpNUyvhzZ48h+ueQ2Hy3cgzX50=">AAACDXicbZDLSgMxFIYz9VbrbdSlm2AVBLHMiKDLoiCupEJv0BlKJs20oUlmSDJiGeYF3Pgqblwo4ta9O9/GtJ2Ftv4Q+PKfc0jOH8SMKu0431ZhYXFpeaW4Wlpb39jcsrd3mipKJCYNHLFItgOkCKOCNDTVjLRjSRAPGGkFw6txvXVPpKKRqOtRTHyO+oKGFCNtrK594MIT6IUS4dTjQfSQ1m+zLMfrGjyG43vXLjsVZyI4D24OZZCr1rW/vF6EE06Exgwp1XGdWPspkppiRrKSlygSIzxEfdIxKBAnyk8n22Tw0Dg9GEbSHKHhxP09kSKu1IgHppMjPVCztbH5X62T6PDCT6mIE00Enj4UJgzqCI6jgT0qCdZsZABhSc1fIR4gE402AZZMCO7syvPQPK24TsW9OytXL/M4imAP7IMj4IJzUAU3oAYaAINH8AxewZv1ZL1Y79bHtLVg5TO74I+szx8ZwZry</latexit>
- 22. Not so fast! Machine learning workflow Profit!Adapt!
- 23. What not to Do (I) Show the adversary what your limits are Message 500 people Message 400 people Message 300 people 🛑 🛑 ✅
- 24. • Introduce delay in blocking response (and/or) • Undo the damage without telling the user. What to do (I) Don't give immediate feedback
- 25. "We don't want to be the ones solving the CAPTCHAs" What not to Do (II) Look for specific content to block
- 26. What to Do (II) Focus on bad behavior, not only bad content
- 27. What to Do (III) Use data the adversary doesn't know/control
- 28. Scoring at Entry Points prevent access to accounts Clustering, Anomaly Detection prevent accounts from doing damage User Reporting find false negatives Behavioral Analysis detect bad activityIncreasing speed More information available What to Do (IV) Defense in depth
- 29. • Think about each step of the ML process. • It's hard to build a good training set. • Adversarial adaptation breaks many assumptions. • Control the data & the response. Take aways Thanks to: Hervé Robert, Isaac Fullinwider, Henry Lu, Sagar Patel, Hongyang Li, Nektarios Leontiadis