Scalable and adaptable typosquatting detection in Apache Metron
Download as PPTX, PDF0 likes372 views
The document discusses Apache Metron, a cybersecurity analytics solution designed for centralized security monitoring and anomaly detection across various data sources. It highlights the framework's use of the Hadoop stack for data processing, detailing methods for enriching data, particularly in relation to typosquatting attacks that mislead users into malicious links. The document also outlines strategies for detecting typosquatting through enrichment techniques while addressing challenges in scalability and recency of threat detection.
Scalable and adaptable typosquatting detection in Apache Metron
- 1. 1 © Hortonworks Inc. 2011–2018. All rights reserved Typosquatting Detection Apache Metron Casey Stella & Mike Miklavcic Principal Software Engineer & VP Apache Metron Mike Miklavcic Staff Software Engineer & PMC Apache Metron
- 2. 2 © Hortonworks Inc. 2011–2018. All rights reserved Apache Metron: A Cybersecurity Analytics Solution • Metron provides a scalable advanced security analytics framework to offer a centralized tool for security monitoring and analysis • Ultimately, this means that we provide a solution to ingest, enrich and detect anomalies in disparate data sources • Metron was initiated at Cisco in 2014 as OpenSOC • Metron was submitted to the Apache Incubator in December 2015 • Metron graduated to a top level project in April 2017
- 3. 3 © Hortonworks Inc. 2011–2018. All rights reserved Apache Metron: The Stack • We aggressively use the Hadoop stack for both batch as well as streaming processing of data. • We use • Apache Zookeeper for distributed configuration management • Apache HBase for random access key/value data • Apache Storm as a stream processing framework • HDFS for long-term storage of processed data • Apache Solr and Elasticsearch for low latency querying • We’ve built a UI to display these alerts to security analysts and allow them to manage them.
- 4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
- 5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
- 6. 6 © Hortonworks Inc. 2011–2018. All rights reserved
- 7. 7 © Hortonworks Inc. 2011–2018. All rights reserved Enrichment by any means necessary • There are a couple of different ways to enrich, each with their own semantics Changing Scope Retrieval Semantics Ingestion Semantics Metron Solution Static/Slow-moving Event Key/Value Lookup Batch HBase Enrichment Static/Slow-moving Event Complex Batch Summarizer Dynamic Event Key/Value Lookup Streaming Streaming HBase Enrichment Static/Slow-moving Event Complex Batch Model as a Service Dynamic Multi-event Complex Streaming Profiler
- 8. 8 © Hortonworks Inc. 2011–2018. All rights reserved Typosquatting • Typosquatting is a form of attack that uses typos in URLs to trick unsuspecting users into clicking on links • Typical types of typos targeted are • homoglyphs (l -> 1) • Subdomain (amazon -> am.az.on) • Kerning errors (m -> rn) • This is an extremely common attack as part of a phishing or spearphishing attack. • Users are sent emails with URLs to click on that are hosted on typosquatted domains • The users’ credentials and personal information are then harvested
- 9. 9 © Hortonworks Inc. 2011–2018. All rights reserved Typosquatting – Approaches • Typical approaches involve • generating the set of typosquatted domains from known good domains • checking incoming traffic against that set of typosquatted domains • Generally the challenges around potential solutions break down into • Scale – Even for modest numbers of domains, the number of records can grow quite large. The Top Alexa 10k domains has on the order of 3 million potential typosquatted domains. • Recency – Strategies involving just reference sets of known good domains can fail to spot targeted phishing attacks from advanced actors
- 10. 10 © Hortonworks Inc. 2011–2018. All rights reserved Typosquatting – Approaches in Metron • Generally, this kind of problem is an enrichment in Metron • We ingest DNS logs • For each DNS log, we want to tag domains visited with potentially typosquatted or not by consulting an enrichment • There are two distinct ways to handle this style of problem in Metron • We could treat this as an Hbase Enrichment and use stellar to lookup the domain in Hbase • is_typosquatted := ENRICHMENT_EXISTS(‘typosquatted_domains’, ip_dst_addr, ‘enrichments’, ‘t’) • This
- 11. 11 © Hortonworks Inc. 2011–2018. All rights reserved Typosquatting – HBase Enrichment • We could treat this as an Hbase Enrichment and use stellar to lookup the domain in Hbase • Use DNS Twist to generate typosquatted domains from regular domains • Ingest via our enrichment loader into HBase • is_typosquatted := ENRICHMENT_EXISTS(‘typosquatted_domains’, ip_dst_addr, ‘enrichments’, ‘t’) • This has a couple of issues • Relies upon a pass over the data locally with DNS twist • Depending on how often we add new domains, typosquatted domains can be large • We have to keep this table up-to-date with known good domains to catch the long tail
- 12. 12 © Hortonworks Inc. 2011–2018. All rights reserved Typosquatting – Summarizer • Note that this boils down to a set inclusion check; Bloom filters are made for this • Are we willing to accept false positives? • Typosquatting is insufficient as a signal to stand on its own, so the impact of a false positive is limited • We can control the false positive rate • The Summarizer is a utility that we have created that • Pass over the data either locally or via a MR job • Applies a transformation (in Stellar) in the map phase • BLOOM_ADD(filter, DOMAIN_TYPOSQUAT(domain)) • Reduces in the reduce phase • BLOOM_MERGE(filters) • Writes the summarized object to HDFS
- 13. 13 © Hortonworks Inc. 2011–2018. All rights reserved Typosquatting – Summarized Enrichment • We can now use the bloom filter that we output from the summarizer to tag a domain as typosquatted • is_typosquatted := BLOOM_EXISTS(OBJECT_GET(‘/path/to/filter’, ip_dst_addr)) • Furthermore, we can use the DOMAIN_TYPOSQUAT stellar function with the profiler to add fill in the gaps • Capture bloom filters for the typosquatted domains which have been visited more than k times in the timerange of your choice • Determine if a domain is a typosquatted domain by consulting the reference bloom filter created by the summarizer and the bloom filter from the profiler • In the future, we’re adding a Stream Summary sketch to make this top-k more scalable
- 14. 14 © Hortonworks Inc. 2011–2018. All rights reserved DEMO
- 15. 15 © Hortonworks Inc. 2011–2018. All rights reserved Questions?
- 16. 16 © Hortonworks Inc. 2011–2018. All rights reserved Thank you Come visit us at the Hortonworks Booth and attend the Cybersecurity Birds of a Feather on Wednesday!