High Performance Cooperative Distributed Systems in Adtech
1 like276 views
The document presents insights from a talk by Stan Rosenberg at QCon New York about building high-performance cooperative distributed systems, particularly in the adtech sector. It highlights the importance of reliability, data quality, and cooperation in fighting ad fraud, while discussing technical challenges such as scaling, low-latency requirements, and managing tech debt. Additionally, it emphasizes the potential of cloud resources and open-source solutions for achieving effective ad fraud detection and data processing.
![Intro
Design
Implementation
Reliability
Lessons Learned
Summary
Know Your Data Structures
1000 references to main memory (e.g., linear scan of linked-list)
is ≈ 100 micros; ( 1
100) × 106 = 10, 000 reqs/second
1000 references to L2 cache is ≈ 7 micros; (1
7) × 106 = 142, 857
reqs/second
linear search is slower than binary, right?
int cnt = 0;
f or ( int i = 0; i n ; i++)
cnt += ( arr [ i ] key ) ;
return cnt n arr [ cnt ] == key ;
QCon, New York, June 26, 2019 20/39](https://image.slidesharecdn.com/untitled-191031084847/85/High-Performance-Cooperative-Distributed-Systems-in-Adtech-22-320.jpg)
![Intro
Design
Implementation
Reliability
Lessons Learned
Summary
Data Ingest ETL
validate each request and apply (payload) limits
translate JSON to snappy-compressed Avro
use Disruptor to consume encoded Avro byte[]
append to Avro data file for current 5-min batch
upload to GCS (throttle to reduce GC pressure)
QCon, New York, June 26, 2019 23/39](https://image.slidesharecdn.com/untitled-191031084847/85/High-Performance-Cooperative-Distributed-Systems-in-Adtech-25-320.jpg)
High Performance Cooperative Distributed Systems in Adtech
- 1. Intro Design Implementation Reliability Lessons Learned Summary High Performance Cooperative Distributed Systems in Adtech Stan Rosenberg VP of Engineering Forensiq New York, NY QCon, New York, June 26, 2019 1/39
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ adtech-fraud-detection/
- 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
- 4. Intro Design Implementation Reliability Lessons Learned Summary Prebid Throughput QCon, New York, June 26, 2019 2/39
- 5. Intro Design Implementation Reliability Lessons Learned Summary GC Pauses QCon, New York, June 26, 2019 3/39
- 6. Intro Design Implementation Reliability Lessons Learned Summary Failure happens all the time Ken Arnold, When you design distributed systems, you have to say, "Failure happens all the time." Fallacies of Distributed Computing (Peter Deutsch), The network is reliable. Latency is zero. Bandwidth is infinite. Transport cost is zero. QCon, New York, June 26, 2019 4/39
- 7. Intro Design Implementation Reliability Lessons Learned Summary Past Work QCon, New York, June 26, 2019 5/39
- 8. Intro Design Implementation Reliability Lessons Learned Summary Present Work QCon, New York, June 26, 2019 6/39
- 9. Intro Design Implementation Reliability Lessons Learned Summary Intro Before, Ph.D., Computer Science; Stevens, Hoboken, 2011 Advisor: David A. Naumann Dissertation Title: Region Logic: Local Reasoning for Java Programs and its Automation Recently, building distributed platforms for startups Appnexus (serving ads faster) PlaceIQ (using location to serve ads) VP of Engineering, Forensiq (fighting ad fraud) QCon, New York, June 26, 2019 7/39
- 10. Intro Design Implementation Reliability Lessons Learned Summary Forensiq Overview Comprehensive Fraud and Verification SaaS (MRC certified) Display Verification (viewability measurements, impression blocking) Performance Fraud (stolen attribution, fake action) Online scoring via Prebid, Postbid and S2S APIs Offline scoring via request log import and reputation lists QCon, New York, June 26, 2019 8/39
- 11. Intro Design Implementation Reliability Lessons Learned Summary Fraud Examples QCon, New York, June 26, 2019 9/39
- 12. Intro Design Implementation Reliability Lessons Learned Summary Fraud Examples QCon, New York, June 26, 2019 10/39
- 13. Intro Design Implementation Reliability Lessons Learned Summary Fraud Examples QCon, New York, June 26, 2019 11/39
- 14. Intro Design Implementation Reliability Lessons Learned Summary Call for Cooperation and Collaboration Let’s improve data quality! provide authentic source ip server-side ad-stitching (e.g., AWS Elemental) hides source ip; triggers datacenter traffic MRC notes, “data center traffic is determined to be a consistent source of non-human traffic”. specify location type (OpenRTB 2.5) and source to strengthen spoofing detection provide campaign/source (aggregate) metrics to help detect client-side JS blocking QCon, New York, June 26, 2019 12/39
- 15. Intro Design Implementation Reliability Lessons Learned Summary Performance Requirements (Prebid API) high-throughput – must scale above 1 mil. RPS low-latency – response p99 < 10ms QCon, New York, June 26, 2019 13/39
- 16. Intro Design Implementation Reliability Lessons Learned Summary Daily Bid Volume 100 ∗ 109/86400 ≈ 1.1 ∗ 106 https://fixad.tech/wp-content/uploads/2019/02/4-appendix-on-market-saturation-of-the-systems.pdf QCon, New York, June 26, 2019 14/39
- 17. Intro Design Implementation Reliability Lessons Learned Summary Common Concerns high-throughput low-latency server backend KV store data ingest ETL data pipelines data pipelines Ad Serving: enrichment, budget, attribution, reporting Fraud Detection: enrichment, scoring, reporting QCon, New York, June 26, 2019 15/39
- 18. Intro Design Implementation Reliability Lessons Learned Summary Guiding Principles use NIO use compare-and-swap instead of locks (affects OOOE) use spatial/temporal locality (prefetch,branch predict) minimize coupling and state–keep it simple minimize GC pressure warmup on startup to trigger JIT measure everything with HdrHistogram benchmark everything with JMH and wrk2 QCon, New York, June 26, 2019 16/39
- 19. Intro Design Implementation Reliability Lessons Learned Summary Cloud is fast (enough) modern hypervisor adds negligible overhead ( 5%) consitent performance–“noisy neighbor” is a myth networking – 2Gbps per core; up to 32Gbps per VM partitions are infrequent; high inter-region throughput local storage – NVMe SSDs; read: 300K IOPS, 2GB/sec cloud storage – high-throughput and high-availability strongly consistent (GCS) fast parallel uploads via compose (GCS) QCon, New York, June 26, 2019 17/39
- 20. Intro Design Implementation Reliability Lessons Learned Summary Mechanical Sympathy Understanding the Hardware Makes You a Better Developer https://mechanical-sympathy.blogspot.com/ https://dzone.com/articles/mechanical-sympathy https://groups.google.com/forum/#!forum/mechanical-sympathy QCon, New York, June 26, 2019 18/39
- 21. Intro Design Implementation Reliability Lessons Learned Summary Latency Little’s Law: L = λ × W , whence throughput is ∝ 1 latency QCon, New York, June 26, 2019 19/39
- 22. Intro Design Implementation Reliability Lessons Learned Summary Know Your Data Structures 1000 references to main memory (e.g., linear scan of linked-list) is ≈ 100 micros; ( 1 100) × 106 = 10, 000 reqs/second 1000 references to L2 cache is ≈ 7 micros; (1 7) × 106 = 142, 857 reqs/second linear search is slower than binary, right? int cnt = 0; f or ( int i = 0; i n ; i++) cnt += ( arr [ i ] key ) ; return cnt n arr [ cnt ] == key ; QCon, New York, June 26, 2019 20/39
- 23. Intro Design Implementation Reliability Lessons Learned Summary Disruptor Pattern–Fast Event Processing Disruptor is like Java’s BlockingQueue but waaaaay faster! RingBuffer one compare-and-swap operation to drain the queue pair of sequence numbers for fast atomic reads/writes exploits speculative racing to eliminate locks consumer message batching results in high-throughput QCon, New York, June 26, 2019 21/39
- 24. Intro Design Implementation Reliability Lessons Learned Summary Disruptor Pattern RingBuffer is pre-allocated (data in Wrapper.message) compact – sizeof(disruptor(524,288)) ≈ 14.5MB QCon, New York, June 26, 2019 22/39
- 25. Intro Design Implementation Reliability Lessons Learned Summary Data Ingest ETL validate each request and apply (payload) limits translate JSON to snappy-compressed Avro use Disruptor to consume encoded Avro byte[] append to Avro data file for current 5-min batch upload to GCS (throttle to reduce GC pressure) QCon, New York, June 26, 2019 23/39
- 26. Intro Design Implementation Reliability Lessons Learned Summary Avro Snappy 16 cores, skylake java version 1.8.0_202 @Threads(24), @BenchmarkMode(Mode.Throughput) Benchmark Score Error Units encode 3741337.244 ±81494.37 ops/s encodeCompress 2699393.673 ±40130.622 ops/s decode 2925509.122 ±37078.569 ops/s decodeDecompress 2771921.410 ±60483.905 ops/s Also see zstd: https://facebook.github.io/zstd/ QCon, New York, June 26, 2019 24/39
- 27. Intro Design Implementation Reliability Lessons Learned Summary Data Ingest ETL early ETL cuts out many downstream inefficiencies Avro’s performance is on par with Protobuf (also see below) throttling uploads and downloads is a must to reduce GC eliminate humongous objects (G1) naive batching/parallel upload with compose works well skip write-ahead log–deal with corrupted Avro blocks Codegen makes Avro encoder 2x faster: https://github.com/RTBHOUSE/avro-fastserde QCon, New York, June 26, 2019 25/39
- 28. Intro Design Implementation Reliability Lessons Learned Summary KV Store–why not Aerospike? Pros founded in 2009 (AppNexus was first large deployment) written in C (better resource management in theory) uses Paxos for distributed consensus; heartbeats for node membership supports migrations, rebalancing support cross-datacenter replication Cons No bulk loading index can get large (RIPEMD is 20 bytes but metadata makes it 64 bytes) log-structured filesystem (copy-on-write); runs compaction in background global 32k bins limit (bins are like column qualifiers) QCon, New York, June 26, 2019 26/39
- 29. Intro Design Implementation Reliability Lessons Learned Summary Low latency KV–Voldemort founded in 2009 by LinkedIn (bulk loading main motivator) written in Java simple get/put API uses consistent hashing (similar to Dynamo) to avoid hotspotting bulk loading and readonly store index is compact – uses only 8 bytes of md5(key) index file is mlocked (sort of) supports rebalancing QCon, New York, June 26, 2019 27/39
- 30. Intro Design Implementation Reliability Lessons Learned Summary Voldemort BuildAndPush QCon, New York, June 26, 2019 28/39
- 31. Intro Design Implementation Reliability Lessons Learned Summary Voldemort Readonly Performance QCon, New York, June 26, 2019 29/39
- 32. Intro Design Implementation Reliability Lessons Learned Summary Custom Voldemort added BloomFilter (client-side to reduce RTT) added Avro schema versioning added Union datastore TCP connection pooling is flawed reloads create short-lived spikes (hard to pin index) 2GB limit per chunk (ByteBuffer 32bit signed addressing) rewrite currently in progress to manage resources more efficiently, rewrite Voldemort backend in C++ use UDP (potentially with Aeron) use GCS instead of HDFS QCon, New York, June 26, 2019 30/39
- 33. Intro Design Implementation Reliability Lessons Learned Summary Putting Things Together QCon, New York, June 26, 2019 31/39
- 34. Intro Design Implementation Reliability Lessons Learned Summary Tech. Debt QCon, New York, June 26, 2019 32/39
- 35. Intro Design Implementation Reliability Lessons Learned Summary Top Two Diseases Legacy and Tech. Debt are the top two diseases of any complex software development avoid them at all costs Google often rewrites legacy before it’s out of control; secondary effect, way of transferring knowledge and ownership to newer team members Henderson, Fergus. Software engineering at Google. arXiv preprint arXiv:1702.01715 (2017). QCon, New York, June 26, 2019 33/39
- 36. Intro Design Implementation Reliability Lessons Learned Summary Rapid Reliable Iteration can’t iterate quickly without automated verification (i.e., tests) invest time into test and benchmarking fixtures early (e.g., write emulators) end-to-end (integration) tests, e.g., Selenium, are must-have instrument with metrics and measure everything use design by contract methodology with code reviews Design by contract was coined by Bertrand Meyer in connection with Eiffel. QCon, New York, June 26, 2019 34/39
- 37. Intro Design Implementation Reliability Lessons Learned Summary GCP Managed Infrastructure distributed, highly available, strongly consistent file system (gcs) global latency-based load balancing zero-downtime rolling deploy fast scaling up/down (new instances take 90 sec. to boot) Bigquery (bulk loading, avro/parquet, partitioned tables) Bigtable (hbase on steroids) syncs (lb logs to bigquery, billing to bigquery, etc.) https://serverfault.com/questions/881698/random-failed-to-connect-to-backend-errors-on-gce-lb QCon, New York, June 26, 2019 35/39
- 38. Intro Design Implementation Reliability Lessons Learned Summary Cloud Tech. is mostly mature https://github.com/googleapis/ cloud-bigtable-client/issues/1348 https://github.com/googleapis/google-cloud-java/ issues/3531 https://github.com/googleapis/google-cloud-java/ issues/3534 https://github.com/GoogleCloudPlatform/ bigdata-interop/issues/106 https://github.com/GoogleCloudPlatform/ bigdata-interop/issues/153 QCon, New York, June 26, 2019 36/39
- 39. Intro Design Implementation Reliability Lessons Learned Summary Trust but Verify cost-effective infrastructure is doable but watch out. . . GCP bait switch product tactics stackdriver glb logging (free until insanely expensive) load-balancer user-defined headers (free until . . . ) cloud armor (firewall for glb) (free until . . . ) Managed services are black boxes (with limited observability) DNS delegation misconfiguration was $54k over 6 months (no metrics, logging or anomaly detection) dataproc transient failures (no useful logging to determine root cause) dataproc job non-determinstically “stuck” while committing output QCon, New York, June 26, 2019 37/39
- 40. Intro Design Implementation Reliability Lessons Learned Summary Summary Cloud and OSS is an extremely powerful combination High-throughout in Cloud is fairly easy through right design Low-latency in Cloud is achievable but takes significantly more effort opportunity to build a managed low-latency KV store storage-as-a-service is still emerging–programmable SSDs Fraud is here to stay–cooperation and collaboration with adtech is vital QCon, New York, June 26, 2019 38/39
- 41. Intro Design Implementation Reliability Lessons Learned Summary Questions? EOF QCon, New York, June 26, 2019 39/39
- 42. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ adtech-fraud-detection/