Architecting Data Lake on AWS by the Data Engineering Team at HiFX IT
- 1. Architecting Data Lakes on AWS with HiFX Established in the year 2001, HiFX is an Amazon Web Services Consulting Partner that has been designing and migrating applications and workloads in the cloud since 2010. We have been helping organisations to become truly data driven by building data lakes in AWS since 2015.
- 2. 2 The Challenges Lack of agility and accessibility for data analysis which would aid the product team to make smart business decisions and improve strategies Increasing volume and velocity of data. With new digital properties getting added, there was a need to design the collection and storage layers that would scale well. Dozens of independently managed collections of data, leading to data silos. Having no single source of truth was leading to difficulties in identifying what type of data is available, getting access to it and integration. Poorly recorded data. Often, the meaning and granularity of the data was getting lost in processing. Dozens of independently managed collections of data, leading to data silos. Having no single source of truth was leading to difficulties in identifying what type of data is available, granting access and integration. 04 03 02 01
- 3. Our Journey from Data to Decisions with an AWS powered Data Lake Connecting dozens of data streams and repositories to a unified data pipeline enabling near realtime access to any data source Engineering well designed big data stores for reporting and and exploratory analysis Architect a secure, well governed data lake to store all data in a raw format. S3 is the fabric with which we have woven the solution. Processing data in streams or batches to aid analytics and machine learning, supplemented by smart workflow management to orchestrate the tasks Dynamic dashboards and visualisations that makes data tell stories and help drive insights. Offering recommendations and predictive analytics off the data in the data lake
- 4. COLLECT STORE PROCESS CONSUME Scribe (Collector) Accumulo (Storage) Acccumulo is the data consumer component responsible for reading data from the event streams (Kinesis Streams), performing rudimentary data quality checks and converting data to Avro Format before loading it to the Data Lake Our Data Lake in S3 captures and store raw data at scale for a low cost. It allows us to store many types of data in the same repository while allowing to define the structure of the data at the time when it is used scribe accumulo Scribe collects data from the trackers and writes them to Kinesis Streams It is written in Go and engineered for high concurrency, low latency and horizontally scalability Currently running on two c4.large instances, our API latency for 50 percentile is 12.6ms and 75 percentile is 36ms. This is made possible because of the consistent and predicable performance of Kinesis
- 5. COLLECT STORE PROCESS CONSUME
- 6. 6 Why Amazon S3 For Data Lake ? Performance relatively lower than an HDFS cluster, but doesn't affect our workloads significantly. EMRFS with consistent view (backed by DynamoDB) works really well Native support for versioning, tiered-storage (Standard, IA, Amazon Glacier) via life-cycle policies and Secure– SSL, client/server-side encryption Dozens of independently managed collections of data, leading to data silos. Having no single source of truth was leading to difficulties in identifying what type of data is available, getting access to it and integration. Unlimited number of objects and volume of data, along with 99.99% availability and 99.999999999% durability. Lower TCO and easier to scale than HDFS Decoupled storage and compute allowing multiple & heterogeneous analysis clusters to use the same data 04 03 02 01
- 7. Prism (Processor) Lens (Consumer) Custom built reporting & visualisation app to help business owners to easily interpret, visualise and record data and derive insights Detailed Analysis of KPIs, Event Segmentation, Funnels, Search Insights, Path Finder, Retention/Addiction Analysis etc powered by Redshift and Druid. Using Pgpool to cache Redshift queries. process consume Unified Processing Engine using Apache Spark running on EMR written in Scala Airflow is used to programmatically author, schedule and monitor workflows Prism generates data for tracking KPIs and perform funnel, pathflow, retention and affinity analysis. It also include machine learning workloads that generate recommendations and predictions COLLECT STORE PROCESS CONSUME
- 8. COLLECT STORE PROCESS CONSUME
- 9. KPIs Product relationship Understand which products are viewed consecutively Product affinity Understand which products are purchased together Sales Hourly, daily, weekly, monthly, quarterly, and annual Average market basket Average order size Cart abandonment rate Shopping cart abandonment rate Days/ Visits To "Purchase" The average number of days and sessions from the first website interaction to purchase. Cost per Acquisition (Total Cost of Marketing Activities) / (# of Conversions) Repeat purchase rate What % of our customers are repeat customers
- 10. Product page performance Measuring product performance The scatter plot compares the number of unique users that view each product with the number of unique users that add the product to basket, with the size of each dot being the number of uniques that buy the product. Any products located in the lower right corner are highly trafficked but low converting - any effort spent fixing those product pages (e.g. by checking the copy, updating the product images or lowering the price) should be rewarded with a significant sales uplift, given the number of people visiting those pages
- 11. Measuring product performance In contrast, products located in the top left of the plot are very highly converting, but low trafficked pages. We should drive more traffic to these pages, either by positioning those products more prominently on catalog pages, for example, or by spending marketing dollars driving more traffic to those pages specifically. Again, that investment should result in a significant uplift in sales, given how highly converting those products are. Similarly, products in the lower left corner are performing poorly - but it is not clear whether this is because they have low traffic levels and /or are poor at driving conversions. We should invest in improving the performance of these pages, but the return on that investment is likely to be smaller (or harder to achieve) than the other two opportunities Product page performance
- 12. Identifying products / content that go well together Market basket analysis is an Association rule learning technique aimed at uncovering the associations and connections between specific products in our store In a market basket analysis, we look to see if there are combinations of products that frequently co-occur in transactions. We can use this type of analysis to: • Inform the placement of content items on sites, or products in catalogue • Drive recommendation engines (like Amazon’s customers who bought this product also bought these products…) • Deliver targeted marketing (e.g. emailing customers who bought products specific products with other products and offers on those products that are likely to be interesting to them.)
Editor's Notes
- #6: Trackers allow to collect data from any type application (web, mobile), service or device. All trackers adhere to the predefined Tracker Protocol. Send data asynchronously, and hence would not affect the performance. Collectors are stateless and horizontally scalable Each shard in kinesis steam can support reads upto 2MB per second and writes upto 1,000 records / 1MB per second. Scribe and Accumulo automatically detects new shards and scales. Accumulo KCL Java App that buffers the events and upload the batches as Avro files to the Data Lake DAGs in Airflow pull dimension and offline data and loads them to the Data Lake
- #9: Streaming workloads for near-realtime reports ( news ) and batch for daily reports (classifieds) EMR with instance fleets provides a cost effective way to process data Data processing involves quality checks, cleansing, reconciling and enrichment. Subset of data ( sans page views, data beyond last 2 years) sent to druid & redshift. All data (historical) stored as parquet in S3 with lifecycle policy. Athena can point to this data for ad hoc analysis Druid for realtime data and aggregate queries that do not require join. Redshift for everything else. LENS built with React using nvd3 charting library. Built for multi tenancy with fine-grained ACLs. Apis powered by Go Recommendations powered by DyanamoDB ( predictable performance and no need to sort on multiple fields)