Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
43 likes6,866 views
The document discusses LinkedIn's evolution in its data ecosystem, emphasizing the importance of adapting to change through better data modeling and unified metrics platforms. It outlines the complexities involved in tracking user engagement and the need for clear communication and contracts between data producers and consumers. The authors advocate for a structured approach to manage data quality and culture while implementing a robust infrastructure to support ongoing changes.
![How do we calculate a metric: ProfileViews
PageViewEvent
Record 1:
{
"header" : {
"memberId" : 12345,
"time" : 1454745292951,
"appName" : {
"string" : "LinkedIn"
"pageKey" : "profile_page"
},
},
"trackingInfo" : {
["vieweeID" : "23456"],
...
}
}
Metric:
ProfileViews = sum(PageViewEvent)
where pageKey = profile_page
PageViewEvent
Record 1:
{
"header" : {
"memberId" : 12345,
"time" : 1454745292951,
"appName" : {
"string" : "LinkedIn"
"pageKey" : "new_profile_page"
},
},
"trackingInfo" : {
["vieweeID" : "23456"],
...
}
}
or new_profile_page](https://image.slidesharecdn.com/strata2016nyc-shirshankaandyael-publish-160930191025/85/Strata-2016-Architecting-for-Change-LinkedIn-s-new-data-ecosystem-10-320.jpg)
![PageViewEvent
Record 1:
{
"header" : {
"memberId" : 12345,
"time" : 1454745292951,
"appName" : {
"string" : "LinkedIn"
"pageKey" : "profile_page"
},
},
"trackingInfo" : {
["vieweeID" : "23456"],
...
}
}
CASE
WHEN trackingInfo["profileIds"] ...
WHEN trackingInfo["profileid"] ...
WHEN trackingInfo["profileId"] ...
WHEN trackingInfo["url$profileIds"] ...
WHEN trackingInfo["11"] LIKE '%profileIds=%' THEN SUBSTRING(trackingInfo["11"],9,60)
WHEN trackingInfo["12"] LIKE '%priceIds=%' THEN SUBSTRING(trackingInfo["12"],9,60)
ELSE NULL
END AS profile_id
Evolution as we mature and grow...
Metric: ProfileViews = sum(PageViewEvent
where
pagekey = profile_page and
memberID != trackinginfo[vieweeID] )](https://image.slidesharecdn.com/strata2016nyc-shirshankaandyael-publish-160930191025/85/Strata-2016-Architecting-for-Change-LinkedIn-s-new-data-ecosystem-11-320.jpg)
![PageViewEvent
Record 1:
{
"header" : {
"memberId" : 12345,
"time" : 1454745292951,
"appName" : {
"string" : "LinkedIn"
"pageKey" : "profile_page"
},
},
"trackingInfo" : {
["vieweeID" : "23456"],
...
}
}
We wanted to move to better data models
LI_ProfileViewEvent
Record 1:
{
"header" : {
"memberId" : 12345,
"time" : 4745292951145,
"appName" : {
"string" : "LinkedIn"
"pageKey" : "profile_page"
},
},
"entityView" : {
"viewType" : "profile-view",
"viewerId" : “12345”,
"vieweeId" : “23456”,
},
}](https://image.slidesharecdn.com/strata2016nyc-shirshankaandyael-publish-160930191025/85/Strata-2016-Architecting-for-Change-LinkedIn-s-new-data-ecosystem-13-320.jpg)
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
- 1. Architecting for change: LinkedIn's new data ecosystem Sept 28, 2016 Shirshanka Das, Principal Staff Engineer, LinkedIn Yael Garten, Director of Data Science, LinkedIn @shirshanka, @yaelgarten
- 2. Design for change. Expect it. Embrace it.
- 3. Product Change Technology Culture & Process Learnings
- 4. The Product Change: Launch a completely rewritten LinkedIn mobile app
- 5. What does this impact? Data driven product
- 6. Tracking data records user activity InvitationClickEvent()
- 7. Tracking data records user activity InvitationClickEvent() Scale fact: ~ 1000 tracking event types, ~ Double-digit TB per day, hundreds of metrics & data products
- 8. user engagement tracking data metric scripts production code Tracking Data Lifecycle TransportProduce Consume Member facing data products Business facing decision making
- 9. Tracking Data Lifecycle & Teams TransportProduce Consume Product or App teams: PMs, Developers, TestEng Infra teams: Hadoop, Kafka, DWH, ... Data teams: Analytics, Relevance Engineers,... user engagement tracking data metric scripts production code Member facing data products Business facing decision making
- 10. How do we calculate a metric: ProfileViews PageViewEvent Record 1: { "header" : { "memberId" : 12345, "time" : 1454745292951, "appName" : { "string" : "LinkedIn" "pageKey" : "profile_page" }, }, "trackingInfo" : { ["vieweeID" : "23456"], ... } } Metric: ProfileViews = sum(PageViewEvent) where pageKey = profile_page PageViewEvent Record 1: { "header" : { "memberId" : 12345, "time" : 1454745292951, "appName" : { "string" : "LinkedIn" "pageKey" : "new_profile_page" }, }, "trackingInfo" : { ["vieweeID" : "23456"], ... } } or new_profile_page
- 11. PageViewEvent Record 1: { "header" : { "memberId" : 12345, "time" : 1454745292951, "appName" : { "string" : "LinkedIn" "pageKey" : "profile_page" }, }, "trackingInfo" : { ["vieweeID" : "23456"], ... } } CASE WHEN trackingInfo["profileIds"] ... WHEN trackingInfo["profileid"] ... WHEN trackingInfo["profileId"] ... WHEN trackingInfo["url$profileIds"] ... WHEN trackingInfo["11"] LIKE '%profileIds=%' THEN SUBSTRING(trackingInfo["11"],9,60) WHEN trackingInfo["12"] LIKE '%priceIds=%' THEN SUBSTRING(trackingInfo["12"],9,60) ELSE NULL END AS profile_id Evolution as we mature and grow... Metric: ProfileViews = sum(PageViewEvent where pagekey = profile_page and memberID != trackinginfo[vieweeID] )
- 13. PageViewEvent Record 1: { "header" : { "memberId" : 12345, "time" : 1454745292951, "appName" : { "string" : "LinkedIn" "pageKey" : "profile_page" }, }, "trackingInfo" : { ["vieweeID" : "23456"], ... } } We wanted to move to better data models LI_ProfileViewEvent Record 1: { "header" : { "memberId" : 12345, "time" : 4745292951145, "appName" : { "string" : "LinkedIn" "pageKey" : "profile_page" }, }, "entityView" : { "viewType" : "profile-view", "viewerId" : “12345”, "vieweeId" : “23456”, }, }
- 14. Two options: 1. Keep the old tracking: a. Cost: producers (try to) replicate it (write bad old code from scratch), b. Save: consumers avoid migrating. 2. Evolve. a. Cost: time on data modeling, and on consumer migration, b. Save: pays down data modeling tech debt How much work would it be?
- 15. How much work would it be? Two options: 1. Keep the old tracking: a. Cost: producers (try to) replicate it (write bad old code from scratch), b. Save: consumers avoid migrating. 2. Evolve. a. Cost: time on data modeling, and on consumer migration, b. Save: pays down data modeling tech debt 2000 daysEstimated cost to update consumers to new tracking with clean, committee-approved data models Estimated cost for producers to attempt to replicate old tracking 5000 days #AnalyticsHappiness
- 16. The Task and Opportunity Must do: So we will do the data modeling, and rewrite all the metrics to account for the changes happening upstream… but… Extra credit points: How do we make sure that the cost is not this high the next time? How do we handle evolution in a principled way?
- 17. Product Change Technology Culture & Process Learnings
- 18. Metrics ecosystem at LinkedIn: 3 yrs ago Operational Challenges Diminished Trust due to multiple sources of truth
- 19. Data Stages Ingest Process Serve VisualizeCreate
- 20. Ingest Process Serve VisualizeCreate Tracking Kafka Espresso …
- 21. Tracking Architecture SDKs in different frameworks (server, client) Tracking front-end Monitoring Tools Components KafkaClient-side Tracking Tracking Frontend Services Tools Create
- 22. Data Stages Ingest Process Serve VisualizeCreate
- 23. Unified Ingestion with Hundreds of TB / day Thousands of datasets 80+% of data ingest Ingest
- 24. In production @ LinkedIn, Intel, Swisscom, NerdWallet, PayPal Ingest
- 25. Ingest Process Serve VisualizeCreate Hadoop
- 26. Processing engines @ LinkedInProcess
- 27. Ingest Process Serve VisualizeCreate Pinot
- 28. Pinot Kafka Hadoop Samza Jobs Pinot minutes hour + Distributed Multi-dimensional OLAP Columnar + indexes No joins Latency: low ms to sub-second Serve
- 29. Site-facing Apps Reporting dashboards Monitoring In production @ LinkedIn, Uber Serve
- 30. Ingest Process VisualizeCreate Hadoop Pinot Raptor Serve
- 31. Ingest Process VisualizeCreate Unified Metrics Platform (UMP) Hadoop Pinot Raptor Serve
- 32. Unified Metrics Platform Metrics Logic Raw Data Pinot UMP Harness Incremental Aggregate Backfill Auto-join Raptor dashboards HDFS Aggregated Data Experiment Analysis Relevance ... HDFS Ad-hoc
- 33. Ingest Process Serve VisualizeCreate RaptorKafka Espresso … Hadoop Pinot Tracking Unified Metrics Platform (UMP)
- 35. How do we handle old and new? PageViewEvent ProfileViewEvent Producers Consumers old new Relevance Analytics
- 36. The Big Challenge load “/data/tracking/PageViewEvent” using AvroStorage() (Pig scripts) My Raw Data Our scripts were doing ….
- 37. My Raw Data My Data API We need “microservices" for Data
- 38. The Database community solved this decades ago... Views!
- 39. We had been working on something that could help... A Data Access Layer for Linkedin Abstract away underlying physical details to allow users to focus solely on the logical concerns Logical Tables + Views Logical FileSystem
- 41. Views ecosystem 41 Producers Consumers LinkedInProfileView JSAProfileView Job Seeker App (JSA) LinkedIn App UnifiedProfileView
- 42. Data Catalog + Discovery (DALI) DaliFileSystem Client Data Source (HDFS) Data Sink (HDFS) Processing Engine (MapReduce, Spark) DALI Datasets (Tables + Views) Query Layers (Hive, Pig, Spark) View Defs + UDFs (Artifactory, Git) Dataflow APIs (MR, Spark, Scalding) DALI CLI Dali: Implementation Details in Context
- 43. From load ‘/data/tracking/PageViewEvent’ using AvroStorage(); To load ‘tracking.UnifiedProfileView’ using DaliStorage(); One small step for a script
- 44. A Few Hard Problems Versioning Views and UDFs Mapping to Hive metastore entities Development lifecycle Git as source of truth Gradle for build LinkedIn tooling integration for deployment
- 45. Early experiences with Dali views How we executed Lots of work to get the infra ready Closed beta model Tons of training and education (hand holding) for all Governance body Feedback from analysts is overwhelmingly positive: + Much simpler to share and standardize data cleansing code with peers + Provides effective insulation to scripts from upstream changes - Harder to debug where problems are due to additional layer
- 46. State of the world today ~100 producer views ~200 consumer views ~30% of UMP metrics use Dali data sources ~80 unique tracking event data sources ProfileViews MessagesSent Searches InvitationsSent ArticlesRead JobApplications ...
- 47. What’s next for Dali? Real-time Views on streaming data Selective materialization Hive is an implementation detail, not a long term bet Open source Data Quality Framework
- 48. Product Change Technology Culture & Process Learnings
- 49. Infrastructure enables, but culture really preserves get_tracking_codes = foreach get_domain_rolled_up generate ..entry_domain_rollup, ( (tracking_code matches 'eml-ced.*' or tracking_code matches 'eml-b2_content _ecosystem_digest.*' or (referer is not null and (referer matches '.*touch.linkedin.com.*trk=eml-ced.*' or referer matches '.*touch.linkedin.com.*trk=eml-b2_content_ecosystem_digest.*')) ? 'Email - CED' : (tracking_code matches 'eml-.*' or (referer is not null and referer matches '.*touch.linkedin .com.*trk=eml-.*') or entry_domain_rollup == 'Email' ? 'Email - Other' : (tracking_code == 'hp-feed-article-title-hpm' and entry_domain_rollup == 'Linkedin' ? 'Homepage Pulse Module' : ((tracking_code matches 'hp-feed-.*' and entry_domain_rollup == 'Linkedin') or (std_user_interface matches '(phone app|tablet app|phone browser|tablet browser)' and tracking_code == 'v-feed') or (tracking_code == 'Organic Traffic' and entry_domain_rollup == 'Linkedin' and (referer == 'https://www.linkedin.com/nhome' or referer == 'http://www.linkedin.com/nhome')) ? 'Feed' : (tracking_code matches 'hb_ntf_MEGAPHONE_.*' and entry_domain_rollup == 'Linkedin' ? 'Desktop Notifications' : (tracking_code == 'm_sim2_native_reader_swipe_right' ? 'Push Notification' : (tracking_code == 'pulse_dexter_stream_scroll' and entry_domain_rollup == 'Linkedin' ? 'Pulse -
- 50. For a great data ecosystem that can handle change: 1. Standardize core data entities 2. Create clear maintainable contracts between data producers & consumers 3. Ensure dialogue between data producers & consumers
- 51. 1. Standardize core data entities • Event types and names: Page, Action, Impression • Framework level client side tracking: views, clicks, flows • For all else (custom) - guide when to create a new Event or Dali view Navigation Page View Control Interaction
- 52. 2. Create clear maintainable contracts 1 1. Tracking specification with monitoring: clear, visual, consistent contract Need tooling to support culture shift Tracking specification Tool 2 2. Dali dataset specification with data quality rules
- 53. 3. Ensure dialogue between Producers & Consumers • Awareness: Train about end-to-end data pipeline, data modeling • Instill communication & collaborative ownership process between all: a step-by-step playbook for who & how to develop and own tracking PM → Analyst → Engineer → All3 → TestEng → Analyst user engagement tracking data metric scripts production code Member facing data products Business facing decision making
- 54. Product Change Technology Culture & Process Learnings
- 55. Our Learnings Culture and Process ● Spend time to identify what needs culture & process, and what needs tools & tech ● Big changes can mean big opportunities ● Very hard to massively change things like data culture or data tech debt; never a good time to invest in “invisible” behind-the-scenes change → Make it non-invisible -- try to clarify or size out the cost of NOT doing it → needed strong leaders, and a village Tech ● Must build tooling to support that culture change otherwise culture will revert ● Work hard to make any new layer as frictionless as possible ● Virtual views on Hadoop data can work at scale! (Dali views)
- 56. For a great data ecosystem that can handle change: 1. Standardize core data entities 2. Create clear maintainable contracts between data producers & consumers 3. Ensure dialogue between data producers & consumers Design for change. Expect it. Embrace it.
- 57. Did we succeed? We just handled another huge change! #AnalyticsHappiness