Flink Forward San Francisco 2018: Andrew Torson - "Extending Flink metrics: Real-time BI atop existing Flink streaming pipelines"
0 likes862 views
This document discusses extending metrics capabilities in Apache Flink for real-time business intelligence (BI) on streaming data pipelines at Walmart Labs. It proposes adding metric calculators and decorators to Flink operators to compute BI metrics as data flows through pipelines. Metrics can be handled in-process for small dimensions or via side outputs for large dimensions, requiring downstream aggregation. The right metrics database and aggregation techniques should be chosen based on dimensionality.
![6
CHI150406 Category Prioritization ...
Basic Code Examples:
Metric calculator :
public class MetricUtils {
public static <T> MetricsCalculator<T> basicMetricsPackage() {
return new CompositeMetricsCalculator<T>()
.withCalculator(new ResettingEventCounterCalculator<T>().withScope("EventCounter"))
.withCalculator(new SimpleRateMeterCalculator<T>().withScope("RateMeter"))
.withCalculator(new SimpleTrafficHistogramCalculator<T>().withScope("TrafficHistogram"));}}
Programmatic pipeline decoration:
stream.map(MeteredMapProvider.<ItemPushRequest,Tuple2<String,String>>of(jsonMarshallerOperator,
MetricUtils.basicMetricsPackage(),"SuccessfulPricing-Mapâ, null).get());
Configuration pipeline decoration:
YAML: { metrics : { packages: [ core : { class: âcom.walmartlabs.smartpricing.streaming.metrics.MetricUtilsâ ,
staticmethods = [ basicMetricsPackage ] } ], pipelines: [ OptimizationPricing : { operators: [ SuccessfulPricing-Map: {
inputcalculators : [ { core : basicMetricsPackage } ] } ] } ] } }
âŠ.
Java: stream.uid(âSuccessfulPricingâ).map(jsonMarshallerOperator);
Annotation pipeline decoration:
@MetricsCalculator(package=âbasicMetricsPackageâ, scope=âSuccessfulPricing-Mapâ)
private MapFunction<ItemPushRequest,Tuple2<String,String>> jsonMarshaller;
âŠ
stream.map(jsonMarshallerOperator);](https://image.slidesharecdn.com/flinkforwardmetricsapr18-180416090408/85/Flink-Forward-San-Francisco-2018-Andrew-Torson-Extending-Flink-metrics-Real-time-BI-atop-existing-Flink-streaming-pipelines-6-320.jpg)
Flink Forward San Francisco 2018: Andrew Torson - "Extending Flink metrics: Real-time BI atop existing Flink streaming pipelines"
- 1. Extending Flink Metrics: real-time BI atop existing streaming pipelines Andrew Torson, Walmart Labs
- 2. 2 CHI150406 Category Prioritization ... Smart Pricing @ Walmart Labs âą Algorithmic: competitive analysis, economic modeling, cost/profit margin/sales/inventory data âą Rich ingestion: Walmart Stores, Walmart.com, Merchants, Competition, 3rd party data âą Large catalog size: ~ 5M 1st party catalog; ~100M 3rd party/marketplace catalog âą Multi-strategy: competitive leadership/revenue management/liquidation/trending/bidding âą Real-time: essential inputs ingestion (competition/trends/availability) âą Real-time: any 1P item price is refreshed within 5 minutes on all important input triggers âą Quasi real-time: push-rate-controlled 3P catalog price valuation âą Throughput control: probabilistic filtering/caching/micro-batching/streaming API/backpressure âą Regular batch jobs for quasi-static data: sales forecast/demand elasticity/marketplace statistics Item data âąMerchant app âąCatalog âą3P sources Competition data âą Matching âą Crawling âą 3P sources Walmart data âą Walmart Stores âą Walmart.com âą SCM/WMS Smart pricing data âą Forecasts âą Elasticity âą Profit margin Pricing algo run âą ML âą Optimization âą Rule-based Price push âą Portal push âą Cache refresh âą Merchant push Tech Stack: Flink, Kafka, Cassandra, Redis, gRPC, Hive, ElasticSearch, Druid, PySpark, Scikit, Azkaban
- 3. 3 CHI150406 Category Prioritization ... Monitoring & BI âą Health-check: metrics/Graphite/Prometheus/Grafana/New Relic âą Reporting: Hive/Tableau âą Auditing: Hive/Presto âą BI: Druid Going beyond Druid: real-time BI âą Multi-dimensional KPI monitoring & alerting: meet/beat score, stop-loss category P&L âą Categorized Top-K counters: merchant âhot-itemâ search ranking, trending items âą Anomaly detection: stateful outlier input detectors, categorized tail statistics âą Price strategy selection: item-level descriptive/prescriptive time-window snapshot If all the data is already flowing through Flink pipelines â why not enrich them to compute real-time BI metrics as well? âą Pros: existing Flink pipelines scalability; data locality; incremental code decoration with metric aspects; powerful Flink function APIs âą Cons: metrics dimensionality curse/abuse; Flink performance side-effects;
- 4. 4 CHI150406 Category Prioritization ... Quick tour of Flink-metrics âą Dropwizard-like metrics: counters/gauges/histograms/meters âą Metric key scoping: system-level(TM/task/operator) + custom user-level (Metric Group feature) âą Flink Runtime Context: allows to register operator-level metric groups via Flink RichFunction API âą Flink Metric Reporter: is notified of metric registry changes and micro-batches the metrics push âą Built-in metrics and reporters: scheduled Dropwizard reporter; Kafka metrics; IO/latency metrics âą Flink web-GUI: basic metric navigation & charting (no aggregates) ĂŒ Good fit for health-checking ĂŒ Simple and extensible x How to define metrics logic? x How to decorate/attach metrics to existing Flink operators? x How to make adding more/new metrics a quick/simple exercise? x How to handle dynamic/BI-like metric dimensions? x How to handle metrics aggregation? Million-dollar question: KSQL vs Flink for real-time BI?
- 5. 5 CHI150406 Category Prioritization ... Extending Flink metrics: âą Define metric logic: add a hierarchy of stateful Flink Metric Calculators; offer built-in calculator bundles âą Attach to existing Flink operators: add a hierarchy of rich Flink Metered Operator Provider decorators; overload calls to StreamingExecutionEnvironment.addOperator() to implicitly decorate pipelines with metrics; âą Rapid metric development: offer metric calculator base classes, extensible with FP lambda arguments âą Ease of new metric set-up: programmatic and/or annotation and/or configuration âą Dynamic metric key dimension extraction: FP lambda to extract key; state per key; register per key MetricCalcul ator<> StatefulMetric Calculator<> ScalarMetric Calculator<> VectorMetri cCalculator <> MeteredFunction<T> extends RichFunction, ResultTypeQueryable<T>, Supplier<MetricCalculatorsRegistry> public interface MetricCalculator<T> extends Consumer<MetricsInvokationContext<T,?>>, Supplier<Map<List<String>, Metric>>, Serializable { void bootstrap(RuntimeContext ctx); } abstract class MeteredOperatorProvider<OPER extends Function> implements Supplier<OPER> { protected OPER innerFunction; protected MetricCalculatorsRegistry metricsCalculators; protected List<String> userMetricGroupScope; MeteredOp eratorProvid er<> MeteredOper ator1Provide r<> SourceOper atorProvider <> SinkOperatorP rovider<> MeteredOper ator2Provider <> MapOperato rProvider<>
- 6. 6 CHI150406 Category Prioritization ... Basic Code Examples: Metric calculator : public class MetricUtils { public static <T> MetricsCalculator<T> basicMetricsPackage() { return new CompositeMetricsCalculator<T>() .withCalculator(new ResettingEventCounterCalculator<T>().withScope("EventCounter")) .withCalculator(new SimpleRateMeterCalculator<T>().withScope("RateMeter")) .withCalculator(new SimpleTrafficHistogramCalculator<T>().withScope("TrafficHistogram"));}} Programmatic pipeline decoration: stream.map(MeteredMapProvider.<ItemPushRequest,Tuple2<String,String>>of(jsonMarshallerOperator, MetricUtils.basicMetricsPackage(),"SuccessfulPricing-Mapâ, null).get()); Configuration pipeline decoration: YAML: { metrics : { packages: [ core : { class: âcom.walmartlabs.smartpricing.streaming.metrics.MetricUtilsâ , staticmethods = [ basicMetricsPackage ] } ], pipelines: [ OptimizationPricing : { operators: [ SuccessfulPricing-Map: { inputcalculators : [ { core : basicMetricsPackage } ] } ] } ] } } âŠ. Java: stream.uid(âSuccessfulPricingâ).map(jsonMarshallerOperator); Annotation pipeline decoration: @MetricsCalculator(package=âbasicMetricsPackageâ, scope=âSuccessfulPricing-Mapâ) private MapFunction<ItemPushRequest,Tuple2<String,String>> jsonMarshaller; ⊠stream.map(jsonMarshallerOperator);
- 7. 7 CHI150406 Category Prioritization ... BI metrics: in-process In-process: fits small BI dimension cardinality & domain cases âą just a regular operator metric extending VectorMetricsCalculator, with custom key extractor FP lambda âą metric keys will be extracted on the fly from data and new Metrics will be registered & reported dynamically âą keyed metric state will be kept per each metric key âą keyed metric state will be updated in-process within each operator invocation (only for the extracted key) âą each task operator will keep a partition of metric keys that it has observed âą no aggregation in Flink: BI metrics are kept & reported without aggregation as regular Metric Groups âą abuse of dimensionality: may bloat the TM memory and/or operator state in Flink and/or Flink metrics reporter public static MetricsCalculator<ItemPushRequest> priceChangePackage(){ return new ResettingVectorCounterCalculator<ItemPushRequest>().withSimpleEvaluator( (data, s) -> Arrays.asList("PriceChangeDescTotalCounter", data.getItemPricePushData().getPriceChangeDesc()) , (k, s) -> !k.get().isEmpty()? 1L: null); }
- 8. 8 CHI150406 Category Prioritization ... BI metrics: side output Side output: fits large BI dimension cardinality & domain cases âą just a regular operator metric extending VectorMetricsCalculator, with custom key extractor FP lambda âą metric keys will be extracted on the fly from data âą no new Metrics will be registered & reported dynamically âą Flink ProcessFunction Operator Provider will be used âą keyed metric state will be updated as async side effect of each operator invocation (only for the extracted key) âą aggregation in Flink: BI metric side-stream must be explicitly handled in Flink, with aggregation & metric push done by another downstream Flink stage (scheduled independently, typically involves a data shuffle) âą abuse of dimensionality: either use a proper aggregation or a firehose sink in the downstream Flink metrics stage https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/side_output.html
- 9. 9 CHI150406 Category Prioritization ... Make a proper choice of the metrics DB/aggregation tech A final bit of advice: Pay attention to the limits of metric dashboarding tools