Tech view on Regulatory Compliance
- 1. Tech view on Regulatory Compliance MarkLogic User Group Benelux Meetup December 2016 Speaker: Alexander L. de Goeij
- 2. About me • Architect / Consultant • Financial Services: Core Trading • Regulations: EMIR, MiFID II • Architecture: Enterprise / Solution / Project Architect • Consulting: IT Strategy, implementations, vendor selection, etc. • Business degree, Tech addiction.
- 3. “Regulations really make my life more fun! ” As said by no-one, ever.
- 4. “Regulations really make my life more fun! ” As said by no-one, ever. everyone who gets to use cool databases! exciting
- 5. The challenge we think we are facing: TransformExtract Source Data Happy Regulator Load Send extractload Some Application
- 6. The actual challenge we are facing: Happy Regulators DB 1Load Source Data Extract Email FTP REST SOAP Tool 2Load Extract Thing NLoad Extract Database you didn’t know still existed
- 7. Current solution: Doesn’t work anymore: • Auditability / Process checks included in Regulations. • Obligation to re-report. • More complex Ad-Hoc requests from the Regulator. • Not suited for Real-Time reporting. • Waste of money…
- 8. What do we need? • Auditability: keep original data in original format to prove results, keep track of ‘who-did-what’ with the data. • Consistency: real-time requirement from regulator demands more than eventual consistency. • Forward Flexibility: we know we don’t know what we will have to report tomorrow.
- 9. Looking to technology for a better answer!
- 10. Your favorite RDBMS • ACID, consistent, and blazing fast if you buy Exadata • Normalize your way out, and fail. • Not fit for processing/reporting across different data objects: e.g. Trades and Mortgages • Try to do NoSQL with SQL (innovative, but terribly slow and impossible to maintain) Example of what not to do: SQL SQL
- 11. MongoDB • Free! Open Source! GridFS! • Have to transform data on ingest (to JSON) as most data is XML • Eventual consistency (AKA data loss) means not real-time. • Good at homogeneous data. • Still master-slave, and scaling issues • Brilliant for RAD / prototyping! Where things go wrong: Source: http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/
- 12. Cassandra (DataStax) • Favors data duplication over normalization • Very fast (if you duplicate well) but does not do JOINs • Used by ING as main component of their Risk grid (YouTube) • Excellent for time series data Source: https://academy.datastax.com/resources/getting-started-time-series-data-modeling
- 14. MarkLogic • Focused on heterogeneously structured data • Bitemporal, if you dare • Semantics / RDF Triples • ACID, Consistent, stores original file • ABAC & redaction in enterprise version • Rules, Workflows, Alerts, Triggers • Not a COTS!
- 15. Ok, so now what?
- 16. Two approaches to a solution Infra approach: • Build everything yourself, use open source components E.g.: • Hadoop • Cassandra + Kafka Platform approach: • Focus on application and business logic, not on infra E.g.: • MarkLogic • Spark (without Hadoop)
- 17. Akka ActorsAkka Actors Spark SparkKafkaKafka Infra approach (SMACK example) • Used (and designed) by Netflix, LinkedIn, Uber, Twitter • Massive amounts of event processing (IoT) • HA and Geo distributed • Scala, Python, R, Java(Script) • Asynchronous everywhere • Near impossible to destroy: reactive, self-healing, back- pressure. Kafka Akka Actors Play REST APIs Cassandra Spark Mesos OS Bare Metal Bare Metal Bare Metal Bare Metal Cassandra Cassandra Zookeeper Marathon Play REST APIsPlay REST APIs
- 19. Platform approach MarkLogic Insert Time Series Database here Spark Source Data Qualitative Quantitative Data Flows Data Stores Analytics Feedback Loop Happy Regulator • Schema transformations • Business Rules • Workflow • Rights management
- 20. Main take-aways • There are no one-stop solutions • Don’t pick bleeding edge stuff if you need it to work • Focus on Business benefit of investment in Regulatory Compliance • Separate the platform from the project! • Start small, think big
- 21. Thank you for listening ! Alexander L. de Goeij [email protected]
- 22. References • https://academy.datastax.com/resources/getting-started-time-series-data-modeling • http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/ • http://hortonworks.com/products/data-center/hdp/ • https://www.linkedin.com/pulse/data-hubs-marklogic-vs-hadoop-kurt-cagle • https://engineering.linkedin.com/blog/2016/04/kafka-ecosystem-at-linkedin • http://www.datanami.com/2015/10/05/how-uber-uses-spark-and-hadoop • https://blog.twitter.com/2015/handling-five-billion-sessions-a-day-in-real-time • http://techblog.netflix.com/2013/12/announcing-suro-backbone-of-netflix.html