Stephen Cantrell, kdb+ Developer at Kx Systems “Kdb+: How Wall Street Tech can Speed up the World"
2 likes•686 views
The document discusses the capabilities and applications of kdb+, a high-performance time-series database developed by KX Systems, widely used in various industries including finance, healthcare, and telecommunications. It highlights the database's extreme performance, low latency, scalability, and ability to manage vast amounts of real-time and historical data. Additionally, it outlines various tools and services that enhance data visualization and management, alongside recruitment opportunities at KX.
Stephen Cantrell, kdb+ Developer at Kx Systems “Kdb+: How Wall Street Tech can Speed up the World"
- 1. HOW WALL STREET TECH CAN SPEED UP THE WORLD STEPHEN CANTRELL – 13th JULY 2017 TECH MEETUPS BERLIN
- 2. • Widely adopted in financial services over two decades • Now in - Hi-TechManufacturing, Utilities, Telco, Energy, Life Sciences, Earth Observation • Software & industry solutions, consulting and implementation services Known for… • Processing & analysis of large volumes of real-time and historical time series data • Extreme performance - low latency • Integrates with and co-exists with other technologies • Scalability without requiring significant infrastructure change • Provide the fastest, most efficient, most flexible tools and dashboards • Worldwide leader in high-volume, high-performance databases 13 Global Offices Inc. NYC, Singapore, London & Tokyo 1800 Employees and growing… Subsidiary of First Derivatives plc Listed on LSE & AIM ABOUT US
- 4. OTHER VERTICALS Health & Life Sciences Automotive Utilities Space and Telco Retail Earth Observation Geospatial Data Analytics Anomaly Detection Genomics Data Processing Connected Health Patient Record Analytics Telemetry Analytics Anomaly Detection Performance Analytics Sensor Analytics Smart Meters Data Predictive Analytics Manufacturing Retail Analytics Marketing Optimization Customer Journey Edge Computing Multivariate Analysis Fault Detection
- 5. kdb+ column based time-series database with in-built programming language q
- 6. CORE TECHNOLOGY • World’s fastest time-series columnar database • Streaming, real-time and historical data in one platform • Runs on Linux, Windows, Solaris, and MacOS • Runs on commodity hardware, cloud, edge devices/appliances • Expressive query (qsql) and programming language (q) • In-memory compute engine for Complex Event Processing • Column-level compression and sensor data noise filtering • Integrates easily into legacy systems for performance augmentation • Multi-core / Multi-processor / Multi-thread / Multi-server
- 7. Kx PERFORMANCE SNIPPETS Streaming Process and store 4.5 million bulk events/second/core Scan Search in-memory tables at 4 billion records/second/core Batch Ingest data at 10 million records/second/core Store Accumulate 10 trillion data points (3 PB) for NYSE Usage Trusted by 17 of the 20 world’s top investment banks Volume Daily volumes of 1.6 TB of streaming data per day Scale From Raspberry Pi, edge devices, to 20,000 cores on AWS Cloud Performance Top performing time-series database according to STAC Research Footprint Tiny 500 KB memory profile (exploits L1/2 Cache) Latency Sub millisecond latency for streaming event processing
- 9. File & DB Sources File & DB Sources Scalability, High Availability, and Fault Tolerance Native Lambda/HTAP Architecture Stream for Kx Application Framework Kx for Flow Kx for Surveillance Kx for Algo Kx for DaaS Kx for Utilities Kx for Cyber Kx for Pharma Vertical Market Solutions Kx for Sensors Kx for Telco Real-Time Sources Core (kdb+) In Memory Database Historical Database In Memory Database Historical Database q language & qsql scripting Develop, configure, deploy, and manage solutions Control for Kx Monitor for Kx Scan, monitor and alerting of issues in software and hardware Third-Party Interoperability Pub/sub, SOA, ODBC, JDBC, web sockets R, Python, MatLab, Java, C#, C/C++ Analyst for Kx Query, explore, transform, and import without programming Dashboards for Kx Build real-time visualizations for multiple devices File & DB Sources Batch Loader Batch Loader Stream Feed Handler Stream Feed Handler Stream Engine Ticker Plant Stream and Ingestion Engine Complex Event Processing • Queries • Transforms • Alerts • Control signals • Notifications • Micro-services Kx ARCHITECTURE
- 10. ENTERPRISE INTERFACES ODBC/ JDBC Web Services Python & Perl R & Matlab TCP Sockets & Web Sockets C#/.NET Java/Scala C/C++ CDC
- 11. STREAM CONTROL ANALYST DASHBOARDS Dashboards for Kx provides rich visualization of real-time streaming, intraday and historical business data. Analyst For Kx provides a complete real- time data transformation, exploration and discovery workflow. Control for Kx allows organizations to manage their application landscape, instantly determining what processes run, who owns them, their parameters and functionality. Stream for Kx is a proven data management solution that can be deployed locally, in the cloud, or as part of a wider Big Data architecture.
- 12. DEMO 2 – NYC MOTOR VEHICLE COLLISONS • Data is available at: https://data.cityofnewyork.us/Public- Safety/NYPD-Motor-Vehicle-Collisions/h9gi-nx95 • Over 1 million rows of historical data between 2012 and 2017 • Weather Data Available at: https://www7.ncdc.noaa.gov/CDO/dataproduct (up to 2013) • Daily Summary Data for a station located in Central Park
- 13. kdb+ Demo Loading data from a source Or if you prefer k
- 14. Reading data into a kdb+ session
- 17. Month by month comparison
- 19. Accidents due to slippery pavement
- 20. Saving your data
- 21. Benchmarks o STAC benchmarks: https://stacresearch.com/kx; includes independently verified benchmarks of the technology using common capital markets use cases. o Intel solution brief: http://www.intel.com/content/www/us/en/processors/xeon/real-time-financial- analysis-with-kx-systems-brief.html o Gartner paper on Kx technology: https://kx.com/gartner-download.php o Community o Kx Wiki: http://code.kx.com/wiki/Main_Page o Kx Community: http://kxcommunity.com/ o Kx Github: http://kxsystems.github.io/ RESOURCES
- 22. 14 Kx TECHNOLOGY FUND Int. Office Access £10 Mil Funding Kx Tech Access Sales & Marketing Support Logistical Support Learn more at www.kx.com/kx-technology-
- 23. 18 RECRUITMENT o Opportunities to travel & live internationally o Paid for city accommodation o Competitive salary and expenses package o Health & travel insurance programme o Generous retirement savings programme