Real time data driven applications (and SQL vs NoSQL databases)
1 like630 views
1) The document discusses building real-time, data driven applications and some of the challenges involved, including dealing with big data, privacy concerns, and real-time analysis and retrieval. 2) It describes an architecture using AngularJS, Python, and a database to build applications and provides examples of using Pandas dataframes and PostgreSQL with PostGIS for managing and querying the data. 3) The key takeaways are that geospatial queries can be difficult and slow down databases, solutions need to be practical given limited resources, and the choice of SQL or NoSQL depends on factors like data size and schema.
![helper.py example
def get_statistics(data, sbi):
sbi_df = data[data.sbi == sbi]
# select * from data where sbi = sbi
hits = sbi_df.hits.sum() # select sum(hits) from …
delta_hits = sbi_df.delta.sum() # select sum(delta) from …
if delta_hits:
percentage = (hits - delta_hits) / delta_hits
else:
percentage = 0
return {"sbi": sbi, "total": hits, "percentage": percentage}](https://image.slidesharecdn.com/nosql-barcelona-150805131057-lva1-app6891/85/Real-time-data-driven-applications-and-SQL-vs-NoSQL-databases-20-320.jpg)
![helper.py example
def get_timeline(data, sbi):
df_sbi = data.groupby([“date”, “hour", “sbi"]).aggregate(sum)
# select sum(hits), sum(delta) from data group by date, hour, sbi
return df_sbi](https://image.slidesharecdn.com/nosql-barcelona-150805131057-lva1-app6891/85/Real-time-data-driven-applications-and-SQL-vs-NoSQL-databases-21-320.jpg)
Real time data driven applications (and SQL vs NoSQL databases)
- 1. GoDataDriven PROUDLY PART OF THE XEBIA GROUP Real time data driven applications Giovanni Lanzani Data Whisperer and SQL vs NoSQL databases
- 2. Who am I? 2008-2012: PhDTheoretical Physics 2012-2013: KPMG 2013-Now: GoDataDriven
- 4. Real-time, data driven app? •No store and retrieve; •Store, {transform, enrich, analyse} and retrieve; •Real-time: retrieve is not a batch process; •App: something your mother could use: SELECT attendees FROM NoSQLMatters WHERE password = '1234';
- 5. Get insight about event impact
- 6. Get insight about event impact
- 7. Get insight about event impact
- 8. Get insight about event impact
- 9. Challenges 1. Big Data; 2. Privacy; 3. Some real-time analysis; 4. Real-time retrieval.
- 10. Is it Big Data? Everybody talks about it Nobody knows how to do it Everyone thinks everyone else is doing it, so everyone claims they’re doing it… Dan Ariely
- 11. Is it Big Data? •Raw logs are in the order of 40TB; •We use Hadoop for storing, enriching and pre- processing.
- 12. 2. Privacy
- 13. 3. (Some) real-time analysis
- 14. •Harder than it looks; •Large data; •Retrieval is by giving date, center location + radius. 4. Real-Time Retrieval
- 15. AngularJS python app REST Front-end Back-end JSON Architecture
- 16. JS-1
- 17. JS-2
- 18. date hour id_activity postcode hits delta sbi 2013-01-01 12 1234 1234AB 35 22 1 2013-01-08 12 1234 1234AB 45 35 1 2013-01-01 11 2345 5555ZB 2 1 2 2013-01-08 11 2345 5555ZB 55 2 2 Data Example
- 19. date hour id_activity postcod e hits delta sbi 2013-01-01 12 1234 1234AB 35 22 1 2013-01-08 12 1234 1234AB 45 35 1 2013-01-01 11 2345 5555ZB 2 1 2 2013-01-08 11 2345 5555ZB 55 2 2 Data Example
- 20. helper.py example def get_statistics(data, sbi): sbi_df = data[data.sbi == sbi] # select * from data where sbi = sbi hits = sbi_df.hits.sum() # select sum(hits) from … delta_hits = sbi_df.delta.sum() # select sum(delta) from … if delta_hits: percentage = (hits - delta_hits) / delta_hits else: percentage = 0 return {"sbi": sbi, "total": hits, "percentage": percentage}
- 21. helper.py example def get_timeline(data, sbi): df_sbi = data.groupby([“date”, “hour", “sbi"]).aggregate(sum) # select sum(hits), sum(delta) from data group by date, hour, sbi return df_sbi
- 22. Who has my data? •First iteration was a (pre)-POC, less data (3GB vs 500GB); •Time constraints; •Oeps: everything is a pandas df!
- 23. Advantage of “everything is a df” Pro: •Fast!! •Use what you know •NO DBA’s! •We all love CSV’s! Contra: •Doesn’t scale; •Huge startup time; •NO DBA’s! •We all hate CSV’s!
- 24. •Set the dataframe index wisely; •Align the data to the index: •Beware of modifications of the original dataframe! source_data.sort_index(inplace=True) If you want to go down this path
- 25. The reason pandas is faster is because I came up with a better algorithm If you want to go down this path
- 26. AngularJS python app REST Front-end Back-end Database JSON ? If you don’t
- 27. A word about (traditional) databases…
- 28. Db: programming language dict
- 29. Postgres for data driven apps?
- 30. Postgres for data driven apps?
- 31. Issues?! •With a radius of 10km, in Amsterdam, you get 10k postcodes.You need to do this in your SQL: •Index on date and postcode, but single queries running more than 20 minutes. SELECT * FROM datapoints WHERE date IN date_array AND postcode IN postcode_array;
- 32. PostGIS is a spatial database extender for PostgreSQL. Supports geographic objects allowing location queries: SELECT * FROM datapoints WHERE ST_DWithin(lon, lat, 1500) AND dates IN ('2013-02-30', '2013-02-31'); -- every point within 1.5km -- from (lat, lon) on imaginary dates Postgres + Postgis (2.x)
- 33. Other db’s?
- 34. How we solved it 1. Align data on disk by date; 2. Use the temporary table trick: 3. Lose precision: 1234AB→1234 CREATE TEMPORARY TABLE tmp (postcodes STRING NOT NULL PRIMARY KEY); INSERT INTO tmp (postcodes) VALUES postcode_array; SELECT * FROM tmp JOIN datapoints d ON d.postcode = tmp.postcodes WHERE d.dt IN dates_array;
- 35. Take home messages 1. Geospatial problems are “hard” and cam kill your queries; 2. Not everybody has infinite resources: be smart and KISS! 3. SQL or NoSQL? (Size, schema)
- 36. GoDataDriven We’re hiring / Questions? / Thank you! @gglanzani [email protected] Giovanni Lanzani Data Whisperer