6 things to expect when you are visualizing (2020 Edition)
1 like•484 views
This document discusses expectations and challenges when visualizing data. The key points are: 1. Expect to find the real need by understanding the audience and goals better than the client. Expect to clean data, which can take a significant amount of time due to multiple sources and formats. 2. Prepare to iterate as the initial visualization may not meet needs or deadlines. Celebrate failures as learning opportunities. 3. Visualization projects include storytelling projects with strict deadlines and analytical tools to support data exploration by technical teams over the long term. The project lifecycle involves identifying needs, prototyping, refining, and maintaining the visualization.
![HAPPY NEW YEAR AROUND THE WORLD
[ PROJECT ]](https://image.slidesharecdn.com/6thingstoexpectwhenyouarevisualizingv6-200408205651/85/6-things-to-expect-when-you-are-visualizing-2020-Edition-67-320.jpg)
![BOBA SCIENCE
[ PROJECT ]](https://image.slidesharecdn.com/6thingstoexpectwhenyouarevisualizingv6-200408205651/85/6-things-to-expect-when-you-are-visualizing-2020-Edition-74-320.jpg)
![GAME OF THRONES
[ PROJECT ]](https://image.slidesharecdn.com/6thingstoexpectwhenyouarevisualizingv6-200408205651/85/6-things-to-expect-when-you-are-visualizing-2020-Edition-76-320.jpg)
![VISUAL ANALYTICS FOR LOG EVENTS
[ PROJECT ]](https://image.slidesharecdn.com/6thingstoexpectwhenyouarevisualizingv6-200408205651/85/6-things-to-expect-when-you-are-visualizing-2020-Edition-137-320.jpg)
![LOG EVENT A.K.A. “CLIENT EVENT”
[Lee et al. 2012]](https://image.slidesharecdn.com/6thingstoexpectwhenyouarevisualizingv6-200408205651/85/6-things-to-expect-when-you-are-visualizing-2020-Edition-146-320.jpg)
![LOG EVENT A.K.A. “CLIENT EVENT”
client : page : section : component : element : action
web : home : timeline : tweet_box : button : tweet
1) User ID
2) Timestamp
3) Event name
4) Event detail
[Lee et al. 2012]](https://image.slidesharecdn.com/6thingstoexpectwhenyouarevisualizingv6-200408205651/85/6-things-to-expect-when-you-are-visualizing-2020-Edition-147-320.jpg)
![DISPLAY CHANGES
iphone home -
- - impression
tweet tweet click
Map of the Market [Wattenberg 1999], StemView [Guerra-Gomez et al. 2013]](https://image.slidesharecdn.com/6thingstoexpectwhenyouarevisualizingv6-200408205651/85/6-things-to-expect-when-you-are-visualizing-2020-Edition-163-320.jpg)
![LABELLA.JS
[ PROJECT ]](https://image.slidesharecdn.com/6thingstoexpectwhenyouarevisualizingv6-200408205651/85/6-things-to-expect-when-you-are-visualizing-2020-Edition-190-320.jpg)
![GRID MAP
[ PROJECT ]](https://image.slidesharecdn.com/6thingstoexpectwhenyouarevisualizingv6-200408205651/85/6-things-to-expect-when-you-are-visualizing-2020-Edition-192-320.jpg)
![VX = REACT + D3
[ PROJECT ]](https://image.slidesharecdn.com/6thingstoexpectwhenyouarevisualizingv6-200408205651/85/6-things-to-expect-when-you-are-visualizing-2020-Edition-197-320.jpg)
6 things to expect when you are visualizing (2020 Edition)
- 1. Krist Wongsuphasawat / @kristw 6 THINGS TO EXPECT WHEN YOU ARE VISUALIZING
- 2. 6 THINGS TO EXPECT WHEN YOU ARE VISUALIZING Krist Wongsuphasawat / @kristw
- 3. Computer Engineer Bangkok, Thailand Chulalongkorn University Krist Wongsuphasawat / @kristw
- 4. Programming + Soccer Computer Engineer Bangkok, Thailand Krist Wongsuphasawat / @kristw
- 5. Programming + Soccer Computer Engineer Bangkok, Thailand Krist Wongsuphasawat / @kristw
- 6. (P.S. These are actually not my robots, but our competitors’.) Krist Wongsuphasawat / @kristw Computer Engineer Bangkok, Thailand
- 7. Krist Wongsuphasawat / @kristw Computer Engineer Bangkok, Thailand PhD in Computer Science Information Visualization Univ. of Maryland
- 8. Krist Wongsuphasawat / @kristw Computer Engineer Bangkok, Thailand IBM Microsoft PhD in Computer Science Information Visualization Univ. of Maryland
- 9. PhD in Computer Science Information Visualization Univ. of Maryland IBM Krist Wongsuphasawat / @kristw Computer Engineer Bangkok, Thailand Data Scientist Analytics, Experiment Twitter Microsoft
- 10. PhD in Computer Science Information Visualization Univ. of Maryland IBM Krist Wongsuphasawat / @kristw Computer Engineer Bangkok, Thailand Engineering Manager Data Experience Airbnb Microsoft Twitter
- 11. #interactive visualizations Open-source projects Visual Analytics Tools interactive.twitter.com Apache Superset committer labella.js (3000+ stars) react-vega Internal tools Academic paperskristw.yellowpigz.com
- 12. DATA =ME+ VIS
- 13. Data, I’m ready!
- 14. Data, I’m ready! Here I come!
- 15. WHAT TO EXPECT?
- 16. 1. EXPECT TO FIND THE REAL NEED
- 17. INPUT (DATA) What clients think they have
- 18. INPUT (DATA) What clients think they have What they usually have
- 19. YOU What clients think you are
- 20. YOU What clients think you are What they will get
- 21. OUTPUT (VIS) What clients ask for
- 22. OUTPUT (VIS) What clients ask for What they really need
- 23. COMMUNICATE
- 24. GOALS Present data Communicate information effectively Analyze data Exploratory data analysis Tools to analyze data Reusable tools for exploration Enjoy Combination of above
- 25. GOALS Present data Communicate information effectively Analyze data Exploratory data analysis Tools to analyze data Reusable tools for exploration Enjoy Combination of above Who are the audience? What do you want to tell? What are the questions? Who will use this? What would they use this for? Who are the audience?
- 26. I need this. Take this.
- 27. I need this. Here you are. I need this. Take this.
- 28. & COMPROMISE
- 29. 2. EXPECT TO CLEAN DATA
- 30. 2. EXPECT TO CLEAN DATA A LOT
- 31. 70-80% of time cleaning data “DATA JANITOR”
- 32. Collect + Clean + Transform DATA WRANGLING
- 33. WHY DOES IT TAKE SO MUCH TIME?
- 34. 2.1 Many sources and data format
- 35. DATA SOURCES Open data Publicly available Internal data Private, owned by clients’ organization Self-collected data Manual, site scraping, etc. Combine the above
- 36. DATA FORMAT Standalone files txt, csv, tsv, json, Google Docs, …, pdf* Databases doesn’t necessary mean they are organized API better quality with more overhead Website Big data*
- 37. NEED TO… Change format e.g. tsv => json Combine data Resolve multiple sources of truth
- 38. 2.2 Data transformation is needed.
- 39. EXAMPLES Convert latitude/longitude into zip code Change country code from 3-letter (USA) to 2-letter (US) Correct time of day based on users’ timezone etc.
- 40. 2.3 Data collection issues
- 42. 2.4 Definition of “clean” data
- 43. IS THIS CLEAN? USER RESTAURANT RATING ======================== A MCDONALD’S 3 B MCDONALDS 3 C MCDONALD 4 D MCDONALDS 5 E IHOP 4 F SUBWAY 4
- 44. IS THIS CLEAN? USER RESTAURANT RATING ======================== A MCDONALD’S 3 B MCDONALDS 3 C MCDONALD 4 D MCDONALDS 5 E IHOP 4 F SUBWAY 4 How many reviews are there? Clean. How many restaurants are there? Not clean. McDonald, McDonald’s, McDonalds
- 45. 2.5 Bigger data, bigger problems
- 46. HAVING ALL TWEETS How people think I feel.
- 47. How people think I feel. How I really feel. HAVING ALL TWEETS
- 48. Hadoop Cluster GETTING BIG DATA Data Storage
- 49. Scalding (slow) GETTING BIG DATA Hadoop Cluster Data Storage Tool
- 50. Scalding (slow) GETTING BIG DATA Hadoop Cluster Data Storage Tool Your laptop Smaller dataset
- 51. Hadoop Cluster Scalding (slow) Data Storage Tool Final dataset Tool node.js / python / excel (fast) Your laptop GETTING BIG DATA Smaller dataset
- 52. CHALLENGES Slow Long processing time (hours) Get relevant Tweets hashtag: #oscars keywords: “parasite” (movie name) Too big Need to aggregate & reduce size Harder to spot problems
- 53. CHALLENGES Slow Long processing time (hours) Get relevant Tweets hashtag: #oscars keywords: “parasite” (movie name) Too big Need to aggregate & reduce size Harder to spot problems
- 54. RAMSAY & RAMSEY
- 55. 2.6 New issues can show up any time.
- 57. RECOMMENDATIONS Always think that you will have to do it again document the process, automation Reusable scripts break a gigantic do-it-all function into smaller ones Reusable data keep for future project
- 58. 3. PREPARE TO ITERATE
- 59. It was a great idea … until I actually tried it.
- 61. TIPS Don’t give up. If stuck, look for inspirations. The vis that gives you insights may or may not be the best vis for sharing. Exploration vs. Communication Keep it as simple as possible but not simpler.
- 62. “Necessity is the mother of invention.” — English Proverb
- 63. “Necessity is the mother of invention.” — English Proverb DEADLINE
- 64. TIPS Don’t give up. If stuck, look for inspirations. The vis that gives you insights may or may not be the best vis for sharing. Exploration vs. Communication Keep it as simple as possible but not simpler. Set milestones and deadline.
- 65. PROJECTS
- 66. STORYTELLING PROJECTS timely Deadline is strict. Also can be unexpected events. wide audience easy to explain and understand, multi-device support one-off project scope analyze data to find stories and find best way to present them
- 67. HAPPY NEW YEAR AROUND THE WORLD [ PROJECT ]
- 73. HAPPY NEW YEAR 2013 twitter.github.io/interactive/newyear2014/
- 74. BOBA SCIENCE [ PROJECT ]
- 76. GAME OF THRONES [ PROJECT ]
- 77. from fans’ conversations Reveal the talking points of every episode of
- 79. Problem Want to know what the audience talk about a TV show from Tweets
- 80. HBO’s Game of Thrones Based on a book series “A Song of Ice and Fire” Medieval Fantasy. Knights, magic and dragons.
- 81. Brief Story
- 82. A King dies. A lot of contenders wage a war to reclaim the throne.
- 83. Minor characters with no claim to the throne set their own plans in action to gain power when all the major characters end up killing each other.
- 84. Brave/Honest/Honorable characters die. Intelligent but shady characters and characters who know nothing continue to live.
- 85. While humans are busy killing each other, ice zombies “White walkers” are invading from the North. The only group who seems to care about this is neutral group called the Night’s Watch.
- 86. HBO’s Game of Thrones Based on a book series “A Song of Ice and Fire” Medieval Fantasy. Knights, magic and dragons. Many characters. Anybody can die. 8 seasons Multiple storylines in each episode
- 87. Problem Want to know what the audience talk about a TV show from Tweets
- 88. Ideas Common words Too much noise
- 89. Ideas Common words Too much noise Characters How o!en each character were mentioned?
- 90. Prototyping Pull sample data from Twitter API Entity recognition and counting naive approach
- 91. List of names Daenerys Targaryen,Khaleesi Jon Snow Sansa Stark Tyrion Lannister Arya Stark Cersei Lannister Khal Drogo Gregor Clegane,Mountain Margaery Tyrell Joffrey Baratheon Bran Stark Theon Greyjoy Jaime Lannister Brienne Eddard Stark,Ned Stark Ramsay Bolton Sandor Clegane,Hound Ygritte Stannis Baratheon Petyr Baelish,Little Finger Robb Stark Bronn Varys Catelyn Stark Oberyn Martell Daario Naharis Davos Seaworth Jorah Mormont Melisandre Myrcella Baratheon Tywin Lannister Tommen Baratheon Grey Worm Tyene Sand Rickon Stark Missandei Roose Bolton Robert Baratheon Jojen Reed Jeor Mormont Tormund Giantsbane Lysa Arryn Yara Greyjoy,Asha Greyjoy Samwell Tarly,Sam Hodor Victarion Greyjoy High Sparrow Dragon Winter Dothraki
- 92. Sample Tweet
- 93. Sample Tweet
- 94. Sample data Character Count Hodor 10000 Jon Snow 5000 Daenerys 4000 Bran Stark 3000 … … *These numbers are made up for presentation, not real data.
- 95. Where to go from here?
- 96. + episodes The Guardian & Google Trends http://www.theguardian.com/news/datablog/ng-interactive/2016/apr/22/game-of-thrones-the-most-googled-characters-episode-by-episode
- 97. + emotion
- 98. + connections
- 99. + connections
- 100. Gain insights from a single episode emotion & connections
- 101. Sample data Character Count Jon Snow+Sansa 1000 Tormund+Brienne 500 Bran Stark+Hodor 300 … … Character Count Hodor 10000 Jon Snow 5000 Daenerys 4000 … … INDIVIDUALS CONNECTIONS + top emojis + top emojis *These numbers are made up for presentation, not real data.
- 102. Graph NODES LINKS + top emojis + top emojis Character Count Jon Snow+Sansa 1000 Tormund+Brienne 500 Bran Stark+Hodor 300 … … Character Count Hodor 1000 Jon Snow 500 Daenerys 400 … … *These numbers are made up for presentation, not real data.
- 103. Network Visualization Node-link diagram Force-directed layout http://blockbuilder.org/kristw/762b680690e4b2b2666dfec15838a384
- 104. Issue: Hairball
- 105. Issue: Occlusions
- 109. + Collision Detection (with clusters) https://bl.ocks.org/mbostock/7881887
- 110. Tormund + Brienne
- 111. Let’s get other episodes.
- 112. More data Hadoop Rewrite the scripts in Scalding to get archived data
- 113. How much data do we need? Whole week? 5 days? 2 days? A day? etc.
- 114. How much data do we need?
- 115. Transitions
- 116. Changing episode
- 119. Colors Default: D3 category10 Distinct but nothing about the context Custom palette Colors related to the groups/houses. Black = Night’s Watch Blue = North Red = Daenerys Gold = Lannister …
- 120. The vis is not enough.
- 122. Legend
- 123. Navigation
- 124. Top 3
- 125. Adjust threshold
- 126. Recap
- 129. Mobile Support
- 130. Self & Peer Does it solve the problem?
- 132. Feedback
- 133. Feedback
- 134. ANALYTICS TOOLS
- 135. VISUAL ANALYTICS TOOL PROJECTS richer, more features to support exploration of complex data more technical audience product managers, engineers, data scientists accuracy designed for dynamic input long-term projects
- 136. PROJECT LIFECYCLE Identify needs Design and prototype Make it work for sample dataset Refine, generalize and productionize Make it work for other cases Document and release Maintain and support Keep it running, Feature requests & Bugs fix
- 137. VISUAL ANALYTICS FOR LOG EVENTS [ PROJECT ]
- 138. USER ACTIVITY LOGS
- 139. UsersUseTwitter
- 141. UsersUse Curious Engineers Log data in Hadoop Write Twitter Instrument Product Managers
- 142. WHAT ARE BEING LOGGED? tweet Activities
- 143. WHAT ARE BEING LOGGED? tweet from home timeline on twitter.com tweet from search page on iPhone Activities
- 144. WHAT ARE BEING LOGGED? tweet from home timeline on twitter.com tweet from search page on iPhone sign up log in retweet etc. Activities
- 145. ORGANIZE?
- 146. LOG EVENT A.K.A. “CLIENT EVENT” [Lee et al. 2012]
- 147. LOG EVENT A.K.A. “CLIENT EVENT” client : page : section : component : element : action web : home : timeline : tweet_box : button : tweet 1) User ID 2) Timestamp 3) Event name 4) Event detail [Lee et al. 2012]
- 148. LOG DATA
- 149. UsersUse Curious Engineers Log data in Hadoop Twitter Instrument Write Product Managers bigger than Tweet data
- 150. UsersUse Curious Engineers Log data in Hadoop Data Scientists Ask Twitter Instrument Write Product Managers
- 151. UsersUse Curious Engineers Log data in Hadoop Data Scientists Find Ask Twitter Instrument Write Product Managers
- 152. LOG DATA
- 153. UsersUse Curious Engineers Log data in Hadoop Data Scientists Find, Clean Ask Twitter Instrument Write Product Managers
- 154. UsersUse Curious Engineers Log data in Hadoop Data Scientists Find, Clean Ask Monitor Twitter Instrument Write Product Managers
- 155. UsersUse Curious Engineers Log data in Hadoop Data Scientists Find, Clean, Analyze Ask Monitor Twitter Instrument Write Product Managers
- 156. Log data EngineersData Scientists Usersin Hadoop Find, Clean, Analyze Use Monitor Ask Curious 1 2 Twitter Instrument Write Product Managers
- 157. client page section component element action Event 50,000+ event types
- 158. client page section component element action Event 50,000+ event types one graph / event x 50,000
- 159. DESIGN
- 160. CLIENT EVENT HIERARCHY iphone home - - - impression tweet tweet click iphone:home:-:-:-:impression iphone:home:-:tweet:tweet:click
- 161. DETECT CHANGES iphone home - - - impression tweet tweet click iphone home - - - impression tweet tweet click TODAY 7 DAYS AGO compared to
- 162. CALCULATE CHANGES +5% +5% +5% +10% +10% +10% -5% -5% -5% DIFF
- 163. DISPLAY CHANGES iphone home - - - impression tweet tweet click Map of the Market [Wattenberg 1999], StemView [Guerra-Gomez et al. 2013]
- 164. DISPLAY CHANGES home - - - impression tweet tweet click iphone
- 165. Demo Demo Demo Demo / Scribe Radar
- 166. Twitter for Banana
- 168. Details separate good and great work 4. RESERVE TIME FOR REFINEMENT
- 169. “The first 90% of the code accounts for the first 90% of the development time. The remaining 10% of the code accounts for the other 90% of the development time.” — Tom Cargill, Bell Labs
- 170. REFINE & POLISH UX / UI + Mobile Support Color Animation / Transition Metadata for SEO Social media preview images Performance Loading time, Data file size “The little of visualisation design” by Andy Kirk http://www.visualisingdata.com/2016/03/little-visualisation-design/
- 171. Example
- 173. x & y only, no radius
- 174. Fix it
- 175. Fix it
- 177. THE ORIGIN From a paper “Interim pre- pandemic planning guidance: community strategy for pandemic influenza mitigation in the United States: early, targeted, layered use of nonpharmaceutical interventions” published in 2007 by the CDC https://stacks.cdc.gov/view/cdc/11425
- 178. REVIVAL Rosamund Pearce, a data journalist at The Economist, rebuild it for a piece about COVID-19. Changed the labeling scheme to assist colorblind readers. https://www.economist.com/briefing/2020/02/29/covid-19-is-now-in-50-countries-and-things-will-get-worse
- 179. THE LINE Drew Harris, an assistant professor at the Thomas Jefferson University, came across the graphic in The Economist. He recalled using it a decade earlier as a pandemic preparedness trainer. So he added the dotted line “healthcare system capacity” https://www.nytimes.com/article/flatten-curve-coronavirus.html
- 180. THEN IT WENT VIRAL
- 184. or find ways to get some 5. PLAN FOR FEEDBACK
- 185. “Feedback is the breakfast of champion.” — Ken Blanchard
- 186. FEEDBACK During development Feedback sessions with clients/potential users After release Logging User study Forum, User group Office hours
- 187. 6. LOOK BACK FOR IMPROVEMENT
- 188. HOW TO BE BETTER? Retrospective What could have been better? Wishlist Expand skillset Learning opportunities Get help Grow the team Improve tooling Solve a problem once and for all Automate repetitive tasks
- 189. REUSABLE WORK
- 192. GRID MAP [ PROJECT ]
- 196. COVID-19 Situation in Thailand by province
- 197. VX = REACT + D3 [ PROJECT ]
- 199. SUMMARY
- 200. 6 STEPS 1. 2. 3. 4. 5. 6. Krist Wongsuphasawat / @kristw kristw.yellowpigz.com Expect to find the real need Expect to clean data a lot Prepare to iterate Reserve time for refinement Plan for feedback Look back for improvement
- 201. My former and current colleagues at Twitter and Airbnb for their collaboration and support in these projects; and my wife for taking care of our two kids while I make these slides. ACKNOWLEDGEMENT
- 202. THANK YOU
- 203. QUESTIONS?