Don't take your data for granted.

Editor's Notes

• #7: This is true of any maturing organization.How many of you work for a company whose web team is just made up of a single webmaster. Years ago, that wouldn’t have been too crazy.
• #8: So why does the data qualify suffer? In the past, easy explanations were used for mysterious discrepancies.It’s also easy to just blame things on the user. “There must be some unique use case that causes this”While there is some truth to these statements, these are generally notacceptable explanations.
• #9: With today’s browsers, generally JavaScript issues are not the culprit. [And I mean with IE6 disappearing from the planet]Analyticsis based on tried and tested web technologies.Precision is more important than accuracy, but it still needs to be measured. And that doesn’t mean you shouldn’t know how accurate your data is.After all, if your business is relying on it, it is production data, and should be treated that way.
• #10: Emphasized.With this, comes the need for risk mitigation, backup data sources, the ability to roll back releases that cause issues, the ability to raise a sev 1 flag, all hands on deck.But then what?
• #11: So you know the technical side is important, but you’re brought back to square one. Where should you focus your obviously limited resources?This is the topic of much debate. A lot of prominent analytics minds have stated analysis should not be held back from quality issuesI also hesitantly agree with this but it’s not a great answer.
• #12: What if you could do both? Being able to split analytics in half.So this is exactly what we chose to do at Beachbody.
• #13: So how does this work?Two independent teams. One goal,By being in different organizations, we’re able to collectively “cover more ground” – whether it be financial planning meetings or engineering roadmap meetings. Analytics can always be represented without sacrifice.Back to the example of that webmaster. Today, web designers and web developers are different people; different teams. This parallels what needs to change with analytics as well. Butwhy not just have developers handle analytics? As you’ll soon see, there is much more to the technical side than just writing code.
• #14: Digging into the technical side, what are our core objectives?We know the data is critical. It would be crazy to not monitor it closely.We feel that we should always be in the loop. On all projects. There are no black boxes. Data doesn’t appear out of thin air.We can’t assume the data is right just because it’s in the tool. There are too many things that can go wrong.Scalability is a constant concern. With growing traffic, growing data, growing complexity, growing users. We look for ways to repeat consistently and automate where possible.Success doesn’t come easy. We need to fight against last minute projects, projects without process, and constantly look for things we can be doing better.
• #15: We can break down our focus into these four areas:ImplementationSupportMonitoringPlatform
• #16: - What most people automatically think of when they think of technical web analytics.- QA needs to be a part of implementation, ensure tracking is working. Validation, ensure tracking is passing correct values.The wiki is used by both analytics team and developersOne key thing to note. My team can read/write JavaScript and HTML, but we aren’t developers.
• #17: Super high level view of our ideal project processDue to different focus, the business analytics team shouldn’t have to be as involved in every engineering meeting.Involved in daily scrums, follow development through the whole process.What’s not shown here is the before and after, where the business side is more involved. Coming up with projects and ongoing analysis of the results.
• #18: No offense to vendors, but usually support is a bit slower than the business would like. Plus there’s a ramp up. We can speed that up for the vast majority of incidents. Adding users, changing access rights. Resetting accountsForensics is big. When a power use spots something that just doesn’t add up, we don’t just let it go. We’ll do analysis to figure out why. Engage who we need to. This can go beyond pure web analytics, deal with other data sources (BI, server logs, etc.). We’re IT so root cause and future prevention is big for us. Issues can happen but the same one should never be repeated. We also do analysis. IT can be a customer. Help find issues in your system. If the data is good, we can find bugs in our site itself. Error messages being thrown. Certain behavioral scenarios, etc.
• #19: - Monitoring is one of the most important of our focuses.- Sanity checking includes errorpagesReconciliations. Manual but transitioning to automated methods for scalability.- bservePoint has been huge here. In addition to analytics issues, we’ve caught issues with our site due to monitoring. The page loads, but something about it renders incorrectly.Ensuring the data / classifications are maintained and don’t go out of date.We’re trying to become more systematic here. Every other system is automatically monitored in great detail, why is analytics not always?And what if something does go haywire? Knowing what caused it is important. Since we’re also in the loop on all other technical issues and site releases, we usually know in a very short timeframe what a root cause for discrepancies can be.
• #20: Where we can really begin to think outside the box. Have fun.Thinking beyond the single vendor.First of all, a roadmap looking out a few months. Outside of project work, we look at trends in the industry, new tracking functionality, future changes in the site. Being a part of IT and the overall engineering roadmap let’s us be smarter about planning.Automation is the only way to scale. Things can start out manually but they should not end up there.Once we’re comfortable with data quality and trust it to be consistent, we think about way to bridge online and offline data. Bringing analytics data into our data warehouse. Custom applications can be built to automate tasks, automate dashboards, reconciliations. But also for deep monitoring, campaign code management, etc.Tools are constantly evolving so it’s important we keep an eye on what is out there.
• #21: Need to fully understand the definitions and disclaimers of the numbers. How people use them. Context is everything.We are constantly working cross-functionally, project management is huge. This wouldn’t be the most efficient use of pure development resources. We manage internal developers, external developers.Work with many customersDeal with vendors
• #22: Another key challenge is going back to multiple teams doing somewhat overlapping work.We’ve done a great job but if you’re not careful it can be a slippery slope.Communication of issues, project status is paramount.Document everything. We use Jira for task management and all work requests.Knowledge transfer between teams is also incredibly important to be more effective. We’re meant to support each other.
• #23: All said and done, there are many advantages to this type of structure.Huge increased confidence. Analytics went from a “nice to have” data source to a critical component of running the business. Executive support is critical.Once analytics data is valid, it can be used for many other reasons. QA for the site is one. If not valid, how do you know what an issue is indicative of?Vendors are not perfect. Although unlikely, data gaps happen. If we’re set up properly with backup systems, we won’t have total data loss on certain key elements (like campaign volume)Analytics doesn’t have to be an intenral war between analysis and integrity. You can have both.
• #24: So even if your organization doesn’t yet have the budget for separate teams, the only way to get there is have analytics prove it’s worth.You can start keeping track of your data quality in simple ways today.Keep the documentation centralized and accessible by everyone. Keep this maintained.Every tool has some level of alerting built in. Use this to react more quickly.Have very “basic” sanity check health dashboards set up. Page views for each of your sites. Trend incoming marketing channel volume.If you’re an analyst that doesn’t understand the technical side, change that. Start learning. Read blogs. Follow #measure.
• #25: - Communication is everything.