Nailing Measurement: a Framework for Measuring Metrics that Matter
0 likes12,544 views
The document outlines Josephine Lee's goals as a product manager at Atlassian, focusing on delivering Confluence in ten languages by the end of 2018 to enhance user satisfaction, particularly among non-English speaking customers. It emphasizes the importance of defining goals, signals, and metrics for measuring success and learning from user interactions. The agenda includes iterative processes to improve user engagement and actionable notifications through analysis and adjustments in strategies.
Nailing Measurement: a Framework for Measuring Metrics that Matter
- 1. JOSEPHINE LEE | PRODUCT MANAGER | ATLASSIAN Nailing Measurement A Framework for Measuring Metrics that Matter
- 2. CONTEXT
- 4. GOAL
- 10. What went wrong?
- 11. OutcomeOutput
- 18. How do you choose the right metric?
- 20. What does feature success look like?
- 21. What does feature success look like? user
- 22. What does user success look like?
- 23. How can we measure that? What does user success look like?
- 27. Goals Signals Metrics What does user success look like?
- 28. Goals Signals Metrics How can we measure that? What does user success look like?
- 30. Goals are what success looks like for your user Goals Signals Metrics
- 31. Goals are what success looks like for your user Goals Signals Metrics E.g. Give a useful talk at Atlas Camp
- 32. Goals are what success looks like for your user Goals Signals Metrics E.g. Give a useful talk at Atlas Camp
- 33. Goals are what success looks like for your user Signals are predictors of the associated goals Goals Signals Metrics E.g. Give a useful talk at Atlas Camp
- 34. Goals are what success looks like for your user Signals are predictors of the associated goals Goals Signals Metrics E.g. Give a useful talk at Atlas Camp E.g. Audience is smiling and nodding
- 35. Goals are what success looks like for your user Signals are predictors of the associated goals Metrics are measurable & define what success looks like Goals Signals Metrics E.g. Give a useful talk at Atlas Camp E.g. Audience is smiling and nodding
- 36. Goals are what success looks like for your user Signals are predictors of the associated goals Metrics are measurable & define what success looks like Goals Signals Metrics E.g. Give a useful talk at Atlas Camp E.g. Audience is smiling and nodding E.g. 4.5+ session approval rating
- 37. GOAL
- 39. Goals Goals are what success looks like for your user
- 41. Post-its Sharpies Whiteboard Markers GSM Play Ingredients
- 42. Setting Goals
- 47. Goals Ensure non-English teams can collaborate and get work done with Confluence
- 48. Goals Signals Ensure non-English teams can collaborate and get work done with Confluence Signals are predictors of the associated goals
- 50. Goals Signals Ensure non-English teams can collaborate and get work done with Confluence
- 51. Goals Signals Ensure non-English teams can collaborate and get work done with Confluence Users working with a non- English language, use the translations.
- 52. Goals Signals Ensure non-English teams can collaborate and get work done with Confluence Users working with a non- English language, use the translations.
- 53. Goals Signals Ensure non-English teams can collaborate and get work done with Confluence Users working with a non- English language, use the translations. Users do not revert back to English.
- 54. Goals Signals Metrics Ensure non-English teams can collaborate and get work done with Confluence Metrics are measurable & define what success looks like Users working with a non-English language, use the translations. Users do not revert back to English.
- 56. Goals Signals Metrics Ensure non-English teams can collaborate and get work done with Confluence Users working with a non-English language, use the translations. Users do not revert back to English.
- 57. Goals Signals Metrics Ensure non-English teams can collaborate and get work done with Confluence Ratio of users in same language as browser is greater than 30%. Users working with a non-English language, use the translations. Users do not revert back to English.
- 58. Goals Signals Metrics Ensure non-English teams can collaborate and get work done with Confluence Ratio of users in same language as browser is greater than 30%. 50% of users that ‘switch' to a translation do not revert to English within 6 months. Users working with a non-English language, use the translations. Users do not revert back to English.
- 59. MetricsSignals
- 60. MetricsSignals Show us whether we’re on the right track
- 61. MetricsSignals Show us whether we’re on the right track Metrics show us whether or not you’ve been successful
- 62. MetricsSignals Show us whether we’re on the right track Metrics show us whether or not you’ve been successful
- 63. SHORT TERM SUCCESS MetricsSignals Show us whether we’re on the right track Metrics show us whether or not you’ve been successful
- 64. SHORT TERM SUCCESS LONG TERM SUCCESS MetricsSignals Show us whether we’re on the right track Metrics show us whether or not you’ve been successful
- 65. SHORT TERM SUCCESS LONG TERM SUCCESS MetricsSignals Show us whether we’re on the right track Metrics show us whether or not you’ve been successful E.g. Audience is smiling and nodding
- 66. SHORT TERM SUCCESS LONG TERM SUCCESS MetricsSignals Show us whether we’re on the right track Metrics show us whether or not you’ve been successful E.g. Audience is smiling and nodding E.g. 4.5+ session approval rating
- 71. USER DRIVEN GOAL Ensurethatnon-English speakingteamscancollaborate andgetworkdonewith Confluence
- 79. Goals Signals Metrics Users are notified for things they care about ITERATION 1
- 80. Goals Signals Metrics Users are notified for things they care about ITERATION 1
- 81. Goals Signals Metrics Users are notified for things they care about Users are opening their notifications ITERATION 1
- 82. Goals Signals Metrics Users are notified for things they care about Users are opening their notifications ITERATION 1
- 83. Goals Signals Metrics Users are notified for things they care about 20% increase in users clicking through to links from a notification Users are opening their notifications ITERATION 1
- 84. Goals Signals Metrics Users are notified for things they care about Users are opening their notifications 20% increase in users clicking through to links from a notification ITERATION 1
- 85. Goals Signals Metrics Users are notified for things they care about Users are opening their notifications 20% increase in users clicking through to links from a notification ITERATION 1
- 87. Informative
- 90. ActionableInformative Notifications that are valuable to users by delivering information
- 91. ActionableInformative Notifications that are valuable to users by delivering information Issue was transitioned to ‘Done’
- 92. ActionableInformative Notifications that are valuable to users by delivering information Issue was transitioned to ‘Done’
- 93. ActionableInformative Notifications that are valuable to users by delivering information Notifications that are valuable to users by driving action Issue was transitioned to ‘Done’
- 94. ActionableInformative Notifications that are valuable to users by delivering information Notifications that are valuable to users by driving action Issue was transitioned to ‘Done’ You were mentioned on a comment
- 95. ActionableInformative Notifications that are valuable to users by delivering information Notifications that are valuable to users by driving action Issue was transitioned to ‘Done’ You were mentioned on a comment
- 96. ActionableInformative NOTIFICATION WAS OPENED Notifications that are valuable to users by delivering information Notifications that are valuable to users by driving action Issue was transitioned to ‘Done’ You were mentioned on a comment
- 97. ActionableInformative NOTIFICATION WAS OPENED NOTIFICATION WAS CLICKED Notifications that are valuable to users by delivering information Notifications that are valuable to users by driving action Issue was transitioned to ‘Done’ You were mentioned on a comment
- 98. Goals Signals Metrics Users are notified for things they care about 20% increase in users clicking through to links from a notification Users are opening their notifications ITERATION 1
- 99. ITERATION 1 Goals Signals Metrics Users are notified for things they care about 20% increase in users clicking through to links from a notification Users are opening their notifications
- 100. ITERATION 1 Goals Signals Metrics Users are notified for things they care about 20% increase in users clicking through to links from a notification ITERATION 2 Users are opening their notifications
- 101. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users clicking through to links from a notification Users are clicking through from their actionable notifications ITERATION 2
- 102. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications Users are clicking through from their actionable notifications 20% increase in users opening informative 20% increase in users clicking through an actionable notification ITERATION 2
- 103. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users opening informative Users are clicking through from their actionable notifications 20% increase in users clicking through an actionable notification ITERATION 2
- 104. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users opening informative Users are clicking through from their actionable notifications 20% increase in users clicking through an actionable notification ITERATION 2
- 105. ActionableInformative NOTIFICATION WAS OPENED NOTIFICATION WAS CLICKED Notifications that are valuable to users by delivering information Notifications that are valuable to users by driving action Issue was transitioned to ‘Done’ You were mentioned on an Issue Issue was shared with you
- 106. Actionable NOTIFICATION WAS CLICKED Notifications that are valuable to users by driving action You were mentioned on an Issue Issue was shared with you
- 107. Actionable NOTIFICATION WAS CLICKED Notifications that are valuable to users by driving action You were mentioned on an Issue Issue was shared with you
- 108. Actionable NOTIFICATION WAS CLICKED Notifications that are valuable to users by driving action You were mentioned on an Issue Issue was shared with you ACTIONED
- 109. Actionable NOTIFICATION WAS CLICKED Notifications that are valuable to users by driving action You were mentioned on an Issue Issue was shared with you ACTIONED
- 110. Actionable NOTIFICATION WAS CLICKED Notifications that are valuable to users by driving action You were mentioned on an Issue Issue was shared with you ACTIONED Notification Product Action
- 111. Actionable NOTIFICATION WAS CLICKED Notifications that are valuable to users by driving action You were mentioned on an Issue Issue was shared with you ACTIONED
- 112. Actionable NOTIFICATION WAS CLICKED Notifications that are valuable to users by driving action You were mentioned on an Issue Issue was shared with you Comments on an issue Likes a comment Transitions an issue Edits an issue Dwells on an issue (30 sec) ACTIONED
- 113. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users opening informative Users are clicking through from their actionable notifications 20% increase in users clicking through to links from a notification ITERATION 2
- 114. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users opening informative Users are clicking through from their actionable notifications 20% increase in users clicking through to links from a notification ITERATION 2
- 115. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users opening informative Users are clicking through from their actionable notifications 20% increase in users clicking through to links from a notification ITERATION 2ITERATION 3
- 116. Goals Signals Metrics Users are notified for things they care about Users are opening their informative notifications 20% increase in users opening informative Users are clicking through from their actionable notifications 20% increase in users actioning a notification ITERATION 3
- 120. Learnings Understand your tradeoffs Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 121. Learnings Understand your tradeoffs Metric: Reduce email volume by x% Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 122. Learnings Understand your tradeoffs Metric: Reduce email volume by x% Counter metric: 0% decrease in Jira MAU Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 123. Learnings Set a target for your team Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 124. Learnings Set a target for your team Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 125. Learnings Set a target for your team What is the minimum % that you would be disappointed with? Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 126. Learnings Set a target for your team Countermetric:0%decreaseinJiraMAU Metric:Reduceemailvolumeby20%x% Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 127. Learnings Set a target for your team Metric:Reduceemailvolumeby20% Countermetric:0%decreaseinJiraMAU Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 128. Learnings Be clear with what you're setting out to achieve Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 129. Learnings Be clear with what you're setting out to achieve Metric: increase email open rates by 20% Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 130. Learnings Be clear with what you're setting out to achieve Metric: increase email open rates by 20% Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 131. Learnings Be clear with what you're setting out to achieve Metric: increase email open rates by 20% Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 132. Learnings Be clear with what you're setting out to achieve Relative: 20%→24% Absolute: 20%→40% Metric: increase email open rates by 20% Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 133. Learnings Counter metrics Avoid x% Be specific Qualitative Pair quantitative metrics with qualitative analysis Keep the balance
- 134. Learnings Counter metrics Avoid x% Be specific Qualitative Pair quantitative metrics with qualitative analysis Quantitative: Tellsyouwhat’shappening Qualitative: Allowsyoutounderstandwhy Keep the balance
- 135. Learnings Balance product execution vs setting the perfect metric Counter metrics Avoid x% Be specific Qualitative Keep the balance
- 136. Takeaways
- 137. 1 Start by determining user success Takeaways
- 138. What does user success look like?
- 139. How can we measure that? What does user success look like?
- 140. 1 Start by determining user success Takeaways
- 141. 1 2 Start by determining user success Use a framework & iterate Takeaways
- 144. 1 2 Start by determining user success Use a framework & iterate Takeaways
- 145. 1 3 2 Start by determining user success Use a framework & iterate Set a target & be specific Takeaways
- 146. 1 3 2 Start by determining user success Use a framework & iterate Set a target & be specific Takeaways
- 147. 1 4 3 2 Start by determining user success Use a framework & iterate Set a target & be specific Balance execution & learning with setting the perfect metric Takeaways
- 148. Thank you! JOSEPHINE LEE | PRODUCT MANAGER | ATLASSIAN