Bitmovin LIVE Tech Talks: 5 Analytics Metrics That Matter

The presentation will start in a few minutes...
1

April 21st
, 2020
5 Video Analytics
metrics that matter
Monitor and improve quality of experience
B I T M O V I N I W E B I N A R
All rights reserved. © Bitmovin Inc 2020

Moderator & Panelists
Andrea
Fassina
Developer
Evangelist
Christoph
Prager
Product
Manager,
Analytics

● Introduction
● Who is Bitmovin
● What are Bitmovin products
● 2020 Market drivers
● OTT app architecture
● Quality of experience and Analytics
● 5 Analytics metrics that matter
○ (Video) startup time
○ Impressions and total hours watched
○ Video bitrate heatmaps
○ Error type and %
○ Rebuffering %
● Conclusion
● Q&A
Agenda

ANALYTICS
Fix playback issues with real-time
data before your viewers see it.
Monitor infrastructure performance
insights, anytime, all the time.
Visualize data, your way.
ENCODER
Deliver content in the highest
quality at the lowest streaming
costs. Prepare content at high speed
and deploy on any environment.
PLAYER
Reach viewers on largest number
of devices using a multi-device
player. It’s ﬂexible and easy to
deploy, utilizing built-in SVOD and
AVOD features, giving complete
control of the viewing experience.

2020 Online Video Market Drivers
Better Viewer
Experience
Increase viewer
satisfaction by
exceeding expectations
on video quality
1
Operational
Optimization
Control cost, mitigate
risk, better manage
complexity, and
maintain relevance
3
Faster Time
to Market
Get assets, products
and services to
market faster and
more efﬁciently
2

Quality of Experience
What is it?
Why does it matter?

Collect data using Bitmovin Analytics:
Quality of Experience: What and How?
POST LAUNCH
Data (in JSON) is
sent every second
from player to REST
API Analytics
endpoint
Collect playback
stats, network
conditions, ads
interaction, user
info (custom data)
Custom data
insertion points to
integrate with
other analytics
services and/or
CMS

5 Metrics that Matter:
Quality of Experience: Metrics that Matter
POST LAUNCH
(Video) startup
time
Video bitrate
heatmap
Impressions &
total hours
watched
Rebuffering %Errors type
and %

How long does the ﬁrst frame take
to load on the client’s device?
(Video) startup time

Multiple terms for startup
time such as:
● Startup delay
● Join time
● Startup time
● Video Startup Time
Working Definition:
Video Startup Time is defined as
the time from when a user presses
“Play” until the first frame of a
video appears.
Startup time(s)

General assumptions
Assumption: The longer the wait the more users will drop off
With increased bandwidth conditions a feeling of immediacy with minimal
loading times has become core to the internet experience
Competing with ofﬂine experiences: Ofﬂine experiences, even though
they had considerable loading times, are still perceived as the benchmark
for immediacy

Data from the past

● Spotify being the service that
brought music streaming to the
mainstream
● Operated under premise that
only a (near) real-time experience
of listening the music will bring
users to online
● They always had the objective of
bringing down the startup time
to a threshold of 200ms
Competing with the
Ofﬂine Experience
Source: Reid Hoffman - Masters of Scale Podcast:
“How to build trust fast” - Interview with Daniel Ek
https://mastersofscale.com/daniel-ek-how-to-build-trust-fast/

Benchmark study
“for every
additional second
of startup delay,
an additional 5.8%
of your viewership
leaves”*
Benchmark study for startup
delay in video comes from
Akamai
*Akamai (2012): “Maximizing
Audience Engagement.”
Whitepaper
Source: Akamai (2012): “Maximizing Audience Engagement.”
https://content.akamai.com/PG2114-Audience-Engagement-WP.html

Ben Dodson (Snap Inc) at Demuxed 2019
suggests viewers have become even
more impatient compared to 2012
● The data suggests that within 2
seconds the entire audience of a
speciﬁc “snap” has disappeared
● Context dependent
○ Very speciﬁc target
audience (teenagers)
○ Length of the video (short)
○ Consumption patterns –
skipping through videos
Impatient Teenagers
Source: Ben Dodson - “Analyzing Video Metrics like
Richard Feynman” (Presented at Demuxed 2019)
https://www.youtube.com/watch?v=fMcCWQrO-vg

● Bitmovin Analytics Industry
Insights shows that Median
Startup Time across our
customer-base is at
1.537 seconds
● Far from the original 200ms
that Spotify had originally
identiﬁed as the perceived
real-time threshold
Bitmovin Analytics Data

Video streaming is a complex process with many inﬂuencing factors across the delivery chain
Video is complex
Video Startup Time
Player Startup Time
DRM Load time
Ad Load time
Ad tag load time

● Licensed content requires protection
via Digital Rights Management
● The licensing server managing the
DRM requests are often not geared to
deal with high concurrencies
● Especially a problem in live situations
● Monitoring this inﬂuence on your
service is key
[SVOD] DRM Load Time

For video platforms relying on ad revenues
knowing how third party ad infrastructure
inﬂuences their Quality of Experience is
even more.
● Two possible inﬂuencing factors
○ Ad Tag Load Time
○ Ad Startup Time
● Can get even more complicated
when using waterfalling and fallbacks
[AVOD] Ad Startup Time

How many and for how long
did users consume the content?
Impressions & total
hours watched

Impressions & total hours watched
POST LAUNCH
Impressions &
total hours
watched
How many and for how long did users consume the content?
Developer Network content

Breaking up impressions by device type
POST LAUNCH
Impressions &
total hours
watched
What are the most used devices and browser types?
Developer Network content
Demos content

Breaking up impressions by device type (cont’d)
POST LAUNCH
Impressions &
total hours
watched
Focusing on playback on TV devices
Demos content

Breaking up impressions by codec type
POST LAUNCH
Impressions &
total hours
watched
What are the most used widely used codecs that your users can support
Demos content
HVC (High-Performance Video Coding)

Impression & total hours watched and custom data
POST LAUNCH
Impressions &
total hours
watched
CRM integration with watching data
Custom data integration into Analytics
data so that you can:
- Better support your users
- Inform your A/B campaigns with
accurate video consumption data
- Aggregate viewing data with
user proﬁles
Custom data

What is the connection speed and context
of your users as they watch the video?
Video bitrate heatmap

Streaming video consumption is increasing
Video services are under pressure to
deliver video with less bitrates
without sacriﬁcing quality

● See if viewers can actually
consume the renditions you offer
○ Can they consume the
renditions under acceptable
startup time and buffering
conditions
● Wasted bitrates are wasted dollars
○ On the CDN side
○ Redundant renditions become
expensive renditions if none
of your viewers are able to
watch them
Optimizing bitrates is key

● A video bitrate heatmap displays
the encoding rendition
consumption patterns of viewers
across the runtime of a video asset
● Left Y-axis: bitrates available
in encoding
● X-axis: video playtime in buckets
● Right Y-axis: Percentage of
viewers that consumed that
bitrate rendition
Using Bitrate Heatmaps

● The middle rendition is hardly
used by the service’s viewers
● The video service provider could
consider removing this rendition
● However, need to consider
potential role of the renditions in
the ABR logic
Redundant renditions

● Customers are centered
around the 400 kbit/s rendition
● This suggests that the
bandwidth available of viewers
is not high enough to
consistently switch to the
highest available rendition
HiRes redundancies

Take away
Optimizing bitrates is a balancing act
Use encoding features like per title encoding, 3 pass to further optimize
videos on the encoding side can help already on the encoding side
But, Analytics data can provide additional insights into what
suitable for your viewers
Especially in bandwidth-constrained regions

Where there any playback errors?
If so what are they and why?
Errors type and %

Errors type and %
POST LAUNCH
Errors type
and %
Where there any playback errors? If so what are they and why?

Seek time
POST LAUNCH
Seek time
The time to start a seek and resume playback is a performance
indicator for network conditions

Are you providing a continuous
playback experience?
Rebuffering %

Are you providing a continuous playback experience?
5. Rebuffering %
POST LAUNCH
Rebuffering %
Bigger size of
segments,
more time do
download
Tablets mean higher
resolution and often
cellular connection
Switch to VVC!

These metrics matter because they
present a holistic health check of the
encoding process and playback client -
insights for iterative improvements on:
Quality of Experience:
Metrics that Matter
POST LAUNCH
Operational efﬁciency
Viewer experience
Faster time to market

Q&A

How to reduce costs with the same - or
better - SLA?
Operational efﬁciency
POST LAUNCH

Are your users enjoying the service?
Viewer experience
POST LAUNCH

Is your content watchable on every
device?
Sky Brazil saved $$$ deploying the
Bitmovin player across all major devices,
instead of creating their own.
Play everywhere
POST LAUNCH

● With people staying at home all the
time, streaming has increased
● Pressure from institutions to reduce
quality to lower strain on internet
pipes
● Video analytics can help meet and
measure those SLA overtime and
encoding settings can be tuned
accordingly to produce the best
quality at the lowest ﬁle size.
● For example: use VP9 for device
population that supports it, do three
pass and per title encoding
Quality of Experience at
the time of COVID-19
POST LAUNCH

Thanks!
Andrea Fassina
Developer Evangelist
Bitmovin

Bitmovin LIVE Tech Talks: 5 Analytics Metrics That Matter

More Related Content

What's hot (20)

Similar to Bitmovin LIVE Tech Talks: 5 Analytics Metrics That Matter (20)

More from Bitmovin Inc (14)

Recently uploaded (20)

Bitmovin LIVE Tech Talks: 5 Analytics Metrics That Matter