I would like to express our apologies for the VOD delivery issues that occurred over the weekend. We understand how important quality of service and performance is to you, as it is to us. 

Identification and Diagnosis

Starting Friday morning, we noticed buffering ratios increase in our QoS system. As we monitored throughout Friday afternoon and evening, we saw that the buffering ratio returned to the expected thresholds across viewership. Saturday morning, we identified that it began increasing again. 

Our team identified the root cause of the issue to be related to a specific header being passed from content delivery network requests. These headers were conflicting with updated browser policies, specifically in relation to updated versions of Google Chrome (Chromium-based browsers) and Firefox handling these headers differently than before. Throughout Saturday, our engineering team worked to investigate, mitigate, and ultimately resolve the underlying issue by updating the CDN requests to respond properly without the headers causing the problem.

Next steps

Our team is focused on ensuring stability and preventing this from happening in the future. We have multiple efforts underway to accomplish this, including:

• Implementing improved monitoring services for faster detection and recovery
• Re-evaluating all CDN endpoints to ensure no configurations will result in future failures
• Re-evaluating multi-CDN approach to ensure the ability to quickly update configurations if needed

We’re committed to improving the quality of our service, and will continue to implement improvements across all systems.