VOD Delivery Issue Identified
Incident Report for Zype

I would like to express our apologies for the VOD delivery issues that occurred over the weekend. We understand how important quality of service and performance is to you, as it is to us. 

Identification and Diagnosis

Starting Friday morning, we noticed buffering ratios increase in our QoS system. As we monitored throughout Friday afternoon and evening, we saw that the buffering ratio returned to the expected thresholds across viewership. Saturday morning, we identified that it began increasing again. 

Our team identified the root cause of the issue to be related to a specific header being passed from content delivery network requests. These headers were conflicting with updated browser policies, specifically in relation to updated versions of Google Chrome (Chromium-based browsers) and Firefox handling these headers differently than before. Throughout Saturday, our engineering team worked to investigate, mitigate, and ultimately resolve the underlying issue by updating the CDN requests to respond properly without the headers causing the problem.

Next steps

Our team is focused on ensuring stability and preventing this from happening in the future. We have multiple efforts underway to accomplish this, including:

  • Implementing improved monitoring services for faster detection and recovery
  • Re-evaluating all CDN endpoints to ensure no configurations will result in future failures
  • Re-evaluating multi-CDN approach to ensure the ability to quickly update configurations if needed

We’re committed to improving the quality of our service, and will continue to implement improvements across all systems.

Posted Oct 26, 2021 - 13:55 EDT

We have seen performance stabilize on VOD delivery as the deployment reaches each CDN endpoint. We will continue monitoring and will provide a post mortem on this incident.
Posted Oct 23, 2021 - 19:50 EDT
As the deployment is updating across CDN nodes, we're seeing performance improve across all endpoints.
Posted Oct 23, 2021 - 18:37 EDT
A fix has been deployed and is propagating across all requests. We are seeing VOD delivery stabilize and are monitoring the results.

If a consumer has issues persist, their requests may be cached with bad request data. To reset that, they should hard refresh their browser by navigating to the refresh option in their browser and selecting "Empty cache and Hard Reload". Over time, cache will be reset automatically.
Posted Oct 23, 2021 - 17:08 EDT
We have deployed an update to mitigate the issue. This update will take around 45 minutes to fully propagate. We will be monitoring the results as this is rolled out.
Posted Oct 23, 2021 - 16:17 EDT
Our team is actively working on deploying a solution that resolves the issue.
Posted Oct 23, 2021 - 16:10 EDT
Our engineers are continuing to work on resolving this issue.
Posted Oct 23, 2021 - 14:54 EDT
We are continuing to work on a fix for this issue.
Posted Oct 23, 2021 - 13:52 EDT
We have identified that this issue is only impacting streaming VOD on Google Chrome. The issue is not prevalent in other browsers.
Posted Oct 23, 2021 - 12:33 EDT
Our engineers are actively investigating an issue with VOD content delivery. We're reporting intermittent loading issues at this time.
Posted Oct 23, 2021 - 11:21 EDT
This incident affected: Content Delivery.