March 5, 2014 2:50 PM
Last week began with news
that Netflix had decided to improve its long-languishing service for customers of Comcast's ISP by directly interconnecting with Comcast, cutting out Internet transit provider Cogent Communications
. The Cogent-Comcast "peering" dispute had been a long-standing topic of media speculation.
Prior to the announcement of the direct interconnection agreement, many media sources
had wished to present this dispute as just another iteration of ISPs attempting to "extort" money from sources of competing content. Netflix's decision to solve the problem itself had pretty much eliminated the "poor Cogent" articles by the end of the week--after Dan Rayburn's detailed explanation
of the deal.
Knowing what we have learned in the past week, it would seem that Netflix's "ISP Speed Index
" is more likely a fairer representation of the performance of Netflix's CDN versus that of its third party transit providers. The Netflix ISP Speed Index
is obviously not attempting to measure "true" ISP speeds, because even gigabit provider Google Fiber hovered around 3.5 Mbps for most of last year.
The ISPs to which Netflix connects with its own CDN generally see speeds in the 2.3 Mbps-2.9 Mbps range. For the customers of ISPs served by a third party transit provider, speeds are often much lower. But how bad is too bad? The "Mendoza Line" for Poor Streaming Transit Performance
Within the past day, Verizon's CEO has said
that their FiOS ISP is close to a direct interconnection agreement with Netflix. Recall, that Cogent also had issues
with refusing to augment inbound capacity to Verizon.
In fairness to Cogent, it does not claim to offer CDN service. Still, you have to wonder, where is Netflix likely to switch away from "plain old transit" service next?
Luckily, Netflix kind of tells us (and its transit/CDN vendors) the levels of service deterioration at which they can expect the "Never mind, I'll do it my damn self" call from Reid Hastings. Here is the performance of several ISPs who offer broadband access at speeds well above 5 Mbps for the last 4 months, as reported by Netflix's ISP Speed Index.
I looked at the Netflix ISP Speed Index for the last 12 months, and, traffic delivery at anywhere between 2-3 Mbps seems to be acceptable. If you're reading about customer service complaints, it's safe to assume that speeds have dropped below 2 Mbps for the ISP in question.
For both Comcast and Verizon customers, Netflix's transit/CDN vendor(s) performed well for most of the year--only deteriorating significantly over the last several months. Like in baseball, though, there is a Mendoza Line
for competent transit service, and it seems to be around 2 Mbps. So . . . if you're a Mediacom customer, you probably don't have much longer to wait.
Netflix's traffic is certainly growing very quickly. But, asymmetric, end-user-destined video traffic is growing very quickly for all ISPs, everywhere. Sandvine's second half 2013 traffic report shows that for the largest Internet consumers (North America
) peak hour video consumption may be as high as 70% for the average North American user, and 75% for the average Asian consumer (I got that by adding the "real time entertainment" and "P2P" categories for both continents). You'd think this would be a good thing for a content delivery network, no?Is "Peering" Net Neutrality Redux?Level 3
explains in this recent ex parte
, that all is not rosy in the world of the CDN--despite Level 3's own recognized success
in the market--because of the nettlesome costs that some ISPs believe should be paid by the party getting paid to carry the traffic to the ISP (namely, Level 3). BTW, this is completely the same issue as the Netflix carrier-v.-every-ISP-in-the-graph issue.
But, Level 3 characterizes what some would call the costs of providing CDN service, as "tolls" that unfairly target large CDNs distributing disproportionately-downstream, bandwidth-intensive traffic. Level 3 tells the FCC, "ISP tolls that facially apply equally to all traffic are effectively tolls on the most bandwidth-intensive services - video services that compete with the ISP's own video services." (ex parte
What types of Level 3 traffic are ISPs targeting? Well, sometimes it looks like this:
You see, HBO Go
is an online service that HBO
(not the subscription TV provider) offers its customers that already buy HBO's premium subscription TV channel. Since HBO, like almost every any other provider of streaming video service, lacks the demand to justify the hassle/cost of direct interconnection with the ISP, it uses the highly-competitive CDN market to deliver its service.
Level 3 is one of HBO Go's CDN partners
, for certain Apple devices. As Level 3 notes, the end-user is often buying a TV service from their ISP. But, poor quality delivery from the TV channel's online content devalues the TV product the ISP is selling
. Thus, it isn't clear why the ISP has an incentive to degrade Level 3's traffic, when the ISP knows that some significant portion of Level 3's traffic is a complement
to the ISP's subscription TV service.
business is focused on how to efficiently deliver asymmetric, bandwidth-intensive traffic to the customers of the ISP. If the ISP had to pay for/build incoming capacity from every CDN to the ISP, then it's good for the CDN, but the ISP can only limit its costs if they CDN traffic stays put.
So, why would Level 3 want the FCC to impose regulations that give an ISP any
kind of interest in which CDN provider a content owner would choose? Maybe, because sometimes those CDN customers decide to build their own CDN, as soon-to-be-former Level 3 customer Apple is reportedly
doing. New CDNs equal new competition for small content owners, and that's not always a great thing . . . for CDN incumbents. Now I get it.
February 25, 2014 3:04 PM
On Sunday, Netflix
and Comcast announced
that they had reached an agreement to provide better quality traffic delivery for the Netflix customers using Comcast's ISP. The terms of the agreement have not been disclosed, but many assume
that Netflix will be paying Comcast for the additional capacity necessary to ensure better service delivery.
Some have correctly explained that this isn't really news. See this excellent piece
by Dan Rayburn, and this one
by Richard Bennett, which is also very good. Both commenters point out that Netflix was, presumably, paying something to its "CDN" partners, Cogent
and Level 3
for the poor performance it was previously receiving, so Netflix was able to improve its position by reaching the direct agreement with Comcast.
Make no mistake, though, it is absolutely normal for content providers (that care about their customers' experience) to pay ISPs for the additional inbound capacity needed to ensure the customer gets good quality service. However, a few articles (e.g., here
, and here
) suggest that the Netflix-Comcast agreement somehow changes the dynamic of "the Internet." It does not.The WWE's Bold Experiment
Yesterday, the WWE launched
its online video channel available through its website
. Most in telecom policy are probably unaware of this launch, because, you know, the content isn't erudite enough. But, since every article lamenting the Netflix-Comcast announcement also predicts devastating effects on the hypothetical "new entrant" it's worth considering the newest streaming video entrant.
The WWE's online channel is notable for a couple reasons: 1) it's the first time an established, successful provider of live entertainment has offered a subscription service directly to the public that is designed to circumvent subscription TV distribution, and 2) its streaming channel is available in beautiful high definition over practically any/every Internet-connected device. Everything you would want to know about the new channel is available in this excellent article
How does a new entrant that competes directly with its existing subscription TV partners make sure its customers get a great online experience over those same companies' ISPs? Well, it certainly helps if the new entrant is a company that offers methyltestosterone
at the coffee machine, just to make the coffee "taste right" to their super macho employees.
But the real "magic" for the WWE, and tons of other quality video providers, is that they use a company that long ago made quality content delivery their main service--Akamai
. Significantly, Akamai has paid ISPs to place its servers are as close as possible to ISP distribution and they have adequate capacity to the ISP's network access point. This is why the Olympics, ESPN, the NBA, NHL and MLB also use Akamai for their live, streaming high definition services. Akamai is a premium CDN (see its customer list
).Why Netflix Traffic Seems to Make News
Netflix's traffic is always in the spotlight, because the company is so successful at acquiring, and creating, video content that customers want--accounting for more than 30% of peak time downstream, fixed line Internet traffic
. But, Akamai also carries (cumulatively) a lot of high definition, premium traffic, and you never hear complaints from the customers of Akamai customers.
If I had to guess, I would say that Netflix has wisely chosen to spend its limited resources on building quality content
, rather than video delivery quality. Why do I say this? Well, recall that Netflix has its own CDN--called Open Connect
. Netflix also publishes
the delivery performance of various ISPs that carry its traffic. The ISP that performed the best in January was Google Fiber
Google Fiber is directly interconnected with Netflix. Google Fiber sells no service to its customers slower than 1,000 mbps, yet its Netflix throughput
was a relatively meager 3.78 mbps. No other Open Connect ISP partner even achieved 3 mbps. Thus, all Netflix customers were paying for much higher speeds from their ISPs than they were getting from Netflix.
So, the Netflix ISP Speed Index
tells you more about Netflix's CDN performance than ISPs' Internet access performance. In fact, GigaOm recently noted
that the best Netflix could say about Open Connect was that it "sucked less" during peak hours for ISP partners than others.
When the minimum necessary speed for even low-level high definition video is at least
4 mbps, it's clear that Netflix has a little ways to go in order to provide higher quality online video delivery. But, direct interconnect agreements with ISPs--like the Comcast deal--position Netflix with much better control over its service quality than it has today. Why Content Delivery Services Need to Exist
A lot of media coverage (e.g
) has mistakenly appropriated any story involving Netflix traffic as a proxy for some kind of meta-online video policy issue with reverberating consequences. But the ISP is not the "troll on the bridge
" and the CDN is not a hapless victim.
The ISP is in the business of providing Internet access from the customer premise to the ISP's point of interconnection with the Internet. If the ISP was responsible for augmenting ingress capacity
from a content provider's backbone to the ISP's point of access every time inbound capacity surged based on an application's popularity, then the ISP would have to involuntarily shoulder the risk of an entirely different business model.
For example, 10 years ago My Space
was the dominant social networking site. Today, it's a relative ghost town. If the ISP's had born the cost of carrying My Space's swelling downstream demand at the time--by building capacity dedicated to carrying My Space traffic from its backbone provider(s) to the ISP point of access--this would now be stranded, wasted capacity. These costs would have been absorbed by all
of the ISP's customers (including the ones that never even used My Space).
Netflix is a great service, but it isn't the ISP's service. Moreover, it can be quickly abandoned by the ISP's customers if a better substitute comes along. The ISP shouldn't have to bear the risk of someone else's Internet business model--especially when there are firms like CDNs that are
in the business of accepting risks associated with delivering content to the ISP.