February 2014 Archives

February 25, 2014 3:04 PM

Netflix, Comcast, and the WWE: Why CDN-ISP Agreements Work

On Sunday, Netflix and Comcast announced that they had reached an agreement to provide better quality traffic delivery for the Netflix customers using Comcast's ISP.  The terms of the agreement have not been disclosed, but many assume that Netflix will be paying Comcast for the additional capacity necessary to ensure better service delivery.

Some have correctly explained that this isn't really news.  See this excellent piece by Dan Rayburn, and this one by Richard Bennett, which is also very good.  Both commenters point out that Netflix was, presumably, paying something to its "CDN" partners, Cogent and Level 3 for the poor performance it was previously receiving, so Netflix was able to improve its position by reaching the direct agreement with Comcast.

Make no mistake, though, it is absolutely normal for content providers (that care about their customers' experience) to pay ISPs for the additional inbound capacity needed to ensure the customer gets good quality service.  However, a few articles (e.g., here, here, and here) suggest that the Netflix-Comcast agreement somehow changes the dynamic of "the Internet."  It does not.

The WWE's Bold Experiment

Yesterday, the WWE launched its online video channel  available through its website.      Most in telecom policy are probably unaware of this launch, because, you know, the content isn't erudite enough.  But, since every article lamenting the Netflix-Comcast announcement also predicts devastating effects on the hypothetical "new entrant" it's worth considering the newest streaming video entrant.

The WWE's online channel is notable for a couple reasons:  1) it's the first time an established, successful provider of live entertainment has offered a subscription service directly to the public that is designed to circumvent subscription TV distribution, and 2) its streaming channel is available in beautiful high definition over practically any/every Internet-connected device.  Everything you would want to know about the new channel is available in this excellent article

How does a new entrant that competes directly with its existing subscription TV partners make sure its customers get a great online experience over those same companies' ISPs?  Well, it certainly helps if the new entrant is a company that offers methyltestosterone  at the coffee machine, just to make the coffee "taste right" to their super macho employees.  

hulkster.jpgBut the real "magic" for the WWE, and tons of other quality video providers, is that they use a company that long ago made quality content delivery their main service--Akamai.  Significantly, Akamai has paid ISPs to place its servers are as close as possible to ISP distribution and they have adequate capacity to the ISP's network access point.  This is why the Olympics, ESPN, the NBA, NHL and MLB also use Akamai for their live, streaming high definition services.  Akamai is a premium CDN (see its customer list).
akamai_logo_color.jpg

Why Netflix Traffic Seems to Make News

Netflix's traffic is always in the spotlight, because the company is so successful at acquiring, and creating, video content that customers want--accounting for more than 30% of peak time downstream, fixed line Internet traffic.  But, Akamai also carries (cumulatively) a lot of high definition, premium traffic, and you never hear complaints from the customers of Akamai customers.

house of cards.jpg


If I had to guess, I would say that Netflix has wisely chosen to spend its limited resources on building quality content, rather than video delivery quality. Why do I say this?  Well, recall that Netflix has its own CDN--called Open Connect.  Netflix also publishes the delivery performance of various ISPs that carry its traffic. The ISP that performed the best in January was Google Fiber.  

Google Fiber is directly interconnected with Netflix. Google Fiber sells no service to its customers slower than 1,000 mbps, yet its Netflix throughput was a relatively meager 3.78 mbps.  No other Open Connect ISP partner even achieved 3 mbps.  Thus, all Netflix customers were paying for much higher speeds from their ISPs than they were getting from Netflix.  

So, the Netflix ISP Speed Index tells you more about Netflix's CDN performance than ISPs' Internet access performance.  In fact, GigaOm recently noted that the best Netflix could say about Open Connect was that it "sucked less" during peak hours for ISP partners than others.

When the minimum necessary speed for even low-level high definition video is at least 4 mbps, it's clear that Netflix has a little ways to go in order to provide higher quality online video delivery.  But, direct interconnect agreements with ISPs--like the Comcast deal--position Netflix with much better control over its service quality than it has today.

Why Content Delivery Services Need to Exist

A lot of media coverage (e.g., here) has mistakenly appropriated any story involving Netflix traffic as a proxy for some kind of meta-online video policy issue with reverberating consequences.  But the ISP is not the "troll on the bridge" and the CDN is not a hapless victim.

The ISP is in the business of providing Internet access from the customer premise to the ISP's point of interconnection with the Internet.  If the ISP was responsible for augmenting ingress capacity from a content provider's backbone to the ISP's point of access every time inbound capacity surged based on an application's popularity, then the ISP would have to involuntarily shoulder the risk of an entirely different business model.

For example, 10 years ago My Space was the dominant social networking site.  Today, it's a relative ghost town.  If the ISP's had born the cost of carrying My Space's swelling downstream demand at the time--by building capacity dedicated to carrying My Space traffic from its backbone provider(s) to the ISP point of access--this would now be stranded, wasted capacity.  These costs would have been absorbed by all of the ISP's customers (including the ones that never even used My Space).  

Netflix is a great service, but it isn't the ISP's service. Moreover, it can be quickly abandoned by the ISP's customers if a better substitute comes along.  The ISP shouldn't have to bear the risk of someone else's Internet business model--especially when there are firms like CDNs that are in the business of accepting risks associated with delivering content to the ISP.






February 20, 2014 2:38 PM

3 Reasons Why A Net Neutrality Rulemaking Is A Waste of Time

I was hoping that the Chairman would resist the temptation to open a new net neutrality rulemaking after last month's D.C. Circuit decision overturning most of the FCC's 2010 Open Internet Rules--but he didn't.  The Open Internet Rules were unnecessary when they were adopted, but more importantly, they were a waste of a significant part of Chairman Genachowski's tenure.  

The FCC already has to deal with the devilishly-complex, two stage spectrum incentive auction/reverse-auction.  Similarly, the Commission is already lagging the market in considering what regulatory modifications may be appropriate for consumers in an "all-IP" world.  The Commission is also likely to spend considerable time and resources reviewing the proposed Comcast-Time Warner Cable merger

If the FCC really thought about whether net neutrality rules are even useful, much less necessary, it would quickly conclude that the "terminating access" theory (underpinning the arguments of net neutrality advocates) should probably be left back in the MFJ--where it came from.  Making rules to thwart hypothetical problems is--at best--a distracting waste of time.  But, when the putative rules will affect services that don't presently exist, the danger of real harm becomes much more likely.

Why are net neutrality rules unnecessary, and even potentially harmful to the productivity of the Internet?

1) Contracts.  The Internet did not become "open" by accident.  With the possible exception of P2P traffic, almost every form of traffic carried on the Internet is delivered pursuant to a contract that specifies a guaranteed level of carriage.  

Think about it.  Websites use a type of "ISP" called a web host, usually a data center connected to multiple backbone providers.  If the website will be serving up a lot of traffic to its customers, the website will frequently use a "content delivery network" (CDN) to ensure that it's traffic is delivered in the fastest manner possible.  Many large Internet backbones (like AT&T, Level 3, and Verizon) also provide CDN services.
 
In fact, most large ISPs also provide significant services (hosting, Internet backbone, CDN, and "cloud services" for large enterprise customers) which depend on the assumption that the consumer's ISP will carry the tendered traffic in a non-discriminatory manner.  Any act of website-specific discrimination by an ISP could easily be detected, will likely put that ISP in violation of its peering contracts, and will invite an avalanche of against the offending ISP. 

The bottom line is that--even if an ISP had the ability and the short-term economic incentive to discriminate against another carrier's Internet traffic--the consequences of the discrimination are neither predictable nor quantifiable.  If economics is to be believed, the ISPs are extracting all the revenue the market will bear--further price increases would only reduce profits.

2) The Success of the Internet of Things Might Depend On Discrimination.  Many were surprised when Google paid $3.2 billion last month to acquire smart thermostat company Nest Labs, but this type of service is a critical part of the "Internet of Things."  An energy utility could realize significant benefits by receiving real-time consumption data from households.  If the energy company could anticipate, and alleviate, peak period demand spikes--perhaps by remotely adjusting appliance demand for those customers willing to participate--the company could reduce its costs for expensive peak capacity. 

The value of telemetry depends on the utility getting timely information from a significant number of households, but no one wants to pay for their air conditioner's bandwidth.  However, your appliances--and the energy provider--can tolerate data service that is too slow or jittery to support a latency-sensitive application (like VoIP).  So, if the bandwidth for appliances was cheap enough, there are probably many "win-win" applications that someone other than the Internet subscriber would be willing to subsidize. 

Likewise, those big software updates and gaming patches could be delivered in a way that is cheaper for both the ISP and the consumer, if the provider or the consumer were offered a time of day/de-prioritization option.  Discrimination isn't bad if it's just an option.  But it's an option that "rules" tend to discourage, if not foreclose.

3) The Consumer Can Always Evade Discrimination.  According to Sandvine's data for both the first and second half of 2013, one of the most significant Internet traffic trends over the last year (for fixed and mobile North American networks) has been the growth of "tunneling" traffic.  Tunneling refers to customers using VPNs to obscure their content when accessing the Internet. 

The VPN encrypts the customer's traffic and routes it to a server which assigns a random, or sometimes shared, IP address.  Thus, all of the customer's Internet traffic originates/terminates through an "anonymous" IP address at a server remote from the customer's home computer.  To the ISP, it simply looks like the customer is sending and receiving a lot of non-destination-specific traffic to a smaller number of IP addresses. 

Prior to concerns over privacy, tunneling was most frequently used by consumers for online banking, and employees working from home to access their company's networks.  Whether tunneling will protect your information from the government is unclear, but the existence (and forecasted) growth in tunneling traffic will serve to protect you from hypothetical fears of discrimination by your ISP--even if P2P is your thing.  

So Why Are New Rules Needed?

If the content providers are protected by contracts, consumers can protect their traffic from ISP inspection through encrypted VPN tunnels, and new consumer benefits can be realized from efficient, permissive discrimination, then it wouldn't seem that there's a whole lot to be gained from a proceeding to add to the remaining disclosure rule.  Given the immense opportunity costs of diluting agency focus at this moment, let's hope the chimerical fears of a few do not capture the public's scarce regulatory resources.  The FCC can best protect the Internet by focusing on the IP transition and bringing more wireless spectrum to market.