Results tagged “Internet Peering”

March 4, 2015 12:42 PM

The FCC Avoided a Bigger Disaster on Interconnection . . . for Now

In case you didn't notice, the FCC's press release describing its decision to reclassify broadband Internet access is a little different on the subject of interconnection than Chairman Wheeler's "Fact Sheet" 3 weeks earlier.  The FCC's press release from last week is substantially similar to the Chairman's fact sheet, except that it contains no reference to the classification of  the "service that broadband providers make available to 'edge providers.'"

The Chairman's "Edge Service" Classification Proposal Was Really Unpopular

On the last day that parties could lobby the Commission, (February 19th) Google had meetings with senior advisors for the Chairman and the two Democratic Commissioners.  In these meetings, Google persuasively argued that "the Commission should not attempt to classify a "service that broadband providers make available to 'edge providers,'" because "this supposed additional service does not exist." Ex Parte (Internal quote to Chairman's "fact sheet.")   On the eve of the Commission's vote, the Wall Street Journal reported that "the FCC tweaked its language to address Google's concern."  

Without this report, it would be hard to know who to credit for killing a truly reckless idea.   That's because the following 5 parties all made arguments against the Chairman's proposed "new service" on the same day as Google:  Akamai, Free Press and New America's Open Technology Institute, National Hispanic Media Coalition, and Cox Communications.

When you somehow manage to get Free Press, New America, and the National Hispanic Media Coalition to oppose an additional Title II regulatory classification--because it makes your original Title II reclassification look even more legally suspect--that's when you know you've gone too far.  

The Importance of Direct Interconnection Agreements

The scariest aspect of the Commission's proposal must have been the FCC's casual willingness to disturb these companies' existing arrangements with ISPs by mandating the future terms on which they and others would be able to obtain interconnection.  For companies with little to no regard for the terms of their existing interconnection agreements, like Netflix and its transit vendors, the Commission could not make their situation worse.  But, for the leading Internet companies, the Commission must have appeared alarmingly ignorant of/indifferent to the importance of these agreements to Internet traffic delivery.

Reason 1:  The Disintermediation of Internet Traffic

In 2009, the University of Michigan, Arbor Networks, and Merit Network presented the results of the largest traffic study since the advent of the commercial Internet.  The study showed that, between 2007 and 2009, Internet traffic delivery changed radically.  Over only 2 years, Internet transit (the traditional "intermediary" between ISPs) became dramatically less important as a traffic delivery vehicle.  

Instead, major content providers began delivering more and more content directly to the consumer's ISP (either through their own networks or CDNs).  Accordingly, since 2009, the "tech giants" have been accelerating their investment in network assets and data centers to route their high bandwidth traffic directly to efficient delivery points in the ISP's networks.

Reason 2: Interconnection Agreements Reflect Valuable Investment

As the Internet has evolved to more efficiently deliver high-bandwidth content to consumers, the largest content providers--including Netflix for the first 4 years of its streaming service--have placed a premium on placing content closer to the customer.  Therefore, the largest traffic sources have entered into agreements to directly exchange traffic with their customers' ISPs.  When these agreements provide for the settlement-free exchange of traffic, it is because this reflects the mutual benefits received by both parties.  

CapEx Graph 1.jpgSince parties to settlement-free "peering" arrangements each provide the other with valuable network facilities, or other benefits, this value can be observed by looking at the investment the parties put into their networks (i.e., capital expenditures).  To get an idea of the importance of those agreements to Internet companies, consider the increase in capex by the largest Internet companies since 2009.

Regulation of Interconnection Terms Could Devalue Previous and Future Investment

As we can see from the chart above the major Internet companies have undertaken a massive amount of capital spending over the past 6 years in order to efficiently deliver content and services to consumers.  To be sure, not all of the Internet companies' capex is driven by traffic delivery interconnection concerns, but the increase in these companies' capex since 2009 correlates with the findings of the University of Michigan, et al., traffic study referenced above.  Moreover, news reports have confirmed spending on improved data networking infrastructure as a capex driver. See, e.g., here and here.   

This capital investment has been made by edge companies and CDNs with the expectation that it will allow these firms to provide a better experience to their customers than their competitors provide.  Indeed, Google notes that it has entered into peering agreements with some of the largest ISPs because it is "unable to use transit to reach users on those networks with reasonable quality." Ex parte at 2 (emphasis added.)

The risk of requiring ISPs to provide interconnection as a separate common carrier service was articulated succinctly by Akamai, which handles 15-30% of the world's Internet traffic.  Akamai argued that the FCC must not mandate the terms and conditions of ISP interconnection, because if the ISPs are required to provide access on equal terms to all:   

This is not technically feasible and the result could be access for none, which would decrease the performance, scalability, reliability and security of the Internet.
Akamai, February 20th ex parte at 1 (emphasis added).  In other words, Akamai understands and accepts that it "must often compete with others for access to ISP facilities."  Akamai, February 9th ex parte, at 3 (emphasis added). But, does the FCC accept interconnection as a legitimate element of competition?

CapEx Graph 2.jpgThe Commission Should Not Displace Competition with Regulation

Netflix, Cogent, and Level 3 assert that they cannot get interconnection with the ISPs--on the terms they would prefer--because of a lack of competition.  But, as Akamai explained, companies seeking the most efficient terms of distribution to the ISP's customers are competing with each other for the best access to these customers.  Could it be that competition is the reason the transit companies aren't getting the terms they want?

Compare the sum of the capex of Netflix's distribution chain over the relevant time period, with their competitors.  Is it surprising that Internet transit wants the FCC to "level the playing field?" 

Transit is at a disadvantage relative to direct interconnection because the Internet has evolved.  For the content distributors sending the most traffic, transit has not been the preferred solution for a long time.  But competition, and not a lack thereof, produced this outcome.  

Unlike the major focus of the larger net neutrality debate--which is concerned with adopting rules to foreclose future ISP service offerings--the regulation of interconnection terms and conditions is fraught with risks to existing service configurations.  The FCC must be careful not to use prescriptive regulation of ISP interconnection terms--in the name of competitive "neutrality"--to foreclose innovative firms (and their customers) from reaping the benefits of their own ideas, risks, and investments.


CapEx Data Table.jpg

December 18, 2014 4:31 PM

The Netflix/Comcast Dispute: Interconnection "Principles" vs. Consumer Rights?

[Note: This is the first post, in a series, in which we'll look at the Netflix/Cogent/Comcast congestion episode from earlier this year.  The focus will be on understanding this event from a different perspective than most of us may have thought about it before.  This series looks further into the question I raised in my last post which is: are existing laws--adequately enforced--sufficient to protect consumers?  For purposes of readability, the full citation of relevant facts has been placed at the end of this post.]

In my last post, I started to look at whether the protracted congestion--and associated consumer service disruptions--caused by the recent Netflix/Cogent/Comcast interconnection dispute indicated that the traditional voluntary agreement structure of the Internet was broken, or whether existing laws might not be enough to prevent protracted consumer disruption.  In a recent article, Prof. Susan Crawford, an advocate for all things Net Neutrality, also highlights the frustrations of customers caught in the middle of the Netflix/Cogent/Comcast dispute,

No federal, state, or local government exercises any oversight over this handful of interconnection points. No Better Business Bureau watches over how your requests for data are being treated.
Prof. Crawford is right to question why consumers got stuck with the short end of this wholesale dispute.  But, I disagree with Prof. Crawford's assumption that new laws--specific to the issue Netflix and Cogent blame for the protracted congestion--are needed. 

As noted in the last post, more specific contracts, and quicker enforcement by wholesale partners are one way to prevent extended periods of consumer frustration.  Similarly, there are also existing laws designed to protect consumers from intentional, or knowing, actions of third parties that prevent consumers from receiving services they believe they are purchasing.

The FTC has expressed concern that it will lose jurisdiction over Net Neutrality-related matters if the FCC decides to reclassify broadband as a Title II service (the FTC Act specifically exempts "common carriers").  Even though I was aware the FTC had asserted jurisdiction to handle Net Neutrality complaints, I didn't really think about the how important the FTC could be. . . until I started looking really closely at the Netflix/Cogent/Comcast congestion episode from earlier this year.   

"It's About the Principle"

When Netflix filed its Petition to Deny the Comcast-TWC merger  in August, I was interested in learning the circumstances that led to Netflix's direct interconnection agreement with Comcast.  I expected to see a pretty basic recitation of how Comcast kept unreasonably increasing interconnection prices, thereby forcing Netflix into lower quality interconnection arrangements.  

Instead, though, the brief spends a lot of time establishing that Netflix's principle--of not paying the ISP any portion of the costs of delivering its content to its customers--was the exclusive factor it relied upon in choosing vendors to deliver its customers' traffic.  The absence of any comparison of various input prices/vendor alternatives available to Netflix seemed odd.  The notion that Netflix was defending a not-purely-economic principle seemed odder still. 

The antitrust analytical framework (which was the ostensible basis for Netflix's opposition to the Comcast-TWC merger) recognizes economic efficiency, and not any unique firm-specific view of how an industry should work.  Yet, Netflix has never indicated that its decisions were based on immediate cost/price effects.  It has even clarified that the costs being imposed by the ISPs are not significant, nor has it raised prices for customers served by the offending ISPs.  See, e.g., this blog post.  Level 3 made the same argument in 2010--that it's not the cost, it's the principle. See this Ars Technica article.   

Prof. Susan Crawford, in the article mentioned above, also observes:

The FCC will find that the money amounts involved in these deals are low at the moment. It's the naked threat posed to the future that is the problem. . . .
The "naked threat posed to the future" may or may not be cause for concern, but--if this threat does not limit the immediate ability of a firm to deliver service--can a firm's reaction to such a threat excuse its performance under its customer contracts?  

It's possible that, for many months at least, Comcast customers (and those of certain other ISPs) were paying for service that Netflix knew to be substandard.  If Netflix failed to take any action to provide the grade of service for which its customers were paying, or to let prospective customers know they would be receiving degraded service for an indeterminate period, then it's possible that enforcement of existing laws might prevent future consumer abuses.    

The Federal Trade Commission Act

Section 5 of the FTC Act prohibits "unfair or deceptive acts or practices" that affect commerce.  An act or practice may be found to be unfair where it "causes or is likely to cause substantial injury to consumers which is not reasonably avoidable by consumers themselves and not outweighed by countervailing benefits to consumers or to competition."  See, FTC "Unfairness Statement."  The FTC is likely to find an act or practice to be deceptive if "there is a representation, omission or practice that is likely to mislead the consumer acting reasonably in the circumstances, to the consumer's detriment." FTC Deception Statement

The standards for unfairness and deception are independent of each other. While a specific act or practice may be both unfair and deceptive, the FTC may find a violation of Section 5 if the act or practice is either unfair or deceptive.

In the next post, we'll look at what happened and whether Netflix's 2013 change in the way it delivered content to the country's largest ISPs seemed reasonably calculated to efficiently deliver content to customers, or, if not, whether it seemed designed to promote another goal.  

***Relevant Facts***

Here is a brief recitation of the relevant facts for purposes of our discussion.  Unless otherwise cited, the facts are taken from the Declaration of Ken Florance  , Netflix's Vice President of Content Delivery, submitted in support of Netflix's Petition to Deny the Comcast-TWC Merger (FCC Docket No. 14-57).

For most of the history of Netflix's streaming video delivery service, Netflix believes that Comcast has required Netflix's third party vendors to pay an additional fee to cover some (or all) of the cost of Netflix-specific capacity augmentation at interconnection points.  Netflix describes 3 instances between 2009 and 2010 where it believes CDNs needed to purchase additional capacity to alleviate congestion issues. Florance Declaration ¶¶ 29-41.

Netflix acknowledges that the volume of its traffic does increase demand for ISP-bound capacity at its vendors' points of interconnection with Comcast.  Moreover, these costs are incremental and specific to the particular point of exchange between Netflix's Internet transit vendor and the ISP.  Florance Declaration at ¶ 46.  

When its traffic was carried on third party CDN networks, Netflix was aware of the costs being incurred on its behalf, but "in the short term Netflix was insulated from a sudden price increase." Florance Declaration at ¶ 39.  While Netflix was using CDNs, its performance over cable systems seemed uniformly better than even on the most advanced telco systems. 

While its service was good using 3rd party CDNs, Netflix explains that, "[a]fter the Akamai, Limelight, and Level 3 CDN congestion episodes [2009-2010], Netflix began transitioning its traffic from CDNs (all of whom, we believed, were paying Comcast's new terminating access fee) to transit providers in our continued effort to avoid terminating access fees."  Florance Declaration at ¶ 40.  (dates in brackets added).  Thus, in February, 2012, Netflix signed an agreement with Cogent for Internet transit service.  Cogent began transitioning traffic to Netflix in August 2012.  Florance Declaration ¶ 41.

Based on customer complaints about service quality, Netflix's service deteriorated immediately upon switching to Cogent transit and progressively deteriorated over the next year. Florance, at ¶ 51.  However, beginning in October 2013, Netflix reports a very high level of customer dissatisfaction and cancellations, due to "Netflix's inability to do anything to change the situation."  Florance ¶ 52 (emphasis added).    

Continue reading The Netflix/Comcast Dispute: Interconnection "Principles" vs. Consumer Rights?
February 25, 2014 3:04 PM

Netflix, Comcast, and the WWE: Why CDN-ISP Agreements Work

On Sunday, Netflix and Comcast announced that they had reached an agreement to provide better quality traffic delivery for the Netflix customers using Comcast's ISP.  The terms of the agreement have not been disclosed, but many assume that Netflix will be paying Comcast for the additional capacity necessary to ensure better service delivery.

Some have correctly explained that this isn't really news.  See this excellent piece by Dan Rayburn, and this one by Richard Bennett, which is also very good.  Both commenters point out that Netflix was, presumably, paying something to its "CDN" partners, Cogent and Level 3 for the poor performance it was previously receiving, so Netflix was able to improve its position by reaching the direct agreement with Comcast.

Make no mistake, though, it is absolutely normal for content providers (that care about their customers' experience) to pay ISPs for the additional inbound capacity needed to ensure the customer gets good quality service.  However, a few articles (e.g., here, here, and here) suggest that the Netflix-Comcast agreement somehow changes the dynamic of "the Internet."  It does not.

The WWE's Bold Experiment

Yesterday, the WWE launched its online video channel  available through its website.      Most in telecom policy are probably unaware of this launch, because, you know, the content isn't erudite enough.  But, since every article lamenting the Netflix-Comcast announcement also predicts devastating effects on the hypothetical "new entrant" it's worth considering the newest streaming video entrant.

The WWE's online channel is notable for a couple reasons:  1) it's the first time an established, successful provider of live entertainment has offered a subscription service directly to the public that is designed to circumvent subscription TV distribution, and 2) its streaming channel is available in beautiful high definition over practically any/every Internet-connected device.  Everything you would want to know about the new channel is available in this excellent article

How does a new entrant that competes directly with its existing subscription TV partners make sure its customers get a great online experience over those same companies' ISPs?  Well, it certainly helps if the new entrant is a company that offers methyltestosterone  at the coffee machine, just to make the coffee "taste right" to their super macho employees.  

hulkster.jpgBut the real "magic" for the WWE, and tons of other quality video providers, is that they use a company that long ago made quality content delivery their main service--Akamai.  Significantly, Akamai has paid ISPs to place its servers are as close as possible to ISP distribution and they have adequate capacity to the ISP's network access point.  This is why the Olympics, ESPN, the NBA, NHL and MLB also use Akamai for their live, streaming high definition services.  Akamai is a premium CDN (see its customer list).

Why Netflix Traffic Seems to Make News

Netflix's traffic is always in the spotlight, because the company is so successful at acquiring, and creating, video content that customers want--accounting for more than 30% of peak time downstream, fixed line Internet traffic.  But, Akamai also carries (cumulatively) a lot of high definition, premium traffic, and you never hear complaints from the customers of Akamai customers.

house of cards.jpg

If I had to guess, I would say that Netflix has wisely chosen to spend its limited resources on building quality content, rather than video delivery quality. Why do I say this?  Well, recall that Netflix has its own CDN--called Open Connect.  Netflix also publishes the delivery performance of various ISPs that carry its traffic. The ISP that performed the best in January was Google Fiber.  

Google Fiber is directly interconnected with Netflix. Google Fiber sells no service to its customers slower than 1,000 mbps, yet its Netflix throughput was a relatively meager 3.78 mbps.  No other Open Connect ISP partner even achieved 3 mbps.  Thus, all Netflix customers were paying for much higher speeds from their ISPs than they were getting from Netflix.  

So, the Netflix ISP Speed Index tells you more about Netflix's CDN performance than ISPs' Internet access performance.  In fact, GigaOm recently noted that the best Netflix could say about Open Connect was that it "sucked less" during peak hours for ISP partners than others.

When the minimum necessary speed for even low-level high definition video is at least 4 mbps, it's clear that Netflix has a little ways to go in order to provide higher quality online video delivery.  But, direct interconnect agreements with ISPs--like the Comcast deal--position Netflix with much better control over its service quality than it has today.

Why Content Delivery Services Need to Exist

A lot of media coverage (e.g., here) has mistakenly appropriated any story involving Netflix traffic as a proxy for some kind of meta-online video policy issue with reverberating consequences.  But the ISP is not the "troll on the bridge" and the CDN is not a hapless victim.

The ISP is in the business of providing Internet access from the customer premise to the ISP's point of interconnection with the Internet.  If the ISP was responsible for augmenting ingress capacity from a content provider's backbone to the ISP's point of access every time inbound capacity surged based on an application's popularity, then the ISP would have to involuntarily shoulder the risk of an entirely different business model.

For example, 10 years ago My Space was the dominant social networking site.  Today, it's a relative ghost town.  If the ISP's had born the cost of carrying My Space's swelling downstream demand at the time--by building capacity dedicated to carrying My Space traffic from its backbone provider(s) to the ISP point of access--this would now be stranded, wasted capacity.  These costs would have been absorbed by all of the ISP's customers (including the ones that never even used My Space).  

Netflix is a great service, but it isn't the ISP's service. Moreover, it can be quickly abandoned by the ISP's customers if a better substitute comes along.  The ISP shouldn't have to bear the risk of someone else's Internet business model--especially when there are firms like CDNs that are in the business of accepting risks associated with delivering content to the ISP.