March 4, 2015 12:42 PM
In case you didn't notice, the FCC's press release
describing its decision to reclassify broadband Internet access is a little different on the subject of interconnection than Chairman Wheeler's "Fact Sheet
" 3 weeks earlier. The FCC's press release
from last week is substantially similar to the Chairman's fact sheet
, except that it contains no reference to the classification of the "service that broadband providers make available to 'edge providers.'"The Chairman's "Edge Service" Classification Proposal Was Really Unpopular
On the last day that parties could lobby the Commission, (February 19th) Google had meetings with senior advisors for the Chairman and the two Democratic Commissioners. In these meetings, Google persuasively argued that "the Commission should not attempt to classify a "service that broadband providers make available to 'edge providers,'" because "this supposed additional service does not exist." Ex Parte
(Internal quote to Chairman's "fact sheet
.") On the eve of the Commission's vote, the Wall Street Journal reported
that "the FCC tweaked its language to address Google's concern."
Without this report, it would be hard to know who to credit for killing a truly reckless idea. That's because the following 5 parties all made arguments against the Chairman's proposed "new service" on the same day as Google: Akamai
, Free Press
and New America's Open Technology Institute, National Hispanic Media Coalition
, and Cox Communications
When you somehow manage to get Free Press, New America, and the National Hispanic Media Coalition to oppose
an additional Title II regulatory classification--because it makes your original Title II reclassification
look even more legally suspect
--that's when you know you've gone too far. The Importance of Direct Interconnection Agreements
The scariest aspect of the Commission's proposal must have been the FCC's casual willingness to disturb these companies' existing arrangements with ISPs by mandating the future terms on which they and others would be able to obtain interconnection. For companies with little to no regard for the terms of their existing interconnection agreements, like Netflix and its transit vendors, the Commission could not make their situation worse. But, for the leading Internet companies, the Commission must have appeared alarmingly ignorant of/indifferent to the importance of these agreements to Internet traffic delivery.Reason 1: The Disintermediation of Internet Traffic
In 2009, the University of Michigan, Arbor Networks, and Merit Network presented the results of the largest traffic study
since the advent of the commercial Internet. The study showed
that, between 2007 and 2009, Internet traffic delivery changed radically. Over only 2 years, Internet transit (the traditional "intermediary" between ISPs) became dramatically less important as a traffic delivery vehicle.
Instead, major content providers began delivering more and more content directly to the consumer's ISP (either through their own networks or CDNs). Accordingly, since 2009, the "tech giants" have been accelerating their investment
in network assets and data centers to route their high bandwidth traffic directly to efficient delivery points in the ISP's networks. Reason 2: Interconnection Agreements Reflect Valuable Investment
As the Internet has evolved to more efficiently deliver high-bandwidth content to consumers, the largest content providers--including Netflix for the first 4 years of its streaming service--have placed a premium on placing content closer to the customer. Therefore, the largest traffic sources have entered into agreements to directly exchange traffic with their customers' ISPs. When these agreements provide for the settlement-free exchange of traffic, it is because this reflects the mutual benefits
received by both parties.
Since parties to settlement-free "peering" arrangements each provide the
other with valuable network facilities, or other benefits, this value
can be observed by looking at the investment the parties put into their
, capital expenditures). To get an idea of the
importance of those agreements to Internet companies, consider the
increase in capex by the largest Internet companies since 2009.Regulation of Interconnection Terms Could Devalue Previous and Future Investment
As we can see from the chart above the major Internet companies have undertaken a massive amount of capital spending over the past 6 years in order to efficiently deliver content and services to consumers. To be sure, not all
of the Internet companies' capex is driven by traffic delivery interconnection concerns, but the increase in these companies' capex since 2009 correlates with the findings of the University of Michigan, et al., traffic study referenced above. Moreover, news reports have confirmed spending on improved data networking infrastructure as a capex driver. See, e.g.
This capital investment has been made by edge companies and CDNs with the expectation that it will allow these firms to provide a better experience
to their customers than their competitors provide. Indeed, Google notes that it has entered into peering agreements with some of the largest ISPs because
it is "unable to use transit to reach users on those networks with reasonable quality
." Ex parte
at 2 (emphasis added.)
The risk of requiring ISPs to provide interconnection as a separate common carrier service was articulated succinctly by Akamai, which handles
15-30% of the world's Internet traffic. Akamai argued that the FCC must not mandate the terms and conditions of ISP interconnection, because if the ISPs are required to provide access on equal terms to all:
This is not technically feasible and the result could be access for none, which would decrease the performance, scalability, reliability and security of the Internet.
Akamai, February 20th ex parte
at 1 (emphasis added). In other words, Akamai understands and accepts that it "must often compete with others for access to ISP facilities
." Akamai, February 9th ex parte
, at 3 (emphasis added). But, does the FCC accept interconnection as a legitimate element of competition?The Commission Should Not Displace Competition with Regulation
Netflix, Cogent, and Level 3 assert that they cannot get interconnection with the ISPs--on the terms they would prefer
--because of a lack of competition. But, as Akamai explained, companies seeking the most efficient terms of distribution to the ISP's customers are competing with each other
for the best access to these customers. Could it be that competition is the reason
the transit companies aren't getting the terms they want?
Compare the sum of the capex of Netflix's distribution chain over the relevant time period, with their competitors. Is it surprising that Internet transit wants the FCC to "level the playing field?"
Transit is at a disadvantage relative to direct interconnection because the Internet has evolved. For the content distributors sending the most traffic, transit has not been the preferred solution for a long time. But competition, and not a lack thereof, produced this outcome.
Unlike the major focus of the larger net neutrality debate--which is concerned with adopting rules to foreclose future
ISP service offerings--the regulation of interconnection terms and conditions is fraught with risks to existing
service configurations. The FCC must be careful not to use prescriptive regulation of ISP
interconnection terms--in the name of competitive "neutrality"--to
foreclose innovative firms (and their customers) from reaping the
benefits of their own ideas, risks, and investments.
December 18, 2014 4:31 PM
[Note: This is the first post, in a series, in which we'll look at the Netflix/Cogent/Comcast congestion episode from earlier this year. The focus will be on understanding this event from a different perspective than most of us may have thought about it before. This series looks further into the question I raised in my last post which is: are existing laws--adequately enforced--sufficient to protect consumers? For purposes of readability, the full citation of relevant facts has been placed at the end of this post.]
In my last post
, I started to look at whether the protracted congestion--and associated consumer service disruptions--caused by the recent Netflix/Cogent/Comcast interconnection dispute indicated that the traditional voluntary agreement structure of the Internet was broken, or whether existing laws might not be enough to prevent protracted consumer disruption. In a recent article
, Prof. Susan Crawford, an advocate for all things Net Neutrality, also highlights the frustrations of customers caught in the middle of the Netflix/Cogent/Comcast dispute,
No federal, state, or local government exercises any oversight over this handful of interconnection points. No Better Business Bureau watches over how your requests for data are being treated.
Prof. Crawford is right to question why consumers got stuck with the short end of this wholesale dispute. But, I disagree with Prof. Crawford's assumption that new laws--specific to the issue Netflix and Cogent blame for the protracted congestion--are needed.
As noted in the last post
, more specific contracts, and quicker enforcement by wholesale partners are one way to prevent extended periods of consumer frustration. Similarly, there are also existing laws designed to protect consumers from intentional, or knowing, actions of third parties that prevent consumers from receiving services they believe they are purchasing.
The FTC has expressed
concern that it will lose jurisdiction over Net Neutrality-related matters if the FCC decides to reclassify broadband as a Title II service (the FTC Act specifically exempts "common carriers"). Even though I was aware the FTC had asserted
jurisdiction to handle Net Neutrality complaints, I didn't really think about the how important the FTC could be. . . until I started looking really closely at the Netflix/Cogent/Comcast congestion episode from earlier this year. "It's About the Principle"
When Netflix filed its Petition
to Deny the Comcast-TWC merger in August, I was interested in learning the circumstances that led to Netflix's direct interconnection agreement with Comcast. I expected to see a pretty basic recitation of how Comcast kept unreasonably increasing interconnection prices, thereby forcing Netflix into lower quality interconnection arrangements.
Instead, though, the brief spends a lot of time establishing that Netflix's principle
--of not paying the ISP any portion of the costs of delivering its content to its customers--was the exclusive factor it relied upon in choosing vendors to deliver its customers' traffic. The absence of any comparison of various input prices/vendor alternatives available to Netflix seemed odd. The notion that Netflix was defending a not-purely-economic principle seemed odder still.
The antitrust analytical framework (which was the ostensible basis for
Netflix's opposition to the Comcast-TWC merger) recognizes economic
efficiency, and not any unique firm-specific view of how an industry
should work. Yet, Netflix has never indicated that its decisions were
based on immediate cost/price effects. It has even clarified that the
costs being imposed by the ISPs are not significant, nor has it raised
prices for customers served by the offending ISPs. See, e.g.
, this blog
. Level 3 made the same argument in 2010--that it's not the cost, it's the principle. See this Ars Technica article
Prof. Susan Crawford, in the article
mentioned above, also observes:
FCC will find that the money amounts involved in these deals are low at
the moment. It's the naked threat posed to the future that is the
problem. . . .
The "naked threat posed to the future"
may or may not be cause for concern, but--if this threat does not limit
the immediate ability of a firm to deliver service--can a firm's reaction
to such a threat excuse its performance under its customer contracts?
possible that, for many months at least, Comcast customers (and those
of certain other ISPs) were paying for service that Netflix knew to be
substandard. If Netflix failed to take any action to provide the grade
of service for which its customers were paying, or to let prospective
customers know they would be receiving degraded service for an
indeterminate period, then it's possible that enforcement of existing
laws might prevent future consumer abuses. The Federal Trade Commission Act
5 of the FTC Act prohibits "unfair or deceptive acts or practices" that
affect commerce. An act or practice may be found to be unfair where it
"causes or is likely to cause substantial injury to consumers which is
not reasonably avoidable by consumers themselves and not outweighed by
countervailing benefits to consumers or to competition." See, FTC "Unfairness Statement
The FTC is likely to find an act or practice to be deceptive if "there
is a representation, omission or practice that is likely to mislead the
consumer acting reasonably in the circumstances, to the consumer's
detriment." FTC Deception Statement
standards for unfairness and deception are independent of each other.
While a specific act or practice may be both unfair and deceptive, the
FTC may find a violation of Section 5 if the act or practice is either
unfair or deceptive.
In the next post, we'll look at what
happened and whether Netflix's 2013 change in the way it delivered
content to the country's largest ISPs seemed reasonably calculated to
efficiently deliver content to customers, or, if not, whether it seemed
designed to promote another goal. ***Relevant Facts***
is a brief recitation of the relevant facts for purposes of our
discussion. Unless otherwise cited, the facts are taken from the
Declaration of Ken Florance , Netflix's Vice
President of Content Delivery, submitted in support of Netflix's
Petition to Deny
the Comcast-TWC Merger (FCC Docket No. 14-57).
most of the history of Netflix's streaming video delivery service,
Netflix believes that Comcast has required Netflix's third party vendors
to pay an additional fee to cover some (or all) of the cost of
Netflix-specific capacity augmentation at interconnection points.
Netflix describes 3 instances between 2009 and 2010 where it believes
CDNs needed to purchase additional capacity to alleviate congestion
issues. Florance Declaration ¶¶ 29-41.
Netflix acknowledges that
the volume of its traffic does increase demand for ISP-bound capacity at
its vendors' points of interconnection with Comcast. Moreover, these
costs are incremental and specific to the particular point of exchange
between Netflix's Internet transit vendor and the ISP. Florance
Declaration at ¶ 46.
When its traffic was carried on third
party CDN networks, Netflix was aware of the costs being incurred on its
behalf, but "in the short term Netflix was insulated from a sudden
price increase." Florance Declaration at ¶ 39. While Netflix was using
CDNs, its performance over cable systems seemed uniformly better
even on the most advanced telco systems.
its service was good using 3rd party CDNs, Netflix explains that,
"[a]fter the Akamai, Limelight, and Level 3 CDN congestion episodes
[2009-2010], Netflix began transitioning its traffic from CDNs (all of
whom, we believed, were paying Comcast's new terminating access fee) to
transit providers in our continued effort to avoid terminating access
fees." Florance Declaration at ¶ 40. (dates in brackets added). Thus,
in February, 2012, Netflix signed an agreement with Cogent for Internet
transit service. Cogent began transitioning traffic to Netflix in
August 2012. Florance Declaration ¶ 41.
Based on customer
complaints about service quality, Netflix's service deteriorated
immediately upon switching to Cogent transit and progressively
deteriorated over the next year. Florance, at ¶ 51. However, beginning
in October 2013, Netflix reports a very high level of customer
dissatisfaction and cancellations, due to "Netflix's inability
anything to change the situation." Florance ¶ 52 (emphasis added).
Continue reading The Netflix/Comcast Dispute: Interconnection "Principles" vs. Consumer Rights?
February 25, 2014 3:04 PM
On Sunday, Netflix
and Comcast announced
that they had reached an agreement to provide better quality traffic delivery for the Netflix customers using Comcast's ISP. The terms of the agreement have not been disclosed, but many assume
that Netflix will be paying Comcast for the additional capacity necessary to ensure better service delivery.
Some have correctly explained that this isn't really news. See this excellent piece
by Dan Rayburn, and this one
by Richard Bennett, which is also very good. Both commenters point out that Netflix was, presumably, paying something to its "CDN" partners, Cogent
and Level 3
for the poor performance it was previously receiving, so Netflix was able to improve its position by reaching the direct agreement with Comcast.
Make no mistake, though, it is absolutely normal for content providers (that care about their customers' experience) to pay ISPs for the additional inbound capacity needed to ensure the customer gets good quality service. However, a few articles (e.g., here
, and here
) suggest that the Netflix-Comcast agreement somehow changes the dynamic of "the Internet." It does not.The WWE's Bold Experiment
Yesterday, the WWE launched
its online video channel available through its website
. Most in telecom policy are probably unaware of this launch, because, you know, the content isn't erudite enough. But, since every article lamenting the Netflix-Comcast announcement also predicts devastating effects on the hypothetical "new entrant" it's worth considering the newest streaming video entrant.
The WWE's online channel is notable for a couple reasons: 1) it's the first time an established, successful provider of live entertainment has offered a subscription service directly to the public that is designed to circumvent subscription TV distribution, and 2) its streaming channel is available in beautiful high definition over practically any/every Internet-connected device. Everything you would want to know about the new channel is available in this excellent article
How does a new entrant that competes directly with its existing subscription TV partners make sure its customers get a great online experience over those same companies' ISPs? Well, it certainly helps if the new entrant is a company that offers methyltestosterone
at the coffee machine, just to make the coffee "taste right" to their super macho employees.
But the real "magic" for the WWE, and tons of other quality video providers, is that they use a company that long ago made quality content delivery their main service--Akamai
. Significantly, Akamai has paid ISPs to place its servers are as close as possible to ISP distribution and they have adequate capacity to the ISP's network access point. This is why the Olympics, ESPN, the NBA, NHL and MLB also use Akamai for their live, streaming high definition services. Akamai is a premium CDN (see its customer list
).Why Netflix Traffic Seems to Make News
Netflix's traffic is always in the spotlight, because the company is so successful at acquiring, and creating, video content that customers want--accounting for more than 30% of peak time downstream, fixed line Internet traffic
. But, Akamai also carries (cumulatively) a lot of high definition, premium traffic, and you never hear complaints from the customers of Akamai customers.
If I had to guess, I would say that Netflix has wisely chosen to spend its limited resources on building quality content
, rather than video delivery quality. Why do I say this? Well, recall that Netflix has its own CDN--called Open Connect
. Netflix also publishes
the delivery performance of various ISPs that carry its traffic. The ISP that performed the best in January was Google Fiber
Google Fiber is directly interconnected with Netflix. Google Fiber sells no service to its customers slower than 1,000 mbps, yet its Netflix throughput
was a relatively meager 3.78 mbps. No other Open Connect ISP partner even achieved 3 mbps. Thus, all Netflix customers were paying for much higher speeds from their ISPs than they were getting from Netflix.
So, the Netflix ISP Speed Index
tells you more about Netflix's CDN performance than ISPs' Internet access performance. In fact, GigaOm recently noted
that the best Netflix could say about Open Connect was that it "sucked less" during peak hours for ISP partners than others.
When the minimum necessary speed for even low-level high definition video is at least
4 mbps, it's clear that Netflix has a little ways to go in order to provide higher quality online video delivery. But, direct interconnect agreements with ISPs--like the Comcast deal--position Netflix with much better control over its service quality than it has today. Why Content Delivery Services Need to Exist
A lot of media coverage (e.g
) has mistakenly appropriated any story involving Netflix traffic as a proxy for some kind of meta-online video policy issue with reverberating consequences. But the ISP is not the "troll on the bridge
" and the CDN is not a hapless victim.
The ISP is in the business of providing Internet access from the customer premise to the ISP's point of interconnection with the Internet. If the ISP was responsible for augmenting ingress capacity
from a content provider's backbone to the ISP's point of access every time inbound capacity surged based on an application's popularity, then the ISP would have to involuntarily shoulder the risk of an entirely different business model.
For example, 10 years ago My Space
was the dominant social networking site. Today, it's a relative ghost town. If the ISP's had born the cost of carrying My Space's swelling downstream demand at the time--by building capacity dedicated to carrying My Space traffic from its backbone provider(s) to the ISP point of access--this would now be stranded, wasted capacity. These costs would have been absorbed by all
of the ISP's customers (including the ones that never even used My Space).
Netflix is a great service, but it isn't the ISP's service. Moreover, it can be quickly abandoned by the ISP's customers if a better substitute comes along. The ISP shouldn't have to bear the risk of someone else's Internet business model--especially when there are firms like CDNs that are
in the business of accepting risks associated with delivering content to the ISP.