Results tagged “peering dispute”

May 29, 2015 2:20 PM

The Historical Evidence Supporting the ISP Discrimination Theory

The primary justification for "strong" net neutrality rules is always that there is insufficient competition in the infrastructure services market.  The theory that insufficient competition in an Internet access market enhances the ISP's incentives/ability to discriminate against "off-net" traffic sources (content, applications, or interconnection facilities of smaller rivals) is a foundational premise of the Commission's recent Net Neutrality/Broadband Reclassification Order

Today, we're going to look at the origins of this theory--in competition law--and whether there is any historical evidence to validate this concern.  This may seem risky, because if historical evidence isn't conclusive, it can be claimed as "support" for a lot of different theories. 

giorgio-wheeler meme3.png         
For example, "ancient astronaut theorists" (as the History Channel calls the UFO crowd) point to surprisingly-advanced ancient wonders throughout the world (e.g., pyramids, or other very large inland structures) as "evidence" that aliens were responsible for these accomplishments.  While some [non-History-Channel-viewers] will be inclined to dismiss ancient astronaut theories, these theories cannot be disproved, either.  On the other hand, if the historical evidence is conclusive, it can rule out inconsistent theories.  Is "ISP market share=interconnection degradation" theory at least as viable as ancient astronaut theories?

WorldCom "Will . . . Become the Internet"

The quote above was from an antitrust complaint, filed by one of WorldCom's smaller rivals in the Internet backbone market, in 1998.  GTE Complaint at ¶2.a.  (emphasis in original)  The smaller rival was no beginner, it was GTE--the country's largest integrated local/long-distance telephone company (at a time when virtually all Internet traffic was carried on telephone company facilities). 

Moreover, GTE wasn't crazy to be concerned.  If you had a crystal ball, you could not have positioned yourself better than WorldCom did through a series of acquisitions from '94-'96, which--you may recall--was when the commercial Internet took shape.  By the end of 1997, WorldCom was the largest provider of Internet backbone capacity in the country. 

So, when WorldCom sought to acquire the second largest Internet backbone--MCI--in November 1997, the DoJ was concerned that the post-merger firm would control between 40-70% of the Internet backbone market.  In early 1998, the Economic Policy Institute estimated that 62% of the entire Internet's revenue would need to traverse the MCI WorldCom backbone, and 50% of all ISPs would be dependent on access to the firm's backbone network. 

A Flawed Theory Takes Root

GTE believed so strongly that the MCI-WorldCom merger threatened its ability to compete in both the Internet backbone, and long-distance, markets that it filed its own private antitrust suit to enjoin the MCI-WorldCom merger in May 1998.  In GTE's Complaint, the company explains its concerns:

all of the major backbones . . . are . . . dependent upon each other for interconnection. They thus find it in their independent interests to cooperate to maintain and upgrade the capacity of interconnection among their networks in order to offer their customers ubiquitous, high-quality access to the whole Internet. . . .  By concentrating . . . the two largest Internet backbone networks to create one dominant national network, the merger will give MCI-WorldCom a stranglehold over the burgeoning Internet and the incentive and ability to stifle competition from all other rival Internet backbone operators, including GTE.
GTE Complaint, at ¶2.a.  Interested readers should look at GTE's full Complaint, as (I promise) you'll find it all sounds very familiar. 

GTE's case never made it to trial, because the facts of the Complaint would soon become less clear.  In July of 1998, MCI agreed to spin off its own Internet backbone to Cable and Wireless; after which the DoJ cleared WorldCom's pending acquisition of MCI with a press release.  The FCC issued its Order clearing the transaction soon after the DoJ's Press Release. 

"So Much for Grand Efforts by Regulators to Dictate Outcomes"

MCI's divestiture of its Internet backbone didn't work out as the DoJ and FCC had hoped.  There were soon allegations that MCI had "pulled a fast one," and had not provided C&W much more than the physical assets of its Internet backbone business.  C&W sued MCI WorldCom in March of 1999. 

Undeterred, WorldCom announced it was buying Sprint for $115B in October of the same year.  In November of 1999, C&W aired its grievances at a Congressional Hearing on the Sprint/WCOM merger.  A little over a month later, Carleen Hawn wrote an incredibly insightful short piece for Forbes, entitled "Swimming with Sharks."   

In her article, Ms. Hawn recounts C&W's complaints, but notes significant industry disagreement about which side was to blame.  According to Jim Crowe, founder of Level 3, the divestiture would never have worked as expected by DoJ, because C&W lacked a domestic U.S. network with which to interconnect the MCI backbone network.  C&W would, therefore, have had to purchase domestic Internet transit capacity.  (Note: this would have been right before Internet transit prices began a swift, steep, and inexorable, period of decline.  See, Dr. Peering table of Internet transit prices.)  

Thus, post-merger MCI WorldCom's market position was unaffected by the divestiture, as its share of the backbone market continued to grow faster than rivals.  In January 2000, Hawn states, "MCI WorldCom is simply more dominant than ever," concluding, "so much for grand efforts by regulators to dictate outcomes."  

A Dominant Gatekeeper . . . Isn't Abusive?

But, if the MCI backbone divestiture did nothing to diminish MCI WorldCom's dominance in the Internet backbone market, what was the outcome?  If GTE's theory was correct, MCI WorldCom would now have "both the incentive and the ability to act opportunistically to degrade the quality of interconnection and increase costs for its rivals."  GTE Complaint ¶23. 

Apparently, though, even with 60% of the revenue-generating traffic on the Internet being dependent on its network, MCI WorldCom never found it profitable to act on its incentives/ability to degrade rivals' interconnection terms.  Indeed, there's no evidence they--or any other ISP (either with respect to access or backbone transit)--has ever acted on such an incentive.  But it would take more than a decade for a court to make this finding.  

The Theory Goes to Court

There are two lasting legacies of GTE's theory of dominant ISP discrimination: 1) the general justification for the FCC's current Net Neutrality/Broadband Reclassification Order, and 2) Cogent's peering strategy.  Beginning with its very first ISP peering dispute, with AOL in 2002, Cogent has been the torch-bearer for GTE's never-tested (at the time) theory.  In fact, Cogent has wheeled out this argument every time it's been de-peered (which is a lot--partial list here), and most of the time it files something new at the FCC, or speaks to the press.

So, it was only a matter of time before Cogent tried to really test the GTE theory in an adjudicatory proceeding.  In 2011, Cogent filed a complaint with the French competition authority, alleging, among other things, that France Telecom's peering ratio (of 2.5X) constituted anticompetitive discrimination, as was France Telecom's refusal to unilaterally add capacity in violation of the firm's prior peering agreement.  Customers of France Telecom experienced congestion when downloading content from Cogent client, Megaupload

In many ways the French competition authority was the most sympathetic forum for this claim.  European competition law--with respect to dominant firm behavior--gives much more weight to preserving opportunities for smaller, and mid-size, firms (U.S. antitrust laws tend to focus primarily on economic efficiency).  See here at pp. 5-6/16.  French competition law is more favorable (than E.U. law) to smaller firms versus dominant rivals, and includes the notion of "abuse of a situation of economic dependence."     

But, Cogent could not make the theory overcome its facts; real-life ISPs simply don't discriminate as predicted.  The French competition authority ruled against Cogent.  See summary here.  Cogent appealed the decision, and a French appellate court affirmed, noting that peering was not an "essential facility," and is in no way functionally different from, or inferior to, transit.  See summary here.    

It Costs More $$ to Believe a Theory than Our Own Lyin' Eyes

In May of 1998, the commercial Internet had only been around for a few years.  At the time, GTE's concern that a "dominant" firm would abuse its position to degrade access to, and raise prices of, interconnection facilities needed by its smaller rivals was not an unreasonable concern. 

However, GTE's concern was immediately put to the test in actual market conditions, and it proved inaccurate.  More recently, as noted, Cogent failed to convince a regulator, under the most lenient standards, that it was harmed by the dominant firm's observance of international peering standards.  A French appellate court agreed, and confirmed this finding.

giorgio_my meme2.png
Yet the "dominant ISPs will harm consumers/competitors" theory--unlike the ancient astronaut theories--continues to pick up traction from sources that should be more skeptical.  But, unlike the ancient astronaut theories, continued belief in the GTE theory is not harmless.  Last week, in a policy memo, Hal Singer of the Progressive Policy Institute describes some of the many costs--in terms of services and innovations foreclosed, and investments forgone--of continuing to believe in the possibility of a theory that our "own lyin' eyes" have never seen happen.


December 4, 2014 3:05 PM

Internet Interconnection: Bad Faith Is No Basis for Good Policy

A few weeks ago, President Obama, acting on some seriously bad advice, formally urged   the FCC to, among other things, consider regulating Internet interconnection agreements.  The "facts" that brought an ordinarily well-functioning market, based on two decades of voluntary agreements, into the President's regulatory cross-hairs were, of course, the highly-publicized disputes surfacing earlier this year involving Netflix, Cogent (one of Netflix's primary Internet transit vendors), and Comcast (at first, and then a series of other large ISPs).  

The only thing that is clear at this point is that there is a lot more information for the FCC to gather, especially from Netflix and Cogent.  The information that is available strongly indicates that the Comcast episode (and each subsequent ISP-specific iteration) was an anomaly, and not likely to repeat itself.  This, alone, should tell us to be wary of rushing to supplant a competitive market with regulation.

Moreover, because of the unique nature of this congestion event--and the fact that such an event had not happened before--the FCC must try to understand everything it can about this event before the Commission even thinks about adopting new rules.  Comprehensive rules are only the answer if the problem is that market participants have no ability/incentive to reach mutually-beneficial voluntary agreements.

Yet, in the present case, the parties were able to reach voluntary agreements; Netflix with Cogent, and Cogent with Comcast.  Therefore, before the Commission concludes that carrier-to-carrier agreements cannot work, it must ask: why didn't the voluntary interconnection agreements produce a timely, efficient outcome in the present instance?    

The Relevant Cogent-Comcast Congestion Facts

For our purposes, we only need to focus on a limited set of facts.  We'll take our facts exactly as presented by Netflix and Cogent (in their bid to obtain regulatory concessions in the FCC's review of the Comcast/TWC merger).  Specifically, we will refer to the Declaration of Ken Florance, Netflix's Vice President of Content Delivery, and the Declaration of Henry Kilmer, Cogent's Vice President of IP Engineering.

--In February, 2012, Netflix signed an agreement with Cogent for Internet transit service, which it would use to deliver traffic coming off CDN agreements later that year.  Cogent began transitioning traffic to Netflix in August 2012.  Florance Declaration ¶ 41.

--Cogent does not provide specific information about its settlement-free agreement with Comcast, but we can discern: 1) the agreement applies to traffic falling within a certain inbound/outbound ratio, 2) the agreement has been in place since sometime in 2008; and 3) for the first 5 years of the agreement, the parties were able to abide by the mutually-agreed-upon terms without issue.  Kilmer Declaration ¶¶ 17, 55, 61-64 and the attached Letter from Arthur Block, General Counsel, Comcast Corp. to Robert Beury, Chief Legal Officer, Cogent, dated June 20, 2013 ("Block Letter").

--Cogent also points out that: 1) it does not believe Comcast is its "peer" and that Cogent only agreed to exchange traffic with Comcast on a settlement-free basis because of Comcast's "market power," and 2) Cogent does not believe there is any reasonable basis for "in/out ratio," which defines the range of traffic volumes subject to exchange on settlement-free terms. Kilmer ¶¶ 42-45, and ¶¶ 55-60.

--According to the Block Letter, Comcast states that, in a capacity planning meeting in the fall of 2012, Cogent told Comcast it did not anticipate needing additional capacity in 2013.  Kilmer at pp. 17-18 of 18.

--In a recent ex parte letter, Cogent only disputes that it affirmatively represented that it would not need additional capacity in 2013. Here at 3.  Cogent does not dispute that it failed to provide any advance notice to Comcast that it anticipated needing additional capacity.   

Good Faith and Bad Faith in the Performance of Contracts

In contract law, there is a general presumption that parties to an agreement will perform their duties fairly and honestly, so as not to deprive the other party of the benefits of their bargain.  This presumption is a part of every contract, and is called the implied covenant of good faith and fair dealing. 

The converse of the implied covenant of good faith is, of course, bad faith.  Bad faith, however, goes beyond simply failing to perform a substantive provision in a contract.  Rather, it is defined as an "intentional dishonest act . . . misleading another, entering into an agreement without the intention or means to fulfill it, or violating basic standards of honesty in dealing with others."   

Defining bad faith in novel circumstances can be difficult, but Professor Stephen Burton, in a Harvard Law Review article in 1980, observes that parties frequently relinquish "future opportunities" to enter into contracts, and these same parties also have some discretion as to how they perform the contract.  Therefore, Professor Burton explains, "[b]ad faith performance occurs precisely when discretion is used to recapture opportunities foregone upon contracting." This test has become a widely-employed benchmark for determining bad faith by state courts. (The Burton article is not available online, but here is a great article by Prof. Robert Summers discussing the Burton test and Good Faith generally).

Did Cogent Exercise Bad Faith By Intentionally Disregarding the Terms of Its Settlement-Free Interconnection Agreement with Comcast?

As an experienced provider of Internet transit services, Cogent would have known how much Netflix traffic it could carry and still be within the terms of its settlement-free interconnection agreement with Comcast.  Instead of limiting the amount of traffic it would accept from Netflix, Cogent went ahead and agreed to accept as much as Netflix wanted to send.  Considering, as well, Cogent's expressly-stated contempt for the traffic ratio (which limited Cogent's future opportunities), it is impossible not to construe Cogent's willful disregard of the traffic ratio as an attempt to "recapture opportunities forgone upon contracting."  

While Cogent tries to insist that Comcast was being unreasonable by asking Cogent to observe the terms of the parties' agreement, the Delaware Supreme Court, not long ago, affirmed that "[a] party does not act in bad faith by relying on contract provisions for which that party bargained, where doing so simply limits advantages to another party." Here, n. 26.  The opinion of the Delaware Supreme Court is relevant because many firms, including Netflix, designate Delaware in contracts designating a choice of law.

Fool Comcast Once . . .

It seems obvious, in retrospect, that Comcast could not anticipate--and was not willing, or prepared, to deal with--Cogent's level of bad faith performance.  It is clear from Comcast's response to Cogent's escalation letter, in June 2013, that Comcast has no intention of treating Cogent's persistent disregard of a crucial term as a "total breach."  Comcast asks only that Cogent purchase transit for that amount of traffic which exceeds the parties agreed-upon ratio.

But, when Cogent refused Comcast's option for preserving the original agreement, while accommodating Cogent's demand for greater capacity, Comcast would have been within its rights to give Cogent notice of its intent to terminate direct interconnection with Cogent.  Because, if Comcast's customers were hitting The Pirate Bay a little too hard (demanding more Cogent-bound capacity), that's what Cogent would have done.  

In 2008, Cogent apparently decided that its settlement-free interconnection agreement with European ISP TeliaSonera had become unappealingly one-sided.  Cogent (probably?) provided whatever notice its agreement with Telia required, and then--fairly suddenly (according to reports)--Cogent simply stopped carrying Telia's traffic. 

In hindsight, Comcast would have best served its customers by simply terminating the agreement.  Though, this course of action would have led to a temporary disruption in service--as Cogent's customers sought other alternatives--it would not have led to the protracted degradation in service that consumers instead had to suffer.  

Nonetheless, the existence of this event will make the system of voluntary network interconnection that comprises the Internet less vulnerable to a future bad faith breach in a critical portion of the supply chain.  Parties to future voluntary interconnection agreements are now much more likely to craft agreements so as to insure against protracted periods of deteriorated service.  A few isolated instances of bad faith should not cause the FCC to abandon its faith in the fundamental structure of the Internet as we know it.  

March 5, 2014 2:50 PM

CDN-ISP Agreements Fuel Content Delivery Competition

Last week began with news that Netflix had decided to improve its long-languishing service for customers of Comcast's ISP by directly interconnecting with Comcast, cutting out Internet transit provider Cogent Communications.  The Cogent-Comcast "peering" dispute had been a long-standing topic of media speculation.

Prior to the announcement of the direct interconnection agreement, many media sources had wished to present this dispute as just another iteration of ISPs attempting to "extort" money from sources of competing content.  Netflix's decision to solve the problem itself had pretty much eliminated the "poor Cogent" articles by the end of the week--after Dan Rayburn's detailed explanation of the deal.

Knowing what we have learned in the past week, it would seem that Netflix's "ISP Speed Index" is more likely a fairer representation of the performance of Netflix's CDN versus that of its third party transit providers.  The Netflix ISP Speed Index is obviously not attempting to measure "true" ISP speeds, because even gigabit provider Google Fiber hovered around 3.5 Mbps for most of last year.

The ISPs to which Netflix connects with its own CDN generally see speeds in the 2.3 Mbps-2.9 Mbps range.  For the customers of ISPs served by a third party transit provider, speeds are often much lower.  But how bad is too bad? 

The "Mendoza Line" for Poor Streaming Transit Performance  

Within the past day, Verizon's CEO has said that their FiOS ISP is close to a direct interconnection agreement with Netflix.  Recall, that Cogent also had issues with refusing to augment inbound capacity to Verizon.   

In fairness to Cogent, it does not claim to offer CDN service.  Still, you have to wonder, where is Netflix likely to switch away from "plain old transit" service next? 

Luckily, Netflix kind of tells us (and its transit/CDN vendors) the levels of service deterioration at which they can expect the "Never mind, I'll do it my damn self" call from Reid Hastings.  Here is the performance of several ISPs who offer broadband access at speeds well above 5 Mbps for the last 4 months, as reported by Netflix's ISP Speed Index.
Netflix CDN Chart-2.jpg

I looked at the Netflix ISP Speed Index for the last 12 months, and, traffic delivery at anywhere between 2-3 Mbps seems to be acceptable.  If you're reading about customer service complaints, it's safe to assume that speeds have dropped below 2 Mbps for the ISP in question. 

For both Comcast and Verizon customers, Netflix's transit/CDN vendor(s) performed well for most of the year--only deteriorating significantly over the last several months.  Like in baseball, though, there is a Mendoza Line for competent transit service, and it seems to be around 2 Mbps.  So . . . if you're a Mediacom customer, you probably don't have much longer to wait.

Netflix's traffic is certainly growing very quickly.  But, asymmetric, end-user-destined video traffic is growing very quickly for all ISPs, everywhere.  Sandvine's second half 2013 traffic report shows that for the largest Internet consumers (North America and Asia) peak hour video consumption may be as high as 70% for the average North American user, and 75% for the average Asian consumer (I got that by adding the "real time entertainment" and "P2P" categories for both continents).  You'd think this would be a good thing for a content delivery network, no?

Is "Peering" Net Neutrality Redux?

Level 3 explains in this recent ex parte, that all is not rosy in the world of the CDN--despite Level 3's own recognized success in the market--because of the nettlesome costs that some ISPs believe should be paid by the party getting paid to carry the traffic to the ISP (namely, Level 3).  BTW, this is completely the same issue as the Netflix carrier-v.-every-ISP-in-the-graph issue.

But, Level 3 characterizes what some would call the costs of providing CDN service, as "tolls" that unfairly target large CDNs distributing disproportionately-downstream, bandwidth-intensive traffic.  Level 3 tells the FCC, "ISP tolls that facially apply equally to all traffic are effectively tolls on the most bandwidth-intensive services - video services that compete with the ISP's own video services." (ex parte at 8/8).  

What types of Level 3 traffic are ISPs targeting?  Well, sometimes it looks like this:

IMAG0015.jpg

You see, HBO Go is an online service that HBO (not the subscription TV provider) offers its customers that already buy HBO's premium subscription TV channel.  Since HBO, like almost every any other provider of streaming video service, lacks the demand to justify the hassle/cost of direct interconnection with the ISP, it uses the highly-competitive CDN market to deliver its service. 

Level 3 is one of HBO Go's CDN partners, for certain Apple devices.  As Level 3 notes, the end-user is often buying a TV service from their ISP.  But, poor quality delivery from the TV channel's online content devalues the TV product the ISP is selling.  Thus, it isn't clear why the ISP has an incentive to degrade Level 3's traffic, when the ISP knows that some significant portion of Level 3's traffic is a complement to the ISP's subscription TV service.  

The CDN business is focused on how to efficiently deliver asymmetric, bandwidth-intensive traffic to the customers of the ISP.  If the ISP had to pay for/build incoming capacity from every CDN to the ISP, then it's good for the CDN, but the ISP can only limit its costs if they CDN traffic stays put.
 
So, why would Level 3 want the FCC to impose regulations that give an ISP any kind of interest in which CDN provider a content owner would choose?  Maybe, because sometimes those CDN customers decide to build their own CDN, as soon-to-be-former Level 3 customer Apple is reportedly doing.  New CDNs equal new competition for small content owners, and that's not always a great thing . . . for CDN incumbents.  Now I get it.