Results tagged “Comcast-Netflix”

December 23, 2014 1:07 PM

The Netflix/Comcast Dispute Pt. 3: Did Netflix Mislead Its Customers?

We've, so far--in parts 1 and 2 of this series--looked at some of the justifications Netflix provides for providing inferior service to its customers served by Comcast for months on end, beginning at some time in 2013.  See, e.g., this Netflix ex parte letter, at p. 2. also, slides 33/34 of this ex parte presentation.   We could finish up by taking a look at each side's claim that the other was the real "cause" of the congestion--but that would miss the point of whether interconnection rules, or the lack thereof, were responsible for consumer service disruptions in this case.

Absent any truly shocking new information coming to light (such as learning that Netflix's contract with Cogent was intended to harm consumers, for example, by expressly forbidding Cogent from purchasing supplemental capacity to relieve congestion for its retail customers), it doesn't matter which party is "right" as a matter of Internet policy.  Rather, the immediate question is whether consumer disruption was the unavoidable consequence of this "policy dispute" (though it was unclear whether the dispute between Netflix, Cogent, and Comcast was a "policy," or ordinary business, dispute when it was settled).  

Netflix Knew Congestion Would Impair It's Service

In its Terms of Use for its customers, Netflix designates that Delaware law control any dispute with its customers.  See, Netflix Terms of Use, para. 11.b.  As was mentioned in the blog  dealing with the Cogent-Comcast interconnection agreement, Delaware law holds that "[a] party does not act in bad faith by relying on contract provisions for which that party bargained, where doing so simply limits advantages to another party."  

Thus, Netflix should/would have anticipated that Comcast would "rely[] on contract provisions for which [it] bargained."  Given that Comcast's actions were entirely predictable (and not in bad faith), Netflix would also have known that adherence to its "principle" (of not paying the ISP for transit) would lead to service-affecting levels of congestion for its customers.  So, if Netflix knew that the combination of its planned course of action and Comcast's predictable reaction would cause its customers to receive congestion-degraded service, did Netflix have any obligation to its customers under Section 5 of the FTC Act?

Was Netflix's Failure to Disclose Congestion an Unfair or Deceptive Practice?

Section 5 of the FTC Act prohibits "unfair or deceptive acts or practices" that affect commerce.  An act or practice may be found to be unfair where it "causes or is likely to cause substantial injury to consumers which is not reasonably avoidable by consumers themselves and not outweighed by countervailing benefits to consumers or to competition."  See, FTC Unfairness Statement.   The FTC is likely to find an act or practice to be deceptive if "there is a representation, omission or practice that is likely to mislead the consumer acting reasonably in the circumstances, to the consumer's detriment." FTC Deception Statement

It seems fairly clear that Netflix's provision of compromised service, without disclosure, was an "unfair practice" in the sense that consumers could not have anticipated, or avoided injury.  Moreover, it would be hard to speculate that Netflix's actions--of not notifying consumers to expect degraded service--had some countervailing consumer benefit.  Thus, let's look at whether Netflix also engaged in a deceptive act or practice.  

Netflix Knew It Was Selling Inferior Service.  According to Netflix, for the entire history of its streaming service, someone (a 3d party CDN) has always paid the ISP for the incremental capacity necessary to ensure its traffic was delivered without congestion. Florance Declaration  ¶¶ 29-41.  Furthermore, as even Comcast points out, Netflix's decision to artificially limit its transit vendors--based on Netflix's "principle" of not paying for ISP capacity or paying anyone that does--meant that these vendors' capacity between their networks and Comcast were bound to become overwhelmed, resulting in congestion.  See, e.g., Declaration of Kevin McElearney, Comcast, at ¶¶ 23-25  

Netflix has established that its Comcast customers received a "substandard" grade of service even though the consumer was paying for, and expected, the "standard" grade of service.  Florance Declaration ¶¶ 51-55.  Nonetheless, Netflix kept selling service to Comcast customers, knowing that--as long as its transit vendors' lacked sufficient throughput capacity at points of interconnection with Comcast--each incremental customer would contribute to the service degradation experienced by all Netflix/Comcast customers.  

Netflix Deceptively Failed to Disclose Its Comcast-Specific Service Level.  The FTC, in its 1983 Deception Statement, states that "the practice of offering a product for sale creates an implied representation that it is fit for the purposes for which it is sold." Deception Stmt., n.4.   The FTC has further explained that,

Where the seller knew, or should have known, that an ordinary consumer would need omitted information to evaluate the product or service . . . materiality will be presumed.
Id. Under the circumstances, and without knowing the extent, length, or degree of degradation to expect, it was impossible for an "ordinary" Comcast broadband consumer to evaluate prospective Netflix service.  Thus, Netflix's knowledge of, and omission of, relevant information about Netflix's Comcast-specific congestion service impairment was "material" and, therefore, deceptive.  

Consumer Protections Can't Be Ignored and Be the Basis for Interconnection Rules

Netflix tells the FCC that, "when Netflix's traffic was congested it did everything in its power--short of paying Comcast an access fee--to alleviate the congestion . . . ."  Netflix Petition to Deny at 62. (emphasis added).  The notion that Netflix "did everything in its power"--short of doing the one thing it knew would resolve its service disruptions, and what it ultimately did do--is fallacious, and simply another way of stating that Netflix did just what it intended to do.  

If the FCC believes that consumers will benefit from interconnection rules, it should adopt rules after careful consideration.  But the Commission should not to deceive itself into thinking that "consumer welfare" is served by preemptively granting concessions to prevent behavior that is otherwise flatly proscribed by existing consumer protection laws.

March 5, 2014 2:50 PM

CDN-ISP Agreements Fuel Content Delivery Competition

Last week began with news that Netflix had decided to improve its long-languishing service for customers of Comcast's ISP by directly interconnecting with Comcast, cutting out Internet transit provider Cogent Communications.  The Cogent-Comcast "peering" dispute had been a long-standing topic of media speculation.

Prior to the announcement of the direct interconnection agreement, many media sources had wished to present this dispute as just another iteration of ISPs attempting to "extort" money from sources of competing content.  Netflix's decision to solve the problem itself had pretty much eliminated the "poor Cogent" articles by the end of the week--after Dan Rayburn's detailed explanation of the deal.

Knowing what we have learned in the past week, it would seem that Netflix's "ISP Speed Index" is more likely a fairer representation of the performance of Netflix's CDN versus that of its third party transit providers.  The Netflix ISP Speed Index is obviously not attempting to measure "true" ISP speeds, because even gigabit provider Google Fiber hovered around 3.5 Mbps for most of last year.

The ISPs to which Netflix connects with its own CDN generally see speeds in the 2.3 Mbps-2.9 Mbps range.  For the customers of ISPs served by a third party transit provider, speeds are often much lower.  But how bad is too bad? 

The "Mendoza Line" for Poor Streaming Transit Performance  

Within the past day, Verizon's CEO has said that their FiOS ISP is close to a direct interconnection agreement with Netflix.  Recall, that Cogent also had issues with refusing to augment inbound capacity to Verizon.   

In fairness to Cogent, it does not claim to offer CDN service.  Still, you have to wonder, where is Netflix likely to switch away from "plain old transit" service next? 

Luckily, Netflix kind of tells us (and its transit/CDN vendors) the levels of service deterioration at which they can expect the "Never mind, I'll do it my damn self" call from Reid Hastings.  Here is the performance of several ISPs who offer broadband access at speeds well above 5 Mbps for the last 4 months, as reported by Netflix's ISP Speed Index.
Netflix CDN Chart-2.jpg

I looked at the Netflix ISP Speed Index for the last 12 months, and, traffic delivery at anywhere between 2-3 Mbps seems to be acceptable.  If you're reading about customer service complaints, it's safe to assume that speeds have dropped below 2 Mbps for the ISP in question. 

For both Comcast and Verizon customers, Netflix's transit/CDN vendor(s) performed well for most of the year--only deteriorating significantly over the last several months.  Like in baseball, though, there is a Mendoza Line for competent transit service, and it seems to be around 2 Mbps.  So . . . if you're a Mediacom customer, you probably don't have much longer to wait.

Netflix's traffic is certainly growing very quickly.  But, asymmetric, end-user-destined video traffic is growing very quickly for all ISPs, everywhere.  Sandvine's second half 2013 traffic report shows that for the largest Internet consumers (North America and Asia) peak hour video consumption may be as high as 70% for the average North American user, and 75% for the average Asian consumer (I got that by adding the "real time entertainment" and "P2P" categories for both continents).  You'd think this would be a good thing for a content delivery network, no?

Is "Peering" Net Neutrality Redux?

Level 3 explains in this recent ex parte, that all is not rosy in the world of the CDN--despite Level 3's own recognized success in the market--because of the nettlesome costs that some ISPs believe should be paid by the party getting paid to carry the traffic to the ISP (namely, Level 3).  BTW, this is completely the same issue as the Netflix carrier-v.-every-ISP-in-the-graph issue.

But, Level 3 characterizes what some would call the costs of providing CDN service, as "tolls" that unfairly target large CDNs distributing disproportionately-downstream, bandwidth-intensive traffic.  Level 3 tells the FCC, "ISP tolls that facially apply equally to all traffic are effectively tolls on the most bandwidth-intensive services - video services that compete with the ISP's own video services." (ex parte at 8/8).  

What types of Level 3 traffic are ISPs targeting?  Well, sometimes it looks like this:

IMAG0015.jpg

You see, HBO Go is an online service that HBO (not the subscription TV provider) offers its customers that already buy HBO's premium subscription TV channel.  Since HBO, like almost every any other provider of streaming video service, lacks the demand to justify the hassle/cost of direct interconnection with the ISP, it uses the highly-competitive CDN market to deliver its service. 

Level 3 is one of HBO Go's CDN partners, for certain Apple devices.  As Level 3 notes, the end-user is often buying a TV service from their ISP.  But, poor quality delivery from the TV channel's online content devalues the TV product the ISP is selling.  Thus, it isn't clear why the ISP has an incentive to degrade Level 3's traffic, when the ISP knows that some significant portion of Level 3's traffic is a complement to the ISP's subscription TV service.  

The CDN business is focused on how to efficiently deliver asymmetric, bandwidth-intensive traffic to the customers of the ISP.  If the ISP had to pay for/build incoming capacity from every CDN to the ISP, then it's good for the CDN, but the ISP can only limit its costs if they CDN traffic stays put.
 
So, why would Level 3 want the FCC to impose regulations that give an ISP any kind of interest in which CDN provider a content owner would choose?  Maybe, because sometimes those CDN customers decide to build their own CDN, as soon-to-be-former Level 3 customer Apple is reportedly doing.  New CDNs equal new competition for small content owners, and that's not always a great thing . . . for CDN incumbents.  Now I get it.




February 25, 2014 3:04 PM

Netflix, Comcast, and the WWE: Why CDN-ISP Agreements Work

On Sunday, Netflix and Comcast announced that they had reached an agreement to provide better quality traffic delivery for the Netflix customers using Comcast's ISP.  The terms of the agreement have not been disclosed, but many assume that Netflix will be paying Comcast for the additional capacity necessary to ensure better service delivery.

Some have correctly explained that this isn't really news.  See this excellent piece by Dan Rayburn, and this one by Richard Bennett, which is also very good.  Both commenters point out that Netflix was, presumably, paying something to its "CDN" partners, Cogent and Level 3 for the poor performance it was previously receiving, so Netflix was able to improve its position by reaching the direct agreement with Comcast.

Make no mistake, though, it is absolutely normal for content providers (that care about their customers' experience) to pay ISPs for the additional inbound capacity needed to ensure the customer gets good quality service.  However, a few articles (e.g., here, here, and here) suggest that the Netflix-Comcast agreement somehow changes the dynamic of "the Internet."  It does not.

The WWE's Bold Experiment

Yesterday, the WWE launched its online video channel  available through its website.      Most in telecom policy are probably unaware of this launch, because, you know, the content isn't erudite enough.  But, since every article lamenting the Netflix-Comcast announcement also predicts devastating effects on the hypothetical "new entrant" it's worth considering the newest streaming video entrant.

The WWE's online channel is notable for a couple reasons:  1) it's the first time an established, successful provider of live entertainment has offered a subscription service directly to the public that is designed to circumvent subscription TV distribution, and 2) its streaming channel is available in beautiful high definition over practically any/every Internet-connected device.  Everything you would want to know about the new channel is available in this excellent article

How does a new entrant that competes directly with its existing subscription TV partners make sure its customers get a great online experience over those same companies' ISPs?  Well, it certainly helps if the new entrant is a company that offers methyltestosterone  at the coffee machine, just to make the coffee "taste right" to their super macho employees.  

hulkster.jpgBut the real "magic" for the WWE, and tons of other quality video providers, is that they use a company that long ago made quality content delivery their main service--Akamai.  Significantly, Akamai has paid ISPs to place its servers are as close as possible to ISP distribution and they have adequate capacity to the ISP's network access point.  This is why the Olympics, ESPN, the NBA, NHL and MLB also use Akamai for their live, streaming high definition services.  Akamai is a premium CDN (see its customer list).
akamai_logo_color.jpg

Why Netflix Traffic Seems to Make News

Netflix's traffic is always in the spotlight, because the company is so successful at acquiring, and creating, video content that customers want--accounting for more than 30% of peak time downstream, fixed line Internet traffic.  But, Akamai also carries (cumulatively) a lot of high definition, premium traffic, and you never hear complaints from the customers of Akamai customers.

house of cards.jpg


If I had to guess, I would say that Netflix has wisely chosen to spend its limited resources on building quality content, rather than video delivery quality. Why do I say this?  Well, recall that Netflix has its own CDN--called Open Connect.  Netflix also publishes the delivery performance of various ISPs that carry its traffic. The ISP that performed the best in January was Google Fiber.  

Google Fiber is directly interconnected with Netflix. Google Fiber sells no service to its customers slower than 1,000 mbps, yet its Netflix throughput was a relatively meager 3.78 mbps.  No other Open Connect ISP partner even achieved 3 mbps.  Thus, all Netflix customers were paying for much higher speeds from their ISPs than they were getting from Netflix.  

So, the Netflix ISP Speed Index tells you more about Netflix's CDN performance than ISPs' Internet access performance.  In fact, GigaOm recently noted that the best Netflix could say about Open Connect was that it "sucked less" during peak hours for ISP partners than others.

When the minimum necessary speed for even low-level high definition video is at least 4 mbps, it's clear that Netflix has a little ways to go in order to provide higher quality online video delivery.  But, direct interconnect agreements with ISPs--like the Comcast deal--position Netflix with much better control over its service quality than it has today.

Why Content Delivery Services Need to Exist

A lot of media coverage (e.g., here) has mistakenly appropriated any story involving Netflix traffic as a proxy for some kind of meta-online video policy issue with reverberating consequences.  But the ISP is not the "troll on the bridge" and the CDN is not a hapless victim.

The ISP is in the business of providing Internet access from the customer premise to the ISP's point of interconnection with the Internet.  If the ISP was responsible for augmenting ingress capacity from a content provider's backbone to the ISP's point of access every time inbound capacity surged based on an application's popularity, then the ISP would have to involuntarily shoulder the risk of an entirely different business model.

For example, 10 years ago My Space was the dominant social networking site.  Today, it's a relative ghost town.  If the ISP's had born the cost of carrying My Space's swelling downstream demand at the time--by building capacity dedicated to carrying My Space traffic from its backbone provider(s) to the ISP point of access--this would now be stranded, wasted capacity.  These costs would have been absorbed by all of the ISP's customers (including the ones that never even used My Space).  

Netflix is a great service, but it isn't the ISP's service. Moreover, it can be quickly abandoned by the ISP's customers if a better substitute comes along.  The ISP shouldn't have to bear the risk of someone else's Internet business model--especially when there are firms like CDNs that are in the business of accepting risks associated with delivering content to the ISP.