As we take a shot at opening people’s eyes to bufferbloat, we should know some of the “objections” we’ll run up against. Even though there’s terrific technical data to back up our research, people seem especially resistant to thinking that bufferbloat might affect their network, even when they’re seeing problems that sound exactly like bufferbloat symptoms. But first, some history:
The very idea of bufferbloat is simply unbelievable. Jim Gettys in 2011 [1] couldn’t believe it, and he’s a smart networking guy,. At the time, it seemed incredible (that is “not credible” == impossible) that something could induce 1.2 seconds of latency into his home network connection. He called in favors from technical contacts at his ISP and at Bell Labs who went over everything with a fine-toothed comb. It was all exactly as spec’d. But he still had the latency.
This led Jim and Dave Täht to start the investigation into the phenomenon known today as “bufferbloat” – the undesirable latency that comes from a router or other network equipment buffering too much data. Over several years, a group of smart people made huge improvements: fq_codel was released 14 May 2012 [3]; it was incorporated into the Linux kernel shortly afterward. CAKE came in 2015, and the Wi-Fi fixes that minimize bufferbloat the drivers arrived in 2018. In 2021 cake-autorate [4] arrived to handle varying capacity ISP links. All these techniques work great: in 2014, my 7mbps DSL link was quite usable. And when the pandemic hit, fq_codel on my OpenWrt router allowed me to use that same 7mbps DSL line for two simultaneous zoom calls.
As one of the authors of [2], I am part of the team that has tried over the years to explain bufferbloat and how to fix it. We’ve spoken with vendors. We’ve spent untold hours responding to posts on assorted boards and forums with the the bufferbloat story.
With these technical fixes in hand, we cockily set about to tell the world about how to fix bufferbloat. We assumed it would be an easy task. Our efforts have been met with skepticism at best, or stony silence. What are the objections?
- This is just the ordinary behavior: Of course things will be slower when there’s more traffic (Willfully ignoring orders of magnitude increase in delay.)
- Besides, I’m the only one using the internet. (Except when my phone uploads photos. Or my computer kicks off some automated process. Or I browse the web. Or …)
- It only happens some of the time. (Exactly. That’s probably when something’s uploading photos, or your computer is doing stuff in the background.)
- Those bufferbloat tests you hear about are bogus. They artificially add load, which isn’t a realistic test. (…and what do you experience if you actually are downloading a file?)
- Bufferbloat only happens when the network is 100% loaded. (True. But when you open a web page, your browser briefly uses 100% of the link. Is this enough to cause momentary lag?)
- It’s OK. I just tell my kids/spouse not to use the internet when I’m gaming. (Huh?)
- I have gigabit service from my ISP. (That helps, but if you’re complaining about “slowness” you still need to rule out bufferbloat in your router.)
- I can’t believe that router manufacturers would ever allow such a thing to happen in their gear. (See the Jim Gettys story above.)
- I mean… wouldn’t router vendors want to provide the best for their customers? (Not necessarily – implementing this (new-ish) code requires engineering effort. They’re selling plenty of routers using the decade-old software. The Boss says, “would we sell more if we make these changes? Probably not.”)
- Why would my ISP provision/sell me a router that gave crappy service? They’re a big company, they must know about this stuff. (Maybe. We have reached out to all the vendors. But remember they profit if you decide your network feels too slow and you upgrade to a higher capacity device/plan.)
- But couldn’t I just tweak the QoS on my router? (Maybe. But see [5])
- Besides, I just spent $300 on a “gaming router”. I just bought the most expensive/best possible solution on the market. (And you’re concerned that you still have lag?)
- You’re telling me that a bunch of pointy-headed academics are smarter than commercial router developers – who sold me that $300 router? I can’t believe it. (Well, these fixes seem to solve the problem when it replaces vendor firmware…)
- And then you say that I should throw away that gaming router and install some “open source firmware”? What the heck is that? And why should I believe you? (Same answer as above.)
- What if it doesn’t solve the problem? Who will give me support? And how will I get back to a vendor-supported system? (Valid point – the first valid point)
- Aren’t there any commercial solutions I can just buy? (Not at the moment. IQrouter was a shining light here – available from Amazon, simple setup, worked a treat – but they have gone out of business. And of course, for the skeptic, this is proof that the “fq_codel-stuff” isn’t really a solution – it seems just like snake oil.)
So… All these hurdles make it hard to convince people that bufferbloat could be the problem, or that they can fix for themselves.
A couple of us have reached out to Consumer Reports, wondering if they would like a story about how vendors would prefer to sell you a shiny new router (or new bigger ISP plan) than fix your bufferbloat. This kind of story seemed to be straight up their alley, but we never heard back after an initial contact. Maybe they deserve another call…
The recent latency results from Starlink give me a modicum of hope. They’re a major player. They (and their customers) can point to an order of magnitude reduction in latency over other solutions. It still requires enough “regular customers” to tell their current ISP that they are switching to Starlink (and spend $600 to purchase a Dishy plus $100/month) to provide a market incentive.
Despite all this doom and gloom, I remain hopeful that things will get better. We know the technology exists for people to take control of their network and solve the problem for themselves. We can continue to respond on forums where people express their dismay at the crummy performance and suggest a solution. We can hope that a major vendor will twig to this effect and bring out a mass-market solution.
I think your suggestion of speaking to eSports people is intriguing. They’re highly motivated to make their personal networks better. And actually solving the problem would have a network effect of bringing in others with the same problem.
Good luck, and thanks for thinking about this.
This is a slightly edited version of a post first published on the bloat and starlink lists.
Rich Brown
[1] https://courses.cs.washington.edu/courses/cse550/21au/papers/bufferbloat.pdf
[2] https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bufferbloat/
[3] https://lists.bufferbloat.net/pipermail/cerowrt-devel/2012-May/000233.html