[Part of the series of blog postings on Netflow]

Netflow is a network protocol invented by Cisco that provides granular visibility on network utilization. Routers and switches send (“export”) Netflow datagrams that summarize traffic through them to a Netflow Collector program that displays the data. This gives visibility into “who’s using the network?”

Virtually all Netflow collectors are able to process Netflow version 5 packets, so they provide a minimal level of useful functionality.

About Other “Flow Protocols”: Netflow v9, IPFIX, and sFlow

Netflow version 9 can handle IPv6 addresses (v5 cannot). Many exporters (routers and software) can also export version 9 datagrams. There are other interesting enhancements of Netflow v9, but they generally are not operationally important for basic installations.

IPFIX is sometimes called “Netflow v10” because it is a superset of v9. It, too, is not generally important for basic installations.

sFlow (“sampled flow”) is an alternate flow generation technique that works well in settings with very high packet rates. Because the router only needs to report on a sample of the traffic (say one in 10 or 100 packets), sFlow decreases load on the router as well as the generated traffic. Many Netflow collectors can also handle sFlow, but it may not be important for a basic installation.

Configuring Cisco, Juniper, and other commercial equipment

There are a zillion vendor and other websites that tell how to configure your commercial routers. Look there.

Configuring LEDE and OpenWrt to export Netflow data

Equipment from Linksys, D-Link, TP-Link, and dozens of other “home router” vendors can export netflow datagrams if they run the LEDE or OpenWrt software and the softflowd package.

To do this, install LEDE/OpenWrt on your router, then install the softflowd package. (Ask on the forums if you have questions about this.)

Edit /etc/config/softflowd to have these settings. (Change 192.168.1.1 below to the IP address of your Netflow Collector program.)

root@LEDE:~# cat /etc/config/softflowd
config softflowd
    option enabled        '1'
    option interface      'br-lan'
    option pcap_file      ''
    option timeout        'maxlife=60'
    option max_flows      '8192'
    option host_port      '192.168.1.1:2055'
    option pid_file       '/var/run/softflowd.pid'
    option control_socket '/var/run/softflowd.ctl'
    option export_version '5'
    option hoplimit       ''
    option tracking_level 'full'
    option track_ipv6     '0'
    option sampling_rate  '0'

Then issue these two commands to start softflowd. Check Troubleshooting (below) to see whether softflowd is working.

service softflowd restart
service softflowd enable

Generating ‘mock’ netflow data

If you don’t have a Netflow-capable router handy, you can still test Netflow collector programs by running a program that creates bogus/fake/mock Netflow data.

One such program is nflow-generator. The Github repo describes how to run it: the instructions have been tested on OSX, but should work for Windows or Linux, if the Go language is installed.

nflow-generator also has a Docker container that has the same functions, but this has not had much testing.

Troubleshooting Netflow Exporters with tcpdump

Use tcpdump to determine if your Netflow exporter (router or program) is sending netflow datagrams to the computer where you wish to run the Netflow Collector program.

tcpdump captures and displays traffic with certain characteristics. To capture netflow datagrams, use:

tcpdump -i eth0 port 2055

Notes:

• You may need sudo (e.g. sudo tcpdump -i eth0...), if you are not running as root.
• Replace eth0 with the interface name that will be receiving the Netflow traffic.
• Port 2055 is the default Netflow port; change it to match the port you are using.

If Netflow datagrams are arriving at the host, you will see single-line outputs for each datagram. Use Control-C to stop tcpdump.

If datagrams are not arriving, then check to see that you have specified the correct interface and port in the command above. Also ensure that there is no firewall (either in the network in general, or on the computer running tcpdump) that might be blocking the traffic.

This article is a part of the Netflow Collector series.

[Part of the series of blog postings on Netflow]

A lightweight Netflow collector and web display based on NFSEN/NFDUMP in a Docker container. NFSEN and NFDUMP are documented and hosted at SourceForge.net

This container listens on ports 2055, 4739, 6343, and 9666 for netflow, ipfix, and sFlow exports. It displays the collected data in a web interface.

Check the Github repo. The files from the /docs directory have more info. Major thanks go to https://github.com/nerdalert/nfsen-dockerized
for a start on this Dockerfile and all the supporting documentation.

Pros:

• Pretty graphs, for varying time periods: daily, weekly, monthly, yearly.
• View data from a specific time period by going to Details, then dragging the thumbs at the bottom of the graph.
• Top-N talkers also available from Details page: scroll to the bottom of the page, configure “Options”, the click “process”
• Automatically detects the netflow exporter(s).
• Lightweight – Runs on a modest computer. Works great if you’re only handling a single home-router’s flow exports. I haven’t tested it with more exporters/more traffic.
• The Docker container displays graphs “out of the box”.

Cons:

• Home page shows 12 charts, one each for flows/sec, packets/sec, and bits/sec, for each of the four time periods. This makes it hard to know what to “focus on” when you’re just starting up, since none of the charts has very much data.
• Also, showing charts with “Flows/sec” makes it seem as if that’s a critical statistic. But Bit/sec is more important (IMHO).
• Selecting a time period to view is a little clunky. (It’d be fun simply to drag across the graph…)
• Alerts and Stats seem a bit inscrutable.
• I briefly tried to install the “FlowDoh” plugin that purports to display Top-N talkers. It was probably my error, but it just didn’t work after a “good try” to use it.
• As-is, this Docker instance only handles a single exporter (my home router), since the Docker networking doesn’t distinguish external exporter addresses.

This article is a part of the Netflow Collector series.

[Part of the series of blog postings on Netflow]

Webview Netflow Reporter is a lightweight Netflow collector and web display tool based on wvnetflow and flow-tools in a Docker container. Webview Netflow Reporter was created by Craig Weinhold craig.weinhold@cdw.com. The original wvnetflow site is hosted at SourceForge.net.

The Dockerfile is available from Github.

Pros

• Pretty graphs – Displays stack charts of the kinds of traffic (web, email, network, telnet, ssh, dns, mysql, sip, p2p, file server, etc) flowing through the router at any time. See the graphic above.
• Click on an interesting point of the graph, and see exactly what traffic was being sent during that period.
• Automatically detects exporters – no configuration required.
• Lightweight – Runs on a modest computer. Works great if you’re only handling a single home-router’s flow exports. I haven’t tested it with more exporters/more traffic.
• The Docker container comes pre-configured to display charts “out of the box”.

Cons

• No automatic way to see “top talkers” – who’s hogging the bandwidth. You have to click the graph, then scroll through a table of hosts that were transmitting at the time.
• One-minute granularity – The lowest granularity is one minute, despite the fact that the flow data has millisecond accuracy.
• Five-minute time chunks – Data only gets updated every five minutes. This probably could be configured to change the processing rate.
• Text-based configuration – Configuration files are arcane.
• As-is, this only handles one exporter. See Known Issues/Questions
• There’s an outstanding issue where there are gaps in the displayed charts. Hopefully this will be resolved.

This article is a part of the Netflow Collector series.

I have the Witi Router working now with SQM-QoS – the bufferbloat-fighting technology that has been implemented in OpenWrt. Read more about Bufferbloat at Bufferbloat and the Ski Shop.

Demetris has made a build of OpenWrt Chaos Calmer (15.05) that has SQM-QoS built-in, so it’s a simple download and flash of the firmware. You can get a copy of the firmware at the mqmaker forum.

I haven’t tried it myself, but it looks as if stas2z’s BB version of the firmware also has the SQM-QoS as well at Alternate openwrt at mqmaker.

I recently got my WiTi board from mqmaker.com – a high performance open-source router platform. It was funded from a IndieGoGo project and was based on OpenWrt, and thus is easy to customize.

The project doesn’t have a lot of documentation yet, so I just updated its description in the OpenWrt wiki to describe the initial out-of-box setup (pretty smooth – no surprises). Read it at: https://wiki.openwrt.org/toh/mqmaker/witi

Update: The deadline for filing has been extended to 9 October 2015, so you now don’t have any excuse for not sending in your opinion.

I filed the following comment on the FCC page regarding the new Proposed Rule that would (ultimately) prohibit individuals from installing third-party firmware (such as OpenWrt, DD-WRT, Tomato, and the myriad others) on their home router.

If you have an opinion on this, you should go to the FCC Comment Page and click the green SUBMIT A FORMAL COMMENT button.

You’re free to plagiarize my note, or take a cue from the Save WiFi page at LibrePlanet.org. I wrote:

I recommend that the FCC RESCIND its Proposed Rule, Document number 2015-18402 regarding wireless devices. The Proposed Rule is overbroad, would harm many communities of Americans, and is not warranted by the facts on the ground.

Although the FCC has the power to regulate equipment that generates radio frequencies, this is a heavy-handed rule that could be addressed other ways. Specifically, I am concerned about the ability of third parties to modify and create new firmware for “consumer routers.”

The proposed rule would require that router manufacturers lock down the RF portion of the router to obtain FCC approval. This “lock down” would prevent modification to the radio’s power, frequencies, etc to prevent it from radiating outside the specified limits. This is a laudable goal, but the application of this rule as written would result in undesirable consequences.

In practice, most radio functions are very tightly wedded to all the other factors of the hardware/software. The most likely way manufacturers would likely lock down the RF operation would be to make it impossible to modify any of the code in the routers.

There would be a number of adverse consequences both for me personally, to consumers in the US, and the networking industry. These consequences can be ameliorated by allowing the owners of routers to install their own code.

1) Security of the router. It is well known that vendor-supplied firmware for consumer routers often contain flaws. Just last week, the CERT released knowledge of a vulnerability to Belkin routers. See http://www.kb.cert.org/vuls/id/201168 The ability to install well-tested, secure firmware into a router benefits all consumers. The ability for a person to update their own router on a regular basis (as opposed to many manufacturer’s seemingly lackadaisical schedule) preserves security.

2) Research into the field of computer networking. Non-traditional research efforts (outside academia) lead to real improvements in the state of computer networking. An example is the CeroWrt project that developed the fq_codel algorithm. http://www.bufferbloat.net/projects/cerowrt The result of this multi-year effort was a major advance in performance for all routers. The fq_codel code has been accepted into the Linux kernel and now runs in hundreds of millions of devices. As a member of the team that worked on this, I assert that without the ease of modification of a consumer router to prove out the ideas, this improvement would likely not have occurred.

3) Personal learning environments. Individuals, as well as network professionals, often use these consumer routers as test beds for increased understanding of network operation. Losing the ability to reprogram the router would make it more expensive, if not prohibitive, for Americans to improve their knowledge and become more competitive.

4) I would incorporate all the other “talking points” listed on the Save WiFI page at: https://libreplanet.org/wiki/Save_WiFi

5) Finally, I want to address the FCC’s original concern – that these consumer routers are SDRs, and they must not be operated outside their original design parameters. While the goal of reducing radio frequency interference is important, the FCC has failed to demonstrate that the widespread practice of installing/updating firmware in consumer routers has caused actual problems. Furthermore, the FCC can use its current enforcement powers to monitor and shut down equipment that is interfering.

Creating a broad, wide-ranging rule to address a theoretical problem harms industry and individuals, and is an overreach of the rules necessary to preserve America’s airwaves.