Decoding Netflow v5 packets with PHP

Lately I have been toying with rewriting the Flowview plugin in Cacti with something that is a bit more usable. The current plugin has some major disadvantages. Namely it relies on the flow-tools package to do all the heavy lifting such as collecting flows, parsing everything, creating reports. In reality, the current plugin is really more of a viewer for flow-tools. Flow-tools itself has several major drawbacks. It doesn’t support Netflow v9 or v10 (IPFIX). The last one is a major problem considering that VMWare 5.1 has moved all of its distributed switches to only outputting v10. I also have to rely on the flow-reports command to spit out a few canned reports, so its not very customizable. It also has major issues when receiving flows from routers that are in different time zones (lots of wackiness when saving the flows per hour, and attempting to use those directories for reports).

To rewrite Flowview. There will be 3 major parts. The Collector (and storage), the Parser, and the Interface. The Collector will be as small and efficient as possible to help it collect large amounts of incoming netflow packets. Spawning multiple threads will help a bit too I think. I bet my friend Greg can help me out with a large stream of Netflow packets to stress test it with. I will be storing the flows in a MySQL database (partitioned per day). As with the Syslog plugin, we have had pretty good luck with performance utilizing MariaDB 5.5 partitioned tables with large data sets. I will be directly inputting into a Memory table (again to help improve performance with the Collector), and will do some processing on those entries (via the Cacti Poller) every minute to move them into long term storage. The UI will be easy enough, and the Parser will be doing the heavy lifting of querying the database. I want to move to a queued job approach of querying for reports, so that we can get a better handle on the load that each report can put on a server.

The first thing to write is the Collector. The easiest Netflow packet to decode is v5. It has completely static length headers, flow record sizes, and data field lengths. You can take a quick glace at the format and fields over at the Cisco site.
NetFlow Export Datagram Format
Each packet you receive will first contain a header telling you a little about the included flows. Following that will be each flow record. You can have multiple flow records in a single packet. A simple loop and a few lines of code to unpack everything is all that is really needed.

Now, for anyone wanting to write their own Netflow parser in PHP, I have included some basic code below to get you started. From here you will want to expand it to do validations, possibly some logging, and decode more Netflow versions. I will show how to decode v10 (IPFIX) in my next blog post.

June 6, 2014 · Jimmy · 4 Comments
Posted in: Cacti, Netflow, Plugins

4 Responses

  1. Dan Farrell - July 3, 2014

    Good to see you’re active on this, it has very real, relevant value to the Cacti user-base. So thanks for keeping up the good work and I look forward to using this when the dust settles!

  2. GCEB - August 5, 2014

    I new Flowview would be awesome! I’d love to be able to use a Netflow/IPFIX tool within Cacti, but because of Flowview’s limitations, I’ve been using the free version (2 interfaces) of ManageEngine’s product. Unfortunately I’m not a programmer and wouldn’t be much help in writing it, but I’d be glad to help test.

  3. Konrad - October 14, 2014

    Why not using already available tools as backend?

    https://tools.netsa.cert.org/silk/

  4. Robert - November 10, 2014

    Awesome !! Thank you for taking on this challenge. Look forward to what you can put together.