Mikrotik Drone

Most people mod their drone with powerful antennas to increase their range.  I wanted to add a different type of antenna to mine for a completely different reason.

I recently ventured into providing Wireless Internet access to myself and all my neighbors.  I live a good distance out in the country, and my only real options for internet were completely useless Satellite based, or a local WISP what was charging me $160 per month for 3 Mbps.   I did the later for seven years before deciding that I needed to find something better.  Since I work from home in a technology based job, I needed high speed internet with low latency.  So I secured access to use a tower in town next to a building (thanks M6 Oilfield!!) where I could purchase a dedicated line.  I now had 75 Mbps of internet that I could do with as I pleased, but I had to get it to my house first.

My first thoughts were to set up an easy Point-to-Point, but after talking with a few of my other neighbors who were in the same predicament as I was, I decided to go with a sector.  This way I could provide access to all of them also.  I chose a Mikrotik NetMetal 5, and a RF Elements 120 Sector, climbed 150 feet up the tower, and got it all rocking and rolling.  On the house side I added a Mikrotik DynaDish 5.  Before the DynaDish came out, I was rocking the Mikrotik QRT5, but the upgrade to an AC link was worth it (for other reasons).  Now, 2 years later both my equipment and the number of neighbors connected has expanded.  Most of them are farmers who just check email / surf the web a little, so the link is still way under utilized.  There are always more that seem to want access though.  I don’t actually charge my neighbors for access to the internet, instead I have them buy their own equipment and then we enact a barter system.  When you live in the country, you are always needing to borrow tools / equipment for various things.  So we instead help each other out in that respect.  They get free internet, and I get to borrow a rototiller when I need it. 🙂

Now for the Drone.  I recently purchased a Yuneec Typhoon Q500 4K.  It has been an awesome drone, and I highly recommend it to anyone looking to get one.  I have used it for various things, such as finding a break in our water main where it crosses the back creek, but I wondered if I could use it for the WISP I was slowly building out.

The furthest neighbor is 13 miles from the tower, while I myself am only 6 miles out.  With different elevations, direction to the tower, and tree lines; finding a solid connection to the tower required lots of trial and error.  You generally put up a temporary 20′ pole and pointed in the general direction and turned it until you found it.  If that connection wasn’t good enough, you moved around until you found one that was.  This may involve moving the temp pole, which was a real hassle.  It may be that you needed to go higher than 20 feet also, which made testing even more problematic.  All this time you also were dragging a laptop around to check the signal quality, and you needed a long Ethernet cable with power to use a POE injector to power the CPE.  So with this in mind, I decided to mod my drone to act as a remote station.

One of the awesome new products that Mikrotik just p ut out was the Mikrotik SXT Lite5 AC.  What makes this awesome is it technically has 2 wifi modules in it, both a 5 Ghz, and a 2.4 Ghz.  The 5 Ghz makes the connection back to the tower, while the 2.4 Ghz acts as an Access Point, allowing a single connection back to manage the device.  This means that I can connect to it wirelessly, without needing a long Ethernet cable / laptop (since I can use my phone).  So I grabbed one of these from my local supplier, ISP Supplies.  One problem solved.  The next problem was power.  I had thought about using a battery pack POE injector like the iDea4Tec Smart PowerBank but not only are they expensive for what they do, but at 1/2 a pound, it was going to be pretty heavy for my drone.  Weight is at a premium, so the next idea was to use the drone’s own battery to power the device.

I was already going to be removing the camera from the drone.  I didn’t really need to see where I was pointing, and since the camera resides on the front end of the drone, it would be directly blocking the CPE signal.  It also saved me some weight.  So removing the camera left me with the power connector freed up.  Since I still planned on using the camera with the drone later, I didn’t want to butcher the existing power cable, so I snagged another one from Carolina Dronz (which also happens to be where I bought the drone to begin with).   I cut off the end that would normally attach to the camera, and proceeded to figure out what voltage we were supplying.  There are 3 wires, so I found the one that supplied consistent power and the ground.  The other supplied power but it was at a lower voltage than I was wanting.  The power line supplied ~11.7v (varies with the battery voltage) while the CPE will accept anywhere from 8v to 30v, so we were in business.  All I needed now as to cut a Cat5 cable, and splice the proper Cat5 wires to the power and ground (Pins 4/5 to power, 7/8 to ground), and I now had a Drone to POE Cable.

Since the drone itself moves a little in the wind, you want the CPE to be able to keep its orientation too.  Later this will be handled by mounting it to one of the GoPro 3-Axis Gimbals (I wasn’t going to mess with my current Gimbal).  For now I have created a temporary mounting system.   This involves making it free hanging beneath the drone by securing it with a rubber cables on each side, and some electrical tape to temporary hold it in place.  I used the tape so I could determine the exact balance point of the CPE to make it always point horizontal while hanging.

So I did a quick pre-configuration of the CPE so it would connect to my tower, and away we went for our first flight.  I powered on the drone, before the drone had finished its initialization, the Mikrotik was already booted up and I had connected to the wireless via my phone.  I flew up ~25 feet and pointed in the general direction of the tower and was awarded with seeing the connection made.  The drone holds its position extremely well (even in some wind).  It also has a Turtle mode, so I was able to pan back and forth super slow until I was able to dial the connection in to find the perfect center.   The Drone Controller itself then provides me with the exact elevation and azimuth of the drone, so now I also know exactly what size pole and the direction it needs to face.  This means I could now find the best possible connection in under 5 minutes, where as normal it might take me 30 minutes or longer.

From here I plan on buying a real Gimbal for it, and doing some more real world tests from further out.  I have a few clients whose connection isn’t as good as I would like, so it will be a good chance to see how I can improve their access.

I promise to post a lot more pictures with detailed instructions once its not so “hackish”.  I would like to note that through out this entire endeavor, I had the support of my good friend Greg Sowell.  He has been educating me all things wireless ever since.

 

January 28, 2017 · Jimmy · One Comment
Posted in: Mikrotik

Decoding Netflow v5 packets with PHP

Lately I have been toying with rewriting the Flowview plugin in Cacti with something that is a bit more usable. The current plugin has some major disadvantages. Namely it relies on the flow-tools package to do all the heavy lifting such as collecting flows, parsing everything, creating reports. In reality, the current plugin is really more of a viewer for flow-tools. Flow-tools itself has several major drawbacks. It doesn’t support Netflow v9 or v10 (IPFIX). The last one is a major problem considering that VMWare 5.1 has moved all of its distributed switches to only outputting v10. I also have to rely on the flow-reports command to spit out a few canned reports, so its not very customizable. It also has major issues when receiving flows from routers that are in different time zones (lots of wackiness when saving the flows per hour, and attempting to use those directories for reports).

To rewrite Flowview. There will be 3 major parts. The Collector (and storage), the Parser, and the Interface. The Collector will be as small and efficient as possible to help it collect large amounts of incoming netflow packets. Spawning multiple threads will help a bit too I think. I bet my friend Greg can help me out with a large stream of Netflow packets to stress test it with. I will be storing the flows in a MySQL database (partitioned per day). As with the Syslog plugin, we have had pretty good luck with performance utilizing MariaDB 5.5 partitioned tables with large data sets. I will be directly inputting into a Memory table (again to help improve performance with the Collector), and will do some processing on those entries (via the Cacti Poller) every minute to move them into long term storage. The UI will be easy enough, and the Parser will be doing the heavy lifting of querying the database. I want to move to a queued job approach of querying for reports, so that we can get a better handle on the load that each report can put on a server.

The first thing to write is the Collector. The easiest Netflow packet to decode is v5. It has completely static length headers, flow record sizes, and data field lengths. You can take a quick glace at the format and fields over at the Cisco site.
NetFlow Export Datagram Format
Each packet you receive will first contain a header telling you a little about the included flows. Following that will be each flow record. You can have multiple flow records in a single packet. A simple loop and a few lines of code to unpack everything is all that is really needed.

Now, for anyone wanting to write their own Netflow parser in PHP, I have included some basic code below to get you started. From here you will want to expand it to do validations, possibly some logging, and decode more Netflow versions. I will show how to decode v10 (IPFIX) in my next blog post.

June 6, 2014 · Jimmy · 4 Comments
Posted in: Cacti, Netflow, Plugins

CactiEZ 0.7 Released

Today, the long awaited release of CactiEZ v0.7 is out.  While the release took much longer than anticipated (I reworked it several times) I believe the final product turned out better because of it.

There are several key things to note about the new CD.

  • OS is CentOS 6 x64 (there is no 32bit version)
  • Now prompts for a few key items during the installation such as Network info, timezone, and password
  • More secure out of the box than the previous builds (some linux hardening, randomized mysql passwords, etc…)
  • PHP Timezone is automatically configured for the timezone entered during install
  • Cacti and all plugins are now installed via RPMs from my custom repository for easy updating of Cacti
  • Switched to RSyslog instead of Syslog-ng
  • NTop removed
  • Now includes a custom setup plugin that walks you through setting up your Cacti server (plugin installs, templates, etc…)
  • Custom jquery based theme donated by an anonymous source (Thanks!)
  • No upgrade path from CactiEZ v0.6 provided

It is currently available for download via torrent at CactiEZ.  EDIT: I have now added a HTTP download link.

I would like to thank all the beta testers that have helped with this project.  I had 500+ beta requests so congrats to those who were selected and got an early peak at it.  For the 1000 other people that asked me why it took so long.  Please realize that every minor change that occurred would require testing that consisted of rebuilding the RPMs (if necessary), recompiling the CD, transferring the CD, re-installing from CD, and then actually starting the testing after all that was done.  All I can say is thanks to VMWare ESXi and a lot of custom scripting, I was able to stream line the process.

For anyone that appreciates the hard work that goes into a project like this, I have provided a Donations link on the CactiEZ page.   I also love Amazon Gift Cards.

For support / questions, please use this thread over at cacti.net – CactiEZ Support

Update: I would like to thank everyone that has made a donation no matter how small.  I shall drink a pint in your honor while I decide what to put in the next CactiEZ (snmp traps maybe?)

October 14, 2012 · Jimmy · 37 Comments
Posted in: Cacti, CactiEZ

Open Source System Management Conference 2012

For anyone interested, I will be speaking at the Open Source System Management Conference 2012 in Bolzano, Italy on May 10th, 2012.  If you would like to attend, please check it out below.  There is going to be a lot of great speakers and informative lectures going on.

Save the date!

 

Thursday, May 10th  –  Bolzano / Italy  –  9.00 a.m.

 

International keynote speakers               Lively discussions                        Shared experiences

The yearly Open Source System Management Conference goes in its 4th edition. The event over the past years has grown to one of the most important meets in Europe for

IT managers, system administrators and everyone interested in Open Source monitoring solutions like Nagios, Cacti, Shinken or Icinga. Join this free event and get in touch with international experts presenting relevant trends, new strategies, actual projects
and practical user experiences.

Date and Location:
May 10th,
Convention Center Four Points Sheraton, Bolzano / Italy

Who should attend: IT managers, system administrators
and everyone interested in Open Source monitoring solutions.

Free entry:
Given the limit of 400 places available we suggest to confirm your participation still today! 

v Register now!

v See the Agenda 

Overview of this year´s speakers:

Jeffrey Hammond, Principal Analyst Forrester Research (US)
Oliver Jan, Founder of the French Monitoring Community (F)
William Leibzon, Monitoring Systems Expert and Nagios Developer (US)
Jimmy Conner, Cacti Plugin Architecture (US)
Luca Deri, Founder of ntop (I)
Franco Stoppini / Leo Renier, IT Department InfoCamere (I)
Georg Kostner, Department Manager Würth Phoenix (I)
Hauke Böttcher, Partner Director OTRS (GER)
Bernd Erk, Managing Director NETWAYS (GER)
Anders Haal, Project Manager Ingby (SWE)

——————————————————————————————————————————————————-

******  NEWS     Find further Conference news on our Blog or go directly to the Conference´s Webpage   –  NEWS ******


 Legge sulla Privacy D.Lgs. n. 196/2003: Würth Phoenix S.r.l., titolare del trattamento, garantisce la massima riservatezza nel trattamento dei dati personali. L’interessato potrà in ogni momento esercitare il diritto secondo l´articolo 7 e richiedere la rettifica o la cancellazione dal nostro archivio, comunicando a Würth Phoenix S.r.l. (mailto:info@wuerth-phoenix.com).

April 20, 2012 · Jimmy · Comments Closed
Posted in: Cacti

Querying Nagios NRPE Client via PHP

I recently had the idea to write an agent for a few windows servers to execute some custom commands so I could graph the data in Cacti.  I have dabbled in making my own Windows SNMP extension agents in the past (and I love SNMP) and while they work well; I really didn’t want anything that complicated.

Now not being someone to re-invent the wheel unless I have to, I started looking around at a few other existing agents.  A few of the requirements I had in mind were that I didn’t want the communication in plain text, it needs to be somewhat secure (password or ACL), it needs to be fairly easy to install and configure custom commands, and most importantly I need to be able to easily communicate with the agent over PHP.  After finding agent after agent that did 1 or 2 of my requirements, I settled on the Nagios NRPE Agent.  While it does have some quirks, it does satisfy the first 3 of my requirements, so all I would have to work out is the 4th.  The NRPE Agent can use SSL to communicate between the monitoring server and the agent.  While it does not have the ability to require a password, it does allow me to set an ACL on the agent to specify the IP addresses allowed to communicate to it.  Configuration is just editing a text config file to add my custom commands, which is really simple.  It also makes it pretty easy to push any new updates to my servers.

As for the 4th requirement, communicating with the agent from PHP.  I wanted to do this, so I could write some Cacti Script Server scripts, and not have to launch a new process to then query the agent.  For NRPE, you normally would have to run the check_nrpe executable with the command line arguments for the command.  This was messy and an extra step which would add some latency and slow down my polling.  I looked around, and really couldn’t find much info on the protocol NRPE uses between the client and host.  I did finally manage to find a perl script that did basically what I wanted, or at least laid out the basics of the required packet for me, so I didn’t have to resort to Wireshark to figure it out.  The only thing required was to figure out exactly what it was doing, and then implement the same thing in PHP.  The perl script was a few hundred lines of code, and had 4-5 required libraries, which was something I was going to have to do away with.

A few of the tidbits I gleamed.  I first made the mistake of attempting to use fsockopen to create the SSL connection, but for SSL to work with NRPE, it seems to require the use of the ADH cipher.  Just trying to do a straight up SSL2 or SSL3 connection is going to fail with an error about the cipher (check the agent log).  This didn’t pose much of a problem though, since PHP also has lots of nice built-in functions for dealing with SSL streams, and its really easy to change the cipher used with those functions.   This also makes it easy, since I really don’t have to deal with certificates or anything else you would normally need to do with SSL.  The next thing is that the NRPE agent expects the packet to always be the exact same size no matter the command.  We are basically going to have to pad the command with nulls to be 1024 in length, we then add all our CRC checksums and other misc headers to this.  Overall the packet itself is very simplistic, and doesn’t include anything really complicated to create.

Anyway, on to some code.  This is the simple script I wrote to test querying the agent.

Overall it took very little code to create the communication which made me extremely pleased.  It turned out to much much smaller than the perl code I was originally working with.  This will ofcourse have to be modified to conform to the Cacti Script Server usage, but that is really the easy part.  Once I get a few templates completed (for the normal NRPE commands) and I will post them for everyone to use.

February 18, 2012 · Jimmy · Comments Closed
Posted in: Cacti