IPTables Bandwidth Monitor
From Soft-hausWiki
Welcome to IPTables Bandwidth Monitor Wiki.
ipt-parse is a tool to manage output created by custom iptables using the Tomato firmware for Broadcom based Linux routers such as the WRT54GL. This program should work with any iptables based distribution and compiles on Linux and Windows.
Original Author: Mark Vejvoda mark_vejvoda@hotmail.com http://www.soft-haus.com/blog
This tool is primarily designed to add support for bandwidth and website monitoring per IP Address in conjunction with the latest tomato firmware.
Where to download the latest package?
This project is officially maintained at launchpad.net [here] The Official Wiki is this page that you are reading (since launchpad.net doesn't offer a Wiki service)
Binary precompiled files available: [here]
Folder Structure of download package
Decompress the ipt-parse.7z archive into an empty folder on your system. If you don't have a program that recognizes .7z files get one [here].
In the root folder there is a README.txt which is a smaller subset of the contents of this Wiki. Below is a brief summary of the sub-folders and their contents:
ipt-parse - all of the source code for this project including a Code Blocks project file. linux32_x86 - binary files compiled for Ubuntu x86 32bit Linux. tomato_files - all files (including binaries) related to running this tool in the Tomato Firmware. win32_x86 - binary files compiled for Windows x86 32bit.
There are two types of installations to choose from before using this bandwidth monitor.
Type 1 - Logfiles Configuration Summary
This method simply saves the output of iptables into logfiles at the specified interval of time that you choose. The ipt-parse program reads these files (parses them) and produces a summary output view. This method is not recommended and remains for backwards compatibility for users who use this method.
Type 2 - SQLITE Configuration Summary
This method saves the output of iptables into a SQLITE database file at the specified interval of time that you choose. The ipt-parse program reads this data from iptables and produces a summary output view. This method has many advantages and allows for a multitude of statistical analysis possibilities. Using the SQLite configuration also allows for tracking websites visited Per Client.
Special Note:
For the two input date parameters you may use the following special notations:
today (meaning use todays date)
today-4 (meaning use todays date minus 4 days as the date)
bw_startday=21 (meaning the 21st is the bandwidth monthly start
date. if today > x then use current month and change day
of month to x, if today < x use last months month and change
day of month to x)
Type 1 - Logfiles Configuration Installation
Here are the steps required to setup your tomato router (or ANY device or operating system that supports iptables)
- NOTE: It is assumed that you have enough free disk space accessible to the tomato scripts
(like a cifs share) for log files to be archived
Step 1:
In Tomato Setup goto Basic / Static DHCP and configure a static IP address for each client that you wish to monitor bandwidth.
Tomato Static DHCP Example:
Step 2:
In Tomato Setup goto Administration / Scripts / Firewall and add the following lines: (repeat the last two lines for each additional static IP address that you configured in the Static DHCP screen)
iptables -N traffic_in iptables -N traffic_out iptables -I FORWARD 1 -j traffic_in iptables -I FORWARD 2 -j traffic_out iptables -A traffic_in -d 192.168.0.100 iptables -A traffic_out -s 192.168.0.100
Tomato Firewall Script Example:
Step 3:
In Tomato Setup goto Administration / Scheduler / Custom 1 and enable the custom scheduled script (at a minimum of every hour or every minute if you want to ensure you don't lose much data if your router loses power or reboots) and select Everyday.
Enter the following script (which periodically archives bandwidth usage statistics:)
cd /cifs1 #change this line to the location accessible to tomato for saving logs iptables -L traffic_in -vn > usagelogs/traffic_in_`date '+%Y%m%d%H%M%S'` iptables -L traffic_out -vn > usagelogs/traffic_out_`date '+%Y%m%d%H%M%S'` ./ipt-parse 1 today-7 today usagelogs/ traffic_in_ traffic_out_ > weeklybandwidth.html
Tomato Custom Scheduler Script Example:
Step 4:
Copy the tomato ipt-parse binary attached to the download archive into /cifs1 and ensure it has execute permission from within tomato (telnet to tomato and from the shell type:)
chmod 777 /cifs1/ipt-parse
- NOTE: there is a version of ipt-parse located in the linux32_x86 folder in the download archive which is the same program compiled under Ubuntu 9.04 (Jaunty) and should be able to run from any regular linux distro due to the minimal dependencies.
- NOTE: The cifs drive must have a linux/unix file system. You can mount a Windows base cifs and acces files, but you won't be able to set the execute permisons as windows file systems doesn't support this. One turn around if you have a windows server in place is to use Linux inside a virtualbox to map a drive.
ALL DONE.
Now depending on how frequently you told the script to execute it will generate and html page in /cifs1/weeklybandwidth.html showing you the bandwidth usage for the last 7 days.
Type 2 - SQLITE Configuration Installation
Here are the steps required to setup your tomato router (or ANY device or operating system that supports iptables)
- NOTE: It is assumed that you have enough free disk space accessible to the tomato scripts (like a cifs share) for the SQLITE database file to be saved. It IS possible to run completed self contained using a JFFS partition (see section lower on this page)
Step 1:
In Tomato Setup goto Basic / Static DHCP and configure a static IP address for each client that you wish to monitor bandwidth.
see #Tomato Static DHCP Example:
Step 2:
The Manual Way to do Step #2
In Tomato Setup goto Administration / Scripts / Firewall and enter the following:
iptables -N traffic_all iptables -I FORWARD 1 -j traffic_all iptables -A traffic_all -d 192.168.0.100 iptables -A traffic_all -s 192.168.0.100
(repeat the last two lines for each static IP address that you configured in the previous step and ensure you replace the example IP address used above if not applicable in your environment)
see #Tomato Firewall Script Example:
The Automatic Way to do Step #2
You may ask ipt-parse to create bandwidth rules using IP addresses that you have already configured in the Static DHCP page. If you use this method, you should NOT manually enter the parameters 'iptables xxx' as shown below but instead add this line to your firewall script (assuming ipt-parse is on cifs1):
/cifs1/ipt-parse 6
If you did not use the automatic way for step #2 (shown above) then remember to enter the IP addresses manually in your firewall script ie.
iptables -N traffic_all iptables -I FORWARD 3 -j traffic_all iptables -A traffic_all -d 192.168.0.100 iptables -A traffic_all -s 192.168.0.100
see #Tomato Firewall Script Example:
- NOTE: If you decided to use the automatic way to auto configure the bandwidth monitoring rules, then for step 3 (the next step) DO NOT pass the additional command-line parameters but instead use:
./ipt-parse 2 BANDWIDTH
as opposed to using the line in step 3 that looks like:
./ipt-parse 2 BANDWIDTH "iptables -L traffic_all -vnx" "iptables -Z traffic_all"
Step 3:
In Tomato Setup goto Administration / Scheduler / Custom 1 and enable the custom scheduled script (at a minimum of every hour or every minute if you want to ensure you don't lose much data if your router loses power or reboots) and select Everyday.
Enter the following script (which periodically archives bandwidth usage statistics, I have mine set to every 5 minutes)
cd /cifs1 #change this line to the location accessible to tomato for saving logs ./ipt-parse 2 BANDWIDTH "iptables -L traffic_all -vnx" "iptables -Z traffic_all" ./ipt-parse 4 today today BANDWIDTH > dailybandwidthlive.html ./ipt-parse 4 today-7 today BANDWIDTH > weeklybandwidthlive.html
see #Tomato Custom Scheduler Script Example:
Step 4:
Copy the attached ipt-parse and BANDWIDTH binaries to /cifs1 and ensure they have execute and read/write permission from within tomato (telnet to tomato and from the shell type:)
chmod 777 /cifs1/ipt-parse chmod 777 /cifs1/BANDWIDTH
- NOTE: the ipt-parse is the same program compiled under Ubuntu 9.04 (Jaunty) and should be able to run from any regular linux distro. Notice that the file BANDWIDTH is a standard sqlite database that may be queried using the standard sqlite3 binary from any platform (we have attached one for both tomato and Ubuntu 9.04 and placed them in the download archive for convenience).
- NOTE: The cifs drive must have a linux/unix file system. You can mount a Windows base cifs and acces files, but you won't be able to set the execute permisons as windows file systems doesn't support this. One turn around if you have a windows server in place is to use Linux inside a virtualbox to map a drive.
ALL DONE.
Now depending on how frequently you told the script to execute it will generate and html page in /cifs1/weeklybandwidth.html showing you the bandwidth usage for the last 7 days.
General Notes:
Generating reports using intra-day date and times
To query using up to the minute intervals (if you are recording stats that frequently) you may pass dates with times to produce a more finite data report like this:
./ipt-parse 3 "20090511 22:15" "20090512 01:45" BANDWIDTH
- NOTICE: the format MUST be exactly as above, with a space between date and time portion)
Importing iptables logfiles into sqlite database
If you have existing iptables logfiles, and it is assumed their values are stored in the format:
iptables -L traffic_out -vn
(notice the -vn as opposed to -vnx) then you may import these logfiles into a sqlite database using this command-line:
./ipt-parse 0 20090401 20090501 usagelogs/ traffic_in_ traffic_out_ flags=import=BANDWIDTH
where all parameters are the same as if you were generating a statistical output for a given date range with the addition of the following parameter:
flags=import=BANDWIDTH
where BANDWIDTH is the path and location to the sqlite database to import into.
It should be safe to run this import multiple times (Even for the same date range) because there are validations to ensure that the same data does not get imported more than once.
Shrinking the sqlite database
It is possible to shrink the size of the database IF, and ONLY IF you are willing to remove granularity by time for a specific date range (for example for and old date range where you really don't care what happened throughout the day) To shrink the database for a range of dates (compress intra-day values into daily values) use this syntax:
./ipt-parse 5 20090301 20090510 ~/Code/sqlite-3.6.13/BANDWIDTH
The command-line above will scan all data between March 1st, 2009 - May 10, 2009 and summarize stats daily, then delete old records for that date range and insert the new daily summarized values, and resize the database.
If you wish to remove old records to reduce the size of the sqlite database you may use the following syntax:
./ipt-parse 7 BANDWIDTH flags=purgeoldrecords=60
The line above will remove records older than 60 days and shrink the database (if enough disk space is free to do so).
Displaying both IP Address and Host names
If you want to see additional host information you may add the optional commandline parameter:
flags=morehostinfo
For CONSOLE output it will append extra host information to the IP Address field, while for HTML output it will turn the IP Address fields into links with pop-up hints showing the additional host information for each IP Address. For example you can generate using a sqlite database with extra host info as follows:
./ipt-parse 4 today today BANDWIDTH flags=morehostinfo > dailybandwidthlive.html
HOW TO Modify the Tomato UI and add new content
Add the following line to the firewall rules, if you want to make use of the modified UI attached with this package:
./SetupUI.sh > SetupUI.log
You will need to copy the folder tomato_dynamic_files and the file SetupUI.sh to /cifs1/ and possibly edit the contents to suit your paths etc. You'll also need to ensure it has execute access by executing (from tomato shell)
chmod 777 /cifs1/SetupUI.sh
To visit the new UI you will need to add /ext/ to the end of the URL that you normally use to enter the Tomato webpage for your router. An example of the updated UI would be:
How to run IPTables Bandwidth Monitor with ONLY the router (without an external CIFS mount)
Yes it is possible to run this tool in Tomato completely on the router itself WITHOUT any external CIFS drives mounted. For this to work you must be using ipt-parse v1.3.1 or later.
What you need for this to work:
- Setup a JFFS2 disk
- Copy files to the /jffs path on the router using scp (secure copy)
- Setup Scripts to capture bandwidth to the /jffs disk
Detailed steps to get ipt-parse working standalone on your router running Tomato
In the following steps we assume your router's IP Address is 192.168.0.1 (so change this to the correct IP for your router)
- Select the Administration Menu Item, then click on JFFS2
- Click on the Enable checkbox, then click on the Format/Erase button.
(Now you should be able to copy ipt-parse files to the /jffs folder on your router.)
- Copy the following two files (from the download archive's tomato_files folder: ipt-parse and BANDWIDTH
by using the secure copy (scp) example using Ubuntu (but any operating system with the scp tool should work too):
scp ipt-parse root@192.168.0.1:/jffs scp BANDWIDTH root@192.168.0.1:/jffs
- Telnet (or SSH) to your router and make ipt-parse executable (you must enable either SSH or Telnet Daemon under Tomato Admin Access):
ssh root@192.168.0.1 <enter your password when prompted> cd /jffs chmod 777 /jffs/ipt-parse
- Setup all of your clients to use Static DHCP in the Tomato Web Interface
caption
- Tell ipt-parse to setup bandwidth monitoring for these static DHCP clients by going to Administration Menu Item, then click on Scripts
then select the Firewall Tab and add the following line:
/jffs/ipt-parse 6
- Now add a timed event to save bandwidth statistics to your sqlite database file BANDWIDTH by going to Administration Menu Item,
then click on Scheduler, then click on Enable for the Custom 1 (or Custom 2) section. Select Everyday and pick every 15 minutes and add the following lines to the script edit window:
cd /jffs ./ipt-parse 2 BANDWIDTH flags=importdailyonly flags=purgeoldrecords=60 > ipt1.log ./ipt-parse 4 today today BANDWIDTH flags=morehostinfo_append > dailybandwidthlive.html mkdir /var/wwwext cp dailybandwidthlive.html /var/wwwext/dailybandwidthlive.asp cp /www/*.css /var/wwwext/
- Reboot your router and bandwidth statistics will be saved to the file BANDWIDTH in your routers /jffs folder.
You may view the statistics by going to a special URL on your router: http://192.168.0.1/ext/dailybandwidthlive.asp
IPT-PARSE Command-line Options
The following shows the help output when running:
./ipt-parse --help ipt-parse v1.3.3 [Nov 26 2010 12:08:08] Copyright (C) 2009-2010 Mark Vejvoda For detailed help please visit: http://soft-haus.com/wiki/index.php5?title=IPTables_Bandwidth_Monitor Usage: flags=commandtype= <required-parameters> <flags=x> : flags=commandtype=value, where value is one of the following command types: : 0 : console report [based on saved logfiles]. : 1 : html report [based on saved logfiles]. : 0 and 1 required parameters: flags=startdate= flags=enddate= flags=logfilepath= : flags=logfile-incomingprefix= flags=logfile-outgoingprefix= : example: ./ipt-parse flags=commandtype=0 flags=startdate=20090401 flags=enddate=20090501 flags=logfilepath=usagelogs/ flags=logfile-incomingprefix=traffic_in_ flags=logfile-outgoingprefix=traffic_out_ : 2 : import data [based on sqlite db] inserts records into sqlite DB from iptables output. : required parameters: flags=database= flags=iptables-get-stats-cmd= flags=iptables-clear-stats-cmd= : *NOTE: the parameters flags=iptables-get-stats-cmd= and flags=iptables-clear-stats-cmd= are ONLY required if iptables was not : previously configured using flags=commandtype=6 : example: ./ipt-parse flags=commandtype=2 flags=database=BANDWIDTH flags=importdailyonly flags=purgeoldrecords=60 : 3 : console report [based on sqlite db]. : 4 : html report [based on sqlite db]. : 5 : summarize data [based on sqlite db] compresses intra-day record data into one record per day. : 3, 4 and 5 required parameters: flags=startdate= flags=enddate= : flags=database= : example: ./ipt-parse flags=commandtype=3 flags=startdate=20090511 22:15 flags=enddate=20090512 01:45 flags=database=BANDWIDTH : 6 : Use this option to auto-discover static DHCP clients when running in the Tomato firmware. : example: ./ipt-parse flags=commandtype=6 : 7 : clean data [based on sqlite db] shrink database by removing empty space. : required parameters: flags=database= : example: ./ipt-parse flags=commandtype=7 flags=database=BANDWIDTH flags=purgeoldrecords=60 : 8 : import DNS data [based on sqlite db] inserts records into sqlite DB from dns message logfile. : required parameters: flags=database= flags=dnslogfile= : example: ./ipt-parse flags=commandtype=8 flags=database=BANDWIDTH flags=purgeoldrecords=60 flags=dnslogfile=/var/log/messages,/var/log/messages.0 flags=dnsignorehosts=127.0.0.1 flags=dnsignorewebsites=*facebook.com flags=dnscombinewebsites=cbc.ca : 9 : console report [based on sqlite db] shows dns website lookup usage via console. : 10 : html report [based on sqlite db] shows dns website lookup usage via html. : required parameters: flags=database= : example: ./ipt-parse flags=commandtype=9 flags=database=BANDWIDTH flags=morehostinfo_append flags=startdate=today flags=enddate=today flags=sortby=host,website flags=filterby=website LIKE '%ebay%' : Parameter details: : ------------------ : flags=startdate= : start date range: : example: flags=startdate=20090425 and default is current date. You may also use the keyword 'today' like this: today : flags=startdate= : end date range: : example: flags=enddate=20090426 and default is current date. You may also use the keyword 'today' like this: today-30 : *NOTE: for flags=commandtype= 3 and 4 you may specify a date and time value using this format: 20090425 19:45 : flags=logfilepath= : optional path to search for the logfiles (default value is ./). : flags=logfile-incomingprefix= : optional log filename prefix used to describe incoming data logs (example: traffic_in_). : flags=logfile-outgoingprefix= : optional log filename prefix used to describe outgoing data logs (example: traffic_out_). : <flags=x> : optional flags containing special values depending on the flags=commandtype=, possible values: : For flags=commandtype=0 or 1 you may import logfile data into sqlite using: : flags=import==BANDWIDTH : where BANDWIDTH = the sqlite database filename to import into (the table must be named bandwidthusage) : For flags=commandtype=0, 1, 3, and 4 you may output additional host information for each client using: : flags=morehostinfo (which displays the Hostname along with the IP Address) : flags=morehostinfo_append (same as above except when using flags=commandtype=1 or 4 the HTML will append Hostname to IP Address) : flags=hostname_only (will display ONLY the Hostname without references to the IP Address) : For flags=commandtype=2 you may want to keep your database smaller and import statistic summarized by date: : flags=importdailyonly (this will allow you to frequently update your database but only store 1 record per Client per day) : flags=purgeoldrecords=x (this flag will delete record older then x days) : example: flags=purgeoldrecords=60 (would delete records older than 60 days) : Special notes: There are some assumed limitations in this tool namely: : Maximum supported client IP's = 101, Maximum supported DateRange length = 91 days : These limits are hard coded but may be easily changed and a new binary produced.
The general command-line usage for ipt-parse is:
ipt-parse <generate-type> <generate-type-parameters> <flags=x>
where possible <generate-type-parameters> are dependent on the <generate-type> selected (see the table below):
<start-date-filter> - The optional start date range to use.
Example: 20090425
Note: the default date is the current date. You may also use the keyword today
<end-date-filter> - The optional end date range to use.
Example: 20090426
Note: The default date is the current date. You may also use the keyword today-30
For <generate-type> 3 and 4 you may specify a date and time value using this
format: 20090425 19:45
Another keyword that may be used for start or end date is: bw_startday=21
This means that bandwidth start period begins on the 21st of each month.
You should change the number 21 to the day of the month your bandwidth cycle begins
<logfiles-path> - The optional path to search for the log files (default value is ./).
<incoming-data-logfileprefix> - The optional log file name prefix used to describe incoming data logs (example: traffic_in_).
<outgoing-data-logfileprefix> - The optional log file name prefix used to describe outgoing data logs (example: traffic_out_).
<flags=x> - The optional value containing special values depending on the <generate-type>, possible values:
For <generate-type> = 0 or 1 you may import logfile data into sqlite using:
flags=import=BANDWIDTH
where BANDWIDTH = the sqlite database filename to import into
(the table must be named bandwidthusage)
For <generate-type> = 0, 1, 3, and 4 you may output additional host information for
each client using:
flags=morehostinfo (which displays the Hostname along with the IP Address)
flags=morehostinfo_append (same as above except when using <generate-type> = 1 or 4 the HTML
will append Hostname to IP Address)
flags=hostname_only (will display ONLY the Hostname without references to the IP Address)
For <generate-type> = 2 you may want to keep your database smaller and import statistic
summarized by date:
flags=importdailyonly (this will allow you to frequently update your database but only
store 1 record per Client per day)
flags=purgeoldrecords= (this flag will delete record older then x days,
example: flags=purgeoldrecords=60 would delete records older than 60 days)
The # of days specified MUST be larger than 0 days. Also notice
that this purge option can also be used with <generate-type> = 7
<sqlite-database-filename> - The file name of the sqlite database
<iptables-showstats-command> - Default value of this parameter is: "iptables -L ipt-parse-traffic-all -vnx"
<iptables-clearstats-command> - Default value of this parameter is: "iptables -Z ipt-parse-traffic-all"
<iptables-chain-name> - Default value of this parameter is: ipt-parse-traffic-all
| generate-type | generate-type-parameters | description |
|---|---|---|
| 0 | <start-date-filter> <end-date-filter> <logfiles-path> <incoming-data-logfileprefix> <outgoing-data-logfileprefix> <flags=x> | Produce console output based on logfiles. |
| 1 | <start-date-filter> <end-date-filter> <logfiles-path> <incoming-data-logfileprefix> <outgoing-data-logfileprefix> <flags=x> | Produce HTML output based on logfiles. |
| 2 | <sqlite-database-filename> <iptables-showstats-command> <iptables-clearstats-command> <flags=x> | INSERT records into sqlite database from iptables output and produce console output based on sqlite database. The parameters <iptables-showstats-command> and <iptables-clearstats-command> are optional and default to the values mentioned above. |
| 3 | <start-date-filter> <end-date-filter> <sqlite-database-filename> <flags=x> | Produce console output based on sqlite database. |
| 4 | <start-date-filter> <end-date-filter> <sqlite-database-filename> <flags=x> | Produce HTML output based on sqlite database. |
| 5 | <start-date-filter> <end-date-filter> <sqlite-database-filename> <flags=x> | Compress data for a date range by taking intra-day records and summarizing them into daily records. |
| 6 | <iptables-chain-name> | Use this option to auto-discover static DHCP clients ONLY when running in the Tomato firmware. In this case the parameter <iptables-chain-name> is optional. |
| 7 | <sqlite-database-filename> | console [based on sqlite db] shrink database by removing empty space. |
Examples:
If you want to see the usage for the current bandwidth period and the period starts on the 21st of each month and you are using sqlite:
./ipt-parse 3 bw_startday=21 today BANDWIDTH
If you want to show BOTH IP Address and Hostname you can always tack this on the end:
./ipt-parse 3 bw_startday=21 today BANDWIDTH flags=morehostinfo
To add the current bandwidth counts to a sqlite database located in /cifs1 called BANDWIDTH run: (Note this format assumes the command: ./ipt-parse 6 was executed first, otherwise additional parameters are required)
./ipt-parse 2 /cifs1/BANDWIDTH
To generate an HTML report from a sqlite database showing only todays usage:
./ipt-parse 4 today today /cifs1/BANDWIDTH > /cifs1/dailybandwidthlive.html
To add current bandwidth counts to a sqlite database using a customized iptables configuration, ONLY store data counts daily (meaning you cannot query based on time of day) and delete bandwidth records older than 60 days:
./ipt-parse 2 BANDWIDTH "iptables -L bw-jffs-all -vnx" "iptables -Z bw-jffs-all" flags=importdailyonly flags=purgeoldrecords=60
IPT-PARSE Website Monitoring
We assume you have already copied the files as is mentioned in the SQLite configuration steps above. To monitor website usage PER Client IP Address do the following:
- Goto Tomato's Advanced Settings then DHCP / DNS and enter the following in dnsmasq:
log-queries
- Add the following to Administration Scheduler Scripts:
./ipt-parse 8 flags=database=BANDWIDTH flags=dnslogfile=/var/log/messages flags=purgeoldrecords=60 flags=dnsignorehosts=127.0.0.1 flags=dnsignorewebsites=*kronos.com >dns.log
./ipt-parse 8 flags=database=BANDWIDTH flags=dnslogfile=/var/log/messages.0 flags=purgeoldrecords=60 flags=dnsignorehosts=127.0.0.1 flags=dnsignorewebsites=*kronos.com >>dns.log
Where can I go to get support?
We have created new support forums at:
http://www.soft-haus.com/smf/index.php/board,2.0.html
IRC on freenode.net in the channel: #softhaus
