« Movin' on up | Main | The generational divide online »

SPLUNK! ZUFF! PAN!! SNUH! BORT! POOO! NEWT! MINT! ZAK!

Over the past few years I've been meaning to evaluate Splunk's main product, a log aggregation and analysis tool by the same name.  Often times as a sysadmin, you have makeshift tools due to budget limitations or other types of hardships.  Many places I've worked have had enterprise level network monitoring capabilities, but unfortunately were monitoring the wrong things.  The items of the most importance on a day-to-day and hour-to-hour basis were done in shell scripts or manually at the command line.  What Splunk does is to give you the configurability of the command line, but packages it up in a nifty web based GUI that allows you do drill down to specific problems (and see the log entries associated) or just skim along at the 40k foot level through graphs and charts.

I recently built a new Ubuntu server box at home (from completely anemic old hardware) and decided that I'd try out Splunk.  Even with the most minimal of hardware Splunk was up and running in no time at all.  From downloading the debian install package to fully functional was about 25 minutes.  My system load churned for about an hour at around 2.0 while all the /var/log and other directories were indexed and pulled into the Splunk database.  It is pretty amazing, as the base version of Splunk can access anything that is local to the system, so if that machine is your syslog server, you can correlate error events over a large network in no time at all.

In the picture below you can see the log entries for a brute force attack against my ssh server (from host 209.239.35.45, which is probably just a hacked intermediary host).  Using Splunk allows me to drill down to see specific attacks by type or host, by very quickly changing the query statements.

 

Below you can see the ebb and flow of 'page not found' 404 errors on my webserver.  I recently started hosting a domain that had been down for about a year.  That domain hosted a bunch of jpg files, which were linked to by some idiot myspace page designer.   Once I track down the individual files that are linked to, I make a symbolic link to this file

 

In the extended entry, I've copied the output from the install.  Just make sure to limit access to port 8000, or whatever other port you'll be using, as there is no access control in the demo version.

 

In a nutshell, Splunk is like a swiss army knife that you never knew you needed, but now you crave.  While it works great as a near real time system monitoring tool, you can also import files from anywhere and process them for historical data.  It would make a great tool for network forensics timeline reconstruction as well as a fine day to day IT operations tool.  I'm sure that there are millions of other things that can be done with this, but just being able to grok so much data at one time is like having some sort of sysadmin super power.

  

Here is how simple the install is:

otterpop:~$ sudo dpkg -i splunk*.deb
Selecting previously deselected package splunk.
(Reading database ... 20705 files and directories currently installed.)
Unpacking splunk (from splunk-3.2.2-34603-linux-2.6-intel.deb) ...
Setting up splunk (3.2.2-34603) ...
----------------------------------------------------------------------
The Splunk Server has been installed in:
        /opt/splunk

To start the Splunk Server, run the command:
        /opt/splunk/bin/splunk start

To use Splunk's web interface, point your browser at:
        http://otterpop:8000

Complete documentation is at http://www.splunk.com/r/docs
----------------------------------------------------------------------

Thanks it folks!!!


The first time you run it under Ubuntu, you get this:

otterpop:~$ sudo /opt/splunk/bin/splunk start

Splunk Free Software License AgreementTHIS SPLUNK SOFTWARE LICENSE AGREEMENT (THE "AGREEMENT") GOVERNS ALL SOFTWARE PROVIDED BY SPLUNK INC. ("SPLUNK") FREE OF CHARGE AND ANY AND ALL UPDATES, UPGRADES, AND MODIFICATIONS THERETO ("FREE SOFTWARE"). SPLUNK SOFTWARE PURCHASED THROUGH SPLUNK'S ONLINE STORE OR OTHER CHANNELS ("PURCHASED SOFTWARE") WILL BE SUBJECT TO APPLICABLE TERMS IN THIS AGREEMENT AND TO THE "ADDITIONAL TERMS FOR PURCHASED SOFTWARE" PROVIDED BY SPLUNK. THE FREE SOFTWARE AND PURCHASED SOFTWARE ARE REFERRED TO COLLECTIVELY AS THE "SOFTWARE". BY CLICKING ON THE "YES" BUTTON, DOWNLOADING OR INSTALLING THE SOFTWARE, OR USING ANY MEDIA THAT CONTAINS THE SOFTWARE, YOU ARE CONSENTING TO BE BOUND BY THIS AGREEMENT.

"Splunk Developer API" means the documentation and functionality enabling the creation of extensions to the Software. "Example Modules" means the source code and binary form of examples that use the Splunk Developer API.

FREE SOFTWARE LICENSE AND RESTRICTIONS. Subject to the terms and conditions of this Agreement, Splunk grants to you a non-exclusive, worldwide, copyright license to use, copy, and distribute the Free Software in binary form only and only to index no more than 500MB of peak daily volume of uncompressed data (the 'Maximum Peak Daily Volume'). The Free Software will be configured to display warnings, reduce available functionality, and/or cease indexing data when the Maximum Peak Daily Volume is reached. Splunk further grants to you a non-exclusive, worldwide, copyright license to use the Splunk Developer API and Example Modules included with the Free Software to develop extensions for the Free Software by addingyour own source code and recompiling (collectively, "Your Extensions"). You agree to assume full responsibility for the performance of the Free Software modified in this way, and shall indemnify, hold harmless, and defend Splunk (includingall of its officers, employees, directors, subsidiaries, representatives, affiliates and agents) and Splunk's suppliers from and against any claims or lawsuits, including attorney's fees and expenses, that arise or result from your distribution of the Free Software and/or Your Extensions pursuant to this Agreement. You retain title to and copyright for Your Extensions, subject to Splunk's title to and copyright for the Free Software, the Splunk Developer API, and the ExampleModules as specified in Ownership, below. You agree that you will include this Agreement with any copy of the Free Software made or distributed by you, and that you will not charge any fee or receive any other consideration in exchange forany distribution of or rights to use Your Extensions. If you want to make any commercial use of Your Extensions you must first enter into a separate agreement with Splunk for such purpose. You shall not (i) decompile, disassemble or reverse engineer the Free Software without the express written authorization of Splunk; (ii)modify, adapt, or create derivative works of the Free Software, except to create Your Extensions in accordance with this Agreement; (iii) rent, lease, loan, or resell the Free Software, the Splunk Developer API, Example Modules, or Your Extensions (including but not limited to offering the functionality of the Free Software on an applications service provider or time sharing basis);or (iv) authorize any third parties to do any of the above.

LIMITATION OF LIABILITY. IN NO EVENT WILL SPLUNK BE LIABLE TO YOU OR TO ANY THIRD PARTY FOR ANY DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, DIRECT, SPECIAL, INDIRECT, INCIDENTAL, CONSEQUENTIAL OR PUNITIVE DAMAGES (INCLUDING LOSS OFUSE, DATA, OR PROFITS, BUSINESS INTERRUPTION, OR COSTS OF PROCURING SUBSTITUTE SOFTWARE) ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT OR THE USE OR PERFORMANCE OF THE FREE SOFTWARE, WHETHER SUCH LIABILITY ARISES FROM CONTRACT, WARRANTY, TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY OR OTHERWISE, AND WHETHER OR NOT SPLUNK HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH LOSS OR DAMAGE. THESE LIMITATIONS WILL SURVIVE AND APPLY EVEN IF ANY REMEDY IS FOUND TO HAVE FAILED OF ITS ESSENTIAL PURPOSE. BECAUSE SOME STATES OR JURISDICTIONS DO NOT ALLOW LIMITATION OR EXCLUSION OF CONSEQUENTIAL OR INCIDENTAL DAMAGES, THE ABOVE LIMITATION MAY NOT APPLY TO YOU.

EXPORT. You agree to comply fully with all relevant export laws and regulationsof the United States ("Export Laws") to ensure that the Software is not (i) exported or re-exported directly or indirectly in violation of Export Laws; or (ii)intended to be used for any purposes prohibited by the Export Laws, including but not limited to nuclear, chemical, or biological weapons proliferation.

GENERAL. This Agreement shall be governed by and construed in accordance with the laws of the State of California, without giving effect to the principles of conflict of law. Any legal action or proceeding arising under this Agreement willbe brought exclusively in the federal or state courts located in the Northern District of California and the parties hereby consent to personal jurisdiction and venue therein. Neither party may assign this Agreement, in whole or in part, except in connection with an internal reorganization or a sale of the business with which this Agreement is associated without Splunk's prior written consent, and any attempt to assign this Agreement other than as permitted above will be null and void. This Agreement is intended for the sole and exclusive benefit of theparties and is not intended to benefit any third party. This Agreement constitutes the complete and exclusive understanding and agreement between the parties regarding its subject matter and supersedes all prior or contemporaneous agreements or understandings, written or oral, relating to its subject matter. Any waiver, modification or amendment of any provision of this Agreement will be effective only if in writing and signed by duly authorized representatives of both parties. Should any term(s) or condition(s) of this Agreement be held to be invalid or unenforceable by a court of competent jurisdiction, the remaining terms and conditions of this Agreement will remain in full force and effect.Do you agree with this license? [y/n]: y

Copying '/opt/splunk/etc/myinstall/splunkd.xml.default' to '/opt/splunk/etc/myinstall/splunkd.xml'.Copying '/opt/splunk/etc/modules/distributedSearch/config.xml.default' to '/opt/splunk/etc/modules/distributedSearch/config.xml'.

Checking prerequisites...

Checking http port [8000]:

openChecking mgmt port [8089]:

openVerifying configuration. 

This may take a while...

Finished verifying configuration.

Checking index directory...

Verifying databases...

Verified databases: _audit, _blocksignature, _internal, _thefishbucket, history, main, metaevents, sampledata, splunklogger

Starting splunkweb...

/opt/splunk/share/splunk/certs does not exist.

Will createGenerating certs for splunkweb server

Generating a 1024 bit RSA private key.............

++++++........................................................++++++

writing new private key to 'privkeySecure.pem'

-----Signature oksubject=/CN=otterpop/O=SplunkUserGetting CA Private Keywriting RSA key

The Splunk web interface is at http://otterpop:8000

If you get stuck, we're here to help.  Feel free to email us at 'support@splunk.com'.

The system this was installed on is a Pentium III 650MHz system with half a gig of ram and a puny 30GB hard drive.  Even on hardware like that, with splunk, apache2, postfix, syslog-ng and a few other processes, the load almost never gets above 0.1.  I'm assuming that you'd need more horsepower than this for a whole enterprise, but you can try out their software on a throwaway PC. 

And if you get the reference in the name of this article, you are a bigger Simpsons fan than I am.  consider yourself enbiggened. 

Bookmark and Share