The LogReport Annual Report 2003
Joost van Baal
Wytze van der Raay
Copyright © 2004 Stichting LogReport Foundation
| Revision History | ||
|---|---|---|
| Revision $Revision: 1.1 $ | $Date: 2007/04/18 15:20:38 $ | |
| $Id: report2003.html,v 1.1 2007/04/18 15:20:38 vanbaal Exp $ | ||
1. Introduction
Log files are often treated like a wasteful by-product of IT activity: they sit somewhere in a dark corner of a computer system and are only examined occasionally, usually in case of after-the-fact reactive problem solving. This is unfortunate. Log files contain the traces of computer activity, and by intelligently analyzing these traces one can learn a lot about the behavior of a system and its users.
Log file analysis is both an essential and tedious part of system administration. It is essential because it's the best way of profiling the usage of the service installed on the network. It's tedious because programs generate a lot of data and tools to report on this data are unavailable or incomplete and when such tools exist, they are specific to one product, which means that you can't compare your Qmail and Exim mail servers.
The Stichting LogReport Foundation, a non-profit foundation, founded August 2000, detected this flaw in system administration and chose to serve a dual purpose: developing and maintaining Lire, our Open Source reporting and analysis software, and serving as a nexus of documentation, ideas, and thought on the topic of log files and their potential applications.
2. Activities in 2003
2.1. Introduction
We report on the activities of the LogReport team during 2003, LogReport's third full year.
A more in-depth overview of what's presented in this document, specifically on Lire development, can be found in the Lire NEWS file, as well as in the various Lire roadmap documents, as shipped with the Lire releases during the year. (There's also a roadmap describing the development plans after August 2003.) A very detailed journal of changes in Lire is in the Lire ChangeLog file. Some highlights can be found on the LogReport history page too.
2.2. People
The main task of the LogReport team is maintenance and development on Lire, LogReport's Free Software tool for performing an integrated analysis of all ones Internet and Intranet Services. Lire does this by automatically generating useful reports from raw logfiles from various services. Next to this work, the team has taken care of administering the LogReport server, hibou.logreport.org. The following changes in the LogReport staff have occured during the year.
On February 28, 2003, Joost van Baal's paid contract with LogReport (for 8 hours a week) ended. On June 30, 2003, Wessel Dankers' paid contract with LogReport (16 hours a week) ended. Both Wessel and Joost however plan to continue doing work for the project, by donating volunteer contributions.
On May 31, 2003, the agreement between Francis Lacoste and the LogReport Foundation ended. Since August 19, 2003, Francis together with Wolfgang Sourdeau are paid for their work on Lire, under a new contract.
Next to these people, involved in the LogReport project by having access to the
LogReport server, or being on the <logreport@logreport.org> alias,
are: Josh Koenig, Wytze van der Raay, and Egon Willighagen.
At the July 3, 2003 LogReport board meeting, a new board got elected. Teus Hagen will no longer be a member of the board; Joost van Baal was elected as a new board member. Teus however will keep doing work for the project, as an advisor to the board. Wytze van der Raay and Jakob Schripsema will continue their positions in the board.
2.3. The LogReport server
The machine hibou.logreport.org hosts our website http://logreport.org/,
provides the public LogReport Online Responder service,
handles our email,
and hosts the 4 public mailinglists <commit@logreport.org>,
<announcement@logreport.org>,
<questions@logreport.org> and
<development@logreport.org>. Furthermore, it hosts a CVS
repository for version control of non-public documents and hibou's
configuration data. The Lire code development is done using CVS on
SourceForge.
Late 2002 the LogReport team noticed that hibou's bandwidth consumption was rising drastically. A search for additional low-cost or free bandwidth was started to limit the heavy bandwidth surcharges by LogReport's ISP. Since February 28, 2003, requests to http://logreport.org/pub/ get redirected to http://download.logreport.org/pub , which is currently pointing to 232pc232.sshunet.nl, a machine administered by Guus Sliepen (who is also donating the bandwidth to us.) Thanks a lot Guus Sliepen!
On July 31 2003, hibou was moved from the VIA Networks facility in Eindhoven, to Amsterdam, The Netherlands: The NLnet Labs foundation has offered LogReport a space in their machine room and free access over NLnet Labs' high-speed internet connection. LogReport of course gladly accepted this very generous offer. Thanks NLnet Labs people!
2.4. Lire Software Releases
Lire is available in .tar.gz source package format, as RPM package to facilitate installation on RPM-based systems like Red Hat Linux and Mandrake Linux, as Debian package for installation on Debian GNU/Linux, and as a FreeBSD port package.
Lire is free in both senses of the word: it is available gratis for download from the internet, and it is Free Software: it is licensed using the GNU General Public License. This means, among other things, anybody is free to study how the program works, and adapt it to ones needs; anybody is free to redistribute copies; and anybody is free to make modifications to the code, and to publish these modifications. The copyright on the major part of the code is held by Stichting LogReport Foundation. See also the Lire Contributor Guidelines for more information on licensing related issues.
In the year 2003, the following Lire versions have been released:
Table 1. Lire releases
| release date | version (filename) |
|---|---|
| December 29, 2002 | Updated Lire 1.2.1 RPMS for Red Hat 7.3, Red Hat 8.0 and Mandrake 9.0 (lire-1.2.1-2.noarch.rpm, lire-html-1.2.1-2.noarch.rpm, lire-pdf-1.2.1-2.noarch.rpm, lire-1.2.1-2mdk.noarch.rpm, lire-html-1.2.1-2mdk.noarch.rpm, lire-pdf-1.2.1-2mdk.noarch.rpm) |
| April 3 and 24 | Lire Debian package 1.2.1-1 and 1.2.1-2 ( lire_1.2.1-1_all.deb, lire_1.2.1-2_all.deb) |
| May 18 | Lire 1.3 source, RPMS for Red Hat Linux 7.3 and 8.0, and Mandrake Linux 9.0 (lire-1.3.tar.gz, lire-full-1.3.tar.gz, lire-1.3-1.noarch.rpm, lire-html-1.3-1.noarch.rpm, lire-pdf-1.3-1.noarch.rpm, lire-1.3-1mdk.noarch.rpm, lire-html-1.3-1mdk.noarch.rpm, lire-pdf-1.3-1mdk.noarch.rpm) |
| September 4 | Lire Debian package 1.3.1-1 (lire_1.3.1-1_all.deb) |
| November 17 | Lire 1.4 source, RPMs for Red Hat Linux 8.0 and 9.0, Debian packages for the stable and unstable Debian release. (lire-1.4.tar.gz, lire-full-1.4.tar.gz, lire-1.4-1.noarch.rpm, lire-html-1.4-1.noarch.rpm, lire-pdf-1.4-1.noarch.rpm lire-doc_1.4-1_all.deb, lire-devel-doc_1.4-1_all.deb) |
| December 15, 17 and 27 | Lire Debian package improvements: versions 1.4-3, 1.4-4 and 1.4-5 released. |
Since 1.4-1, Wolfgang has taken over maintenance of the Lire Debian package from Joost. Lire will very likely get shipped with the Debian `sarge' release, which will be released someday in spring 2004 (based upon information in this 21 Feb 2004 post).
Since August 20, 2002, Lire is distributed with the FreeBSD ports collection (thanks to Edwin Groothuis' efforts.) See e.g. this URL.
We list the main improvements in the various releases, as posted on the LogReport Announcement list. A more detailed overview is in the NEWS file, as distributed with Lire.
Within Lire, we use the term service. A service coincides with one well-defined log file format. So, a service generally coincides with one application: the sendmail service handles sendmail log files. However, a lot of webservers use W3C defined formats, and a lot of commercial firewalls use the WELF format. Therefore, w3c_extended and welf are services. Each service has its 2dlf-convertor, to convert the log file to the more generic Lire DLF format. We provide e.g. sendmail2dlf and w3c_extended2dlf. A superservice is a class of services which share the same DLF format, and which will generally give the same reports.
- Lire 1.3
Lire 1.3 shipped with a major upgrade to the Lire internals. This resulted in a new and drastically improved DLF converter API: the new API supports multiple DLF schemas as well as log continuation. Furthermore, the configuration API was revised; it is now much easier to make extensions like DLF converter, analyzers, etc. configurable. The new improved lr_config(1) command makes use of this: it is now easier to configure periodical reports.
Documentation was improved. The new internal APIs which should be used by extension developers are documented in manpages. A tutorial on how to write new DLF converters and DLF schemas is added to the Developer's manual. An extensive suite of unit tests and regression tests was added to Lire, making it easier to make new Lire extensions.
New DLF converters were contributed by Lire's users:
Of course, these contributions are very much appreciated. Thank you, volunteers!- tinydns
DNS DLF converter for tinydns, as shipped with djbdns. Contributed by Christopher Boumenot.
- s1ms
Email DLF converter for SunONE Messaging Server. Contributed by Jean-Yves Monnier.
- snort
Firewall DLF converter for the Snort Open Source Network Intrusion Detection System. Contributed by Torsten Fellhauer.
- fw1_lea
Firewall DLF converter for CheckPoint Firewall-1 NG 5.0, logging via OPSEC's LEA API. Contributed by Torsten Fellhauer.
- Lire 1.4
Next to many bugfixes, release 1.4 introduces a new report generation algorithm which improves performance a lot. DLF data is now stored in an SQLite database and the report is generated directly from the database. Tests show performance increases of ~18%-55% depending on the superservice used.
Here is a table showing performance comparisons between running lr_log2report from Lire 1.3 and Lire 1.4 on various log files. This benchmark was run on a Debian GNU/Linux system running on an Intel Pentium III 600Mhz with 512Megs RAM.
The Lire 1.4 release has undergone extensive testing on lots of platforms. We've been able to add FreeBSD, OpenBSD, NetBSD and Mac OS/X to our group of test platforms.Table 2. Lire 1.4 performance vs 1.3
Service Lines in Log Sys+User 1.3 Sys+User 1.4 1.4 Improvement % cisco_ios 5,000 121.73s 73.00s 40.03% cisco_ios 100,000 1463.47s 630.93s 56.89% cisco_ios 240,338 3206.67s 1393.20s 56.55% iis_ftp 5,000 31.27s 26.97s 13.75% ipchains 5,000 71.21s 47.00s 34.00% ipfilter 5,000 36.68s 33.02s 9.98% iptables 5,000 67.72s 49.24s 27.29% pix 5,000 86.34s 18.84s 78.18% squid_access 5,000 97.12s 46.47s 52.15% w3c_extended 5,000 145.86s 119.61s 18.00% w3c_extended 100,000 2088.76s 1708.71s 18.20% w3c_extended 541,385 11325.90s 9834.71s 13.17% watchguard 5,000 68.51s 46.64s 31.92% welf 5,000 53.04s 41.81s 21.17% xferlog 5,000 76.92s 4.16s 94.59%
Table 3. Lire 1.4.1 services
| superservice | services | products |
|---|---|---|
| database | mysql, pgsql | The MySQL Database, by MySQL AB; The PostgreSQL Database, by the PostgreSQL Global Development Group |
| dialup | isdnlog | isdn4k-utils, part of isdn4linux, a set of Linux kernel modules |
| dns, dnszone | bind8, bind9, tinydns | BIND, the Berkeley Internet Name Domain by the Internet Software Consortium; tinydns as shipped with Daniel Bernsteins djbdns. |
| argomail, exim, nms, postfix, qmail, s1ms, sendmail | ArGoSoft Mail Server, exim Internet Mailer, Netscape Messaging Server, Wietse Venema's Postfix, D.J. Bernstein's qmail message transfer agent, SunONE Messaging Server, the Sendmail Consortium's Sendmail MTA. | |
| firewall | cisco_ios, fw1_lea, ipchains, ipfilter, iptables, pix, snort, watchguard, welf | The Cisco IOS router, the Cisco PIX firewall, Linux ipchains, Linux iptables, Darren Reed's IP Filter, WatchGuard Firebox, all products supporting the WebTrends Enhanced Log Format like firewalls from Check Point, Check Point Firewall-1 5.0 using the LEA engine, Microsoft, Netscape and Sun Microsystems, the Snort IDS system. |
| ftp | iis_ftp, xferlog | Microsoft IIS FTP Server, all FTP servers supporting the xferlog log format, like ProFTPD and WU-FTPD. |
| msgstore | dbmail, nmsmmp, nmsstore | DBMAIL IMAP and POP server, Netscape Messaging Server Mail Multi Plexor, Netscape Messaging Server IMAP and POP engines. |
| cups_pagelog, lprng_account | The CUPS printing system, LPRng | |
| proxy | ms_isa, squid_access, welf_proxy | Microsoft Internet Security and Acceleration Server, SQUID Web Proxy Cache, all Proxy servers supporting the WebTrends Enhanced Log Format |
| spamfilter | spamassassin | SpamAssassin email UCE spam filter |
| syslog | syslog | BSD syslog daemon, Solaris 8 syslog daemon, Netscape Messaging Server logging daemon, WebTrends syslog daemon, Kiwi Syslog, Sendmail Switch logging daemon |
| www | combined, common, modgzip, referer, w3c_extended | All web servers supporting the NSCA combined log format, like Apache; all web servers supporting the CERN Common Log Format, like Apache, IIS, Boa and PureFTPD |
DocBook XML, HTML (in multiple and in one page), XHTML (in multiple pages and in one page), Lire XML
Excel 95, RTF
PDF
Plain ASCII
2.5. Statistics
We show some statistics, giving some indications about the work done by the LogReport team, as well as about the effectiveness of this work.
2.5.1. Downloads and installs
We do have figures about the number of downloads from our webserver. They're depicted in the graphic.
Number of Lire tarball downloads from http://logreport.org/ in 4-week periods during 2003.
The number of tarball downloads has been growing during the year. Looking back at 2002, we're doing better also: in 2002, we've had 4-week periods with less than 500 downloads (which we hadn't in 2003), in 2002, the best period had less than 900 downloads (in 2003, we've had 2 periods with more than 1000 downloads).
RPM downloads
Number of Lire RPM package downloads from http://logreport.org/ in 4-week periods during 2003.
The number of RPM downloads has been rising a lot during 2003. When comparing these figures with 2002, the difference is even more convincing: in 2002, we've had 2 periods with less than 200 downloads (in 2003, we've had more than 300 downloads in every period), in 2002, we never had more than 400 downloads (in 2003, we've had over 800 downloads during 3 periods). We have about doubled the RPM download traffic in 2003, compared with 2002.
Based on the Debian Popularity Contest, the number of people who've installed a Lire Debian package grew from 330 in January 2003 to 600 in December 2003.
Please note that Lire is also available for download directly from Sourceforge. The Sourceforge-supplied statistics show us the number of monthly downloads fluctuated during the year, with a total of about 1500 downloads for 2003.
2.5.2. Submits to the LogReport Online Responder
During the entire year, we've been running a so called Online Responder on
<log@<service>.logreport.org>. People can send their log
files via email or submit them via a web page, and get a report back.
During the year, the number of weekly users using the Responder has grown from about 25 to about 50. About 100 log files get submitted for analysis weekly.
2.5.3. CVS Commits to the Lire code
There were between 30 and 140 commits on the Concurrent Version System holding the Lire code done monthly during 2003, for a total of 1148. This number represents the number of changes to the Lire code. (Beware: the size and impact of one change can vary a lot! Therefore, these number are to be interpreted cum grano salis.)
2.5.4. Traffic on the LogReport mailing lists
During the year, the number of external people subscribed to the announcement mailing list grew from about 90 to about 200. (Early 2002, we had 24 announcement subscribers.) Subscribers on the development list grew from 35 to 45, while the number of subscribers on the questions list grew from 55 to 90. This is an overall growth of more than 50% for the 3 lists during 2003. We're very happy at this growing interest in LogReport and Lire!
About once every other day an external interested person posts to any of LogReport's mailing lists or group adresses.
We've tackled the spam problem we've been suffering from in early 2003. This made being subscribed to the lists unattractive in the late 2002 period. We now have a policy which executes automatic distribution of posts only from subscribers. Posts from others are moderated manually.
Hosting these open lists has been very helpful at building an active and contributing user community: in the third year, the LogReport project consolidated its true Open Source character. See the next section for the details.
2.6. Cooperation with developers
Lots of people suggested improvements and reported bugs, using our mailinglists. Furthermore, code was contributed by external developers.
2.6.1. Contacts with external developers
We name a few of the people who have contacted us via our public mailing lists during the year, suggesting improvements and reporting bugs. These contacts give reason to believe these people do use Lire to do their work.
In 2003, we've been contacted by various developers and users. We'll give a list of the main contacts.
Long time users Roberto Dalzilli and Edward Quick contacted us
Tony Allen, from Psiframe, Inc., San Francisco, USA
Vitaly V. "Agri" Manushkin from desnol.ru, Russia
Michael Grau from the Kansas, US administration
Bob Kennington from the US government, Department of the Treasury
Ing. Massimiliano Ciccarelli from Defence division of Italian Software and IT Service supplier Datamat S.p.A.
Hubert Lubaczewski from the Eo Networks Sp. z o.o. Network Operations Center in Poland
Mohamed Sami from ECS, Egyptian administration
Andrea Ghirardini from Italian pilasecurity.com
Jean-Marie Kubek from Université Paul Valery, Montpellier, France
Torsten Fellhauer, from iXpoint Informationssysteme GmbH, Ettlingen, Germany
Richard Crane, from Yale University / Haskins Laboratories, New Haven, CT, USA
Alexandre Terplak, from VIA Net.works, France
Stewart James, from Victoria University, Australia
Michael Schwartzkopff, from MultiNET Services GmbH, Grasbrunn, Germany
Christopher V. Browne, from US dialup ISP copper.net
Wouter Verhelst, from Template, Belgium
Joe Dewberry, from Vodafone US
Apart from these people, the LogReport developers have been contacted privately by lots of ISP's, network hardware manufacturers and players in the Financial and Industry markets. It's clear we have established quite a user base: lots of downloads and lots of feedback.
2.6.2. Code contributions
The following code contributions have been made to Lire in 2003. On January 8, 2003, a convertor for the SunONE Messaging Server by Jean-Yves Monnier from Sun Professional Services was contributed. In March 2003, Torsten Fellhauer, of FW1-Loggrabber fame, has contributed a Firewall DLF converter for the Snort Open Source Network Intrusion Detection System, as well as a Firewall DLF converter for CheckPoint Firewall-1 NG 5.0, logging via OPSEC's LEA API. In March 2003 Christopher Boumenot contributed a DNS DLF converter for tinydns, as shipped with djbdns. All these contributions have been integrated and come with Lire, since Lire 1.3.
2.7. Publicity, demo
Mika Boström has written a book on home users' basic computer security, "Kotimikron Tietoturva", ISBN 951-762-813-7, published by Talentum (the book is available in Finnish only). It includes a cover CD with suitable tools and software, including the Lire 1.2.1 source release.
LogReport is mentioned on the LogAnalysis Generic Log Parsing Tools webpage.
In July, Wessel Dankers gave a Lire demo at Tunix, the people working on the Tunix Firewall product. Wessel worked together with Jan E. Kuba van Bijnen, installing Lire on a Tunix test box, and explaining Lire internals.
2.8. Looking back, looking ahead
2.8.1. Comparing 2002 and 2003
In 2002, we had on average about four people working on the LogReport project (Josh Koenig, Wessel Dankers, Joost van Baal, Francis Lacoste). In 2003, we had only two most of the time: Francis with either Wessel or Wolfgang. During approximately two months, nobody was working on paid development for LogReport. In 2002, we shipped four major Lire releases. In 2003, we shipped two.
Still the LogReport team has managed to keep the extended LogReport community vivid. The LogReport board is very grateful and would like to thank the 2003 LogReport team for this amazing achievement.
2.8.2. LogReport Future
In July 2003 Francis Lacoste wrote a proposal for implementing substantial improvements of the Lire software, culminating in a Lire 2.0 release which would be able to survive by true Free Software maintenance. The plan was submitted to Stichting NLnet to request funding, and the funding (EUR 38.000) was granted. Many thanks to Stichting NLnet for (once more) providing major financial support to the LogReport project!
Since August 19, Francis is contracted by LogReport, to complete Lire 2.0. The work is done by both Francis and Wolfgang Sourdeau. The development contract will end 3 weeks after the public Lire 2.0 release, or on August 1, 2004, whichever comes first.
No activities from the LogReport Foundation are expected after the release of Lire 2.0. However, with the Lire 2.0 framework in place, and given the stable and growing Lire user base, we foresee a bright future for the Lire Free Software product. We look forward to see further community-contributed evolution, but also to see e.g. consultants offering tailor-made solutions based on the Lire 2.0 framework.