Log files are like internal surveillance cameras, recording everything that happens inside servers, network devices, and some applications. Here's how to make the best use of this data.
Not long ago, one of Citizens & Northern Bank's servers came close to running out of disk space. With more than 100 servers, the bank's small IT staff can't always keep track of space usage, says Peter Boergermann, the IT security officer. Fortunately, the servers have devices installed that monitor their logs, and are programmed to send an automatic email to IT staff members in response to certain events. "One thing we track is when a server runs low on disk space," Boergermann says. "Getting the alert gave us time to see what was going on and clean it up before it caused a disruption."
Originally invented to help IT administrators keep systems running smoothly, log files capture every action on servers, network devices, and certain applications such as database management systems. Though log files resided in their equipment, until recently, most small businesses didn't bother with them.
Regulations forcing the issue
But that's changing. "In the past few years, an increasing number of regulations and industry standards forced companies to pay attention to the data stored in logs," says Mark Nicolett, vice president and distinguished analyst in the security team at Gartner. "Payment Card Industry standards (PCI) pushed the need for log management to a really wide variety of companies." (For more on PCI, see related IncTechnology story.)
Log management is a necessity for meeting PCI compliance, as well as complying with many governmental regulations such as the Health Insurance Portability and Accountability Act (HIPAA) and Sarbanes-Oxley. But, as with the Citizens & Northern server, it can offer benefits that go well beyond compliance.
"In one hospital I talked to, the CIO had used permanent glue to seal up USB ports on a computers containing sensitive information," says Patricia Sueltz, CEO of LogLogic, Inc. which makes log tracking appliances. "He didn't want any information leaving or coming in. Well, there was no information leaving, but they couldn't load things onto their systems, and the equipment couldn't be returned at the end of their lease." Logging could have helped accomplish the same objective without going to such an extreme, she says.
In addition to alerting you to security and hardware issues, data logs can provide all sorts of useful information -- for instance, about what users do on an e-commerce site. And, they can even help you get better service from a hosted software provider. "If a provider isn't fulfilling its service level agreement, you can use logs to prove that," Sueltz notes.
"Log data is ugly"
If you're ready to get more out of your logs, the first step is to choose a log management system, either an application, or an appliance such as those LogLogic provides. "You need a log management system because raw log data is ugly, and very hard to work with," Nicolett says. Besides, simply storing raw log files isn't enough to meet such requirements as PCI standards. "A PCI assessor wants to see centralized collection and the ability to create reports from data," he says.
By storing data to a separate location or device, log management can also help protect log files. "You wouldn't want to keep your web logs on a Web server," Boergermann says. "A hacker who compromises a server will make sure to alter the log so as to conceal the breach. You have to make sure those files are stored somewhere else so that you can do forensics."
Indeed, it's important to remember that the logs themselves contain valuable and sensitive information, and need to be handled with security in mind. "I might be tracking data on whether people use credit cards or debit cards more on weekends," Sueltz says. "Those log files will contain people's card numbers. So what happens to that information? Does it all go to accounting? Is anyone able to look at it who shouldn't?"
Finally, Nicolett warns, log data can be a valuable way to learn more about what's going on within your network -- but only if you act on the information. "It's helpful to understand what's happening, but to do any real good, you have to be willing to follow up on discrepancies or problems that may be found. Otherwise, all it does is document your weaknesses and lack of follow-up. So if you don't act on whatever you learn, the technology can turn into a liability."