Speaking of which, the squidguard blacklists might make really good input for MT-Blacklist. Remember this and try it.
Month: November 2004
My Monday LISA tutorial was on system log aggregation, analysis, and statistics. mjr taught it, and he’s as good a public speaker as ever. Also the topic was pretty damned fascinating. I’ll be dumping a pile of links into del.icio.us sometime soonish now.
Highlights, some of which are significant and some of which are just cool:
You can set up an invisible loghost. What you do is you specify a non-existent host as the loghost on all your DMZ servers. You’re gonna need to manually stuff an entry into the arp table so that your DMZ servers will blithely send syslog packets off into thin air. Then you hook the real loghost up to the DMZ with no IP address in promiscuous mode. Run tcpdump on it to capture all the packets, and write some cheap perl to strip syslog payloads out of the captured packets.
Or use mjr’s plog instead of tcpdump, since it’ll automate all that complex stuff for you. Neat.
Artifical ignorance. Cute term. It’s basically the same rule of thumb as “block everything, then permit what you want” but reversed. “It’s interesting unless I’ve explicitly said it’s boring.” At a very basic level, it looks like this:
First seen anomaly detection. It’s sort of like artificial ignorance, but different. You alert every time something completely new appears in the logs. There is a tool for this, also written by mjr, called NBS (Never Before Seen). It uses Berkeley DB and is very fast. You feed it input for a specified dataset and it tells you if it’s seen that particular chunk of input before. It can report on its database in a bunch of useful ways.
Example: record DHCP servers giving out IP addresses. (Sample string after a bit of log parsing: “10.0.0.10 gives IP 10.0.1.1 to MAC 0:2:2d:10:10:10”.) If a new MAC address shows up, it’ll be flagged by NBS as a new chunk of input, because that string is guaranteed to differ in that case. If an old MAC address gets a different IP address, that’ll show up too, but only the first time it gets that particular IP. As a bonus, you’ll find out if any new DHCP servers show up. Pure gold.
Another example, which happens to be the first use I thought of: turn it loose on my HTTPD log files. Filter said log files for referrer and URL pairs; report the first time a new referrer/URL pair is seen. I have something like this in place now but it’s written in perl and it’s fairly fragile; this will be better.
Or just dump URLs into the database. “Hm, someone just tried to load /cgi/foobar.exe for the first time; looks like a new exploit.”
So yeah, a very cool tutorial. I’m all jazzed about the possibilities. Check out his web site on the topic.
I kinda think I haven’t found the heart of Atlanta yet. I took MARTA up to the Buckhead station, and found a wasteland of shopping malls, alleviated only by a Borders with a stunningly friendly woman behind the counter. Midtown was nicer this morning — the Flying Biscuit is a short walk from the train station, and they do an awesome breakfast. Even if they only have turkey bacon. So maybe Midtown is the right place to be, but there weren’t all that many pedestrians. Hard to figure.
People are authentically nice. You can tell because they aren’t just nice to you; they act in a manner which expects niceness back. I was walking past Piedmont Park and a guy in his twenties said “Hey, excuse me?” He was in the middle of parking. He wanted help parallel parking so he wouldn’t ding the bumper of the car behind him. That’s pretty nice, and he assumed I’d be nice and help out, so I did.
(He had three feet in front of him. A Bostonian would have been embarassed to ask for help in a situation like that. There’s some kind of tradeoff here.)
In any case, I have the pleasure of having gone from this:
To this:
The latter is Grady High School’s football stadium, by the by. It is the alma mater of Earthwind Moreland, New England Patriot.
There is some desire for a Jon Udell-esque library bookmarklet for Harvard’s Hollis Catalog. A little bit of research, and there you go. It’s not perfect, since Hollis doesn’t seem to use ISBNs as the primary index, but the first book listed on the results page will be the book you’re looking for if it’s in the catalog.
My friend Rob MacDougall has a nifty new blog, which I cannot recommend strongly enough if you are interested in the history of technology. Share and enjoy.
My pal Jamie’s doing a music exchange — burn a CD with your favorite songs of all time on it, send it to everyone on the list, you know the drill. In one of those fleeting moments of personal revelation I sometimes indulge in, here’s mine. (Yeah, that’s a pretty weak excuse for personal revelation.)
Ashcroft has resigned. That’s good.
If anyone was paying attention to my list of criteria for judging Bush’s success, it’s been updated.
The Incredibles is really really good, but if you’ve been reading reviews, you don’t need me to tell you that. I teared up a bit, I forgot it wasn’t a live-action movie, yes it really is one of the best superhero movies ever period. I got nothing much to say beyond “Wow, awesome.” Go go go.
“Listen…
There’s another national anthem playing,
Not the one you cheer
At the ball park.”
“We’re the other national anthem, folks,
The ones that can’t get in
To the ball park.”
Available from iTunes, happily enough.