Who told you that?

This guy popped up in my referer logs the other day, and it turns out that he actually links back to an old post of mine. I’ve changed my opinion a bit since then, after I realized that it’s fairly trivial to write a script that validates referers. All need to you do is grab the page listed as a referer and check to see if it really contains a link back to your site. It’s only a first level technique — there are ways around it — but it would certainly catch what Joel is doing.

Thus, while Joel says there’s nothing that can be done about his technique… he’s wrong. Admittedly, I haven’t integrated my script with my general purpose log analysis scripts but in the cases where I have noticed referer spam I just update my config file and tell the scripts to ignore those referers.

I stuck my script in after the cut. It runs over an active log file, tosses out referers it’s seen before, validates new referers as per the technique above, and emails me a note when it sees a valid new referer. It will not work out of the box on your server, but it should be kind of clear what needs to be updated if you’re a perl coder. Vanity, vanity, all is vanity.

#!/usr/bin/perl -w
use strict;
####

#### Modules

####
use LWP::Simple;

use IO::File::Log;

use Carp;

use Fcntl;

use NDBM_File;

use Mail::Mailer;
####

#### Initialize variables

####
my $logfile = "LOGFILE GOES HERE";

my $baseURL = "BASE URL FOR YOUR SITE GOES HERE";

my $datafile = "DATA FILE FOR KEEPING TRACK OF REFERERS GOES HERE";

my $email = "YOUR EMAIL GOES HERE";

my %urlhist;

my $fh = new IO::File::Log($logfile);
####

#### Main loop -- over and over and over...

####
while (my $line = $fh->getline) {

my @line = split(" ", $line);

my $page = $baseURL . $line[6];

my $referer = substr($line[10], 1, -1);

my $tag = $referer . " " . $page;

(my $date = $line[3]) =~ s/^\[(.*?):(.*)/$1 $2/;
        # A little code to fix /index.php vs / confusion
        $page =~ s/index.(php|html)$//;
        # NDBM_File doesn't write data until untie(), so we tie and

# untie as we go
        tie(%urlhist, 'NDBM_File', "$datafile", O_RDWR, 0) ||

die "couldn't tie";
        next if (

($referer eq "-") ||                    # No referer

($page =~ /\.cgi/) ||                   # Don't do CGIs

($referer =~ /^http:\/\/popone/) ||     # Don't do self-refs

($referer =~ /search|query/) ||         # Search results

($urlhist{$tag})                        # We've seen it before

);
        $urlhist{$tag} = 1;                             # Mark it as seen
        if (scan($referer, $page)) {

notify($referer, $page, $email, $date);

}
        untie(%urlhist);

}
####

#### scan: check for URL in a web page

####   takes: URL to scan, URL to look for

####   returns: true if the URL to look for is found

####
sub scan {

if ($#_ != 1) {

carp "scan() takes two arguments";

return 0;

}
        my $huntURL = shift;

my $findURL = shift;
        my $page = get($huntURL);

if (! $page) {

carp $huntURL, " couldn't be downloaded";

return 0;

}
        return ($page =~ /$findURL/);

}
####

#### notify: let us know we found a match

####    takes: URL scanned, URL found, email address, date

####    returns: nothing at all

####

sub notify {

if ($#_ != 3) {

carp "notify() takes four arguments";

return 0;

}
        my $huntURL = shift;

my $findURL = shift;

my $email = shift;

my $date = shift;
        open(MAIL, "|/usr/sbin/sendmail -t");

print MAIL <<EOM;

To: $email

From: www\@YOURSITE

Subject: Link Found: $huntURL
${date}:

Referer $huntURL contains a link to $findURL.

EOM
        close(MAIL);

}