Bots and spiders

Last night, my host asked me if I know what causes some abnormal spikes of bandwidth on our webserver. Learner took a look, and noticed that one IP was spidering the online players page like crazy, a few times a second.

For example, here is some small sample:
64.72.119.194 - - [24/Oct/2006:19:52:03 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:03 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:04 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:04 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:05 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:05 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:06 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:06 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:07 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
64.72.119.194 - - [24/Oct/2006:19:52:07 +0200] "GET /online_players.htm HTTP/1.0" 403 295 "-" "-"
As you can see, the server sends a 403 response. This is because I banned that IP, and a few others from our webserver. The bot is still trying to spider, but is not able to anymore. And it's been doing this for 8 days now. Too bad that the person who is doing this is not playing the game from the same IP, or he would be permanently banned.

Our policy on bots and spiders that access the online players page is simple: not more than once a minute. 120 times a minute is really a little bit too much.