Block Bad Bots from slowing down WordPress

Sometimes it’s hard to know why your WordPress website is going slow.

And your hosting company says it’s your fault ‘too many plugins’.

The truth is shared web hosting is really only able to service very small WordPress sites.

Once the WordPress site get a bit larger in terms of posts/pages or the number of plugins, it starts to suck resources up, and eventually the host will CPU limit you. This happens for a few reasons:

1. larger site don’t cache well. Too many URLs to cache eventually leads to uncached pages, or too many pages needing refreshing.
2. more pages/post means exponentially more CPU. WordPress lists pages by category, tags, and other ways they reference each other. Each page takes a certain amount of CPU to calculate. More pages/posts more CPU is used.
3. more plugins and heavier themes makes each page harder to calculate.

Now wait until the Bots come past – to see what you have on every page. They don’t care too much about sucking down your resources. Once you get CPU limited the bots just make it worse.

Something like fail2ban or WordFence can stop some of the attack bots. But I don’t think either try to stop the web spider bots. You need some web spider bots, like bingbot and google bot – but a lot of the others aren’t need.

Here is a formula to kill a few of the worst bots – put this in your .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Baiduspider|icevikatam|AhrefsBot|spbot|MJ12|majestic12|Ahrefs) [NC]
RewriteRule .* - [F,L]

 

By Scott Farrell on June 7th, 2018 , Follow @scott_WordPress Tweet to @scott_WordPress