Blackhole for Bad Bots

Description

Add your own virtual Blackhole trap for bad bots. The Blackhole plugin includes a hidden link to your pages. You then add a line to your robots.txt file that forbids bots from following the hidden link. Bots that ignore or disobey your robots rules will crawl the link and fall into the honeypot trap. Once trapped, bad bots are denied further access to your WordPress-powered website.

I call it the “one-strike” rule: bots have one chance to obey your site’s robots.txt rule. Failure to comply results in immediate banishment. The best part is that the Blackhole only affects bad bots: human users never see the hidden link, and good bots obey the robots rules in the first place. Win-win! 🙂

Using a caching plugin? Check out the Installation notes for important info.

Features

  • Easy to set up
  • Squeaky clean code
  • Focused and modular
  • Born of simplicity, no frills
  • Lightweight, fast and flexible
  • Built with the WordPress API
  • Works with other security plugins
  • Easy to reset the list of bad bots
  • Easy to delete any bot from the list
  • Blackhole link includes “nofollow” attribute
  • Plugin options configurable via settings screen
  • Works silently behind the scenes to protect your site
  • Optionally receive an email alert with WHOIS lookup for blocked bots
  • All major search engine bots are whitelisted so they will never get blocked
  • Focused on flexibility, performance, and security
  • Complete inline documentation via the Help tab
  • Provides setting to whitelist any IP addresses
  • Customize the message displayed to bad bots 😉
  • One-click restore plugin default options

Blackhole for Bad Bots protects your site against bad bots, spammers, scrapers, scanners, and other automated threats.

Not using WordPress? Check out the standalone PHP version of Blackhole!

Pro version now available »

Whitelist

By default, this plugin does NOT block any of the major search engines:

  • AOL.com
  • Baidu
  • Bingbot/MSN
  • DuckDuckGo
  • Googlebot
  • Teoma
  • Yahoo!
  • Yandex

These search engines (and all of their myriad variations) are whitelisted. They always are allowed full access to your site, even if they disobey your robots.txt rules. This list can be customized in the plugin settings.

Header Image Courtesy NASA/JPL-Caltech.

Support development of this plugin

I develop and maintain this free plugin with love for the WordPress community. To show support, you can make a cash donation, bitcoin donation, or purchase one of my books:

And/or purchase one of my premium WordPress plugins:

  • BBQ Pro – Pro version of Block Bad Queries
  • Blackhole Pro – Pro version of Blackhole for Bad Bots
  • SES Pro – Super-simple & flexible email signup forms
  • USP Pro – Pro version of User Submitted Posts

Links, tweets and likes also appreciated. Thanks! 🙂

Screenshots

  • Blackhole Settings Screen (showing default options)
  • Blackhole Bad Bots Screen (showing some example bots)

FAQ

What other security plugins do you recommend?

I recently recorded a video tutorial series for Lynda.com on how to secure WordPress sites. That’s a good place to learn more about the best techniques and WP plugins for protecting your site against threats.

How is this plugin different than a firewall?

Blackhole uses its own “smart bot technology” that only blocks bots if they have demonstrated bad behavior. Firewalls typically are “static” and block requests based on a predefined set of patterns. That means that firewalls sometimes block legitimate visitors. Blackhole never blocks regular visitors, and only it blocks bots that disobey your site’s robots.txt rules. So the rate of false positives is close to zero.

The trigger link is not appearing in the source code of my pages.

In order for the plugin to add the trigger link to your pages, your theme must include the template tag, wp_footer(). This is a recommended tag for all WordPress themes, so your theme should include it. If not, you can either add it yourself or contact the theme developer and ask for help. Here is more information about wp_footer(). Once the footer tag is included, the plugin will be able to add the trigger link to your pages.

Will this block good bots like Google and Bing?

No. Never. All the major search engine bots are whitelisted and will never be blocked. Unless you remove them from the whitelist setting, which is not recommended.

I think the plugin is blocking Chrome, Firefox, etc.?

Impossible because the plugin never blocks by user agent. It only blocks by IP address. No other criteria are used to block anything.

Is it possible to block some bots by just adding them to blocked list and deny them Access to my website?

Not possible with the free version, but the Pro version includes an easy way to add bots manually (via the Bad Bot Log).

How do I add other bots to the whitelist?

Visit the plugin settings and add to the list.

How do I reset the list of blocked bots?

Visit the plugin settings and click the button.

How do I delete the example/default bot from the log?

Not possible with the free version, but can do with the Pro version.

How can I disable the email alerts?

Visit the plugin settings and click the button.

Is there a standalone version of the Blackhole?

Yes. Visit Perishable Press to download a PHP-based version that does not require WordPress.

Is there a Pro version of Blackhole?

Yes, the Pro version is available at Plugin Planet.

Is Multisite supported?

Not yet, but it’s on the to-do list.

Can I manually include the blackhole link?

Yes, you can add the following code anywhere in your theme template:

<?php if (function_exists('blackhole_trigger')) blackhole_trigger(); ?>
Should whitelisted bots contain exact names, or can I just use partial names?

You can use partial names or full names, depending on how specific you would like to be with blocking. If you look at the default whitelisted bot strings, you will see that they are just portions of the full user agent. So for example you can block all bots that include the string “whateverbot” by including that string in the whitelist setting. It makes it easier to block bots, but you have to be careful about false positives.

What about WordPress automatic (hidden) robots.txt?

By default, WordPress will automatically serve a hidden, “virtual” robots.txt file to anything that requests it. Once you add your own “real” robots.txt file, WordPress will stop generating the virtual one. So when it comes to WordPress and robots.txt, real trumps virtual. Blackhole Pro requires that you add some rules to an actual robots.txt file, but it does not create/add any robots rules or the robots.txt file for you. Check out the plugin’s Help tab for more infos.

Which WP caching plugins are compatible with Blackhole?

Blackhole for Bad Bots is not yet tested with all caching plugins and scripts. So it may not be compatible with all of them. Here is a list of known compatible caching plugins and the steps required for full compatibility:

If you are using a caching plugin or script, and Blackhole isn’t working, you can try one of the the following solutions:

  • Add the blackhole URL (available via the “Robots Rules” setting) to your caching plugin’s “ignore” list
  • Add the IP(s) of your caching script or load balancer to the Blackhole setting, “Whitelisted IPs”

If neither of those tricks work, it’s recommended to deactivate the plugin. Please test thoroughly (for more info, refer to the “Testing” section in the readme.txt).

Got a question?

Send any questions or feedback via my contact form

Reviews

Not compatible with all cache plugins

wplike75

Hello Jeff Star, I'm updating this review, thanks for adding the infos about the cache plugins, and for all your work, you are a great developers. About my comment in your web site perishable press, I do not know why you did not received it, when I have posted it will be marked with "waiting for moderation" and it never appair after, but anyway... Best regards.

Read all 28 reviews

Changelog

1.5 (2017/03/08)

  • Fixes some incorrect translation strings
  • Adds some style to the Robots Rules setting
  • Adds complete contextual help (via the Help tab)
  • Adds info about the default bot to the Bad Bots screen
  • Replaces delete link with blackhole icon for default bot
  • Removes line breaks from plugin-activation warning messages
  • Replaces admin_url() with admin_url('plugins.php') for activation warning
  • Replaces get_template_directory with get_stylesheet_directory
  • Updates plugin links on the WP Plugins screen
  • Adds default option for ip_whitelist setting
  • Adds option to set the address for the “From” email header
  • Improves loading of PHP include files
  • Changes link text for clearing list of bad bots
  • Adds new action hook: blackhole_settings_contextual_help
  • Adds new action hook: blackhole_badbots_contextual_help
  • Refines the Settings and Bad Bots screens
  • Improves logic for script/style enqueue
  • Adds blurb about Pro version
  • Generates new translation template
  • Tests on WordPress version 4.8 (alpha)

1.4 (2016/11/14)

Important: if you are upgrading from a previous version and are using a Custom Warning Template, please move it to your theme directory. Otherwise it will be deleted when you upgrade the plugin. Check out the “Custom Warning Template” section under the Installation documentation for more information.

  • Fixes missing settings notices via settings_errors
  • Adds strong tags to admin notices
  • Changes directory for custom warning template
  • Removes the /custom/ directory
  • Removes Save Changes button from Bad Bot screen
  • Updates some default settings page styles
  • Changes stable tag from trunk to latest version
  • Adds &raquo; to rate plugin link on Plugins screen
  • Improves escaping for localization tags
  • Updates URl for rate this plugin link
  • Tests on WordPress version 4.7 (beta)

1.3.1 (2016/08/11)

  • Fixes bug with IP-address whitelisting
  • Fixes bug with user-agent whitelisting
  • Makes a small change to settings label
  • Generates new translation template

1.3 (2016/08/08)

  • Updates WP Admin Notices
  • Replaces _e() with esc_html_e()
  • Replaces __() with esc_html__()
  • Removes unnecessary settings_errors()
  • Renames text domain from “bbb” to “blackhole-bad-bots”
  • Renames /lang/ directory to /languages/
  • Updates load_i18n()
  • Refines blackhole_callback_robots()
  • Fixes bug in blackhole_whois()
  • Improves performance by eliminating redundant whois lookup
  • Adds new hook to filter IP address validation: blackhole_ip_filter
  • Improves IP address handling (adds three new IP-related functions)
  • Improves readability of whois email notifications
  • Adds “Whitelisted IPs” setting
  • Adds a rate this plugin link
  • Tests on WordPress 4.6

1.2 (2016/03/28)

  • Updates URL for BBB_HOME
  • Tweaks display of “Blocked Bots” in plugin settings
  • Tests on WordPress 4.5 beta

1.1 (2016/02/22)

  • Adds “baidu” to the search engine whitelist
  • Adds credit to documentation for header image

1.0 (2016/02/22)

  • Initial release

Contributors & Developers

This is open source software. The following people have contributed to this plugin.

Contributors

Browse the code