Free WordPress plugin
See which AI crawlers request pages on your site, choose what each one is told, and check whether it took any notice.
Writing a rule and the rule working are not the same thing. You can add a line to robots.txt asking an AI crawler to stay away, and you have no way of knowing whether it made any difference.
AI Crawler Stats records every request from a known AI crawler that reaches WordPress, along with what your site was telling that crawler at the moment it arrived. If a crawler kept coming after you asked it not to, you see it.
Which AI crawlers requested pages, how often, and which pages. Daily totals for trends, and exact per-address counts when you need certainty.
Allow, monitor, or disallow, per crawler. Rules are written into robots.txt for you, and the advisory header is sent on matching requests.
The part nobody else shows. Requests that arrived while the rule in force said no, listed per crawler and per address, using the rule that applied at the time rather than today's setting.
Any single post, page or product can override the site setting, from the edit screen, quick edit or bulk edit, with a filterable column in the posts list.
There are three ways to tell a crawler to stay away. Only one of them is enforced. The plugin repeats this on every screen where you make the choice, because the alternative is a customer who believes they are protected and is not.
| Mechanism | What it really is |
|---|---|
| A robots.txt rule | A request. Well behaved crawlers honour it. One that ignores it can still fetch your pages. |
X-Robots-Tag: noai, noimageai |
Also a request, and a weaker one. noai is not an agreed standard and most crawlers ignore it. |
| Refusing the request with a 403 | Enforced at your server. The page is never sent. It still matches on a user agent, which can be faked. |
This is not a firewall. A scraper that does not announce itself is unaffected, and no plugin can honestly promise to stop AI companies using your content. What this one gives you is evidence.
It also never sends noindex, nofollow or none under any setting, because those would take your pages out of ordinary Google and Bing results.
The plugin records requests that reach WordPress. Plenty never do, and rather than quietly under report, it tells you so in a banner above every figure that you cannot dismiss.
Twenty four crawlers, each one checked against the operator's own published documentation. Every entry records the page it came from, and the plugin links to it.
Three of them publish no documentation at all, and the plugin labels those as undocumented rather than pretending otherwise. Three more are documented by their own operators as fetching pages with robots.txt rules that may not apply, and those are flagged next to the setting so you find out before you rely on it.
No, and no plugin can promise that. Two of the three mechanisms are requests a crawler is free to ignore. The third is enforced at your server but matches on a user agent, which can be faked. What you get is evidence of what actually happened.
No. Neither company offers a submission form, an endpoint or an API, and any plugin that says otherwise is describing something that does not exist. The only lever you have is whether a crawler is allowed, which is why the plugin warns you before you disallow one that feeds a product linking back to its sources.
Almost certainly page caching. When a page comes from a cache, WordPress never runs, so the request is not recorded. The plugin has no way to know how much it is missing, so it says that plainly instead of guessing.
Per site. Activate it on each site individually. Network activation is refused with a message explaining why, because everything it stores belongs to one site.
For a human visitor the plugin does one lowercase and a short string comparison, with no database read at all. Recording happens after the page has been sent to the browser, and only for requests from a crawler it recognises.
Made by Azhar Ali. Free, GPL licensed, no premium tier, no locked settings and no upsell notices in your dashboard.
Questions or a bug to report? Get in touch.
