What Custom Bots should I add to my Robots-only deployment?

Modified on Fri, 26 Jul at 2:33 PM

You're looking to add some additional bots but don't want to look for all the information? You've come to the right place.

Here is a list (which we will continue to add to) of bots you may be interested in adding to your Robots-Only deployment. This link provides instructions for enabling Robots-Only deployment, and customizing the list of robots.

General Bots

Bot Name (User Agent Token)	Full User Agent String
360Spider	Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
AhrefsBot	Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
AOLBuild	Mozilla/5.0 (compatible; MSIE 9.0; AOL 9.7; AOLBuild 4343.19; Windows NT 6.1; WOW64; Trident/5.0; FunWebProducts)
Ask Jeeves/Teoma	Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
botify	Mozilla/5.0 (compatible; botify; http://botify.com)
ContentKing	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com)
DuckDuckBot	Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)
Baiduspider	Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Qwantify	Mozilla/5.0 (compatible; Qwantify/Bleriot/1.2.1; +https://help.qwant.com/bot)
SemrushBot	Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
Daum	Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
SeznamBot	Mozilla/5.0 (compatible; SeznamBot/4.0; +http://napoveda.seznam.cz/seznambot-intro/)
Screaming Frog SEO Spider	Screaming Frog SEO Spider/5.1
LinkedInBot	LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
Sogou web spider	Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Twitterbot	Twitterbot/1.0
ichiro	ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
Yeti	Mozilla/5.0 (compatible; Yeti/1.1; +https://naver.me/spd)

AI Bots

There are also a number of AI search engines and chat bots using their own crawlers to collect data for training purposes.

Bot Name (User Agent Token)	Full User Agent String
ChatGPT-User	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot)
Diffbot	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
GPTBot	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
OAI-SearchBot	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)
PerplexityBot	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Note that while there are sometimes multiple different full user agent strings for a given Bot, we only need one to be added for the custom bots' deployment to work.

If you have additional suggestions for this list, please don't hesitate to reach out to [email protected].