What Custom Bots should I add to my bots only deployment?

Modified on Fri, 31 May 2024 at 05:01 PM

You're looking to add some additional bots but don't want to look for all the information? You've come to the right place. 


Here is a list (which we will continue to add to) of bots you may be interested in adding to your Bots Only deployment. This link provides instructions about how to do this. 


Bot Name (User Agent Token)Full User Agent String
ContentKingMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com)
AOLBuild
Mozilla/5.0 (compatible; MSIE 9.0; AOL 9.7; AOLBuild 4343.19; Windows NT 6.1; WOW64; Trident/5.0; FunWebProducts)
AhrefsBot
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
Ask Jeeves/Teoma
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
botify
Mozilla/5.0 (compatible; botify; http://botify.com)
Screaming Frog SEO SpiderScreaming Frog SEO Spider/5.1

DuckDuckBot

Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)
GPTBot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
Baiduspider
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
QwantifyMozilla/5.0 (compatible; Qwantify/Bleriot/1.2.1; +https://help.qwant.com/bot)
SemrushBot
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
DaumMozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
SeznamBotMozilla/5.0 (compatible; SeznamBot/4.0; +http://napoveda.seznam.cz/seznambot-intro/)
360SpiderMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
LinkedInBot
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
Sogou web spiderSogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Twitterbot
Twitterbot/1.0
ichiro
ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
PerplexityBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Yeti
Mozilla/5.0 (compatible; Yeti/1.1; +https://naver.me/spd)


Note that while there are sometimes multiple different full user agent strings for a given Bot, we only need one to be added for the custom bots' deployment to work.


If you have additional suggestions for this list, please don't hesitate to reach out to support@schemaapp.com


Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select atleast one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article