What Custom Bots should I add to my Robots-only deployment?

Modified on Thu, 18 Jul at 12:40 PM

You're looking to add some additional bots but don't want to look for all the information? You've come to the right place. 


Here is a list (which we will continue to add to) of bots you may be interested in adding to your Robots-Only deployment. This link provides instructions for enabling Robots-Only deployment, and customizing the list of robots. 


Bot Name (User Agent Token)Full User Agent String
360Spider
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
AhrefsBot
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
AOLBuild
Mozilla/5.0 (compatible; MSIE 9.0; AOL 9.7; AOLBuild 4343.19; Windows NT 6.1; WOW64; Trident/5.0; FunWebProducts) 
Ask Jeeves/Teoma
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
botify
Mozilla/5.0 (compatible; botify; http://botify.com)
ContentKing
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com)

Diffbot
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)

DuckDuckBot

Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)
GPTBot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
Baiduspider
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
QwantifyMozilla/5.0 (compatible; Qwantify/Bleriot/1.2.1; +https://help.qwant.com/bot)
SemrushBot
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
DaumMozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
SeznamBotMozilla/5.0 (compatible; SeznamBot/4.0; +http://napoveda.seznam.cz/seznambot-intro/)
Screaming Frog SEO Spider
Screaming Frog SEO Spider/5.1
LinkedInBot
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
Sogou web spiderSogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Twitterbot
Twitterbot/1.0
ichiro
ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
PerplexityBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Yeti
Mozilla/5.0 (compatible; Yeti/1.1; +https://naver.me/spd)



Note that while there are sometimes multiple different full user agent strings for a given Bot, we only need one to be added for the custom bots' deployment to work.


If you have additional suggestions for this list, please don't hesitate to reach out to support@schemaapp.com


Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select at least one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article