Schema App Robots-Only Deployment

This new feature from Schema App lets you choose whether to deploy your markup to all users or select web crawlers. It is designed to improve the performance of sites using our JavaScript deployment for normal web and mobile users while still providing your markup to search engines. This will reduce the impact of Schema App's scripts on your pages' performance for all other users. 

Robots-Only filters traffic at Schema App’s CDN using the request User-Agent. For stronger privacy and performance, many enterprises also configure their tag manager (Tealium, Google Tag Manager, Adobe Launch) so Schema App tags fire only for crawlers. See Tag manager: bots-only (Tealium and others) below.

TABLE OF CONTENTS

• How does Robots-Only Deployment work?
• What deployment methods support this?
• How do I set up Robots-Only Deployment?
• What Robots does Schema App Deploy to by Default?
• How can I test my Schema Markup once I've enabled Robots-Only Deployment?
• What if I need to add my own bots?
• How long before my updates to deployment management deploys?
• Should I do anything specific if I use a robot to pre-render my pages?
• How does Robots-Only Deployment differ from cloaking?
• Should I also restrict tags in my tag manager?
• Tag manager: bots-only (Tealium and others)
• Visitor data and regulated industries

How does Robots-Only Deployment work?

Robots-Only Deployment is configured in the Deployment Management section of your Schema App projects. No modification to your scripts is needed.

Once you have set your project to deploy to Robots Only, requests from non-bots will be filtered by Schema App's Content Delivery Network (CDN) so that the highlighter is not active on your pages for those users. This means when your page makes the normal async request for highlight.js, it will receive an empty script with no functionality which takes no further actions. For normal web users this is fine since the markup doesn't provide any utility. This means any subsequent work the highlighter would normally take will not impact the user.

Web crawlers like Google and Bing, as well as other customizable user agents, will receive the entire script and the highlighter will work as expected. This filtering is done using the user agent sent by a web browser (or crawler) with each request to the CDN. When Robots Only is enabled, the highlighter will only load and produce markup when the user agent visiting the page matches an agent in the Robots Only list. 

What deployment methods support this?

Robots-Only Deployment is only applicable to Schema App projects using JavaScript deployments (including Crawler deployment), with the exception of JavaScript with Caching. Projects using deployment methods that are not eligible for this feature will not see the option to enable it in the project set up interface.

When you deploy via Tealium, Google Tag Manager, or Adobe Launch, enable Robots Only in Schema App and use bot-only load rules in your tag manager for the strictest visitor-data posture.

How do I set up Robots-Only Deployment?

1. Log into Schema App and expand the project settings with the arrow on the right side of your project.
2. Scroll down to the Deployment Management section.
3. Select "Robots Only".
4. Save your changes.

NOTE: If you have more than one project with the same subdomain i.e. https://www.example.com and https://www.example/about then the bots only configuration of one project will apply to all projects that have a website with this subdomain. If this is case you should only configure this on one project. The configuration is based on the subdomain not at the project level so will apply to all. 

Should I also restrict tags in my tag manager?

Schema App Robots-Only is configured in Deployment Management inside app.schemaapp.com. It does not require changes to your Tealium, GTM, or Adobe Launch containers.

For customers in regulated industries (for example healthcare or financial services), or anyone who wants human browsers to avoid calling Schema App’s CDN, we recommend both:

1. Tag manager: Load rules or triggers so Schema App tags run only when the visitor User-Agent matches an allowed crawler list.
2. Schema App: Enable Robots Only under Deployment Management (backup if a tag is published to all pages by mistake).

Tag manager: bots-only (Tealium and others)

Follow your standard Schema App tag-manager integration, but replace an All Pages load rule with a User-Agent filter. Use the same bot tokens as in the default list below (and any additional bots or custom bots in Schema App).

Also enable Robots Only in Schema App Deployment Management so the CDN still blocks non-bots if a tag fires for every visitor.

Tealium iQ example

Full Tealium setup: How to: Set Up Tealium for Schema App.

1. User-Agent variable

• Go to Data Layer and confirm dom.user_agent exists (Type: DOM Variable, Source: user_agent).

2. Create the "Crawler & AI bots only" load rule

• Load Rules → + Add Rule
• Title: Schema App - Bots Only
• Condition: Variable dom.user_agent, operator matches regex (ignore case)
• Value (example—extend for your SEO tools, e.g. SEMrush):

googlebot|google-inspectiontool|bingbot|applebot|baiduspider|facebookexternalhit|linkedinbot|twitterbot|gptbot|oai-searchbot|perplexitybot|claudebot|schemabot

3. JavaScript extension and generic tag

• Add a Javascript Code extension scoped to your Schema App deployment tag, with window.schema_highlighter from your project settings (see Tealium setup article).
• Add a Tealium Generic Tag (not the legacy marketplace Schema App tag), Base URL: https://cdn.schemaapp.com/javascript/highlight.js, Type: script.

4. Apply the load rule

• On both the extension and the tag: uncheck All Pages; check Schema App - Bots Only.
• Publish to Dev/QA first.

Verification (Tealium)

You cannot verify by visiting the site as a normal user. Instead:

1. Use Chrome DevTools Network Conditions to set User-Agent to Googlebot (see testing section below)—enable Disable cache.
2. Confirm the tag fires in Tealium Web Companion.
3. Run the URL through the Schema Markup Validator or Rich Results Test.

GTM / Adobe Launch: Use an equivalent trigger (User-Agent matches RegEx or built-in bot detection). See Considerations for JavaScript Tag Deployment.

Note: Highlighter authoring in app.schemaapp.com may still require SchemaBot or IP allowlisting—separate from production site traffic.

Visitor data and regulated industries

Tag-manager bots-only deployment means most human visitors never load Schema App JavaScript or call Schema App CDN endpoints from the browser. That supports privacy reviews where organizations want to minimize third-party requests from real users.

Schema App does not record or retain end-user IP addresses in its content delivery layer. For full detail (including optional AWS WAF logging), see How Schema App handles website visitor data.

What Robots does Schema App Deploy to by Default?

By default Schema App deploys to all major search engine bots. Review the list of default Bots in the table below. 

For a list of other possible Bots to include, review this list.

How can I test my Schema Markup once I've enabled Robots-Only Deployment?

When you use the default bots that Schema App maintains, you should be able to use the Rich Result Test tool or Schema Validator as you normally would to test your markup. If you are using your own set of custom bots be sure to enable Googlebot otherwise these tools will not work. 

If you want to test locally in your browser it is possible to alter the user agent for these purposes. In Google Chrome, open the developer tools (CMD|CTRL+SHIFT+I) and navigate to the Hamburger Menu on the right side of the screen. Select "More Tools" > "Network Conditions". 

Once you click on Network Conditions, panel should appear at the very bottom of your window. Under "Network conditions" you can set the user agent. For most cases choosing Googlebot will show you the results. You can test your own user agents as well by selecting custom. Also make sure that "Disable cache" is selected so that in between user agent selection the browser doesn't cache the empty script or vice versa.

What if I need to add my own bots?

You may have other SEO tools that use bots to report on schema markup. If you'd like to include these bots, Schema App provides an interface to do so, however it is only recommended for advanced users. This configuration is only available by going to https://app.schemaapp.com/home?customBots=true. This URL will show a third Deployment Management option:  "Custom Robots". The default list contains bots Schema App expects to support. You can prevent or allow other bots (non-matches are excluded by default). 

Add your own bot by providing the name of the bot and the user agent Schema App should match with. The name must appear in the user agent or it will not be matched. This is required because some bots have multiple user agents but typically contain the same name, this helps reduce duplication on the Schema App side. 

How long before my updates to deployment management deploys?

This can take about 1 hour to fully deploy so you might still see the CDN deploying to all user agents for a little while. This is normal since the changes need to roll out to all locations within the CDN. After an hour, you should see the markup only deploying to the bots that are registered. 

Should I do anything specific if I use a robot to pre-render my pages?

Yes! If your pre-rendering bot uses a specific user agent, you will want to add that to the list of bots by navigating to https://app.schemaapp.com/home?customBots=true and adding that information to the list of rules. If you are not sure of your specific pre-rendering user agent, you may want to consult with your CSM before implementing Robots Only deployment.

The other import aspect is making sure your pre-rendering solution is aware when highlight.js is done rendering. To do this you can leverage the final event highliight.js emits. The event is called highlightJsOnComplete and it is emitted after the final pagecount call is made. 

This will vary based on your pre-render solution but many will allow you to set a flag or trigger another event to let the pre-render solution it is complete. Below is an example with prerender.io that explains how you could integrate it. Doing it using events as opposed to a timeout is for accuracy, the amount of time will vary based on the complexity of pages since the highlighter relies on other resources and it could take longer in some cases. Responding to the event ensures all the necessary actions have been performed. 

// Outside the event
window.prerenderReady = false;

window.addEventListener('highlightJsOnComplete', (e) => {
        // either set the value to true or notify another process this is complete. 
        window.prerenderReady = true;
});

How does Robots-Only Deployment differ from cloaking?

Google Search Central's spam policy defines cloaking as presenting different content to users and search engines with the intent to manipulate rankings or mislead users. Typical examples include showing unrelated or misleading content to bots compared to human visitors. Additionally, Google's guidelines on dynamic rendering clarify that as long as similar content is served to both users and crawlers, it is not considered cloaking.

Robot-Only Deployment is a form of dynamic rendering, and given the above definitions, does not qualify as cloaking:

• The content served to bots is a machine-readable translation of what is already visible to users, not different or misleading information.
• There is no attempt to manipulate rankings or conduct bait-and-switch tactics; the schema markup simply makes the existing content more accessible for search engines.
• Human users are not presented with different information—they just don’t receive the schema markup, which is unnecessary for their experience and reduces impact on page performance.