Published 2026-10-19

How to Allow Shoply AI’s Crawler Through Your Firewall

A brick firewall opens one gate so Shoply's crawler can carry website pages to the chat assistant

Shoply AI keeps its answers accurate by regularly reading your website. If your site uses Cloudflare or another firewall or bot-protection service, that service may block Shoply’s crawler. When that happens, the chatbot keeps answering from an old copy of your site.

The fix is usually one rule: allow requests whose User-Agent contains ShoplyAIBot. This guide explains why the crawler is needed, how to recognize it, and how to allow it in Cloudflare and other firewalls.

Who needs to do this?

SiteChat website owners. SiteChat is Shoply’s chat and search for any website that is not a Shopify store, including Magento stores.
Agencies and developers who manage hosting, a CDN (content delivery network), or security settings for those websites.

Shopify app merchants normally don’t need to change anything. Shoply receives Shopify product data through Shopify’s API, not by crawling your storefront. However, if you added pages from a non-Shopify website as knowledge sources, such as a help center or blog hosted elsewhere, that website’s firewall could still block Shoply. See how to teach Shoply about your store for where those pages are listed.

Why does Shoply need to read my website?

Shoply reads the pages you listed in your crawl settings so that the chatbot’s answers, prices, and links match your live site.

For Magento stores, Shoply also reads your public product catalog through your store’s GraphQL endpoint (/graphql). It sends read-only product queries for details such as names, SKUs, prices, variants, and product URLs.

The crawler never logs in, never places orders, and never submits forms. To learn what Shoply does with this content, read what information Shoply uses to answer customer questions.

What happens if the crawler is blocked?

Crawls fail. Shoply cannot fetch your pages or product data.
The AI keeps answering from the last successful crawl. Shoppers may see old prices, products you have removed, and outdated or wrong links. In one real case, a store moved from a development site to its production domain, but its firewall blocked the crawler, so the chatbot kept linking shoppers to the old development site.
“Last updated … ago” stops advancing on your Shoply admin dashboard.

Common signs of a blocked crawler

In the Shoply admin, the crawl time falls behind. On the dashboard, the “Last updated at: … ago” time under Content crawled by Shoply AI keeps growing instead of resetting every few hours, for example “770 hours ago” (about a month). It stays behind even after you click Recrawl Now, because a blocked crawl never finishes.

If you check your firewall logs, the usual symptom is Cloudflare returning HTTP 403 to Shoply’s requests, in one of these forms:

Error 1010, meaning the site owner banned the request based on its browser signature. Cloudflare’s own fix for this error points to Browser Integrity Check.
A “Just a moment…” challenge page, which a crawler cannot complete.

How do I recognize Shoply’s crawler?

Every request from Shoply’s crawler uses this User-Agent:

text


ShoplyAIBot/1.0 (https://shoplyai.ai; support@shoplyai.ai; For sites installed Shoply AI Search & Chatbot)

Write your rules to match the shorter text ShoplyAIBot (for example, “User-Agent contains ShoplyAIBot”). Your rule will then keep working if the version number changes.

Allow by user agent, not by IP address. Shoply’s crawler does not use fixed IP addresses, so an IP allowlist will not work reliably.

How the crawler behaves

It only crawls the sites you configured in Shoply.
It obeys robots.txt when reading pages. If your robots.txt blocks a page, Shoply will skip it, so check that it does not block the pages you want the AI to learn from.
It is gentle on your server: at most 2 requests at a time, with a 2-second delay between requests.
Page count is capped by your Shoply plan.

When the crawler runs

Automatically on a schedule. How often depends on your plan.
When you click “Recrawl Now” in the Shoply admin.
Every 2 days while crawls keep failing. Shoply retries automatically, so once you add the allow rule, the next retry or a manual recrawl will pick up your site.

How do I allow Shoply in Cloudflare?

In Cloudflare, create a custom rule with the Skip action. A skip rule lets matching requests bypass the security checks that would otherwise block or challenge them.

Step 1: Create the skip rule

In the Cloudflare dashboard, open your website (zone) and go to Security rules.
Select Create rule > Custom rules.
In Rule name, enter something clear, such as Allow Shoply AI crawler.
Under When incoming requests match, set:
- Field: User Agent
- Operator: contains
- Value: ShoplyAIBot
If you prefer the expression editor, use:
text
```
(http.user_agent contains "ShoplyAIBot")
```
Under Then take action > Choose action, select Skip.
Choose what to skip (see the table below).
Leave Log matching requests turned on. Shoply’s requests will then appear in Cloudflare’s Security Events, which helps with troubleshooting.
Select Deploy.

Step 2: Choose what to skip

Skip option in Cloudflare	Skip it?	Why
Browser Integrity Check	Yes	Causes error 1010 for crawlers
Security Level	Yes	Stops Security Level checks from challenging the crawler
All Super Bot Fight Mode rules	Yes, on Pro plans and above	Super Bot Fight Mode can challenge or block crawlers
User Agent Blocking	If you use it	Only matters if you created user-agent blocking rules
All rate limiting rules	Only if Security Events show a rate limiting rule blocking Shoply	Shoply’s request rate is low, but a strict limit can still catch it
All managed rules	Only if Security Events show a managed rule blocking Shoply	Keeps Cloudflare’s attack-pattern rules active for everyone else
All remaining custom rules	If you have other custom rules that block bots	Stops your later custom rules from blocking Shoply

Start with the “Yes” rows. If Cloudflare’s Security Events still shows Shoply’s requests being blocked or challenged, the event tells you which feature acted. Add that feature to the skip rule.

Step 3: Put the rule first

Cloudflare evaluates custom rules in order, from first to last. If a rule above yours blocks or challenges the request, your skip rule never runs. Move the Shoply rule to the top of your custom rules list.

If you use Bot Fight Mode (Cloudflare Free plan)

Bot Fight Mode cannot be bypassed with a skip rule. Cloudflare runs it separately from custom rules, so skip and allow rules have no effect on it. You have two options:

Turn off Bot Fight Mode: go to Security > Settings, filter by Bot traffic, and turn Bot fight mode off.
Upgrade to a Pro plan or higher and use Super Bot Fight Mode, which does support skip rules.

If Under Attack mode is on

Under Attack mode shows a challenge page to every visitor, and crawlers cannot pass it. Cloudflare recommends using it only during an active attack. Shoply will not be able to read your site until it is turned off.

Is it safe to allow a user agent?

Any client can claim to be ShoplyAIBot, so this rule skips some bot checks for anyone who sends that User-Agent. To keep the risk small, skip only the features that block the crawler, and keep your login, checkout, and admin protections in place. Shoply never needs access to those areas.

How do I allow Shoply in other firewalls?

The principle is the same everywhere: allow requests whose User-Agent contains ShoplyAIBot. Exact menus differ by product, so follow your vendor’s documentation for creating an allow or exception rule. Common places to check:

Cloud web application firewalls (WAFs) such as Sucuri, AWS WAF, and Akamai: add an allow or exception rule that matches the User-Agent header.
WordPress security plugins such as Wordfence: look for bot blocking, rate limiting, or user-agent blocking settings.
Server-level filters such as ModSecurity rules, and bot blockers provided by your hosting company: whitelist the user agent, or ask your host to do it.

Magento stores: make sure the rule also covers POST requests to /graphql. Some firewalls treat API requests separately from page views. If pages are allowed but /graphql is blocked, your product catalog will not update.

Do not rely on an IP allowlist. Shoply’s crawler does not use fixed IP addresses.

How do I check that it worked?

In the Shoply admin, click Recrawl Now.
Wait for the crawl to finish, then check that “Last updated” on your dashboard has reset to a recent time.
Ask the chatbot about a product or page you changed recently, and confirm that the answer and link match your live site.

You can also run a quick check from your own computer. Replace https://your-site.com/ with your website’s address:

bash


curl -I -A "ShoplyAIBot/1.0 (https://shoplyai.ai; support@shoplyai.ai; For sites installed Shoply AI Search & Chatbot)" https://your-site.com/

A response of HTTP/2 200 (or HTTP/1.1 200 OK) means your rule lets that user agent through. A 403 means something is still blocking it.

This test runs from your computer’s IP address, not Shoply’s servers. Some firewalls treat traffic differently depending on where it comes from, so the recrawl in the Shoply admin is the final check.

Need help?

Email support@shoplyai.ai with your website address, the firewall or hosting provider you use, and a screenshot of any blocked request in your firewall’s logs. Please do not send passwords or API keys.