TheRealDealBot
The Real Deal is a South African deal comparison site. This page explains our crawler, so that site owners can identify it, control it, or ask us to stop.
How to identify it
Every request we make sends this as its User-Agent header:
TheRealDealBot/1.0 (+https://www.therealdeal.shop/bot; crawler@therealdeal.shop)
We send it on its own, not dressed up as a browser, including from the
headless browser we use to check some product pages. Matching on
TheRealDealBot will find every request we make.
Signed requests
Our crawler's requests are also signed with Web Bot Auth (HTTP Message Signatures, RFC 9421, Ed25519), so a site can check that a request really comes from us and is not someone borrowing our name. Signed requests carry:
Signature-Agent: "https://www.therealdeal.shop"
Our public keys are published at
/.well-known/http-message-signatures-directory.
Pages loaded in our headless browser carry the same User-Agent but are not signed.
What it does
| Requests | Public promotion, sale and clearance listings, and the public product pages they link to. On WooCommerce stores we may read a single product's public Store API record to confirm its price. We do not use Shopify's APIs, and we read a store's product feed only when the store has agreed to it. We never touch checkout, cart, account or customer data. |
|---|---|
| Frequency | One listing pass per site per night, beginning around 23:00 SAST. A site whose pass failed may be retried once the next morning. Products already listed on The Real Deal are re-checked after each pass, and early in the morning, so expired deals come off quickly. |
| Rate | One request per second per domain (one every 2.3 seconds on Shopify stores), two concurrent at most, with automatic back-off up to thirty seconds if a site responds slowly. A 403 refusal is never retried. |
| robots.txt | Fetched and obeyed before crawling, including Crawl-delay. A disallow rule is honoured without exception. |
| AI training | We do not train models on crawled content, and we honour ai-train=no content signals. |
How to control it
Any of these work, and we will not work around them:
User-agent: TheRealDealBot Disallow: /
Or set a Crawl-delay for us, or block the token at your edge or
firewall. If you would rather talk to a person, email
crawler@therealdeal.shop and we
will action a removal request without requiring a legal letter.
Prefer a feed?
We would rather take a feed than crawl you, and a growing number of retailers on the platform already send us one. A feed gives you control of exactly what is listed, accurate stock and pricing, immediate removal of expired promotions, and no load on your origin. We can consume Google Shopping XML, Merchant Center feeds, and standard JSON product feeds. Email the address above.
Listing removal
If you want your products removed from The Real Deal, email crawler@therealdeal.shop with your domain. We will disable crawling and remove existing listings, and confirm back to you once done.