Our page fetcher

How the OpenShopGraph fetcher reads your shop, and how to exclude it

OpenShopGraph reads publicly reachable shop pages (shipping, returns, payment, imprint) to publish source-backed shop facts. This page describes our fetcher so you can recognise it and decide about it.

Who we are

OpenShopGraph is an open, source-backed shop-data project. More about the project is on the About page. You reach us at info@openshopgraph.com.

What the fetcher does

It identifies itself as osg-shop-enrichment/0.1 and points to this page. It only reads public pages of a shop over HTTPS: home page, imprint, shipping, returns, payment, terms and FAQ. It requests one page at a time with a pause of at least one second between requests, reads at most 3 MB per page, does not log in, and does not solve CAPTCHAs or work around bot protection. Where a page needs a browser to render, it is loaded with a headless browser from the same servers.

robots.txt

Before reading a page, the fetcher reads your robots.txt and follows the Disallow rules for osg-shop-enrichment and for all agents. One exception applies to the browser fallback: a few public, customer-facing policy pages (shipping, returns, terms) may still be read there when a platform disallows the whole policy folder by default, because shoppers need these facts. If you do not want that either, tell us and we exclude your domain.

Exclude your shop

Email info@openshopgraph.com with your shop domain and the subject "Exclude". We will stop fetching that domain. You can also disallow osg-shop-enrichment in your robots.txt.