XML sitemap generator that leaves the wrong URLs out

Yes, though not for the reason most people think. A sitemap does not make Google index a page and it does not improve rankings. What it does is fix discovery for URLs your internal linking does not reach well — new pages, deep pages, pages with few internal links — and give you a clean denominator in Search Console, where "submitted vs indexed" is one of the few honest coverage signals you get. On a well-linked twenty-page site the benefit is close to zero. On a thousand-page site it is substantial.

Because a sitemap is a list of URLs you are asking Google to index, and asking it to index something you have told it not to index is a contradiction it notices. We drop four categories: URLs that do not return 200, URLs that redirect, URLs carrying noindex in the robots meta tag or the X-Robots-Tag header, and URLs whose canonical tag points somewhere else. Every exclusion is listed with its reason on the Excluded tab, so you can check our reasoning rather than trust it.

Always, and it is not optional. We fetch your robots.txt before the first page and filter every discovered URL through it using the same RFC 9309 engine behind our robots.txt Tester. URLs your own rules disallow are recorded as blocked and never fetched. A tool that ignored the target site's crawl rules would be indefensible regardless of who was running it.

Five hundred pages per crawl. The crawl runs from our edge network at a deliberately moderate rate, and an unbounded free crawler pointed at someone else's server is a tool that eventually gets used as a weapon. For sites larger than that, use paste mode with an export from your CMS or a desktop crawler — it handles up to 50,000 URLs per file and writes a sitemap index automatically past that.

No, which is why both default to omitted. Google has stated plainly that it ignores both. They are legal in the protocol and harmless, but they add bytes and create a false sense that you are steering something. Lastmod is different — Google does use it, but only for as long as it stays truthful. A sitemap that stamps every URL with today's date teaches Google to ignore your lastmod entirely, which costs you the one signal in the file that was working.

It is a proposed markdown file at your site root that tells AI assistants what your site is and which pages are worth reading, rather than making them infer it from your HTML. It is a convention, not a standard — no AI engine currently commits to reading it, and anyone promising otherwise is selling something. It costs a few minutes to publish and it makes your site legible to anything that does look. We generate one alongside the sitemap for that reason, not because it is a ranking factor.

Upload sitemap.xml to your site root so it lives at yourdomain.com/sitemap.xml, add the Sitemap: line we generate to your robots.txt, and submit the URL in Google Search Console under Indexing → Sitemaps. Search Console is where the value is: it tells you how many of the submitted URLs actually got indexed, and the gap between those numbers is usually the most useful technical SEO diagnostic on the site.

Usually one of four reasons. The pages are only reachable through JavaScript navigation, which this crawler does not execute — and neither do most AI crawlers, so that is worth knowing. They are orphaned, with no internal link pointing at them. They are behind a login. Or robots.txt disallows the path they live under. Check the Excluded tab first; if the missing pages are not there, they were never discovered, and orphaned pages are a real problem the sitemap only papers over.