Skip to content
LumetLumet home

Methodology

What this scan is, and is not

Lumet v1 is a public, account-free fetch of one URL. It does not log in, execute JavaScript, crawl the rest of the site, or ask ChatGPT whether your brand is mentioned. The report is a reading of the document a non-JS client received.

Fetch

The server requests the URL over HTTP(S) with a 12 second timeout and a 1.5 MB cap. Redirects are followed only after each hop is checked. Private, loopback, link-local, and cloud-metadata addresses are refused. The User-Agent is Lumet/1.0 (+https://lumet.io/methodology).

robots.txt

GET /robots.txt on the final origin. Groups are parsed for User-agent, Allow, Disallow, and Sitemap. Lumet then asks whether the scanned path is allowed for * and for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. A missing file is treated as default-allow, which is how most crawlers behave. Longest matching rule wins; Allow ties beat Disallow. This is a practical parser, not a claim of bit-identical Googlebot behavior.

JS-less render

The HTML is parsed as received. script, style, template, iframe, object, and embed are removed. Remaining body text is counted. Fewer than 15 words is treated as an empty shell. A short page that still has an H1 is not. We do not run a headless browser, Crawl4AI, or any embedding index.

JSON-LD

application/ld+json blocks are parsed as JSON. @type values are collected, including nodes under @graph. Lumet looks for common Schema.org types (Organization, WebSite, SoftwareApplication, Product, LocalBusiness, Article, and related). Invalid JSON is skipped, not repaired.

Metadata and headings

title, meta description, rel=canonical, Open Graph, Twitter tags, and meta robots (including noindex) are read from the static HTML. Headings h1–h3 are outlined; empty or multiple H1s are noted. Counts are real. There is no 0–100 score.

Sitemap

If robots.txt lists Sitemap: URLs, those are fetched first (up to three). Otherwise GET /sitemap.xml. A 200 response that contains urlset or sitemapindex counts as found. Lumet does not download every loc.

Verdict

Invisible: the document failed, crawlers are told to skip this path, meta robots says noindex, there is no title, or the JS-less text is an empty shell. Partly invisible: a named AI crawler is disallowed, or the static text is thin without an H1. Visible: the HTML has readable text and is not blocked. Warnings (missing schema, Open Graph, sitemap) stay on the report without automatically downgrading the verdict.

Out of scope

  • No JavaScript execution, screenshots, or visual diff.
  • No ChatGPT / Claude / Perplexity prompt queries (“shadow queries”).
  • No vector database, pgvector, or generated content studio.
  • No site-wide crawl, auth, or billing in this version.

If a check cannot be done reliably, Lumet omits it rather than faking a widget. The shareable /scan/[id] URL stores the report JSON on the instance that produced it (local disk, /tmp on Vercel, or Vercel Blob when BLOB_READ_WRITE_TOKEN is set).