Most of the manuals I wanted were never PDFs. Apple’s Final Cut Pro guide is 501 separate web pages. A vendor help centre is a Zendesk site. I had 89 of them saved by hand with no record of where any one came from, so nothing could tell me which had been revised since, or go and get a fresh copy.
What it does
Turn a 500-page web manual into one file
pagespring ingest pointed at Apple’s Final Cut Pro guide walks all 501 topic pages and writes one HTML file. Site chrome is gone and the headings follow the guide’s own table of contents. No saving pages by hand, no scraper written for that one site.
Point it at a URL without knowing the platform
Thirteen patterns cover Apple guides, GitBook, Read the Docs, Zendesk, Microsoft and Adobe help, OpenStax, PDFs, zip/tar/epub bundles and OpenAPI specs. Anything else is content-probed at fetch time. A site nothing recognizes exits 2 and names what it probed rather than staging junk. classify <url> shows the routing with no fetch.
Find out which manuals changed, not all of them
refresh --all re-checks every staged manual against the URL it came from and prints changed / unchanged / failed / skipped, one line per slug. A PDF whose server answers 304 costs one request and no download. A dead source fails on its own line, the sweep continues, and the old file stays.
Catch a broken crawl before you index it
A crawl can finish clean and still be wrong. The llms.txt pattern once staged a single page for a site with 170. audit --all --strict catches that, along with files that no longer hash to what was recorded, image references pointing at nothing, a crawl that stopped dead on the 6000-page cap, and two slugs pulled from the same URL. It runs offline. Wire it as audit --all --strict && <hand off> and a bad crawl never reaches the indexer.
Re-clean a crawl without re-crawling the site
Ingest once with --keep-raw, then renormalize <slug> runs your current cleanup rules over the pages already on disk. No network, so you can iterate on extraction against a real crawl. Byte-identical output re-stages nothing and prints unchanged, which is how you know an edit changed no behavior.
Pull a textbook's 1,100 images over several runs
localize <slug> fetches a staged file’s images and re-points its references without re-crawling the site. Each reference is fixed the moment its image lands and progress is checkpointed every 50, so a run you kill keeps what it got and the next one skips it. OpenStax Chemistry 2e came to 1,148 images.
What that looks like in practice
Two jobs, and they are the whole tool: get the manual no matter what shape the manufacturer published it in, and notice when they change it.
refresh --all tells you which of your manuals changed and re-fetches only those.Where it sits
pagespring is the front of a three-stage pipeline that ends in local search. It acquires; pagespeak converts; QMD indexes. Each stage writes files and stops — nothing calls the next one, so any of them can be re-run alone.
That separation is what makes the search good rather than merely possible. A manual that arrives as 501 separate pages has no usable structure: navigation chrome on every page, headings that restart, no way to tell which page a passage came from. pagespring normalises it into one document with its headings intact, which is the thing pagespeak needs to cut sections on hierarchy and stamp each one with a breadcrumb. QMD then retrieves a section that knows which manual and which chapter it came from, instead of a fragment that could be from anywhere.
incoming/<slug>/ and stops. pagespeak reads that, writes sections, and stops before
embeddings. QMD indexes the sections and answers queries locally. Any stage can be re-run without the
others — which is why renormalize and refresh exist.