Crawler / audit evidence
ILCrawler
Mixed-language crawler/audit platform for turning raw crawl, rendered, Lighthouse, resource, issue, and export evidence into reviewable SEO handoff data.
- runs the high-volume raw path through Go workers with robots.txt, sitemap, URL normalization, frontier state, and NDJSON spool batches
- uses a Rust importer to bulk-load raw crawl facts into PostgreSQL before Go post-processing creates derived data
- keeps Node.js Lighthouse, Go rendered audits, Next.js UI, exports, issue workflow, and operations controls visible enough to operate