Firecrawl crawls web pages and returns them clean, as Markdown or structured data, ready for a language model. What it adds, and the responsibilities that come with it.
A web page is easy for a person to read.
For a language model it's a mess of menus, ads, scripts and cookie banners.
Firecrawl's job is to clean up that mess.
Firecrawl is a project that takes a web address (or a whole website) and returns it in a format a language model understands well: clean Markdown or structured data. It can read a single page, go through every page on a site, map its links or extract specific fields from a description.
It's offered as a paid service and the code is published under the AGPL licence, so you can also run it yourself. By default it respects what each site says in its robots.txt file.
For the same reason MarkItDown interests me with documents: what AI produces depends on how clean the input is.
If you want an assistant to answer questions about your product documentation, compare public prices or summarise what your competitors say on their sites, you first have to turn those pages into useful text. Doing it by hand for every site takes forever. Programming an extractor for each one does too.
AI doesn't replace the work of preparing information. It just makes it more obvious when you skip it.
Being able to read a website doesn't mean you can do anything you like with its content.
Many sites forbid crawling in their terms. There's copyrighted content and personal data you shouldn't collect without a legal basis. And crawling someone else's site aggressively costs money for whoever runs it.
My rule is simple: if you wouldn't be comfortable explaining to the site owner what you're doing with their content, don't do it.
That many of the best AI tools aren't models. They're pipes: pieces that carry information from where it lives to where it can be used, clean and organised.
It's unglamorous work. And it's exactly the work you notice most when it's missing.
P.S. The best place to try a tool like this is your own website. It'll tell you a lot about how clear it is (or isn't).