Recurring revenue: what changes when your business doesn't start from zero every month
What recurring revenue is, what MRR and ARR mean without the jargon, and why thinking in recurring terms changes how you run a business.
Leer más →Building websites with AI makes the visible part fast. The invisible part hasn't changed: our own site was answering ChatGPT with a 404, and nobody had noticed.
Your website can say "here I am" to a person and "nothing here" to ChatGPT.
At the same time. Same address.
Ours did. So did this blog.
A few days ago we asked our server provider to stop blocking the AI crawlers: the ones from ChatGPT, Perplexity and Claude. The idea was simple. If someone asks an assistant something, we want it to be able to read our site before it answers. Nothing more exotic than that.
Quick, friendly reply: you can switch that on from the panel. Plus a fair warning: some of those bots crawl like there's no tomorrow to train their models, and that puts load on the server.
We switched it on. We hit the "check again" button. The check still said no.
Requested as a browser, the server answered 200: all good. Requested as ChatGPT's search crawler, it answered 404: nothing here. The block wasn't in the website. It was in front of it, in a server rule that neither the site nor we could see.
We wrote back. That same afternoon, support found it: the crawler was still on the blocked list. They removed it. Done.
Done, plus one awkward question: how long had it been like that?
I don't know. That's the part that interests me.
If you search for how to build a website with AI, you'll mostly find builders that put together a template in minutes from a short description. They work. For a simple page, they get you out of trouble. Most guides admit the small print: you still have to customise it yourself, and your freedom is whatever the builder allows.
What I see every day is something else: a developer who knows what they're doing goes several times faster with AI. Forms, dashboards, screens, placeholder copy. That's why I wrote that custom development is no longer a luxury, and that code is getting cheaper, but judgement isn't.
Can you build a website with AI? Yes. No argument.
The better question is a different one: who checks what the AI can't see?
A website isn't just code. It's the server it lives on, the security rules standing in front of it, the certificate, the backups, the person who answers when something breaks. I said it when I wrote about cheap hosting: what sits underneath ends up in the shop window.
The AI that writes your page doesn't know what rule your server has against bots. It has never seen it. It has no reason to. And you, looking at the site in your browser, don't see it either: to you it answers 200. All green.
That's the kind of failure that worries me most. Not the one that breaks the site. The one that leaves it perfect for you and invisible to someone else.
A website that can't be read doesn't exist, however pretty the AI made it.
That's the other question that comes up when you search for this. My view: Google doesn't care who wrote the code. It cares about being able to read the page, about it loading, and about it answering what people are looking for. A site built with AI can rank just as well, or just as badly, as one built by hand.
Assistants work the same way, only more bluntly. If their crawler can't read your site, they can't cite it when someone asks. There's no position eight. There's nothing.
And there's a decision here worth taking calmly, because not all crawlers do the same job:
We also learned something as a bonus: that setting applied to every site on that server, not just ours. We asked afterwards. Better to ask before.
If you have a website, the check takes a minute: request it the way a crawler does, not only the way a browser does. I can't tell you how much time it would have saved us. Not knowing is precisely the problem.
Build the website with AI. Really, do. It's faster and cheaper than ever.
Then ask someone what's standing in front of it.
P.S. The funny part: to tell this story on a blog that wants AIs to read it, we first had to get the AIs able to read the blog.
Comments
No comments yet. Want to start?