The website’s robots.txt tells every crawler not to visit any page. Search engines stop crawling it, and its pages gradually fall out of search results.
What it means
robots.txt is the file at the root of the website that tells crawlers where they may go. A group for all crawlers (User-agent: *) with Disallow: / shuts them out of the whole website.
What triggers it
Once a day, as part of Uptime monitoring, SaaSMaestro reads the website’s robots.txt and looks for that rule. Finding it raises a Critical issue. If robots.txt can’t be read that day, the last result is kept. Any change to robots.txt is also added to the timeline as a note, with the lines added and removed.
Common causes
- A staging or development robots.txt copied to the live website.
- A CMS or plugin setting that hides the website from search engines.
- A new website launched with the pre-launch block still in place.
How to fix it
- Open /robots.txt on the website and find the “Disallow: /” line under “User-agent: *”.
- Remove it, or change it to block only the folders that should stay private, unless the whole website is meant to be hidden.
- Check the timeline note to see when it changed, and make sure your deploy process won’t put it back.
- In Search Console, open the robots.txt report to have Google fetch the new file.
When it clears
The issue goes at the next daily check that reads a robots.txt without the rule.
After 7 days without the problem, the action drops off the board. If you marked it done and the problem is still detected 14 days later, it returns as “Fix didn’t work”.