Robots.txt is a small text file that sits on every website. It tells search engines and bots what they’re allowed to see and what they’re not, working like a digital “do not enter” sign. In the early ...