Robots.txt
Start Here
When should you use robots.txt, noindex, canonicals, or authentication to control search access?
Robots.txt, meta robots, canonicals, and access controls are often used interchangeably even though they solve different problems. What is the decision framework for choosing the right control?
Read the cornerstone solution Open Technical SEO Community