How to Create file | What is | What are the main points to Create

A file tells search engine crawlers what URLs the crawler can access on your site. This is mainly used to avoid overloading your site with requests; This is not a system to keep any web page out of Google. To keep a web page away from Google, block indexing with Noindex or password protect the page. A file can be found at the base of your website. For example, for the website , the file is located at The file is a plain text file that follows the robots exclusion standard. The file contains one or more rules. Each rule restricts or allows access to a single file location on the domain or subdomain where the file is hosted. A) Here is a simple file with two rules:- User-agent: Googlebot Disallow: /nogooglebot/ User-agent: * Allow: / Sitemap: B) Basic guidelines for creating a file There are four