Smart Tools
博客文章
Theme
Search
User login
  • 全局设置

  • 限制目录

  • Sitemap

  • 国内搜索引擎

  • 国外搜索引擎

开始生成 复制 清空
What is a robots.txt file:

1. robots.txt (always lowercase) is a text file stored in the root directory of a website. It typically tells web search engine spiders which content on the site can be indexed and which cannot.

2. The filename of robots.txt should be in all lowercase. robots.txt should be placed in the website’s root directory

3. If you want to define search engine crawler behavior for subdirectories separately, you can merge the custom settings into the robots.txt file in the root directory

4. The robots.txt protocol is not a formal standard but rather a convention, so it does not guarantee website privacy

5. Note that robots.txt uses string comparison to determine whether to crawl a URL, so the presence or absence of a trailing slash “/” in a directory path represents different URLs

robots.txt file content

1. Whether search engine spiders can access or crawl the site

2. Accessibility of directories or files for search engine spiders

3. Definition of the sitemap path

4. Limits on the crawl frequency of search engine spiders

About the robots.txt Generator

1. Configure the desired settings via the web interface, then click "Generate" to create the robots.txt file

2. Create a blank text file named "robots.txt," then copy and paste the content above into it

3. Place the "robots.txt" file in your website's root directory and verify that search engine spiders can access it

Recommended Tools
Home Search Favorites Language