分享本人修改过的robots.txt 减少蜘蛛爬取动态页面

先上图：百度蜘蛛爬取得路径


不浪费爬虫的重复爬行
#
# robots.txt for Discuz! X3
#

User-agent: *
Disallow: /api/
Disallow: /data/
Disallow: /source/
Disallow: /install/
Disallow: /template/
Disallow: /config/
Disallow: /uc_client/
Disallow: /uc_server/
Disallow: /static/
Disallow: /admin.php
Disallow: /search.php
Disallow: /member.php
Disallow: /api.php
Disallow: /misc.php
Disallow: /connect.php
Disallow: /forum.php?mod=redirect*
Disallow: /forum.php?mod=post*
Disallow: /home.php?mod=spacecp*
Disallow: /userapp.php?mod=app&*
Disallow: /*?mod=misc*
Disallow: /*?mod=attachment*
Disallow: /*mobile=yes*
Disallow: /forum.php?mod=forumdisplay&fid=*&filter=hot
Disallow: /home.php?mod=space&uid=*&do=profile
Disallow: /home.php?mod=magic&mid=*&idtype=tid&id=*
Disallow: /home.php?mod=space&uid=*&do=thread&type=thread&view=me&from=space
Sitemap: http://www.52zutuan.com/sitemap.html
Sitemap: http://www.52zutuan.com/sitemap.xml

支持下分享。。。

谢谢分享。。。