zoukankan      html  css  js  c++  java
  • 网站迁移服务器后CPU、内存飙升,设置robots.txt 问题

    User-agent: SemrushBot
    Disallow: /
    User-agent: SemrushBot-SA
    Disallow: /
    User-agent: SemrushBot-BA
    Disallow: /
    User-agent: YandexBot/3.0
    Disallow: /
    User-agent: coccocbot-web/1.0
    Disallow: /
    User-agent: linkdexbot/2.0
    Disallow: /
    User-agent: DotBot/1.1
    Disallow: /
    User-Agent: YisouSpider
    Disallow: /
    User-Agent: MJ12bot
    Disallow: /
    User-Agent: BOT
    Disallow: /
    User-Agent: CrawlDaddy
    Disallow: /
    User-Agent: ApacheBench
    Disallow: /
    User-Agent: Swiftbot
    Disallow: /
    User-Agent: AhrefsBot
    Disallow: /
    User-Agent: ZmEu
    Disallow: /
    User-Agent: WinHttp
    Disallow: /
    User-Agent: EasouSpider
    Disallow: /
    User-Agent: HttpClient
    Disallow: /
    User-Agent: YYSpider
    Disallow: /
    User-Agent: jaunty
    Disallow: /
    User-Agent: oBot
    Disallow: /
    User-Agent: Linguee Bot
    Disallow: /
    User-Agent: Bytespider
    Disallow: /
    User-Agent: BLEXBot
    Disallow: /
    User-Agent: CompSpyBot
    Disallow: /
    User-Agent: Exabot
    Disallow: /
    User-Agent: ZoominfoBot
    Disallow: /
    User-Agent: ExtLinksBot
    Disallow: /
    User-Agent: AlphaBot
    Disallow: /
    User-Agent: perl
    Disallow: /
    User-Agent: Wget
    Disallow: /
    User-Agent: ZmEu
    Disallow: /
    User-Agent: Python
    Disallow: /
    User-Agent: mail.RU
    Disallow: /
    User-Agent: ApacheBench
    Disallow: /
    User-Agent: Swiftbot
    Disallow: /
    User-Agent: AhrefsBot
    Disallow: /
    User-Agent: ZmEu
    Disallow: /
    User-Agent: WinHttp
    Disallow: /
    User-Agent: EasouSpider
    Disallow: /
    User-Agent: HttpClient
    Disallow: /
    User-Agent: YYSpider
    Disallow: /
    User-Agent: jaunty
    Disallow: /
    User-Agent: oBot
    Disallow: /
    User-Agent: Linguee Bot
    Disallow: /
    User-Agent: Bytespider
    Disallow: /
    User-Agent: BLEXBot
    Disallow: /
    User-Agent: CompSpyBot
    Disallow: /
    User-Agent: Exabot
    Disallow: /
    User-Agent: ExtLinksBot
    Disallow: /
    User-Agent: AlphaBot
    Disallow: /
    User-Agent: perl
    Disallow: /
    User-Agent: Wget
    Disallow: /
    User-Agent: ZmEu
    Disallow: /
    User-Agent: Python
    Disallow: /
    User-Agent: mail.RU
    Disallow: /
    User-Agent: Go-http-client
    Disallow: /

    User-agent: *
    Disallow: /admin/
    Disallow: /adminlogin/
    Disallow: /log/
    Disallow: /update/
    Disallow: /history/
    Disallow: /test/
    Disallow: /data/

    都是一些无效的爬虫访问

  • 相关阅读:
    node.js是什么
    python基础 filter ,列表,字典,集合 中根据 条件 筛选 数据
    nginx 自动补全www,当不输入www时候自动补全www
    python爬虫,接口是post请求,参数是request payload 的形式,如何传参
    python使用with开启线程锁
    linux nohup后台执行脚本并指定文件输出 ,nohup 修改默认日志输出文件
    python线程锁
    nginx yum安装启动
    redis desktop manager 远程连接服务器上的redis
    职位列表中英对照
  • 原文地址:https://www.cnblogs.com/zhian/p/12967875.html
Copyright © 2011-2022 走看看