zoukankan      html  css  js  c++  java
  • 网站迁移服务器后CPU、内存飙升,设置robots.txt 问题

    User-agent: SemrushBot
    Disallow: /
    User-agent: SemrushBot-SA
    Disallow: /
    User-agent: SemrushBot-BA
    Disallow: /
    User-agent: YandexBot/3.0
    Disallow: /
    User-agent: coccocbot-web/1.0
    Disallow: /
    User-agent: linkdexbot/2.0
    Disallow: /
    User-agent: DotBot/1.1
    Disallow: /
    User-Agent: YisouSpider
    Disallow: /
    User-Agent: MJ12bot
    Disallow: /
    User-Agent: BOT
    Disallow: /
    User-Agent: CrawlDaddy
    Disallow: /
    User-Agent: ApacheBench
    Disallow: /
    User-Agent: Swiftbot
    Disallow: /
    User-Agent: AhrefsBot
    Disallow: /
    User-Agent: ZmEu
    Disallow: /
    User-Agent: WinHttp
    Disallow: /
    User-Agent: EasouSpider
    Disallow: /
    User-Agent: HttpClient
    Disallow: /
    User-Agent: YYSpider
    Disallow: /
    User-Agent: jaunty
    Disallow: /
    User-Agent: oBot
    Disallow: /
    User-Agent: Linguee Bot
    Disallow: /
    User-Agent: Bytespider
    Disallow: /
    User-Agent: BLEXBot
    Disallow: /
    User-Agent: CompSpyBot
    Disallow: /
    User-Agent: Exabot
    Disallow: /
    User-Agent: ZoominfoBot
    Disallow: /
    User-Agent: ExtLinksBot
    Disallow: /
    User-Agent: AlphaBot
    Disallow: /
    User-Agent: perl
    Disallow: /
    User-Agent: Wget
    Disallow: /
    User-Agent: ZmEu
    Disallow: /
    User-Agent: Python
    Disallow: /
    User-Agent: mail.RU
    Disallow: /
    User-Agent: ApacheBench
    Disallow: /
    User-Agent: Swiftbot
    Disallow: /
    User-Agent: AhrefsBot
    Disallow: /
    User-Agent: ZmEu
    Disallow: /
    User-Agent: WinHttp
    Disallow: /
    User-Agent: EasouSpider
    Disallow: /
    User-Agent: HttpClient
    Disallow: /
    User-Agent: YYSpider
    Disallow: /
    User-Agent: jaunty
    Disallow: /
    User-Agent: oBot
    Disallow: /
    User-Agent: Linguee Bot
    Disallow: /
    User-Agent: Bytespider
    Disallow: /
    User-Agent: BLEXBot
    Disallow: /
    User-Agent: CompSpyBot
    Disallow: /
    User-Agent: Exabot
    Disallow: /
    User-Agent: ExtLinksBot
    Disallow: /
    User-Agent: AlphaBot
    Disallow: /
    User-Agent: perl
    Disallow: /
    User-Agent: Wget
    Disallow: /
    User-Agent: ZmEu
    Disallow: /
    User-Agent: Python
    Disallow: /
    User-Agent: mail.RU
    Disallow: /
    User-Agent: Go-http-client
    Disallow: /

    User-agent: *
    Disallow: /admin/
    Disallow: /adminlogin/
    Disallow: /log/
    Disallow: /update/
    Disallow: /history/
    Disallow: /test/
    Disallow: /data/

    都是一些无效的爬虫访问

  • 相关阅读:
    Fix RICHTX32.OCX Issue on Windows 7
    Solved: c:\windows\system32\config\systemprofile\desktop
    递归的现实应用
    Win 8 app 获取窗口的宽度和高度, 本地化, 及文本读取
    均匀设计U Star 665
    GridView.ScrollIntoView() doesn't work
    XML节点访问与更新
    Keyboard supports in Software Testing
    WordPress程序备受喜爱的原因:十八般武艺
    paypal注册教程(PP注册教程)paypal使用方法
  • 原文地址:https://www.cnblogs.com/zhian/p/12967875.html
Copyright © 2011-2022 走看看