User-Agent: wget Disallow: / User-agent: libwww Disallow: / User-agent: grub-client Disallow: / User-agent: WebReaper Disallow: / User-agent: k2spider Disallow: / # Some bots are known to be trouble, particularly those designed to copy # entire sites. Please obey robots.txt. User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: AhrefsBot disallow: / User-Agent: SputnikBot Disallow: / # Misbehaving: requests much too fast: User-agent: fast Disallow: / User-agent: dotbot Disallow: / User-agent: Baiduspider Disallow: / User-agent: grapeshot Disallow: / User-agent: MJ12bot Disallow: / User-agent: Cliqzbot Nicht erlauben: / Disallow: / User-agent: Taboolabot Disallow: / User-agent: BLEXBot Disallow: / User-agent: SMTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Riddler Disallow: / User-agent: SemrushBot Disallow: / User-agent: spbot Disallow: / User-agent: megaindex.com Disallow: / User-agent: HealthSumBot Disallow: / User-agent: msnbot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: UCBrowser Disallow: / User-agent: Yeti Disallow: / User-agent: YisouSpider Disallow: / User-agent: PetalBot Disallow: / User-agent: Go-http-client Disallow: / User-agent: barkrowler Disallow: / User-agent: Zoominfobot Disallow: / User-agent: adsbot Disallow: / User-agent: YandexBot Allow: /UKR/ Allow: /DONATIONS/ Allow: /ETIQUETTE/ Allow: /KULINAR/ Allow: /MAMBURUNIX/ Allow: /MEDICAL/ Allow: /MONEY/ Allow: /MUSIC/ Allow: /WOMAN/ Allow: /MEN/ Allow: /UNIX/ Allow: /$ Disallow: / Disallow: /*.plain.txt Disallow: /*.support.dhtml #Disallow: /*_feedBack$ Host: lib.misto.kiev.ua Sitemap: http://lib.misto.kiev.ua/sitemap.txt Crawl-Delay: 10 User-Agent: * Disallow: /win Disallow: /koi Disallow: /iso Disallow: /mac Disallow: /alt Disallow: /lat Disallow: /cgi-bin Disallow: /AQUARIUM Disallow: /MPECHKIN Disallow: /rk Disallow: /Forum #Disallow: /lib Disallow: /*.plain.txt.rem Disallow: /*.support.dhtml #Disallow: /*_feedBack$ Sitemap: http://lib.misto.kiev.ua/sitemap.txt Crawl-Delay: 10 Request-rate: 1/5 Visit-time: 2000-0500