TestCrawler
TestCrawler is a bot, but at the moment it is not known who develops it.
Latest user agents:
- Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots) python-requests/2.2.1 CPython
- hgfAlphaXCrawl/0.1 (+https://www.fim.uni-passau.de/data-science/forschung/open-search) RobotsTxtFetcher/1.3
- https://developers.snap.com/robots
- Mozilla/5.0 (compatible) facebookexternalhit/1.1 (+https://faviconkit.com/robots)
- Mozilla/5.0 (compatible; OrtingoBot/2.0; +https://ortingo.com/ortingobot) [More detail: OrtingoBot is a web crawler for the search engine, Ortingo. If you would like your entire site, or specific parts of your site to not be crawled by our bot, please inquire us at hello@ortingo.com, and we will happily cease the crawling. Note: Version 2.0 of OrtingoBot is a re-engineered version of our older version, 1.0, designed for being more accurately compliant with robots.txt files, or the Robots Exclusion Standard.]
- Snap URL Preview Service; bot; snapchat; https://developers.snap.com/robots
- Mozilla/5.0 (compatible; OrtingoBot/2.0; +https://ortingo.com/ortingobot) [More detail: OrtingoBot is a web crawler for the search engine, Ortingo. If you would like your entire site, or specific parts of your site not to be crawled by our bot, please inquire us at hello@ortingo.com, and we will happily cease the crawling. Note: Version 2.0 of OrtingoBot is a re-engineered version of our older version, 1.0, designed for being more accurately compliant with robots.txt files, or the Robots Exclusion Standard.]
- https://user-agents.net/string/mozilla-5-0-compatible-linux-x86-64-mail-ru-bot-fast-2-0-http-go-mail-ru-help-robots-python-requests-2-2-1-cpython-2-7-6-linux-4-9-80-c9
- Mozilla/5.0 (compatible; ThinkChaos/0.3.0; +In_the_test_phase,_if_the_ThinkChaos_brings_you_trouble,_please_add_disallow_to_the_robots.txt._Thank_you.)
- RobotsTxtParser-VIPnytt/2.1 (+https://github.com/VIPnytt/RobotsTxtParser/blob/master/README.md)
- Snap URL Preview Service; bot; https://developers.snap.com/robots
- Flok's crawler (https://flok.codes). I respect robots.txt
- Snap URL Preview Service; bot; https://developer.snapchat.com/robots
- SSERobots
- Linux MozillaLinux Mozilla 5/0Thunderstorm/1.0 (Linux)Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20070308 Minefield/3.0a1Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0Mozilla/5.0 (Linux; Android 4.2.2; QMV7A Build/JDQ39) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.59 Safari/537.36Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.169 Safari/537.36Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0
- RobotsChecker/0.6 ( http://www.blocked.org.uk)
- RobotsTxtParser-VIPnytt/2.0 (+https://github.com/VIPnytt/RobotsTxtParser/blob/master/README.md)
- robots