Skip to content

NagetBot

NagetBot is the web crawler behind Naget search.

NagetBot visits public web pages and adds them to the index that Naget searches. When a page from your site helps answer a search, Naget shows it as a source with a link to your page.

How to recognize NagetBot

Requests from NagetBot carry this user-agent string:

Mozilla/5.0 (compatible; NagetBot/1.0; +https://naget.ai/bot)

The link in the string leads to this page.

Control what NagetBot crawls

Before fetching pages from a host, NagetBot reads the host’s robots.txt. It follows the rules in the group for the NagetBot user agent, or the * group if there is none. If a host has no robots.txt, NagetBot treats every page as allowed.

To keep NagetBot off your whole site:

User-agent: NagetBot
Disallow: /

To keep it out of one section and slow it down:

User-agent: NagetBot
Disallow: /private/
Crawl-delay: 10

Crawl-delay sets the minimum number of seconds between NagetBot’s requests to your site. A delay of one second or more also limits NagetBot to one request at a time.

NagetBot keeps a copy of each robots.txt for up to 24 hours, so a change can take a day to apply.

How NagetBot behaves

  • It spaces out its requests to each site. When a site returns errors, it waits twice as long after each error in a row, up to 10 seconds between requests.
  • It fetches HTML pages only, reads at most 5 MB of each page, and follows up to five redirects.
  • It does not run JavaScript, fill in forms, or sign in.

Contact

If NagetBot is causing problems on your site, or you have a question about it, email [email protected]. Include your domain and, if you can, a few lines from your access log.