Our crawler
BounteeProfiler
If you found this page in your server logs: BounteeProfiler read some of your website because someone using Bountee asked it to, almost always the business that owns the site, to learn what it sells and who buys it.
How it identifies itself
BounteeProfiler/1.0 (+https://bountee.ai/bot)
We never disguise it as a browser or another crawler.
What it reads, and how much
- Only the website it was given and its www twin. It does not follow links to other sites or to other subdomains.
- At most 60 pages, at most three links deep, and no more than two requests a second.
- Only once each time someone asks for it, not on a schedule.
- It skips privacy, terms, cookie and other legal pages, images, video and downloads.
- It reads the text on the page. It does not fill in forms, sign in, or run your scripts.
robots.txt
BounteeProfiler reads your robots.txt first and obeys the rules for User-agent: * and for BounteeProfiler. To keep it out of your whole site:
User-agent: BounteeProfiler Disallow: /
If your site turns the crawler away, Bountee may read the same public pages through its AI provider's web fetch tool, which follows that provider's own rules. Blocking that tool in your robots.txt keeps those pages out as well.
Questions
Write to hello@bountee.ai. If BounteeProfiler ever misbehaves on your site, tell us and we will fix it.