Crawler
AITEITbot.
AITEITbot is our small crawler. It reads a handful of public pages on company websites (home, about, team, contact, press, careers) to understand what a business makes and who runs marketing.
How it behaves
- Follows robots.txt. If robots.txt cannot be read, it does not crawl the site.
- At most one request per second per site, and it backs off on errors.
- Never logs in, never submits forms, never reads pages behind a login.
- Keeps only business information: company details, published business emails, names and job titles of team members listed on the site.
How to block it
Add this to your robots.txt:
User-agent: AITEITbot
Disallow: /
Remove your data
Email manas@aiteitai.com and we delete what we hold and add your domain to our do-not-contact list.