Effective date: 1st September, 2024

About Our Robot

Web Crawler Usage

Pillion uses an automated web crawler to retrieve publicly available data from websites for analysis purposes to provide our service. Our crawler is designed to be polite and to minimize its impact on the websites it visits. It follows best practices for crawl rate and frequency.

Crawler Identification

Our web crawler can be identified by the following user agent string:

Blocking our crawler

Our crawler is designed to respect the robots.txt file of websites it visits. If you wish to prevent our crawler from accessing specific parts of your website, you can update your robots.txt file accordingly. For example, to prevent our bot from crawling any page on your site:

Contact

If you have any questions or concerns about our web crawler, please contact us at [email protected] .