BaliPropertyIndexBot

The crawler behind Bali Property Index. It identifies itself on every request as BaliPropertyIndexBot/0.1 (+https://baliindex.com/bot; mailto:index@baliindex.com).

What it collects

Facts about property listings that are already public: asking price and currency, land and building size, bedrooms and bathrooms, tenure and years remaining on a lease, the listing URL, and the dates a listing was first and last seen.

It does not store photos or descriptions. It keeps a numeric fingerprint of the main photo, used only to recognise the same property when it appears on more than one site. The photo itself is not kept.

How often it visits

Twice a week, on Tuesday and Thursday evenings, Bali time. Between visits nothing is requested.

How it behaves

• It reads robots.txt before anything else and follows it.
• It makes at most one request every three seconds to any one site.
• It reopens a listing page only when the listing is new or its price has changed. Pages it has already read are cached.
• If a site refuses it, it stops. It does not retry through other addresses or disguise itself.

To be excluded

A site owner who does not want to be indexed can email index@baliindex.com with the site's address. The exclusion is honoured within a week and stays in place. A Disallow rule for BaliPropertyIndexBot in robots.txt has the same effect from the next visit.