UKHomeIndexBot
A crawler that reads for-sale listing pages on UK estate agents' websites.
To stop it, add this to your robots.txt:
User-agent: UKHomeIndexBot
Disallow: /
It is read before every visit, so it takes effect on the next one. Or email
crawler@ukhomeindex.uk with the domain — no reason needed, no form.
What it keeps
Price, number of bedrooms, address as published, property type, tenure, and the dates those changed, with a link back to your page. One display-sized image and the floor plan per listing; other photographs are not copied. Descriptions are not republished.
What it does not do
- Fill in forms, follow links behind a login, or touch anything under
/wp-admin, an account area, or a valuation enquiry. - Train AI models. Nothing collected is used for that.
- Take anything from Rightmove, Zoopla, OnTheMarket or any other portal.
How it behaves
robots.txtis read before every visit and honoured, includingCrawl-delay.- One request at a time per site, several seconds apart, and slower if your server is slow or returns errors.
- Conditional requests (
If-Modified-Since,ETag), so an unchanged page costs you a few bytes. - It identifies itself on every request:
Mozilla/5.0 (compatible; UKHomeIndexBot/0.1; +https://ukhomeindex.uk/)