UKHomeIndexBot

A crawler that reads for-sale listing pages on UK estate agencies' own websites, for a private, non-commercial record of what was on the market and when. It is used by one household. It is not a portal and nothing it reads comes from one.

To stop it

It is read before every visit, so it takes effect on the next one. Or email crawler@ukhomeindex.uk with the domain — no reason needed, no form; applied within a day. Either way your agency is still recorded as existing, and its homes are counted as unread rather than pretended away.

What it keeps

• A copy of each listing page as your server sent it, so the record can be re-read later without fetching it again.
• From that page: price, bedrooms, the address as published, property type, tenure, status, your own feature list, and the dates those changed, with a link back to your page.
• The addresses of the images and a fingerprint of each, so the same photographs can be recognised. It does not copy images or floor plans, and it does not republish descriptions.

What it does not do

• Fill in forms, follow links behind a login, or touch anything under /wp-admin, an account area, or a valuation enquiry.
• Work around a block or a challenge. If your site refuses it, it stops.
• Train AI models. Nothing collected is used for that.
• Take anything from Rightmove, Zoopla, OnTheMarket or any other portal.

How it behaves

• robots.txt is read before every visit and honoured, including Crawl-delay.
• One request at a time per site, several seconds apart, and slower if your server is slow or returns errors.
• It identifies itself on every request:
  Mozilla/5.0 (compatible; UKHomeIndexBot/0.1.0; +https://ukhomeindex.uk/)