About Our Data & Our Crawler

SmallClaims.com publishes small claims court information — claim limits, filing procedures, courthouse locations, and related details — compiled from official state court websites and primary government sources. Because this information changes (courts move, fees change, rules are revised), we re-verify our data on a recurring schedule. Part of that work is automated by a crawler, documented on this page for transparency and for the website administrators whose sites it visits.

SmallClaimsFreshnessBot

SmallClaimsFreshnessBot is the automated agent we use to check whether the official court web pages we cite are still live and current. It exists for one purpose: catching stale or moved information before it can mislead a self-represented litigant. It identifies itself on every request with this User-Agent:

SmallClaimsFreshnessBot/1.0 (+https://smallclaims.com/about-our-data; legal@smallclaims.com)

How the crawler behaves

• Identification: every request carries the User-Agent above, with a link to this page and a contact address.
• robots.txt: the crawler requests and honors robots.txt directives, including rules addressed to SmallClaimsFreshnessBot.
• Rate limiting: requests are limited to no more than one per second per domain, and most sites we monitor receive only a handful of requests spread across weeks or months.
• Scope: the crawler makes simple GET requests to publicly available pages we already cite as sources. It does not log in, submit forms, or attempt to access non-public content, and it does not execute JavaScript.

For website administrators: verifying our crawler

Traffic claiming to be SmallClaimsFreshnessBot can be verified with forward-confirmed reverse DNS — the same method commonly used to verify major search engine crawlers. Our crawler operates from a single static address with matching forward and reverse DNS records:

# Reverse lookup of the connecting IP:
dig +short -x 167.172.249.66
# -> crawler.smallclaims.com.

# Forward lookup of that hostname:
dig +short crawler.smallclaims.com
# -> 167.172.249.66

If both lookups match as shown, the traffic is ours. Requests using our User-Agent from any other address are not from us — we would appreciate a report to the contact address below.

Opting out

To exclude your site, add the following to your robots.txt — the crawler will honor it:

User-agent: SmallClaimsFreshnessBot
Disallow: /

You can also email us at legal@smallclaims.com and we will exclude your site from monitoring.

Contact

Questions about the crawler, our data practices, or a specific request from our address can be sent to legal@smallclaims.com. For how we source and verify the legal information on this site, see our editorial standards.