If you've seen EchelonGraphBot in your logs, here's exactly what it does and how to block it
Source Reddit NetSec
Discussion
EchelonGraphBot is a web crawler operated by EchelonGraph, Inc. that scans public websites to check configurations such as HTTPS enforcement, certificate expiration, TLS versions, security headers, cookie flags, and redirects. It visits a large number of third-party hosts approximately once per day, respecting robots.txt and rate limits. Operators can opt out via a DNS TXT record or robots.txt. The bot's activity is transparent and publicly documented, with no evidence of malicious intent or exploitation. The operator offers to adjust probing frequency upon request and honors opt-out requests.
r/netsec (opens in a new tab)u/Foreign_Score_40210 points0 comments
We run a crawler that checks how public websites are set up. Things like whether HTTPS is enforced, when the certificate expires, which TLS version is offered, what security headers are set, how cookies are flagged, and where redirects go.
If it has been hitting your servers, you probably want to know what it is. So we wrote the whole thing down on one page: echelongraph.io/bot It covers how often it visits a single host, how it handles robots.txt and Retry-After, which address it comes from, what it will never do, and how to opt out with a DNS record. No signup, no popup.
Two notes on the opt-out, since that is usually the first question. It is a DNS TXT record, so you do not need an account with us to use it. And robots.txt works the normal way if you would rather do it there.
The data it collects is also public at echelongraph.io/radar if you want to look up your own domain, or anyone else's. Free, no account.
Things that are not finished, so you hear it from us first:
- We say checks can run every 30 seconds. They actually land about once a minute. The scheduler does not honour the shorter setting yet.
- Checks stop a couple of minutes after you close the tab.
Happy to answer anything about the crawler's behaviour. If it has done something in your logs that the bot page does not explain, tell me and I will fix either the bot or the page.