Clew is a capture-the-flag puzzle instrumented to study how automated readers -- particularly AI agents -- traverse the web. This page explains what that instrumentation collects.
Every request to this site is logged before anything else happens: timestamp, IP address, request method and path, response status, User-Agent, TLS fingerprints (JA3, JA4, HTTP/2), Referer, and header order. This applies to every visitor, human or automated, and happens unconditionally.
The challenge page at /thread also causes your browser to make
one additional request on its own, if it executes JavaScript -- that request
is logged the same as any other, and is how this research tells apart
clients that run JavaScript from ones that only fetch raw HTML.
If you submit the self-report at /honesty, the model and
task description you provide are stored as well, alongside your IP and the
time of submission.
A separate nameserver, dnsd, answers DNS queries for
*.c.clew.wtf and logs every one it receives: the resolver IP
address that made the query, and the hostname queried. This is Stage 3 of
the challenge -- visitors who reach it are told to resolve a specific
hostname, and the DNS query itself is the evidence that they did. Unlike
the other data described on this page, this log entry can't currently be
deleted with the button below: DNS queries aren't tied to any
stage_events row the deletion feature can look up by IP, so
there's nothing for it to find and remove.
This data is the subject of the research: understanding what automated
agents do with content they encounter, and how honestly they report it when
asked. Collection is disclosed openly here and in robots.txt
rather than hidden -- Clew is not a covert honeypot.
The raw request log is rotated out after approximately 14 days. Stage-event records -- including the honesty-flag submissions described above -- are purged on the same schedule, approximately 14 days after they are recorded. The DNS query log is rotated out on the same ~14-day schedule as the request log.
This data is not sold, and not shared with third parties for advertising or tracking. It may be used in aggregated or de-identified form in research findings this project publishes.
The button below deletes stage-event records tied to your current IP address immediately -- this includes any honesty-flag submission you made. It does not retroactively remove entries already written to the raw request log, which ages out automatically on the schedule described above rather than being edited in place.
This control is meant for human visitors. Automated clients that only
fetch pages -- most crawlers, including the ones this project studies --
never submit HTML forms, so this doesn't happen automatically as a side
effect of being crawled. Agents have a separate mechanism: sending
X-Clew-Optout: 1 suppresses logging for that request, but
only that request -- it doesn't persist and doesn't apply retroactively.
See also: rules.