# Mindjack - robots.txt
# https://mindjack.xyz

User-agent: *
Allow: /
Content-Signal: search=yes,ai-train=no

# Endpoints a crawler can never index, but which cost real crawl budget.
# Measured 2026-08-11 from origin logs: 118 of 408 real-Googlebot requests
# (66.249.x) over 13 days landed here. The two stream endpoints are the worst
# of it. They are Server-Sent Events, so Googlebot opens the connection and
# holds it until it gives up; six of those ended in a 499.
#
# Deliberately NOT blocked: the endpoints that feed visible content to the
# pages Google already has indexed (market, prices, leaderboard, recent-trades,
# monitor/tokens, intel/stats, chart data). Blocking those would render those
# pages empty for Google and risk losing the little we have indexed.
Disallow: /api/monitor/stream
Disallow: /api/kols/stream
Disallow: /api/monitor/heartbeat
Disallow: /api/kols/heartbeat
Disallow: /api/monitor/time-remaining
Disallow: /api/kols/time-remaining
Disallow: /api/monitor/my-analyses
Disallow: /api/monitor/active-analyses
Disallow: /api/monitor/live/active
Disallow: /api/kols/tracker/status

# AI crawlers are allowed here, deliberately.
#
# This site sells data to agents, so the crawlers that tell agents what exists
# are distribution, not theft. Measured over the seven days to 2026-08-17 from
# origin logs: BuiltWith-MCPRegistryScrape asked this host for /llms.txt,
# ChatGPT-User and Claude-SearchBot and OAI-SearchBot asked for /openapi.json,
# and every one of them left with a 404. Those doors are open now, and a
# blanket Disallow would have been a second lock on the same door.
#
# The training preference is still declared, above, in Content-Signal:
# search and reference yes, training no. That is a stated preference rather
# than an access rule, which is the right shape for the ask. Blocking the
# crawler blocks the reference too, and reference is what sends us customers.

Sitemap: https://mindjack.xyz/sitemap-pages.xml
Sitemap: https://mindjack.xyz/sitemap-tokens.xml