# CrossCompute Crawler Access Policy

Last updated: 2026-03-11

This policy explains what crawlers and agent-discovery systems should index.

Primary goal:

- allow indexing of machine-readable docs
- avoid crawler load on authenticated APIs and dashboard pages

## 1) Indexable Public Resources

These URLs are intended for indexing and repeated fetch:

- `https://www.crosscompute.io/`
- `https://www.crosscompute.io/agent-skill.md`
- `https://www.crosscompute.io/llms.txt`
- `https://www.crosscompute.io/openapi.json`
- `https://www.crosscompute.io/api-parameters.md`
- `https://www.crosscompute.io/.well-known/agent-trust-policy.json`
- `https://www.crosscompute.io/.well-known/ai-plugin.json`
- `https://www.crosscompute.io/.well-known/security.txt`
- `https://www.crosscompute.io/sitemap.xml`
- `https://www.crosscompute.io/robots.txt`

## 2) Non-Index Targets

These paths are not designed for crawl/index loops:

- `/api/*` (account, agent registration, usage, topup)
- `/v1/*` (inference and model APIs)
- `/app.html` (authenticated dashboard)
- `/admin/*`

Even if a crawler fetches these endpoints, access control and auth rules still apply.

## 3) Cloudflare Browser Rendering /crawl Compatibility

CrossCompute is compatible with Cloudflare Browser Rendering crawl workflows.

Operational notes:

- apply crawl policy by URL path, not only by user-agent string
- crawler identity may be indicated by request headers such as:
  - `cf-brapi-request-id`
  - `Signature-agent`
- keep machine docs public and cacheable
- keep authenticated API endpoints non-indexed and protected by auth

## 4) Caching Guidance For Crawlers

- honor `ETag` and `Last-Modified` when available
- prefer conditional requests (`If-None-Match`, `If-Modified-Since`)
- avoid aggressive recrawl of API endpoints

## 5) Canonical Agent Discovery Sequence

Recommended fetch order for new agent runtimes:

1. `GET /llms.txt`
2. `GET /openapi.json`
3. `GET /agent-skill.md`
4. `GET /api-parameters.md`
5. `GET /.well-known/agent-trust-policy.json`

From there, runtime API sequence should be:

1. `POST /api/agents/register`
2. `GET /api/agents/usage`
3. `POST /api/agents/topup` (x402 challenge-response)
4. `POST /v1/chat/completions`