# URL → Markdown API

Converts a web page into clean Markdown suitable for LLM input.

Base URL: `http://127.0.0.1:8788`

All responses are JSON. Every successful call returns `ok: true` with a `result`;
every refusal returns `ok: false` with a typed `error`.

## Authentication

Send your key on every request except `/health`:

```
X-API-Key: <your key>
```

## `GET|POST /v1/scrape`

Convert one URL to Markdown.

| Parameter | Type | Default | Meaning |
| --- | --- | --- | --- |
| `url` | string | **required** | The page to convert. Must be `http` or `https`. Max 2048 characters. |
| `formats` | string[] or comma-separated string | `["markdown"]` | Any of `markdown`, `html`, `rawHtml`, `links`, `summary`. |
| `onlyMainContent` | boolean | upstream default | Strip navigation, headers and footers, keeping the article body. |
| `fresh` | boolean | `false` | Bypass the response cache and fetch the page live. Slower; use only when you need the very latest version of a page. |

GET takes query parameters; POST takes a JSON body. They are equivalent.

```bash
curl -s 'http://127.0.0.1:8788/v1/scrape?url=https://example.com/article' \
  -H 'X-API-Key: YOUR_KEY'
```

```bash
curl -s http://127.0.0.1:8788/v1/scrape \
  -H 'X-API-Key: YOUR_KEY' \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com/article","formats":["markdown"],"onlyMainContent":true}'
```

### Response

```json
{
  "ok": true,
  "result": {
    "url": "https://example.com/article",
    "markdown": "# Article title\n\nBody text…",
    "metadata": {
      "title": "Article title",
      "description": "A short summary of the page.",
      "language": "en",
      "sourceURL": "https://example.com/article",
      "statusCode": 200,
      "contentType": "text/html"
    },
    "tookMs": 934
  }
}
```

`result` carries one key per requested format (`markdown`, `html`, `links`, …)
plus `metadata` and `tookMs`. `metadata` fields are best-effort: a page that
declares no description or language simply omits those keys, so read them
defensively.

## `GET /v1/stats`

Current capacity and remaining quota. Poll this before a large batch to decide
whether to start now or wait.

```json
{
  "ok": true,
  "status": "ok",
  "capacity": { "availableNow": 8, "inFlight": 0, "queued": 0 },
  "quota": {
    "remainingThisPeriod": 4097,
    "resetsAt": "2026-10-01T00:00:00.000Z",
    "daysUntilReset": 23.4
  },
  "usage": { "requestsServed": 12, "requestsFailed": 0 }
}
```

| Field | Meaning |
| --- | --- |
| `status` | `ok`, `saturated` (no free slot this instant), or `exhausted` (no quota left this period). |
| `capacity.availableNow` | How many more scrapes can start immediately without queueing. |
| `capacity.queued` | Requests currently waiting for a slot. |
| `quota.remainingThisPeriod` | Conversions still available before `resetsAt`. |

## `GET /health`

Never authenticated — this is what a load balancer probes.

- `200 {"ok":true,"status":"ready"}` — serving
- `503 {"ok":false,"status":"starting"}` — still warming up; retry shortly

## `GET /docs`

This document, as raw Markdown.

## Errors

Refusals are returned as **HTTP 200** with `ok: false`. This is deliberate: a
capacity refusal is a scheduling answer that carries a machine-readable retry
time, not a failure of the service. Mapping it onto 429/503 would discard that
timestamp and invite intermediate proxies and client libraries to retry on their
own terms.

```json
{
  "ok": false,
  "error": {
    "code": "CAPACITY",
    "message": "no capacity for this request right now",
    "retryAtMs": 1789500000000,
    "attempts": 2,
    "upstreamStatus": [429]
  }
}
```

| Code | HTTP | Meaning | What to do |
| --- | --- | --- | --- |
| `BAD_REQUEST` | 400 | Missing/invalid `url`, or an unsupported format. | Fix the request; retrying unchanged will not help. |
| `UNAUTHORISED` | 401 | Missing or wrong `X-API-Key`. | Send the correct key. |
| `CAPACITY` | 200 | No capacity right now. | Wait until `retryAtMs`, then retry. |
| `QUEUE_TIMEOUT` | 200 | Waited its allowance without getting a slot. | Wait until `retryAtMs`, then retry. |
| `QUEUE_FULL` | 200 | Too many requests are already waiting. | Back off and retry with jitter. |
| `ALL_ATTEMPTS_FAILED` | 200 | The page could not be fetched. | Check `upstreamStatus`; a 4xx usually means the URL itself is bad. |
| `NO_ELIGIBLE_SOURCE` | 200 | The service has no capacity configured. | Operator problem — do not retry in a loop. |
| `CLOSED` | 200 | The service is starting or shutting down. | Retry in a few seconds. |
| `INTERNAL` | 500 | Unexpected failure. | Retry once; report if it persists. |

### Retry guidance

`retryAtMs` is an epoch-milliseconds timestamp. When it is present, wait until
it rather than using a fixed backoff — it is the earliest moment the request can
actually succeed. A `Retry-After` header (in seconds) is set alongside it for
clients that prefer the header.

```js
const res  = await fetch(url, { headers: { 'X-API-Key': KEY } });
const body = await res.json();

if (!body.ok) {
  if (body.error.retryAtMs) {
    await sleep(Math.max(0, body.error.retryAtMs - Date.now()));
    // retry once here
  }
  throw new Error(body.error.code);
}
return body.result.markdown;
```

Do not retry `BAD_REQUEST`, `UNAUTHORISED` or `NO_ELIGIBLE_SOURCE` — they will
not succeed on a second attempt.

## Notes for automated clients

- **One request per URL.** The service handles its own internal retries and
  failover; wrapping calls in your own retry loop multiplies load without
  improving your success rate.
- **Respect `retryAtMs`.** It is the only reliable signal for when to come back.
- **Check `metadata.statusCode`.** A `200` envelope means the *service* worked;
  `metadata.statusCode` tells you what the target page returned.
- **Concurrency.** Read `capacity.availableNow` from `/v1/stats` and keep your
  parallelism at or below it to avoid `CAPACITY` responses.