---
title: "Web reader API for AI agents: 120 requests a minute"
description: "Web reader API for AI agents: public pages and PDFs over HTTP and MCP, 120 requests a minute per key, stops at declared paywalls and TDMRep reservations."
canonical: "https://laboetie.io/products/source-access"
language: en
---
# Source Access: web reader API

Source Access is a web reader API for AI agents that reads public web pages and documents over HTTP and MCP. Each consumer key gets 120 requests a minute. A read ends on a publisher page when that page declares a paywall or a TDMRep reservation on text and data mining.

Status: Live. Access: In house. Stops at declared paywalls and TDMRep reservations.

## Capabilities

- Law and science queries go to official sources first.
- The service handles PDF, OCR and archives.
- Each consumer has its own key on the HTTP and MCP interface.
- Limits are 120 requests a minute per key and six concurrent reads.

## Web reader API definition

A web reader API takes a URL and returns the readable text of the page, so an agent works from clean text in place of raw HTML. Source Access covers public web pages and documents in one service. Its calls cover text reading, raw byte reading and web search. Agents connect through an MCP endpoint. Law and science queries go to official sources first.

## Reading long PDFs and scanned files

Source Access reads PDFs one window at a time. A call sets a character offset, a page range or a search term, and the response returns that window with the next offset. Scanned PDFs go through OCR. The service keeps a source in memory for 30 minutes after a read, so the next window of a long PDF needs no new read.

## Publisher rights: paywalls and TDMRep

Source Access reads two machine readable publisher signals. Each one ends the read on that publisher page. A paywall declared with the schema.org property isAccessibleForFree set to false returns the reason paywall. A TDMRep reservation under article 4 of EU directive 2019/790 returns the reason tdm_reserved. That reservation can sit in a response header, a meta tag or a tdmrep.json policy file. We left paywall bypass and TDMRep reservation override out of the service.

## Rate limits per API key

Each consumer key gets 120 requests a minute. The service runs six reads at once across REST and MCP, and further reads wait in a queue. A request body over 64 KiB returns 413. A request the caller got wrong returns 400 with the reason.

## MCP tools for AI agents

Source Access exposes two MCP tools over Streamable HTTP. The source_fetch tool reads a URL, and the web_search tool returns results with a title, a URL and a snippet. Each consumer holds its own bearer key on HTTP and MCP alike. Agents use source_fetch for every PDF and whenever a quote must stay verbatim in its language.

## Public addresses only

Source Access reads public addresses only. Its refusal of private addresses covers a redirect to 127.0.0.1 or 169.254.169.254 and a DNS answer that changed after the first check. Each redirect hop and each subresource opens its own connection, which goes through the same check.


## Why AI agents need a web reader

An AI agent that works from raw HTML wastes its context on navigation, scripts and markup. A long PDF can fill that context in one read. A page that fails with no explanation leaves the agent guessing or inventing an answer.

A web reader API sits between the agent and the web. It fetches the address, removes the noise and returns text the model can use, along with the facts a developer needs: the final URL, the content type and what went wrong when a read fails.

## What one read returns

A call is a POST with a URL and a bearer key.

```bash
curl https://your-host/fetch \
  -H "Authorization: Bearer $SOURCE_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.org/report.pdf", "firstPage": 12, "lastPage": 18 }'
```

| Field | Meaning |
|---|---|
| `ok` | True when the read succeeded |
| `finalUrl` | The address after redirects |
| `contentType` | The format of the source |
| `contentText` | The text of this window |
| `nextOffset` | Where the next window starts |
| `provenance` | Set when the text came from a copy, such as an archive |

The `provenance` field matters for agents that cite sources. A quote from an archived capture should be labelled as one, and the application can make that decision from the response.

## The endpoints

| Endpoint | Purpose |
|---|---|
| `POST /fetch` | Read text from a page or a document |
| `POST /raw` | Fetch the original bytes, in base64, with status and headers |
| `POST /search` | Search the web and return titles, URLs and snippets |
| `POST /mcp` | The same reads for agents, through MCP |
| `GET /health` | Service health |

Use `/raw` when a pipeline needs the original file, and `/fetch` when text feeds a model.

## Reading long documents in windows

A 400 page report would swallow a model's context. Source Access returns a window and the position of the next one, so the agent reads what it needs.

A typical sequence looks like this.

1. Ask for the first window and read the page count.
2. Call again with `find` to locate a term, such as a clause or a figure.
3. Read the page range around the match.

The service keeps a source in memory for 30 minutes after a read, so the second and third calls do not fetch the document again. That saves time on the agent's side and load on the publisher's side.

Scanned PDFs go through OCR, which turns an image of a page into text. Archives are unpacked as well. OCR text can contain errors, particularly in tables, so an application that relies on exact figures should check them against the page.

## Sources in order of authority

For law and science, an official source beats a page found through a search, so the service prefers official copies and open access versions when they exist. Each result states its origin, so a citation can name where the text came from.

## Publisher rights

A publisher can state in machine readable form that its content is closed or reserved. Source Access reads two such signals.

| Signal | Where it appears | Reason returned |
|---|---|---|
| Declared paywall | The schema.org property `isAccessibleForFree` set to false | `paywall` |
| TDMRep reservation | A `tdm-reservation` header, a meta tag or `/.well-known/tdmrep.json` | `tdm_reserved` |

**TDMRep** is a protocol that lets a publisher reserve its rights on text and data mining, under article 4 of EU directive 2019/790. The read on that publisher page ends with the stated reason. An agent should treat that as a decision by the publisher, report it, and look for another source.

Your own legal position still needs its own review. The service reads these two signals, and terms of use written in ordinary prose fall outside what it can parse.

## Failed reads

A failed read returns `ok` set to false, a reason, a message and the list of attempts.

| Reason type | What to do |
|---|---|
| `paywall`, `tdm_reserved` | Stop and find another source |
| 400 with a reason | Fix the request |
| 413 | Reduce a body above 64 KiB |
| Other failure | Read the attempts and retry later |

An agent that receives a clear reason can say so to its user. That removes a common cause of invented answers, where an empty result leads a model to fill the gap from memory.

## MCP tools for agents

**MCP**, the Model Context Protocol, is a standard way for an agent to call outside tools. Source Access exposes two tools over Streamable HTTP.

```json
{
  "mcpServers": {
    "source-access": {
      "type": "http",
      "url": "https://your-host/mcp",
      "headers": { "Authorization": "Bearer ${SOURCE_KEY}" }
    }
  }
}
```

| Tool | Returns |
|---|---|
| `source_fetch` | The same read as `/fetch` |
| `web_search` | Results with a title, a URL and a snippet |

## Rate limits and concurrency

Each consumer has its own key, with 120 requests a minute. Six reads run at once across REST and MCP together, and others wait in a queue. A shared service stays fair this way, and an agent with a large task cannot starve the others.

## Public addresses only

A service that fetches any URL can be turned against its own network: a request for `127.0.0.1` or for a cloud metadata address could expose internal systems. This attack is called server side request forgery.

Source Access refuses private addresses, including a redirect to `127.0.0.1` or `169.254.169.254` and a DNS answer that changed after the first check. Each redirect hop and each subresource opens its own connection through the same check.

The fetch call has no cookie or credential field, so content behind a login stays out of reach.

## Web reader, scraper or search API

| Tool | Best for | Output |
|---|---|---|
| Web reader API | One known address, for a model | Clean text and metadata |
| Scraper | Structured data from many pages of a site | Fields you define |
| Search API | Finding addresses | Links and snippets |

Most agents need a reader and a search. Source Access offers both behind one key. Bulk crawling of a whole site is a different job, and it falls outside the design of this service.

## Next step

[Send a brief](https://laboetie.io/brief) with the kinds of source your agents need to read. [Cortex](https://laboetie.io/products/cortex) uses this service for research, and [Agent Runtime](https://laboetie.io/products/agent-runtime) runs the agents that call it.


## Frequently asked questions

### What is a web reader API?

A web reader API is a service that turns a URL into text an application or an agent can use. The service handles the fetch and the decoding, so the caller gets page content with no HTML to parse. Over HTTP or MCP, Source Access returns the text with its final URL, content type and read window.

### Can a web reader API read PDF files?

Source Access reads PDF files and sends scanned PDFs through OCR. A call can ask for a page range, a character window or the matches of a search term, which keeps a long document out of the context of the agent. The response gives the page count and the next offset. A repeat read within 30 minutes answers from memory.

### Can Source Access read pages behind a login?

No. The fetch call accepts a URL with language, window and search options. It has no cookie or credential field, so the service holds no session on any site. Source Access covers public addresses, so content behind an account stays outside what the service can read. A read of such an address reaches the public sign in page at most.

### What happens when a publisher reserves text and data mining?

Source Access ends the read on that publisher page and returns the reason tdm_reserved. The reservation can come from a tdm-reservation response header, a meta tag or a policy file at /.well-known/tdmrep.json. The rule follows article 4 of EU directive 2019/790. Agents treat the reason as a publisher refusal of machine reading. A declared paywall ends the read in the same way.

### What does a failed read return?

A failed read returns ok set to false, a reason, a message and the list of attempts. Two reasons reflect a publisher decision: paywall and tdm_reserved. A request the caller got wrong returns 400 with its reason. A body over 64 KiB returns 413. Any other failure returns 500 with its detail in the service log.

### How long does Source Access keep a page?

The service answers from memory for a source read in the last 30 minutes, which lets an agent walk through a long document window by window with one read of the source. After 30 minutes the next call reads the source again. The cache holds its entries per consumer key and per language.

### Who can use Source Access?

Access to Source Access is in house. The service reads public pages and files for our own services and agents. Each of them holds its own bearer key. No public signup or price list exists. A team that wants a reader for its own agents can send a brief. The scope stays public pages under the rights limits on this page.