Source Access is a web reader API for AI agents that reads public web pages and documents over HTTP and MCP. Each consumer key gets 120 requests a minute. A read ends on a publisher page when that page declares a paywall or a TDMRep reservation on text and data mining.
Stops at declared paywalls and TDMRep reservations
A web reader API takes a URL and returns the readable text of the page, so an agent works from clean text in place of raw HTML. Source Access covers public web pages and documents in one service. Its calls cover text reading, raw byte reading and web search. Agents connect through an MCP endpoint. Law and science queries go to official sources first.
Source Access reads PDFs one window at a time. A call sets a character offset, a page range or a search term, and the response returns that window with the next offset. Scanned PDFs go through OCR. The service keeps a source in memory for 30 minutes after a read, so the next window of a long PDF needs no new read.
Source Access reads two machine readable publisher signals. Each one ends the read on that publisher page. A paywall declared with the schema.org property isAccessibleForFree set to false returns the reason paywall. A TDMRep reservation under article 4 of EU directive 2019/790 returns the reason tdm_reserved. That reservation can sit in a response header, a meta tag or a tdmrep.json policy file. We left paywall bypass and TDMRep reservation override out of the service.
Each consumer key gets 120 requests a minute. The service runs six reads at once across REST and MCP, and further reads wait in a queue. A request body over 64 KiB returns 413. A request the caller got wrong returns 400 with the reason.
Source Access exposes two MCP tools over Streamable HTTP. The source_fetch tool reads a URL, and the web_search tool returns results with a title, a URL and a snippet. Each consumer holds its own bearer key on HTTP and MCP alike. Agents use source_fetch for every PDF and whenever a quote must stay verbatim in its language.
Source Access reads public addresses only. Its refusal of private addresses covers a redirect to 127.0.0.1 or 169.254.169.254 and a DNS answer that changed after the first check. Each redirect hop and each subresource opens its own connection, which goes through the same check.
An AI agent that works from raw HTML wastes its context on navigation, scripts and markup. A long PDF can fill that context in one read. A page that fails with no explanation leaves the agent guessing or inventing an answer.
A web reader API sits between the agent and the web. It fetches the address, removes the noise and returns text the model can use, along with the facts a developer needs: the final URL, the content type and what went wrong when a read fails.
A call is a POST with a URL and a bearer key.
curl https://your-host/fetch \
-H "Authorization: Bearer $SOURCE_KEY" \
-H "Content-Type: application/json" \
-d '{ "url": "https://example.org/report.pdf", "firstPage": 12, "lastPage": 18 }'
| Field | Meaning |
|---|---|
ok | True when the read succeeded |
finalUrl | The address after redirects |
contentType | The format of the source |
contentText | The text of this window |
nextOffset | Where the next window starts |
provenance | Set when the text came from a copy, such as an archive |
The provenance field matters for agents that cite sources. A quote from an archived capture should be labelled as one, and the application can make that decision from the response.
| Endpoint | Purpose |
|---|---|
POST /fetch | Read text from a page or a document |
POST /raw | Fetch the original bytes, in base64, with status and headers |
POST /search | Search the web and return titles, URLs and snippets |
POST /mcp | The same reads for agents, through MCP |
GET /health | Service health |
Use /raw when a pipeline needs the original file, and /fetch when text feeds a model.
A 400 page report would swallow a model's context. Source Access returns a window and the position of the next one, so the agent reads what it needs.
A typical sequence looks like this.
find to locate a term, such as a clause or a figure.The service keeps a source in memory for 30 minutes after a read, so the second and third calls do not fetch the document again. That saves time on the agent's side and load on the publisher's side.
Scanned PDFs go through OCR, which turns an image of a page into text. Archives are unpacked as well. OCR text can contain errors, particularly in tables, so an application that relies on exact figures should check them against the page.
A publisher can state in machine readable form that its content is closed or reserved. Source Access reads two such signals.
| Signal | Where it appears | Reason returned |
|---|---|---|
| Declared paywall | The schema.org property isAccessibleForFree set to false | paywall |
| TDMRep reservation | A tdm-reservation header, a meta tag or /.well-known/tdmrep.json | tdm_reserved |
TDMRep is a protocol that lets a publisher reserve its rights on text and data mining, under article 4 of EU directive 2019/790. The read on that publisher page ends with the stated reason. An agent should treat that as a decision by the publisher, report it, and look for another source.
Your own legal position still needs its own review. The service reads these two signals, and terms of use written in ordinary prose fall outside what it can parse.
A failed read returns ok set to false, a reason, a message and the list of attempts.
| Reason type | What to do |
|---|---|
paywall, tdm_reserved | Stop and find another source |
| 400 with a reason | Fix the request |
| 413 | Reduce a body above 64 KiB |
| Other failure | Read the attempts and retry later |
An agent that receives a clear reason can say so to its user. That removes a common cause of invented answers, where an empty result leads a model to fill the gap from memory.
MCP, the Model Context Protocol, is a standard way for an agent to call outside tools. Source Access exposes two tools over Streamable HTTP.
{
"mcpServers": {
"source-access": {
"type": "http",
"url": "https://your-host/mcp",
"headers": { "Authorization": "Bearer ${SOURCE_KEY}" }
}
}
}
| Tool | Returns |
|---|---|
source_fetch | The same read as /fetch |
web_search | Results with a title, a URL and a snippet |
Each consumer has its own key, with 120 requests a minute. Six reads run at once across REST and MCP together, and others wait in a queue. A shared service stays fair this way, and an agent with a large task cannot starve the others.
A service that fetches any URL can be turned against its own network: a request for 127.0.0.1 or for a cloud metadata address could expose internal systems. This attack is called server side request forgery.
Source Access refuses private addresses, including a redirect to 127.0.0.1 or 169.254.169.254 and a DNS answer that changed after the first check. Each redirect hop and each subresource opens its own connection through the same check.
The fetch call has no cookie or credential field, so content behind a login stays out of reach.
| Tool | Best for | Output |
|---|---|---|
| Web reader API | One known address, for a model | Clean text and metadata |
| Scraper | Structured data from many pages of a site | Fields you define |
| Search API | Finding addresses | Links and snippets |
Most agents need a reader and a search. Source Access offers both behind one key. Bulk crawling of a whole site is a different job, and it falls outside the design of this service.
Send a brief with the kinds of source your agents need to read. Cortex uses this service for research, and Agent Runtime runs the agents that call it.
A web reader API is a service that turns a URL into text an application or an agent can use. The service handles the fetch and the decoding, so the caller gets page content with no HTML to parse. Over HTTP or MCP, Source Access returns the text with its final URL, content type and read window.
Source Access reads PDF files and sends scanned PDFs through OCR. A call can ask for a page range, a character window or the matches of a search term, which keeps a long document out of the context of the agent. The response gives the page count and the next offset. A repeat read within 30 minutes answers from memory.
No. The fetch call accepts a URL with language, window and search options. It has no cookie or credential field, so the service holds no session on any site. Source Access covers public addresses, so content behind an account stays outside what the service can read. A read of such an address reaches the public sign in page at most.
Source Access ends the read on that publisher page and returns the reason tdm_reserved. The reservation can come from a tdm-reservation response header, a meta tag or a policy file at /.well-known/tdmrep.json. The rule follows article 4 of EU directive 2019/790. Agents treat the reason as a publisher refusal of machine reading. A declared paywall ends the read in the same way.
A failed read returns ok set to false, a reason, a message and the list of attempts. Two reasons reflect a publisher decision: paywall and tdm_reserved. A request the caller got wrong returns 400 with its reason. A body over 64 KiB returns 413. Any other failure returns 500 with its detail in the service log.
The service answers from memory for a source read in the last 30 minutes, which lets an agent walk through a long document window by window with one read of the source. After 30 minutes the next call reads the source again. The cache holds its entries per consumer key and per language.
Access to Source Access is in house. The service reads public pages and files for our own services and agents. Each of them holds its own bearer key. No public signup or price list exists. A team that wants a reader for its own agents can send a brief. The scope stays public pages under the rights limits on this page.