Skip to main content
NanoScraper

NanoScraper · collection pipeline

How a scrape becomes a record you can trust

From the URL someone submits to the row they export. Every layer assumes the one below it is untrusted — that idea explains most of the design.

Where the boundary sits

  • Ours

    The API, job records, the queue, the worker, the connector interface and the guarded fetch.

  • untrusted

    The engines

    Third-party containers on a network of their own. They supply bytes and decide nothing else.

  • The store

    Immutable raw observations, then canonical records carrying field-level provenance.

The dashed rules are where trust stops. An engine supplies bytes and decides nothing else.

Stage A–F

The whole path, at once

Six stages, four groups, three trust zones. Tenancy, lifecycle, persistence, deduplication and provenance stay ours — which is why swapping an engine touches one module and nothing downstream of it.

Fig. 01 — The whole path2779 × 718
The collection pipeline. A trusted zone holds the UI, API, job records, queue and worker. The connector interface and the SSRF-guarded fetch are NanoScraper code running inside the worker process, and reach the public web directly. Only the third-party engine containers are isolated on the engines network; the connector calls them over HTTP and they reach the web themselves. Observations flow into an immutable raw store, then into canonical companies and contacts with field-level provenance, then out to the API and exports.

Only the engine containers are isolated — the connector and its guarded fetch are ours, inside the worker. Two connectors open the socket through fetch.py; two hand it to a container, and nothing downstream can tell which.

Stage A · B

Creating and dispatching a job

Two refusals land before any work starts, in order: readiness, then config. Validating the config of a connector that cannot run would imply a correct config leads somewhere.

Quota admission comes before the run row exists — admitting afterwards leaves an orphaned pending run behind every refusal, and pending runs count toward the next check.

Fig. 02 — Create and dispatch1579 × 1309
A sequence from operator to worker. The API authenticates, resolves the workspace and verifies an active membership, then asks the registry whether the connector is ready, refusing with a reason if not. Only then is the config validated and the job created. Dispatch admits the quota first, snapshots the config, and enqueues scalars. The worker re-validates membership on arrival.

Authorization happens twice, on purpose. The task carries a workspace id as a scalar, never a trusted object, and the worker proves membership again — the queue is not a trust boundary.

Stage C

Four connectors, three ways of reaching the web

The connectors differ in one thing that matters: who opens the socket, and therefore where the SSRF check sits.

Scrapling is a parser only. Letting it fetch would open a second way out that knows nothing about our blocked ranges. Parsing alone avoids about 1.5 GB of Chromium and still buys JSON-LD — a business stating its own phone number in machine-readable form.

Fig. 03 — Fetch strategies1148 × 672
Two groups. In the first, web_extract and scrapling both fetch through a single guarded egress module. In the second, google_maps and firecrawl hand the work to an external engine; the firecrawl path validates the URL before submitting it, and google_maps takes keywords rather than URLs so no user-supplied target exists.

Firecrawl fetches from inside our network, so its target is validated before being handed over. Skip that and SSRF is not avoided, only outsourced to a fetcher with none of our controls.

Stage C·1

Inside the guarded fetch

The threat is rarely http://127.0.0.1. It is five subtler shapes, and one URL is checked for every one of them.

  1. Private answerA public hostname whose DNS record resolves somewhere private.
  2. Second lookupA name that answers differently the second time it is asked.
  3. RedirectA hop that lands in the metadata service.
  4. Mapped spellingAn IPv4-mapped IPv6 form of a blocked range.
  5. No endA body that never stops arriving.
Fig. 04 — The guarded fetch933 × 1685
A decision flow for one URL: scheme allow-list, then reject embedded credentials, then resolve DNS and check every returned address against the blocked ranges, then pin the validated address and connect to it while sending the original Host header and TLS SNI. Redirects loop back to the start of validation with a hop limit. Server errors are transient and retried; policy failures are terminal.

Pinning the address is what closes DNS rebinding. The Host header and TLS SNI still carry the original name, so virtual hosting and certificates keep working.

Stage C·2

What an engine can reach if it is compromised

Application validation is not the last line. Three things it cannot cover:

Its own redirect
A hop the engine follows itself, after our check has passed.
A changed answer
DNS that moves between our check and its connect.
A defect
A flaw in the engine itself.
Fig. 05 — Engine isolation960 × 608
Three groups of containers. PostgreSQL, Redis and the scheduler sit on the internal network. The backend and worker sit on both networks, since they are ours and already hold the credentials. The collection engines sit only on the engines network, and their attempts to reach PostgreSQL and Redis are marked as verified failures.

The engines network has no route to PostgreSQL or Redis, tested from inside the engine rather than inferred. A topology check asserts it against both the declared configuration and the live network.

Network isolation holds then, and both controls are required. It bounds what a compromised engine can reach; which workspace's data a legitimate request may touch is a different boundary, on /docs/security.

Stage D

Storing what was observed, and what happens when it fails

A connector failure is a normal operating condition, not a crash. The run stores a classified reason — transient for a timeout or an upstream 5xx, invalid_config for a bad configuration — and stays inspectable rather than escaping as a server error.

Classification does not reopen the run. A failed run is terminal — the lifecycle table declares no transition out of it — and re-running opens a new attempt, so the failure stays in the history.

Fig. 06 — Run lifecycle1575 × 573
A run moves from pending to running, then to succeeded, partial, or failed. A failure is classified as transient or terminal and the reason is stored on the run, but both kinds end it: there is no path back to pending. A successful run dispatches normalization.

Uniqueness is scoped to a single run, so a redelivered task cannot store the same observation twice. Re-running opens a new attempt with its own copies, and the count recorded is what was persisted.

Stage E

Raw becomes canonical, and stays traceable

Normalization is a separate task, so a normalization bug cannot roll back the collection that produced the evidence, and re-normalizing after a rule change is a repair path rather than a gamble.

Raw observations are never mutated. Saving one twice raises.

Fig. 07 — Raw to canonical681 × 1196
From an immutable raw observation, payload fragments are coerced, then matched against existing companies by domain or by an unambiguous name key. Ambiguous matches create a new record rather than merging silently. Existing records have gaps filled but are never overwritten. Field provenance is then written with both the stored value and what the source actually said.

Provenance records both halves, because they are not the same claim. Where the source said nothing the observed value stays blank — inventing it would put fabricated evidence in the table whose job is to be checkable.

Stage F

The catalogue, and the two that never collect

Whether a connector can run right now is answered by the application, not this page. /api/v1/connectors/ re-checks the operator's switches on every request and asks each engine-backed connector whether its engine answers.

Readiness travels with the catalogue, so the interface can disable a control rather than offer an action the API would refuse anyway. The table states only what each connector declares in code.

Collection connectors, whether each is on by default, and which side opens the network connection
ConnectorDeclared in codeWho fetchesNotes
web_extracton by defaultus, guardedparses pages we fetched ourselves
scraplingon by defaultus, guardedparser only — never its fetchers; adds JSON-LD
google_mapson by defaultthe enginekeywords, not URLs — no SSRF surface from input
firecrawlopt-inthe engineAGPL, separate unmodified service; target pre-validated
referencediagnostic, off by defaultnobodyits records are invented
linkedinpermanently refusednobodydeclared and permanently withheld

Six connectors, registered by apps/connectors/apps.py.

reference
Fabricates records. Once stored they are indistinguishable from collected ones, so it is labelled diagnostic and stays off.
linkedin
Its readiness check never consults the enable flags, because an operator cannot switch off a licence contradiction — and the module imports no LinkedIn library.
extraction
The seventh, registered by apps/extraction/apps.py. It runs saved extraction definitions through the same boundary and appears in the same catalogue response.

Where these pictures come from

Every claim here comes from the code and from runs executed against production. The Mermaid sources are committed alongside the SVGs they produce, and the renderer between them is a container pinned by digest.

The security boundariesWhat has actually been verifiedSign in