Data and provenance
Where every value came from.
Every canonical field records two things: the value NanoScraper stored, and the fragment the source actually printed. Those are different claims, and the system keeps both.
One provenance rowfield: phone
- valuewhat was stored
- 02079460958
- raw_valuewhat the source printed
- (020) 7946 0958
Recognisably the same number. Demonstrably not the same string.
The row also holds the observation, the connector and its version, the source reference, and the time it was seen.
Two tables, not one.
Raw observations and canonical entities are separate models, and the evidence side is insert-only in the strict sense. Normalisation reads it and writes somewhere else, so it cannot write back over what it was derived from.
backend/apps/jobs/models.py · backend/apps/canonical/normalize.py
- RawObservation.save()
- Raises outright if the row already exists. Insert-only is enforced by the model, not by convention.
- canonical/normalize.py
- Imports
re,unicodedataandurllib.parse. No Django, no models, no connector, no I/O. - The enforcing test
- The whole normaliser suite runs without a database, which is possible only because nothing in there can reach one.
The field that records both halves.
FieldProvenance holds both values for one field of one entity. Its shape is not a convention: three database constraints enforce it, and the generated schema publishes both fields as required.
backend/apps/canonical/models.py
- Exactly one target
- A check constraint requires a company or a contact — never both, never neither.
- One row each
- Unique constraints give one row per workspace, entity, field and observation.
- Held in the migration
- All three are database constraints, so the wrong shape is rejected by PostgreSQL rather than by a validator someone can skip.
- Both halves published
- The serializer lists both fields and the generated schema marks both required, so the typed client cannot quietly drop one.
How to read it
Downward, in one direction. A payload fragment is coerced, matched, and only then written as provenance. Nothing on that path writes back to the observation it started from.
The two branches are where it declines to improve on the evidence: an ambiguous match makes a new record, and an existing record has gaps filled rather than values replaced.
The defect this repaired
A column read “as observed”. It wasn't.
Before the repairone phone number, one stored string
value
02079460958
as observed
02079460958
The second heading was a claim about a source. No source printed that string — the normaliser produced it. The page it came from had written (020) 7946 0958.
Nothing errored. Nothing was logged. The value was even correct — it was the claim about the value that was false, which is the harder kind to notice.
Held in three independent places
- The write path
- Stores an empty string when a source fragment is absent, rather than echoing the canonical value.
- The migration
- Adds the column with no backfill. A historical raw form is not recoverable, so old rows stay blank.
- The interface
- Hides the “as observed” line when the raw value is blank or identical, instead of printing one string under two headings.
Declining to back-fill was the deliberate part. Inventing a plausible original would have put fabricated evidence in the one table whose job is to be checkable. Blank reads as unknown, which is the truth.
backend/apps/canonical/tests/test_provenance_semantics.py — a dedicated module whose tests fail if any of the three is removed
Matching, and refusing to match.
An ambiguous name creates a new record rather than merging.
The lookup takes two candidates and uses the match only when exactly one comes back. Guessing which of two businesses a row belongs to is a wrong answer that looks exactly like a right one.
Deduplication happens inside one workspace only. Domain is tried first; a name key is a fallback, used only when it is at least three characters long and only among companies with no domain at all.
- 01Gap-filling only
- Guards over domain, website, description and language let a later observation add what was missing — never replace what was there.
- 02Names are not rewritten
- An existing company's name is never changed by a later observation.
- 03A merge keeps the duplicate
- Contacts and provenance repoint to the survivor; the duplicate stays, pointing at it, with its domain cleared so a partial unique index accepts the row.
No observation is touched by any of it, which is what makes a mistaken merge something to rebuild from rather than restore from a backup.
backend/apps/canonical/services.py
From records to a dataset that stays current.
A dataset holds one record per entity, stable across updates. Each accepted value points at its observation and provenance row through nullable foreign keys rather than copying them — nullable because a value a person resolved by hand has no observation behind it.
An update either covered the whole source or it did not, and that single flag gates removal. A partial update observed nothing about the records it never reached, so it cannot nudge one towards being written off.
- Removal threshold
- Absence becomes removal at the dataset's threshold — two by default — and only a complete update raises the counter.
- One change record each
- Every field that moves becomes one change record; a record appearing or disappearing becomes one of its own.
- Versions are snapshots
- The whole value map, not deltas, so a past state is exact rather than replayed. Fifty per record by default, up to five hundred.
- Trimmed history refuses
- Past the cap the time machine declines the question: the oldest version still held would present a later state as the one asked for.
A single absence never removes a record. The coverage flag is the difference between “this business closed” and “the update stopped early” — two conclusions an absent row cannot distinguish on its own.
backend/apps/datasets/services.py · backend/apps/datasets/timemachine.py
What a run records, and one gap that stays open.
A run carries a normalized_at stamp. That the step ran is recorded separately from what it produced, because an empty result cannot tell those two facts apart.
- Stamped when it finishes
- Written whether or not canonical rows resulted, so “the source held nothing usable” stays distinct from “this has not run yet”.
- Dispatched only when there is something
- Only a run that stored at least one observation and did not fail. A run that collected nothing carries no stamp — correctly.
A known gap, stated rather than solved
When normalisation skips an observation the reason is logged and never surfaced, so three observations and zero companies can appear with no on-screen explanation. It was checked against every contract during release closure — no acceptance criterion requires it — and kept as declared debt.
backend/apps/canonical/tasks.py · backend/apps/jobs/tasks.py · docs/execution/phases/phase-08-release-gate.md