Principles
Principle
Adaptor input
Adaptors do not read the operational system itself. They read a landed, source-shaped copy — silver in lakehouse language, SRC or L2 in programme talk. Keep that copy honest: the source’s own keys, and usually the source’s own grain. It is cleaned enough to query. It is not the enterprise model.
Two jobs often land here. A source stores a range — cover from January through May — and you need a row per month. Or the same person is captured twice in one feed — once on the customer, again on the policy — and the birthdays should match. You cannot see that one row at a time. Do that in silver when you can, then point CRISP at those tables. If silver is frozen, or the check only makes sense after programme codes exist, the Rules catalogue can drive the same work — sometimes before anything is written to Conform.
Source tables, then CORE
Silver makes the source safe to query. CORE makes it enterprise-true.
01
Landed
What arrived from the source
02
Source tables
Split and check here if you can
03
Adaptor
Map into the programme shape
04
Conform
First enterprise write
Rules catalogue
Maps and sign still complete Conform rows. The same store can also say “split this range” or “these two birthdays must match” — on source tables, before the write, when silver cannot do it itself.
Splitting rows, and comparing across them
The examples will change. The kinds will not. One is turning one source row into several. The other is a check that needs more than one row.
One row → a row per month
What the source stored
Cover from 1 January through 31 March — one row
January
still points at the original
February
still points at the original
March
still points at the original
Same idea for a list of children, or a condition that creates more than one row
If the source gives you a from–to range and the programme needs a row per month (or per day, or per child in a list), split it here and keep a pointer back to the original. If you only needed to report by month, you might keep one row with a date range and let Span answer as-of questions — splitting is for when each month is a real fact, not a view.
The same person, captured twice
Customer
How the feed defines them
Date of birth
same person · same feed
On the policy
Captured again in the details
Date of birth
Do the dates match? Same idea for “header total equals the lines”
A feed often stores the same person twice — once in the customer record, again in the policy details. Check the birthdays agree. That is ordinary data quality, not Identity. Identity asks whether two systems are talking about the same real-world person. The same kind of check asks whether a header total matches its lines.
Prefer silver. Use Rules when silver cannot.
| If… | Then… |
|---|---|
| The columns are already on the source tables, and that team can change them | Do the split or the check in silver. Point adaptors at those tables. |
| You only have a file or a stream, not a table yet | Land it first, then check. You cannot compare rows on raw bytes. |
| Silver is frozen, or another team will not change grain | Keep the rule in the Rules catalogue. The adaptor can apply it before Conform. |
| The check needs programme codes or signed amounts that only exist after Rules stamp Conform | Run the check after that stamp — still Rules, just later. |
| Silver must keep one row per source row, but CORE needs a finer grain | Split in the adaptor, driven by Rules, still before the Conform write. |
CRISP still reads Conform · Rules · Identity · Span · Publish. Conform is where enterprise facts are first written. Completing those facts with catalogues and maps still happens on the written rows. The same Rules store can also be consulted earlier, on source tables, so you do not write Conform only to fail a check you already could have seen.
If silver is a table, the adaptor can read Rules first
Yes. Silver holds the rows; Rules holds the agreed checks; the adaptor (or a model just in front of it) runs those checks before Conform. Failures stay in a holding area for that source — they never become enterprise assertions.
Adaptor consults Rules first
Source tables
Already landed — you can join them
Rules catalogue
“These two dates must match” — agreed, versioned
Only the good rows go to Conform
Failures stay with that source — they never become enterprise facts
Those early checks can only see what silver already has. If the rule needs a programme code that is stamped later, it waits until after that stamp.
A file is not enough to compare rows
To ask whether two copies of a birthday agree, you need a table you can join — a landing table, an external table, a staged model. Land the file first. Do not write Conform just to get something you can query.
Pseudocode
# Prefer: split and check in silver (source keys still visible)
silver = landed_source_tables(lane) # a table — never a raw file
# Optional: adaptor reads the Rules catalogue against that silver
checks = rules_for_source_tables() # compare across rows, native columns only
valid = silver minus held_aside(checks)
shaped = split_rows_if_needed(valid) # e.g. one range → one row per month
mapped = adaptor_to_canon(shaped)
conform_load(mapped) # enterprise write
# Only if the check needs programme codes or signed amounts:
check_again_on_conform()
stamp_catalogues_and_maps() # usual Rules pass on written rows