A number that remembers where it came from
A grid intensity figure is 412 gCO2e per kWh. Six months later somebody asks where that came from, and the honest answer is usually a shrug: it was copied out of a portal, the portal has restated it since, and the parameters that produced it are gone. This instrument is the record that stops that happening. It holds the source, the dataset, the version of the dataset, the instant of retrieval, the licence and the exact query, and it keeps holding them through arithmetic. Divide one cited value by another and the answer still knows about both retrievals.
01A cited value
Five fields are mandatory. A record will not construct without them, and the point is not that the boxes are full but that the gaps are written down: if a source states no licence, the honest entry is the word unstated, not an empty field and not a guess.
Value A
Value B
02Arithmetic keeps the lineage
This is the part that makes the record worth keeping. A cited value is not a number with a note attached to it: the note travels through the operation. Divide two figures in the same unit and the unit cancels while the sources do not. The rule the Python pins with a sweep over four hundred randomly generated expression trees is that no operation can drop a source, and that combining values in a different order cannot change which sources the answer admits to.
03Auditing the document
A published figure is only as current as the oldest retrieval underneath it. This panel reads the two values above and the one derived from them as a single document, and says what is wrong with it: what has aged past the threshold, what carries no address to go back to, what recorded no query, and whether the licences underneath have been mixed. The Python has a strict mode that exits non zero on any of these, meant for a build rather than for a person: a repository of derived figures should fail when its inputs age out, not when somebody eventually re-reads them.
04Method, and what this refuses
Identity is the canonical form
A record's identity is a hash of its canonical form: sorted keys at every depth, no insignificant whitespace, and the timestamp normalised to UTC. Two records identify as the same retrieval exactly when they describe the same retrieval, however the fields were assembled and whichever offset the timestamp was written in. The query is part of that identity, so the same dataset at the same version and the same instant but a different country code is two different numbers and must never deduplicate against the other.
Lineage is a set union
Merging two lineages is union over the identities. That makes it commutative, associative and idempotent, with the empty lineage as the identity element. Those laws are what guarantee that the order in which you combine values cannot change which sources the answer admits to, which is why the Python tests them directly rather than trusting them.
Three things this refuses to do
- Construct a record with a missing mandatory field, because a figure whose licence nobody wrote down is not a figure you can publish
- Accept a timestamp with no timezone, because an instant that could be any of twenty six is not an instant
- Say whether two licences may be combined in a published figure. It will tell you they have been mixed. Whether that is permitted is a legal question with a jurisdiction attached, and neither this page nor the library behind it answers it
Method behind it: the provenance case study