provenance

A grid intensity figure is 412 gCO2e per kWh. Six months later somebody asks where it came from, and the honest answer is usually a shrug: the number was copied out of a portal, the portal has restated it since, and the parameters that produced it are gone. Nothing about the way the figure was stored made that outcome avoidable. It was stored as a number.

Keeping a citation beside the number does not survive contact with a calculation. The moment two sourced figures are divided, multiplied or summed, the result is an ordinary float and the notes are somewhere else, in a comment or a cell to the right or a document nobody opens. Published work is almost entirely derived figures, so the place the provenance is needed is precisely the place it has been dropped.

There is a quieter failure underneath. A dataset has versions, and a retrieval has an instant. A figure read in March from the 2024 edition is not the same figure as one read in September from the same dataset after a restatement, and stored as bare numbers the two are indistinguishable. Neither is a figure whose query nobody wrote down: same dataset, same version, same instant, different country code, and two entirely different numbers that look identical in a spreadsheet.

A cited value is a magnitude, a unit and a lineage, and the lineage is what the arithmetic preserves. Five fields are mandatory on every record: source, dataset, version, retrieval instant and licence. A record will not construct without them, and where a source states no licence the entry is the word unstated, so that the gap is recorded rather than hidden. Identity is the canonical form of the record, meaning sorted keys at every depth and the timestamp normalised to UTC, hashed. Two records identify as the same retrieval exactly when they describe the same retrieval, however the fields were assembled and in whichever offset the time was written.

Merging two lineages is union over those identities. That makes the operation commutative, associative and idempotent, with the empty lineage as the identity element, and those laws are exactly what guarantees that the order in which values are combined cannot change which sources the answer admits to. The Python tests them directly rather than trusting them, and sweeps the central invariant over four hundred randomly generated arithmetic expression trees: for each one the resulting lineage must be precisely the set of leaves that contributed, and the resulting magnitude must equal the same expression computed on plain floats.

Two records side by side, every field editable. A button constructs them and a second button takes a mandatory field away so that the refusal can be seen rather than described. Below that, an operation selector combines the two and prints the result with the sources it admits to. A third panel audits both records and the derived value as one document: what has gone stale, what has no address, what recorded no query, and whether the licences have been mixed.

It refuses to construct a record with a mandatory field missing. There is a button on the page whose only job is to empty the licence field, because the refusal is the feature and a described refusal is not the same as a seen one. A figure whose licence nobody wrote down is not a figure you can publish, and the library raises rather than filling the gap.

It refuses a timestamp with no timezone. An instant that could be any of twenty six is not an instant, and assuming UTC would silently make two different retrievals identify as one. The page names the shape it wants instead of guessing at the shape it was given.

It refuses to say whether two licences may be combined in a published figure. It will tell you they have been mixed, and it will name them. Whether combining them is permitted is a legal question with a jurisdiction attached, and answering it is not something a library can do from inside a calculation. Making sure you know before you publish rather than after is something it can do, and that is where it stops.

This is the foundation the rest of that repository is built on, and it is the piece of it that generalises furthest. The claim any published figure makes is not really about the number. It is about whether anybody can get back to where the number came from, and that is a property of how it was stored long before it is a property of how carefully it was quoted.

• github.com/Keremozdemirra/open-climate-data
• Read this as a standalone page