Kerem Özdemir

provenance

A figure that carries its source, and keeps carrying it through the arithmetic.

5 minute read

The problem

A grid intensity figure is 412 gCO2e per kWh. Six months later somebody asks where it came from, and the honest answer is usually a shrug: the number was copied out of a portal, the portal has restated it since, and the parameters that produced it are gone. Nothing about the way the figure was stored made that outcome avoidable. It was stored as a number.

Keeping a citation beside the number does not survive contact with a calculation. The moment two sourced figures are divided, multiplied or summed, the result is an ordinary float and the notes are somewhere else, in a comment or a cell to the right or a document nobody opens. Published work is almost entirely derived figures, so the place the provenance is needed is precisely the place it has been dropped.

There is a quieter failure underneath. A dataset has versions, and a retrieval has an instant. A figure read in March from the 2024 edition is not the same figure as one read in September from the same dataset after a restatement, and stored as bare numbers the two are indistinguishable. Neither is a figure whose query nobody wrote down: same dataset, same version, same instant, different country code, and two entirely different numbers that look identical in a spreadsheet.

The method

A cited value is a magnitude, a unit and a lineage, and the lineage is what the arithmetic preserves. Five fields are mandatory on every record: source, dataset, version, retrieval instant and licence. A record will not construct without them, and where a source states no licence the entry is the word unstated, so that the gap is recorded rather than hidden. Identity is the canonical form of the record, meaning sorted keys at every depth and the timestamp normalised to UTC, hashed. Two records identify as the same retrieval exactly when they describe the same retrieval, however the fields were assembled and in whichever offset the time was written.

Merging two lineages is union over those identities. That makes the operation commutative, associative and idempotent, with the empty lineage as the identity element, and those laws are exactly what guarantees that the order in which values are combined cannot change which sources the answer admits to. The Python tests them directly rather than trusting them, and sweeps the central invariant over four hundred randomly generated arithmetic expression trees: for each one the resulting lineage must be precisely the set of leaves that contributed, and the resulting magnitude must equal the same expression computed on plain floats.

identity(record)   = hash of the canonical form
                     sorted keys at every depth, timestamp in UTC

merge(L1, L2)      = union over identity, first seen order kept
                     commutative, associative, idempotent

value op value     magnitude by ordinary arithmetic
                   lineage by merge, and never by anything else

The example on the page is two invented agencies reporting grid intensity for two invented countries, 412 and 188 gCO2e per kWh. Divide the first by the second and the answer is 2.1915, dimensionless because identical units cancel, and it still lists both retrievals with both licences. The second record was written with a +02:00 offset and is normalised to 20:00Z before hashing, which is what lets two records written in different offsets identify as the same retrieval.

The document panel then reads the whole thing as one artefact. Both retrievals are past the staleness threshold, one carries no address to return to and one recorded no query, and the two licences differ. In the Python that combination is what strict mode exits non zero on, and the intended reader of that exit code is a build rather than a person.

What it does

Two records side by side, every field editable. A button constructs them and a second button takes a mandatory field away so that the refusal can be seen rather than described. Below that, an operation selector combines the two and prints the result with the sources it admits to. A third panel audits both records and the derived value as one document: what has gone stale, what has no address, what recorded no query, and whether the licences have been mixed.

Mandatory fieldssource, dataset, version, retrieval instant, licence
Optional fieldsthe query and the address, both reported when absent rather than ignored
Operationsdivide, multiply, add, subtract, with addition refused across different units
Identitythe canonical form, shown in full, and its fingerprint
Auditstaleness against an adjustable threshold and an adjustable reading date

What it refuses to do

It refuses to construct a record with a mandatory field missing. There is a button on the page whose only job is to empty the licence field, because the refusal is the feature and a described refusal is not the same as a seen one. A figure whose licence nobody wrote down is not a figure you can publish, and the library raises rather than filling the gap.

It refuses a timestamp with no timezone. An instant that could be any of twenty six is not an instant, and assuming UTC would silently make two different retrievals identify as one. The page names the shape it wants instead of guessing at the shape it was given.

It refuses to say whether two licences may be combined in a published figure. It will tell you they have been mixed, and it will name them. Whether combining them is permitted is a legal question with a jurisdiction attached, and answering it is not something a library can do from inside a calculation. Making sure you know before you publish rather than after is something it can do, and that is where it stops.

Try it

Press the button that takes the licence away and watch three panels stop at once. Then put it back and change the retrieval instant on value B to any other offset: the fingerprint does not move, because the record describes the same retrieval.

Open provenance

github.com/Keremozdemirra/open-climate-data

This is the foundation the rest of that repository is built on, and it is the piece of it that generalises furthest. The claim any published figure makes is not really about the number. It is about whether anybody can get back to where the number came from, and that is a property of how it was stored long before it is a property of how carefully it was quoted.

All tools