Back to Ideas 3 min read

Reconstructing the Past

E
EkoHacks Team
·
Reconstructing the Past

You connect a repository that has a year of history behind it. The platform was not watching when those pull requests were opened and merged, it had never heard of this repo until a moment ago. Yet the moment you connect it, a profile fills in and a trend chart draws a curve reaching back months. How does a system show you a past it never witnessed? Two moves: onboarding turns strangers into records, and backfill replays history into the shape the dashboards expect.

Onboarding: strangers become records

The repository's pull requests were written by people, identified by their GitHub usernames. The platform attributes work to participants, its own internal records. So the first job is to bridge the two: for each pull request author, is this someone we already know?

The importer holds a map from GitHub username to participant, and for each author it does a match or create: if the username is already a participant, attribute the PR to them; if not, and they are a real person (bots are skipped), create a participant so the work has somewhere to land. This is a pattern you will meet everywhere data crosses a boundary: an external identity (a GitHub login, an email, a customer id) has to be resolved to an internal one, and you either find the existing record or mint a new one.

There is a deliberate choice hiding here. When you connect your own repo on the free tier, the importer runs with onboarding turned off, so it attributes pull requests to people already on the platform but does not create accounts for every stranger who ever contributed. Connecting your side project should show your stats, not silently enrol a dozen people who never asked to be here. The same code, one flag, two very different social contracts.

Backfill: replaying history into snapshots

Now the harder half. The trend chart is not drawn from raw pull requests; it is drawn from weekly snapshots, one row per developer per week holding that week's merge rate, throughput and the rest. During normal operation those snapshots are written as events arrive. But the history predates us, so there are no snapshots for it. We have to backfill: compute what each week's snapshot would have been had we been watching.

So the importer walks each active developer week by week, and for every week they opened at least one pull request, it computes that week's snapshot from the pull requests that fall in it. Empty weeks are skipped, so the trend is made of real points rather than a run of zeroes. The result is a curve that looks exactly as if the platform had been recording all along, reconstructed from the one source of truth it does have: the pull requests themselves.

How far back? Deliberately bounded. The pull requests are all imported, the whole history. But the weekly snapshots reach back at most one year (52 weeks) from a developer's earliest pull request. A developer with three years of history gets every PR stored, and a trend covering their most recent year. The bound is a choice: enough history to be useful, not so much that a single connect drags the database through a decade of dead weeks.

The lesson

Backfill is a specific idea worth naming: derived state can be recomputed from the source of truth at any time. The snapshots are not precious; they are a function of the pull requests. Because that function exists, a repo we met five seconds ago can have a year of trend, and if we ever change how a snapshot is computed, we can rebuild every one of them.

The mindset: keep a clear source of truth, keep derived data honestly derived, and you can always reconstruct the past, whether you were watching when it happened or not. That is also why "connect and see your history in minutes" is a feature we can actually promise.

E

Written by

EkoHacks Team

More from Ideas

·6 min read

What the Nullable Gave Back

One file, seven behaviours held fixed, the database swapped for a Nullable: about 180 times less time inside the tests, and coverage flat to two decimals.

E
EkoHacks Team
·6 min read

Twenty Six More Tests, Four Fewer Behaviours

Removing the mocks grew the suite from 44 tests to 70 and quietly deleted four behaviours, every one of them a failure path. Test count is not coverage.

E
EkoHacks Team
·6 min read

The Best Coverage Number in the Room

Same commit, same spec, same test count. The mocked suite ran 5.6 times faster, covered 3.5 fewer points of real code, and posted the best branch coverage.

E
EkoHacks Team