Retiring a Board: User Testing a CLI Before You Ship It

We had a Linear board to retire. Gleamcave, 50 stories across four columns, a project making way for its successor. We also had a brand-new CLI command that promised to do the whole job: eko stories fetch to read a column, eko stories archive to empty it, eko stories create to push the next project's backlog. Every behaviour was pinned by a test. The suite was green. So we were done, right?

We spent an afternoon user testing it against the real board before shipping. The suite had proven every behaviour we designed. The afternoon found four things we hadn't designed. That gap is what this post is about.

Start With a Baseline You Didn't Build

Here is the trap: if you check a tool by looking at the tool's own output, you are testing whether it agrees with itself. It always does.

So before the CLI touched anything, we read the board without it, using three raw GraphQL queries with curl. Who am I, and which workspace is this key for? What columns does this team actually have? And the big one: every issue, with its column.

curl -s https://api.linear.app/graphql \
  -H "Authorization: $LINEAR_API_KEY" -H "Content-Type: application/json" \
  -d '{"query":"{ issues(filter:{team:{key:{eq:\"GLE\"}}}, first:100) { nodes { identifier state { name } } } }"}'

Grouped by column: Backlog 33, Done 9, In Progress 4, Todo 4. Fifty issues, and now we had their identifiers from a source the CLI had never seen. Everything the CLI claimed from here on could be checked against that list.

If you have ever written a characterisation test, this is the same move. The baseline is the pin.

Read Everything Before You Change Anything

fetch is read-only, which makes it free to test. So we wrung it dry first.

$ eko stories fetch GLE Backlog
  page 1: 33 stories
GLE-6 [Story,Website,Feature] Homepage: fast bilingual hero
GLE-7 [Story,Website] Language switch EL/EN with localized URLs + SEO
...

We diffed the identifiers against the baseline: exact match, all four columns. Then the backups. A third argument writes the full JSON, six fields per story, so we saved one file per column outside the repo. Those files matter in a minute, because we were about to archive everything they describe.

The negative tests came next. Unset the API key and the command stops before making a single request:

$ LINEAR_API_KEY= eko stories fetch GLE Backlog
stopped: stories fetch needs LINEAR_API_KEY

Sequence the risky after the safe. By the time anything could mutate the board, the reading half had earned our trust.

The Wrong-Workspace Episode

Then the first real run of create fell over:

$ eko stories create GLE gle-test-backlog.json
stopped: no team with key GLE

The CLI was fine. The terminal was not: it held a stale API key from an earlier session. A valid key, happily authenticating, for the wrong Linear workspace. A workspace with no team called GLE. The command's named stop did exactly its job.

But here is the uncomfortable part. The same wrong key under fetch doesn't stop. It prints page 1: 0 stories and exits zero, a lie with a straight face. An empty result and a wrong environment look identical.

No unit test would ever have caught this. In the test suite, the null Linear never has a wrong key. We never designed that failure, so we never pinned it. Environment is part of the system under test, and only the real environment can play its part.

Create, and Prove the Tree

With the right key exported, we pushed a disposable test backlog: two phases, nested children, one grandchild three levels deep, all titled [test]. First the preview:

$ eko stories create GLE gle-test-backlog.json --dry-run
  would have created [test] Phase 1: walking skeleton
    would have created [test] Deploy hello-world to staging
      would have created [test] Provision staging DNS
    would have created [test] Smoke test on staging
  ...
  would have created 7 stories in GLE

Then the real run, answering n at the confirm first and checking nothing appeared. Then y. Seven cards, GLE-51 through GLE-57, minted depth-first. And again: don't take the tool's word for it. A GraphQL query for the new cards' parent links:

GLE-51  parent=-       [test] Phase 1: walking skeleton
GLE-52  parent=GLE-51  [test] Deploy hello-world to staging
GLE-53  parent=GLE-52  [test] Provision staging DNS
GLE-54  parent=GLE-51  [test] Smoke test on staging

The nesting mirrors the JSON exactly, grandchild included. The parent-before-children promise, proven by the board itself.

A confession from the transcript: the create ran twice, so GLE-58 through GLE-64 were duplicates of 51 through 57. Nothing malfunctioned; the command is append-only by design. But now we know something the design never states out loud: a second "y" on the same file doubles a backlog. Finding, noted, documented.

The Typo Episode

Now the destructive half. First target: a small column.

$ eko stories archive GLE InProgress
  nothing to archive in GLE/InProgress

Except the column is called "In Progress", with a space. Four live stories sat there, untouched, while the command reported something indistinguishable from success. The same ambiguity that hid the wrong workspace key had just bitten a plain human typo. For the second time in an hour, "does not exist" wore the costume of "is empty".

One occurrence is an anecdote. Two in an hour is a specification. That evening it became a written story for the next release: unknown teams and columns are named stops. The fix is one query away, because the team's real column names are right there to ask for, and the stop can simply say what exists: no column "InProgress" in GLE (it has: Backlog, Todo, In Progress, ...).

This is the honest economics of user testing. The suite pins every behaviour you thought of. An hour of real use writes the stories you didn't.

The Tidy, and Proving "Archive, Not Delete"

With the quotes in place, the sweep: every column, backed up first, then archived. The confirm names the count before anything happens; the mutations go up in batches of 25 behind GraphQL aliases, so 40 stories cost two round trips, not forty; and a re-run converges immediately on "nothing to archive", which is what makes a partial failure safe to retry.

Then the claim that justifies the whole design. Archiving is reversible, so the board's record survives its tidying. We proved it the same way as everything else, with a query the CLI didn't run:

• The live board: 0 issues.
• The same query with includeArchived: true: all 64, every one stamped with archivedAt.

The column went back to the present tense. The history is one query flag away. That is the difference between tidying and shredding, and it is a claim worth proving rather than assuming.

What the Afternoon Bought

Four findings, none of them from the test suite:

1. An unknown team or column reads as an empty one. Bit a wrong key, then bit a typo. Now a written story for the next release.
2. No whole-board overview. We reached for "show me every column with its count" mid-test and it wasn't there. Story candidate.
3. create is append-only. A second approved run duplicates the backlog. Documented.
4. Unlabelled stories print an empty []. Cosmetic, queued as polish.

And none of them blocked the release. Every claim the release actually makes had now been proven twice, once by the suite against the null and once by the real board. So we shipped.

The suite proves the behaviours you designed. User testing finds the behaviours you didn't. You need both, and in that order, because an afternoon on a real board is only cheap when the suite has already made the tool boring.