Runmark

You are changing an API. Often an agent is helping. Before and after each change you need to run a real sequence of HTTP calls — authenticate, create something, read it back, exercise the failure path — and know what happened.

The usual tools handle the first request well and the sequence badly. The flow lives in a desktop application, so it is invisible in review and drifts from the code it tests. The run state is inside a session you cannot inspect. If step four fails, step one runs again. And when an agent is the one running the flow, none of it is legible: there is no artifact to read, only a screenshot or a paraphrase.

Runmark answers that shape of problem.

The whole design rests on one split. The runmark directory describes what should happen, and git tracks it alongside the code it tests. The artifacts directory records what did happen. Git never sees it.

In practice that split decides the product. Because intent is tracked, a request sequence appears in a pull request next to the change it validates, and a reviewer can argue about an assertion the same way they would argue about a test. Because runtime state is separate and durable, a run can pause at a checkpoint and resume later without repeating completed steps, and the record of a failure survives the terminal it happened in.

# yaml-language-server: $schema=https://raw.githubusercontent.com/exit-zero-labs/runmark/main/packages/contracts/schemas/request.schema.json
kind: request
title: Ping
method: GET
url: "{{baseUrl}}/ping"
expect:
  status: 200

# yaml-language-server: $schema=https://raw.githubusercontent.com/exit-zero-labs/runmark/main/packages/contracts/schemas/run.schema.json
kind: run
title: Smoke
env: dev
steps:
  - kind: request
    id: ping
    uses: ping

It also gives secrets an obvious home. Session ledgers are written redacted. Anything sensitive goes into an owner-only companion file beside them, and the audit export path never reads those files. Nobody has to remember to scrub a run record before sharing it.

Runmark ships a CLI and an MCP stdio server in a single installed package, and the rule between them is parity: the same definitions, the same session model, the same artifacts, the same safety rules. Not a reduced agent surface, and not a separate agent product.

Fourteen MCP tools cover what the CLI covers — listing and describing definitions, validating a project, running and resuming, reading session state and artifacts, pulling stream chunks, cancelling, explaining where a variable came from, and exporting an audit summary.

Building both against one engine kept the design honest. Anything an agent could not inspect was something a person could not debug, and the fix was the same in both cases: write it down as an artifact.

The packaging follows from the same idea. The MCP SDK is imported lazily, so someone who only ever runs the CLI does not pay for the agent surface at startup. Agents also tend to launch a server from somewhere other than the repository being tested, so every MCP call has to state its project root explicitly. Inheriting a working directory would quietly test the wrong project.

Validation runs before execution, not during it. Schema problems, broken wiring between steps, and unsafe configurations are caught while nothing has left the machine, because the alternative is discovering them halfway through a sequence that has already written to a real system.

Resume has a similar rule. If the tracked definitions changed since a session paused, or another process holds the session lock, continuing could mean executing a flow that no longer matches the record. Runmark refuses and exits with a distinct code, so a script or an agent can tell an unsafe resume apart from a failed request and decide what to do.

The tool sends no product telemetry. The only network traffic is the requests you configured.

Runmark is pre-1.0 and says so. It is published on npm, open source under MIT, and has been through several releases, including one that folded the separate MCP package into the CLI and one that tightened how agents identify a project.

The execution model and the file layout are settled. The command surface is still moving.