testing
── THE RULE THIS MODULE EXISTS TO ENFORCE ──────────────────────────────────Metrics come from the runner's OWN machine-readable reporter, never scraped
from the human-readable console output. A console format is presentation: it
changes with a runner upgrade, with `--verbose`, with terminal width, and it
changes without a version bump. A regex over it does not fail when it breaks —
it returns zero, and a card that says "0 tests" reads exactly like a card for a
lane that has no tests. That is the one kind of green this pipeline cannot tell
from success (contract §4.7: the floors are part of the gate, not a statistic).
So: `flutter test --machine` (NDJSON, one event per line) and
`vitest run --reporter=json` (a Jest-shaped object). Both are documented
protocols with stable field names. The same holds for the report-only inputs
added with `TestReport`: JUnit XML, Stryker's `mutation.json`, lcov and
istanbul's `coverage-summary.json` are all machine formats with a schema.
── TWO SHAPES, SIDE BY SIDE ────────────────────────────────────────────────
`TestMetrics` is the original wire contract with `slack` and does not move.
`TestReport` (`wildbit.test-report/v1`) is the richer org-wide standard —
durations, suites, slowest, flaky, scenarios, features, coverage, mutation,
perf — and travels ALONGSIDE it. `toMetrics` turns one into the other, so a
caller pinned to an older `slack` keeps working.
── WHERE THIS CODE COMES FROM ──────────────────────────────────────────────
Ported verbatim in behaviour from `pacha/app` (`parseFlutterMachine`,
`parseFlutterFailures`, `firstFrameIn`) and from `pacha-api` + `pacha-site`
(`parseVitestJson`, byte-identical in both). Every measured behaviour those
carried is preserved and the measurement is recorded next to it; a couple of
them look like bugs until you read why, and deleting one costs a red run or,
worse, a green one.
── COMPLEX VALUES TRAVEL AS JSON STRINGS ───────────────────────────────────
Dagger's TypeScript SDK exposes structural types poorly across a module
boundary, and a shape mismatch there fails at call time with an unreadable
error. Everything non-scalar crosses as a JSON string. The consumer does
`JSON.parse` and hands the same string straight to `slack.render` and
`slack.breakdown`.
Installation
dagger install github.com/wildbitca/daggerverse/testing@v0.2.0Entrypoint
Return Type
Testing Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
func (m *MyModule) Example() *dagger.Testing {
return dag.
Testing()
}@function
def example() -> dagger.Testing:
return (
dag.testing()
)@func()
example(): Testing {
return dag
.testing()
}Types
Testing 🔗
flutterMachine() 🔗
Parse a flutter test --machine report into metrics plus failure detail.
Returns the JSON of a FlutterReport: the six TestMetrics fields that
slack.render and slack.breakdown consume, plus failures with the file,
line, kind and message of every red test. Extra keys are ignored by slack,
so the same string goes to both without stripping anything.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | String ! | - | the raw NDJSON, one event per line. An unparseable line is skipped, not fatal — a truncated tail on a killed run must not discard the events before it. |
| exitCode | String ! | "0" | the suite’s REAL exit status, as the wrapper wrote it
( |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
flutter-machine --report string --exit-code stringfunc (m *MyModule) Example(ctx context.Context, report string, exitCode string) string {
return dag.
Testing().
Fluttermachine(ctx, report, exitCode)
}@function
async def example(report: str, exitcode: str) -> str:
return await (
dag.testing()
.fluttermachine(report, exitcode)
)@func()
async example(report: string, exitCode: string): Promise<string> {
return dag
.testing()
.flutterMachine(report, exitCode)
}flutterMachineFile() 🔗
flutterMachine, reading the report from a File.
Not sugar: pacha/app collects 4576 tests and its NDJSON runs to several
megabytes. A File stays in the engine and is read once here, instead of
crossing the module boundary as a multi-megabyte GraphQL string argument that
the orchestrator also has to hold in memory.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | File ! | - | the report file, e.g. |
| exitCode | String ! | "0" | see |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
flutter-machine-file --report file:path --exit-code stringfunc (m *MyModule) Example(ctx context.Context, report *dagger.File, exitCode string) string {
return dag.
Testing().
Fluttermachinefile(ctx, report, exitCode)
}@function
async def example(report: dagger.File, exitcode: str) -> str:
return await (
dag.testing()
.fluttermachinefile(report, exitcode)
)@func()
async example(report: File, exitCode: string): Promise<string> {
return dag
.testing()
.flutterMachineFile(report, exitCode)
}vitestJson() 🔗
Parse a vitest run --reporter=json report into metrics.
Returns the JSON of a TestMetrics — exactly the shape slack.render and
slack.breakdown consume.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | String ! | - | the raw JSON. An empty or malformed report yields zeroed
metrics rather than an error, on purpose: a runner that died
before writing |
| exitCode | String ! | "0" | the suite’s REAL exit status, as the wrapper wrote it
( |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
vitest-json --report string --exit-code stringfunc (m *MyModule) Example(ctx context.Context, report string, exitCode string) string {
return dag.
Testing().
Vitestjson(ctx, report, exitCode)
}@function
async def example(report: str, exitcode: str) -> str:
return await (
dag.testing()
.vitestjson(report, exitcode)
)@func()
async example(report: string, exitCode: string): Promise<string> {
return dag
.testing()
.vitestJson(report, exitCode)
}vitestJsonFile() 🔗
vitestJson, reading the report from a File.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | File ! | - | the report file, e.g. |
| exitCode | String ! | "0" | see |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
vitest-json-file --report file:path --exit-code stringfunc (m *MyModule) Example(ctx context.Context, report *dagger.File, exitCode string) string {
return dag.
Testing().
Vitestjsonfile(ctx, report, exitCode)
}@function
async def example(report: dagger.File, exitcode: str) -> str:
return await (
dag.testing()
.vitestjsonfile(report, exitcode)
)@func()
async example(report: File, exitCode: string): Promise<string> {
return dag
.testing()
.vitestJsonFile(report, exitCode)
}flutterReport() 🔗
flutter test --machine → TestReport: suites with durations, the 10
slowest tests, failures with lines, and flaky tests (failed, retried and
passed within the run).
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | String ! | - | the raw NDJSON (see |
| lane | String ! | - | what this run is, e.g. |
| exitCode | String ! | "0" | the suite’s REAL exit status (see |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
flutter-report --report string --lane string --exit-code stringfunc (m *MyModule) Example(ctx context.Context, report string, lane string, exitCode string) string {
return dag.
Testing().
Flutterreport(ctx, report, lane, exitCode)
}@function
async def example(report: str, lane: str, exitcode: str) -> str:
return await (
dag.testing()
.flutterreport(report, lane, exitcode)
)@func()
async example(report: string, lane: string, exitCode: string): Promise<string> {
return dag
.testing()
.flutterReport(report, lane, exitCode)
}flutterReportFile() 🔗
flutterReport, reading the report from a File (the NDJSON of a large suite runs to megabytes).
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | File ! | - | No description provided |
| lane | String ! | - | No description provided |
| exitCode | String ! | "0" | No description provided |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
flutter-report-file --report file:path --lane string --exit-code stringfunc (m *MyModule) Example(ctx context.Context, report *dagger.File, lane string, exitCode string) string {
return dag.
Testing().
Flutterreportfile(ctx, report, lane, exitCode)
}@function
async def example(report: dagger.File, lane: str, exitcode: str) -> str:
return await (
dag.testing()
.flutterreportfile(report, lane, exitcode)
)@func()
async example(report: File, lane: string, exitCode: string): Promise<string> {
return dag
.testing()
.flutterReportFile(report, lane, exitCode)
}vitestReport() 🔗
vitest run --reporter=json → TestReport. Also the Angular path: ng test
with @angular/build:unit-test runs vitest and takes --reporters=json.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | String ! | - | the raw JSON. Empty or malformed yields a zero report, and
|
| lane | String ! | - | see |
| exitCode | String ! | "0" | the suite’s REAL exit status |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
vitest-report --report string --lane string --exit-code stringfunc (m *MyModule) Example(ctx context.Context, report string, lane string, exitCode string) string {
return dag.
Testing().
Vitestreport(ctx, report, lane, exitCode)
}@function
async def example(report: str, lane: str, exitcode: str) -> str:
return await (
dag.testing()
.vitestreport(report, lane, exitcode)
)@func()
async example(report: string, lane: string, exitCode: string): Promise<string> {
return dag
.testing()
.vitestReport(report, lane, exitCode)
}vitestReportFile() 🔗
vitestReport, reading the report from a File.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | File ! | - | No description provided |
| lane | String ! | - | No description provided |
| exitCode | String ! | "0" | No description provided |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
vitest-report-file --report file:path --lane string --exit-code stringfunc (m *MyModule) Example(ctx context.Context, report *dagger.File, lane string, exitCode string) string {
return dag.
Testing().
Vitestreportfile(ctx, report, lane, exitCode)
}@function
async def example(report: dagger.File, lane: str, exitcode: str) -> str:
return await (
dag.testing()
.vitestreportfile(report, lane, exitcode)
)@func()
async example(report: File, lane: string, exitCode: string): Promise<string> {
return dag
.testing()
.vitestReportFile(report, lane, exitCode)
}junitReport() 🔗
JUnit XML → TestReport: Maestro --format junit, Cypress, vitest’s junit
reporter, or any other producer.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | String ! | - | No description provided |
| lane | String ! | - | No description provided |
| runner | String ! | "junit" |
|
| exitCode | String ! | "0" | the runner’s REAL exit status. Maestro exits non-zero when a flow fails and also when it never reached a device. |
| asScenarios | Boolean ! | true | true (default) fills |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
junit-report --report string --lane string --runner string --exit-code string --as-scenarios booleanfunc (m *MyModule) Example(ctx context.Context, report string, lane string, runner string, exitCode string, asScenarios bool) string {
return dag.
Testing().
Junitreport(ctx, report, lane, runner, exitCode, asScenarios)
}@function
async def example(report: str, lane: str, runner: str, exitcode: str, asscenarios: bool) -> str:
return await (
dag.testing()
.junitreport(report, lane, runner, exitcode, asscenarios)
)@func()
async example(report: string, lane: string, runner: string, exitCode: string, asScenarios: boolean): Promise<string> {
return dag
.testing()
.junitReport(report, lane, runner, exitCode, asScenarios)
}junitReportFile() 🔗
junitReport, reading the report from a File. A missing file is an empty report, folded by exitCode.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | File ! | - | No description provided |
| lane | String ! | - | No description provided |
| runner | String ! | "junit" | No description provided |
| exitCode | String ! | "0" | No description provided |
| asScenarios | Boolean ! | true | No description provided |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
junit-report-file --report file:path --lane string --runner string --exit-code string --as-scenarios booleanfunc (m *MyModule) Example(ctx context.Context, report *dagger.File, lane string, runner string, exitCode string, asScenarios bool) string {
return dag.
Testing().
Junitreportfile(ctx, report, lane, runner, exitCode, asScenarios)
}@function
async def example(report: dagger.File, lane: str, runner: str, exitcode: str, asscenarios: bool) -> str:
return await (
dag.testing()
.junitreportfile(report, lane, runner, exitcode, asscenarios)
)@func()
async example(report: File, lane: string, runner: string, exitCode: string, asScenarios: boolean): Promise<string> {
return dag
.testing()
.junitReportFile(report, lane, runner, exitCode, asScenarios)
}strykerReport() 🔗
Stryker mutation.json → TestReport carrying only mutation. REPORT-ONLY:
nothing here fails a build on a score.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | String ! | - | No description provided |
| lane | String ! | - | No description provided |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
stryker-report --report string --lane stringfunc (m *MyModule) Example(ctx context.Context, report string, lane string) string {
return dag.
Testing().
Strykerreport(ctx, report, lane)
}@function
async def example(report: str, lane: str) -> str:
return await (
dag.testing()
.strykerreport(report, lane)
)@func()
async example(report: string, lane: string): Promise<string> {
return dag
.testing()
.strykerReport(report, lane)
}strykerReportFile() 🔗
strykerReport, reading the report from a File.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | File ! | - | No description provided |
| lane | String ! | - | No description provided |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
stryker-report-file --report file:path --lane stringfunc (m *MyModule) Example(ctx context.Context, report *dagger.File, lane string) string {
return dag.
Testing().
Strykerreportfile(ctx, report, lane)
}@function
async def example(report: dagger.File, lane: str) -> str:
return await (
dag.testing()
.strykerreportfile(report, lane)
)@func()
async example(report: File, lane: string): Promise<string> {
return dag
.testing()
.strykerReportFile(report, lane)
}lcovReport() 🔗
lcov tracefile(s) → TestReport carrying only coverage. Several shards’
files may be concatenated into one string: records are unioned per line,
so a line covered by two shards counts once. Give it the SAME lane as the
suite it measured and merge folds it in without touching the counts.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | String ! | - | No description provided |
| lane | String ! | - | No description provided |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
lcov-report --report string --lane stringfunc (m *MyModule) Example(ctx context.Context, report string, lane string) string {
return dag.
Testing().
Lcovreport(ctx, report, lane)
}@function
async def example(report: str, lane: str) -> str:
return await (
dag.testing()
.lcovreport(report, lane)
)@func()
async example(report: string, lane: string): Promise<string> {
return dag
.testing()
.lcovReport(report, lane)
}lcovReportFile() 🔗
lcovReport, reading the tracefile from a File.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | File ! | - | No description provided |
| lane | String ! | - | No description provided |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
lcov-report-file --report file:path --lane stringfunc (m *MyModule) Example(ctx context.Context, report *dagger.File, lane string) string {
return dag.
Testing().
Lcovreportfile(ctx, report, lane)
}@function
async def example(report: dagger.File, lane: str) -> str:
return await (
dag.testing()
.lcovreportfile(report, lane)
)@func()
async example(report: File, lane: string): Promise<string> {
return dag
.testing()
.lcovReportFile(report, lane)
}istanbulReport() 🔗
istanbul coverage-summary.json (reporter json-summary) → TestReport carrying only coverage.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | String ! | - | No description provided |
| lane | String ! | - | No description provided |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
istanbul-report --report string --lane stringfunc (m *MyModule) Example(ctx context.Context, report string, lane string) string {
return dag.
Testing().
Istanbulreport(ctx, report, lane)
}@function
async def example(report: str, lane: str) -> str:
return await (
dag.testing()
.istanbulreport(report, lane)
)@func()
async example(report: string, lane: string): Promise<string> {
return dag
.testing()
.istanbulReport(report, lane)
}istanbulReportFile() 🔗
istanbulReport, reading the summary from a File.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | File ! | - | No description provided |
| lane | String ! | - | No description provided |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
istanbul-report-file --report file:path --lane stringfunc (m *MyModule) Example(ctx context.Context, report *dagger.File, lane string) string {
return dag.
Testing().
Istanbulreportfile(ctx, report, lane)
}@function
async def example(report: dagger.File, lane: str) -> str:
return await (
dag.testing()
.istanbulreportfile(report, lane)
)@func()
async example(report: File, lane: string): Promise<string> {
return dag
.testing()
.istanbulReportFile(report, lane)
}withPerf() 🔗
Attach perf measurements to a report. REPORT-ONLY: a value over its
budget is printed as over budget and fails nothing.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | String ! | - | JSON TestReport |
| perf | String ! | - | JSON |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
with-perf --report string --perf stringfunc (m *MyModule) Example(ctx context.Context, report string, perf string) string {
return dag.
Testing().
Withperf(ctx, report, perf)
}@function
async def example(report: str, perf: str) -> str:
return await (
dag.testing()
.withperf(report, perf)
)@func()
async example(report: string, perf: string): Promise<string> {
return dag
.testing()
.withPerf(report, perf)
}merge() 🔗
Merge reports — shards of one lane, or several lanes — into one.
The rules (scenario retries, coverage weighting, lanes) are documented on
mergeReports; the short version: same lane sums, different lanes are
summed into the top line AND kept per lane in lanes.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| reports | String ! | - | JSON array of TestReports (a single object is accepted too) |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
merge --reports stringfunc (m *MyModule) Example(ctx context.Context, reports string) string {
return dag.
Testing().
Merge(ctx, reports)
}@function
async def example(reports: str) -> str:
return await (
dag.testing()
.merge(reports)
)@func()
async example(reports: string): Promise<string> {
return dag
.testing()
.merge(reports)
}features() 🔗
Assign tests and scenarios to features and report where the mapping drifts.
Returns JSON {report, ok, problems, unassignedTests, unknownFeatures,
unmappedScenarios, undeclaredFeatures}; report.features is filled.
Report-only by default: drift is RETURNED, and the consumer’s guard decides.
Run it after merge — the feature rows are computed from the final suites.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | String ! | - | JSON TestReport |
| specsDir | Directory ! | - | the |
| map | File | - | a coverage map elsewhere (e.g. |
| strict | Boolean ! | false | throw with the problems instead of returning |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
features --report string --specs-dir DIR_PATH --strict booleanfunc (m *MyModule) Example(ctx context.Context, report string, specsDir *dagger.Directory, strict bool) string {
return dag.
Testing().
Features(ctx, report, specsDir, strict)
}@function
async def example(report: str, specsdir: dagger.Directory, strict: bool) -> str:
return await (
dag.testing()
.features(report, specsdir, strict)
)@func()
async example(report: string, specsDir: Directory, strict: boolean): Promise<string> {
return dag
.testing()
.features(report, specsDir, strict)
}featureMap() 🔗
features without the filesystem: the feature ids and the map text given
directly. Pure, so the mapping is testable on fixtures.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | String ! | - | No description provided |
| featureIds | String ! | - | JSON string array, e.g. |
| map | String ! | "" | the coverage.tsv text; empty means no rows at all |
| strict | Boolean ! | false | No description provided |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
feature-map --report string --feature-ids string --map string --strict booleanfunc (m *MyModule) Example(ctx context.Context, report string, featureIds string, map string, strict bool) string {
return dag.
Testing().
Featuremap(ctx, report, featureIds, map, strict)
}@function
async def example(report: str, featureids: str, map: str, strict: bool) -> str:
return await (
dag.testing()
.featuremap(report, featureids, map, strict)
)@func()
async example(report: string, featureIds: string, map: string, strict: boolean): Promise<string> {
return dag
.testing()
.featureMap(report, featureIds, map, strict)
}toMetrics() 🔗
TestReport → legacy TestMetrics, for a caller whose pinned slack only
reads metrics. failedNames also lists red scenarios.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| report | String ! | - | No description provided |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
to-metrics --report stringfunc (m *MyModule) Example(ctx context.Context, report string) string {
return dag.
Testing().
Tometrics(ctx, report)
}@function
async def example(report: str) -> str:
return await (
dag.testing()
.tometrics(report)
)@func()
async example(report: string): Promise<string> {
return dag
.testing()
.toMetrics(report)
}summaryMarkdown() 🔗
Markdown for $GITHUB_STEP_SUMMARY: totals, lanes, failures, flaky, the 10
slowest, suites, scenarios, the feature matrix, coverage, mutation and perf.
Return Type
String !Arguments
| Name | Type | Default Value | Description |
|---|---|---|---|
| reports | String ! | - | JSON array of TestReports (or one object); one section each. Pass the merged report for one combined section. |
Example
dagger -m github.com/wildbitca/daggerverse/testing@b28df5b99999d2299ff8be12ede84bb70d2860c0 call \
summary-markdown --reports stringfunc (m *MyModule) Example(ctx context.Context, reports string) string {
return dag.
Testing().
Summarymarkdown(ctx, reports)
}@function
async def example(reports: str) -> str:
return await (
dag.testing()
.summarymarkdown(reports)
)@func()
async example(reports: string): Promise<string> {
return dag
.testing()
.summaryMarkdown(reports)
}