Retries

Marathon Cloud retries individual tests, not whole suites. The goal is to separate tests that are genuinely broken from tests that fail intermittently, without you re-running the pipeline.

Kinds of retries

Reactive retries

When a test fails, it is scheduled again on a different device than the one it failed on. A test that passes on any attempt is reported as flaky; a test that fails every attempt is reported as failed. The number of reactive retries per test is set by the platform from your suite’s history, and can be overridden with --retry-quota-test-reactive.

Preventive retries

Marathon Cloud keeps a per-test history of outcomes across your runs. When a test has been unstable recently, the scheduler may run it more than once from the start, without waiting for a failure. This costs a little device time but returns a verdict sooner than failing first and retrying later, which matters for the 15-minute target. The preventive quota is set by the platform and can be overridden with --retry-quota-test-preventive.

Uncompleted executions

A test that never produces a result — the device crashed, the process was killed, the batch timed out — is treated as uncompleted rather than failed. Uncompleted executions are re-run under a separate quota, --retry-quota-test-uncompleted, so infrastructure problems do not consume the retries reserved for real test failures.

What the report shows

Each test in the report is one of:

Verdict	Meaning
Passed	Passed on the first attempt.
Flaky	Failed at least once, then passed on a retry.
Failed	Failed on every attempt.
Ignored	Skipped by the test framework.

Flaky tests count as passing for the run’s exit code, and are listed separately so you can find them. The flakiness history used by preventive retries is internal to the scheduler and is not currently shown in the console.

Configuration

Retries are on by default with quotas chosen by the platform. Override them per run:

marathon-cloud run android \
  --application app-debug.apk \
  --test-application app-debug-androidTest.apk \
  --retry-quota-test-reactive 2 \
  --retry-quota-test-preventive 0 \
  --retry-quota-test-uncompleted 1

Disable all retries:

marathon-cloud run android \
  --application app-debug.apk \
  --test-application app-debug-androidTest.apk \
  --no-retries

Both examples read the API key from MARATHON_CLOUD_API_KEY.

--no-retries sets every quota to zero and cannot be combined with the --retry-quota-* flags.

Note

Turning retries off is useful when you are measuring raw flakiness or debugging a specific test. For gating pull requests, leave them on.

Reducing flakiness at the source

Retries hide flakiness; they do not fix it. Tests that show up as flaky repeatedly are candidates for:

• Isolation: run with --isolated to rule out state leaking between tests in a batch. See batching.
• Concurrency: if flakiness correlates with suite size, your shared backend may be the bottleneck. See limiting concurrency.
• Refactoring: replace sleeps with idling resources or explicit waits, and remove dependencies on test ordering.