Batching and parallelism

Marathon Cloud runs your suite on a pool of devices in parallel. Two decisions shape every run: how many devices to use, and how many tests to send to a device at once. Both are made for you from your suite’s history; this page explains the defaults and the two flags that override them.

Parallelism

When a run starts, the scheduler estimates the total execution time of the suite from previous runs (or from a conservative default the first time) and sizes the device pool so the run finishes in about 15 minutes. A suite that needs two hours of device time gets roughly eight devices; a suite that needs ten minutes gets one or two.

Fifteen minutes is a target, not a hard limit. Very long individual tests, a suite with no history yet, or an unusually slow batch can push a run past it.

Batching

A batch is a group of tests sent to one device together. The tests run in sequence and their results, logs, and videos come back as a group. Between batches the device pulls the previous batch’s artifacts, receives the next batch, and starts it; the app is installed once when the device is set up, not per batch.

Batching amortizes that per-batch overhead. Sending tests one at a time would spend more time on transfer than on testing. The scheduler picks batch sizes from each test’s historical duration so that no single batch dominates the run. You do not configure this directly.

Within a batch, tests run in the same app process in sequence, the same way they do when you run a class locally. State a test leaves behind, such as logged-in sessions, files, or preferences, is visible to the tests that follow it in the batch.

Isolated execution

--isolated sets the batch size to one, so no test shares a batch with another.

marathon-cloud run android \
  --application app-debug.apk \
  --test-application app-debug-androidTest.apk \
  --isolated

Use it to confirm whether a failure depends on tests that ran before it. Expect the run to take longer and cost more, since per-batch overhead is now paid once per test. Once you have found the leaking state, fix the test and drop the flag.

On iOS, --batch-isolation uninstall_app removes and reinstalls the app between batches without going down to one test per batch.

Limiting concurrency

--concurrency-limit N caps the pool at N devices.

marathon-cloud run android \
  --application app-debug.apk \
  --test-application app-debug-androidTest.apk \
  --concurrency-limit 4

This is an escape hatch, not a tuning knob. Use it when the tests, or the systems they talk to, cannot cope with parallel execution:

• Tests share a backend account or fixture and interfere with each other when run at the same time.
• A staging backend falls over under the load of many devices hitting it at once.
• Tests depend on a rate-limited third-party API.

A limit slows the run in proportion: a two-hour suite capped at four devices takes about thirty minutes. The better fix is to make tests independent and give the backend headroom, then remove the limit.

Debugging “passes locally, fails on Cloud”

The usual cause is state your local emulator has accumulated that the test silently depends on: a logged-in account, granted permissions, cached data, a snapshot restored at boot.

To reproduce the Cloud environment locally:

1. Wipe the emulator’s data and delete its snapshots.
2. Cold boot it.
3. Run the failing test alone, then as part of its class.

If it passes alone but fails in the class, it depends on another test’s state; --isolated on Cloud will confirm it. If it fails alone on a clean emulator, the test depends on environment that the cloud device does not have, such as a pre-installed app, a system setting, or network access to something internal.

Note

See retries for how failures inside a batch are re-run.