Skip to main content
Version: develop

Configuration

To indicate that you're using a vendor config for iOS/watchOS/tvOS/visionOS you have to specify the type in configuration as following:

vendorConfiguration:
type: "iOS"
additional_option1: ...
additional_option2: ...
caution

All the options below should be placed under the vendorConfiguration with appropriate yaml indentation

Required options

Test bundle

Marathon can run both XCUITests and XCTests. Test bundle requires you to specify application under test as well as test application. After preprocessing both of these inputs are distilled into an application bundle (e.g. my.app) and xctest bundle (e.g. my-tests.xctest) You can specify .ipa application archives as well as .zip with the same content as application archive. They will be searched for the application and xctest bundles. If there are multiple entries matching description - marathon will fail.

Raw bundles (.app + .xctest)

application: "build/my.app"
testApplication: "build/my.xctest"

Archive bundles (.ipa/.zip)

tip

It is much easier to supply the .app application bundle and .xctest bundle directly instead of wasting time on packaging a signed application archive and depending on marathon's runtime type discovery of your bundles

If you want to specify your bundles as .ipa/.zip:

application: "build/my.ipa"
testApplication: "build/my.zip"

Derived data dir

derivedDataDir: "derivedDataDir/"

Test type

Marathon will detect if the specified .xctest is a XCUITest bundle or XCTest bundle. If you want to save some execution time you can explicitly specify this:

testType: "xcuitest"

Extra applications

Marathon can install additional applications that might be required for testing:

extraApplications:
- "/path/to/additional.app"

Devices

By default, marathon will look for a file Marathondevices in the same folder as Marathonfile for the configuration of workers. You can override this location with the following property:

devices: my/devices.yaml

For the documentation of the format of this file refer to worker's documentation.

SSH

tip

This section is only required if you're using remote workers and want to provide the same ssh configuration for all of those workers. You can always specify and/or override this explicitly for each worker

For each ssh connection you want to specify authentication, identifying known hosts and keep-alive:

Public Key Authentication

To authenticate using private key and a username:

ssh:
authentication:
type: "publicKey"
username: "username"
key: "/home/user/.ssh/id_rsa"

Password Authentication

To authenticate using username and password:

caution

sshpass is required to allow rsync to pick up username+password credentials.

ssh:
authentication:
type: "password"
username: "username"
password: "storing-password-here-is-a-bad-idea"
tip

Storing ssh password in a configuration file directly is a bad idea. Refer to dynamic configuration and utilize the envvar interpolation to provide the password for your test runs during runtime.

Known hosts

When ssh establishes connection to a remote host it tries to verify the identity of the remote host to mitigate potential men-in-the-middle attack. You can specify the known_hosts file in the OpenSSH format as following:

ssh:
knownHostsPath: "/home/user/.ssh/known_hosts"
caution

If you omit this configuration then marathon will trust any remote host. This is a bad idea for production.

Secure Shell debug

If you are experiencing issues with ssh connections and want to have more information use the following debug flag. Caution: a lot of data is written to stdout when using this flag.

ssh:
debug: true

Optional

Collecting xcresult

By default, marathon will pull the xcresult bundle into the output folder under device files and cleanup the remote worker to not bloat the worker storage. To change this override the following:

info

As of the time of writing marathon doesn't support merging the xcresult and treats them as just regular file artifacts.

  xcresult:
pullingPolicy: ALWAYS
remoteClean: true

Possible values for the pullingPolicy are ALWAYS (by default), NEVER and ON_FAILURE - which means pulling the xcresult only for failed batches (if at least one test in the batch is failed), including retried failures.

Attachment lifetime

Marathon generates the xctestrun file for each batch and can specify custom lifecycle attachments. By default, system attachments will be deleted on success and user attachments will always be kept in the xcresult, but you can override this:

  xcresult:
attachments:
systemAttachmentLifetime: DELETE_ON_SUCCESS
userAttachmentLifetime: KEEP_ALWAYS

Possible values for the lifetime are KEEP_ALWAYS, DELETE_ON_SUCCESS and KEEP_NEVER.

Screen recorder configuration

By default, marathon will record a h264-encoded video of the internal display with black mask if it is supported. If you want to force screenshots or configure the recording parameters you can specify this as follows:

screenRecordConfiguration:
preferableRecorderType: "screenshot"

Video recorder configuration

Apple's video recorder can encode videos using codec h264 and hevc.

caution

HEVC encoded videos are not supported by some web browsers. Such videos might not be playable in html reports that marathon produces

screenRecordConfiguration:
videoConfiguration:
enabled: true
codec: h264
display: internal
mask: black

The display field can be either internal or external. The mask field can be either black or ignored.

Screenshot configuration

Marathon can resize and combine screenshots from device into a GIF image

screenRecordConfiguration:
screenshotConfiguration:
enabled: true
type: jpeg
display: internal
mask: black
width: 720
height: 1280
# ISO_8601 duration
delay: PT1S

The display and mask fields have the same options as the video recorder. The type specifies the format of a single frame and is advised not to be changes. The delay field specifies the minimal delay between frames using ISO 8601 notation.

Push media files to apple simulator before test run

Sometimes you need to push some media files such as photos, live photos, videos, or contacts to the library of a device. Here is how you can configure it:

info

All files will be pushed using the simctl addmedia command, which is only available for apple simulators.

This import will only be done once per device provisioning, so in the simplest case once before a test run. So if the test manipulates these in any way - it's the test's responsibility to reset it to the state needed.

danger

The imported files will remain on the device throughout its entire life cycle. New test runs will produce new file imports. To avoid increasing the simulator storage growth or side effects from the previous test runs - read how to erase it before/after test runs - Test run lifecycle

importMedia:
- "media/photo.jpg"
- "media/video.mp4"
- "media/contact.vcf"

xctestrun Environment and TestingEnvironment variables

You can specify additional Environment and TestingEnvironment variables for your test run:

xctestrunEnv:
app: # EnvironmentVariables
MY_ENV_VAR_1: A
MY_ENV_VAR_2: B
test: # TestingEnvironmentVariables
MY_TEST_ENV_VAR_1: AA
MY_TEST_ENV_VAR_2: BB

These will be placed in the generated xctestrun property list file under the EnvironmentVariables and TestingEnvironmentVariables keys.

info

Marathon generates required values for DYLD_FRAMEWORK_PATH, DYLD_LIBRARY_PATH and DYLD_INSERT_LIBRARIES for test environment. If you specify custom ones then your values will be placed as a lower priority path elements at the end of the specified envvar.

xcodebuild test-without-building arguments

You can specify additional arguments to pass to the underlying xcodebuild test-without-building invocation.

xcodebuildTestArgs:
"-test-timeouts-enabled": "YES"
"-maximum-test-execution-time-allowance": "60"

It is impossible to override the following reserved arguments:

  • -xctestrun
  • -enableCodeCoverage
  • -resultBundlePath
  • -destination-timeout
  • -destination

Test run lifecycle

Marathon provides two lifecycle hooks: onPrepare and onDispose. For each you can specify one of the following actions: SHUTDOWN (shutdown simulator), ERASE (erase simulator) and TERMINATE (terminate simulator).

These can be useful during provisioning of workers, e.g. you might want to erase the existing simulators before using them

danger

If you specify TERMINATE marathon will kill -SIGKILL the simulators. This usually results in simulators unable to boot with black screen as well as a number of zombie processes and can only be resolved by erasing the state. In most cases SHUTDOWN is the recommended action.

tip

If you specify ERASE then marathon will first shut down the simulator since it's impossible to erase it otherwise

An example for a more clean test run:

lifecycle:
onPrepare:
- ERASE
onDispose:
- SHUTDOWN
tip

Booting simulators is an expensive operation: terminating and erasing simulators is advisable only if you can't accept side effects from the previous test runs or other usage of simulators

Shutdown unused simulators

Marathon will automatically detect if some running simulators are not required by the test run and will shut down them. If you want to override this behaviour:

lifecycle:
shutdownUnused: false

Permissions

Marathon can grant permissions to application before test run (default) or before each test batch execution:

permissions:
grant:
- contacts
- photos-add
PermissionDescription
allApply the action to all services
calendarAllow access to calendar
contacts-limitedAllow access to basic contact info
contactsAllow access to full contact details
locationAllow access to location services when app is in use
location-alwaysAllow access to location services at all times
photos-addAllow adding photos to the photo library
photosAllow full access to the photo library
media-libraryAllow access to the media library
microphoneAllow access to audio input
motionAllow access to motion and fitness data
remindersAllow access to reminders
siriAllow use of the app with Siri
user-trackingAllow user tracking
permissions:
lifecycle: BEFORE_TEST_RUN
grant:
- contacts
- photos-add

Timeouts

All the timeouts for test run can be overridden, here is an example configuration with default values:

timeoutConfiguration:
# ISO_8601 duration
shell: PT30S
shellIdle: PT30S
reachability: PT5S
screenshot: PT10S
video: PT300S
erase: PT30S
shutdown: PT30S
delete: PT30S
create: PT30S
boot: PT30S
install: PT30S
uninstall: PT30S
testDestination: PT30S
importMedia: PT30S
NameDescription
shellTimeout for generic shell commands, unless a more specific action is specified
shellIdleIdle timeout for generic shell commands, any input from stdout/stderr will refresh the time window
reachabilityTimeout for considering remote worker unreachable
screenshotTimeout for taking a screenshot
videoTimeout for recording a video. Should be longer than the duration of your longest test
createTimeout for creating a simulator
bootTimeout for booting a simulator
shutdownTimeout for shutting down a simulator
eraseTimeout for erasing a simulator
deleteTimeout for deleting a simulator
installTimeout for installing applications (does not apply for the app bundle or test bundle)
uninstallTimeout for uninstalling applications
testDestinationTimeout for waiting for simulator specified to xcodebuild
importMediaTimeout for importing a media file to simulator

Threading

Marathon allows you to tweak the number of threads that are used for executing coroutines:

tip

This can be important if you're connecting a lot of devices to the test execution, say 100 or a 1000. Default 8 threads in the devices provider will take a long time to process all of those devices.

threadingConfiguration:
deviceProviderThreads: 8
deviceThreads: 2

deviceThreads is the number of threads allocated for processing each device's coroutines. This includes screenshots, parsing results, etc. It is an advanced setting that should not be changes unless you know what you're doing. A minimal value is 2 for the run to be stable.

Hide xcodebuild output

By default, marathon will print the xcodebuild output during testing. You can disable it as following:

hideRunnerOutput: true

Compact output

By default, marathon will print the timestamp of each entry on each line.

foo@bar $ marathon
D 23:08:45.855 [main] <AppleDeviceProvider> Initializing AppleDeviceProvider
D 23:08:45.879 [AppleDeviceProvider-1] <AppleSimulatorProvider> Establishing communication with ...
D 23:08:46.226 [AppleDeviceProvider-2] <c.m.m.i.c.r.s.s.c.PerformanceDefaultConfig> Available cipher factories: ...

If you want to make the output more compact by removing the timestamps:

compactOutput: true

Remote rsync configuration

tip

This section is relevant only if you're using remote workers

Override rsync binary on the remote worker

rsync:
remotePath: "/usr/bin/rsync-custom"

Clear state between test batch executions

By default, marathon does not clear state between test batch executions. To mitigate potential test side effects, one could add an option to clear the container data between test runs. Keep in mind that test side effects might still be present. If you want to isolate tests even further, then you should consider reducing the batch size.

dataContainerClear: true

Test parser

tip

If you need to parallelize the execution of tests generated at runtime (i.e. flutter) - xctest parser is your choice.

Test parsing (collecting a list of tests expected to execute) can be done using either binary inspection using nm, or injecting marathon's proprietary blob and allows marathon to collect a list of tests expected to run without actually running them.

note

We don't provide source code for the libxctest-parser module. By using libxctest-parser you're automatically accepting it' s EULA

YAML typeProsConst
"nm"Doesn't require installation of apps onto the deviceDoesn't support runtime-generated tests, e.g. flutter
"xctest"Supports precise test parsing and any runtime-generated tests hence allows marathon to parallelize their executionRequires a booted device for parsing and a fake test run including installation of test app under test

Default test parser is nm.

testParserConfiguration:
type: "nm"
testClassRegexes:
- "^((?!Abstract).)*Test[s]*$"

Capture device log

By default, marathon does not pull device system logs. To investigate potential issues users might need to collect these. System logs will be available at $output-folder/device-files per each batch (not per test!) due to latency issues

deviceLog: true