Skip to main content
Version: 0.10.1

Configuration

To indicate that you're using a vendor config for macOS you have to specify the type in configuration as following:

vendorConfiguration:
  type: "macOS"
  additional_option1: ...
  additional_option2: ...
caution

All the options below should be placed under the vendorConfiguration with appropriate yaml indentation

Required options​

Test bundle​

Marathon can run both XCUITests and XCTests. Test bundle requires you to specify application under test as well as test application. After preprocessing both of these inputs are distilled into an application bundle (e.g. my.app) and xctest bundle (e.g. my-tests.xctest) You can specify .ipa [application archives][4] as well as .zip with the same content as application archive. They will be searched for the application and xctest bundles. If there are multiple entries matching description - marathon will fail.

Raw bundles (.app + .xctest)​

application: "build/my.app"
testApplication: "build/my.xctest"

Archive bundles (.ipa/.zip)​

tip

It is much easier to supply the .app application bundle and .xctest bundle directly instead of wasting time on packaging a signed application archive and depending on marathon's runtime type discovery of your bundles

If you want to specify your bundles as .ipa/.zip:

application: "build/my.ipa"
testApplication: "build/my.zip"

Derived data dir​

derivedDataDir: "derivedDataDir/"

Test type​

Marathon will detect if the specified .xctest is a XCUITest bundle or XCTest bundle. If you want to save some execution time you can explicitly specify this:

testType: "xcuitest"

Devices​

By default, marathon will look for a file Marathondevices in the same folder as Marathonfile for the configuration of workers. You can override this location with the following property:

devices: my/devices.yaml

For the documentation of the format of this file refer to worker's documentation.

SSH​

tip

This section is only required if you're using remote workers and want to provide the same ssh configuration for all of those workers. You can always specify and/or override this explicitly for each worker

For each ssh connection you want to specify authentication, identifying known hosts and keep-alive:

Public Key Authentication​

To authenticate using private key and a username:

ssh:
  authentication:
    type: "publicKey"
    username: "username"
    key: "/home/user/.ssh/id_rsa"

Password Authentication​

To authenticate using username and password:

caution

sshpass is required to allow rsync to pick up username+password credentials.

ssh:
  authentication:
    type: "password"
    username: "username"
    password: "storing-password-here-is-a-bad-idea"
tip

Storing ssh password in a configuration file directly is a bad idea. Refer to dynamic configuration and utilize the envvar interpolation to provide the password for your test runs during runtime.

Known hosts​

When ssh establishes connection to a remote host it tries to verify the identity of the remote host to mitigate potential men-in-the-middle attack. You can specify the known_hosts file in the OpenSSH format as following:

ssh:
  knownHostsPath: "/home/user/.ssh/known_hosts"
caution

If you omit this configuration then marathon will trust any remote host. This is a bad idea for production.

Secure Shell debug​

If you are experiencing issues with ssh connections and want to have more information use the following debug flag. Caution: a lot of data is written to stdout when using this flag.

ssh:
  debug: true

Optional​

Collecting xcresult​

By default, marathon will pull the xcresult bundle into the output folder under device files and cleanup the remote worker to not bloat the worker storage. To change this override the following:

info

As of the time of writing marathon doesn't support merging the xcresult and treats them as just regular file artifacts.

  xcresult:
    pull: true
    remoteClean: true

Attachment lifetime​

Marathon generates the xctestrun file for each batch and can specify custom lifecycle attachments. By default, system attachments will be deleted on success and user attachments will always be kept in the xcresult, but you can override this: 

  xcresult:
    attachments:
      systemAttachmentLifetime: DELETE_ON_SUCCESS
      userAttachmentLifetime: KEEP_ALWAYS

Possible values for the lifetime are KEEP_ALWAYS, DELETE_ON_SUCCESS and KEEP_NEVER.

xctestrun Environment and TestingEnvironment variables​

You can specify additional Environment and TestingEnvironment variables for your test run:

xctestrunEnv:
  app: # EnvironmentVariables
    MY_ENV_VAR_1: A
    MY_ENV_VAR_2: B
  test: # TestingEnvironmentVariables
    MY_TEST_ENV_VAR_1: AA
    MY_TEST_ENV_VAR_2: BB

These will be placed in the generated xctestrun property list file under the EnvironmentVariables and TestingEnvironmentVariables keys.

info

Marathon generates required values for DYLD_FRAMEWORK_PATH, DYLD_LIBRARY_PATH and DYLD_INSERT_LIBRARIES for test environment. If you specify custom ones then your values will be placed as a lower priority path elements at the end of the specified envvar.

xcodebuild test-without-building arguments​

You can specify additional arguments to pass to the underlying xcodebuild test-without-building invocation.

xcodebuildTestArgs:
  "-test-timeouts-enabled": "YES"
  "-maximum-test-execution-time-allowance": "60"

It is impossible to override the following reserved arguments:

  • -xctestrun
  • -enableCodeCoverage
  • -resultBundlePath
  • -destination-timeout
  • -destination

Permissions​

Marathon can grant permissions to application during device setup:

permissions:
  grant:
    - accessibility
    - developer-tool
danger

Granting permissions to macOS requires disabling System Integrity Protection

Timeouts​

All the timeouts for test run can be overridden, here is an example configuration with default values:

timeoutConfiguration:
  # ISO_8601 duration
  shell: PT30S
  shellIdle: PT30S
  reachability: PT5S
  screenshot: PT10S
  video: PT300S
  erase: PT30S
  shutdown: PT30S
  delete: PT30S
  create: PT30S
  boot: PT30S
  install: PT30S
  uninstall: PT30S
  testDestination: PT30S
NameDescription
shellTimeout for generic shell commands, unless a more specific action is specified
shellIdleIdle timeout for generic shell commands, any input from stdout/stderr will refresh the time window
reachabilityTimeout for considering remote worker unreachable
screenshotTimeout for taking a screenshot
videoTimeout for recording a video. Should be longer than the duration of your longest test
createTimeout for creating a simulator
bootTimeout for booting a simulator
shutdownTimeout for shutting down a simulator
eraseTimeout for erasing a simulator
deleteTimeout for deleting a simulator
installTimeout for installing applications (does not apply for the app bundle or test bundle)
uninstallTimeout for uninstalling applications
testDestinationTimeout for waiting for simulator specified to xcodebuild

Threading​

Marathon allows you to tweak the number of threads that are used for executing coroutines:

tip

This can be important if you're connecting a lot of devices to the test execution, say 100 or a 1000. Default 8 threads in the devices provider will take a long time to process all of those devices.

threadingConfiguration:
  deviceProviderThreads: 8
  deviceThreads: 2

deviceThreads is the number of threads allocated for processing each device's coroutines. This includes screenshots, parsing results, etc. It is an advanced setting that should not be changes unless you know what you're doing. A minimal value is 2 for the run to be stable.

Hide xcodebuild output​

By default, marathon will print the xcodebuild output during testing. You can disable it as following:

hideRunnerOutput: true

Compact output​

By default, marathon will print the timestamp of each entry on each line.

foo@bar $ marathon
D 23:08:45.855 [main] <AppleDeviceProvider> Initializing AppleDeviceProvider
D 23:08:45.879 [AppleDeviceProvider-1] <AppleSimulatorProvider> Establishing communication with ...
D 23:08:46.226 [AppleDeviceProvider-2] <c.m.m.i.c.r.s.s.c.PerformanceDefaultConfig> Available cipher factories: ...

If you want to make the output more compact by removing the timestamps:

compactOutput: true

Remote rsync configuration​

tip

This section is relevant only if you're using remote workers

Override rsync binary on the remote worker

rsync:
  remotePath: "/usr/bin/rsync-custom"

Clear state between test batch executions​

By default, marathon does not clear state between test batch executions. To mitigate potential test side effects, one could add an option to clear the container data between test runs. Keep in mind that test side effects might still be present. If you want to isolate tests even further, then you should consider reducing the batch size.

dataContainerClear: true

Test parser​

tip

If you need to parallelize the execution of tests generated at runtime (i.e. flutter) - xctest parser is your choice.

Test parsing (collecting a list of tests expected to execute) can be done using either binary inspection using nm, or injecting marathon's proprietary blob and allows marathon to collect a list of tests expected to run without actually running them.

note

We don't provide source code for the libxctest-parser module. By using libxctest-parser you're automatically accepting it's EULA

YAML typeProsConst
"nm"Doesn't require installation of apps onto the deviceDoesn't support runtime-generated tests, e.g. flutter
"xctest"Supports precise test parsing and any runtime-generated tests hence allows marathon to parallelize their executionRequires a booted device for parsing and a fake test run including installation of test app under test

Default test parser is nm.

testParserConfiguration:
  type: "nm"
  testClassRegexes:
  - "^((?!Abstract).)*Test[s]*$"

Contents