Advanced Testing and Continuous Integration

(1)

Developer Tools

#WWDC16

Session 409

Advanced Testing and

Continuous Integration

Zoltan Foley-Fisher Xcode Engineer

(2)

(3)

(4)

(5)

(6)

(7)

Agenda

(8)

Agenda

Testing Concepts  Xcode

(9)

Agenda

• Crash Log Gathering Xcode Server

• Advanced Triggers

• Issue Tracking and Blame

(10)

Agenda

• Configurable Integration User xcodebuild

(11)

(12)

(13)

(14)

Testing framework

XCTest

(15)

Testing framework

XCTest

Sources TestClassA.m TestClassB.swift TestClassC.swift

(16)

Testing framework

XCTest

Compile Build Products UI.xctest Unit.xctest Sources TestClassA.m TestClassB.swift TestClassC.swift

(17)

(18)

Test harness

Xcode

(19)

Test harness

Xcode

(20)

Test harness

Xcode

Author tests Execute tests

(21)

Test harness

Xcode

Author tests Execute tests

(22)

(23)

Continuous Integration

Xcode Server

(24)

Continuous Integration

Xcode Server

(25)

Continuous Integration

Xcode Server

Schedules bot runs Generates reports

(26)

Continuous Integration

Xcode Server

Schedules bot runs Generates reports Sends notifications

(27)

(28)

Command line tool

xcodebuild

(29)

Command line tool

xcodebuild

(30)

Command line tool

xcodebuild

Used by Xcode Server Build and execute tests

(31)

Command line tool

xcodebuild

Used by Xcode Server Build and execute tests For custom CI systems

(32)

Previous Years’ Sessions

Testing with Xcode 6 WWDC 2014

Continuous Integration with Xcode 6 WWDC 2014

UI Testing in Xcode WWDC 2015

(33)

(34)

Behind the scenes

How Testing Works

(35)

Fundamental concepts

Testing Timeline

(36)

How testing starts

Test Hosting

(37)

How testing starts

Test Hosting

MyApp.app

(38)

How testing starts

Test Hosting

MyApp.app

(39)

How testing starts

Test Hosting

MyApp.app

Unit Tests UI Tests

(40)

How testing starts

Test Hosting

MyApp.app UI Test Runner MyApp.app

Unit Tests UI Tests

(41)

How testing starts

Test Hosting

Unit Tests UI Tests

(42)

How testing starts

Test Hosting

Unit Tests UI Tests

(43)

Comparison

Test Hosting

Unit Tests UI Tests

Direct access to host app data and APIs Uses Accessibility to access target app All tests run in single app launch Tests launch app for every test case

(44)

Comparison

Test Hosting

Unit Tests UI Tests

Direct access to host app data and APIs Uses Accessibility to access target app All tests run in single app launch Tests launch app for every test case

(45)

Fundamental concepts

Testing Timeline

(46)

Fundamental concepts

Testing Timeline

(47)

Test case structure

(48)

Test case structure

Test Observation

(49)

Test case structure

Test Observation

(50)

Test case structure

Test Observation

(51)

Test case structure

Test Observation

Test Suite A Test Suite B

(52)

Test case structure

Test Observation

Setup or tear down work

(53)

Test case structure

Test Observation

Setup or tear down work Custom logging

(54)

XCTestObservation Protocol

Test Observation

Test Bundle

optional public func testBundleWillStart(_ testBundle: AnyObject!)

(55)

XCTestObservation Protocol

Test Observation

Test Bundle

optional public func testSuiteWillStart(_ testSuite: AnyObject!)

(56)

XCTestObservation Protocol

Test Observation

Test Bundle

optional public func testCaseWillStart(_ testCase: AnyObject!)

(57)

XCTestObservation Protocol

Test Observation

Test Bundle

optional public func testCaseDidFinish(_ testCase: AnyObject!)

(58)

XCTestObservation Protocol

Test Observation

Test Bundle

optional public func testCase(_ testCase: AnyObject!, didFailWithDescription

description: AnyObject!, inFile filePath: AnyObject!, atLine lineNumber: AnyObject!)

(59)

XCTestObservation Protocol

Test Observation

Test Bundle

optional public func testSuiteDidFinish(_ testSuite: AnyObject!)

(60)

XCTestObservation Protocol

Test Observation

Test Bundle

optional public func testBundleDidFinish(_ testBundle: AnyObject!)

(61)

// Test Observation

// Example LoggingObserver

class LoggingObserver: NSObject, XCTestObservation {

(62)

// Test Observation

override init() { super.init()

let sharedCenter = XCTestObservationCenter.shared()

sharedCenter.addTestObserver(self)

}

(63)

// Test Observation

}

internal func testBundleWillStart(_ testBundle: Bundle) { print("Lights down, show is about to start")

}

(64)

// Test Observation

}

internal func testCase(_ testCase: AnyObject!, didFailWithDescription

print("That sounds off key…") }

(65)

// Test Observation

}

internal func testCase(_ testCase: AnyObject!, didFailWithDescription

print("That sounds off key…") }

internal func testBundleDidFinish(_ testBundle: Bundle) { print("Lights up, don’t forget your coat!")

} }

(66)

Use NSPrincipalClass

(67)

Use NSPrincipalClass

Test Observation

(68)

Use NSPrincipalClass

Test Observation

Test Bundle Info.plist

(69)

Summary

Testing Concepts

(70)

(71)

Crashes during testing

(72)

Crashes during testing

(73)

Crashes during testing

Crash Log Gathering

(74)

Crashes during testing

Crash Log Gathering

Test host or target application may crash Xcode completes test suite

(75)

Crashes during testing

Crash Log Gathering

Test host or target application may crash Xcode completes test suite

(76)

Improved test reports

(77)

Improved test reports

(78)

Improved test reports

Crash Log Gathering

For Unit and UI tests

(79)

Improved test reports

Crash Log Gathering

Local and Xcode Server crash logs

(80)

Improved test reports

Crash Log Gathering

Local and Xcode Server crash logs Logs included in test report

(81)

Improved test reports

Crash Log Gathering

Local and Xcode Server crash logs Logs included in test report

View crashes in Xcode’s Debug Navigator

(82)

Demo

(83)

Improved test reports

(84)

Crash logs gathered in test reports

Improved test reports

(85)

Crash logs gathered in test reports

View crashes in Xcode’s Debug Navigator

Improved test reports

(86)

Continuous Integration in Xcode

Xcode Server

Eric Dudiak

(87)

Overview

(88)

Overview

Xcode Server

(89)

Overview

Xcode Server

Custom environment variables Advanced trigger editing

(90)

Overview

Xcode Server

(91)

Overview

Xcode Server

Issue tracking and blame Upgrade integrations

(92)

Overview

Xcode Server

Issue tracking and blame Upgrade integrations

(93)

New in Xcode 7.3

(94)

Integration scripts

(95)

Email notifications

(96)

Email notifications

(97)

Email notifications

(98)

Email notifications

(99)

Email notifications

(100)

Email notifications

(101)

We all have issues

(102)

We all have issues

Issue Tracking and Blame

(103)

We all have issues

Issue Tracking and Blame

Nobody is perfect • Tests will fail

(104)

We all have issues

Issue Tracking and Blame

(105)

We all have issues

Issue Tracking and Blame

• Errors will come up

(106)

We all have issues

Issue Tracking and Blame

• Xcode will blame you Even if you are perfect

(107)

We all have issues

Issue Tracking and Blame

• Xcode will blame you Even if you are perfect • SDKs change

(108)

We all have issues

Issue Tracking and Blame

(109)

We all have issues

Issue Tracking and Blame

• Language improvements • Smarter tools

(110)

Identifying issue owners

(111)

Identifying issue owners

Issue Tracking and Blame

Lets you know when it’s time to act

(112)

Identifying issue owners

Issue Tracking and Blame

Lets you know when it’s time to act You broke it

(113)

Identifying issue owners

Issue Tracking and Blame

• You introduced an issue

(114)

Identifying issue owners

Issue Tracking and Blame

• Issue is on or near line you modified

(115)

Identifying issue owners

Issue Tracking and Blame

• Issue is on or near line you modified • You are the only committer

(116)

Identifying issue owners

Issue Tracking and Blame

You can help fix it

(117)

Identifying issue owners

Issue Tracking and Blame

You can help fix it

• You probably know how to fix an issue

(118)

Identifying issue owners

Issue Tracking and Blame

You can help fix it

• You seem to know about the area involved

(119)

Identifying issue owners

Issue Tracking and Blame

You can help fix it

• You seem to know about the area involved • Fuzzy matching

(120)

Bot configuration changes

(121)

Bot configuration changes

Issue Tracking and Blame

Tracks changes to your bot configuration

(122)

Bot configuration changes

Issue Tracking and Blame

Attributes issues to changes where possible

(123)

Bot configuration changes

Issue Tracking and Blame

Attributes issues to changes where possible Included in email reports

(124)

When it’s not your fault

(125)

When it’s not your fault

Upgrade Integrations

Re-integrates your project

(126)

When it’s not your fault

Upgrade Integrations

Same revision as the last integration

(127)

When it’s not your fault

Upgrade Integrations

Any new issues are due to the upgrade

(128)

When it’s not your fault

Upgrade Integrations

Any new issues are due to the upgrade Prevents blaming you for broken builds

(129)

(130)

Configurable Integration User

(131)

Configurable Integration User

Improved visibility into your integrations Allows customization

(132)

Configurable Integration User

You own and manage how your integrations run

(133)

Configurable Integration User

(134)

Configurable Integration User

Can be any user on the system Runs as a menu extra

(135)

Custom Integration User

(136)

Custom Integration User

••••••••••••••• •••••••••••••••

(137)

Custom Integration User

(138)

Custom Integration User

(139)

Custom Integration User

(140)

Custom Integration User

(141)

Custom Integration User

(142)

Custom Integration User

(143)

(144)

Custom Integration User

(145)

Demo

(146)

(147)

Configurable Integration User

(148)

Configurable Integration User

Improved visibility

(149)

Configurable Integration User

Improved visibility

Customized simulators Normal macOS user

(150)

Configurable Integration User

Improved visibility

Customized simulators Normal macOS user

(151)

Best practices

(152)

Best practices

Configurable Integration User

(153)

Best practices

Configurable Integration User

Dedicate a new user

(154)

Best practices

Configurable Integration User

Dedicate a new user

Avoid using administrator accounts

(155)

Best practices

Configurable Integration User

Dedicate a new user

Stay logged in with Fast User Switching Disable screen lock

(156)

Best practices

Configurable Integration User

Dedicate a new user

(157)

Best practices

Configurable Integration User

Dedicate a new user

Customize for your needs • Simulators

(158)

Best practices

Configurable Integration User

Dedicate a new user

(159)

Best practices

Configurable Integration User

Dedicate a new user

• Networking (VPN and proxies) • User data and settings

(160)

Best practices

Configurable Integration User

Dedicate a new user

• Networking (VPN and proxies) • User data and settings

(161)

Building block for Continuous Integration Systems

(162)

Test action

xcodebuild

xcodebuild test -workspace <path> -scheme <name>

(163)

Test action

xcodebuild

Builds sources

(164)

Test action

xcodebuild

Builds sources

Installs app on destination if needed

(165)

Test action

xcodebuild

Builds sources

Installs app on destination if needed Runs tests

(166)

Test action

xcodebuild

Builds sources

Installs app on destination if needed Runs tests

Reports results in console

(167)

Customizing tests

Test Options

NEW

-destination <specifier>

-only-testing:TestBundleA/TestSuiteA/TestCaseA -only-testing:TestBundleB/TestSuiteB

(168)

Customizing tests

Test Options

NEW

-destination <specifier>

-only-testing:TestBundleA/TestSuiteA/TestCaseA -only-testing:TestBundleB/TestSuiteB

-only-testing:TestBundleC

xcodebuild test -scheme <name>

(169)

Separating building from testing

Additional Test Actions

NEW

(170)

Separating building from testing

Additional Test Actions

NEW

(171)

Only building

Building for Testing

NEW

xcodebuild build-for-testing -workspace <path> -scheme <name>

(172)

Builds sources for testing

Only building

Building for Testing

NEW

(173)

Outputs products in Derived Data

Only building

Building for Testing

NEW

(174)

Outputs products in Derived Data Generates an xctestrun file

Only building

Building for Testing

NEW

(175)

Only testing

Test Without Building

NEW

xcodebuild test-without-building -workspace <path> -scheme <name>

(176)

Only testing

Test Without Building

Finds binary products in Derived Data

NEW

(177)

Only testing

Test Without Building

Finds binary products in Derived Data Installs if necessary

NEW

(178)

Only testing

Test Without Building

Finds binary products in Derived Data Installs if necessary

Runs tests and report results as before

NEW

(179)

Only testing

Test Without Building

NEW

xcodebuild test-without-building -xctestrun <path>

(180)

Only testing

Test Without Building

Ingests the xctestrun file

NEW

(181)

Only testing

Test Without Building

Finds binary products relative to xctestrun file

NEW

(182)

Only testing

Test Without Building

Finds binary products relative to xctestrun file Runs tests, reports results as before

NEW

(183)

Distributed testing

xcodebuild

Build Machine Test Machine #2

(184)

Distributed testing

xcodebuild

(185)

Distributed testing

xcodebuild

Test Machine #1

(186)

Distributed testing

xcodebuild

Test Machine #1

(187)

xcodebuild test-without-building

Distributed testing

xcodebuild

Test Machine #1

(188)

xcodebuild test-without-building

Distributed testing

xcodebuild

Test Machine #1

(189)

Test manifest

(190)

Test manifest

(191)

Test manifest

xctestrun

Tests to run or skip

(192)

Test manifest

xctestrun

Environment variables

(193)

Test manifest

xctestrun

Command-line arguments

(194)

Test manifest

xctestrun

Documentation in man pages

(195)

Test manifest

xctestrun

Documentation in man pages

man xcodebuild.xctestrun

(196)

(197)

(198)

Summary

• Configurable Integration User xcodebuild

(199)

(200)