Additional UI tests / also disabling animations to speed up UI tests by wmathurin · Pull Request #3994 · forcedotcom/SalesforceMobileSDK-iOS

wmathurin · 2026-02-24T20:44:30Z

Overview

A number of scenarios described in the 13.2 high level test plans were not covered.
Converted test scheme to test plans

Test scenarios described in this Salesforce internal doc

wmathurin · 2026-02-24T20:44:46Z

.github/workflows/ui-test-nightly.yaml

-          - MigrationTests
+          - LoginWithRestartTests
          - MultiUserLoginTests
+          - RefreshTokenMigrationTests


More descriptive name

wmathurin · 2026-02-24T20:45:15Z

native/SampleApps/AuthFlowTester/AuthFlowTester/Classes/AppDelegate.swift

+
+        // Speed up UI tests by disabling animations
+        if ProcessInfo.processInfo.arguments.contains("-UITesting") {
+            UIView.setAnimationsEnabled(false)


It seems to be a bit faster (~20% faster).

wmathurin · 2026-02-24T20:46:07Z

native/SampleApps/AuthFlowTester/AuthFlowTesterUITests/Tests/AdvancedAuthBeaconLoginTests.swift

 /// This class runs the same tests as BeaconLoginTests but uses the advanced_auth login host.
 ///
-/// NB: Tests use the first user from ui_test_config.json (advanced_auth host)
+/// NB: Tests use the second user from ui_test_config.json (advanced_auth host)


Since we have more than 5 test suites and they run in parallel, we don't want the same user logging in more than 5 times at once.

wmathurin · 2026-02-24T20:46:47Z

native/SampleApps/AuthFlowTester/AuthFlowTesterUITests/Tests/LegacyLoginTests.swift

+
    // MARK: - CA Non-hybrid Web Server Flow Tests

+    /// Login with CA opaque using default scopes and (non-hybrid) web server flow.


Not new - was in a separate suite previously.

wmathurin · 2026-02-24T20:47:38Z

native/SampleApps/AuthFlowTester/AuthFlowTesterUITests/Tests/LoginWithRestartTests.swift

@@ -1,8 +1,8 @@
 /*
- DynamicConfigLoginTests.swift
+ LoginWithRestartTests.swift


Focus of the class is tests that involve restart - they mostly deal with dynamic config since it is defined in memory only.

wmathurin · 2026-02-24T20:48:01Z

native/SampleApps/AuthFlowTester/AuthFlowTesterUITests/Tests/LoginWithRestartTests.swift

+
+    // MARK: - Multi-User Restart
+
+    /// Login multiple users with dynamic config, restart app, and verify all users persist correctly.


That's a new test - but was in our high level test plan.

wmathurin · 2026-02-24T20:48:27Z

native/SampleApps/AuthFlowTester/AuthFlowTesterUITests/Tests/MultiUserLoginTests.swift

        logout()
    }
+
+    // MARK: - Beacon and Non-Beacon Multi-User


wmathurin · 2026-02-24T20:49:13Z

...eApps/AuthFlowTester/AuthFlowTesterUITests/Tests/RefreshTokenMigrationWithRestartTests.swift

+///
+/// NB: Tests use the third and fourth user from ui_test_config.json
+///
+class RefreshTokenMigrationWithRestartTests: BaseAuthFlowTester {


New tests - maybe overkill??

github-actions · 2026-02-24T20:59:36Z

	Tests	Passed ☑️	Skipped	Failed ❌️
SalesforceSDKCore iOS ^18 Test Results	591 ran	590 ✅		1 ❌

Test	Result
SalesforceSDKCore iOS ^18 Test Results
DomainDiscoveryCoordinatorTests.testMissingMyDomain()	❌ failure

View SalesforceSDKCore iOS ^18 Test Results

github-actions · 2026-02-24T21:03:57Z

	Tests	Passed ☑️	Skipped	Failed ❌️
SalesforceSDKCore iOS ^26 Test Results	591 ran	590 ✅		1 ❌

Test	Result
SalesforceSDKCore iOS ^26 Test Results
DomainDiscoveryCoordinatorTests.testNonCallbackURL()	❌ failure

View SalesforceSDKCore iOS ^26 Test Results

…e new tests

github-actions · 2026-02-24T22:39:15Z

	1 Warning
⚠️	Big PR, try to keep changes smaller if you can.

Generated by 🚫 Danger

codecov · 2026-02-24T23:07:24Z

Codecov Report

❌ Patch coverage is 58.33333% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.82%. Comparing base (06e487f) to head (5abe185).
⚠️ Report is 10 commits behind head on dev.

Files with missing lines	Patch %	Lines
...ore/Classes/Login/DevConfig/BootConfigEditor.swift	58.33%	10 Missing ⚠️
...lasses/Login/DevConfig/DiscoveryResultEditor.swift	54.54%	10 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              dev    #3994      +/-   ##
==========================================
- Coverage   69.27%   66.82%   -2.45%     
==========================================
  Files         253      254       +1     
  Lines       22260    22408     +148     
==========================================
- Hits        15421    14975     -446     
- Misses       6839     7433     +594

Components	Coverage Δ
Analytics	`70.78% <ø> (ø)`
Common	`69.76% <ø> (ø)`
Core	`59.54% <58.33%> (-3.76%)`	⬇️
SmartStore	`74.70% <ø> (+0.11%)`	⬆️
MobileSync	`87.41% <ø> (-0.12%)`	⬇️

Files with missing lines	Coverage Δ
...s/Login/DevConfig/LoginOptionsViewController.swift	`98.70% <100.00%> (ø)`
...ore/Classes/Login/DevConfig/BootConfigEditor.swift	`59.40% <58.33%> (-37.47%)`	⬇️
...lasses/Login/DevConfig/DiscoveryResultEditor.swift	`64.51% <54.54%> (-6.28%)`	⬇️

... and 30 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…simulated discovery results) -> saving on a few taps

…ng with simulated discovery results) -> saving on a few taps" This reverts commit f74afc8.

…ert - that saves a bit of time Unfortunately, we have to deal with the "Allow Paste" permission pop-up Also converted test scheme to use test plan

wmathurin · 2026-02-25T19:18:53Z

native/SampleApps/AuthFlowTester/AuthFlowTesterUITests/PageObjects/LoginOptionsPageObject.swift

+        let springboard = XCUIApplication(bundleIdentifier: "com.apple.springboard")
+        let semaphore = DispatchSemaphore(value: 0)
+
+        DispatchQueue.global().async {


Ugly code to handle the paste permission alert.
One of the reason I migrated the test scheme to use a test plan is because I thought the paste permission could be granted there.
Also I tried to make it work on export too but couldn't get the alert to dismiss properly.

…le tests in parallel

Saving a few taps here and there.

Locally running the command gets 59/65 to pass (last 6 pass if retried in xcode) and take 1700s

wmathurin · 2026-02-26T04:45:13Z

Ran it locally (from the command line) and got all the tests to pass. It took 39 minutes.

wmathurin · 2026-02-26T21:35:38Z

native/SampleApps/AuthFlowTester/AuthFlowTester.xctestplan

+        "locationScenario" : {
+          "identifier" : "com.apple.dt.IDEFoundation.CurrentLocationScenarioIdentifier"
+        },
+        "maximumTestExecutionTimeAllowance" : 360,


Test will be killed if taking more than 6'

wmathurin · 2026-02-26T21:36:39Z

native/SampleApps/AuthFlowTester/AuthFlowTester.xctestplan

+        },
+        "testExecutionOrdering" : "random",
+        "testRepetitionMode" : "retryOnFailure",
+        "maximumTestRepetitions" : 3,


The retry has allowed me to get all tests to pass locally from a single invocation of xcodebuild test.

wmathurin · 2026-02-26T21:37:25Z

native/SampleApps/AuthFlowTester/AuthFlowTester.xctestplan

+  },
+  "testTargets" : [
+    {
+      "parallelizable" : true,


Running test suites on different simulator (the number depends on the capability of the machine I think). Locally I get all the tests to complete in under 40 minutes.

wmathurin · 2026-02-26T21:39:11Z

native/SampleApps/AuthFlowTester/AuthFlowTester.xctestplan

+        "testRepetitionMode" : "retryOnFailure",
+        "maximumTestRepetitions" : 3,
+        "testTimeoutsEnabled" : true,
+        "uiTestingScreenshotsLifetime" : "deleteOnSuccess"


The screenshots are really useful to see what happened. Not sure if there will be an easy way to get them on CI.

wmathurin added 2 commits February 24, 2026 11:50

Re-organizing UI tests - adding more tests also

3bcf81f

Disabling animations to speed up tests and reduce flappiness

7677881

wmathurin commented Feb 24, 2026

View reviewed changes

wmathurin changed the title ~~Additional UI. tests / also disabling animations to speed up UI tests~~ Additional UI tests / also disabling animations to speed up UI tests Feb 24, 2026

Misc changes - renaming helper method for clarity - fixing some of th…

542900b

…e new tests

wmathurin added 3 commits February 24, 2026 19:06

No need to click save after doing Import config (and same thing with …

f74afc8

…simulated discovery results) -> saving on a few taps

Revert "No need to click save after doing Import config (and same thi…

6d51ea3

…ng with simulated discovery results) -> saving on a few taps" This reverts commit f74afc8.

Import buttons read directly from pasteboard instead of showing an al…

3368123

…ert - that saves a bit of time Unfortunately, we have to deal with the "Allow Paste" permission pop-up Also converted test scheme to use test plan

wmathurin commented Feb 25, 2026

View reviewed changes

wmathurin added 3 commits February 25, 2026 14:28

Moving away from using paste board - is problematic if running multip…

493017b

…le tests in parallel

No longer turning off animations - didn't quite work

8941da3

Saving a few taps here and there.

Adding retry on failure in test plans.

5abe185

Locally running the command gets 59/65 to pass (last 6 pass if retried in xcode) and take 1700s

wmathurin marked this pull request as ready for review February 26, 2026 04:45

wmathurin requested a review from bbirman February 26, 2026 21:34

wmathurin commented Feb 26, 2026

View reviewed changes

brandonpage approved these changes Feb 27, 2026

View reviewed changes

wmathurin merged commit 1815820 into forcedotcom:dev Feb 27, 2026
20 of 29 checks passed


		// MARK: - CA Non-hybrid Web Server Flow Tests

		/// Login with CA opaque using default scopes and (non-hybrid) web server flow.


		// MARK: - Multi-User Restart

		/// Login multiple users with dynamic config, restart app, and verify all users persist correctly.

Conversation

wmathurin commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 24, 2026

Uh oh!

codecov bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wmathurin commented Feb 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wmathurin commented Feb 24, 2026 •

edited

Loading

github-actions bot commented Feb 24, 2026 •

edited

Loading

github-actions bot commented Feb 24, 2026 •

edited

Loading

codecov bot commented Feb 24, 2026 •

edited

Loading