Skip to content

ci: Use Cirrus Runners for iOS integration tests #4951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Flash0ver wants to merge 12 commits into
base: main
Choose a base branch
from
+61 −24
Draft

ci: Use Cirrus Runners for iOS integration tests #4951

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
1df212b
ci: Use Cirrus Runners for iOS integration tests
Flash0ver Feb 23, 2026
192b825
test: add deliberate failure
Flash0ver Feb 24, 2026
25b97b3
Apply suggestions from code review
Flash0ver Feb 24, 2026
04a5b89
ci: Reorder matrix and fix Xcode image tag for net10.0
Flash0ver Feb 24, 2026
7f51356
ci: Add FIFO concurrency groups for Cirrus iOS runners
Flash0ver Feb 24, 2026
175a733
ci: Install PowerShell via dotnet global tool on Cirrus runners
Flash0ver Feb 24, 2026
ebc5740
ci: Add dotnet tools dir to PATH after installing PowerShell
Flash0ver Feb 24, 2026
aff5bf1
build: Update XHarness CLI to 10.0.0-prerelease.25622.2
Flash0ver Feb 25, 2026
841dbc3
ci: Select correct Xcode version before building iOS test app
Flash0ver Feb 25, 2026
7d8d1ed
cI: add debug info
Flash0ver Feb 26, 2026
3776ef8
ci: fix invalid workflow file
Flash0ver Feb 26, 2026
6ec1266
ci: fix invalid workflow file
Flash0ver Feb 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 23 additions & 11 deletions .github/workflows/device-tests-ios.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,16 @@ on:

jobs:
ios-tests:
runs-on: macos-15
name: iOS Tests (${{ matrix.tfm }})
strategy:
fail-fast: false
matrix:
include:
- tfm: net10.0
runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.2.0", "runner_group_id:12"]
Copy link

@itaybre itaybre Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like these never triggered, I am not sure about the reason.

The tag might be wrong, it should be just 26.2 (see https://github.com/cirruslabs/macos-image-templates/pkgs/container/macos-tahoe-xcode/676398842?tag=26.2)

But I can suggest adding this to ensure they execute in FIFO: https://cirrus-runners.app/setup/#concurrency-groups-fifo-scheduling

Copy link
Member Author

@Flash0ver Flash0ver Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • changed to runner_group_id 10: 25b97b3
  • fixed image template version: 04a5b89
  • using Concurrency groups: 7f51356

- tfm: net9.0
runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.0.1", "runner_group_id:12"]
runs-on: ${{ matrix.runs-on }}
env:
DOTNET_CLI_TELEMETRY_OPTOUT: 1
DOTNET_NOLOGO: 1
Expand All @@ -72,39 +81,42 @@ jobs:
uses: ./.github/actions/environment

- name: Build iOS Test App
run: pwsh ./scripts/device-test.ps1 ios -Build
run: pwsh ./scripts/device-test.ps1 ios -Build -Tfm ${{ matrix.tfm }}

- name: Run Tests
id: first-test-run
continue-on-error: true
timeout-minutes: 40
run: pwsh scripts/device-test.ps1 ios -Run
run: pwsh scripts/device-test.ps1 ios -Run -Tfm ${{ matrix.tfm }}

- name: Retry Tests (if previous failed to run)
if: steps.first-test-run.outcome == 'failure'
timeout-minutes: 40
run: pwsh scripts/device-test.ps1 ios -Run
run: pwsh scripts/device-test.ps1 ios -Run -Tfm ${{ matrix.tfm }}

- name: Checkout github-workflows
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: getsentry/github-workflows
ref: 747c4c2906d373f5cd809abe94a7fd732a2421a7 # 3.2.1
path: modules/github-workflows

- name: Run Integration Tests
id: first-integration-test-run
continue-on-error: true
timeout-minutes: 40
uses: getsentry/github-workflows/sentry-cli/integration-test@747c4c2906d373f5cd809abe94a7fd732a2421a7 # 3.2.1
with:
path: integration-test/ios.Tests.ps1
run: pwsh integration-test/ios.Tests.ps1 -dotnet_version ${{ matrix.tfm }}

- name: Retry Integration Tests (if previous failed to run)
if: steps.first-integration-test-run.outcome == 'failure'
timeout-minutes: 40
uses: getsentry/github-workflows/sentry-cli/integration-test@747c4c2906d373f5cd809abe94a7fd732a2421a7 # 3.2.1
with:
path: integration-test/ios.Tests.ps1
run: pwsh integration-test/ios.Tests.ps1 -dotnet_version ${{ matrix.tfm }}

- name: Upload results
if: success() || failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: device-test-ios-results
name: device-test-ios-results-${{ matrix.tfm }}
path: |
test_output
integration-test/mobile-app/test_output
16 changes: 11 additions & 5 deletions integration-test/ios.Tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
param(
[string] $dotnet_version = "net10.0"
)

# This file contains test cases for https://pester.dev/
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
Expand All @@ -11,12 +15,14 @@ BeforeDiscovery {
$script:simulator = Get-IosSimulatorUdid -PreferredStates @('Booted')
}

$ios_tpv = switch ($dotnet_version) {
'net9.0' { '26.0' }
default { '26.2' }
}

Describe 'iOS app (<tfm>, <configuration>)' -ForEach @(
# Note: we can't run against net10 and net9 becaus .NET 10 requires Xcode 26.2 and .NET 9 requires Xcode 26.0.
# The macOS GitHub Actions runners only have Xcode 26.1+ installed and no support for Xcode 26.2 is planned for
# net9.0-ios: https://github.com/dotnet/macios/issues/24199#issuecomment-3819021247
@{ tfm = "net10.0-ios26.2"; configuration = "Release" }
@{ tfm = "net10.0-ios26.2"; configuration = "Debug" }
@{ tfm = "$dotnet_version-ios$ios_tpv"; configuration = "Release" }
@{ tfm = "$dotnet_version-ios$ios_tpv"; configuration = "Debug" }
) -Skip:(-not $script:simulator) {
BeforeAll {
. $PSScriptRoot/../scripts/device-test-utils.ps1
Expand Down
2 changes: 1 addition & 1 deletion integration-test/net9-maui/Sentry.Maui.Device.IntegrationTestApp.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<!-- Pin target iOS version so that our tests don't break when new versions of Xcode are released.
'netX.0-ios' resolves the latest version of the iOS SDK otherwise.
-->
<TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('OSX'))">$(TargetFrameworks);net10.0-ios26.2</TargetFrameworks>
<TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('OSX'))">$(TargetFrameworks);net9.0-ios26.0;net10.0-ios26.2</TargetFrameworks>

<OutputType>Exe</OutputType>
<RootNamespace>Sentry.Maui.Device.IntegrationTestApp</RootNamespace>
Expand Down
9 changes: 3 additions & 6 deletions test/Sentry.Maui.Device.TestApp/Sentry.Maui.Device.TestApp.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,7 @@

<PropertyGroup>
<TargetFrameworks>$(TargetFrameworks);net9.0-android;net10.0-android</TargetFrameworks>
<!-- Note: we can't run against net9-ios because .NET 10 requires Xcode 26.2 and .NET 9 requires Xcode 26.0.
The macOS GitHub Actions runners only have Xcode 26.1+ installed and no support for Xcode 26.2 is planned for
net9.0-ios: https://github.com/dotnet/macios/issues/24199#issuecomment-3819021247
-->
<TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('OSX'))">$(TargetFrameworks);net10.0-ios</TargetFrameworks>
<TargetFrameworks Condition="$([MSBuild]::IsOSPlatform('OSX'))">$(TargetFrameworks);net9.0-ios;net10.0-ios</TargetFrameworks>
<DefineConstants Condition="'$(EnableMauiDeviceTestVisualRunner)' == 'true'">$(DefineConstants);VISUAL_RUNNER</DefineConstants>
</PropertyGroup>

Expand Down Expand Up @@ -38,7 +34,8 @@
<SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'android'">21.0</SupportedOSPlatformVersion>
<!-- Pin target iOS version so that our tests don't break when new versions of Xcode are released.
'netX.0-ios' resolves the latest version of the iOS SDK otherwise. -->
<TargetPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios'">26.2</TargetPlatformVersion>
<TargetPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios' And '$(TargetFramework)' == 'net10.0-ios'">26.2</TargetPlatformVersion>
<TargetPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios' And '$(TargetFramework)' == 'net9.0-ios'">26.0</TargetPlatformVersion>
<EmbedAssembliesIntoApk>true</EmbedAssembliesIntoApk>

<!-- Avoid errors of Sentry.Testing not being a self-contained executable, while including it in a self-contained executable (this).
Expand Down
6 changes: 6 additions & 0 deletions test/Sentry.Tests/SentrySdkTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,12 @@
_logger = Substitute.ForPartsOf<TestOutputDiagnosticLogger>(testOutputHelper);
}

[Fact]
public void This_Test_Should_Fail()
Copy link
Member Author

@Flash0ver Flash0ver Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: remove

I added this to see a test failure in CI ... to double check against a false negative

{
Assert.True(SentrySdk.IsEnabled);

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-musl-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-musl-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-musl-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-musl-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-musl-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-musl-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (linux-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-arm64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

Check failure on line 32 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False
}

Check failure on line 33 in test/Sentry.Tests/SentrySdkTests.cs

View workflow job for this annotation

GitHub Actions / .NET (win-x64)

Sentry.Tests.SentrySdkTests.This_Test_Should_Fail 

Assert.True() Failure
Expected: True
Actual:   False

[Fact]
public void IsEnabled_StartsOffFalse()
{
Expand Down
Loading