Flaky builds

[rschili]

Our builds were never completely stable, but lately it seems about 50% of them are failing. Given that the pipeline runs for 1.5 hours, this is a real concern. The pipeline is barely a trustworthy gating signal anymore.

Collecting flaky failures that I hit during recent PRs:

#1204

• imodel-native on linux: Failed the setup node 22 step
• integration-tests (typescript) one test failed because of unresponsive server

#1205 (comment)

• Linux imodel-native failed during download secrets step

Downloading secret value for: 4787-BD44.
##[error]
4787-BD44: "Could not fetch access token for Azure. Status code: network_error, status message: Network request failed"
Restarted:

• Linux "Use node 22" step failed without message after 3 seconds

[rschili]

I've been thinking hard about this. Given that the issues are on so many random places it seems very hard to work on this. Still I have a few ideas:

• The length of the pipeline is a problem. We could explore having another "light" version (<20 minutes) of the pipeline which runs on every push, while the full pipeline (all tests, all systems, 80+ minutes) only runs before pushing to main
• We can try and detach flaky tests from failing the pipeline. Some test frameworks can flag certain tests as "known flaky" which will make them not fail the pipeline.
• There may be a way to repeat a failing test N times before we consider it a hard failure

[rschili]

Another one, I just published a PR updating a markdown file in imodel native and the build on android failed after 37 minutes:

C:\agent_work\3\b\Tools\gradle\bin\gradle --project-dir C:\agent_work\3\b\AndroidX64\DgnPlatTst\prj --no-daemon --quiet build assembleAndroidTest
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
[CXX1104] NDK from ndk.dir at C:\agent_work\3\b\Tools\androidndk_r28c had version [28.2.13676358] which disagrees with android.ndkVersion [27.0.12077973]
warning: [options] source value 8 is obsolete and will be removed in a future release
warning: [options] target value 8 is obsolete and will be removed in a future release
warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
Note: C:\agent_work\3\b\AndroidX64\DgnPlatTst\prj\app\src\main\java\com\bentley\test\TestActivity.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
3 warnings
warning: [options] source value 8 is obsolete and will be removed in a future release
warning: [options] target value 8 is obsolete and will be removed in a future release
warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
Note: C:\agent_work\3\b\AndroidX64\DgnPlatTst\prj\app\src\main\java\com\bentley\test\TestActivity.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
3 warnings

FAILURE: Build failed with an exception.

• What went wrong:
  Execution failed for task ':app:packageDebug'.

A failure occurred while executing com.android.build.gradle.tasks.PackageAndroidArtifact$IncrementalSplitterRunnable

• Try:

Run with --stacktrace option to get the stack trace.
Run with --info or --debug option to get more log output.
Run with --scan to get full insights.
Get more help at https://help.gradle.org./

BUILD FAILED in 1m 29s
BBMake: call trace
line: 42, C:\agent_work\3\b\AndroidX64\static\BuildContexts\iModelPlatform\SubParts\BeGTest\Android\BuildApk.mke
line: 232, C:\agent_work\3\b\AndroidX64\static\BuildContexts\iModelPlatform\SubParts\BeGTest\Android\MakeJUnitTestProject.mke
Fri Nov 14 10:29:49 2025, elapsed time: 2:37