Introduce synctest-based framework for deterministic testing #9099

GaetanoMar96 · 2026-01-22T13:09:29Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR introduces a new testing framework for Cluster Autoscaler based on synctest.

The new framework utilizes a "bubble" environment and virtual time to:

Ensure Determinism: Background goroutines are managed within the synctest bubble, preventing leaks.
Accelerate Test Execution: Full scale-up/scale-down lifecycles that normally take minutes of simulated time now execute in milliseconds of real time.
Structured Lifecycle Management: Provides a TestContext and Builder pattern for easier setup of complex cluster states.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

The PR is structured into three logical commits to provide a clear path from architecture to implementation:

Refactor for Dependency Injection (Builder Pattern): I introduced a Builder Pattern in the main initialization logic. This allows us to inject fakes (KubeClient, CloudProvider, PodObservers) during setup. This decoupling is essential for moving away from heavyweight integration tests toward lightweight, controlled simulations.
Lifecycle & Leak Management: To support synctest's requirement for a clean "bubble" exit, I refactored internal components to properly accept and propagate context.Context. This ensures that background goroutines observe shutdown signals immediately, preventing the "durable sleep" leaks common in components that previously relied on static timers.
Framework & Lifecycle Tests: This commit introduces the TestContext and TestBuilder. It includes a full-lifecycle regression test (Scale-up -> Stabilization -> Scale-down) that demonstrates the framework's ability to orchestrate complex state transitions deterministically using the encapsulated state of our fake K8s and Cloud Provider implementations.

Usage: See staticautoscaler_test.go for an example of the new RunTest and ctx.Step pattern.

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

linux-foundation-easycla · 2026-01-22T13:09:37Z

The committers listed above are authorized under a signed CLA.

✅ login: GaetanoMar96 / name: Gaetano Maranzino (30580e9, 5056a24, ae452ea)

k8s-ci-robot · 2026-01-22T13:09:38Z

Welcome @GaetanoMar96!

It looks like this is your first PR to kubernetes/autoscaler 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/autoscaler has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2026-01-22T13:09:39Z

Hi @GaetanoMar96. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

GaetanoMar96 · 2026-01-23T10:05:13Z

/assign @kawych @mtrqq

cluster-autoscaler/builder/autoscaler.go

kawych · 2026-01-22T17:13:38Z

cluster-autoscaler/builder/autoscaler.go

+	var nodeInfoComparator nodegroupset.NodeInfoComparator
+	if len(autoscalingOptions.BalancingLabels) > 0 {
+		nodeInfoComparator = nodegroupset.CreateLabelNodeInfoComparator(autoscalingOptions.BalancingLabels)
+	} else {


Can we pull it outside, to main?

I just noticed that the core package calls the NewCloudProvider() -_-

My proposal of target state:

move that branch outside, to main

move initialization from core to main as well:

autoscaler/cluster-autoscaler/core/autoscaler.go

Line 139 in 872d31f

opts.CloudProvider = cloudBuilder.NewCloudProvider(opts, informerFactory)

in the core repo (and in the builder) just error out if CloudProvider is nil

split the cloudprovider/, the interface should be clearly separated from provider-specific stuff (including cloudprovider/builder)

Add some form of dependency enforcement, e.g. unit test or play with dependency visibility: at minimum, the core repo cannot have dependency on any of the cloud-specific providers.

I think 4 and 5 would require some larger discussion, but in this CL I think 1-3 are achievable.

Cloud provider depends on informer factory and core autoscaler options, which are initialized only later. If we move all this code in main then in test config we should handle all of them injecting them at build initialization. Are we sure we want to do this?

Right... that might be a bit unsafe, since we don't know 100% which options does the NewCloudProvider really depend on.

Happy case: we can just move it out and see if any of the tests fail. But I'm not too confident that we have sufficient test coverage to detect all issues.

Another solution would be to leave it as is. IMO in this case we should at least move this initialization out of the core package, e.g. to the builder package. And ideally add a comment in the code that a target state is initialization in main.go, but it wasn't done because we don't have ability to test it thoroughly.

I'd like to get some feedback from other CA maintainers @mtrqq @towca

And regarding the nodeInfoComparator initialization, I think it's better to leave it in main.go for now, since the code is different depending on cloud provider (also it's in the spirit of the TODO below to eventually migrate it somewhere in the CloudProvider package).

Migrated the nodeInfoComparator. I will leave the conversation unresolved for now waiting from CA maintainers feedback

cluster-autoscaler/test/integration/framework.go

cluster-autoscaler/builder/bigunittest/staticautoscaler_test.go

cluster-autoscaler/builder/bigunittest/framework.go

cluster-autoscaler/test/integration/framework.go

cluster-autoscaler/builder/bigunittest/framework.go

cluster-autoscaler/builder/bigunittest/testconfig.go

cluster-autoscaler/cloudprovider/builder/cloud_provider_builder.go

cluster-autoscaler/builder/autoscaler.go

cluster-autoscaler/utils/fake/kubernetes.go

cluster-autoscaler/test/integration/config.go

cluster-autoscaler/test/integration/inmemory/staticautoscaler_test.go

cluster-autoscaler/cloudprovider/test/fake_cloud_provider.go

kawych

Added couple more comments, but they're increasingly minor. The biggest one is what we already agreed on offline. Thanks for the great work!

cluster-autoscaler/test/integration/inmemory/staticautoscaler_test.go

cluster-autoscaler/builder/autoscaler_test.go

cluster-autoscaler/test/integration/framework.go

kawych · 2026-02-04T16:19:18Z

cluster-autoscaler/test/integration/framework.go

+
+	c.Autoscaler, _, err = autoscalerbuilder.New(*c.Options).
+		WithKubeClient(c.Fakes.K8s.Client).
+		WithInformerFactory(c.Fakes.K8s.InformerFactory).


Would it be possible to initialize that inside the builder? Or does the additional option defined in main.go make it harder?

I am not entirely sure about this question, can you make it a bit more clear?

cluster-autoscaler/cloudprovider/test/fake_cloud_provider.go

mtrqq

Thanks for the changes, it will greatly simplify E2E testing of some autoscaler features while not going into expensive E2E tests territory

I like the direction that you've picked, having a set of reusable components without locking outselves in a specific framework or even synctest. It is good choice to keep direct interaction with synctest puts you in the right mindset when writing the test while also not forcing new tests to pick between framework and manual labour or mocking all the things around

Another data point to consider - while synctest is a great tool to verify correctness of the autoscaler, it's not optimal to use it with benchmarks, see testing.B reference in golang/go#73567

cluster-autoscaler/test/integration/utils.go

cluster-autoscaler/builder/autoscaler.go

cluster-autoscaler/cloudprovider/test/fake_cloud_provider.go

mtrqq · 2026-02-05T13:53:12Z

cluster-autoscaler/test/integration/inmemory/staticautoscaler_test.go

+			WithKubeClient(kubeClient).
+			WithInformerFactory(fakeK8s.InformerFactory).
+			WithCloudProvider(fakeCloud).
+			WithPodObserver(&loop.UnschedulablePodObserver{}).


Isn't it something which we may default in the builder and add an override if needed? So far I don't see observer value being changed in the call sites

I believe this is what we are actually doing, see this code in the builder
if b.podObserver == nil { b.podObserver = loop.StartPodObserver(ctx, b.kubeClient) }

mtrqq · 2026-02-05T13:59:10Z

cluster-autoscaler/test/integration/inmemory/staticautoscaler_test.go

+	unneededTime = 1 * time.Minute
+)
+
+func TestStaticAutoscaler_FullLifecycle(t *testing.T) {


I would add add some baseline integration test which devs can refer to when adding new ones. The main point of which would obviously be to provide a skeleton for such a test, but also it may serve as a decent source of documentation for using synctest.

For me the current most lacking point is to outline a boundary between what autoscaler components can be configured in the test bauble and what can't or are not supposed to. Adding this information may steer new tests into the correct direction straight away resulting in a lot of time saved when debugging failures

Added it in the last commit

elmiko · 2026-02-05T17:44:23Z

i think this is a great contribution. i am still reviewing but i like what i see so far.

elmiko · 2026-02-09T19:51:21Z

/ok-to-test

elmiko

this is making sense to me, no specific request for changes at the moment.

kawych

Added couple more nitpicks, but these are just polishes, I think it's good to go!

@mtrqq @towca Would be great if you can give your oppinion on the thread about dependencies

kawych · 2026-02-11T15:58:12Z

cluster-autoscaler/test/integration/utils.go

+}
+
+// MustCreateManager creates a controller-runtime manager with metrics and health probes disabled.
+func MustCreateManager(t *testing.T) manager.Manager {


nit: "Manager" is one of the most generic and unhelpful terms, can you add some substance to the naming here?

NewControllerRuntimeMgr? Or even NewControllerRuntime if that doesn't collide with anything?

kawych · 2026-02-11T16:00:40Z

cluster-autoscaler/test/integration/config.go

+
+// Config is the "blueprint" for a test. It defines the entire
+// initial state of the world before the test runs.
+type Config struct {


nit: consider OptionsBuilder

kawych · 2026-02-11T16:01:55Z

cluster-autoscaler/test/integration/synctest/template_test.go

+	// Initialize fakes and configuration options.
+	// This happens outside the synctest bubble to keep the setup clean.
+	config := integration.NewConfig().WithOverrides() // override CA options if needed
+


uber-nit: consider removing excessive whitespaces

kawych · 2026-02-11T16:03:10Z

cluster-autoscaler/test/integration/synctest/template_test.go

+	unneededTime = 1 * time.Minute
+)
+
+func TestStaticAutoscaler_Template(t *testing.T) {


nit: consider adding a top-level comment explaining the purpose of this test.

k8s-ci-robot · 2026-02-12T09:09:06Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: GaetanoMar96, kawych
Once this PR has been reviewed and has the lgtm label, please assign towca for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

cluster-autoscaler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/feature Categorizes issue or PR as related to a new feature. labels Jan 22, 2026

k8s-ci-robot added do-not-merge/needs-area area/cluster-autoscaler labels Jan 22, 2026

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 22, 2026

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Jan 22, 2026

k8s-ci-robot requested review from elmiko and vadasambar January 22, 2026 13:09

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed do-not-merge/needs-area cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jan 22, 2026

GaetanoMar96 marked this pull request as draft January 22, 2026 13:20

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 22, 2026

GaetanoMar96 force-pushed the synctest-framework branch 5 times, most recently from 35ac017 to eb540b2 Compare January 23, 2026 09:59

GaetanoMar96 marked this pull request as ready for review January 23, 2026 10:01

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 23, 2026

k8s-ci-robot requested review from aleksandra-malinowska and feiskyer January 23, 2026 10:01

k8s-ci-robot assigned kawych and mtrqq Jan 23, 2026

kawych suggested changes Jan 23, 2026

View reviewed changes

GaetanoMar96 requested a review from kawych February 3, 2026 12:37

mtrqq reviewed Feb 3, 2026

View reviewed changes

GaetanoMar96 force-pushed the synctest-framework branch 3 times, most recently from 5180f1d to 58e27de Compare February 3, 2026 15:05

GaetanoMar96 requested a review from mtrqq February 3, 2026 15:29

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 4, 2026

GaetanoMar96 force-pushed the synctest-framework branch from 58e27de to 5233df5 Compare February 4, 2026 15:52

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 4, 2026

kawych reviewed Feb 4, 2026

View reviewed changes

GaetanoMar96 force-pushed the synctest-framework branch from 5233df5 to d529731 Compare February 4, 2026 18:10

mtrqq reviewed Feb 5, 2026

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 6, 2026

Choraden mentioned this pull request Feb 9, 2026

Draft: Add performance benchmark for the CA RunOnce control loop #9199

Draft

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Feb 9, 2026

GaetanoMar96 force-pushed the synctest-framework branch from d529731 to 919781f Compare February 10, 2026 10:28

k8s-ci-robot removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 10, 2026

GaetanoMar96 added 2 commits February 10, 2026 10:42

Refactor main to allow dependency injection

30580e9

Allow synctest usage passing context.

5056a24

GaetanoMar96 force-pushed the synctest-framework branch from 919781f to 5c82692 Compare February 10, 2026 10:43

Add a full cycle CA test.

ae452ea

GaetanoMar96 force-pushed the synctest-framework branch from 5c82692 to ae452ea Compare February 10, 2026 12:20

GaetanoMar96 requested review from kawych and mtrqq February 10, 2026 12:59

elmiko reviewed Feb 10, 2026

View reviewed changes

kawych approved these changes Feb 12, 2026

View reviewed changes

Introduce synctest-based framework for deterministic testing #9099

Are you sure you want to change the base?

Introduce synctest-based framework for deterministic testing #9099

Uh oh!

Conversation

GaetanoMar96 commented Jan 22, 2026

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

linux-foundation-easycla bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jan 22, 2026

Uh oh!

k8s-ci-robot commented Jan 22, 2026

Uh oh!

GaetanoMar96 commented Jan 23, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kawych left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mtrqq left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elmiko commented Feb 5, 2026

Uh oh!

elmiko commented Feb 9, 2026

Uh oh!

linux-foundation-easycla bot commented Jan 22, 2026 •

edited

Loading