Add configurable option to scale-down during task run time for cost-based autoscaler by Fly-Style · Pull Request #18958 · apache/druid

Fly-Style · 2026-01-27T13:32:24Z

To test the cost-based autoscaler in real-world scenarios, we temporarily allow scale-downs during task runtime. In case if we don't want to scale down immediately.
Two configuration options has been added:

scaleDownDuringTaskRolloverOnly indicates if task scaling down is limited to periods during task rollovers only. false by default, allows scaling down during normal task run time.
scaleDownBarrier, A number of successful scale down attempts which should be skipped to allow prevention of the auto-scaler from scaling down tasks immediately when the computed cost-based metrics fall below this barrier. Setting it to 0 removes the threshold.

There is a parallel PR that tries to implement the most correct behaviour for scaling during task rollover: #18954

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
been tested in a test Druid cluster.

…ased autoscaler

capistrant

serde test needs fixing up it seems. in addition to that left some lower stakes comments questions. Overall I think it is mergeable as is, but I do wonder about best default for scaleDownDuringTaskRolloverOnly in the long term

capistrant · 2026-01-27T16:48:05Z

...java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/CostBasedAutoScaler.java

+    } else if (!config.isScaleDownOnTaskRolloverOnly() && optimalTaskCount < currentTaskCount &&
+               (config.getScaleDownBarrier() == 0 || ++scaleDownCounter == config.getScaleDownBarrier())) {


nit suggestion: might be easier cognitively to drop the ||?

Suggested change

} else if (!config.isScaleDownOnTaskRolloverOnly() && optimalTaskCount < currentTaskCount &&

(config.getScaleDownBarrier() == 0 || ++scaleDownCounter == config.getScaleDownBarrier())) {

} else if (!config.isScaleDownOnTaskRolloverOnly() && optimalTaskCount < currentTaskCount && ++scaleDownCounter >= config.getScaleDownBarrier()) {

Yep, good catch!

Fixed: 688833e

capistrant · 2026-01-27T16:50:35Z

docs/ingestion/supervisor.md

 |`idleWeight`|The weight of extracted poll idle value in cost function. | No | 0.75 |
 |`defaultProcessingRate`|A planned processing rate per task, required for first cost estimations. | No | 1000 |
+|`scaleDownBarrier`| A number of successful scale down attempts which should be skipped  to prevent the auto-scaler from scaling down tasks immediately.  | No | 5 |
+|`scaleDownDuringTaskRolloverOnly`| Indicates whether task scaling down is limited to periods during task rollovers only. | No | False |


Is it possible that we want this to be true as default, if that is like the production behavior we think will be preferred? I say if that is like the production behavior we think will be preferred because of this in the description:

To test the cost-based autoscaler in real-world scenarios, we temporarily allow scale-downs during task runtime

like instead of temporarily defaulting for testing, default for long term productionalization plans and we can override for folks testing behavior more rapidly?

Actually, that's the case. Scale down during task rollover is now in unsatisfactory condition, and the existence of that PR with scaleDownDuringTaskRolloverOnly set to false is required for smooth autoscaler testing.
There is a parallel PR: #18954, I tried to integrate a new way to gradually stop tasks and scale down, but it is very risky and need a very careful review and testing.

Well, by bad, I did not produce a good PR description. I updated the it.

capistrant · 2026-01-27T16:58:06Z

...rg/apache/druid/indexing/seekablestream/supervisor/autoscaler/CostBasedAutoScalerConfig.java

    this.idleWeight = Configs.valueOrDefault(idleWeight, DEFAULT_IDLE_WEIGHT);
    this.defaultProcessingRate = Configs.valueOrDefault(defaultProcessingRate, DEFAULT_PROCESSING_RATE);
+    this.scaleDownBarrier = Configs.valueOrDefault(scaleDownBarrier, DEFAULT_SCALE_DOWN_BARRIER);
+    this.scaleDownDuringTaskRolloverOnly = Configs.valueOrDefault(scaleDownDuringTaskRolloverOnly, false);


nit: should this follow pattern of default at top of file or are we only doing that for magic numbers with context added in comments?

Sorry, did not get your point :)

Just wondering if inlining the default when the rest are constants at the top of the file for just this one config is intentional. I think it is no big deal

capistrant · 2026-01-27T17:13:50Z

...rg/apache/druid/indexing/seekablestream/supervisor/autoscaler/CostBasedAutoScalerConfig.java

 {
-  private static final long DEFAULT_SCALE_ACTION_PERIOD_MILLIS = 15 * 60 * 1000; // 15 minutes
-  private static final long DEFAULT_MIN_TRIGGER_SCALE_ACTION_FREQUENCY_MILLIS = 1200000; // 20 minutes
+  private static final long DEFAULT_SCALE_ACTION_PERIOD_MILLIS = 10 * 60 * 1000; // 10 minutes


looks like this caused the default serde test to fail. I assume the one below would also fail assertions if not fixed up as well.

Fixed: 688833e

capistrant

thanks for the fixing up @Fly-Style. LGTM

kfaraz · 2026-02-03T09:07:09Z

...pache/druid/indexing/seekablestream/supervisor/autoscaler/CostBasedAutoScalerConfigTest.java

+    Assert.assertEquals(DEFAULT_SCALE_ACTION_PERIOD_MILLIS, config.getScaleActionPeriodMillis());
+    Assert.assertEquals(DEFAULT_MIN_TRIGGER_SCALE_ACTION_FREQUENCY_MILLIS, config.getMinTriggerScaleActionFrequencyMillis());
+    Assert.assertEquals(DEFAULT_LAG_WEIGHT, config.getLagWeight(), 0.001);
+    Assert.assertEquals(DEFAULT_IDLE_WEIGHT, config.getIdleWeight(), 0.001);
+    Assert.assertEquals(DEFAULT_PROCESSING_RATE, config.getDefaultProcessingRate(), 0.001);
+    Assert.assertEquals(DEFAULT_SCALE_DOWN_BARRIER, config.getScaleDownBarrier());


Rather than importing the constants, it would be nice to verify the numerical values directly here, so that when the constants change, we consciously update the tests as well.

kfaraz · 2026-02-03T09:11:33Z

docs/ingestion/supervisor.md

 |`idleWeight`|The weight of extracted poll idle value in cost function. | No | 0.75 |
 |`defaultProcessingRate`|A planned processing rate per task, required for first cost estimations. | No | 1000 |
+|`scaleDownBarrier`| A number of successful scale down attempts which should be skipped  to prevent the auto-scaler from scaling down tasks immediately.  | No | 5 |
+|`scaleDownDuringTaskRolloverOnly`| Indicates whether task scaling down is limited to periods during task rollovers only. | No | False |


Thanks for adding this config, it makes sense to have this option. But I agree with @capistrant in that we would want the default of this flag to be true so that scale down happens only on task rollover by default. But we can fix it up once we have addressed the issues with the task rollover bit.

That's a reason for this option's existence, actually :D

kfaraz · 2026-02-03T09:23:08Z

docs/ingestion/supervisor.md

 |`lagWeight`|The weight of extracted lag value in cost function.| No| 0.25|
 |`idleWeight`|The weight of extracted poll idle value in cost function. | No | 0.75 |
 |`defaultProcessingRate`|A planned processing rate per task, required for first cost estimations. | No | 1000 |
+|`scaleDownBarrier`| A number of successful scale down attempts which should be skipped  to prevent the auto-scaler from scaling down tasks immediately.  | No | 5 |


This config does not seem to be aligned with the whole cost-based story.
It is a little counterintuitive as it essentially negates the decision taken by the auto-scaler.

IIUC, this has been put in as a hack until we have fixed the issue with scaling on task rollover.

Please remove this config and we should try to evaluate the following instead:

increase the window over which we compute the average lag

AND/OR add more terms to the cost function

AND/OR add a config like minScaleDownDelay (which serves a role similar to the scaleDownBarrier but in terms of time rather than count)

minScaleDelay idea looks better, I like it more 👍🏻

github-actions bot added the Area - Ingestion label Jan 27, 2026

Fly-Style mentioned this pull request Jan 27, 2026

Enhance autoscaling during task rollover #18954

Open

6 tasks

Fly-Style force-pushed the cost-base-autoscaler-scaledown-with-barrier branch from e689f4e to e26868e Compare January 27, 2026 13:48

Add configurable option to scale-down during task run time for cost-b…

4fd0f86

…ased autoscaler

Fly-Style force-pushed the cost-base-autoscaler-scaledown-with-barrier branch from e26868e to 4fd0f86 Compare January 27, 2026 14:06

Docs

0f66c0a

github-actions bot added the Area - Documentation label Jan 27, 2026

capistrant self-requested a review January 27, 2026 15:44

capistrant requested changes Jan 27, 2026

View reviewed changes

Address review comments, compress tests a bit

688833e

capistrant approved these changes Jan 28, 2026

View reviewed changes

capistrant merged commit ed0264a into apache:master Jan 28, 2026
40 checks passed

Fly-Style deleted the cost-base-autoscaler-scaledown-with-barrier branch January 28, 2026 15:39

kfaraz reviewed Feb 3, 2026

View reviewed changes

		} else if (!config.isScaleDownOnTaskRolloverOnly() && optimalTaskCount < currentTaskCount &&
		(config.getScaleDownBarrier() == 0 \|\| ++scaleDownCounter == config.getScaleDownBarrier())) {

Conversation

Fly-Style commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

capistrant left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fly-Style Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

capistrant left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Fly-Style commented Jan 27, 2026 •

edited

Loading

Fly-Style Jan 28, 2026 •

edited

Loading