Cling JIT out-of-memory error when using large number of expression-based NormFactors

[tianzhuli2002]

Check duplicate issues.

• Checked for duplicates

Description

Hello,

We are encountering a reproducible out-of-memory error in ROOT when creating a large number of expression-based parameters during RooFit workspace construction.

Similar issues:
#14156
https://root-forum.cern.ch/t/cling-jit-session-error-cannot-allocate-memory/56744

As a stress test, we programmatically create many parameters defined via expressions (schematically of the form):

define a base parameter SF_i

define a derived parameter SquaredSF_i = SF_i * SF_i

Each derived parameter is implemented as a RooFit expression (e.g. via RooFormulaVar / expression factories), and all are inserted into a single workspace.

When scaling this up to O(10⁴–10⁵) such expression-based parameters (≈30,000 in our test), the job crashes during workspace creation with:

cling JIT session error: Cannot allocate memory
LLVM ERROR: out of memory

Our practical use case is a large simultaneous likelihood fit with many regions and samples, where custom normalisation scale factors are defined through algebraic expressions and applied consistently across subsets of regions and samples. In our real configuration, this corresponds to:

O(500) regions

O(4000) samples

O(70) expression-based scale factors (slightly more complex than a simple square)

While we do not require O(30,000) such expressions in production, we observe that the job already crashes reliably at this realistic scale, with the same Cling JIT out-of-memory failure. Reducing the number of expression-based scale factors to ~10–20 makes the problem disappear, even with the same regions and samples.

Reproducer

To run:

root -l
.L cling_test.C+

cling_test(30000)

cling_test.C

ROOT version

ROOT 6.37.01

Installation method

~/miniforge3/bin/root

Operating system

Platform: linuxx8664gcc

Additional context

No response

[guitargeek]

Thanks for the report! I think RooFit/HistFactory should not force you to use the interpreter. It's nice to have it for some of the high-level abstractions like the RooWorkspace factory language, but there should be always a way to do everything without the interpreter if possible.

In the case of Measurement::AddPreprocessFunction, I think the solution would be to add an additional overload that can take any C++ function callback as an argument, which specifies how the preprocess functions are created. Something like:

void createPreprocessFunction(RooWorkspace &ws) {
   // you write this function
   ws.import(RooProduct("SquaredSF", ... etc));
}

/// later you do:
meas.AddPreprocessFunctionCallback(createPreprocessFunction);

You think that would work for your use case?

By the way, do you intend to create the workspaces from Python? Then I'd have to take extra case that this add preprocess function callback also works from Python.

[TomasDado]

Thanks @guitargeek, I think this could work. But to add a bit of context, one very nice advatange of the JITting code is that one can pass a string from some config file without modifying the code. In this example there is a simple formula (prodcut) that can be easily passed in the code, but not so easily via the config. So the solution would work in this particular case, but its not a general solution unfortunately.

[guitargeek]

I see. We could consider writing a little expression to RooFit parser, that translates the expression to existing RooFit classes like RooProduct etc. Or to reuse already jitted code if the math is the same. I think the latter would not be a big change but could have a big positive impact on memory usage and runtime, provided the expressions are similar.

[tianzhuli2002]

Hi yes I think adding the additional overload would work in our case! Also reusing already jitted code would also be a nice add as well. Thanks!

[tianzhuli2002]

Also sorry - forgot to answer your question on creating the workspaces from python! We don't plan to use python - we are using trex-fitter to build this RooFit/HistFactory workspace. Really appreciate the help on this!