Skip to content

Conversation

@FBumann
Copy link
Contributor

@FBumann FBumann commented Nov 27, 2025

PR Title

Performance: Up to 50x faster printing via O(log n) label lookup


Summary

  • Up to 52x speedup for printing operations on models with many variable/constraint arrays
  • Optimizes get_label_position() from O(n) linear search to O(log n) binary search using LabelPositionIndex class
  • Replaces slow .sel() xarray lookups with direct numpy array indexing in print_single_variable()
  • Optimizes nvars and ncons properties to avoid expensive .flat.labels.unique() calls

Changes proposed in this Pull Request

New LabelPositionIndex class (linopy/common.py)

  • Lazy-built sorted index of label ranges
  • Binary search via np.searchsorted for fast O(log n) lookups
  • find_single() returns (name, coord) tuple
  • find_single_with_index() returns (name, coord, numpy_index) for direct array access
  • Cache invalidation when variables/constraints are added or removed

Optimized print_single_variable() (linopy/common.py)

  • Uses get_label_position_with_index() to get raw numpy indices
  • Accesses lower/upper bounds via direct var.values[index] instead of .sel(coord)

Optimized nvars and ncons properties (linopy/variables.py, linopy/constraints.py)

  • Replaced .flat.labels.unique() with direct iteration and counting
  • Properly handles scalar constraints and masked variables

Cache management (linopy/variables.py, linopy/constraints.py)

  • Added _label_position_index field to Variables and Constraints classes
  • Cache is automatically invalidated on add() and remove() operations

Benchmark Results

Scaling benchmark across model sizes (n_arrays = number of variable/constraint arrays, 10 vars each):

newplot (1) newplot

Print all variables

n_arrays Original (ms) Optimized (ms) Speedup
10 15.1 4.6 3.3x
100 351.1 47.2 7.4x
500 6231.6 240.6 25.9x
1000 24446.5 468.2 52.2x

Print all constraints

n_arrays Original (ms) Optimized (ms) Speedup
10 4.1 3.0 1.4x
100 157.7 31.0 5.1x
500 3389.1 155.5 21.8x
1000 13928.5 317.9 43.8x

nvars property

n_arrays Original (ms) Optimized (ms) Speedup
10 3.0 1.6 1.9x
100 28.1 16.0 1.7x
1000 308.8 160.3 1.9x

ncons property

n_arrays Original (ms) Optimized (ms) Speedup
10 8.9 0.2 45.8x
100 88.1 2.0 45.1x
1000 922.4 20.0 46.1x

Summary at largest model size (1000 arrays)

  • Print all variables: 52.2x faster (24.4s → 0.47s)
  • Print all constraints: 43.8x faster (13.9s → 0.32s)
  • ncons property: 46.1x faster (922ms → 20ms)
  • nvars property: 1.9x faster (309ms → 160ms)

Index building cost is negligible (~0.08ms for 100 arrays, ~0.75ms for 1000 arrays).

Before Merging

Remove development/benchmark files from dev-scripts/:

  • benchmark_printing.py
  • benchmark_scaling.py
  • benchmark_bottlenecks.py
  • benchmark_document_model.py
  • benchmark_scaling_plot.html
  • PR_DESCRIPTION.md

Checklist

  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

@FabianHofmann
Copy link
Collaborator

Wonderful @FBumann ! I can review on Monday. But looks promising and shouldn't take long to pull in. The scripts in dev-scripts are intended? This is normally ignored. Any breaking changes I should look out for/we should mention are n the release notes?

@FBumann
Copy link
Contributor Author

FBumann commented Nov 28, 2025

@FabianHofmann The dev scripts were used to produce the benchmark and i wanted to show them for transparency. But no, they should not get merged (as stated in #Before Merging).
I will remove them. If you want to take a look you can checkout the intermediate commit i guess.

@FabianHofmann
Copy link
Collaborator

@FBumann I have made a rigorous test on the consistency of printouts, it all looks good. this is a wonderful pr, kudos and thank you very much!

@FabianHofmann FabianHofmann merged commit c6c64a7 into PyPSA:master Dec 1, 2025
19 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants