-
Notifications
You must be signed in to change notification settings - Fork 72
Speed up printing #526
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Speed up printing #526
Conversation
for more information, see https://pre-commit.ci
|
Wonderful @FBumann ! I can review on Monday. But looks promising and shouldn't take long to pull in. The scripts in dev-scripts are intended? This is normally ignored. Any breaking changes I should look out for/we should mention are n the release notes? |
|
@FabianHofmann The dev scripts were used to produce the benchmark and i wanted to show them for transparency. But no, they should not get merged (as stated in #Before Merging). |
…ed-up-printing/main # Conflicts: # dev-scripts/benchmark_bottlenecks.py # dev-scripts/benchmark_printing.py
|
@FBumann I have made a rigorous test on the consistency of printouts, it all looks good. this is a wonderful pr, kudos and thank you very much! |
PR Title
Performance: Up to 50x faster printing via O(log n) label lookup
Summary
get_label_position()from O(n) linear search to O(log n) binary search usingLabelPositionIndexclass.sel()xarray lookups with direct numpy array indexing inprint_single_variable()nvarsandnconsproperties to avoid expensive.flat.labels.unique()callsChanges proposed in this Pull Request
New
LabelPositionIndexclass (linopy/common.py)np.searchsortedfor fast O(log n) lookupsfind_single()returns (name, coord) tuplefind_single_with_index()returns (name, coord, numpy_index) for direct array accessOptimized
print_single_variable()(linopy/common.py)get_label_position_with_index()to get raw numpy indiceslower/upperbounds via directvar.values[index]instead of.sel(coord)Optimized
nvarsandnconsproperties (linopy/variables.py,linopy/constraints.py).flat.labels.unique()with direct iteration and countingCache management (
linopy/variables.py,linopy/constraints.py)_label_position_indexfield toVariablesandConstraintsclassesadd()andremove()operationsBenchmark Results
Scaling benchmark across model sizes (n_arrays = number of variable/constraint arrays, 10 vars each):
Print all variables
Print all constraints
nvars property
ncons property
Summary at largest model size (1000 arrays)
Index building cost is negligible (~0.08ms for 100 arrays, ~0.75ms for 1000 arrays).
Before Merging
Remove development/benchmark files from
dev-scripts/:benchmark_printing.pybenchmark_scaling.pybenchmark_bottlenecks.pybenchmark_document_model.pybenchmark_scaling_plot.htmlPR_DESCRIPTION.mdChecklist
doc.doc/release_notes.rstof the upcoming release is included.