SpineOpt input data structure update

[nhniina]

Tasku: These are outdated (2025-12-09).

SpineOpt input data structure will see major changes in the forthcoming versions. The plan for the changes is desribed here.

The following classes will be updated:

old class	new class
unit__from_node	node__to_unit
unit__from_node__investment_group	unit_flow__investment_group
unit__from_node__user_constraint	unit_flow__user_constraint
unit__node__node	unit_flow__unit_flow
unit__to_node__investment_group	unit_flow__investment_group
unit__to_node__user_constraint	unit_flow__user_constraint

In addition, the following new class will be created:

new class	additional info
unit_flow	superclass for node__to_unit and unit__to_node subclasses

Several parameters will be renamed. They are listed here:

old class	old parameter name	new class	new_parameter
connection	connection_availability_factor	connection	availability_factor
connection	connection_investment_econ_lifetime	connection	econ_lifetime
connection	connection_investment_tech_lifetime	connection	tech_lifetime
connection	connection_investment_variable_type	connection	investment_variable_type
connection	connection_reactance	connection	reactance
connection	connection_resistance	connection	resistance
model	db_lp_solver	model	lp_solver
model	db_lp_solver_options	model	lp_solver_options
model	db_mip_solver	model	mip_solver
model	db_mip_solver_options unit__to_node	model	mip_solver_options
node	frac_state_loss	node	storage_self_discharge
node	state_coeff	node	storage_state_coeff
node	storage_investment_econ_lifetime	node	storage_econ_lifetime
node	storage_investment_tech_lifetime	node	storage_tech_lifetime
temporal_block	representative_periods_mapping	temporal_block	periods_represented?
unit	number_of_units	unit	existing_units
unit	unit_availability_factor	unit	availability_factor
unit	unit_investment_econ_lifetime	unit	econ_lifetime
unit	unit_investment_tech_lifetime	unit	tech_lifetime
unit	unit_investment_variable_type	unit	investment_variable_type
unit__from_node	fuel_cost	node__to_unit	flow_cost
unit__from_node	unit_capacity	node__to_unit	capacity_per_unit
unit__from_node	vom_cost	node__to_unit	flow_cost
unit__to_node	fuel_cost	unit__to_node	flow_cost
unit__to_node	unit_capacity	unit__to_node	capacity_per_unit
unit__to_node	vom_cost	unit__to_node	flow_cost
unit__node__node	fix_ratio_in_in_unit_flow	unit_flow__unit_flow	equality_constraint
unit__node__node	fix_ratio_in_out_unit_flow	unit_flow__unit_flow	equality_constraint
unit__node__node	fix_ratio_out_in_unit_flow	unit_flow__unit_flow	equality_constraint
unit__node__node	fix_ratio_out_out_unit_flow	unit_flow__unit_flow	equality_constraint
unit__node__node	fix_units_on_coefficient_in_in	unit_flow__unit_flow	equality_constant
unit__node__node	fix_units_on_coefficient_in_out	unit_flow__unit_flow	equality_constant
unit__node__node	fix_units_on_coefficient_out_in	unit_flow__unit_flow	equality_constant
unit__node__node	fix_units_on_coefficient_out_out	unit_flow__unit_flow	equality_constant
unit__node__node	max_ratio_in_in_unit_flow	unit_flow__unit_flow	greater_than_constraint
unit__node__node	max_ratio_in_out_unit_flow	unit_flow__unit_flow	greater_than_constraint
unit__node__node	max_ratio_out_in_unit_flow	unit_flow__unit_flow	greater_than_constraint
unit__node__node	max_ratio_out_out_unit_flow	unit_flow__unit_flow	greater_than_constraint
unit__node__node	max_units_on_coefficient_in_in	unit_flow__unit_flow	greater_than_constant
unit__node__node	max_units_on_coefficient_in_out	unit_flow__unit_flow	greater_than_constant
unit__node__node	max_units_on_coefficient_out_in	unit_flow__unit_flow	greater_than_constant
unit__node__node	max_units_on_coefficient_out_out	unit_flow__unit_flow	greater_than_constant
unit__node__node	min_ratio_in_in_unit_flow	unit_flow__unit_flow	less_than_constraint
unit__node__node	min_ratio_in_out_unit_flow	unit_flow__unit_flow	less_than_constraint
unit__node__node	min_ratio_out_in_unit_flow	unit_flow__unit_flow	less_than_constraint
unit__node__node	min_ratio_out_out_unit_flow	unit_flow__unit_flow	less_than_constraint
unit__node__node	min_units_on_coefficient_in_in	unit_flow__unit_flow	less_than_constant
unit__node__node	min_units_on_coefficient_in_out	unit_flow__unit_flow	less_than_constant
unit__node__node	min_units_on_coefficient_out_in	unit_flow__unit_flow	less_than_constant
unit__node__node	min_units_on_coefficient_out_out	unit_flow__unit_flow	less_than_constant

In addition to simple renaming, some parameters will be changed from a flat structure to Map parameters, allowing for easier grouping of parameters that belong to certain features. These are listed here:

old class	old parameter name	new class	new parameter name	map index
connection	candidate_connections	connection	investment_limits	[max_new]
connection	connections_invested_big_m_mga	connection	mga	[investment_big_m]
connection	connections_invested_mga	connection	mga	[investment]
connection	connections_invested_mga_weight	connection	mga	[investment_weight]
connection	fix_connections_invested	connection	investment_limits	[fix_new]
connection	fix_connections_invested_available	connection	investment_limits	[fix_cumulative]
connection	initial_connections_invested	connection	investment_limits	[min_new]?
connection	initial_connections_invested_available	connection	investment_limits	[min_cumulative]?
model	max_gap	model	decomposition	[max_gap]
model	max_iterations	model	decomposition	[max_iterations]
model	max_mga_iterations	model	mga	[max_iterations]
model	max_mga_slack	model	mga	[max_slack]
model	min_iterations	model	decomposition	[min_iterations]
node	candidate_storages	node	storage_investment_limits	[max_new]
node	fix_node_pressure	node	pressure_limits	[fix]
node	fix_node_state	node	storage_state_limits	[fix]
node	fix_node_voltage_angle	node	voltage_angle_limits	[fix]
node	fix_storages_invested	node	storage_investment_limits	[fix_new]
node	fix_storages_invested_available	node	storage_investment_limits	[fix_cumulative]
node	initial_node_pressure	node	pressure_limits	[initial]
node	initial_node_state	node	storage_state_limits	[initial]
node	initial_node_voltage_angle	node	voltage_angle_limits	[initial]
node	initial_storages_invested	node	storage_investment_limits	[min_new]?
node	initial_storages_invested_available	node	storage_investment_limits	[min_cumulative]?
node	max_node_pressure	node	pressure_limits	[max]
node	max_voltage_angle	node	voltage_angle_limits	[max]
node	min_node_pressure	node	pressure_limits	[min]
node	min_voltage_angle	node	voltage_angle_limits	[min]
node	node_state_cap	node	storage_state_limits	[max]
node	node_state_min	node	storage_state_limits	[min]
node	storages_invested_big_m_mga	node	storage_mga	[investment_big_m]
node	storages_invested_mga	node	storage_mga	[investment]
node	storages_invested_mga_weight	node	storage_mga	[investment_weight]
unit	candidate_units	unit	investment_limits	[max_new]
unit	fix_units_invested	unit	investment_limits	[fix_new]
unit	fix_units_invested_available	unit	investment_limits	[fix_cumulative]
unit	fix_units_on	unit	units_online	[min, max]
unit	initial_units_invested	unit	investment_limits	[min_new]?
unit	initial_units_invested_available	unit	investment_limits	[min_cumulative]?
unit	initial_units_on	unit	units_online	[initial]
unit	units_invested_big_m_mga	unit	mga	[investment_big_m]
unit	units_invested_mga	unit	mga	[investment]
unit	units_invested_mga_weight	unit	mga	[investment_weight]
unit__from_node	fix_unit_flow	node__to_unit	flow_limits	[fix]
unit__from_node	initial_unit_flow	node__to_unit	flow_limits	[initial]
unit__from_node	max_total_cumulated_unit_flow_from_node	node__to_unit	flow_limits	[max_cumulative]
unit__from_node	min_total_cumulated_unit_flow_from_node	node__to_unit	flow_limits	[min_cumulative]
unit__from_node	min_unit_flow	node__to_unit	flow_limits	[min]
unit__from_node	ramp_down_limit	node__to_unit	ramp_limits	[ramp_down]
unit__from_node	ramp_up_limit	node__to_unit	ramp_limits	[ramp_up]
unit__from_node	shut_down_limit	node__to_unit	ramp_limits	[shutdown]
unit__from_node	start_up_limit	node__to_unit	ramp_limits	[startup]
unit__to_node	fix_unit_flow	unit__to_node	flow_limits	[fix]
unit__to_node	initial_unit_flow	unit__to_node	flow_limits	[initial]
unit__to_node	max_total_cumulated_unit_flow_to_node	unit__to_node	flow_limits	[max_cumulative]
unit__to_node	min_total_cumulated_unit_flow_to_node	unit__to_node	flow_limits	[min_cumulative]
unit__to_node	min_unit_flow	unit__to_node	flow_limits	[min]
unit__to_node	ramp_down_limit	unit__to_node	ramp_limits	[ramp_down]
unit__to_node	ramp_up_limit	unit__to_node	ramp_limits	[ramp_up]
unit__to_node	shut_down_limit	unit__to_node	ramp_limits	[shutdown]
unit__to_node	start_up_limit	unit__to_node	ramp_limits	[startup]

A few parameters will be moved from one class to another class. Some of these also renamed at the same time. The parameters that will be moved are:

old class	old parameter name	new class	new parameter name
unit	curtailment_cost	unit__to_node, node__to_unit	curtailment_cost
unit	fom_cost	unit__to_node, node__to_unit	fixed_annual_cost
unit	shut_down_cost	unit__to_node, node__to_unit	shutdown_cost
unit	start_up_cost	unit__to_node, node__to_unit	startup_cost

Finally, some parameters will be moved to another class, the parameter may be renamed, and some calculation may be performed on the parameter. These parameters are the following:

old class	old parameter name	new class	new parameter name	additional info
unit	unit_investment_cost	unit__to_node, node__to_unit	investment_cost	divided by unit capacity
unit	units_on_cost	unit__to_node, node__to_unit	online_cost	divided by unit capacity

Additional changes

• node: balance_type will be renamed to node_type and updated to include options storage_node and storage_group (based on has_state, which will be removed)

[manuelma]

Great stuff @nhniina - this level of exhaustivity is really great to assess these changes.

I have some comments:

• node_type instead of balance_type? I might prefer to keep the latter, it is way more explicit.
• periods_represented instead of representative_periods_mapping? I don't like any of them too much but I feel we could do a little bit better than periods_represented, maybe representative_temporal_blocks? In any case we should name mapping parameters after the values and not after the keys if we want to follow the computer science standard. Another option is the 'value_by_key' approach, so in this case something like representative_block_by_period might work? And yet another option is the 'key_to_value' convention (period_to_representative_block).
• investment_method instead of investment_variable_type? I again prefer to keep the latter, it's more explicit. The other one throws me off as something completely different than the type of variable used to represent investments.
• The map_index for initial_storages_invested shouldn't be [min_new] but rather [initial] I believe. This might also be the case for other initial_* parameters so could be useful to double check.

But again, thank you so much for going through all this.

[manuelma]

Here is a list with some of the toolbox/spinedb_api requirements we identified in our last meeting:

1. Having values from parameter_value_list within indexed values (i.e., array, map, time-series, time-pattern), also validated against the list.
2. Have the option to initialize parameter values with a map with the valid indexes (e.g., fix, min, max), so the user can then edit the values directly.
3. Have a new visualization option to expand indexed values into the parameter value pane in DB editor (similar to what we do in the pivot table where we add a new column with the index).
4. Validate map indexes against a list, validate uniqueness. Also show the valid indexes in a combo box in DB editor.

Also, these are some issues related to SpineOpt/Interface.

1. SpineInterface needs to check inside maps to determine if a value is specified or not, and then use the default in that case. And it needs to use the default for the right index.
2. SpineOpt needs to include the index argument in the call to map parameters.
3. Does SpineInterface need to handle superclass_subclass in a way?

Did I miss something @nhniina @DillonJ @Tasqu ?

[DillonJ]

Thinking about this a little more - in addition to the above, an issue with the map component of the plan - that is, having sub-parameters inside maps is that inside DB editor, when you click on an entity, say a unit, all I will see in the parameter panes is map, map, map... no only do I not see the values, but I don't see what parameters are defined either and would constantly have to click inside each map to see what is going on. We could implement additional functionality for visualisation of maps.... say we could auto expand the contents of maps - which I would enable by default so I can see what is going on... then we end up back where we are no and I don't quite see the point of all the effort. Additionally when a user is searching for a parameter value, these sub parameters won't appear in the list. They will have to know the parent parameter name and somehow know what sub parameters each contains.

I feel like the itention behind the map thing is to more conveniently organise parameters for the user. But my feeling is that using maps either results in a lot of work to implement the missing functionality, or a significant loss in utility for the user.

If we want this parameter grouping, maybe we should consider implementing the functionality directly and not use maps. For example, I was looking at the SCIP solver options... they look like this:

conflict/maxlploops = 2
conflict/maxvarsfac = 0.15
conflict/lpiterations = 10

Maybe it would be simpler to implement some kind of parameter grouping intead of hiding things in maps?

I just spent a few minutes applying the changes to a random unit from the DB I'm working on now. This is what the changes look like. In this real case, there is no reduction in the number of parameters and the main impact for the user is that in place of 4 scalar values, there are now 4 maps instead, needing to be created by hand and each having to be clicked into to see the actual values.

Now:

After:

[nhniina]

That's a valid point. Instead of using maps or grouping, there could also be a new data type in Spine Toolbox that would be intended for these types of parameters where the indices are predefined. Its contents could also be shown in a summarised way in the value column.

In your example, just the candidate_units parameter would be converted to a map parameter or similar. Equality/less_than_constraints are meant to be scalars / arrays also in the new version, but they would move to a new class unit_flow__unit_flow.

[manuelma]

Thank you @DillonJ - this is also the way I feel about it. Changing the data structure for visualization purposes is generally wrong, instead you should implement the views that satisfy you on top of your existing data structure. Data structure changes are generally more risky and less isolated than visualization changes.

So if we want parameter grouping in the view, we should do just that, possibly adding new features to the existing structure like a new column 'group' to the parameter definition, but never assume the view is fixed and that we need to play with the data structure to be able to see what we want.

[nhniina]

Summarising the plan forward

• Let's use the grouping option instead of maps or structs
• Let's take the grouping into account in the renaming of the parameters
• Let's use a single underscore to separate the future group name and the rest of the parameter name
• Whenever sensible, let's rename parameters so that they are ordered logically when sorted alphabetically
• Spine Toolbox will implement the visualisation of groups later
• After groups are enabled from Spine Toolbox side, we can introduce them in the SpineOpt template

[Tasqu]

So just a quick progress update:

• I've merged the "old" changes by @nhniina with the current master branch and fixed the unit tests. See the datastructure_update branch.
• Because the new structure relies on superclasses, you also need the #139_attempt4 branch of SpineInterface.

Next steps:

• Go through the latest plans and see what still remains to be implemented.
• Implement said changes, test them, and try to have something ready for user testing ASAP.

[Tasqu]

Another progress update:

• I've completed the first round of changes started by @nhniina (not exactly the ones outlined in the table above, since that's outdated), mainly consisting of simple parameter/class renaming and the unit_flow__unit_flow and fix_ratio rework for units. These changes are in the datastructure_update branch, but due to the required superclass functionality you will also need the #139_attempt4 branch of SpineInterface.
• From here, the question is whether we want to merge these in pieces, or if we want to keep them separate until a full overhaul. There are still a few changes that have been proposed, that I've yet to implement (if we want to implement them at all).

Next steps would be to:

1. Test the changes prior to merging, preferably. (merge needs to wait for Julia v1.12 fix anyhow)
2. Update spineopt_template.json to the new entity class structure?
3. Discuss and potentially implement some remaining smallish changes (e.g. moving curtailment_cost from unit to unit__to_node).
4. Discuss some more radical proposed changes to connection.

[Tasqu]

Very rough draft of still proposed possible parameter renamings and changes:

2025-12-16: Rest of the changes left for later if we have the time.

"Easy" renaming improvements:

• compression_factor -> ??? # Can we make this clearer that it's related to gas flow?
• connection_capacity -> capacity_per_connection # Should we reuse capacity_per_unit? Or would just capacity suffice?
• downward_reserve -> reserve_downward(?) # Make reserve parameters group when sorting. So this is a boolean flag like is_reserve_node? Why do we have this flag in so many places?
• fix_binary_gas_connection_flow -> binary_gas_flow_limits_fix # Rename gas parameters to group better?
• fix_connection_flow -> flow_limits_fix(?) # Reuse existing parameter names
• fix_connection_intact_flow -> flow_limits_fix_intact(?) # Group with similar params
• has_binary_gas_flow -> binary_gas_flow_activate(??) # Group with other gas flow params?
• has_storage -> storage_activate(?) # Group with storage params?
• initial_binary_gas_connection_flow -> binary_gas_flow_limits_initial # Group with other gas flow params?
• initial_connection_flow -> flow_limits_initial(??) # Reuse flow limits
• initial_connection_intact_flow -> flow_limits_initial_intact(??) # Group with flow limits
• is_longterm_storage -> storage_longterm_activate(?) # Group with other storage params
• is_reserve_node -> reserve_activate(?) # Group with other reserve params. Should _activate be _active for conciseness?
• node_balance_penalty -> balance_penalty(?) # To align with "balance_type"?
• node_balance_sense -> balance_sense(?) # To align with "balance_type"?
• unit_capacity -> capacity_per_unit(??) # Can we better align this with connection_capacity and conv_cap_to_flows?
• unit_decommissioning_cost -> decommissioning_cost(?) # Hasn't been implemented yet?
• unit_start_flow -> ??? # Group with other flow limits? No, this is actually completely different as it affects the flow ratio constraints.
• upward_reserve -> reserve_upwards(?) # Group reserve parameters better?
• vom_cost -> flow_cost(??) # Should align with other cost renamings, if any? Need to consider renaming fuel_cost if this is changed?

Sweeping renamings?

• Change all _activate to _active for conciseness and better code readability?
• Should cost parameters all start with cost_ to make them group alphabetically?

Remove? Or at least Rename to something clearer?

• connection_conv_cap_to_flow -> capacity_to_flow_conversion_factor(???) # Do we really need this?
• unit_conv_cap_to_flow -> capacity_to_flow_conversion_factor(???) # Do we really need this?

User constraint related, I guess we need to refer to the variable names for clarity?

• connection_flow_coefficient -> coefficient_for_connection_flow
• connections_invested_available_coefficient -> coefficient_for_connections_invested_available
• connections_invested_coefficient -> coefficient_for_connections_invested
• demand_coefficient -> coefficient_for_demand
• node_state_coefficient -> coefficient_for_node_state # This is for the node__user_constraint coefficient...
• storages_invested_available_coefficient -> coefficient_for_storages_invested_available
• storages_invested_coefficient -> coefficient_for_storages_invested
• unit_flow_coefficient -> coefficient_for_unit_flow
• units_invested_available_coefficient -> coefficient_for_units_invested_available # Rename these user_constraint coefficients to better group?
• units_invested_coefficient -> coefficient_for_units_invested
• units_on_coefficient -> coefficient_for_units_on
• units_started_up_coefficient -> coefficient_for_units_started_up # Rename these user_constraint coefficients to better group?

Requires some class changes:

• curtailment_cost -> unit__node | curtailment_cost(?) # Point is to make this a per MW capacity, making it flow-specific?
• fom_cost -> unit__node | fixed_annual_cost # Point is to make this a per MW capacity, making it flow-specific?
• shut_down_cost -> unit__node | shutdown_cost(??) # Point is to make this a per MW capacity, making it flow-specific?
• start_up_cost -> unit__node | startup_cost (??) # Point is to make this a per MW capacity, making it flow-specific?
• unit_investment_cost -> unit__node | investment_cost(??) # Point is to make this a per MW capacity, making it flow-specific?
• units_on_cost -> unit__node | online_cost(??) # Point is to make this a per MW capacity, making it flow-specific?

Connection rework related (see later comment)

• fix_ratio_out_in_connection_flow -> constraint_equality_flow_ratio(?) # Reuse unit flow ratio params.
• max_ratio_out_in_connection_flow -> constraint_greater_than_flow_ratio(?) # Align with unit ratio params
• min_ratio_out_in_connection_flow -> constraint_less_than_flow_ratio(?) # Align with unit ratio params

Not sure what these are or if they should be renamed for clarity

• fix_nonspin_units_shut_down -> ???
• fix_nonspin_units_started_up -> ???
• fixed_pressure_constant_0 -> ???
• fixed_pressure_constant_1 -> ???
• initial_nonspin_units_shut_down -> ???
• initial_nonspin_units_started_up -> ???
• is_non_spinning -> nonspinning_activate(??) # Align terminology?
• is_renewable -> renewable_activate(???) # Align terminology?
• mp_min_res_gen_to_demand_ratio -> ???
• mp_min_res_gen_to_demand_ratio_slack_penalty -> ???
• node_opf_type -> ???
• ordered_unit_flow_op -> ???
• tax_in_unit_flow -> ??? # Why are these node properties?
• tax_net_unit_flow -> ??? # Why are these node properties?
• tax_out_unit_flow -> ??? # Why are these node properties?
• units_on_non_anticipativity_margin -> ???
• units_on_non_anticipativity_time -> ???

[Tasqu]

Ok, we'll likely stick with these changes for now. The rest would require discussions (i.e. take too long), be of dubious use, or require larger redefining what a connection is (i.e. take too long).

The rest of the changes can be implemented later if so desired (and if we find the time). For now, it's more pressing to get at least some changes done, tested, and hopefully merged.

[DillonJ]

Hi @nhniina @Tasqu

Not sure if this is the best place to discuss, but it's the first potentially relevant place I found.

I tried running our SpineH2IRL model in the SpineOpt datastructure_update branch and SpineInterface #139_attempt4 branch

This is how I got on:

Running SpineOpt...
WARNING: using JuMP.Parameter in module SpineInterface conflicts with an existing identifier.
[SpineOpt version 0.11.0 (git hash: f22bcade0ad554ad0ac359480db7357a555a0ddb)]
[SpineInterface version 0.16.1 (git hash: df1bb5088b6576e3aab6c81fb84017dc53498185)]
Execution started at 2025-12-16T10:24:39.329
Reading input data from sqlite:///C:\Workspace\Spine\Projects\Pathways_Model_Test\.spinetoolbox\items\pathway_scenario_db\H2_IRL_AllScen.sqlite?spinedbfilter=cfg%3Ascenario%3AHN...
Upgrading data structure to the latest version... 
Replacing `use_economic_representation` and `use_milestone_years` 
	by `multiyear_economic_discounting`...
Parameter value list `multiyear_economic_discounting_value_list` already exists
Running several migrations to update the class structure and to rename, modify and move parameters...
Creating superclasses...
Note: Check entity alternatives in classes related to the unit_flow superclass...
Update ordering of multidimensional classes...
there's already a parameter_definition with {'entity_class_name': 'unit_flow__user_constraint', 'name': 'graph_view_position'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__user_constraint', 'name': 'unit_flow_coefficient'}
Renaming parameters...
there's already a parameter_definition with {'entity_class_name': 'unit', 'name': 'out_of_service_count_fix'}
Renaming classes...
Moving parameters to multidimensional classes...
non-existent elements in byname ('H2_ELECTRO_PB (1)', 'H2_DUB', 'EL_PB_220_4462', 'H2_ELECTRO_PB (1)') for class unit_flow__unit_flow
ERROR: LoadError:
Stacktrace:
  [1] error()
    @ Base .\error.jl:44
  [2] check_run_request_return_value(value_to_be_checked::Vector{Any}, log_level::Int64, print_value::Bool)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\versions\major_upgrade_to_17.jl:375
  [3] check_run_request_return_value(value_to_be_checked::Vector{Any}, log_level::Int64)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\versions\major_upgrade_to_17.jl:371
  [4] move_parameter_to_multidimensional_class(db_url::URIs.URI, old_class_name::String, old_par_name::String, new_class_name::String, new_par_name::String, mapping::Vector{Int64}, log_level::Int64)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\versions\major_upgrade_to_17.jl:732
  [5] move_parameters_to_multidimensional_classes(db_url::URIs.URI, parameters_to_multidimensional_classes::Vector{Tuple{Tuple{String, String}, Tuple{String, String, Vector{Int64}}}}, log_level::Int64)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\versions\major_upgrade_to_17.jl:680
  [6] major_upgrade_to_17(db_url::URIs.URI, log_level::Int64)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\versions\major_upgrade_to_17.jl:355
  [7] _run_migration(url::URIs.URI, version::Int64, log_level::Int64)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\migration.jl:119
  [8] (::SpineOpt.var"#593#594"{Int64})(clean_url::URIs.URI)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\migration.jl:109
  [9] (::SpineInterface.var"#214#215"{SpineOpt.var"#593#594"{Int64}})(db::URIs.URI)
    @ SpineInterface c:\workspace\spine\SpineInterface\src\api\db.jl:776
 [10] _db(f::SpineInterface.var"#214#215"{SpineOpt.var"#593#594"{Int64}}, url::String; upgrade::Bool)
    @ SpineInterface c:\workspace\spine\SpineInterface\src\api\db.jl:817
 [11] _db
    @ c:\workspace\spine\SpineInterface\src\api\db.jl:814 [inlined]
 [12] without_filters
    @ c:\workspace\spine\SpineInterface\src\api\db.jl:771 [inlined]
 [13] run_migrations
    @ c:\workspace\spine\SpineOpt\src\data_structure\migration.jl:108 [inlined]
 [14] #_do_upgrade_db#85
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:333 [inlined]
 [15] _do_upgrade_db
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:331 [inlined]
 [16] _check_version(url_in::String; log_level::Int64, upgrade::Bool)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\run_spineopt.jl:325
 [17] _check_version
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:315 [inlined]
 [18] prepare_spineopt(url_in::String; log_level::Int64, upgrade::Bool, filters::Dict{String, String}, templates::Tuple{}, mip_solver::Nothing, lp_solver::Nothing, use_direct_model::Bool, use_model_names::Bool, add_bridges::Bool)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\run_spineopt.jl:264
 [19] prepare_spineopt
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:251 [inlined]
 [20] _run_spineopt(f::SpineOpt.var"#64#65", url_in::String, url_out::String; upgrade::Bool, filters::Dict{String, String}, templates::Tuple{}, mip_solver::Nothing, lp_solver::Nothing, use_direct_model::Bool, use_model_names::Bool, add_bridges::Bool, log_level::Int64, alternative::String, kwargs::@Kwargs{optimize::Bool, update_names::Bool, write_as_roll::Int64, resume_file_path::Nothing})
    @ SpineOpt c:\workspace\spine\SpineOpt\src\run_spineopt.jl:149
 [21] #67
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:104 [inlined]
 [22] _log_to_file
    @ c:\workspace\spine\SpineOpt\src\util\misc.jl:271 [inlined]
 [23] #run_spineopt#66
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:103 [inlined]
 [24] #run_spineopt#63
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:67 [inlined]
 [25] top-level scope
    @ C:\Users\xxxx\.spinetoolbox\work\run_spineopt_local__bad54b2f84fa483d8c41307f472f53e3__toolbox\run_spineopt.jl:3
 [26] include(fname::String)
    @ Main .\sysimg.jl:38
 [27] top-level scope
    @ none:1
in expression starting at C:\Users\xxxx\.spinetoolbox\work\run_spineopt_local__bad54b2f84fa483d8c41307f472f53e3__toolbox\run_spineopt.jl:3
julia> 

[DillonJ]

Hi @Tasqu

Nodes:
'H2_DUB'
'EL_PB_220_4462'

Unit:
'H2_ELECTRO_PB (1)'

[Tasqu]

@DillonJ Ok, I showed this to @nhniina and she recommended to check the database. The migration is likely trying to convert a unit__node__node parameter into a unit_flow__unit_flow parameter, but runs into a non-existent unit_flow. E.g. if a ('H2_ELECTRO_PB (1)', 'H2_DUB', 'EL_PB_220_4462') fix_ratio_... parameter exists without BOTH the corresponding unit_to_node and unit_from_node being defined.

It's also possible our migration script has it's indices mixed, though, so that it's looking for a non-existent unit flow. I'll have to be very careful with those when implementing unit tests.

[DillonJ]

[Tasqu]

Yeah, it's complaining about the H2_DUB | EL_PB_220_4462. There's no flow to that. Only to ..._4472

[DillonJ]

Indeed! I'll remove that and try again!

[DillonJ]

We're getting further:

2-element Vector{String}:
 "http://127.0.0.1:1742"
 "http://127.0.0.1:1744"
WARNING: using JuMP.parameter_value in module SpineInterface conflicts with an existing identifier.

Running SpineOpt...
WARNING: using JuMP.Parameter in module SpineInterface conflicts with an existing identifier.
[SpineOpt version 0.11.0 (git hash: f22bcade0ad554ad0ac359480db7357a555a0ddb)]
[SpineInterface version 0.16.1 (git hash: df1bb5088b6576e3aab6c81fb84017dc53498185)]
Execution started at 2025-12-16T12:40:11.164
Reading input data from sqlite:///C:\Workspace\Spine\Projects\Pathways_Model_Test\.spinetoolbox\items\pathway_scenario_db\H2_IRL_AllScen.sqlite?spinedbfilter=cfg%3Ascenario%3AHN...
Upgrading data structure to the latest version... 
Replacing `use_economic_representation` and `use_milestone_years` 
	by `multiyear_economic_discounting`...
Parameter value list `multiyear_economic_discounting_value_list` already exists
Running several migrations to update the class structure and to rename, modify and move parameters...
Creating superclasses...
Note: Check entity alternatives in classes related to the unit_flow superclass...
Update ordering of multidimensional classes...
there's already a parameter_definition with {'entity_class_name': 'unit_flow__user_constraint', 'name': 'graph_view_position'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__user_constraint', 'name': 'unit_flow_coefficient'}
Renaming parameters...
there's already a parameter_definition with {'entity_class_name': 'unit', 'name': 'out_of_service_count_fix'}
Renaming classes...
Moving parameters to multidimensional classes...
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_equality_flow_ratio'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_equality_flow_ratio'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_equality_flow_ratio'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_greater_than_flow_ratio'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_greater_than_flow_ratio'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_greater_than_flow_ratio'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_less_than_flow_ratio'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_less_than_flow_ratio'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_less_than_flow_ratio'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_equality_online_coefficient'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_equality_online_coefficient'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_equality_online_coefficient'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_greater_than_online_coefficient'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_greater_than_online_coefficient'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_greater_than_online_coefficient'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_less_than_online_coefficient'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_less_than_online_coefficient'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__unit_flow', 'name': 'constraint_less_than_online_coefficient'}
there's already a parameter_definition with {'entity_class_name': 'unit_flow__user_constraint', 'name': 'coefficient_for_unit_flow'}
Removing classes...
Renaming parameter value lists...
Renaming list values...
Merging variable type lists...
there's already a parameter_value_list with {'name': 'variable_type_list'}
Could not rename list unit_online_variable_type_list.
value integer of investment_variable_type2 for ('H2_GAL_SHAN',) is not in variable_type_list
ERROR: LoadError:
Stacktrace:
  [1] error()
    @ Base .\error.jl:44
  [2] check_run_request_return_value(value_to_be_checked::Vector{Any}, log_level::Int64, print_value::Bool)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\versions\major_upgrade_to_17.jl:375
  [3] check_run_request_return_value(value_to_be_checked::Vector{Any}, log_level::Int64)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\versions\major_upgrade_to_17.jl:371
  [4] merge_variable_type_lists(db_url::URIs.URI, log_level::Int64)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\versions\major_upgrade_to_17.jl:864
  [5] major_upgrade_to_17(db_url::URIs.URI, log_level::Int64)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\versions\major_upgrade_to_17.jl:363
  [6] _run_migration(url::URIs.URI, version::Int64, log_level::Int64)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\migration.jl:119
  [7] (::SpineOpt.var"#593#594"{Int64})(clean_url::URIs.URI)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\data_structure\migration.jl:109
  [8] (::SpineInterface.var"#214#215"{SpineOpt.var"#593#594"{Int64}})(db::URIs.URI)
    @ SpineInterface c:\workspace\spine\SpineInterface\src\api\db.jl:776
  [9] _db(f::SpineInterface.var"#214#215"{SpineOpt.var"#593#594"{Int64}}, url::String; upgrade::Bool)
    @ SpineInterface c:\workspace\spine\SpineInterface\src\api\db.jl:817
 [10] _db
    @ c:\workspace\spine\SpineInterface\src\api\db.jl:814 [inlined]
 [11] without_filters
    @ c:\workspace\spine\SpineInterface\src\api\db.jl:771 [inlined]
 [12] run_migrations
    @ c:\workspace\spine\SpineOpt\src\data_structure\migration.jl:108 [inlined]
 [13] #_do_upgrade_db#85
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:333 [inlined]
 [14] _do_upgrade_db
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:331 [inlined]
 [15] _check_version(url_in::String; log_level::Int64, upgrade::Bool)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\run_spineopt.jl:325
 [16] _check_version
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:315 [inlined]
 [17] prepare_spineopt(url_in::String; log_level::Int64, upgrade::Bool, filters::Dict{String, String}, templates::Tuple{}, mip_solver::Nothing, lp_solver::Nothing, use_direct_model::Bool, use_model_names::Bool, add_bridges::Bool)
    @ SpineOpt c:\workspace\spine\SpineOpt\src\run_spineopt.jl:264
 [18] prepare_spineopt
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:251 [inlined]
 [19] _run_spineopt(f::SpineOpt.var"#64#65", url_in::String, url_out::String; upgrade::Bool, filters::Dict{String, String}, templates::Tuple{}, mip_solver::Nothing, lp_solver::Nothing, use_direct_model::Bool, use_model_names::Bool, add_bridges::Bool, log_level::Int64, alternative::String, kwargs::@Kwargs{optimize::Bool, update_names::Bool, write_as_roll::Int64, resume_file_path::Nothing})
    @ SpineOpt c:\workspace\spine\SpineOpt\src\run_spineopt.jl:149
 [20] #67
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:104 [inlined]
 [21] _log_to_file
    @ c:\workspace\spine\SpineOpt\src\util\misc.jl:271 [inlined]
 [22] #run_spineopt#66
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:103 [inlined]
 [23] #run_spineopt#63
    @ c:\workspace\spine\SpineOpt\src\run_spineopt.jl:67 [inlined]
 [24] top-level scope
    @ C:\Users\xxxx\.spinetoolbox\work\run_spineopt_local__c33cab4ba3434c6a9191099d6b0d8068__toolbox\run_spineopt.jl:3
 [25] include(fname::String)
    @ Main .\sysimg.jl:38
 [26] top-level scope
    @ none:1
in expression starting at C:\Users\xxxx\.spinetoolbox\work\run_spineopt_local__c33cab4ba3434c6a9191099d6b0d8068__toolbox\run_spineopt.jl:3

[DillonJ]

This is the row in question:

[Tasqu]

Merging variable type lists...
there's already a parameter_value_list with {'name': 'variable_type_list'} # CHECK THIS!
Could not rename list unit_online_variable_type_list.
value integer of investment_variable_type2 for ('H2_GAL_SHAN',) is not in variable_type_list
ERROR: LoadError:
...

We might have ran into this in one example .json file earlier: For some reason, there was already a parameter value list by the name variable_type_list in the data, which was preventing the rest of the migration from going through. As far as we know, this list hasn't been used for ages for anything, so removing it should not cause issues, BUT BE CAREFUL.

In any case, I'll have to check if we can handle this automatically in the migration. (assuming this is the cause)

[DillonJ]

So I have compared the solution with latest master and the datastructure update branch and there are some differences:

<style> </style>

Technology	Investment Type	Datastructure Update Investment Units	Master Investment Units
2hBatt	Storage	0	0
6hBatt	Storage	0	0
H2	Connection	14	7
H2CCGT	Unit	12	12
H2ELECTRO	Unit	108	108
H2OCGT	Unit	37	37
Solar	Unit	9000	9000
Wind-H2	Unit	0	0
Wind-off	Unit	11100	11271
Wind-on	Unit	6500	6500
H2Salt	Storage	2836752	2861962

The difference in storage and offshore wind suggests some difference between master and the datastructure update branch regarding the treatment of storage? Did we include something to limit intraday storage level in rep periods?

Next step is to look at the model formulation

Here are the solution stats differences:

Datastructure Update:
build_time (model = "ie_transmission_model",) 3 minutes, 53 seconds, 898 milliseconds
elapsed_time 8 minutes, 23 seconds, 76 milliseconds
solve_time (model = "ie_transmission_model",) 3 minutes, 21 seconds, 319 milliseconds

Master:
build_time (model = "ie_transmission_model",) 3 minutes, 38 seconds, 428 milliseconds
elapsed_time 10 minutes, 4 seconds, 901 milliseconds
solve_time (model = "ie_transmission_model",) 5 minutes, 35 seconds, 622 milliseconds

[Tasqu]

I don't think we've made any core alterations to storages, only a bunch of parameter renamings, which shouldn't really affect the outcome (assuming we've renamed things everywhere, but we should see errors if that wasn't the case). However, there was a bug with unit_flow__user_constraint and unit_flow__investment_group not migrating correctly that we noticed and fixed while working on unit tests (parameters were only migrated for "one direction", this fix isn't yet pushed). If your model was relying on those, they might be a potential cause.

The only storage-related "change" I can think of at the moment is changing the storage_investment_variable_type parameter value lists. Maybe check those get mapped correctly?

[Tasqu]

Well, I just realized another quite serious bug that could affect storages depending on how they are modelled: back-and-forth flow limits (and similar) don't work properly. E.g. if you have flows both unit1 -> node1 and node1 -> unit1 with different capacities, you get one value for both, but not sure which one.

This goes all the way down to SpineInferface, as EntityClass and Parameter calls don't care about the keyword argument order, but for these kinds of situations they would need to. I was aware of this, but I don't remember the reason for not "fixing" it before. Hopefully I just failed to deem it necessary and avoided making changes with potential consequences for users.

If we're extremely unlucky, this might completely prevent the planned structural changes. I had to implement the superclass functionality in SpineInterface in a pretty hacky way to preserve code optimizations by @manuelma. If this issue reaches that level, I'm not sure if I'm capable of fixing it in any reasonable amount of time. Guess we'll see 😅

[Tasqu]

Crisis averted. However, I had to change a pretty fundamental property of SpineInterface in the process: The order of EntityClass related keyword arguments now matters for RelationshipClass and Parameter calls. Fortunately, there were only a handful of places in SpineOpt where the entity class kwargs were not already in the correct order, but this still potentially breaks other code that relies on SpineInterface. I also had to rewrite some pretty eldritch code by @manuelma, which could potentially have an adverse impact on performance.

Latest SpineOpt datastructure_update and SpineInterface #139_attempt4 branches are required for these changes (along with migration unit tests).