Mapping values to node_state_min and node_state_cap

[SeanByrne08]

Describe your problem

Hi,
I am encountering an issue when I try to map time series to node_state_min and node_state_cap parameters.

I am trying to vary the EV battery size for different stochastic scenarios. The node_state_cap is a fixed value e.g. 60.0 kWh, the node_state_min is represented by a time-series where the required minimum SOC may vary throughout the day.

The model runs for multiple stochastic scenarios correctly when the node state min/cap remain the same across the scenarios. When I try to map values for different stochastic scenarios I encounter the following error.

SpineOpt console message:
2-element Vector{String}:
"http://127.0.0.1:64086"
"http://127.0.0.1:64090"

Execution started at 2024-07-06T11:52:47.513
Preparing SpineOpt for sqlite:///C:\Users\Sean\Downloads\V2H Model Example.spinetoolbox\items\Input_Data\Input data - Copy.sqlite?spinedbfilter=cfg%3Atool%3Aobject_activity_control...
Initializing data structure from db... 5.015164 seconds (1.83 M allocations: 328.023 MiB, 1.58% gc time)
Preprocessing data structure... 0.016777 seconds (14.07 k allocations: 1.204 MiB)
Checking data structure... 0.000742 seconds (6.41 k allocations: 618.758 KiB)
Running SpineOpt...
Creating temporal structure... 0.013090 seconds (27.02 k allocations: 1.776 MiB)
Creating stochastic structure... 0.010518 seconds (8.86 k allocations: 767.500 KiB)
Adding variables...

• [variable_units_available] 0.081974 seconds (413.65 k allocations: 47.301 MiB, 43.04% gc time)
• [variable_units_on] 0.068192 seconds (528.66 k allocations: 55.745 MiB)
• [variable_units_started_up] 0.053987 seconds (407.53 k allocations: 46.515 MiB)
• [variable_units_shut_down] 0.080405 seconds (407.53 k allocations: 46.363 MiB, 39.17% gc time)
• [variable_unit_flow] 0.171008 seconds (1.24 M allocations: 141.143 MiB, 18.98% gc time)
• [variable_unit_flow_op] 0.000379 seconds (460 allocations: 43.781 KiB)
• [variable_unit_flow_op_active] 0.000131 seconds (242 allocations: 25.609 KiB)
• [variable_connection_flow] 0.000255 seconds (157 allocations: 13.000 KiB)
• [variable_connection_intact_flow] 0.000134 seconds (154 allocations: 12.859 KiB)
• [variable_connections_invested] 0.000104 seconds (115 allocations: 11.453 KiB)
• [variable_connections_invested_available] 0.000175 seconds (121 allocations: 14.000 KiB)
• [variable_connections_decommissioned] 0.000043 seconds (57 allocations: 6.422 KiB)
• [variable_storages_invested] 0.000148 seconds (115 allocations: 11.453 KiB)
• [variable_storages_invested_available] 0.000074 seconds (115 allocations: 11.500 KiB)
• [variable_storages_decommissioned] 0.000022 seconds (57 allocations: 6.422 KiB)
  ERROR: LoadError: MethodError: no method matching set_lower_bound(::JuMP.VariableRef, ::SpineInterface.Map{Symbol, SpineInterface.TimeSeries{Float64}})

Closest candidates are:
set_lower_bound(::JuMP.GenericVariableRef, ::Number)
@ JuMP C:\Users\Sean.julia\packages\JuMP\jZvaU\src\variables.jl:660
set_lower_bound(::JuMP.VariableRef, ::SpineInterface.Call)
@ SpineInterface C:\Users\Sean.julia\packages\SpineInterface\aLG6G\src\update_model.jl:120

Stacktrace:
[1] _set_lower_bound(var::JuMP.VariableRef, lb::SpineInterface.Map{Symbol, SpineInterface.TimeSeries{Float64}})
@ SpineInterface C:\Users\Sean.julia\packages\SpineInterface\aLG6G\src\update_model.jl:133
[2] set_lower_bound(var::JuMP.VariableRef, call::SpineInterface.Call)
@ SpineInterface C:\Users\Sean.julia\packages\SpineInterface\aLG6G\src\update_model.jl:122
[3] _variable(m::JuMP.Model, name::Symbol, ind::NamedTuple{(:node, :stochastic_scenario, :t), Tuple{SpineInterface.Object, SpineInterface.Object, SpineInterface.TimeSlice}}, bin::Nothing, int::Nothing, lb::SpineInterface.Parameter, ub::Nothing, fix_value::SpineInterface.Parameter, internal_fix_value::Nothing, replacement_value::Nothing)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\variables\variable_common.jl:129
[4] #1015
@ .\none:0 [inlined]
[5] iterate
@ .\generator.jl:47 [inlined]
[6] _all(f::Base.var"#372#374", itr::Base.Generator{Vector{NamedTuple{(:node, :stochastic_scenario, :t), Tuple{SpineInterface.Object, SpineInterface.Object, SpineInterface.TimeSlice}}}, SpineOpt.var"#1015#1016"{Nothing, Nothing, SpineInterface.Parameter, Nothing, SpineInterface.Parameter, Nothing, Nothing, JuMP.Model, Symbol}}, #unused#::Colon)
@ Base .\reduce.jl:1292
[7] all
@ .\reduce.jl:1278 [inlined]
[8] Dict(kv::Base.Generator{Vector{NamedTuple{(:node, :stochastic_scenario, :t), Tuple{SpineInterface.Object, SpineInterface.Object, SpineInterface.TimeSlice}}}, SpineOpt.var"#1015#1016"{Nothing, Nothing, SpineInterface.Parameter, Nothing, SpineInterface.Parameter, Nothing, Nothing, JuMP.Model, Symbol}})
@ Base .\dict.jl:111
[9] add_variable!(m::JuMP.Model, name::Symbol, indices::typeof(node_state_indices); bin::Nothing, int::Nothing, lb::SpineInterface.Parameter, ub::Nothing, initial_value::SpineInterface.Parameter, fix_value::SpineInterface.Parameter, internal_fix_value::Nothing, replacement_value::Nothing, non_anticipativity_time::Nothing, non_anticipativity_margin::Nothing)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\variables\variable_common.jl:56
[10] add_variable_node_state!(m::JuMP.Model)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\variables\variable_node_state.jl:43
[11] macro expansion
@ .\timing.jl:273 [inlined]
[12] macro expansion
@ C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\util\misc.jl:51 [inlined]
[13] _add_variables!(m::JuMP.Model; add_user_variables::SpineOpt.var"#54#64", log_level::Int64)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt_standard.jl:141
[14] macro expansion
@ .\timing.jl:273 [inlined]
[15] macro expansion
@ C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\util\misc.jl:51 [inlined]
[16] init_model!(m::JuMP.Model; add_user_variables::Function, add_constraints::Function, alternative_objective::Function, log_level::Int64)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt_standard.jl:91
[17] init_model!
@ C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt_standard.jl:88 [inlined]
[18] rerun_spineopt_standard!(m::JuMP.Model, url_out::String; add_user_variables::Function, add_constraints::Function, alternative_objective::Function, log_level::Int64, optimize::Bool, update_names::Bool, alternative::String, write_as_roll::Int64, resume_file_path::Nothing)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt_standard.jl:37
[19] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Any, NTuple{9, Symbol}, NamedTuple{(:add_user_variables, :add_constraints, :log_level, :optimize, :update_names, :alternative, :write_as_roll, :resume_file_path, :alternative_objective), Tuple{SpineOpt.var"#54#64", SpineOpt.var"#55#65", Int64, Bool, Bool, String, Int64, Nothing, SpineOpt.var"#84#90"}}})
@ Base .\essentials.jl:818
[20] rerun_spineopt(url_out::String; mip_solver::Nothing, lp_solver::Nothing, add_user_variables::Function, add_constraints::Function, alternative_objective::Function, log_level::Int64, optimize::Bool, update_names::Bool, alternative::String, write_as_roll::Int64, resume_file_path::Nothing, use_direct_model::Bool)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt.jl:274
[21] _run_spineopt(url_in::String, url_out::String; upgrade::Bool, mip_solver::Nothing, lp_solver::Nothing, add_user_variables::Function, add_constraints::Function, log_level::Int64, optimize::Bool, update_names::Bool, alternative::String, write_as_roll::Int64, use_direct_model::Bool, filters::Dict{String, String}, resume_file_path::Nothing)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt.jl:187
[22] _run_spineopt
@ C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt.jl:167 [inlined]
[23] #run_spineopt#47
@ C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt.jl:98 [inlined]
[24] run_spineopt(url_in::String, url_out::String)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt.jl:79
[25] top-level scope
@ C:\Users\Sean.spinetoolbox\work\run_spineopt__492da65db92f4e7dad25048aaa801513__toolbox\run_spineopt.jl:3
[26] include(fname::String)
@ Base.MainInclude .\client.jl:478
[27] top-level scope
@ none:1
in expression starting at C:\Users\Sean.spinetoolbox\work\run_spineopt__492da65db92f4e7dad25048aaa801513__toolbox\run_spineopt.jl:3

caused by: MethodError: no method matching set_lower_bound(::JuMP.VariableRef, ::SpineInterface.Map{Symbol, SpineInterface.TimeSeries{Float64}})

Closest candidates are:
set_lower_bound(::JuMP.GenericVariableRef, ::Number)
@ JuMP C:\Users\Sean.julia\packages\JuMP\jZvaU\src\variables.jl:660
set_lower_bound(::JuMP.VariableRef, ::SpineInterface.Call)
@ SpineInterface C:\Users\Sean.julia\packages\SpineInterface\aLG6G\src\update_model.jl:120

Stacktrace:
[1] _set_lower_bound(var::JuMP.VariableRef, lb::SpineInterface.Map{Symbol, SpineInterface.TimeSeries{Float64}})
@ SpineInterface C:\Users\Sean.julia\packages\SpineInterface\aLG6G\src\update_model.jl:133
[2] set_lower_bound(var::JuMP.VariableRef, call::SpineInterface.Call)
@ SpineInterface C:\Users\Sean.julia\packages\SpineInterface\aLG6G\src\update_model.jl:122
[3] _variable(m::JuMP.Model, name::Symbol, ind::NamedTuple{(:node, :stochastic_scenario, :t), Tuple{SpineInterface.Object, SpineInterface.Object, SpineInterface.TimeSlice}}, bin::Nothing, int::Nothing, lb::SpineInterface.Parameter, ub::Nothing, fix_value::SpineInterface.Parameter, internal_fix_value::Nothing, replacement_value::Nothing)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\variables\variable_common.jl:129
[4] #1015
@ .\none:0 [inlined]
[5] iterate
@ .\generator.jl:47 [inlined]
[6] Dict{NamedTuple{(:node, :stochastic_scenario, :t), Tuple{SpineInterface.Object, SpineInterface.Object, SpineInterface.TimeSlice}}, JuMP.VariableRef}(kv::Base.Generator{Vector{NamedTuple{(:node, :stochastic_scenario, :t), Tuple{SpineInterface.Object, SpineInterface.Object, SpineInterface.TimeSlice}}}, SpineOpt.var"#1015#1016"{Nothing, Nothing, SpineInterface.Parameter, Nothing, SpineInterface.Parameter, Nothing, Nothing, JuMP.Model, Symbol}})
@ Base .\dict.jl:85
[7] dict_with_eltype
@ .\abstractdict.jl:581 [inlined]
[8] dict_with_eltype
@ .\abstractdict.jl:588 [inlined]
[9] Dict(kv::Base.Generator{Vector{NamedTuple{(:node, :stochastic_scenario, :t), Tuple{SpineInterface.Object, SpineInterface.Object, SpineInterface.TimeSlice}}}, SpineOpt.var"#1015#1016"{Nothing, Nothing, SpineInterface.Parameter, Nothing, SpineInterface.Parameter, Nothing, Nothing, JuMP.Model, Symbol}})
@ Base .\dict.jl:109
[10] add_variable!(m::JuMP.Model, name::Symbol, indices::typeof(node_state_indices); bin::Nothing, int::Nothing, lb::SpineInterface.Parameter, ub::Nothing, initial_value::SpineInterface.Parameter, fix_value::SpineInterface.Parameter, internal_fix_value::Nothing, replacement_value::Nothing, non_anticipativity_time::Nothing, non_anticipativity_margin::Nothing)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\variables\variable_common.jl:56
[11] add_variable_node_state!(m::JuMP.Model)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\variables\variable_node_state.jl:43
[12] macro expansion
@ .\timing.jl:273 [inlined]
[13] macro expansion
@ C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\util\misc.jl:51 [inlined]
[14] _add_variables!(m::JuMP.Model; add_user_variables::SpineOpt.var"#54#64", log_level::Int64)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt_standard.jl:141
[15] macro expansion
@ .\timing.jl:273 [inlined]
[16] macro expansion
@ C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\util\misc.jl:51 [inlined]
[17] init_model!(m::JuMP.Model; add_user_variables::Function, add_constraints::Function, alternative_objective::Function, log_level::Int64)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt_standard.jl:91
[18] init_model!
@ C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt_standard.jl:88 [inlined]
[19] rerun_spineopt_standard!(m::JuMP.Model, url_out::String; add_user_variables::Function, add_constraints::Function, alternative_objective::Function, log_level::Int64, optimize::Bool, update_names::Bool, alternative::String, write_as_roll::Int64, resume_file_path::Nothing)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt_standard.jl:37
[20] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Any, NTuple{9, Symbol}, NamedTuple{(:add_user_variables, :add_constraints, :log_level, :optimize, :update_names, :alternative, :write_as_roll, :resume_file_path, :alternative_objective), Tuple{SpineOpt.var"#54#64", SpineOpt.var"#55#65", Int64, Bool, Bool, String, Int64, Nothing, SpineOpt.var"#84#90"}}})
@ Base .\essentials.jl:818
[21] rerun_spineopt(url_out::String; mip_solver::Nothing, lp_solver::Nothing, add_user_variables::Function, add_constraints::Function, alternative_objective::Function, log_level::Int64, optimize::Bool, update_names::Bool, alternative::String, write_as_roll::Int64, resume_file_path::Nothing, use_direct_model::Bool)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt.jl:274
[22] _run_spineopt(url_in::String, url_out::String; upgrade::Bool, mip_solver::Nothing, lp_solver::Nothing, add_user_variables::Function, add_constraints::Function, log_level::Int64, optimize::Bool, update_names::Bool, alternative::String, write_as_roll::Int64, use_direct_model::Bool, filters::Dict{String, String}, resume_file_path::Nothing)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt.jl:187
[23] _run_spineopt
@ C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt.jl:167 [inlined]
[24] #run_spineopt#47
@ C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt.jl:98 [inlined]
[25] run_spineopt(url_in::String, url_out::String)
@ SpineOpt C:\Users\Sean.julia\packages\SpineOpt\2Cdsi\src\run_spineopt.jl:79
[26] top-level scope
@ C:\Users\Sean.spinetoolbox\work\run_spineopt__492da65db92f4e7dad25048aaa801513__toolbox\run_spineopt.jl:3
[27] include(fname::String)
@ Base.MainInclude .\client.jl:478
[28] top-level scope
@ none:1

Attach your model as a JSON file (encouraged)

project.json

What versions are you using?

Julia: 1.9.2
spinetoolbox 0.7.0
spinedb_api 0.30.0
spine_engine 0.23.0
spine_items 0.21.0
Python 3.9.13
SpineOpt 0.1.5

[tarskul]

Your 'project.json' is the json for Spine Toolbox, not the model itself so it is not possible for me to reproduce the error (although I understand it may be difficult to disclose, perhaps make a minimal working example that reproduces the error?). However, your problem description seems sufficiently detailed at first glance.

To me this seems like a bug. It has happened before that there was a hole in the stochastic structure.

Thanks for reporting!

I'm not sure whether we can resolve this in the short term though...

[tarskul]

Maybe I should have mentioned that I'm using version 0.8 of both spine toolbox and spineopt. From your post I see that you have an older version, i.e. 0.7 for spine toolbox and 0.1.5 for spineopt. 0.8 is not backwards compatible with 0.7 due to a different database structure under the hood.

I'm wondering whether I should install an older version to try it there, or to simply ask you to upgrade your installation. Do you have a lot of old projects? And/or, more importantly, do you have many specific scripts where you hardcode the data into the old database structure?

[SeanByrne08]

I do have a lot of projects developed on this version and my model has numerous scripts which generate data and import it into the spine db. What impact will updating my versions of spine toolbox and spineopt have on these projects?

Would you be available for a quick call where I could share my screen on explain my issue?

[tarskul]

It is possible to have multiple versions of spine tools on the same system, so in principle the impact should be limited. It depends a bit on how you configured your system. As for the scripts, it may be some work to convert them all.

Though maybe we should check whether it actually is an issue in 0.7 or simply a mistake somewhere. Don't know whether I have time to check that though... Also, if it is an issue with 0.7, it is unlikely to be patched in 0.7 as we are focusing a lot on 0.8 at the moment.

Anyway, next week tuesday at 3pm Brussels time, there is the spine tools user meeting. Is it ok if we take the discussion there?

[SeanByrne08]

Ok, I am going to hold off updating for the moment, I will try and see if I have made a mistake somewhere. I will also attend the meeting next Thursday, thanks for all your help!

Sean

[tarskul]

* Next Tuesday

;)