Defining constraint for renewable generation with availability factors

mbah · March 20, 2022, 12:30am

Hello,

I am trying to code up a straightforward constraint for renewable generation (wind, solar etc…) without much success. As an example, the general formulation for a wind generation constraint is as follows:

My availability factor data file is structured by node (i.e. zone) over time

availability_factor

The wind installed capacity paramter is accessed by calling the g_max coloumn in this file

wind plant

A snippet of the code i’ve tried so far that is generating a key error

@constraint(dispatch_model, 
  WNDGeneration[t in T, n in N, w in WND], 
  WIND[t,n,w] <= data[:wind_cf][n][t] * data[:plants_aggregation][:g_max][w][n]  
)

ERROR: LoadError: KeyError: key (1, "ZONE_A", "ZONE_A_WIND") not found
Stacktrace:

Any ideas/solutions to this problem would be greatly appreciated.

Thanks

odow · March 20, 2022, 7:17pm

Please provide a reproducible example.

Do you have the full stacktrace of the error? My guess is that your WIND variable isn’t defined over the same indices as the WNDGeneration constraint.

mbah · March 20, 2022, 11:21pm

Thanks @odow. I will work on a reproducible example. In the mean time, this is the full stacktrace error message:

ERROR: KeyError: key (1, "ZONE_A", "ZONE_A_WIND") not found
Stacktrace:
  [1] to_index(A::JuMP.Containers.DenseAxisArray{VariableRef, 2, Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}}, Tuple{JuMP.Containers._AxisLookup{Dict{Int64, Int64}}, JuMP.Containers._AxisLookup{Dict{String, Int64}}}}, idx::Tuple{Int64, String, String})
    @ JuMP.Containers C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\DenseAxisArray.jl:257
  [2] getindex(::JuMP.Containers.DenseAxisArray{VariableRef, 2, Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}}, Tuple{JuMP.Containers._AxisLookup{Dict{Int64, Int64}}, JuMP.Containers._AxisLookup{Dict{String, Int64}}}}, ::Int64, ::String, ::String)
    @ JuMP.Containers C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\DenseAxisArray.jl:266
  [3] macro expansion
    @ C:\Users\mbahe\.julia\packages\MutableArithmetics\8xkW3\src\rewrite.jl:279 [inlined]
  [4] macro expansion
    @ C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\macros.jl:676 [inlined]
  [5] (::var"#44#45")(t::Int64, n::String, w::String)
    @ Main C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\macro.jl:194
  [6] #38
    @ C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\container.jl:105 [inlined]
  [7] iterate
    @ .\generator.jl:47 [inlined]
  [8] collect(itr::Base.Generator{JuMP.Containers.VectorizedProductIterator{Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}, WeakRefStrings.StringVector{String}}}, JuMP.Containers.var"#38#39"{var"#44#45"}})
    @ Base .\array.jl:678
  [9] map(f::Function, A::JuMP.Containers.VectorizedProductIterator{Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}, WeakRefStrings.StringVector{String}}})
    @ Base .\abstractarray.jl:2323
 [10] container(f::Function, indices::JuMP.Containers.VectorizedProductIterator{Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}, WeakRefStrings.StringVector{String}}}, #unused#::Type{JuMP.Containers.DenseAxisArray})
    @ JuMP.Containers C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\container.jl:105
 [11] container(f::Function, indices::JuMP.Containers.VectorizedProductIterator{Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}, WeakRefStrings.StringVector{String}}})
    @ JuMP.Containers C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\container.jl:66
 [12] macro expansion
    @ C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\macros.jl:142 [inlined]
 [13] top-level scope

odow · March 21, 2022, 12:08am

How did you define WIND? It looks like it has two dimensions, but you’re indexing over three.

jd-foster · March 21, 2022, 12:42am

As @odow said, the definition of the variable WIND is the issue, not your constraint declaration (shown above).
MWE:

using JuMP
model = JuMP.Model()
T = [1, 2, 3]
N = ["ZONE_A", "ZONE_B", "ZONE_C"]
WND = ["ZONE_A_WIND", "ZONE_B_WIND"]
@variable(model, WIND[N, WND]) ## THIS needs to be:
# @variable(model, WIND[T, N, WND])
@constraint(model, WNDGeneration[t in T, n in N, w in WND], WIND[t,n,w] <= 1.0)

mbah · March 22, 2022, 3:52pm

Thank you @odow and @jd-foster for the previous answers. I am trying to code up a relatively simple dispatch model. Since this is a network model, ideally the results i expect is dispatch of conventionals and output of renewables (wind, solar, etc) across the zones and over time. As you pointed out, my issue seems to be with indexing, but i’m not sure where exactly.

using JuMP, Clp, DataFrames, CSV 
using Base: Symbol

#Define sets
P= plants.index
N = nodes.index
N = nodes.index
T = demand_2018.index
L= lines.index

#Specifying all plant types

C = filter(r -> any(occursin.(["conventional"], r.plant_type)), readDataDFs[:plants]).index
ROR = filter(r -> any(occursin.(["ROR"], r.plant_type)), readDataDFs[:plants]).index #Run-of-river
SOL = filter(r -> any(occursin.(["solar"], r.plant_type)), readDataDFs[:plants]).index  #solar
WND = filter(r -> any(occursin.(["wind"], r.plant_type)), readDataDFs[:plants]).index #wind

dispatch_model = Model(Clp.Optimizer)

#Output variables

@variables(dispatch_model, begin
  GEN[T, C] >=0 #Conventional plant generation
  FLOW[T, L]  #Line power flow 
  RUNOFRIVER[T,ROR] >=0 #Run-of-River
  SOLAR[T,SOL] >=0
  WIND[T,WND] >=0
  VoltageAngle[T, N] #Voltage angle
end) 

#Objective function
@objective(dispatch_model, 
  Min, 
  sum(sum(GEN[t, c]*data[:plants][:cost][c] for c in C) for t in T))

#conventional plant capacity constraint
@constraint(dispatch_model, 
CapConstraint[t in T, c in C], 
GEN[t,c] <= data[:plants][:g_max][c] 
) 

#I have issues with renewable generation constraints

#Run-of-river generation constraint
@constraint(dispatch_model, 
  RORGeneration_new[t in T, n in N, r in ROR], 
  RUNOFRIVER[t,n,r] <= data[:ror_cf][t] * data[:plants][:g_max][r][n]   
)

#Wind generation constraint
@constraint(dispatch_model, 
  WNDGeneration[t in T,  n in N, w in WND], 
  WIND[t,n,w] <= data[:wind_cf][n][t] * data[:plants][:g_max][w][n]  
)

#Solar generation constraint
@constraint(dispatch_model, 
  SOLARGeneration[t in T,  n in N, s in SOL], 
  SOLAR[t,n,s] <= data[:solar_cf][n][t] * data[:plants][:g_max][s][n]  
)

...

Initial error messsage for Wind

ERROR: KeyError: key (1, "ZONE_A", "ZONE_A_WIND") not found
Stacktrace:
  [1] to_index(A::JuMP.Containers.DenseAxisArray{VariableRef, 2, Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}}, Tuple{JuMP.Containers._AxisLookup{Dict{Int64, Int64}}, JuMP.Containers._AxisLookup{Dict{String, Int64}}}}, idx::Tuple{Int64, String, String})
    @ JuMP.Containers C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\DenseAxisArray.jl:257
  [2] getindex(::JuMP.Containers.DenseAxisArray{VariableRef, 2, Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}}, Tuple{JuMP.Containers._AxisLookup{Dict{Int64, Int64}}, JuMP.Containers._AxisLookup{Dict{String, Int64}}}}, ::Int64, ::String, ::String)
    @ JuMP.Containers C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\DenseAxisArray.jl:266
  [3] macro expansion
    @ C:\Users\mbahe\.julia\packages\MutableArithmetics\8xkW3\src\rewrite.jl:279 [inlined]
  [4] macro expansion
    @ C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\macros.jl:676 [inlined]
  [5] (::var"#44#45")(t::Int64, n::String, w::String)
    @ Main C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\macro.jl:194
  [6] #38
    @ C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\container.jl:105 [inlined]
  [7] iterate
    @ .\generator.jl:47 [inlined]
  [8] collect(itr::Base.Generator{JuMP.Containers.VectorizedProductIterator{Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}, WeakRefStrings.StringVector{String}}}, JuMP.Containers.var"#38#39"{var"#44#45"}})
    @ Base .\array.jl:678
  [9] map(f::Function, A::JuMP.Containers.VectorizedProductIterator{Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}, WeakRefStrings.StringVector{String}}})
    @ Base .\abstractarray.jl:2323
 [10] container(f::Function, indices::JuMP.Containers.VectorizedProductIterator{Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}, WeakRefStrings.StringVector{String}}}, #unused#::Type{JuMP.Containers.DenseAxisArray})
    @ JuMP.Containers C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\container.jl:105
 [11] container(f::Function, indices::JuMP.Containers.VectorizedProductIterator{Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}, WeakRefStrings.StringVector{String}}})
    @ JuMP.Containers C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\container.jl:66
 [12] macro expansion
    @ C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\macros.jl:142 [inlined]
 [13] top-level scope

I am aware that the variables are not defined the same as the constraints. I attempted to make them similar,

@variables(dispatch_model, begin
  GEN[T, C] >=0 #Conventional plant generation
  FLOW[T, L]  #Line power flow 
  RUNOFRIVER[T,N,ROR] >=0 #Run-of-River
  SOLAR[T,N,SOL] >=0
  WIND[T,N,WND] >=0
  VoltageAngle[T, N] #Voltage angle
end) 

#Wind generation constraint
@constraint(dispatch_model, 
  WNDGeneration[t in T,  n in N, w in WND], 
  WIND[t,n,w] <= data[:wind_cf][n][t] * data[:plants][:g_max][w][n]  
)

but I get the following error message (example of wind).

ERROR: KeyError: key "ZONE_A" not found
Stacktrace:
  [1] getindex(h::Dict{Symbol, Dict{Int64, V} where V}, key::String)
    @ Base .\dict.jl:482
  [2] macro expansion
    @ C:\Users\mbahe\.julia\packages\MutableArithmetics\8xkW3\src\rewrite.jl:279 [inlined]
  [3] macro expansion
    @ C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\macros.jl:676 [inlined]
  [4] (::var"#42#43")(t::Int64, n::String, w::String)
    @ Main C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\macro.jl:194
  [5] #38
    @ C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\container.jl:105 [inlined]
  [6] iterate
    @ .\generator.jl:47 [inlined]
  [7] collect(itr::Base.Generator{JuMP.Containers.VectorizedProductIterator{Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}, WeakRefStrings.StringVector{String}}}, JuMP.Containers.var"#38#39"{var"#42#43"}})
    @ Base .\array.jl:678
  [8] map(f::Function, A::JuMP.Containers.VectorizedProductIterator{Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}, WeakRefStrings.StringVector{String}}})
    @ Base .\abstractarray.jl:2323
  [9] container(f::Function, indices::JuMP.Containers.VectorizedProductIterator{Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}, WeakRefStrings.StringVector{String}}}, #unused#::Type{JuMP.Containers.DenseAxisArray})
    @ JuMP.Containers C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\container.jl:105
 [10] container(f::Function, indices::JuMP.Containers.VectorizedProductIterator{Tuple{Vector{Int64}, WeakRefStrings.StringVector{String}, WeakRefStrings.StringVector{String}}})
    @ JuMP.Containers C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\Containers\container.jl:66
 [11] macro expansion
    @ C:\Users\mbahe\.julia\packages\JuMP\klrjG\src\macros.jl:142 [inlined]

I would appreciate any help on this issue.

Thanks

odow · March 22, 2022, 7:51pm

The new error is saying that your data is wrong:

Try calling this for different t, n, and w:

data[:wind_cf][n][t] * data[:plants][:g_max][w][n]

If I had to guess the problem, it’s that your data[:wind_df] expects a Symbol as the key, but you passed a String. You might want data[:wind_cf][Symbol(n)][t], or you could change how you create the data.

mbah · March 22, 2022, 8:36pm

Thanks @odow for this insight. I suspect the problem may be when I try to convert my input dataframes into dictionaries as follows;

for inputDataItem in inputDataItems 
  readDataDFs[Symbol(inputDataItem)] = DataFrame(CSV.File(data_dir * inputDataItem * ".csv"), copycols=true) 
  data[Symbol(inputDataItem)] = df_to_dict_with_id(readDataDFs[Symbol(inputDataItem)]) 
  set[Symbol(inputDataItem)] = readDataDFs[Symbol(inputDataItem)].index
end

#Here index is set to the first coloumn of all csv files

Any way around this?

jd-foster · March 22, 2022, 11:33pm

We would have to know what this function looks like.

mbah · March 23, 2022, 9:31am

Sure @jd-foster , I have added the complete data reading code below

function df_to_dict_with_id(df::DataFrame)

    Dict(col[1] => Dict(zip(df[!, 1], col[2])) for col in pairs(eachcol(df)))

  end

data = Dict{Symbol,Any}()  
set = Dict{Symbol,Any}()
readDataDFs = Dict()
inputDataItems = ["plants", "demand_2018", "nodes","lines","solar_cf","wind_cf","ror_cf"] # read csv file names for input data 

for inputDataItem in inputDataItems #loop starts~ for each data file inputDataItems
  readDataDFs[Symbol(inputDataItem)] = DataFrame(CSV.File(data_dir * inputDataItem * ".csv"), copycols=true) #read in all the CSV files into the dictionary readDataDFs
  data[Symbol(inputDataItem)] = df_to_dict_with_id(readDataDFs[Symbol(inputDataItem)]) #use the dataframe to dictionary function to convert all dataframes into dictionaries and insert the dic files to data. Data is now main working file
  set[Symbol(inputDataItem)] = readDataDFs[Symbol(inputDataItem)].index #set is set to the first coloumn (index) of all the csv file 
end

If i can add a bit of explanation to my previous response to odow. The files ‘plants,demand_2018,nodes and lines’ all have a coloumn called index which is mostly a combination of strings and numbers identifying each element. A few example below

Lines
lines

Plants
plants

Nodes
nodes

However, when it comes to the capacity factor csv files, the zones are ordered in seperate columns and the index is the time

Example: wind
wind_cf

Could this be the cause of the problem?

Thanks a lot!

jd-foster · March 31, 2022, 4:16am

Did you figure this one out? I’m pretty sure this is the solution to the last error you gave:

The reason that it is expecting a Symbol as key in the second set of brackets, e.g

data[:wind_cf][:MHKVL_E][2]

is that you are using the pairs function in df_to_dict_with_id on DataFrame columns for your wind_cf table, hence the column headers are interpreted as Symbols for your dictionary keys. You can do what @odow recommended, but if you really want to have Strings for your keys, then change df_to_dict_with_id to

function df_to_dict_with_id(df::DataFrame)

    Dict( String(col[1]) => Dict(zip(df[!, 1], col[2])) for col in pairs(eachcol(df)))

end

BTW, rather than

you might find that this is easier to read:

CSV.read(joinpath(data_dir,inputDataItem * ".csv"), DataFrame)

noting copycols=true is the default here anyway.