How to add/edit values in an NDSparse?

Specific Domains Data
1

(crosposting from this so question)

How to add or edit values in an sparse indexed table ? From the documentation I understood that the NDSparse object itself is immutable but not the underlying data, so I “understand” why something like this doesn’t work, but I don’t know how to obtain a new table with the new data:


julia> myTable = ndsparse((
                    region      = ["US","US","US","US","EU","EU","EU","EU"],
                    product     = ["apple","apple","banana","banana","apple","apple","banana","banana"],
                    year        = [2011,2010,2011,2010,2011,2010,2011,2010]
                  ),(
                    production  = [3.3,3.2,2.3,2.1,2.7,2.8,1.5,1.3],
                    consumption = [4.3,7.4,2.5,9.8,3.2,4.3,6.5,3.0]
                 ))
3-d NDSparse with 8 values (2 field named tuples):
region  product   year │ production  consumption
───────────────────────┼────────────────────────
"EU"    "apple"   2010 │ 2.8         4.3
"EU"    "apple"   2011 │ 2.7         3.2
"EU"    "banana"  2010 │ 1.3         3.0
"EU"    "banana"  2011 │ 1.5         6.5
"US"    "apple"   2010 │ 3.2         7.4
"US"    "apple"   2011 │ 3.3         4.3
"US"    "banana"  2010 │ 2.1         9.8
"US"    "banana"  2011 │ 2.3         2.5

julia> myTable["EU","banana","2011"] = (2.5, 7.5) # Editing data
(2.5, 7.5)

julia> myTable["EU","banana","2012"] = (2.5, 7.5) # Adding row
(2.5, 7.5)

julia> flush!(myTable) # If ever needed...

julia> myTable # No changes !
3-d NDSparse with 8 values (2 field named tuples):
region  product   year │ production  consumption
───────────────────────┼────────────────────────
"EU"    "apple"   2010 │ 2.8         4.3
"EU"    "apple"   2011 │ 2.7         3.2
"EU"    "banana"  2010 │ 1.3         3.0
"EU"    "banana"  2011 │ 1.5         6.5
"US"    "apple"   2010 │ 3.2         7.4
"US"    "apple"   2011 │ 3.3         4.3
"US"    "banana"  2010 │ 2.1         9.8
"US"    "banana"  2011 │ 2.3         2.5
2

You have a typo:

myTable["EU","banana","2011"] = (2.5, 7.5)

should throw an error (maybe open an issue on GitHub).

But:

myTable["EU","banana",2011] = (production = 2.5, consumption = 7.5)

works!

3

Please, change the title of the question for future reference to something like “How to add/edit rows in an NDSparse?”.

I am asking because the answer for IndexedTable is different.

4

Done, thank you… but when I type your answer (using integers for the year) I receive the error “type Tuple has no field region”.

5

That is odd! Can you run:

using JuliaDB

myTable = ndsparse((
                    region      = ["US","US","US","US","EU","EU","EU","EU"],
                    product     = ["apple","apple","banana","banana","apple","apple","banana","banana"],
                    year        = [2011,2010,2011,2010,2011,2010,2011,2010]
                  ),(
                    production  = [3.3,3.2,2.3,2.1,2.7,2.8,1.5,1.3],
                    consumption = [4.3,7.4,2.5,9.8,3.2,4.3,6.5,3.0]
                 ))

myTable["EU","banana",2011] = (production = 2.5, consumption = 7.5)

in a fresh Julia session? Please, post the error and version numbers for JuliaDB and IndexedTables import Pkg; Pkg.status() and versioninfo().

6

I was using IndexedTables not JuliaDB (is the former deprecated ?)

Still, I have added and loaded JuliaDB instead (and restarted Julia session of course) but the problem is the same.

This is my packages status:

(v1.1) pkg> status
    Status `~/.julia/environments/v1.1/Project.toml`
  [c52e3926] Atom v0.8.7
  [6e4b80f9] BenchmarkTools v0.4.2
  [336ed68f] CSV v0.5.5
  [861a8166] Combinatorics v0.7.0
  [8f4d0f93] Conda v1.3.0
  [a0b5b9ef] Cxx v0.3.2
  [a93c6f00] DataFrames v0.18.3
  [1313f7d8] DataFramesMeta v0.4.1
  [864edb3b] DataStructures v0.15.0
  [31a5f54b] Debugger v0.5.0
  [31c24e10] Distributions v0.20.0
  [c91e804a] Gadfly v1.0.1
  [bd48cda9] GraphRecipes v0.4.0
  [f67ccb44] HDF5 v0.11.1
  [cd3eb016] HTTP v0.8.2
  [7073ff75] IJulia v1.18.1
  [6deec6e2] IndexedTables v0.12.0
  [b6b21f68] Ipopt v0.5.4
  [682c06a0] JSON v0.20.0
  [4076af6c] JuMP v0.19.2
  [a93385a2] JuliaDB v0.12.0
  [aa1ae85d] JuliaInterpreter v0.6.0
  [e5e0dc1b] Juno v0.7.0
  [aba2b823] LAJuliaUtils v0.0.0 #master (https://github.com/sylvaticus/LAJuliaUtils.jl.git)
  [2fda8390] LsqFit v0.8.1
  [d6bdc55b] Mads v0.7.0
  [e4e893b0] Mimi v0.9.1
  [51fcb6bd] NamedColors v0.2.0
  [56b0d19f] OdsIO v0.5.0
  [b98c9c47] Pipe v1.1.0
  [91a5bcdd] Plots v0.25.1
  [c46f51b8] ProfileView v0.4.1
  [438e738f] PyCall v1.91.2
  [d330b81b] PyPlot v2.8.1
  [1a8c2f83] Query v0.11.0
  [6f49c342] RCall v0.13.2
  [df47a6cb] RData v0.6.0
  [295af30f] Revise v2.1.6
  [27aeedcb] RobustPmap v0.4.1
  [60ddc479] StatPlots v0.9.2
  [f3b207a7] StatsPlots v0.11.0
  [fd094767] Suppressor v0.1.1
  [24249f21] SymPy v1.0.3
  [bd369af6] Tables v0.2.6
  [37b6cedf] Traceur v0.3.0
  [0ae4a718] VegaDatasets v0.5.0
  [112f6efa] VegaLite v0.6.0
  [fdbf4ff8] XLSX v0.5.3
  [c2297ded] ZMQ v1.0.0
  [9a3f8284] Random 
  [1a1011a3] SharedArrays

Versioninfo:

julia> versioninfo()
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = atom  -a
  JULIA_NUM_THREADS = 4

Full error stack:

julia> myTable["EU","banana",2011] = (production = 2.5, consumption = 7.5)
ERROR: type Tuple has no field region
Stacktrace:
 [1] _getproperty at /home/lobianco/.julia/packages/StructArrays/RkrVr/src/utils.jl:25 [inlined]
 [2] macro expansion at /home/lobianco/.julia/packages/StructArrays/RkrVr/src/utils.jl:42 [inlined]
 [3] foreachfield(::Type{NamedTuple{(:region, :product, :year),Tuple{String,String,Int64}}}, ::typeof(push!), ::StructArrays.StructArray{NamedTuple{(:region, :product, :year),Tuple{String,String,Int64}},1,NamedTuple{(:region, :product, :year),Tuple{Array{String,1},Array{String,1},Array{Int64,1}}},Int64}, ::Tuple{String,String,Int64}) at /home/lobianco/.julia/packages/StructArrays/RkrVr/src/utils.jl:42
 [4] foreachfield(::Function, ::StructArrays.StructArray{NamedTuple{(:region, :product, :year),Tuple{String,String,Int64}},1,NamedTuple{(:region, :product, :year),Tuple{Array{String,1},Array{String,1},Array{Int64,1}}},Int64}, ::Tuple{String,String,Int64}) at /home/lobianco/.julia/packages/StructArrays/RkrVr/src/utils.jl:45
 [5] push!(::StructArrays.StructArray{NamedTuple{(:region, :product, :year),Tuple{String,String,Int64}},1,NamedTuple{(:region, :product, :year),Tuple{Array{String,1},Array{String,1},Array{Int64,1}}},Int64}, ::Tuple{String,String,Int64}) at /home/lobianco/.julia/packages/StructArrays/RkrVr/src/structarray.jl:173
 [6] _setindex_scalar!(::NDSparse{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},Tuple{String,String,Int64},StructArrays.StructArray{NamedTuple{(:region, :product, :year),Tuple{String,String,Int64}},1,NamedTuple{(:region, :product, :year),Tuple{Array{String,1},Array{String,1},Array{Int64,1}}},Int64},StructArrays.StructArray{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},1,NamedTuple{(:production, :consumption),Tuple{Array{Float64,1},Array{Float64,1}}},Int64}}, ::NamedTuple{(:production, :consumption),Tuple{Float64,Float64}}, ::Tuple{String,String,Int64}) at /home/lobianco/.julia/packages/IndexedTables/DgX0A/src/indexing.jl:156
 [7] _setindex!(::NDSparse{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},Tuple{String,String,Int64},StructArrays.StructArray{NamedTuple{(:region, :product, :year),Tuple{String,String,Int64}},1,NamedTuple{(:region, :product, :year),Tuple{Array{String,1},Array{String,1},Array{Int64,1}}},Int64},StructArrays.StructArray{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},1,NamedTuple{(:production, :consumption),Tuple{Array{Float64,1},Array{Float64,1}}},Int64}}, ::NamedTuple{(:production, :consumption),Tuple{Float64,Float64}}, ::Tuple{String,String,Int64}) at /home/lobianco/.julia/packages/IndexedTables/DgX0A/src/indexing.jl:152
 [8] setindex!(::NDSparse{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},Tuple{String,String,Int64},StructArrays.StructArray{NamedTuple{(:region, :product, :year),Tuple{String,String,Int64}},1,NamedTuple{(:region, :product, :year),Tuple{Array{String,1},Array{String,1},Array{Int64,1}}},Int64},StructArrays.StructArray{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},1,NamedTuple{(:production, :consumption),Tuple{Array{Float64,1},Array{Float64,1}}},Int64}}, ::NamedTuple{(:production, :consumption),Tuple{Float64,Float64}}, ::String, ::String, ::Int64) at /home/lobianco/.julia/packages/IndexedTables/DgX0A/src/indexing.jl:140
 [9] top-level scope at none:0
7

IndexedTables is fine. JuliaDB is a wrapper around IndexedTables and offers extra features.

I see that your IndexedTables is at 0.12.0. Please, update import Pkg; Pkg.@pkg_str("add IndexedTables#master") because version 0.12.1 had some changes to some constructors.

8

Works with master.

However when I try to insert a new row (e.g. myTable["EU","banana",2012] = (production = 2.5, consumption = 7.5)) it “seems” all fine, but when I try to print the table I get the error:

[UPDATE]
Sorry it works fine, I may not have restarted Julia after the update??? When the master will be released ?
[/UPDATE]

julia> myTable
Error showing value of type NDSparse{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},Tuple{String,String,Int64},StructArrays.StructArray{NamedTuple{(:region, :product, :year),Tuple{String,String,Int64}},1,NamedTuple{(:region, :product, :year),Tuple{Array{String,1},Array{String,1},Array{Int64,1}}},Int64},StructArrays.StructArray{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},1,NamedTuple{(:production, :consumption),Tuple{Array{Float64,1},Array{Float64,1}}},Int64}}:
ERROR: index and data must have the same number of elements
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] #ndsparse#103(::Function, ::Bool, ::Nothing, ::Bool, ::Function, ::Val{:serial}, ::NamedTuple{(:region, :product, :year),Tuple{Array{String,1},Array{String,1},Array{Int64,1}}}, ::NamedTuple{(:production, :consumption),Tuple{Array{Float64,1},Array{Float64,1}}}) at /home/lobianco/.julia/packages/IndexedTables/iteNw/src/ndsparse.jl:80
 [3] #ndsparse at ./none:0 [inlined]
 [4] #ndsparse#102 at /home/lobianco/.julia/packages/IndexedTables/iteNw/src/ndsparse.jl:65 [inlined]
 [5] #ndsparse at ./none:0 [inlined]
 [6] #ndsparse#106 at /home/lobianco/.julia/packages/IndexedTables/iteNw/src/ndsparse.jl:112 [inlined]
 [7] #ndsparse at ./none:0 [inlined]
 [8] #ndsparse#107 at /home/lobianco/.julia/packages/IndexedTables/iteNw/src/ndsparse.jl:116 [inlined]
 [9] #ndsparse at ./none:0 [inlined]
 [10] #NDSparse#109 at /home/lobianco/.julia/packages/IndexedTables/iteNw/src/ndsparse.jl:123 [inlined]
 [11] Type at ./none:0 [inlined]
 [12] flush!(::NDSparse{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},Tuple{String,String,Int64},StructArrays.StructArray{NamedTuple{(:region, :product, :year),Tuple{String,String,Int64}},1,NamedTuple{(:region, :product, :year),Tuple{Array{String,1},Array{String,1},Array{Int64,1}}},Int64},StructArrays.StructArray{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},1,NamedTuple{(:production, :consumption),Tuple{Array{Float64,1},Array{Float64,1}}},Int64}}) at /home/lobianco/.julia/packages/IndexedTables/iteNw/src/indexing.jl:230
 [13] show(::IOContext{REPL.Terminals.TTYTerminal}, ::NDSparse{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},Tuple{String,String,Int64},StructArrays.StructArray{NamedTuple{(:region, :product, :year),Tuple{String,String,Int64}},1,NamedTuple{(:region, :product, :year),Tuple{Array{String,1},Array{String,1},Array{Int64,1}}},Int64},StructArrays.StructArray{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},1,NamedTuple{(:production, :consumption),Tuple{Array{Float64,1},Array{Float64,1}}},Int64}}) at /home/lobianco/.julia/packages/IndexedTables/iteNw/src/ndsparse.jl:275
 [14] show(::IOContext{REPL.Terminals.TTYTerminal}, ::MIME{Symbol("text/plain")}, ::NDSparse{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},Tuple{String,String,Int64},StructArrays.StructArray{NamedTuple{(:region, :product, :year),Tuple{String,String,Int64}},1,NamedTuple{(:region, :product, :year),Tuple{Array{String,1},Array{String,1},Array{Int64,1}}},Int64},StructArrays.StructArray{NamedTuple{(:production, :consumption),Tuple{Float64,Float64}},1,NamedTuple{(:production, :consumption),Tuple{Array{Float64,1},Array{Float64,1}}},Int64}}) at ./sysimg.jl:194
 [15] display(::REPL.REPLDisplay, ::MIME{Symbol("text/plain")}, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:131
 [16] display(::REPL.REPLDisplay, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:135
 [17] display(::Any) at ./multimedia.jl:287
 [18] #invokelatest#1 at ./essentials.jl:742 [inlined]
 [19] invokelatest at ./essentials.jl:741 [inlined]
 [20] print_response(::IO, ::Any, ::Any, ::Bool, ::Bool, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:155
 [21] print_response(::REPL.AbstractREPL, ::Any, ::Any, ::Bool, ::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:140
 [22] (::getfield(REPL, Symbol("#do_respond#38")){Bool,getfield(Atom, Symbol("##172#173")),REPL.LineEditREPL,REPL.LineEdit.Prompt})(::Any, ::Any, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:714
 [23] #invokelatest#1 at ./essentials.jl:742 [inlined]
 [24] invokelatest at ./essentials.jl:741 [inlined]
 [25] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/LineEdit.jl:2273
 [26] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:1035
 [27] run_repl(::REPL.AbstractREPL, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/REPL/src/REPL.jl:192
 [28] (::getfield(Base, Symbol("##734#736")){Bool,Bool,Bool,Bool})(::Module) at ./client.jl:362
 [29] #invokelatest#1 at ./essentials.jl:742 [inlined]
 [30] invokelatest at ./essentials.jl:741 [inlined]
 [31] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:346
 [32] exec_options(::Base.JLOptions) at ./client.jl:284
 [33] _start() at ./client.jl:436
9

Great! Try downgrading first: import Pkg; Pkg.@pgk_str("add IndexedTables@0.12.1")

If it does not work anymore, please tell me and I will ask the maintainers to make a new release.

10

v0.12.1 works, thank you again.