Problem with Base.push! in 0.7.0-alpha (?)

f-fathurrahman · June 25, 2018, 6:13am

Dear all,

I am a computational physicist and a novice Julia programmer.I have written several simple codes using Julia, although mostly they still look like Fortran .

With the release of Julia 0.7.0-alpha, I have been trying to update my code.
So far so good until I came to this serious problem (IMHO) which make me to hard restart my laptop several times.

I am sorry because I could not present the minimal code that reproduce the problem I have faced because I could not risk to do any more hard restart again on my laptop.

I will just give the link to github of the problematic code below.

https://github.com/f-fathurrahman/ffr-ElectronicStructure.jl/blob/master/LO_Gaussian/test_BasisSet.jl

The code is my effort to refactor PyQuante.jl code of Rick Mueller.

Using Julia 0.6.3 the code runs fine. However, when I use 0.7.0-alpha, I got the following error:

ERROR: LoadError: LoadError: error in method definition: function Base.push! must be explicitly imported to be extended
Stacktrace:
 [1] top-level scope
 [2] include at ./boot.jl:314 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1071
 [4] include(::Module, ::String) at ./sysimg.jl:29
 [5] include(::String) at ./client.jl:393
 [6] top-level scope
 [7] include at ./boot.jl:314 [inlined]
 [8] include_relative(::Module, ::String) at ./loading.jl:1071
 [9] include(::Module, ::String) at ./sysimg.jl:29
 [10] exec_options(::Base.JLOptions) at ./client.jl:267
 [11] _start() at ./client.jl:427
in expression starting at /home/efefer/WORKS/my_github_repos/ElectronicStructure.jl/ffr-ElectronicStructure.jl/LO_Gaussian/CGBF.jl:44
in expression starting at /home/efefer/WORKS/my_github_repos/ElectronicStructure.jl/ffr-ElectronicStructure.jl/LO_Gaussian/test_BasisSet.jl:9

So, I try to import Base.push! in file CGBF.jl:

import Base.push!
# This should become the constructor along with center and power
function push!(cbf::CGBF,expn,coef)
    Base.push!(cbf.pgbfs, PGBF(expn, cbf.center, cbf.power))
    Base.push!(cbf.coefs, coef)
    normalize!(cbf)
end

When I do it like this the code run at some point before call to this function and then it hang up. I first thought that this is not a Julia problem and simply my laptop hangs (again because of unknown problem, which happened sometimes).
However, I run into this problem again when running julia-0.7.0-alpha test_BasisSet.jl. After several hard restart, I tried to make sense of this problem and try to see what happened to my laptop when I run this. So, I open up htop and observed that memory usage of Julia jumps up to 55% and before my laptop hang up.

I have encountered the hang up problems when I used Julia before (mainly because I tried to construct sparse matrix using kron), but this problem is the most severe I have encountered so far.

This problem might be solved if I avoid using push and allocate memory beforehand which is the usual way I am handling this problem in Fortran. However, I might use push! somewhere and be aware of this problem.

Any help will be appreciated.
Best regards,

Fadjar Fathurrahman

I am using Ubuntu 16.04, 64 bit

julia> Sys.total_memory()/2^20
3835.27734375

julia> Sys.cpu_
cpu_info    cpu_summary
julia> Sys.cpu_info()
2-element Array{Base.Sys.CPUinfo,1}:
 Intel(R) Pentium(R) CPU B980 @ 2.40GHz: 
        speed         user         nice          sys         idle          irq
      840 MHz      37364 s        141 s       7177 s     272661 s          0 s
 Intel(R) Pentium(R) CPU B980 @ 2.40GHz: 
        speed         user         nice          sys         idle          irq
      971 MHz      40149 s        239 s       7743 s     268268 s          0 s

Tamas_Papp · June 25, 2018, 6:50am

You are probably affected by this language change:

The syntax using A.B can now only be used when A.B is a module, and the syntax using A: B can only be used for adding single bindings

Use

import Base: push!

instead.

f-fathurrahman · June 25, 2018, 7:47am

Thank you very much for quick reply.

I have tried your proposed solution however I still get the problem (I have to hard restart again).

By the way, I have used the syntax import Base.println without any problem in my code.

f-fathurrahman · June 25, 2018, 7:47am

By the way, is it possible to upload zip file of the code ?
(In case it is annoying to download the code one-by-one or you don’t want to download the entire git repo).

Tamas_Papp · June 25, 2018, 7:58am

It is best to make a minimal working example. You learn about the problem in the process (and may even solve it), and reviewing smaller files makes it easier to help you.

f-fathurrahman · June 27, 2018, 8:30am

Sorry I was panicked at that time.

Here is some code I put in COMMON.jl:

const Tuple3F64 = Tuple{Float64,Float64,Float64}
const Tuple3I64 = Tuple{Int64,Int64,Int64}

struct PGBF
    center::Tuple3F64
    power::Tuple3I64
    expn::Float64
    NORM::Float64
end

function PGBF( expn::Float64, center::Tuple3F64, power::Tuple3I64 )
    NORM = 1.0
    p = PGBF( center, power, expn, NORM )
    return p
end

function PGBF( expn::Float64, center::Tuple3F64 )
    power = (0,0,0)
    NORM = 1.0
    p = PGBF( center, power, expn, NORM )
    return p
end

function PGBF( expn::Float64 )
    center = (0.0,0.0,0.0)
    power = (0,0,0)
    NORM = 1.0
    p = PGBF( center, power, expn, NORM )
    return p
end

mutable struct CGBF
    center::Tuple3F64
    power::Tuple3I64
    pgbfs::Array{PGBF,1}
    coefs::Array{Float64,1}
    NORM::Float64
end

function init_CGBF( center::Tuple3F64, power::Tuple3I64 )
    return CGBF( center, power, PGBF[], Float64[], 1.0 )
end

init_CGBF(x=0,y=0,z=0,I=0,J=0,K=0) = CGBF( (x,y,z), (I,J,K), PGBF[], Float64[], 1.0 )

#import Base: push!  # The problem is here
function push!(cbf::CGBF,expn,coef)
    Base.push!(cbf.pgbfs, PGBF(expn, cbf.center, cbf.power))
    Base.push!(cbf.coefs, coef)
end

const ZATOMS = Dict(
    "H"  => 1,
    "He" => 2,
    "Li" => 3 )

    const sto3g = Any[
      # H
      [("S",
        [(3.4252509099999999, 0.15432897000000001),
         (0.62391373000000006, 0.53532813999999995),
         (0.16885539999999999, 0.44463454000000002)])],
      # He
      [("S",
        [(6.3624213899999997, 0.15432897000000001),
         (1.1589229999999999, 0.53532813999999995),
         (0.31364978999999998, 0.44463454000000002)])],
    ]
    const basis_set_data = Dict{Any,Any}("sto3g" => sto3g);


const BasisSet = Array{CGBF,1}

function push!(basis::BasisSet,cbf::CGBF)
    Base.push!(basis,cbf)
end

function build_basis( Natoms::Int64, atsymbs::Array{String,1},
                      positions::Array{Float64,2}; name="sto3g" )
    data = basis_set_data[name]
    basis = BasisSet()

    for ia = 1:Natoms

        atno = ZATOMS[atsymbs[ia]]
        x = positions[1,ia]
        y = positions[2,ia]
        z = positions[3,ia]

        for btuple in data[atno]
            sym,primlist = btuple
            for (I,J,K) in sym2power[sym]
                cbf = init_CGBF( (x,y,z), (I,J,K) )
                push!(basis,cbf)
                for (expn,coef) in primlist
                    push!(cbf,expn,coef)
                end
            end
        end

    end

    return basis
end

const sym2power = Dict{Any,Any}(
  "S" => [(0,0,0)],
  "P" => [(1,0,0),(0,1,0),(0,0,1)],
  "D" => [(2,0,0),(0,2,0),(0,0,2),(1,1,0),(1,0,1),(0,1,1)]
  )

Now, in the calling function:

if VERSION > v"0.6.3"
    using Printf
end

include("COMMON.jl")

function test_main()
    Natoms = 1
    positions = zeros(Float64,3,Natoms)
    basis = build_basis(1, ["H"], positions)
    println(basis)
end

test_main()

Using julia-0.6.3:

julia test_BasisSet.jl 
CGBF[CGBF((0.0, 0.0, 0.0), (0, 0, 0), PGBF[PGBF((0.0, 0.0, 0.0), (0, 0, 0), 3.42525, 1.0), PGBF((0.0, 0.0, 0.0), (0, 0, 0), 0.623914, 1.0), PGBF((0.0, 0.0, 0.0), (0, 0, 0), 0.168855, 1.0)], [0.154329, 0.535328, 0.444635], 1.0)]

Julia 0.7.0-alpha will complain:

julia-0.7.0-alpha test_BasisSet.jl 
ERROR: LoadError: LoadError: error in method definition: function Base.push! must be explicitly imported to be extended
Stacktrace:
 [1] top-level scope
 [2] include at ./boot.jl:314 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1071
 [4] include(::Module, ::String) at ./sysimg.jl:29
 [5] include(::String) at ./client.jl:393
 [6] top-level scope
 [7] include at ./boot.jl:314 [inlined]
 [8] include_relative(::Module, ::String) at ./loading.jl:1071
 [9] include(::Module, ::String) at ./sysimg.jl:29
 [10] exec_options(::Base.JLOptions) at ./client.jl:267
 [11] _start() at ./client.jl:427
in expression starting at /home/efefer/WORKS/JULIA/LO_gaussian_2_MWE/COMMON.jl:47
in expression starting at /home/efefer/WORKS/JULIA/LO_gaussian_2_MWE/test_BasisSet.jl:5

Last time I tried, adding Base.push! will hang up.

Reversing the lines:

                cbf = init_CGBF( (x,y,z), (I,J,K) )
                push!(basis,cbf)
                for (expn,coef) in primlist
                    push!(cbf,expn,coef)
                end

to:

                cbf = init_CGBF( (x,y,z), (I,J,K) )
                for (expn,coef) in primlist
                    push!(cbf,expn,coef)
                end
                push!(basis,cbf)

might solve the problem.

fredrikekre · June 27, 2018, 9:38am

Instead of

function push!(cbf::CGBF,expn,coef)
    Base.push!(cbf.pgbfs, PGBF(expn, cbf.center, cbf.power))
    Base.push!(cbf.coefs, coef)
end

Did you try

import Base: push!
function push!(cbf::CGBF,expn,coef)
    push!(cbf.pgbfs, PGBF(expn, cbf.center, cbf.power))
    push!(cbf.coefs, coef)
end

alternatively

function Base.push!(cbf::CGBF,expn,coef)
    push!(cbf.pgbfs, PGBF(expn, cbf.center, cbf.power))
    push!(cbf.coefs, coef)
end

f-fathurrahman · June 27, 2018, 9:56am

Thank you very much for your response.

I believed that I have tried:

# ...
import Base: push!
function push!(cbf::CGBF,expn,coef)
    push!(cbf.pgbfs, PGBF(expn, cbf.center, cbf.power))
    push!(cbf.coefs, coef)
end
# ...

along with the following lines in function build_basis:

                cbf = init_CGBF( (x,y,z), (I,J,K) )
                push!(basis,cbf)
                for (expn,coef) in primlist
                    push!(cbf,expn,coef)
                end

which caused my laptop to hang.
I’m sorry if I was mistaken, but currently I could not risk trying this on my laptop to make sure. I will report back soon as I can get access to my work PC.

Tamas_Papp · June 27, 2018, 10:07am

Are you sure it wasn’t something unrelated? Julia is powerful, but merely running code as a non-root user should not be able to affect the system.

f-fathurrahman · June 27, 2018, 10:16am

Yes this is not related to root or non root.

What I mean is that I have encountered several hang-up after running that script (probably because of memory-related problem?) which caused me to hard restart my laptop.

Actually, I can only reproduce the problem on Linux systems: my laptop and and my work PC, which both running Linux 16.04.

When I tried to ‘overload’ Base.push!:

import Base: push! 
function push!(cbf::CGBF,expn,coef)
    Base.push!(cbf.pgbfs, PGBF(expn, cbf.center, cbf.power))
    Base.push!(cbf.coefs, coef)
end

and running the file test_BasisSet.jl

if VERSION > v"0.6.3"
    using Printf
end

include("COMMON.jl")

function test_main()
    Natoms = 1
    positions = zeros(Float64,3,Natoms)
    basis = build_basis(1, ["H"], positions)
    println(basis)
end

test_main()

I got my laptop hang up.

I’m sorry if I am not clear in the problem statement.

f-fathurrahman · June 27, 2018, 12:56pm

UPDATE

I got stackoverflow error when running on Juliabox.

Tamas_Papp · June 27, 2018, 12:59pm

Again, having a minimal working example would make it so much easier to help you.

f-fathurrahman · June 27, 2018, 1:47pm

I am sorry with the ugly code. The truth is I am still learning the original code which is written by R.P Muller (https://github.com/rpmuller/pyquante2/tree/master/julia).

In the mean time I think I have solved the problem of hang up laptop and reproduced stackoverflow error on my laptop.

In my .bashrc, I used to add the statement ulimit -s unlimited because this is required by code generated by Intel Fortran compiler. It seems that this is no longer required and I have removed that line from my bashrc. My work PC also have this statement in its .bashrc.

@Tamas_Papp I think you are right about the problem. Julia is not wrong here, it is simply my setting which is not safe. Also thank you for pointing out the import Base: push! syntax, I got no deprecation messages about this.

I will come back again if I can isolate the real problem here.

kristoffer.carlsson · June 27, 2018, 1:49pm

I think the problem is quite simple. You have managed to define a method that calls itself over and over (and that is what the stacktrace in JuliaBox is showing) giving you an infinite recursion until stackoverflow.

f-fathurrahman · June 27, 2018, 1:55pm

Ups, I think it is . The problem vanish when I comment

const BasisSet = Array{CGBF,1}

#function push!(basis::BasisSet,cbf::CGBF)
#    Base.push!(basis,cbf)
#end

I guess I have panicked because my laptop hang up due to statement ulimit -s unlimited in my .bashrc.