At a glance, why don't I have better performances between Julia and PHP?

As mentioned, use a typed Dict. And It seems you can remove some of the Dict lookups. Does this work:


function cross(internoVassoi, dictVassoioFila)

    println("Calcolo cross  ...")
    # ---------------------------------    
    nrCellePerVassoio = Dict{String,Vector{Int64}}();
    for row in internoVassoi
        if row>" "          
            s = split(row,";")
            codiceArticolo = s[4]                 
            nrVassoio = s[2]   
            
            if haskey(dictVassoioFila, nrVassoio)     
                fila = dictVassoioFila[nrVassoio]  

                nrTotali, nrPiene, nrVuote = get(nrCellePerVassoio, nrVassoio, [0,0,0])
            
                nrTotali += 1
                if codiceArticolo > ""                    
                    nrPiene += 1 
                else                 
                    nrVuote += 1        
                end   
                nrCellePerVassoio[ nrVassoio ] = [nrTotali,nrPiene,nrVuote]          
                
            end    
        end
    end
    return nrCellePerVassoio
end

I can’t do better then the PHP code :-/

I also try in this way, splitting the “cross” function , but performances are the same :-/


function crossdo(row, nrCellePerVassoio, dictVassoioFila)
    if row>" " 
        s = split(row,";")
        codiceArticolo = s[4]                 
        nrVassoio = s[2]
        if haskey(dictVassoioFila, nrVassoio)     
            fila = dictVassoioFila[nrVassoio]  
            nrTotali, nrPiene, nrVuote = [0,0,0]

            if haskey(nrCellePerVassoio, nrVassoio)
                nrTotali, nrPiene, nrVuote = nrCellePerVassoio[ nrVassoio ] 
            else
                nrCellePerVassoio[ nrVassoio ] = [nrTotali,nrPiene,nrVuote]  
            end 
        
            nrTotali += 1
            if codiceArticolo > ""                    
                nrPiene += 1 
            else                 
                nrVuote += 1        
            end   
            nrCellePerVassoio[ nrVassoio ][1] = nrTotali
            nrCellePerVassoio[ nrVassoio ][2] = nrPiene
            nrCellePerVassoio[ nrVassoio ][3] = nrVuote           
            
        end  
    end
end

function cross(internoVassoi, dictVassoioFila)

    println("Calcolo cross  ...")
    # ---------------------------------    
    xnrCellePerVassoio = Dict{String,Array{Int64}}();  
    for xrow in internoVassoi        
            crossdo(xrow, xnrCellePerVassoio, dictVassoioFila)       
    end
    return xnrCellePerVassoio
end

Dict{String,Array{Int64}}()

is not a concrete type. Use Vector{Int64}() or Array{Int64,1}() for 1D arrays. There are many ways to get more speed, but getting type stability (see also @code_warntype) is one of the first things I would fix.

2 Likes

I tried to replace the type as you suggested, but the processing time didn’t improve.:-/

did you try the code I posted above? If yes, I recommend profiling to see where the bottleneck is.

Are you comparing the julia and the php code in script mode?
Starting julia and compiling the code can take quite a bit of time.
If you really need the script mode, maybe this helps: ANN: DaemonMode.jl, a package to run faster scripts in Julia

1 Like

I would like to test it, but i doesn’t install …
What is wrong?


julia> import Pkg

julia> Pkg.add("DaemonMode")
   Updating registry at `C:\Users\Renzo\.julia\registries\General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
ERROR: The following package names could not be resolved:
 * DaemonMode (not found in project, manifest or registry)

Stacktrace:
 [1] pkgerror(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\Types.jl:53
 [2] ensure_resolved(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; registry::Bool) at D:\buildbot\worker\packag
e_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\Types.jl:802
 [3] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; preserve::Pkg.Types.PreserveLevel, platform::Pkg.BinaryP
latforms.Windows, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at D:\buildbot\worker\pa
ckage_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\API.jl:148
 [4] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at D:\buildbot\worker\package_win64\build\usr\share\juli
a\stdlib\v1.4\Pkg\src\API.jl:112
 [5] #add#27 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\API.jl:109 [inlined]
 [6] add at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\API.jl:109 [inlined]
 [7] #add#24 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\API.jl:107 [inlined]
 [8] add at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\API.jl:107 [inlined]
 [9] add(::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at D:\buildbot\worker\p
ackage_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\API.jl:106
 [10] add(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\API.jl:106
 [11] top-level scope at REPL[2]:1

julia>

it seems it is not registered.
Try this

using Pkg
dm = PackageSpec(url="https://github.com/dmolina/DaemonMode.jl")
Pkg.add(dm)

So we are talking about 0.9s vs. 0.7s when running as script, which involves compilation time. I am not sure if you can get much faster than that, other than using DaemonMode or putting the compiled packages into your Julia distribution using PackageCompiler.

I have rather another question: is this really a problem? Are you running this scripts thousands/millions of times? If so, consider running it once and do the loop over multiple files inside the Julia process. I am absolutely sure that the actual code in Julia is much faster than the PHP one, but negligible to the compilation time overhead.

4 Likes

exactly. when you call the script Julia compiles your code and then executes it. if the time is 0.9 seconds probably 0.895 is compilation and 0.005 is execution.

1 Like

I try calling the same elaboration 100 times, so to be sure that the compiled time has been spent for the first call, but not for the 99 next calls … here the Julia code:

folderInput ="c:/vmswap/robot/tabelle"
folderOutput ="c:/vmswap/robot/"

function daFileAdArray(nomeTabella)
    fn = folderInput  * "/" * nomeTabella
    st = open(fn) do file
        read(file, String)
    end
    rows = split(st, "\r\n") 
    deleteat!(rows,1); #cancello prima riga nome campi
    return rows
end

function daiLocazioneFila(ubicazioni)
    println("Calcolo dizionario id locazione / fila ...")
    dictIdLocazioneFila= Dict{String, String}()
    for row in ubicazioni
        if row>" "
        s = split(row,";")   
            idLocazione = s[1]  
            ubicazione = s[8]
            ubicaziones = split(ubicazione,".")
            fila=ubicaziones[3]
            dictIdLocazioneFila[idLocazione]=fila 
        end
    end
    return  dictIdLocazioneFila
end

function daiVassoioFila(testaVassoi, dictIdLocazioneFila)

    println("Calcolo dizionario nr vassoio / fila ...")
    dictVassoioFila = Dict{String,String}();  
    for row in testaVassoi
        if row>" "
           s = split(row,";")   
           nrVassoio = s[1]  
           idLocazione = s[4]
           if haskey(dictIdLocazioneFila, idLocazione)     
             fila = dictIdLocazioneFila[ idLocazione ]
             dictVassoioFila[nrVassoio]=fila
           end
        end
    end
    return dictVassoioFila
end
 
function crossdo(row, nrCellePerVassoio, dictVassoioFila)
    if row>" " 
        s = split(row,";")
        codiceArticolo = s[4]                 
        nrVassoio = s[2]
        if haskey(dictVassoioFila, nrVassoio)     
            fila = dictVassoioFila[nrVassoio]  
            nrTotali, nrPiene, nrVuote = [0,0,0]

            if haskey(nrCellePerVassoio, nrVassoio)
                nrTotali, nrPiene, nrVuote = nrCellePerVassoio[ nrVassoio ] 
            else
                nrCellePerVassoio[ nrVassoio ] = [nrTotali,nrPiene,nrVuote]  
            end 
        
            nrTotali += 1
            if codiceArticolo > ""                    
                nrPiene += 1 
            else                 
                nrVuote += 1        
            end   
            nrCellePerVassoio[ nrVassoio ][1] = nrTotali
            nrCellePerVassoio[ nrVassoio ][2] = nrPiene
            nrCellePerVassoio[ nrVassoio ][3] = nrVuote           
            
        end  
    end
end

function cross(internoVassoi, dictVassoioFila)

    println("Calcolo cross  ...")
    # ---------------------------------    
    xnrCellePerVassoio = Dict{String,Array{Int64}}();  
    for xrow in internoVassoi        
            crossdo(xrow, xnrCellePerVassoio, dictVassoioFila)       
    end
    return xnrCellePerVassoio
end

function calcolaVassoi(testaVassoi, nrCellePerVassoio)
    # ---------------------------------
    println("Calcolo finale  ...") 
    vassoi= copy([])

    for row in testaVassoi
        s = split(row,";")
        nrVassoio = s[1]
        if haskey(nrCellePerVassoio, nrVassoio)
        
                #println( nrVassoio, " = " , nrCellePerVassoio[nrVassoio] )
                #print( nrVassoio, ", "  )
                push!(vassoi,nrVassoio )
        
        end
    end
    println("nr totale vassoi ubicati = " , length(vassoi))
    println("========================================") 
    return vassoi
end


function calcoloFinale(testaVassoi, nrCellePerVassoio,dictVassoioFila)
    vassoiConQualcosa = copy([])
    vassoiTuttiPieni = copy([])
    vassoiTuttiVuoti = copy([])

    vassoiConQualcosa_1 = copy([])
    vassoiTuttiPieni_1 = copy([])
    vassoiTuttiVuoti_1 = copy([])

    vassoiConQualcosa_2 = copy([])
    vassoiTuttiPieni_2 = copy([])
    vassoiTuttiVuoti_2 = copy([])

    vassoiConQualcosa_3 = copy([])
    vassoiTuttiPieni_3 = copy([])
    vassoiTuttiVuoti_3 = copy([])

    vassoiConQualcosa_4 = copy([])
    vassoiTuttiPieni_4 = copy([])
    vassoiTuttiVuoti_4 = copy([])

    for row in testaVassoi
        s = split(row,";")
        nrVassoio = s[1]
        if haskey(nrCellePerVassoio, nrVassoio)
            valoriPerVassoio =  nrCellePerVassoio[ nrVassoio ] 
            filaVassoio = dictVassoioFila[nrVassoio]
            if (valoriPerVassoio[2] > 0) && (!(valoriPerVassoio[1] == valoriPerVassoio[2]) )
                #println( nrVassoio, " = " , nrCellePerVassoio[nrVassoio] )
                #print( nrVassoio, ", "  )
                push!(vassoiConQualcosa,nrVassoio )

                if filaVassoio == "F01"
                    push!(vassoiConQualcosa_1,nrVassoio )
                end
                if filaVassoio == "F02"
                    push!(vassoiConQualcosa_2,nrVassoio )
                end
                if filaVassoio == "F03"
                    push!(vassoiConQualcosa_3,nrVassoio )
                end
                if filaVassoio == "F04"
                    push!(vassoiConQualcosa_4,nrVassoio )
                end


            end
            if valoriPerVassoio[2] == 0
                #println( nrVassoio, " = " , nrCellePerVassoio[nrVassoio] )
                #print( nrVassoio, ", "  )
                push!(vassoiTuttiVuoti,nrVassoio )
                if filaVassoio == "F01"
                    push!(vassoiTuttiVuoti_1,nrVassoio )
                end
                if filaVassoio == "F02"
                    push!(vassoiTuttiVuoti_2,nrVassoio )
                end
                if filaVassoio == "F03"
                    push!(vassoiTuttiVuoti_3,nrVassoio )
                end
                if filaVassoio == "F04"
                    push!(vassoiTuttiVuoti_4,nrVassoio )
                end            
            end          
            if valoriPerVassoio[2] == valoriPerVassoio[1]
                #println( nrVassoio, " = " , nrCellePerVassoio[nrVassoio] )
                #print( nrVassoio, ", "  )
                push!(vassoiTuttiPieni,nrVassoio )
                if filaVassoio == "F01"
                    push!(vassoiTuttiPieni_1,nrVassoio )
                end
                if filaVassoio == "F02"
                    push!(vassoiTuttiPieni_2,nrVassoio )
                end
                if filaVassoio == "F03"
                    push!(vassoiTuttiPieni_3,nrVassoio )
                end
                if filaVassoio == "F04"
                    push!(vassoiTuttiPieni_4,nrVassoio )
                end           
            end    
            
        end
    end
    tab = "\t\t"
    println(" ")
    println("Numerosità totali e per fila   : ", "totali", tab ,"fila1",tab , "fila2",tab ,"fila3",tab ,"fila4")
    println("Numerosità vassoi con qualcosa : ", length(vassoiConQualcosa), tab ,length(vassoiConQualcosa_1),tab , length(vassoiConQualcosa_2),tab ,length(vassoiConQualcosa_3),tab ,length(vassoiConQualcosa_4))
    println("Numerosità vassoi tutti pieni  : ", length(vassoiTuttiPieni), tab ,length(vassoiTuttiPieni_1), tab ,length(vassoiTuttiPieni_2),tab ,length(vassoiTuttiPieni_3),tab ,length(vassoiTuttiPieni_4))
    println("Numerosità vassoi tutti vuoti  : ", length(vassoiTuttiVuoti), tab ,length(vassoiTuttiVuoti_1),tab , length(vassoiTuttiVuoti_2),tab , length(vassoiTuttiVuoti_3),tab ,length(vassoiTuttiVuoti_4) )
end

function statistica()
     xubicazioni = daFileAdArray("TAUTOMAGLOC.TXT")
     xtestaVassoi = daFileAdArray("TAUTOMAGVAS.TXT")
     xinternoVassoi = daFileAdArray("TAUTOMAGVASDETT.TXT")
     xdictIdLocazioneFila = daiLocazioneFila(xubicazioni)
     xdictVassoioFila = daiVassoioFila(xtestaVassoi,xdictIdLocazioneFila) 
     xnrCellePerVassoio = cross(xinternoVassoi, xdictVassoioFila)
    vassoi = calcolaVassoi(xtestaVassoi, xnrCellePerVassoio)
      calcoloFinale(xtestaVassoi, xnrCellePerVassoio,xdictVassoioFila)
end

 
for i in 1:100
    statistica()
    println("was the ",i,"° computation.")
end
 

here the php code, again a loop of 100 times:

<?php

$folderInput ="c:/vmswap/robot/tabelle";
$folderOutput ="c:/vmswap/robot/";

function daFileAdArray($nomeTabella) {
    global $folderInput;
    $fn = $folderInput  . "/" . $nomeTabella;
    $t = file_get_contents($fn);
    $rows = explode("\r\n",$t);
    array_shift($rows);  
    return $rows;
} 

for($k=1; $k<=100; $k++){
        $ubicazioni = daFileAdArray("TAUTOMAGLOC.TXT");
        $testaVassoi = daFileAdArray("TAUTOMAGVAS.TXT");
        $internoVassoi = daFileAdArray("TAUTOMAGVASDETT.TXT");

        # cross vassoi ubicazioni
        echo "\nCalcolo dizionario id locazione / fila ...";

        $dictIdLocazioneFila = [];

        foreach($ubicazioni as $row) { 
            if($row>" ") {
               $s = explode(";",$row);   
               $idLocazione = $s[0];  
               $ubicazione = $s[7];
               $ubicaziones = explode(".",$ubicazione);
               $fila=$ubicaziones[2];
               $dictIdLocazioneFila[$idLocazione]=$fila; 
            }
        }

        echo "\nCalcolo dizionario nr vassoio / fila ...";
        $dictVassoioFila = [];
        foreach($testaVassoi as $row) { 
            if($row>" "){
               $s = explode(";",$row);   
               $nrVassoio = $s[0];  
               $idLocazione = $s[3];
               if (array_key_exists($idLocazione, $dictIdLocazioneFila)) {

                 $fila = $dictIdLocazioneFila[ $idLocazione ];
                 $dictVassoioFila[$nrVassoio]=$fila;
               }
            }
        }

        echo "\nCalcolo  ...";
        # ---------------------------------
        $indiceMasterSuSlave = 1;
        $nrCellePerVassoio = [];  
        foreach($internoVassoi as $row) {
            if($row>" "){          
                $s = explode(";",$row);
                $codiceArticolo = $s[3];

                $nrVassoio = $s[$indiceMasterSuSlave];   
                if (array_key_exists($nrVassoio, $dictVassoioFila)) {

                    $fila = $dictVassoioFila[$nrVassoio]; 

                    if (array_key_exists($nrVassoio, $nrCellePerVassoio)) {
                        $x = $nrCellePerVassoio[ $nrVassoio ]; 
                        $nrTotali = $x[0];
                        $nrPiene = $x[1];
                        $nrVuote = $x[2];

                    } else {           
                         $nrTotali = 0;
                         $nrPiene = 0;
                         $nrVuote = 0;
                    }    
                    $nrTotali = $nrTotali + 1;

                    if($codiceArticolo > "") {
                        $nrPiene = $nrPiene + 1;

                    } else {
                        $nrVuote = $nrVuote + 1;        
                    }

                    $nrCellePerVassoio[ $nrVassoio ] = [$nrTotali,$nrPiene,$nrVuote];   
                }    
            }
        }

        # ---------------------------------
        echo "\nCalcolo finale  ...";
        $vassoi = [];

        foreach($testaVassoi as $row) {
            $s = explode(";",$row);
            $nrVassoio = $s[0];
            if (array_key_exists($nrVassoio, $nrCellePerVassoio)) {

                    $vassoi[]= $nrVassoio;      
            }
        }
        echo "\nnr totale vassoi ubicati = ".count($vassoi);

        echo "n========================================";

        # ---------------------------------

        $vassoiConQualcosa = [];
        $vassoiTuttiPieni =[];
        $vassoiTuttiVuoti = [];

        $vassoiConQualcosa_1 = [];
        $vassoiTuttiPieni_1 = [];
        $vassoiTuttiVuoti_1 = [];

        $vassoiConQualcosa_2 = [];
        $vassoiTuttiPieni_2 = [];
        $vassoiTuttiVuoti_2 = [];

        $vassoiConQualcosa_3 = [];
        $vassoiTuttiPieni_3 = [];
        $vassoiTuttiVuoti_3 = [];

        $assoiConQualcosa_4 = [];
        $vassoiTuttiPieni_4 = [];
        $vassoiTuttiVuoti_4 = [];

        foreach($testaVassoi as $row){
            $s = explode(";",$row);
            $nrVassoio = $s[0];
            if (array_key_exists($nrVassoio, $nrCellePerVassoio)) {

                $valoriPerVassoio =  $nrCellePerVassoio[ $nrVassoio ]; 
                $filaVassoio = $dictVassoioFila[$nrVassoio];
                if( 
                         ($valoriPerVassoio[1] > 0) && 
                         (!($valoriPerVassoio[0] == $valoriPerVassoio[1]) )
                   ) {

                    $vassoiConQualcosa[] =  $nrVassoio ;

                    if($filaVassoio == "F01") {
                        $vassoiConQualcosa_1[] = $nrVassoio;
                    }
                    if($filaVassoio == "F02") {
                        $vassoiConQualcosa_2[] = $nrVassoio;
                    }
                    if($filaVassoio == "F03") {
                        $vassoiConQualcosa_3[] = $nrVassoio;
                    }
                    if($filaVassoio == "F04") {
                        $vassoiConQualcosa_4[] = $nrVassoio;
                    }


                }
                if($valoriPerVassoio[1] == 0){
                    #println( nrVassoio, " = " , nrCellePerVassoio[nrVassoio] )
                    #print( nrVassoio, ", "  )
                    $vassoiTuttiVuoti[] = $nrVassoio;

                    if($filaVassoio == "F01") {
                        $vassoiTuttiVuoti_1[] = $nrVassoio;
                    }
                    if($filaVassoio == "F02") {
                        $vassoiTuttiVuoti_2[] = $nrVassoio;
                    }
                    if($filaVassoio == "F03") {
                        $vassoiTuttiVuoti_3[] = $nrVassoio;
                    }
                    if($filaVassoio == "F04") {
                        $vassoiTuttiVuoti_4[] = $nrVassoio;
                    }                    

                }

                if($valoriPerVassoio[1] == $valoriPerVassoio[0]) {
                    #println( nrVassoio, " = " , nrCellePerVassoio[nrVassoio] )
                    #print( nrVassoio, ", "  )
                    $vassoiTuttiPieni[] = $nrVassoio;


                    if($filaVassoio == "F01") {
                        $vassoiTuttiPieni_1[] = $nrVassoio;
                    }
                    if($filaVassoio == "F02") {
                        $vassoiTuttiPieni_2[] = $nrVassoio;
                    }
                    if($filaVassoio == "F03") {
                        $vassoiTuttiPieni_3[] = $nrVassoio;
                    }
                    if($filaVassoio == "F04") {
                        $vassoiTuttiPieni_4[] = $nrVassoio;
                    }    
                } 
            }
        }

        $tab = "\t\t";
        echo " ";
        echo "\nNumerosità totali e per fila   : ". "totali". $tab ."fila1".$tab . "fila2".$tab ."fila3".$tab ."fila4";
        echo "\nNumerosità vassoi con qualcosa : ". count($vassoiConQualcosa). $tab .count($vassoiConQualcosa_1).$tab . count($vassoiConQualcosa_2).$tab .count($vassoiConQualcosa_3).$tab .count($vassoiConQualcosa_4);
        echo "\nNumerosità vassoi tutti pieni  : ". count($vassoiTuttiPieni). $tab . count($vassoiTuttiPieni_1). $tab .count($vassoiTuttiPieni_2).$tab .count($vassoiTuttiPieni_3).$tab .count($vassoiTuttiPieni_4);
        echo "\nNumerosità vassoi tutti vuoti  : ". count($vassoiTuttiVuoti). $tab .count($vassoiTuttiVuoti_1).$tab . count($vassoiTuttiVuoti_2).$tab . count($vassoiTuttiVuoti_3).$tab .count($vassoiTuttiVuoti_4) ;

  echo "\nWas the $k ° computation.";        
}

?>

And here the Julia results:

Numerosità vassoi tutti pieni  : 474            150             189             72              63
Numerosità vassoi tutti vuoti  : 677            177             129             221             150
was the 98° computation.
Calcolo dizionario id locazione / fila ...
Calcolo dizionario nr vassoio / fila ...
Calcolo cross  ...
Calcolo finale  ...
nr totale vassoi ubicati = 1734
========================================

Numerosità totali e per fila   : totali         fila1           fila2           fila3           fila4
Numerosità vassoi con qualcosa : 583            128             164             113             178
Numerosità vassoi tutti pieni  : 474            150             189             72              63
Numerosità vassoi tutti vuoti  : 677            177             129             221             150
was the 99° computation.
Calcolo dizionario id locazione / fila ...
Calcolo dizionario nr vassoio / fila ...
Calcolo cross  ...
Calcolo finale  ...
nr totale vassoi ubicati = 1734
========================================

Numerosità totali e per fila   : totali         fila1           fila2           fila3           fila4
Numerosità vassoi con qualcosa : 583            128             164             113             178
Numerosità vassoi tutti pieni  : 474            150             189             72              63
Numerosità vassoi tutti vuoti  : 677            177             129             221             150
was the 100° computation.

Execution time: 18.165 s

c:\prg\julia>

Here the php results:

Numerosità vassoi tutti pieni  : 474            150             189             72              63
Numerosità vassoi tutti vuoti  : 677            177             129             221             150
Was the 98 ° computation.
Calcolo dizionario id locazione / fila ...
Calcolo dizionario nr vassoio / fila ...
Calcolo  ...
Calcolo finale  ...
nr totale vassoi ubicati = 1734n========================================
Numerosità totali e per fila   : totali         fila1           fila2           fila3           fila4
Numerosità vassoi con qualcosa : 583            128             164             113             17622
Numerosità vassoi tutti pieni  : 474            150             189             72              63
Numerosità vassoi tutti vuoti  : 677            177             129             221             150
Was the 99 ° computation.
Calcolo dizionario id locazione / fila ...
Calcolo dizionario nr vassoio / fila ...
Calcolo  ...
Calcolo finale  ...
nr totale vassoi ubicati = 1734n========================================
Numerosità totali e per fila   : totali         fila1           fila2           fila3           fila4
Numerosità vassoi con qualcosa : 583            128             164             113             17800
Numerosità vassoi tutti pieni  : 474            150             189             72              63
Numerosità vassoi tutti vuoti  : 677            177             129             221             150
Was the 100 ° computation.
Execution time: 20.094 s

c:\xampp\htdocs\robot>

There is a gain of only 2 seconds between the Julia and the PHP code.

:-/
I try also to run with 1000 iterations … running in the same moment the two script …jula ends consuming 218 seconds, PHP consuming 248 seconds. So , i see, the gain (from test with 100 iterations to the other with 1000) is about 10-12% … it seems to me to much little…

The much of time consumed is, using @time macro, here:

function crossdo(row, nrCellePerVassoio, dictVassoioFila)
    if row>" " 
        s = split(row,";")
        codiceArticolo = s[4]                 
        nrVassoio = s[2]
        if haskey(dictVassoioFila, nrVassoio)     
            fila = dictVassoioFila[nrVassoio]  
            nrTotali, nrPiene, nrVuote = [0,0,0]

            if haskey(nrCellePerVassoio, nrVassoio)
                nrTotali, nrPiene, nrVuote = nrCellePerVassoio[ nrVassoio ] 
            else
                nrCellePerVassoio[ nrVassoio ] = [nrTotali,nrPiene,nrVuote]  
            end 
        
            nrTotali += 1
            if codiceArticolo > ""                    
                nrPiene += 1 
            else                 
                nrVuote += 1        
            end   
            nrCellePerVassoio[ nrVassoio ][1] = nrTotali
            nrCellePerVassoio[ nrVassoio ][2] = nrPiene
            nrCellePerVassoio[ nrVassoio ][3] = nrVuote           
            
        end  
    end
end

function cross(internoVassoi, dictVassoioFila)

    println("Calcolo cross  ...")
    # ---------------------------------    
    xnrCellePerVassoio = Dict{String,Array{Int64}}();  
    for xrow in internoVassoi        
            crossdo(xrow, xnrCellePerVassoio, dictVassoioFila)       
    end
    return xnrCellePerVassoio
end

But i can’t imagine what is wrong … perhaps a wrong method of looking for the items in the Dict? i don’t understand …

Try using BenchmarkTools to @benchmark the statistica function rather than writing your own loop. I’m wondering if you are maybe allocating a lot of memory… the benchmark will tell you

1 Like

Benchmark modules gives these results:

...
Calcolo dizionario id locazione / fila ...
Calcolo dizionario nr vassoio / fila ...
Calcolo cross  ...
Calcolo finale  ...
nr totale vassoi ubicati = 1734
========================================

Numerosità totali e per fila   : totali         fila1           fila2           fila3           fila4
Numerosità vassoi con qualcosa : 583            128             164             113             178
Numerosità vassoi tutti pieni  : 474            150             189             72              63
Numerosità vassoi tutti vuoti  : 677            177             129             221             150
   143.401 ms (2787482 allocations: 105.75 MiB)

c:\prg\julia>

having done this instead of the loop:


#for i in 1:1000
#@benchmark begin
    @btime statistica()
#end
 #       println("was the ",i,"° computation.")
#end
 

So what does it means?

Here is the csv files … dimension are normals for me.

Not that I think this will have a big impact but why are you calling copy here?

Might also be useful to make Arrays with concrete types instead of Any (as will create).

It means the calculations are taking 143 milliseconds and allocating 105 Megabytes of RAM.

It seems you might benefit from reducing allocations, given that your file is only 1MB or less.

I suggest to call statistica once to ensure compilation, and then call @profile on it. you might need to return to using a loop to get good coverage

https://docs.julialang.org/en/v1/manual/profile/

Mmmm, i obtain this … Do you understand something?

The output is more than 1500 lines and discourse here do not permit post of this entity, so i copy/paste only the portion of profile output that contain cross and crossdo that are the 2 functions surely slow (becase the @time macro showed me this).

  ╎    68  c:\prg\julia\robot6.jl:86; cross(::Array{SubString{String...
   ╎     2   c:\prg\julia\robot6.jl:52; crossdo(::SubString{String}, :...
   ╎    ╎ 2   @Base\operators.jl:294; >
   ╎    ╎  2   @Base\operators.jl:268; <
   ╎    ╎   2   ...\strings\basic.jl:315; isless
  1╎    ╎    1   ...\strings\basic.jl:271; cmp(::String, ::SubString{S...
   ╎    ╎    1   ...\strings\basic.jl:272; cmp(::String, ::SubString{S...
   ╎    ╎     1   @Base\iterators.jl:1213; Stateful
   ╎    ╎    ╎ 1   ...ngs\substring.jl:70; iterate
   ╎    ╎    ╎  1   ...ngs\substring.jl:70; iterate
  1╎    ╎    ╎   1   @Base\int.jl:53; +
   ╎     48  c:\prg\julia\robot6.jl:53; crossdo(::SubString{String}, :...
   ╎    ╎ 48  @Base\strings\util.jl:313; split
   ╎    ╎  48  ...e\strings\util.jl:313; #split#326
   ╎    ╎   1   @Base\array.jl:394; getindex
   ╎    ╎    1   @Base\boot.jl:424; Array
  1╎    ╎     1   @Base\boot.jl:405; Array
  1╎    ╎   1   ...e\strings\util.jl:0; _split(::SubString{String},...
   ╎    ╎   3   ...e\strings\util.jl:327; _split(::SubString{String},...
   ╎    ╎    3   ...strings\search.jl:104; findfirst
   ╎    ╎     3   ...trings\search.jl:275; findnext
   ╎    ╎    ╎ 3   ...trings\search.jl:239; _search
   ╎    ╎    ╎  2   ...rings\search.jl:150; _searchindex(::SubString{...
   ╎    ╎    ╎   2   ...rings\search.jl:133; findnext(::Base.Fix2{typ...
   ╎    ╎    ╎    2   @Base\generator.jl:44; iterate
   ╎    ╎    ╎     2   ...e\iterators.jl:334; iterate
   ╎    ╎    ╎    ╎ 2   ...e\iterators.jl:343; _zip_iterate_all
   ╎    ╎    ╎    ╎  1   ...\iterators.jl:351; _zip_iterate_some
   ╎    ╎    ╎    ╎   1   ...ings\basic.jl:540; iterate
   ╎    ╎    ╎    ╎    1   ...\substring.jl:96; nextind
  1╎    ╎    ╎    ╎     1   ...gs\string.jl:145; _nextind_str(::SubSt...
   ╎    ╎    ╎    ╎  1   ...\iterators.jl:353; _zip_iterate_some
   ╎    ╎    ╎    ╎   1   ...\iterators.jl:351; _zip_iterate_some
   ╎    ╎    ╎    ╎    1   ...\substring.jl:72; iterate
   ╎    ╎    ╎    ╎     1   ...gs\string.jl:183; iterate
  1╎    ╎    ╎    ╎    ╎ 1   @Base\int.jl:53; +
  1╎    ╎    ╎  1   ...rings\search.jl:154; _searchindex(::SubString{...
   ╎    ╎   8   ...e\strings\util.jl:333; _split(::SubString{String},...
   ╎    ╎    4   @Base\array.jl:914; push!
  4╎    ╎     4   @Base\array.jl:871; _growend!
   ╎    ╎    1   ...\strings\basic.jl:457; prevind
   ╎    ╎     1   ...strings\basic.jl:465; prevind(::SubString{String...
   ╎    ╎    ╎ 1   ...ngs\substring.jl:86; isvalid
   ╎    ╎    ╎  1   ...rings\string.jl:299; isvalid(::String, ::Int64)
   ╎    ╎    ╎   1   ...trings\basic.jl:186; checkbounds
  1╎    ╎    ╎    1   @Base\int.jl:410; <=
   ╎    ╎    3   ...ings\substring.jl:44; SubString
   ╎    ╎     3   ...ings\substring.jl:38; SubString
  1╎    ╎    ╎ 1   ...ngs\substring.jl:0; SubString
   ╎    ╎    ╎ 2   ...ngs\substring.jl:34; SubString
   ╎    ╎    ╎  2   ...rings\string.jl:137; nextind
  2╎    ╎    ╎   2   ...rings\string.jl:145; _nextind_str(::String, :...
   ╎    ╎   34  ...e\strings\util.jl:338; _split(::SubString{String},...
   ╎    ╎    34  ...strings\search.jl:275; findnext
   ╎    ╎     33  ...trings\search.jl:239; _search
  2╎    ╎    ╎ 2   ...trings\search.jl:144; _searchindex(::SubString{...
   ╎    ╎    ╎ 7   ...trings\search.jl:148; _searchindex(::SubString{...
   ╎    ╎    ╎  7   @Base\iterators.jl:544; peel
   ╎    ╎    ╎   7   @Base\iterators.jl:518; rest
   ╎    ╎    ╎    7   @Base\iterators.jl:500; Rest
  7╎    ╎    ╎     7   ...e\iterators.jl:500; Rest
   ╎    ╎    ╎ 10  ...trings\search.jl:150; _searchindex(::SubString{...
  1╎    ╎    ╎  1   ...rings\search.jl:0; findnext(::Base.Fix2{type...
  1╎    ╎    ╎  1   ...rings\search.jl:128; findnext(::Base.Fix2{type...
   ╎    ╎    ╎  6   ...rings\search.jl:131; findnext(::Base.Fix2{type...
   ╎    ╎    ╎   1   @Base\generator.jl:44; iterate
   ╎    ╎    ╎    1   @Base\iterators.jl:333; iterate
   ╎    ╎    ╎     1   ...e\iterators.jl:343; _zip_iterate_all
   ╎    ╎    ╎    ╎ 1   ...e\iterators.jl:351; _zip_iterate_some
   ╎    ╎    ╎    ╎  1   ...ings\basic.jl:540; iterate
   ╎    ╎    ╎    ╎   1   ...ings\basic.jl:540; iterate
   ╎    ╎    ╎    ╎    1   ...\substring.jl:96; nextind
  1╎    ╎    ╎    ╎     1   ...gs\string.jl:141; _nextind_str(::SubSt...
   ╎    ╎    ╎   5   ...gs\substring.jl:39; SubString
   ╎    ╎    ╎    5   ...gs\substring.jl:44; SubString
   ╎    ╎    ╎     5   ...gs\substring.jl:38; SubString
  2╎    ╎    ╎    ╎ 5   ...s\substring.jl:34; SubString
   ╎    ╎    ╎    ╎  3   ...ngs\string.jl:137; nextind
  2╎    ╎    ╎    ╎   2   ...ngs\string.jl:141; _nextind_str(::String...
  1╎    ╎    ╎    ╎   1   ...ngs\string.jl:145; _nextind_str(::String...
   ╎    ╎    ╎  2   ...rings\search.jl:133; findnext(::Base.Fix2{type...
   ╎    ╎    ╎   2   @Base\generator.jl:44; iterate
   ╎    ╎    ╎    2   @Base\iterators.jl:334; iterate
   ╎    ╎    ╎     2   ...e\iterators.jl:343; _zip_iterate_all
   ╎    ╎    ╎    ╎ 2   ...e\iterators.jl:351; _zip_iterate_some
   ╎    ╎    ╎    ╎  2   ...ings\basic.jl:540; iterate
   ╎    ╎    ╎    ╎   2   ...\substring.jl:96; nextind
  2╎    ╎    ╎    ╎    2   ...ngs\string.jl:141; _nextind_str(::SubStr...
   ╎    ╎    ╎ 3   ...trings\search.jl:152; _searchindex(::SubString{...
   ╎    ╎    ╎  3   ...ngs\substring.jl:96; nextind
  1╎    ╎    ╎   2   ...rings\string.jl:141; _nextind_str(::SubString...
  1╎    ╎    ╎    1   @Base\promotion.jl:398; ==
   ╎    ╎    ╎   1   ...rings\string.jl:145; _nextind_str(::SubString...
  1╎    ╎    ╎    1   @Base\int.jl:53; +
   ╎    ╎    ╎ 8   ...trings\search.jl:153; _searchindex(::SubString{...
  8╎    ╎    ╎  8   @Base\iterators.jl:1213; Stateful
   ╎    ╎    ╎ 3   ...trings\search.jl:154; _searchindex(::SubString{...
   ╎    ╎    ╎  1   @Base\reduce.jl:815; all
   ╎    ╎    ╎   1   @Base\reduce.jl:819; _all(::Base.var"#68#69"{...
   ╎    ╎    ╎    1   @Base\iterators.jl:333; iterate
   ╎    ╎    ╎     1   ...e\iterators.jl:341; _zip_iterate_all
   ╎    ╎    ╎    ╎ 1   ...e\iterators.jl:373; _zip_isdone
  1╎    ╎    ╎    ╎  1   ...essentials.jl:205; tail
   ╎    ╎    ╎  2   ...gs\substring.jl:39; SubString
   ╎    ╎    ╎   1   ...trings\basic.jl:159; lastindex
  1╎    ╎    ╎    1   ...gs\substring.jl:95; thisind(::SubString{Stri...
   ╎    ╎    ╎   1   ...gs\substring.jl:44; SubString
   ╎    ╎    ╎    1   ...gs\substring.jl:38; SubString
  1╎    ╎    ╎     1   ...gs\substring.jl:34; SubString
  1╎    ╎     1   ...trings\search.jl:240; _search
   ╎    ╎   1   ...e\strings\util.jl:344; _split(::SubString{String},...
   ╎    ╎    1   @Base\array.jl:914; push!
  1╎    ╎     1   @Base\array.jl:871; _growend!
   ╎     2   c:\prg\julia\robot6.jl:56; crossdo(::SubString{String}, :...
   ╎    ╎ 2   @Base\dict.jl:545; haskey
   ╎    ╎  2   @Base\dict.jl:289; ht_keyindex(::Dict{String,St...
   ╎    ╎   2   @Base\operators.jl:123; isequal
   ╎    ╎    2   ...\strings\basic.jl:295; ==
   ╎    ╎     1   ...strings\basic.jl:273; cmp(::SubString{String}, :...
   ╎    ╎    ╎ 1   @Base\iterators.jl:333; iterate
   ╎    ╎    ╎  1   @Base\iterators.jl:345; _zip_iterate_all
   ╎    ╎    ╎   1   @Base\iterators.jl:351; _zip_iterate_some
   ╎    ╎    ╎    1   ...e\iterators.jl:1263; iterate
   ╎    ╎    ╎     1   ...e\iterators.jl:1263; iterate
   ╎    ╎    ╎    ╎ 1   ...\iterators.jl:1256; popfirst!
   ╎    ╎    ╎    ╎  1   ...s\substring.jl:72; iterate
   ╎    ╎    ╎    ╎   1   ...ngs\string.jl:183; iterate
  1╎    ╎    ╎    ╎    1   @Base\int.jl:53; +
  1╎    ╎     1   ...strings\basic.jl:276; cmp(::SubString{String}, :...
   ╎     1   c:\prg\julia\robot6.jl:57; crossdo(::SubString{String}, :...
   ╎    ╎ 1   @Base\dict.jl:476; getindex
   ╎    ╎  1   @Base\dict.jl:289; ht_keyindex(::Dict{String,St...
   ╎    ╎   1   @Base\operators.jl:123; isequal
   ╎    ╎    1   ...\strings\basic.jl:295; ==
   ╎    ╎     1   ...strings\basic.jl:274; cmp(::SubString{String}, :...
   ╎    ╎    ╎ 1   @Base\operators.jl:193; !=
   ╎    ╎    ╎  1   @Base\char.jl:208; ==
  1╎    ╎    ╎   1   @Base\promotion.jl:398; ==
   ╎     2   c:\prg\julia\robot6.jl:58; crossdo(::SubString{String}, :...
   ╎    ╎ 2   @Base\array.jl:130; vect
   ╎    ╎  2   @Base\array.jl:657; _array_for
   ╎    ╎   2   ...\abstractarray.jl:670; similar
   ╎    ╎    2   ...\abstractarray.jl:671; similar
   ╎    ╎     2   @Base\boot.jl:414; Array
  2╎    ╎    ╎ 2   @Base\boot.jl:405; Array
   ╎     3   c:\prg\julia\robot6.jl:60; crossdo(::SubString{String}, :...
   ╎    ╎ 3   @Base\dict.jl:545; haskey
   ╎    ╎  1   @Base\dict.jl:282; ht_keyindex(::Dict{String,Ar...
   ╎    ╎   1   @Base\dict.jl:168; hashindex
   ╎    ╎    1   @Base\hashing.jl:18; hash
  1╎    ╎     1   @Base\hashing2.jl:179; hash
   ╎    ╎  2   @Base\dict.jl:289; ht_keyindex(::Dict{String,Ar...
   ╎    ╎   2   @Base\operators.jl:123; isequal
   ╎    ╎    2   ...\strings\basic.jl:295; ==
   ╎    ╎     1   ...strings\basic.jl:272; cmp(::SubString{String}, :...
  1╎    ╎    ╎ 1   @Base\iterators.jl:1213; Stateful
   ╎    ╎     1   ...strings\basic.jl:274; cmp(::SubString{String}, :...
   ╎    ╎    ╎ 1   @Base\iterators.jl:334; iterate
   ╎    ╎    ╎  1   @Base\iterators.jl:345; _zip_iterate_all
   ╎    ╎    ╎   1   @Base\iterators.jl:353; _zip_iterate_some
   ╎    ╎    ╎    1   @Base\iterators.jl:351; _zip_iterate_some
   ╎    ╎    ╎     1   ...e\iterators.jl:1263; iterate
  1╎    ╎    ╎    ╎ 1   ...\iterators.jl:1256; popfirst!
   ╎     3   c:\prg\julia\robot6.jl:61; crossdo(::SubString{String}, :...
   ╎    ╎ 1   @Base\dict.jl:476; getindex
   ╎    ╎  1   @Base\dict.jl:289; ht_keyindex(::Dict{String,Ar...
   ╎    ╎   1   @Base\operators.jl:123; isequal
   ╎    ╎    1   ...\strings\basic.jl:295; ==
   ╎    ╎     1   ...strings\basic.jl:272; cmp(::SubString{String}, :...
  1╎    ╎    ╎ 1   @Base\iterators.jl:1213; Stateful
   ╎    ╎ 2   @Base\tuple.jl:82; indexed_iterate
  2╎    ╎  2   @Base\tuple.jl:82; indexed_iterate
   ╎     1   c:\prg\julia\robot6.jl:63; crossdo(::SubString{String}, :...
   ╎    ╎ 1   @Base\dict.jl:376; setindex!(::Dict{String,Arra...
   ╎    ╎  1   @Base\dict.jl:381; setindex!(::Dict{String,Arra...
   ╎    ╎   1   @Base\dict.jl:348; ht_keyindex2!(::Dict{String...
   ╎    ╎    1   @Base\dict.jl:191; rehash!(::Dict{String,Array...
   ╎    ╎     1   @Base\array.jl:500; zeros
   ╎    ╎    ╎ 1   @Base\array.jl:504; zeros
   ╎    ╎    ╎  1   @Base\boot.jl:414; Array
  1╎    ╎    ╎   1   @Base\boot.jl:405; Array
   ╎     2   c:\prg\julia\robot6.jl:72; crossdo(::SubString{String}, :...
   ╎    ╎ 2   @Base\dict.jl:476; getindex
   ╎    ╎  2   @Base\dict.jl:289; ht_keyindex(::Dict{String,Ar...
   ╎    ╎   2   @Base\operators.jl:123; isequal
   ╎    ╎    2   ...\strings\basic.jl:295; ==
   ╎    ╎     1   ...strings\basic.jl:272; cmp(::SubString{String}, :...
  1╎    ╎    ╎ 1   @Base\iterators.jl:1213; Stateful
   ╎    ╎     1   ...strings\basic.jl:274; cmp(::SubString{String}, :...
   ╎    ╎    ╎ 1   @Base\iterators.jl:334; iterate
   ╎    ╎    ╎  1   @Base\iterators.jl:345; _zip_iterate_all
   ╎    ╎    ╎   1   @Base\iterators.jl:351; _zip_iterate_some
   ╎    ╎    ╎    1   ...e\iterators.jl:1263; iterate
   ╎    ╎    ╎     1   ...e\iterators.jl:1256; popfirst!
   ╎    ╎    ╎    ╎ 1   ...s\substring.jl:70; iterate
  1╎    ╎    ╎    ╎  1   @Base\int.jl:53; +
   ╎     1   c:\prg\julia\robot6.jl:73; crossdo(::SubString{String}, :...
   ╎    ╎ 1   @Base\dict.jl:476; getindex
   ╎    ╎  1   @Base\dict.jl:289; ht_keyindex(::Dict{String,Ar...
   ╎    ╎   1   @Base\operators.jl:123; isequal
   ╎    ╎    1   ...\strings\basic.jl:295; ==
   ╎    ╎     1   ...strings\basic.jl:272; cmp(::SubString{String}, :...
  1╎    ╎    ╎ 1   @Base\iterators.jl:1213; Stateful
   ╎     3   c:\prg\julia\robot6.jl:74; crossdo(::SubString{String}, :...
   ╎    ╎ 3   @Base\dict.jl:476; getindex
  1╎    ╎  1   @Base\dict.jl:279; ht_keyindex(::Dict{String,Ar...
   ╎    ╎  2   @Base\dict.jl:289; ht_keyindex(::Dict{String,Ar...
   ╎    ╎   2   @Base\operators.jl:123; isequal
   ╎    ╎    2   ...\strings\basic.jl:295; ==
   ╎    ╎     1   ...strings\basic.jl:272; cmp(::SubString{String}, :...
   ╎    ╎    ╎ 1   @Base\iterators.jl:1213; Stateful
   ╎    ╎    ╎  1   ...ngs\substring.jl:70; iterate
   ╎    ╎    ╎   1   ...gs\substring.jl:71; iterate
  1╎    ╎    ╎    1   ...trings\basic.jl:194; checkbounds
   ╎    ╎     1   ...strings\basic.jl:274; cmp(::SubString{String}, :...
   ╎    ╎    ╎ 1   @Base\iterators.jl:334; iterate
   ╎    ╎    ╎  1   @Base\iterators.jl:345; _zip_iterate_all
   ╎    ╎    ╎   1   @Base\iterators.jl:351; _zip_iterate_some
   ╎    ╎    ╎    1   ...e\iterators.jl:1263; iterate
   ╎    ╎    ╎     1   ...e\iterators.jl:1256; popfirst!
   ╎    ╎    ╎    ╎ 1   ...s\substring.jl:72; iterate
  1╎    ╎    ╎    ╎  1   @Base\Base.jl:33; getproperty

It seems to me that it shows many calls to split in strings library, also calls to findnext and other string processing.

it probably means most of the time is spent on string processing and the php string code probably has string processing written in c so it would have similar execution time.

I’d look back at how you process strings and see if that could be improved.

For example rather than splitting your string it seems you only use entry 2 and 4 so perhaps just find the first 4 instances of ‘;’ and create substrings of the 2 and 4 components…

I understand the following.
PHP takes 0.758 s
from your post above, Julia currently takes 0.143 s (if all functions are complied / run at least once).

Thus if you get DaemonMode working, you will be at 0.143 s
This can clearly be improved, but it is already a bit faster than PHP. (factor 5)
Admittedly I cannot say whether this comparison is fair/meaningful as I have zero knowledge about PHP (and its compilation/startup time)

you may want to consider these two packages



which may help you identify problematic code pieces
1 Like

You could consider CSV.jl https://github.com/JuliaData/CSV.jl .
This will be MUCH slower for the first time reading the data. For subsequent calls, it should be ‘ok’ though.
But it will allow you to specify the delimiter as ;
s = split(row,";")
should not be necessary anymore, as you can directly read into a DataFrame (or similar).
but this would mean that you also need DataFrames.jl.
hard to say whether this is worth it for your use case, without having the data at hand.