Julia just... vanished in the middle of the night during a large task. Help troubleshooting/optimising would be appreciated

I have quite a large task to run and in doing so I’ve encountered unfortunate issues. Below there are two blocks of code, one with a bunch of for loops (attempt 1), the other using functions (attempt 2). I’m running these on a Windows 11 laptop via a Jupyter Notebook opened from IJulia in Anaconda prompt. Attempt 1 had a time counter. It got to 2% within half an hour, but unfortunately it only got to 13% after 8 hours (overnight). The for loop is embarrassingly parallel, so I’m not sure why it would slow down so much with time. I woke up, begin using my computer, and then it ran out of memory (I think due to other memory-intensive programs I was using during the day).

I tried Attempt 2 the second night. 0.1% of the task (changing the value of T) took 15 seconds, so I thought it should complete overnight. However, when I woke up, Julia had… vanished. My computer had signed out of Windows but not restarted (I had set it never to sleep or sign out). Firefox was still open (I was running in a Jupyter Notebook) but the Notebook showed nothing on screen. I tried reopening the Notebook but I couldn’t connect to localhost:8888 - Anaconda prompt (in which I opened Julia) had seemingly shut. I had no error visible or anything - very strange. Julia just vanished mid-task.

I’ve attempted to use functions to write better code in Attempt 2. I’m quite new to programming so might be making significant rookie errors. Any guidance would be appreciated. For a MWE, simply change the value of T to tau or tau+1, tau+2, etc.

ATTEMPT 1:

using DataFrames, CSV, Dates, StatsBase, ProgressMeter,Statistics
const T,n,tau,M=5788,366,90,1000  #CHANGE T to tau+1
const daily_rf_rate = 0.0;
price_mat=rand(T,n);
empty_array = Array{Float64}(undef, T-tau+1, 91, 3);

# #The main loop is over t.
@showprogress for t in tau:T
    for k in 10:100
        sharpe_list=Array{Float64}(undef,M)
        for i in 1:M
            my_sample=sample(1:n,k,replace=false)
            drawn_prices=price_mat[t-tau+1:t,my_sample]

            # Calculate daily returns for each stock
            returns = diff(drawn_prices, dims=1) ./ drawn_prices[1:end-1, :]

            # Calculate daily portfolio returns using equal weights (0.1 for each stock since there are 10 stocks)
            portfolio_returns = sum(returns, dims=2) * 1/k

            # Compute average return and standard deviation for the portfolio
            average_portfolio_return = mean(portfolio_returns)
            stddev_portfolio_return = std(portfolio_returns)
            


# Calculate the Sharpe ratio for the portfolio
            sharpe_ratio_portfolio = (average_portfolio_return - daily_rf_rate) / stddev_portfolio_return
            ##empty_array[t,k,i]=sharpe_ratio_portfolio ##I'll save them later
            sharpe_list[i]=sharpe_ratio_portfolio
        end
        empty_array[t-tau+1,k-9,1]=quantile(sharpe_list,0.1); #FILL IN THE QUANTILE VALUES INTO ARRAY
        empty_array[t-tau+1,k-9,2]=quantile(sharpe_list,0.5);
        empty_array[t-tau+1,k-9,3]=quantile(sharpe_list,0.9);

    end
end

ATTEMPT 2

using DataFrames, CSV, Dates, StatsBase, ProgressMeter,Statistics
const T,n,tau,M=5788,366,90,1000  #CHANGE T to tau+1
const daily_rf_rate = 0.0;
price_mat=rand(T,n);

function compute_sharpe(t, k)
    sharpe_list = Array{Float64}(undef, M)
    for i in 1:M
        my_sample = sample(1:n, k, replace=false)
        drawn_prices = price_mat[t-tau+1:t, my_sample]
        returns = diff(drawn_prices, dims=1) ./ drawn_prices[1:end-1, :]
        portfolio_returns = sum(returns, dims=2) * 1/k
        average_portfolio_return = mean(portfolio_returns)
        stddev_portfolio_return = std(portfolio_returns)
        sharpe_ratio_portfolio = (average_portfolio_return - daily_rf_rate) / stddev_portfolio_return
        sharpe_list[i] = sharpe_ratio_portfolio
    end
    return sharpe_list
end

function compute_quantiles(t,k)
    sharpe_list=compute_sharpe(t,k)
return [quantile(sharpe_list,0.1), quantile(sharpe_list,0.5), quantile(sharpe_list,0.9)]
end 

I=tau:T;J=10:100
Ans_full=compute_quantiles.(I',J);

I’m guessing actually Windows decided to run an update and restart overnight

4 Likes

No that’s not it, because Firefox wouldn’t be open then, and my settings are do not update automatically.

Out of memory and the Julia process got killed? That also explains the slowdown at first.

1 Like

Yeah maybe but the final answer I’m filling out is only 5000 x 91 x 3, that’s not too large, right? My computer memory is 8GB and nothing else was running. My code doesn’t seem like it’d be too bad with memory, for example sharpe_list keeps getting overwritten. Am I doing something wrong with memory?

Yeah these things can be annoying to debug. There can be many reasons but I think it doesn’t make a lot of sense to speculate when it’s possible to get an error message.
To get this it would probably make sense to run the code by starting Julia in the shell instead. That way, even if Julia crashes you will at least be able to see the error.

It doesn’t look like it’s the source of your problem on first glance but in my opinion one should always have very good reason to use under initializers instead of zeros. If you make a mistake somewhere and don’t write to all positions of the array afterwards you can get very strange bugs indeed. I think it’s better to not use it in most cases.

Note that this is a non-const global so there will be a lot of runtime allocations that can easily be avoided by passing this matrix to the function as an argument instead.

1 Like

I haven’t looked at your code in detail, but at a glance, there’s a number of places that allocate quite a bit:

These allocate a copy - slicing arrays in julia copies. Use the @view macro to create a lightweight view into the data instead.

Similarly, the broadcast ./ also creates a new array. You can allocate the returns array once outside of the loop and reuse it, by writing returns .= ....

This leads to a type instability. Array{Float64} is an abstract type, and so the sizes on indexing are not known, because the dimensionality is a unknown. Use Vector{Float64}, or equivalently Array{Float64, 1}, instead.


I’d also recommend checking the Performance Tips section of the manual.

1 Like

I don’t want to belabor this point too much (and reducing the memory consumption of the code is worthwhile in any case), but like @dlakelan mentioned, it does sound a lot like Windows did an update during your second attempt. At least Windows 10 was notorious for doing updates even though it’s seemingly told not to…

If you set it to never sign out, but it actually signed out, it doesn’t seem related to Julia.

On my Windows machine, Firefox automatically re-opens and restores the last sessions after an update is installed. If that’s the case for you as well, it would explain why the notebook was “empty” (the Jupyter server in the background didn’t get restarted).

You could check the event/update logs to see whether it really didn’t restart or not if you want to investigate further. But then again, reducing the memory allocations might be more fruitful.

5 Likes

Absolutely, I don’t know what happened exactly, but I’m new to Julia and appreciate all tips to improve the time/space efficiency.

1 Like

AI generated answer to Linux syslog equivalent:

When a program crashes or runs out of memory on Windows, it is usually logged in the Windows Event Log. […]

You can view the Windows Event Log using the Event Viewer. To do this, press the Windows key + X and select Event Viewer from the menu. Once you have opened Event Viewer, you can navigate to the Windows Logs section and select Application to view application-related events. You can then filter the events by level, source, or date to find the relevant event.

Alternatively, you can write to a text file using a logging framework such as log4cxx. This approach is more pragmatic than using the Windows Event Log, which has a high barrier to entry and is not widely used by applications.

The OS log will not help you about the cause, only confirm, hopefully that the OS killed your program (its only option in the end if you try to use too much memory, you can limit that with CLI option, if possible in Julia 1.10). Julia has a memory profiler you can look into.

I didn’t think that was possible in recent Windows (since 10 or earlier) any more. Except allowed in certain enterprise versions. I’m not a Windows user, I think what you have is a suggestion to the OS regarding your preference (maybe it’s only respected at certain times, e.g. in the middle of a game?).

If all users could opt into not restarting, then most likely would… and most computers on the Internet would be insecure fast.

2 Likes

Oh thanks, I thought empty arrays were the best bet.

price_mat is generated by a previous step, so exists in memory as a global. So you’re saying I should just modify the function to take a matrix argument A and then call it on price_mat?

I actually find that empty (undef) arrays are preferable. If some elements are not filled and later you get an error, that a very good source of information that your code had a bug since the intention was to fill all elements.

1 Like

Thanks for your tips above - I did read this part of the manual before posting (it informed things like using const for the variables) but I’m new to Julia so most of it was difficult to understand. I’ve changed the code to allocate returns outside the inner loop, but its length changes with k, so I still have to allocate it 91 times. Thanks for your tip on Vector too - I had confused that with Array. Regarding your tip with the @view macro - do you also mean assigning an array drawn_prices first and then filling it with the following?

draw_prices .=@view price_mat[t-tau+1:t,my_sample]

You should better fill it with NaN’s then in my opinion. In most cases, using undef only means that you take whatever state the allocated memory is in and interpret that to be your datatype:

function UseUndef(N)
    a = Vector{Float64}(undef,N)

    return sum(a)
end
for i in 1:10
    println(UseUndef(5))
end

which printed for me

1.390198155611988e-308
1.390198008661865e-308
NaN
1.6e-321
1.51e-321
1.453e-321
1.45e-321
1.393e-321
1.235e-321
4.588893417874244e233

I.e. it does not error and, perhaps even more troublesome, it often returns an array full of zeros.
I have once had a bug in my code that came from undef initialization but was extremely rare since the undef’s were usually zero (which caused no issues). It was super annoying to debug because the problem would occur very rarely, randomly and be completely silent.

1 Like

I must be confusing something because I remember to have hit errors of UndefVarError, which is what I had in mind as error. I mean, a numeric equivalent of what we get with Strings.

Vector{String}(undef,2)
2-element Vector{String}:
 #undef
 #undef
1 Like

The difference is that String is an object of variable length (isbits("") gives false) and thus a String array only stores references to actual String objects. These references are initialized as “undef”. Float64 on the other have known length (isbits(1.0) is true) and thus are stored directly in the array. That’s why there is no “undef” reference and whatever happens to be in the memory is just interpreted as Float64

2 Likes