# Julia just... vanished in the middle of the night during a large task. Help troubleshooting/optimising would be appreciated
**URL:** https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834
**Category:** Performance
**Tags:** question, crash
**Created:** [September 14, 2023, 4:40am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834 "2023-09-14T04:40:51Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@MaxM](https://discourse.julialang.org/u/MaxM)
#### Post date: [September 14, 2023, 4:40am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/1 "2023-09-14T04:40:51Z")
I have quite a large task to run and in doing so I’ve encountered unfortunate issues. Below there are two blocks of code, one with a bunch of for loops (attempt 1), the other using functions (attempt 2). I’m running these on a Windows 11 laptop via a Jupyter Notebook opened from IJulia in Anaconda prompt. Attempt 1 had a time counter. It got to 2% within half an hour, but unfortunately it only got to 13% after 8 hours (overnight). The for loop is embarrassingly parallel, so I’m not sure why it would slow down so much with time. I woke up, begin using my computer, and then it ran out of memory (I think due to other memory-intensive programs I was using during the day).
I tried Attempt 2 the second night. 0.1% of the task (changing the value of T) took 15 seconds, so I thought it should complete overnight. However, when I woke up, Julia had… vanished. My computer had signed out of Windows but not restarted (I had set it never to sleep or sign out). Firefox was still open (I was running in a Jupyter Notebook) but the Notebook showed nothing on screen. I tried reopening the Notebook but I couldn’t connect to localhost:8888 - Anaconda prompt (in which I opened Julia) had seemingly shut. I had no error visible or anything - very strange. Julia just vanished mid-task.
I’ve attempted to use functions to write better code in Attempt 2. I’m quite new to programming so might be making significant rookie errors. Any guidance would be appreciated. For a MWE, simply change the value of T to tau or tau+1, tau+2, etc.
ATTEMPT 1:
```julia
using DataFrames, CSV, Dates, StatsBase, ProgressMeter,Statistics
const T,n,tau,M=5788,366,90,1000 #CHANGE T to tau+1
const daily_rf_rate = 0.0;
price_mat=rand(T,n);
empty_array = Array{Float64}(undef, T-tau+1, 91, 3);
# #The main loop is over t.
@showprogress for t in tau:T
for k in 10:100
sharpe_list=Array{Float64}(undef,M)
for i in 1:M
my_sample=sample(1:n,k,replace=false)
drawn_prices=price_mat[t-tau+1:t,my_sample]
# Calculate daily returns for each stock
returns = diff(drawn_prices, dims=1) ./ drawn_prices[1:end-1, :]
# Calculate daily portfolio returns using equal weights (0.1 for each stock since there are 10 stocks)
portfolio_returns = sum(returns, dims=2) * 1/k
# Compute average return and standard deviation for the portfolio
average_portfolio_return = mean(portfolio_returns)
stddev_portfolio_return = std(portfolio_returns)
# Calculate the Sharpe ratio for the portfolio
sharpe_ratio_portfolio = (average_portfolio_return - daily_rf_rate) / stddev_portfolio_return
##empty_array[t,k,i]=sharpe_ratio_portfolio ##I'll save them later
sharpe_list[i]=sharpe_ratio_portfolio
end
empty_array[t-tau+1,k-9,1]=quantile(sharpe_list,0.1); #FILL IN THE QUANTILE VALUES INTO ARRAY
empty_array[t-tau+1,k-9,2]=quantile(sharpe_list,0.5);
empty_array[t-tau+1,k-9,3]=quantile(sharpe_list,0.9);
end
end
```
ATTEMPT 2
```julia
using DataFrames, CSV, Dates, StatsBase, ProgressMeter,Statistics
const T,n,tau,M=5788,366,90,1000 #CHANGE T to tau+1
const daily_rf_rate = 0.0;
price_mat=rand(T,n);
function compute_sharpe(t, k)
sharpe_list = Array{Float64}(undef, M)
for i in 1:M
my_sample = sample(1:n, k, replace=false)
drawn_prices = price_mat[t-tau+1:t, my_sample]
returns = diff(drawn_prices, dims=1) ./ drawn_prices[1:end-1, :]
portfolio_returns = sum(returns, dims=2) * 1/k
average_portfolio_return = mean(portfolio_returns)
stddev_portfolio_return = std(portfolio_returns)
sharpe_ratio_portfolio = (average_portfolio_return - daily_rf_rate) / stddev_portfolio_return
sharpe_list[i] = sharpe_ratio_portfolio
end
return sharpe_list
end
function compute_quantiles(t,k)
sharpe_list=compute_sharpe(t,k)
return [quantile(sharpe_list,0.1), quantile(sharpe_list,0.5), quantile(sharpe_list,0.9)]
end
I=tau:T;J=10:100
Ans_full=compute_quantiles.(I',J);
```
---
### Author:  [@dlakelan](https://discourse.julialang.org/u/dlakelan)
#### Post date: [September 14, 2023, 4:45am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/2 "2023-09-14T04:45:59Z")
> [@MaxM](#):
>
> However, when I woke up, Julia had… vanished. My computer had signed out of Windows but not restarted
I’m guessing actually Windows decided to run an update and restart overnight
---
### Author:  [@MaxM](https://discourse.julialang.org/u/MaxM)
#### Post date: [September 14, 2023, 4:49am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/4 "2023-09-14T04:49:42Z")
> [@dlakelan](#):
>
> I’m guessing actually Windows decided to run an update and restart overnight
No that’s not it, because Firefox wouldn’t be open then, and my settings are do not update automatically.
---
### Author:  [@Sukera](https://discourse.julialang.org/u/Sukera)
#### Post date: [September 14, 2023, 5:08am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/5 "2023-09-14T05:08:47Z")
Out of memory and the Julia process got killed? That also explains the slowdown at first.
---
### Author:  [@MaxM](https://discourse.julialang.org/u/MaxM)
#### Post date: [September 14, 2023, 5:14am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/6 "2023-09-14T05:14:49Z")
> [@Sukera](#):
>
> Out of memory and the Julia process got killed? That also explains the slowdown at first.
Yeah maybe but the final answer I’m filling out is only 5000 x 91 x 3, that’s not too large, right? My computer memory is 8GB and nothing else was running. My code doesn’t seem like it’d be too bad with memory, for example sharpe\_list keeps getting overwritten. Am I doing something wrong with memory?
---
### Author:  [@Salmon](https://discourse.julialang.org/u/Salmon)
#### Post date: [September 14, 2023, 6:25am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/7 "2023-09-14T06:25:56Z")
Yeah these things can be annoying to debug. There can be many reasons but I think it doesn’t make a lot of sense to speculate when it’s possible to get an error message.
To get this it would probably make sense to run the code by starting Julia in the shell instead. That way, even if Julia crashes you will at least be able to see the error.
> [@MaxM](#):
>
> `sharpe_list = Array{Float64}(undef, M)`
It doesn’t look like it’s the source of your problem on first glance but in my opinion one should always have very good reason to use under initializers instead of zeros. If you make a mistake somewhere and don’t write to all positions of the array afterwards you can get very strange bugs indeed. I think it’s better to not use it in most cases.
> [@MaxM](#):
>
> `price_mat=rand(T,n);`
Note that this is a non-const global so there will be a lot of runtime allocations that can easily be avoided by passing this matrix to the function as an argument instead.
---
### Author:  [@Sukera](https://discourse.julialang.org/u/Sukera)
#### Post date: [September 14, 2023, 6:28am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/8 "2023-09-14T06:28:01Z")
I haven’t looked at your code in detail, but at a glance, there’s a number of places that allocate quite a bit:
> [@MaxM](#):
>
> `price_mat[t-tau+1:t, my_sample]`
> [@MaxM](#):
>
> `drawn_prices[1:end-1, :]`
These allocate a copy - slicing arrays in julia copies. Use the `@view` macro to create a lightweight view into the data instead.
Similarly, the broadcast `./` also creates a new array. You can allocate the `returns` array once outside of the loop and reuse it, by writing `returns .= ...`.
> [@MaxM](#):
>
> ` sharpe_list = Array{Float64}(undef, M)`
This leads to a type instability. `Array{Float64}` is an abstract type, and so the sizes on indexing are not known, because the dimensionality is a unknown. Use `Vector{Float64}`, or equivalently `Array{Float64, 1}`, instead.
* * *
I’d also recommend checking the [Performance Tips](https://docs.julialang.org/en/v1/manual/performance-tips/) section of the manual.
---
### Author:  [@Sevi](https://discourse.julialang.org/u/Sevi)
#### Post date: [September 14, 2023, 7:41am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/9 "2023-09-14T07:41:51Z")
I don’t want to belabor this point too much (and reducing the memory consumption of the code is worthwhile in any case), but like @dlakelan mentioned, it does sound a lot like Windows did an update during your second attempt. At least Windows 10 was notorious for doing updates even though it’s seemingly told not to…
> [@MaxM](#):
>
> My computer had signed out of Windows but not restarted (I had set it never to sleep or sign out). Firefox was still open (I was running in a Jupyter Notebook) but the Notebook showed nothing on screen. I tried reopening the Notebook but I couldn’t connect to localhost:8888 - Anaconda prompt (in which I opened Julia) had seemingly shut. I had no error visible or anything - very strange. Julia just vanished mid-task.
If you set it to never sign out, but it actually signed out, it doesn’t seem related to Julia.
On my Windows machine, Firefox automatically re-opens and restores the last sessions after an update is installed. If that’s the case for you as well, it would explain why the notebook was “empty” (the Jupyter server in the background didn’t get restarted).
You could check the event/update logs to see whether it really didn’t restart or not if you want to investigate further. But then again, reducing the memory allocations might be more fruitful.
---
### Author:  [@MaxM](https://discourse.julialang.org/u/MaxM)
#### Post date: [September 14, 2023, 8:31am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/10 "2023-09-14T08:31:35Z")
> [@Sevi](#):
>
> You could check the event/update logs to see whether it really didn’t restart or not if you want to investigate further. But then again, reducing the memory allocations might be more fruitful.
Absolutely, I don’t know what happened exactly, but I’m new to Julia and appreciate all tips to improve the time/space efficiency.
---
### Author:  [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [September 14, 2023, 9:54am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/11 "2023-09-14T09:54:24Z")
> [@MaxM](#):
>
> > [@Sukera](#):
> >
> > Out of memory and the Julia process got killed? That also explains the slowdown at first.
>
> Yeah maybe […] Am I doing something wrong with memory?
AI generated answer to Linux syslog equivalent:
> When a program crashes or runs out of memory on Windows, it is usually logged in the Windows Event Log. […]
>
> You can view the Windows Event Log using the Event Viewer. To do this, press the Windows key + X and select Event Viewer from the menu. Once you have opened Event Viewer, you can navigate to the Windows Logs section and select Application to view application-related events. You can then filter the events by level, source, or date to find the relevant event.
>
> Alternatively, you can write to a text file using a logging framework such as log4cxx. This approach is more pragmatic than using the Windows Event Log, which has a high barrier to entry and is not widely used by applications.
The OS log will not help you about the cause, only confirm, hopefully that the OS killed your program (its only option in the end if you try to use too much memory, you can limit that with CLI option, if possible in Julia 1.10). Julia has a memory profiler you can look into.
> [@MaxM](#):
>
> my settings are do not update automatically.
I didn’t think that was possible in recent Windows (since 10 or earlier) any more. Except allowed in certain enterprise versions. I’m not a Windows user, I think what you have is a suggestion to the OS regarding your preference (maybe it’s only respected at certain times, e.g. in the middle of a game?).
If all users could opt into not restarting, then most likely would… and most computers on the Internet would be insecure fast.
---
### Author:  [@MaxM](https://discourse.julialang.org/u/MaxM)
#### Post date: [September 14, 2023, 11:21am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/12 "2023-09-14T11:21:38Z")
> [@Salmon](#):
>
> It doesn’t look like it’s the source of your problem on first glance but in my opinion one should always have very good reason to use under initializers instead of zeros
Oh thanks, I thought empty arrays were the best bet.
> [@Salmon](#):
>
> Note that this is a non-const global so there will be a lot of runtime allocations that can easily be avoided by passing this matrix to the function as an argument instead.
price\_mat is generated by a previous step, so exists in memory as a global. So you’re saying I should just modify the function to take a matrix argument A and then call it on price\_mat?
---
### Author:  [@joa-quim](https://discourse.julialang.org/u/joa-quim)
#### Post date: [September 14, 2023, 11:24am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/13 "2023-09-14T11:24:30Z")
> [@MaxM](#):
>
> Oh thanks, I thought empty arrays were the best bet.
I actually find that empty (undef) arrays are preferable. If some elements are not filled and later you get an error, that a very good source of information that your code had a bug since the intention was to fill all elements.
---
### Author:  [@MaxM](https://discourse.julialang.org/u/MaxM)
#### Post date: [September 14, 2023, 11:51am UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/14 "2023-09-14T11:51:42Z")
> [@Sukera](#):
>
> I’d also recommend checking the [Performance Tips](https://docs.julialang.org/en/v1/manual/performance-tips/) section of the manual.
Thanks for your tips above - I did read this part of the manual before posting (it informed things like using const for the variables) but I’m new to Julia so most of it was difficult to understand. I’ve changed the code to allocate `returns` outside the inner loop, but its length changes with k, so I still have to allocate it 91 times. Thanks for your tip on `Vector` too - I had confused that with `Array`. Regarding your tip with the `@view` macro - do you also mean assigning an array `drawn_prices` first and then filling it with the following?
`draw_prices .=@view price_mat[t-tau+1:t,my_sample]`
---
### Author:  [@Salmon](https://discourse.julialang.org/u/Salmon)
#### Post date: [September 14, 2023, 12:42pm UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/15 "2023-09-14T12:42:07Z")
You should better fill it with NaN’s then in my opinion. In most cases, using undef only means that you take whatever state the allocated memory is in and interpret that to be your datatype:
```julia
function UseUndef(N)
a = Vector{Float64}(undef,N)
return sum(a)
end
for i in 1:10
println(UseUndef(5))
end
```
which printed for me
```julia
1.390198155611988e-308
1.390198008661865e-308
NaN
1.6e-321
1.51e-321
1.453e-321
1.45e-321
1.393e-321
1.235e-321
4.588893417874244e233
```
I.e. it does not error and, perhaps even more troublesome, it often returns an array full of zeros.
I have once had a bug in my code that came from undef initialization but was extremely rare since the undef’s were usually zero (which caused no issues). It was super annoying to debug because the problem would occur very rarely, randomly and be completely silent.
---
### Author:  [@joa-quim](https://discourse.julialang.org/u/joa-quim)
#### Post date: [September 14, 2023, 12:57pm UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/16 "2023-09-14T12:57:49Z")
I must be confusing something because I remember to have hit errors of `UndefVarError`, which is what I had in mind as error. I mean, a numeric equivalent of what we get with Strings.
```julia
Vector{String}(undef,2)
2-element Vector{String}:
#undef
#undef
```
---
### Author:  [@abraemer](https://discourse.julialang.org/u/abraemer)
#### Post date: [September 14, 2023, 1:07pm UTC](https://discourse.julialang.org/t/julia-just-vanished-in-the-middle-of-the-night-during-a-large-task-help-troubleshooting-optimising-would-be-appreciated/103834/17 "2023-09-14T13:07:21Z")
The difference is that `String` is an object of variable length (`isbits("")` gives false) and thus a `String` array only stores references to actual `String` objects. These references are initialized as “undef”. `Float64` on the other have known length (`isbits(1.0)` is true) and thus are stored directly in the array. That’s why there is no “undef” reference and whatever happens to be in the memory is just interpreted as `Float64`