Time-space trade-off algorithms: memory management


#1

I am currently writing a time-space trade-off library for stacks structures (exchange little time efficiency for huge memory improvement). I write features done either in c++ or Julia, and then port them into the other language.

I am currently porting some technical parts from c++ to Julia that relies mostly on the use of c++ smart pointers. What is the best Julia usage to have a behavior close of smart pointers?


#2

Depends what you want to implement with smart pointers. If it is some application of RAII, sharing some details and perhaps a minimal working example would be helpful.


#3

Thank you for your answer. Actually, the library is open source and available as CompressedStacks.jl. As it is now, it misses a few features to be practical outside some specific problems, and I am trying to optimize the memory management in Julia. Unfortunately, I am not yet familiar enough with the behavior of the GC and what would be equivalent to (smart) pointer.
The general compression system of the library is to keep explicit some of the top data in the stack (per say a vector) and compressed the rest with different levels of compression depending on how likely they are to be accessed. When a compressed data needs to be accessed, the whole compressed block with this data is reconstructed.

So, my problem is that those blocks share some information that can be expensive, and I would rather point to them than store them explicitly. Additionally, I would like to be sure that when an information is not in any block anymore, it should be deleted in memory by the next GC call.
Maybe a minimal example of the actual behavior could be as follow:

mutable struct Block{T1,T2}
  unique::T1
  shareable::T2
end

A = Block("A unique content", "shared content")
B = Block("B unique content", A.shareable)

# Do things with A and B
# Clear A and B (or reassign them to 0 here for the sake of freeing their shared content)
A = 0
B = 0

# Next GC call (here manual for the sake of the example)
gc()

I was mainly wondering if something better could be done to handle the part of shared data within those blocks. I saw a pointer structure Ptr appeared in v0.6. Is there any advantage to use them over field assignation in the Block structure?
If yes, is there any interest to look at smart pointer behavior in Julia (so yes, a kind of RAII)?
My aim is to reduce memory usage, so I want every GC call to be as efficient as possible, wether I call it manually or not.


#4

See https://github.com/JuliaLang/julia/issues/11207