I’m developing my first Julia package Paillier.jl and wanted to know if I’m on the right track for handling an optional rng? All through the library should I be creating fallback function signatures that create a default rng if not given - like this:
generate_paillier_keypair(n_length=2048) = generate_paillier_keypair(RandomDevice(), n_length)
function generate_paillier_keypair(rng::AbstractRNG, n_length=2048)
Many thanks
I think that sounds reasonable, and is how it’s done within Random as well: source here.
The duplication is not that pretty IMO. There are a few ways to get around it. One is to use keyword arguments to make both arguments optional so that you only need a single method. Another is to use macros/generated code (at the cost of less readable code). Finally, do what you’re doing now, but extract repeated constants/logic:
const DEFAULT_KEY_LENGTH = 2048
default_rng() = RandomDevice()
generate_paillier_keypair(n_length = DEFAULT_KEY_LENGTH) =
generate_paillier_keypair(default_rng(), n_length)
function generate_paillier_keypair(rng::AbstractRNG, n_length = DEFAULT_KEY_LENGTH)
Btw, is it intended to create a new instance of RandomDevice for each call?
Thanks for the advice. I think I’ll copy your example with the extracted logic.
Good question about creating new instances of RandomDevice() - I’m thinking no. On linux it looks to be a wrapper around /dev/urandom (which is what I want), and I would assume that just having one open file would be better.
I ran into a weird behavior when trying to use a single RandomDevice. This code freezes/locks up while trying to run my unittests:
If I have the code:
default_rng = RandomDevice()
generate_paillier_keypair(n_length=DEFAULT_KEY_LENGTH) = generate_paillier_keypair(default_rng, n_length)
function generate_paillier_keypair(rng::AbstractRNG, n_length=DEFAULT_KEY_LENGTH)
So for now I’ll go back to create a new RandomDevice instance each call, but it would be good to understand what is going on.
Unless you want to dispatch on the type of various RNGs, I would just use keyword arguments.
Hmm. Strange. Yes, it’d be interesting to know what causes that. Perhaps you could try to create a minimal reproducible example, which in the best case will help you find what causes it, or at least you can post it here for us to test.
Are you using multiple threads by any chance? That will cause problems if using the same RNG:
julia> rng = RandomDevice()
RandomDevice(IOStream(<file /dev/urandom>), true)
julia> Threads.@threads for n = 1:100000 rand(rng) end
signal (11): Segmentation fault: 11
in expression starting at no file:0
jl_ios_get_nbyte_int at /Users/osx/buildbot/slave/package_osx64/build/src/sys.c:331
read at ./iostream.jl:402 [inlined]
rand at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.2/Random/src/RNGs.jl:29 [inlined]
...
Does constructing a new random device open /dev/urandom as a file? And are you perhaps doing this a very large number of times? If so, then perhaps your operating system is running out of file descriptors.
It does:
julia> @btime RandomDevice()
ERROR: SystemError: opening file "/dev/urandom": Too many open files
But if I understand OP correctly, he’s saying that the problem occurs when he’s not constructing a new RandomDevice for each call (i.e. when reusing the same RNG).
Yeah that is the issue I ran into. I’ve tried to make a minimal demonstration but couldn’t recreate the issue until I started playing with the include statement.
My tests were running examples as well as unit tests - to ensure I didn’t break them:
# File: tests/runtests.jl
include("../src/Paillier.jl")
using Main.Paillier
...
@testset "Test raw_cryptosystem.jl example" begin
@capture_stdout include("../examples/raw_cryptosystem.jl")
@test 70 == decrypt(priv, c)
end
And the examples themselves were importing the Paillier module (not Main.Paillier) like the tests.
# file examples/encoded_numbers.jl
import Paillier
keysize = 2048
publickey, privatekey = Paillier.generate_paillier_keypair(keysize)
...
I was also running the tests from InteliJ using the -L src/Paillier.jl argument. When I change the runtests.jl file to using Paillier and don’t use the -L I get the same locking behavior (with no output).
So I guess I jumped to the /dev/urandom file but it is probably that I’m making a beginner mistake with my imports etc. Anyone able to point me in the right direction? How do people usually write and test examples as well as tests for a Julia library?
Many thanks for the help so far!
It should never be necessary for your test or example files to include("../src/Paillier.jl"), and you should not need to do -L src/Paillier.jl either.
Instead, since you have already created a nice Project.toml file, you should be able to load your package (via import Paillier or using Paillier) whenever you have that Project active. The way you activate a project in Julia is:
cd /whatever/Paillier
julia --project
or, if you are already in Julia:
pkg> activate /whatever/Paillier
(where you get the pkg> prompt by pressing ] ).
I would recommend trying all of this in a terminal for now. I’m not sure what exactly intelliJ is doing, and it will be easier to save debugging that part of the puzzle for later. In a terminal, you should be able to do:
cd /whatever/Paillier
julia --project
julia> using Paillier
If that doesn’t work, I would remove all your includes until you just have a plain Paillier/src/Paillier.jl file with just module Paillier in it and then add your includes one by one.
Assuming that works, you should be able to change your runtests.jl to just do:
using Paillier # no include required
then you can try it with:
cd /whatever/Paillier
julia --project
pkg> test Paillier
(note that you don’t actually need to do the cd and julia --project steps every time. You can just keep the same Julia terminal open. I’m including them every time for completeness).
Assuming that works, you should be able to change your examples to also do:
using Paillier
(or import Paillier if you prefer. There’s no difference in the way using and import look for packages).
And you should be able to run your examples with:
cd /whatever/Paillier
julia --project example.jl
Once that’s all working, we can figure out how IntelliJ factors into it.
Thanks so much for your help understanding this! Good idea to remove IntelliJ from the equation for now.
I’ve taken your advice with removing the -L and includes, and now I’m back to suspecting an issue with the RandomDevice is involved. If I run an example it still locks up but on Ctrl-C outputs the following:
[brian@hardbyte-nzxt Paillier.jl]$ julia --project examples/raw_cryptosystem.jl
^C
signal (2): Interrupt
in expression starting at /home/brian/dev/Paillier.jl/examples/raw_cryptosystem.jl:3
__select at /usr/bin/../lib/libc.so.6 (unknown line)
unknown function (ip: 0x7f60641025eb)
unknown function (ip: 0x7f6064104c9d)
ios_readprep at /usr/bin/../lib/libjulia.so.1 (unknown line)
jl_ios_get_nbyte_int at /usr/bin/../lib/libjulia.so.1 (unknown line)
read at ./iostream.jl:402 [inlined]
rand at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/RNGs.jl:29 [inlined]
rand! at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/Random.jl:235 [inlined]
rand! at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/Random.jl:231 [inlined]
rand! at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/Random.jl:231 [inlined]
macro expansion at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/generation.jl:325 [inlined]
macro expansion at ./gcutils.jl:87 [inlined]
rand at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/generation.jl:322
rand at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/Random.jl:219 [inlined]
n_bit_random_number at /home/brian/dev/Paillier.jl/src/utilities.jl:9
nbit_prime_of_size at /home/brian/dev/Paillier.jl/src/utilities.jl:17 [inlined]
generate_paillier_keypair at /home/brian/dev/Paillier.jl/src/keygeneration.jl:20
unknown function (ip: 0x7f603e89edd7)
jl_fptr_trampoline at /usr/bin/../lib/libjulia.so.1 (unknown line)
jl_apply_generic at /usr/bin/../lib/libjulia.so.1 (unknown line)
generate_paillier_keypair at /home/brian/dev/Paillier.jl/src/keygeneration.jl:10
unknown function (ip: 0x7f603e89e773)
jl_fptr_trampoline at /usr/bin/../lib/libjulia.so.1 (unknown line)
jl_apply_generic at /usr/bin/../lib/libjulia.so.1 (unknown line)
unknown function (ip: 0x7f606411c797)
unknown function (ip: 0x7f606411c540)
unknown function (ip: 0x7f606411d508)
unknown function (ip: 0x7f606411dbe4)
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f6054a0a3ff)
unknown function (ip: (nil))
unknown function (ip: 0x7f606411e0ac)
unknown function (ip: 0x7f6063ffa3ac)
unknown function (ip: 0x7f6063fd255e)
jl_load at /usr/bin/../lib/libjulia.so.1 (unknown line)
unknown function (ip: 0x7f6058d4da32)
unknown function (ip: 0x7f6058d58b5b)
jl_apply_generic at /usr/bin/../lib/libjulia.so.1 (unknown line)
unknown function (ip: 0x7f6058d5815a)
unknown function (ip: 0x7f6058d58867)
jl_apply_generic at /usr/bin/../lib/libjulia.so.1 (unknown line)
unknown function (ip: 0x55b57596c6e3)
unknown function (ip: 0x55b57596c0a8)
__libc_start_main at /usr/bin/../lib/libc.so.6 (unknown line)
unknown function (ip: 0x55b57596c15d)
unknown function (ip: 0xffffffffffffffff)
Allocations: 875529 (Pool: 875327; Big: 202); GC: 1
When I change the RandomDevice line I get a working example:
_default_rng = RandomDevice()
# broken line (used for output above)
#default_rng() = _default_rng
# working line (used for output below)
default_rng() = RandomDevice()
[brian@hardbyte-nzxt Paillier.jl]$ julia --project examples/raw_cryptosystem.jl
PublicKey(bits=1024, hash=7225169492579635362)
PrivateKey(hash=3823826835132048832)
decrypt(a) = 10
decrypt(b) = 50
decrypt(a + 5) = 15
decrypt(2a + b) = 70
A similar behavior from the package manager:
(Paillier) pkg> test Paillier
Testing Paillier
Resolving package versions...
^C
ERROR: LoadError: InterruptException:
Stacktrace:
[1] read at ./iostream.jl:402 [inlined]
[2] rand at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/RNGs.jl:29 [inlined]
[3] rand! at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/Random.jl:235 [inlined]
[4] rand! at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/Random.jl:231 [inlined] (repeats
2 times)
[5] macro expansion at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/generation.jl:325 [inlined]
[6] macro expansion at ./gcutils.jl:87 [inlined]
[7] rand(::Random.RandomDevice, ::Random.SamplerBigInt) at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/generation.jl:322
[8] rand at /build/julia/src/julia-1.0.3/usr/share/julia/stdlib/v1.0/Random/src/Random.jl:219 [inlined]
[9] n_bit_random_number(::Random.RandomDevice, ::BigInt) at /home/brian/dev/Paillier.jl/src/utilities.jl:9
[10] nbit_prime_of_size at /home/brian/dev/Paillier.jl/src/utilities.jl:17 [inlined]
[11] generate_paillier_keypair(::Random.RandomDevice, ::Int64) at /home/brian/dev/Paillier.jl/src/keygeneration.jl:20
[12] generate_paillier_keypair(::Int64) at /home/brian/dev/Paillier.jl/src/keygeneration.jl:10
All this code is open source if that helps in any way - https://github.com/hardbyte/Paillier.jl/tree/refactor-rng
I would guess this is a precompilation issue. By creating a single RandomDevice as a global variable in your module, that value is saved during precompilation of the module, but it may not actually be safe to do so (since that device might contain pointers or file handles which are not safe to use in another Julia process). When you define your function to return a new random device each time it’s called, the problem goes away because a new valid random device is constructed each time you ask for one.
If this is the problem, and you still want to share a single random device, then it’s easy to fix: you just need to delay construction of the random device until your module is actually loaded. You do that with an __init__ function inside your module:
module Foo
const _device = Ref{RandomDevice}()
default_rng = _device[]
function __init__()
_device[] = RandomDevice()
end