Deploying julia packages once in the cloud (google dataflow) (pipeline)

General Usage
3

I’m using v0.5.2, whereas, we are working in adapting code to v0.6.2

The errors I’m obtaining are:
ERROR: LoadError: ArgumentError: Module ODBC not found in current path. Run Pkg.add(“ODBC”) to install the ODBC package. in require(::Symbol) at ./loading.jl:365 in include_from_node1(::String) at ./loading.jl:488 in process_options(::Base.JLOptions) at ./client.jl:265 in _start() at ./client.jl:321 while loading /usr/local/lib/python2.7/dist-packages/config/julia.jl, in expression starting on line 21

To contextualize a little bit more:
I’m working with Google Cloud Dataflow, more precisely with Python SDK. So, I have created a Pipeline which takes data as input, a ParDo(Python coded) transofrms data and this data is used in de model.jl
To be able to use my scripts Julia in the cloud:

  1. I have installed Julia via a ParDo function of my pipeline (using subprocess with curl, tar, cp,…)
s = Popen(['curl', 'https://julialang-s3.julialang.org/bin/linux/x64/0.5/julia-0.5.2-linux-x86_64.tar.gz',
                   '-o', '/tmp/julia.tar.gz'], stdin=PIPE, stdout=PIPE, bufsize=1)
        logging.info("***********************")
        logging.info(s.stdout.read())
        logging.info("***********************")

        s = Popen(['tar', 'xzf', '/tmp/julia.tar.gz', '-C', '/tmp'],
                  stdin=PIPE, stdout=PIPE, bufsize=1)
        logging.info("***********************")
        logging.info(s.stdout.read())
        logging.info("***********************")
  1. I have uploaded all my Julia files and libraries via the creation of a Python Package and using setup.py to set up my VM python environnement.
  2. I need to be able to use Julia Packages in the cloud (where each time I run my model a new VM turns on (and installs Julia) and each time models turns off all installatin deseapears). However, I’m not getting to install the Julia Pacakges within the Julia Installation process, so I can’t not use the model)

I hope it’s now clearer.
Guillem