Package for genome graphics?

question

#1

Are there any graphing capabilities in Julia 1.x specifically for genomics (“genome graphs”), for example a genome viewer?

Ben Ward illustrated the Julia package GenomeBrowsers in his presentation at JuliaCon 2018. It has not been tested on Julia 1.x so perhaps it is unmaintained? I was unable to install it on Julia 1.1:

(v1.1) pkg> add GenomeBrowsers
  Updating registry at `C:\Users\aalexandersson\.julia\registries\General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
ERROR: The following package names could not be resolved:
 * GenomeBrowsers (not found in project, manifest or registry)
Please specify by known `name=uuid`.

The most promising package I found is the Python package genomeview. There are Python (3.3 or higher) installation instructions. There is a bioRxiv preprint paper. The package is downloadable from PyPi. I assume Julia 1.x could call the Python package (when it is installed) but I have not tried it.


#2

Hi Alex. Ben will be along in a moment I am sure.
One note though - you can add unregistered packages for development and test. You can also add specific git branches too.

add https://github.com/BioJulia/GenomeBrowsers.jl #master

However you are correct - GenomeBrowsers has a dependency on HttpParser which cannot be satisfied.
Over to Ben!


#3

Hi everyone,

Alex, yes you could use PyCall with genomeview, that would probably work. I don’t think GenomeBrowsers has a proper release yet, mostly because we haven’t settled on a system for doing visualisation with BioJulia… Will we use custom plots recipes? Makie? Will we wrap JS elements up with WebIO? all of these things? How will the functionality be organised? So it’s really a stand alone example of how an existing JS project can be integrated with BioJulia. But you should still be able to install it as John points out, so I will have to have a look at why HttpParser was not updated, or what do do about it, perhaps there is a HttpParser replacement GenomeBrowsers could use instead.


#4

@johnh @Ward9250
Thank you, John and Ben.

I can install the genomeview dependency pyBigWig in Linux (but not in Windows), with these commands:

using PyCall
using Conda
Conda.add_channel("Bioconda")
Conda.add("pyBigWig") # Windows produces error "failed process"

I am unable to use PyCall with genomeview because genomeview is a PyPi package, see Conda issue 50.

Please share if you come up with a Julia 1.x solution for genome graphics – whether the Python package genomeview or the Julia package GenomeBrowsers or something else.

Anders


#5

https://github.com/orangeSi/ClustersPloter this is not a package, just a tool to plot genome features(gene or reads or other) by perl.


#6

If you haven’t installed anything important in Conda.jl’s environment and you are ready to wipe it off whenever something bad happens [^1], you can try:

using PyCall
run(`$(PyCall.pyprogramname) -m pip install genomeview`)

as I noted in the issue you linked.

[^1]: There is also experimental conda-pip interop. https://www.anaconda.com/conda-4-6-release/


#7

I happen to have something related to this! I’ve quickly added an example and some documentation and made the repository public:

I’ve been developing this entirely to suit my own needs until now so it might not have the functionalities you are looking for, but over time I’m hoping to make it more general. Feel free to ask, or add it yourself, if you miss anything.


#8

Thank you. What exactly do you suggest to type and run? I am a Julia beginner, so I most likely made at least one beginner’s mistake somewhere. I have a PC with Windows and a laptop with Linux. If I first run

module Pip

as you commented in the Conda issue 50 the result is the same error on both computers:

syntax: incomplete: premature end of input

Then, when I run

using PyCall
run(`$(PyCall.pyprogramname) -m pip install genomeview`)

I get two different error messages. I get this very verbose error message on my Windows machine:

Collecting genomeview
  Using cached https://files.pythonhosted.org/packages/81/34/db88cf0c01f9dc78da709e55ae6c7f271608e19f1ec72c851cd85926bf0a/genomeview-1.0.1.tar.gz
    Complete output from command python setup.py egg_info:
    '.' is not recognized as an internal or external command,
    operable program or batch file.
    '.' is not recognized as an internal or external command,
    operable program or batch file.
    make: ./version.sh: Command not found
    make: ./version.sh: Command not found
    # pysam: no cython available - using pre-compiled C
    # pysam: htslib mode is shared
    # pysam: HTSLIB_CONFIGURE_OPTIONS=None
    # pysam: htslib configure options: None
    # pysam: htslib_config LDFLAGS=
    # pysam: htslib_config LIBHTS_OBJS=kfunc.o knetfile.o kstring.o bcf_sr_sort.o bgzf.o errmod.o faidx.o hfile.o hfile_net.o hts.o hts_os.o md5.o multipart.o probaln.o realn.o regidx.o sam.o synced_bcf_reader.o vcf_sweep.o tbx.o textutils.o thread_pool.o vcf.o vcfutils.o cram/cram_codecs.o cram/cram_decode.o cram/cram_encode.o cram/cram_external.o cram/cram_index.o cram/cram_io.o cram/cram_samtools.o cram/cram_stats.o cram/files.o cram/mFILE.o cram/open_trace_file.o cram/pooled_alloc.o cram/rANS_static.o cram/sam_header.o cram/string_alloc.o
    # pysam: htslib_config LIBS=-lz -lm -lbz2 -llzma
    # pysam: htslib_config PLATFORM=MSYS_NT-6.1-WOW
    # pysam: config_option: ENABLE_PLUGINS=0
    # pysam: config_option: HAVE_COMMONCRYPTO=0
    # pysam: config_option: HAVE_GMTIME_R=0
    # pysam: config_option: HAVE_HMAC=0
    # pysam: config_option: HAVE_IRODS=0
    # pysam: config_option: HAVE_LIBCURL=0
    # pysam: config_option: HAVE_MMAP=0
    package init file 'samtools\__init__.py' not found (or not a regular file)
    package init file 'bcftools\__init__.py' not found (or not a regular file)
    package init file 'samtools\win32\__init__.py' not found (or not a regular file)
    package init file 'htslib\__init__.py' not found (or not a regular file)
    package init file 'htslib\htslib\__init__.py' not found (or not a regular file)
    warning: no files found matching 'KNOWN_BUGS'
    warning: no files found matching 'THANKS'
    warning: manifest_maker: MANIFEST.in, line 15: path 'tests/' cannot end with '/'

    warning: no files found matching 'samtools\configure'
    warning: no files found matching 'samtools\config.mk.in'
    warning: no files found matching 'samtools\config.h.in'
    libchtslib.c
    C:\Users\AALEXA~1\AppData\Local\Temp\easy_install-2o4py0hv\pysam-0.15.2\htslib\htslib/kstring.h(152): warning C4267: 'return': conversion from 'size_t' to 'int', possible loss of data
    C:\Users\AALEXA~1\AppData\Local\Temp\easy_install-2o4py0hv\pysam-0.15.2\htslib\htslib/kstring.h(183): warning C4267: 'return': conversion from 'size_t' to 'int', possible loss of data
    C:\Users\AALEXA~1\AppData\Local\Temp\easy_install-2o4py0hv\pysam-0.15.2\htslib\htslib/kstring.h(191): warning C4146: unary minus operator applied to unsigned type, result still unsigned
    C:\Users\AALEXA~1\AppData\Local\Temp\easy_install-2o4py0hv\pysam-0.15.2\htslib\htslib/kstring.h(220): warning C4146: unary minus operator applied to unsigned type, result still unsigned
    c:\users\aalexandersson\appdata\local\temp\easy_install-2o4py0hv\pysam-0.15.2\htslib\htslib\hts.h(810): warning C4244: 'return': conversion
from 'int64_t' to 'int', possible loss of data
    C:\Users\AALEXA~1\AppData\Local\Temp\easy_install-2o4py0hv\pysam-0.15.2\htslib\htslib/hfile.h(123): warning C4244: 'return': conversion from '__int64' to 'off_t', possible loss of data
    C:\Users\AALEXA~1\AppData\Local\Temp\easy_install-2o4py0hv\pysam-0.15.2\htslib\htslib/hfile.h(245): warning C4018: '<': signed/unsigned mismatch
    C:\Users\AALEXA~1\AppData\Local\Temp\easy_install-2o4py0hv\pysam-0.15.2\htslib\htslib/bgzf.h(35): fatal error C1083: Cannot open include file: 'zlib.h': No such file or directory
    Traceback (most recent call last):
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\_msvccompiler.py", line 423, in compile
        self.spawn(args)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\_msvccompiler.py", line 542, in spawn
        return super().spawn(cmd)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\ccompiler.py", line 909, in spawn
        spawn(cmd, dry_run=self.dry_run)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\spawn.py", line 38, in spawn
        _spawn_nt(cmd, search_path, dry_run=dry_run)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\spawn.py", line 81, in _spawn_nt
        "command %r failed with exit status %d" % (cmd, rc))
    distutils.errors.DistutilsExecError: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\command\bdist_egg.py", line 172, in run
        cmd = self.call_command('install_lib', warn_dir=0)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\command\bdist_egg.py", line 158, in call_command
        self.run_command(cmdname)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\command\install_lib.py", line 11, in run
        self.build()
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\command\install_lib.py", line 107, in build
        self.run_command('build_ext')
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\command\build_ext.py", line 78, in run
        _build_ext.run(self)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\command\build_ext.py", line 339, in run
        self.build_extensions()
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\command\build_ext.py", line 448, in build_extensions
        self._build_extensions_serial()
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\command\build_ext.py", line 473, in _build_extensions_serial
        self.build_extension(ext)
      File "C:\Users\AALEXA~1\AppData\Local\Temp\easy_install-2o4py0hv\pysam-0.15.2\cy_build.py", line 86, in build_extension
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\command\build_ext.py", line 199, in build_extension
        _build_ext.build_extension(self, ext)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\command\build_ext.py", line 533, in build_extension
        depends=ext.depends)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\_msvccompiler.py", line 425, in compile
        raise CompileError(msg)
    distutils.errors.CompileError: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\sandbox.py", line 154, in save_modules
        yield saved
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\sandbox.py", line 195, in setup_context
        yield
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "C:\Users\AALEXA~1\AppData\Local\Temp\easy_install-2o4py0hv\pysam-0.15.2\setup.py", line 439, in <module>
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\core.py", line 163, in setup
        raise SystemExit("error: " + str(msg))
    SystemExit: error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\command\easy_install.py", line 1138, in run_setup
        run_setup(setup_script, args)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\sandbox.py", line 253, in run_setup
        raise
      File "C:\Users\aalexandersson\.julia\conda\3\lib\contextlib.py", line 99, in __exit__
        self.gen.throw(type, value, traceback)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\sandbox.py", line 195, in setup_context
        yield
      File "C:\Users\aalexandersson\.julia\conda\3\lib\contextlib.py", line 99, in __exit__
        self.gen.throw(type, value, traceback)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\sandbox.py", line 166, in save_modules
        saved_exc.resume()
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\sandbox.py", line 141, in resume
        six.reraise(type, exc, self._tb)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\_vendor\six.py", line 685, in reraise
        raise value.with_traceback(tb)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\sandbox.py", line 154, in save_modules
        yield saved
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\sandbox.py", line 195, in setup_context
        yield
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "C:\Users\AALEXA~1\AppData\Local\Temp\easy_install-2o4py0hv\pysam-0.15.2\setup.py", line 439, in <module>
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\distutils\core.py", line 163, in setup
        raise SystemExit("error: " + str(msg))
    SystemExit: error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\AALEXA~1\AppData\Local\Temp\pip-install-g5w7aw1v\genomeview\setup.py", line 48, in <module>
        python_requires=">=3.3"
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\__init__.py", line 128, in setup
        _install_setup_requires(attrs)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\__init__.py", line 123, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\dist.py", line 514, in fetch_build_eggs
        replace_conflicting=True,
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\pkg_resources\__init__.py", line 770, in resolve
        replace_conflicting=replace_conflicting
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\pkg_resources\__init__.py", line 1053, in best_match
        return self.obtain(req, installer)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\pkg_resources\__init__.py", line 1065, in obtain
        return installer(requirement)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\dist.py", line 581, in fetch_build_egg
        return cmd.easy_install(req)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\command\easy_install.py", line 673, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\command\easy_install.py", line 699, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\command\easy_install.py", line 884, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\command\easy_install.py", line 1152, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "C:\Users\aalexandersson\.julia\conda\3\lib\site-packages\setuptools\command\easy_install.py", line 1140, in run_setup
        raise DistutilsError("Setup script exited with %s" % (v.args[0],))
    distutils.errors.DistutilsError: Setup script exited with error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTool\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\AALEXA~1\AppData\Local\Temp\pip-install-g5w7aw1v\genomeview\

I use Julia 1.1.0 on Windows:

julia> versioninfo()
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7 CPU         870  @ 2.93GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, nehalem)
Environment:
  JULIA_EDITOR = "C:\Users\aalexandersson\AppData\Local\atom\app-1.29.0\atom.exe" -a
  JULIA_NUM_THREADS = 4

This is my status on Windows:

(v1.1) pkg> status
    Status `C:\Users\aalexandersson\.julia\environments\v1.1\Project.toml`
  [c52e3926] Atom v0.7.14
  [9e28174c] BinDeps v0.8.10
  [00701ae9] BioAlignments v1.0.0
  [37cfa864] BioCore v2.0.5
  [7e6ae17a] BioSequences v1.1.0
  [159f3aea] Cairo v0.5.6
  [34da2185] Compat v1.5.1
  [8f4d0f93] Conda v1.2.0
  [a93c6f00] DataFrames v0.17.0
  [186bb1d3] Fontconfig v0.2.0
  [c91e804a] Gadfly v1.0.1
  [c43c736e] Genie v0.1.0 #master (https://github.com/essenciary/Genie.jl)
  [00d57c2a] GenomicAnnotations v0.1.0 #master (https://github.com/kdyrhage/GenomicAnnotations.jl.git)
  [db448a0f] GenomicMaps v0.1.0 #master (https://github.com/kdyrhage/GenomicMaps.jl.git)
  [e5e0dc1b] Juno v0.5.4
  [7240a794] Oracle v0.0.5
  [438e738f] PyCall v1.18.5
  [ce6b1742] RDatasets v0.6.1
  [b8865327] UnicodePlots v1.0.1

This is the much shorter error message on my Linux machine, also with Julia 1.1.0:

julia> run(`$(PyCall.pyprogramname) -m pip install genomeview`)
/usr/bin/python3: No module named pip
ERROR: failed process: Process(`/usr/bin/python3 -m pip install genomeview`, ProcessExited(1)) [1]
Stacktrace:
 [1] pipeline_error at ./process.jl:785 [inlined]
 [2] #run#515(::Bool, ::Function, ::Cmd) at ./process.jl:726
 [3] run(::Cmd) at ./process.jl:724
 [4] top-level scope at none:0

#9

module Pip code is a proposal to add this in PyCall.jl. If you want to use it right now, run(`$(PyCall.pyprogramname) -m pip install ...`) is the way to go (where ... part is replaced with appropriate options or arguments).

In Windows it fails while installing pysam. Reading https://github.com/pysam-developers/pysam, it looks like you can install it via conda. If you are using conda environment, it’s always good to install dependencies using conda as much as possible.

In Linux, it looks like you don’t have pip. If you didn’t do any customization, PyCall.pyprogramname should be a path to the system’s Python like "/usr/bin/python3". If that’s the case, install pip for Python 3 using system’s package manager (e.g., in Ubuntu sudo apt-get install python3-pip). After that’s done, you need to invoke pip install with --user; i.e., run(`$(PyCall.pyprogramname) -m pip install --user genomeview`).

Note that all of these are pip usage questions, not Julia questions. We are just invoking command line program with Julia’s run function. You can println(`$(PyCall.pyprogramname) -m pip install ...`), copy-and-paste whatever printed in bash or whatever the system shell you are using, and then execute it outside Julia. You can then ask questions about these commands to the authors of the Python packages.