# Parquet on M1

**URL:** <https://discourse.julialang.org/t/parquet-on-m1/83323>\
**Category:** General Usage\
**Created:** [June 24, 2022, 10:00pm UTC](https://discourse.julialang.org/t/parquet-on-m1/83323 "2022-06-24T22:00:42Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![JackStrauss](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JackStrauss](https://discourse.julialang.org/u/JackStrauss)\
**Post date:** [June 24, 2022, 10:00pm UTC](https://discourse.julialang.org/t/parquet-on-m1/83323/1 "2022-06-24T22:00:42Z")

</div>

Hello Everybody

Parquet files are great, and we happily read and write them in Julia on Linux.

But not on a Mac M1, there are 2 competing libraries, [Expanding Man / Parquet2.jl · GitLab](https://gitlab.com/ExpandingMan/Parquet2.jl) and [GitHub - JuliaIO/Parquet.jl: Julia implementation of Parquet columnar file format reader](https://github.com/JuliaIO/Parquet.jl) both of which depend on snappy which does not work on a M1. Previously discussed [M1 Mac Faile to install BinaryProvider Package - #2 by giordano](https://discourse.julialang.org/t/m1-mac-faile-to-install-binaryprovider-package/74103/2)

We solved the problem by using RCall to get R to read parquet files and then send them to Julia, but that is very clumsy and slow.

So was just wondering if there was some other Parquet implementation in Julia that works on M1?

best, Jack.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [June 24, 2022, 10:20pm UTC](https://discourse.julialang.org/t/parquet-on-m1/83323/2 "2022-06-24T22:20:26Z")

</div>

You’ll have to ask @ExpandingMan what’s the problem with libsnappy 1.1.9: [updated to use Clang.jl... still baffling errors by ExpandingMan · Pull Request #33 · JuliaIO/Snappy.jl · GitHub](https://github.com/JuliaIO/Snappy.jl/pull/33#issuecomment-1083462863). Also, Parquet.jl should allow Snappy 0.4 at  
[Parquet.jl/Project.toml at ab6d68278713b400465b9d294736c89dfbc30f35 · JuliaIO/Parquet.jl · GitHub](https://github.com/JuliaIO/Parquet.jl/blob/ab6d68278713b400465b9d294736c89dfbc30f35/Project.toml#L32)

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [June 24, 2022, 10:28pm UTC](https://discourse.julialang.org/t/parquet-on-m1/83323/3 "2022-06-24T22:28:03Z")

</div>

> [@giordano](#):
>
> You’ll have to ask @ExpandingMan what’s the problem with libsnappy 1.1.9

Also because if I dev Snappy.jl to allow snappy\_jll 1.1.9 and dev Parquet to allow Snappy 0.4.0, then tests of both Snappy and Parquet are successful for me on M1:

```julia
     Testing Running tests...
Test Summary: | Pass Total Time
parquet tests | 3910 3910 21.3s
     Testing Parquet tests passed

[...]

     Testing Running tests...
Test Summary: | Pass Total Time
Low-level Interfaces | 54 54 0.1s
Test Summary: | Pass Total Time
High-level Interfaces | 113 113 0.7s
     Testing Snappy tests passed

```

So I have no idea what was so catastrophically wrong.

---

<div class="post-metadata">

**Author:** ![JackStrauss](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JackStrauss](https://discourse.julialang.org/u/JackStrauss)\
**Post date:** [June 25, 2022, 8:55am UTC](https://discourse.julialang.org/t/parquet-on-m1/83323/4 "2022-06-25T08:55:18Z")

</div>

Thanks @giordano

yes I take it one can dev it, but is a bit worrying that one needs to resort to such tricks for production data flow, and manually making it happen.  
But we will try it on one machine.

best, jack

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [June 25, 2022, 9:05am UTC](https://discourse.julialang.org/t/parquet-on-m1/83323/5 "2022-06-25T09:05:36Z")

</div>

I opened a PR to use snappy\_jll 1.1.9: [Allow installation of `snappy_jll` 1.1.9 by giordano · Pull Request #35 · JuliaIO/Snappy.jl · GitHub](https://github.com/JuliaIO/Snappy.jl/pull/35). Tests are indeed failing on Linux, but only on that platform. And only a single test. I also have zero knowledge of the package and less than zero interest in pursuing a fix, I never used it and have no need for it.

---

<div class="post-metadata">

**Author:** ![JackStrauss](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JackStrauss](https://discourse.julialang.org/u/JackStrauss)\
**Post date:** [June 25, 2022, 11:28am UTC](https://discourse.julialang.org/t/parquet-on-m1/83323/6 "2022-06-25T11:28:24Z")

</div>

@giordano thanks!

I can see that. Seems unfortunate that such important data science package as reading and writing Parquet files depend on packages that are so poorly maintained. I wish we knew more Julia and hence be able to rectify this.

@bkamins What do you think?

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [June 25, 2022, 11:36am UTC](https://discourse.julialang.org/t/parquet-on-m1/83323/7 "2022-06-25T11:36:17Z")

</div>

Despite my zero interest in snappy, I’m digging into it. I believe it’s an upstream bug in [GitHub - google/snappy: A fast compressor/decompressor](https://github.com/google/snappy) which has been fixed in the development version of the library.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [June 25, 2022, 11:44am UTC](https://discourse.julialang.org/t/parquet-on-m1/83323/8 "2022-06-25T11:44:18Z")

</div>

Yes, I can confirm using [this upstream patch](https://github.com/google/snappy/commit/8dd58a519f79f0742d4c68fbccb2aed2ddb651e8) solves the issue for me on Linux.

---

<div class="post-metadata">

**Author:** ![isaacsas](https://avatars.discourse-cdn.com/v4/letter/i/f6c823/32.png) [@isaacsas](https://discourse.julialang.org/u/isaacsas)\
**Post date:** [June 25, 2022, 11:47am UTC](https://discourse.julialang.org/t/parquet-on-m1/83323/9 "2022-06-25T11:47:38Z")

</div>

It seems strange that any open source project that wants to work across multiple platforms would make Snappy a required dependency given this statement on the Snappy lib Github:

> We are unlikely to accept contributions to the build configuration files, such as `CMakeLists.txt`. We are focused on maintaining a build configuration that allows us to test that the project works in a few supported configurations inside Google. We are not currently interested in supporting other requirements, such as different operating systems, compilers, or build systems.

That seems pretty explicit that there is going to be limited support for this compression library for non-Google users…

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [June 25, 2022, 12:55pm UTC](https://discourse.julialang.org/t/parquet-on-m1/83323/10 "2022-06-25T12:55:44Z")

</div>

> [@isaacsas](#):
>
> That seems pretty explicit that there is going to be limited support for this compression library for non-Google users…

@ExpandingMan - do we need to depend on Snappy?

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [June 25, 2022, 1:48pm UTC](https://discourse.julialang.org/t/parquet-on-m1/83323/11 "2022-06-25T13:48:25Z")

</div>

> [@bkamins](#):
>
> @ExpandingMan - do we need to depend on Snappy?

In short: yes.

The compressed buffers are buried pretty deep in the parquet format, it’s not just a matter of compressing an entire file or even compressing the entire file minus a header.

> [@giordano](#):
>
> Yes, I can confirm using [this upstream patch](https://github.com/google/snappy/commit/8dd58a519f79f0742d4c68fbccb2aed2ddb651e8) solves the issue for me on Linux.

This is great, thanks!

Tests are passing for me now on Parquet2 with snappy 1.1.9. Presumably you can now use it on any architecture that 1.1.9 is built for.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [June 25, 2022, 1:52pm UTC](https://discourse.julialang.org/t/parquet-on-m1/83323/12 "2022-06-25T13:52:47Z")

</div>

Fixed by

- [[snappy] Add upstream patch to fix compilation of asm statements by giordano · Pull Request #5067 · JuliaPackaging/Yggdrasil · GitHub](https://github.com/JuliaPackaging/Yggdrasil/pull/5067)
- [Allow installation of `snappy_jll` 1.1.9 by giordano · Pull Request #35 · JuliaIO/Snappy.jl · GitHub](https://github.com/JuliaIO/Snappy.jl/pull/35)
- [Add compat for `Snappy` v0.4 by giordano · Pull Request #166 · JuliaIO/Parquet.jl · GitHub](https://github.com/JuliaIO/Parquet.jl/pull/166)