# Arrow stream usage clarification

**URL:** <https://discourse.julialang.org/t/arrow-stream-usage-clarification/89508>\
**Category:** Data\
**Tags:** dataframes, arrow\
**Created:** [October 30, 2022, 2:27am UTC](https://discourse.julialang.org/t/arrow-stream-usage-clarification/89508 "2022-10-30T02:27:09Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [December 1, 2022, 11:21am UTC](https://discourse.julialang.org/t/arrow-stream-usage-clarification/89508/6 "2022-12-01T11:21:15Z")

</div>

You need `Arrow.write("outputFile.arrow",largerTableOutput, file=false)` to allow for streaming.

However, this is not needed, you can just use `Arrow.append`:

```julia
MassivePartionedTable_Input = Arrow.Stream("inputFile.arrow")
for eachPartition in MassivePartionedTable_Input
  df_eachPartition = DataFrame(eachPartition) 
  largerTableOuput = DoesSomeThingOnThisPartition( df_eachPartition )
  Arrow.append("outputFile.arrow", largerTableOutput)
end

```

Example (input has 2 partitions):

```julia
julia> s = Arrow.Stream("test.arrow")
Arrow.Stream(Arrow.ArrowBlob[Arrow.ArrowBlob(UInt8[0xff, 0xff, 0xff, 0xff, 0xa8, 0x00, 0x00, 0x00, 0x10, 0x00 … 0x00, 0x00, 0xff, 0xff, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00], 1, 600)], 1, nothing, Symbol[], Type[], nothing, Dict{Int64, Arrow.DictEncoding}(), Dict{Int64, Arrow.Flatbuf.Field}(), true, Base.RefValue{Union{Nothing, Symbol}}(nothing))

julia> collect(s)
2-element Vector{Any}:
 Arrow.Table with 1 rows, 2 columns, and schema:
 :a Int64
 :b Int64
 Arrow.Table with 1 rows, 2 columns, and schema:
 :a Int64
 :b Int64

julia> for x in s
           df = DataFrame(x)
           @show df
           Arrow.append("test2.arrow", df)
       end
df = 1×2 DataFrame
 Row │ a b
     │ Int64 Int64
─────┼──────────────
   1 │ 1 2
df = 1×2 DataFrame
 Row │ a b
     │ Int64 Int64
─────┼──────────────
   1 │ 2 3

julia> read("test.arrow") == read("test2.arrow")
true

```

---

_[View the full topic](https://discourse.julialang.org/t/arrow-stream-usage-clarification/89508)._