# Readbytes! is bugging me
**URL:** https://discourse.julialang.org/t/readbytes-is-bugging-me/23396
**Category:** New to Julia
**Tags:** binaryio, data
**Created:** [April 22, 2019, 10:03am UTC](https://discourse.julialang.org/t/readbytes-is-bugging-me/23396 "2019-04-22T10:03:23Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)
#### Post date: [April 22, 2019, 10:03am UTC](https://discourse.julialang.org/t/readbytes-is-bugging-me/23396/1 "2019-04-22T10:03:23Z")
Hey guys
I’ve been trying to read a binary file in which I know there exists n = 3186, Int32 values. The way I’ve done it is like this:
```julia
raw_data = zeros(UInt8,n*4)
readbytes!(filestream, raw_data,n*4)
```
By doing this I get:
```julia
raw_data
12744-element Array{UInt8,1}:
0x00
0x00
0x00
0x1b
0x00
0x00
0x00
0x1c
0x00
0x00
0x00
0x1d
⋮
```
Which is great! The problem arises when I try to reinterpret these numbers, I know that in this array piece I’ve shown you I should get 27, 28 and 29. But something is going wrong with Endian representation or when I use reinterpret because I get:
```julia
reinterpret(Int32,raw_data)
3-element reinterpret(Int32, ::Array{UInt8,1}):
452984832
469762048
486539264
```
I can get the right numbers if I use ‘reverse’ but I have to use it twice, first to get the right numbers, then again to get the right indices - but I feel this is a wrong approach:
```julia
reverse(reinterpret(Int32,reverse(raw_data)))
3-element Array{Int32,1}:
27
28
29
```
There must be an easier way and I assume that reversing twice makes a big performance hit? Currently my code reading line by line is 200 ms faster than readbytes! which seems a bit wrong. Hope someone can help.
Kind regards
---
### Author:  [@tkluck](https://discourse.julialang.org/u/tkluck)
#### Post date: [April 22, 2019, 10:09am UTC](https://discourse.julialang.org/t/readbytes-is-bugging-me/23396/2 "2019-04-22T10:09:17Z")
You’re looking for the `ntoh` function:
```julia
map(ntoh, reinterpret(Int32, raw_data))
```
---
### Author:  [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)
#### Post date: [April 22, 2019, 10:20am UTC](https://discourse.julialang.org/t/readbytes-is-bugging-me/23396/3 "2019-04-22T10:20:59Z")
Thanks, you came in clutch! 🙂
Now I am only 130 ms slower, so will just have to fix my implementation now
Kind regards
---
### Author:  [@tkluck](https://discourse.julialang.org/u/tkluck)
#### Post date: [April 22, 2019, 10:36am UTC](https://discourse.julialang.org/t/readbytes-is-bugging-me/23396/4 "2019-04-22T10:36:23Z")
May want to try in-place as well, using broadcasting:
```julia
data = reinterpret(Int32, raw_data)
data .= ntoh.(data)
```
And just in case : don’t forget to benchmark your code by wrapping it in a function; don’t benchmark at top level in the REPL. All top-level variables have type `Any` so the compiler can’t optimize it very well.
---
### Author:  [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)
#### Post date: [April 22, 2019, 11:06am UTC](https://discourse.julialang.org/t/readbytes-is-bugging-me/23396/5 "2019-04-22T11:06:00Z")
> [@tkluck](#):
>
> data .= ntoh.(data)
I’ve put it into a function and am benchmarking now using your in-line tip. I’ve gotten it down to 330 ms, which is still about 70 ms slower than my reading line by line.
```julia
#Preallocate an array depending on datatype and of chosen size
arrayVal::Array{UInt8,1} = zeros(UInt8,size[1]*4)
readbytes!(fd, arrayVal,sizeof(arrayVal))
#Close the open file
close(fd)
data = reinterpret(Int32,arrayVal)
return ntoh.(data)
end
```
Maybe reading line by line is just superior in Julia instead of reading a whole array of bytes and making operations on it?
Down under; using readbytes!
```julia
@benchmark k = readVtkArray("PartAll",Idp)
BenchmarkTools.Trial:
memory estimate: 477.45 MiB
allocs estimate: 28701
--------------
minimum time: 328.899 ms (0.00% GC)
median time: 342.392 ms (0.00% GC)
mean time: 431.651 ms (20.01% GC)
maximum time: 1.405 s (73.79% GC)
--------------
samples: 12
evals/sample: 1
```
Down under; read line by line
```julia
@benchmark k = readVtkArray("PartAll",Idp)
BenchmarkTools.Trial:
memory estimate: 261.69 MiB
allocs estimate: 28691
--------------
minimum time: 276.426 ms (0.00% GC)
median time: 289.531 ms (0.00% GC)
mean time: 379.285 ms (21.72% GC)
maximum time: 1.744 s (80.30% GC)
--------------
samples: 17
evals/sample: 1
```
Kind regards
---
### Author:  [@tkluck](https://discourse.julialang.org/u/tkluck)
#### Post date: [April 22, 2019, 12:03pm UTC](https://discourse.julialang.org/t/readbytes-is-bugging-me/23396/6 "2019-04-22T12:03:29Z")
Two more things to try:
1. Skip initializing to zero:
```julia
arrayVal = Vector{UInt8}(undef, size[1]*4)
```
1. replace the copy operation `return ntoh.(data)` by changing `data` in-place:
```julia
data .= ntoh.(data)
return data
```
ON top of that, it might be helpful to paste a working example for both of your benchmarks.
---
### Author:  [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)
#### Post date: [April 22, 2019, 2:17pm UTC](https://discourse.julialang.org/t/readbytes-is-bugging-me/23396/7 "2019-04-22T14:17:55Z")
I’ve done your suggestion 1 and can see some marginal improvement, so will keep it, but in this case your 2nd suggestion is slowing things down dramatically. I’ve made it as such:
```julia
#Preallocate an array depending on datatype and of chosen size
arrayVal::Array{UInt8,1} = Vector{UInt8}(undef, size[1]*4)
#readbytes!(fd, arrayVal,sizeof(arrayVal))
read!(fd,arrayVal)
#Close the open file
close(fd)
data = reinterpret(Int32,arrayVal)
data .= ntoh.(data)
return data
```
And now the results are:
```julia
@benchmark k = readVtkArray("PartAll",Idp)
BenchmarkTools.Trial:
memory estimate: 477.90 MiB
allocs estimate: 28706
--------------
minimum time: 534.921 ms (0.00% GC)
median time: 556.051 ms (0.00% GC)
mean time: 719.914 ms (22.26% GC)
maximum time: 2.025 s (71.23% GC)
--------------
samples: 9
evals/sample: 1
```
Which is a major decrease (if I’ve done it correctly). I am trying to make a minimal working example available in my other post ([How fast is binary reading capabilities in Julia compared with other languages? - #5 by Ahmed\_Salih](https://discourse.julialang.org/t/how-fast-is-binary-reading-capabilities-in-julia-compared-with-other-languages/23401/5)), I will try to post it as soon as possible.
Kind regards
---
### Author:  [@tkluck](https://discourse.julialang.org/u/tkluck)
#### Post date: [April 22, 2019, 3:25pm UTC](https://discourse.julialang.org/t/readbytes-is-bugging-me/23396/8 "2019-04-22T15:25:36Z")
Ah, yes, I’m seeing the same:
```julia
julia> function f()
a = Vector{UInt8}(undef, 100_000)
data = reinterpret(Int32, a)
data .= ntoh.(data)
data
end
f (generic function with 1 method)
julia> using BenchmarkTools
julia> @btime f();
284.997 μs (3 allocations: 97.80 KiB)
julia> function g()
a = Vector{UInt8}(undef, 100_000)
data = reinterpret(Int32, a)
return ntoh.(data)
end
g (generic function with 1 method)
julia> @btime g();
14.242 μs (5 allocations: 195.56 KiB)
```
it seems related to the `reinterpret` call. I wonder if it’s actually necessary; it seems this might work too:
```julia
julia> function h()
data = Vector{Int32}(undef, 100_000)
data .= ntoh.(data)
return data
end
h (generic function with 1 method)
julia> @btime h();
19.140 μs (2 allocations: 390.70 KiB)
```
… Okay, that’s still a bit slower. Then I see no reason to take my second suggestion.
---
### Author:  [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)
#### Post date: [April 22, 2019, 3:56pm UTC](https://discourse.julialang.org/t/readbytes-is-bugging-me/23396/9 "2019-04-22T15:56:30Z")
I’ve also arrived at the conclusion that for bigger files, it is much faster to just read every byte/element in a for loop and handle conversion instantly, instead of reading the whole array and then reinterpreting. I still can’t make it make logical sense why this is the case, but I’ve tried a lot of different approaches and have not been able to beat line by line.
Thanks for your time.
Kind regards