# Reading multiple files and also one file containg limits to read other file's content upto

**URL:** <https://discourse.julialang.org/t/reading-multiple-files-and-also-one-file-containg-limits-to-read-other-files-content-upto/115060>\
**Category:** Data\
**Tags:** hdf5, python, io\
**Created:** [June 1, 2024, 4:41pm UTC](https://discourse.julialang.org/t/reading-multiple-files-and-also-one-file-containg-limits-to-read-other-files-content-upto/115060 "2024-06-01T16:41:43Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [June 1, 2024, 4:41pm UTC](https://discourse.julialang.org/t/reading-multiple-files-and-also-one-file-containg-limits-to-read-other-files-content-upto/115060/1 "2024-06-01T16:41:43Z")

</div>

I have python code like:

```julia
from pyfiles import * 
runname='./' #define start dir and file, uncomment below
sane = 1#0 for MAD98, 1 for SANE00
if sane == 1:
    inputfile=runname+'sane00.athinput'
    datafile='sane00.prim.01800.athdf'
    data=read_data(datafile,athinput=inputfile)
d = data.rho # density

```

Here

> **inputfile**
>
> ```julia
> <job>
> problem_id = sane00
> 
> <time>
> cfl_number = 0.25
> nlim = -1
> tlim = 4.0E4
> 
> <output1>
> file_type = hdf5
> variable = prim
> id = prim
> dt = 10.0
> xdmf = false
> 
> <output2>
> file_type = hdf5
> variable = uov
> id = user
> dt = 100.0
> xdmf = false
> 
> <output3>
> file_type = rst
> dt = 1.E4
> 
> <mesh>
> nx1 = 72
> x1min = 1.6
> x1max = 1200.0
> x1rat = 1.0963050292915717
> ix1_bc = user
> ox1_bc = user
> 
> nx2 = 32
> x2min = 0.0
> x2max = 3.141592653589793
> ix2_bc = polar
> ox2_bc = polar
> 
> nx3 = 16
> x3min = 0.0
> x3max = 6.283185307179586
> ix3_bc = periodic
> ox3_bc = periodic
> 
> refinement = static
> 
> <meshblock>
> nx1 = 18
> nx2 = 4
> nx3 = 16
> 
> <refinement1>
> level = 1
> x1min = 1.6
> x1max = 200.0
> x2min = 0.41
> x2max = 2.731592653589793
> x3min = 0.0
> x3max = 6.283185307179586
> 
> <refinement2>
> level = 2
> x1min = 1.6
> x1max = 30.0
> x2min = 0.61
> x2max = 2.531592653589793
> x3min = 0.0
> x3max = 6.283185307179586
> 
> <refinement3>
> level = 1
> x1min = 8.5
> x1max = 1200.0
> x2min = 0.0
> x2max = 0.380
> x3min = 0.0
> x3max = 6.283185307179586
> 
> <refinement4>
> level = 1
> x1min = 8.5
> x1max = 1200.0
> x2min = 2.761592653589793
> x2max = 3.141592653589793
> x3min = 0.0
> x3max = 6.283185307179586
> 
> <refinement5>
> level = 2
> x1min = 44.4
> x1max = 1200.0
> x2min = 0.0
> x2max = 0.18
> x3min = 0.0
> x3max = 6.283185307179586
> 
> <refinement6>
> level = 2
> x1min = 44.4
> x1max = 1200.0
> x2min = 2.961592653589793
> x2max = 3.141592653589793
> x3min = 0.0
> x3max = 6.283185307179586
> 
> <coord>
> m = 1.0
> a = 0.0
> 
> <hydro>
> gamma = 1.3333333333333333
> dfloor = 1.0e-6
> pfloor = 1.0e-8
> rho_min = 1.0e-2
> rho_pow = -1.5
> pgas_min = 1.0e-2
> pgas_pow = -2.5
> sigma_max = 100.0
> beta_min = 1.0e-3
> gamma_max = 50.0
> 
> <problem>
> k_adi = 1.0
> r_edge = 40.5
> r_peak = 80.0
> l = 0.0
> rho_max = 1.0
> 
> field_config = loops
> potential_cutoff = 0.2
> potential_start = 25.0
> potential_end = 550.0
> potential_wavelength = 3.75
> beta_min = 0.05833
> 
> potential_r_pow = 0.0
> potential_rho_pow = 1.0 
> 
> sample_n_r = 288
> sample_n_theta = 128
> sample_r_rat = 1.0232525877105085
> sample_cutoff = 0.2
> 
> pert_amp = 1.00E-1
> pert_kr = 6.28E-2
> pert_kz = 6.28E-2
> 
> ```

is address of `.txt` configuration file that determines the limit to read data of **sane00.prim.01800.athdf** upto.

How to accomplish this in Julia? What is equivalent/alternative of **python** [cis.read\_data()](https://cis.readthedocs.io/en/1.2.0/api/cis.html#cis.read_data) function in Julia?  
@mkitti

---

<div class="post-metadata">

**Author:** ![feanor12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/feanor12/32/8212_2.png) [@feanor12](https://discourse.julialang.org/u/feanor12)\
**Post date:** [June 2, 2024, 1:04pm UTC](https://discourse.julialang.org/t/reading-multiple-files-and-also-one-file-containg-limits-to-read-other-files-content-upto/115060/2 "2024-06-02T13:04:02Z")

</div>

I think for HDF5 files you can use [Home · HDF5.jl](https://docs.juliahub.com/General/HDF5/stable/)

You can modify the read\_data function to take a config file (or better a predefined config type) instead.

```julia
function read_data(file,dataset)
   data = nothing
   h5open(file) do h5
      data = read(h5[dataset])
   end
   data
end

files = readdir("path_to_target",join=true)
data = files |> x->read_data.(x,Ref("prim")) |> x->reduce(vcat,x)

```

The bigger challenge will be writing a parser for the txt file you want to input.

something like this

```julia
struct ReadConfig
   fields::Dict{String,Dict{String,Any}}
end

import Base.parse
function parse(::Type{ReadConfig},filepath)
   rc = ReadConfig(Dict())
   for line in readlines(filepath)
     # parsing logic like:
     # keyDict[keyname] = value
     # rc.fields[fieldname] = keyDict
   end
   rc
end

function read_data(file,config::ReadConfig)
   # adapted logic
end

```

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [June 2, 2024, 4:28pm UTC](https://discourse.julialang.org/t/reading-multiple-files-and-also-one-file-containg-limits-to-read-other-files-content-upto/115060/3 "2024-06-02T16:28:50Z")

</div>

Hi Michael, I got some idea from your post but i still don’t have solution yet.

 ![image](https://global.discourse-cdn.com/julialang/original/3X/9/2/927ec1f25ae99854f0ce5196a2904894d7508fd5.png)

---

<div class="post-metadata">

**Author:** ![feanor12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/feanor12/32/8212_2.png) [@feanor12](https://discourse.julialang.org/u/feanor12)\
**Post date:** [June 2, 2024, 4:56pm UTC](https://discourse.julialang.org/t/reading-multiple-files-and-also-one-file-containg-limits-to-read-other-files-content-upto/115060/4 "2024-06-02T16:56:56Z")

</div>

I think you read in a files that is not a hdf5 file. You can add a filter to the file list.

```julia
file_root = raw"/home/raman/Pictures"
files = readdir(file_root,join=true) |> filter(endswith(".athdf"))

```

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [June 2, 2024, 5:25pm UTC](https://discourse.julialang.org/t/reading-multiple-files-and-also-one-file-containg-limits-to-read-other-files-content-upto/115060/5 "2024-06-02T17:25:35Z")

</div>

There is still error.

 ![image](https://global.discourse-cdn.com/julialang/original/3X/7/6/76790675160d15f1108b05bfd151541d0369d3c3.png)

---

<div class="post-metadata">

**Author:** ![feanor12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/feanor12/32/8212_2.png) [@feanor12](https://discourse.julialang.org/u/feanor12)\
**Post date:** [June 2, 2024, 5:47pm UTC](https://discourse.julialang.org/t/reading-multiple-files-and-also-one-file-containg-limits-to-read-other-files-content-upto/115060/6 "2024-06-02T17:47:42Z")

</div>

Try calling it like that

```julia
dataset_key = "/prim"
filepath = raw"/home/raman/Pictures/sane00.prim.01800.athdf"
read_data(filepath,dataset_key)

```

In case you don’t know the datset\_keys run

```julia
using HDF5
filepath = raw"/home/raman/Pictures/sane00.prim.01800.athdf"
h5open(filepath,"r") do io
  HDF5.show_tree(io)
end

```

to print the internal structure of the file

---

<div class="post-metadata">

**Author:** ![feanor12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/feanor12/32/8212_2.png) [@feanor12](https://discourse.julialang.org/u/feanor12)\
**Post date:** [June 2, 2024, 5:53pm UTC](https://discourse.julialang.org/t/reading-multiple-files-and-also-one-file-containg-limits-to-read-other-files-content-upto/115060/7 "2024-06-02T17:53:12Z")

</div>

Here is another example:

> **Console example on how to read a hdf5 file**
>
> ```julia
> julia> using HDF5
> 
> julia> function read_data(file,dataset)
> data = nothing
> h5open(file) do h5
> data = read(h5[dataset])
> end
> data
> end
> read_data (generic function with 1 method)
> 
> julia> # Generate an example file with one dataset called signal
> 
> julia> file = "temp.h5"
> "temp.h5"
> 
> julia> fid = h5open(file,"w")
> 🗂️ HDF5.File: (read-write) temp.h5
> 
> julia> fid["signal"] = rand(10);
> 
> julia> fid
> 🗂️ HDF5.File: (read-write) temp.h5
> └─ 🔢 signal
> 
> julia> signal = read(fid,"signal")
> 10-element Vector{Float64}:
> 0.013432897038968217
> 0.9148075310100701
> 0.8004580824186582
> 0.7207654878373057
> 0.8584511594948295
> 0.7538087575791116
> 0.47729849299353155
> 0.9576824437692589
> 0.7710724755633548
> 0.32544648906230933
> 
> julia> close(fid)
> 
> julia> # Read data using our custom function
> 
> julia> read_data(file,"signal")
> 10-element Vector{Float64}:
> 0.013432897038968217
> 0.9148075310100701
> 0.8004580824186582
> 0.7207654878373057
> 0.8584511594948295
> 0.7538087575791116
> 0.47729849299353155
> 0.9576824437692589
> 0.7710724755633548
> 0.32544648906230933
> 
> julia> # Read two files (the same file twice)
> 
> julia> [file,file] |> x->read_data.(x,"signal")
> 2-element Vector{Vector{Float64}}:
> [0.013432897038968217, 0.9148075310100701, 0.8004580824186582, 0.7207654878373057, 0.8584511594948295, 0.7538087575791116, 0.47729849299353155, 0.9576824437692589, 0.7710724755633548, 0.32544648906230933]
> [0.013432897038968217, 0.9148075310100701, 0.8004580824186582, 0.7207654878373057, 0.8584511594948295, 0.7538087575791116, 0.47729849299353155, 0.9576824437692589, 0.7710724755633548, 0.32544648906230933]
> 
> julia> # Combine both datasets
> 
> julia> [file,file] |> x->read_data.(x,"signal") |> x->reduce(vcat,x)
> 20-element Vector{Float64}:
> 0.013432897038968217
> 0.9148075310100701
> 0.8004580824186582
> 0.7207654878373057
> 0.8584511594948295
> 0.7538087575791116
> 0.47729849299353155
> 0.9576824437692589
> 0.7710724755633548
> 0.32544648906230933
> 0.013432897038968217
> 0.9148075310100701
> 0.8004580824186582
> 0.7207654878373057
> 0.8584511594948295
> 0.7538087575791116
> 0.47729849299353155
> 0.9576824437692589
> 0.7710724755633548
> 0.32544648906230933
> 
> ```

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [June 3, 2024, 5:52am UTC](https://discourse.julialang.org/t/reading-multiple-files-and-also-one-file-containg-limits-to-read-other-files-content-upto/115060/8 "2024-06-03T05:52:05Z")

</div>

> [@feanor12](#):
>
> Here is another example:

Here you are creating a sub-directory _“signal”_ in _fid_ file, but in my case

> [@raman\_kumar](#):
>
> `datafile='sane00.prim.01800.athdf'`

is a HDF5 file and

> [@raman\_kumar](#):
>
> ` inputfile=runname+'sane00.athinput'`

is a text file containing limits of variables in `datafile` to read upto. In my case `inputfile` is not a sub-directory of `datafile` but a separate folder.  
@feanor12 How can i make it work in my case? Please see [cis/cis/data\_io/hdf.py at c207b4e3b47f770da404991eedeeb55422817772 · cedadev/cis · GitHub](https://github.com/cedadev/cis/blob/c207b4e3b47f770da404991eedeeb55422817772/cis/data_io/hdf.py#L107)  
@mkitti

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [June 17, 2024, 3:32pm UTC](https://discourse.julialang.org/t/reading-multiple-files-and-also-one-file-containg-limits-to-read-other-files-content-upto/115060/9 "2024-06-17T15:32:42Z")

</div>

Sorry, I found that this read\_data() function is not part of python language but it belongs to an independent [cis library](https://cis.readthedocs.io/en/stable/cis_api.html) that don’t have api in Julia. so i have to install cis library and then use PythonCall.jl to use python api of that library.