Reading multiple files and also one file containg limits to read other file's content upto

Specific Domains Data
, ,
1

I have python code like:

from pyfiles import * 
runname='./' #define start dir and file, uncomment below
sane = 1#0 for MAD98, 1 for SANE00
if sane == 1:
    inputfile=runname+'sane00.athinput'
    datafile='sane00.prim.01800.athdf'
    data=read_data(datafile,athinput=inputfile)
d = data.rho # density

Here

inputfile 
<job>
problem_id = sane00

<time>
cfl_number = 0.25
nlim       = -1
tlim       = 4.0E4

<output1>
file_type = hdf5
variable  = prim
id        = prim
dt        = 10.0
xdmf      = false

<output2>
file_type = hdf5
variable  = uov
id        = user
dt        = 100.0
xdmf      = false

<output3>
file_type = rst
dt        = 1.E4

<mesh>
nx1    = 72
x1min  = 1.6
x1max  = 1200.0
x1rat  = 1.0963050292915717
ix1_bc = user
ox1_bc = user

nx2    = 32
x2min  = 0.0
x2max  = 3.141592653589793
ix2_bc = polar
ox2_bc = polar

nx3    = 16
x3min  = 0.0
x3max  = 6.283185307179586
ix3_bc = periodic
ox3_bc = periodic

refinement = static

<meshblock>
nx1 = 18
nx2 = 4
nx3 = 16

<refinement1>
level = 1
x1min = 1.6
x1max = 200.0
x2min = 0.41
x2max = 2.731592653589793
x3min = 0.0
x3max = 6.283185307179586

<refinement2>
level = 2
x1min = 1.6
x1max = 30.0
x2min = 0.61
x2max = 2.531592653589793
x3min = 0.0
x3max = 6.283185307179586

<refinement3>
level = 1
x1min = 8.5
x1max = 1200.0
x2min = 0.0
x2max = 0.380
x3min = 0.0
x3max = 6.283185307179586

<refinement4>
level = 1
x1min = 8.5
x1max = 1200.0
x2min = 2.761592653589793
x2max = 3.141592653589793
x3min = 0.0
x3max = 6.283185307179586

<refinement5>
level = 2
x1min = 44.4
x1max = 1200.0
x2min = 0.0
x2max = 0.18
x3min = 0.0
x3max = 6.283185307179586

<refinement6>
level = 2
x1min = 44.4
x1max = 1200.0
x2min = 2.961592653589793
x2max = 3.141592653589793
x3min = 0.0
x3max = 6.283185307179586

<coord>
m = 1.0
a = 0.0

<hydro>
gamma     = 1.3333333333333333
dfloor    = 1.0e-6
pfloor    = 1.0e-8
rho_min   = 1.0e-2
rho_pow   = -1.5
pgas_min  = 1.0e-2
pgas_pow  = -2.5
sigma_max = 100.0
beta_min  = 1.0e-3
gamma_max = 50.0

<problem>
k_adi   = 1.0
r_edge  = 40.5
r_peak  = 80.0
l       = 0.0
rho_max = 1.0

field_config         = loops
potential_cutoff     = 0.2
potential_start      = 25.0
potential_end        = 550.0
potential_wavelength = 3.75
beta_min             = 0.05833

potential_r_pow      = 0.0
potential_rho_pow    = 1.0 

sample_n_r     = 288
sample_n_theta = 128
sample_r_rat   = 1.0232525877105085
sample_cutoff  = 0.2

pert_amp = 1.00E-1
pert_kr  = 6.28E-2
pert_kz  = 6.28E-2

is address of .txt configuration file that determines the limit to read data of sane00.prim.01800.athdf upto.

How to accomplish this in Julia? What is equivalent/alternative of python cis.read_data() function in Julia?
@mkitti

2

I think for HDF5 files you can use Home · HDF5.jl

You can modify the read_data function to take a config file (or better a predefined config type) instead.

function read_data(file,dataset)
   data = nothing
   h5open(file) do h5
      data = read(h5[dataset])
   end
   data
end

files = readdir("path_to_target",join=true)
data = files |> x->read_data.(x,Ref("prim")) |> x->reduce(vcat,x)

The bigger challenge will be writing a parser for the txt file you want to input.

something like this

struct ReadConfig
   fields::Dict{String,Dict{String,Any}}
end

import Base.parse
function parse(::Type{ReadConfig},filepath)
   rc = ReadConfig(Dict())
   for line in readlines(filepath)
     # parsing logic like:
     # keyDict[keyname] = value
     # rc.fields[fieldname] = keyDict
   end
   rc
end

function read_data(file,config::ReadConfig)
   # adapted logic
end
3

Hi Michael, I got some idea from your post but i still don’t have solution yet.

image
image816×1026 102 KB

4

I think you read in a files that is not a hdf5 file. You can add a filter to the file list.

file_root = raw"/home/raman/Pictures"
files = readdir(file_root,join=true) |> filter(endswith(".athdf"))
5

There is still error.

image
image816×1062 100 KB

6

Try calling it like that

dataset_key = "/prim"
filepath = raw"/home/raman/Pictures/sane00.prim.01800.athdf"
read_data(filepath,dataset_key)

In case you don’t know the datset_keys run

using HDF5
filepath = raw"/home/raman/Pictures/sane00.prim.01800.athdf"
h5open(filepath,"r") do io
  HDF5.show_tree(io)
end

to print the internal structure of the file

7

Here is another example:

Console example on how to read a hdf5 file 
julia> using HDF5

julia> function read_data(file,dataset)
          data = nothing
          h5open(file) do h5
             data = read(h5[dataset])
          end
          data
       end
read_data (generic function with 1 method)

julia> # Generate an example file with one dataset called signal

julia> file = "temp.h5"
"temp.h5"

julia> fid = h5open(file,"w")
🗂️ HDF5.File: (read-write) temp.h5

julia> fid["signal"] = rand(10);

julia> fid
🗂️ HDF5.File: (read-write) temp.h5
└─ 🔢 signal

julia> signal = read(fid,"signal")
10-element Vector{Float64}:
 0.013432897038968217
 0.9148075310100701
 0.8004580824186582
 0.7207654878373057
 0.8584511594948295
 0.7538087575791116
 0.47729849299353155
 0.9576824437692589
 0.7710724755633548
 0.32544648906230933

julia> close(fid)

julia> # Read data using our custom function

julia> read_data(file,"signal")
10-element Vector{Float64}:
 0.013432897038968217
 0.9148075310100701
 0.8004580824186582
 0.7207654878373057
 0.8584511594948295
 0.7538087575791116
 0.47729849299353155
 0.9576824437692589
 0.7710724755633548
 0.32544648906230933

julia> # Read two files (the same file twice)

julia> [file,file] |> x->read_data.(x,"signal")
2-element Vector{Vector{Float64}}:
 [0.013432897038968217, 0.9148075310100701, 0.8004580824186582, 0.7207654878373057, 0.8584511594948295, 0.7538087575791116, 0.47729849299353155, 0.9576824437692589, 0.7710724755633548, 0.32544648906230933]
 [0.013432897038968217, 0.9148075310100701, 0.8004580824186582, 0.7207654878373057, 0.8584511594948295, 0.7538087575791116, 0.47729849299353155, 0.9576824437692589, 0.7710724755633548, 0.32544648906230933]

julia> # Combine both datasets

julia> [file,file] |> x->read_data.(x,"signal") |> x->reduce(vcat,x)
20-element Vector{Float64}:
 0.013432897038968217
 0.9148075310100701
 0.8004580824186582
 0.7207654878373057
 0.8584511594948295
 0.7538087575791116
 0.47729849299353155
 0.9576824437692589
 0.7710724755633548
 0.32544648906230933
 0.013432897038968217
 0.9148075310100701
 0.8004580824186582
 0.7207654878373057
 0.8584511594948295
 0.7538087575791116
 0.47729849299353155
 0.9576824437692589
 0.7710724755633548
 0.32544648906230933
8

Here you are creating a sub-directory “signal” in fid file, but in my case

is a HDF5 file and

is a text file containing limits of variables in datafile to read upto. In my case inputfile is not a sub-directory of datafile but a separate folder.
@feanor12 How can i make it work in my case? Please see cis/cis/data_io/hdf.py at c207b4e3b47f770da404991eedeeb55422817772 · cedadev/cis · GitHub
@mkitti

9

Sorry, I found that this read_data() function is not part of python language but it belongs to an independent cis library that don’t have api in Julia. so i have to install cis library and then use PythonCall.jl to use python api of that library.