# Combining data of different GCNs in a single file

**URL:** <https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013>\
**Category:** Data\
**Tags:** strings, data, loops, dataframes\
**Created:** [July 24, 2023, 12:18pm UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013 "2023-07-24T12:18:32Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [July 24, 2023, 12:18pm UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/1 "2023-07-24T12:18:32Z")

</div>

I want to create an empty table having headers 👉

- **Time(s) | Magnitude | MagErr | Filter | System | Telescope | GalExtCorr | GCN\_no. | GRB\_no.**

My code work looks like 👇

```julia
using HTTP,DataFrames,CSV

```

```julia
function doanalysis()
       dfg=nothing
       for x in 34020:34038 
           print("\r peeking at GCN $x ")
           try
               url = "https://gcn.nasa.gov/circulars/$x/raw"
               resp = HTTP.get(url) 
               status=resp.status
               print(" ",status," "); 
               if status == 404 ; println("status=",status); continue; end          
               txt = String(resp.body)
			   if occursin("V. Lipunov", txt)
                   println(" MASTER report") 
                   he=first(findfirst(r"^Tmid"im,txt))
                   lr=first(findnext("\nFilter",txt,he))-1
                   cltxt=txt[he:lr]
                   
                   df=CSV.read(IOBuffer(cltxt), DataFrame, delim='|', skipto=3)
                   df.GCN=[x for i in 1:nrow(df)]
                  
                   if isnothing(dfg) 
                       dfg=df
                   else
                       dfg=vcat(dfg,df)
                   end # if x is first
                   CSV.write("data-$(x).csv",df)
               end # if occursin
           catch e
               println("error ")
              
           end # trycatch
       end # for loop
       println()
       if !isnothing(dfg)
           CSV.write("data-all.csv",dfg)
       else
           @info "no dfg to write"
       end # !isnothing
       end # function doanalysis

```

```julia
doanalysis()

```

How to add codes for different telescopes in loop for different GCNs ?

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [July 24, 2023, 5:21pm UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/2 "2023-07-24T17:21:33Z")

</div>

One way to get a table from multiple tables is using the vcat() function.  
below an example  
But to handle the various cases you’re dealing with you have to patiently enter all the if then else that you need

```julia
using CSV, DataFrames, HTTP
function gettablegcn(tbln)
    url="https://gcn.nasa.gov/circulars/$tbln/raw"
    txt=String((HTTP.get(url)))
    #m treats the ^ and $ tokens as matching the start and end of individual lines, as opposed to the whole string.
    readuntil(IOBuffer(txt),"\nTmid-T0")
    he=first(findfirst(r"^Tmid"im,txt))

    cltxt=readuntil(IOBuffer(txt[he:end]),"\nFilter")
    # lr=first(findnext("\nFilter",txt,he))-1
    # cltxt=txt[he:lr]
    df=CSV.read(IOBuffer(cltxt), DataFrame, delim='|', skipto=3, dateformat="yyyy-mm-dd HH:MM:SS")
end

# Time(s) Magnitude MagErr Filter System Observatory/Telescope/Instrument GalExtCorr Source GCN_no, GRB_no,

vcat(gettablegcn.(34026:34028)..., cols=:union)

julia> vcat(gettablegcn.(34026:34028)..., cols=:union)
3×8 DataFrame
 Row │ Tmid-T0 Date Time Site Coord (J2000) Filt. Expt. Limit ⋯
     │ Int64 Dates.DateTime String31 String String7 Int64 Float6 ⋯
─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ 39166 2023-06-19T22:40:12 MASTER-OAFA (09h 47m 21.03s , +27d 22m 11.4s) C 180 17. ⋯
   2 │ 38724 2023-06-19T22:40:12 MASTER-OAFA (09h 47m 21.03s , +27d 22m 11.4s) C 180 17.  
   3 │ 12859 2023-06-19T22:40:12 MASTER-OAFA (09h 47m 21.03s , +27d 22m 11.4s) C 180 17.  

```

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [July 24, 2023, 5:26pm UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/3 "2023-07-24T17:26:12Z")

</div>

> [@rocco\_sprmnt21](#):
>
> `vcat(gettablegcn.(34026:34028)..., cols=:union)`

`cols =:union ` is the correct solution.

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [July 25, 2023, 6:08am UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/4 "2023-07-25T06:08:57Z")

</div>

What is `$tbln` in your code?

---

<div class="post-metadata">

**Author:** ![jdad](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jdad/32/4739_2.png) [@jdad](https://discourse.julialang.org/u/jdad)\
**Post date:** [July 26, 2023, 4:35pm UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/5 "2023-07-26T16:35:07Z")

</div>

`$tbln` is (in string `url`) the interpolated value of the `tbln` variable which is the argument of the function `gettablegcn`

---

<div class="post-metadata">

**Author:** ![jdad](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jdad/32/4739_2.png) [@jdad](https://discourse.julialang.org/u/jdad)\
**Post date:** [July 26, 2023, 4:38pm UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/6 "2023-07-26T16:38:48Z")

</div>

I refactored somewhat the code, using as in @rocco_sprmnt21’s proposal the raw version of the circular, and putting dedicated code into functions for

1. extracting latest circular GCN

2. extracting GRB if present

3. extracting table if present,

plus some corrections for special cases encountered whoile testing (see comments inside the functions)

```julia
@time using HTTP,DataFrames,CSV

function getlatestcirc()
    urln="https://gcn.nasa.gov/circulars/" # no raw version => we will work with html
    resp = HTTP.get(urln) # throw exception if page not found
    body=String(resp.body)

    # search for <li value="12345"> , where n=12345, c1=c2=", c3=>
    lid=findfirst("<li value=",body)
    lid=last(lid) # convert to position of '='
    lid=lid+1; c1=body[lid:lid]; if c1 != "\""; error("getlastcirc: invalid c1 $c1 $(body[lid-10:lid+10])");end
    gcn=body[lid+1:lid+5]
    lid=lid+5 # skip gcn
    lid=lid+1; c2=body[lid:lid]; if c2 != "\""; error("getlastcirc: invalid c2 $c2 $(body[lid-10:lid+10])");end
    lid=lid+1; c3=body[lid:lid]; if c3 != ">" ; error("getlastcirc: invalid c3 $c3 $(body[lid-10:lid+10])");end

    # and decode it
    gcn=parse(Int,gcn)
    gcn
end

function getgrb(lines,gcn)
	grb=nothing
    grbt="no_grb"
	for line in lines
		if occursin("GRB",line)
			# println("GRB:",line)
			t=line
			t=replace(t,r".*GRB" => "")
			t=replace(t,r"^ " => "")
			t=replace(t,r"^:" => "")
			t=replace(t,r"[|:].*$" => "")
			if isnothing(grb)
				grb=t
				grbt=grb
			elseif grb != t
				tt=t
				tt=replace(tt,r"\..*$" => "")
				if !startswith(grb,tt)
					error("Error, multiple GRB ($grb,$t) in circ $gcn")
				end
			end
		end
		# if occursin("robotic elescope",line)
		# # println("robotic:",line)
		# elseif occursin("elescope",line)
		# # println("escope:",line)
		# end
	end
	grb,grbt
end

function getdf(lines,gcn)
    dbg=false
	lfirsttable=nothing; llasttable=nothing
	# gcn 33228 table starts by Sources | Tmid-T0 (day) | UT (start) ...
	# gcn 33228 table has blank lines inside table, table is terminated by "----"
	for (i,line) in pairs(lines)
	   # println(" i=",i," line=",line)
	   if startswith(line,"Tmid-T0")
		   if isnothing(lfirsttable)
			   lfirsttable=i
		   else
			   error("Error, multiple tables (Tmid-T0)in circ $gcn")
		   end
	   end
	   # gcn 33064 is terminated by blank line, not Filter
	   
	   if !isnothing(lfirsttable) && isnothing(llasttable) # check table not terminater by filter but by blankline
		   if (i>lfirsttable+3) && isnothing(llasttable)
			   if line == ""
				   llasttable=i-1 # premature end of table (no Filter line)
				   dbg && println("setting llasttable (blk) ",llasttable)
			   end
		   end
	   end
	   if !isnothing(lfirsttable) && startswith(line,"Filter")
		   if isnothing(llasttable)
			   llasttable=i-1
			   dbg && println("setting llasttable (Filter) ",llasttable)
		   else
			   error("Error, multiple tables (Filter)in circ $gcn")
		   end
	   end
	end 
	# normalize names in header line
	lines[lfirsttable]=replace(lines[lfirsttable],"." => "")
	cltxt=join(lines[lfirsttable:llasttable],'\n')
	if gcn == 33064
	    cltxt=replace(cltxt,"15.3 |" => "15.3")
	end
	haspvert=occursin(" P| ",cltxt)
	# if gcn == 34262
	if haspvert
		dbg && println("gcn $gcn haspvert")
	    cltxt=replace(cltxt," P| " => " P! ")
	end
	dbg && println("\n cltxt=\n",cltxt,"\n endofcltxt")
	df = CSV.read(IOBuffer(cltxt), DataFrame, delim='|',skipto=3, normalizenames=true)
	dbg && @show df
	if haspvert
		println("rechange")
	    df.Filt=replace.(df.Filt," P! " => " P| ")
		dbg && @show df
	end
	df
end

function doanalysis()
	writedetailedcsv=false
	clearline="\r" * (" "^60) * "\r"
    dfg=nothing
    circfirst=33037
    # circfirst=33546
    # circfirst=33411
    # circfirst=34242
    # circfirst=33728
	# circlast=33044
	# circlast=33228
	# circlast=33124
    circlast=getlatestcirc()
    println("lastcirc is ",circlast)

    # circlast=circfirst
	# circlast=circfirst=33064
	# circlast=circfirst=34262
    println("Will peek from $circfirst to $circlast")

    # for gcn in 33037:33287 # was 33037:33187, but mention of 33211 as good candidate
    for gcn in circfirst:circlast
	    print(clearline)
        print("\r peeking at gcn: $gcn ")
        try
			# we use the "raw" version of circ bulletin, because it is plain text and not html
            url = "https://gcn.nasa.gov/circulars/$gcn/raw"
			# with "exception=false", a 404 error does not raise an exception
            resp = HTTP.get(url;status_exception=false) 
            status=resp.status
            print(" ",status," "); # println();
            if status == 404 ; println("page not found"); continue; end          
            txt = String(resp.body)
            hastelescope=occursin("elescope",txt)
            hasGRB=occursin("GRB",txt)
            hasrobotic=occursin("robotic telescope",txt)
			#hastmidt0=occursin("Tmid-T0",txt) # not : gcn 3328 have table with tmi-to in the middle of the line, but no Filter endline, etc.
            hastmidt0=occursin("\nTmid-T0",txt)
            ok= hastelescope && hastmidt0
            if hastelescope || hasGRB || hasrobotic || hastmidt0
                cmt=""
                if hasrobotic; cmt=cmt * " robotic"; end
                if hastelescope; cmt=cmt * " telescope"; end
                if hasGRB; cmt=cmt * " GRB"; end
                if hastmidt0; cmt=cmt * " tmidt0"; end
                if ok; cmt=cmt * " OK"; end
				cmt=cmt*" "
                print(" :",cmt)
            end
			if hastmidt0 && occursin("Tmid-T0 (day)",txt)
				# @info "skipping bad header Tmid-T0 (day)"
				println("skipping bad header Tmid-T0 (day)")
				continue
			end
            grb=nothing
			grbt="no_grb"
            if ok
                lines=readlines(IOBuffer(txt))
				grb,grbt=getgrb(lines,gcn)
                if isnothing(grb)
                    if hasGRB
						println(" !no grb with hasGRB !")
                        error("\ncirc $gcn : no grb when hasGRB!!!")
                    else
                        # println("no_grb")
						println("grb:",grbt)
                    end
                else
                    println("grb:",grbt)
                end
                    
				df=getdf(lines,gcn)
                df.GCN=[gcn for i in 1:nrow(df)]
				df.GRB=[grbt for i in 1:nrow(df)]
                if isnothing(dfg) # gcn == 33037
                    dfg=df
                else
                    dfg=vcat(dfg,df,cols=:union)
                end # if gcn is first
                if writedetailedcsv
				    CSV.write("data-$(gcn).csv",df)
				end
            end
        catch e
		    # error("error at circ $gcn") # e
            println("error caught at circ $gcn ",e)
            rethrow()
        end # trycatch
    end # for loop

    println()

    if !isnothing(dfg)
		namedfg="data-all.csv"
        CSV.write(namedfg,dfg)
		@info "namedfg written successfully"
		println("\ndescription"); println(describe(dfg))
		println("\ncontent"); show(dfg)
		println()
    else
        @info "no dfg to write"
    end # !isnothing
	
    dfg
	
end # function doanalysis

dfg=doanalysis()

println("tkumar done.")

```

Hope it helps.

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [July 26, 2023, 4:52pm UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/7 "2023-07-26T16:52:17Z")

</div>

I want to compile the tabular data of as much as possible telescopes available at GCN and combine them in a single file . It must also contain columns of GCN number , Magnitude and GRB identity .  
My code for SWIFT / UVOT Telescope looks like :

```julia
using HTTP,DataFrames,CSV
function doanalysis()
       dfg=nothing
       for x in 33211:33212
           print("\r peeking at GCN $x ")
           try
               url = "https://gcn.nasa.gov/circulars/$x"
               resp = HTTP.get(url) 
               status=resp.status
               print(" ",status," "); 
               if status == 404 ; println("status=",status); continue; end          
               txt = String(resp.body)

			   if occursin(r"GRB ?\d{6}([A-G]|(\.\d{2}))?",txt)
				    m=match(r"GRB ?\d{6}([A-G]|(\.\d{2}))?",txt)
				    print(m.match)
			   end
			   # code for SWIFT/UVOT 
			   if occursin("report on behalf of the Swift/UVOT team",txt)
               hb,he=findfirst(r"^Filter"im,txt)
			   lr,_=findnext("\n\nThe",txt,he)
			   cltxt=replace(txt[hb:lr], " +/- "=>"+/-", r" +(\w)"=>s"\t\1",r" +(&gt;)"=>s"\t>")
			   df=CSV.read(IOBuffer(cltxt), DataFrame, delim='\t')
			    df.GCN=[x for i in 1:nrow(df)]
				df.GRB=[m.match for i in 1:nrow(df)]
				rename!(df, :Mag => :Magnitude)
			   @show DataFrame(df)
              end
			   
		   catch e
			   print("Error")
		   end
	   end
end   
doanalysis()

```

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [July 26, 2023, 5:45pm UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/8 "2023-07-26T17:45:30Z")

</div>

My code for MASTER robotic telescopes is 😆

```julia
using HTTP,DataFrames,CSV
function doanalysis()
                            dfg=nothing
                            for x in 30000:34246
                          print("\r peeking at GCN $x ")
                         try
                             url = "https://gcn.nasa.gov/circulars/$x/raw"
                             resp = HTTP.get(url) 
                             status=resp.status
                             print(" ",status," "); 
                             if status == 404 ; println("status=",status); continue; end          
                             txt = String(resp.body)
                             if occursin(r"GRB ?\d{6}([A-G]|(\.\d{2}))?",txt)
				    m=match(r"GRB ?\d{6}([A-G]|(\.\d{2}))?",txt)
				    print(m.match)
			   end

                             if occursin("V. Lipunov", txt)
                                 println(" MASTER report")
                                 
                                 he=first(findfirst(r"^Tmid"im,txt))
                                 lr=first(findnext("\nFilter",txt,he))-1
                                 cltxt=txt[he:lr]
                                 
                                 df=CSV.read(IOBuffer(cltxt), DataFrame, delim='|', skipto=3)
                                 df.GCN=[x for i in 1:nrow(df)]
                                 df.GRB=[m.match for i in 1:nrow(df)]
                                 rename!(df, :Filt. => :Filter)
                                 if isnothing(dfg) 
                                     dfg=df
                                 else
                                     dfg=vcat(dfg,df)
                                 end # if x is first
        end # if occursin
                         catch e
                             println("error ")
                            
                         end # trycatch
                     end # for loop
                     println()
                     if !isnothing(dfg)
                         CSV.write("data-all.csv",dfg)
                     else
                         @info "no dfg to write"
                     end # !isnothing
                     end # function doanalysis
doanalysis()

```

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [July 27, 2023, 3:11am UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/9 "2023-07-27T03:11:17Z")

</div>

I have tried to merge them together but it is giving error possibly due to different header names.

```julia
using HTTP , CSV, DataFrames
function doanalysis()
                            dfg=nothing
                            for x in 34000:34146
                          print("\r peeking at GCN $x ")
                         try
                             url = "https://gcn.nasa.gov/circulars/$x/raw"
                             resp = HTTP.get(url) 
                             status=resp.status
                             print(" ",status," "); 
                             if status == 404 ; println("status=",status); continue; end          
                             txt = String(resp.body)
                             if occursin(r"GRB ?\d{6}([A-G]|(\.\d{2}))?",txt)
				                  m=match(r"GRB ?\d{6}([A-G]|(\.\d{2}))?",txt)
				                  print(m.match)
			                  end

                             if occursin("V. Lipunov", txt)
                                 println(" MASTER report")
                                 
                                 he=first(findfirst(r"^Tmid"im,txt))
                                 lr=first(findnext("\nFilter",txt,he))-1
                                 cltxt=txt[he:lr]
                                 
                                 df=CSV.read(IOBuffer(cltxt), DataFrame, delim='|', skipto=3)
                                 df.GCN=[x for i in 1:nrow(df)]
                                 df.GRB=[m.match for i in 1:nrow(df)]
                                 rename!(df, :Filt. => :Filter)
                                 if isnothing(dfg) 
                                     dfg=df
                                 else
                                     dfg=vcat(dfg,df)
                                 end
                                elseif occursin("report on behalf of the Swift/UVOT team",txt)
                                       println(" SWIFT/UVOT report ")
                                       hb,he=findfirst(r"^Filter"im,txt)
			                           lr,_=findnext("\n\nThe",txt,he)
			                           cltxt=replace(txt[hb:lr], " +/- "=>"+/-", r" +(\w)"=>s"\t\1",r" +(&gt;)"=>s"\t>")
			                           df=CSV.read(IOBuffer(cltxt), DataFrame, delim='\t')
			                           df.GCN=[x for i in 1:nrow(df)]
				                       df.GRB=[m.match for i in 1:nrow(df)]
				                       rename!(df, :Mag => :Magnitude)
				                      if isnothing(dfg) 
                                          dfg=df
                                      else
                                          dfg=vcat(dfg,df)
                                      end # if x is first
        end # if occursin
                         catch e
                             println("error ")
                            
                         end # trycatch
                     end # for loop
                     println()
                     if !isnothing(dfg)
                         CSV.write("data-all.csv",dfg)
                     else
                         @info "no dfg to write"
                     end # !isnothing
                     end # function doanalysis
doanalysis()

```

---

<div class="post-metadata">

**Author:** ![jdad](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jdad/32/4739_2.png) [@jdad](https://discourse.julialang.org/u/jdad)\
**Post date:** [July 27, 2023, 9:47am UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/10 "2023-07-27T09:47:36Z")

</div>

Here is a modified version. Modifications are

```julia
tku3$ diff tku3-org.jl tku3.jl
13,14c13,16
< if occursin(r"GRB ?\d{6}([A-G]|(\.\d{2}))?",txt)
< m=match(r"GRB ?\d{6}([A-G]|(\.\d{2}))?",txt)
---
> grb_rexp=r"GRB ?\d{6}([A-G]|(\.\d{2}))?"
> m=match(grb_rexp,txt)
> grb="nogrb"
> if occursin(grb_rexp,txt)
15a18
> grb=m.match
27c30
< df.GRB=[m.match for i in 1:nrow(df)]
---
> df.GRB=[grb for i in 1:nrow(df)]
31c34
< dfg=vcat(dfg,df)
---
> dfg=vcat(dfg,df,cols=:union)
40,41c43,44
< df.GRB=[m.match for i in 1:nrow(df)]
< rename!(df, :Mag => :Magnitude)
---
> df.GRB=[grb for i in 1:nrow(df)]
> if "Mag" in names(df); rename!(df, :Mag => :Magnitude); end
45c48
< dfg=vcat(dfg,df)
---
> dfg=vcat(dfg,df,cols=:union)
49a53
> rethrow()

```

Explanations

- the `m` variable was not set at the function level but only in the `if`. It is better defined it at the function level (and factorised the regular expression so it is typed only once), and then you can get directly the `grb` value to work with after (and set it to `"nogrb"` in case regexp does not match), instead of the match variable
- added `cols=:union` to the `vcat`s so we add new field instead of failing. By the way, having two sequences of contructing df and vcat-ing in dfg is prone to problems in the future, you should factorize that into one common sequence (function? common if branch ?) if possible
- protected the `rename!`by checking tbefore that `:Mag` is present in the dataframe
- added `rethrow()` in the `catch`branch so we know what errors triggered the failure

Good luck!

here is the full (modified) program

```julia
using HTTP , CSV, DataFrames
function doanalysis()
                            dfg=nothing
                            for x in 34000:34146
                          print("\r peeking at GCN $x ")
                         try
                             url = "https://gcn.nasa.gov/circulars/$x/raw"
                             resp = HTTP.get(url) 
                             status=resp.status
                             print(" ",status," "); 
                             if status == 404 ; println("status=",status); continue; end          
                             txt = String(resp.body)
							 grb_rexp=r"GRB ?\d{6}([A-G]|(\.\d{2}))?"
							 m=match(grb_rexp,txt)
							 grb="nogrb"
                             if occursin(grb_rexp,txt)
				                  print(m.match)
								  grb=m.match
			                  end

                             if occursin("V. Lipunov", txt)
                                 println(" MASTER report")
                                 
                                 he=first(findfirst(r"^Tmid"im,txt))
                                 lr=first(findnext("\nFilter",txt,he))-1
                                 cltxt=txt[he:lr]
                                 
                                 df=CSV.read(IOBuffer(cltxt), DataFrame, delim='|', skipto=3)
                                 df.GCN=[x for i in 1:nrow(df)]
                                 df.GRB=[grb for i in 1:nrow(df)]
                                 if isnothing(dfg) 
                                     dfg=df
                                 else
                                     dfg=vcat(dfg,df,cols=:union)
                                 end
                                elseif occursin("report on behalf of the Swift/UVOT team",txt)
                                       println(" SWIFT/UVOT report ")
                                       hb,he=findfirst(r"^Filter"im,txt)
			                           lr,_=findnext("\n\nThe",txt,he)
			                           cltxt=replace(txt[hb:lr], " +/- "=>"+/-", r" +(\w)"=>s"\t\1",r" +(&gt;)"=>s"\t>")
			                           df=CSV.read(IOBuffer(cltxt), DataFrame, delim='\t')
			                           df.GCN=[x for i in 1:nrow(df)]
				                       df.GRB=[grb for i in 1:nrow(df)]
				                       if "Mag" in names(df); rename!(df, :Mag => :Magnitude); end
				                      if isnothing(dfg) 
                                          dfg=df
                                      else
                                          dfg=vcat(dfg,df,cols=:union)
                                      end # if x is first
        end # if occursin
                         catch e
                             println("error ")
                            rethrow()
                         end # trycatch
                     end # for loop
                     println()
                     if !isnothing(dfg)
                         CSV.write("data-all.csv",dfg)
                     else
                         @info "no dfg to write"
                     end # !isnothing
                     end # function doanalysis
doanalysis()

```

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [July 27, 2023, 11:44am UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/11 "2023-07-27T11:44:13Z")

</div>

`cols =:union ` is the correct solution.

> [@jdad](#):
>
> added `cols=:union` to the `vcat`s so we add new field instead of failing.

---

<div class="post-metadata">

**Author:** ![jdad](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jdad/32/4739_2.png) [@jdad](https://discourse.julialang.org/u/jdad)\
**Post date:** [July 27, 2023, 11:49am UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/12 "2023-07-27T11:49:35Z")

</div>

That is what your `rename!` does, effectively. But if the column name is already `Magnitude` it will fail, so you have to protect it by the `if "Mag" in names(df)`.

You then just have to add a tailored if-protected-rename for each of the name you want to normalize (Mag,Mag.,Filt,Filt.,…)

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [July 29, 2023, 9:38am UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/13 "2023-07-29T09:38:38Z")

</div>

> [@jdad](#):
>
> `if "Mag" in names(df)`

rename is not working for this case 😜

```julia
using HTTP , CSV, DataFrames
function doanalysis()
    dfg=nothing
    for x in 10819
    print("\r peeking at GCN $x ")
        try
            url = "https://gcn.nasa.gov/circulars/$x/raw"
            resp = HTTP.get(url) 
            status=resp.status
            print(" ",status," "); 
            if status == 404 ; println("status=",status); continue; end          
            txt = String(resp.body)
            if occursin(r"GRB ?\d{6}([A-G]|(\.\d{2}))?",txt)
				m=match(r"GRB ?\d{6}([A-G]|(\.\d{2}))?",txt)
				print(m.match)
			end

            if occursin("CrAO", txt)
                println(" CrAO report")                
                he=first(findfirst(r"^(Date)|(UT start)|(T0+)"m,txt))
                lr=first(findnext(r"^\nThe |(Photometry)"m,txt,he))-1
                cltxt=replace(txt[he:lr], ","=>" ",r"(�+)"=>" " , "+/-"=>" ")
                df=CSV.read(IOBuffer(cltxt), DataFrame, delim="|", skipto=3)
                df.GCN=[x for i in 1:nrow(df)]
                df.GRB=[m.match for i in 1:nrow(df)] 
				if "mag." in names(df); rename!(df, :mag. => :Magnitude); end
				if isnothing(dfg) 
                    @show dfg=df
                else
                    @show dfg=vcat(dfg,df)
                end # if x is first
            end # if occursin
        catch e
            println("error ")                    
        end # trycatch
    end # for loop
end
doanalysis()

```

---

<div class="post-metadata">

**Author:** ![jdad](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jdad/32/4739_2.png) [@jdad](https://discourse.julialang.org/u/jdad)\
**Post date:** [July 29, 2023, 9:50am UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/14 "2023-07-29T09:50:19Z")

</div>

I would suppose this is because there is a dot in `mag.` . IIUC, a dot cannot appear in a symbol, `:mag.` is bad, `:mag`. is ok. Suggestion : in the rename, use strings instead of symbols.

> [@raman\_kumar](#):
>
> `rename!(df, :mag. => :Magnitude)`

```julia
rename!(df, "mag." => "Magnitude")

```

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [July 29, 2023, 9:53am UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/15 "2023-07-29T09:53:59Z")

</div>

No , `rename!(df, "mag." => "Magnitude")` is also not working. It is not issue of `.` . In _if_ loop i also can’t rename **Filter** header.

---

<div class="post-metadata">

**Author:** ![jdad](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jdad/32/4739_2.png) [@jdad](https://discourse.julialang.org/u/jdad)\
**Post date:** [July 29, 2023, 10:54am UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/16 "2023-07-29T10:54:26Z")

</div>

There is no mention of Filter is this code, so it is probably a try with another version you did…

Here, inserting a `@show names(df)` before and after the `rename`, I see that while it seems the CSV.read was successful, in reality it was not, it did swoop all fields before GCN in one gigantic field.

To correct that, change the `delim` to blank, and add the option to ignore repeated delimiters.

`delim=" ", ignorerepeated=true`

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [July 29, 2023, 11:54am UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/17 "2023-07-29T11:54:02Z")

</div>

Thanks, It is working fine now.  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/8/6/86074d67b3f91cd14e72d13ad2a77bdb39d68e55.png)

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [July 30, 2023, 12:28pm UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/18 "2023-07-30T12:28:09Z")

</div>

> [@jdad](#):
>
> added `cols=:union` to the `vcat`s

I want to have data from equivalent headers of different telescopes in a single column. For example single `Magnitude` column should contain data of headers titled `mag` ,`mag.`, `Mag` and `Limit` in different telescope’s GCN.

---

<div class="post-metadata">

**Author:** ![jdad](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jdad/32/4739_2.png) [@jdad](https://discourse.julialang.org/u/jdad)\
**Post date:** [July 30, 2023, 12:42pm UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/19 "2023-07-30T12:42:11Z")

</div>

Just add the corresponding `if … rename!` for each of the column headers you see in the final dfg after the line you already have for “mag.”

---

<div class="post-metadata">

**Author:** ![raman\_kumar](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raman_kumar/32/26782_2.png) [@raman\_kumar](https://discourse.julialang.org/u/raman_kumar)\
**Post date:** [July 30, 2023, 12:44pm UTC](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013/20 "2023-07-30T12:44:17Z")

</div>

I tried that `if … rename!` but it skips a lot of GCNs.

> **See code**
>
> ```julia
> using HTTP , CSV, DataFrames
> function doanalysis()
> dfg=nothing
> for x in 34000:34146
> print("\r peeking at GCN $x ")
> try
> url = "https://gcn.nasa.gov/circulars/$x/raw"
> resp = HTTP.get(url) 
> status=resp.status
> print(" ",status," "); 
> if status == 404 ; println("status=",status); continue; end          
> txt = String(resp.body)
> if occursin(r"GRB ?\d{6}([A-G]|(\.\d{2}))?",txt)
> m=match(r"GRB ?\d{6}([A-G]|(\.\d{2}))?",txt)
> print(m.match)
> end
> 
> if occursin("V. Lipunov", txt)
> println(" MASTER report")
>                                  
> he=first(findfirst(r"^Tmid"im,txt))
> lr=first(findnext("\nFilter",txt,he))-1
> cltxt=txt[he:lr]
>                                  
> df=CSV.read(IOBuffer(cltxt), DataFrame, delim='|', skipto=3)
> df.GCN=[x for i in 1:nrow(df)]
> df.GRB=[m.match for i in 1:nrow(df)]
> if "Limit" in names(df); rename!(df, :Limit => :Magnitude); end
> if isnothing(dfg) 
> dfg=df
> else
> dfg=vcat(dfg,df,cols=:union)
> end
> elseif occursin("report on behalf of the Swift/UVOT team",txt)
> println(" SWIFT/UVOT report ")
> hb,he=findfirst(r"^Filter"im,txt)
> lr,_=findnext("\n\nThe",txt,he)
> cltxt=replace(txt[hb:lr], " +/- "=>"+/-", r" +(\w)"=>s"\t\1",r" +(&gt;)"=>s"\t>")
> df=CSV.read(IOBuffer(cltxt), DataFrame, delim='\t')
> df.GCN=[x for i in 1:nrow(df)]
> df.GRB=[m.match for i in 1:nrow(df)]
> if "Mag" in names(df); rename!(df, :Mag => :Magnitude); end
> if isnothing(dfg) 
> dfg=df
> else
> dfg=vcat(dfg,df,cols=:union)
> end # if x is first
>                                       
> elseif occursin("CrAO", txt)
> println(" CrAO report")                
> he=first(findfirst(r"^(Date)|(UT start)|(T0+)"m,txt))
> lr=first(findnext(r"^\nThe |(Photometry)"m,txt,he))-1
> cltxt=replace(txt[he:lr], ","=>" ",r"(�+)"=>" " , "+/-"=>" ",">"=>" ")
> df=CSV.read(IOBuffer(cltxt), DataFrame, skipto=3,delim=" ", ignorerepeated=true)
> df.GCN=[x for i in 1:nrow(df)]
> df.GRB=[m.match for i in 1:nrow(df)] 
> if "mag." in names(df); rename!(df, "mag." => "Magnitude"); end
> if isnothing(dfg) 
> dfg=df
> else
> dfg=vcat(dfg,df,cols=:union)
> end # if x is first
> end # if occursin
> catch e
> println("error in try catch ")
>                             
> end # trycatch
> end # for loop
> println()
> if !isnothing(dfg)
> CSV.write("data-all.csv",dfg)
> else
> @info "no dfg to write"
> end # !isnothing
> end # function doanalysis
> doanalysis()
> 
> ```

[Next page](https://discourse.julialang.org/t/combining-data-of-different-gcns-in-a-single-file/102013.md?page=2)