# \[ANN\] PowerFlowData.jl: a parser for PSS/E-format \`.raw\` power flow data files
**URL:** https://discourse.julialang.org/t/ann-powerflowdata-jl-a-parser-for-pss-e-format-raw-power-flow-data-files/71722
**Category:** Package Announcements
**Tags:** package, announcement
**Created:** [November 18, 2021, 1:30pm UTC](https://discourse.julialang.org/t/ann-powerflowdata-jl-a-parser-for-pss-e-format-raw-power-flow-data-files/71722 "2021-11-18T13:30:05Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@npr](https://discourse.julialang.org/u/npr)
#### Post date: [November 18, 2021, 1:30pm UTC](https://discourse.julialang.org/t/ann-powerflowdata-jl-a-parser-for-pss-e-format-raw-power-flow-data-files/71722/1 "2021-11-18T13:30:05Z")
[PowerFlowData.jl](https://github.com/nickrobinson251/PowerFlowData.jl/) is a new package for parsing `.raw` [PSS/E](https://en.wikipedia.org/wiki/Power_system_simulator_for_engineering) files. ([Documentation](https://nickrobinson251.github.io/PowerFlowData.jl/dev/index.html))
This file format is widely used in power systems research, and I hope PowerFlowData.jl will help researchers who need to work with these files.
PowerFlowData.jl currently supports both v30 and v33 of the PSS/E data format. And should be ~100x faster and lower memory than existing open-source parsers, such as the `parse_psse` functions included in PowerModels.jl and PowerSystems.jl. For example, on a large file from PowerSystemsTestData:
```nohighlight
Timing PowerModels.jl (v0.18.3) on v33 file: PowerSystemsTestData/ACTIVSg70k/ACTIVSg70k.RAW
1st run : 91.756136 seconds (695.83 M allocations: 15.238 GiB, 5.62% gc time, 0.59% compilation time)
2nd run : 89.145086 seconds (692.60 M allocations: 15.049 GiB, 5.78% gc time)
Timing PowerSystems.jl (v1.15.1) on v33 file: PowerSystemsTestData/ACTIVSg70k/ACTIVSg70k.RAW
1st run : 88.938766 seconds (683.01 M allocations: 14.586 GiB, 5.63% gc time, 0.15% compilation time)
2nd run : 89.891755 seconds (681.18 M allocations: 14.480 GiB, 5.90% gc time)
Timing PowerFlowData.jl (v1.0.0) on v33 file: PowerSystemsTestData/ACTIVSg70k/ACTIVSg70k.RAW
1st run : 0.837682 seconds (639.34 k allocations: 135.744 MiB, 74.52% compilation time)
2nd run : 0.248546 seconds (1.16 k allocations: 96.066 MiB)
```
(timings in Julia v1.6.3, and with logging output disabled)
I would welcome feedback from users, here or as issues on the GitHub repo.
* * *
(p.s. thanks to Jacob Quinn for a lot of advice on how to make a fast parser, as well as for Parsers.jl and Tables.jl which make the package possible.)
---
### Author:  [@npr](https://discourse.julialang.org/u/npr)
#### Post date: [November 18, 2021, 2:46pm UTC](https://discourse.julialang.org/t/ann-powerflowdata-jl-a-parser-for-pss-e-format-raw-power-flow-data-files/71722/2 "2021-11-18T14:46:42Z")
Possibly of interest to @ccoffrin and developers of PowerModels.jl đ
---
### Author:  [@ccoffrin](https://discourse.julialang.org/u/ccoffrin)
#### Post date: [November 18, 2021, 3:55pm UTC](https://discourse.julialang.org/t/ann-powerflowdata-jl-a-parser-for-pss-e-format-raw-power-flow-data-files/71722/3 "2021-11-18T15:55:05Z")
Nice to know about this @npr
cc @claytonpbarrows who may also be interested.
---
### Author:  [@frapac](https://discourse.julialang.org/u/frapac)
#### Post date: [November 18, 2021, 6:07pm UTC](https://discourse.julialang.org/t/ann-powerflowdata-jl-a-parser-for-pss-e-format-raw-power-flow-data-files/71722/4 "2021-11-18T18:07:21Z")
Nice job!
We would be fairly interested by a generic parser for powerflow data. Is there any plan to integrate MATPOWER `.m` format as well?
Right now, we are using a code derived from PowerModels.jl, which is doing quite a good job. But we canât rely explicitly on PowerModels.jl, as this would imply to add JuMP in our dependencies (+ a non negligible precompilation cost).
---
### Author:  [@ccoffrin](https://discourse.julialang.org/u/ccoffrin)
#### Post date: [November 18, 2021, 6:36pm UTC](https://discourse.julialang.org/t/ann-powerflowdata-jl-a-parser-for-pss-e-format-raw-power-flow-data-files/71722/5 "2021-11-18T18:36:23Z")
@frapac there has long been a dream of @claytonpbarrows and I to have a âparser-onlyâ package that could be used by the variety of power system tools that are out there in Julia. The challenge in the past has been, what data format would such a parser target? Maybe @npr has found a reasonable proposal for that challenge in this package.
---
### Author:  [@npr](https://discourse.julialang.org/u/npr)
#### Post date: [November 18, 2021, 6:38pm UTC](https://discourse.julialang.org/t/ann-powerflowdata-jl-a-parser-for-pss-e-format-raw-power-flow-data-files/71722/6 "2021-11-18T18:38:40Z")
> Is there any plan to integrate MATPOWER `.m` format as well?
No plans. Iâm not actually familiar with that format, whereas Iâve come across PSS/E in a work context. The PowerFlowData.jl code is very PSS/E specific; basically thereâs [a big file](https://github.com/nickrobinson251/PowerFlowData.jl/blob/v1.1.0/src/types.jl) where I write out the specification as Julia typesâŚ
This does mean that (most) of the actual parsing code is fairly simple/generic, and (mostly) relies on the types describing the format⌠so if the MATPOWER format were similar enough that the same approach would work (i.e. we wouldnât have to write too much format-specific _parsing_ code, only types), then i could _maybe_ see that being something we could add.
hmm⌠đ¤
---
### Author:  [@ccoffrin](https://discourse.julialang.org/u/ccoffrin)
#### Post date: [November 18, 2021, 7:16pm UTC](https://discourse.julialang.org/t/ann-powerflowdata-jl-a-parser-for-pss-e-format-raw-power-flow-data-files/71722/7 "2021-11-18T19:16:35Z")
> [@npr](#):
>
> This does mean that (most) of the actual parsing code is fairly simple/generic, and (mostly) relies on the types describing the format⌠so if the MATPOWER format were similar enough that the same approach would work (i.e. we wouldnât have to write too much format-specific _parsing_ code, only types), then i could _maybe_ see that being something we could add.
A major challenge with the Matpower format is that the files are effectively Matlab code, not a formalized data specification! So, for example, you donât nessiary know the types of the fields until you are parsing them. We have kind of turned these Matpower files into a data format but under fairly strong assumptions about what is allowed and not in the `.m` file. The most robust solution would be if there was a parser that would read matlab `.m` files into Julia native commands and then we could translate from there. If you would be interested in adapting our current parser into the approach you have taken here the code is here, [https://github.com/lanl-ansi/InfrastructureModels.jl/blob/master/src/io/matlab.jl](https://github.com/lanl-ansi/InfrastructureModels.jl/blob/master/src/io/matlab.jl)
---
### Author:  [@frapac](https://discourse.julialang.org/u/frapac)
#### Post date: [November 18, 2021, 7:59pm UTC](https://discourse.julialang.org/t/ann-powerflowdata-jl-a-parser-for-pss-e-format-raw-power-flow-data-files/71722/8 "2021-11-18T19:59:17Z")
> @frapac there has long been a dream of @claytonpbarrows and I to have a âparser-onlyâ package that could be used by the variety of power system tools that are out there in Julia
I think it would be very nice to have all the parsers in a single place đ
For the input data, I think having both PSSE and Matpower would cover almost every use cases we have so far.
For the output data, I am used to PowerModelsâ `Dict` structures, which are quite versatile. The `Table` structure uses in `PowerFlowData.jl` should do the job too.
It would be interesting to move the MATPOWER parser of PowerModels.jl in PowerflowData.jl (and modifies it to use the same `Table` structure). In fact, that would be perfect for us. @npr do you think it would make sense to integrate that in your package?
---
### Author:  [@npr](https://discourse.julialang.org/u/npr)
#### Post date: [November 19, 2021, 12:35am UTC](https://discourse.julialang.org/t/ann-powerflowdata-jl-a-parser-for-pss-e-format-raw-power-flow-data-files/71722/9 "2021-11-19T00:35:24Z")
> It would be interesting to move the MATPOWER parser of PowerModels.jl in PowerflowData.jl
I think adding support for MATPOWER data is an interesting idea, and one i hadnât considered before.
Weâd need to investigate if the same approach would be suitable, otherwise it would make more sense as itâs own package, i think.
I opened an issue to track it
>
>
> Prompted by discussion in https://discourse.julialang.org/t/ann-powerflowdata-jlâŚ-a-parser-for-pss-e-format-raw-power-flow-data-files/71722
>
> Turns out power systems research has more than one very silly data format to deal with đ đ
>
> We could consider broadening the scope of the package to also parse matpower data...
> I think this only makes sense in this package (rather than its own package) if we can re-use a decent amount of what we have here, and broadly take the same approach (parsing the files byte-by-byte, having sections described by their own struct, etc).
>
> Some specific ways in which matpower data is even more gross than PSS/E:
> \* it's not even its own file format, it's a matlab file.
> \* I \_think\_ there's something approaching \[a specification for the \_matrices\_ in the file which contain the actual data\](https://matpower.app/manual/matpower/DataFileFormat.html) (if there's not then we're sunk)
> \* but the file itself just has to be a valid matlab file (i think), so actually finding the matrices amongst the rest of the file sounds very painful. đ
> \* since it's a matlab file, we have to deal with all types of comments, including end-of-line comments
> \* and not sure if this will be harder because whitespace is the delimiter
> \* we can probably give up on mid-line comments appearing in the data (is that a thing in matlab?), because something are just too silly.
> \* I don't think there's any requirement on the order in which different categories of data appear, meaning we'd have to somehow identify what data
> \* possibly the (second part of) \_names\_ are meaningful? e.g. bus data has to be \`foo.bus = ...\`?
> \* the first part of this name isn't meaningful. convention seems to be \`mpc.\*\` but i think that's convention only, and not actually required.
> \* the "Generator Cost Data" doesn't not have a fixed number of columns đ
> \* on a quick look i think we could have the last column be a \`Vector{Vector{Float64}}\` and \`push!\` new columns if/when we encounter them... but it'll mena custom parsing code, not just a custom type
>
> Probably other ways too... I'd never looked at this format before, so likely there's more to discover!
>
> I think worth investigating further
---
### Author:  [@metab0t](https://discourse.julialang.org/u/metab0t)
#### Post date: [November 19, 2021, 2:43am UTC](https://discourse.julialang.org/t/ann-powerflowdata-jl-a-parser-for-pss-e-format-raw-power-flow-data-files/71722/10 "2021-11-19T02:43:43Z")
In fact, I use MATLAB.jl to call corresponding case function and convert the result to julia dataframes đ, because MATPOWER sometimes contains post-processing code.