[ANN] PowerFlowData.jl: a parser for PSS/E-format `.raw` power flow data files

npr · November 18, 2021, 1:30pm

PowerFlowData.jl is a new package for parsing .raw PSS/E files. (Documentation)

This file format is widely used in power systems research, and I hope PowerFlowData.jl will help researchers who need to work with these files.

PowerFlowData.jl currently supports both v30 and v33 of the PSS/E data format. And should be ~100x faster and lower memory than existing open-source parsers, such as the parse_psse functions included in PowerModels.jl and PowerSystems.jl. For example, on a large file from PowerSystemsTestData:

Timing PowerModels.jl (v0.18.3) on v33 file: PowerSystemsTestData/ACTIVSg70k/ACTIVSg70k.RAW
1st run : 91.756136 seconds (695.83 M allocations: 15.238 GiB, 5.62% gc time, 0.59% compilation time)
2nd run : 89.145086 seconds (692.60 M allocations: 15.049 GiB, 5.78% gc time)
Timing PowerSystems.jl (v1.15.1) on v33 file: PowerSystemsTestData/ACTIVSg70k/ACTIVSg70k.RAW
1st run : 88.938766 seconds (683.01 M allocations: 14.586 GiB, 5.63% gc time, 0.15% compilation time)
2nd run : 89.891755 seconds (681.18 M allocations: 14.480 GiB, 5.90% gc time)
Timing PowerFlowData.jl (v1.0.0) on v33 file: PowerSystemsTestData/ACTIVSg70k/ACTIVSg70k.RAW
1st run :  0.837682 seconds (639.34 k allocations: 135.744 MiB, 74.52% compilation time)
2nd run :  0.248546 seconds (1.16 k allocations: 96.066 MiB)

(timings in Julia v1.6.3, and with logging output disabled)

I would welcome feedback from users, here or as issues on the GitHub repo.

(p.s. thanks to Jacob Quinn for a lot of advice on how to make a fast parser, as well as for Parsers.jl and Tables.jl which make the package possible.)

npr · November 18, 2021, 2:46pm

Possibly of interest to @ccoffrin and developers of PowerModels.jl

ccoffrin · November 18, 2021, 3:55pm

Nice to know about this @npr

cc @claytonpbarrows who may also be interested.

frapac · November 18, 2021, 6:07pm

Nice job!
We would be fairly interested by a generic parser for powerflow data. Is there any plan to integrate MATPOWER .m format as well?

Right now, we are using a code derived from PowerModels.jl, which is doing quite a good job. But we can’t rely explicitly on PowerModels.jl, as this would imply to add JuMP in our dependencies (+ a non negligible precompilation cost).

ccoffrin · November 18, 2021, 6:36pm

@frapac there has long been a dream of @claytonpbarrows and I to have a “parser-only” package that could be used by the variety of power system tools that are out there in Julia. The challenge in the past has been, what data format would such a parser target? Maybe @npr has found a reasonable proposal for that challenge in this package.

npr · November 18, 2021, 6:38pm

Is there any plan to integrate MATPOWER .m format as well?

No plans. I’m not actually familiar with that format, whereas I’ve come across PSS/E in a work context. The PowerFlowData.jl code is very PSS/E specific; basically there’s a big file where I write out the specification as Julia types…

This does mean that (most) of the actual parsing code is fairly simple/generic, and (mostly) relies on the types describing the format… so if the MATPOWER format were similar enough that the same approach would work (i.e. we wouldn’t have to write too much format-specific parsing code, only types), then i could maybe see that being something we could add.

hmm…

ccoffrin · November 18, 2021, 7:16pm

A major challenge with the Matpower format is that the files are effectively Matlab code, not a formalized data specification! So, for example, you don’t nessiary know the types of the fields until you are parsing them. We have kind of turned these Matpower files into a data format but under fairly strong assumptions about what is allowed and not in the .m file. The most robust solution would be if there was a parser that would read matlab .m files into Julia native commands and then we could translate from there. If you would be interested in adapting our current parser into the approach you have taken here the code is here, https://github.com/lanl-ansi/InfrastructureModels.jl/blob/master/src/io/matlab.jl

frapac · November 18, 2021, 7:59pm

@frapac there has long been a dream of @claytonpbarrows and I to have a “parser-only” package that could be used by the variety of power system tools that are out there in Julia

I think it would be very nice to have all the parsers in a single place
For the input data, I think having both PSSE and Matpower would cover almost every use cases we have so far.
For the output data, I am used to PowerModels’ Dict structures, which are quite versatile. The Table structure uses in PowerFlowData.jl should do the job too.

It would be interesting to move the MATPOWER parser of PowerModels.jl in PowerflowData.jl (and modifies it to use the same Table structure). In fact, that would be perfect for us. @npr do you think it would make sense to integrate that in your package?

npr · November 19, 2021, 12:35am

It would be interesting to move the MATPOWER parser of PowerModels.jl in PowerflowData.jl

I think adding support for MATPOWER data is an interesting idea, and one i hadn’t considered before.

We’d need to investigate if the same approach would be suitable, otherwise it would make more sense as it’s own package, i think.

I opened an issue to track it

github.com/nickrobinson251/PowerFlowData.jl

Add ability to parse MATPOWER data?!

opened 12:27AM - 19 Nov 21 UTC

nickrobinson251

idea

Prompted by discussion in https://discourse.julialang.org/t/ann-powerflowdata-jl…-a-parser-for-pss-e-format-raw-power-flow-data-files/71722 Turns out power systems research has more than one very silly data format to deal with 😂 😭 We could consider broadening the scope of the package to also parse matpower data... I think this only makes sense in this package (rather than its own package) if we can re-use a decent amount of what we have here, and broadly take the same approach (parsing the files byte-by-byte, having sections described by their own struct, etc). Some specific ways in which matpower data is even more gross than PSS/E: * it's not even its own file format, it's a matlab file. * I _think_ there's something approaching [a specification for the _matrices_ in the file which contain the actual data](https://matpower.app/manual/matpower/DataFileFormat.html) (if there's not then we're sunk) * but the file itself just has to be a valid matlab file (i think), so actually finding the matrices amongst the rest of the file sounds very painful. 😈 * since it's a matlab file, we have to deal with all types of comments, including end-of-line comments * and not sure if this will be harder because whitespace is the delimiter * we can probably give up on mid-line comments appearing in the data (is that a thing in matlab?), because something are just too silly. * I don't think there's any requirement on the order in which different categories of data appear, meaning we'd have to somehow identify what data * possibly the (second part of) _names_ are meaningful? e.g. bus data has to be `foo.bus = ...`? * the first part of this name isn't meaningful. convention seems to be `mpc.*` but i think that's convention only, and not actually required. * the "Generator Cost Data" doesn't not have a fixed number of columns 😒 * on a quick look i think we could have the last column be a `Vector{Vector{Float64}}` and `push!` new columns if/when we encounter them... but it'll mena custom parsing code, not just a custom type Probably other ways too... I'd never looked at this format before, so likely there's more to discover! I think worth investigating further

metab0t · November 19, 2021, 2:43am

In fact, I use MATLAB.jl to call corresponding case function and convert the result to julia dataframes , because MATPOWER sometimes contains post-processing code.