[ANN] PowerFlowData.jl: a parser for PSS/E-format `.raw` power flow data files

PowerFlowData.jl is a new package for parsing .raw PSS/E files. (Documentation)

This file format is widely used in power systems research, and I hope PowerFlowData.jl will help researchers who need to work with these files.

PowerFlowData.jl currently supports both v30 and v33 of the PSS/E data format. And should be ~100x faster and lower memory than existing open-source parsers, such as the parse_psse functions included in PowerModels.jl and PowerSystems.jl. For example, on a large file from PowerSystemsTestData:

Timing PowerModels.jl (v0.18.3) on v33 file: PowerSystemsTestData/ACTIVSg70k/ACTIVSg70k.RAW
1st run : 91.756136 seconds (695.83 M allocations: 15.238 GiB, 5.62% gc time, 0.59% compilation time)
2nd run : 89.145086 seconds (692.60 M allocations: 15.049 GiB, 5.78% gc time)
Timing PowerSystems.jl (v1.15.1) on v33 file: PowerSystemsTestData/ACTIVSg70k/ACTIVSg70k.RAW
1st run : 88.938766 seconds (683.01 M allocations: 14.586 GiB, 5.63% gc time, 0.15% compilation time)
2nd run : 89.891755 seconds (681.18 M allocations: 14.480 GiB, 5.90% gc time)
Timing PowerFlowData.jl (v1.0.0) on v33 file: PowerSystemsTestData/ACTIVSg70k/ACTIVSg70k.RAW
1st run :  0.837682 seconds (639.34 k allocations: 135.744 MiB, 74.52% compilation time)
2nd run :  0.248546 seconds (1.16 k allocations: 96.066 MiB)

(timings in Julia v1.6.3, and with logging output disabled)

I would welcome feedback from users, here or as issues on the GitHub repo.


(p.s. thanks to Jacob Quinn for a lot of advice on how to make a fast parser, as well as for Parsers.jl and Tables.jl which make the package possible.)

10 Likes

Possibly of interest to @ccoffrin and developers of PowerModels.jl :slight_smile:

Nice to know about this @npr

cc @claytonpbarrows who may also be interested.

3 Likes

Nice job!
We would be fairly interested by a generic parser for powerflow data. Is there any plan to integrate MATPOWER .m format as well?

Right now, we are using a code derived from PowerModels.jl, which is doing quite a good job. But we can’t rely explicitly on PowerModels.jl, as this would imply to add JuMP in our dependencies (+ a non negligible precompilation cost).

@frapac there has long been a dream of @claytonpbarrows and I to have a “parser-only” package that could be used by the variety of power system tools that are out there in Julia. The challenge in the past has been, what data format would such a parser target? Maybe @npr has found a reasonable proposal for that challenge in this package.

1 Like

Is there any plan to integrate MATPOWER .m format as well?

No plans. I’m not actually familiar with that format, whereas I’ve come across PSS/E in a work context. The PowerFlowData.jl code is very PSS/E specific; basically there’s a big file where I write out the specification as Julia types…

This does mean that (most) of the actual parsing code is fairly simple/generic, and (mostly) relies on the types describing the format… so if the MATPOWER format were similar enough that the same approach would work (i.e. we wouldn’t have to write too much format-specific parsing code, only types), then i could maybe see that being something we could add.

hmm… :thinking:

A major challenge with the Matpower format is that the files are effectively Matlab code, not a formalized data specification! So, for example, you don’t nessiary know the types of the fields until you are parsing them. We have kind of turned these Matpower files into a data format but under fairly strong assumptions about what is allowed and not in the .m file. The most robust solution would be if there was a parser that would read matlab .m files into Julia native commands and then we could translate from there. If you would be interested in adapting our current parser into the approach you have taken here the code is here, InfrastructureModels.jl/matlab.jl at master · lanl-ansi/InfrastructureModels.jl · GitHub

@frapac there has long been a dream of @claytonpbarrows and I to have a “parser-only” package that could be used by the variety of power system tools that are out there in Julia

I think it would be very nice to have all the parsers in a single place :slight_smile:
For the input data, I think having both PSSE and Matpower would cover almost every use cases we have so far.
For the output data, I am used to PowerModels’ Dict structures, which are quite versatile. The Table structure uses in PowerFlowData.jl should do the job too.

It would be interesting to move the MATPOWER parser of PowerModels.jl in PowerflowData.jl (and modifies it to use the same Table structure). In fact, that would be perfect for us. @npr do you think it would make sense to integrate that in your package?

It would be interesting to move the MATPOWER parser of PowerModels.jl in PowerflowData.jl

I think adding support for MATPOWER data is an interesting idea, and one i hadn’t considered before.

We’d need to investigate if the same approach would be suitable, otherwise it would make more sense as it’s own package, i think.

I opened an issue to track it

1 Like

In fact, I use MATLAB.jl to call corresponding case function and convert the result to julia dataframes :grinning_face_with_smiling_eyes:, because MATPOWER sometimes contains post-processing code.