The PQDS File Specification

The PQio application provides a quick means to convert PQDIF files to the PQDS.csv format so that event waveform data can be easily reviewed, obfuscated or edited if necessary, and shared with colleagues. The Power Quality Data Sharing (PQDS) format prescribes the way to structure a CSV file to exchange PQ data -- both waveform and trending data -- in a simple, human readable form. In many GPA applications, like openSEE and TrenDAP, when clicking on the "Export as CSV" button the CSV file produced will comply with the PQDS format standard so that this file can be parsed and used by PQDS compatible systems. Version 1.2 of this specification is now available in pre-release for review.

...
  1. Introduction
  2. Metadata
  3. Data
  4. Expected PQDS Parser Behavior
  5. PQDS file naming convention
  6. Example PQDS.csv File
  7. PQDS Log Files
  8. Meta-data PQDS Codes Descriptions
  9. References

1. Introduction

The Power Quality Data Sharing (PQDS) format is a comma-separated value file that enables the easy review, editing and sharing of electric power disturbance waveforms and trending data. It is designed to be a simple human-to-human data format that complements the standard system-to-system waveform data formats of PQDIF and COMTRADE.

The Comma-Separated Value (CSV) file format is supported by almost all spreadsheets and database management systems as well as by standard libraries that are available in most programming languages. In addition, many software applications can split files based on the comma separated values. [1]

The PQDS format CSV file includes two information sections -- (1) a meta-data section with key-value information about the waveform, and (2) a data section. This data section can contain either time-series voltage and current waveform measurements or time series trending measurments consistsing of minimum, maximum and average values. Each of these two sections individually complies with the RFC 4180 CSV format specification with header data to identify the names of the fields that follow. Parsers for PQDS files will expect these files to comply with the seven RFC 4180 rules for CSV formatted files. [2]

The PQDS format and tools to convert PQDIF and COMTRADE waveform data to PQDS has been developed by the Electric Power Research Institute (EPRI) to support utility disturbance monitoring research. The EPRI PQio program is building a library of waveforms that can be used to evaluate and test automated waveform analytics -- including analytics involving artificial intelligence (AI) techniques.

1.1 Simplicity

All meta-data and waveform values in PQDS files are in standard engineering units without the need for subsequent scaling or adjustment.
Scientific notation for values is not allowed. The time associated with a measurement is in decimal seconds.

While more complex event cases can be be included in the waveform data itself, no provisions are made for describing multiple events in the waveform meta-data. If included, PQDS meta-data can be used to provide details about a single event in the waveform or general information about the dataset.

1.2 Extensibility

Users and groups of users can easily extend both the PQDS meta-data as well as PQDS time-series waveform data measurement values with low risk of breaking any automation that may have been implemented by other users who have implemented different extensions.

1.3 Shareability

The PQDS format is designed to facilitate event waveform data sharing in all its forms -- as waveform data may need to be shared in full detail among work colleagues, or as anonymous data may need to be contributed to repositories for power system phenomenon research and analytics improvement. The PQDS supports including (or not including) just the waveform data that is needed for an individual use case. If necessarily, the waveform data itself can also be easily trimmed down to just to portion of the event that is of interest.

PQDS is designed to make it easier for the data-owner to improve productivity through collaboration with others -- the business-basis for development of the PQDS format.

  • As a human-readable format with easy-to-use spreadsheet tools for review, there is no ambiguity on the data that is being shared.
  • The ability to selectivity reduce, anatomize or correct the meta-data associated with the waveform allows waveform data sharing to align with business confidentiality, security and regulatory requirements on a case-by-case basis.
  • The data is provided to the recipient in the flexible, ubiquitous CSV format so that it can be quickly and easily used without the need for specialized tools.
  • The PQDS format is sufficiently well-structured that it can be used as a format for the automated exchange of waveform data such as can be done through spreadsheet macros at the user-level or through automated programming interfaces. For example, the csv-based format of PQDS enables power quality data to be ported to machine learning tools and other statistical analytic tools that are now commonly available and expect data in this form.

2. Metadata

The meta-data section begins with a header record than contains the following field names:

Column Field Name Requirements
A key Not case sensitive. Note 1
B value The information associated with the key Note 2
C value-units The SI units for numeric values (if any) Note 1
D expected-data-type The expected meta-data value type that can be used for validation Note 3
E note Any other descriptive information about the value Note 2
  1. The "key" and "value-units" fields must not exceed 32 characters, must begin with a letter can only contain letters, numerals, the hyphen, underscore and dot characters. Key and value-units fields may be encapsulated in double quotes. The key "waveform-data" is reserved and cannot be used.
  2. "Value" must not exceed 254 characters and "note" must not exceed 4096 characters and both are limited to standard ASCII characters (32 thru 126). Value and note fields must be encapsulated in double-quotes to avoid truncation at the first comma. Value fields that are intended to parsed as numbers should not contain commas -- even if these are wrapped in quotes.
  3. Expected data types are (The data-type field is optional with the data-type of "Text" assumed.):
    • Enumeration (E): zero or positive integer less than 32,768
    • Numeric (N): digits, decimal point and leading sign with a total length less than 16 numerals
    • Alpha-numeric (A): letters, numbers, dot, hyphen and underscore with a total length less than 255 characters
    • Text (T): any ASCII character (32 through 126). Total length limited to 4096 characters.
    • Binary (B): the numeral 0 or 1

Importantly, the entire meta-data section of the PQDS.csv file is not required and may be omitted entirely without impact to processing the waveform or trending data.

2.1 Reserved PQDS Metadata keys

The following key names are reserved to facilitate use of PQDS to support automated data sharing. Users are welcome to add keys as they may be needed.

2.1.1 Meta-data about device making the measurements:

Meta-data Key Description PQDS Data-type
DeviceName Meter or measurement device name T
DeviceAlias Alternate meter or measurement device name T
DeviceLocation Meter or device location name T
DeviceLocationAlias Alternate meter or device location name T
DeviceLatitude Latitude T (35.041058)
DeviceLongitude Longitude T (-85.315074)
AccountName Name of customer or account T
AccountNameAlias Alternate customer or account name T
DeviceDistanceToXFMR The distance (feet) to the upstream transformer N
DeviceConnectionTypeCode The PQDS code for meter connection type E (0-5)
DeviceNotes Description of the device T
DeviceOwner The utility name T

2.1.2 Meta-data about the electrical asset or load being monitored:

Meta-data Key Description PQDS Data-type
AssetName Name of the monitored asset T
NominalVoltage-LG Volts - line-to-ground N
NominalFrequency Hertz N (60 default)
UpstreamXFMR-kVA Upstream transformer size in kVA N
LineLength Length of line in units specified (default miles) N

2.1.3 Meta-data about the nature of the disturbance in the waveform:

Note that these fields are primarily associated with disturbance data and might not apply in the trending-data case.

Meta-data Key Description PQDS Data-type
EventGUID Globally Unique Event Identifier A (e.g., d571d945-e950-41b5-a3ef-960c410c395e)
EventID A user defined ID for this event A
EventYear Year E (4 digit, e.g., 2019)
EventMonth Month E (1-12)
EventDayofMonth Day E (1-31)
EventHour Hour E (0-23)
EventMinute Minute E (0 to 59)
EventSecond Second E (0 to 59)
EventNanoSeconds Nanosecond N (0 to 999999999)
EventDate Event Date T (e.g., 01/12/2019)
EventTime Event Time T (e.g., 10:09:23.999999999)
EventTimeZone Event Time Zone T (UTC, ET, CT, MT, PT, AT, HT)
EventTypeCode PQDS Event Type Code E (0-5)
EventFaultTypeCode PQDS Fault Type Code E (0 - x)
EventPeakCurrent Amperes N
EventPeakVoltage Volts N
EventMaxVA RMS Maximum A-Phase Volts N
EventMaxVB RMS Maximum B-Phase Volts N
EventMaxVC RMS Maximum C-Phase Volts N
EventMinVA RMS Minimum A-Phase Volts N
EventMinVB RMS Minimum B-Phase Volts N
EventMinVC RMS Maximum C-Phase Volts N
EventMaxIA RMS Maximum A-Phase Amperes N
EventMaxIB RMS Maximum B-Phase Amperes N
EventMaxIC RMS Maximum C-Phase Amperes N
EventFaultI2T Amperes^2 (for fault duration) N
EventPreEventCurrent Amperes (RMS) N
EventPreEventVoltage Volts (RMS) N
EventDuration Decimal seconds N (0-999.999999999)
EventCauseCode PQDS Event Cause Code E (0-12)
EventNotes Description of the event T
EventCauseNotes Description of the event cause T
DistanceToFault Distance from the meter to the fault N

2.1.4 Meta-data about the nature of the trending data in the file:

Note that these fields are primarily associated with tredning data and might not apply in the disturbance data case.

Meta-data Key Description PQDS Data-type
StartDate Start Date for Trending Data T (in the format "DD-MM-YYYY")

2.1.5 Meta-data about the measurements in the time-series data:

Meta-data Key Description PQDS Data-type
WaveformDataTypeFlag 0 = point-on-wave measurements; 1 = RMS measurements B (0 or 1)

2.1.6 Meta-data about use restrictions or information classification of the waveform:

Meta-data Key Description PQDS Data-type
WaveformSensitivityCode The PQDS Info Classification Code E (0-3)
WaveformSensitivityNotes Description of waveform sensitivity T

2.2 Adding New Keys -- User Extensions to Meta-data

Additional keys and values can be added to the meta-data section as needed. As these new keys are created the pattern of "domain"."key" should be followed with the "domain" typically being a corporate or organizational abbreviation. This approach avoids key conflicts as meta-data is independently extended by multiple parties. e.g., extended meta-data key = "EPRI.SequenceID"

3. Data Section

Following the meta-data section is the data. Since PQDS can be used to either store disturbance wavefor recordings or trending-data, there are two possible formats for this section. In either case the first line after the meta-data is the header record (line) which describes the type of data ( "waveform" or "trending") and the asociated signals. The following two sections describe each format in more detail

3.1 Disturbance Waveform Data

The header record (line) for waveform-data is comprised of the following fields and MUST begin with the string "waveform-data" (not case sensitive).

Column Field Name Requirements
A waveform section flag T (MUST be set to the string "waveform-data") Note 5
B thu J measurement T (MUST one of nine measurement names) Note 6
  1. Column A (the first value) contains time data in decimal seconds and should not exceed 999 seconds or 12 decimal places of precision (picoseconds). Optionally, negative time can be shown with time zero representing the trigger point for the disturbance.
  2. Valid values for measurement names to be included in the waveform data header are "va", "vb", "vc", "vn", "ia", "ib", "ic", "in", "f". These names are not case sensitive and can be in any order. Field names and their respective data can be omitted for data that is not available.

3.1.1 Adding New Measurements -- User Extensions to Waveform Data

Additional measurements can be added as additional 'columns' to the standard voltage and current measurement quantities shown in Note 6 above.

3.2 Trending Data

The header record (line) for trending-data is comprised of the following fields and MUST begin with the string "trending-data" (not case sensitive).

Column Field Name Requirements
A waveform section flag T (MUST be set to the string "trending-data") Note 7
B thu L measurement T (MUST one of eleven measurement names) Note 8
  1. Column A (the first value) contains time data in the "HH:MM" format. >>HH<< is the the 2 digit hour in 24H format (00-23) and >>MM<< is the 2 digit minute (00-59). For trending data spanning a single day the date is stored in the "StartDate" field in the meta-data section.
  2. Valid values for measurement names to be included in the waveform data header are "va", "vb", "vc", "vn", "ia", "ib", "ic", "in", "f", "vthd", "ithd", "". These names are not case sensitive and can be in any order. Field names and their respective data can be omitted for data that is not available.

The data itself following the header record consists of the Time stamp as described in Note 7 followed by a tripplet of values for each measurement. These values are seperated by >>,<< and are always ordered as minimum, average, maximum.

3.1.1 Trending Data Spanning multiple Days

Since trending data can be taken at slow sampling rates it is possible the data contained in the PQDS files goes beyond 23:59. In that case an additional day-header is inserted between two days. This header can be skipped for the first day (Day 0)

Column Field Name Requirements
A DayCount N (MUST be set to the day follwing this header) Note 9
B Date T (MUST be in the Date Format described in Note 10)
C Spacing T (MUST be a set of 20 "#")
  1. Column A (the first value) represents the contious count of days starting with Day 0 which is the Date stored in the "StartDate" field in the meta-data section.
  2. Column B represents the date of the day following this header. This is optional but if supplied should be in the following format "DD-MM-YYYY", where DD is the 2 digit day of the month, MM is the 2 digit month of the year and YYYY is the 4 digit Year.

3.1.2 Adding New Measurements -- User Extensions to Waveform Data

Additional measurements can be added as additional 'columns' to the standard voltage and current measurement quantities shown in Note 8 above.

4. Expected PQDS Parser Behavior

As part parsing PQDS files, if the the waveform meta-data section is present it must begin with the three header field names of "key", "value", "value-units" (not case sensitive). Meta-data keys do not need to appear in any particular order. Empty lines in the meta-data section will be ignored. All fields will be ignored for invalid keys (e.g., one containing special characters or with a length greater than 32 characters). Valid keys will contain 'null' for other fields that are invalid. Other fields may be added to each meta-data line (meta-data record), but will not be parsed.

Parsers are expected to validate meta-data values based on the data-type rules.

4.1 Including embedded quotes and commas

Parsers of PQDS should be expected to behave like Excel when handling strings that include commas and quotes. For example the string: >> "I find that the harder I work, the more luck I seem to have.", Thomas Jefferson << is placed in the CSV as """I find that the harder I work, the more luck I seem to have."", Thomas Jefferson" That is, the overall string is encapsulated in quotes with quotes that are intended represented as double quotes.

5. PQDS file naming convention

There is purposely no standard for PQDS file naming except that is recommended that the file extension be ".csv"

6. Example PQDS.csv File

6.1 Wave Form Data File

                            
                                DeviceName,"(FL1)",,T,"Meter or measurement device name"
                                DeviceAlias,"e6b51727-f747-11cf-9d89-0080c72e70a3",,T,"Alternate meter device name"
                                DeviceLocation,"N/A",,T,"Meter or measurement device location name"
                                AccountName,"e6b5170a-f747-11cf-9d89-0080c72e70a3",,T,""
                                DeviceOwner,"ptco004",,T,"Utility name"
                                AssetName,"Asset 1 (e6b5170a-f747-11cf-9d89-0080c72e70a3)",,T,"Asset name"
                                EventGUID,"3a07e27c-0800-98d9-fc11-0747fa0715b6e",,A,"Unique Event Identifier"
                                EventID,"Event 6",,T,"A user defined Event Name"
                                WaveFormSensitivityCode,"1",,E,"PQDS Data Sensitivity Code"
                                ContactEmail,"clackner@gridprotectionalliance.org",,T,"Email of the author"
                                Utility,"GridProtectionAlliance",,T,"Utility that generated the PQDS file"
                                waveform-data,va,vb,vc,ia,ib,ic
                                0,-148.404,0.7707,149.76,-3.51931,-0.0487,-0.082
                                0.0013,-145.0518,-8.03,152.65,-0.8474,-0.0906,-0.082

6.2 Trending Data File

                        
                            DeviceName,"(FL1)",,T,"Meter or measurement device name"
                            DeviceAlias,"e6b51727-f747-11cf-9d89-0080c72e70a3",,T,"Alternate meter device name"
                            DeviceLocation,"N/A",,T,"Meter or measurement device location name"
                            AccountName,"e6b5170a-f747-11cf-9d89-0080c72e70a3",,T,""
                            DeviceOwner,"ptco004",,T,"Utility name"
                            AssetName,"Asset 1 (e6b5170a-f747-11cf-9d89-0080c72e70a3)",,T,"Asset name"
                            StartDate,"02-01-2019",,T,"Start Date for Trending Data"
                            WaveFormSensitivityCode,"1",,E,"PQDS Data Sensitivity Code"
                            ContactEmail,"clackner@gridprotectionalliance.org",,T,"Email of the author"
                            Utility,"GridProtectionAlliance",,T,"Utility that generated the PQDS file"
                            trending-data,va,vb,vc
                            23:40,148.404,149.750,156.081,145.203,148.756,150.046,135.214,146.736,151.124
                            23:50,148.213,148.124,155.234,144.142,149.253,151.770,136.213,145.998,149.235
                            1,03-01-2019,#####################						
                            0:00,149.143,149.986,156.876,145.585,148.125,150.124,144.008,146.341,149.432
                            0:10,148.134,141.231,156.236,145.234,148.235,150.156,140.145,144.444,150.999

7. PQDS Log Files

7.1 Introduction

The Power Quality Data Sharing (PQDS) log file format is a comma-separated value file that enables easy tracking and sharing of the metadata not included in the PQDS file for various reasons, such as anomyizing the electric power disturbance waveforms. It is designed to be a simple and extensible human-to-human data format that complements the PQDS file format.

The PQDS log file format can include multiple sections, each corresponding to a single PQDS file generated by the EPRI PQDS application. The EPRI PQDS application will append a new section to the existing log file for each PQDS file generated.

7.2 Header Section

Each section corresponding to a PQDS file is separated by a single line of 42 >>-<< characters. The first section skips this seperator and imidiately starts with the identifying header.

7.3 Identifying Header

Each section contains a line consisting of two fields separated by a comma. The first field is the date and time at which the corresponding PQDS file is generated. The field is encapsulated in double quotations and formated as d/m/YYYY hh:mm:ss. The date and time are given in UTC to avoid any confusion over the time zone in which the file was generated and formated using a 24 hour format. The second field is a uniquely generated GUID associated with the PQDS file. This GUID corresponds to the EventGUID field in the meta data section of the PQDS file as found in Section 2.1 of this document. Note that the field is also encapsulated by ".

7.4 Meta Data Section

Imidiately following the identifying header the meta data section of the PQDS file is included. Unlike the PQDS file itself all available meta data should be included in the log file. The format of this section is the same as that described in Section 2.1 this document.

7.5 Data Section

The log file does not contain any actual data. However, in order to better describe the content of the PQDS file a header similar to that of the wave form or tredning data section in the PQDS file is included. The entry starts with the word >>waveform-data<< or >>trending-data<< , followed by comma separated list of the channels included in the PQDS file. Suggested values include >>va<<, >>vb<<, >>vc<<, >>vn<<, >>ia<<, >>ib<<, >>ic<<, >>in<<, and >>f<<.

7.6 Example PQDS log File

                            
                                "1/11/2019 09:34:50","3a07e27c-0800-98d9-fc11-0747fa0715b6e"
                                DeviceName,"(FL1)",,T,"Meter or measurement device name"
                                DeviceAlias,"e6b51727-f747-11cf-9d89-0080c72e70a3",,T,"Alternate meter name"
                                DeviceLocation,"N/A",,T,"Meter or measurement device location name"
                                AccountName,"e6b5170a-f747-11cf-9d89-0080c72e70a3",,T,""
                                DeviceOwner,"ptco004",,T,"Utility name"
                                AssetName,"Asset 1 (e6b5170a-f747-11cf-9d89-0080c72e70a3)",,T,"Asset name"
                                EventGUID,"00000000-0000-0000-0000-000000000000",,A,"Unique Event Identifier"
                                EventID,"Event 6",,T,"A user defined Event Name"
                                WaveFormSensitivityCode,"1",,E,"PQDS Data Sensitivity Code"
                                ContactEmail,"clackner@gridprotectionalliance.org",,T,"Email of the author"
                                Utility,"GridProtectionAlliance",,T,"Utility that generated the PQDS file"
                                waveform-data,va,vb,vc,ia,ib,ic
                                ---------------------------------------------
                                "1/12/2019 19:35:04","f7c6281b-0858-22df-ae9a-1191d83f81b4"
                                DeviceName,"(FL1)",,T,"Meter or measurement device name"
                                DeviceAlias,"e6b51727-f747-11cf-9d89-0080c72e70a3",,T,"Alternate meter name"
                                DeviceLocation,"N/A",,T,"Meter or measurement device location name"
                                AccountName,"e6b5170a-f747-11cf-9d89-0080c72e70a3",,T,""
                                DeviceOwner,"ptco004",,T,"Utility name"
                                AssetName,"Asset 1 (e6b5170a-f747-11cf-9d89-0080c72e70a3)",,T,"Asset name"
                                EventGUID,"00000000-0000-0000-0000-000000000000",,A,"Unique Event Identifier"
                                EventID,"Event 6",,T,"A user defined Event Name"
                                WaveFormSensitivityCode,"1",,E,"PQDS Data Sensitivity Code"
                                ContactEmail,"clackner@gridprotectionalliance.org",,T,"Email of the author"
                                Utility,"GridProtectionAlliance",,T,"Utility that generated the PQDS file"
                                waveform-data,va,vb,vc,ia,ib,ic

8. Meta-data PQDS Codes Descriptions

8.1 DeviceConnectionTypeCode

The PQDS code for meter connection type with the following codes predefined:

Code Description
0 unknown
1 3 Phase, 4 wires
2 3 Phase, 3 wires
3 1 Phase, 2 wires
4 1 Phase, 3 wires
5 2 Phase, 3 wires

8.2 EventTypeCode

The PQDS code for the event type with the following codes predefined:

Code Description
0 Unknown
1 Fault
2 Voltage sag
3 Voltage swell
4 Interruption
5 Harmonics

8.3 EventFaultTypeCode

The PQDS code for the fault type with the following codes predefined:

Code Description
0 Unknown
1 Single Line to Ground (LG)
2 Line to Line (LL)
3 Double Line to Ground (LLG)
4 3 Phase (LLL)
5 3 Phase to Ground (LLLG)

Note that the PQDS code for the event type is not the same as the event fault type. The event fault type is a subset of the event type.

For example, a voltage sag is an event type. The event fault type can be a single line to ground fault or a double line to ground fault.

8.4 EventCauseCode

The PQDS code for the event cause with the following codes predefined:

Code Description
0 Unknown
1 Lightning
2 Animal
3 Vegetation
4 Contamination
5 Smoke
6 Ice or Snow
7 Wind
8 Utility equipment issue
9 Customer equipment issue
10 Vandalism
11 Human error
12 Other

8.5 WaveformSensitivityCode

The PQDS code for the waveform data sensitivity with the following codes predefined:

Code Description
0 Internal use only (i.e., protected, confidential information)
1 Business sensitive (shared under NDA only)
2 Reliability sensitive (shared with other grid entities under NERC data sharing or other reliability or marketing service agreements)
3 Public

9. References

  1. Wikipedia, CSV, 2019, https://en.wikipedia.org/wiki/Comma-separated_values.
  2. Super CSV Package for JAVA, Kasper/GitHub, 2015, http://super-csv.github.io/super-csv/csv_specification.html.
  3. RFC 4180, Format for Comma-Separated Values, 2008 https://tools.ietf.org/html/rfc4180 https://www.loc.gov/preservation/digital/formats/fdd/fdd000323.shtml.