GPA Grid Protection Alliance
Open Source Software & Services for Electric Utilities
The PQDS File Specification |
The PQio application provides a quick means to convert PQDIF files to the PQDS.csv format so that event waveform data can be easily reviewed, obfuscated or edited if necessary, and shared with colleagues. The Power Quality Data Sharing (PQDS) format prescribes the way to structure a CSV file to exchange PQ data -- both waveform and trending data -- in a simple, human readable form. In many GPA applications, like openSEE and TrenDAP, when clicking on the "Export as CSV" button the CSV file produced will comply with the PQDS format standard so that this file can be parsed and used by PQDS compatible systems. Version 1.2 of this specification is now available in pre-release for review. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Version 1.1Version 1.1 - February 2021 -- This version of the PQDS file standard includes changes that enable trending, or interval, data to be included in PQDS files. 1. IntroductionThe Power Quality Data Sharing (PQDS) format is a comma-separated file that facilitates the convenient review, editing and sharing of Power Quality (PQ) data -- i.e., disturbance waveforms and trending data values. PQDS is a simple human-to-human format that complements the standard system-to-system data formats of PQDIF and COMTRADE. The Comma-Separated Value (CSV) file format used by PQDS is supported by almost all spreadsheets and database management systems as well as by standard libraries that are available in most programming languages. In addition, many software applications have the ability to split files based on comma separated values. [1] The PQDS file format contains two sections -- (1) a metadata section with key-value pair information that describe the measurement values, and (2) a section for measurement values themselves. The measurement values section can contain either time-series voltage and current waveform measurements or time series trending measurements consisting of minimum, maximum and average values over an interval, e.g., 10 minutes. Each of these two sections individually comply with the RFC 4180 CSV format specification with header data to identify the names of the fields that follow. Parsers for PQDS files will expect these files to comply with the seven RFC 4180 rules for CSV formatted files. [2] The PQDS file format and tools to convert PQDIF and COMTRADE waveform data to PQDS has been developed by the Electric Power Research Institute (EPRI) to support utility disturbance monitoring research. The EPRI PQio program is building a library of waveforms that can be used to evaluate and test automated waveform analytics -- including analytics involving artificial intelligence (AI) techniques. 1.1 SimplicityAll metadata and measurement values included within PQDS files are in standard engineering units without the need for subsequent scaling or adjustment. If included, the optional PQDS metadata can be used to provide details about a single event in the waveform or general information about the dataset. While complex multi-event cases can be be included in the measurement data itself, no provisions are made for describing multiple events in the PQDS metadata section. 1.2 ExtensibilityUsers and groups of users can easily extend both the PQDS metadata as well as PQDS time-series waveform data measurement values with low risk of breaking any automation that may have been implemented by other users who have implemented different extensions. 1.3 ShareabilityPQDS format is designed to make it easier for the data-owner to improve productivity through collaboration with others -- the business-basis for development of the PQDS format.
The PQDS format is designed to facilitate time-series measurement value sharing in all its forms -- for example, as waveform data may need to be shared in full detail among work colleagues, or as data may need to be contributed to repositories for power system phenomenon research and analytics improvement. The PQDS supports including (or not including) just the waveform data that is needed for an individual use case. If necessarily, the waveform data itself can also be easily trimmed down to just to portion of the event that is of interest. 2. PQDS MetadataThe PQDS metadata section begins with a header record than contains the following field names:
Importantly, the entire metadata section of the PQDS.csv file is NOT required and may be omitted entirely without impact to processing the waveform or trending data. 2.1 Reserved PQDS Metadata keysThe following key names are reserved to facilitate use of PQDS to support automated data sharing. Users are welcome to add keys as they may be needed. 2.1.1 metadata about device making the measurements:
2.1.2 metadata about the electrical asset or load being monitored:
2.1.3 metadata about the nature of the disturbance in the waveform:Note that these fields are primarily associated with disturbance data and might not apply in the trending-data case.
2.1.4 Metadata about the nature of the trending data in the fileNote that these fields are primarily associated with tredning data and might not apply in the disturbance data case.
2.1.5 Metadata about the measurements in the time-series data
2.1.6 Metadata about use restrictions or information classification of the waveform
2.2 Adding New Keys -- User Extensions to metadataAdditional keys and values can be added to the metadata section as needed. As these new keys are created the pattern of "domain"."key" should be followed with the "domain" typically being a corporate or organizational abbreviation. This approach avoids key conflicts as metadata is independently extended by multiple parties. e.g., extended metadata key = "EPRI.SequenceID" 3. PQDS Measurement ValuesThe measurement values section in the PQDS file follows the optional PQDS metadata section. There are two measurement data formats -- one for disturbance waveform values one for trending data (min, max, avg) values, In both cases, the first line after the PQDS metadata section is a header record (or line) which describes the type of data that follows ("waveform" or "trending") as well as the labels for the measurement values. The following two sections describe each format in more detail. 3.1 Waveform ValuesThe header record (line) for waveform-data is comprised of the following fields and MUST begin with the string "waveform-data" (not case sensitive).
3.1.1 Adding New Measurements / User Extensions to Waveform DataAdditional measurements can be added as additional 'columns' to the standard voltage and current measurement quantities shown in Note 6 above. 3.2 Trending ValuesThe header record (line) for trending, or trending, data is comprised of the following fields and MUST begin with the string "trending-data" (not case sensitive).
The mesurement data following the header record consists of the time-stamp as described in Note 7 followed by a tripplet of values for each measurement. These values are seperated by >>,<< and are always ordered as minimum, average, maximum. 3.1.1 Trending Values Spanning Multiple DaysSince trending values are typically recorded over relatively long intervals (e.g., 10 minutes), a common use case for PQDS will be to share trending data for a long periods -- many days. In this case an additional day-header is inserted to define the start of a new day. This header can be skipped for the first day (Day 0)
3.1.2 Adding New Measurements -- User Extensions to Waveform DataAdditional measurements can be added as additional 'columns' to the standard voltage and current measurement quantities shown in Note 8 above. 4. Expected PQDS Parser BehaviorIf the optional PQDS metadata section is included in the PQDS file, it must begin with the three header field names of "key", "value", "value-units" (not case sensitive). metadata keys do not need to appear in any particular order. Empty lines in the metadata section will be ignored. All fields will be ignored for invalid keys (e.g., one containing special characters or with a length greater than 32 characters). Valid keys will contain 'null' for other fields that are invalid. Other fields may be added to each metadata line (metadata record), but will not be parsed. PQDS parsers are expected to validate metadata values based on the data-type rules. 4.1 Including Embedded Quotes and CommasParsers of PQDS should be expected to behave like Excel when handling strings that include commas and quotes. For example the string: >> "I find that the harder
I work, the more luck I seem to have.", Thomas Jefferson << is placed in the CSV as 5. The PQDS File Naming ConventionThere is purposely no standard for PQDS file naming requirements. It is recommended that the PQDS file extension be ".csv" 6. Example PQDS.csv File6.1 PQDS Containing Waveform Values
6.2 PQDS Containing Trending Values
7. PQDS Log Files7.1 IntroductionThe Power Quality Data Sharing (PQDS) log file format is a comma-separated value file that enables easy tracking and sharing of the metadata not included in the PQDS file for various reasons, such as anomyizing the electric power disturbance waveforms. It is designed to be a simple and extensible human-to-human data format that complements the PQDS file format. The PQDS log file format can include multiple sections, each corresponding to a single PQDS file generated by the EPRI PQDS application. The EPRI PQDS application will append a new section to the existing log file for each PQDS file generated. 7.2 Header SectionEach section corresponding to a PQDS file is separated by a single line of 42 >>-<< characters. The first section skips this seperator and imidiately starts with the identifying header. 7.3 Identifying HeaderEach section contains a line consisting of two fields separated by a comma. The first field is the date and time at which the corresponding PQDS file is generated. The field is encapsulated in double quotations and formated as d/m/YYYY hh:mm:ss. The date and time are given in UTC to avoid any confusion over the time zone in which the file was generated and formated using a 24 hour format. The second field is a uniquely generated GUID associated with the PQDS file. This GUID corresponds to the EventGUID field in the meta data section of the PQDS file as found in Section 2.1 of this document. Note that the field is also encapsulated by ". 7.4 Meta Data SectionImidiately following the identifying header the meta data section of the PQDS file is included. Unlike the PQDS file itself all available meta data should be included in the log file. The format of this section is the same as that described in Section 2.1 this document. 7.5 Data SectionThe log file does not contain any actual data. However, in order to better describe the content of the PQDS file a header similar to that of the wave form or tredning data section in the PQDS file is included. The entry starts with the word >>waveform-data<< or >>trending-data<< , followed by comma separated list of the channels included in the PQDS file. Suggested values include >>va<<, >>vb<<, >>vc<<, >>vn<<, >>ia<<, >>ib<<, >>ic<<, >>in<<, and >>f<<. 7.6 Example PQDS log File
8. metadata PQDS Codes Descriptions8.1 DeviceConnectionTypeCodeThe PQDS code for meter connection type with the following codes predefined:
8.2 EventTypeCodeThe PQDS code for the event type with the following codes predefined:
8.3 EventFaultTypeCodeThe PQDS code for the fault type with the following codes predefined:
8.4 EventCauseCodeThe PQDS code for the event cause with the following codes predefined:
8.5 WaveformSensitivityCodeThe PQDS code for the waveform data sensitivity with the following codes predefined:
9. References
|