|
Grid Solutions Framework - GSF on Github
The Grid Solutions Framework (GSF) is a comprehensive collection of classes and methods useful for any .NET project. The GSF is the foundational code library for all GPA products. It includes hundreds of class libraries that extend or expand the functionality included in the .NET Framework. The GSF makes many of the more complex .NET features (e.g., sockets and encryption) easier to implement and adds functions not included within .NET. Currently, the GSF has over 490,000 lines of actual code and 148,000 lines of comments spanning more than 40 assemblies - reference the online documentation for complete class details.
As a few examples: the GSF communications library defines an abstract asynchronous communications framework, i.e., IServer / IClient, for socket (TCP, UDP, Multicast), serial and file-based communications; the GSF security framework includes components for implementing role-based security integrated with local, database driven, LDAP or ActiveDirectory accounts; and the GSF includes various utility protocol libraries, e.g., PQDIF, COMTRADE, IEEE C37.118, IEEE 1344, IEC 61850-90-5, UTK F-NET, SEL Fast Message and MMS.
The Grid Solutions Framework was created in 2012 as a combination of GPA's .NET 3.5/4.0 Time Series Framework and TVA's Code Library projects on CodePlex. Since its creation, a significant number of new code components have been added and the libraries have been re-factored to make the integrated GSF framework more secure and significantly better performing.
Primary GSF libraries are available as packages from nuget:
|
|
Time-Series Library – GSF on Github
The Time-Series Library (TSL) included within the Grid Solutions Framework is a powerful collection of methods to process real-time streaming data. Implemented with an adapter-based extensibility model for pluggable inputs, actions and outputs, the TSL allows fully customizable real-time data processing. Systems using the TSL inherit the ability to manage, process and respond to fast-moving time-series data at low latency.
The TSL can also be used to manage time-series data that includes an accurate time-stamp at the moment of measurement, such as values acquired from source devices with GPS time-sources. Time-series values with accurate time-stamps can be routed to user-developed adapters that inherit the ability to time-align, i.e., concentrate, values with other time-series values that were measured at the same time. Once time-series values are aligned by time, an action - or calculation - can be performed over a slice of all the data received for the same time-stamp.
Because of its modular design, systems implemented using the TSL become general purpose distributed stream processing engines. New fully self-contained, derived time-series based applications can be easily developed using the TSL that include all the needed user interface components for configuration management, database schemas for various databases systems (e.g., SQL Server, MySQL, Oracle and SQLite), and historical time-series data archiving. See the GSF Time-Series Library components overview for more information.
|
|
Project Alpha is a Visual Studio project template that provides developers a convenient starting point
from which to build new products (proprietary and open source) from the Time-Series Library.
|
|
|
Gateway Exchange Protocol - GSF on Github
The Gateway Exchange Protocol (GEP) is an open source measurement-based publish/subscribe transport protocol used for exchanging time-series data and automatically synchronizing meta-data between two applications. The protocol supports sending real-time and historical data at full or down-sampled resolutions. When sending historical data the replay speed can be controlled dynamically to provide data as quickly as possible, e.g., a data download, or slowed for visualization streaming.
The wire protocol employed by GEP implements a publish/subscribe data exchange model using a simple command driven service with tightly compressed, fast binary serialization of time-series values. The protocol does not require a predefined or fixed configuration – that is, the time-series values arriving in one data packet can be different than those arriving in another. Each packet data consists of a collection of time-series values; each time-series value is a structure containing an ID, a time-stamp, a value and associated flags.
The GEP code is a part of the Grid Solutions Framework. GEP subscription protocol API libraries are available in .NET (including Mono and Unity 3D), C++, and Java. See the GEP Overview and GEP Use Tutorial (ppt) for more information. A spreadsheet is available to estimate the bandwidth requirements for both IEEE C37.118 and GEP. See: GEP Bandwidth Estimator Spreadsheet (zipped)
The GEP Subscription Tester is a multi-platform graphical application that can be used to verify connectivity to applications implementing a GEP data publisher.
|
|
The SNAPdb Engine is a high-performance key/value pair archiving system. The SNAPdb is the archiving engine for the openHistorian 2.0. This technology was developed as a core data archival and retrieval system designed to significantly improve the ability to archive extremely large volumes of real-time streaming data and directly serve these large data volumes to consuming applications and systems. The SNAPdb code is contained within the Grid Solutions Framework.
The "SNAP" of the SNAPdb engine is an acronym for:
Serialized, NoSQL, ACID Compliant and Performant.
You can think of the SNAPdb engine like a high-speed, file-backed dictionary or hash-table.
The primary development goals of the SNAPdb engine were:
Speed
- In-memory cache for very high speed extraction of near-real time data
- Low data insertion lag time
- High-speed API for historical data extraction
Reliability
- ACID-based system design objectives with emphasis on “durability”
- File structure resistant to data corruption
Capability
- B+Tree based that supports out-of-sequence insertion
- Transaction-like data updates allowed
- Loss-less data compression
- Time support to +/- 100 nanoseconds (with extended time precision fields available)
- Accommodate multiple data types (any data type to value structure size)
Other Key Features
- Store data sequentially in compressed block structures
- Distribute and archive real-time data with low latencies
- Support for high-frame rate application refresh
- Quick query and application response time
- Ability to extract or process large and very large data blocks, e.g., post-process day of high-resolution data
- Use sequential I/O to increase disk performance with lower hardware burden (SSDs not required)
|
|
Broccoli Sharp – https://github.com/GridProtectionAlliance/BroccoliSharp
Developed as part of the ARMORE project, Broccoli Sharp is a C#-based API wrapper for the Bro intrusion detection system API called Broccoli, the "Bro client communications library". This interface allows Bro users to leverage existing .NET technologies such as the the Grid Solution Framework when using Bro. The library consists of a set of managed C# based .NET classes and structures that wrap Broccoli functionality.
The Broccoli Sharp library fully implements the Broccoli API and will allow Bro applications, e.g., ARMORE, to create client sensors, receive Bro IDs, send and receive Bro events, as well as send and receive event requests to and from peering Bros. Additionally, just as in the native C Broccoli API, this .NET library can be used to create and receive values of pure types like integers, counters, timestamps, IP addresses, port numbers, booleans, and strings. As Bro is normally only deployed in Linux environments, the BroccoliSharp library was developed to run in conjunction with Mono, an open source implementation of the .NET framework that works on various platforms, including Linux.
BroccoliSharp, along with complete documentation, is available as an open source project on GitHub. In order to be fully compatible with other Bro projects, this library is published under terms of the BSD license.
|
|
GPA utilizes leading-edge technologies in the development of new products. The specific technology selected varies based on the target platform and suitability to the deliverable. Unless there is a clear advantage from a competing technology, GPA uses .NET when possible because of its rich set of development tools and wide deployment options; however, our expertise includes many other platforms and technologies.
Development Technologies
A strength of GPA is in its ability to rapidly develop new, practical technologies for immediate application in industry environments using development technologies that allow the widest possible deployments.
- .NET - Powerful, multi-language, open source, general purpose software development framework
- Mono - Cross-platform, open source implementation of the .NET framework
- Unity 3D - High-speed graphical development platform for next generation visualizations
- C / C++ - General purpose language for multi-platform system level development
- Log4Net - Hierachical logging component that supports many output targets
- Java - Commonly used, open source, cross-platform development framework
- WiX Toolset - XML driven toolset used to create Windows-based installation packages
Data Technologies
When dealing with both traditional data and Big Data, GPA uses various technologies and brings years of data driven experience to bear for new development.
- SQL Server - Fast and powerful DBMS with embedded integration with .NET technologies for complete data management
- SQL Server Integration Services - (SSIS) Data-warehousing tool commonly used for extract, transform and load (ETL) functionality
- SQL Server Reporting Services - (SSRS) Reporting development and deployment framework
- SQL Server Analysis Services - (SSAS) Multidimensional analysis (OLAP) tool for powerful data mining
- Oracle - Widely deployed and powerful DBMS with rich technology integration for general purpose data storage
- MySQL - Commonly used open source DBMS for general purpose data storage
- SQLite - Public domain, light-weight, open source DBMS for general purpose data storage
- Hadoop - Framework for the distributed processing of large data sets across clusters of computers
- OSI-PI - Widely used commercial time-series data historian system
- ZeroMQ - A light-weight messaging technology with adapters available in many languages that scales well and which enables message queuing, routing and filtering. ZeroMQ can be deployed with or without a dedicated message broker.
Web Technologies
For browser-based tool development, GPA uses all of the latest technologies and techniques to bring the best possible experience to the user
- AJAX - The overarching architecture for rich client side web experience
- HTML5 - Markup and Canvas support
- CSS3 - Styling and customized themes
- JavaScript - Client side processing and user interface behavior language
- jQuery - Cross browser and mobile JavaScript library that provides rich functionality
- JQWidgets - jQuery plugin that provides a multitude of controls tightly integrated with jQuery
- JQuery User Interface - Curated set of user interface interactions, effects, widgets, and themes built on top of the jQuery JavaScript Library
- OpenLayers - A high-performance, feature-packed library for all mapping needs
- HighCharts - Highcharts is a charting library written in pure JavaScript, offering an easy way of adding interactive charts to your web site or web application
|
|
|
