Rapid predictive analysis of very large data sets using the distributed computational graph

What is this patent about?

'424 is related to the field of analyzing large datasets using a distributed computational graph. Traditional data analysis pipelines are often linear and limited in their ability to handle complex, changing data streams. The background highlights the increasing volume of data and the need for tools that can analyze this data in real-time, drawing meaningful conclusions and enabling effective action. Existing systems lack the ability to self-assess and adapt to optimize performance.

The underlying idea behind '424 is to create a system that combines real-time processing of streaming data with the ability to retrieve and analyze relevant stored data. This is achieved through a distributed computational graph where data transformations are represented as nodes and data flow as edges. The system monitors its own operations and intermediate results, adapting its function to optimize performance and ensure reliable conclusions. This involves both a streaming pathway and a batch analysis pathway, allowing for predictive analysis based on both current and historical data.

The claims of '424 focus on a distributed computing cluster comprising a first and second plurality of computer systems. The first plurality stores portions of a computational graph, describing the flow of data from a first transformation pipeline to a second. A first computer system receives a stream of input data, processes it using the first pipeline, determines information about the second pipeline, and transmits the output to a second computer system. The second computer system then processes the received data using the second pipeline. A third computer system causes a fourth to execute at least one of the pipelines.

In practice, the system receives streaming data from various sources, filters it, and splits it into two pathways: a streaming pathway and a batch pathway. The streaming pathway uses a transformation pipeline to perform real-time analysis, while the batch pathway stores and analyzes historical data. The system sanity and retrain module monitors the progress of the analysis, optimizing coordination between the two pathways and adapting the system's behavior as needed. The messaging module facilitates communication between the different components, ensuring that the system can respond to changing conditions and new information.

The invention differentiates itself from prior approaches by enabling non-linear transformation pipelines, including afferent branch, efferent branch, and cyclical configurations. This allows for more complex and iterative analyses. The system also incorporates a system sanity and retrain module that monitors the progress of the analysis and adjusts the system's behavior to optimize performance. This self-adaptive capability is crucial for handling the vast amounts of data and the dynamic nature of the analysis tasks.