Rapid predictive analysis of very large data sets using the distributed computational graph

What is this patent about?

'425 is related to the field of predictive analysis using distributed computational graphs. The background involves the increasing volume of data and the need for systems that can analyze streaming data in conjunction with vast amounts of stored data to make meaningful conclusions and enable effective action. Existing data pipelines are limited in their capabilities and are often linear, which restricts their use in complex situations requiring branching or recurrent modification.

The underlying idea behind '425 is to create a system that intelligently combines the processing of a current data stream with the ability to retrieve relevant stored data, enabling predictive conclusions or actions. This involves a distributed computational graph where data transformations are represented as nodes, and the flow of data between transformations is represented as edges. The system also monitors its own operations and intermediate factors to optimize function and maximize the probability of reliable conclusions.

The claims of '425 focus on a distributed computing network comprising first, second, and third pluralities of computer systems. The first computer system executes software instructions to receive a stream of data, process portions of it using a first transformation pipeline, and transmit the output. A second computer system processes the output using a second transformation pipeline. A third computer system stores data representing a portion of the distributed computational graph, describing the data flow between the pipelines. The third computer system monitors the execution of the pipelines and, in response, causes a fourth computer system to execute instructions that perform processing using either the first or second transformation pipeline.

In practice, the system receives streaming data from various sources, filters it, and splits it into two pathways: a streaming pathway and a batch pathway. The streaming pathway uses a transformation pipeline to perform real-time analysis, while the batch pathway stores the data for later analysis. A system sanity and retrain module monitors the progress of the analysis and adjusts the parameters of the system to optimize performance. The results of the analysis are then output in a pre-defined format.

This system differentiates itself from prior approaches by using a distributed computational graph that allows for non-linear transformation pipelines. This enables the system to handle more complex situations and to adapt to changing data streams. The system also includes a self-assessment mechanism that monitors its own operations and makes adjustments to optimize performance, which is a significant improvement over rigidly programmed data pipelines.