Overview: Python Streams

IoTPy is a small Python library that helps you to build applications that process streams of data. Sensors, social media, news feeds, webcams and other sources generate streams of data. Stream applications ingest and analyze streams to control actuators, generate alerts, and feed continuous displays. Structural health monitoring applications analyze streams from accelerometers and strain gauges; social media applications analyze Tweets, and messages; power-grid applications analyze streams of voltage and current readings.

Stream applications have two key characteristics.

  1. They often run forever. Power grid, health monitoring, social media, tsunami warning and other S&R applications are designed to keep operating. As a consequence data structures such as streams can have an unbounded size.

  2. They run on heterogeneous platforms. Many stream applications process data at the edge, where it is generated, and send summarized information to other computers for further processing. For example an accelerometer may generate 250 samples per second, but only a few seconds a day may be relevant to crisis monitoring application. Concurrent computations occur at multiple levels. (1) Sensors and actuators run in separate threads. (2) Multiple processes in a multicore computer execute using shared memory(3) Multiple machines send messages across a distributed network.

Programs in most Python libraries do not run forever: they read input, produce output and terminate. Each execution of a program operates on independent data of bounded size. Most programs cannot be mapped easily across multithreaded, multiprocessor and distributed memory platforms.

THE TWO GOALS OF IOTPY

  1. Python Libraries: Superb Python libraries, such as NumPy, SciPy, and SciKit were not designed to support streaming. IoTPy leverages their power to build streaming applications.

  2. Python Concurrency: Create applications on concurrent platforms by simply connecting streams. IoTPy uses a single construct - connect input and output streams together - to deal with all types of concurrency.

IoTPy is motivated by applications in "Event Processing: Designing IT Systems for Agile Companies" with Roy Schulte of Gartner.

Disadvantages of IoTPy

IoTPy is a work in progress. IoTPy has not been optimized for performance. IoTPy does not have its own security protocols for execution on distributed computing systems; it uses the protocols of the platforms on which it runs.

Getting Started

The code is on GitHub:

https://github.com/AssembleSoftware/IoTPy.git

You will need to download fewer than 5 files from the GitHub repository for most applications. The repository has a single core file, stream.py, and many files containing example programs for a variety of streaming applications. Download stream.py and the handful of files containing stream operators that fit your specific problem. You do not need to download the entire repository.

The best way to look at the documentation and code is to run Jupyter notebooks. Look at

You can also look at the source code for the corresponding Python files; the Python file corresponding to the Jupyther notebook x.ipynb is x.py

Next: Streams