Stream processing in IoT environment

is designed for low-latency processing of streaming data at the edge of the network. IoT devices frequently generate large volumes of unstructured streaming data, such as video and audio streams. Even if the data streams are structured, they may be meaningless if their temporal characteristics are not considered. Cloud-based services are generally not good at processing these kinds of data. Preprocessing data streams before they are sent to the cloud makes large scale data processing in the cloud more efficient and reduces the usage of network bandwidth. To handle such data streams in the challenging environment of IoT, SensorBee was designed to be:

Stateful

Extracting useful information from unstructured data often requires machine learning. SensorBee works well with online machine learning algorithms by storing machine learning models as user-defined states. It supports various machine learning toolkits, including Chainer and some components of Jubatus. In addition, SensorBee’s query language enables stateful temporal data handling.

Expressive

Operations and queries are written in BQL, a dialect of CQL (Continuous Query Language). BQL is a very powerful language and is easy to learn for people already familiar with SQL. BQL is schemaless, and uses a JSON-like data structure as its internal data structure. It also provides many features useful for dealing with JSON.

Lightweight

SensorBee is lightweight: it has a minimum footprint of less than 30MB and can be run on a small computer such as Raspberry Pi. Although SensorBee is currently not small enough to be embedded in a very small device, a future development goal is to enable BQL statements to run on a very weak chip.

Getting Started

Install Go

is written in Go and it requires that Go be installed in advance. Please look at the official installation guide to install Go tools and set up its development environment (i.e. setting GOROOT and GOPATH environment variables).

Install SensorBee

$ go get gopkg.in/sensorbee/sensorbee.v0/...
sensorbee and build_sensorbee commands are installed under $GOPATH/bin.

Run SensorBee

$ sensorbee run

Build Custom sensorbee Command

Because SensorBee is written in Go, all plugins are statically linked. In order to add a plugin, a custom sensorbee command must be built. 
$ ls
build.yaml
$ cat build.yaml
plugins:
  - github.com/sensorbee/twitter/plugin
$ build_sensorbee
sensorbee_main.go
$ ls
build.yaml
sensorbee
sensorbee_main.go