Use the Pipeline API

A guide to generating and processing streams with the Pipeline API

Beneath Pipelines provide an abstraction over the basic Beneath APIs that makes it easier to develop, test, and deploy stream processing logic.

Beneath pipelines are currently quite basic and do not yet support joins and aggregations. They are still well-suited for generating streams, one-to-N stream derivation, as well as syncing and alerting records.

Install the Beneath SDK

If you haven’t already, follow the Install the Beneath SDK quick start to install and authenticate Beneath on your local computer.

Find a stream and head to API “Pipelines” tab

Browse the Beneath web console and navigate to a stream you want to process.

Click through to the stream’s “Pipelines” API: API > Python > Pipelines. There you’ll find a stream-specific tutorial on how to create a pipeline.

For an example, look at the API for earthquakes or r/wallstreetbets posts streams.

Select a code template and insert your logic

The Pipelines tab has two code templates:

  • Generate records for this stream. Follow this guide to consume an external data source and write the data to Beneath. You can also adapt the stream_path and schema params to make the pipeline produce an entirely new stream.
  • Derive a new stream. Follow this guide to apply processing logic, such as filtering or enrichment, to the stream.

Run and deploy your pipeline

At the bottom of the Pipelines tab, you’ll find instructions for how to run and deploy a pipeline. Once your pipeline is live, use the web console to view the pipeline’s output and monitor its activity.