To clear out any confusion: Beneath has a broad interpretation of the term “stream” that includes bounded data collections (also known as batch data or static data). This interpretation is based on the philosophy that even bounded collections comes into existance, and will in turn be consumed, in some kind of sequence.
Streams in Beneath have some unique properties that you can learn more about below.
All streams have an associated schema that defines the fields of its records. The schema must also define a key comprised of one or more columns (also known as a primary key or unique key). Beneath rejects records that do not adhere to the stream’s schema.
Records are not stored directly in a stream, but rather in a stream instance, which a stream can have many of. You can think of them as different versions of the stream.
Every stream instance has two boolean flags: a) It can be marked as finalized, which indicates that no more records will be written to it, and b) it can be marked as primary, which makes it the default instance for the stream, causing it to be shown in the Terminal.
The following list gives some examples that clarify the usefulness of stream instances:
Every record in a stream instance has an associated timestamp. When you write a record, you can specify a custom timestamp or use the current timestamp of the Beneath server (the default). The timestamp comes in handy when several records with the same key are written to a stream instance. In these cases, the record with the highest timestamp is added in the operational data index.
Records written to Beneath are automatically made stored in several systems, which are useful for different purposes. These are:
To learn more about these systems, see unified data system.
The following pages in this section explain more about how to use these systems in practice.