To clear out any confusion: Beneath has a broad interpretation of the term “stream” that includes bounded data collections (also known as batch data or static data). This interpretation is based on the philosophy that even bounded collections comes into existance, and will in turn be consumed, in some kind of sequence.
Streams in Beneath have some unique properties that you can learn more about below.
Records written to Beneath are automatically stored in a streaming log, a data warehouse and an operational data store. This means you can consume and query streams in the way that makes the most sense for your use case. You can learn a lot more about this feature in Unified data system.
All streams have an associated schema that defines the fields of its records. The schema must also define a key comprised of one or more columns (also known as a primary key or unique key). Beneath rejects records that do not adhere to the stream’s schema. See Schema definition for more details.
Records are not stored directly in a stream, but rather in a stream instance, which a stream can have many of. Each instance has an associated version number, which is a positive integer.
Every stream instance has two boolean flags: a) It can be marked as finalized, which indicates that no more records will be written to it, and b) it can be marked as primary, which makes it the default instance for the stream.
The following list gives some examples that clarify the usefulness of stream instances:
Every record in a stream instance has an associated timestamp. When you write a record, you can specify a custom timestamp or use the current timestamp of the Beneath server (the default). The timestamp comes in handy when several records with the same key are written to a stream instance. In these cases, the record with the highest timestamp is added in the operational data index.