Beneath aims to use a single git repository (monorepo) for all code and documentation. Hopefully that makes it easier to manage dependencies and get a full view of development.
Checkout the git repo with:
git clone https://gitlab.com/beneath-hq/beneath.git
Client libraries (i.e. language-specific libraries that read from and write to Beneath) are found in the
clients/ directory. As an example, see the Python client library.
The entire system is implemented in Go with the exception of the frontend, which is implemented in TypeScript (and React), and the client libraries, which are implemented for a variety of different languages.
The core system spans multiple separate executable services, namely:
control/), which manages users, projects, streams, etc. It exposes a GraphQL API and is backed by Postgres and Redis.
control/taskqueue/), which executes background jobs issued by the control server.
gateway/), which reads and writes data to Beneath. It exposes REST and gRPC APIs and interfaces with the underlying infrastructure using its pipeline and the engine (found in
gateway/pipeline/), which in the background receives data from the gateway and writes it to the engine.
Here’s a rough guide to the project structure:
assets: Assets such as logos, images, etc.
build: Dockerfiles for the deployed executables in
clients: Language-specific libraries used to interface with Beneath, including reading and writing data
cmd: contains a package with a
main.gofile for each executable program
configs: should contain
.test.envfiles that configure the platform in local development
control: code related to the control server, including Postgres models, GraphQL resolvers, and the task queue
deployments: Kubernetes and Helm manifests for deploying to production
docs/contributing: documentation that describes how the codebase is structured and how to contribute to it
engine: drivers for interfacing with the underlying data infrastructure, e.g., Pubsub, BigTable, BigQuery, etc.
gateway: code related to the data gateway server (excluding data infrastructure code – that’s in
internal: utility libraries that are not directly related to any specific service (e.g., HTTP middleware)
pkg: stand-alone utility libraries that are not directly related to any specific service (e.g., the stream schema parser)
scripts: contains helper scripts for code generation, etc.
test: contains end-to-end and integration tests
web: contains the frontend (data console) code