Deploying Cube Core with Docker

This is an example of a production-ready deployment, but real-world deployments can vary significantly depending on desired performance and scale.

If you’d like to deploy Cube to Kubernetes, please refer to the following resources with Helm charts: gadsme/charts or OpstimizeIcarus/cubejs-helm-charts-kubernetes.These resources are community-maintained, and they are not maintained by the Cube team. Please direct questions related to these resources to their authors.

Prerequisites

Docker Desktop

Configuration

Create a Docker Compose stack by creating a docker-compose.yml. A production-ready stack would at minimum consist of:

One or more Cube API instance
A Cube Refresh Worker
A Cube Store Router node
One or more Cube Store Worker nodes

An example stack using BigQuery as a data source is provided below:

Using macOS or Windows? Use CUBEJS_DB_HOST=host.docker.internal instead of localhost if your database is on the same machine.

Using macOS on Apple Silicon (arm64)? Use the arm64v8 tag for Cube Store Docker images, e.g., cubejs/cubestore:arm64v8.

Note that it’s a best practice to use specific locked versions, e.g., cubejs/cube:v0.36.0, instead of cubejs/cube:latest in production.

services:
  cube_api:
    restart: always
    image: cubejs/cube:latest
    ports:
      - 4000:4000
    environment:
      - CUBEJS_DB_TYPE=bigquery
      - CUBEJS_DB_BQ_PROJECT_ID=cube-bq-cluster
      - CUBEJS_DB_BQ_CREDENTIALS=<BQ-KEY>
      - CUBEJS_DB_EXPORT_BUCKET=cubestore
      - CUBEJS_CUBESTORE_HOST=cubestore_router
      - CUBEJS_API_SECRET=secret
    volumes:
      - .:/cube/conf
    depends_on:
      - cube_refresh_worker
      - cubestore_router
      - cubestore_worker_1
      - cubestore_worker_2

  cube_refresh_worker:
    restart: always
    image: cubejs/cube:latest
    environment:
      - CUBEJS_DB_TYPE=bigquery
      - CUBEJS_DB_BQ_PROJECT_ID=cube-bq-cluster
      - CUBEJS_DB_BQ_CREDENTIALS=<BQ-KEY>
      - CUBEJS_DB_EXPORT_BUCKET=cubestore
      - CUBEJS_CUBESTORE_HOST=cubestore_router
      - CUBEJS_API_SECRET=secret
      - CUBEJS_REFRESH_WORKER=true
    volumes:
      - .:/cube/conf

  cubestore_router:
    restart: always
    image: cubejs/cubestore:latest
    environment:
      - CUBESTORE_WORKERS=cubestore_worker_1:10001,cubestore_worker_2:10002
      - CUBESTORE_REMOTE_DIR=/cube/data
      - CUBESTORE_META_PORT=9999
      - CUBESTORE_SERVER_NAME=cubestore_router:9999
    volumes:
      - .cubestore:/cube/data

  cubestore_worker_1:
    restart: always
    image: cubejs/cubestore:latest
    environment:
      - CUBESTORE_WORKERS=cubestore_worker_1:10001,cubestore_worker_2:10002
      - CUBESTORE_SERVER_NAME=cubestore_worker_1:10001
      - CUBESTORE_WORKER_PORT=10001
      - CUBESTORE_REMOTE_DIR=/cube/data
      - CUBESTORE_META_ADDR=cubestore_router:9999
    volumes:
      - .cubestore:/cube/data
    depends_on:
      - cubestore_router

  cubestore_worker_2:
    restart: always
    image: cubejs/cubestore:latest
    environment:
      - CUBESTORE_WORKERS=cubestore_worker_1:10001,cubestore_worker_2:10002
      - CUBESTORE_SERVER_NAME=cubestore_worker_2:10002
      - CUBESTORE_WORKER_PORT=10002
      - CUBESTORE_REMOTE_DIR=/cube/data
      - CUBESTORE_META_ADDR=cubestore_router:9999
    volumes:
      - .cubestore:/cube/data
    depends_on:
      - cubestore_router

Set up reverse proxy

In production, the Cube API should be served over an HTTPS connection to ensure security of the data in-transit. We recommend using a reverse proxy; as an example, let’s use NGINX.

You can also use a reverse proxy to enable HTTP 2.0 and GZIP compression

First we’ll create a new server configuration file called nginx/cube.conf:

server {
  listen 443 ssl;
  server_name cube.my-domain.com;

  ssl_protocols               TLSv1 TLSv1.1 TLSv1.2;
  ssl_ecdh_curve              secp384r1;
  # Replace the ciphers with the appropriate values
  ssl_ciphers                 "ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384 OLD_TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 OLD_TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256";
  ssl_prefer_server_ciphers   on;
  ssl_certificate             /etc/ssl/private/cert.pem;
  ssl_certificate_key         /etc/ssl/private/key.pem;
  ssl_session_timeout         10m;
  ssl_session_cache           shared:SSL:10m;
  ssl_session_tickets         off;
  ssl_stapling                on;
  ssl_stapling_verify         on;

  location / {
    proxy_pass http://cube:4000/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }
}

Then we’ll add a new service to our Docker Compose stack:

services:
  ...
  nginx:
    image: nginx
    ports:
      - 443:443
    volumes:
      - ./nginx:/etc/nginx/conf.d
      - ./ssl:/etc/ssl/private

Don’t forget to create a ssl directory with the cert.pem and key.pem files inside so the Nginx service can find them. For automatically provisioning SSL certificates with LetsEncrypt, this blog post may be useful.

Security

Use JSON Web Tokens

Cube can be configured to use industry-standard JSON Web Key Sets for securing its API and limiting access to data. To do this, we’ll define the relevant options on our Cube API instance:

If you’re using queryRewrite for access control, then you must also configure scheduledRefreshContexts so the refresh workers can correctly create pre-aggregations.

services:
  cube_api:
    image: cubejs/cube:latest
    ports:
      - 4000:4000
    environment:
      - CUBEJS_DB_TYPE=bigquery
      - CUBEJS_DB_BQ_PROJECT_ID=cube-bq-cluster
      - CUBEJS_DB_BQ_CREDENTIALS=<BQ-KEY>
      - CUBEJS_DB_EXPORT_BUCKET=cubestore
      - CUBEJS_CUBESTORE_HOST=cubestore_router
      - CUBEJS_API_SECRET=secret
      - CUBEJS_JWK_URL=https://cognito-idp.<AWS_REGION>.amazonaws.com/<USER_POOL_ID>/.well-known/jwks.json
      - CUBEJS_JWT_AUDIENCE=<APPLICATION_URL>
      - CUBEJS_JWT_ISSUER=https://cognito-idp.<AWS_REGION>.amazonaws.com/<USER_POOL_ID>
      - CUBEJS_JWT_ALGS=RS256
      - CUBEJS_JWT_CLAIMS_NAMESPACE=<CLAIMS_NAMESPACE>
    volumes:
      - .:/cube/conf
    depends_on:
      - cubestore_worker_1
      - cubestore_worker_2
      - cube_refresh_worker

Securing Cube Store

All Cube Store nodes (both router and workers) should only be accessible to Cube API instances and refresh workers. To do this with Docker Compose, we simply need to make sure that none of the Cube Store services have any exposed

Monitoring

All Cube logs can be found by through the Docker Compose CLI:

docker-compose ps

           Name                           Command               State                    Ports
---------------------------------------------------------------------------------------------------------------------------------
cluster_cube_1                 docker-entrypoint.sh cubej ...   Up      0.0.0.0:4000->4000/tcp,:::4000->4000/tcp
cluster_cubestore_router_1     ./cubestored                     Up      3030/tcp, 3306/tcp
cluster_cubestore_worker_1_1   ./cubestored                     Up      3306/tcp, 9001/tcp
cluster_cubestore_worker_2_1   ./cubestored                     Up      3306/tcp, 9001/tcp

docker-compose logs

cubestore_router_1    | 2021-06-02 15:03:20,915 INFO  [cubestore::metastore] Creating metastore from scratch in /cube/.cubestore/data/metastore
cubestore_router_1    | 2021-06-02 15:03:20,950 INFO  [cubestore::cluster] Meta store port open on 0.0.0.0:9999
cubestore_router_1    | 2021-06-02 15:03:20,951 INFO  [cubestore::mysql] MySQL port open on 0.0.0.0:3306
cubestore_router_1    | 2021-06-02 15:03:20,952 INFO  [cubestore::http] Http Server is listening on 0.0.0.0:3030
cube_1                | 🚀 Cube API server (vX.XX.XX) is listening on 4000
cubestore_worker_2_1  | 2021-06-02 15:03:24,945 INFO  [cubestore::cluster] Worker port open on 0.0.0.0:9001
cubestore_worker_1_1  | 2021-06-02 15:03:24,830 INFO  [cubestore::cluster] Worker port open on 0.0.0.0:9001

Update to the latest version

Find the latest stable release version from Docker Hub. Then update your docker-compose.yml to use a specific tag instead of latest:

services:
  cube_api:
    image: cubejs/cube:v0.36.0
    ports:
      - 4000:4000
    environment:
      - CUBEJS_DB_TYPE=bigquery
      - CUBEJS_DB_BQ_PROJECT_ID=cube-bq-cluster
      - CUBEJS_DB_BQ_CREDENTIALS=<BQ-KEY>
      - CUBEJS_DB_EXPORT_BUCKET=cubestore
      - CUBEJS_CUBESTORE_HOST=cubestore_router
      - CUBEJS_API_SECRET=secret
    volumes:
      - .:/cube/conf
    depends_on:
      - cubestore_router
      - cube_refresh_worker

Extend the Docker image

If you need to use dependencies (i.e., Python or npm packages) with native extensions inside configuration files or dynamic data models, build a custom Docker image. You can do this by creating a Dockerfile and a corresponding .dockerignore file:

touch Dockerfile
touch .dockerignore

Add this to the Dockerfile:

FROM cubejs/cube:latest

COPY . .
RUN apt update && apt install -y pip
RUN pip install -r requirements.txt
RUN npm install

And this to the .dockerignore:

model
cube.py
cube.js
.env
node_modules
npm-debug.log

Then start the build process by running the following command:

docker build -t <YOUR-USERNAME>/cube-custom-image .

Finally, update your docker-compose.yml to use your newly-built image:

services:
  cube_api:
    image: <YOUR-USERNAME>/cube-custom-image
    ports:
      - 4000:4000
    environment:
      - CUBEJS_API_SECRET=secret
      # Other environment variables
    volumes:
      - .:/cube/conf
    depends_on:
      - cubestore_router
      - cube_refresh_worker
      # Other container dependencies

Note that you shoudn’t mount the whole current folder (.:/cube/conf) if you have dependencies in package.json. Doing so would effectively hide the node_modules folder inside the container, where dependency files installed with npm install reside, and result in errors like this: Error: Cannot find module 'my_dependency'. In that case, mount individual files:

    # ...
    volumes:
      - ./model:/cube/conf/model
      - ./cube.js:/cube/conf/cube.js
      # Other necessary files

Production checklist

Thinking of migrating to the cloud instead? Click here to learn more about migrating a self-hosted installation to Cube Cloud.

This is a checklist for configuring and securing Cube for a production deployment.

Disable Development Mode

When running Cube in production environments, make sure development mode is disabled both on API Instances and Refresh Worker. You can read more about what development mode changes here.

Development mode is an authentication bypass. Cube is in development mode when CUBEJS_DEV_MODE=true, and also whenever NODE_ENV is not production. When Cube is started through the cubejs CLI — which is what the official Docker images run — setting CUBEJS_DEV_MODE=true additionally forces NODE_ENV=development, which switches off JWT verification on the REST (JSON) and GraphQL APIs: they then accept requests with no token at all.Development mode also mounts Playground and its supporting endpoints with no authentication whatsoever. Anyone who can reach the instance is handed a ready-to-use API token, and can mint further ones carrying any security context signed with your API secret — and so query every data API as any user, bypassing member-level access control and row-level security. The same endpoints read your data model files and the table schema of every connected data source, and overwrite your data model and your .env. With CUBEJS_DEV_MODE=true and no CUBEJS_SQL_PASSWORD set, the SQL API accepts any credentials as well, allowing arbitrary SQL against connected data sources.This is intentional. Development mode is designed to run on a developer’s local machine for ease of use and debugging. Never run it where anyone else can reach it, never expose it to the internet, and never use it in production. Using development mode in the Cube cloud platform is highly discouraged — it bypasses the platform’s security model.To keep it off: cubejs server and the official Docker images already set NODE_ENV=production, so leaving CUBEJS_DEV_MODE unset — its default — is enough there. If you embed @cubejs-backend/server-core directly rather than starting Cube through the cubejs CLI, set NODE_ENV=production yourself, since an unset NODE_ENV puts the instance in development mode whatever the flag says.

CUBEJS_DEV_MODE defaults to false, and the official Docker images already set NODE_ENV=production — as does cubejs server itself — so on a Docker deployment leaving the flag unset is enough. Setting NODE_ENV below as well is belt-and-braces; it matters only if you embed @cubejs-backend/server-core directly, where an unset NODE_ENV puts the instance in development mode whatever the flag says.

# Keep development mode off. Under the `cubejs` CLI and the official images,
# setting this to `true` also forces NODE_ENV=development, switching off JWT
# verification on the data APIs
CUBEJS_DEV_MODE=false

# Already set by the official images and by `cubejs server`; set it explicitly if
# you start Cube some other way
NODE_ENV=production

Set up Refresh Worker

To refresh in-memory cache and pre-aggregations in the background, we recommend running a separate Cube Refresh Worker instance. This allows your Cube API Instance to continue to serve requests with high availability.

# Set to true so a Cube instance acts as a refresh worker
CUBEJS_REFRESH_WORKER=true

Set up Cube Store

While Cube can operate with in-memory cache and queue storage, there’re multiple parts of Cube which require Cube Store in production mode. Replicating Cube instances without Cube Store can lead to source database degraded performance, various race conditions and cached data inconsistencies.

Cube Store manages in-memory cache, queue and pre-aggregations for Cube. Follow the instructions here to set it up. Depending on your database, Cube may need to “stage” pre-aggregations inside your database first before ingesting them into Cube Store. In this case, Cube will require write access to a dedicated schema inside your database. The schema name is prod_pre_aggregations by default. It can be set using the pre_aggregations_schema configration option.

You may consider enabling an export bucket which allows Cube to build large pre-aggregations in a much faster manner. It is currently supported for BigQuery, Redshift, Snowflake, and some other data sources. Check the relevant documentation for your configured database to set it up.

Secure the deployment

If you’re using JWTs, you can configure Cube to correctly decode them and inject their contents into the Security Context. Add your authentication provider’s configuration under the jwt property of your cube.js configuration file, or if using environment variables, see CUBEJS_JWK_*, CUBEJS_JWT_* in the Environment Variables reference.

Set up health checks

Cube provides Kubernetes-API compatible health check (or probe) endpoints that indicate the status of the deployment. Configure your monitoring service of choice to use the /readyz and /livez API endpoints so you can check on the Cube deployment’s health and be alerted to any issues.

Appropriate cluster sizing

There’s no one-size-fits-all when it comes to sizing a Cube cluster and its resources. Resources required by Cube significantly depend on the amount of traffic Cube needs to serve and the amount of data it needs to process. The following sizing estimates are based on default settings and are very generic, which may not fit your Cube use case, so you should always tweak resources based on consumption patterns you see.

Memory and CPU

Each Cube cluster should contain at least 2 Cube API instances. Every Cube API instance should have at least 3GB of RAM and 2 CPU cores allocated for it. Refresh workers tend to be much more CPU and memory intensive, so at least 6GB of RAM is recommended. Please note that to take advantage of all available RAM, the Node.js heap size should be adjusted accordingly by using the --max-old-space-size option:

NODE_OPTIONS="--max-old-space-size=6144"

The Cube Store router node should have at least 6GB of RAM and 4 CPU cores allocated for it. Every Cube Store worker node should have at least 8GB of RAM and 4 CPU cores allocated for it. The Cube Store cluster should have at least two worker nodes.

RPS and data volume

Depending on data model size, every Core Cube API instance can serve 1 to 10 requests per second. Every Core Cube Store router node can serve 50-100 queries per second. As a rule of thumb, you should provision 1 Cube Store worker node per one Cube Store partition or 1M of rows scanned in a query. For example if your queries scan 16M of rows per query, you should have at least 16 Cube Store worker nodes provisioned. Please note that the number of raw data rows doesn’t usually equal the number of rows in pre-aggregation. At the same time, queries don’t usually scan all the data in pre-aggregations, as Cube Store uses partition pruning to optimize queries. EXPLAIN ANALYZE can be used to see scanned partitions involved in a Cube Store query. Cube Cloud ballpark performance numbers can differ as it has different Cube runtime.

Optimize usage

See this recipe to learn how to optimize data source usage.

Getting Started

Deployment

Prerequisites

Configuration

Set up reverse proxy

Security

Use JSON Web Tokens

Securing Cube Store

Monitoring

Update to the latest version

Extend the Docker image

Production checklist

Disable Development Mode

Set up Refresh Worker

Set up Cube Store

Secure the deployment

Set up health checks

Appropriate cluster sizing

Memory and CPU

RPS and data volume

Optimize usage

​Prerequisites

​Configuration

​Set up reverse proxy

​Security

​Use JSON Web Tokens

​Securing Cube Store

​Monitoring

​Update to the latest version

​Extend the Docker image

​Production checklist

​Disable Development Mode

​Set up Refresh Worker

​Set up Cube Store

​Secure the deployment

​Set up health checks

​Appropriate cluster sizing

​Memory and CPU

​RPS and data volume

​Optimize usage

Prerequisites

Configuration

Set up reverse proxy

Security

Use JSON Web Tokens

Securing Cube Store

Monitoring

Update to the latest version

Extend the Docker image

Production checklist

Disable Development Mode

Set up Refresh Worker

Set up Cube Store

Secure the deployment

Set up health checks

Appropriate cluster sizing

Memory and CPU

RPS and data volume

Optimize usage