Follow these steps to build the Datalogz services on your virtual machine.
To build the Datalogz application, follow these steps:
ssh
into a linux virtual machine to build the following repositories.
datalogz-bi-diagnostics (ELT)
datalogz-bi-gateway (API)
datalogz-bi-frontend (APP)
git pull
the main/master
branches of each repository
Read the README.md
for each repository
Build the services:
Build datalogz-bi-diagnostics
Confirm non-sensitive environment variables listed here have been added to .prod.env
and that sensitive enviornment variables have been added to the Key Vault
Add the following environment variables to .prod.env
ENV=PROD
DBT_ENV=prod
WAREHOUSE_TYPE=[SNOWFLAKE | POSTGRES]
If you are running Azure Managed Identity VMs, add the following:
AZURE_KEY_VAULT_URL=
AZURE_KEY_VAULT_NAME=
AZURE_VM_NAME=
AZURE_RESOURCE_GROUP_NAME=
From the project directory, run source ./init_env.sh
to run through the interactive build script:
Choose env: prod
Choose warehouse: postgres, snowflake
Choose cloud: azure, aws
Choose IAM method for VM: env, identity
Choose vm setup: mono
or split
The ./init_env.sh
script will build the correct docker compose file based on the options that are chosen.
Build datalogz-bi-gateway
Confirm non-sensitive environment variables listed here have been added to .prod.env
and that sensitive enviornment variables have been added to the Key Vault
Add the following environment variables to .prod.env
ENV=PROD
PYTHONDONTWRITEBYTECODE=1
CRON_SERVICE_URL=https://airflow_webserver:8080
HTTP_SCHEME=https
HOST_NAME=localhost
Change localhost
to your host DNS or private IP of the VM
If you are running Azure Managed Identity VMs, add the following:
AZURE_RESOURCE_GROUP_NAME=
AZURE_VM_NAME=
AZURE_KEY_VAULT_NAME=
AZURE_KEY_VAULT_URL=
Build the services from the project directory:
Monolith VM:
Run docker compose -f docker-compose.mono.prod.yml up --build -d
Split VM:
Run docker compose -f docker-compose.prod.yml up --build -d
Build datalogz-bi-frontend
Add the following environment variables to .env
REACT_APP_IDENTITY_PROVIDER = 'MICROSOFT'
VM_MANAGED_IDENTITY = 'TRUE'
Values for REACT_APP_IDENTITY_PROVIDER can be:
MICROSOFT
TABLEAU_SSO_FOR_CLOUD
TABLEAU_SA
WORKOS
If REACT_APP_IDENTITY_PROVIDER
is set, MANAGED_IDENTITY
can also be set to TRUE to embed Service Principal credentials into API build for new Connectors to use by default.
Build the services from the project directory:
Monolith VM:
Run docker compose -f docker-compose.mono.prod.yml up --build -d
OR
Split VM:
Run docker compose -f docker-compose.dev.yml up --build -d