🏗️Build

Follow these steps to build the Datalogz services on your virtual machine.

To build the Datalogz application, follow these steps:

1. ssh into a linux virtual machine to build the following repositories.

   1. datalogz-bi-diagnostics (ELT)

   2. datalogz-bi-gateway (API)

   3. datalogz-bi-frontend (APP)

2. git pull the main/master branches of each repository

3. Read the README.md for each repository

4. Build the services:

   1. Build datalogz-bi-diagnostics

      1. Confirm non-sensitive environment variables listed here have been added to .prod.env and that sensitive enviornment variables have been added to the Key Vault

      2. Add the following environment variables to .prod.env

         1. ENV=PROD

         2. DBT_ENV=prod

         3. WAREHOUSE_TYPE=[SNOWFLAKE | POSTGRES]

      3. If you are running Azure Managed Identity VMs, add the following:

         1. AZURE_KEY_VAULT_URL=

         2. AZURE_KEY_VAULT_NAME=

         3. AZURE_VM_NAME=

         4. AZURE_RESOURCE_GROUP_NAME=

      4. From the project directory, run source ./init_env.sh to run through the interactive build script:

         1. Choose env: prod

         2. Choose warehouse: postgres, snowflake

         3. Choose cloud: azure, aws

         4. Choose IAM method for VM: env, identity

         5. Choose vm setup: mono or split

      5. The ./init_env.sh script will build the correct docker compose file based on the options that are chosen.

   2. Build datalogz-bi-gateway

      1. Confirm non-sensitive environment variables listed here have been added to .prod.env and that sensitive enviornment variables have been added to the Key Vault

      2. Add the following environment variables to .prod.env

         • ENV=PROD

         • PYTHONDONTWRITEBYTECODE=1

         • CRON_SERVICE_URL=https://airflow_webserver:8080

         • HTTP_SCHEME=https

         • HOST_NAME=localhost

           • Change localhost to your host DNS or private IP of the VM

      3. If you are running Azure Managed Identity VMs, add the following:

         • AZURE_RESOURCE_GROUP_NAME=

         • AZURE_VM_NAME=

         • AZURE_KEY_VAULT_NAME=

         • AZURE_KEY_VAULT_URL=

      4. Build the services from the project directory:

         1. Monolith VM:

            • Run docker compose -f docker-compose.mono.prod.yml up --build -d

         2. Split VM:

            • Run docker compose -f docker-compose.prod.yml up --build -d

   3. Build datalogz-bi-frontend

      1. Add the following environment variables to .env

         • REACT_APP_IDENTITY_PROVIDER = 'MICROSOFT'

         • VM_MANAGED_IDENTITY = 'TRUE'

         Values for REACT_APP_IDENTITY_PROVIDER can be:

         • MICROSOFT

         • TABLEAU_SSO_FOR_CLOUD

         • TABLEAU_SA

         • WORKOS

         • GOOGLE

         If REACT_APP_IDENTITY_PROVIDER is set, MANAGED_IDENTITY can also be set to TRUE to embed Service Principal credentials into API build for new Connectors to use by default.

      2. Build the services from the project directory:

         1. Monolith VM:

            • Run docker compose -f docker-compose.mono.prod.yml up --build -d

         OR

         1. Split VM:

            • Run docker compose -f docker-compose.dev.yml up --build -d