Secrets Manager

Datalogz uses AWS Secrets Manager to store sensitive secrets required to run the application.

1. Access the Secrets Manager service and select "Store".

2. Choose "Other Type of secret" as the secret type.

3. Input your secret's key and value. Use "Add Row" to add multiple secrets from the following list.

4. For Encryption key, select "aws/secretsmanager".

5. Select "Next".

6. Name your secret "datalogz_secrets". Add a description such as "secret values used by Datalogz to securely access credentials".

7. Select "Next".

8. This step is optional. Datalogz recommends creating a lambda function to enable secrets rotation. Once configured, select "Next".

9. Review your secrets. If everything looks good, select "Next". Otherwise, select "Previous" to make updates where necessary.

Environment Variables List

Repository: datalogz-bi-diagnostic

File: .prod.env

ENV=PROD
DBT_ENV=prod
WAREHOUSE_TYPE=POSTGRES

# Warehouse
# If using Snowflake, change WAREHOUSE_TYPE to SNOWFLAKE

Secrets Manager

# Specifies the URL or connection string to the Celery result backend
# e.g. db+postgresql://<user>:<pass>@<host>/airflow
# if SSL is required include parameters: ?sslmode=require&sslrootcert=/opt/airflow/<name_of_cert>.crt.pem
AIRFLOW--CELERY--RESULT-BACKEND

# Specifies the URL or connection string to the Airflow metadata database.
# e.g. postgresql+psycopg2://<user>:<pass>@<host>/airflow
# if SSL is required include parameters: ?sslmode=require&sslrootcert=/opt/airflow/<name_of_cert>.crt.pem
AIRFLOW--DATABASE--SQL-ALCHEMY-CONN

# Specifies the Fernet key used for encrypting and decrypting Airflow connections and variables. 
# This must be a 32 base64 encoded string
# e.g. Example format: jHfPb-mvRhWyofw8bzyCJym-HyKjSNNbwS8bLJjK0Vo=
AIRFLOW-FERNET-KEY

# Specifies the hostname or IP address of the PostgreSQL database server used for Airflow metadata storage.
AIRFLOW-POSTGRES-DB-HOST

# Specifies the name of the PostgreSQL database, user and password used for Airflow metadata storage.
AIRFLOW-POSTGRES-DB-NAME=airflow
AIRFLOW-POSTGRES-DB-USER
AIRFLOW-POSTGRES-DB-PASSWORD

# Specifies the username and password for the Airflow web UI.
AIRFLOW-WWW-USER
AIRFLOW-WWW-PASSWORD

# Specifies the private access token used for callbacks to gateway on task success/fail
# e.g. eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c
AIRFLOW-TO-GATEWAY-ACCESS-TOKEN
GATEWAY-SERVICE-BASE-URL=http://bi_gateway_service:5000

# if using Postgres Warehouse
DIAGNOSTICS-POSTGRES-HOST
DIAGNOSTICS-POSTGRES-DATABASE-NAME=datalogz_wh
DIAGNOSTICS-POSTGRES-PASSWORD
DIAGNOSTICS-POSTGRES-SCHEMA-NAME=public
DIAGNOSTICS-POSTGRES-USERNAME=datalogz_diagnostics_admin

# if using Snowflake Warehouse
SNOWFLAKE-ACCOUNT-IDENTIFIER
DIAGNOSTICS-SNOWFLAKE-WAREHOUSE-NAME=DATALOGZ_BIOPS
DIAGNOSTICS-SNOWFLAKE-DATABASE-NAME=DATALOGZ_WH
DIAGNOSTICS-SNOWFLAKE-PASSWORD
DIAGNOSTICS-SNOWFLAKE-ROLE-NAME=DATALOGZ_DIAGNOSTICS_ADMIN_ROLE
DIAGNOSTICS-SNOWFLAKE-SCHEMA-NAME=PUBLIC
DIAGNOSTICS-SNOWFLAKE-USERNAME=DATALOGZ_DIAGNOSTICS_ADMIN

Repository: datalogz-bi-gateway

File: .env.prod

ENV=PROD
HOST_NAME=app.your_hostname.com
HTTP_SCHEME=https
CRON_SERVICE_URL=https://airflow_webserver:8080

# Include either "MICROSOFT" or "TABLEAU_SERVER"
CONFIGURED_IDPS=["MICROSOFT"]

# Create this SQL user in advance following the SQL code in Databases section
ENTITLEMENT_USER_NAME=datalogz_gateway_user

Secrets Manager

# Specifies the connection string to the application's database
# e.g. postgresql://datalogz_gateway_admin:<password>@<host>:<port>/datalogz_bi
# If SSL is required include parameters: ?sslmode=require&sslrootcert=/opt/airflow/<cert_name>.crt.pem
APP-DB-CONNECTION-STRING

# Specifies the connection string to the application's OLAP warehouse
# e.g. postgresql://datalogz_diagnostics_admin:<password>@<host>:<port>/datalogz_wh?options=-csearch_path%3Dbiops_marts%2Dbiops_general
# If SSL is required include parameters: &sslmode=require&sslrootcert=/opt/airflow/<cert_name>.crt.pem
BI-DB-CONNECTION-STRING

# Specifies the private access token used for callbacks to gateway on task success/fail
# e.g. eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c
AIRFLOW-TO-GATEWAY-ACCESS-TOKEN

# Specifies the algorithm used to encode and decode JWT tokens
# the secret should at least be 32 characters long, but the longer the better.
JWT-ALGORITHM=HS256
JWT-SECRET-KEY

# Specifies the API key for the mail client to send emails
MAIL-CLIENT-API-KEY

# If using Microsoft Active Directory
# Specifies the client ID and Decret for the Microsoft OAuth2 application
MICROSOFT-CLIENT-ID
MICROSOFT-CLIENT-SECRET

# If using PowerBI
# Specifies the client ID and Secret for the Power BI OAuth2 application
POWERBI-CLIENT-ID
POWERBI-CLIENT-SECRET