System Design
The Datalogz system design provides customers a flexible approach for deploying Datalogz according to their preferences.
The Datalogz application uses OLTP and OLAP databases to optimize performance and scalability between the two workloads. Customers may choose which database technologies they would like to use for both the application and warehouse databases.
Typical OLTP Workloads include:
- Create and manage Accounts and Environments
- Create and manage Roles, Users, and Permissions
- Create and manage Connectors
- Create and manage BI Activity Dashboard
- Create and manage Operations and Actions
- Create and manage Impact Reports
Typical OLAP Workloads include:
- Transform raw JSON data into enriched dimensional datasets
- Identify issues in the BI environment related to ROI, security and compliance
- Generate Change History for all BI metadata endpoints
- Produce Context Logs for identifying root causes of Issues
- Generate Recommendations for improving BI environments
The following options are available today:
1. Postgres Only
- App DB: PostgreSQL (OLTP)
- BI WH: PostgreSQL (OLAP)
2. Postgres + Snowflake
- App DB: PostgreSQL (OLTP)
- BI WH: Snowflake (OLAP)
This option utilizes a single PostgreSQL server with two databases -- one for OLTP workloads and another for OLAP workloads.
PostgreSQL OLTP and OLAP
This option utilizes a PostgreSQL database for OLTP workloads and a Snowflake database for OLAP workloads.
PostgreSQL OLTP and Snowflake OLAP
The Datalogz application uses Apache Airflow for connector management, providing BI Admins with pre-built metadata pipelines they can choose to schedule daily, weekly or hourly basis. New issues and recommendations will be generated after each connector refresh based on the latest data that has changed.
Connectors must be configured by BI Admins to approve the Datalogz application. This will provide read-only access to standard and admin level APIs based on a selection of Groups.
Groups are generally defined as follows for each system:
- PowerBI: Workspaces
- Tableau: Projects
- Looker: Folders
The admin level APIs unlock the most insight for your BI Admins when it comes to types of Issues and Recommendations Datalogz is able to provide. After a new connector is created, BI Admins can use Datalogz RBAC to assign fine-grained permissions to Users who should only have access to certain metadata from certain Groups.
The Service Principal you create for Datalogz utilizes a combination of admin and standard endpoints to retrieve the activity, lineage, query expression, and inventory metadata.
The following data flow diagram shows the endpoints from the PowerBI Rest API used to build the metadata model for Datalogz BI Ops. This will run on a daily or weekly basis defined when creating your Datalogz connector.
PowerBI Rest API Data Flow diagram
All warehouse transformations are source-controlled and executed using an open-source technology named
dbt-core. Metadata loaded into the BI Warehouse is transformed using dbt-core to produce insights and recommendations that can be used to improve BI operations. The dbt-core service is included as part of the ELT API and shown in the diagram below:
The ELT API uses dbt-core to transform data from Ingest to Marts for frontend consumption.
Last modified 1mo ago