As BI Ops advocates, we believe that maintaining a healthy and efficient Business Intelligence (BI) environment requires focusing on three essential pillars: Security Monitoring, BI Governance, and Platform Administration. These pillars serve as the foundation of our product and define how we help organizations ensure their BI environments are secure, compliant, and performant.

Security Monitoring

Security Monitoring is the continuous oversight of the BI environment to ensure that data access, usage, and overall security protocols are compliant with organizational policies and industry standards. In a world where data breaches and unauthorized access can have serious consequences, security monitoring is a critical aspect of BI management.

BI Governance

BI Governance is the framework that controls the structure, management, and usage of BI assets across an organization. Effective governance ensures that data is accurate, consistent, and compliant with organizational policies and industry standards. It encompasses data quality, usage, lifecycle management, and access control.

Platform Administration

Platform Administration refers to the oversight, configuration, and management of the BI infrastructure, ensuring that the system operates smoothly and supports the organization's reporting needs. It includes managing users, permissions, system updates, performance optimizations, and capacity usage (PowerBI only).

How it works?

Datalogz helps address these three pillars by leveraging Datalogz Default Monitors.

These monitors are pre-configured or user-customized SQL-based queries designed to track issues across each pillar. They continuously scan the BI environment for anomalies, rule breaches, or inefficiencies. For example, Datalogz Monitors can identify unauthorized access attempts for Security Monitoring, flag outdated or inconsistent reports for BI Governance, track user permissions and system configurations for Platform Administration. For PowerBI users, we can help monitor resource usage to prevent capacity overloads. Alerts generated by these monitors are surfaced directly in the platform, allowing users to take immediate action and maintain optimal performance, security, and governance across their BI tools.

Additionally, Datalogz offers the option for users to create their own custom monitors, allowing organizations to define their own pillars and track their BI environment based on internal governance and specific business needs. This flexibility enables users to tailor monitoring to unique security, performance, and governance goals, ensuring that every aspect of their BI infrastructure is optimized and aligned with internal standards. Alerts generated by both default and custom monitors are surfaced within the platform, enabling immediate action and ensuring optimal performance and compliance across all BI tools.

Introduction

Welcome to the Datalogz Infrastructure Deployment Guide. This document provides comprehensive instructions for deploying the Datalogz platform either on Amazon Web Services (AWS) or on bare metal servers. It is intended for DevOps engineers, system administrators, and IT professionals responsible for setting up and managing Datalogz infrastructure within your organization.

The guide includes step-by-step procedures, diagrams, and best practices to ensure a smooth deployment. All diagrams are dynamically generated using D2 and can be found in the diagrams directory.

Prerequisites

Before you begin the deployment process, ensure that you have met the following prerequisites:

• Access Permissions:

  • For AWS deployment, an AWS account with permissions to create and manage resources such as EC2 instances, RDS Instances, S3 buckets, and IAM roles.

  • For bare metal deployment, administrative access to the physical or virtual servers.

• Required Software Installed:

  • Terraform

  • AWS CLI

  • Ansible

  • Git

• Network Requirements:

  • Open ports as required by Datalogz services (e.g., HTTP/HTTPS ports, database ports).

  • SSH access to servers for Ansible to run playbooks.

• SSH Keys:

  • A valid SSH key pair for accessing the servers. The private key should be accessible from the machine where Ansible is run.

AWS Deployment

Overview

Deploying Datalogz on AWS involves provisioning infrastructure using Terraform and configuring instances using Ansible. This approach automates the deployment process, ensuring consistency and scalability.

Installation and Configuration

Step 1: Install Required Tools

Ensure that the following tools are installed on your local machine:

1. Terraform

   • Installation Guide: Install Terraform

   • Verify installation: terraform -v

2. AWS CLI

   • Installation Guide: Install AWS CLI

   • Verify installation: aws --version

3. Ansible

   • Installation Guide: Install Ansible

   • Verify installation: ansible --version

Step 2: Configure AWS Credentials

Set up your AWS credentials so that the AWS CLI and Terraform can authenticate with your AWS account.

• Option 1: Using Environment Variables

export AWS_ACCESS_KEY_ID="your-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
export AWS_DEFAULT_REGION="your-default-region"

• Option 2: Using AWS CLI Configuration

  Run the AWS configure command: aws configure

  You will be prompted to enter your AWS Access Key ID, Secret Access Key, Default Region, and Output Format.

Step 3: Verify AWS Configuration

Test your AWS configuration by trying to list your current S3 buckets: aws s3 ls

Terraform Deployment

Step 4: Navigate to the AWS Terraform Directory

Change your current directory to the AWS Terraform configuration folder: cd aws

Step 5: Initialize Terraform

Initialize the Terraform working directory. This command downloads the necessary provider plugins: terraform init

Step 6: Review Terraform Variables

Inspect the variables.tf file to understand the variables required for deployment, such as AWS Region, S3 Bucket name, and EC2 Key Name.

Step 7: Plan the Terraform Deployment

Generate and review the execution plan to understand the resources that will be created: terraform plan

Review the Plan:

• Ensure that the resources and configurations match your expectations.

• Check for any unintended changes to existing infrastructure.

Step 8: Apply the Terraform Plan

Apply the execution plan to deploy the infrastructure: terraform apply

• Confirmation:

  • You will be prompted to confirm the deployment. Type yes to proceed.

• Deployment Duration:

  • The deployment typically takes around 15-25 minutes.

Step 9: Post-Deployment Verification

After Terraform completes:

• State File:

  • Ensure that the terraform.tfstate file has been updated.

Ansible Configuration

Step 10: Update Ansible Inventory

Ansible uses a dynamic inventory plugin to find the IP address of the EC2 instance based on the instance Name tag.

• Location: Change directory to the Ansible folder: cd ../ansible

Step 11: Configure Ansible Settings

Update the ansible.cfg file:

• Private Key File:

  • Set the private_key_file parameter to the path of your SSH private key.

  Copy

  [defaults]
  inventory = inventory.yml
  private_key_file = /path/to/your/private/key.pem
  host_key_checking = False

Step 12: Run the Ansible Playbook

Execute the Ansible playbook to configure the EC2 instances: ansible-playbook deploy.yml -v

• Verbose Mode:

  • The v flag enables verbose output. Use vvv for even more detailed logs.

• Common Issues:

  • If you encounter SSH authentication errors, verify that the SSH key has the correct permissions (chmod 600 key.pem).

Step 13: Verify Deployment

After the playbook runs:

• Services Check:

  • SSH into the instances and verify that the Datalogz containers are running.

• Application Test:

  • Access the application URL to ensure it is functioning correctly.

Basic Concepts

Assets

An asset refers to any resource within your Business Intelligence (BI) environment that holds value in terms of data analysis, reporting, or data management.

Common types of BI assets include datasets, reports, dashboards, databases, and other data-related objects created and maintained by data users within an organization. These assets form the backbone of your BI environment, as they are the entities that store, process, and present the data used for decision-making. Each asset comes with associated metadata, such as creation date, ownership, usage statistics, and compliance with established rules.

Platform Context

Platform Context is a term we introduced to provide a unified reference for organizational structures across various BI tools. Since each BI tool uses its own terminology for organizing resources (e.g., workspace, project, streams), Platform Context serves as a generic term to simplify communication and enhance usability across the platform.

Monitors

A Monitor is a tool used to continuously track and evaluate the health and performance of Business Intelligence (BI) environment. Essentially, a monitor is a pre-written SQL query designed to extract and flag BI assets that breach predefined thresholds or criteria. It acts as a bulk filter, scanning your BI environment for any assets that deviate from set standards or expectations, such as outdated datasets, underused reports, or security vulnerabilities.

Monitors are highly customizable, allowing users to edit the SQL query, assign statuses, and determine the team responsible for resolving any triggered alerts. This makes monitors a powerful, scalable way to manage large sets of assets and ensure that any issues are detected and handled efficiently.

Alerts

An Alert is a notification generated when an asset breaches a threshold or criteria set within a Monitor. It's designed to notify users of problems within their BI environment and prompt them to take corrective actions.

Each alert provides key details about the issue, including the specific asset involved (e.g., datasets, reports, dashboards), the type of breach (e.g., unused report, failed refresh), and associated metadata (e.g., asset owner, creation date). Alerts serve as actionable items that help users identify and resolve issues quickly and efficiently.

Insights

Insights page is the visualization hub of BI (Business Intelligence) alerts and metadata within our platform. It provides users with a comprehensive view of their BI environment through summary of historical and real-time data. Insight delivers critical information in an easily digestible format, helping users monitor the health of their BI systems.

Connectors

A Connector in our platform is the critical gateway that allows the Datalogz team to access and interact with a client’s Business Intelligence (BI) environment. It serves as the key that connects our platform to various BI tools used by the client, such as PowerBI, Tableau, and Qlik. Connectors are typically made up of API keys, admin secrets, or similar authentication credentials provided by the BI tool, which grant the necessary permissions for our platform to monitor and gather data from these systems.

Teams

A Team in our platform represents a group of individuals within an organization who collaborate toward shared business goals. Teams are typically organized around functional roles or departments, such as marketing, finance, or operations, and members of each team often require access to the same sets of BI (Business Intelligence) content, such as datasets, reports, and dashboards.

In essence, a team functions as a collection of users with similar data needs, providing a way to efficiently manage access to BI resources and organize collaborative efforts around the BI environment.

Creating Monitors

Default Monitors

These are out-of-the-box monitors created and owned by Datalogz team. Default Monitors are pre-configured to help users quickly begin monitoring common and critical aspects of their BI environment. These monitors are set up with default SQL queries and thresholds based on industry best practices, allowing users to start receiving alerts and insights without any initial setup.

My Monitors

These are user-created monitors. Currently, users can only create new monitors by duplicating an existing Default Monitor and modifying it according to their needs. My Monitors provide users the flexibility to adjust the monitoring criteria, such as SQL queries and thresholds, to fit their specific requirements.

Overview

Datalogz offers multiple options for users to log in and allows Root User and Admin to restrict methods for added security.

Log-in Methods

Google Authentication

Users can authenticate into Datalogz using Google authentication when using a Google-supported email address.

Microsoft Authentication

Users can authenticate into Datalogz using Microsoft authentication using a Microsoft-supported email address.

SAML

With SAML enabled, users can log in through their identity provider's website by selecting the SAML option on the login page.

Once the organization enables SAML, all members must log in using SAML.

Overview

A Monitor in our platform is a tool used to continuously track and evaluate the health and performance of Business Intelligence (BI) assets. Essentially, a monitor is a pre-written SQL query designed to extract and flag BI assets that breach predefined thresholds or criteria. It acts as a bulk filter, scanning your BI environment for any assets that deviate from set standards or expectations, such as outdated datasets, underused reports, or security vulnerabilities.

Monitors are highly customizable, allowing users to edit the SQL query, assign statuses, and determine the team responsible for resolving any triggered alerts. This makes monitors a powerful, scalable way to manage large sets of assets and ensure that any issues are detected and handled efficiently.

Cross-tab: Another name for a text table or a table of numbers. It is a tabular representation of data where rows and columns intersect to display values.

Dashboard: A collection of views shown in a single location where you can compare and monitor a variety of data simultaneously. Dashboards provide a consolidated and interactive way to analyze data.

Data source: The underlying data that Tableau Reader is connected to. You can't change the data source in Tableau Reader. It serves as the foundation for creating visualizations and reports in Tableau.

Filter: A control on a view that limits the data shown in a view. For example, a filter on Region that only includes the West. Filters help users focus on specific subsets of data within a visualization.

Marks: A visual representation of one or more rows in a data source. Mark types can be bar, line, square, and so on. Marks are the individual data points or elements displayed on a visualization.

Packaged workbook: A type of workbook created in either Tableau Desktop or Tableau Server. These files contain both the workbook as well as copies of the referenced local file data sources and background images. They allow for easy sharing and collaboration.

Pane: The row and columns areas in a view. Panes divide the view into sections, often used for arranging headers, rows, and columns within a worksheet or dashboard.

Repository: A folder located in your My Documents folder that stores workbooks. The repository is where Tableau stores its files, including workbooks and data sources.

View: The visual representation of your data in a worksheet or dashboard. Views are the charts, graphs, and tables that display data to users for analysis and interpretation.

Workbook: A collection of one or more worksheets and dashboards. Workbooks serve as containers for organizing and presenting data visualizations and analyses.

Worksheet: A single view of data. Each worksheet can be connected to a single data source. Worksheets are where you build and design visualizations and reports.

Dimension: A qualitative field that can be used to categorize, segment, and reveal the details in your data. Examples include dates, customer names, or geographical data. Dimensions provide context for analysis.

Measure: A quantitative field that can be aggregated and is suitable for mathematical operations, such as sums or averages. Measures would be data like sales amount, temperature readings, or counts of events. Measures provide numeric values for analysis.

Calculated Field: A user-defined field created by applying calculations to existing fields in the data source. This allows for more advanced analysis within a Tableau workbook. Calculated fields are created using mathematical, logical, or custom expressions.

Parameter: A dynamic placeholder that allows users to replace a constant value in a calculation, filter, or reference line. For instance, a parameter can let end-users change the threshold value displayed in a view. Parameters enable user interactivity and customization.

Extract: A saved subset of a data source that you can use to improve performance and support offline data analysis. An extract is a snapshot of the data taken at a specific point in time. It can be useful for working with large datasets efficiently.

Live Connection: A direct connection to a data source that allows real-time access to the latest data, but can be slower if the data set is very large or the database is not optimized. Live connections ensure that data is always up-to-date.

Hierarchy: An organizational structure that allows for drilling down into dimensions. Hierarchies are used in Tableau to define levels of data granularity from higher to lower levels of aggregation. They help in organizing and navigating data.

Tooltip: A message that appears when a user hovers over a mark in the view. Tooltips can be customized to display relevant information about the data point. They provide additional context and details about data.

Blending: The ability to combine data from two different data sources on a single sheet and visualize them together, even if they're not joined or related at the database level. Blending allows for integrated analysis of disparate data sources.

Sets: Custom fields that define a subset of data based on some conditions. A set can be used for comparative analysis, like comparing the performance of top products against all others. Sets help in creating segments within data.

Bins: User-defined containers of equal size that can be used to divide the dimension data into distinct ranges, which are often used for histograms. Bins are used to group continuous data into discrete intervals for analysis.

Story: A sequence of visualizations that work together to show different facets of data and insights. A story can explain how data leads to the conclusions you've made. It allows for storytelling through data visualization.

Release date: October 1st, 2024

What's New

Enhanced Insights Page

• BI Tool-Specific Insights: The Insights page has been enhanced to provide customized visualizations based on the type of BI tool being used. Whether you're using PowerBI, Tableau, or Qlik, the Insights page now adapts to show tailored metrics and trends specific to each tool, helping users better understand their BI environments in the context of the tools they rely on.

Enhanced Default Monitors

• More Comprehensive Default Monitors: We’ve upgraded our Default Monitors to be more robust and comprehensive, addressing a broader range of issues under the four pillars of Security Monitoring, BI Governance, Platform Administration, and Capacity Monitoring. These enhanced monitors provide deeper insights into your BI environment, identifying more potential issues and improving the overall monitoring experience.

New Default Monitor Features

• Show/Hide Monitors: Users can now choose to show or hide specific monitors, allowing for a more personalized and clutter-free view of their monitoring dashboard.

• Clone Monitors: You can now clone default monitors, making it easy to create custom monitors based on existing configurations.

• Detailed Monitor Data: Access richer, more detailed data in each monitor. View query logic, underlying metadata, and track changes over time to gain more granular control over your BI monitoring.

Custom Monitors

• Create Custom Monitors by Duplicating Default Monitors: Users now have the ability to create custom monitors by duplicating Default Monitors and editing their SQL queries to tailor the monitor to specific needs. These custom monitors can be deleted or further modified, providing full flexibility in how you monitor your BI environment.

• My Monitors vs Default Monitors: To keep things organized, we’ve introduced two folders in the Monitor section: My Monitors and Default Monitors. The Default Monitors folder houses the pre-built monitors created by our team, while the My Monitors folder holds user-created monitors, giving you a clear separation between system and custom configurations.

Enhanced Alerts Table

• Improved Filtering, Search, and Column Data: We’ve enhanced the Alerts Table to make managing and resolving alerts even more efficient. Users can now apply more granular filters, perform faster searches, and access more detailed column data to quickly identify and address critical issues.

Launch of Teams

• Teams Feature: We’ve introduced Teams functionality! Organizations can now create and manage teams within the platform. Teams allow users to collaborate more effectively by organizing groups with shared BI goals and content. Assign BI connectors, workspaces, and assets to specific teams, improving collaboration, accountability, and management across departments.

Data catalog: A component in Data manager and Data load editor that enables you to select and load data from all the datasets to which you have access. It serves as a catalog or repository of available data sources.

Data connection: Used to let data tasks access data sources and external storage and cloud data warehouses used in a data project. Data connections are the links or interfaces that allow data to be transferred or accessed.

Qlik Data Gateway - Data Movement: Allows you to move firewalled data from your enterprise data sources to cloud and on-premises targets over a strictly outbound encrypted and mutually authenticated connection. It facilitates secure data transfer between different environments.

Data Gateway Direct Access: Allows Qlik Sense SaaS applications to securely access firewalled data over a strictly outbound encrypted and mutually authenticated connection. It provides direct access to otherwise restricted data sources.

Data leakage: An undesired phenomenon in machine learning where an algorithm is trained with data that it will use for generating predictions, leading to unrealistically high model performance from memorization rather than actual learning. It can result in biased or overfit models.

Data load editor: A script editor that allows you to build and customize the script that loads data into your app. It provides a way to manipulate and transform data during the loading process.

Data manager: An app component that allows you to load and manage data sources in an app. Data managers are responsible for organizing and maintaining data within the application.

Data mart: Part of your data pipeline containing a subset of data from Storage or Transform data assets, ideally containing summarized data collected for analysis on specific sections or units within an organization. Data marts are specialized databases optimized for specific purposes.

Data model viewer: An app component that allows you to view the structure of the data added to an app and metadata about tables and fields. It provides insights into the organization of data within an application.

Data pipeline: A set of tasks for integrating data in a data project, which can be a simple linear pipeline or a complex one consuming several data sources and generating many outputs. Data pipelines define the flow of data processing within a project.

Data profiling: Displays statistics and information about your data sets. It provides insights into the characteristics and quality of data, helping in data preparation and analysis.

Data project: A workspace where you create your data pipeline using data assets, associated with a data platform used as the target for all outputs. Data projects are where data integration and transformation activities are managed.

Data task: The main unit of work in a data project for moving, storing, transforming data, and creating data marts. Data tasks define specific actions within a data project.

Dataset: Synonymous with table, referring to original source tables, transformed tables, or the fact and dimension tables in a data mart. Datasets are organized collections of data.

Dimension: An entity used in Analytics Services to categorize data in a chart, and in Data Integration, a dataset in a data mart forming part of the star schema. Dimensions provide context for data analysis.

Dynamic views: Allows you to query and view relevant subsets of large datasets from another app in a chart, with the ability to refresh dynamically as selections are made. Dynamic views provide a flexible way to interact with data.

Fact: A table that holds data to be analyzed, working together with dimension tables to store data on the ways in which fact table data can be analyzed. Facts contain the measurable data points in a data model.

Favorites: A section available to all users to add apps, datasets, automations, notes, experiments, and charts from the hub, which are private. Favorites allow users to bookmark and access frequently used items.

Feature (machine learning): A variable in a machine learning problem that can influence the value of the target column, recognized as columns in a dataset within Qlik AutoML. Features are input variables used to make predictions.

Field: Contains values loaded from a data source, corresponding to a column in a table and used to create dimensions and measures in visualizations. Fields represent individual data attributes.

Full load: Refers to the initial replication of data from the data source to the landing in Qlik Cloud Data Integration. It involves transferring all data without incremental updates.

Sheet: A sheet in Qlik Sense is a canvas where you can create a customized view of your data, arranged in a way that tells a story or answers specific questions. Sheets are used for data visualization and analysis.

Sheet objects: Components used to create an interface on a sheet, which can include data visualizations like tables and charts, as well as other objects such as buttons and text objects. Sheet objects are elements placed on sheets for interaction.

Snapshot: Graphical representations of a visualization at a certain point in time, used to create stories. Snapshots capture the state of visualizations for storytelling purposes.

Space data: Governed areas of the Qlik Cloud tenant used to create and store data projects, manage new data connections, and access Data Movement gateways. Space data is where data integration and management activities occur.

Space managed: Controlled spaces used to share apps with a limited group of users. Managed spaces provide a controlled environment for collaborative app development.

Space personal: A private space belonging to users where they can develop apps. Personal spaces are individual workspaces for app development.

Space shared: Areas where apps and data sources can be shared with other users for collaborative development. Shared spaces facilitate teamwork and sharing of resources.

Storage: Part of the data pipeline containing ready-to-consume datasets in Qlik Cloud from data copied from the landing zone. Storage is where data is stored and made available for analysis.

Story: A tool that allows the sharing of data insights and discoveries made in an app with other users, combining reporting, presentation, and exploratory analysis. Stories enable data-driven narratives.

Subscription: Reports that let you schedule recurring emails containing a PDF of selected sheets or charts. Subscriptions automate the delivery of data insights to users.

Synthetic key: A composite key between two tables in the data model, created when two or more tables have common fields, which may need to be reviewed if it results in a data model error. Synthetic keys are generated to link related tables.

Tables: ODS, HDS, and Change: Types of tables in a data project such as the Current table (ODS), the Prior table (HDS), and the Change table, serving different purposes within the data architecture. These tables are used to manage historical data changes.

Target: The destination or endpoint where data is intended to be transferred, stored, or loaded, in data movement, migration, or synchronization processes. Targets define where data should be placed.

Tenant: The deployment of Qlik Cloud, holding items such as users, apps, and spaces. Tenants represent the individual environments within Qlik Cloud.

Training dataset: The dataset used to train a machine learning model in Qlik AutoML, designed to learn patterns and make predictions on new data. Training datasets are used to teach models.

Transform: A task that allows creation of reusable data transformations in a data pipeline with rules and custom SQL. Transformations modify data to prepare it for analysis or storage.

Type 1 - Operational Data Store (ODS): In ODS datasets, new information overwrites the original information, i.e., no historical data is kept. Type 1 ODS tables update existing data with new information.

Type 2 - Historical Data Store (HDS): In HDS datasets, a new record representing the new information is added to the table, including both the original and the new record. Type 2 HDS tables maintain historical data.

Variable: A variable in Qlik Sense is a value container which can store a static or a calculated value, like a numeric or an alphanumeric value. Variables hold values that can be used in expressions and calculations.

Views: Virtual representations of physical datasets in data projects, which can query and fetch relevant data dynamically without occupying significant disk space. Views provide efficient data access.

Visualization: Charts, extensions, and other objects that help visualize data for exploration on a sheet. Visualizations are used to represent data graphically.

Vocabulary: A business logic feature in Qlik that allows the addition of synonyms and custom analyses to Insight Advisor Search and Chat. Vocabulary customization enhances business analysis capabilities.

Sheet view: A view in Qlik Sense representing a canvas where users can arrange data visualizations and other objects to tell a story or answer questions. Sheet views are used for creating customized data presentations.

Working in spaces in Qlik Cloud Data Integration: Refers to the governed areas in Qlik Cloud where users can create and store data projects, manage data connections, and access Data Movement gateways. Working in spaces involves data management and integration activities.

Working in managed spaces: Involves utilizing tightly controlled spaces to share applications with a select group of users. Managed spaces ensure controlled access and collaboration.

Working in personal spaces: Related to a user's private space where they can develop applications independently. Personal spaces provide individual workspaces for app development.

Working in shared spaces: Pertains to areas in Qlik Cloud where users can collaborate and share applications and data sources. Shared spaces facilitate teamwork and sharing of resources.

Storing datasets: Involves keeping datasets up-to-date in the Qlik Cloud data pipeline without manual intervention after data is transferred from the landing zone. Data storage ensures data availability for analysis.

Using data storytelling: A feature that enables users to share insights and discoveries from data analysis through a narrative combining reporting, presentation, and exploratory analysis. Data storytelling enhances data communication.

Scheduling reports with subscriptions: Allows users to configure and send recurring reports via email containing selected sheets or charts. Subscriptions automate report delivery.

Synthetic keys: Composite keys in the data model created when common fields between two or more tables exist, which may require review if data model errors are present. Synthetic keys are generated to link related tables.

Machine learning concepts: Encompass general principles of machine learning, such as targets for predictions in Qlik AutoML and the concept of data movement to a target destination. Machine learning concepts provide the foundation for predictive analytics.

Working with visualizations: Encompasses creating and interacting with charts, extensions, and other objects on a sheet to explore and understand data patterns. Working with visualizations is a key aspect of data analysis.

Business logic vocabulary: A feature in Qlik that allows adding synonyms and custom analyses to Insight Advisor Search and Chat, enhancing business analysis capabilities. Vocabulary customization improves data understanding and search capabilities.

Account: Use your work or school account to sign in to Power BI. Administrators manage work or school accounts in Azure Active Directory. Your level of access is determined by the Power BI license associated with that account and the capacity type where content is stored. See license and Premium.

Admin portal: The location where Power BI admins manage users, features, and settings for Power BI in their organization.

Aggregates: When the values of multiple rows are grouped together as input on criteria to form a single value of more significant meaning or measurement. Only implicit measures can be aggregated.

Alert (alerts): A feature that notifies users of changes in the data based on limits they set. Alerts can be set on tiles pinned from report visuals. Users receive alerts on the service and on their mobile app.

Annotate: To write lines, text, or stamps on a snapshot copy of a tile, report, or visual on the Power BI mobile app for iOS and Android devices.

App (apps): A bundle of dashboards, reports, and datasets. It also refers to the mobile apps for consuming content such as the Power BI app for iOS.

AppSource: Centralized online repository where you can browse and discover dashboards, reports, datasets, and apps to download.

ArcGIS for Power BI: ArcGIS is a mapping and analytics platform created by the company Esri. The name of the visual included in the Power BI visuals library is called ArcGIS for Power BI.

Auto Insights: Now called Quick Insights.

BI: Business intelligence.

Bookmark: A view of data captured in the Bookmarks pane of a report in Power BI Desktop or service. In Desktop, the bookmarks are saved in the pbix report file for sharing on the PowerBI service.

Breadcrumbs: The navigation at the top left to quickly navigate between reports and dashboards.

Calculation: A mathematical determination of the size or number of something.

Capacity (Power BI Premium): Data models running on hardware fully managed by Microsoft in Microsoft cloud data centers to help ensure consistent performance at scale. BI solutions are delivered to the entire organization regardless of Power BI license.

Card (visual type): A Power BI visualization type.

Card (Power BI Home): Power BI Home displays rectangular and square pictures that represent dashboards, reports, apps, and more. These pictures are referred to as cards.

Certified custom visual: A Power BI custom visual that met requirements and passed strict security testing.

Connect live: A method of connecting to SQL Server Analysis Services data models. Also called a live connection.

Connector: Power BI Desktop includes an ever-growing collection of data connectors that are built to connect to a specific data source. Examples include GitHub, MailChimp, Power BI dataflows, Google Analytics, Python, SQL Server, Zendesk, and more than 100 additional data sources.

Container: The areas on the navigation pane are containers. In the nav pane, you'll find containers for: Browse, Data hub, Apps, Metrics, Deployment pipelines, Learn, Workspaces, and Home.

Content: Content for the Power BI service is generally dashboards, reports, and apps. It can also include workbooks and datasets.

Content list: The content index for an app.

Content view: The view that lists Power BI content you created or content that was shared by other designers.

Continuous variable: A continuous variable can be any value between its minimum and maximum limits; otherwise, it is a discrete variable. Examples are temperature, weight, age, and time. Continuous variables can include fractions or portions of the value. The total number of blue skateboards sold is a discrete variable since we can't sell half a skateboard.

Correlation: A correlation tells us how the behavior of things are related. If their patterns of increase and decrease are similar, then they're positively correlated. And if their patterns are opposite, then they're negatively correlated. For example, if sales of our red skateboard increase each time we run a TV marketing campaign, then sales of the red skateboard and the TV campaign are positively correlated.

Cross-filter: Applies to visual interactions. Cross-filtering removes data that doesn't apply. For example, selecting Moderation in the doughnut chart cross-filters the line chart. The line chart now displays only data points that apply to the Moderation segment.

Cross-highlight: Applies to visual interactions. Cross-highlighting retains all the original data points but dims the portion that doesn't apply to your selection. For example, selecting Moderation in the doughnut chart cross-highlights the column chart. The column chart dims all the data that doesn't apply to the Moderation segment and highlights all the data that does apply to the Moderation segment.

Custom visual: Visuals that are created by the community and Microsoft. They can be downloaded from the Microsoft Store for use in Power BI reports.

Dashboard: In the Power BI service, a dashboard is a single page, often called a canvas, that uses visualizations to tell a story. Because it is limited to one page, a well-designed dashboard contains only the most important elements of that story. Dashboards can be created and viewed only in the Power BI service, not in Power BI Desktop.

Data connector: See connector.

Data model (Excel data model): In Power BI content, a data model refers to a map of data structures in a table format. The data model shows the relationships that are being used to build databases. Report designers, administrators, and developers create and work with data models to create Power BI content.

Dataflow: Dataflows ingest, transform, integrate, and enrich big data by defining data source connections, Extract Transform Load (ETL) logic, refresh schedules, and more. Formerly called 'data pool'.

Dataset: A dataset is a collection of data used to create visualizations and reports.

Desktop (or Power BI Desktop): Free Power BI tool used primarily by report designers, admins, and developers.

Diamond (Power BI Premium): The shape of the icon that signifies a workspace is a Premium capacity workspace.

Dimension: Dimensions are categorical (text) data. A dimension describes a person, object, item, product, place, and time. In a dataset, dimensions are a way to group measures into useful categories. For our skateboard company, some dimensions might include looking at sales (a measure) by model, color, country/region, or marketing campaign.

Drill up, drill down, drillthrough: In Power BI, 'drill down' and 'drill up' refer to the ability to explore the next level of detail in a report or visual. 'Drill through' refers to the ability to select a part of a visual and be taken to another page in the report filtered to the data that relates to the part of the visual you selected on the original page. Drill to details commonly means to show the underlying records.

Editing view: The mode in which report designers can explore, design, build, and share a report.

Ellipsis: (...) menu. Selecting an ellipsis displays additional menu options. Also referred to as the More actions or More options menu depending on the menu options.

Embed code: A common standard across the internet. In Power BI, the customer can generate an embed code and copy it to place content such as a report visual on a website or blog.

Embedded: See Power BI Embedded.

Embedding: In the Power BI developer offering, the process of integrating analytics into apps using the Power BI REST APIs and the Power BI SDK.

Environment: [Power BI Desktop, Power BI Mobile, the Power BI service, and others] Another way to refer to one of the Power BI tools. It's OK to use Power BI environment (tenant) in documentation where it might help business analysts who are familiar with the term 'tenant' to know it's the same thing.

Explicit measures: Power BI uses explicit measures and implicit measures (see definition). Explicit measures are created by report designers and saved with the dataset. They are displayed in Power BI as fields and can therefore be used over and over. For example, a report designer creates an explicit measure TotalInvoice that sums all invoice amounts. Colleagues who use that dataset and who have edit access to the report can select that field and use it to create a visual. When an explicit measure is added or dragged onto a report canvas, Power BI does not apply an aggregation. Creating explicit measures requires edit access to the dataset.

Filter versus highlight: A filter removes data that does not apply. A highlight grays out the data that does not apply.

Focus mode: Use focus mode to pop out a visual or tile to see more detail. You can still interact with the visual or tile while in focus mode.

Full-screen mode: Use full-screen mode to view Power BI content without the distraction of menus and navigation panes.

Gateway or on-premises data gateway: A bridge to underlying data sources. It provides quick and secure data transfer between the Power BI service and on-premises data sources that support refresh. Usually managed by IT.

High-density visuals: Visuals with more data points than Power BI can render. Power BI samples the data to show the shape and outliers.

Home: The default landing page for Power BI service users. Doesn't modify anything. Can be called Power BI Home or simply Home.

Implicit measures: Power BI uses implicit measures and explicit measures. Implicit measures are created dynamically when you drag a field onto the report canvas to create a visual, and Power BI automatically aggregates the value using one of the built-in standard aggregations (SUM, COUNT, MIN, AVG, and others). Creating implicit measures requires edit access to the report.

Insights: See quick insights.

KPIs: Key performance indicators. A type of visual.

Left navigation (left nav): This was replaced with nav pane but might still appear in some documentation. The controls along the left edge of Power BI service. First instance: navigation pane. Subsequent mentions or tight spaces: nav pane.

License: Your level of access is determined by the Power BI license associated with your account and the capacity type where content is stored.

List page or content list: One of the section pages for the elements in the nav pane. For example, Create Data hub or My workspace.

Measure: A measure is a quantitative (numeric) field that can be used to do calculations. Common calculations are sum, average, and minimum.

Microsoft R (R): R is a programming language and software environment for statistical computing and graphics.

Mobile app: Apps that allow you to run Power BI on iOS, Android, and Windows mobile devices.

Modeling (Power BI Desktop): Getting the data you've connected to ready for use in Power BI. This includes creating relationships between tables in multiple data sources, creating measures, and assigning metrics.

My workspace: The workspace for each Power BI customer to use to create content.

Native: Included with the product. For example, Power BI comes with a set of native visualization types. But you can also import other types such as Power BI visuals.

Navigation pane or nav pane: The controls along the left edge of the Power BI service.

Notification: Messages sent by and to the Power BI Notification center.

Notification center: The location in the service where messages are delivered to users such as notice of sunsetting certain features.

OneDrive for work or school vs OneDrive: OneDrive is a personal account and OneDrive for work or school is for work accounts.

On-premises: The term used to distinguish local computing (in which computing resources are located on a customer's own facilities) from cloud computing.

On-premises data gateways: See gateways or on-premises data gateways.

PaaS: Platform as a service, for example, Power BI Embedded.

Page: Reports have one or more pages. Each tab on the report canvas represents a page.

Paginated reports: Paginated reports are designed to be printed or shared. They display all the data in a table even if the table spans multiple pages.

pbiviz: The file extension for a Power BI custom visual.

pbix: The file extension for a Power BI Desktop file.

Permissions: What a user can and can't do in Power BI is based on permissions.

Phone report: The name for a Power BI report that's been formatted for viewing on a phone.

Phone view: The user interface in the Power BI service for laying out a phone report.

Pin unpin: The action a report designer takes when placing a visual, usually from a report, onto a dashboard.

Power BI, Power BI service, Power BI Desktop, Power BI mobile: Some of the Power BI offerings. Power BI is the general term.

Power BI Desktop: Also referred to as Desktop. The free Windows application of Power BI you can install on your local computer.

Power BI Embedded: A product used by developers to embed Power BI dashboards and reports into their own apps, sites, and tools.

Power BI Premium: An add-on to the Power BI Pro license that enables organizations to predictably scale BI solutions.

Power BI Pro: A monthly per-user license that provides the ability to build reports and dashboards, collaborate on shared data, and more.

Power BI Report Builder: A free standalone Windows Desktop application used for authoring paginated reports.

Power BI Report Server: An on-premises report server with a web portal in which you display and manage reports and KPIs.

Power BI service: An online SaaS (software as a service).

Premium workspace: A workspace running in a capacity signified to customers by a diamond icon.

Pro license or Pro account: See account and license.

Publish: Power BI service report designers bundle the contents of a Power BI workspace to make it available to others as a Power BI app.

Q&A: The Power BI feature that allows natural language questions about a dataset and get responses in the form of visualizations.

Q&A virtual analyst: [Power BI Mobile] For iOS, the conversational UI for Q&A.

QR codes: [Power BI Mobile] A matrix barcode that can be generated for dashboards or tiles in the Power BI service to identify products.

query string parameter: Add one to a URL to pre-filter the results seen in a Power BI report.

Quick Insights: Automatically generated insights that reveal trends and patterns in data.

Reading view: Read-only view for reports (as opposed to Editing View).

real-time streaming: The ability to stream data and update dashboards in real time from sources such as sensors, social media, usage metrics, and more.

recent: The container on the home page that holds all the individual items that were accessed last.

related content: Shows the individual pieces of content that contribute to the current content. For example, for a dashboard, you can see the reports and datasets providing the data and visualizations on the dashboard.

relative links: Links from dashboard tiles to other dashboards and reports that have been shared directly or distributed through a Power BI app. This enables richer dashboards that support drillthrough.

report: A multi-perspective view into a single dataset with visualizations that represent different findings and insights from that dataset. Can have a single visualization or many, a single page or many pages.

report editor: The report editor is the tool in which new reports are created and changes are made to existing reports by report designers.

report measures: Also called custom calculations. Excel calls these calculated fields. See also measures.

responsive visuals: Visuals that change dynamically to display the maximum amount of data and insights no matter the screen size.

row-level security (RLS): Power BI feature that enables database administrators to control access to rows in a database table based on the characteristics of the user executing a query (for example, group membership).

SaaS: Software as a service is a way of delivering applications over the internet as a web-based service. Also referred to as web-based software, on-demand software, or hosted software.

screenshot: Simple screenshots of a report can be emailed using the 'send a screenshot' feature.

service: See Power BI service. A standalone resource available to customers by subscription or license. A service is a product offering delivered exclusively via the cloud.

settings: The location for Power BI users to manage their own general settings such as whether to preview new features, set the default language, close their account, and more.

share (sharing): In Power BI, sharing typically means directly sharing an individual item (a dashboard or report) with one or more people by using their email address.

Shared with me: The container in the nav pane that holds all the individual items that were directly shared by another Power BI user.

snapshot: In Power BI, a snapshot is a static image compared with a live image of a tile, dashboard, or report.

SQL Server Analysis Services (SSAS): An online analytical data engine used in decision support and business analytics, providing the analytical data for business reports and client applications such as Power BI, Excel, Reporting Services reports, and other data visualization tools.

SQL Server Reporting Services (SSRS): A set of on-premises tools and services to create, deploy, and manage report servers and paginated reports.

streaming data: See real-time streaming. The ability to stream data and update dashboards in real time.

subscriptions (subscribe): You can subscribe to report pages, apps, and dashboards and receive emails containing a snapshot. Requires a Power BI Pro license.

summarization: [Power BI Desktop] The operation being applied to the values in one column.

tiles: Power BI dashboards contain report tiles.

time series: A time series is a way of displaying time as successive data points. Those data points could be increments such as seconds, hours, months, or years.

value (values): Numerical data to be visualized.

visual (visualization): A chart. Some visuals are bar chart, treemap, doughnut chart, map.

visual interaction: One of the great features of Power BI is the way all visuals on a report page are interconnected. If you select a data point on one of the visuals, all the other visuals on the page that contain that data change based on that selection.

Visualizations pane: Name for the visualization templates that ship in the shared report canvas for Power BI Desktop and the Power BI service.

workbook: An Excel workbook to be used as a data source. Workbooks can contain a data model with one or more tables of data loaded into it by using linked tables, Power Query, or Power Pivot.

workspace: Containers for dashboards, reports, and datasets in Power BI. Users can collaborate on the content in any workspace except My workspace.

x-axis: The axis along the bottom, the horizontal axis of a line graph.

y-axis: The axis along the side, the vertical axis of a line graph.

The first step to using Datalogz is to create a connector for the organization.

Overview

A connector is an access for Datalogz to connect to a BI environment. We recommend organizations create single connector per each BI platform. Currently, Datalogz support connection with PowerBI, Tableau, Qlik, and Spotfire.

When a connector is created, Datalogz automatically extracts BI metadata and creates default monitors.

Connector Settings

Navigate to Connectors and choose a connector by clicking the connector name.

Admins will be able to:

• Edit connector name

• Edit projects/workspaces/streams

• Modify fresh schedule

• Export connector history

• Manually refresh connector

• Disable connector

Only Root User and Admin have access to Connector Settings.

Disable Connector

Toggle on/off to enable/disable the connector.

Multiple Connectors

You can create multiple connectors in Datalogz under a single account, as the organization might use multiple BI tools (e.g. Tableau, Qlik).

Only Root Users can create new connectors.

To create a new connector:

1. Click New Connector

2. Choose the type of connector

Create teams in your organization to assign different projects/workspaces/streams.

Overview

After the initial onboarding, by default, Datalogz will generate a team with the highest privileges for your organization. Root Users can create and split teams. Any user can be part of one or many teams.

To group users

• who work together frequently

• who work on one area of work, such as finance, marketing, etc.

• who share the same level of access, such as executives, global admin, etc.

View Your Team(s)

The teams you are a member of will be listed in Team and in the top-left dropdown.

The current team you navigate in will be shown in the field of the dropdown.

Create a Team

• Go to Teams, click on New Team in the upper right of the table.

• Create a name for the team

• Assign a connector

• Assign Platform Contexts

• (Optional) Select Users

Timezone

Timezone is automatically set at localtime based on the user's IP address.

Access Control

All members of a team can view their team. Only Root User can create new teams. Both Root User and Admin can invite members to their teams. Anyone can view default insights, default monitors, and default alerts. Members in teams can view team insights, team monitors, and team alerts, as long as they are not private.

Delete a team

Deleting a team will not delete a user. But it will permanently delete any connectors and platform contexts associated with it. This can't be undone, and the data cannot be recovered.

This guide will walk you through how to set up PowerBI Connector in Datalogz via service principal. This method allows read-only access to PowerBI's administrative APIs for a single PowerBI tenant.

Time estimated: 6 mins

To start, we need the following:

1. Azure App registration → https://portal.azure.com/#view/Microsoft_AAD_RegisteredApps/ApplicationsListBlade

2. Azure Group creation → https://portal.azure.com/#view/Microsoft_AAD_IAM/GroupsManagementMenuBlade/~/Overview

3. PowerBI configuration → https://app.powerbi.com/admin-portal/tenantSettings

4. Datalogz account & license activation -> contact customers@datalogz.io

5. Log in to your Datalogz account → https://app.datalogz.io/#/organization/connectors

Step 1: Create a new App Registration

1. Open App registrations link -> https://portal.azure.com/#view/Microsoft_AAD_RegisteredApps/ApplicationsListBlade

2. Create New Registration
3. Create a name for your app, select single tenant (first option) as a supported account type, ignore the optional field as needed, and click Register at the bottom.

In this new app, we will need three pieces of information

1. Application (client) ID & Directory (tenant) ID

1. Client Secret Value

   1. Under Manage in the side nav

   2. Click Certificates & secrets & create new

   3. Find the Client Secret Value

Step 2: Create a new Group in Azure

1. Click and open Azure Group link https://portal.azure.com/#view/Microsoft_AAD_IAM/GroupsManagementMenuBlade/~/Overview

2. Create a new group
3. Add members by adding the new app to the Enterprise applications
4. Create new group

Step 3: Configure PowerBI with the Group

1. Open PowerBI Tenant Settings: https://app.powerbi.com/admin-portal/tenantSettings

2. In the Admin Portal, search "admin api" in the search box.
3. Enable all three Admin API settings and apply all newly created Groups.
4. This configuration could take 15-25 mins for changes to work its way through PowerBI.

Step 4: Create a New Connector in Datalogz

1. Log in to your Datalogz account

2. Navigate to Connector in the sidebar, and click New Connector
3. Choose PowerBI, and select Connect using a Service Principal (SP)
4. Copy-paste the new application information.

   1. Directory (tenant) ID

   2. Application (client) ID

   3. Client Secret Value

Step 5: Reload the connection in Datalogz

1. Select a PowerBI workspace

2. Reload Workspace and select all, option to expand and choose personal workspace

3. Rename Connection and Continue to Next Step.

4. Schedule the Connector Refresh rate.

5. Refresh and click to view the new connector.

For more questions, please contact Datalogz support support@datalogz.io

Capacity Monitoring (Additional Setup)

This additional setup for Fabric Capacity Metrics data is optional, and is summarized below. The full documentation for this setup can be found in Microsoft's Doc pages.

1. Install the Microsoft Fabric Capacity App in your Power BI environment.
2. Navigate to the "Microsoft Fabric Capacity Metrics" Workspace that is created when the app is installed. Manage access and grant "Admin" permissions to the Datalogz service principal Groups.

The Datalogz Framework is our method for governing the business intelligence environment. It influences how we approach BI issues and design the product.

We recommend reading our

This guide will walk you through how to set up Qlik Connector in Datalogz using the Virtual Proxy Authentication Method.

Time estimated: 6 mins.

To start, We need:

1. Hostname, the address at which we access Click.

2. Virtual Proxy details

3. (Optional) N-Printing details

Create a New Connector in Datalogz

1. Log in to your Datalogz account

2. Navigate to Connector in the sidebar, and click New Connector
3. Choose Qlik

1. Copy-paste

   1. Hostname, the address at which we access Click.

   2. Virtual Proxy details

   3. (Optional) N-Printing details

1. Select Streams to extract.

For more questions, please contact Datalogz support support@datalogz.io

This guide will walk you through how to set up a Tableau connector in Datalogz via Tableau Cloud.

Time estimated: 4 mins.

To start, log in to the following:

1. Log in to your Datalogz account → https://app.datalogz.io/#/organization/connectors

2. Sign in to Tableau Cloud https://sso.online.tableau.com/public/idp/SSO

Step 1: Create a new connector in Datalogz

1. Navigate to Connector in the sidebar, and click New Connector
2. Choose Tableau

Step 2: Fill in the Connector details

1. Copy-paste Host and Site Name in the URL
2. Find the latest Rest API Version

   https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_whats_new.htm

Create Access Token Name

1. In the Tableau home screen, click on the user profile icon. Under account settings, find personal access tokens.
2. Name & create a new token
3. Copy-paste Token details, and connect the connector

   1. Secret to Access Token Secret < Warning: this Secret will disappear after closing this dialog box>

   2. Token Name to Access Token Name

4. Name the connector select projects, and continue.

5. Set refresh schedule and finish the connector set up.

More information on personal access tokens and managing your account settings.

Or contact Datalogz support support@datalogz.io

Overview

Insight page is the visualization hub of BI (Business Intelligence) alerts and metadata within our platform. It provides users with a visual view of their BI environment through the summary of historical and real-time data. Insight delivers critical information in an easily digestible format, helping users monitor the health of their BI systems.

Datalogz offers two types of Insights:

Global Insights: A consolidated view of BI metrics and alerts across all connected BI tools (e.g., PowerBI, Tableau, Qlik), giving a comprehensive perspective of the entire BI ecosystem.

Team-based Insights: These are more focused visualizations, offering insights for specific BI tools that are assigned to individual teams. This allows teams to drill down into the data that is most relevant to them.

What's unique about Insight is that it provides customized visualizations based on the type of BI tool being monitored, adjusting for the specific nuances and metrics associated with tools like PowerBI, Tableau, and Qlik.

How to create Insights

At this time, users or clients cannot create new Insight tabs by themselves within the platform. To add new Insights or customize existing ones, users must contact our team.

If a client needs a specific type of Insight or wants to customize the visualizations for their BI tools, they can reach out to our team, and we will work with them to design and implement the desired insights. This ensures that all Insights align with the unique requirements and data goals of the organization.

Overview

Root users and admins can manually invite, suspend, and promote members from the user's settings page.

Invite members

Send an invitation

To send an invitation:

1. Go to the Teams > Users

2. Click Invite Users.

3. Enter the invitee(s) email address, and assign the user type.

4. Click Add.

5. Click Send invites. New members will receive an invite link via email along with steps to join the workspace.

User Roles

Root Admin

Root Admin have the highest organizational privilege and can only be assigned by the Datalogz team during the onboarding.

Root Admin are the only users who can:

• View or edit connectors

• Grant or remove Admin roles

• Manage login methods

• Create a new Team

Admin

Admins have the second highest privilege in assigned Teams and synced projects/workspace/streams.

Admins can

• Invite and manage users

• Rename Teams

• Edit monitor and alerts

Admins can not

• View or edit connectors

• Delete a Team

Member

Members are viewers and collaborators in assigned Teams, projects/workspaces/streams.

Delete member

To delete a user from Datalogz

1. Go to Teams > Users

2. Find the member's name, and click on the delete icon in the same column

The user will be removed from Datalogz and unable to access it unless they are re-invited.

To remove a member from a Team

1. Go to Teams

2. Click on the Team's name

3. Deselect a member's name

The member will be removed from the Team and unable to access assigned projects/workspaces/streams. The member will remain a user in Datalogz.

Overview

My Monitors are user-created monitors. Currently, users can only create new monitors by duplicating an existing Default Monitor and modifying it according to their needs. My Monitors provide users the flexibility to adjust the monitoring criteria, such as SQL queries and thresholds, to fit their specific requirements.

How does it work?

• Duplicating Default Monitors: Users can create My Monitors by duplicating an existing Default Monitor. This duplication feature allows users to build upon the existing structure of the default SQL queries and modify them according to their specific needs.

• Customizable Queries: Once duplicated, users can modify the SQL query, thresholds, and alert criteria of the monitor. This enables users to tailor the monitor to focus on specific assets, custom data conditions, or unique business requirements that are not covered by the default configuration.

• Full Control: Users have full control over their My Monitors, including the ability to edit the monitor title, assign it to specific teams, and customize alert priorities or resolutions.

Overview

Default Monitors are out-of-the-box monitors created and owned by our team. Default Monitors are pre-configured to help users quickly begin monitoring common and critical aspects of their BI environment. These monitors are set up with default SQL queries and thresholds based on industry best practices, allowing users to start receiving alerts and insights without any initial setup.

How does it work?

• Pre-Built by Our Team: Default Monitors are developed and maintained by our team, designed to cover a wide range of use cases for managing BI environments. These monitors come preconfigured with SQL queries that evaluate common thresholds and criteria for data performance, security, usage, and compliance.

• Ready to Use: Users can immediately begin using Default Monitors after setting up their BI environment. These monitors will automatically generate alerts based on predefined rules, requiring no further configuration from the user.

• Fixed Configuration: Users cannot directly modify the SQL queries or thresholds of Default Monitors, ensuring that these monitors remain consistent and reliable as standard tools for all users.

Introduction

This document provides an in-depth look at Datalogz, a SaaS-based, read-only metadata application, detailing how it interfaces with various BI tools, handles authentication, and ensures data security.

Overview of Datalogz SaaS Solution

Datalogz is a specialized tool designed to administer BI environments effectively through the analysis of non-proprietary metadata. It operates exclusively in a SaaS environment, ensuring enhanced efficiency and security.

Metadata Read from BI Tools

Connector Pipeline

The Datalogz application uses Apache Airflow for connector management, providing BI Admins with pre-built metadata pipelines they can choose to schedule daily, weekly or hourly basis. New alerts will be generated after each connector refresh based on the latest data that has changed.

Your connectors will retrieve metadata from the following API endpoints:

Connectors must be configured by BI Admins to approve the Datalogz application. This will provide read-only access to standard and admin-level APIs based on a selection of Groups. Groups are generally defined as follows for each system:

• PowerBI: Workspaces

• Tableau: Projects

• Qlik: Streams

The admin-level APIs unlock the most insight for your BI Admins when it comes to types of Issues and Recommendations Datalogz is able to provide. After a new connector is created, BI Admins can use Datalogz RBAC to assign fine-grained permissions to Users who should only have access to certain metadata from certain Groups.

Authentication and Security Measures

Secrets used to authenticate to the BI Metadata APIs are encrypted and stored in a managed-identity Azure Key Vault within a private subnet of the Datalogz private virtual network. Only the Datalogz backend virtual machine used for running data extraction pipelines has network access and adequate privileges to use these secrets for authentication.

Datalogz as a "Read Only" Metadata Application

Datalogz is able to provide BI Ops insights and recommendations using metadata about the existence of BI reports, users, activities, and other BI assets. Datalogz does not require any data access to generate these recommendations, so we do not require this level of permission to be granted to service principal credentials or personal access tokens. This means Datalogz does not have access to query any data that is used in BI reports. Datalogz only requires access to metadata about the nature of those BI reports, such as the title, description, configuration, asset lineage, usage patterns, refresh durations, successful uptime, and governance features. From this information, the Datalogz recommendations are generated without data access, which can also be described as “read-only access to metadata only.”

Comparison with On-Prem Deployment

Compared to the On-Prem Deployment, a SaaS Deployment has the following benefits:

• Datalogz monitors and maintains all infrastructure required to run the Datalogz BI Ops platform.

• Datalogz monitors and maintains all services, code, and images required to run the backend and frontend services.

• Datalogz upgrades, tests, and deploys all new versions of Datalogz to give you a seamless experience between versions.

• No resources on your end are required to commit to the following activities that are part of an on-prem deployment engagement:

  • Monitoring and maintaining the infrastructure.

  • Monitor and maintain the services, code, and images required to run the services.

  • Upgrading and deploying new versions of Datalogz using Docker Desktop and some Windows or Linux commands.

Architecture Diagram

Resource Group: rg-biops-prod-eastus2-001

Resources:

• Virtual Machines:

• Frontend VM: Hosting services running in containers using Docker Compose behind an nginx web server in the public subnet.

• Backend VM: Hosting backend services (ELT API, Fast API, etc.) in containers using Docker Compose behind an nginx web server in the private subnet.

• Virtual Network:

  • Public Subnet for frontend services.

  • Private Subnet for backend services.

• Databases:

  • Azure PostgreSQL database for operational data.

  • Snowflake data warehousing service for BI metadata ingestion and analysis using Azure Storage Integration and Snowflake Network Policy.

• Storage:

  • File Storage configured for secure data ingestion via Snowflake External Stage using Azure blob storage for staging data before loading it into Snowflake.

• Key Vault:

  • Key Vault for secure storage of keys and passwords, with Managed Identity access exclusively from the backend VM.