Introduction

Welcome to the Datalogz Infrastructure Deployment Guide. This document provides comprehensive instructions for deploying the Datalogz platform either on Amazon Web Services (AWS) or on bare metal servers. It is intended for DevOps engineers, system administrators, and IT professionals responsible for setting up and managing Datalogz infrastructure within your organization.

The guide includes step-by-step procedures, diagrams, and best practices to ensure a smooth deployment. All diagrams are dynamically generated using D2 and can be found in the diagrams directory.

Prerequisites

Before you begin the deployment process, ensure that you have met the following prerequisites:

• Access Permissions:

  • For AWS deployment, an AWS account with permissions to create and manage resources such as EC2 instances, RDS Instances, S3 buckets, and IAM roles.

  • For bare metal deployment, administrative access to the physical or virtual servers.

• Required Software Installed:

  • Terraform

  • AWS CLI

  • Ansible

  • Git

• Network Requirements:

  • Open ports as required by Datalogz services (e.g., HTTP/HTTPS ports, database ports).

  • SSH access to servers for Ansible to run playbooks.

• SSH Keys:

  • A valid SSH key pair for accessing the servers. The private key should be accessible from the machine where Ansible is run.

AWS Deployment

Overview

Deploying Datalogz on AWS involves provisioning infrastructure using Terraform and configuring instances using Ansible. This approach automates the deployment process, ensuring consistency and scalability.

Installation and Configuration

Step 1: Install Required Tools

Ensure that the following tools are installed on your local machine:

1. Terraform

   • Installation Guide: Install Terraform

   • Verify installation: terraform -v

2. AWS CLI

   • Installation Guide: Install AWS CLI

   • Verify installation: aws --version

3. Ansible

   • Installation Guide: Install Ansible

   • Verify installation: ansible --version

Step 2: Configure AWS Credentials

Set up your AWS credentials so that the AWS CLI and Terraform can authenticate with your AWS account.

• Option 1: Using Environment Variables

export AWS_ACCESS_KEY_ID="your-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
export AWS_DEFAULT_REGION="your-default-region"

• Option 2: Using AWS CLI Configuration

  Run the AWS configure command: aws configure

  You will be prompted to enter your AWS Access Key ID, Secret Access Key, Default Region, and Output Format.

Step 3: Verify AWS Configuration

Test your AWS configuration by trying to list your current S3 buckets: aws s3 ls

Terraform Deployment

Step 4: Navigate to the AWS Terraform Directory

Change your current directory to the AWS Terraform configuration folder: cd aws

Step 5: Initialize Terraform

Initialize the Terraform working directory. This command downloads the necessary provider plugins: terraform init

Step 6: Review Terraform Variables

Inspect the variables.tf file to understand the variables required for deployment, such as AWS Region, S3 Bucket name, and EC2 Key Name.

Step 7: Plan the Terraform Deployment

Generate and review the execution plan to understand the resources that will be created: terraform plan

Review the Plan:

• Ensure that the resources and configurations match your expectations.

• Check for any unintended changes to existing infrastructure.

Step 8: Apply the Terraform Plan

Apply the execution plan to deploy the infrastructure: terraform apply

• Confirmation:

  • You will be prompted to confirm the deployment. Type yes to proceed.

• Deployment Duration:

  • The deployment typically takes around 15-25 minutes.

Step 9: Post-Deployment Verification

After Terraform completes:

• State File:

  • Ensure that the terraform.tfstate file has been updated.

Ansible Configuration

Step 10: Update Ansible Inventory

Ansible uses a dynamic inventory plugin to find the IP address of the EC2 instance based on the instance Name tag.

• Location: Change directory to the Ansible folder: cd ../ansible

Step 11: Configure Ansible Settings

Update the ansible.cfg file:

• Private Key File:

  • Set the private_key_file parameter to the path of your SSH private key.

  Copy

  [defaults]
  inventory = inventory.yml
  private_key_file = /path/to/your/private/key.pem
  host_key_checking = False

Step 12: Run the Ansible Playbook

Execute the Ansible playbook to configure the EC2 instances: ansible-playbook deploy.yml -v

• Verbose Mode:

  • The v flag enables verbose output. Use vvv for even more detailed logs.

• Common Issues:

  • If you encounter SSH authentication errors, verify that the SSH key has the correct permissions (chmod 600 key.pem).

Step 13: Verify Deployment

After the playbook runs:

• Services Check:

  • SSH into the instances and verify that the Datalogz containers are running.

• Application Test:

  • Access the application URL to ensure it is functioning correctly.

The first step to using Datalogz is to create a connector for the organization.

Overview

A connector is an access for Datalogz to connect to a BI environment. We recommend organizations create single connector per each BI platform. Currently, Datalogz support connection with PowerBI, Tableau, Qlik, and Spotfire.

When a connector is created, Datalogz automatically extracts BI metadata and creates default monitors.

Connector Settings

Navigate to Connectors and choose a connector by clicking the connector name.

Admins will be able to:

• Edit connector name

• Edit projects/workspaces/streams

• Modify fresh schedule

• Export connector history

• Manually refresh connector

• Disable connector

Only Root User and Admin have access to Connector Settings.

Disable Connector

Toggle on/off to enable/disable the connector.

Multiple Connectors

You can create multiple connectors in Datalogz under a single account, as the organization might use multiple BI tools (e.g. Tableau, Qlik).

Only Root Users can create new connectors.

To create a new connector:

1. Click New Connector

2. Choose the type of connector

This guide will walk you through how to set up Qlik Connector in Datalogz using the Virtual Proxy Authentication Method.

Time estimated: 6 mins.

To start, We need:

1. Hostname, the address at which we access Click.

2. Virtual Proxy details

3. (Optional) N-Printing details

Create a New Connector in Datalogz

1. Log in to your Datalogz account

2. Navigate to Connector in the sidebar, and click New Connector
3. Choose Qlik

1. Copy-paste

   1. Hostname, the address at which we access Click.

   2. Virtual Proxy details

   3. (Optional) N-Printing details

1. Select Streams to extract.

This guide will walk you through how to set up PowerBI Connector in Datalogz via service principal. This method allows read-only access to PowerBI's administrative APIs for a single PowerBI tenant.

Time estimated: 6 mins

To start, we need the following:

1. Azure App registration → https://portal.azure.com/#view/Microsoft_AAD_RegisteredApps/ApplicationsListBlade

2. Azure Group creation → https://portal.azure.com/#view/Microsoft_AAD_IAM/GroupsManagementMenuBlade/~/Overview

3. PowerBI configuration → https://app.powerbi.com/admin-portal/tenantSettings

4. Datalogz account & license activation -> contact customers@datalogz.io

5. Log in to your Datalogz account → https://app.datalogz.io/#/organization/connectors

Step 1: Create a new App Registration

1. Open App registrations link -> https://portal.azure.com/#view/Microsoft_AAD_RegisteredApps/ApplicationsListBlade

2. Create New Registration
3. Create a name for your app, select single tenant (first option) as a supported account type, ignore the optional field as needed, and click Register at the bottom.

In this new app, we will need three pieces of information

1. Application (client) ID & Directory (tenant) ID

1. Client Secret Value

   1. Under Manage in the side nav

   2. Click Certificates & secrets & create new

   3. Find the Client Secret Value

Grant Permissions for the new App Registration Here add permissions specifically for the Power

• Tenant.ReadWrite.All

• Dataset.ReadWrite.All

Step 2: Create a new Group in Azure

1. Click and open Azure Group link https://portal.azure.com/#view/Microsoft_AAD_IAM/GroupsManagementMenuBlade/~/Overview

2. Create a new group
3. Add members by adding the new app to the Enterprise applications
4. Create new group

Step 3: Configure PowerBI with the Group

1. Open PowerBI Tenant Settings: https://app.powerbi.com/admin-portal/tenantSettings

2. In the Admin Portal's keyword search box type "XMLA", and in the Integration settings section enable "Allow XMLA endpoints and Analyze in Excel with on-premises semantic models"
3. In the Admin Portal's keyword search box type "fabric api" and enable "Service Princepals can use Fabric APIs"
4. In the Admin Portal's keyword search box type "admin api".
5. Enable all three Admin API settings and apply all newly created Groups.
6. Note: This configuration can take 15-25 mins for changes to work its way through PowerBI.

Step 4: Create a New Connector in Datalogz

1. Log in to your Datalogz account

2. Navigate to Connector in the sidebar, and click New Connector
3. Choose PowerBI, and select Connect using a Service Principal (SP)
4. Copy-paste the new application information.

   1. Directory (tenant) ID

   2. Application (client) ID

   3. Client Secret Value

Step 5: Reload the connection in Datalogz

1. Select a PowerBI workspace

2. Reload Workspace and select all, option to expand and choose personal workspace

3. Rename Connection and Continue to Next Step.

4. Schedule the Connector Refresh rate.

5. Refresh and click to view the new connector.

For more questions, please contact Datalogz support support@datalogz.io

Capacity Monitoring (Additional Setup)

1. Navigate to the Admin Portal and enable the Integration Setting for Semantic Model Execute Queries REST API. When enabled, users in the organization can query semantic models by using Data Analysis Expressions (DAX) through Power BI REST APIs.
2. The full documentation for the "Microsoft Fabric Capacity App" can be found in Microsoft's Doc pages. While you can use an existing Fabric Capacity Metrics app setup in PowerBI it's suggested to get the app again and setup a workspace specifically for use with Datalogz.

   1. Install the Microsoft Fabric Capacity App in your Power BI environment. Navigate to the "Apps" section on the side bar of Power BI. There you can search for the "Microsoft Fabric Capacity App" and add it.

   2. Navigate to the "Microsoft Fabric Capacity Metrics" Workspace that is created with the app. Go to Manage Access and grant "Admin" permissions to the Datalogz service principal Groups.

   3. Run the app for the first time to start data flowing into the semantic model.

3. Provide the Dataset ID for the "Fabric Capacity Metrics" to Datalogz. This can be found in the URL when viewing the semantic model in your web browser.

   1. Sample :

Overview

Root users and admins can manually invite, suspend, and promote members from the user's settings page.

Invite members

Send an invitation

To send an invitation:

1. Go to the Teams > Users

2. Click Invite Users.

3. Enter the invitee(s) email address, and assign the user type.

4. Click Add.

5. Click Send invites. New members will receive an invite link via email along with steps to join the workspace.

User Roles

Root Admin

Root Admin have the highest organizational privilege and can only be assigned by the Datalogz team during the onboarding.

Root Admin are the only users who can:

• View or edit connectors

• Grant or remove Admin roles

• Manage login methods

• Create a new Team

Admin

Admins have the second highest privilege in assigned Teams and synced projects/workspace/streams.

Admins can

• Invite and manage users

• Rename Teams

• Edit monitor and alerts

Admins can not

• View or edit connectors

• Delete a Team

Member

Members are viewers and collaborators in assigned Teams, projects/workspaces/streams.

Delete member

To delete a user from Datalogz

1. Go to Teams > Users

2. Find the member's name, and click on the delete icon in the same column

The user will be removed from Datalogz and unable to access it unless they are re-invited.

To remove a member from a Team

1. Go to Teams

2. Click on the Team's name

3. Deselect a member's name

The member will be removed from the Team and unable to access assigned projects/workspaces/streams. The member will remain a user in Datalogz.

This guide will walk you through how to set up a Tableau connector in Datalogz via Tableau Cloud.

Time estimated: 4 mins.

To start, log in to the following:

1. Log in to your Datalogz account → https://app.datalogz.io/#/organization/connectors

2. Sign in to Tableau Cloud https://sso.online.tableau.com/public/idp/SSO

Step 1: Create a new connector in Datalogz

1. Navigate to Connector in the sidebar, and click New Connector
2. Choose Tableau

Step 2: Fill in the Connector details

1. Copy-paste Host and Site Name in the URL
2. Find the latest Rest API Version

   https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_whats_new.htm

Create Access Token Name

1. In the Tableau home screen, click on the user profile icon. Under account settings, find personal access tokens.
2. Name & create a new token
3. Copy-paste Token details, and connect the connector

   1. Secret to Access Token Secret < Warning: this Secret will disappear after closing this dialog box>

   2. Token Name to Access Token Name

4. Name the connector select projects, and continue.

5. Set refresh schedule and finish the connector set up.

More information on personal access tokens and managing your account settings.

Or contact Datalogz support support@datalogz.io

Overview

Datalogz offers multiple options for users to log in and allows Root User and Admin to restrict methods for added security.

Log-in Methods

Google Authentication

Users can authenticate into Datalogz using Google authentication when using a Google-supported email address.

Microsoft Authentication

Users can authenticate into Datalogz using Microsoft authentication using a Microsoft-supported email address.

SAML

With SAML enabled, users can log in through their identity provider's website by selecting the SAML option on the login page.

Once the organization enables SAML, all members must log in using SAML.

Create teams in your organization to assign different projects/workspaces/streams.

Overview

After the initial onboarding, by default, Datalogz will generate a team with the highest privileges for your organization. Root Users can create and split teams. Any user can be part of one or many teams.

To group users

• who work together frequently

• who work on one area of work, such as finance, marketing, etc.

• who share the same level of access, such as executives, global admin, etc.

View Your Team(s)

The teams you are a member of will be listed in Team and in the top-left dropdown.

The current team you navigate in will be shown in the field of the dropdown.

Create a Team

• Go to Teams, click on New Team in the upper right of the table.

• Create a name for the team

• Assign a connector

• Assign Platform Contexts

• (Optional) Select Users

Timezone

Timezone is automatically set at localtime based on the user's IP address.

Access Control

All members of a team can view their team. Only Root User can create new teams. Both Root User and Admin can invite members to their teams. Anyone can view default insights, default monitors, and default alerts. Members in teams can view team insights, team monitors, and team alerts, as long as they are not private.

Delete a team

Deleting a team will not delete a user. But it will permanently delete any connectors and platform contexts associated with it. This can't be undone, and the data cannot be recovered.