On-prem Deployment

Introduction

Welcome to the Datalogz Infrastructure Deployment Guide. This document provides comprehensive instructions for deploying the Datalogz platform either on Amazon Web Services (AWS) or on bare metal servers. It is intended for DevOps engineers, system administrators, and IT professionals responsible for setting up and managing Datalogz infrastructure within your organization.

The guide includes step-by-step procedures, diagrams, and best practices to ensure a smooth deployment. All diagrams are dynamically generated using D2 and can be found in the diagrams directory.

---

Prerequisites

Before you begin the deployment process, ensure that you have met the following prerequisites:

• Access Permissions:

  • For AWS deployment, an AWS account with permissions to create and manage resources such as EC2 instances, RDS Instances, S3 buckets, and IAM roles.

  • For bare metal deployment, administrative access to the physical or virtual servers.

• Required Software Installed:

  • Terraform

  • AWS CLI

  • Ansible

  • Git

• Network Requirements:

  • Open ports as required by Datalogz services (e.g., HTTP/HTTPS ports, database ports).

  • SSH access to servers for Ansible to run playbooks.

• SSH Keys:

  • A valid SSH key pair for accessing the servers. The private key should be accessible from the machine where Ansible is run.

AWS EC2 Instance

• Backend VM

  • Recommended size: 8 vCPU, 16 GB RAM

  • Recommended instance type: c7i.2xlarge

  • Security Group associated to backend VM

• Frontend VM

  • Recommended size: 4 vCPU, 8 GB RAM

  • Recommended instance type: c7i.xlarge

  • Security Group associated to frontend VM

---

AWS Deployment

Overview

Deploying Datalogz on AWS involves provisioning infrastructure using Terraform and configuring instances using Ansible. This approach automates the deployment process, ensuring consistency and scalability.

Installation and Configuration

Step 1: Install Required Tools

Ensure that the following tools are installed on your local machine:

1. Terraform

   • Installation Guide: Install Terraform

   • Verify installation: terraform -v

2. AWS CLI

   • Installation Guide: Install AWS CLI

   • Verify installation: aws --version

3. Ansible

   • Installation Guide: Install Ansible

   • Verify installation: ansible --version

Step 2: Configure AWS Credentials

Set up your AWS credentials so that the AWS CLI and Terraform can authenticate with your AWS account.

• Option 1: Using Environment Variables

export AWS_ACCESS_KEY_ID="your-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
export AWS_DEFAULT_REGION="your-default-region"

• Option 2: Using AWS CLI Configuration

  Run the AWS configure command: aws configure

  You will be prompted to enter your AWS Access Key ID, Secret Access Key, Default Region, and Output Format.

Step 3: Verify AWS Configuration

Test your AWS configuration by trying to list your current S3 buckets: aws s3 ls

---

Terraform Deployment

Step 4: Navigate to the AWS Terraform Directory

Change your current directory to the AWS Terraform configuration folder: cd aws

Step 5: Initialize Terraform

Initialize the Terraform working directory. This command downloads the necessary provider plugins: terraform init

Step 6: Review Terraform Variables

Inspect the variables.tf file to understand the variables required for deployment, such as AWS Region, S3 Bucket name, and EC2 Key Name.

Step 7: Plan the Terraform Deployment

Generate and review the execution plan to understand the resources that will be created: terraform plan

Review the Plan:

• Ensure that the resources and configurations match your expectations.

• Check for any unintended changes to existing infrastructure.

Step 8: Apply the Terraform Plan

Apply the execution plan to deploy the infrastructure: terraform apply

• Confirmation:

  • You will be prompted to confirm the deployment. Type yes to proceed.

• Deployment Duration:

  • The deployment typically takes around 15-25 minutes.

Step 9: Post-Deployment Verification

After Terraform completes:

• State File:

  • Ensure that the terraform.tfstate file has been updated.

---

Ansible Configuration

Step 10: Update Ansible Inventory

Ansible uses a dynamic inventory plugin to find the IP address of the EC2 instance based on the instance Name tag.

• Location: Change directory to the Ansible folder: cd ../ansible

Step 11: Configure Ansible Settings

Update the ansible.cfg file:

• Private Key File:

  • Set the private_key_file parameter to the path of your SSH private key.

  [defaults]
  inventory = inventory.yml
  private_key_file = /path/to/your/private/key.pem
  host_key_checking = False

Step 12: Run the Ansible Playbook

Execute the Ansible playbook to configure the EC2 instances: ansible-playbook deploy.yml -v

• Verbose Mode:

  • The v flag enables verbose output. Use vvv for even more detailed logs.

• Common Issues:

  • If you encounter SSH authentication errors, verify that the SSH key has the correct permissions (chmod 600 key.pem).

Step 13: Verify Deployment

After the playbook runs:

• Services Check:

  • SSH into the instances and verify that the Datalogz containers are running.

• Application Test:

  • Access the application URL to ensure it is functioning correctly.