Welcome to the Datalogz Infrastructure Deployment Guide. This document provides comprehensive instructions for deploying the Datalogz platform either on Amazon Web Services (AWS) or on bare metal servers. It is intended for DevOps engineers, system administrators, and IT professionals responsible for setting up and managing Datalogz infrastructure within your organization.
The guide includes step-by-step procedures, diagrams, and best practices to ensure a smooth deployment. All diagrams are dynamically generated using D2 and can be found in the diagrams directory.
Before you begin the deployment process, ensure that you have met the following prerequisites:
Access Permissions:
For AWS deployment, an AWS account with permissions to create and manage resources such as EC2 instances, RDS Instances, S3 buckets, and IAM roles.
For bare metal deployment, administrative access to the physical or virtual servers.
Network Requirements:
Open ports as required by Datalogz services (e.g., HTTP/HTTPS ports, database ports).
SSH access to servers for Ansible to run playbooks.
SSH Keys:
A valid SSH key pair for accessing the servers. The private key should be accessible from the machine where Ansible is run.
Deploying Datalogz on AWS involves provisioning infrastructure using Terraform and configuring instances using Ansible. This approach automates the deployment process, ensuring consistency and scalability.
Ensure that the following tools are installed on your local machine:
Terraform
Installation Guide: Install Terraform
Verify installation: terraform -v
AWS CLI
Installation Guide: Install AWS CLI
Verify installation: aws --version
Ansible
Installation Guide: Install Ansible
Verify installation: ansible --version
Set up your AWS credentials so that the AWS CLI and Terraform can authenticate with your AWS account.
Option 1: Using Environment Variables
Option 2: Using AWS CLI Configuration
Run the AWS configure command: aws configure
You will be prompted to enter your AWS Access Key ID, Secret Access Key, Default Region, and Output Format.
Test your AWS configuration by trying to list your current S3 buckets: aws s3 ls
Change your current directory to the AWS Terraform configuration folder: cd aws
Initialize the Terraform working directory. This command downloads the necessary provider plugins: terraform init
Inspect the variables.tf file to understand the variables required for deployment, such as AWS Region, S3 Bucket name, and EC2 Key Name.
Generate and review the execution plan to understand the resources that will be created: terraform plan
Review the Plan:
Ensure that the resources and configurations match your expectations.
Check for any unintended changes to existing infrastructure.
Apply the execution plan to deploy the infrastructure: terraform apply
Confirmation:
You will be prompted to confirm the deployment. Type yes to proceed.
Deployment Duration:
The deployment typically takes around 15-25 minutes.
After Terraform completes:
State File:
Ensure that the terraform.tfstate file has been updated.
Ansible uses a dynamic inventory plugin to find the IP address of the EC2 instance based on the instance Name tag.
Location: Change directory to the Ansible folder: cd ../ansible
Update the ansible.cfg file:
Private Key File:
Set the private_key_file parameter to the path of your SSH private key.
Execute the Ansible playbook to configure the EC2 instances: ansible-playbook deploy.yml -v
Verbose Mode:
The v flag enables verbose output. Use vvv for even more detailed logs.
Common Issues:
If you encounter SSH authentication errors, verify that the SSH key has the correct permissions (chmod 600 key.pem).
After the playbook runs:
Services Check:
SSH into the instances and verify that the Datalogz containers are running.
Application Test:
Access the application URL to ensure it is functioning correctly.