g4ti0r

g4ti0r

In a cyberpunk world, dare to stand alone and stay true to yourself
HomeArchives

Build Openstack & Distributed Log Query Tool

#Project59
AI Translation
This post is translated from Chinese into English through AI.View Original
AI-generated summary
**Summary:** This project involves setting up an OpenStack environment with one controller node and four compute nodes, along with a distributed log query tool. The network structure includes specific IP addresses for each node. **Setup Steps:** 1. **Preparation:** Download necessary resources and configure virtual machines using VMware 16 Pro, setting up two network interfaces for each VM. 2. **Initial Configuration:** Install CentOS 7 on each VM, configure network settings, disable the firewall and security policies, and set up hostnames. 3. **Clone VMs:** Clone the initial VM to create compute nodes, adjusting IP addresses and connecting via SSH. 4. **YUM Repository Configuration:** Set up local YUM repositories on the controller and compute nodes for package management. 5. **OpenStack Installation:** Install OpenStack components on both the controller and compute nodes, configuring necessary files and installing main components like MySQL, Keystone, Glance, and Nova. 6. **Distributed Log Query Setup:** Install Go on both machines and configure the environment variables, followed by uploading the log query code from GitHub. The project concludes with the basic setup of OpenStack and the log querying tool.

Introduction#

A small project that sets up one control node and four compute nodes using OpenStack, and then queries logs using a distributed log querier.

Network Structure#

controllercomputecompute02compute03compute04
ens33192.168.10.10192.168.10.20192.168.10.30192.168.10.40192.168.10.50
ens34192.168.20.10192.168.20.20192.168.20.30192.168.20.40192.168.20.50
ens33 and 34 are network interfaces

OpenStack Setup#

Preliminary Preparation#

Resource Preparation#

Click to download the resource package
Cloud Disk

Virtual Machine Configuration#

I am using VMware 16 Pro version.
Select to change settings.
Add a new network card, both select Host-only mode, and fill in the corresponding IP addresses.

  • Network 1 Host-only 192.168.10.0
  • Network 2 Host-only 192.168.20.0

Create a New Virtual Machine#

Create a new virtual machine -> select centos7 iso -> allocate disk size 40G -> click finish.
Modify the virtual machine configuration.
Configure memory and processor, then add network adapters, selecting the two network cards just set up.

Initial Configuration of the First Virtual Machine#

After booting, set the installation location, select the 40G hard disk, done.
Configure username and password, restart.
After the restart is complete, enter the username root, enter the password (note that the password is not displayed when entering in Linux), and press Enter after completion.
Next, edit the network card settings.

vi /etc/sysconfig/network-scripts/ifcfg-ens33
# Press i to enter edit mode
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.10.10
NETMASK=255.255.255.0
# After modifying the above content, press esc->:wq to exit

Restart it.

systemctl restart network

Then modify the other network card.

vi /etc/sysconfig/network-scripts/ifcfg-ens34
IPADDR=192.168.20.10
NETMASK=255.255.255.0

Turn off the firewall.

systemctl stop firewalld
systemctl disable firewalld

Turn off the security policy.

setenforce 0
vi /etc/selinux/config
SELINUX=disable

Set hosts.

vi /etc/hosts
192.168.10.10 controller
192.168.10.20 compute
192.168.10.30 compute02
192.168.10.40 compute03
192.168.10.50 compute04

At this point, the basic steps are complete!

Initial Configuration of Other Virtual Machines#

Shut down the first machine, right-click to clone.
Just click next all the way.
I suggest cloning one machine first, and after configuring one compute node, clone the configured compute.

Configure Compute Nodes#

Similarly, first change the IP address and then connect via SSH.

vi /etc/sysconfig/network-scripts/ifcfg-ens33
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.10.20
NETMASK=255.255.255.0

systemctl restart network

vi /etc/sysconfig/network-scripts/ifcfg-ens34
IPADDR=192.168.20.20
NETMASK=255.255.255.0

Configure Repositories#

# In controller
hostnamectl set-hostname controller
# In compute
hostnamectl set-hostname compute

Upload two ISOs.

# In controller
Yum repository configuration
vi /etc/yum.repos.d/local.repo
[centos]
name=centos
enabled=1
gpgcheck=0
baseurl=file:///opt/centos
[iaas]
name=iaas
enabled=1
gpgcheck=0
baseurl=file:///opt/iaas/iaas-repo

mkdir /opt/centos
mkdir /opt/iaas
----------------------------------------------------
# In compute
Yum repository configuration
rm -rf /etc/yum.repos.d/*
vi /etc/yum.repos.d/ftp.repo
[centos]
name=centos
enabled=1
gpgcheck=0
baseurl=ftp://192.168.10.10/centos
[iaas]
name=iaas
enabled=1
gpgcheck=0
baseurl=ftp://192.168.10.10/iaas/iaas-repo

# In controller
mount -o loop CentOS-7-x86_64-DVD-2009.iso /mnt/
cp -rvf /mnt/* /opt/centos
umount /mnt/

mount -o loop openstack.iso /mnt/
cp -rvf /mnt/* /opt/iaas/
umount /mnt/

yum repolist
rm -rf /etc/yum.repos.d/C*
yum install vsftpd -y
vi /etc/vsftpd/vsftpd.conf
anon_root=/opt

systemctl restart vsftpd
systemctl enable vsftpd

Install OpenStack#

# In controller & compute
yum install openstack-iaas -y

After installation is complete, refresh the terminal.
Next, we need to configure the OpenStack files.
In the controller.

vi /etc/openstack/openrc.sh
ctrl+v+G+d
:%s/PASS=/PASS=000000/g
HOST_IP=192.168.10.10
# Controller Server hostname. example:controller
HOST_NAME=controller
# Compute Node Manager IP. example:x.x.x.x
HOST_IP_NODE=192.168.10.20,192.168.10.30,192.168.10.40,192.168.10.50
# Compute Node hostname. example:compute
HOST_NAME_NODE=compute
#--------------------Chrony Config-------------------##
# Controller network segment IP. example:x.x.0.0/16(x.x.x.0/24)
network_segment_IP=192.168.0.0/16
# External Network Interface. example:eth1
INTERFACE_NAME=ens34

# External Network The Physical Adapter. example:provider
Physical_NAME=provider

# First Vlan ID in VLAN RANGE for VLAN Network. example:101
minvlan=1

# Last Vlan ID in VLAN RANGE for VLAN Network. example:200
maxvlan=400

# Disk can be filled arbitrarily
# Cinder Block Disk. example:md126p3
BLOCK_DISK=sdb
# The NODE Object Disk for Swift. example:md126p4.
OBJECT_DISK=sdc
# The NODE IP for Swift Storage Network. example:x.x.x.x.
STORAGE_LOCAL_NET_IP=192.168.10.20

Other places can be written according to the prompt examples.
Then transfer this configuration file to the compute node.

scp /etc/openstack/openrc.sh [email protected]:/etc/openstack/
Enter password

Install Main Components#

# In controller & compute
iaas-pre-host.sh
Reconnect SSH
# In controller
iaas-install-mysql.sh
iaas-install-keystone.sh
iaas-install-glance.sh
iaas-install-placement.sh
iaas-install-nova-controller.sh
iaas-install-neutron-controller.sh
iaas-install-dashboard.sh
# In compute
iaas-install-nova-compute.sh
iaas-install-neutron-compute.sh

Distributed Log Query#

Install Go on both machines.

tar -zxf go1.17.2.linux-amd64.tar.gz -C /usr/local
vi /etc/profile
# Go environment variables
export GO111MODULE=on
export GOROOT=/usr/local/go
export GOPATH=/home/gopath
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

source /etc/profile

Code link
https://github.com/44maker/DistributedLogQuerier
Upload the code to both machines.
At this point, all configuration work is complete!
You can shut down the machines, then clone the compute node, and copy three more machines, install the configurations mentioned above, set the IP addresses, and connect via SSH.

Test#

# On all compute nodes
go run server.go
# In controller
go build client.go
./client [query] [log file name]
example:  ./client name

Unit Test#

# Generate test log files
go run generate_testfiles.go
# On all compute nodes
go run server.go
# In controller
go test -v client_test.go

Reference results
Results.png

Results2.png

Loading...
Ownership of this post data is guaranteed by blockchain and smart contracts to the creator alone.