Deployment Guides (Others)

(1)

Deployment Guides (Others)

Issue 02

(2)

No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.

All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

(3)

1 Elasticsearch Deployment Guide (CentOS 7.6 & openEuler 20.03)...1

1.1 Introduction... 1

1.2 Environment Requirements... 2

1.3 Configuring the Deployment Environment... 3

1.4 Deploying Elasticsearch...4

2 Redis Deployment Guide (CentOS 7.6 & openEuler 20.03)...7

2.1 Introduction... 7

2.2 Environment Requirements... 8

2.3 Configuring the Deployment Environment... 9

2.4 Deploying Redis... 9

(4)

1

Elasticsearch Deployment Guide

(CentOS 7.6 & openEuler 20.03)

1.1 Introduction

1.2 Environment Requirements

1.3 Configuring the Deployment Environment 1.4 Deploying Elasticsearch

1.1 Introduction

Elasticsearch Overview

This document describes the Elasticsearch deployment procedure and does not include the source code compilation procedure.

All programs required in this document are downloaded from the official websites. Most of these programs are compiled based on the x86 platform and may contain modules that are implemented in platform-dependent languages (such as C/C++). Therefore, incompatibility issues may occur if these programs are directly run on TaiShan servers. To resolve the problem, download and compile the source code and then deploy the programs. The deployment procedure is the same regardless of the program compilation platform.

Recommended Versions

Softwar

e Version How to Obtain

(5)

Softwar

e Version How to Obtain

ElasticSe

arch 6.7.1 Download the software package of the required version fromthe official website:

https://artifacts.elastic.co/downloads/elasticsearch/ elasticsearch-6.7.1.tar.gz

1.2 Environment Requirements

Hardware Requirements

Minimum configuration: any CPU, one DIMM of any capacity, and one drive of any capacity

The configuration varies depending on the application scenario.

OS Requirements

CentOS 7.4 to 7.6. NO TE

This document uses CentOS 7.6 as an example to describe how to deploy an Elasticsearch cluster.

Cluster Environment

For example, the cluster has nodes 1 to 4. Table 1-1 lists the data plan of each node.

Table 1-1 Cluster data plan

Node IP Address Number of Drives OS & JDK

Node 1 IPaddress1 System drive: 1 x 4 TB HDD

Data drives: 12 x 4 TB HDDs

CentOS 7.6 & OpenJDK jdk8u252-b09

Node 2 IPaddress2 Node 3 IPaddress3 Node 4 IPaddress4

Software Planning

(6)

Table 1-2 Software plan Node Service Node 1 -Node 2 Elasticsearch*2 Node 3 Elasticsearch*2 Node 4 Elasticsearch*2

1.3 Configuring the Deployment Environment

Step 1 Log in to nodes 1 to 4 in sequence and change the host names to server1, agent1, agent2 and agent3.

hostnamectl set-hostname host_name --static

Step 2 Log in to all nodes and modify the /etc/hosts file.

Add the mapping between the IP addresses and host names of nodes in the cluster to the hosts file.

IPaddress1 server1 IPaddress2 agent1 IPaddress3 agent2 IPaddress4 agent3

Step 3 Log in to each node and disable the firewall.

systemctl stop firewalld.service systemctl disable firewalld.service

Step 4 Log in to each node and configure SSH password-free login.

1. Generate a key and press Enter if any message prompted.

ssh-keygen -t rsa

2. Configure SSH password-free login on each node (including password-free login for the local node):

ssh-copy-id -i ~/.ssh/id_rsa.pub root@node_IP_address

Step 5 Log in to each node and install OpenJDK.

1. Install OpenJDK. ARM:

wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/ OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gz

tar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gz -C /usr/local

x86:

wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/ OpenJDK8U-jdk_x64_linux_hotspot_8u252b09.tar.gz

tar -zxf OpenJDK8U-jdk_x64_linux_hotspot_8u252b09.tar.gz -C /usr/local

2. Add environment variables.

vim /etc/profile

export JAVA_HOME=/usr/local/jdk8u252-b09 export PATH=$JAVA_HOME/bin:$PATH

3. Make the environment variables take effect.

(7)

4. Check whether the OpenJDK is successfully installed.

java -version

The installation is successful if information similar to the following is displayed:

----End

1.4 Deploying Elasticsearch

Obtaining Elasticsearch

Step 1 Download the Elasticsearch package from the following website:

https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.7.1.tar.gz

Step 2 Place elasticsearch-6.7.1.tar.gz in the /usr/local directory on agent1 and

decompress it.

mv elasticsearch-6.7.1.tar.gz /usr/local tar -zxvf elasticsearch-6.7.1.tar.gz

Step 3 Create a soft link for subsequent version updates.

ln -s elasticsearch-6.7.1 elasticsearch

----End

Adding Elasticsearch to Environment Variables

Step 1 Run the following command to edit the /etc/profile file:

Step 2 Add the following environment variables to the end of the file:

export ES_HOME=/usr/local/elasticsearch export PATH=$ES_HOME/bin:$PATH

Step 3 Make the environment variables take effect.

source /etc/profile

----End

Copying the Configuration Directories

cd /usr/local/elasticsearch cp -r config config0 cp -r config config1

NO TE

(8)

Modifying the Elasticsearch Configuration Files

NO TE

All Elasticsearch configuration files are stored in the $ES_HOME/config{0,1} directory. Step 1 Modify the config0/elasticsearch.yml file.

cluster.name: my-application # Cluster name, which can be changed.

node.name: node-{ip}-9200 # Instance name. Ensure that each instance name is unique in the cluster. path.data: /data/data1/elastic,/data/data2/elastic,/data/data3/elastic,/data/data4/elastic,/data/data5/ elastic,/data/data6 # Data directory of the ESMaster instance.

path.logs: /var/log/elastic-9200 # Log directory of the ESMaster instance. bootstrap.memory_lock: false

network.host: $agent1 # You can enter the IP address of agent1. http.port: 9200 # Port of the ESMaster instance.

transport.tcp.port: 9300 discovery.zen.ping.unicast.hosts: ["agent1:9300","agent1:9301","agent2:9300","agent2:9301", "agent3:9300","agent3:9301"] discovery.zen.minimum_master_nodes: 1 node.max_local_storage_nodes: 5 node.master: true node.data: true xpack.ml.enabled: false

Step 2 Modify the config1/elasticsearch.yml file.

cluster.name: my-application # Cluster name, which can be changed.

node.name: node-{ip}-9201 # Instance name. Ensure that each instance name is unique in the cluster. path.data: /data/data7/elastic,/data/data8/elastic,/data/data9/elastic,/data/data10/elastic,/data/data11/ elastic,/data/data12/elastic # Data directory of the ESMaster instance.

path.logs: /var/log/elastic-9201 # Log directory of the ESMaster instance. bootstrap.memory_lock: false

network.host: agent1 # You can enter the IP address of agent1. http.port: 9201 # Port of the ESMaster instance.

transport.tcp.port: 9301 discovery.zen.ping.unicast.hosts: ["agent1:9300","agent1:9301","agent2:9300","agent2:9301", "agent3:9300","agent3:9301"] discovery.zen.minimum_master_nodes: 1 node.max_local_storage_nodes: 5 node.master: true node.data: true xpack.ml.enabled: false

Step 3 Modify the config{0,1}/jvm.options file. The memory of the ESMaster instance

does not need to be too large. Set it based on site requirements.

-Xms31g -Xmx31g

Step 4 Modify the bin/elasticsearch-env file and add the following configuration to the

beginning of the script:

export JAVA_HOME=/usr/local/jdk8u252-b09 export PATH=${JAVA_HOME}/bin:${PATH}

----End

Synchronizing the Configuration to Other Nodes

Step 1 Synchronize the configuration.

1. Copy elasticsearch-6.7.1 to the /usr/local directory of agent2 and agent3.

scp -r /usr/local/elasticsearch-6.7.1 root@agent2:/usr/local scp -r /usr/local/elasticsearch-6.7.1 root@agent3:/usr/local

2. Log in to agent2 and agent3 and create a soft link for elasticsearch-6.7.1.

cd /usr/local

(9)

Step 2 Create data and log directories.

1. Create the ESMaster data directory on agent1 to agent3.

mkdir -p /usr/local/elasticsearch/esmaster

2. Create the ESNode1 data directory on agent1 to agent3.

mkdir -p /data/data{1,2,3,4,5,6,7,8,9,10,11,12}/elastic

3. Create the ESMaster and ESNode1 data directories on agent1 to agent3.

mkdir -p /var/log/elastic-{9200,9201}

Step 3 Modify node parameters.

1. On agent2, modify config0/elasticsearch.yml.

node.name: node-{ip}-9200 # Instance name. Ensure that each instance name is unique in the cluster. network.host: agent2 # You can enter the IP address of agent2.

2. Modify config1/elasticsearch.yml.

3. On agent3, modify config0/elasticsearch.yml.

4. Modify config1/elasticsearch.yml.

----End

Starting and Stopping Elasticsearch

Step 1 Create an Elasticsearch user.

1. Log in to agent1 to agent3 and create an Elasticsearch user. By default, Elasticsearch cannot be started by the root user.

useradd elasticsearch

2. Change the owner of the Elasticsearch directories.

chown -R elasticsearch:elasticsearch /usr/local/elasticsearch

chown -R elasticsearch:elasticsearch /data/data{1,2,3,4,5,6,7,8,9,10,11,12}/elastic chown -R elasticsearch:elasticsearch /var/log/elastic-{9200,9201}

Step 2 On agent1 to agent3, modify /etc/security/limits.conf.

elasticsearch soft nofile 190000 elasticsearch hard nofile 190000 elasticsearch soft nproc 190000 elasticsearch hard nproc 190000

Step 3 On agent1 to agent3, start the Elasticsearch cluster in the ${ES_HOME} directory.

sysctl -w vm.max_map_count=655360

sudo -u elasticsearch ES_PATH_CONF=/usr/local/elasticsearch/config0/ bin/elasticsearch -p /usr/local/ elasticsearch/es-{ip}-9200.pid -d

sudo -u elasticsearch ES_PATH_CONF=/usr/local/elasticsearch/config1/ bin/elasticsearch -p /usr/local/ elasticsearch/es-{ip}-9201.pid -d

NO TE

If your permission is insufficient, add the permission.

chmod -R 777 /usr/local/elasticsearch

chmod -R 777 /data/data{1,2,3,4,5,6,7,8,9,10,11,12}/elastic

Step 4 Kill the Elasticsearch process to stop the Elasticsearch cluster.

kill -9 $(ps aux | grep Elasticsearch | grep -v 'grep --color=auto Elasticsearch' | awk '{print \\\$2})

(10)

2

Redis Deployment Guide (CentOS 7.6 &

openEuler 20.03)

2.1 Introduction

2.2 Environment Requirements

2.3 Configuring the Deployment Environment 2.4 Deploying Redis

2.1 Introduction

Redis Overview

This document describes the Redis deployment procedure and does not include the source code compilation procedure.

All programs required in this document are downloaded from the official websites. Most of these programs are compiled based on the x86 platform and may contain modules that are implemented in platform-dependent languages (such as C/C++). Therefore, incompatibility issues may occur if these programs are directly run on TaiShan servers. To resolve the problem, you need to download and compile the source code and then deploy the programs. The deployment procedure is the same regardless of the program compilation platform.

Recommended Versions

Software Version How to Obtain

Redis 5.0.4 Download the software package of the required version from the official website:

(11)

2.2 Environment Requirements

Hardware Requirements

Minimum configuration: any CPU, one DIMM of any capacity, and one drive of any capacity

The configuration varies depending on the application scenario.

OS Requirements

CentOS 7.4 to 7.6 NO TE

This document uses CentOS 7.6 as an example to describe how to deploy a Redis cluster.

Cluster Environment

For example, the cluster has nodes 1 to 4. Table 2-1 lists the data plan of each node.

Table 2-1 Cluster data plan

Node IP Address Number of Drives OS

Node 1 IPaddress1 System drive: 1 x 4 TB HDD Data drives: 12 x 4 TB HDDs CentOS 7.6 Node 2 IPaddress2 Node 3 IPaddress3 Node 4 IPaddress4

Software Planning

Table 2-2 lists the software plan of each node in the cluster.

Table 2-2 Software plan

Node Service

(12)

2.3 Configuring the Deployment Environment

Step 1 Log in to nodes 1 to 4 in sequence and change the host names to server1, agent1, agent2 and agent3.

hostnamectl set-hostname Host name --static

Step 2 Log in to all nodes and modify the /etc/hosts file.

Add the mapping between the IP addresses and host names of nodes in the cluster to the hosts file.

IPaddress1 server1 IPaddress2 agent1 IPaddress3 agent2 IPaddress4 agent3

Step 3 Log in to each node and disable the firewall.

systemctl stop firewalld.service systemctl disable firewalld.service

Step 4 Log in to each node and configure SSH password-free login.

1. Generate a key and press Enter if any message is prompted.

ssh-keygen -t rsa

2. Run the following command on each node to configure SSH password-free login (including password-free login for the local node):

ssh-copy-id -i ~/.ssh/id_rsa.pub root@IP address of each node

----End

2.4 Deploying Redis

Preparations

NO TE

Perform the operations on agent1. Step 1 Install Ruby.

yum install ruby ruby-devel rubygems rpm-build

Step 2 Download redis-3.0.5.gem and upload it to the /usr/local directory of agent1.

Download link: https://rubygems.org/gems/redis/versions

gem install redis-3.0.5.gem

Step 3 (Optional) Modify the /etc/sysctl.conf file.

1. Add the following content to the /etc/sysctl.conf file on agent1 to agent3:

vm.overcommit_memory=1

(13)

NO TE

If the system displays a message indicating that the memory is insufficient, you can perform this operation.

----End

Installing Redis

Step 1 Save redis-5.0.4.tar.gz to the /usr/local directory on agent1 and decompress it.

tar -zxvf redis-5.0.4.tar.gz

Step 2 Go to the folder that is generated after the package is decompressed, and install

Redis.

cd /usr/local/redis-5.0.4 make -j8

make PREFIX=/usr/local/redis-5.0.4 install

----End

Copying Redis to Other Nodes

Synchronize Redis to other nodes.

scp -r /usr/local/redis-5.0.4 root@agent2:/usr/local scp -r /usr/local/redis-5.0.4 root@agent3:/usr/local

Creating a Redis Soft Link and Adding Environment Variables

Step 1 Create a Redis soft link on agent1 to agent3.

cd /usr/local ln -s redis-5.0.4 redis

Step 2 Open the configuration file.

Step 3 Add Redis to environment variables.

export REDIS_HOME=/usr/local/redis export PATH=$REDIS_HOME/bin:$PATH

Step 4 Make the environment variables take effect.

source /etc/profile

----End

Deploying a Cluster

NO TE

In this example, six instances (three primary and three secondary) are deployed. By default, Redis uses the three instances with the smallest port numbers as the primary instances. Therefore, [7001, 7004], [7002, 7005], and [7003, 7006] are deployed on agent1 to agent3. Step 1 Create run directories.

1. On agent1:

mkdir -p /usr/local/redis-server/{7001,7004}

2. On agent2:

(14)

3. On agent3:

mkdir -p /usr/local/redis-server/{7003,7006}

Step 2 Copy the default configuration file.

1. On agent1:

cd /usr/local/redis-5.0.4

cp redis.conf /usr/local/redis-server/redis-default.conf

scp redis.conf root@agent2:/usr/local/redis-server/redis-default.conf scp redis.conf root@agent3:/usr/local/redis-server/redis-default.conf

2. Modify the default configuration file on agent1 to agent3.

vim redis-default.conf

Modify the three values. ${ip} is the IP address of the agent node, and $

{port} is the port of the primary node.

bind ${ip} port ${port}

pidfile /var/run/redis_${port}.pid

Step 3 Create an independent configuration file.

1. On agent1, open the /usr/local/redis-server/7001/redis.conf file.

vim /usr/local/redis-server/7001/redis.conf

2. Add the following content:

# Include the default configuration.

include /usr/local/redis-server/redis-default.conf pidfile /var/run/redis-7001.pid

port 7001

# Enable the cluster mode. cluster-enabled yes

cluster-config-file /usr/local/redis-server/nodes-7001.conf cluster-node-timeout 15000

Step 4 Copy the configuration.

On agent1: cd /usr/local/redis-server/7001/ cp redis.conf /usr/local/redis-server/7004/ scp redis.conf root@agent2:/usr/local/redis-server/7002 scp redis.conf root@agent2:/usr/local/redis-server/7005 scp redis.conf root@agent3:/usr/local/redis-server/7003 scp redis.conf root@agent3:/usr/local/redis-server/7006

Step 5 Modify the port in the configuration file.

1. On agent1: sed -i "s/7001/7004/g" /usr/local/redis-server/7004/redis.conf 2. On agent2: sed -i "s/7001/7002/g" /usr/local/redis-server/7002/redis.conf sed -i "s/7001/7005/g" /usr/local/redis-server/7005/redis.conf 3. On agent3: sed -i "s/7001/7003/g" /usr/local/redis-server/7003/redis.conf sed -i "s/7001/7006/g" /usr/local/redis-server/7006/redis.conf

Step 6 Start the Redis instances.

NO TE

Replace the ports in the following commands with the actual ones. 1. Start the Redis instance on agent1.

/usr/local/redis/bin/redis-server /usr/local/redis-server/7001/redis.conf & /usr/local/redis/bin/redis-server /usr/local/redis-server/7004/redis.conf &

(15)

3. Start the Redis instance on agent3.

Step 7 Enable the cluster mechanism. A Redis cluster has been set up.

NO TE

Run the following command during the first startup. The redis-trib.rb script is used to implement handshakes between cluster instances, slot allocation, and primary/secondary mapping.

/usr/local/redis-5.0.4/src/redis-cli --cluster create --cluster-replicas 1 agent1:7001 agent2:7002 agent3:7003 agent1:7004 agent2:7005 agent3.116:7006

----End

Connecting to the Cluster

Connect to any instance. The -c parameter must be added. Otherwise, automatic switchover between the instances cannot be performed. The -h parameter is optional.

/usr/local/redis/bin/redis-cli -h agent1 -p 7001 -c

Stopping the Cluster

NO TE

Replace the ports in the following commands with the actual ones. 1. Stop the Redis instance on agent1.

/usr/local/redis/bin/redis-cli -h agent1 -p 7001 shutdown /usr/local/redis/bin/redis-cli -h agent1 -p 7004 shutdown

2. Stop the Redis instance on agent2.

/usr/local/redis/bin/redis-cli -h agent2 -p 7002 shutdown /usr/local/redis/bin/redis-cli -h agent2 -p 7005 shutdown

3. Stop the Redis instance on agent3.

(16)

A

Change History

Date Description

2021-07-13 This issue is the second official release.

Deployment Guides (Others)