Load balancing with nginx
Nginx is a widely used reverse proxy server that enables efficient load balancing. The core role of load balancing is to distribute requests from clients to multiple backend servers, thus balancing the pressure on each server. With Nginx, we can implement various load balancing algorithms such as polling, IP hashing, etc.
vi /etc/nginx/ stickhttpbottom line
upstream app {
server 172.16.162.215:80;
server 172.16.162.216:80;
}
server {
listen 80;
listen [::]:80;
server_name localhost;
location / {
proxy_pass http://app;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
In the above configuration, the proxy_pass directive proxies all requests matching the root path to the pool of upstream servers defined in the back-end. proxy_set_header is used to set the request header information that is passed to the back-end server during the forwarding process to ensure that the client's real IP address and other information can be correctly passed to the back-end server to ensure that the back-end server can obtain complete information about the client when it logs and audits security. to ensure that the back-end server can obtain complete client information during logging and security auditing.
Two different hosts for load
[root@iZbp1fry44ac0aglan54biZ ~]# hostnamectl set-hostname program1
[root@iZbp1fry44ac0aglan54biZ ~]# bash
[root@program1 ~]# echo my id is 1 >
[root@program1 ~]# ls
[root@program1 ~]# python3 -m 80
Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) ...
[root@iZbp11cxs557bjnz6el8roZ ~]# hostnamectl set-hostname program2
[root@iZbp11cxs557bjnz6el8roZ ~]# bash
[root@program2 ~]# echo my id is 2 >
[root@program2 ~]# ls
[root@program2 ~]# python3 -m 80
Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) ...
Successful load balancing
[root@iZbp11cxs557bjnz6el8roZ nfs]# curl 47.97.245.17
my id is 1
[root@iZbp11cxs557bjnz6el8roZ nfs]# curl 47.97.245.17
my id is 2
[root@iZbp11cxs557bjnz6el8roZ nfs]# curl 47.97.245.17
my id is 1
[root@iZbp11cxs557bjnz6el8roZ nfs]# curl 47.97.245.17
my id is 2
[root@iZbp11cxs557bjnz6el8roZ nfs]#
The basic load balancing of nginx is configured here.
A real program has more than just a front-end page, it has a database, it has php, it has static pages and so on!
Infrastructure for simple programs
In modern distributed systems, it has become common practice to deploy applications using Docker containers, which package an application and its dependent environments into a single unit, ensuring consistent operation regardless of the operating system environment. The advantages of containerization are rapid deployment, good resource isolation, and support for lightweight horizontal scaling. No virtual operating system hardware resources are required
Configuring docker
# step 1: Install some of the necessary system tools
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
# Step 2: Adding Software Source Information
sudo yum-config-manager --add-repo /docker-ce/linux/centos/
# Step 3
sudo sed -i 's++/docker-ce+' /etc//
# Step 4: Update and installDocker-CE
sudo yum makecache fast
sudo yum -y install docker-ce
# Step 4: opensDockerservice
sudo service docker start
[root@iZbp1fry44ac0aglan54biZ ~]# vi /etc/docker/
{
"registry-mirrors": [
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
""
]
}
[root@iZbp1fry44ac0aglan54biZ ~]# systemctl restart docker
Configuration Database
docker run -d \
--name mariadb \
-e MYSQL_ROOT_PASSWORD=Supermao666 \
-e MYSQL_DATABASE=nextcloud \
-p 3306:3306 \
-v /opt/db:/var/lib/mysql \
mysql:5.7
4f8398c27b1b2104aae735d839f4f07cb39c44dd70370dad03275953d54dd53c
[root@program1 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4f8398c27b1b mysql:5.7 "…" 4 seconds ago Up 3 seconds 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mariadb
LAN Intranet connectivity testing
[root@program1 ~]# mysql -uroot -pSupermao666 -h172.16.162.215
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.7.44 MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| nextcloud |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.00 sec)
MySQL [(none)]> Bye
Configuring the nextcloud Program
docker networking
docker run -d \
--name nextcloud1 \
-e MYSQL_PASSWORD=Supermao666 \
-e MYSQL_DATABASE=nextcloud \
-v /opt/nextcloud1:/var/www/html \
-v /opt/nextdata:/var/www/html/data \
-p 80:80 \
--network bridge \
nextcloud:latest
[root@program1 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
69cb522262fa nextcloud:latest "/ apac…" About a minute ago Up About a minute 0.0.0.0:80->80/tcp, :::80->80/tcp nextcloud1
4f8398c27b1b mysql:5.7 "…" 6 minutes ago Up 6 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mariadb
#over the Internetipto access the initialization program
[root@program1 ~]# curl
121.43.116.224
[root@program1 ~]#
Upload the file successfully and complete the simple program deployment
The number of user visits becomes larger, the database, cloud hosting is gradually full
Use redis middleware to ease the pressure on mysql and use caching instead of querying the database every time.
Pull up the specification of cloud hosts or expand the number of cloud hosts horizontally for load balancing. Here horizontal expansion is used
Use shared nfs storage to ensure that all horizontally expanding cloud hosts have consistent configuration files and data. (If it is not consistent, it is not expanding, and all cloud hosts are working on their own)
nfs configuration
[root@iZbp11cxs557bjnz6el8roZ ~]# mkdir /nfs
[root@iZbp11cxs557bjnz6el8roZ /]# chmod -R 777 nfs/
[root@iZbp11cxs557bjnz6el8roZ /]# sudo vi /etc/exports
[root@iZbp11cxs557bjnz6el8roZ /]# cat /etc/exports
/nfs 172.16.162.0/24(rw,sync,no_root_squash,no_subtree_check)
## Internal LAN access sync: Synchronized write to disk for data security.
## no_root_squash: Allow remote root users to have root privileges on files shared by NFS (can be adjusted according to security requirements).
## no_subtree_check: Disables subtree checking, which improves performance.
[root@iZbp11cxs557bjnz6el8roZ /]# sudo systemctl start nfs
[root@iZbp11cxs557bjnz6el8roZ /]# sudo systemctl enable nfs
Created symlink from /etc/systemd/system// to /usr/lib/systemd/system/.
[root@iZbp11cxs557bjnz6el8roZ /]#
[root@iZbp11cxs557bjnz6el8roZ /]# sudo vi /etc/exports
[root@iZbp11cxs557bjnz6el8roZ /]# sudo systemctl start nfs
[root@iZbp11cxs557bjnz6el8roZ /]# sudo systemctl enable nfs
Created symlink from /etc/systemd/system// to /usr/lib/systemd/system/.
[root@iZbp11cxs557bjnz6el8roZ nfs]# touch flag
[root@iZbp11cxs557bjnz6el8roZ nfs]# ls
flag
[root@iZbp11cxs557bjnz6el8roZ nfs]# pwd
/nfs
[root@iZbp11cxs557bjnz6el8roZ nfs]#
nfs verification test
[root@program1 ~]# cd /mnt/
[root@program1 mnt]# ls
[root@program1 mnt]# sudo mount -t nfs 172.16.162.217:/nfs /mnt/
[root@program1 mnt]# ls
[root@program1 mnt]# cd ..
[root@program1 /]# cd -
/mnt
[root@program1 mnt]# ls
flag
[root@program1 mnt]# df -hT
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs tmpfs 1.8G 580K 1.8G 1% /run
tmpfs tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/vda1 ext4 40G 6.1G 32G 17% /
tmpfs tmpfs 365M 0 365M 0% /run/user/0
overlay overlay 40G 6.1G 32G 17% /var/lib/docker/overlay2/e3c8da8d16909c43dd674566d7effbd34ec017300453cedacc82d6e72b809056/merged
172.16.162.217:/nfs nfs4 40G 2.6G 35G 7% /mnt
[root@program1 mnt]#
Delete the previous nextcloud Start nextcloud again
Put nextcloud's data disk and configuration files on the mnt shared disk.
[root@program1 mnt]# docker run -d \
--name nextcloud21 \
-e MYSQL_PASSWORD=Supermao666 \
-e MYSQL_DATABASE=nextcloud \
-v /mnt/nextcloud1:/var/www/html \
-v /mnt/nextdata:/var/www/html/data \
-p 80:80 \
--network bridge \
nextcloud:latest
ec3e2f63df4a71383be910d97a021a402ae9477eb7acefdfe9d166f61acc2c5e
[root@program1 mnt]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ec3e2f63df4a nextcloud:latest "/ apac…" 3 seconds ago Up 2 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp nextcloud21
4f8398c27b1b mysql:5.7 "…" 28 minutes ago Up 28 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mariadb
[root@program1 mnt]# du -sh
5.9M .
[root@program1 mnt]# docker logs ec3e2f63df4a
Initializing nextcloud 29.0.6.1 ...
[root@program1 mnt]#
[root@program1 mnt]# du -sh
29M .
[root@program1 mnt]#
Waiting to write to a shared disk
## Configuration complete
[root@program1 mnt]# du -sh
684M .
[root@program1 mnt]# docker logs ec3e2f63df4a
Initializing nextcloud 29.0.6.1 ...
New nextcloud instance
Next step: Access your instance to finish the web-based installation!
Hint: You can specify NEXTCLOUD_ADMIN_USER and NEXTCLOUD_ADMIN_PASSWORD and the database variables _prior to first launch_ to fully automate initial installation.
Initializing finished
=> Searching for scripts (*.sh) to run, located in the folder: //before-starting
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.3. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.3. Set the 'ServerName' directive globally to suppress this message
[Tue Sep 10 14:47:18.756670 2024] [mpm_prefork:notice] [pid 1:tid 1] AH00163: Apache/2.4.62 (Debian) PHP/8.2.23 configured -- resuming normal operations
[Tue Sep 10 14:47:18.756745 2024] [core:notice] [pid 1:tid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
27.17.161.6 - - [10/Sep/2024:14:47:37 +0000] "POST /apps/text/session/20/sync HTTP/1.1" 404 655 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Edg/128.0.0.0"
27.17.161.6 - - [10/Sep/2024:14:47:37 +0000] "POST /apps/text/session/20/push HTTP/1.1" 404 655 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Edg/128.0.0.0"
27.17.161.6 - - [10/Sep/2024:14:47:37 +0000] "GET /apps/files/api/v1/stats HTTP/1.1" 404 655 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Edg/128.0.0.0"
[root@program1 mnt]#
[root@program1 mnt]# curl
121.43.116.224
Delete the previous database
[root@program1 mnt]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ec3e2f63df4a nextcloud:latest "/ apac…" 9 minutes ago Up 9 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp nextcloud21
4f8398c27b1b mysql:5.7 "…" 38 minutes ago Up 38 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mariadb
[root@program1 mnt]# docker rm -f 4f8398c27b1b
4f8398c27b1b
[root@program1 mnt]# rm -rf /opt/db/
[root@program1 mnt]#
The previous database has been written tosupermao
Restarting the database
[root@program1 mnt]# docker run -d --name mariadb -e MYSQL_ROOT_PASSWORD=Supermao666 -e MYSQL_DATABASE=nextcloud -p 3306:3306 -v /opt/db:/var/lib/mysql mysql:5.7
Once again, go to the public ip for access and then initialize the
Cloud hosting program2 configuration redis
In highly concurrent Web systems, frequent database queries can lead to performance bottlenecks. To solve this problem, Redis can be introduced as a caching system to reduce direct access to the database.Redis is a memory-based key-value database that provides very fast access to data, and is especially suitable for caching data that is frequently queried.
[root@program2 ~]# docker run -d --name redis-container -p 6379:6379 redis
545f756f22f2c768c44891aa256708699ec199ad5a4ae0788727c506b4111985
[root@program2 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
545f756f22f2 redis "…" 5 seconds ago Up 4 seconds 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp redis-container
## beta (software)rediskey-value database
[root@program2 ~]# redis-cli -h 172.16.162.216 -p 6379
172.16.162.216:6379> set mykey "Hello Redis"
OK
172.16.162.216:6379> get mykey
"Hello Redis"
172.16.162.216:6379>
Interfacing nextcloud with redis
redis exists on top of program2 with ip 172.16.162.216
First monitor redis
[root@program1 ~]# redis-cli -h 172.16.162.216 MONITOR
OK
Update the nextcloud configuration file, which will be automatically synchronized into the docker container
Source files
<?php
$CONFIG = array (
'' => '/', '' => '\\OC\\\\Memory', '/', '/')
'' => '\\OC\\\Memcache\\\APCu',
'apps_paths' =>
array (
0 =>
array (
'path' => '/var/www/html/apps', 'url' => array (
'url' => '/apps',
'writable' => false,
),
1 =>
array (
'path' => '/var/www/html/custom_apps',
'url' => '/custom_apps',
'writable' => true,
),
),
After changing
<?php
$CONFIG = array (
'' => '/',
'' => '\\OC\\\\Memcache\\\APCu',
'' => '\OC\Memcache\Redis',
'' => 'true', 'redis'.
'redis' => array(
'host' => '172.16.162.216', 'port' => 'redis' => array(
'port' => '6379'.
),
'apps_paths' =>
array (
0 =>
array (
'path' => '/var/www/html/apps',
'url' => '/apps',
'writable' => false,
),
1 =>
array (
'path' => '/var/www/html/custom_apps',
'url' => '/custom_apps',
'writable' => true,
),
),
Improve security by only allowing access to nextcloud instances from 47.97.245.17. Prevent hijacking of DNS records or spoofing of requests
'trusted_domains' =>
array (
0 => '47.97.245.17', #load-balanced public address for first part
),
If redis keeps refreshing then it is hitting the cache and the redis configuration is successful
[root@program1 ~]# redis-cli -h 172.16.162.216 MONITOR
OK. 1725980850.546
1725980850.548579 [0 172.16.162.215:51728] "TTL" "b5f1517db060bd92ddc8106a7d81bbcf/lockfiles/b947cec9e58799d4586628341ec933a4"
1725980850.548951 [0 172.16.162.215:51728] "INCRBY" "b5f1517db060bd92ddc8106a7d81bbcf/lockfiles/b947cec9e58799d4586628341ec933a4" "1"
1725980850.549195 [0 172.16.162.215:51728] "EXPIRE" "b5f1517db060bd92ddc8106a7d81bbcf/lockfiles/b947cec9e58799d4586628341ec933a4" "3600 "
1725980850.550609 [0 172.16.162.215:51728] "TTL" "b5f1517db060bd92ddc8106a7d81bbcf/lockfiles/c5e0b4791fd3a7d14e54b84edeb73034"
1725980850.550870 [0 172.16.162.215:51728] "INCRBY" "b5f1517db060bd92ddc8106a7d81bbcf/lockfiles/c5e0b4791fd3a7d14e54b84edeb73034" "1"
1725980850.551147 [0 172.16.162.215:51728] "EXPIRE" "b5f1517db060bd92ddc8106a7d81bbcf/lockfiles/c5e0b4791fd3a7d14e54b84edeb73034" "3600 "
1725980850.552536 [0 172.16.162.215:51728] "TTL" "b5f1517db060bd92dd
program2 starts another nextcloud.
Mounted Data Disk
[root@program2 ~]# sudo mount -t nfs 172.16.162.217:/nfs /mnt/
[root@program2 ~]# docker run -d \
--name nextcloud21 \
-e MYSQL_PASSWORD=Supermao666 \
-e MYSQL_DATABASE=nextcloud \
-v /mnt/nextcloud1:/var/www/html \
-v /mnt/nextdata:/var/www/html/data \
-p 80:80 \
--network bridge \
nextcloud:latest
b93e787b84b47394e88398f64325540e2c54370e5a4b4f0d20aeb343f8b0ab84
[root@program2 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b93e787b84b4 nextcloud:latest "/ apac…" 3 seconds ago Up 2 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp nextcloud21
545f756f22f2 redis "…" 29 minutes ago Up 29 minutes 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp redis-container
[root@program2 ~]#
High Availability Testing
downlose (value, weight etc)program1(used form a nominal expression) nextcloud
nextcloudStill available，在高负载(used form a nominal expression)情况下program2It will also be polled to share the traffic pressure
[root@program1 opt]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a626845e9e90 nextcloud:latest "/ apac…" 7 seconds ago Up 6 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp nextcloud21
84fe8c4fc4bf mysql:5.7 "…" 3 minutes ago Up 3 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp mariadb
[root@program1 opt]# docker stop a62
Trying to downgrade the service on any node, the service is still available
Finally, only the elastic ip for load balancing is left
Architecture can still be optimized
When it comes to larger data volumes, you can then expand horizontally. Cloud vendors' elastic scaling groups are also a great option
nfs can become a performance bottleneck with a lot of writes. For heavy traffic consider ceph or s3.
nginx-bal will just go down when it encounters very high traffic. consider doing disaster recovery and triage