impunity
Feedback in the group management background can not log on, I access, the entire interface blank, no prompts, open F12, found that the console prompts js, css and other static resources reported net::ERR_HTTP2_PROTOCOL_ERROR, the client can be downloaded to the server-side resources, the first time I encountered this, * go up!
What is net::ERR_HTTP2_PROTOCOL_ERROR about?
There are a lot of possible problems, including full disks, nginx configuration problems, request header problems, browser problems, cdn problems. Since this service has not been updated and no one has touched the configuration, it's unlikely to be a problem with nginx, the service, and also every time the js file is downloaded, it doesn't seem to be complete, even though the server is responding.
So the initial judgment is that there is a problem with the server, but there is no ssh configured, and the phone that logs into the AliCloud backend is also at work, so there is no way to see the server monitoring.
At this point, I found a few more times to refresh, using the browser's local cache, the home page can still be accessed, but the login function can not be responded to, if you fill out the wrong code, you can immediately return the error message. This means that MySQL writing is affected, and basically it can be concluded that the server's disk and memory problems.
The next day when I got to the office, I looked at the console monitor, and man, 99% of the disk was occupied.
Refreshed the admin backend and the response is very, very slow; clean up the space first and then verify if this is the problem!
settle (a dispute)
Lists all mounted filesystems and displays the total space, used space, free space, and utilization for each filesystem.
df -h
#Filesystem Size Used Avail Use% Mounted on
#udev 16G 0 16G 0% /dev
#tmpfs 3.1G 724K 3.1G 1% /run
#/dev/vda1 99G 94G 343M 100% /
#tmpfs 16G 0 16G 0% /dev/shm
#tmpfs 5.0M 0 5.0M 0% /run/lock
#tmpfs 16G 0 16G 0% /sys/fs/cgroup
#tmpfs 3.1G 0 3.1G 0% /run/user/0
At this point, be careful, if directly from the root directory recursive lookup, will consume a lot of CPU and IO resources, may lead to the server becomes very slow, or even temporarily unresponsive, so a layer to layer to
# /* Just replace it with the corresponding directory
du -sh /* | sort -rh | head -n 10
I finally found two directories, the nacos access log and the MySQL binlog, totaling 80 gigabytes of space, most of which is seven days old and can be processed.
For nacos logs, just delete the previous ones. For MySQL binlog files, it is recommended that you run the command from the console
# Delete the binlog from 7 days ago
PURGE BINARY LOGS BEFORE NOW() - INTERVAL 7 DAYS.
In the end, 40% of the space was disposed of
validate (a theory)
Multiple forced refreshes of the front-end page and faster response times for static resources and interfaces.
Next, looking at yesterday's logs, MySQL reported no errors except that it could not acquire a lock when writing data
Includes nginx logs, /var/log/syslog system logs, /var/log/mysql/ hints for insufficient space
2024-08-13T02:27:02.537301Z 1256492 [ERROR] [MY-000035] [Server] Disk is full writing './binlog.000556' (OS errno 28 - No space left on device). Waiting for someone to free space... Retry in 60 secs. Message reprinted in 600 secs.
2024-08-13T02:37:02.571302Z 1256492 [ERROR] [MY-000035] [Server] Disk is full writing './binlog.000556' (OS errno 28 - No space left on device). Waiting for someone to free space... Retry in 60 secs. Message reprinted in 600 secs.
2024-08-13T02:47:02.605275Z 1256492 [ERROR] [MY-000035] [Server] Disk is full writing './binlog.000556' (OS errno 28 - No space left on device). Waiting for someone to free space... Retry in 60 secs. Message reprinted in 600 secs.
2024-08-13T02:57:02.640862Z 1256492 [ERROR] [MY-000035] [Server] Disk is full writing './binlog.000556' (OS errno 28 - No space left on device). Waiting for someone to free space... Retry in 60 secs. Message reprinted in 600 secs.
2024-08-13T03:07:02.674508Z 1256492 [ERROR] [MY-000035] [Server] Disk is full writing './binlog.000556' (OS errno 28 - No space left on device). Waiting for someone to free space... Retry in 60 secs. Message reprinted in 600 secs.
2024-08-13T03:17:02.710238Z 1256492 [ERROR] [MY-000035] [Server] Disk is full writing './binlog.000556' (OS errno 28 - No space left on device). Waiting for someone to free space... Retry in 60 secs. Message reprinted in 600 secs.
There is no logging information to further confirm this other than the page is actually getting faster
Follow-up
Low disk space due to logging issues, need to enable timer or bring your own tool to clean up regularly
nacos access log
For nacos does not provide size and split configuration for access logs, only switches, and it is not recommended to turn them off online, so you need to write a shell script to add it to the timer
Write a Crontab for the production environment, put the script in the /etc/ directory, and give it executable permissions.
/bin/bash /bin/bash
logFile="/data/nacos/bin/logs/nacos_del_access.log"
# Keep the log for 14 days
date=`date -d "$date -14 day" +"%Y-%m-%d"`
# Adjustable location
delFilePath="/data/nacos/bin/logs/access_log.${date}.log"
if [ ! -f "${logFile}" ];then
echo 'access log file prints logs frequently. /etc// will delete the access log file regularly' >>${logFile}
fi
# Delete the log file if it exists
if [ -f "${delFilePath}" ];then
rm -rf ${delFilePath}
curDate=`date --date='0 days ago' "+%Y-%m-%d %H:%M:%S"`
echo '['${curDate}'] Delete file' ${delFilePath} >>${logFile}
fi
MySQL binlog
It is recommended to keep it for seven or fourteen days, using MySQL's own configuration.
# My default configuration of thirty days
# -- binlog_expire_logs_seconds 2592000
# SHOW VARIABLES LIKE 'binlog_expire_logs_seconds';
SET GLOBAL binlog_expire_logs_seconds = 604800;
show VARIABLES like 'expire_logs_days';
set global expire_logs_days = 7;
Previously these two commands were used, but they are deprecated in the new version of MySQL.
Permanently in effect,, need to reboot
[mysqld]
binlog_expire_logs_seconds = 604800
To add logs later, you need to manage and rotate log files, consider using logrotate
Reference:
- /questions/58215104/whats-the-neterr-http2-protocol-error-about
- /wtzvae/article/details/107212870
- https://blog./haibo0668/5486115
- /mr_wanter/article/details/112515814
Here's a block of anti-climbing code, I don't mind articles being crawled, but please give credit where it's coming from
("Author's homepage:/Go-Solo");;
("Original source address: /Go-Solo/p/18358836");;