Linux CentOS 7 Installation of Kafka 2.8.2 - Standalone & JDK 11 & Switching JDK Versions

Kafka uses Java 11 by default starting with 2.6.0 , and Java 8 is no longer supported starting with 3.0.0, see details:/downloads

• Producer: The message producer, the client that sends messages to the kafka broker.
• Consumer: the message consumer, the client that fetches messages from the kafka broker.
• ConsumerGroup: consumer group, composed of multiple consumers. Each consumer in a consumer group is responsible for consuming data in different partitions, a partition can only be consumed by a consumer in the group; consumer groups do not affect each other. All consumers belong to a consumer group, that is, the consumer group is logically a subscriber.
  Different groups can consume the same message and it can only be consumed by one consumer within the consuming group
• Broker :A kafka server is a broker. a cluster consists of multiple brokers. A cluster consists of multiple brokers. A broker can hold multiple topics.
• Topic: can be interpreted as a queue, producers and consumers are oriented to a topic.
• Partition: in order to achieve scalability, a very large topic can be distributed to multiple brokers (i.e., servers), a topic can be divided into multiple partitions, each partition is an ordered queue.
• Replica: copy, in order to ensure that a node failure in the cluster, the node on the partition data is not lost, and Kafka can still continue to work, kafka provides a replica mechanism, a topic for each partition has a number of replicas, a Leader and a number of follower
• leader: the "master" of each partition with multiple replicas, the object to which the producer sends data, and the object to which the consumer consumes data are all leaders.
• follower: A "follower" of multiple replicas per partition that synchronizes data from the leader in real time to keep it in sync with the leader's data. when the leader fails, a follower becomes the new follower.

Installing JDK 11

[root@kafka1host ~]# mkdir /usr/local/java
# Unpack the JDK
[root@kafka1host ~]# tar -zxvf jdk-11.0.17_linux-x64_bin. -C /usr/local/java
# Note: Since jdk1.8 does not have jre, you need to create a jre folder in the jdk root directory with the following command, or else you will get a missing jre error when running tomcat.
. /bin/jlink --module-path jmods --add-modules --output jre

# Configure environment variables
[root@localhost~]# vi /etc/profile
export JAVA_HOME=/usr/local/java/jdk-11.0.17
export JRE_HOME=${JAVA_HOME}
export PATH=$PATH:${JAVA_HOME}/bin
export CLASSPATH=. /:${JAVA_HOME}/lib:${JAVA_HOME}/lib

# Make environment changes take effect
[root@localhost~]# source /etc/profile
# At this point java -version is still showing the old version [1. delete old version, 2. switch Java version] [root@kafka1host~]# source /etc/profile
[root@kafka1host kafka]# java -version
openjdk version "1.8.0_102"
OpenJDK Runtime Environment (build 1.8.0_102-b14)
OpenJDK 64-Bit Server VM (build 25.102-b14, mixed mode)
[root@kafka1host kafka]#

Switch the JAVA version, (If you need to uninstall an older version, click here)

# View installed Java versions and their paths. /etc/alternatives/java is a symbolic link to the current Java version.
[root@kafka1host ~]# ls -l /usr/bin/java
lrwxrwxrwx. 1 root root 22 Apr 27 2021 /usr/bin/java -> /etc/alternatives/java
# View the path to the executable for the current Java version: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64/jre/bin/java is the path to the executable for the current Java version.
[root@kafka1host ~]# ls -l /etc/alternatives/java
lrwxrwxrwx. 1 root root 71 Apr 27 2021 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64/jre/bin/java
# alternatives registering Java version information
[root@kafka1host ~]# alternatives --install /usr/bin/java java /usr/local/java/jdk-11.0.17/bin/java 1
# alternatives --install <link> <name> <path> <priority> # install means install.
# install means install
# link is a symbolic link
# name is an identifier
# path is the path to the executable
# priority is the priority
# Select the Java configuration version
[ithoth@kafka1host ~]# sudo alternatives --config java

There are 3 programs which provide 'java'.

  Selection Command
-----------------------------------------------
*+ 1 java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64/jre/bin/java)
   2 java-1.7.0-openjdk.x86_64 (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111-2.6.7.8.el7.x86_64/jre/bin/java)
   3 /usr/local/java/jdk-11.0.17/bin/java
# Enter the version number and press Enter.
Enter to keep the current selection[+], or type selection number: 3
# Check the Java version
[ithoth@kafka1host ~]# java -version
java version "11.0.17" 2022-10-18 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.17+10-LTS-269)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.17+10-LTS-269, mixed mode)
[ithoth@kafka1host ~]#

Note: Since there is no jre after jdk1.8, if you need to run tomcat, you need to go to the root directory of jdk and use the following command to generate a jre folder, otherwise you will get a missing jre error when running tomcat.
./bin/jlink --module-path jmods --add-modules --output jre

Installing Kafka

Download Kafka 2.8.2

/dist/kafka/2.8.2/kafka_2.13-2.8.

firewalls

Configuring Firewall Rules

# View firewall state
[root@kafka1host ~]# firewall-cmd --state
running
# View default scopes -- usually not needed to see them
[root@kafka1host ~]# firewall-cmd --get-default-zone
public
# Add firewall rules to allow access to ports 2181, 9092
[root@kafka1host ~]# firewall-cmd --permanent --zone=public --add-port=2181/tcp
# -permanent : It means to make the setting effective permanently, if you don't add it, it will be invalid after reboot of the machine.
# --add-port=2181/tcp : It means to add a port and protocol rule.
# --zone=public : the scope (default is public, can be left out)
[root@kafka1host ~]# firewall-cmd --permanent --add-port=9092/tcp
# Update firewall rules
[root@kafka1host ~]# firewall-cmd --reload
# View all open ports
[root@kafka1host ~]# firewall-cmd --list-port
2181/tcp 9092/tcp

Modify Configuration

Modify Configuration
Note that there can be no spaces before or after the "=" in the configuration

# kafka broker actually listening to the address and port, inter-cluster configuration to use, if not configured will use the hostname to cause the program can not be accessed, such as the report: can not connect: kafka1host:9092
listeners=PLAINTEXT://172.0.30.100:9092

# Allow deletion of the topic needs to be set to =true otherwise it will just mark it for deletion.
=true

# Message logs, by default logs are stored in the /tmp directory.
# The logs in the program directory are the logs of kafka runs, it is not recommended to put them in the program directory.
= /tmp/kafka-logs

# The following can not be configured ------- more detailed configuration Baidu
# Provide the address to the public, it will be registered to zookeeper, if not configured, it will use the value of listeners above.
# =PLAINTEXT://10.100.25.230:9092

operational test

Generally zookeeper is deployed separately. For this single-node deployment, we'll use the Zookeeper built into the kafka package to save you the trouble.

Starting zookeeperWindow A

# Start zookeeper first
[root@kafka1host ~]# /usr/local/kafka_2.13-2.8.2/bin/ /usr/local/kafka_2.13-2.8.2/config/

Start Kafka , open a new command lineWindow B

# Start Kafka again
[root@kafka1host ~]# /usr/local/kafka_2.13-2.8.2/bin/ /usr/local/kafka_2.13-2.8.2/config/

Create topic and open another command lineWindow Cbeta (software)

# establish topic
[root@kafka1host ~]# /usr/local/kafka_2.13-2.8.2/bin/ --create --topic kafka-vipsoft --replication-factor 1 --partitions 1 --zookeeper localhost:2181
Created topic kafka-vipsoft.

# ferret out topic
[root@kafka1host ~]# /usr/local/kafka_2.13-2.8.2/bin/ --list --zookeeper localhost:2181
__consumer_offsets
kafka-vipsoft
# removing topic，in the event that =true If you don't.，existkafkaReboot for it to take effect
[root@kafka1host ~]# /usr/local/kafka_2.13-2.8.2/bin/ --delete --topic kafka-vipsoft --zookeeper localhost:2181
Topic kafka-vipsoft is marked for deletion.
Note: This will have no impact if is not set to true.

kick-start producerWindow C

# Start the producer, can't use localhost:9092 need to be consistent with the listeners in the configuration
[root@kafka1host ~]# /usr/local/kafka_2.13-2.8.2/bin/ --broker-list 172.0.30.100:9092 --topic kafka-vipsoft
> Hello 123

Start the consumer and open another command lineWindow D

#Start the consumer
[root@kafka1host ~]# /usr/local/ --bootstrap-server 172.16.3.203:9092 --topic kafka-vipsoft --from-beginning
Hello 123

self-starting

Creating startup commands
vi /usr/local/kafka_2.13-2.8.2/kafka_start.sh

/bin/sh /bin/sh

/usr/local/kafka_2.13-2.8.2/bin/ /usr/local/kafka_2.13-2.8.2/config/ &

sleep 5 #Start zookeeper first, wait 5 seconds and then start kafka.

#Start kafka
/usr/local/kafka_2.13-2.8.2/bin/ /usr/local/kafka_2.13-2.8.2/config/ & sleep 5 #Start zookeeper first.

Creating a stop command
vi /usr/local/kafka_2.13-2.8.2/kafka_stop.sh

/bin/sh /bin/sh

/usr/local/kafka_2.13-2.8.2/bin/ /usr/local/kafka_2.13-2.8.2/config/ &

sleep 3 # Start zookeeper first, wait 3 seconds and then start kafka.

/usr/local/kafka_2.13-2.8.2/bin/ /usr/local/kafka_2.13-2.8.2/config/ & sleep 3 #Start zookeeper first, wait 3 seconds.

Modify script execution privileges

chmod 775 kafka_start.sh
chmod 775 kafka_stop.sh

Validation Scripts

sh /usr/local/kafka_2.13-2.8.2/kafka_start.sh

Setting up boot-up
vi /etc//

# Add the following script
sh /usr/local/kafka_2.13-2.8.2/kafka_start.sh &

Verify Port

View Multi-Port Status

netstat -ntpl | grep ':2181\|:9020'
[root@kafka1host ~]# netstat -ntpl | grep ':2181\|:9092'
tcp6       0      0 172.16.3.203:9092       :::*                    LISTEN      5419/java
tcp6       0      0 :::2181                 :::*                    LISTEN      4182/java