Location>code7788 >text

Dynamic Authorization Authentication for Kafka Based on SASL/SCRAM

Popularity：811 ℃/2024-07-29 09:33:31

I. Description

In big data processing and analysisApache Kafka has become a core component. However, deploying in a production environmentKafka When it comes to security, it is an important factor that must be considered.SASL(simple authentication and security layer) andSCRAM(Saltation Challenge Response Authentication Mechanism for Password-Based Authentication Mechanisms) provides a way to augment theKafka Cluster security.

In this article, we will start deploying from scratchZooKeeper cap (a poem)Kafka and by configuring theSASL/SCRAM cap (a poem)ACL(access control lists) to enhanceKafka The safety of the

 

Kafka's security mechanism

The kafka community has made a number of changes to the0.9.0.0 The version formally adds security features to meet a variety of security requirements, including:

  1. Secure communication between Kafka and ZooKeeper;
  2. Secure communication between Kafka cluster ZooKeeper;
  3. Secure communication between client and server;
  4. Message level privilege control, you can control the client (producer or consumer) read and write operation privileges.

 

Authentication Methods introduced version Applicable Scenarios
SSL 0.9.0 SSL does more channel encryption, SSL authentication is not as good as SASL so SSL is generally used for communication encryption.
SASL/GSSAPI 0.9.9 It is mainly for Kerberos. If your company already does Kerberos authentication (e.g. using Active Directory), then using GSSAPI is the most convenient. Because you don't need to set up additional Kerberos, just have your Kerberos administrator request the principal for each Broker and OS user that will be accessing the Kafka cluster.
SASL/PLAIN 0.10.2 Simple username-password authentication, usually used in conjunction with SSL, is more appropriate for small companies that don't have the need to build company-level Kerberos.
SASL/SCRAM 0.10.2 An enhanced version of PLAIN that supports dynamic user additions and subtractions.
Deleation Token 1.1 The Delegation Token, introduced in version 1.1, is a lightweight authentication mechanism designed to complement existing SASL or SSL authentication. To use a Delegation Token, you need to configure SASL authentication first, and then use the API provided by Kafka to get the corresponding Delegation Token, so that the Broker and the client can use this token directly when doing authentication, instead of having to go to the KDC to get the corresponding ticket (Kerberos authentication or transferring a Keystore file) each time. Kerberos authentication) or transferring Keystore files (SSL authentication).
SASL/OAUTHBEARER 2.0 OAuth 2 framework integration.

 

III. Environment and software preparation

through (a gap)Apache Kafka official website Download the corresponding version of Kafka and extract it to a directory of your choice.

Ensure that you have Java installed to run Kafka by running thejava -version to check the Java environment.

 

Deploying Zookeeper

Using Kafkainternally installedZookeeper

4.1. Enabling SASL Authentication

Go to the config directory and change the The configuration file adds the following:

authProvider.1=
jaasLoginRenew=3600000
requireClientAuthScheme=sasl
=true

 

4.2. Configuring JAAS

In the config directory, create thezk_jaas.conf file, which reads as follows:

Server {
     required
    username="admin"
    password="admin"
    user_admin="admin"
    user_zkclient="zkclient";
};

What it does is create a Server node where the

  • required is the processing class for the authentication logic;
  • username、password is the username and password for communication between zookeepers;
  • user_admin="admin" The structure of user_[username]=[password] defines the username and password used by the kafka-broker (zookeeper client) to connect to the zookeeper.

Note: The last line of the Server's internal; and } after; Can't be missing!

 

4.3 Modification of start-up files

Go to the bin directory and modify the Documentation;

existexport KAFKA_HEAP_OPTS= After the parameters of the configuration item, addJAAS The configuration of the

export KAFKA_HEAP_OPTS="-Xmx512M -Xms512M -=../config/zk_jaas.conf"

 

4.4. Starting Zookeeper

Execute the command:./ -daemon ../config/

The -daemon parameter configures background operation

 

4.5.

Can be downloaded from the official websiteApache ZooKeeper Download the corresponding version of ZooKeeper and unzip it;

increaseJAAS configuration, create thezk_client_jaas.conf Documentation:

Client{
     required
    username="zkclient"
    password="zkclient";
};

Modify the file, add the JAAS configuration to the startup command:

"$JAVA" "-=${ZOO_LOG_DIR}" "-=${ZOO_LOG4J_PROP}" "-=${ZOO_LOG_FILE}" \
     -cp "$CLASSPATH" $CLIENT_JVMFLAGS $JVMFLAGS \
     "-=../conf/zk_client_jaas.conf" \
      "$@"

fulfillment Connect to a locally activatedZooKeeper

Go to Kafka's log directory to view the logs of the built-in zk The following is displayed:

INFO adding SASL authorization for authorizationID: zkclient ()

SASL authentication for ZooKeeper has been successfully configured.

 

V. Deploying Kafka

5.1. Configuring the Kafka Broker

Go to the config directory and change the The configuration file adds the following:

listeners=SASL_PLAINTEXT://:9092
=SASL_PLAINTEXT://localhost:9092
=SASL_PLAINTEXT
=SCRAM-SHA-256
=SCRAM-SHA-256
=
=false
=User:admin
  • Enables the ACL authorization mechanism and specifies the implementation class;
  • Whether to allow any operation if no ACL (Access Control List) configuration is found; here set tofalse means that users other than the super administrator must be configured with an ACL to access the resource;
  • Super Administrator, no need to configure ACLs Users with all privileges.

 

5.2. Configuring JAAS

In the config directory, create thekafka_server_jaas.conf file, which reads as follows:

KafkaServer {
     required
    username="admin"
    password="admin";
};

Client{
     required
    username="zkclient"
    password="zkclient";
};
  • KafkaServer hit the nail on the headusername，password The account password used for communication between Kafka cluster Broker nodes;
  • KafkaServer hit the nail on the headuser_test="test" Used for Kafka clients (producer, consumer) to log in with the account password configured in the user_[username]=[password] structure under this configuration when connecting to the broker;
  • Client Used for authentication between broker and zookeeper, corresponds to the [user_zkclient="zkclient"] configuration in zk_jaas.conf;
  • user_admin="admin" The structure of user_[username]=[password] defines the username and password used by the kafka-broker (zookeeper client) to connect to the zookeeper.

 

5.3 Modification of start-up files

Go to the bin directory and modify the Documentation;

existexport KAFKA_HEAP_OPTS= After the parameters of the configuration item, addJAAS The configuration of the

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G -=../config/kafka_server_jaas.conf"

 

5.4 Creating SCRAM users

Before starting Kafka, you need to create a user and execute the following in the bin directory:

Create separateadmin(Super Administrator) andtest(client user)

./ --zookeeper localhost:2181 --alter --add-config 'SCRAM-SHA-256=[password=admin]' --entity-type users --entity-name admin

./ --zookeeper localhost:2181 --alter --add-config 'SCRAM-SHA-256=[password=test]' --entity-type users --entity-name test

SASL/SCRAM Authenticated user information is dynamically created and stored in the ZooKeeper, due to the above configuration.kafka_server_jaas.conf The communication between Brokers in theadmin of the user, if the user does not exist it willStartup Error。

 

5.5. Starting Kafka

Execute the command:./ -daemon ../config/

The -daemon parameter configures background operation

 

VI. Verifying SASL/SCRAM Authentication

6.1. Client Authentication Configuration

6.1.1 Administrator configuration

Go to the config directory and create the The content is as follows:

=SASL_PLAINTEXT
=SCRAM-SHA-256
= required username="admin" password="admin";

Configure the type of authentication as well as the processing class and user of the login logic, using the super administrator admin

take note of final; is a must add.

6.1.2 Producer configuration

Modify the Add the following:

=SASL_PLAINTEXT
=SCRAM-SHA-256
= required username="admin" password="admin";

The producer also uses the super administrator admin to send messages.

6.1.3 Consumer Configuration

Modify the Add the following:

=SASL_PLAINTEXT
=SCRAM-SHA-256
= required username="test" password="test";

Consumers use the test user to receive messages.

 

6.2 Creating a topic

Execute the following command in the bin directory:

./ --bootstrap-server localhost:9092 --create --topic test --partitions 1 --replication-factor 1 --command-config ../config/
  • bootstrap-server Configure the address of the Kafka server
  • topic Specify the topic name
  • command-config Specify the authentication configuration for the command, here use the above createdAdministrator Configuration

Once created, you can view the list of existing topics with the following command:

./ --bootstrap-server localhost:9092 --list --command-config ../config/

 

6.3 Creating consumers

6.3.1. executing kafka-console-consumer

Execute the following command in the bin directory:

./ --bootstrap-server localhost:9092 --topic test -- ../config/

Executing the command reveals the followingreport an error Information:

ERROR Error processing message, terminating consumer process:  ($)
: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256
Processed a total of 0 messages

Authentication failed Authentication fails because the consumer's authentication uses the test user, which has not been configured with any ACL permissions.

6.3.2. Configuring User ACL Privileges

Kafka's ACLs (Access Control Lists) allow you to define which users have access to which topics and can perform which operations (e.g. read, write, create, delete, etc.).

Execute the following command:

./ --authorizer-properties =localhost:2181 --add --allow-principal User:test --operation Read --topic test --group test-consumer-group

To create a resource for the test user in thetopic[test] Assign read-only permissions under

The execution was successful. you can view a list of all ACLs assigned to the resource with the following command:

./ --bootstrap-server localhost:9092 --topic test --list --command-config ../config/

Re-create the consumer:

./ --bootstrap-server localhost:9092 --topic test -- ../config/

After successful execution, the shell window will remain blocked waiting for a message.

 

6.4 Creating producers

Open a new shell window Execute the following command in the bin directory:

./ --bootstrap-server localhost:9092 --topic test -- ../config/

Since the producer's authentication uses the admin for thesuper-administrator Therefore, there is no need to configure ACL permissions.

After successful execution, the> symbol, and after entering the content, switch to theconsumers The window will be visible.

<Previous Next>