So many kafka problems, look at the solutions!

At the end of this article, there is a group to share the interview experience

I've already shared a few interviews, this is a detailed interview about a more project and technology oriented interview, first time I met asking so many kafka questions, see how this fan answered.

Let's take a look.Job Description：

Job Description:

• Responsible for the design, development and maintenance of Go-based backend services;
• Participate in system architecture design to ensure high availability, performance and scalability;
• Write high quality, maintainable code and perform code reviews;
• Solve complex technical problems and optimize system performance;
• Work with cross-functional teams to ensure projects are delivered on time;
• Write technical documentation and development specifications to improve the overall development efficiency of the team;
• Participate in the research and introduction of new technologies and promote technological innovation.

Job Requirements:

1. Bachelor's Degree or higher in Computer Science or related field with a clear and logical mind;
2. 5+ years of software development experience, with at least 3 years of Go language development experience;
3. Familiarity with the standard libraries and common frameworks of the Go language, and an understanding of Go's memory management and concurrency model;
4. Experience in microservices architecture design and development, familiar with gRPC, Protobuf and other technologies;
5. Familiarity with common databases (e.g. MySQL, PostgreSQL) and caching technologies (e.g. Redis);
6. Good code style and habit of writing high quality unit tests;
7. Familiar with container technologies such as Docker, Kubernetes, etc. with some DevOps experience;
8. Strong analytical and problem solving skills, good communication and teamwork skills;
9. Contributors to open source projects or personal technical blogs are preferred.

Extra credit:

• Experience with Rust development and familiarity with other programming languages (e.g. Python, C++, etc.);
• Experience in the development of trading systems in the financial industry, such as quotes, trading, and risk control.

interview process

Take a look at the full interview process:

1. self-introduction

2. Tell us about the business of the project, how it was done, what were the difficulties and how were they solved?

   • Message queue: data skewing problem, no data loss, repeated consumption
   • redis: using hash as well as Pipline, bigkey disassembly
   • Database: performance optimization, split library and split table

3. Database optimization sets lock_timeout which lock is timeout

   The lock_timeout in the database optimization settings is primarily for theline lockThe timeout setting of lock_timeout. When a transaction is waiting to acquire a row lock, if the lock_timeout setting is exceeded, a timeout error is thrown to prevent the transaction from waiting for a long period of time, which may result in a degradation of system performance.

4. kafka cluster size

   Our Kafka cluster is sized at 5 servers and is capable of supporting 500,000 concurrency. With proper configuration and optimization, we can further improve the performance and concurrency of our Kafka cluster.

5. Why kafka supports high concurrency

   Kafka supports high concurrency for several main reasons:

   1. Partitioning: Kafka divides the data into multiple partitions, each of which can be stored and processed on a different server, thus enabling horizontal scaling and increasing the concurrency of the system.
   2. Zero-copy: Kafka uses zero-copy technology, which reduces the number of times data is copied in memory and improves data transfer efficiency.
   3. Network compression: Kafka supports network compression, which can reduce the amount of data transmitted over the network and improve the performance and concurrency of the system.

6. Tell us what a zero-copy is.

   Zero-copy is a technique that improves data transfer efficiency by reducing the number of times data is copied in memory. In Kafka, the zero-copy technique is implemented by using the mmap function, which maps a file into memory so that the application can access the contents of the file directly without having to copy the data. This reduces the number of times data is copied in memory and improves data transfer efficiency.

7. Why kafka

   First of all, Kafka has a high concurrency processing capability, which can meet the high throughput requirements of financial transaction systems. Second, Kafka scales very well and it is easy to add server nodes to improve system performance. In addition, Kafka is inherently suitable for cluster deployment and is written in Java and Scala, which makes it easy to troubleshoot and maintain. In contrast, RabbitMQ's cluster deployment is relatively complex and uses the Erlang language, which may be unfamiliar to some developers and difficult to troubleshoot.

8. What is the Api Gateway used for?

   Api Gateway is mainly used for traffic limitation and security control of third-party calls. It can filter, route and load balance requests to ensure system stability and security. At the same time, the Api gateway can also provide some additional functions, such as authentication, authorization, monitoring and statistics.

9. What about the payment module piece?

   In the project, the payment module is developed by other colleagues. I mainly realize the related business functions by calling the API provided by the payment module. When calling the API, I will ensure the correctness and safety of the parameters, and at the same time, I will make reasonable processing and error handling for the returned results.

10. Tell us about your microservices architecture.

    Our system utilizes a hybrid architecture of serviceless and microservices. This architecture can fully utilize the elasticity and efficiency of the serviceless architecture, and at the same time take advantage of the flexibility and scalability of the microservice architecture. In the concrete implementation, we encapsulate some general-purpose functional modules into serviceless functions, which are invoked by event-driven way. And for some complex business modules, we adopt microservice architecture for independent development and deployment.

11. How differential upgrading is achieved

    By comparing different versions of the code, we find out the differences, and then extract these differences for operations such as calculating MD5 values to determine whether an upgrade is needed. During the upgrade process, we will ensure data integrity and consistency while minimizing the impact on the system.

12. Compilation services similar to CI/CD

    By modifying the Git commands, a function similar to Web Hook is realized. When a code is submitted and tagged, the system will automatically perform a Git clone operation, and then test and compile. This ensures the quality and stability of the code and improves development efficiency.

13. C/C++ is for developing firmware, right?

    Yes, I have developed smart gateways using a combination of C/C++ and Lua. However, since I am applying for a back-end development position this time, I did not include that part in my resume.

14. You're a big fan of redis?

    Yes, I really like to use Redis, which is like a data-structured database that provides a rich set of data structures and operation interfaces that are very easy to use. In our project, we use Redis to implement many business functions, such as caching hot data, implementing distributed locking, and task queue management.

15. ask (a question) in reply

wind up

As you can see, in thisIn-depth technical interviewsin which candidates are required to have strong technical skills and the ability to deal with a variety of challenges. The direction of the interviewer's inquiries seems to be mostlyProblems with Kafka's High Concurrency Handling, I wonder if this interview has given you any learning points? I also hope that by sharing this interviewer's experience, it will inspire and help you if you're looking for a job or working on technology enhancement.

Welcome ❤

We got one.Free group for sharing interview questions, intercommunicate and brush up on your progress together.

Maybe it will enable you to brush up on the latest interview questions for your intended company.

Interested friends can add my wechat:wangzhongyang1993, Remarks: [Blog Park].