More technical exchanges, job search opportunities, welcome to pay attention to theByteDance Data PlatformWeChat Public，Reply [1]Enter the official communication group.
Sales data is an important indicator reflecting market trends, consumer behavior and product performance, and is also a key basis for enterprises to make accurate decisions. Therefore, comprehensive utilization, efficient analysis and compliant management of sales data occupy an important position in enterprise operation.
In order to use sales data more efficiently and securely, a company introduced the open source ClickHouse as a data analysis engine to unify dispersed sales data into a set of visual analytics platform, and adopted the forensic ACL model to fine-tune the management of internal staff's authority to see and use the data.
However, in fact, the company's sales data platform suffered from insufficient performance and impaired user experience after the introduction of forensic ACLs. First, the performance of ClickHouse is difficult to meet the demand of complex and huge-volume queries, which makes the cluster complexity deteriorate; second, the CPU utilization of ClickHouse cluster is in the state of hitting full for a long time which affects the user experience.
To solve these problems, ByteHouse, which offers significant advantages in complex queries and is fully compatible with ClickHouse, was the company's first choice for migration.
It is understood that ByteHouse supports optimizer and MPP execution model, which can better support complex join and aggregation computing scenarios. Among them, ByteHouse's optimizer in the direction of RBO and CBO respectively carried out a large number of self-research optimization, and the implementation of the dynamic Filter push, materialized view rewriting, plan reuse and result reuse and other high-level capabilities. Thus, it can generate the optimal query execution plan according to the table structure, index and other information, improve query execution efficiency, reduce resource consumption, and overall improve the query performance of ByteHouse in complex scenarios.
With ByteHouse's support, the company has now achieved significant query efficiency improvements in both the non-ACL query and ACL query directions for sales data. Taking the 60M advertiser DI scenario of ACL query as an example, the query efficiency has been drastically shortened from 16 seconds before the optimization to 1 second nowadays, which is an improvement of up to 16 times.
Sampling test results from a data set of the company's sales platform
As a new generation of cloud-native data warehouse products, ByteHouse has been continuously optimized in terms of offline and online complex analysis performance, convenient and elastic expansion and contraction, full-scenario analysis engine, and other core capabilities, and has been widely used in the Internet, games, finance, meteorology and other fields. In the future, ByteHouse will continue to empower more business systems with its excellent data analysis capabilities and help enterprises transform and upgrade their digital intelligence.
click to jumpVolcano Engine Cloud Native Data Warehouse ByteHouse Learn more.