Introduction to VictoriaMetrics Chinese Tutorial (10) Cluster Edition

VictoriaMetrics Chinese tutorial article series:

• VictoriaMetrics Chinese Tutorial (01) Introduction
• VictoriaMetrics Chinese Tutorial (02) Installation
• VictoriaMetrics Tutorial (03) How to Configure Prometheus to Write Data Remotely to VictoriaMetrics
• VictoriaMetrics Tutorial (04) Docking to Grafana while Introducing vmui
• VictoriaMetrics Chinese Tutorial (05) Docking Various Monitor Data Collectors
• VictoriaMetrics Chinese Tutorial (06) Capacity Planning
• VictoriaMetrics Chinese Tutorial (07) High availability (High availability) program
• VictoriaMetrics Chinese Tutorial (08) VictoriaMetrics Storage
• VictoriaMetrics Tutorial (09) 18 Troubleshooting Suggestions and Tips for VictoriaMetrics

Introduction to Cluster Edition

The VictoriaMetrics cluster version is also open source, but is more complex to maintain - there are more components after all. If the data volume is less than one million data points per second, it is recommended to use the single-node version instead of the cluster version. The single-node version is perfectly adapted to the number of CPU cores, RAM and available storage space. The single-node version is easier to configure and operate than the clustered version, so think twice before choosing the clustered version.

Outstanding features

• All features of the single-node version are supported.
• Performance and capacity level scaling.
• Supports multiple independent namespaces (aka multi-tenancy) for time series data.
• Supports replication replication.

VictoriaMetrics Architecture Overview

The VictoriaMetrics cluster consists of the following services:

• vmstorage - stores raw data and returns query data for a given tag filter in a given time range
• vminsert - accepts extracted data and distributes it among vmstorage nodes based on a consistent hash of the metric name and all its tags
• vmselect - performs a query by fetching the required data from all configured vmstorage nodes

Each service scales independently and can run on the most appropriate hardware. vmstorage nodes do not know about each other, do not communicate with each other, and do not share any data. This is a shareless architecture. It improves cluster availability and simplifies cluster maintenance as well as cluster scaling.

Note that even though vmselect is stateless, it still requires some disk space (a few GB) for temporary caching. For more details, see-cacheDataPath Command line flags.

multi-tenant

VictoriaMetrics clusters support multiple mutually isolated tenants (aka namespaces). Tenants are identified by accountID oraccountID:projectID These identifiers are placed in the request URLs for writes and reads. Some facts about VictoriaMetrics tenants:

• Each accountID and projectID is defined by the[0 .. 2^32) Any 32-bit integer identifier in the range. If the projectID is missing, it is automatically assigned to 0. It is expected that additional information about the tenant (e.g., authentication token, tenant name, restrictions, accounting, etc.) will be stored in a separate relational database. This database must be managed by a separate service (such as vmauth or vmgateway) located in front of the VictoriaMetrics cluster.
• Tenants are automatically created when the first data point is written to a given tenant.
• All tenants' data is evenly distributed across the available vmstorage nodes. This ensures load uniformity among vmstorage nodes when data volumes and query loads vary across tenants.
• Database performance and resource utilization is independent of the number of tenants. It depends primarily on the total number of active time series across all tenants. A time series is considered active if it has received at least one sample in the past hour or has been affected by a query in the past hour.
• This can be done byhttp://<vmselect>:8481/admin/tenants url to get a list of registered tenants.
• VictoriaMetrics publicizes individual tenant statistics through metrics.

As many of you know, the read and write addresses for the clustered version are:

• Read:http://{vmselect}:8481/select/0/<suffix>For examplehttp://{vmselect}:8481/select/0/prometheus/api/v1/query
• Write:http://{vminsert}:8480/insert/0/<suffix>For examplehttp://{vminsert}:8480/insert/0/prometheus/api/v1/write

Here.0 is the accountID.<suffix> It's the specific path.0 means that accountID is 0 and projectID is also 0, i.e., the default tenant.

Cluster Edition Startup

The minimum cluster must contain the following nodes:

• have-retentionPeriod cap (a poem)-storageDataPath Single vmstorage node for command line arguments
• involve-storageNode=<vmstorage_host> A single vminsert node of the
• involve-storageNode=<vmstorage_host> A single vmselect node of the

For high availability, it is recommended to run at least two nodes for each service. In this case, when a single node is temporarily unavailable, the cluster will continue to work and the remaining nodes can handle the increased workload. The node may be temporarily unavailable during underlying hardware failures, software upgrades, migrations, or other maintenance tasks.

It is better to run many small vmstorage nodes rather than a few large vmstorage nodes because this reduces the increase in workload on the remaining vmstorage nodes when some vmstorage nodes are temporarily unavailable.

An http load balancer, such as vmauth or nginx, must be placed in front of the vminsert and vmselect nodes. it must contain the following routing configuration based on the url format:

• in order to/insert Requests that begin with must be routed to port 8480 on the vminsert node.
• in order to/select Requests that begin with must be routed to port 8481 on the vmselect node.

The port can be changed by setting -httpListenAddr on the appropriate node.

read-only mode

(coll.) fail (a student)-storageDataPath The directory pointed to contains less free space than- When it does, the vmstorage node automatically switches to read-only mode. vminsert nodes stop sending data to such nodes and begin rerouting data to the remaining vmstorage nodes.

When vmstorage is in read-only mode, it willhttp://vmstorage:8482/metrics uppervm_storage_is_read_only The indicator is set to 1. It is set to 0 when vmstorage is not in read-only mode.

Replication and data security

By default, VictoriaMetrics recommends that users put the-storageDataPath The data is guaranteed to be persistent by pointing to highly available underlying storage, such as Google Compute Persistent Disks. If for some reason a replicated persistent persistent disk is not available, VictoriaMetrics supports application-level replication.

This can be done by placing the-replicationFactor=N Command line flags are passed to vminsert to enable replication. This instructs vminsert to store N copies of each ingested sample on N different vmstorage nodes. This ensures that if up toN-1 If a vmstorage node is not available, all stored data is still available for querying.

commander-in-chief (military)-replicationFactor=N Command line flags are passed to vmselect to indicate that it will not be able to query if during the query less than the-replicationFactor of the vmstorage node is not available, then do not mark the response as partial.

The cluster must contain at least2*N-1 vmstorage nodes, where N is the replication factor for theN-1 A given replication factor for maintaining newly extracted data when a storage node is unavailable.

VictoriaMetrics stores timestamps with millisecond precision, so when replication is enabled you must set the-=1ms Command line flags are passed to vmselect nodes so that they can de-duplicate samples obtained from different vmstorage nodes during a query. If duplicates are pushed to VictoriaMetrics from identically configured vmagent instances or Prometheus instances, the de-duplication document must be passed the- is set to thescrape_interval。

Please note that replication cannot prevent disasters, so regular backups are recommended.

Note that copying increases resource usage (CPU, RAM, disk space, network bandwidth) by up to-replicationFactor=N times, because vminsert stores N copies of the incoming data to different vmstorage nodes, and vmselect needs to deduplicate the replicated data obtained from the vmstorage nodes during the query. Therefore, it is not a good idea to leave the replication to the-storageDataPath Pointed underlying replicated persistent storage, such as Google Compute Engine persistent disks, is more cost-effective, which prevents data loss and data corruption. It also provides consistent high performance and can be resized without downtime. An HDD-based persistent disk should be sufficient for most use cases. It is recommended that you use persistent replicated persistent volumes in Kubernetes.

Deduplication

The clustered version of VictoriaMetrics supports data deduplication in the same way as the single-node version. The only difference is that deduplication is not guaranteed when samples and sample replicates of the same time series end up on different vmstorage nodes. This can happen in the following cases:

• When a vmstorage node is added/removed, new samples of the time series will be rerouted to another vmstorage node;
• When a vmstorage node is temporarily unavailable (for example, during a restart). New samples are then rerouted to the remaining available vmstorage nodes;
• When a vmstorage node does not have enough capacity to handle the incoming data stream, vminsert reroutes the new samples to other vmstorage nodes.

It is recommended that vmselect and vmstorage nodes be set with the same- Command line flag value to ensure consistency of query results, even if the storage tier has not completed deduplication.

summarize

VictoriaMetrics Cluster Edition is more suitable for large-scale monitoring data storage and querying than the standalone edition. However, VictoriaMetrics Cluster Edition is more expensive to maintain and require more hardware resources. Please think twice before choosing the clustered version, VictoriaMetrics supports replication, hanging some nodes does not affect data security, however, it is recommended not to turn on, but to leave it to cloud storage to ensure data persistence. If you don't have the conditions to use cloud storage, then consider turning on replication.