Getting Started with Go Monitoring and Alerting Opentelemetry

preamble

Opentelemetry

Distributed Link Trace (Distributed Tracing ) concept was first proposed by Google, the development of technology has been relatively mature, there are also some protocol standards can be referred to. Currently inTracing Two of the more influential open-source technology frameworks are Netflix's open-sourceOpenTracing and Google's open sourceOpenCensus. Both frameworks have a relatively high developer base. In order to form a unified technical standard, the two frameworks eventually merged to form theOpenTelemetry The project, referred to asotelThe otel has two main sections, link tracing and monitoring alarms, for monitoring alarms, check out another article:Getting Started with Go Link Tracing Opentelemetry

Prometheus

Prometheus Sourced from SoundCloud, it has a full suite of open-source system monitoring and alerting toolkits, and is one of the most important tools to support theOpenTelemetry One of the systems that isCNCF of the second project.

Grafana

Grafana is an open source analytics and visualization platform that allows you to query, visualize, and alert on data from a variety of data sources. It provides a user-friendly interface for creating and sharing dashboards, charts, and alerts.Grafana supports a wide range of data sources, among which arePrometheus

basic concept

Here in order to get started, as simple as possible to introduce some abstract concepts, combined with the code to understand, if you can not understand it is okay, the code is written to write a natural understanding:

Meter Provider
for interfacing to manage the globalMeter Created, the global equivalent of a monitoring metrics management factory.

Meter
Used to interface to create and manage globalInstrumentdifferentMeter can be seen as different program components.

Instrument
Used to manage each of the different types of metrics under different components, such as

Measurements
Corresponding indicators reported for specificDataPoint Indicator data, a series of numerical items.

Metric Reader
Used to implement the data stream reading of the indicator and internally defines the data structure for the specific operation of the indicator.OpenTelemetry The official community offers a wide range of flexibleReader Realizations such asPeridRader、ManualReader etc.

Metric Exporter
Exporter Used to expose local metrics to corresponding third-party vendors, for example:Promtheus、Zipkin etc.

Type of indicator

OpenTelemetry metrics There are many different indicator types, which can be visualized as similar to theint, float This type of variable:

Counter：Indicators that are only increasing or decreasing, such ashttp Total number of requests, byte size;

Asynchronous Counter：Asynchronous Counter;

UpDownCounter：Indicators that can be added or subtracted, such ashttp Number of active connections;

Asynchronous UpDownCounter：Asynchronous Counter;

Gauge：Indicators that can be increased or decreased, values that are measured instantaneously, such asCPU use, it is asynchronous;

Histogram: Grouping Aggregation Metrics, which is a little more difficult to understand, can be moved tohere (literary)Check it out, and of course, there will be a detailed example of how to use it later.

Practical: collecting indicators

After a bunch of nonsense, it's finally time for the real thing. Let's start withhttp The total number of requests is used as an example to walk through the entire collection metrics process. Install the extension:

go get /prometheus/client_golang
go get /otel/exporters/prometheus
go get /otel/metric
go get /otel/sdk/metric

show (a ticket), write the following code:

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"os"
	"os/signal"

	"/prometheus/client_golang/prometheus/promhttp"
	"/otel/exporters/prometheus"
	api "/otel/metric"
	"/otel/sdk/metric"
)

const meterName = "oldme_prometheus_testing"

var (
	requestHelloCounter api.Int64Counter
)

func main() {
	ctx := ()

	// establish prometheus exporter
	exporter, err := ()
	if err != nil {
		(err)
	}

	// establish meter
	provider := ((exporter))
	meter := (meterName)

	// establish counter Type of indicator
	requestHelloCounter, err = meter.Int64Counter("requests_hello_total")
	if err != nil {
		(err)
	}

	go serveMetrics()

	ctx, _ = (ctx, )
	<-()
}

func serveMetrics() {
	("serving metrics at localhost:2223/metrics")
	("/metrics", ())

	("/index", (func(w , r *) {
		// record (in sports etc) counter norm
		((), 1)

		_, _ = ([]byte("Hello, Otel!"))
	}))

	err := (":2223", nil) //nolint:gosec // Ignoring G114: Use of net/http serve function that has no support for setting timeouts.
	if err != nil {
		("error serving http: %v", err)
		return
	}
}

In our code, we define a name for therequests_hello_total (used form a nominal expression)Int64Counter Type of indicator.Int64Counter Representing that this is an only increasingint64 value, which is just right to use as a record of the total number of requests. Running our program, without error, accesses thehttp://localhost:2223/index can be seenHello, Otel!. And we visithttp://localhost:2223/metrics Indicator data can be seen:

Here the data has not been visualized yet, let's walk through the process and visit a few more timeshttp://localhost:2223/index can be seenrequests_hello_total It will increase:

Histogram

Next, we'll capture it.Histogram Indicators, statistics on0.1, 0.2, 0.5, 1, 2, 5 straighthttp The number of requests in the Add the relevant code in it, and you can copy it directly over:

package main

import (
	"context"
	"fmt"
	"log"
	"math/rand"
	"net/http"
	"os"
	"os/signal"
	"time"

	"/prometheus/client_golang/prometheus/promhttp"
	"/otel/exporters/prometheus"
	api "/otel/metric"
	"/otel/sdk/metric"
)

const meterName = "oldme_prometheus_testing"

var (
	requestHelloCounter api.Int64Counter
	requestDurationHistogram api.Float64Histogram
)

func main() {
	ctx := ()

	// establish prometheus exporter
	exporter, err := ()
	if err != nil {
		(err)
	}

	// establish meter
	provider := ((exporter))
	meter := (meterName)

	// establish counter Type of indicator
	requestHelloCounter, err = meter.Int64Counter("requests_hello_total")
	if err != nil {
		(err)
	}

	// establish Histogram Type of indicator
	requestDurationHistogram, err = meter.Float64Histogram(
		"request_hello_duration_seconds",
		("record (in sports etc) Hello Time-consuming statistics for requests"),
		(0.1, 0.2, 0.5, 1, 2, 5),
	)
	if err != nil {
		(err)
	}

	go serveMetrics()
	go goroutineMock()

	ctx, _ = (ctx, )
	<-()
}

func serveMetrics() {
	("serving metrics at localhost:2223/metrics")
	("/metrics", ())

	("/index", (func(w , r *) {
		// record (in sports etc) counter norm
		((), 1)

		// Calculate request processing time
		startTime := ()
		// Simulate request processing time
		(((3)) * )
		defer func() {
			duration := (startTime).Seconds()
			((), duration)
		}()

		_, _ = ([]byte("Hello, Otel!"))
	}))

	err := (":2223", nil) //nolint:gosec // Ignoring G114: Use of net/http serve function that has no support for setting timeouts.
	if err != nil {
		("error serving http: %v", err)
		return
	}
}

// Randomly simulate a number of co-programs
func goroutineMock() {
	for {
		go func() {
			// Wait a few seconds.
			var s = ((10))
			(s * )
		}()
		(1 * )
	}
}

Going here, the code level is over, it's already halfway there, the code is open source in theGithub. After that we can install thePrometheus server-side andGrafana to perform data visualization.

Installing Prometheus

Prometheus There are several ways to install it, but I'm still using theDocker Installation, of course, you can also use other ways to install, the specific installation can refer to other articles, the subsequentGrafana Ditto, without further ado, in Fill in thetargets Our address:

scrape_configs:
  - job_name: "prometheus"

    static_configs:
      - targets: ["localhost:2223"]

Prometheus go{{target}}/metrics Pull our indicator in. Afterwards, open thePromethues address, e.g. mine is: http://localhost:9090, if it all works you can find it in thestatus:targets See our metrics in:

existPromethues Home Page Enquiryrequests_hello_total Indicators can be seen in visualized charts:

Installing Grafana

myGrafana Once installed, this is what it looks like when you log in (I changed the default colors):

existData source AddPrometheus server, and then in theDashboard Add the metrics we want to monitor to see a better looking chart: