Location>code7788 >text

Net8 pushes Serilog logs to ES with video

Popularity：140 ℃/2024-07-31 11:39:00

This is a hands-on demo for Serilog that includes distinguished record storage, AOP logging, EF execution logging, and pushing logs to Elastic Search.

A word in advance.

Since AI came out, the learning curve has instantly slowed down and the path to learning has changed dramatically.
With me before most of the first know the theory and then find the relevant framework official website or blog, and then go to practice Demo, and then added to the code, that is, theTheory - Practice - Application - Skills。
And now, after a little knowledge of the concept, you can directly approach the AI to ask for a demo to run first, apply it to the code, and then reverse the combination of the code to ask the AI about the logic and theory of each step of the implementation, and then digested into their own skills, that is, theConcepts - Applications - Theoretical Knowledge - Skills。

Therefore, this time, we try a different way of thinking about writing, and start thinking and practicing from the point of view of the reader who gets the source code.

As usual, the source code is at the end of the article, so pick it up if you need it.

environmental preparation

  • Docker Desktop
  • VS2022 SDK .NET8
  • Elastic Search 8.0+ Kibana（Docker）
  • SqlServer (Docker or local SqlServer)
  • Chat Gpt (optional)

Effect at a glance

Fulfillment requirements

  1. Separate EF, non-EF logs, stored to different paths
  2. In which non-EF logging to AOP thinking, record the execution of the method before and after the information, API interface call details and return results.
  3. Push all messages to elastic search.

Realization effects

Docker

Starting ES and SqlServer in Docker

F5 again to start the project

日志分别存储

  • EF logs are stored in the Logs/EFSql folder.
  • Non-EF logs are stored under the Logs/AuditLogs folder

推送至ES

All logs pushed to Elastic Search

The first glimpse of Program

OK, I'm sure you've pulled the code off by now, so you don't have to skip this section if you're not trying to chew through the code raw.

net 8 and .net 6 can use the top-level statements, as opposed to .net 5 and previous versions, removed the Startup as the web configuration program, that is, the entry point to get started with the major projects, here directly look at the Program to explore how to configure the Demo Web, to get a glimpse of it!

using Autofac.
using ;
using ;
using ;using ;using ;using ;using ;using ;using ;using
using ;using ;using ;using ;using ;using ;using ;using
using ;using ;using ;using ;using ;using ;using ;using
The use of the term "using" in the context of a new technology
using ;using ;using ;using ;using ;using ;using ;using
using Serilog ;using
using ILogger = ;

var builder = (args);; // Configure the configuration of a new system.
// Config
(new AppSettings()); // Serilog = (args); var builder = (args); // Config
// Serilog
(); }
// EF
<TestContext>
    (o => (("NicoLocaldbStr")))
#if DEBUG
    .EnableSensitiveDataLogging()
#endif
    .UseLoggerFactory((builder => ())
    )));
// Http Info
(); // Http Info.
// Auto Fac
(new AutofacServiceProviderFactory()); // Http Info (); // Auto Fac
<ContainerBuilder>(containerBuilder =>.
{
    // Register the AopDemo interceptor for singleton mode
    <AopDemo>().SingleInstance();

    // InterceptedBy the specified interceptor.
    <ValuesService>().As<IValuesService>().EnableInterfaceInterceptors().InterceptedBy(typeof(AopDemo));
    // Or label the IValuesService with [Intercept(typeof(AopDemo))].
    //<ValuesService>().As<IValuesService>().EnableInterfaceInterceptors();

    //Register the existing logger instance with the Autofac container, specifying this instance as an implementation of the ILogger interface, which is Serilog's ILogger.
    // When components in the application request an instance of the ILogger interface through dependency injection, they receive this singleton object
    ().As<ILogger>();
});

(); var app = ();; }); }); }); }); })
As<ILogger>(); }); (); var app = ();
// Middleware
<ResponseLoggingMiddleware>(); // Returns the logging of the client response
(); var app = (); // Returns a logging record of the client's response.
();

();

Here I'll just put All 51 lines of code are pulled out, so you can see roughly what's being done.

  • You can see that using references theAutofacas well as his extension pack, which introduces theSerilogand specifies the ILogger as , for now.

  • (new AppSettings()) Here the configuration is encapsulated as a single instance in theAppSettingsclass, and then fetch the configuration directly from the AppSettings class.

  • (); Here with an extension method to write Serilog's injection into the AddCustomLog() method, subsequently this will be the core method of the Demo, focusing on the

<TestContext>
    (o => (("NicoLocaldbStr"))
#if DEBUG
    .EnableSensitiveDataLogging()
#endif
    .UseLoggerFactory((builder => ())
    ));

  • The above code, injected into EF as DbContext pooling, and turned on sensitive logging in Debug mode and configured Serilog as the logging utility

  • (); Add information about access to Http requests for subsequent Reponse middleware development to provide HTTP information

  • as well as Configuring Auto Fac, Note Detail

  • < ResponseLoggingMiddleware >(); ResponseLoggingMiddleware middleware returns logging of client responses.

Another look at the Serilog expansion

It is clear from the above that the use ofAddCustomLog() Such an extended method, injected Serilog, OK! Ctrl + F12 to find out!

    public static class CustomLogger
    {
        public static IHostBuilder AddCustomLog(this IHostBuilder builder)
        {
             = new LoggerConfiguration()
                .()
                .()

                // Output to console
                .WriteToConsole()
                // Output to file
                .WriteToFile()
                // output toEs
                .WriteToElasticsearch()
                .CreateLogger();

            ();
            return builder;
        }
    }

  • .()：Instructs Serilog to read configuration information from the application's configuration file (e.g. ). It is assumed that there is an AppSettings class that holds configuration information for the application.

  • .()： Add the ability to enrich log events from the log context, for example by automatically adding associated HTTP request information to the log.

  • .WriteToConsole()： Configure Serilog to output logs to the console.

  • .WriteToFile()： Configure Serilog to output logs to a file.

  • .WriteToElasticsearch()： Configure Serilog to output logs to Elasticsearch. this is useful for centrally managing and analyzing logs in Elasticsearch.

WriteToConsole and WriteToFile are also extension methods, then continue Ctrl + F12 to continue.

public static class LoggerConfigurationExtensions
{
    /// <summary>
    /// Output on the console
    /// </summary>
    /// <param name="loggerConfiguration"></param>
    /// <returns></returns>
    public static LoggerConfiguration WriteToConsole(this LoggerConfiguration loggerConfiguration)
    {
        // Output general log
        loggerConfiguration = (lg =>
            ().());

        // exportsSQL
        loggerConfiguration = (lg =>
            ().());

        return loggerConfiguration;
    }

    /// <summary>
    /// write to a file
    /// </summary>
    /// <param name="loggerConfiguration"></param>
    /// <returns></returns>
    public static LoggerConfiguration WriteToFile(this LoggerConfiguration loggerConfiguration)
    {
        // SQLstatements are written to the
        loggerConfiguration = (lg =>
            ()
                .(s => ((, @""), rollingInterval: ,
                    outputTemplate: , retainedFileCountLimit: 31)));

        // in-SQLwrite
        loggerConfiguration = (lg =>
            ()
                .(s => ((, @""), rollingInterval: ,
                outputTemplate: , retainedFileCountLimit: 31)));
        return loggerConfiguration;
    }

    /// <summary>
    /// push ES
    /// </summary>
    /// <param name="loggerConfiguration"></param>
    /// <returns></returns>
    public static LoggerConfiguration WriteToElasticsearch(this LoggerConfiguration loggerConfiguration)
    {
        var esUri = ("ElasticConfiguration:Uri");
        loggerConfiguration = (new ElasticsearchSinkOptions(
                new Uri(esUri))
        {
            AutoRegisterTemplate = true,
            IndexFormat = "nico-api-log-{0:}",
            ModifyConnectionSettings = con => ((sender, certificate, chain, sslPolicyErrors) => true)
            .BasicAuthentication(
                ("ElasticConfiguration:Es_UserName"),
                ("ElasticConfiguration:Es_Pwd"))
        });
        return loggerConfiguration;
    }

    /// <summary>
    /// filter out SQLLog of statements
    /// </summary>
    /// <param name="lc"></param>
    /// <returns></returns>
    public static LoggerConfiguration FilterSqlLog(this LoggerConfiguration lc)
    {
        return (e =>
        ("SourceContext") &&
        ["SourceContext"].ToString().Contains(""));
    }

    /// <summary>
    /// 过滤in- SQLLog of statements
    /// </summary>
    /// <param name="lc"></param>
    /// <returns></returns>
    public static LoggerConfiguration FilterRemoveSqlLog(this LoggerConfiguration lc)
    {
        return (e =>
        ("SourceContext") &&
        ["SourceContext"].ToString().Contains(""));
    }
}

The notes are very detailed, and without further ado, it is necessary to explain the core issue, how to distinguish between SQL and non-SQL?

Logging Context (SourceContext):Serilog allows filtering by attributes of log events. In this example, the SourceContext property is used, which is a property commonly used to identify the source of a log. For logs generated by Entity Framework Core, SourceContext will typically contain the value "", which indicates that the log entry was generated by EF Core during the execution of a database command.

Filter ():This method allows to configure the logging system to contain only log entries that meet certain conditions. In this case, the condition is that the log event must contain SourceContext in its properties and its value contains "". In this way, only database command logs generated by EF Core will be included.

If this section is read and understood, the following practice is definitely not difficult for the reader.

Deployment of ES

I encountered a lot of pitfalls in the deployment of ES, see some of the big brother's blog is also a pass, here with Docker desktop step by step all covered in place, follow the article step by step, absolutely get it.

Starting elasticsearch, kibana

Let's start by posting the official ES documentation startup tutorial/guide/en/elasticsearch/reference/current/

Download elasticsearch, kibana mirrors, this article is as of 7/30/24, use the latest ES version!8.14.3

镜像列表

docker create network

docker network create elastic-net

创建好的网络

Start ES in docker and remember the login password to join the cluster's Token

docker run --name es01 --net elastic -p 9200:9200 -p 9300:9300 -e "=single-node" -t elasticsearch:8.14.3

After startup, it will show the password in the terminal, and the Token to join the cluster, you will need to use it later when you start kibana, if you accidentally turn it off, you can find it in docker desktop, find the container log of es.

包含了密码，Token等

Here ES startup is complete, next start Kibana

Start kibana in docker, join the cluster

docker run --name kib01 --net elastic -p 5601:5601 kibana:8.14.3

提示进入后台配置页面

http://0.0.0.0/5601 You may not be able to get in (the reason for changing the host file), use thehttp://localhost/5601can immediately (do sth)

Pass in the token, wait for the deployment to complete, and you're in the login screen.

The account defaults to elastic and the password (lowercase) is the one you saved at the beginning, in this case _*5O7P+5FKSV7nC_EDEERemember to change it to your own If you have successfully logged in, you will be taken to the backend page.

ES backend data set creation (DataView)

Create a Data view, at this point you need to pay attention to, Index pattern here you need to fill in the name of the corresponding index, match successfully in order to see the log in Discover.

Now that the ES build is complete, it's time to go ahead and push the Log to the ES platform.

Start!!!

The ES has been deployed above, now you need to run the project and push the logs into the ES platform.

Modifying the Project Configuration

The database is SQL Server, you need to change it to your own SQL link string.

    "ConnectionStrings": {
    // SQL Server
    "NicoLocaldbStr": "Data Source=localhost;Initial Catalog=nicolocaldb;User ID=sa;Password=MyStrongPwd!2#;Trust Server Certificate=True"
  }

We need to add a table here.


-- SQL Server
CREATE TABLE TestEFTable (
    Id INT PRIMARY KEY IDENTITY(1,1), -- The Id column is the primary key and is set to incremental.
    Name NVARCHAR(255) NULL, -- Name column, allow NULL, use NVARCHAR type to support Unicode characters.
    LastModifiedAt DATETIMEOFFSET NULL, -- LastModifiedAt column, allow NULL, use DATETIMEOFFSET type
    LastModifiedBy INT NOT NULL -- LastModifiedBy column, NULL is not allowed!
);

  "ElasticConfiguration": {
    "Uri": "https://localhost:9200", // ES address
    "Es_UserName": "elastic", // account number
    "Es_Pwd": "_*5O7P+5FKSV7nC_EDEE" // password
  }

The configuration of the ES is similarly modified to your own.

Proto-God Initiation! (bushi

F5 to start! (Here's a video of it in action)
/video/BV1w3vUeWE4n/

summarize

To summarize a little.

  • Demo integrated Elasticsearch, Serilog, interceptor, autofac, EF.
  • Distinguishing between EF and non-EF logging and storage
  • Recorded full how-to videos are included in the package!

source code repository [GitHub] /OrzCoCo-Y/LogSampleDemo
[Gitee] /yi_zihao/LogSampleDemo

bibliography

【ChatGPT】
Maomi Framework Borrowed from one of the HTTP logging, instead of writing your own middleware
[The philosophy of the old eight open source framework/anjoy8/ Draws on its service registration

<Previous Next>