StarBlog Blog Development Notes (33): Brand new access statistics function, asynchronous queue, library storage

Preface

Although the focus of work is now on AI, compared to the grand narratives of various large models, I still prefer to conceive functions, write code, and enjoy the process of solving problems and publishing them online.

When the StarBlog series was updated, I also mentioned that as the function updates are updated, I will continue to write extras after the tutorial series is completed. This is not the first extra.

This time it is a brand new design access statistics function.

Visit statistics

The access statistics function has been implemented very early, in this previous articleDeveloping blog project StarBlog - (11) Based on .NetCore, implementing access statistics

Problems with the old implementation

A middleware was added beforeVisitRecordMiddleware, each request is written to the database

This will cause two problems:

1. Influence performance
2. The database is too large and difficult to backup

New implementation

I've always been dissatisfied with this implementation

This time, it simply redesigned and solved all the problems mentioned above at one time.

I used mermaid to draw a simple picture (I tried to insert the mermaid picture in the article for the first time, but I don’t know what the effect is)

/syntax/

The new implementation uses a queue to temporarily store access logs

And added background tasks, and the access logs are regularly taken out from the queue to write to the database.

This will not affect access speed

This is basically the introduction of this new feature

Of course, there will be some details to pay attention to in the specific implementation, and the following code section will be introduced

New technology stack

This time I used EFCore as ORM

The reasons and how to introduce me in this previous article have been introduced:Asp-Net-Core Development Notes: Quickly introduce efcore to existing projects

The main purpose is to use EFCore to enable the partitioning of the library more conveniently.

Specific implementation

Next is the specific code implementation

queue

exist/ServicesAdded indocument

public class VisitRecordQueueService {
   private readonly ConcurrentQueue<VisitRecord> _logQueue = new ConcurrentQueue<VisitRecord>();
   private readonly ILogger<VisitRecordQueueService> _logger;
   private readonly IServiceScopeFactory _scopeFactory;

   /// <summary>
   /// Batch size
   /// </summary>
   private const int BatchSize = 10;

   public VisitRecordQueueService(ILogger<VisitRecordQueueService> logger, IServiceScopeFactory scopeFactory) {
     _logger = logger;
     _scopeFactory = scopeFactory;
   }

   // Add logs to queue
   public void EnqueueLog(VisitRecord log) {
     _logQueue.Enqueue(log);
   }

   // Bulk writing to the database regularly
   public async Task WriteLogsToDatabaseAsync(CancellationToken cancellationToken) {
     if (_logQueue.IsEmpty) {
       // Wait for a while to avoid high-frequency meaningless checks
       await (1000, cancellationToken);
       return;
     }

     var batch = new List<VisitRecord>();
     // Get a batch of logs from the queue
     while (_logQueue.TryDequue(out var log) && < BatchSize) {
       (log);
     }

     try {
       using var scope = _scopeFactory.CreateScope();
       var dbCtx = <AppDbContext>();
       await using var transaction = await (cancellationToken);
       try {
         (batch);
         await (cancellationToken);
         await (cancellationToken);
         _logger.LogInformation("Access log Successfully wrote {BatchCount} logs to the database", );
       }
       catch (Exception) {
         await (cancellationToken);
         throw;
       }
     }
     catch (Exception ex) {
       _logger.LogError(ex, "Access Log Error writing logs to the database: {ExMessage}", );
     }
   }
 }

Used here:

• ConcurrentQueueThis thread-safe FIFO queue
• When writing to the database in batches, transactions are used, and when an error is reported, it will automatically roll back

middleware

Modify /Middlewares/

public class VisitRecordMiddleware {
  private readonly RequestDelegate _next;

  public VisitRecordMiddleware(RequestDelegate requestDelegate) {
    _next = requestDelegate;
  }

  public Task Invoke(HttpContext context, VisitRecordQueueService logQueue) {
    var request = ;
    var ip = ()?.ToString();
    var item = new VisitRecord {
      Ip = ip?.ToString(),
      RequestPath = ,
      RequestQueryString = ,
      RequestMethod = ,
      UserAgent = ,
      Time = 
    };
    (item);

    return _next(context);
  }
}

Nothing special, it is to replace the previous database operation with adding it to the queue

Note that dependency injection cannot be in the middleware construction method.IApplicationBuilderThe dependency injection container is not fully prepared when registering the middleware

Backstage tasks

Add in /Servicesdocument

public class VisitRecordWorker : BackgroundService {
   private readonly ILogger<VisitRecordWorker> _logger;
   private readonly IServiceScopeFactory _scopeFactory;
   private readonly VisitRecordQueueService _logQueue;
   private readonly TimeSpan _executeInterval = (30);

   public VisitRecordWorker(ILogger<VisitRecordWorker> logger, IServiceScopeFactory scopeFactory, VisitRecordQueueService logQueue) {
     _logger = logger;
     _scopeFactory = scopeFactory;
     _logQueue = logQueue;
   }

   protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
     while (!) {
       await _logQueue.WriteLogsToDatabaseAsync(stoppingToken);
       await (_executeInterval, stoppingToken);
       _logger.LogDebug("Background Task VisitRecordWorker ExecuteAsync");
     }
   }
 }

It should be noted that BackgroundService is a singleton lifecycle, while the database is related to the scoped lifecycle, so you must first get scope before use, rather than directly inject it.

Used hereIServiceScopeFactoryInsteadIServiceProvider

In a multi-threaded environment, it is guaranteed that the root container instance can be obtained, which is also recommended in Microsoft's documentation.

Library division and reconstruction

Introducing EFCore

As mentioned above, the access log is relatively large. In a few months after the launch of this function, hundreds of thousands of data were accumulated, and it occupied more than 100M in the database, although this is far from reaching the database. The bottleneck

But for our lightweight project, when I want to backup, compared to a few MB of blog data, the hundreds of MB of access logs become redundant data, and there is almost no backup in this part. significance

So the division of the database is inevitable

This time I used EFCore to operate this new database separately

The previous article has been introduced in detail and will not be repeated in this article.

Asp-Net-Core Development Notes: Quickly introduce efcore to existing projects

Refactoring the service

Because EFCore is used, the services involved also need to be adjusted, and switch from FreeSQL to EFCore

Modify /Services/

public class VisitRecordService {
   private readonly ILogger<VisitRecordService> _logger;
   private readonly AppDbContext _dbContext;

   public VisitRecordService(ILogger<VisitRecordService> logger, AppDbContext dbContext) {
     _logger = logger;
     _dbContext = dbContext;
   }

   public async Task<VisitRecord?> GetById(int id) {
     var item = await _dbContext.(e => == id);
     return item;
   }

   public async Task<List<VisitRecord>> GetAll() {
     return await _dbContext.(e => ).ToListAsync();
   }

   public async Task<IPagedList<VisitRecord>> GetPagedList(VisitRecordQueryParameters param) {
     var querySet = _dbContext.();

     // search
     if (!()) {
       querySet = (a => ());
     }

     // Sort
     if (!()) {
       var isDesc = ("-");
       var orderByProperty = ('-');
       if (isDesc) {
         orderByProperty = $"{orderByProperty} desc";
       }

       querySet = (orderByProperty);
     }
    
     IPagedList<VisitRecord> pagedList = new StaticPagedList<VisitRecord>(
       await (, ).ToListAsync(),
       , ,
       Convert.ToInt32(await ())
     );
     return pagedList;
   }

   /// <summary>
   /// Overview of the data
   /// </summary>
   public async Task<object> Overview() {
     var querySet = _dbContext.VisitRecords
       .Where(e => !("/Api"));

     return new {
       TotalVisit = await (),
       TodayVisit = await (e => == ).CountAsync(),
       YesterdayVisit = await querySet
         .Where(e => == (-1).Date)
         .CountAsync()
     };
   }

   /// <summary>
   /// Trend data
   /// </summary>
   /// <param name="days">View the data in the last few days, the default is 7 days</param>
   public async Task<object> Trend(int days = 7) {
     var startDate = (-days).Date;
     return await _dbContext.VisitRecords
       .Where(e => !("/Api"))
       .Where(e => >= startDate)
       .GroupBy(e => )
       .Select(g => new {
         time = ,
         date = $"{}-{}",
         count = ()
       })
       .OrderBy(e => )
       .ToListAsync();
   }

   /// <summary>
   /// Statistics
   /// </summary>
   public async Task<object> Stats(DateTime date) {
     return new {
       Count = await _dbContext.VisitRecords
         .Where(e => == date)
         .Where(e => !("/Api"))
         .CountAsync()
     };
   }
 }

The main changes are the GetPagedList and Overview interfaces

• EFCore does not support sorting by field names by default, for this purpose I have introduced a library to implement it
• EFCore does not seem to have FreeSQL's Aggregate API, which can be replaced with native SQL, but I didn't do this and I still did multiple queries, but the impact is not big.

Others are syntax differences, just simply modify them.

summary

After a long time, I developed new features for StarBlog again, and the development experience of C# is still so smooth

However, the warning "Packages with vulnerabilities have been detected" also reminds me that the SDK version of this project has been outdated

So I'll find time to upgrade as soon as possible

A trailer: The next function is related to backup

References

• /zh-cn/dotnet/core/extensions/scoped-service
• /wucy/p/