NET Screen Recording

Window / screen capture applies to screenshots, annotations and other tools scene, when the window / screen image data stream to obtain it, the following talk about video conferencing shared desktop, remote desktop these scenarios is how to realize the screen recording.

Common screen picture time capture program, mainly GDI, WGC, DXGI.

GDI

GDI (Graphics Device Interface) is the use of user32 under WindowsAPI to realize, is the earliest and most basic graphics device interface in the Windows operating system, meet all windows platform. Screenshots of the screen/window can be seen in detail:NET window/screenshot - TangSongYuanMingQing2188 - Blogland ()

GDI performance is not very good, especially for high frame rate and high resolution needs, to achieve more than 20 frames per second interception, CPU consumption is a little high. In addition, GDI can not get the mouse, you need to draw the mouse on the intercepted image.

So GDI is easy to use, does not rely on the GPU, and screenshot scenarios with low performance requirements are recommended to use this solution directly.

WGC

Windows Graphics Capture, a new mechanism introduced by Win10 to capture the screen and the contents of the window.Screen capture - UWP applications | Microsoft Learn

WinRT provides interface access, Csproj properties add: <UseWinRT> true</UseWinRT>

Screenshot code implementation example:

 1     public WgcCapture(IntPtr hWnd, CaptureType captureType)
 2     {
 3         if (!())
 4         {
 5             throw new Exception("Windows Graphics Capture API not supported");
 6         }
 7         var item = captureType ==  ? (hWnd) : (hWnd);
 8         CaptureSize = new Size(, );
 9 
10         var d3dDevice = (false);
11         _device = (d3dDevice);
12         _framePool = (d3dDevice, pixelFormat: DirectXPixelFormat.B8G8R8A8UIntNormalized, numberOfBuffers: 1, );
13         _desktopImageTexture = CreateTexture2D(_device, );
14         _framePool.FrameArrived += OnFrameArrived;
15          += (i, _) =>
16         {
17             _framePool.FrameArrived -= OnFrameArrived;
18             StopCapture();
19             ItemClosed?.Invoke(this, i);
20         };
21         _session = _framePool.CreateCaptureSession(item);
22     }
23     private void OnFrameArrived(Direct3D11CaptureFramePool sender, object args)
24     {
25         try
26         {
27             using var frame = _framePool.TryGetNextFrame();
28             if (frame == null) return;
29             var data = CopyFrameToBytes(frame);
30             var captureFrame = new CaptureFrame(CaptureSize, data);
31             FrameArrived?.Invoke(this, captureFrame);
32         }
33         catch (Exception)
34         {
35             // ignored
36         }
37     }

The API is responsible for actually grabbing pixels from the screen, theGraphicsCaptureItem class represents the captured window or display.GraphicsCaptureSession for starting and stopping capture operations.Direct3D11CaptureFramePool class maintains the buffer into which the screen content is to be copied for the frame.

WGC screenshot process:

Create Capture Item: Use CreateCaptureItemForMonitor or CreateCaptureItemForWindow to create a capture item.
Create D3D11 device and context: Call D3D11CreateDevice to create a Direct3D 11 device and device context. The DXGI screenshot is not used here, but it references the DXGI device type
Convert to Direct3D Device: Convert a D3D11 device to a SharpDX Direct3D device object.
Create frame pools and sessions: Use Direct3D11CaptureFramePool and GraphicsCaptureSession.
Start Capture: Call StartCapture to start the session and register the frame arrival event.
Processing Frames: Processing captured frames in a frame arrival event

Here we are using the more mature SharpDX for Direct3D, quoting the following Nuget version

Get the intercepted D3D object frame, frame to data stream:

 1     private byte[] CopyFrameToBytes(Direct3D11CaptureFrame frame)
 2     {
 3         using var bitmap = Direct3D11Utils.CreateSharpDxTexture2D();
 4         _device.(bitmap, _desktopImageTexture);
 5         // Mapping Texture2D resources to CPU memory
 6         var mappedResource = _device.(_desktopImageTexture, 0, , );
 7         //Bgra32
 8         var bytesPerPixel = 4;
 9         var width = _desktopImageTexture.;
10         var height = _desktopImageTexture.;
11         using var inputRgbaMat = new Mat(height, width, MatType.CV_8UC4, , );
12 
13         var data = new byte[ *  * bytesPerPixel];
14         if ( != width ||  != height)
15         {
16             var size = new (, );
17             (inputRgbaMat, inputRgbaMat, size, interpolation: );
18         }
19         var sourceSize = new Size(, );
20         if (CaptureSize == sourceSize)
21         {
22             var rowPitch = ;
23             for (var y = 0; y < height; y++)
24             {
25                 var srcRow =  + y * rowPitch;
26                 var destRowOffset = y * width * bytesPerPixel;
27                 (srcRow, data, destRowOffset, width * bytesPerPixel);
28             }
29         }
30         else
31         {
32             (, data, 0, );
33         }
34 
35         _device.(_desktopImageTexture, 0);
36         return data;
37     }

Converts a Surface object to a Texture2D that acquires SharpDX, mapping to the CPU to output image byte data as a memory copy.

The above default is to output three-channel 8-bit Bgr24, if it is four-channel Bgra32 can be copied from memory as follows:

1 using var inputRgbMat = new Mat();
2 (inputRgbaMat, inputRgbMat, ColorConversionCodes.BGRA2BGR);
3 (, data, 0, );

Get the byte data and you can save it locally or display it in the interface .

Screenshot Demo shows:

 1     private void CaptureButton_OnClick(object sender, RoutedEventArgs e)
 2     {
 3         var monitorHandle = ().First().MonitorHandle;
 4         var wgcCapture = new WgcCapture(monitorHandle, );
 5          += WgcCapture_FrameArrived;
 6         ();
 7     }
 8 
 9     private void WgcCapture_FrameArrived(object? sender, CaptureFrame e)
10     {
11         (() =>
12         {
13             var stride =  * 4; // 4 bytes per pixel in BGRA format
14             var bitmap = (, , 96, 96, PixelFormats.Bgra32, null, , stride);
15             ();
16              = bitmap;
17         });
18     }

WGC utilizes modern graphics hardware and operating system features to provide high-performance and low-latency screen capture for real-time scenarios such as screen recording, video conferencing, and other applications.

For more, you can refer to the official websiteScreen capture to video - UWP applications | Microsoft Learn. Also browse and run my Demo:kybs00/CaptureImageDemo ()

DXGI

The full name is DirectX Graphics Infrastructure, from Win8, Microsoft introduced a new set of interfaces Desktop Duplication API, and because the Desktop Duplication API is through the DXGI to provide desktop images, the speed is very fast.

DXGI uses GPUs, so the cpu usage is very low and the performance is very high. documentation on the official DXGI website:DXGI - Win32 apps | Microsoft Learn

Because DXGI also uses DirectX, so many interfaces are similar to WGC. That is, through D3D, all kinds of QueryInterface, all kinds of Enum, the core method is AcquireNextFrame.

It has a disadvantage, there is no way to capture the contents of the window. So sharing windows in a videoconference is not possible with DXGI.

Let's look at the Demo call code.

 1     private void CaptureButton_OnClick(object sender, RoutedEventArgs e)
 2     {
 3         var monitorDxgiCapture = new MonitorDxgiCapture();
 4          += WgcCapture_FrameArrived;
 5         ();
 6     }
 7 
 8     private void WgcCapture_FrameArrived(object? sender, CaptureFrame e)
 9     {
10         ?.(() =>
11         {
12             var stride =  * 4; // 4 bytes per pixel in BGRA format
13             var bitmap = (, , 96, 96, PixelFormats.Bgra32, null, , stride);
14 
15             ();
16              = bitmap;
17         });
18     }

Capture screen frame data:

 1     [HandleProcessCorruptedStateExceptions]
 2     private CaptureFrame CaptureFrame()
 3     {
 4         try
 5         {
 6             var data = new byte[ *  * 4];
 7             var result = _mDeskDupl.TryAcquireNextFrame(TimeOut, out _, out var desktopResource);
 8             if () return null;
 9 
10             using var tempTexture = desktopResource?.QueryInterface<Texture2D>();
11             _mDevice.(tempTexture, _desktopImageTexture); //Copy Image Texture: GPU Hardware Accelerated Texture Replication
12             desktopResource?.Dispose();
13 
14             var desktopSource = _mDevice.(_desktopImageTexture, 0, , );
15             using var inputRgbaMat = new Mat(_screenSize.Height, _screenSize.Width, MatType.CV_8UC4, );
16             if ( != _screenSize.Width ||  != _screenSize.Height)
17             {
18                 var size = new (, );
19                 (inputRgbaMat, inputRgbaMat, size, interpolation: );
20             }
21             (, data, 0, );
22 
23             var captureFrame = new CaptureFrame(CaptureSize, data);
24             _mDevice.(_desktopImageTexture, 0);
25             //release frame
26             _mDeskDupl.ReleaseFrame();
27             return captureFrame;
28         }
29         catch (AccessViolationException)
30         {
31             return null;
32         }
33         catch (Exception)
34         {
35             return null;
36         }
37     }

It also uses hardware acceleration to copy 2D texture resources and then outputs them as byte data via memory copy.

The 1080P local recording and display, CPU and GPU usage is as follows:

There is no significant difference between the 1080P and WGC solutions, and the latency is close. However, at 4K and 8K resolutions, the DXGI solution is superior, being able to directly manage graphics hardware and provide high-performance rendering. It communicates with kernel mode drivers and system hardware, borrowing the architecture diagram from the official website:

So in 4K scenes where very low latency and high frame rates are required, DXGI can provide the necessary performance optimizations.

Above code example, detailed Demo see github:kybs00/CaptureImageDemo ()

To summarize the three options，

GDI: For all Windows versions, but with lower performance.

DXGI: Suitable for high performance requirements, higher complexity and learning costs, and only for screen recording, no window support.

WGC (Windows Graphics Capture): Win10 version 1803 or higher, high performance and low latency, both screen and window support.

Recording is mainly screen recording, live streaming, remote desktop, video conferencing, screen transfer and other scenarios. Recording screen/window is recommended to prioritize the use of WGC, then use DXGI compatible with win8