NET Camera Capture

This paper focuses on how cameras (camcorders) capture data for use in camera-like local display software, as well as streaming data transfer scenarios such as screen transfer and video conferencing.

Camera capture has a variety of programs, such as, WPFMediaKit, OpenCvSharp, EmguCv,, MediaCaptre (UWP), some online articles as well as the github has been a lot of introduction, summarized here

1.

AForge video library is based on DirectShow technology development , provides capture , processing and display of video streaming interfaces , as well as image-rich image processing features such as filters , feature extraction and object detection . See the official website for details open source repositoryandrewkirillov/()

We'll look at the AForge recording code below, theInstall the Nuget package dependencies:

1     <PackageReference Include="" Version="2.2.5" />
2     <PackageReference Include="" Version="2.2.5" />
3     <PackageReference Include="" Version="8.0.8" />

The camera shows:

 1     private void StartButton_OnClick(object sender, RoutedEventArgs e)
 2     {
 3         // Get all video input devices
 4         var videoDevices = new FilterInfoCollection();
 5         if ( > 0)
 6         {
 7             // Select the first video input device
 8             var videoSource = new VideoCaptureDevice(videoDevices[0].MonikerString);
 9             // Register NewFrame event handler
10              += new NewFrameEventHandler(videoSource_NewFrame);
11             // Start camera video feed
12             ();
13         }
14     }
15 
16     private async void videoSource_NewFrame(object sender, NewFrameEventArgs eventArgs)
17     {
18         // Get the current video frame to display
19         var image = ToBitmapImage();
20         await (() => {  = image; });
21     }

The camera records the video stream:

 1     private async void videoSource_NewFrame1(object sender, NewFrameEventArgs eventArgs)
 2     {
 3         // Get the current video frame
 4         // Convert Bitmap to byte[] for Streaming
 5         byte[] byteArray = BitmapToByteArray(, out int stride);
 6         // Convert byte[] to BitmapImage for temporary presentation
 7         BitmapImage image = ByteArrayToBitmapImage(byteArray, , , stride, );
 8         await (() => {  = image; });
 9     }

One of the data conversion, here the resolution stride with byte [] data to be stored together, otherwise the subsequent data can not be processed:

 1     // Convert Bitmap to byte[]
 2     public byte[] BitmapToByteArray(Bitmap bitmap, out int stride)
 3     {
 4         Rectangle rect = new Rectangle(0, 0, , );
 5         BitmapData bitmapData = (rect, , );
 6         //stride is the resolution level value, e.g. 3840
 7         stride = ;
 8         int bytes = () * ;
 9         byte[] rgbValues = new byte[bytes];
10 
11         // Copy bitmap data to byte array
12         (bitmapData.Scan0, rgbValues, 0, bytes);
13 
14         (bitmapData);
15         return rgbValues;
16     }
17 
18     // Convert byte[] to BitmapImage
19     public BitmapImage ByteArrayToBitmapImage(byte[] byteArray, int width, int height, int stride, PixelFormat pixelFormat)
20     {
21         var bitmapImage = new BitmapImage();
22         using (var memoryStream = new MemoryStream())
23         {
24             var bmp = new Bitmap(width, height, stride, pixelFormat, (byteArray, 0));
25             // Save to MemoryStream
26             (memoryStream, );
27             (0, );
28             ();
29              = memoryStream;
30              = ;
31             ();
32             ();
33         }
34         return bitmapImage;
35     }

See Demo code for detailskybs00/AForgeNETDemo ()

Verification.Large latency and lack of clarity compared to Windows cameras

2. WPFMediaKit

WPFMediaKit is also based on DirectShow , it provides some packaging to facilitate the use of media features in WPF applications .

Using WPFMediaKit To record camera video, you need to combine the video capture functionality provided by WPFMediaKit with other libraries (e.g. AForge or FFmpeg) to realize the recording functionality.

Here we refer to WPFMediaKit, two Nuget packages, and then capture the current video frame by timer:

 1     var bitmap = new Bitmap(, , .Format24bppRgb);
 2     var bitmapData = (new Rectangle(0, 0, , ),,);
 3     try
 4     {
 5         (out IntPtr frame);
 6         (frame, 0, bitmapData.Scan0,  *  * 3);
 7     }
 8     finally
 9     {
10         (bitmapData);
11     }

The implementation of this timer is rather low.Lower latency and smoother, but also lacks clarity compared to Win system cameras

（UWP）

MediaCapture is a WinRT API for Windows 8 and above, designed for capturing audio, video and photos.

MediaCaptuer is an API for UWP apps!Capture basic photos, video and audio with MediaCapture - UWP applications | Microsoft LearnTo use it in WPF, you need to introduce two Nuget packages:

1     <PackageReference Include="" Version="6.1.2" />
2     <PackageReference Include="" Version="10.0.26100.1" />

Initialize MediaCapture.

1     var mediaCapture =new MediaCapture();
2     var videos = await ();
3     var settings = new MediaCaptureInitializationSettings()
4     {
5         VideoDeviceId = videos[0].Id,
6         StreamingCaptureMode = ,
7     };
8     await (settings);

In several scenarios, respectively, the output Demo. recording local files, the notes are already very detailed and do not repeat the explanation, directly look at the code:

 1     private MediaCapture _mediaCapture;
 2     private InMemoryRandomAccessStream _randomAccessStream;
 3     private async void StartButton_OnClick(object sender, RoutedEventArgs e)
 4     {
 5         // 1. Initialize MediaCapture object
 6         var mediaCapture = _mediaCapture = new MediaCapture();
 7         var videos = await ();
 8         var settings = new MediaCaptureInitializationSettings()
 9         {
10             VideoDeviceId = videos[0].Id,
11             StreamingCaptureMode = ,
12         };
13         await (settings);
14 
15         // 2. Setting up the data stream to be recorded
16         var randomAccessStream = _randomAccessStream = new InMemoryRandomAccessStream();
17         // 3. Configure recorded video settings
18         var mediaEncodingProfile = MediaEncodingProfile.CreateMp4();
19         // 4. Starting recording
20         await (mediaEncodingProfile, randomAccessStream);
21     }
22 
23     private async void StopButton_OnClick(object sender, RoutedEventArgs e)
24     {
25         // Stop recording
26         await _mediaCapture.StopRecordAsync();
27         // Process the recorded data and save it to "C:\Users\XXX\Videos\RecordedVideo.mp4".
28         var storageFolder = ;
29         var file = await ("RecordedVideo.mp4", );
30         using var fileStream = await ();
31         await (_randomAccessStream.GetInputStreamAt(0), (0));
32         _randomAccessStream.Dispose();
33     }

Camera display, hosting screen (top) via UWP-WindowsXamlHost:

 1     private async void StartButton_OnClick(object sender, RoutedEventArgs e)
 2     {
 3         _mediaCapture = new MediaCapture();
 4         var videos = await ();
 5         var settings = new MediaCaptureInitializationSettings()
 6         {
 7             VideoDeviceId = videos[0].Id,
 8             StreamingCaptureMode = ,
 9         };
10         await _mediaCapture.InitializeAsync(settings);
11         //Show WindowsXamlHost
12          = ;
13         //Binding screen source
14         _captureElement.Source = _mediaCapture;
15         await _mediaCapture.StartPreviewAsync();
16     }

MediaCapture is a UWP platform implementation that binds the screen source directly to the CaptureElement assignment.Rendering directly with CaptureElement is very fast, and the logic of this implementation is the same as the windows camera.

Alternatively, use MediaCapture to capture screen frame events for streaming data capture collection:

1 // Configuring the Video Frame Reader
2 var frameSource = (source =>  == );
3 _frameReader = await (frameSource, MediaEncodingSubtypes.Argb32);
4 _frameReader.FrameArrived += FrameReader_FrameArrived;
5 await _frameReader.StartAsync();

As shown below, listen to FrameArrived and use the rendered display (for display only, with high latency):

 1     private async void FrameReader_FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
 2     {
 3         var frame = ();
 4         if (frame != null)
 5         {
 6             var bitmap = ?.SoftwareBitmap;
 7             if (bitmap != null)
 8             {
 9                 // Process each frame here
10                 await (async () =>
11                 {
12                     var bitmapImage = await ConvertSoftwareBitmapToBitmapImageAsync(bitmap);
13                     _captureImage.Source = bitmapImage;
14                 });
15             }
16         }
17     }

If you need to convert SoftwareBitmap to buffer byte data, you can process it as follows:

 1     public async Task<byte[]> SoftwareBitmapToByteArrayAsync(SoftwareBitmap softwareBitmap)
 2     {
 3         // Using InMemoryRandomAccessStream to store image data
 4         using var stream = new InMemoryRandomAccessStream();
 5         // Creating a Bitmap Encoder
 6         var encoder = await (, stream);
 7         // Convert to BGRA8 format, if the current format is different
 8         var bitmap = (softwareBitmap, BitmapPixelFormat.Bgra8, );
 9         (bitmap);
10         await ();
11         ();
12 
13         // Read byte data
14         using var reader = new DataReader((0));
15         byte[] byteArray = new byte[];
16         await ((uint));
17         (byteArray);
18 
19         return byteArray;
20     }

Above, MediaCapture can realize the camera display and recording related functions.MediaCapture code demo details see Github:kybs00/CameraCaptureDemo: camera preview, capture DEMO ()

3. Other

OpenCvSharp is a wrapper for OpenCV in the C# environment , providing cross-platform computer vision and image processing capabilities .2K video is smoother, 4K video has lower latency but poorer displays

Others such as EmguCV is another C# wrapper component library based on OpenCV with the same powerful features as OpenCVSharp.

NET Framework can directly use the powerful features of DirectShow for video capture and processing. DirectShow itself is better, but as a hosted package, performance will be affected. Delay effect to be verified

I won't write a demo here.

Let's take a look at the performance data, 4K screen device + 4K camera, preview display via local camera. Borrowed from the group's partner, Big Brother Jiankai's latency statistics for each program:

Verified that the MediaCaptre latency is about the same as the system camera. Through the task manager we can see that the system camera CPU usage is 3%, but the GPU is 15%. With hardware acceleration, the performance is very good, so we recommend the MediaCaptre solution for camera acquisition.

Reference List:

Capture basic photos, video and audio with MediaCapture - UWP applications | Microsoft Learn

[C#] Use, implement camera frame capture, video save and crop video area, efficient rendering with WriteableBitmap - Lonely Into Pie - Blogland ()

How to Capture Camera Video via .Net and Comparison (How to Capture Camera Video via .Net) - Wuya - Blogland ()

C# Using AForge for camera information acquisition - I know the way. - blogosphere ()

WPF Camera Usage (WPFMediaKit) - Late Autumn Without Trace - Blogland ()

C# use OpenCvSharp4 library to read computer camera data and display it in real time_c#Display camera screen-CSDN Blog

SharpCaptureDemo: C#, vb, .net capture camera , desktop screen , microphone microphone sound , camera screen , and support for mixing , but also supports simultaneous capture recording . High efficiency, the underlying directshow technology, stability and compatibility. ()