ffmpeg simple player (3)--use ffmpeg to decode video and use opencv to display video

Installation of ffmpeg

Here I installed ffmpeg by compiling the source code under Linux. Of course, you can also use apt-get to install it. However, when I used apt-get to install ffmpeg, I couldn't find the ffmpeg library using cmake, so I chose to compile it. Source code method.

goOfficial websiteDownload the latest source code package

Then unzip it into the folder and configure it using the built-in configure program.

./configure --prefix=/path/to/your/ffmpeg

Only the installation path is configured here. This path determines where ffmpeg will be installed after make install. Other configuration options can be viewed through ./configure --help or searched online.

Then enter in the terminal

make -j4
 # After executing the previous step, enter
 make install
 # If the specified directory requires root permissions, don’t forget to add sudo.

Add ffmpeg using cmake

Since ffmpeg does not provide a similar configuration file, it uses pkg-config for configuration.

Generally speaking, the pkg-config configuration file .pc or the cmake configuration file will be placed in the /lib/pkgconfig or /lib/cmake/XXX/ directory of the library installation package.

Since I wanted to use cmake to manage packages, and the size of ffmpeg itself is actually very small, I wrote a file directly.

# 
set(FIND_FFMPEG TRUE)
set(FFMPEG_SOURCE /path/to/your/ffmpeg)

set(FFMPEG_INCLUDE_DIRS ${FFMPEG_SOURCE}/include)
set(FFMPEG_LIBDIRS_DIRS ${FFMPEG_SOURCE}/lib)

find_library(FFMPEG_AVCODEC_LIBRARY avcodec ${FFMPEG_LIBDIRS_DIR})
find_library(FFMPEG_AVFORMAT_LIBRARY avformat ${FFMPEG_LIBDIRS_DIR})
find_library(FFMPEG_AVUTIL_LIBRARY avutil ${FFMPEG_LIBDIRS_DIR})
find_library(FFMPEG_SWSCALE_LIBRARY swscale ${FFMPEG_LIBDIRS_DIR})
find_library(FFMPEG_SWRESAMPLE_LIBRARY swresample ${FFMPEG_LIBDIRS_DIR})

set(FFMPEG_LIBS ${FFMPEG_AVCODEC_LIBRARY} ${FFMPEG_AVFORMAT_LIBRARY} ${FFMPEG_AVUTIL_LIBRARY} ${FFMPEG_SWSCALE_LIBRARY} ${FFMPEG_SWRESAMPLE_LIBRARY})

The main method is to find the ffmpeg library file through find_library, and then set the variables through set. Place this configuration file in the same directory as

#
 ......# omitted
 # start
 ...
 set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${CMAKE_CURRENT_SOURCE_DIR})

 find_package(FFMPEG REQUIRED)

 include_directories(${FFMPEG_INCLUDE_DIRS})

 ......#omitted
 # Generate executable file
 ...

 target_link_libraries(${PROJECT_NAME} ${FFMPEG_LIBS})

This contains references to several libraries I want to use. If you want to use other libraries, you can add the corresponding find_library and set the corresponding variables.

As for why I do this, you can read my other articlefindpackage() usage guide。

As for ffmpeg installed using apt install ffmpeg, you can search for how to reference the packages managed by pkg-config through cmake, or you can refer to my other articleUse of zbar libraryHow to reference the zbar library.

Using ffmpeg

Since ffmpeg is written in pure C language, and for C language, there is no function overloading, so the function signature generated when the pure C file is compiled is the function name. In C++, due to function overloading, the function name will be reorganized (called symbol modification) during compilation. The modified function name will contain information such as the type and number of function parameters. This results in different function signatures generated when compiling c files and c++ files, causing the compiler to not be able to find the corresponding function entry when connecting.

Therefore, if you want to use ffmpeg's function in a C++ file, you need to use extern "C" modification in the C++ file to tell the compiler that this is a C language function and do not modify it symbolically.

extern "C"
 {
 #include 
 #include 
 ...
 }

 ## Common structures

 When using ffmpeg to decode audio and video, we need to use some commonly used structures. Here is a brief introduction.
 ```c++
 AVFormatContext //Format context, used to store audio and video format information such as the number of audio and video streams, encoding information of audio and video streams, etc.
 AVCodecContext //Coding and decoding context, used to store codec and parameter information during coding and decoding, such as codec type, codec parameters, etc.
 AVCodec // Codec, used to store codec information, such as codec name, codec type, etc.
 AVPacket //Storage encoded audio and video data
 AVFrame //Storage decoded audio and video data
 SwsContext //Context for image conversion, which can convert images in different formats into the format we need
 SwrContext //Context for audio conversion, which can resample audio in different formats into the format we need

Decode video

extern "C" 
{
#include <libavformat/>
#include <libavcodec/>
#include <libswscale/>
#include <libavutil/>
}
int main()
{
    .......
}

This completes the preparations. The next step is to obtain video information, open the video file, obtain the index of the video stream, and other operations.

// av_register_all() The first sentence of many tutorials is this, but it has been deprecated after ffmpeg4.0 and does not need to be called anymore.
     AVFormatContext *pFormatCtx = nullptr;
     /*Find the information in the audio and video files and pass it into the AVFormatContext structure*/
     if (avformat_open_input(&pFormatCtx, videoPath.c_str(), nullptr, nullptr) != 0)
     {
         //Note that AVFormatContext must be a null pointer (nullptr) or memory allocated by avformat_alloc_context(), otherwise an error will be reported.  The former allocates memory by the above function.
         cerr << "Failed to open video file" << endl;
         return -1;
     }
     /*Get stream information, including video stream, audio stream, etc.*/
     if (avformat_find_stream_info(pFormatCtx, nullptr) < 0)
     {
         cerr << "Failed to obtain information" << endl;
         return -1;
     }
     /*Print video information*/
     av_dump_format(pFormatCtx, 0, videoPath.c_str(), 0);

     /*Get the index of the video stream*/
     videoStreamIdx = -1;
     for (int i = 0; i < pFormatCtx->nb_streams/*number of streams*/; i++)
     {
         if (pFormatCtx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) /*If it is a video stream, find the record index and exit*/
         {
             videoStreamIdx = i;
             break;
         }
     }
     if (videoStreamIdx == -1)
     {
         cerr << "Video stream not found" << endl;
         return -1;
     }

Then get the decoding context for decoding

/*Get decoder*/
     AVCodec *pCodec = nullptr;
     AVCodecParameters *pCodecParameters = pFormatCtx->streams[videoStreamIdx]->codecpar;
     pCodec = avcodec_find_decoder(pCodecParameters->codec_id);
     if (pCodec == nullptr)
     {
         cerr << "Decoder not found" << endl;
         return -1;
     }
     /*Get decoding context*/
     AVCodecContext *pCodecCtx = avcodec_alloc_context3(pCodec);
     if (avcodec_parameters_to_context(pCodecCtx, pCodecParameters) < 0)
     {
         cerr << "Failed to obtain decoding context" << endl;
         return -1;
     }
     /*Open decoder*/
     if (avcodec_open2(pCodecCtx, pCodec, nullptr) < 0)
     {
         cerr << "Failed to open decoder" << endl;
         return -1;
     }

Then the data is decoded and displayed, here using opencv for display.

/*Initialize the conversion context and convert yuv to rgb*/
     SWsContext *pSwsCtx = sws_getContext(pCodecCtx->width, pCodecCtx->height,
                              pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height,
                              AV_PIX_FMT_RGB24, SWS_BILINEAR, nullptr, nullptr, nullptr);

     /*Allocate container memory*/
     AVFrame *pFrame = av_frame_alloc();
     AVFrame *pFrameRGB = av_frame_alloc();
     AVPackeat *pPacket = av_packet_alloc();
     uint8_t *buffer = nullptr;
     // Get the size of a frame of image
     int bufferSz = av_image_get_buffer_size(AV_PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height, 1);
     buffer = (uint8_t *)av_malloc(bufferSz); // Memory to store one frame of image
     // Bind the buffer to pFrameRGB and specify the storage format as RBG24 bit
     av_image_fill_arrays(pFrameRGB->data, pFrameRGB->linesize, buffer, AV_PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height, 1);
     //Read a packet
     while (av_read_frame(pFormatCtx, pPacket) >= 0)
     {
         if (pPacket->stream_index == videoStreamIdx)
         {
             //Send the packet to the decoder
             int ret = avcodec_send_packet(pCodecCtx, pPacket);
             if (ret < 0)
             {
                 cerr << "Sending failed" << endl;
                 return -1;
             }
             while (ret >= 0)
             { // Get a decoded frame from the decoder
                 ret = avcodec_receive_frame(pCodecCtx, pFrame);
                 if (ret == AVERROR_EOF)
                 {
                     /*Exit if the end is reached*/
                     break;
                 }
                 // Convert, convert yuv format to RBG
                 sws_scale(pSwsCtx, (uint8_t const *const *)pFrame->data, pFrame->linesize, 0, pCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);

                 // Convert data to opencv format
                 Mat img(pCodecCtx->height, pCodecCtx->width, CV_8UC3, pFrameRGB->data[0]);
                 // Note that the color space in opencv is BGR
                 cvtColor(img, img, COLOR_RGB2BGR);
                 imshow("video", img);
                 waitKey(1);
             }
         }
         av_packet_unref(pPacket);
     }

     // Release resources
     av_frame_free(&pFrame);
     av_frame_free(&pFrameRGB);
     av_packet_free(&pPacket);
     avcodec_free_context(&pCodecCtx);
     avformat_close_input(&pFormatCtx);
     sws_freeContext(pSwsCtx);
     av_free(buffer);
     return 0;