OpenCV development notes (seventy-nine): based on the Stitcher class to achieve panoramic image stitching

preamble

  When one camera field of view is not large, we want to perform a merge of the two fields of view, which allows the front view to increase so that a wider standard field of view can be seen. The method of splicing is divided into two paths, the first path is the stitcher class and the second idea is feature point matching.
  This one uses stitcher matching for two graphs to field of view merge and splice.

Demo

The process of stitching the two images together

Step 1: Open the image

cv::Mat mat = cv::imread("D:/qtProject/openCVDemo/openCVDemo/modules/openCVManager/images/");
cv::Mat mat2 = cv::imread("D:/qtProject/openCVDemo/openCVDemo/modules/openCVManager/images/");

Step 2: Add images to the queue

std::vector<cv::Mat> vectorMat;
vectorMat.push_back(mat);
vectorMat.push_back(mat2);

Step 3: Create the splice class

cv::Ptr<cv::Stitcher> pStitcher = cv::Stitcher::create(cv::Stitcher::PANORAMA, false);
//cv::Ptr<cv::Stitcher> pStitcher = cv::Stitcher::create(cv::Stitcher::SCANS, false);

Step 4: Splicing

cv::Ptr<cv::Stitcher> pStitcher = cv::Stitcher::create(cv::Stitcher::SCANS, false);
LOG;
cv::Stitcher::Status status = pStitcher->stitch(vectorMat, resultMat);
LOG;
if(status != cv::Stitcher::OK)
{
    std::cout << "Failed to stitch, status =" << status << std::endl;
    return;
}

  For splices showing all:
  

cv::namedWindow("mat", cv::WINDOW_NORMAL);
cv::imshow("mat", mat);
cv::resizeWindow("mat", cv::Size(400, 300));

cv::namedWindow("mat2", cv::WINDOW_NORMAL);
cv::imshow("mat2", mat2);
cv::resizeWindow("mat2", cv::Size(400, 300));

cv::namedWindow("resultMat", cv::WINDOW_NORMAL);
cv::imshow("resultMat", resultMat);
cv::resizeWindow("resultMat", cv::Size(400, 300));

Step 5: Crop the image in width and height with black border (omitted)

  Directly write an algorithm for rectangular detection of the surrounding black area, and then crop can be, there are many ways to do this, usually we take pictures are not all black, and the black edge is all black, this algorithm is written with obvious features.

time-consuming test

Original image 1701x1280 pixels, took about 477ms

  The original image is 1701x1280 pixels and the stitching consumes about 477ms:

Image reduced to 400x300 pixels in about 390ms

  Then test its time consumption after scaling its images:
  

  Scaling the images uniformly to 800x600 resolution takes around 390ms for their stitching.

Image enlargement to 1920x1080 pixels takes about 530ms

  Scaling up the image to 1920x1080 resolution takes around 530ms for stitching

take note of

  This test is not rigorous, based on the scaling of the same figure, simply control the pixel scaling to compare, but the conclusions drawn can reflect the impact of the image size, the final elapsed time is affected by a number of factors, including, but not limited to, the number of detected feature electricity, the size of overlapping regions, pixel resolution, multiple images.

reach a verdict

  This way is suitable for splicing photos, after the black edge processing, the effect is very good, but, call stitcher class implementation of the picture feature matching requirements, some feature points are not enough pictures can not be spliced, and, when the picture is larger or more pictures spliced, the speed is slow. So, if you put it on video, 25-60fps a second, it will definitely not work.
  SIFT algorithm splicing, SIFT algorithm can provide high accuracy, the obtained images need to be processed again to get a relatively good image, the
  ORB algorithm splicing, the algorithm is very fast, but the most prone to problems, and the resulting image needs to be processed again to get a relatively good image, the

function prototype

Function cv::Stitcher::create

static Ptr<Stitcher> create(Mode mode = PANORAMA, bool try_use_gpu = false);

• Parameter 1: Splice mode enumeration, only 2 values PANORAMA and SCANS
  PANORAMA: A mode that creates a panoramic view of a photo, expecting the image to be in perspective;
  SCANS: mode for synthetic scanning. Expect an image under affine transformation with no exposure compensation by default. (Since we generally always have an angular offset, this mode has high requirements for stitching images)
• Parameter 2: whether to use gpu, this way to compile opencv have to bring gpu compilation, compile opencv open support gpu, on arm, you need to confirm whether the chip supports GPU, and then install the GPU driver, and then compile opencv to support the GPU option, only then you can.

Function cv::Stitcher::stitch

CV_WRAP Status stitch(InputArrayOfArrays images, OutputArray pano);

• Parameter 1: Input image list
• Parameter 2: Output splicing result

Status stitch(InputArrayOfArrays images, const std::vector<std::vector<Rect> > &rois, OutputArray pano);

• Parameter 1: Input image list
• Parameter 2: input image list in order of the region to be stitched
• Parameter 3: Output splicing result

Demo Source Code

void OpenCVManager::testStitchImages()
{
    cv::Mat mat = cv::imread("D:/qtProject/openCVDemo/openCVDemo/modules/openCVManager/images/");
    cv::Mat mat2 = cv::imread("D:/qtProject/openCVDemo/openCVDemo/modules/openCVManager/images/");

#if 0
    // Stitching a circular panorama, the feature points are completely insufficient to use this method, and at the same time, even if they could be stitched, they would not be able to meet the new energy requirements.
    cv::Mat mat = cv::imread("D:/qtProject/openCVDemo/openCVDemo/modules/openCVManager/images/front_2024-08-22_17-15-08_result.png");
    cv::Mat mat2 = cv::imread("D:/qtProject/openCVDemo/openCVDemo/modules/openCVManager/images/left_2024-08-22_17-15-10_result.png");
    cv::Mat mat2 = cv::imread("D:/qtProject/openCVDemo/openCVDemo/modules/openCVManager/images/right_2024-08-22_17-15-11_result.png");
#endif

#if 1
    // Scale the image to test its stitching time.
    cv::resize(mat, mat, cv::Size(1920, 1080));
    cv::resize(mat2, mat2, cv::Size(1920, 1080));
#endif

    std::vector<cv::Mat> vectorMat;
    vectorMat.push_back(mat);
    vectorMat.push_back(mat2);


    cv::Mat resultMat;

    cv::Ptr<cv::Stitcher> pStitcher = cv::Stitcher::create(cv::Stitcher::PANORAMA, false);
//    cv::Ptr<cv::Stitcher> pStitcher = cv::Stitcher::create(cv::Stitcher::SCANS, false);
    LOG;
    cv::Stitcher::Status status = pStitcher->stitch(vectorMat, resultMat);
    LOG;
    if(status != cv::Stitcher::OK)
    {
        std::cout << "Failed to stitch, status =" << status << std::endl;
        return;
    }


    cv::namedWindow("mat", cv::WINDOW_NORMAL);
    cv::imshow("mat", mat);
    cv::resizeWindow("mat", cv::Size(400, 300));

    cv::namedWindow("mat2", cv::WINDOW_NORMAL);
    cv::imshow("mat2", mat2);
    cv::resizeWindow("mat2", cv::Size(400, 300));

    cv::namedWindow("resultMat", cv::WINDOW_NORMAL);
    cv::imshow("resultMat", resultMat);
    cv::resizeWindow("resultMat", cv::Size(400, 300));

    cv::