introductory
Three years ago, when writing a smart car, there was a camera on the car needs to be collected to achieve the operation of the push-pull flow, technology selection at that time the first version of the nginx rtmp push-pull flow, the server configuration environment is centos, 2H4G3M a configuration, nginx rtmp latency is 20 seconds, ultra-slow, and then studied the SRS as well as ZLMediaKit these two open source push-pull flow server, if I remember correctly, the two are based on the development of c++, performance is great, and then replaced the push-pull flow server, the latency of the car in less than a second, performance has increased dramatically, at the time of the study, and then did not record, this time the use of SRS to achieve the audio and video calls, as well as the sharing of desktop features, hereby record and share to the I'd like to share this with you.
Technology Overview
This demo involves a total of three services, the deployment environment is Tencent Cloud's lightweight servers, the configuration is 2H2G3M servers, CENTOS system, nginx deployment of the front-end, docker deployment of the back-end and SRS services.
SRS
SRS is an open source (MIT protocol) simple and efficient real-time video server that supports RTMP, WebRTC, HLS, HTTP-FLV, SRT, MPEG-DASH, and GB28181 protocols. SRS media server and FFmpeg, OBS, VLC, WebRTC and other clients to provide the ability to receive and distribute streams, is a typical publish (push stream) and subscription (play) server model. SRS supports the conversion of audio and video protocols widely used on the Internet, such as RTMP or SRT to HLS or HTTP-FLV or WebRTC.
back end
The back-end is based on anet8.0 base framework developed a webapi interface thatThe front-end and back-end communication uses thesignalr, used to implement a message change notification between different users, this project does not involve any database aspects.
forward part of sth.
The front end is based on thenode20.16.0，vue3。where the address of the push and pull streams and the address of the back-end server are configured in theIn the configuration, pushUrl is the address of the push stream, pullUrl is the address of the pull stream, apiUrl is the address of the back-end interface, and signalrUrl is the address of the front-end and back-end websocket communication. The ports in apiUrl and signalrUrl need to be consistent with the ports on which the backend is running in order to access and communicate with each other.See the legend below for specific configuration documentation。
Function Overview
The overall functionality is divided into a few:
Device testing/parameter setting
This page is mainly used to configure the camera and detect whether the camera is available or not, audio detect whether the audio is available or not, and configure the audio device, the subsequent audio and video calls are based on the device selected here for the push-pull stream, the user name, because this project is a demo, so it is all based on the memory, the user is set up by yourself, after the demo to communicate with the person a logo; in this page Configure the video device, audio device, and user name, then you can go to the subsequent page for group chat or single chat, or desktop sharing function.
group chat
Group chat into the page, you can see the existing group chat information, as well as the number of people online, you can join or delete the group chat operation, if there is a new group, here will also use Signalr to synchronize the update to get the latest list of group data, this demo does not determine the deletion of the time permissions and if someone is chatting with the group to delete the group, in the equipment detection page If you don't select the audio/video device, you can also enter the group, you can view an audio/video of the online personnel with audio/video, as shown in the figure, enter the name of the group you want to create at the top, click Add Group to get to the following page, enter the video page, it will first go to push the stream to the server, so that the next person who enters the video stream can see your video stream, at the same time, if you don't select the audio/video device, click Join to get the following page, and this page won't show the group's data, and the page won't show the video stream. If you don't select audio/video device, you can click join to get the following page, and the page will not show the online people of the group, you can check which user's audio/video from the top of the camera of the video.
chat
Single chat is only two people to make audio and video calls, but also supports no choice of device can make calls, no text chat support, single chat list, will also use signalr to synchronize update, in the user set the user name, here you can see the list of data updates. Click call to make an audio/video call with someone online, but you can't make a call with yourself, and you can't make an audio/video call with the person you selected if he/she is in a single chat. After selecting the user you want to make a call with, you will enter the chat page, and the other user will receive a reminder if he/she accepts the audio/video call, and he/she can either reject or accept it, and after rejecting the call, the requesting party will exit the page of the online list. After rejecting the call, the requesting party will exit to the online list page, and clicking Accept will enter an audio/video call page for both parties, which allows them to have a voice chat and watch the video stream pushed by the other party's webcam. After accepting, either party will leave the chat and the other party will exit the chat page. You will be returned to the user's online list.
shared desktop
The operation of the shared desktop is the same as that of the group, enter the name of the shared desktop at the top and click the button to enter the page of the shared desktop. Entering the page, it will prompt the screen content to be shared, you can choose the whole screen, browser or window can be, and you can also choose whether you want to share the audio of this page, after clicking Share, you can see the shared video content, the right side shows the list of participants, and if you choose the audio device, it will also capture the voice push, more than one participant in the voice chat, the voice will show who is speaking and Whose voice stream is received. At the same time, when the sharing person clicks Stop Sharing, other participants can also share their desktops, and when the sharing desktop button is clicked, the sharing desktop buttons of other participants will be hidden, and they will see the video information captured on your screen.
Program Configuration
About the configuration, mainly the front-end configuration and SRS configuration, front-end configuration, in the beginning has been elaborated, mainly SRS push and pull stream configuration, as well as API and Signalr address configuration, if the server is an extranet, the address must be the address of the extranet server, the configuration of SRS, after downloading SRS, SRS provides a default configuration folder, the name of the folder is called conf folder, which provides a variety of different functions of the conf configuration file, here we are based on the transformation of the following is the configuration information. folder, the folder name is called conf folder, which provides a variety of different functions of the conf configuration file, here we are based on the transformation, the following is the configuration information, the first line is listen, the default is 1935, on behalf of SRS startup RMP live service port. max_connection on behalf of the maximum number of connections, the default is 1000, srs_log_tank is the print mode of the log, the default is console, that is, console, optional configuration has file and console, file is the output form of the file, if the form of the file, you must configure the path of the print file of the log by configuring the srs_log_file parameter, the default is the file of the .objs /. The default is .objs/ file. daemon, on behalf of the form of whether to run in the background, on behalf of the on, off behalf of the off. http_api, SRS provides a number of api, this configuration represents whether to enable http_api, and port 8080. below, shows the SRS official website to provide the navigation of the API. rtc_server to enable the functionality of webrtc, port is udp 8000, and the port is udp 8000. The port is udp 8000, the default is 8000, which is the last configuration is candidata, if the local deployment, this can not need to configure, if you need to deploy the server outside the network, then this needs to be set to the server address of the outside network.
SRS's video push and pull streaming, there is the concept of virtual host, the default provides a defaultVhost virtual host in the configuration file, there is a separate configuration about the host within the virtual host, hls on behalf of turning on the hls video protocol function, http_remux, whether or not to open the live streaming service for http, mout on behalf of the mounted http streams, which can is a different format, the default is flv, optional ts, that is, the suffix is .ts, there are protocols such as mp3, aac, etc. rtc on behalf of whether to open the webrtc function for the virtual host, and whether to open the rtmp to rtc, as well as the rtc to rtmp function. http_hooks is an interface callback configured by the SRS in the case of a client pushing or pulling the stream. http_hooks is an interface callback configured by SRS when a client pushes or pulls a stream, this callback can be a back-end service written by us, the configuration here is the interface address of my actual back-end service, which can be adjusted according to your own actual situation, enable stands for turn on the HOOK, publish is the callback for pushing a stream, unpublish is the callback for stopping pushing a stream, and play and stop are the callbacks for pulling a stream to play and stop. play is the configuration for pulling the stream, gop_cache is whether to cache the last frame, if it is on, the client can play it quickly, if it is off, the client pulls the latest one all the time, queue_length is the length of the queue for caching the frames for one second, if it is more than this one, it will remove the previous frames, mw_latency, the delay of merging and writing about streams, the unit of millisecond. Publish is a configuration for pushing the stream, mr is the latency of merge reads, in milliseconds.
About the above said so many configurations, the main configuration or http_hooks configuration, this configuration, is the SRS in the receipt of the push stream request, or to stop pushing the stream, as well as pulling the stream, stop pulling the stream of the time to our back-end service of a callback, enabled on behalf of whether or not to open the callback, publish is the start of pushing the stream of the callback, unpublish is to stop the push stream play and stop are the callbacks for the client to play or stop the stream, if deployed in an extranet and the srs and backend services are on the same server, the address can be the LAN address, which is different from the front-end configuration, the front-end configuration must be the extranet address.
The full configuration of the above on all can be found at the following URL:
http_api:/lts/zh-cn/docs/v6/doc/http-api。
http_server:/lts/zh-cn/docs/v6/doc/http-server。
rtc_server/rtc:/lts/zh-cn/docs/v6/doc/webrtc。
hls:/lts/zh-cn/docs/v6/doc/hls。
deployments
As mentioned above, our release environment is centos, nginx and docker, so after installing nginx, we need to configure a port reverse proxy for nginx, here I use port 443, and generate an ssl certificate on the Internet, because if you use a URL that begins with http, the browser mechanism will not allow the use of the webcam, and you need to change the browser configuration to make the http URL can access the webcam, because if you deploy and debug this service locally, it is recommended to use 127.0.0.1 or localhost to obtain the webcam information. change the browser configuration to make the http URL can access the camera, if the local deployment and debugging of this service, it is recommended to use 127.0.0.1 or localhost and other ways to get the camera's information, because the browser's security mechanism, non-https and the url address is the IP address is not 127.0.0.1 or localhost The way to get the webcam is restricted, the way to contact the restriction is to enter chrome://flags in the browser address, enter, search for Insecure origins treated as secure in the pop-up page, as shown in the figure, enter, you need to enter the IP address of the service in the input box below, and change the disabled to enabled, after the change is complete, the browser will appear below the panel in Figure 3, click Relauch to restart the browser can be accessed through the IP address of the service, and load the camera, if the Chinese disable is disabled, enable is enabled, Relauch is restarted.
So based on the http issue above, an ssl certificate needs to be generated to allow the web to be accessed using https. Below I put a copy of my nginx configuration, in the configuration of 443 below, specified servicename, as well as root points to the address of the web front-end after we publish. Specified, and ssl certificate related configuration address, configure the root path to access the web front-end, and then the route including api, pull, push, pointing to the corresponding port, such as the above picture of the port configuration, 1985, using webrtc to push the flow, 8080, to pull the flow of the port, video is the reverse proxy address of the signalr.
The above is a configuration of nginx deployed on my server itself.
# For more information on configuration, see:
# * Official English Documentation: /en/docs/
# * Official Russian Documentation: /ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/;
pid /run/;
# Load dynamic modules. See /usr/share/doc/nginx/.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/ main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 4096;
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
include /etc/nginx/;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/ directory.
# See /en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx//*.conf;
server {
listen 80;
listen [::]:80;
server_name _;
root /video/dist/;
index ;
# Load configuration files for the default server block.
include /etc/nginx//*.conf;
location / {
try_files $uri $uri/ /;
}
error_page 404 /;
location = / {
}
error_page 500 502 503 504 /;
location = / {
}
}
server {
listen 443 ssl;
listen [::]:443;
server_name 49.233.22.57;
root /video/dist/;
index ;
ssl_certificate /sslfile/49.233.22.; # ssl certificate storage path
ssl_certificate_key /sslfile/49.233.22.; # Secret key storage path
# Some configuration of ssl
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 5m;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Enable the TLS protocol
location / {
try_files $uri $uri/ /;
}
location /api {
proxy_pass http://10.2.20.12:5000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}
location /rtc {
proxy_pass http://10.2.20.12:1985;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}
location /pull {
rewrite ^/pull/(.*)$ /$1 break;
proxy_pass http://10.2.20.12:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}
location /video {
proxy_pass http://10.2.20.12:5000;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_cache_bypass $http_upgrade != '';
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}
}
# Settings for a TLS enabled server.
#
# server {
# listen 443 ssl http2;
# listen [::]:443 ssl http2;
# server_name _;
# root /usr/share/nginx/html;
#
# ssl_certificate "/etc/pki/nginx/";
# ssl_certificate_key "/etc/pki/nginx/private/";
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 10m;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
#
# # Load configuration files for the default server block.
# include /etc/nginx//*.conf;
#
# error_page 404 /;
# location = / {
# }
#
# error_page 500 502 503 504 /;
# location = / {
# }
# }
}
The backend is deployed in docker, the environment is the development environment of net8, the dockerfile configuration is as follows, put the released file and dockerfile together, execute docker build -t videoimg . command to package the docker image. Remember there is a dot . After entering, the docker image of videoimg is generated. After generating the image, you need to start the container, docker run --name videoc --security-opt seccomp=unconfined -d -p 5000:8080 videoimg After entering, you can start the container of the back-end service. If the local port is 5000, you need to open port 5000 in your firewall or cloud server.
#See /customizecontainer to learn how to customize your debug container and how Visual Studio uses this Dockerfile to build your images for faster debugging. FROM /dotnet/aspnet:8.0 AS base WORKDIR /app EXPOSE 8080 EXPOSE 8081 COPY . . FROM base AS final WORKDIR /app ENTRYPOINT ["dotnet", ""]
docker to start SRS, if SRS is deployed on an external server, you need to configure an environment variable CANDIDATE="IP address of external server" before starting the SRS container, write the IP address of the external server into the environment variable, and SRS startup will read the configuration from it, or you can directly modify the configuration of the CANDIDATE in the conf to directly change it to the address of your external server. You can also directly modify the configuration of CANDIDATE in conf to change it to the address of your external server. After configuration, start SRS, docker run --rm --name srs -it -d -p 1935:1935 -p 1985:1985 -p 8080:8080 -p 1990:1990 -p 8088:8088 -p 8000:8000/udp -v /video/conf:/conf -- env CANDIDATE=$$CANDIDATE env CANDIDATE=$CANDIDATE /ossrs/srs:5 . /objs/srs -c /conf/ The above command, -v, mounts the container's internal conf folder, and then later specifies that it uses the config file as the startup, and exposes the port information, which is required for push and pull streams, as well as web servers.
SRS provides a web management system, after the firewall and other configurations, enter IP:8080 in the browser to open the following page, in the console you can see the video of the push and pull streaming information, as well as the inflow and outflow of traffic.
Port Configuration
The server needs to open up a few ports
1: The port of the back-end service, e.g. port 5000 in this example
2: SRS related ports: 1935, 1985, 8080, 1990, 8088, 8000/udp
Code and detailed Windows documentation address
For a more detailed explanation, check out the documented deployment address for written windows, it's all the same except for the differences in deployment methods.
git address:/cxd199645/
summarize
On doing an audio-video call technology to share here, if and do not understand, you can add QQ/WX:934550201.
1. SRS
SRS is an open source (MIT protocol) Simple and efficient real-time video server that supportsRTMP, WebRTC, HLS, HTTP-FLV, SRT, MPEG-DASH and GB28181 protocols. SRS media server andFFmpeg、OBS、VLC、 WebRTCand other clients work together to provideReceipt and distribution of streamsThe ability to be a typical release (push streaming) and subscription (playback) server models. SRS supports the conversion of audio and video protocols that are widely used on the Internet, such as the ability to convert theRTMPmaybeSRT， convertHLSmaybeHTTP-FLVmaybeWebRTCand other agreements.