TWI798867B - Video processing method and associated system on chip - Google Patents

Abstract

The present invention provides a SoC, wherein the SoC includes a human recognition circuit, a sound detection circuit and a processing circuit. The human recognition circuit is used to obtain an image data from an image capturing device in real time, and perform human recognition on the image data to generate a recognition result. The sound detection circuit is used to obtain a plurality of sound signals from a plurality of microphones, to determine a voice characteristic value of a main sound. The processing circuit is coupled to the human recognition circuit and the sound detection circuit, and is configured to determine a specific region of the image data according to the recognition result and the voice characteristic value of the main sound, and the processing circuit processes the image data to highlight the specific area.

Description

Video processing method and related system chip

The invention relates to a video processing method for real-time streaming.

Live streaming (live streaming) is currently widely used in many aspects of society, for example, it can be applied in remote video conferencing. However, when one of the parties in the remote video conference includes multiple participants in the image frame, sometimes it may be difficult for the other party's participants to distinguish who is speaking in the image frame. Specifically, assume that the first party and the second party are conducting a remote video conference, and the first party has multiple participants in the physical conference room, and captures the video and audio information of the physical conference room through a microphone and a camera If it is transmitted to the remote second-party participants through the network, due to the posture and position of the first-party participants, the second-party participants may not be able to see which one is speaking, resulting in Participants of the second party are disturbed and affect the efficiency of the meeting.

Therefore, one of the objectives of the present invention is to propose a person tracking technology applied to remote video communication, which can emphasize the person who is currently speaking in the video frame, so as to solve the problems described in the prior art.

In one embodiment of the present invention, a system chip is disclosed, which includes a person recognition circuit, a sound detection circuit and a processing circuit. The person recognition circuit is used to obtain an image data from an image capture device in real time, and perform person recognition on the image data to generate a recognition result; the sound detection circuit is used to obtain a plurality of sound signals from a plurality of microphones, to determine a sound characteristic value of a main sound; and the processing circuit is coupled to the person identification circuit and the sound detection circuit, and is used for judging according to the recognition result and the sound characteristic value of the main sound A specific area in the image data is extracted, and the image data is processed to emphasize the specific area.

In one embodiment of the present invention, a video processing method is disclosed, which includes the following steps: obtaining an image data from an image capture device in real time, and performing character recognition on the image data to generate a recognition result; A plurality of microphones obtain a plurality of sound signals to determine a sound feature value of a main sound; according to the recognition result and the sound feature value of the main sound, a specific area in the image data is judged; and for The image data is processed to emphasize the specific area.

110: Electronic device

120: electronic device

200: system chip

202: image capture device

204_1~204_N: Microphone

210: Character identification circuit

220: Voice activity detection circuit

230: Sound direction detection circuit

240: Processing circuit

300~314: steps

410~450: area

FIG. 1 is a schematic diagram of a remote video conference.

FIG. 2 is a schematic diagram of an electronic device according to an embodiment of the present invention.

FIG. 3 is a flowchart of a video processing method according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a plurality of people in an image frame recognized by the person recognition circuit.

FIG. 5 is a schematic diagram of emphasizing a person who is speaking in a video frame.

FIG. 1 is a schematic diagram of a remote video conference. As shown in Figure 1, there is a There is an electronic device 110 for taking pictures of the first meeting room in real time, and recording the sound in the first meeting room in real time, and then sending it to the second meeting room through the network for the electronic device in the second meeting room 120 plays the image and sound of the first meeting room; at the same time, the electronic device 120 in the second meeting room also captures the image of the second meeting room and records the sound of the second meeting room in real time, and transmits them to the second meeting room through the network. A meeting room for the electronic device 110 in the first meeting room to play the image and sound of the second meeting room. In this embodiment, the electronic device 110 and the electronic device 120 may be any electronic devices with video and audio sending and receiving functions and network communication functions, such as TV, notebook computer, tablet computer, mobile phone, etc.

As described in the prior art, when one party in the remote video conference includes multiple participants in the video frame, sometimes it may be difficult for the other party's participants to distinguish who is speaking in the video frame. For example, if the participants in the second meeting room are not familiar with the voices of the participants in the first meeting room, or the participants in the first meeting room are not facing the camera, or other image transmission factors , the participants in the second conference room may sometimes have difficulty passing through the audio and video played by the electronic device 120 , thus causing confusion. Therefore, the system chip in the electronic device 110 in this embodiment designs a method that can emphasize the participant who is speaking in the image, so that the participant in the second conference room can clearly know which conference room is the first conference room. participants are speaking to address the above questions.

FIG. 2 is a schematic diagram of an electronic device 110 according to an embodiment of the present invention. As shown in FIG. 2 , the electronic device 110 includes a system chip 200 , an image capture device 202 and a plurality of microphones 204_1 - 204_N, wherein N is any suitable positive integer greater than one. In addition, the system chip 200 includes a person recognition circuit 210, a voice activity detection circuit 220, a sound detection circuit (in this embodiment, a sound direction detection circuit 230 is taken as an example) and A processing circuit 240 . In this embodiment, the image capture device 202 can be a camera or a video camera, so as to continuously capture images in the first conference room in real time to generate image data to the system chip 200, wherein the system chip The image data received at 200 may be original image data or data after some image processing operations have been performed. The microphones 204_1 ˜ 204_N may be digital microphones, which are disposed at different positions of the electronic device 110 to generate multiple audio signals to the system chip 200 respectively. It should be noted that in the embodiment of FIG. 2, the image capture device 202 and the microphones 204_1~204_N are installed in the electronic device 110, however, in other embodiments, the image capture device 202 and the microphones 204_1~204_N It can be externally connected to the electronic device 110 .

In the system chip 200, the person recognition circuit 210 is used to perform person recognition on the image data received from the image capture device 202, to determine whether there is a person in the received image data, and to determine each person eigenvalues and the position/area of each character in the screen. Specifically, the person recognition circuit 210 can use deep learning or neural network-like methods to process each frame in the image data, for example, use multiple different convolution filters to process the frame Perform multiple convolution operations to identify whether there is a person in the frame; in addition, for the detected person, a feature value of each person is determined through the previously adopted deep learning or neural network method (or, the feature value of the area where each character is located), where the feature value can be represented as a multi-dimensional vector, for example, a vector with a dimension of '512'. It should be noted that the above-mentioned circuit design related to person identification is well known to those skilled in the art. In addition, one of the key points of this embodiment is the application of the person identified by the person identification circuit 210 and its eigenvalues. , so other details of the person identification circuit 210 will not be repeated here.

The voice activity detection circuit 220 is used to receive the voice signals from the microphones 204_1~204_N, and determine whether there are voice components in the voice signals. Specifically, the voice activity detection circuit 220 can mainly perform the following operations: perform noise reduction processing on the received audio signal, convert the audio signal into a frequency domain and then process a block to obtain a feature value, convert the obtained Eigenvalues are compared with a reference value Perform comparison to determine whether the sound signal is a voice signal. It should be noted that since the circuit design related to voice activity detection is well known to those skilled in the art, and one of the key points of this embodiment is to perform subsequent steps according to the judgment result of the voice activity detection circuit 220 operation, so other details of the voice activity detection circuit 220 will not be repeated here. In addition, in another embodiment, the voice activity detection circuit 220 may only receive audio signals from some of the microphones 204_1~204_N, instead of receiving audio signals from all the microphones 204_1~204_N.

Regarding the operation of the sound direction detection circuit 230, since the positions of the microphones 204_1~204_N on the electronic device 110 are known, the sound direction detection circuit 230 can be based on the time difference of the sound signals from the microphones 204_1~204_N (that is, The phase difference of the received sound signal) to determine the azimuth of the main sound in the first conference room, that is, the direction and angle of the main speaker relative to the electronic device 110 . In this embodiment, the sound direction detection circuit 230 will only determine one direction, that is, if there are multiple people speaking in the first conference room at the same time, it will be based on some characteristics of the received sound signals ( For example, signal strength) to determine which direction the main sound is coming from. It should be noted that since the related circuit design of sound direction detection is well known to those skilled in the art, and one of the key points of this embodiment is to perform subsequent steps according to the judgment result of the sound direction detection circuit 230 Therefore, other details of the sound direction detection circuit 230 will not be repeated here.

Regarding the overall operation of the system chip 200 , refer to the flow chart of the video processing method according to an embodiment of the present invention shown in FIG. 3 . In step 300, the process starts, the electronic device 110 is powered on and the connection with the electronic device 120 in the second meeting room is completed. In step 302, the voice activity detection circuit 220 receives the sound signals from the microphones 204_1~204_N, and judges whether there is a voice component in these sound signals, if yes, the process enters step 304; if not, the process stays in step 302 for continuous detection Whether the received audio signal contains speech components. In step 304, the processing circuit 240 learns that the voice activity detection circuit After 220 detects that the audio signal has a voice component, enable the person recognition circuit 210, so that the person recognition circuit 210 starts to perform person recognition on the received image data, so as to determine whether there is a person in the received image data , and determine the feature value of each character and the position/area of each character in the screen. Taking Figure 4 as an example for illustration, the person recognition circuit 210 detects that there are 5 people in the image, so it can determine the area 410-450 of each person in the screen, and determine the image in the area 410-450 The characteristic value of the content is used as the characteristic value of each character respectively. In step 306, the processing circuit 240 enables the sound direction detection circuit 230, and the sound direction detection circuit 230 starts to judge the direction and angle of the main sound relative to the electronic device 110 according to the time difference of the sound signals from the microphones 204_1~204_N. . It should be noted that step 304 and step 306 can be executed simultaneously, that is, the execution of this embodiment is not limited to the sequence shown in FIG. 3 .

In step 308, the processing circuit 240 determines the area of each person in the video frame determined by the person identification circuit 210 (for example, the areas 410-450 in FIG. 4 ), plus the detection of the sound direction detection circuit 230 Based on the direction and angle of the main speaking character relative to the electronic device 110 , it can be determined which character in the image is speaking. In step 310 , after determining the speaking person in the image frame, the processing circuit 240 processes the image data from the image capture device 202 to emphasize the main speaking person in the image data. Specifically, referring to FIG. 5, assuming that the processing circuit 240 judges that the person in the area 440 is the main speaking person, the processing circuit 240 can process the image data to enlarge or add a label to the person in the area 440. / arrows, or any other image processing method, to enhance the visual effect of the characters in the area 440 . After processing the image data to enhance the visual effect of the characters in the area 440, the processing circuit 240 sends the processed image data to the back-end circuit for other image processing, and then sends it to the second conference room through the network The electronic device 120 in the second meeting room, so that the participants in the second meeting room can clearly know who is currently speaking in the first meeting room.

It should be noted that the above-mentioned embodiment of enhancing the visual effects of the characters in the area 440 does not necessarily have to perform visual enhancement on the entire area 440, but only a part of the area 440 can be visually enhanced, which can also achieve the same effect . Taking FIG. 5 as an example for illustration, the area 440 includes the head and body of the character, and the processing circuit 240 can only enlarge the head part.

In step 312 , the processing circuit 240 keeps tracking the previously emphasized person, and continues to process the image data from the image capture device 202 to emphasize the person in the image data. Specifically, the character identification circuit 210 can continue to determine the area where each character is located in the image frame and its characteristic value, and the processing circuit 240 can continue to identify the character in the current and subsequent images according to the characteristic value of the previously emphasized character. The character is emphasized in the picture. Taking the area 440 in FIG. 5 as an example, the processing circuit 240 can track the area/person whose feature value is similar to the feature value of the area 440 in the subsequently received image frame (for example, the difference of the feature value is within a certain range), so as to continuously The person is emphasized in the subsequent image frame, even if the emphasized person does not speak for a short period of time in the subsequent image frame, and the sound direction detection circuit 230 does not detect that the person has a voice.

It should be noted that since the person who is speaking may move and may not continue to speak, step 312 can avoid the continuous opening and closing of the video screen to enhance the visual effect of the speaking person, which will affect the participants in the second meeting room feelings.

In step 314, the processing circuit 240 determines the area of each character in the video frame determined by the character recognition circuit 210, plus the direction and direction of the main speaking character detected by the sound direction detection circuit 230 relative to the electronic device 110. Angle, and whether the voice activity detection circuit 220 detects whether someone is speaking (that is, the received sound signal has a voice component), to determine whether the person speaking has changed, if not, the process returns to step 312 to continue Track who is currently speaking; if so, go back to step 308 to determine a new speaker. Specifically, since the sound direction detection circuit 230 can only detect the directionality of the sound, it is impossible to know whether the sound in the determined direction is a human voice. Therefore, by cooperating with the operation of the voice activity detection circuit 220, When the voice activity detection circuit 220 detects that there is a voice component in the current voice signal, if the voice direction detection circuit 230 detects that the direction and angle of the main speaker relative to the electronic device 110 is changed to the position of another character , the processing circuit 240 can determine that the speaker has changed. It should be noted that, in order to prevent the processing circuit 240 from constantly changing the highlighted person in the image data, the execution of step 314 will require a long detection period before making a judgment.

In another embodiment, in order to further confirm whether the speaker has changed, the processing circuit 240 may additionally include a voiceprint recognition mechanism to assist the detection result of the sound direction detection circuit 230 . Specifically, since the voice of each person has unique speech characteristics, the voiceprint recognition mechanism in the processing circuit 240 can continuously capture some sound clips to determine whether the sound feature values of these sound clips belong to the same person, for Make a judgment on the speaker. For example, if it is determined by the character identification circuit 210, the voice activity detection circuit 220 and the voice direction detection circuit 230 that the person speaking has changed, but the voiceprint recognition mechanism judges that the sound feature value of the sound segment belongs to the same person, Then the processing circuit 240 can suspend judging whether the speaker has changed, and make a judgment after detecting for a period of time.

In the previous embodiments, the sound direction detection circuit 230 is used as the sound detection circuit for illustration, however, the present invention is not limited thereto. In other embodiments, the voiceprint recognition mechanism can be used to replace the voice direction detection circuit 230 of the above-mentioned embodiment, and only the voiceprint recognition result is used to judge the speaking person and determine the emphasized object accordingly. In other words, the sound detection circuit of the present invention can obtain a plurality of sound signals from a plurality of microphones to determine the sound feature value of a main sound, and the sound feature value can be an azimuth of the main sound or a sound supply. The sound feature value of the sound clip of the grain recognition mechanism.

To briefly summarize the present invention, in the video processing method of the present invention, by detecting the person who is currently speaking and emphasizing the person in the video data, the participants in the remote conference room can clearly know who is currently speaking, so it can Effectively improve meeting efficiency.

The above descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of the present invention.

110: Electronic device

200: system chip

202: image capture device

204_1~204_N: Microphone

210: Character identification circuit

220: Voice activity detection circuit

230: Sound direction detection circuit

240: Processing circuit

Claims (9)

A system chip includes: a human identification circuit, used to obtain an image data from an image capture device in real time, and perform identification on the image data to generate a recognition result; a sound detection circuit, used to automatically A plurality of microphones obtain a plurality of sound signals to determine a sound feature value of a main sound; and a processing circuit, coupled to the person identification circuit and the sound detection circuit, is used to obtain the identification result and the The sound feature value of the main sound is used to determine a specific area in the image data, and the image data is processed to emphasize the specific area; wherein the recognition result includes a plurality of areas, and each area includes a person; and the processing circuit is used to select an area from the plurality of areas as the specific area according to the sound feature value of the main sound; and the recognition result further includes a plurality of feature values respectively corresponding to the plurality of areas , and the processing circuit tracks the feature value of the specific area to determine the position of the specific area in the subsequent image data, and processes the subsequent image data to emphasize the specific area. The system chip as described in item 1 of the scope of the patent application further includes: a voice activity detection circuit, which is used to judge whether the at least a part of the sound signals contains at least a part of the sound signals according to the plurality of sound signals There is a speech component; wherein the processing circuit determines whether to determine the specific area in the image data according to the recognition result and the sound feature value of the main sound according to whether the at least part of the sound signal contains a speech component , and process the imagery to emphasize that particular area. The system chip as described in item 2 of the scope of the patent application, wherein when the voice activity detection When the circuit indicates that the at least part of the sound signal contains speech components, the processing circuit will judge the specific area in the image data according to the recognition result and the sound feature value of the main sound, and the image data Treated to emphasize that particular area. The system chip as described in item 1 of the scope of the patent application further includes: a voice activity detection circuit, which is used to judge whether the at least a part of the sound signals contains at least a part of the sound signals according to the plurality of sound signals There are speech components; wherein the processing circuit determines the plurality of feature values corresponding to the plurality of regions according to the person identification circuit, the sound feature value of the main sound detected by the sound detection circuit, and The speech activity detection circuit detects whether the at least a part of the sound signal contains a speech component, so as to determine whether the person speaking has changed, so as to determine whether to select another region from the plurality of regions as the specific region. The system chip as described in item 1 of the scope of the patent application, wherein the processing circuit processes the image data to enlarge the figures in the specific area. The system chip as described in item 1, 2, 3, 4 or 5 of the scope of patent application, wherein the sound detection circuit is a sound direction detection circuit, and the sound feature value of the main sound is a value of the main sound Azimuth. A video processing method, including obtaining an image data from an image capture device in real time, and performing character recognition on the image data to generate a recognition result; Obtaining a plurality of sound signals from a plurality of microphones to determine a sound feature value of a main sound; judging a specific area in the image data according to the identification result and the sound feature value of the main sound; and The image data is processed to emphasize the specific region; wherein the recognition result includes a plurality of regions, each region contains a person, and the recognition result further includes a plurality of feature values respectively corresponding to the plurality of regions; and the video The processing method further includes: tracking the feature value of the specific area to determine the position of the specific area in the subsequent image data, and processing the subsequent image data to emphasize the specific area. The video processing method described in item 7 of the scope of the patent application further includes: judging whether the at least a part of the sound signals contains speech components according to at least a part of the sound signals of the plurality of sound signals; The plurality of characteristic values of the plurality of regions, the sound characteristic value of the main sound, and whether the at least a part of the sound signal contains speech components, so as to determine whether the person speaking has changed, so as to judge whether it is from the plurality of Region Select another region as the specific region. According to the video processing method described in item 7 of the scope of the patent application, the step of processing the subsequent image data to emphasize the specific area includes: processing the image data to enlarge the characters in the specific area.

Images