CN114625245A - Control method, device, equipment and storage medium - Google Patents

Abstract

The embodiment of the application discloses a control method, a control device, control equipment and a storage medium, wherein the method comprises the following steps: at least acquiring first information of a user; responding to at least the acquired first information of the user, and performing first operation on a first area according to the first information; wherein the first information includes image information of the user, the first region being determined based on the first information.

Description

A control method, device, equipment and storage medium

technical field

The present application relates to computer processing technology, and in particular, to a control method, apparatus, device, and storage medium.

Background technique

In the related art, in the process of controlling the robot, it is necessary to issue accurate voice instructions to it. For example, for the sweeping robot, when putting forward cleaning requirements (such as cleaning the living room, cleaning the master bedroom, etc.) The named area puts forward cleaning requirements, but it is impossible to achieve cleaning requirements for areas that do not have accurate names (such as: clean here, etc.), which is not flexible and convenient to use, and cannot meet the needs of users.

SUMMARY OF THE INVENTION

In view of this, embodiments of the present application provide a control method, apparatus, device, and storage medium.

In a first aspect, an embodiment of the present application provides a control method, which is applied to an electronic device, and the method includes:

Obtain at least the first information of the user;

In response to at least the acquired first information of the user, performing a first operation on the first area according to the first information;

The first information includes first image information of the user, and the first area is determined based on the first information.

In a second aspect, an embodiment of the present application provides a control device, the device comprising:

a first acquiring module, configured to acquire at least the first information of the user;

a first processing module, configured to perform a first operation on the first area according to the first information in response to at least the acquired first information of the user;

The first information includes first image information of the user, and the first area is determined based on the first information.

In a third aspect, an embodiment of the present application provides an electronic device, where the electronic device includes: a processor, a memory, and a communication bus;

the communication bus is used to realize the communication connection between the processor and the memory;

The processor is used for executing the program in the memory, so as to realize the above-mentioned control method.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors, to realize the above control method.

In this embodiment of the present application, first, at least the first information of the user is obtained; secondly, in response to at least the obtained first information of the user, a first operation is performed on the first area according to the first information; The first information includes the first image information of the user, and the first area is determined based on the first information; in this way, performing the first operation on the first area determined by the user's image information, in use More flexible and convenient, can meet more needs of users.

Description of drawings

FIG. 1 is a schematic flowchart of the implementation of a control method provided by an embodiment of the present application;

FIG. 2 is a schematic flowchart of the implementation of a control method provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of an implementation flowchart of a control method provided by an embodiment of the present application;

FIG. 4 is a schematic flowchart of the implementation of a control method provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of an implementation flowchart of a control method provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of an implementation flowchart of a control method provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of an implementation flowchart of a control method provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of the composition and structure of a control device provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed ways

The technical solutions of the present application will be further elaborated below with reference to the accompanying drawings and embodiments.

An embodiment of the present application provides a control method, which is applied to an electronic device. FIG. 1 is a schematic flowchart of the implementation of the control method provided by the embodiment of the present application. As shown in FIG. 1 , the method includes:

Step S101: Obtain at least the first information of the user.

In some embodiments, the first information includes first image information of the user; further, the first image information may be image features, such as color features, texture features, shape features, and spatial relationship features.

In some embodiments, the first image information may be image information of one frame of image, or may be image information of multiple frames of images that are consecutive in time.

In some embodiments, the first image information may be obtained by performing feature extraction on the collected image information of the user.

Step S102: In response to at least the acquired first information of the user, perform a first operation on the first area according to the first information.

In some embodiments, the first region is determined based on the first information.

In some embodiments, the first area can be understood as the area to be treated; here, the first area is related to the function of the electronic device, for example, when the electronic device is a cleaning robot or a car washing robot, the first area can be the area to be cleaned area; when the electronic device is a home service robot, the first area can be the area to be served, or the auxiliary area of the auxiliary object; for example, the area to be served can be the sink, and the auxiliary area can be arms, elbows, legs, etc. ; Obviously, the electronic device can also be a robot with other functions, which is not limited here.

In some embodiments, the first area may be determined by image information of one frame of image, or may be determined by image information of multiple frames of images that are consecutive in time; in a possible implementation manner, the first area may be determined by the following Process determination: first, identify the user's first image information, and determine the user's instruction information; secondly, combine the spatial information, and determine the area corresponding to the instruction information in the spatial information as the first area; for example: when the electronic device is a sweeping robot , combined with the user's instruction information and house graphic modeling, to determine the first area; in this way, the cleaning robot can clean the area that does not have a name (such as: here), which is more flexible and convenient to use.

In some embodiments, the first operation may be a default operation; in a possible implementation manner, the default operation may be specified by the system or determined based on user selection; generally, when leaving the factory, it will be an electronic The device pre-sets the default operation; after leaving the factory, the user can select from the preset options to modify the default operation preset before leaving the factory.

In this embodiment of the present application, first, at least the first information of the user is obtained; secondly, in response to at least the obtained first information of the user, a first operation is performed on the first area according to the first information; that is, , the first area is determined by the user's image information, and the first operation is performed on the first area; in this way, the use is more flexible and convenient, and more needs of the user can be met.

In some embodiments, the first operation may be a default operation. FIG. 2 is a schematic flowchart of the implementation of the control method provided by the embodiment of the present application. As shown in FIG. 2 , step S102 includes:

Step S201: Determine a first action of the user based on the first image information.

In some embodiments, the first motion may be a physical motion of the user.

In some embodiments, the first action may be determined by performing image recognition on the first image information; in a possible implementation, it may be determined by the following process: first, collecting multiple frames of images of the user that are consecutive in time; Secondly, the image information of each image is extracted to obtain the first image information; thirdly, based on the first image information, the information of the limb position change is determined; finally, the information of the limb position change is identified and processed to obtain the first action.

Step S202: Based on the first action, determine the first area indicated by the user.

In some embodiments, the first area may be determined in combination with the first action and spatial information; for example, when the electronic device is a sweeping robot or a car washing robot, step S202 may be implemented through the following process: first, based on the first action, determine Indication information; secondly, combined with the spatial information, the area corresponding to the indication information in the spatial information is determined as an approximate area; finally, the image after historical cleaning is compared with the image of the above approximate location, and the location with stains is determined as the first an area.

Step S203: Perform the first operation on the first area.

In some embodiments, the first operation (default operation) may be determined according to the application scenario; for example: when the electronic device is a sweeping robot or a car washing robot, the default operation may be a cleaning operation; further, the cleaning operation may include sweeping , mopping, sweeping and then mopping, water spraying, waxing, polishing, polishing and then waxing, etc., which are not limited here.

In the embodiment of the present application, firstly, based on the first image information, determine the first action of the user; secondly, based on the first action, determine the first area indicated by the user; The first operation is performed in the first area; in this way, the first operation can be performed on the first area specified by the user (for example, an area that does not have an accurate name), which is more flexible and convenient in use, and is applicable to a wider range of people. , eg: Disabled (deaf).

In some embodiments, the first information further includes the first voice information of the user. In this case, the first operation may be determined through the first voice information. In this way, the determined first operation may be the same as the default operation, or may be the same as the default operation. The operations are different; FIG. 3 is a schematic diagram of the implementation flow of the control method provided by the embodiment of the present application. As shown in FIG. 3 , before step S203, the method further includes:

Step S301: Determine a first operation based on the first voice information.

In some embodiments, steps 201 to 203 can be understood as a process of performing a default operation based on the user's first image information; adding step S301 on the basis of steps 201 to 203 can be understood as a process based on the user's first image The process of performing the first operation on the information and the first voice information. At this time, the first operation may be the same as the default operation, or may be different from the default operation; in this way, the electronic device can adapt to more scenarios.

In some embodiments, the first operation may identify and determine the first voice information through voice recognition.

In some embodiments, the first voice information can be collected while collecting the first image information; in this way, the first area can be determined by the first image information, and the first operation can be determined by the first voice information, which can facilitate the user to publish the first An instruction of an operation, for example: when the first voice information is "robot, come here to clean", the first operation is determined to be "sweep" based on the first voice information, the first area is determined based on the first image information obtained at the same time, and then , and perform the first operation "Scan" on the first area.

In some embodiments, the electronic device may be an auxiliary robot. In this case, the first operation includes an auxiliary operation. FIG. 4 is a schematic flowchart of the implementation of the control method provided by the embodiment of the present application. As shown in FIG. 4 , step S102 includes:

Step S401: Based on the first information, determine a second action of the user.

In some embodiments, the first information includes the first image information of the user. In this case, the implementation methods of step S401 and step S201 are the same.

In some embodiments, the second action is different from the first action.

Step S402: Based on the second action, determine the auxiliary operation indicated by the user.

In some embodiments, the auxiliary operation may be an operation of assisting the user, such as: assisting standing, assisting sitting up, etc., which are not limited here.

In some embodiments, the mapping relationship between the second action and the auxiliary operation may be pre-established, for example, one hand or both palms up corresponds to assisting standing, one hand or both palms down corresponds to assisting sitting up, etc. Here Not limited.

Step S403: Determine the first area on the user based on the auxiliary operation.

In some embodiments, the first area can be understood as the implementation position of the auxiliary operation; in a possible implementation manner, step S403 can be implemented through the following process: first, combining the auxiliary operation and the human body image information, in the human body image information Determine the implementation positions corresponding to the auxiliary operations; secondly, determine all the implementation positions as the first area.

Step S404: Perform the auxiliary operation on the first area to assist the user.

In this embodiment of the present application, firstly, based on the first information, the second action of the user is determined; secondly, based on the second action, the auxiliary operation indicated by the user is determined; thirdly, based on the second action In the auxiliary operation, the first area is determined on the user; finally, the auxiliary operation is performed on the first area; in this way, the disabled person, such as the deaf-mute person with limited mobility, can be assisted.

In some embodiments, the first information may also include the first image information and the first voice information of the user. In this case, the methods of steps S401 to S404 are used to determine the first area for performing the auxiliary operation, and the determination is based on the first voice information. Auxiliary operation (same implementation method as step S301), so that it is convenient for the user to issue the instruction of the first operation, for example: when the first voice information is "robot, scratch here", the auxiliary operation is determined based on the first voice information For "flexing", a first region where the auxiliary operation is performed on the user is determined based on the simultaneously acquired first image information, and then the auxiliary operation "flexing" is performed on the first region.

In some embodiments, the second action includes: the eye action of the user; here, a mapping relationship between the eye action and the auxiliary operation may be established in advance, and the auxiliary operation is performed by using the methods of steps S401 to S404. Here, the mapping relationship is for example: blinking once corresponds to assisting standing, blinking twice corresponds to assisting sitting up, etc., which are not limited here. In this way, more usage modes can be provided for users, and more groups of people can be applied.

In some embodiments, after obtaining the first information, other information may be continuously obtained; FIG. 5 is a schematic flowchart of the implementation of the control method provided by the embodiment of the present application. As shown in FIG. 5 , after step S102, the method further include:

Step S501: Acquire second information of the user.

In some embodiments, the second information includes: second image information, and/or second voice information; that is, the second information may be the user's image information or second voice information, or the user's image information and voice messages.

In some embodiments, the second information may be acquired within a preset period of time after acquiring the first information, or within a preset period of time after the first operation is performed, or may be acquired after the first information is acquired to the first operation. Obtained within a preset time period after execution, which is not limited here.

Step S502: In response to the acquired second information of the user, perform a second operation on the first area according to the second information.

In some embodiments, the second operation may be the same as the first operation, or may be different from the first operation.

In this embodiment of the present application, first, second information of the user is acquired; secondly, in response to the acquired second information of the user, a second operation is performed on the first area according to the second information ; wherein, the second information includes: second image information, and/or second voice information; in this way, the user can continuously issue instructions to the electronic device.

In some embodiments, the electronic device needs to move to the vicinity of the user to obtain the first information of the user. FIG. 3 is a schematic flowchart of the implementation of the control method provided by the embodiment of the present application. As shown in FIG. 6 , step S101 includes:

Step S601: Determine the location information of the user.

In some embodiments, the location information of the user may be determined by locating the sound source of the user to determine the location information of the user; further, step S201 may be implemented by the following steps: first, acquiring the user's voice instruction; secondly, by Microphone array algorithm spatial modeling locates the user's location.

Step S602: Based on the location information, move to the vicinity of the user and capture images of the user to obtain the first information.

In the embodiment of the present application, first, the location information of the user is determined; secondly, based on the location information, moving to the vicinity of the user and capturing images of the user to obtain the first information; in this way, it is possible to Move to the vicinity of the user, and capture images of the user to obtain image information of the user.

Hereinafter, an exemplary application of the embodiments of the present application in a practical application scenario will be described, taking the electronic device as a cleaning robot as an example for description.

FIG. 7 is a schematic diagram of the implementation flow of a control method provided by an embodiment of the present application. As shown in FIG. 7 , the method includes:

Step S701: Perform graphical modeling on the house to obtain a three-dimensional model of the house.

Step S702: In the case of receiving the wake-up instruction, locate the position of the sound source.

In some embodiments, the sound source localization is the process of judging the position of the sound-emitting object by the auditory system, which includes horizontal sound source localization, vertical sound source localization and identification of the distance from the listener.

Step S703: Determine the user's position based on the sound source position and the three-dimensional model.

In some embodiments, the user position in step S703 may be determined by combining the sound source position with the three-dimensional model.

Step S704: Plan a path based on the user position and the three-dimensional model, and move to the vicinity of the user according to the path.

Step S705: in the case of receiving the task instruction, capture the user image.

In some embodiments, the task instructions may be voice instructions.

In some embodiments, in order to prevent the user from being interrupted and make the cleaning robot focus on the same user, after acquiring the task instruction, the timbre can be recognized, and after determining that the issuer of the task instruction and the issuer of the wake-up instruction are the same After the user, the user image is collected again.

Step S706: Determine a user-specified area based on the user image;

In some embodiments, by recognizing the user image, the user's instruction information can be obtained; therefore, step S706 can be realized by the following steps: first, recognizing the user image, and determining the user's actions (body movements, facial movements, etc.); Secondly, based on the user action, combined with the three-dimensional model, a designated area is determined.

Step S707: in the designated area, determine the area to be cleaned;

In some embodiments, the area to be cleaned is the above-mentioned first area.

In some embodiments, the designated area may be a larger area; generally, the area to be cleaned is determined in the designated area; therefore, step S706 can be implemented by the following steps: first, collecting images of the designated area; secondly, after retrieving historical cleaning The image is compared with the designated area to determine the area to be cleaned (dirty area).

Step S708: Clean the area to be cleaned.

An embodiment of the present application provides a control device. FIG. 8 is a schematic structural diagram of a control device provided by an embodiment of the present application. As shown in FIG. 8 , the control device 800 includes:

a first obtaining module 801, configured to obtain at least the first information of the user;

a first processing module 802, configured to perform a first operation on the first area according to the first information in response to at least the acquired first information of the user;

The first information includes image information of the user, and the first area is determined based on the first information.

In some embodiments, the first processing module 802 includes:

a second determining module, configured to determine the first action of the user based on the first image information;

a third determining module, configured to determine the first area indicated by the user based on the first action;

A second processing module, configured to perform the first operation on the first region.

In some embodiments, the first information further includes first voice information of the user, and the first processing module 802 further includes:

A fourth determining module, configured to determine a first operation based on the first voice information.

In some embodiments, the first operation includes an auxiliary operation, and the first processing module 802 includes:

a fifth determining module, configured to determine the second action of the user based on the first information;

a sixth determining module, configured to determine the auxiliary operation indicated by the user based on the second action;

a seventh determination module, configured to determine the first region on the user based on the auxiliary operation;

A third processing module, configured to perform the auxiliary operation on the first area.

In some embodiments, the second action includes an eye action of the user.

In some embodiments, the control device 800 further includes:

a second obtaining module, configured to obtain the second information of the first user;

a fourth processing module, configured to perform a second operation on the first area according to the second information in response to the acquired second information of the user; wherein the second information includes: a second image information, and/or second voice information.

In some embodiments, the control device 800 further includes:

an eighth determining module, configured to determine the location information of the user;

The fifth processing module is configured to move to the vicinity of the user based on the location information and collect information from the user to obtain the first information.

An embodiment of the present application provides an electronic device, and FIG. 9 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As shown in FIG. 9 , the electronic device 900 includes: a processor 901, a memory 902, and a communication bus 903;

The communication bus 903 is used to realize the communication connection between the processor 901 and the memory 902;

The processor 901 is configured to execute the program in the memory 902 to implement the above control method.

An embodiment of the present application provides a computer-readable storage medium, where one or more programs are stored in the computer-readable storage medium, and the one or more programs can be executed by one or more processors to implement the above control method .

It is to be understood that reference throughout the specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic associated with the embodiment is included in at least one embodiment of the present application. Thus, appearances of "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily necessarily referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be dealt with in the embodiments of the present application. implementation constitutes any limitation. The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling, or direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms. of.

The unit described above as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit; it may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may all be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integration The unit can be implemented either in the form of hardware or in the form of hardware plus software functional units.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by program instructions related to hardware, the aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, the execution includes: The steps of the above method embodiments; and the aforementioned storage medium includes: a removable storage device, a read only memory (Read Only Memory, ROM), a magnetic disk or an optical disk and other media that can store program codes.

Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence or in the parts that make contributions to the prior art. The computer software products are stored in a storage medium and include several instructions for A computer device (which may be a personal computer, a server, or a network device, etc.) is caused to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a removable storage device, a ROM, a magnetic disk, or an optical disk.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (10)

1. A control method, comprising:

at least acquiring first information of a user;

responding to at least the acquired first information of the user, and performing first operation on a first area according to the first information;

wherein the first information includes first image information of the user, the first region being determined based on the first information.

2. The method of claim 1, wherein the performing a first operation on a first area according to the first information in response to at least the obtained first information of the user comprises:

determining a first action of the user based on the first image information;

determining the first region indicated by the user based on the first action;

and performing the first operation on the first area.

3. The method of claim 2, the first information further comprising first voice information of the user, the method comprising, prior to the first operation on the first region:

based on the first voice information, a first operation is determined.

4. The method of claim 1, wherein the first operation comprises an auxiliary operation, and the performing, in response to at least obtaining first information of a user, a first operation on a first area according to the first information comprises:

determining a second action of the user based on the first information;

determining the secondary operation indicated by the user based on the second action;

determining the first region on the user based on the secondary operation;

and performing the auxiliary operation on the first area.

5. The method of claim 4, the second action comprising: an eye movement of the user.

6. The method of claim 1, further comprising:

acquiring second information of the user;

responding to the acquired second information of the user, and performing second operation on the first area according to the second information; wherein the second information comprises: second image information, and/or second voice information.

7. The method according to claim 1, before the performing, in response to at least acquiring first information of a user, a first operation on a first area according to the first information, the method further comprising:

determining location information of the user;

and moving the mobile terminal to the vicinity of the user and acquiring information of the user based on the position information to obtain the first information.

8. A control device, comprising:

the first acquisition module is used for at least acquiring first information of a user;

the first processing module is used for responding to at least the acquired first information of the user and carrying out first operation on a first area according to the first information;

wherein the first information includes first image information of the user, the first region being determined based on the first information.

9. An electronic device, the electronic device comprising: a processor, a memory, and a communication bus;

the communication bus is used for realizing communication connection between the processor and the memory;

the processor is used for executing the program in the memory so as to realize the control method.

10. A computer readable storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the control method of any one of claims 1 to 7.

Images