CN116252307A - Robot assistant and method for controlling the same - Google Patents

Abstract

The invention provides a robot assistant and a method for controlling the robot assistant, wherein the robot assistant comprises: a base; a lifting mechanism positioned on the base; a display rotatably mounted on the lifting mechanism; a camera positioned on the display, the camera configured to capture an image of a user; and a control system that receives command instructions; wherein, in response to a command instruction, the control system is configured to detect movement of the user's face in a vertical direction based on the image; wherein, in response to detecting movement of the user's face in a vertical direction, the control system is configured to rotate the display and actuate the lift mechanism to move the display up and down to face the user's face during movement of the user's face in the vertical direction. When the user makes a deep squat, the camera can be controlled to always face the face of the user, so that the face of the user appears in the center of the display screen. The robotic assistant may provide guidance by displaying information on a display.

Description

Robot assistant and method of controlling robot assistant

technical field

The present invention relates to robots, and more particularly to an intelligent robotic assistant that can provide walking assistance, walking training and physical training.

Background technique

The demand for robotics in the service industry has grown over the years. As a result, robotic assistants have attracted a great deal of attention in recent years.

For example, a type of robotic assistant could be designed to help support part of the user's body weight to reduce the load on the user's legs while walking, thereby reducing fatigue and physical exertion. For example, a large body of research can be found on assistive robotics, including applications for upper-body, lower-body, and whole-body assistance or training.

These robotic assistants typically include wheels for movement and a vertical body with a handle for the user to grasp. Some robotic assistants may include displays for displaying guided information. However, due to the fixed nature of the vertical body and display, these robotic assistants may lack the ability to self-reconfigure to adapt to different application scenarios.

Therefore, it is necessary to provide a robot assistant to overcome the above problems.

Contents of the invention

Therefore, the present invention provides a robot assistant to solve the above problems.

In order to achieve the above object, the present invention provides a robot assistant, comprising: a base; a lifting mechanism on the base; a display rotatably installed on the lifting mechanism; a camera on the display, the camera is configured as taking an image of the user; and a control system that receives a command instruction; wherein, in response to the command instruction, the control system is configured to detect a vertical movement of the user's face based on the image; wherein, in response to detecting that the user's face moves in a vertical direction For vertical movement, the control system is configured to rotate the display and actuate the lift mechanism to move the display up and down to face the user's face during the vertical movement of the user's face.

Optionally, the robot assistant also includes a display bracket on the top of the lifting mechanism and a motor fixed on the display bracket, wherein the display is indirectly mounted on the lifting mechanism through the display bracket, and the motor is configured to drive the display Swivel relative to the monitor arm.

Optionally, the robotic assistant further includes a rotation damper connected to the display bracket, wherein the rotation damper is configured to control the rotation speed of the display.

Optionally, the robotic assistant further includes two handles fixedly connected to the lifting mechanism, wherein the two handles are configured to fit the user's hands to provide two handles.

Optionally, the robotic assistant further includes a limit switch fixedly connected to the display stand, wherein the limit switch is configured to be activated in response to the display being rotated to a predetermined position, and the control system is configured to The limit switch is activated to stop the rotation of the display.

Optionally, the lifting mechanism includes an actuator installed on the base, a main body vertically arranged on the base, and a sliding part slidably connected to the main body, the display is located on the sliding part, and the actuator configured to drive the slider up and down.

The present invention also provides a robot assistant, comprising: a base; a lifting mechanism vertically arranged and installed on the base; a display rotatably connected to the lifting mechanism; a camera fixed on the display, and the camera is configured as taking an image of a user; one or more processors; memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, The one or more programs include: instructions for detecting a vertical movement of a user's face based on the image; and for rotating the display in response to detecting the vertical movement of the user's face and actuate the lifting mechanism to move the display up and down to allow the camera to face the user's face during the movement of the user's face in the vertical direction.

The present invention also provides a method for controlling a robotic assistant, comprising: providing a base; providing a lift mechanism positioned on the base; providing a display rotatably connected to the lift mechanism; providing a a camera; obtaining a plurality of images of the user using the cameras; detecting a vertical movement of the user's face based on the images; and rotating the display in response to detecting the vertical movement of the user's face And actuate the lifting mechanism to move the display up and down to allow the camera to face the user's face during the vertical movement of the user's face.

Optionally, rotating the display includes: determining a key point of the user's face in the current image in the images; determining the angle between a line passing through the key point and the center of a camera and the optical axis of the camera; determining the user's face a moving direction in the vertical direction; and rotating the display based on the moving direction of the user's face in the vertical direction and the included angle.

Optionally, after determining the moving direction of the user's face in the vertical direction, actuating the lifting mechanism to move the display up and down includes: activating the lifting mechanism to move upward according to the moving direction of the user's face in the vertical direction. Or move the display down.

The technical solution of the present invention has the following advantages: the robot assistant can promote the active lifestyle of the elderly. Robotic assistants could allow them to do more exercise to maintain their mobility. Moving around also provides older adults with more opportunities to interact with others (especially in a nursing or assisted living facility), thereby reducing their feelings of isolation. When the squatting user is properly standing behind the robot assistant, the camera can be controlled to face the user's face all the time, so that the user's face appears in the center of the display. Robotic assistants can provide guidance/assistance by displaying information on a display, such as squat times.

Description of drawings

The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the disclosure and, together with the description, further serve to explain the principles of the disclosure and to enable those skilled in the relevant art to make and use the disclosure. In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings that need to be used in the description of the specific embodiments or prior art. Obviously, for those skilled in the art As far as people are concerned, other drawings can also be obtained based on these drawings on the premise of not paying creative work.

Figure 1 is a schematic perspective view of a robotic assistant according to one embodiment.

Figure 2 is a schematic perspective view of a robotic assistant with certain components omitted for clarity.

Figure 3a is similar to Figure 2, but viewed from a different angle.

Figure 3b is similar to Figure 3 but showing the display in the extended position.

FIG. 4 is an enlarged view of part A of FIG. 2 .

Fig. 5 is an enlarged view of part B of Fig. 3a.

Figure 6 is a schematic diagram showing the display in two different positions.

Figure 7 is a schematic block diagram of a robotic assistant according to one embodiment.

Fig. 8 is a schematic flowchart of a method for controlling a robotic assistant according to one embodiment.

Fig. 9 is a schematic flowchart of a method for controlling a robotic assistant according to one embodiment.

Fig. 10 shows two exemplary images taken consecutively by the camera of the robotic assistant according to one embodiment.

Fig. 11 is a schematic flowchart of a method for controlling a robotic assistant according to one embodiment.

Figure 12 illustrates an exemplary image showing key points of a user's face, according to one embodiment.

Figure 13 is a schematic diagram of a simplified model of a robotic assistant according to one embodiment.

Fig. 14a is a diagram showing a relationship between a user's face and a face image in an image plane when the user stands at a predetermined position from the center of the camera.

Fig. 14b is a diagram showing a relationship between a user's face and a face image in an image plane when the user stands at a random position.

Figure 15 is a schematic block diagram of a robotic assistant according to one embodiment.

16 is a flowchart of a method of controlling a display in an automatic control mode and a manual control mode, according to one embodiment.

Detailed ways

The technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Apparently, the described embodiments are part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The present disclosure is illustrated by way of example and not limitation in the drawings, in which like reference numerals indicate like elements. It should be noted that references to "one" embodiment in this disclosure are not necessarily references to the same embodiment, and that such references may mean "at least one" embodiment.

While specific configurations and arrangements are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other configurations and arrangements may be used without departing from the spirit and scope of the present disclosure. It will be apparent to those skilled in the relevant art that the present disclosure can also be used in various other applications.

It is noted that references in the specification to "one embodiment," "example embodiments," "some embodiments," "certain embodiments," etc. indicate that the described embodiments may include particular features. , structure or characteristic (characteristic), but each embodiment may not necessarily include a specific feature, structure or characteristic. Moreover, these phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature, structure or characteristic is described in connection with one embodiment, it is within the knowledge of those skilled in the relevant art to implement such feature, structure or characteristic in relation to other embodiments whether explicitly described or not.

In general, a term can be understood at least in part from its usage in context. For example, the term "one or more" as used herein may be used in the singular to describe any feature, structure or characteristic or may be used in the plural to describe a combination of features, structures or characteristics, depending at least in part on the context. Similarly, terms such as "a" or "the" may also be read to convey singular usage or to convey plural usage, depending at least in part on the context. Furthermore, the term "based on" may be understood as not necessarily intended to convey an exclusive set of factors, and may instead allow for the presence of additional factors not necessarily explicitly described, again depending at least in part on context.

Although features and elements of the present disclosure have been described as embodiments in particular combinations, within the principles of the disclosure, each feature or element can be used alone or in various other combinations to the maximum extent indicated by the expression. The broad, ordinary meanings of the terms in the appended claims are indicated.

In addition, the technical features involved in the different embodiments of the present invention described below may be combined with each other as long as there is no conflict with each other.

FIG. 1 shows a perspective view of a robotic assistant 100 . In one embodiment, the robotic assistant 100 may be designed to help support a portion of the user's body weight to reduce the load on the user's legs when the user (eg, a care receiver or patient) is walking. The robot assistant 100 can provide support/guidance for people during their walking so that they can maintain balance and walk safely. In one embodiment, robotic assistant 100 may be used in settings such as healthcare settings, aged care settings, assisted living settings, etc., to assist elderly people while they are walking. However, robotic assistant 100 may be used in other settings. For example, robotic assistant 100 may be used in hospitals to provide walking assistance, physical training, and fall prevention to people who have temporarily lost the ability to walk due to accident or illness.

2, 3a and 3b, in one embodiment, the robot assistant 100 may include a base 10, a lifting mechanism 20 positioned on the base 10, a display 30 rotatably mounted on the lifting mechanism 20, a camera positioned on the display 30 40. Control system 50 (see FIG. 7). Control system 50 receives commands from a host computer and a graphical user interface (GUI) displayed on display 30 to allow users (eg, healthcare professionals and care receivers) to directly control robotic assistant 100 . In response to command instructions, control system 50 controls movement of lift mechanism 20 and rotation of display 30 , and/or other mechanical or software aspects of robotic assistant 100 . In response to command instructions, control system 50 controls movement of lift mechanism 20 and rotation of display 30 , and/or other mechanical or software aspects of robotic assistant 100 .

In one embodiment, base 10 may provide a movement mechanism for robotic assistant 100 to move from one location to another. In one embodiment, base 10 includes a body, two differential drive wheel mechanisms connected to the body, and one or more other wheels. The wheel mechanism allows the base 10 to move along a desired path, while the one or more other wheels achieve balance and stability of the base 10 . The one or more other wheels may be castor wheels or omni-directional drive wheels.

In one embodiment, the lifting mechanism 20 is located on top of the base 10 . By actuation of the lift mechanism 20, the display 30 can move up and down in the vertical direction. When the display 30 is in the lowest retracted position, the lift mechanism 20 gives the robotic assistant 100 a limited height, which facilitates the stability of the robotic assistant 100 during movement and travel. The lifting mechanism 20 can be actuated to adjust the robotic assistant 100 to different heights, so that the robotic assistant 100 can flexibly adapt to users of different heights. Further description of lift mechanism 20 is provided below.

In one embodiment, the robotic assistant may include sensors that enable the robotic assistant 100 to perceive the environment in which the robotic assistant 100 is working. In one embodiment, the sensors may include ranging sensors that do not require physical contact with the detected object. They allow robotic assistant 100 to sense an obstacle without actually touching it. Range sensors may include infrared (IR) sensors, ultrasonic sensors, one or more light detection and ranging (LiDAR) sensors, near field communication (NFC), and RFID sensors/readers. In one embodiment, the sensors may include inertial measurement unit (IMU) sensors, each of which includes at least one accelerometer and at least one gyroscope. One or more LiDAR sensors are used to create a map of the environment. In conjunction with the IMU sensor, the LiDAR sensor is used to determine the real-time location of the robotic assistant 100 in the map of the environment. Data from the ranging sensors are used to detect obstacles, such as bumps, overhanging objects, spills, and other hazards, during the movement of the robotic assistant 100, and the robotic assistant 100 can alert the user to go around detected obstacles. These sensors may be positioned along base 10 or elsewhere on robotic assistant 100 .

The control system 50 is electrically connected with the base 10 , the lifting mechanism 20 and the sensors, and is configured to receive commands to control the robot assistant 100 . Commands may be received from control system 50 in response to movements/motions of robotic assistant 100 , or control system 50 may receive commands from a host computer wirelessly or through a wired connection or through a GUI on display 30 . The control system 50 can also directly receive command instructions from the user. For example, the robotic assistant 100 may detect whether the handle of the robotic assistant 100 is being held by the user. In some modes, the control system 50 receives commands after the user grips the handle. In response to the commands, the control system 50 controls the movement of the base 10 and controls the lifting mechanism 20 to drive the vertical movement of the display 30 . Further description of control system 50 is provided below.

In one embodiment, base 10 may be a differential drive platform. Base 10 may include two independently driven wheel mechanisms and a castor mechanism. The two wheel mechanisms are spaced apart from each other on opposite sides of the base 10 , their rotation axes are aligned with each other and extend along the width direction of the base 10 . The caster mechanism may include omni-directional wheels and is disposed adjacent to the end of the base 10 opposite to the wheel mechanism. It should be noted that the number and arrangement of the wheel mechanisms and caster mechanisms can be changed according to actual needs. For example, in an alternative embodiment, two wheel mechanisms and two caster mechanisms may be respectively provided at the four corners of the base 10 .

Referring to FIG. 3 b , in one embodiment, the lifting mechanism 20 may include an actuator 21 installed on the base 10 , a main body 23 vertically arranged on the base 10 , and a sliding mechanism slidably accommodated in the main body 23 . piece 25. The actuator 21 is used to drive the slider 25 to move up and down. The display 30 is thus movable between a lowermost retracted position (see Figures 1-3a) and a defined extended position (see, for example, Figure 3b).

In another embodiment, the lifting mechanism 20 may include a lifting mechanism disposed within the main body 23 and the slider 25 . The actuator 21 may be a linear motor, which is used to drive the lifting mechanism to extend or retract in the vertical direction. The actuator 21 is used to apply push force or pull force to the lifting mechanism, so as to drive the lifting mechanism to extend or retract in the vertical direction, thereby driving the sliding member 25 to move up and down in the vertical direction. In one embodiment, the lift mechanism may include a lead screw coupled to the output shaft of the motor, and a threaded collar coupled to and slidable along the lead screw. Rotational motion from the actuator 21 is converted into translational motion through the cooperation of the threaded collar and the lead screw. The lifting mechanism can thus drive the display 30 to move up and down.

In yet another embodiment, the lift mechanism may be a scissor lift mechanism. Specifically, the lifting mechanism may include one or more pairs of supports that are rotatably connected to each other and each pair of supports forms a crossing "X" pattern. The arrangement of these pairs of supports is well known and will not be described in detail here. It is important to note that lead screws and threaded collars and scissor lift mechanisms are just examples of lift mechanisms. The lifting mechanism can adopt other configurations according to actual needs.

In one embodiment, the robot assistant 100 may further include a first housing 201 installed on the top of the base 10 (see FIG. 1 ). The lifting mechanism 30 is disposed inside the first housing 201 .

Referring to FIG. 2 and FIG. 3 a , in an embodiment, the robot assistant 100 may further include a display support 301 located on the top of the lifting mechanism 20 and a motor 302 fixed on the display support 301 . The monitor 30 is indirectly installed on the lifting mechanism 20 through the monitor bracket 301 . The motor 302 is used to drive the display 30 to rotate relative to the display stand 301 . In one embodiment, the display stand 301 is made of a base plate 3011 and two vertical plates 3012 and 3013 . The base plate 3011 is fixed on the top of the slider 25 of the lifting mechanism 20 . Two vertical plates 3012 and 3013 are disposed on opposite sides of the base plate 3011 . The display 30 is rotatably connected to the upper ends of the vertical plates 3012 and 3013 . In one embodiment, the display 30 may define a U-shaped recess 31 in which the upper ends of the two vertical plates 3012 and 3013 are received and rotatably connected to the inner side surface of the recess 31 .

In one embodiment, the motor 302 is disposed in the space between the vertical plates 3012 and 3013 and fixed on the vertical plate 3012 . In this case, the rotating motor shaft of the motor 302 passes through a hole defined in the vertical plate 3012 and is fixed to the display 30 . The display 30 is thus able to rotate together with the motor shaft.

Referring to FIG. 4 and FIG. 5 , in one embodiment, the robot assistant 100 may further include a rotation damper 303 connected to the display stand 301 . The rotation damper 303 is configured to control the rotation speed of the display 30 . The rotary damper 303 is fixed to the vertical plate 3013 . In one embodiment, the display 30 is connected to the vertical plate 3013 through the connector 304 and the rotation damper 303 . The rotational damper 303 may define a through hole 3031 . In one embodiment, the through hole 3031 is defined in the rotor of the rotary damper 303 and is a square hole. The connecting piece 304 includes a main body 3041 and a rotating shaft 3042 . One end of the main body 3041 is fixed on the display 30 , and the other end is provided with a rotating shaft 3042 whose size and shape are determined according to the square through hole 3031 of the rotation damper 303 . The main body 3041 passes through the through hole 3014 on the vertical plate 3013 , and the shaft 3042 passes through the square through hole 3031 of the rotation damper 303 , so that the rotation can be transmitted from the display 30 to the rotation damper 303 . Specifically, when the link 304 rotates together with the display 30, the rotor of the rotary damper 303 is thus driven to rotate. There are various types of dampers to choose from. For example, rotational damper 303 may utilize the principles of fluid resistance to dampen motion. In this example, the rotation damper 303 may include a main body, a rotor, a cover, and oil filled in a space defined by the main body, the rotor, and the cover. The viscosity of the oil is used to provide a braking force to slow down the rotation of the display 30, which can ensure that the rotation of the display 30 is smooth and gentle. It should be noted that FIG. 4 is only an illustrative example, and other types of dampers can be used to control the speed of the display 30 according to actual needs.

Referring to FIG. 4 , in one embodiment, the robot assistant 100 may further include a limit switch 305 fixedly connected to the display stand 301 . The limit switch 305 is configured to be actuated in response to the display 30 being rotated to a predetermined position. Control system 50 is configured to stop rotation of display 30 in response to limit switch 305 being actuated. In one embodiment, the limit switch 305 is an optical limit switch and is positioned adjacent to the rotary damper 303 . Block 306 is secured to the end of shaft 3042 of connector 304 . Block 306 may thus rotate with display 30 . The limit switch 305 may be an infrared slotted optical switch, which may include an infrared source and filtered infrared phototransistor detector mounted exactly opposite each other with a small open gap between them. The limit switch 305 can detect the presence of an object in the gap that blocks light. When the end of the block 306 moves into the gap of the limit switch 305 , the limit switch 305 is activated and the control system 50 sends a signal to the motor 302 to stop the rotation of the display 30 . It should be noted that the limit switch 305 may be other types of switches, such as mechanical limit switches. In one embodiment, the predetermined position refers to the initial position as shown in FIGS. 1 and 2 . The end of the stopper 306 is accommodated in the gap of the limit switch 305 when the display 30 is in the initial position.

Referring again to FIGS. 2 and 3 a , in one embodiment, the robotic assistant 100 may also include two handles 60 that are fixedly connected to the lifting mechanism 20 . Two handles 60 are configured to fit the user's hands to provide two grips. The user can hold the two handles 60 when walking/standing, which enables the robot assistant 100 to provide upward support for the user, thereby helping the user maintain balance while walking/standing. In one embodiment, the two handles 60 are connected to the lifting mechanism 20 by a generally U-shaped rod 61 . The robot assistant 100 may further include a second housing 62 (see FIG. 1 ) disposed above the first housing 201 . The second housing 62 accommodates the U-shaped bar 61 and is fixed to the U-shaped bar 61 .

In one embodiment, displays 30 may be touch-sensitive display devices and each provide an input interface and an output interface between robotic assistant 100 and a user. Display 30 may display visual output to a user. Visual output can include graphics, text, icons, video, and any combination thereof. In one embodiment, when display 30 is in an initial position as shown in FIG. 1 , display 30 faces forward of robotic assistant 100 to display general information, or to allow telepresence for users who are not actively using the walking function. When the display 30 is rotated to the rearward facing position, the display 30 can display walking/exercise related information.

In one embodiment, the camera 40 can be an RGB camera and is arranged in the frame of the display 30 . As shown in FIG. 6 , when the display 30 is at an initial position, the camera 40 faces forward, and the camera 40 can be rotated together with the display 30 to a desired position to face rearward. The motion range of the display 30/camera 40 can be set to 165 degrees. However, the range of movement of the display 30/camera 40 can be changed according to actual needs.

Referring to FIG. 7 , in one embodiment, a control system 50 may include a processor 51 and a memory 52 storing computer readable instructions. Processor 51 runs or executes various software programs and/or instruction sets stored in memory 52 to perform various functions of robotic assistant 100 and process data. The processor 51 may be a central processing unit (CPU), a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, discrete gates, transistor logic device, discrete hardware components, or a combination of some or all of these components. A general purpose processor may be a microprocessor or any conventional processor or the like. Memory 52 may store software programs and/or sets of computer readable instructions and may include high speed random access memory and may include nonvolatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other nonvolatile solid state storage equipment.

In one embodiment, the robotic assistant 100 may include a plurality of sensors 70, including a 3D camera 72, a LiDAR sensor 73, a plurality of IR sensors 74, a plurality of ultrasonic sensors 75, and a plurality of IMU sensors 76. The 3D camera 72 may be provided at the on a casing 201 . The IR sensor 74 and the ultrasonic sensor 75 may be disposed on the first housing 201 . An IMU sensor 76 may be provided on the base 10 . The sensors 72 to 76 are configured to output data to the control system 50 so that the control system 50 can perform localization, motion planning, trajectory tracking control and obstacle avoidance for the robotic assistant 100 . In one embodiment, an electrocardiogram (ECG) sensor 77 may be embedded in the handle 60 to measure the heartbeat of a user holding the handle 60 . It should be noted that robotic assistant 100 may have many more sensors than shown.

In one embodiment, the robotic assistant 100 also includes a power supply system 81 for powering all key components of the robotic assistant 100 . Power system 81 is mounted on base 10 and may include a battery management system (BMS), one or more power sources (e.g., batteries, alternating current (AC)), a charging system, power failure detection circuitry, a power converter, or an inverter , power status indicators (eg, light emitting diodes (LEDs)), and any other components related to the generation, management, and distribution of power. Power system 81 may also include a self-charging unit that may engage with a fixed-location docking station to allow robotic assistant 100 to be charged. A battery management system manages a rechargeable battery such as protecting it from operation outside its safe operating area, monitoring its status, computing auxiliary data, reporting on that data, controlling its environment, validating it, and/or balancing it.

In one embodiment, the robotic assistant 100 may also include a speaker 82 and a microphone 83 that provide an audio interface between the user and the robotic assistant 100 . The microphone 83 receives audio data, converts the audio data into electrical signals, and transmits the electrical signals to the control system 50 as commands. Speaker 82 converts the electrical signal into sound waves audible to humans. Speaker 82 and microphone 83 enable voice interaction between the user and the robot assistant. Speaker 82 may play music or other audio content to the user for entertainment purposes. The robotic assistant 100 may also include a wireless communication interface 84, such as WiFi and Bluetooth modules. The robotic assistant 100 may also include an NFC subsystem 85, which may include an NFC chip and an antenna to communicate with another device/tag, which allows the NFC subsystem 85 to have NFC read functionality. NFC subsystem 85 may be used for authorization purposes. That is, NFC subsystem 85 may serve as a security mechanism to determine user privileges or access levels relative to system resources.

It should be noted that FIG. 7 shows only one example of robotic assistant 100, and that robotic assistant 100 may have more or fewer components than shown, may combine two or more components, or may have different Component configuration or arrangement. For example, robotic assistant 100 may include front and rear light strips to illuminate the path for the user when the environment is dark. The robotic assistant 100 may include a storage unit for storing items so that the robotic assistant 100 may deliver the items to desired locations. The various components shown in Figure 7 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

FIG. 8 is a flowchart illustrating a method of controlling the robot assistant 100 according to one embodiment, which includes the following steps. It should be noted that the sequence of steps shown in FIG. 8 is not limiting, and can be changed according to actual needs.

Step S101: Receive a command instruction. The processor 51 of the control system 50 receives the command instructions. For example, processor 51 may receive commands from a user (eg, a care receiver) requesting robotic assistant 100 to retrieve an object from one location and deliver the object to another location.

Step S201: move the base 10 in response to the first command instruction. The processor 51 may analyze each command instruction and move the base 10 to a determined position in response to the first command instruction. The first command instruction may include a description of where the robotic assistant 100 needs to go. For example, when a user (eg, a care recipient) requests the robot assistant 100 to fetch and deliver an object, the first command instruction may include a description of the starting location where the object is stored and the target location where the object needs to be delivered. Processor 51 may execute software programs and/or instruction sets stored in memory 52 to perform localization, motion planning, and trajectory tracking so that base 10 may determine its real-time position on a known map during movement along a planned path. If there is a dynamic obstacle on the planned path, the processor 51 can plan a new path to avoid the obstacle. In other words, the base 10 can be controlled to move along a prescribed path, which path will be adjusted if there is an obstacle in the path. The base 10 can first move autonomously to the starting position and then to the target position. Additionally, the base 10 may be controlled through on-screen commands or control inputs inferred from the handle, which may be connected to a load cell. This allows the user to directly control the movement of the base 10 .

Step S301: Control the lifting mechanism 20 to move the display 30 and the handle 60 up and down in response to the second command. The processor 51 can analyze each command instruction and control the lifting mechanism 20 to move the display 30 and the handle 60 up and down in response to the second command instruction. For example, the processor 51 may receive command instructions from a user (eg, a care receiver) and control the robotic assistant 100 to autonomously move between determined locations. In this case, the processor 51 controls the lifting mechanism 20 to move the display 30 and the handle 60 down to the lowest retracted position (see FIG. 1 ), so that the robot assistant 100 can have a limited height, which is helpful for moving the robot assistant 100. Movement of assistant 100 and stability during travel. Processor 51 may receive command instructions from a user (e.g., a care receiver) who requests assistance from robotic assistant 100 while the user is walking, processor 51 may then determine the height of the user, and may align display 30 according to the height of the user. and the handle 60 moves upward to the extended position. In this case, the extended position is not a fixed position and can vary according to the height of the user. With this configuration, the robotic assistant 100 can have the flexibility to adapt to different users of different heights, which allows different users to walk and push the robotic assistant 100 in a substantially upright posture.

Step S401: Rotate the display 30 in response to a third command. The processor 51 can analyze each command instruction, and rotate the display 30 according to the third command instruction. For example, the processor 51 may receive command instructions from a user (eg, a care receiver) and control the robotic assistant 100 to autonomously move between determined locations. In this case, the processor 51 rotates the display 30 to its initial position as shown in FIG. 1 so that the camera 40 faces forward and can detect objects in front of the robot assistant 100 so that the robot assistant 100 can sense the environment. The processor 51 may receive a command instruction from a user (for example, a care receiver) who requests the robot assistant 100 to provide assistance while the user is walking, the processor 51 rotates the display 30 to a position where the camera 40 faces backward and may detect the user's Facial expressions or other biometrics. As a result, the robotic assistant 100 can monitor the user's fatigue.

In one embodiment, robotic assistant 100 can work in different modes. For example, robotic assistant 100 may operate in a first mode or an autonomous mode. In this mode, the control system 50 can perform positioning, motion planning, trajectory tracking control and obstacle avoidance according to the data output by the sensors 72 to 76, which enables the robot assistant 100 to move autonomously between the initial position and the target position. thereby completing the assigned tasks. The robotic assistant 100 can work in the second mode or sleep mode. In this mode, robotic assistant 100 enters a low power state and remains in this state. When the robotic assistant 100 in the first mode does not receive a user input for a preset period of time (eg, 10 minutes) or the robotic assistant 100 is charged, the robotic assistant 100 is switched to the second mode. The robotic assistant 100 may switch back to the first mode after receiving a command from the user, such as a voice command, a touch on the display 30, or the like.

Robotic assistant 100 can work in a third mode or stand assist mode. In this mode, robotic assistant 100 acts as a stable structure from which the user can grasp handle 60 and stand up from a sitting position. After the robotic assistant 100 in the first mode approaches a seated user, the robotic assistant 100 may switch to the third mode. The robotic assistant 100 in the third mode can switch to the first mode when there is no physical task. Robotic assistant 100 may operate in a fourth mode or walking assistance mode. In this mode, robotic assistant 100 is ready to be pushed by the user and to help support some of the user's body weight while the user is walking. After the robotic assistant 100 in the first mode approaches a standing user, the robotic assistant 100 may switch to the fourth mode. The robotic assistant 100 in the fourth mode can switch to the first mode when there is no physical task.

Robotic assistant 100 may operate in a fifth mode or training mode. In this mode, robotic assistant 100 is ready to be pushed by the user and to help support some of the user's body weight while the user is walking. After the robotic assistant 100 in the first mode approaches a standing user, the robotic assistant 100 may switch to the fifth mode. The robotic assistant 100 in the fifth mode can switch to the first mode when there is no physical task. The difference between the training mode and the walking assistance mode is that the robotic assistant 100 in the training mode can apply additional resistance to the user, so he/she has to make an extra effort to push the robotic assistant forward or side to side, thus giving enough training time Increases muscle strength and coordination. In one embodiment, the base 10 may also include a stopper. When the robotic assistant switches to training mode, the processor 51 controls the brakes to press against the moving wheels of the base 10 to create friction. In this case, the user needs to exert more force on the robot assistant 100 to increase muscle strength and coordination with sufficient training time. It should be noted that robotic assistant 100 may have more modes of operation than those discussed above.

In one embodiment, in the training mode, the robotic assistant 100 may provide assistance/guidance to the user doing squats. Here, the squat is a strength exercise in which the trainee lowers the hips from a standing position and then rises back up. FIG. 9 shows an exemplary flowchart of a method for controlling a robotic assistant when a user performs a squat. The method may include the following steps.

Step S1001: Detect the movement of the user's face in the vertical direction according to the images captured by the camera 40 .

If the user wants to get help/guidance from the robot assistant 100 while he/she is doing squats, he/she needs to stand near and behind the robot assistant 100 . After receiving the crouching exercise command from the user, the processor 51 controls the display 30 to rotate so that the camera 40 can face backward to capture images of the environment behind the robot assistant 100 . During the user's squatting exercise, the camera 40 is controlled to take images of the environment behind the robot assistant 100 at predetermined intervals. The processor 51 may detect the movement of the user's face in the vertical direction based on the image of the environment behind the robot assistant 100 . Processor 51 may compare two or more images taken consecutively.

In one embodiment, processor 51 compares two consecutively captured images. Specifically, image 1 in FIG. 10 represents a previously captured image, and image 2 represents a currently captured image. Processor 51 may identify the user's face in Image 1 and Image 2 and determine the location of the face in Image 1 and Image 2 . In one embodiment, the location of the face refers to the center of the bounding box of the face in images 1 and 2 . By comparing the positions of the faces in images 1 and 2, the processor 51 can determine that the user's face has moved downward.

Step S1002: In response to detecting the vertical movement of the user's face, rotate the display 30 and activate the lifting mechanism 20 to move the display 30 up and down to allow the camera 40 to face the user's face during the vertical movement of the user's face.

In one embodiment, when the user's face moves downward, the processor 51 controls the lifting mechanism 20 to move the display 30 downward by a predetermined distance, and when the user's face moves upward, controls the lifting mechanism 30 to move the display 30 upward by a predetermined distance. Processor 51 then rotates display 30 until camera 40 faces the user's face. In this way, the camera 40 can keep facing the user's face, which allows the user's face to continuously appear in the middle of the display 30 to obtain a better display operation experience.

Referring to FIG. 11 , in one embodiment, rotating the display 30 may include the following steps. Step S2001: Determine the key points of the user's face in the current image captured by the camera 40 .

Referring to FIG. 12 , in one embodiment, key points may be the center between the user's eyes, the center of the user's mouth, the tip of the user's nose, and the like. In this embodiment, the key point is the center P between the user's eyes. The processor 51 may first determine the center of the user's eyes, and then determine the midpoint of a line segment formed by connecting the two centers of the user's eyes. The midpoint is then identified as a keypoint.

和/>

其中P_x表示关键点P的x坐标，A_x，B_x，C_x，D_x分别表示顶点A、B、C、D的x坐标，P_y表示关键点P的y坐标，A_y、B_y、C_y、D_y表示顶点A、B、C的y坐标。在一个实施例中，当/>

时确定用户的面部位于显示器30的中间，其中，H代表图1所示图像的高度。图12中的坐标系定义如下：坐标系原点为图像的左上角，x轴和y轴分别沿图像的宽度和高度延伸。In one embodiment, points A, B, C and D in Figure 12 can represent four vertices of the bounding box, and the position of the key point P can be calculated according to the following formula:

and />

Among them, P _x represents the x coordinate of key point P, A _x , B _x , C _x , D _x represent the x coordinates of vertices A, B, C, D respectively, P _y represents the y coordinate of key point P, A _y , B _y , C _y , and D _y represent the y-coordinates of vertices A, B, and C. In one embodiment, when />

It is determined that the user's face is located in the middle of the display 30, where H represents the height of the image shown in FIG. 1 . The coordinate system in Figure 12 is defined as follows: the origin of the coordinate system is the upper left corner of the image, and the x-axis and y-axis extend along the width and height of the image, respectively.

Step S2002: Determine the angle between the line passing through the key point P and the center of the camera and the optical axis of the camera 40 .

FIG. 13 is a schematic diagram showing a simplified model of robotic assistant 100 with camera 40 facing backwards. The simplified model of the robotic assistant 100 has a vertical translational degree of freedom (DOF) and a rotational degree of freedom. A coordinate system x ₃ y ₃ z ₃ is constructed with the camera center as the origin C, and the z-axis of the coordinate system x ₃ y ₃ z ₃ extends along the optical axis of the camera 40, which starts from the focal point and is perpendicular to the image plane straight line. In one embodiment, a pinhole camera model is used to model the camera 40 . As shown in Figures 14a and 14b, in this model, conceptually, all light passes through a small pinhole placed in a nearly disappearing pinhole and illuminates the image plane below it. The image formed on the image plane obeys the laws of projective geometry. The pinhole of the pinhole camera model is defined as the "camera center" above. Therefore, the angle θ _obj between the z-axis and the line segment CP in FIG. 13 is the angle between the line passing through the key point P and the center of the camera and the optical axis of the camera 40 . The angle θ _obj may also be referred to as the pitch angle of the user's face.

并且/>

其中，f_{focal_length}表示摄像头中心到图像平面的距离。根据这两个方程，可以得到如下方程：/>

根据三角形相似定理，/>

由于/>

并且/>

可以得到如下方程：/>

AD和M⁰C可以预先测量，A⁰D⁰是通过计算点A⁰和D⁰之间的像素数确定的，N¹Q¹是通过计算点N¹和Q¹之间的像素数确定的。以这种方式，因此可以确定站在机器人助理100后面的随机当前位置处的用户面部的俯仰角θ¹ _obj。The principle of calculating the angle θ _obj is described below. Fig. 14a is a diagram showing the relationship between the user's face and the image of the user's face in the image plane when the user stands at a predetermined position from the center C of the camera. Fig. 14b is a diagram showing a relationship between a user's face and an image of the user's face in an image plane when the user stands at a random current position. In Figures 14a and 14b, the user's face is represented by a line segment AD perpendicular to the main axis passing through the camera center C and perpendicular to the image plane. The projection points of the line segment AD on the main axis are denoted by M ⁰ and M ¹ . Points M ⁰ and M ¹ are mapped/projected to N ⁰ and N ¹ in the image plane. Endpoints A and D are mapped/projected as A ⁰ and D ⁰ in the image plane of Figure 14a, and as A ¹ and D ¹ in the image plane of Figure 14b. The keypoint P in Figures 14a and 14b is mapped/projected as ^Q0 and ^Q1 in the image plane in Figures 14a and 14b. According to the triangle similarity theorem,

and />

Among them, f _{focal_length} represents the distance from the camera center to the image plane. According to these two equations, the following equation can be obtained: />

According to the triangle similarity theorem,

due to />

and />

The following equation can be obtained: />

AD and M ⁰ C can be measured in advance, A ⁰ D ⁰ is determined by calculating the number of pixels between points A ⁰ and D ⁰ , N ¹ Q ¹ is determined by calculating the number of pixels between points N ¹ and Q ¹ . In this way, it is thus possible to determine the pitch angle θ ¹ _obj of the user's face standing behind the robotic assistant 100 at a random current position.

Step S2003: Determine the moving direction of the user's face in the vertical direction. In one embodiment, the processor 51 may determine the direction of movement of the user's face in the vertical direction by comparing two or more images taken continuously, which has been discussed in conjunction with FIG. 10 .

Step S2004: activate the lifting mechanism to move the display up or down based on the moving direction of the user's face in the vertical direction. Specifically, when the user's face moves downward, the lifting mechanism 20 is controlled to move the display 30 downward by a predetermined distance, and when the user's face moves upward, the lifting mechanism 20 is controlled to move the display 30 upward by a predetermined distance.

Step S2005: Rotate the display based on the moving direction of the user's face in the vertical direction and the angle between the line passing through the key point P and the center of the camera and the optical axis of the camera 40 . In this embodiment, the processor 51 rotates the display 30 and at the same time controls the lift mechanism 20 to move the display 30 up or down by a predetermined distance until the camera 40 faces the user's face.

(见图12)。PID控制器可以包括比例控制器，其针对关键点P的目标位置和关键点P的当前位置之间的差异应用适当的比例变化。PID控制器可以包括积分控制器，其检查关键点P随时间的位置和关键点P的目标位置的偏移，然后在必要时校正控制器输出。PID控制器可以包括微分控制器，该微分控制器监测关键点P的位置变化率，并在出现异常变化时相应地改变控制器输出。Referring to FIG. 15 , in one embodiment, the control system 50 may include a visual servoing system including a proportional-integral-derivative (PID) controller. The PID controller may receive the difference between the target position of the keypoint P and the current position of the keypoint P. The target position here is the position of the key point P in the middle of the display 30, that is,

(See Figure 12). The PID controller may comprise a proportional controller that applies an appropriate proportional change for the difference between the target position of the keypoint P and the current position of the keypoint P. The PID controller may include an integral controller that checks the position of the keypoint P over time for deviations from the target position of the keypoint P, and then corrects the controller output if necessary. The PID controller may include a differential controller that monitors the rate of change of the position of the key point P and changes the controller output accordingly when there is an abnormal change.

The control system 50 may include a torso control system that receives a controller output from a PID controller of the visual servoing system. The pitch angle θ ¹ _obj of the user's face at the current position standing behind the robotic assistant 100 is also input into the trunk control system. The trunk control system may include a PID speed controller for controlling the lifting mechanism 20 . After determining the moving direction of the user's face, the PID speed controller controls the lifting mechanism 20 to move the display 30 up or down by a certain distance, so that the pitch angle θ ¹ _obj decreases by θ ^1" _obj . The trunk control system may include The PID position controller is used to control the display 30 to rotate so that the pitch angle θ ¹ _obj decreases by θ ^1' _obj . θ ^1' _obj and θ ^1" _obj satisfy the following equation: θ ^1' _obj + θ ^1" _obj = θ ¹ _obj . Therefore, after the display 30 is moved up or down by a certain distance and rotated by the angle θ ^1' _obj , the pitch angle θ ¹ _obj is equal to 0, which means that the key point P has moved from the current position to the target position.

Control system 50 may include a dual mode controller that may receive an output from a PID position controller to rotate display 30 . The dual mode controller can also release the motor 302 so that the display 30 can be manually rotated by the user. FIG. 16 is a flowchart of a method for controlling the display 30 in the automatic control mode and the manual control mode. The method may include the following steps.

Step S3001: Receive the angle signal from the PID position controller.

The dual-mode controller receives an angle signal from the PID position controller to rotate the display 30 by an angle θ ^1' _obj .

Step S3002: Measure the current of the motor 302 used to rotate the display 30.

When no external force is applied to the display 30, the current of the motor 302 will be less than the minimum threshold. When the user applies an external force to the display 30 to manually rotate the display 30, the current of the motor 302 will be greater than the maximum threshold. By measuring and monitoring the current of the motor 302 , it can be judged whether the user exerts an external force on the display 30 .

Step S3003: Determine whether the current is greater than a threshold within a preset time.

For example, if the current is greater than the maximum threshold for 2 seconds, it is determined that the user has applied an external force to the display 30 . If so, the flow proceeds to step S3004; otherwise, the flow proceeds to step S3005.

Step S3004: Release the motor 302 for manual operation.

After detecting the external force from the user, the processor 51 will release the motor 302 . For example, motor 302 may be disengaged from display 30 , which releases display 30 and allows a user to manually rotate display 30 .

Step S3005: continue to send the position command to the motor 302 .

If no external force is exerted on the display 30, the processor 51 will continue to send position commands to the motor 302, so that the display 30 can rotate to a desired position according to the angle signal from the PID position controller.

Step S3006: Measure the current of the motor 302 for rotating the display 30.

After the motor 302 is released, the current of the motor 302 will be measured and monitored so that it can be determined whether the external force is still exerted on the display 30 .

Step S3007: Determine whether the current is less than a threshold within a preset time period.

When the current is less than the minimum threshold for a preset time (for example, 2 seconds), it is determined that the external force applied to the display 30 has stopped; otherwise, it is determined that the external force is still applied to the display 30 . If the current is less than the minimum threshold for a preset time, the process returns to step S3002. If the current is not less than the minimum threshold within the preset time period, the process returns to step S3006.

The method shown in FIG. 16 allows the display 30 to automatically rotate to a position where the camera 40 faces the user's face, and allows the user to manually rotate the display 30 to a desired position. After the external force ceases, the display 30 will switch from manual control mode to automatic control mode.

It should be appreciated that the above disclosure details several embodiments of a robotic assistant 100 that may provide walking assistance and fall prevention. As noted above, robotic assistant 100 may be used in assisted living or healthcare settings. However, the present disclosure is not limited thereto. In other exemplary usage scenarios, robotic assistant 100 may be used in a hospital.

With the above configuration, the robotic assistant can facilitate the active lifestyle of the elderly. Robotic assistants could allow them to do more exercise to maintain their mobility. Moving around also provides older adults with more opportunities to interact with others (especially in a nursing or assisted living facility), thereby reducing their feelings of isolation. When the squatting user is properly standing behind the robot assistant, the camera can be controlled to face the user's face all the time, so that the user's face appears in the center of the display. Robotic assistants can provide guidance/assistance by displaying information on a display, such as squat times.

It is intended that the specification and examples be considered exemplary only, with the true scope indicated by the claims and their equivalents.

The foregoing description, for purposes of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiment was chosen and described in order to best explain the principles of the disclosure and its practical application, to thereby enable others skilled in the art to best utilize the disclosure and the various embodiments with various modifications as are suited to the particular use contemplated. .

Claims (10)

1. A robotic assistant, comprising:

a base;

a lifting mechanism positioned on the base;

a display rotatably mounted on the lifting mechanism;

a camera positioned on the display, the camera configured to capture an image of a user; and

a control system that receives command instructions;

wherein, in response to a command instruction, the control system is configured to detect movement of the user's face in a vertical direction based on the image;

wherein, in response to detecting movement of the user's face in a vertical direction, the control system is configured to rotate the display and actuate the lift mechanism to move the display up and down to face the user's face during movement of the user's face in the vertical direction.

2. The robotic assistant of claim 1, further comprising a display bracket positioned on top of the lift mechanism and a motor secured to the display bracket, wherein the display is mounted to the lift mechanism indirectly through the display bracket, the motor configured to drive the display to rotate relative to the display bracket.

3. The robotic assistant of claim 2, further comprising a rotational damper coupled to the display stand, wherein the rotational damper is configured to control a rotational speed of the display.

4. The robotic assistant of claim 1, further comprising two handles fixedly connected to the lifting mechanism, wherein the two handles are configured to fit a user's hand to provide two grips.

5. The robotic assistant of claim 2, further comprising a limit switch fixedly connected to the display bracket, wherein the limit switch is configured to be activated in response to rotation of the display to a predetermined position, and the control system is configured to stop rotation of the display based on the limit switch being activated.

6. The robotic assistant of claim 1, wherein the lifting mechanism comprises an actuator mounted on the base, a body vertically disposed on the base, and a slider slidably coupled to the body, the display being located on the slider, the actuator being configured to drive the slider up and down.

7. A robotic assistant, comprising:

a base;

the lifting mechanism is vertically arranged and installed on the base;

a display rotatably connected to the lifting mechanism;

a camera secured to the display, the camera configured to capture an image of a user;

One or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising:

instructions to detect movement of a user's face in a vertical direction based on the image; and

in response to detecting movement of the user's face in a vertical direction, rotating the display and actuating the lift mechanism to move the display up and down to allow the camera to face the user's face during movement of the user's face in the vertical direction.

8. A method for controlling a robotic assistant, comprising:

providing a base;

providing a lifting mechanism positioned on the base;

providing a display rotatably connected to the lifting mechanism;

providing a camera secured to the display;

obtaining a plurality of images of a user by using the camera;

detecting movement of the face of the user in a vertical direction based on the image;

in response to detecting movement of the user's face in a vertical direction, the display is rotated and the lift mechanism is actuated to move the display up and down to allow the camera to face the user's face during movement of the user's face in the vertical direction.

9. The method of controlling a robotic assistant of claim 8, wherein rotating the display comprises:

determining key points of a user face in a current image in the images;

determining an included angle between a connecting line passing through the key point and the center of a camera and the optical axis of the camera;

determining the moving direction of the face of the user in the vertical direction; and

the display is rotated based on the direction of movement of the user's face in the vertical direction and the angle.

10. The method of controlling a robotic assistant of claim 9, wherein after determining a direction of movement of a user's face in a vertical direction, actuating the lifting mechanism to move the display up and down comprises:

the lifting mechanism is activated to move the display up or down depending on the direction of movement of the user's face in the vertical direction.

Images