CN108524121A - The intelligent wheel chair of screen is controlled based on eyeball - Google Patents

Abstract

The present invention proposes an intelligent wheelchair based on an eyeball control screen, which aims to effectively expand the range of eye control, including an electric wheelchair, and an information collection device, an information processing device, an information display device and a fixing device fixed on the electric wheelchair The first bracket and the second bracket, wherein, the information collection device is used to capture the user's face image, and the information processing device extracts eye information from the captured image, judges the state of eye opening and closing, and pupil rotation, and uses the judged While the eye movement state is converted into commands to control the movement of the wheelchair, it is also converted into commands to control the movement of the cursor and click on the content displayed on the electronic display according to different rules. The user controls the starting, braking and left and right of the electric wheelchair through effective blinking and pupil rotation. Rotating, when the wheelchair is stationary, the user controls the mouse to click and move on the electronic display screen through effective blinking and pupil rotation respectively, realizing the communication between the user and the outside world through the eyes.

Description

Smart wheelchair based on eyeball control screen

technical field

The invention belongs to the field of computer technology, and relates to an intelligent wheelchair, in particular to an intelligent wheelchair based on an eyeball-controlled screen, which can be used by disabled persons such as patients with "progressive anthropomorphism" who can only communicate with the outside world through their eyes. While the movement is realized by the rotation of the pupil of the eye and the change of the opening and closing of the eye, the movement of the cursor and the click on the content displayed on the electronic display screen can be controlled through the eyes, so that the user can communicate with the outside world through the eyes.

technical background

At present, wheelchairs have become the means of transportation for the elderly and the disabled with limited mobility. Especially for the patients with ALS in the late stage of the disease, only their eyes can move freely in their whole body, and they can use ordinary eye-controlled smart wheelchairs to move around, but they cannot operate other equipment normally. A kind of torment, with this as the background, transform the wheelchair device that is indispensable in the life of "gradually frozen people" into a set that can not only help them control the movement of the electric wheelchair through their eyes, but also help them open and close the state of the eyes and the rotation of the pupils. The state is converted into an intelligent wheelchair system that controls the movement of the cursor and clicks on the content displayed on the electronic display screen.

For example, the application publication number is CN107260420A, and the patent application titled "Intelligent Wheelchair Human-Computer Interaction Control System and Method Based on Eye Movement Recognition" discloses an intelligent wheelchair human-computer interaction control system based on eye movement recognition. The eye electromechanical acquisition module collects eye electromechanical signals, extracts conscious blink signals from the obtained electromechanical signals, and realizes the control of the direction of wheelchair movement according to the combination of various conscious eye blink signals. The fuzzy controller part uses signal myoelectric integration The value and signal slope are used as input, and the driving speed of the intelligent wheelchair is used as the output, so as to realize the precise control of the driving speed of the wheelchair. This invention realizes using the eyes to control the movement of the wheelchair. The shortcoming is that when the sensor collects the electromechanical signals of the user's eyes when the user blinks, the left and right eyes blink in different order and times to represent different commands. This operation method lacks Versatility, and the action of blinking controls different commands, and one-to-many commands are prone to mistakes.

In 2017, He Rentian and others disclosed a method of "design and realization of an eyeball-controlled intelligent wheelchair" in the first issue of the "Information and Communication" journal. They designed an eyeball-signal-controlled intelligent wheelchair, and obtained user information through an information collection device. The eyeball signal is then extracted by the information processing device to judge the state of the eye, and the change of the eye information is converted into commands to control the wheelchair's forward, backward, left turn and right turn, so as to realize the motion control of the wheelchair by the eyes. This invention realizes the use of eyes to control the movement of electric wheelchairs, and at the same time solves the lack of versatility and one-to-many commands in the operation method when the eyes control the movement of wheelchairs in the published patent "Intelligent Wheelchair Human-Computer Interaction Control System and Method Based on Eye Movement Recognition" The problem that is easy to cause mistakes. The shortcoming of this invention is that it only realizes the control of the wheelchair through the eye information, and does not further explore the processing ability of the information processing device for the eye information. Wider range of eye control for people with disabilities who only have eye movement.

Contents of the invention

The object of the present invention is to address the defects in the above-mentioned prior art, and propose a smart wheelchair based on the eyeball control screen. When the user controls the motion state of the electric wheelchair through the opening and closing state of the eyes and the rotating state of the pupil, it can realize The control of cursor movement and clicking on the content displayed on the electronic display screen effectively broadens the scope of eye control and enables users to communicate with the outside world through their eyes.

In order to achieve the above object, the technical scheme that the present invention takes is:

1. An intelligent wheelchair based on an eyeball control screen, comprising an electric wheelchair 1, an information collection device 2 and an information processing device 5, and the information collection device 2 includes a camera fixed in front of the electric wheelchair 1 through a first bracket 3 for taking images of the user's face The information processing device 5 includes a central processing unit, which is used to judge the opening and closing state of the user's eyes and the rotation state of the pupil by extracting the opening and closing information of the user's eyes and the position information of the pupil in the multi-frame user's facial image taken by the camera, and The judgment result is converted into a control command for controlling the motion state of the electric wheelchair 1; an electronic display device 4 is also fixed on the electric wheelchair 1, and the electronic display device 4 includes an electronic display screen fixed on the front of the electric wheelchair 1 by a second bracket 6; the central processing When the electric wheelchair 1 stops moving, the device converts the opening and closing state of the user's eyes and the movement state of the pupil into control commands for controlling the movement of the cursor and clicking on the content displayed on the electronic display screen, so that the user can interact with the outside world through the eyes.

In the smart wheelchair based on the eyeball control screen, the first bracket 3 includes a first bracket body composed of a horizontal bar and a vertical bar. The vertical bar is fixed on the foot frame on the front end side of the electric wheelchair 1. The direction of the central axis of the camera can be adjusted, and can rotate around a fixed point; the base for fixing the camera is installed on the cross bar, which can realize the camera rotating up and down around the installation point, and can move left and right along the cross bar.

The above-mentioned smart wheelchair based on the eyeball control screen, the second bracket 6 includes a second bracket body composed of a horizontal bar and a vertical bar, and the vertical bar is fixed on the front end of the electric wheelchair 1 opposite to the foot on the other side of the first bracket 3 on the shelf.

The above-mentioned smart wheelchair based on the eyeball control screen converts the result of the CPU judging the opening and closing state of the user's eyes and the rotation state of the pupil into a control command for controlling the motion state of the electric wheelchair 1, wherein:

Convert the opening and closing state of the eyes to its corresponding control command, and the conversion rule is:

When the user blinks 4 times in a row, the electric wheelchair starts to move, and when the user blinks 3 times in a row, the electric wheelchair stops moving;

Convert the rotation state of the pupil of the eye into its corresponding control command, and the conversion rule is:

The position of the pupil when the user looks straight ahead is the reference point, and the forward direction is the starting point. When the user looks to the left side of the body, the pupil position is shifted to the left based on the reference point, and the electric wheelchair will move to the left. Rotation, the rotation angle is proportional to the distance the pupil deviates from the reference point; when the user looks to the right side of the body, the pupil position shifts to the right based on the reference point, and the electric wheelchair will turn to the right, the rotation angle is proportional to the distance the pupil deviates from the reference point Proportional; when the user's eyes look at the ground, the pupil position is shifted downward based on the reference point, and the electric wheelchair moves backwards.

The above-mentioned smart wheelchair based on the eyeball control screen converts the opening and closing state of the eyes and the rotation state of the pupil into control commands for controlling the movement of the cursor and clicking on the content displayed on the electronic display screen, wherein:

Convert the opening and closing state of the eyes to its corresponding control command, and the conversion rule is:

When the user's eyes stare at a certain point on the electronic display screen, effectively blinking once means clicking the point, and effectively blinking twice in a row means double-clicking the point, and the effective blinking means that the eye closure time period is 1 second to 2 seconds;

Convert the rotation state of the pupil of the eye into its corresponding control command, and the conversion rule is:

The user looks around the electronic display screen for a week along the edge, and divides the movement range of the pupil in the eye socket into n×n equal rectangular small grids, n≥2, and at the same time divides the electronic display screen into n×n equal The large rectangular grid of the above-mentioned two n×n rectangular grids establishes the mapping relationship of the corresponding positions. When the user's line of sight is fixed on a certain position of the electronic display screen, the corresponding eye pupil also stays in the corresponding small grid. Relative position, according to the mapping relationship between the two grids, the cursor will move to a certain position on the electronic display; when the eyes look at the lower edge of the electronic display, the pupil is located at the lower edge of the small grid, and the opened document or web page window The scroll bar scrolls down; when the pupil is on the upper edge of the electronic display, the open document or webpage scroll bar scrolls up.

Compared with the prior art, the present invention has the following characteristics:

1. Since the present invention is equipped with an electronic display screen in front of the electric wheelchair, when the user realizes the control of the motion state of the electric wheelchair through the opening and closing state of the eyes and the rotation state of the pupil, it realizes the movement of the cursor and clicking on the electronic display screen. The control of the displayed content enables the user to communicate with the outside world through the eyes. Compared with the prior art, the scope of eye control is effectively widened and the life of the user is enriched.

2. The present invention converts the opening and closing of the user's eyes and the state of eyeball rotation into commands for cursor movement and clicking on the content displayed on the electronic display screen according to certain conversion rules through the central processing unit. The conversion rules are simple and easy to operate.

4. The first bracket and the second bracket used to fix the electronic display device and the information collection device in the present invention can be stretched up and down and rotate around the corresponding fixed points with the two sides of the electric wheelchair, which increases the comfort of the user. And can satisfy different users.

Description of drawings

Fig. 1 is the overall structural representation of the present invention;

Fig. 2 is the realization flowchart of the present invention.

Detailed ways

Below in conjunction with accompanying drawing and specific implementation method, the present invention is described in further detail:

Referring to Fig. 1, the intelligent wheelchair based on the eyeball control screen includes an electric wheelchair 1, an information collection device 2 and an information processing device 5, wherein the information collection device 2 includes a camera fixed in front of the electric wheelchair 1 through a first bracket 3 for Take the user's facial image; the information processing device 5 includes a central processing unit for judging the opening and closing state of the user's eyes and the rotation of the pupil by extracting the opening and closing information of the user's eyes and the position information of the pupils in the multi-frame user's facial images captured by the camera state, and the judgment result is converted into a control command to control the motion state of the electric wheelchair 1; an electronic display device 4 is also fixed on the electric wheelchair 1, and the electronic display device 4 includes an electronic display that is fixed on the front of the electric wheelchair 1 through the second bracket 6 screen; when the electric wheelchair 1 stops moving, the central processing unit converts the opening and closing state of the user's eyes and the movement state of the pupils into operation commands for controlling the movement of the cursor and clicking on the content displayed on the electronic display screen.

The first bracket 3 is formed by vertical welding of a horizontal bar and a vertical bar. The specific structure and composition are as follows: the vertical bar is a telescopic bar that can expand and contract along the central axis, so that the fixed information collection device 2 can move up and down. , the vertical rod is fixed on the right foot frame of the electric wheelchair in a direction perpendicular to the base of the electric wheelchair, and can rotate along with the right foot frame along the fixed point of the right foot frame to provide convenience for users to get on and off the wheelchair. In addition, the vertical rod There is a groove in the same direction as the horizontal bar in the lower part of the mobile phone; the horizontal bar is located in front of the user and is used to fix the information collection device. The significance of the first bracket 3 is to fix the information collection device 2 and realize the movement of the information collection device 2 up, down, left, and right, so as to improve user comfort and meet the needs of different users.

The second support 6 is vertically welded by a horizontal bar and a vertical bar: wherein the vertical bar is a telescopic bar that can expand and contract along the central axis, which can realize the up and down movement of its fixed electronic display device 4, and the vertical bar is vertically It is fixed on the left leg of the electric wheelchair in the direction of the base of the electric wheelchair, and can rotate along with the left leg along the fixed point of the left leg to provide convenience for the user to get on and off the wheelchair; the horizontal bar is located in front of the user and is located in the first bracket 3 Just below the cross bar is used to fix the electronic display screen through a V-shaped bracket. When the user starts to use, the cross bar can be embedded in the groove of the vertical bar of the first bracket 3 to protect the user. The significance of the second bracket 6 is to fix the electronic display device 4 and realize the up and down movement of the electronic display device 4, so as to improve user comfort and meet the needs of different users.

The information collection device 2 includes a camera and a camera base. The camera is clamped on the first bracket 3 through the camera base, facing the user, and is used to collect user facial images. The camera base can move left and right on the first bracket 3, and the camera can also rotate up and down. . The camera is wirelessly connected to the central processing unit, and sends the captured user's face image to the central processing unit in real time.

The information processing device 5 is mainly a central processing unit. The central processing unit is connected to the electronic display screen through wires, and is packaged with an information processing device shell with holes, and is wirelessly connected to the camera to obtain real-time user face images. The entire information processing device 5 is placed under the support of the electric wheelchair 1 to avoid extra space occupation.

The electronic display device 4 includes an electronic display screen and a V-shaped bracket. The V-shaped bracket is fixed at the midpoint of the crossbar of the second bracket 6 and faces the user. And the command after conversion, realize the opening and closing state of the user's eyes and the rotation state of the user's eye pupil to the control of the cursor movement, click and window scroll wheel of the electronic display screen. In this example, a 14.1-inch ordinary computer display screen is used.

When the user converts the opening and closing state of the eyes and the rotation state of the pupils of the eyes into their corresponding control commands, the conversion rule is: when the user's eyes blink effectively 4 times in a row, in this example, the effective blinking is defined as the eyeball closing time lasting 1 second At 1.5 seconds, the electric wheelchair starts to move, the user effectively blinks 3 times in a row, and the electric wheelchair stops moving; the position of the pupil when the user looks straight ahead is used as the starting reference point, and the forward direction is taken as the starting direction, when the user Looking to the left side of the body, the pupil position is shifted to the left based on the reference point, and the electric wheelchair will turn to the left, and the rotation angle is proportional to the distance of the pupil from the reference point; when the user looks to the right side of the body, the pupil position is Right deviation, the electric wheelchair will turn to the right, and the rotation angle is proportional to the distance of the pupil from the reference point; when the user's eyes look at the ground, the pupil position will shift downward based on the reference point, and the electric wheelchair will move backwards.

When the user converts the opening and closing state of the eyes and the rotation state of the eyeball into control commands for controlling the movement of the cursor and clicking on the content displayed on the electronic display screen, the user looks around the electronic display screen along the edge for a week, and the pupil in the figure is in the process of surrounding the line of sight. The range of motion in the eye socket is divided into n×n equal rectangular small grids, n≥2, and the electronic display screen is divided into n×n equal rectangular large grids at the same time, and the above two n×n rectangular grids are established The mapping relationship of the corresponding position in , the size of n is related to the electronic display screen used by the user and the specific operation to be realized. When the user's line of sight is fixed on a certain position of the electronic display screen, the corresponding pupil of the eye also stays in the corresponding small grid According to the mapping relationship between the two grids, the cursor will move to a certain position on the electronic display screen; when the eyes look at the lower edge of the electronic display screen, the pupil is located at the lower edge of the small grid, and the opened document or web page The window scroll bar scrolls down; when the pupil is on the upper edge of the electronic display, the open document or webpage scroll bar scrolls up. In this example, a 14.1-inch ordinary computer display screen is divided into 15×15 grids, and each grid is about 2.1×1.2. A normal Windows operating system is installed in the central processing unit used, and video, Music players and chat software, etc., based on this, when using it, patients with "gradual freezing syndrome" can switch the video player or music player on and off through the opening and closing state of the eyes and the rotation state of the pupils of the eyes and through the keyboard. The input method to express oneself has broadened the control range of the eyes and enriched the user's entertainment life.

Referring to Fig. 2, the process of using the present invention for "ALS" patients is:

Step 1. The auxiliary personnel do the preparatory work before use:

Step 1a) The assistant assists the patient with ALS to get on the wheelchair, closes the first bracket 3 and the second bracket 6, and adjusts the height of the two brackets;

Step 1b) The assistant turns on the central processing unit, the electronic display screen, the camera and the electric wheelchair 1 .

Step 2. Patients with ALS control the movement of the wheelchair through their eyes:

Step 2a) Patients with ALS can blink effectively according to certain rules and look left, right and down by turning pupils;

Step 2b) The camera transmits the facial images of the multi-frame "gradually frozen human syndrome" patient to the central processing unit;

Step 2c) The central processing unit extracts eye features and judges the state of opening and closing of the eyes and the state of eyeball rotation through the obtained multi-frame facial images, and converts these eye states into corresponding controls for controlling the electric wheelchair to move forward, Commands to turn left, turn right, and stop;

The conversion rules are as follows: the patient with ALS effectively blinks 4 times in a row, and the electric wheelchair starts to move forward; when the eyes look to the left, the electric wheelchair 1 turns to the left; when the eyes look to the right, the electric wheelchair 1 turns to the right When turning and looking down, the electric wheelchair 1 moves backwards; when blinking effectively 3 times in a row, the electric wheelchair 1 stops.

Step 2d) The electric wheelchair executes the converted command of the central processing unit.

Step 3. When the electric wheelchair 1 is stationary, the patient with ALS controls the electronic display through his eyes:

Step 3a) The user first looks around the electronic display screen along the edge for a week;

Step 3b) The camera transmits the pictures of the face of the multi-frame "gradual freezing syndrome" patient to the central processing unit;

Step 3c) The central processing unit extracts eye features and judges the eyeball rotation state of the obtained multi-frame facial images, and divides the movement range of the pupils in the eye sockets into 15×15 equal rectangular small meshes during the line of sight circle At the same time, the electronic display screen is divided into 15×15 equal rectangular large grids, and the mapping relationship between the corresponding positions in the above two 15×15 rectangular grids is established;

Step 3d) The patient with "ALS" effectively blinks according to certain rules and looks at a certain point on the electronic display screen by turning the pupil of the eyeball;

Step 3e) The camera transmits the pictures of the face of the multi-frame "gradually frozen human syndrome" patient to the central processing unit;

Step 3f) The central processing unit extracts eye features from the obtained multi-frame facial images and judges the opening and closing state of the eyes and the state of eyeball rotation, and converts these eye states into controlling the movement of the cursor and clicking on "gradual freezing syndrome" The control command of the content displayed on the electronic display screen that the patient wants to turn on and off, realizes watching videos, reading and playing games independently;

The conversion rules are as follows: if a patient with ALS looks at the video player or document and effectively blinks twice, the video player or document can be opened; , the scroll bar of the document window scrolls down to continue reading; when the eyes are fixed on the upper edge of the electronic display screen, the scroll bar of the opened document scrolls up.

Step 3g) The electronic display screen is used to execute the converted command of the central processing unit.

Step 4. Patients with ALS should stop using and leave the electric wheelchair:

Step 4a) The assistant turns off the central processing unit, the electronic display screen, the camera and the electric wheelchair 1;

Step 4b) The assistant opens the first support 3 and the second support 6, and assists the patient with "Asymptomatic frostbite" to get out of the electric wheelchair.

Claims (5)

1. controlling the intelligent wheel chair of screen, including electric wheelchair (1), information collecting device (2) and information processing apparatus based on eyeball It sets (5), described information harvester (2) includes the camera for passing through first support (3) and being fixed in front of electric wheelchair (1), is used In shooting user's face-image；Described information processing unit (5) includes central processing unit, for by extracting camera shooting The opening and closing information of eyes of user and the location information of pupil in multiframe user's face image, judge eyes of user opening and closing state and The rotary state of pupil, and will determine that result converts the control command of electric wheelchair (1) motion state in order to control；Its feature exists In, be further fixed on electronic display unit (4) on the electric wheelchair (1), the electronic display unit (4) include pass through second support (6) electronic display being fixed in front of electric wheelchair (1)；The central processing unit, will when electric wheelchair (1) moves stopping Cursor moves and click on electronic display in order to control for the opening and closing state for the eyes of user judged and the rotary state conversion of pupil The control command of displayed content realizes that user passes through eyes and extraneous interaction.

2. the intelligent wheel chair according to claim 1 for controlling screen based on eyeball, which is characterized in that the first support (3), include the first support body being made of cross bar and montant, the montant is fixed on the tripod of electric wheelchair (1) front end side On, which can realize the central axis direction adjusting along bar, and can be rotated around fixed point；Fixed camera shooting is installed on the cross bar The pedestal of head is, it can be achieved that camera is rotated upwardly and downwardly around installation point, and can be moved left and right along the cross bar.

3. the intelligent wheel chair according to claim 1 for controlling screen based on eyeball, which is characterized in that the second support (6), include the second support body being made of transverse bar and vertical bar, the vertical bar is fixed on electric wheelchair (1) front end and the On the tripod of one holder (3) opposite other side.

4. the intelligent wheel chair of screen is controlled based on eyeball according to claim 1, which is characterized in that described by central processing unit Judge the control of the result conversion of the opening and closing state of eyes of user and the rotary state of pupil electric wheelchair (1) motion state in order to control System order, wherein：

It is its corresponding control command by the opening and closing condition conversion of eyes, transformation rule is：

When eyes of user continuous effective is blinked 4 times, electric wheelchair setting in motion, user's continuous effective is blinked 3 times, and electric wheelchair stops Only move；

Convert the rotary state of eye pupil to its corresponding control command, transformation rule is：

Position with pupil when user's fixating straight ahead is the datum mark of starting, and is the direction of advance originated with front, when User sees that, to left side of body, pupil position is deviated to the left based on datum mark, and electric wheelchair can turn left, rotational angle and pupil The distance for deviateing datum mark is directly proportional；When user sees that, to right side of body, pupil position is deviated to the right based on datum mark, electric wheelchair It can turn right, rotational angle deviates directly proportional at a distance from datum mark with pupil；When eyes of user is seen earthward, pupil position base It is offset downward in datum mark, electric wheelchair retreats.

5. the intelligent wheel chair according to claim 1 for controlling screen based on eyeball, which is characterized in that by eyes The rotary state of conjunction state and pupil converts the control command that cursor in order to control moves and click on electronic display displayed content, Wherein：

It is its corresponding control command by the opening and closing condition conversion of eyes, transformation rule is：

When eyes of user stares certain point in electronic display, effectively blink is primary indicates to click the point, twice in succession effectively Blink indicates that the double-click point, effective blink indicate that the eyes closed period is 1 second to 1.5 seconds；

Convert the rotary state of eye pupil to its corresponding control command, transformation rule is：

User regards electronic display one week along edge ring, and by sight, motion range of the pupil in eye socket is divided into during N × n impartial rectangle small grid, n >=2, while electronic display is divided into the n × n impartial big grid of rectangle, it establishes The mapping relations of corresponding position in above-mentioned two n × n rectangular mesh, when the sight of user watches a certain position of electronic display attentively not When dynamic, corresponding eye pupil also stays at the relative position of corresponding small grid, and according to the mapping relations of two grids, cursor is just The a certain position of electronic display can be moved to；When eye gaze electronic display lower edge, pupil is located under its small grid Edge, the document or web page windows scroll bar of opening scroll down through；When pupil is located at electronic display top edge, opening Document or webpage scroll bar scroll up.