CN103699226B - A kind of three mode serial brain-computer interface methods based on Multi-information acquisition - Google Patents

Abstract

The invention discloses a three-mode serial brain-computer interface method based on multi-information fusion, comprising the following steps: using two visual stimulation paradigms to stimulate the subjects; extracting the subjects' EEG data; setting correlation Parameters, read EEG data, perform preprocessing, feature extraction and pattern recognition on EEG data, and obtain the final pattern recognition result; convert the final pattern recognition result into control instructions, and complete specific tasks by executing control instructions. The hybrid paradigm BCI introduces electrophysiological control signals in addition to EEG signals, which expands the applicable environment and objects of BCI to some extent. It has the advantages of high stability, multiple options, and a wide range of applications, laying the foundation for the brain-computer interface to enter the stage of large-scale time application as soon as possible. This invention can be used in electronic entertainment, industrial control and other fields, and can obtain a perfect brain-computer interface system, which is expected to obtain considerable social and economic benefits.

Description

A three-modal serial brain-computer interface method based on multi-information fusion

technical field

The invention relates to the field of man-machine interface, in particular to a three-mode serial brain-computer interface method based on multi-information fusion.

Background technique

The human-machine interface refers to the interface of the input/output device that establishes contact and exchanges information in the human-computer interaction interface. Human-computer interaction is a study of the interactive relationship between the system and the user. A system can be a variety of machines, or it can be computerized systems and software. The human-machine interface is a control circuit that realizes information transmission between a computer and a human-computer interaction device. It completes the conversion of information form and the function of information transmission control together with human-computer interaction equipment. The design of the human-computer interaction interface should include the user's understanding of the system (that is, the mental model), in order to improve the usability or user-friendliness of the system.

As a new type of human-computer interface, the brain-computer interface (BCI) is a communication control system that does not depend on the normal output channels of the peripheral nerves and muscles of the brain. In the current research results, it mainly collects and analyzes human EEG signals in different states, and then uses certain engineering techniques to establish a direct communication and control channel between the human brain and computers or other electronic devices, thereby To realize a brand-new information exchange and control technology can directly express wishes or manipulate external devices by controlling brain electricity without language or body movements. The two commonly used eliciting methods of the existing brain-computer interface are endogenous ERP (event-related potential) and visual evoked potential. These two methods can better build a communication and control channel between the user and the device.

Some operations and activities in real life are complex, and completing a specific task requires more steps and different operating procedures. Therefore, a single-paradigm BCI system is not enough to support users to complete a specific task and action in daily life. For example, to assist in grasping a water cup to drink water, two basic functions of movement and grasping need to be completed without considering the movement speed and grasping degree, which cannot be realized by a single paradigm BCI at the same time. Existing brain-computer interface devices use a single type of EEG signal to manipulate the human-machine interface, resulting in a narrow range of applications, poor operational flexibility, and fewer operating instruction sets, which cannot reflect the user-friendliness and usability of the system. For example, the P300-Speller in the traditional stimulation coding mode is not conducive to the information transmission of a large instruction set, and there are problems such as low information transmission efficiency and limited number of optional characters, which are difficult to meet the requirements of practical applications.

Contents of the invention

The present invention provides a three-mode serial brain-computer interface method based on multi-information fusion. The present invention increases the number of operating instruction sets of the BCI system, has better operational flexibility, and is more suitable for real life scenes. For details, see Described below:

A three-mode serial brain-computer interface method based on multi-information fusion, said method comprising the following steps:

(1) Using two visual stimulation paradigms to stimulate the subjects;

(2) Extract the EEG data of the subjects;

(3) Set relevant parameters, read the EEG data, perform preprocessing, feature extraction and pattern recognition on the EEG data, and obtain the final pattern recognition result;

(4) Convert the final pattern recognition results into control instructions, and complete specific tasks by executing control instructions.

The steps of extracting the EEG data of the subject are specifically:

Using the TCP/IP protocol provided by the Scan4.5 software, connect the BCI2000 with the acquisition software, and realize the real-time acquisition and reading of the EEG data through the FieldTrip toolkit.

The steps of setting relevant parameters, reading the EEG data, performing preprocessing, feature extraction and pattern recognition on the EEG data, and obtaining the final pattern recognition result are as follows:

1) Call the parameters first;

2) Read real-time EEG data, and officially enter the data processing stage from the second second;

3) When processing data, first enter the judgment of the cursor movement state according to the order, intercept the EEG data 2s before the current moment, and perform canonical correlation analysis to obtain the largest canonical correlation coefficient, and compare it with the largest canonical correlation coefficient obtained from the previous 1s data Accumulate, after accumulating 3 times, compare the accumulative result with the set threshold, if it is greater than the threshold, count the results of 3 times and make pattern recognition, and then send out the pattern recognition result;

4) If it is not greater than the threshold, judge the occlusal operation, process and analyze the intercepted 2s data for the time domain analysis of the 1s data of the occlusal operation, extract the time domain features and make a judgment. If it is judged as a long-term occlusal operation, it represents Enter the mouse click state and send the pattern recognition result; if it is a short-term occlusal operation, enter the pre-opening mode of the character spelling state; if it is judged as no occlusal operation, it means it is in an idle state;

5) After one round of result output, proceed to the next round of status judgment and result output.

The moving state of the cursor is to detect and judge that the user is watching the SSVEP flashing light paradigm, and then convert the final pattern recognition result into a control instruction for cursor movement;

The character spelling state is to detect and judge that the user opens the P300 mode, and watch the target character in the Oddball paradigm, and then convert the final pattern recognition result into a control command for keyboard input;

The mouse click state is to detect and judge that the user has performed a long-term occlusal action, and then convert the final pattern recognition result into a control instruction for clicking the left mouse button;

The idle state is when no information feature is detected from the user data, it is determined that the user is in the idle state, and in the idle state, no result is output and no operation is executed.

The long-term occlusal operation has an occlusal duration greater than 600 ms; the short-term occlusal operation has an occlusal duration less than 600 ms.

The beneficial effect of the technical solution provided by the present invention is: the mixed paradigm brain-computer interface designed by this method introduces electrophysiological control signals other than EEG signals, which expands the applicable environment and environment of the brain-computer interface to a certain extent. object. It has the advantages of high stability, multiple options, and wide application range, which lays the foundation for the brain-computer interface to enter the stage of large-scale time application as soon as possible. This invention can be used in electronic entertainment, industrial control and other fields, and further research can lead to a perfect brain-computer interface system, which is expected to obtain considerable social and economic benefits.

Description of drawings

Figure 1 is a schematic diagram of a three-mode serial brain-computer interface method based on multi-information fusion;

Figure 2(a) is a schematic diagram of the Oddball rank-and-column paradigm;

Figure 2(b) is a schematic diagram of the steady-state visual evoked potential flash paradigm;

Fig. 3 is the schematic diagram of extracting the EEG data of the subject;

Fig. 4 is the flowchart of obtaining final pattern recognition result;

FIG. 5 is a schematic diagram of analyzing EEG signals using the CCA algorithm.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the implementation manner of the present invention will be further described in detail below in conjunction with the accompanying drawings.

A hybrid brain-computer interface (hBCI) is a brain-computer interface that combines at least one other system or device to help humans send information. Other communication systems here can be: another brain-computer interface device (pure hBCI); devices based on other physiological signals (physiological hBCI), such as myoelectricity, oculoelectricity and heart rate, etc.; other communication devices (hybrid hBCI), Can be assistive devices for people with disabilities or common input devices (keyboard and mouse, etc.). Hybrid-paradigm BCIs can be divided into two basic types according to their hybrid control methods, serial mode and parallel mode. In the serial mode, different signals are controlled in sequence, and this type of system can effectively reduce the false positive rate of the system; while in the parallel mode, the two sensory modes are controlled simultaneously, which is equivalent to increasing the recognizable tasks of the system number.

Daily activities and tasks can generally be broken down into fixed steps to complete. Therefore, the serial mixed-paradigm brain-computer interface solves the application bottleneck of the single-paradigm BCI, to a certain extent, it can realize daily operation activities, and to a certain extent, it can meet the needs of daily life.

In order to increase the number of operating instruction sets of the BCI system and improve the flexibility of operation, the embodiment of the present invention provides a three-mode serial brain-computer interface based on multi-information fusion, see Figure 1, and see the following description for details:

101: Using two visual stimulation paradigms to stimulate the subjects;

The designed visual stimuli included the traditional Oddball determinant paradigm (evoked event-related potentials) ^[1] and the steady-state visual evoked potential (SSVEP) ^[2] flashing light paradigm. The schematic diagrams of the paradigms are shown in Figure 2-a and b, respectively. Among them, the traditional Oddball paradigm is designed and realized by the Eprime software platform, and the flashing light paradigm is designed and realized by the FPGA platform. Both visual stimulus paradigms are presented to the user in real time.

P300 refers to the endogenous ERP components that appear 300-400 ms after the stimulus, and are usually induced by the Oddball paradigm. The so-called Oddball paradigm refers to the ERP evoked by deviant stimuli or target stimuli (small probability stimuli, the general target stimulus probability is 10-30%) in the sequence of standard stimuli (high probability stimuli). The smaller the probability of the deviant stimulus or the target stimulus, the larger the amplitude of the evoked P300. P300-Speller is an important human-machine interface paradigm that uses EEG signals to realize text selection input. It can realize the purpose of direct dialogue between patients and the outside world, and can effectively improve the quality of life of paralyzed patients. It is convenient for clinical application, and the P300 signal has It has the advantages of stable features and no need for training, so it can be seen that P300-Speller has a good application prospect.

Steady-state visual evoked potential is one of the most commonly used and most effective modes in the research of brain-computer interface. It does not need to train the subjects, and the experiment is simple and convenient. With a high signal-to-noise ratio, strong SSVEP signals can be recorded on the scalp, and very few electrodes are required, and enough information can be collected with one or two electrodes, which has strong operability. Based on the above advantages, we can see that in-depth research on SSVEP will help us understand our brains more clearly and realize real human-computer interaction, which has strong theoretical and application value.

102: Extracting the EEG data of the subject;

The completion of the hybrid brain-computer interface first requires real-time collection of EEG data. The EEG data collection adopts the 40-lead NuAmp EEG amplifier of Neuroscan Company, and a total of 6 EEG signals are collected: Fz, Cz, Pz, Oz, P7 and P8, arranged according to the international 10-20 system. The EEG signals of all leads are referenced to the right mastoid and grounded to the left mastoid, and the impedance values are all below 5K. The subjects sat quietly on a chair about 60cm away from the screen, watched the corresponding visual stimulus paradigm and performed voluntary occlusal movements to complete the web browsing operation. This method uses the TCP/IP protocol provided by the Scan4.5 software to connect the BCI2000 with the acquisition software, and realizes the real-time acquisition and reading of EEG data through the FieldTrip toolkit, which lays the foundation for subsequent online data processing. The specific data The collection diagram is shown in Figure 3.

103: Set relevant parameters, read the EEG data, perform preprocessing, feature extraction and pattern recognition on the EEG data, and obtain the final pattern recognition result;

Figure 4 shows the extraction and processing flow of online EEG data. The entire processing flow includes four user states, namely: cursor movement state, character spelling state, mouse click state and idle state. Among them, the cursor movement state is to detect and judge that the user is staring at the SSVEP flashing light paradigm, and then convert the final pattern recognition result into a control command for cursor movement; the character spelling state is to detect and judge that the user turns on the P300 mode and stares at the target character in the Oddball paradigm (referring to the character that the user expects to output, and the user needs to pay attention to the character that is expected to be output during the process), and then convert the final pattern recognition result into a control command input by the keyboard; the mouse click state is to detect and judge that the user has performed a long-term occlusal action (Long-term occlusal action refers to the occlusal action with a long duration, generally greater than 600ms), and then converts the final pattern recognition result into a control command for clicking the left mouse button; the idle state means that no information is detected from the user's data When the feature is used, it is determined that the user is in an idle state. When the user is in an idle state, no results are output and no operations are performed. In the data processing process of this design, the analysis of the user status is judged sequentially, and the data processing flow is as follows:

1) First call the parameters (such as the location of the EEG data cache file, the flashing frequency, the trained classifier data, etc.).

2) Read real-time EEG data, and officially enter the data processing stage from the second second. When the new data packet reaches the length of 1s, a data processing and analysis is performed, that is, a data processing and analysis is performed every second.

3) When processing data, first enter the judgment of the cursor movement state according to the order, intercept the EEG data 2s before the current moment, and perform canonical correlation analysis to obtain the largest canonical correlation coefficient (Gao et al. of Tsinghua University proposed a method using multi-channel The CCA-based frequency identification method of the signal is used for the frequency feature extraction and identification of SSVEP ^[3] , that is, the Cvalue value in Figure 4 is obtained through feature extraction), and is accumulated with the maximum typical correlation coefficient obtained from the previous 1s data. After 3 times, compare the cumulative result with the set threshold (determined by the result of offline analysis), if it is greater than the threshold, count the results of 3 times (each canonical correlation analysis will get the maximum canonical correlation coefficient and the corresponding Target frequency, the target frequency refers to the frequency of the flashing light that the user is looking at) and pattern recognition (the one with more target frequency among the 3 results is the pattern recognition result) and then the pattern recognition result is sent out.

CCA analyzes two sets of variables: one set is the multi-channel EEG signal recorded from a certain source area, denoted as x(t), and the other set of variables is the stimulus signal. Each flashing light stimulus of SSVEP flickers according to a certain frequency, and the flickering is also triggered by an electrical signal of a certain frequency (a square wave with a fixed period). It is also known that a periodic signal can be decomposed into a set of Fourier sequences, that is to say, a stimulus signal presented in the form of a square wave, denoted as y(t), has a fixed period (frequency is f), so it can be decomposed into f and its harmonics The Fourier sequence of the wave (sin(2πft), cos(2πft), sin(4πft), cos(4πft)...), such as

$\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left(\begin{array}{c}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 6</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>\end{array}\right)<span\; class="notranslate">\; =</span>\left(\begin{array}{c}\mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;ft</span>}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; cos</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;ft</span>}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}\mathrm{<span\; class="notranslate">\; \&pi;ft</span>}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; cos</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}\mathrm{<span\; class="notranslate">\; \&pi;ft</span>}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}\mathrm{<span\; class="notranslate">\; \&pi;ft</span>}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; cos</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}\mathrm{<span\; class="notranslate">\; \&pi;ft</span>}<span\; class="notranslate">\; )</span>\end{array}\right)<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; ,</span>\frac{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; ,</span>\frac{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Among them, f is the fundamental frequency, T is the number of data sampling points, and S is the sampling rate of the signal. Figure 5 illustrates how to use the CCA algorithm to analyze EEG signals. Since the brain behaves as a low-pass filter in dynamics, some high-frequency components in the square wave signal will be filtered out, so the low-frequency fundamental and harmonics are generally used (the fundamental and primary, primary, and harmonics are used in the above formula) second harmonic 6 components). CCA can be used as a feature extraction method for the detection and identification of SSVEP based on the assumption that the output is the SSVEP that responds to the electrical activity of the brain, and the system that outputs the stimulus signal is a linear system, that is to say, the SSVEP response contains The frequency content is consistent with the stimulus signal. In the algorithm, by calculating the typical correlation coefficients of EEG signals and all frequency stimuli in the system, the frequency corresponding to the largest coefficient is the frequency of SSVEP, which is also the frequency of flash light stimulation.

The core problem of the SSVEP-based brain-computer interface system is to detect the frequency of the SSVEP component in the subject's EEG signal. Assume that there are K stimulation frequencies f ₁ , f ₂ ,...,f _K and N leads to L seconds of EEG data. If the stimulus frequency is denoted as f _S , it satisfies

f _S = max _f ρ(f), f = f ₁ , f ₂ ,..., f _K (2)

Among them, ρ(f) is the typical correlation coefficient of x (EEG signal) and y (stimulus signal) (as shown in Equation 2).

4) If it is not greater than the threshold, judge the occlusal operation. Process and analyze the intercepted 2s data of the last 1s data to analyze the time domain of the occlusal operation, extract the time domain features and make a judgment. If it is judged as a long-term occlusal operation, it means entering the mouse click state and sending the pattern recognition result; if If it is a short-term occlusal operation, it will enter the pre-opening mode of the character spelling state; if it is judged that there is no occlusal operation, it means it is in an idle state.

Occlusion evoked scalp-myoelectricity (OES-ME) is the myoelectric artifact collected at the scalp when the masticatory muscles drive the scalp to move during the occlusal movement. As one of the noises of common EEG signals, occlusal-induced scalp EMG has the characteristics of large amplitude and easy to distinguish. Due to the large difference between occlusal-induced scalp EMG and EEG signals, it is easier to identify than EEG signals. , the bite-induced scalp myoelectricity and other myoelectricity signals can be used as a supplementary input method for a man-machine interface with EEG as the main input method to trigger some frequently executed instruction sets to increase the response speed and accuracy of the system , to improve the user-friendliness of the system. But at the same time, the addition of occlusion-induced scalp myoelectricity may cause some problems. As a kind of myoelectric signal, occlusal-induced scalp myoelectricity may interfere with the acquisition of EEG signals. affect the stability of the system. Bite action has strong temporal characteristics. The duration characteristics of the signal are extracted by time domain analysis, and the duration is defined as greater than 30 ms to be judged as occlusal operation, otherwise it is judged as no occlusal action. And the occlusal operation with a duration greater than 600ms is defined as a long-term occlusal operation, and the occlusal operation with a duration of less than 600ms is defined as a short-term occlusal operation.

Since the user is required to perform a short-term occlusal action before spelling characters, that is, only when a short-term occlusal action is detected can the user enter the character spelling state, but short-term occlusal actions are prone to misuse. Therefore, in order to prevent the misjudgment of the state caused by misoperation, the character spelling pre-opening mode is set, that is, after detecting the short-term occlusal action, check whether all the labels in the data of the next 5 seconds contain the start label. If it is not detected, skip this state and directly judge it as an idle state; if it is detected, it will continue to wait for the end tag to be detected. When all are detected, intercept the EEG data between the start tag and the end tag for P300 processing (including preprocessing, feature extraction and pattern recognition (Fisher linear discriminant analysis ^[4] ), and the judged results will be recorded in voice Prompt feedback to the user, if it is the same as the user's expectation, the result needs to be executed within 3s before the bite operation is performed, that is, it is judged whether there is a bite operation within 3s, and the pattern recognition result is output; otherwise, the judgment is considered wrong and the pattern recognition result is not output .

The collected EEG data includes EEG signals and labels (also called event codes) and the positions of labels (corresponding to EEG signals). The label is the data representing the rank and column information and the stimulus start information sent by the Eprime software program to the EEG amplifier through the parallel port while the Oddball rank and rank paradigm is displayed. For example, at the beginning of each round of stimulation, a short-term occlusal operation will be prompted on the screen, and the start label will be sent after 3s, which represents the beginning of a new round of character spelling tasks. character of. During the stimulation period, a label representing the row and column information will be sent after each row and column is stimulated. For example, when the fifth row is on, the sending label is 5; when the second column is on, the sending label is 8. An end tag is also sent at the end of a round of stimuli, representing the end of the round. Since the Eprime software program sends label data to the EEG amplifier through the parallel port, the amplifier will synchronously integrate the label data and EEG data in real time, so the label position and signal correspond in real time, and the corresponding part can be intercepted through the start label and end label. EEG signals under the stimulation of chakras can be analyzed and processed.

Fisher linear discriminant analysis is generally applicable to pattern recognition of two types of samples. For the case of binary classification, the best classification effect should make the two types of one-dimensional sample features after projection satisfy the largest inter-class distance and the smallest intra-class distance, so as to satisfy the linear separability of the two types of samples to the greatest extent.

The Fisher criterion function is the ratio of the between-class dispersion to the intra-class dispersion, defined as:

Wherein, it is assumed that there are two types of samples W ₁ and W ₂ , and the corresponding numbers of samples are N ₁ and N ₂ respectively. Define μ ₁ and μ ₂ as the mean value of two types of original samples, and the mean value of the one-dimensional feature data after sample projection is defined as and The dispersion of the two types of samples after projection is defined as and The intra-class dispersion of two types of samples is S _W and the inter-class dispersion is S _b . Satisfying the largest dispersion between classes and the smallest dispersion within classes, that is, when the Fisher function takes the maximum value, the classification effect is the best, that is, the value of the variable parameter ω at this time ω ^* is the best projection vector, and ω ^T is the value of ω Transpose. Differentiate the formula and take 0 and derive the optimal projection vector ω ^* for the binary classification:

For the P300 pattern recognition in this design, all data samples are divided into two categories, namely EEG signals under target and non-target stimuli. The target stimulus is the stimulus state when the row and column where the user expects to output the character is lit, and the stimulus state when the rest of the rows and columns are lit is the non-target stimulus.

5) After one round of result output, proceed to the next round of status judgment and result output.

104: Convert the final pattern recognition result into a control instruction, and complete a specific task (web browsing) by executing the control instruction.

The entire operation of this step is completed under the framework of the MFC platform, which mainly realizes two functions: serial port communication and command control conversion. Wherein, the final pattern recognition result is received through serial port transmission, and the final pattern recognition result is converted into corresponding control instructions. Control instructions include P300 control instructions, SSVEP control instructions and OES-ME control instructions. The execution of the control command will be displayed directly on the computer screen or as a voice prompt. Such as the movement of the cursor on the screen (realized by changing the current cursor position); call the keyboard operation program to simulate the keyboard operation to realize the input of characters; call the mouse action event to simulate the mouse operation (click the left mouse button) to realize the confirmation click function.

references

[1] Farwell L.A., Donchin E., Talking off the top of your head: A mental prosthesis utilizing event-related brain potentials, Electroencephalogr. Clin. Neurophysiol., 1988 70:510–523.

[2] Vialatte F.B., Maurice M., Dauwels J., et al. Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Progress In Neurobiology, 2010, 90(4): 418～438.

[3]Lin Z., Zhang C., Wu W. et al., Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs.IEEE Trans.Biomed.Eng.,2007,54(6):1172～1176.

[4] Sun Changcheng. Visual P300-Speller-Induced ERP Research Based on Three-Dimensional Coded Stimulus Sequences: [Master's Thesis], Tianjin; Tianjin University, 2011.

Those skilled in the art can understand that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the above-mentioned embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within range.

Claims (2)

1. A three-mode serial brain-computer interface method based on multi-information fusion is characterized by comprising the following steps:

(1) stimulating the testee by adopting two visual stimulation paradigms;

(2) extracting electroencephalogram data of a subject;

(3) setting related parameters, reading electroencephalogram data, preprocessing the electroencephalogram data, extracting features and identifying modes to obtain a final mode identification result;

(4) converting the final mode recognition result into a control instruction, and completing a specific task by executing the control instruction;

the steps of setting relevant parameters, reading electroencephalogram data, preprocessing the electroencephalogram data, extracting features and identifying modes, and obtaining a final mode identification result specifically comprise:

1) firstly, calling parameters;

2) reading real-time electroencephalogram data, and formally entering a data processing stage from the second;

3) when data is processed, firstly, judging the cursor moving state according to the sequence, intercepting electroencephalogram data 2s before the current time, performing typical correlation analysis to obtain a maximum typical correlation coefficient, accumulating the maximum typical correlation coefficient with the maximum typical correlation coefficient obtained by the data 1s, comparing an accumulated result with a set threshold value after accumulating for 3 times, counting the result 3 times if the accumulated result is greater than the threshold value, and sending a mode identification result after mode identification is carried out;

4) if the time domain characteristics are not larger than the threshold value, judging the occlusion operation, processing and analyzing the last 1s data of the intercepted 2s data, performing time domain analysis of the occlusion operation, extracting the time domain characteristics, judging, if the time domain characteristics are judged to be the long-term occlusion operation, indicating that the mouse enters a mouse clicking state, and sending a mode identification result; if the short-time occlusion operation is performed, entering a pre-opening mode of a character spelling state; if judging that no occlusion operation exists, representing that the mobile terminal is in an idle state;

the long-time-range occlusion operation is that the occlusion duration is longer than 600 ms; the short-time meshing operation is that the meshing duration is less than 600 ms;

5) after the result of one round is output, continuing the judgment of the state of the next round and the result output;

wherein,

the cursor movement state is to detect and judge that the user is watching the SSVEP flashing light paradigm, and then convert the final pattern recognition result into a control instruction of cursor movement;

the spelling state of the character is to detect and judge that a user starts a P300 mode, and watch a target character in an Oddball paradigm, so that a final mode recognition result is converted into a control instruction input by a keyboard;

the mouse clicking state is used for detecting and judging that a user executes long-term occlusion action, and then a final mode recognition result is converted into a control instruction for clicking a left mouse button;

the idle state is that when no information characteristic is detected from the data of the user, the user is judged to be in the idle state, and when the idle state is detected, no result is output and no operation is executed.

2. The multi-information fusion-based three-modality serial brain-computer interface method according to claim 1, wherein the step of extracting the electroencephalogram data of the subject specifically comprises:

the BCI2000 is connected with acquisition software by using a TCP/IP protocol provided by Scan4.5 software, and the real-time acquisition and reading of electroencephalogram data are realized by a FieldTrip tool package.