CN116543455A

CN116543455A - Method, equipment and medium for establishing parkinsonism gait damage assessment model and using same

Info

Publication number: CN116543455A
Application number: CN202310387454.8A
Authority: CN
Inventors: 霍卫光; 韩建达; 于宁波
Original assignee: Shenzhen Research Institute Of Nankai University
Current assignee: Shenzhen Research Institute Of Nankai University
Priority date: 2023-04-03
Filing date: 2023-04-03
Publication date: 2023-08-04

Abstract

The application discloses a method, equipment and medium for establishing a parkinsonism gait impairment evaluation model and using the same, wherein the method for establishing the model comprises the following steps: preprocessing gait videos, and extracting a skeleton diagram sequence and a multi-period gait energy diagram; and training the neural network by using a skeleton diagram sequence and a multi-cycle gait energy diagram and combining gait impairment scoring labels obtained by a doctor according to gait video diagnosis so as to obtain the parkinsonism gait impairment assessment model, wherein the gait impairment scoring labels are used for dividing the gait impairment degree. According to the method, the outline extracted from the gait video and the data flow of two different forms of the gait energy cycle chart and the skeleton chart are utilized, the neural network is combined with the neural network training to generate the corresponding neural network model, the neural network model can analyze the gait video of the patient with the Parkinson disease to be evaluated, the gait score is obtained, the gait damage degree of the patient with the Parkinson disease can be conveniently and rapidly evaluated based on the gait score, and the accuracy is high.

Description

Method, equipment and medium for establishing parkinsonism gait damage assessment model and using same

Technical Field

The application belongs to the field of medical image processing, and particularly relates to a method, equipment and medium for establishing a parkinsonism gait impairment evaluation model and using the same.

Background

Parkinson's Disease (PD) is a common neurodegenerative disease and is characterized clinically by mainly resting tremor, bradykinesia, stiffness, postural instability, etc., which severely affects the quality of life of the patient. Early and timely diagnosis is critical to the initiation of neuroprotective therapies and post-management.

The main basis for the current evaluation of the motor functions of the Parkinson's disease is an evaluation scale MDS-UPDRS. The third part of the gauge evaluates the motor symptoms of PD, requiring the grader to score 18 items of gait, upper limb mobility, leg mobility, etc. for 0-4 points, with 0 indicating normal and 4 indicating severe, depending on the observed patient condition.

Clinically, a neurologist with a high experience will quantitatively evaluate the motor and non-motor symptoms of the patient according to the scale. However, this manual evaluation has two challenges: on the one hand, the evaluation by the clinician is time consuming and subjective. On the other hand, this assessment method requires the patient to go to the hospital regularly, which is disadvantageous for long-range disease management and control. The most parkinsonism in China exists in the world, but the early symptoms are not obvious, the cost of the doctor is high, and the doctor's rate is only less than 40%.

Deep learning is the inherent law and presentation hierarchy of learning sample data, and the information obtained in these learning processes greatly helps the interpretation of data such as text, images and sounds. Its final goal is to have the machine have analytical learning capabilities like a person, and to recognize text, image, and sound data. Therefore, how to use a deep learning model to perform gait impairment assessment on the parkinson's disease is a current urgent problem to be studied.

Disclosure of Invention

In order to solve the problems that in the prior art, the evaluation time for manually evaluating symptoms of a parkinsonism patient is long and not objective enough, and the patients cannot be subjected to home automatic evaluation, the application provides a method, equipment and medium for establishing a parkinsonism gait damage evaluation model and using the same, and the technical scheme is as follows:

in one aspect, the present application provides a method for establishing a parkinson's disease gait impairment assessment model, comprising the steps of:

s10, preprocessing gait videos, and extracting a skeleton diagram sequence and a multi-cycle gait energy diagram, wherein the gait videos are walking videos of a parkinson patient photographed from the side;

and S20, training a neural network by using a skeleton diagram sequence and a multi-cycle gait energy diagram and combining gait impairment scoring labels obtained by a doctor according to gait video diagnosis so as to obtain a parkinsonism gait impairment evaluation model, wherein the gait impairment scoring labels are used for dividing the gait impairment degree.

Further, the parkinsonism gait impairment evaluation model is a double-flow neural network, and comprises a skeleton flow, a contour flow, a vector splicing unit, a full-connection layer and a softmax layer;

the skeleton flow comprises a plurality of ST-GCN units and is used for processing an input skeleton diagram sequence to obtain skeleton flow output vectors, the outline flow comprises a plurality of VGG units and is used for processing an input multi-period gait energy diagram to obtain outline flow output vectors, the skeleton flow output vectors and the outline flow output vectors are input into a vector splicing unit to be spliced, double-flow information fusion is achieved through a full-connection layer, and finally predicted probability values of all scores are obtained through a softmax layer, wherein the score with the highest predicted probability value is a gait score estimated value of a gait video.

Further, in step S10, the gait video is preprocessed, and the extracting the multi-cycle gait energy diagram specifically includes:

s101, extracting a contour map from each frame of a gait video, and arranging the contour maps in time sequence to obtain a contour map sequence;

s102, determining a gait energy period according to the profile diagram sequence;

the gait energy period is a time interval between adjacent peaks and valleys in a change curve of the interval between two feet in the profile sequence along with time, and the starting point of the gait energy period is a time point corresponding to the peak or the valley;

S103, dividing the profile diagram sequence into a group a according to the gait energy period, and superposing the profile diagrams in each group to form a gait energy diagram, so as to correspondingly obtain a gait energy diagrams;

and S104, connecting the a Zhang Butai energy diagrams in the horizontal direction in time sequence, and correspondingly obtaining a multi-cycle gait energy diagram.

Further, the calculation for superimposing the profile maps in each group in step S103 to form a gait energy map is specifically as follows:

wherein I is _c (x, y, N) is a contour map extracted from an nth frame of the gait video, N is a number of frames of the video contained in one gait energy period, x is a horizontal coordinate of a pixel point in the contour map, y is a vertical coordinate of the pixel point in the contour map, and a value of GEI (x, y) is a pixel.

Further, in step S10, the gait video is preprocessed, and the skeleton map sequence extracting specifically includes:

s111, extracting two-dimensional coordinates of joints from each frame of gait video;

s112, calculating a skeleton diagram sequence corresponding to the gait video according to the two-dimensional coordinates of the joints and the spatial connection relation among the joints;

further, in step S112, the calculation of the skeleton map sequence corresponding to the gait video according to the two-dimensional coordinates of the joints and the spatial connection relationship between the joints is specifically as follows:

J＝(V，E)，

Wherein the node set V includes two-dimensional coordinates of joints in all frames of the gait video; the edge set E includes two types of spatial connection relationships: a proximal end, a distal end; the proximal end comprises a left shoulder, a right shoulder, a vertebra, a left hip, a right hip, a left upper arm, a right upper arm, a left thigh, a right thigh, and the distal end comprises a left lower arm, a right lower arm, a left lower leg;

the spatial connection relation of the edge set E is represented by an adjacent matrix A, and the number a of the ith row and the jth column in the matrix A _ij Representing the spatial connection relationship between joint i and joint j, and if the spatial connection relationship is proximal, then a _ij X, if the spatial connection relationship is far-end, a _ij Y, where x, y is a positive integer, and 1<x<y。

Further, the edge set E further includes a time connection relationship: a time sequence connection; the time sequence connection comprises the connection between the node at the current moment of each joint and the node at the next moment of each joint;

the time sequence connection of the edge set E is realized by one-dimensional convolution of the joint along the time direction, as follows:

wherein X is _t The sequence of two-dimensional coordinate time delay direction of a certain joint is a matrix of t rows and 2 columns, t is the number of frames contained in gait video, N is the number of convolution kernels of time sequence convolution, and W _k Is the kth time sequence convolution kernel, is a one-dimensional vector, and B is a learnable linear bias.

Further, the calculation of the ST-GCN unit is as follows:

Wherein X is _in To input features, X _out For output characteristics, A is an adjacency matrix, D ⁱⁱ ＝∑ _j (A ^ij +I ^ij ) The degree matrix is used for normalizing the adjacent matrix A, W is a learnable weight coefficient matrix, and B is a learnable bias coefficient.

Further, the VGG unit includes a two-dimensional convolution layer and a maximum pooling layer.

Further, training the neural network in step S20 further includes:

calculating a loss value through a loss function, reversely spreading the loss value, and stopping training when the error converges;

the loss function adopts cross entropy and is specifically calculated as follows:

wherein N is the number of samples, K is the number of categories, p _ic Representing the probability that the class of the ith sample is c, i.e

Further, y _ic E {0,1}, when the class of the ith sample is c, corresponding to y _ic =1, otherwise y _ic ＝0。

In another aspect, the present application provides a method for parkinson's disease gait impairment assessment, comprising the steps of:

s1: preprocessing gait videos, and extracting a skeleton diagram sequence and a multi-cycle gait energy diagram, wherein the gait videos are walking videos of a person to be evaluated, which is photographed from the side;

s2: inputting the skeleton diagram sequence and the multi-cycle gait energy diagram into the parkinsonism gait impairment evaluation model obtained by the method for establishing the parkinsonism gait impairment evaluation model in any one of the above, and obtaining the gait score of the person to be evaluated.

Further, step S2 includes obtaining gait score optimization results using a voting mechanism, specifically:

grouping the gait videos of the same person to be evaluated, correspondingly and respectively obtaining a plurality of groups of gait scores, and selecting the score with the largest occurrence number from the plurality of groups of gait scores, namely obtaining the gait score optimization result of the person to be evaluated.

Further, after step S2, the method further includes the steps of:

s3: calculating joint response values of the skeleton diagram sequences in the parkinsonism gait impairment evaluation model;

s4: performing classification calculation on the joint response values to obtain an average response value of each body part;

s5: and obtaining a refined gait evaluation result according to the joint response value and the average response value.

Further, the joint response value is calculated as follows:

wherein S is a joint response value vector, N is the number of joints, each value in the joint response value vector correspondingly represents the response value of the Parkinson' S disease gait damage evaluation model at the joint, T is the length of an input skeleton diagram sequence, and C _out Number of output channels for skeleton stream, O _it Output matrix X for skeleton stream _out Is a component of the group.

Further, the joints are divided into 6 groups representing 6 body parts: the step S3 of classifying and calculating the joint response values to obtain an average response value of each body part specifically includes:

The average response values for each set of joints are calculated separately to obtain an average joint response value for each body part.

In another aspect, the present application provides a computer device, where the computer device includes a computer readable storage medium, a processor, and a computer program stored on the computer readable storage medium and executable on the processor, where the processor when executing the program implements a method of establishing a parkinson's disease gait impairment evaluation model as defined in any one of the above, or implements a method of parkinson's disease gait impairment evaluation as defined in any one of the above.

In yet another aspect, the present application provides a readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method of modeling parkinson's disease gait impairment assessment of any of the above, or implements a method of parkinson's disease gait impairment assessment of any of the above.

The beneficial effects of this application are: according to the method, the outline extracted from the gait video and the data flow of two different forms of the gait energy cycle chart and the skeleton chart are utilized, the neural network is combined with the neural network training to generate the corresponding neural network model, the neural network model can analyze the gait video of the patient with the Parkinson disease to be evaluated so as to obtain the gait score, the condition of the patient is evaluated based on the gait score, the gait damage degree of the patient with the Parkinson disease can be evaluated conveniently and rapidly through the neural network model, and the accuracy is high.

Drawings

FIG. 1A is a schematic diagram of an embodiment of a method of modeling a Parkinson's disease gait impairment evaluation in accordance with the present application;

FIG. 1B is a schematic block diagram of one embodiment of a method of parkinsonism gait impairment assessment of the present application;

FIG. 2A is a flow chart of one embodiment of a method of modeling a parkinsonism gait impairment evaluation in accordance with the present application;

FIG. 2B is a sub-flowchart of an embodiment of step S10 in FIG. 2;

FIG. 2C is a sub-flowchart of an embodiment of step S10 in FIG. 2;

FIG. 3 is a flow chart of one embodiment of a method of parkinsonism gait impairment assessment of the present application;

FIG. 4 is a flow chart of one embodiment of a method of fine evaluation of parkinsonism gait impairment of the present application;

FIG. 5 is a flow chart of one embodiment of a method of parkinsonism gait impairment quantification of the present application;

FIG. 6 is a schematic diagram of a framework of the Parkinson's disease gait impairment assessment model of the present application;

FIG. 7 is a schematic diagram of one embodiment of gait video acquisition of the present application;

FIG. 8 is a schematic diagram of a full flow chart of one embodiment of a method of parkinsonism gait impairment assessment of the present application;

FIG. 9 is a schematic diagram of one embodiment of a skeleton diagram of the present application;

FIG. 10 is a waveform diagram of an embodiment of the change of the distance between two feet with time in the gait video of the application;

FIG. 11 is a schematic diagram of one embodiment of a multi-cycle gait energy diagram of the application;

FIG. 12 is a graph showing the average response value distribution of each joint in one embodiment of the present application;

FIG. 13 is a graph of average response value for each body part in one embodiment of the present application;

FIG. 14 is a graph of ROC versus different scores in an embodiment of the present application;

FIG. 15 is a confusion matrix in one embodiment of the present application;

fig. 16 is a schematic structural view of a hardware operating environment of an embodiment of the method for modeling parkinson's disease gait impairment evaluation or method for parkinson's disease gait impairment evaluation of the present application.

Detailed Description

In order to facilitate an understanding of the present application, the present application will be described in more detail below with reference to the accompanying drawings and specific examples. Preferred embodiments of the present application are shown in the drawings. This application may, however, be embodied in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

It is noted that all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs unless defined otherwise. The terminology used in the description of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. For example, the term "plurality" includes two and more.

The following are some term interpretations of the present application:

ST-GCN: spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition, a space-time diagram convolutional network for human motion recognition.

VGG: VGG is an image recognition network proposed by Visual Geometry Group of Oxford in 2014, using a deeper network architecture, demonstrating that increasing network depth can affect network performance to some extent.

Gait is a useful biomarker for the identification of early parkinson's disease. At present, many researches try to acquire gait information by using wearable equipment such as a bracelet and the like so as to realize early screening of the Parkinson's disease. However, since these wearable devices can only measure the motion of a specific portion, it is difficult to completely characterize the motion characteristics of the whole body, and key factors such as sidedness and coordination are difficult to detect.

In view of the above, the present application provides a method, apparatus and medium for establishing a parkinson's disease gait impairment evaluation model and using the same, in which video samples can be collected only by a smart phone, and gait analysis is performed based on the collected video samples, so as to obtain an evaluation result of better overall motion characteristics.

Referring to fig. 1A-1B, the manner in which a model is built and used is shown.

Specifically, referring to fig. 1A, in a manner of establishing a parkinson's disease gait impairment evaluation model, after acquiring gait videos, the gait videos need to be evaluated manually, and a label is marked, for example, a doctor views each section of acquired gait videos and then gives an MDS-UPDRS score, and a label file X corresponding to the section of gait videos includes the score information. And preprocessing the gait video, sending the preprocessed data into a neural network for model training to obtain a predicted result Y, and stopping training when the difference between the predicted result Y and the label file X meets a preset threshold value, wherein the neural network is the parkinsonism gait damage assessment model after training is completed.

Further, referring to fig. 1B, a trained parkinson's disease gait impairment assessment model is used for assessment. At this time, the collected gait video is not manually evaluated and labeled any more, but is directly sent to the parkinsonism gait impairment evaluation model after being preprocessed, and a predicted result Z is output after being calculated by the parkinsonism gait impairment evaluation model, and the predicted result Z is the score given by the model to the gait video, so that the machine can be operated without intervention of doctors.

Further, the process of building a model (training a model) and the process of using the model are separate. Once trained, a model can be used an unlimited number of times (e.g., thousands of times) and can be run on different hardware environments.

Further, referring to fig. 7, an embodiment of a gait video acquisition mode is shown. As shown, the cell phone (or camera) is mounted on the cell phone support, the long side is parallel to the ground, and the distance from the sagittal plane of the subject is about 5 meters, so that when the subject walks along the direction parallel to the sagittal plane of the subject, the field of view of the cell phone (or camera) is about 5 meters. The subject needs to walk from the left side to the right side of the video picture as required, then turn around and return, and repeat three times. The recorded video is cut into 6 sections, the turning part is removed, and only the straight walking section is reserved, so that 6 sections of gait video can be obtained.

The gait video of the application is not limited to the above acquisition mode, and can be used as long as the acquired gait video meets the requirement of shooting from the side, the level and stability are maintained during shooting, and the whole body of a subject is always and completely presented in a picture. This is because gait videos photographed at a side view angle are more advantageous for extracting human body contour features, so that analysis of gait features is performed based on changes in human body contour diagrams.

The gait video of the application is also not limited to the above range, and a plurality of videos with different durations and different distances can be recorded for the same subject.

Further, subjects in gait videos include patients with parkinson's disease and normal persons of different degrees, in the training process shown in fig. 1A, the subjects are called parkinson's disease patients, scores are given manually by doctors, in the evaluation process shown in fig. 1B, the subjects are called evaluators, and scores are given by the trained models.

In one aspect, and with reference to fig. 2A-2C, the present application provides a method of establishing a parkinson's disease gait impairment assessment model, comprising the steps of:

The skeleton diagram sequence mainly comes from joint characteristics in gait videos, and the multi-cycle gait energy diagram mainly comes from outline characteristics in the gait videos.

By the gait video photographed from the side, in addition to the skeleton map features of the human body based on the joints, human body profile features, particularly gait changes of the human body profile during walking, can be extracted, so that analysis of the gait features is performed based on the changes of the human body profile. For the video shot from the front, the change of the human body profile map cannot be accurately perceived, for example, the change of the step length, the step speed and the like cannot be observed, the change of the swing arm angle and the like, the inclination angle of the body and the like cannot be known, and the change features are extracted from the side face.

Therefore, the method aims at the gait video shot by the side face, extracts the skeleton graph sequence and the multi-period gait energy graph based on the joints and the contours, combines the data streams reflecting the gait characteristics at two different angles to perform neural network training, so that a dual-flow network model combining the contours and the joints is obtained.

Further, referring to fig. 2B, in step S10, the gait video is preprocessed, and the extracting the multi-cycle gait energy diagram specifically includes:

s103, dividing the profile diagram sequence into a groups according to gait energy periods, and superposing the profile diagrams in each group to form a gait energy diagram, wherein a gait energy diagram is correspondingly obtained;

Specifically, referring to fig. 8, the contour map is a black-and-white human contour map, each frame corresponds to one contour map, and all contour maps extracted according to the time sequence in the step video are finally formed into a contour map sequence (i.e. a contour sequence in the map) in sequence.

Further, referring to fig. 10, one gait energy cycle is a step, i.e., the time interval between adjacent peaks and valleys, and the start of the gait energy cycle is the time point corresponding to the peak or valley. For example, when the start of a gait energy cycle is a valley value of 0.0, the gait energy cycle is the time interval between the adjacent peak value of 0.5 and the valley value of 0.0. Superimposing the profiles over the gait energy cycle, a gait energy pattern is obtained, corresponding to the first of the 6 consecutive human profile variations in figure 11. The superposition mode is to superpose all the contour diagrams in the gait energy period by taking the midpoint of the vertex as a reference.

Further, in fig. 11, a=6, and b is the number of contour diagrams in one gait energy cycle. Thus, fig. 11 shows the 6 gait energy patterns taken in succession, and the 6 gait energy patterns are stitched together in the horizontal direction, resulting in a multi-cycle gait energy pattern.

wherein I is _c (x, y, N) is a contour map extracted from the nth frame of gait video, N is a gait energyThe video frame number contained in the period is x, the x is the horizontal coordinate of the pixel point in the contour map, y is the vertical coordinate of the pixel point in the contour map, and the value of GEI (x, y) is the pixel.

Further, referring to fig. 2C, in step S10, the gait video is preprocessed, and the extracting skeleton map sequence specifically includes:

and S112, calculating a skeleton diagram sequence corresponding to the gait video according to the two-dimensional coordinates of the joints and the spatial connection relation among the joints.

Specifically, referring to fig. 8, each frame of gait video corresponds to a joint map and one gait video corresponds to a set of joint sequences. The joint sequences have a certain spatial connection relationship, a skeleton diagram corresponding to each frame of gait video can be calculated and obtained according to the spatial connection relationship, and the skeleton diagrams are arranged according to the time sequence, so that the skeleton diagram sequence shown in fig. 8 is obtained.

Further, in step S112, the calculation of the skeleton map sequence corresponding to the gait video according to the two-dimensional coordinates and the spatial connection relation of the joints is specifically as follows:

J＝(V，E)，

referring to fig. 9, for the joint classification provided in an embodiment of the present application, a total of 15 joints are shown in the following numbers: upper cervical portion, lower cervical portion 1, left shoulder 2, left elbow 3, left wrist 4, right shoulder 5, right elbow 6, right wrist 7, tail vertebra 8, left hip joint 9, left knee 10, left ankle 11, right hip joint 12, right knee 13, right ankle 14.

Wherein the node set V includes two-dimensional coordinates of joints in all frames of the gait video; the edge set E includes two types of spatial connection relationships: a proximal end, a distal end; time connection relation: a time sequence connection; the proximal end comprises a left shoulder, a right shoulder, a vertebra, a left hip, a right hip, a left upper arm, a right upper arm, a left thigh, a right thigh, and the distal end comprises a left lower arm, a right lower arm, a left lower leg; the time sequence connection comprises the connection between the node at the current moment of each joint and the node at the next moment;

in other embodiments, it may also be an edge set E that is selected based on other spatial or temporal conditions.

In an embodiment of the present application, the spatial connection relation of the edge set E is expressed in the adjacent matrix a in actual operation, and the number a of the ith row and the jth column in the matrix a _ij Representing the spatial connection relationship between joint i and joint j, and if the spatial connection relationship is proximal, then a _ij X, if the spatial connection relationship is far-end, a _ij Y, where x, y is a positive integer, and 1<x<And y. For example: x=2, y=3.

wherein X is _t The sequence of two-dimensional coordinate time delay direction of a certain joint is a matrix of t rows and 2 columns, t is the number of frames contained in video, N is the number of convolution kernels of time sequence convolution, and W _k The k time sequence convolution kernel is a one-dimensional vector, the length of the one-dimensional vector can be freely adjusted according to the requirement, and B is a learnable linear bias.

Further, referring to fig. 6, the neural network used for training, and the parkinson's disease gait impairment evaluation model obtained after training are a dual-flow neural network, including a skeleton flow, a contour flow, a vector stitching unit, a full connection layer, a softmax layer;

The full name of the ST-GCN is Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition, namely a space-time diagram convolution network for human motion recognition.

VGG is an image recognition network proposed by Visual Geometry Group of Oxford in 2014, using a deeper network architecture, demonstrating that increasing network depth can affect network performance to some extent.

Further, the calculation of the ST-GCN unit of the present application is as follows:

Specifically, referring to fig. 8, in an embodiment of the present application, the skeleton flow includes 10 ST-GCN network elements, and the profile flow includes 5 VGG network elements. Each VGG unit consists of 2 or 3 two-dimensional convolution layers and 1 layer maximum pooling layer, and the number of two-dimensional convolution layers and the number of channels in five VGG units are shown in table 1. Wherein, the convolution layers all adopt 3*3 convolution kernels, and the step length is 1.

TABLE 1

VGG block sequence number	Convolution kernel size	Number of channels	Number of convolution layers
				1	3	64	2
2	3	128	2
				3	3	256	3
4	3	512	3
				5	3	512	3

In an embodiment of the present application, the dual-flow neural network realizes the fusion of dual-flow information through vector stitching and full-connection layers, wherein the length of a skeleton flow output vector is 139264, the length of a joint flow output vector is 25088, a vector stitching unit stitches the two vectors into a long vector with a length of 164362, and then a predicted probability value of a neural network model for different preset score categories is obtained through two full-connection layers and one softmax layer, and the highest probability value is the estimated gait score value of the model for a patient to which the sample belongs.

For example, referring to fig. 8, when the preset score categories include: 0. 1, 2, respectively obtaining three fractional predictive probability values of 0,1 and 2 through softmax, wherein the highest probability is the final gait score estimated value.

Further, training the neural network in step S20 further includes:

By introducing the loss function, error convergence can be accelerated more quickly, and training efficiency is improved.

In another aspect, referring to fig. 3, the present application provides a method for gait impairment assessment of parkinson's disease, comprising the steps of:

The method is to use the Parkinsonism gait impairment evaluation model after the model is established. Based on the unified concept, the specific flow of extracting the skeleton diagram sequence and the multi-cycle gait energy diagram is described above, and will not be described herein.

grouping gait videos of the same person to be evaluated, correspondingly and respectively obtaining a plurality of groups of gait scores, and selecting the score with the largest occurrence number from the plurality of groups of gait scores, namely obtaining the gait score optimization result of the person to be evaluated.

For example, when 6 pieces of gait video of the same person to be evaluated are input, 6 scoring results are obtained: 1,1,1,2,0,0. The maximum number of times is 1, so the gait score optimization result of the person to be evaluated should be 1.

Further, when the scores with the largest occurrence number are multiple, the higher one of the scores is selected as the gait score optimization result of the person to be evaluated.

For example, when 6 pieces of gait video of the same person to be evaluated are input, 6 scoring results are obtained: 1,1,1,0,0,0. The maximum number of times is 1 and 0, and 1 is greater than 0, so the gait score optimization result of the person to be evaluated should be 1.

Specifically, in one embodiment of the present application, an experimental dataset was constructed based on the Tianjin city, ring lake hospital platform, containing 54 PD patients and age-matched 26 healthy persons for comparison. In the evaluation process, accuracy (precision), recall (recall), F1 score (F1-score), and subject work characteristic curve (ROC) and area under the curve (AUC) are used as classification results of the performance index evaluation model, and specific analysis of the experimental results is as follows:

1. classification result analysis

Five-fold cross validation is adopted on an experimental data set to evaluate the parkinsonism gait impairment evaluation model obtained by the method for establishing the parkinsonism gait impairment evaluation model, and experimental results show that the method can obtain 71.3% of total accuracy. As shown in table 2, classification performance of the parkinson's disease gait impairment evaluation model obtained by the method for establishing the parkinson's disease gait impairment evaluation model in each category is given, wherein the accuracy, recall, F1 score and AUC value of each score category are calculated, and each score category obtains satisfactory accuracy. This is also shown by the confusion matrix shown in fig. 15. ROC curves under each score category are shown in fig. 14, where healthy person category (0 score) performs best and 1 score 2 score performs nearly. Therefore, the parkinsonism gait impairment evaluation model obtained by the method for establishing the parkinsonism gait impairment evaluation model can better distinguish parkinsonism patients from healthy people, and has the classification capability of early disease screening.

TABLE 2

	Accuracy rate of	Recall rate of recall	F1 fraction	AUC
					0	0.840	0.808	0.824	0.892
1	0.586	0.654	0.618	0.745
					2	0.731	0.679	0.704	0.733

2. In contrast to advanced parkinsonian gait assessment work

In table 3, comparing the present application with the world leading related field work, the results show that the method provided by the present application obtains better classification performance in a larger sample set, and the classification task is significantly superior to the existing methods and works.

TABLE 3 Table 3

Further, referring to fig. 4, after obtaining the gait score, the application further provides a refined evaluation method, and the damaged condition of each joint is obtained by calculating the neural network response value, which is specifically as follows:

after step S2, the method further comprises the steps of:

Further, the joint response value is calculated as follows:

wherein S is a joint response value vector, N is the number of joints, each value in the joint response value vector correspondingly represents the response value of the Parkinson' S disease gait damage evaluation model at the joint, T is the length of an input skeleton diagram sequence, and C _out Is a frameworkNumber of output channels of stream, O _it Output matrix X for skeleton stream _out Is a component of the group.

Further, the joints are divided into 6 groups representing 6 body parts: neck (joint 0, joint 1), torso (joint 1, joint 8), left arm (joint 5-7), right arm (joint 2-4), left leg (joint 12-14), right leg (joint 9-11), and step S3 of classifying the joint response values to obtain an average response value for each body part specifically includes:

the average response values for each set of joints are calculated separately to obtain an average joint response value for each body part. Thereby further obtaining a representation of the damage to each of the above-mentioned body parts of the subject.

Specifically, referring to fig. 12 and 13, in an embodiment of the present application, a statistical analysis is performed on the response value vectors of 54 PD patients in the experimental data set, and the average response value of each joint and the average response value of each body part are obtained, and the obtained results are shown in fig. 12 and 13.

Fig. 12 shows a distribution of average response values of each joint, wherein the average response value of each joint is a value after normalization, specifically, the average response value of each joint is divided by the maximum value thereof. For example, if the average response value of the joint 0 is 20 and is the maximum value, the average response value of the joint 0 is normalized to be 1, and accordingly, the average response value of the other joints is normalized to be 0.8, 0.6, 0.4, 0.2, and the like.

As can be seen from fig. 13, for PD patients, the neural network has a larger response to the neck, torso and arms, meaning that the neural network is more concerned about the movement of the upper body of the patient. Wherein, three body parts with highest response values are respectively: neck, torso, and left arm.

By the above method, after the score is obtained, the specific damaged body parts of the patient, and the degree of damage of each body part, can be further determined.

Further, referring to fig. 5, another refined gait impairment quantification method is also proposed, and a finer granularity of motion impairment situation is obtained through gait quantification parameters, which is specifically as follows:

s6: preprocessing gait videos, and extracting a skeleton diagram sequence, wherein the gait videos are walking videos of a person to be evaluated, which are photographed from the side;

s7: calculating and obtaining quantization parameters according to the skeleton diagram sequence;

s8: and evaluating the motion characteristics of the person to be evaluated according to the quantization parameter.

The method for acquiring the skeleton map sequence in step S6 is as described above, and is not described herein.

In one aspect, the gait impairment quantification method may be a complementary quantification evaluation method after the gait score is obtained by using the model, and may be performed after step S1, after step S2, or after step S5;

On the other hand, the gait impairment quantification method can also be used alone.

Further, the motion characteristics and the corresponding quantization parameters and calculation modes are as follows:

when the motion characteristic is a step length, the quantization parameter is a step length, and the calculation mode is as follows: in the time-dependent curve of the distance between the left ankle joint 11 and the right ankle joint 14, the peak corresponds to a value;

when the motion characteristic is pace, the quantization parameter is trunk speed, and the calculation mode is as follows: the horizontal velocity of the lower cervical joint 1.

The motion characteristics are that when the arm swings, the quantization parameters comprise the swing angle of the arm and the asymmetric coefficient of the two sides of the swing of the arm,

wherein the arm swing angle is calculated by the maximum angle of the front and back swing of the upper arm (comprising the left side and the right side),

the calculation mode of the arm swinging bilateral asymmetry coefficient is as follows:

the swing angle of the larger side is theta _max The smaller side is theta _min 。

When the motion characteristic is neck forward flexion, the quantitative parameter is neck forward flexion angle, and the calculation mode is as follows: the connecting line of the upper cervical joint 0 and the lower cervical joint 1 forms an included angle with the vertical direction.

The motion characteristic is that when the trunk is forward bent, the quantization parameter is the forward bending angle of the trunk, and the calculation mode is the included angle formed by the connecting line of the lower neck joint 1 and the tail cone joint 8 and the vertical direction.

When the motion characteristic is gait cycle, the quantization parameter is gait cycle, and the calculation mode is the time interval between two adjacent peaks or the time interval between two adjacent valleys in the time-dependent change curve of the distance between the left ankle joint 11 and the right ankle joint 14. It should be noted that the gait cycle is different from the gait energy cycle described above, and the gait cycle is two steps and one cycle, and the gait energy cycle is one step and one cycle.

Specifically, referring to fig. 10, from time node 0.5 corresponding to the first peak to time node 1.75 corresponding to the second peak, a gait cycle is shown. Alternatively, one gait cycle is from time node 0.0 corresponding to the first valley to time node 1.25 corresponding to the second valley.

Further, the step S7 specifically includes:

s71: judging whether the value of the quantization parameter exceeds a preset threshold value interval, and if the value exceeds the preset threshold value interval, determining that the motion characteristic is abnormal.

The quantized parameters with physical significance are valued, so that the state of the corresponding motion characteristic can be represented, and therefore, effective reference is provided for finely quantizing gait damage and personalized gait evaluation. Based on the MDS-UPDRS gait score, the specific motion characteristics can be further known, so that the body parts with problems can be further judged.

Further, after determining that there is an abnormality in the motion characteristics, step S7 further includes the steps of:

s72: carrying out Spearman correlation analysis on the multiple quantization parameters and gait scores of the parkinsonism of the person to be evaluated to obtain correlation coefficients, wherein the gait scores are MDS-UPDRS scores corresponding to gait videos;

s73: and sequencing the relative numbers, and correspondingly sequencing the degree of abnormality of the obtained motion characteristics.

Specifically, in one embodiment of the present application, 7 quantization parameters described above were counted in 54 PD patients, and the Spearman correlation coefficient between these quantization values and the MDS-UPDRS gait score was calculated separately. Recording a certain quantized value set of 54 patients as X and gait score set of the 54 patients as Y, and arranging the two sets in order from small to large to obtain R (X) and R (Y), wherein the calculation method of the Spearman correlation coefficient rho is as follows:

where cov is the covariance calculation of the two variables and σ is the standard deviation of the variables.

The results of Spearman correlation analysis of the seven quantized values with gait score are shown in table 4 below:

TABLE 4 Table 4

From the above table, the three indices most relevant to gait score are the maximum values of torso velocity, neck forward flexion angle and arm swing angle, respectively. This is consistent with the three body part situations described above where the response value is highest, further verifying the validity of the index of neural network response values, i.e. its ability to respond to impaired gait.

In yet another aspect, the present application provides a computer device, which is characterized in that the computer device includes a computer readable storage medium, a processor, and a computer program stored on the computer readable storage medium and executable on the processor, where the processor implements the above-described method for establishing a parkinson's disease gait impairment evaluation model, or the method for parkinson's disease gait impairment evaluation, or the method for parkinson's disease gait impairment quantification when executing the computer program.

In yet another aspect, the present application provides a readable storage medium, on which a computer program is stored, which when executed by a processor, implements the above-described method for establishing a parkinson's disease gait impairment evaluation model, or the method for parkinson's disease gait impairment evaluation, or the method for parkinson's disease gait impairment quantification.

Referring specifically to fig. 16, in practical application, fig. 16 is a schematic structural diagram of a hardware operation environment related to a method for establishing a parkinson's disease gait impairment evaluation model, a method for parkinson's disease gait impairment evaluation, or a method for parkinson's disease gait impairment quantification in the present application.

As shown in fig. 16, the hardware runtime environment may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.

Those skilled in the art will appreciate that the hardware architecture of the method operation described in the present application illustrated in fig. 16 is not limiting of the method operation apparatus, and may include more or fewer components than illustrated, or may combine certain components, or a different arrangement of components.

As shown in fig. 16, an operating system, a network communication module, a user interface module, and a computer program may be included in the memory 1005 as one type of readable storage medium. The operating system is a management and control program and supports the operation of a network communication module, a user interface module, a computer program and other programs or software; the network communication module is used to manage and control the network interface 1004; the user interface module is used to manage and control the user interface 1003.

In the hardware configuration shown in fig. 16, the network interface 1004 is mainly used for connecting to a background server, and performing data communication with the background server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; the processor 1001 may call a computer program stored in the memory 1005 and perform the steps of the method of establishing the parkinson's disease gait impairment evaluation model, or the method of parkinson's disease gait impairment evaluation, or the method of parkinson's disease gait impairment quantification described above.

In summary, the present application has the following beneficial effects:

the gait video of the parkinsonism patient to be evaluated can be analyzed by utilizing the data streams of two different forms of the gait energy cycle chart and the skeleton chart, which are extracted from the gait video, and combining with the neural network training to generate a corresponding neural network model, so as to obtain the gait score, the gait damage degree of the parkinsonism patient can be conveniently and rapidly evaluated based on the gait score, and the accuracy is high.

Further, the gait video photographed from the side can extract human body contour features, particularly gait changes of human body contours in walking, in addition to skeleton map features of a human body based on joints, so that analysis of gait features is performed based on the changes of human body contour maps. For the video shot from the front, the change of the human body profile map cannot be accurately perceived, for example, the change of the step length, the step speed and the like cannot be observed, the change of the swing arm angle and the like, the inclination angle of the body and the like cannot be known, and the change features are extracted from the side face.

Further, a voting mechanism is used for obtaining gait score optimization results, and more accurate gait scores are obtained.

Further, by calculating the network response value, after the score is obtained, the damage condition of each joint can be further obtained.

Further, by classifying the joints, corresponding to different body parts, after obtaining the score, the patient's specific damaged body parts, and the degree of damage to each body part, can be further determined.

Furthermore, the gait quantized parameters are used for acquiring the motion damage condition with finer granularity, and on the basis of acquiring the gait score, the specific motion characteristics are further known to be problematic, so that the body parts are further judged to be problematic.

Further, a plurality of quantization parameters and gait scores of the parkinson's disease of the person to be evaluated are subjected to correlation analysis to obtain correlation coefficients, the correlation coefficients are ranked, and the ranking of the abnormality degrees of the motion features is correspondingly obtained, so that the relative abnormality degree of the motion features with problems is further known, and the direction which should be focused is given.

Claims

1. A method of establishing a parkinson's disease gait impairment assessment model, comprising the steps of:

And S20, training a neural network by using the skeleton diagram sequence and the multi-cycle gait energy diagram and combining gait impairment scoring labels obtained by doctors according to the gait video diagnosis so as to obtain a Parkinson' S disease gait impairment evaluation model, wherein the gait impairment scoring labels are used for dividing the gait impairment degree.

2. The method for establishing a parkinsonism gait impairment evaluation model according to claim 1, wherein the parkinsonism gait impairment evaluation model is a dual-flow neural network comprising a skeleton flow, a contour flow, a vector stitching unit, a full connection layer, a softmax layer;

the skeleton flow comprises a plurality of ST-GCN units and is used for processing an input skeleton graph sequence to obtain skeleton flow output vectors, the outline flow comprises a plurality of VGG units and is used for processing an input multi-period gait energy graph to obtain outline flow output vectors, the skeleton flow output vectors and the outline flow output vectors are input into the vector splicing unit to be spliced, double-flow information fusion is achieved through a full-connection layer, finally, predicted probability values for scores are obtained through a softmax layer, and the score with the highest predicted probability value is a gait score estimated value of the gait video.

3. The method for establishing a parkinson' S disease gait impairment evaluation model according to claim 1, wherein the preprocessing of gait videos in step S10, extracting a multi-cycle gait energy map, comprises:

s101, extracting a contour map from each frame of the gait video, and arranging the contour maps in time sequence to obtain a contour map sequence;

the gait energy period is a time interval between adjacent peaks and valleys in a change curve of the distance between two feet in the profile sequence along with time, and the starting point of the gait energy period is a time point corresponding to the peaks or the valleys;

4. A method for modeling parkinson' S disease gait impairment evaluation according to claim 3, wherein the contour maps within each group in step S103 are superimposed to form a gait energy map calculated as follows:

Wherein I is _c (x, y, N) is a contour plot extracted from an nth frame of the gait video, N being the sum of the gait energy periodX is the horizontal coordinate of the pixel point in the contour map, y is the vertical coordinate of the pixel point in the contour map, and the value of GEI (x, y) is the pixel.

5. The method for establishing a parkinson' S disease gait impairment evaluation model according to claim 4, wherein the preprocessing of gait video in step S10, extracting a skeleton map sequence, specifically comprises:

s111, extracting two-dimensional coordinates of a joint from each frame of the gait video;

and S112, calculating a skeleton diagram sequence corresponding to the gait video according to the two-dimensional coordinates of the joints and the spatial connection relation between the joints.

6. The method for building a parkinson' S disease gait impairment evaluation model according to claim 5, wherein in step S112, the skeleton map sequence corresponding to the gait video is calculated according to the two-dimensional coordinates of the joints and the spatial connection relationship between the joints, specifically as follows:

J＝(V，E)，

wherein node set V comprises two-dimensional coordinates of joints in all frames of the gait video; the edge set E includes two types of the spatial connection relations: a proximal end, a distal end; the proximal end comprises a left shoulder, a right shoulder, a vertebra, a left hip, a right hip, a left upper arm, a right upper arm, a left thigh and a right thigh, and the distal end comprises a left lower arm, a right lower arm and a left lower leg;

7. The method of modeling parkinson's disease gait impairment evaluation according to claim 6, wherein the edge set E further comprises a temporal connection: a time sequence connection; the time sequence connection comprises the connection between the node at the current moment of each joint and the node at the next moment of each joint;

the time sequence connection of the edge set E is realized by one-dimensional convolution of the joint along the time direction, and the time sequence connection is as follows:

wherein X is _t The sequence of two-dimensional coordinate time delay direction of a certain joint is a matrix of t rows and 2 columns, t is the number of frames contained in the gait video, N is the number of convolution kernels of time sequence convolution, and W _k Is the kth time sequence convolution kernel, is a one-dimensional vector, and B is a learnable linear bias.

8. The method of modeling parkinson's disease gait impairment evaluation according to claim 7, wherein the calculation of the ST-GCN unit is as follows:

Wherein X is _in To input features, X _out For output characteristics, A is the adjacency matrix, D ⁱⁱ ＝∑ _j (A ^ij +I ^ij ) And the degree matrix is used for normalizing the adjacent matrix A, W is a learnable weight coefficient matrix, and B is a learnable bias coefficient.

9. The method of modeling parkinson's disease gait impairment evaluation according to claim 2, wherein the VGG unit comprises a two-dimensional convolution layer and a max pooling layer.

10. The method for modeling parkinson' S disease gait impairment evaluation according to claim 2, wherein training the neural network in step S20 further comprises:

calculating a loss value through a loss function, back-propagating the loss value, and stopping training when the error converges;

Further, y _ic E {0,1}, when the class of the ith said sample is c, corresponding to y _ic =1, otherwise y _ic ＝0。

11. A method for parkinson's disease gait impairment assessment, comprising the steps of:

s1: preprocessing gait videos, and extracting a skeleton diagram sequence and a multi-period gait energy diagram, wherein the gait videos are walking videos of a person to be evaluated, which is photographed from the side;

S2: inputting the skeleton diagram sequence and the multi-cycle gait energy diagram into the parkinson's disease gait impairment assessment model obtained by the method for establishing the parkinson's disease gait impairment assessment model according to any one of claims 1 to 9, and obtaining the gait score of the person to be assessed.

12. The method for parkinson' S disease gait impairment assessment according to claim 11, wherein step S2 comprises using a voting mechanism to obtain gait score optimization results, in particular:

grouping the gait videos of the same person to be evaluated, correspondingly and respectively obtaining a plurality of groups of gait scores, and selecting the score with the largest occurrence number from the plurality of groups of gait scores, namely, obtaining the gait score optimization result of the person to be evaluated.

13. The method for parkinson' S disease gait impairment assessment according to claim 11, further comprising, after step S2, the steps of:

14. The method of parkinson's disease gait impairment assessment according to claim 13, wherein the joint response values are calculated as follows:

wherein S is a joint response value vector, N is the number of joints, each value in the joint response value vector correspondingly represents the response value of the parkinsonism gait impairment evaluation model at the joint, T is the input skeleton diagram sequence length, and C _out For the number of output channels of the skeleton stream, O _it Output matrix X for the skeleton stream _out Is a component of the group.

15. The method of parkinson's disease gait impairment assessment according to claim 14, wherein the joints are divided into 6 groups representing 6 body parts: the step S3 of classifying and calculating the joint response values to obtain an average response value of each body part specifically includes:

the average response values of each set of joints are calculated separately to obtain an average joint response value for each body part.

16. Computer device, characterized in that it comprises a computer readable storage medium, a processor and a computer program stored on the computer readable storage medium and executable on the processor, wherein the processor implements the method of building a parkinsonism gait impairment evaluation model according to any one of claims 1-9 or the method of parkinsonism gait impairment evaluation according to any one of claims 10-15 when executing the program.

17. A readable storage medium, characterized in that it has stored thereon a computer program, which when executed by a processor, implements a method of modeling a parkinson's disease gait impairment evaluation according to any one of claims 1-9 or implements a method of parkinson's disease gait impairment evaluation according to any one of claims 10-15.