CN104408775B - Depth perception based three-dimensional shadow play production method - Google Patents

Abstract

The invention discloses a three-dimensional shadow play production method based on depth perception, and belongs to the technical field of digital shadow play production. The present invention aims to solve the problem that the existing shadow puppet play inheritance methods cannot truly reflect the virtual-real transformation of the shadow puppet show, and cannot adaptively match the shadow puppet pattern. It first takes the overall image of the shadow play figure to be made as the front view of the three-dimensional model making module as a reference for the three-dimensional model of the shadow play figure; then makes a plane model of all body parts of the shadow play figure; carries out surface color mapping to the plane model; Make the plane model of all body parts of the shadow puppet figure and join them in order to form a 3D model of the shadow puppet figure; bind the 3D model of the shadow play figure with all the bones; establish a mapping between all the bones of the 3D shadow play figure model and the 3D coordinate information of the corresponding human bone joints relationship; and then use the human skeleton movement sequence to control the skeleton movement of the 3D model of the shadow puppet figure. The invention is used for making three-dimensional shadow puppets.

Description

Three-dimensional shadow play production method based on depth perception

technical field

The invention relates to a three-dimensional shadow play production method based on depth perception, and belongs to the technical field of digital shadow play production.

Background technique

Shadow puppetry originated in the Han Dynasty and is the earliest screen shadow art for people to watch in the world. Shadow puppet art has experienced thousands of years of development and inheritance. It bears mutual bearing with folk customs and integrates with local traditions and folk culture. Its modeling technique is unique, with special expressive force and formal beauty. It is a traditional rap performance art and folk craft in my country The unique art variety formed by the perfect combination of fine arts was once an important part of people's daily life entertainment. However, with the emergence and rapid development of other art forms and information technology, people gradually get used to obtaining information and interacting with entertainment through TV, personal computer and the Internet without leaving home. Technology products have caused a serious loss of audience for shadow puppetry. In addition, shadow puppetry is still mainly passed on in the form of master-apprentice teaching. The production process of shadow puppetry is complicated and difficult to preserve. The performance needs to be performed in a specific venue. It is a reappearance of traditional performances, and rarely presents new scripts, new styles, and new performance forms, which makes the inheritance of shadow puppetry in the contemporary dilemma. Take the Wangkui shadow puppetry, which is also included in the world's intangible cultural heritage, as an example. The average age of the inheritors of shadow puppetry and other performers is over 70 years old, but there is still no clear successor. This intangible cultural heritage is almost at risk of being lost. dangerous situation.

The fact that shadow puppetry is on the verge of extinction has attracted the attention of many scholars, who have begun to use different ideas to protect and inherit shadow puppetry. Some scholars start with the inheritance of the performance form of shadow puppetry. For example, Fu Shaofeng of Xidian University and others used intelligent robots instead of shadow puppets to perform shadow puppetry, and Tang Aofei of Xi'an University of Technology and others replaced traditional The shadow puppeteer uses a joystick to control the movement of the shadow puppet in the performance space. However, robot or mechanically controlled performances always need to be carried out according to the preset program, which is rigid and unnatural. It can only mechanically imitate the shape of shadow puppetry, but cannot express its spirit. Another group of scholars started with the plight of shadow puppets with complex carving techniques and difficulty in preservation. For example, Hua Qingyi and others from Northwest University scanned and digitized the various components of traditional shadow puppets and shadow puppet backgrounds, and proposed that two-dimensional shadow puppet figures and The modeling method of the shadow puppet background realizes an Internet-based digital shadow puppet production method using a mouse for interaction. Cai Jianping from Beijing University of Technology used hand-drawn traditional shadow puppets as materials to build a shadow play digital performance platform based on Flash two-dimensional animation software, allowing users to use the mouse and touch screen to realize interactive shadow play animation production. However, these methods only form a two-dimensional animation with a shadow play style and a similar style of background music. difficult to express properly. Wen Yan et al. from East China Normal University used a Kinect-based somatosensory control method to establish the action link between the shadow puppet performer and the two-dimensional digital shadow puppet figure, and tried to use the performer's body movements to control the shadow puppet figure to perform, so that the remote client computer You can watch shadow puppetry performances on the screen. However, this method only uses two-dimensional information of human body movement to imitate the horizontal movement of shadow puppet characters on the screen plane, and does not truly reflect the virtual-real transformation effect presented by moving away from and approaching the shadow screen during shadow puppetry performances. That is to say, although the performance of traditional shadow puppets is realized through the light transmission effect of the shadow screen, the shadow puppets do not move close to the shadow screen, but imitate reality in the three-dimensional space formed by the plane of the shadow screen and the back of the shadow screen. human body movement. Qiu Hang and others from the University of Electronic Science and Technology of China disclosed a method for making a 3D shadow puppet model by stylizing an ordinary 3D model into a 3D shadow puppet model with the aesthetic characteristics of traditional shadow puppetry. The shadow puppet model has almost no thickness, and simply stylizing some traditional shadow puppet patterns into the generated shadow puppet model is also contrary to the digital protection and inheritance of shadow puppetry.

Contents of the invention

The purpose of the present invention is to solve the problem that the existing inheritance method of shadow puppetry cannot truly reflect the virtual-real transformation of shadow puppetry performance, and cannot adaptively match the shadow puppet pattern, and provides a three-dimensional shadow puppet production method based on depth perception.

The three-dimensional shadow play production method based on depth perception of the present invention, it comprises the following steps:

Step 1: use the overall image of the shadow play figure to be made as the front view of the three-dimensional model making module, and the front view is used as a reference for the three-dimensional model of the shadow play figure, and the overall image of the shadow play figure to be produced presents 90-degree movements of hands and feet;

Step 2: Establish a plane model of all body parts of the shadow puppet figure to be made according to the front view;

Step 3: Carry out surface color mapping on the plane model of all body parts of the shadow puppet figure to be produced;

Step 4: Adjust the positions of the plane models of all the body parts of the shadow puppet figure to be made, and connect them in order to form a three-dimensional model of the shadow puppet figure;

Step 5: Create all the bones of the 3D model of the shadow puppet, and then bind the 3D model of the shadow puppet to all the bones through the skinning of the bones;

Step 6: Use the depth sensor to capture the movement of human bones, obtain the three-dimensional coordinate information of all human bone joints, and establish a mapping relationship between all the bones of the shadow puppet figure's three-dimensional model and the three-dimensional coordinate information of the corresponding human bone joints; then use the captured human bone movement sequence Control all the skeletal movements of the 3D model of shadow puppets, and play the recorded vocals or accompaniment music synchronously to realize the production of 3D shadow puppets.

The method for obtaining the overall image of the shadow puppet figure to be produced in step 1 is: take the overall photo of the shadow puppet figure to be produced, use an image processing tool to remove the background from the overall photo of the shadow puppet figure to be produced, and then save it as a png format file as the overall figure of the shadow puppet figure to be produced image.

The specific method of establishing the plane model of all body parts of the shadow puppet figure to be made in step 2 is as follows:

In the three-dimensional model making module, according to the overall image of the shadow puppet figure to be made as the front view of the three-dimensional model making module described in step 1, a new plane model is created, and the parameters of the new plane model are modified to obtain the head crop parts P1 plane model, P2 plane model of upper body, P3 plane model of right upper arm, P4 plane model of right forearm, P5 plane model of right hand, P6 plane model of left forearm, P7 plane model of left forearm, P8 plane model of left hand, P9 plane model of right upper leg, P10 plane model of lower leg, P11 plane model of right foot, P12 plane model of left upper leg, P13 plane model of left lower leg and P14 plane model of left foot;

The basis for modifying the parameters of the new plane model is to disassemble the shadow puppet figure to be made into head stubble, two upper arms, two forearms, two hands, one body, two thighs, two calves and two There are 14 body parts in total, and the front and back photos of the 14 body parts are taken respectively, and the background is removed from the front and back photos of the 14 body parts with image processing tools, and then saved as png format files as corresponding The basis for modifying the parameters of the newly created plane model.

In step 3, the specific method for mapping the surface color of all the body parts of the shadow puppet figure to be produced is as follows:

In the 3D model making module, follow the order from left to right and from top to bottom to texture the plane models of all body parts: first select the plane model of the head stubble part P1, create its plane texture in the Z-axis direction, and then use the The plane model of the head crop part P1 is corresponding to the png format file of the body part, and the surface color mapping is carried out on the plane model of the head crop part P1;

The specific method of forming the three-dimensional model of shadow puppet figure in the step 4 is:

When head stubble part P1 faces to the right:

Move the Z-axis direction, the positive direction means moving forward, and the negative direction means backward; join the 14 body parts plane models in the following hierarchical order: the upper body P2 plane model covers the head part P1 plane model; the right hand P5 plane model covers the right forearm P4 plane model, the P4 plane model of the right forearm covers the P3 plane model of the right arm, the P3 plane model of the right arm covers the P2 plane model of the upper body; the P11 plane model of the right foot covers the P10 plane model of the lower right leg, and the P10 plane model of the right lower leg covers the upper right leg P9 plane model, right upper leg P9 plane model covers upper body P2 plane model; upper body P2 plane model covers left forearm P6 plane model, left forearm P6 plane model covers left forearm P7 plane model, left forearm P7 plane model covers left hand P8 Plane model; P2 plane model of the upper body covers the P12 plane model of the left upper leg, P12 plane model of the left upper leg covers P13 plane model of the left lower leg, P13 plane model of the left lower leg covers the P14 plane model of the left foot; a three-dimensional model of a shadow puppet figure is formed.

In Step 5, the specific method of binding the 3D model of the shadow puppet figure to all the bones is as follows:

All the bones for building the 3D model of shadow puppets are 15 joint points: head joint S1, shoulder center S2, right shoulder joint S3, right elbow joint S4, right wrist joint S5, left shoulder joint S6, left elbow joint S7, left wrist joint S8 , central hip joint S9, right hip joint S10, right knee joint S11, right ankle joint S12, left hip joint S13, left knee joint S14, and left ankle joint S15; The skeleton is bound to the 3D model of the shadow puppet figure.

In step six, the specific method for establishing a mapping relationship between all the bones of the three-dimensional model of the shadow play figure and the three-dimensional coordinate information of the corresponding human bone joints is:

The depth sensor is used to capture the human skeleton movement of the performer to obtain the three-dimensional coordinate information of the bone joint points during the movement of the performer; at the same time, the microphone array at both ends of the depth sensor is used to obtain the performer's singing voice and the opera played by the band; the obtained performer is saved in the whole Skeletal movement sequences and corresponding audio signals during the performance;

Extract the three-dimensional coordinate information of 20 skeleton joint points of the performer from each frame of the skeleton motion sequence, and combine the three-dimensional coordinate information of 15 corresponding skeleton joint points in the three-dimensional coordinate information of the 20 skeleton joint points with the 15 three-dimensional coordinate information of the shadow puppet figure. The mapping relationship of the joint points is as follows: assign the three-dimensional coordinate information of the head joint point in the head skeleton position module to the head joint S1; assign the three-dimensional coordinate information of the shoulder joint point in the shoulder skeleton position module to the center of the shoulders S2; Assign the three-dimensional coordinate information of the right shoulder joint point in the shoulder skeleton position module to the right shoulder joint S3; assign the three-dimensional coordinate information of the right elbow joint point in the right elbow skeleton position module to the right elbow joint S4; assign the three-dimensional coordinate information of the right wrist joint point in the right wrist skeleton position module Assign the coordinate information to the right wrist joint S5; assign the three-dimensional coordinate information of the left shoulder joint point in the left shoulder skeleton position module to the left shoulder joint S6; assign the three-dimensional coordinate information of the left elbow joint point in the left elbow skeleton position module to the left elbow joint S7; assign the left wrist skeleton position module Assign the three-dimensional coordinate information of the left wrist joint point in the position module to the left wrist joint S8; assign the three-dimensional coordinate information of the hip central joint point in the hip central skeleton position module to the hip central joint S9; assign the right hip joint point three-dimensional coordinate information in the right hip skeleton position module Assign the value to the right hip joint S10; assign the three-dimensional coordinate information of the right knee joint point in the right knee skeleton position module to the right knee joint S11; assign the three-dimensional coordinate information of the right ankle joint point in the right ankle skeleton position module to the right ankle joint S12; Assign the three-dimensional coordinate information of the left hip joint point in the left hip skeleton position module to the left hip joint S13; assign the three-dimensional coordinate information of the left knee joint point in the left knee skeleton position module to the left knee joint S14; assign the left ankle joint point information in the left ankle skeleton position module The three-dimensional coordinate information of joint points is assigned to the left ankle joint S15;

The specific method for controlling all the bone motions of the 3D model of the shadow puppet figure by using the captured human bone motion sequence is as follows:

Use the relative distance in the x-axis direction of two adjacent frames corresponding to the image corresponding to the center of the shoulders S2 in the skeletal motion sequence to control the face orientation of the shadow puppet figure in the 3D model of the shadow puppet figure: the positive semi-axis of the x-axis extends to the left as the standard, when two adjacent frames When the difference between the x coordinates of the image corresponding to the center of the shoulders S2 in the skeleton sequence is positive, turn the plane model of the stubble part P1 controlled by the head joint S1 to the left, and when the images corresponding to the center of the shoulders S2 in the two adjacent frames of the skeleton sequence When the difference of the x coordinates of the two is negative, turn the plane model of the stubble part P1 controlled by the head joint S1 to the right; combine the other corresponding image actions in the application bone motion sequence to control all other bone motions of the shadow puppet 3D model, and then synchronize Play the recorded vocals or accompaniment music of performers to realize the production of 3D shadow puppetry.

Advantages of the present invention: the present invention is a digital shadow puppet production method, which generates a realistic three-dimensional shadow puppet figure model on the basis of traditional shadow puppet image materials, and then uses a depth sensor to capture the bones corresponding to the performance actions of traditional opera performers The movement sequence and recording of the performer's singing and accompaniment music, and finally the captured skeleton data is bound to the skeleton of the 3D shadow puppet character model to generate a realistic 3D shadow puppetry that is different from traditional opera.

The method of the invention can control the three-dimensional model of the virtual shadow puppet figure in real time, it captures the movement characteristics of the human body by means of a specific depth sensor, and then establishes the relationship between the human skeleton and the shadow puppet skeleton, so as to realize the movement of the shadow puppet skeleton controlled by the human motion skeleton sequence. In this way, a real-time, immersive and realistic three-dimensional shadow puppet show is generated, which truly reflects the virtual-real transformation of the shadow puppet show, and realizes adaptive matching of shadow puppet patterns.

Description of drawings

Fig. 1 is the schematic diagram of all body parts planar models of shadow puppet figures to be made;

Fig. 2 is a schematic diagram of dismantling a real shadow puppet figure;

Fig. 3 is the three-dimensional model of the shadow puppet figure after splicing in order;

Fig. 4 is the schematic diagram of all skeletal articulation points of shadow puppet figure three-dimensional model;

Fig. 5 is a schematic diagram of all the skeletons of the three-dimensional model of the shadow play figure.

detailed description

Specific embodiment one: below in conjunction with Fig. 1 to Fig. 5 illustrate this embodiment, the three-dimensional shadow puppet production method based on depth perception described in this embodiment, it comprises the following steps:

Step 1: use the overall image of the shadow play figure to be made as the front view of the three-dimensional model making module, and the front view is used as a reference for the three-dimensional model of the shadow play figure, and the overall image of the shadow play figure to be produced presents 90-degree movements of hands and feet;

Step 2: Establish a plane model of all body parts of the shadow puppet figure to be made according to the front view;

Step 3: Carry out surface color mapping on the plane model of all body parts of the shadow puppet figure to be produced;

Step 4: Adjust the positions of the plane models of all the body parts of the shadow puppet figure to be made, and connect them in order to form a three-dimensional model of the shadow puppet figure;

Step 5: Create all the bones of the 3D model of the shadow puppet, and then bind the 3D model of the shadow puppet to all the bones through the skinning of the bones;

Step 6: Use the depth sensor to capture the movement of human bones, obtain the three-dimensional coordinate information of all human bone joints, and establish a mapping relationship between all the bones of the shadow puppet figure's three-dimensional model and the three-dimensional coordinate information of the corresponding human bone joints; then use the captured human bone movement sequence Control all the skeletal movements of the 3D model of shadow puppets, and play the recorded vocals or accompaniment music synchronously to realize the production of 3D shadow puppets.

Specific embodiment two: this embodiment will further explain embodiment one, the method for obtaining the overall image of the shadow puppet figure to be made in the step 1 is: take and obtain the overall photo of the shadow puppet figure to be made, and use an image processing tool to process the background of the overall photo of the shadow puppet figure to be produced Cut out, and then save it as a png format file as the overall image of the shadow puppet figure to be made.

The image processing tool in this embodiment can use Photoshop.

Specific embodiment three: below in conjunction with Fig. 1 to Fig. 3 illustrate present embodiment, present embodiment is described further to embodiment two, and the concrete method of setting up the whole body parts plane model of shadow puppet figure to be made in step 2 is:

In the three-dimensional model making module, according to the overall image of the shadow puppet figure to be made as the front view of the three-dimensional model making module described in step 1, a new plane model is created, and the parameters of the new plane model are modified to obtain the head crop parts P1 plane model, P2 plane model of upper body, P3 plane model of right upper arm, P4 plane model of right forearm, P5 plane model of right hand, P6 plane model of left forearm, P7 plane model of left forearm, P8 plane model of left hand, P9 plane model of right upper leg, P10 plane model of lower leg, P11 plane model of right foot, P12 plane model of left upper leg, P13 plane model of left lower leg and P14 plane model of left foot;

The basis for modifying the parameters of the new plane model is to disassemble the shadow puppet figure to be made into head stubble, two upper arms, two forearms, two hands, one body, two thighs, two calves and two There are 14 body parts in total, and the front and back photos of the 14 body parts are taken respectively, and the background is removed from the front and back photos of the 14 body parts with image processing tools, and then saved as png format files as corresponding The basis for modifying the parameters of the newly created plane model.

In this embodiment, the overall image of the shadow puppet figure to be made is imported into the front view of the three-dimensional model making module, placed at the center point of the front view, and used as a reference for modeling to form a shadow play figure whose hands and feet are at 90 degrees. According to the overall image of the shadow puppets to be made in the front view, adjust the length and width ratio of the corresponding plane model so that the proportion of the plane model matches the size of the actual shadow puppets, as shown in Figure 1. Secondly, after the size matching of the planar model is completed, the points of the planar model are added, the positions of the corresponding points are adjusted, and the material of the shadow puppet in the background is combined to make the appearance ratio of the planar model closer to the actual shadow puppet.

Specific implementation mode four: present implementation mode is described further to implementation mode three, and in step 3, the specific method of carrying out the texture map of surface color to be made shadow puppet figure whole body parts plane model is:

In the 3D model making module, follow the order from left to right and from top to bottom to texture the plane models of all body parts: first select the plane model of the head stubble part P1, create its plane texture in the Z-axis direction, and then use the The plane model of the head crop part P1 is corresponding to the png format file of the body part, and the surface color mapping is carried out on the plane model of the head crop part P1;

Specific embodiment five: below in conjunction with Fig. 1, Fig. 2 and Fig. 3 illustrate present embodiment, present embodiment is described further to embodiment 4, the concrete method that forms shadow puppet character three-dimensional model in the step 4 is:

When head stubble part P1 faces to the right:

Move the Z-axis direction, the positive direction means moving forward, and the negative direction means backward; join the 14 body parts plane models in the following hierarchical order: the upper body P2 plane model covers the head part P1 plane model; the right hand P5 plane model covers the right forearm P4 plane model, the P4 plane model of the right forearm covers the P3 plane model of the right arm, the P3 plane model of the right arm covers the P2 plane model of the upper body; the P11 plane model of the right foot covers the P10 plane model of the lower right leg, and the P10 plane model of the right lower leg covers the upper right leg P9 plane model, right upper leg P9 plane model covers upper body P2 plane model; upper body P2 plane model covers left forearm P6 plane model, left forearm P6 plane model covers left forearm P7 plane model, left forearm P7 plane model covers left hand P8 Plane model; P2 plane model of the upper body covers the P12 plane model of the left upper leg, P12 plane model of the left upper leg covers P13 plane model of the left lower leg, P13 plane model of the left lower leg covers the P14 plane model of the left foot; a three-dimensional model of a shadow puppet figure is formed.

Since the shadow puppet itself connects the various parts of the shadow puppet figure through lines, there is a positional relationship of the order in which the shadow puppet parts are attached, so there should also be a corresponding order between the parts of the established shadow play plane model. Taking the orientation of the head as a standard, in this embodiment, the position of the planar model is adjusted by taking the head facing right as an example. First split the shadow puppet model, as shown in Figure 2. In this embodiment, the fourteen shadow puppet parts plane models are joined together, which can form a sense of distance that the left side of the shadow puppet figure is farther away and the right side is closer. ,As shown in Figure 3.

Specific embodiment six: the present embodiment is described below in conjunction with Fig. 4 and Fig. 5, and present embodiment is further described to embodiment five, and in step five, the specific method that the three-dimensional model of shadow puppet figure is established binding with all skeletons is:

All the bones for building the 3D model of shadow puppets are 15 joint points: head joint S1, shoulder center S2, right shoulder joint S3, right elbow joint S4, right wrist joint S5, left shoulder joint S6, left elbow joint S7, left wrist joint S8 , central hip joint S9, right hip joint S10, right knee joint S11, right ankle joint S12, left hip joint S13, left knee joint S14, and left ankle joint S15; The skeleton is bound to the 3D model of the shadow puppet figure.

Since the shape of shadow puppet figures is mainly profile and half profile, the bones have their own unique characteristics in the process of movement, so the bone binding of shadow puppet figures also has its own characteristics. According to the characteristics of the shadow puppet figure, the main skeleton of the shadow puppet figure is divided into the above 15 joint points, as shown in Figure 4.

In the 3D model making software, according to the characteristics of the shadow puppet figure, the bones from the head to the feet and from the left hand to the right hand are established in sequence, as shown in Figure 5. Then, through the skinning of the skeleton, the movement of the skeleton can really establish a connection with the 3D model of the shadow puppet figure. Finally, brush weights to make individual bones affect individual model parts. So far, the 3D model of the shadow puppet figure and its corresponding skeleton have been established and bound.

Specific embodiment seven: the present embodiment is described below in conjunction with Fig. 4 and Fig. 5, and present embodiment is further described to embodiment six, and in step six, all skeletons of the three-dimensional model of the shadow play figure are mapped with the three-dimensional coordinate information of corresponding human skeleton joints The specific method of the relationship is:

The depth sensor is used to capture the human skeleton movement of the performer to obtain the three-dimensional coordinate information of the bone joint points during the movement of the performer; at the same time, the microphone array at both ends of the depth sensor is used to obtain the performer's singing voice and the opera played by the band; the obtained performer is saved in the whole Skeletal movement sequences and corresponding audio signals during the performance;

Extract the three-dimensional coordinate information of 20 skeleton joint points of the performer from each frame of the skeleton motion sequence, and combine the three-dimensional coordinate information of 15 corresponding skeleton joint points in the three-dimensional coordinate information of the 20 skeleton joint points with the 15 three-dimensional coordinate information of the shadow puppet figure. The mapping relationship of the joint points is as follows: assign the three-dimensional coordinate information of the head joint point in the head skeleton position module to the head joint S1; assign the three-dimensional coordinate information of the shoulder joint point in the shoulder skeleton position module to the center of the shoulders S2; Assign the three-dimensional coordinate information of the right shoulder joint point in the shoulder skeleton position module to the right shoulder joint S3; assign the three-dimensional coordinate information of the right elbow joint point in the right elbow skeleton position module to the right elbow joint S4; assign the three-dimensional coordinate information of the right wrist joint point in the right wrist skeleton position module Assign the coordinate information to the right wrist joint S5; assign the three-dimensional coordinate information of the left shoulder joint point in the left shoulder skeleton position module to the left shoulder joint S6; assign the three-dimensional coordinate information of the left elbow joint point in the left elbow skeleton position module to the left elbow joint S7; assign the left wrist skeleton position module Assign the three-dimensional coordinate information of the left wrist joint point in the position module to the left wrist joint S8; assign the three-dimensional coordinate information of the hip central joint point in the hip central skeleton position module to the hip central joint S9; assign the right hip joint point three-dimensional coordinate information in the right hip skeleton position module Assign the value to the right hip joint S10; assign the three-dimensional coordinate information of the right knee joint point in the right knee skeleton position module to the right knee joint S11; assign the three-dimensional coordinate information of the right ankle joint point in the right ankle skeleton position module to the right ankle joint S12; Assign the three-dimensional coordinate information of the left hip joint point in the left hip skeleton position module to the left hip joint S13; assign the three-dimensional coordinate information of the left knee joint point in the left knee skeleton position module to the left knee joint S14; assign the left ankle joint point information in the left ankle skeleton position module The three-dimensional coordinate information of joint points is assigned to the left ankle joint S15;

The specific method for controlling all the bone motions of the 3D model of the shadow puppet figure by using the captured human bone motion sequence is as follows:

Use the relative distance in the x-axis direction of two adjacent frames corresponding to the image corresponding to the center of the shoulders S2 in the skeletal motion sequence to control the face orientation of the shadow puppet figure in the 3D model of the shadow puppet figure: the positive semi-axis of the x-axis extends to the left as the standard, when two adjacent frames When the difference between the x coordinates of the image corresponding to the center of the shoulders S2 in the skeleton sequence is positive, turn the plane model of the stubble part P1 controlled by the head joint S1 to the left, and when the images corresponding to the center of the shoulders S2 in the two adjacent frames of the skeleton sequence When the difference of the x coordinates of the two is negative, turn the plane model of the stubble part P1 controlled by the head joint S1 to the right; combine the other corresponding image actions in the application bone motion sequence to control all other bone motions of the shadow puppet 3D model, and then synchronize Play the recorded vocals or accompaniment music of performers to realize the production of 3D shadow puppetry.

In this embodiment, the depth sensor can use a Kinect depth camera, which can perform complete bone tracking on the human body and provide three-dimensional coordinate information of 20 bone joint points. Therefore, the mapping relationship between the 15 skeletal joint points in the 3D model of the shadow play figure and the corresponding joint points provided by Kinect can be established to realize the movement of the 3D model of the shadow play figure by using the human body motion captured by Kinect.

When the skeletal nodes captured by Kinect are used to directly control the movement of the 3D model of shadow puppets, the orientation of their faces cannot be distinguished. According to the characteristics of shadow puppetry in which the neck is used as the supporting point to control the turning of the shadow puppet figure, here the relative distance of the central joint point S2 of the shoulders in the x-axis direction in two adjacent frames is used to control the orientation of the shadow puppet figure's face. Here, from the perspective of the Kinect depth camera, the positive semi-axis of the x-axis extends to the left as the standard. When the difference between the x-coordinates of S2 in two adjacent frames of skeleton sequences is positive, the head part P1 controlled by the head joint S1 Flip to the left, when the difference between the x coordinates of S2 in two adjacent frames of skeleton sequences is negative, flip the head part P1 controlled by the head joint S1 to the right. This solves the problem that the head of the shadow puppet model controlled by Kinect is always facing one side.

When the captured skeletal movement sequence is used to control the 3D model of the shadow puppet figure, the recorded vocals and accompaniment music of the performer are played synchronously to present a corresponding realistic 3D shadow play. Applying the method of the present invention can not only generate a three-dimensional shadow puppet show in which shadow puppet performers use their limbs to perform action performances, but also can control the shadow puppet character model by combining the recorded singing and accompaniment music by the performance actions of other traditional opera performers to generate other shadow puppet performance styles. Traditional opera three-dimensional shadow puppetry.

Claims (1)

1. a kind of three-dimensional shadow show manufacture method based on depth perception, it comprises the following steps：

Step one：Figure for shadow-play personage general image to be produced is made into the front view of module, the front view conduct as threedimensional model The reference of figure for shadow-play personage's threedimensional model, the figure for shadow-play personage general image to be produced are presented 90 degree of actions of trick；

Step 2：Figure for shadow-play personage whole body part areal model to be produced is set up according to the front view；

Step 3：The pinup picture of surface color is carried out to figure for shadow-play personage to be produced whole body part areal model；

Step 4：The position of figure for shadow-play personage to be produced whole body part areal model is adjusted, and which is sequentially sewed is connect, form skin Actor or actress's thing threedimensional model；

Step 5：All skeletons of figure for shadow-play personage's threedimensional model, then the eyelid covering by skeleton are set up, by figure for shadow-play personage's threedimensional model Set up with all skeletons and bind；

Step 6：Skeleton motion is caught using depth transducer, the three-dimensional coordinate information of all human body skeletal joints is obtained, Three-dimensional coordinate information of all skeletons of figure for shadow-play personage's threedimensional model with corresponding human body skeletal joint is set up into mapping relations；It is sharp again It is with all skeleton motions of catcher's body skeleton motion sequence control figure for shadow-play personage's threedimensional model, synchronous to play the performance recorded Person's music for voices in a Chinese opera or accompaniment melody, realize that three-dimensional shadow show makes；

In step one, the preparation method of figure for shadow-play personage general image to be produced is：Shoot and obtain figure for shadow-play personage entirety phase to be produced Piece, carries out background to figure for shadow-play personage to be produced entirety photograph using image processing tool and removes, then save as png formatted files As figure for shadow-play personage general image to be produced；

The concrete grammar that figure for shadow-play personage to be produced whole body part areal model is set up in step 2 is：

Make in module in threedimensional model, according to the front view for making module described in step one as threedimensional model wait make Make figure for shadow-play personage's general image, newly-built areal model, the parameter for changing the newly-built areal model obtain first planting part P1 successively and put down Surface model, upper body P2 areal models, right large arm P3 areal model, right forearm P4 areal models, right hand P5 areal models, left large arm P6 areal models, left forearm P7 areal models, left hand P8 areal models, upper right lower limb P9 areal models, bottom right lower limb P10 plane moulds Type, right crus of diaphragm P11 areal models, upper left leg P12 areal model, left lower leg P13 areal models and left foot P14 areal models；

The modification of the parameter for changing the newly-built areal model is according to being：By figure for shadow-play personage to be produced disassemble for the first planting, two Upper arm, two forearms, two handss, a body, two thighs, two shanks and two feet totally 14 body parts, respectively The positive and negative photograph for obtaining 14 body parts is shot, using image processing tool respectively to the positive and negative of 14 body parts Face photograph carries out background and removes, then save as respectively png formatted files as corresponding newly-built areal model parameter modification according to According to；

The concrete grammar of the pinup picture of surface color is carried out in step 3 to figure for shadow-play personage to be produced whole body part areal model For：

Make in module in threedimensional model, it then follows from left to right, order from top to bottom is entered to all body part areal models Row pinup picture：First planting part P1 areal models are selected first, are created the Planar Mapping of its Z-direction, are then used and first planting part The png formatted files of P1 areal models correspondence body part carry out the pinup picture of surface color to first planting part P1 areal models；So Afterwards in this way, successively 14 body part areal models are carried out with the pinup picture of surface color；

In step 4, the concrete grammar of formation figure for shadow-play personage's threedimensional model is：

When first planting part P1 is towards right side：

Mobile Z-direction, positive direction represent reach, and negative direction represents retrogressing；14 body parts are put down by following hierarchical sequence Surface model is sewed and is connect：Upper body P2 areal models cover first planting part P1 areal models；Right hand P5 areal models cover right forearm P4 and put down Surface model, right forearm P4 areal models cover right large arm P3 areal model, and right large arm P3 areal model covers upper body P2 plane moulds Type；Right crus of diaphragm P11 areal models cover bottom right lower limb P10 areal models, and bottom right lower limb P10 areal models cover upper right lower limb P9 plane moulds Type, upper right lower limb P9 areal models cover upper body P2 areal models；Upper body P2 areal models cover left large arm P6 areal model, Zuo great Arm P6 areal models cover left forearm P7 areal models, and left forearm P7 areal models cover left hand P8 areal models；Upper body P2 puts down Surface model covers upper left leg P12 areal model, and upper left leg P12 areal model covers left lower leg P13 areal models, left lower leg P13 Areal model covers left foot P14 areal models；Form figure for shadow-play personage's threedimensional model；

In step 5 by the concrete grammar that figure for shadow-play personage threedimensional model sets up binding with all skeletons it is：

The all skeletons for setting up figure for shadow-play personage's threedimensional model are respectively 15 articulares：Joint of head S1, both shoulders central authorities S2, right shoulder Joint S3, right elbow joint S4, right carpal joint S5, left shoulder joint S6, left elbow joint S7, left carpal joint S8, buttocks central authorities joint S9, RG joint S10, right knee joint S11, right ankle joint S12, left buttocks joint S13, left knee joint S14 and left ankle joint S15； Again by the eyelid covering of skeleton, all skeletons of figure for shadow-play personage's threedimensional model are bound with figure for shadow-play personage three-dimension modeling；

Characterized in that, the three-dimensional in step 6 by all skeletons of figure for shadow-play personage's threedimensional model with corresponding human body skeletal joint is sat Mark information sets up the concrete grammar of mapping relations：

The skeleton motion of performing artist is caught using depth transducer, skeletal joint point three-dimensional when performing artist moves is obtained and is sat Mark information；The music for voices in a Chinese opera of performing artist and the opera of band performance are obtained with the microphone array at depth transducer two ends simultaneously；Preserve The performing artist of acquisition skeleton motion sequence and corresponding audio signal during whole performance；

By 20 skeletal joint point three-dimensional coordinate informations of performing artist are extracted in each frame of skeleton motion sequence, by 20 skeletons 15 of 15 corresponding skeletal joint point three-dimensional coordinate informations and figure for shadow-play personage's threedimensional model in articulare three-dimensional coordinate information It is as follows that articulare sets up mapping relations：Joint of head point three-dimensional coordinate information in head skeleton position module is assigned to into head pass Section S1；Shoulder joints point three-dimensional coordinate information in shoulder frame position module is assigned to into both shoulders central authorities S2；By right shoulder skeleton position In putting module, right shoulder joint node three-dimensional coordinate information is assigned to right shoulder joint S3；By right elbow joint point in right elbow frame position module Three-dimensional coordinate information is assigned to right elbow joint S4；Right carpal joint point three-dimensional coordinate information in right carpal bone rack position module is assigned to Right carpal joint S5；Left shoulder joint node three-dimensional coordinate information in left shoulder frame position module is assigned to into left shoulder joint S6；By left elbow In frame position module, left elbow joint point three-dimensional coordinate information is assigned to left elbow joint S7；By left wrist in left carpal bone rack position module Articulare three-dimensional coordinate information is assigned to left carpal joint S8；Will be buttocks central authorities articulare in buttocks central authorities frame position module three-dimensional Coordinate information is assigned to buttocks central authorities joint S9；RG articulare three-dimensional coordinate information in RG frame position module is assigned It is worth and gives RG joint S10；Right knee joint point three-dimensional coordinate information in right kneecap rack position module is assigned to into right knee joint S11；Right ankle joint point three-dimensional coordinate information in right anklebone rack position module is assigned to into right ankle joint S12；By left buttocks skeleton In position module, left buttocks articulare three-dimensional coordinate information is assigned to left buttocks joint S13；Will be left in left kneecap rack position module Knee joint point three-dimensional coordinate information is assigned to left knee joint S14；By left ankle joint point three-dimensional coordinate in left anklebone rack position module Information is assigned to left ankle joint S15；

Using the concrete grammar of all skeleton motions of catcher's body skeleton motion sequence control figure for shadow-play personage's threedimensional model it is：

Using the adjacent two frames x-axis direction relative distance control figure for shadow-play people in skeleton motion sequence with both shoulders central authorities S2 correspondence images The figure for shadow-play character facial direction of thing threedimensional model：Standard is extended to the left with x-axis positive axis, when in adjacent two frames skeleton sequence with The difference of the x coordinate of both shoulders central authorities S2 correspondence images be timing, the first planting part P1 areal models controlled by joint of head S1 to Left upset, when the difference in adjacent two frames skeleton sequence with the x coordinate of both shoulders central authorities S2 correspondence images is negative, by joint of head S1 The first planting part P1 areal models for being controlled are overturn to the right；Remaining respective image action control in connected applications skeleton motion sequence Remaining all skeleton motion of figure for shadow-play personage's threedimensional model, resynchronisation play the performing artist's music for voices in a Chinese opera or accompaniment melody recorded, and realize Three-dimensional shadow show makes.