CN109416679B - Multiple electronic controls and tracking for mixed reality interactions - Google Patents

Abstract

A system for using a plurality of electronic control and tracking devices for mixed reality interaction, comprising an electronic control device that communicates with the plurality of electronic devices via a network, receives user interactions and communicates them to the devices via the network, and receives user interactions via the network from other electronic devices. A device receives and communicates interactions to a host device, and a method for nested communication between a plurality of electronic control devices and electronic systems.

Description

Multiple electronic controls and tracking for mixed reality interactions

Cross-references associated with the application

This application is a PCT application entitled "MULTIPLE ELECTRONIC CONTROL ANDTRACKING DEVICES FOR MIXED-REALITY INTERACTION" filed on June 21, 2016 and claims priority, and is hereby incorporated by reference in its entirety The entire description thereof is incorporated herein.

technical field

The present disclosure relates to the field of electronic devices, and more particularly to the field of input and output methods for interacting within software applications.

Background technique

In the field of electronic devices, discrete handheld controllers are often used to enable a user to provide interaction or receive feedback from a host device such as a personal computer or video game console. These controllers may be connected via wired or wireless connections, and are typically only paired with a single host device at any given time. When a user wishes to utilize multiple controllers, they must be separately connected to the host device (eg, using two separate ports on the computer to connect both the keyboard and mouse to the personal computer). This requires a number of separate communication connections between the host device and the controller, and if the host device or controller have mismatched communication hardware, they are incompatible and cannot be used. Additionally, controllers are often designed for a specific purpose, such as for a specific type of video game or computer application, and require the user to interact with them in a specific way in order to hold the controller in a specific way to have manual access to all of its functions. This can be awkward or even unhealthy for the user and limits the way they can interact with the device.

Additionally, users typically interact with the fitness device, whether viewing a static screen or not. For example, a user may choose to watch a static screen showing a television show while running on a treadmill. Elements in television programs are static in that the elements do not change behavior based on user interaction with the television program, but instead perform predetermined actions. An example of a dynamic screen is a video game. A user interacts with a remote device and affects the activity of elements in a video game. At most, the user may interact with the screen performing an activity unrelated to the action associated with the fitness device interaction.

Further, with the rapid expansion of the virtual reality industry, new interaction methods have been explored, including various controllers for games, canes, and motion-based input devices including gloves and camera-based hand tracking. However, these devices focus on the interaction with the user's hand and ignore other parts of the body that can be used to improve interaction and immersion while also expanding data collection possibilities.

What is needed is a means to allow users to connect various control and tracking devices to a host device independent of their communication means, and also addresses the need for a controller that users can interact with regardless of their position or movement for Interact with the Mixed Reality Electron app.

Contents of the invention

Accordingly, the present inventors have conceived and implemented in a preferred embodiment of the present invention multiple electronic control and tracking devices that can pair with multiple host devices and that can send used nested communications for mixed reality interaction.

In accordance with a preferred embodiment of the present invention, a system for using a plurality of electronic control and tracking devices for mixed reality interaction is disclosed, comprising an electronic control device including an electronic control device stored in memory and running on a processor of a networked computing device at least a plurality of programming instructions and configured to communicate with a plurality of electronic devices via a network and configured to operate a plurality of hardware control elements configured to receive manual interactions from a user and configured to transmit a plurality of interactions data to at least a portion of the plurality of electronic devices, the interaction data being based at least in part on at least a portion of a manual interaction from a human user, and configured to receive communication from at least a portion of the plurality of electronic devices, the communication comprising at least mixed reality data, wherein The mixed reality data includes at least a plurality of sensor data readings and is configured to pass at least a portion of the traffic to at least a portion of the plurality of electronic devices.

According to another preferred embodiment of the present invention, a method for nested communication between a plurality of electronic control devices and electronic systems is disclosed, comprising the steps of including at least a plurality of programming instructions on a processor of the device and configured to communicate with a plurality of electronic devices via a network and configured to operate a first electronic control device of a plurality of hardware control elements configured to receive a plurality of manual commands from a human user interacting and configured to send a plurality of interaction information to at least a portion of the plurality of electronic devices, the interaction information being based at least in part on at least a portion of a plurality of manual interactions from a human user, and configured to receive communications from at least a portion of the plurality of electronic devices traffic and at least a portion of the traffic to at least a portion of a plurality of communication devices connected to the host electronic device via a network; connected to the first electronic control device using an additional electronic control device, the additional electronic control device comprising at least one stored in a memory and a plurality of programming instructions running on a processor of a networked computing device and configured to communicate with a plurality of electronic devices via a network and configured to operate a plurality of hardware control elements configured to receive data from a human user human interaction, and configured to send a plurality of interaction information to at least a portion of the plurality of electronic devices, the interaction information is based at least in part on at least a portion of the plurality of human interactions from a human user, and is configured for transmission from the plurality of electronic devices receiving at least a portion of the traffic and passing at least a portion of the traffic to at least a portion of the plurality of communication devices; receiving a plurality of user interactions at an additional electronic control device; sending a plurality of interaction messages to the first electronic control device, the interaction messages being at least in part based on at least a portion of the user interaction; and communicating at the first electronic control device at least a portion of the interaction information to the host electronic device.

Description of drawings

The drawings illustrate several embodiments of the invention and together with the description serve to explain principles of the invention according to the embodiments. It should be appreciated by those skilled in the art that the specific embodiments shown in the drawings are exemplary only and should not be considered in any way to limit the invention or the scope of the claims herein.

1 is a diagram of an exemplary hardware arrangement of a device for natural torso tracking and feedback for electronic interaction, illustrating the use of multiple tethers and a movable torso harness, according to a preferred embodiment of the present invention.

Figure 2 is a diagram illustrating various alternative tether arrangements.

3 is a diagram of an additional exemplary hardware arrangement of an apparatus for natural tether tracking and feedback for electronic interaction, illustrating the use of an angle sensor to detect angular movement of the tether, according to a preferred embodiment of the present invention.

Figure 4A is a diagram illustrating an exemplary physical arrangement of paired, nested communicating wireless control devices in accordance with a preferred embodiment of the present invention.

4B is a diagram illustrating an exemplary physical arrangement of paired nested communicating wireless control devices communicating via an adapter in accordance with a preferred embodiment of the present invention.

5 is a diagram illustrating various exemplary physical configurations of nested communication wireless controllers configured for various purposes.

FIG. 6 is a flowchart illustrating an exemplary method for nested communication between a control device and an electronic system according to a preferred embodiment of the present invention.

FIG. 7 is a block diagram illustrating an exemplary system architecture using a nested communication control device with various electronic systems according to a preferred embodiment of the present invention.

8 is a block diagram illustrating components of an adaptable fitness system according to an embodiment of the present invention.

FIG. 9A is a block diagram illustrating components of a motion sensing device according to an embodiment of the present invention.

Figure 9B illustrates a motion sensing device according to an embodiment of the invention.

Figure 9C illustrates a handheld controller in accordance with an embodiment of the present invention.

9D illustrates an example of a handheld controller mounted to a handle of an exercise device in accordance with an embodiment of the present invention.

10 is a flowchart illustrating an exemplary method for interacting with a fitness application based on a user's interaction with the fitness system, according to an embodiment of the invention.

Figure 11 is a block diagram illustrating an exemplary hardware architecture for a computing device in an embodiment of the present invention.

Figure 12 is a block diagram illustrating an exemplary logical architecture for a client device according to an embodiment of the present invention.

Fig. 13 is a block diagram illustrating an exemplary architectural arrangement of a client, a server, and an external service according to an embodiment of the present invention.

Figure 14 is another block diagram illustrating an exemplary hardware architecture for a computing device in various embodiments of the invention.

Detailed ways

The inventors have conceived and implemented in a preferred embodiment of the present invention multiple electronic control and tracking devices for mixed reality interaction that can pair with multiple host devices and can send used nested communications.

One or more different inventions may be described in this application. Further, several alternative embodiments may be described for one or more inventions described herein; it should be understood that these are shown for illustrative purposes only and are not intended to limit in any way the inventions contained herein or described herein. Display claims. One or more inventions may be applied broadly to several embodiments, as is apparent from this disclosure. Generally, the embodiments are described in sufficient detail to enable those skilled in the art to practice one or more of the inventions, and it is to be understood that other embodiments may be utilized and structural, logical, software, electrical, and other changes may be made without departing from the specific the scope of the invention. Accordingly, those skilled in the art will recognize that one or more inventions may be practiced with various modifications and variations. Certain features of one or more inventions described herein may be described with reference to certain embodiments or drawings which form a part of this disclosure and in which are shown by way of schematic illustration specific embodiments of one or more inventions. It should be understood, however, that these features are not limited to use in the particular embodiment(s) or figures with which they are described. This disclosure is neither a literal description of all embodiments of one or more inventions nor a listing of features of one or more inventions that must be present in all embodiments.

Section headings, as well as the headings of this patent application, are provided in this patent application for convenience only and should not be construed as limiting the disclosure in any way.

Devices that are in communication with each other need not be in continuous communication with each other, unless expressly specified otherwise. In addition, devices that are in communication with each other may communicate directly or indirectly through one or more logical or physical communicating devices or intermediaries.

A description of an embodiment intercommunicating with several components does not imply that all such components are required. On the contrary, various optional components may be described to illustrate a wide variety of possible embodiments of one or more inventions and to more fully describe one or more characteristic aspects of the invention. Similarly, although process steps, method steps, algorithms, etc. may be described in a sequential order, these processes, methods and algorithms can generally be configured to work in an alternate order unless specifically stated to the contrary. In other words, any order or order in which the steps may be described in this patent application does not, by itself, indicate that the steps need to be performed in that order. The steps of the described processes may be performed in virtually any order. Further, some steps may be performed concurrently, although described or implied as not occurring concurrently (for example because one step is described after another). Furthermore, the illustration of a process in the drawings by its description does not imply that the illustrated process excludes other variations and modifications, that the illustrated process or any step thereof is essential to one or more inventions, and does not It is implied that the process shown is preferred. Furthermore, steps are generally described once per embodiment, but this does not mean that they must occur once, or that they may occur only once per execution or operation of a process, method, or algorithm. Some steps may be omitted in some embodiments or situations, or some steps may be performed more than once in a given embodiment or situation.

While a single device or item is described herein, it will be apparent that more than one device or item may be used instead of a single device or item. Similarly, when more than one device or article is described herein, it will be apparent that a single device or article may be used instead of more than one device or article.

A function or feature of a device may alternatively be embodied by one or more other devices not explicitly described as having that function or feature. Accordingly, other embodiments of one or more inventions need not themselves include devices.

Technologies and mechanisms described or referenced herein will sometimes be described in the singular for clarity. However, it should be appreciated that particular embodiments may include multiple iterations of techniques or multiple instantiations of mechanisms unless otherwise noted. Process descriptions or blocks in the drawings should be understood to represent modules, segments or code portions that include one or more executable instructions for implementing specific logical functions or steps in the process. Alternative implementations are included within the scope of embodiments of the invention in which, for example, functions may be performed in an order other than that shown or described, including substantially simultaneously or in the reverse order, depending on the functions involved, as understood by those skilled in the art. understood by technicians.

conceptual framework

Figure 7 is a block diagram illustrating an exemplary system architecture 700 using nested communication control devices 701, 702a-n with various electronic systems 710 in accordance with a preferred embodiment of the present invention. According to an embodiment, the control device 701 may be connected to a plurality of electronic systems 710 such as including (but not limited to) a personal computer 711, a video game console 712, a media center (such as a home theater system) 713, or a mobile device 714 (such as a smartphone or tablet computing devices). Connection can occur via various means, but according to an embodiment and as contemplated by the inventors, the ideal method of connection is via a wireless connection mechanism using radio. Exemplary communication protocols or technologies for this connection may include, but are not limited to, cellular radio communications, BLUETOOTH ^™ , ANT ^™ , WiFi, Near Field Communication (NFC), or other connection mechanisms.

Depending on the embodiment, multiple additional control devices 702a-n may be paired with the main control device 701 via various connection mechanisms. When connected in this manner, input from users interacting via the additional control devices 702 a - n can be sent to the master control device 701 , which can then relay the interactions to a plurality of connected electronic systems 710 . Feedback may then be received by the main control device 701, if provided by the electronic system 710, and passed on to the additional control devices 102a-n if desired (e.g., if the user interacts via multiple additional control devices 102a and the electronic system 110 provides Feedback about the user or the particular design of the device). It should be appreciated that not all control devices need utilize similar connection mechanisms. For example, if the main control device 101 runs BLUETOOTH ^™ and a WiFi radio for wireless communication, the additional control device 102a can connect via BLUETOOTH ^™ and the second additional control device 102b can use a WiFi connection. In this manner, the master control device 101 may be viewed as a means for combining various connection devices to facilitate communication between the plurality of electronic systems 110 and the plurality of additional control devices 102a-n of varying design and operation.

Detailed Description of Exemplary Embodiments

1 is a diagram of an exemplary hardware arrangement 100 for natural torso tracking and feedback for electronic interaction, illustrating the use of multiple tethers 110a-n and a movable torso harness 120, in accordance with a preferred embodiment of the present invention. . According to an embodiment, the plurality of tethers 110a-n may be fixed or integrally formed as part of a handle or rail 130, such as the handles found on exercise equipment such as treadmills, ellipticals, stair climbers, and the like. In alternative arrangements, specially designed equipment with integral tethers 110a-n may be used, but it will be appreciated that a modular design with fixed and freely removable tethers 110a-n may be desirable for Facilitate structural elements for use in various fitness equipment or buildings, according to the particular use or situation of the user. The tethers 110a-n can then be fitted or integrally formed to the torso harness 120, as shown in the form of a strap, which can be worn by the user so that movement of their body affects the tethers 110a-n and acts upon them in various ways. Apply stress. It should be appreciated that although a strap design for the torso harness 120 is shown for clarity, various physical arrangements may be used, such as including (but not limited to) a vest, a series of harness-like straps similar to climbing or draping equipment , a backpack, a strap designed to be worn under the user's body or in place of clothing (such as in a medical setting for collecting precise data), or multiple specially formed clips or attachment points that can be easily attached to the user's clothing. Additionally, the torso harness 120 may be constructed with movable components, such as having an inner strap 121 that allows a user a degree of movement within the harness 120 without restricting their movement. Movement of the inner strap 121 (or other movable portion) can be measured in various ways, such as using accelerometers, gyroscopes, or optical sensors, and this data can be used as an interaction with the software application, except from the tether as described below. 110a-n in addition to collected data.

As users move, their bodies naturally shift in position and orientation. These deflections can be detected and measured via the tethers 110a-n, for example by measuring patterns of tension or strain on the tethers 110a-n to indicate body orientation, or by measuring small changes in strain on the tethers 110a-n to determine more precisely Movements such as the user's body posture when speaking, or the user's stride or specific characteristics of gait. Additionally, by varying the number and arrangement of tethers 110a-n, more precise or specific patterns of movement can be detected and measured (such as, for example, using a special arrangement of multiple tethers attached to specific areas of the user's body to detect very fine movements for medical diagnosis or fitness coaching). This data can be used in interactions with software applications, such as for virtual reality applications as user input to control in-game characters. In this arrangement, as the user moves, that movement can be translated to the in-game character or avatar to convey a more natural sense of interaction and presence. For example, in a multiplayer role-playing game, this can be used to facilitate non-verbal communication and recognition between players, as their distinct styles and postures can be communicated during the game by detecting natural torso positions and movements. In fitness or health applications, this data can be used to track and monitor the user's posture or ergonomics, or to aid in specific fitness activities such as holding yoga poses, stretching, or proper running form during use on a treadmill. train them. In medical applications, this data can be used to help diagnose injuries or deficiencies that may require attention, such as by detecting abnormalities in movement or physiological adaptations to unrecognized injuries (such as when a user subconsciously shifts their weight away from an injured foot or knee). , without consciously recognizing a problem).

Various arrangements of tethers 110a-n and tether sensors (as described below with reference to FIGS. 2-3 ) may enable users to interact with, and receive haptic feedback from, software applications in various immersive ways. For example, by detecting the rotation, tension, stress, or angle of the tether, by using natural body movements and positioning such as leaning, jumping, squatting, kneeling, turning, or offsetting their center in various directions to trigger when configured for An action within a software application that accepts torso-tracking input, allowing users to interact with applications such as virtual reality games or simulations. By applying haptic feedback of varying forms and intensities (as described in more detail below, with reference to FIG. Directional movement or "jerking" of the user, or changing the response to multiple tethers such as pulling and releasing in varying order or sequence to simulate more complex effects such as (e.g. in the case of game use) explosions, rides in vehicles, or walking through the leaves.

It should be appreciated that although reference is made to virtual reality applications, a wide variety of use cases may be possible in accordance with embodiments. For example, torso tracking can be used in fitness and health applications to monitor a user's posture or gait while walking without using additional virtual reality equipment or software.

Figure 2 is a diagram illustrating various alternative tether arrangements. Depending on the respective use case and hardware arrangement, the tethers 110a-n may utilize various purpose-driven designs as shown. For example, a "stretchable" tether 210 may be used to measure strain during movement of the user, when the tether 210 is stretched or compressed (eg, using a piezoelectric material and measuring the electrical change). This arrangement may be suitable for precise measurements, but may lack mechanical strength or durability for total movement detection or continued use. Alternative configurations may utilize a non-deformable tether 220 such as a steel cable or similar non-stretch material. Instead of measuring the strain on the tether 220, some degree of movement of the alternative tether 220 within the housing 222 (e.g., an attachment point on the torso harness 120 or handle 130) may be allowed and the strain on the tether 220 may be measured, such as via an optical sensor. Position or Move 221. In a third exemplary arrangement, the tether 230 can be coiled around an axle or pulley 231 and can be unwound when force is applied during movement by the user. The rotation of the pulley 231 can be measured, or alternatively a tension device such as a coil spring (not shown) can be utilized and the tension or strain on the device measured when the tether 230 is extended or retracted. In this manner, it will be appreciated that various mechanisms may be used to facilitate the use of tethers and accessories for detecting and measuring natural torso position and movement, and that various additional or alternative mechanisms may be utilized in accordance with the embodiments disclosed herein. Hardware layout.

Additionally, by using various hardware configurations, it becomes possible to utilize "passive" tethers that only measure movement or strain, as well as "active" tethers that can apply resistance or movement to provide tactile feedback to the user. For example, in arrangements utilizing a coiled spring or roller 231, the spring or roller 231 may coil to retract the tether and guide or resist movement of the user if desired. In this way, various new forms of feedback-based interactions are made possible, and in the case of virtual reality use, user engagement and immersion are improved through more natural physical feedback during their interactions.

By applying various forms and strengths of feedback using various tether arrangements, various feedback types can be used to provide tactile output to the user in response to software events. For example, tension on a tether can be used to simulate constrained movement such as wading through water or dense foliage, walking up a sloped surface, magnetism or gravity, or other forms of physical resistance or resistance that can be simulated by directional or non-directional tension . Pulling, retracting or pulling on a tether can be used to simulate sudden forces such as recoil forces from gunfire, explosions, grasping or knocking by software entities such as objects or characters, deploying parachutes, bungee jumping, sliding or falling , or other momentary force or event delivered by a tugging or tugging sensation. By utilizing various modes of haptic feedback, more complex event communications can be sent to the user, such as riding on horseback or in a vehicle, standing on the deck of a ship in the sea, turbulence in an airplane, weather, or haptic feedback can be used Additional virtual events on display. In this way, virtual environments and events can be made more immersive and tangible to the user, by enabling the user to interact using natural body movements and positioning, and by providing haptic feedback in a manner that feels natural and expected for the user. For example, if a user is controlling a character in a game application through a first-person viewpoint, it can look natural when their character is hit, and there will be a physical sensation corresponding to the event; however, this is not possible with conventional interactive devices, detracting from any sense of immersion or reality for the user. By providing this physical sensation along with the virtual event, the experience becomes more engaging and encourages users to interact more naturally as their actions result in natural and believable feedback, satisfying their knee-jerk expectations and avoiding excessive "immersion breaking" moments , which in turn reduces the user's adoption of unusual behavior or unhealthy postures as a result of adapting to a limited interaction schedule.

Haptic feedback can be provided to notify users of non-gaming events, such as for desktop notifications for email or application updates, or to provide feedback on their gestures for fitness or health coaching. For example, users may be encouraged to maintain a particular stance, posture, or posture while exercising or for a set length of time (e.g., for a yoga training application), and provide feedback to remind them to adjust their posture if their posture deviates from an acceptable range. attitude. This can be used for sports, fitness, health or ergonomics applications that do not need to take advantage of other characteristic aspects of virtual reality and can operate as traditional software applications on non-specialized computing hardware. For example, a user at their desk can use an ergonomic training application to monitor their body posture throughout the workday and provide haptic cues to correct poor posture when it is detected, helping the user maintain a healthy working posture to reduce stress caused by poor posture. Posture-induced fatigue or injury (eg, repetitive stress injuries that can be associated with poor posture when working at a computer).

3 is a diagram of an additional exemplary hardware arrangement 300 for natural torso tracking and feedback for electronic interaction, illustrating the use of angle sensors 312, 321a-n to detect the angle of a tether 320, in accordance with a preferred embodiment of the present invention. move. According to one exemplary arrangement, the tether 310 may be fixed to or pass through a swivel joint such as a ball bearing 311 or the like to allow free angular movement. During movement, the angular movement or deflection 312 of the protruding rod, rod, or tether segment 313 may be measured (eg, using optical, magnetic, or other sensors) to determine the corresponding angle of the tether 310 . In this way, precise angular measurements can be collected without impeding range of motion or introducing unnecessary mechanical complexity.

In an alternative hardware arrangement, the use of angle sensors 321a-n enables tracking of the vertical angle of tether 320 to detect and optionally measure vertical movement or orientation of the user's torso. When the tether 320 contacts the sensors 321a-n, it can be registered and used to detect substantially vertical movement (ie, whether the tether is angled up or down). For more precise measurements, the specific hardware configuration of the sensors 321a-n can be varied, such as using a pressure sensing switch to detect how much force is being applied and use that measurement to determine the corresponding angle (as given a known configuration of the tether 320 can be possible). It should be appreciated that various combinations of hardware may be used to provide the desired method or degree of angle detection or measurement, such as using a conductive tether 320 and capacitive sensors 321a-n to detect contact, or using a mechanical or rubber dome switch (as typically used in keyboard configurations) to detect physical contact without employing the conductive tether 320.

Using angle detection or measurements can expand the interaction possibilities to include more detailed and natural movements of the user's body. For example, if the user crouches, all tethers 110a-n can simultaneously detect the downward angle. Additionally, by combining inputs from multiple applicable sensors when possible (e.g., utilizing adaptive software to collect data from any sensor it detects, no particular sensor type is required for operation), such as by combining inputs from tether 110a -n, as well as data from hardware sensors such as accelerometers or gyroscopes, can enhance data accuracy or applicability, enabling multiple methods of similar or varying types or levels of precision of position or movement detection. Similarly, when the user jumps then all tethers can simultaneously detect the upward angle. However, if the user tilts in one direction, it can be known that not all tethers 110a-n will detect the same angle. For example, tethers 110a-n in the direction the user is leaning may detect a downward angle, while those on the opposite side will detect an upward angle (due to the orientation of the user's torso and thus the worn torso harness 120). In this way, more precise torso interaction can be facilitated by improved detection and recognition of orientation and movement. Additionally, it will be appreciated that sensors 321a-n may be used for other angle measurements, such as to detect horizontal angles. For example, if a user is wearing a non-rotating torso harness 120, similar stress may be applied to all attached tethers 110a-n when they twist their body. Without angle detection, the precise nature of this movement would be unclear, but with horizontal angle detection, it becomes possible to identify that all tethers 110a-n are straining in a similar direction (e.g. in a clockwise pattern when viewed from above because The user may watch the tethers 110a-n) during use, and thus interpret the interaction as a twisting motion (rather than, for example, the user crouching or kneeling, which would apply similar stress to the tethers 110a-n but would have a different angle measurement).

Figure 4A is a diagram illustrating an exemplary physical arrangement of paired, nested communicating wireless control devices 400a-b in accordance with a preferred embodiment of the present invention. According to an embodiment, the plurality of wireless control devices 400a-b may communicate via a wireless communication protocol such as including (but not limited to) BLUETOOTH ^™ , WiFi, NFC, ZIGBEE ^™ , cellular radio frequency (eg, via an existing GSM or other cellular network, or via Using cellular frequencies to create an ad hoc or "mesh" network between the devices if desired) or other suitable wireless communication protocols or techniques to communicate with multiple electronic host devices 410 (such as, but not limited to, game console devices, personal A computer, a mobile device such as a smart phone or tablet computing device, a media device, or other electronic device that can receive user interaction via a physically separate controller or controllers) is "paired." According to an embodiment, two physically separate controllers 400a-b may be paired with the host 410 such that the user holds one controller in each hand for ease of manipulation and ergonomic comfort. For example, a user may hold a controller at a comfortable angle and distance when seated, making possible a greater degree of comfort than manipulating a single controller with two hands. For example, a user may grab one controller in each hand during workouts or full-body interactive games, such as when running on a treadmill or interacting with a virtual fitness trainer such as when using a NINTENDO WII ^™ game console. It should also be appreciated that the control devices 400a-b can optionally be paired with more than one host device 410, such as using a single controller to interact with multiple computers or mobile devices, such as for giving demonstrations or guiding instructions, where the teacher can Interact directly with students' devices.

Figure 4B is a diagram illustrating an exemplary physical arrangement of paired, nested communicating wireless control devices 400a-b, illustrating communication via adapter 420 in accordance with a preferred embodiment of the present invention. According to an embodiment, a communication adapter 420 may be employed to facilitate communication between multiple controllers 400a-b and multiple host devices 410, for example to connect multiple physical controllers and present them to the host as a single "virtual controller". ” (as may be required depending on the particular software application being interacted with, such as a video game requiring a single controller design) or between the host and may otherwise be unable to communicate (for example, if one device only relies on BLUETOOTH ^TM for connectivity and the other Bridge communication between a controller using WiFI, or if two different communication protocols are being used. Additionally, the communication adapter may be operated by the control device, such as in an arrangement where multiple additional or "secondary" controllers communicate with a controller labeled "master", which in turn communicates with the host device, such as in A "nested communication" arrangement, in which the master controller acts as a communication adapter between the other controllers and the host, works by receiving traffic from each additional controller and submitting all traffic to the host device (as described below, with reference to Figure 5 -Figure 6). For example, the master controller may operate the communication adapter 420 to bridge communications between fitness trackers or wearable devices (eg, such as FITBIT ^™ , APPLE WATCH ^™ , or MICROSOFT BAND ^™ devices) to multiple host devices 410 to facilitate linking fitness Or health information (as typically tracked by wearable devices) integrated with video games or other software applications. This arrangement can also be used to bridge data between devices from different manufacturers or running different software or operating systems, for example to integrate health data from an APPLE WATCH ^™ device with a MICROSOFT XBOX ^™ gaming console, or from a FITBIT ^™ fitness tracker The player's fitness information is integrated with the SONY PLAYSTATION ^™ game console, or other combinations of hardware, software and manufacturers.

For example, specific wireless protocols or hardware technologies may be employed, or hardware devices may be used to configure the control devices 400a-b to interact with home game consoles (eg, MICROSOFT XBOX ^™ or SONY PLAYSTATION ^™ ), depending on a purpose specifically designed for use, such as for interacting with a home gaming console (e.g., MICROSOFT XBOX™ or SONY PLAYSTATION™). A particular application utilizes many communication protocols, and the user may be given the option to select the protocol or frequency desired to be used. For example, a control device 400a-b configured for a desktop computer may be configured to communicate using BLUETOOTH ^™ radios in the 2.4GHz frequency range. A hardware switch or button 401b may optionally be provided whereby the user can select a different frequency if desired, e.g. a radio frequency in the 5GHz range for use over a WiFi network instead of BLUETOOTH ^™ , e.g. to increase range or speed or to avoid interference from other devices (for example, many microwave ovens used in homes or offices can cause interference in the 2.4GHz band). The use of various wireless communication devices allows users to hold multiple control devices 400a-b in any position or manner they choose, improving ergonomics and reducing discomfort or use Conventional controls that grip them in a particular way can be at risk of repetitive stress injuries (eg, a computer keyboard that often requires the user's hands and arms to be in awkward alignment).

By utilizing wireless connections according to embodiments, increased movement or positioning of multiple control devices can be used as an additional method of interaction with the user, for example using gesture-based input methods or position via internal hardware sensors such as accelerometers or gyroscopes track. In this way, additional forms of input may be enabled in accordance with user preferences, increasing the number or quality of possible interactions with a particular control and optionally reducing the need for special hardware buttons. For example, a gesture-enabled control device can be programmed with a specific motion gesture that simulates the press of a "volume up" button, eliminating the need for a hardware button to perform this function.

Depending on the embodiment, multiple hardware buttons may be provided on the control device 400a-b, or the physical design or arrangement may vary. For example, a diaphragm or mechanical switch based button could be utilized to provide a tactile sensation once actuated by the user (as is common in designing dedicated hardware keyboards such as in gaming or professional designs), or capacitive touch based buttons could be utilized To provide interaction in a more compact physical arrangement when a tactile sensation may not be required (as is common in mobile applications, such as screen lighting controls on smartphones or tablet computing devices).

Hardware buttons or other controls may be placed in various arrangements depending on the particular use or desired function, for example a digital or analog "trigger" type button 404 may be placed on the lower side 400c of the control (shown in side cutaway), For example for gaming applications where a trigger actuation function may be desired. Another exemplary arrangement may be to arrange a number of buttons 402a-n for navigation in a specific way so as to form a directional pad (D-pad) for various games or applications, such as for navigating a media player interface or to Play videos from a top-down perspective, such as a strategy or simulated board game. Other buttons may include large easy-reach buttons 401a for games or applications when the user may need to quickly select and not look at the actuation buttons. These buttons are provided with various identifying features such as readable text labels, colors, or physical shapes including, but not limited to, square 401a, circle 401b, triangle 403a, or star 403b control buttons, for example.

Depending on the embodiment, a specific exemplary arrangement may be a pair of controllers that effectively act as a single "split controller," where the user grabs one device with one hand and divides functions between the two devices to facilitate two-handed interaction without Use both hands on a single device. Controls can be arranged in a combination, with directional buttons positioned around the top portion of each device, and function buttons positioned at the center of the directional buttons to provide easy-to-reach buttons for interaction (e.g. to "accept" or "accept" on an on-screen menu item). "click"). Additionally, these combined control arrangements can then be coupled with multiple internal hardware sensors, such as accelerometers, gyroscopes, Hall effect sensors, or other hardware devices that facilitate physical interaction through the capture and interpretation of sensor data, for example to Interaction is enabled through the use of predetermined or customizable gesture or position tracking.

Various alternative or additional control device arrangements may be utilized in various combinations through nested communication protocols. For example, depending on their intended use (such as for a particular game or application, or for a particular device or console), a user may select their preferred "primary" control device and then pair a "secondary" control device with it. In this way it can be appreciated that the specific number, style or arrangement of buttons or other interactive features or physical designs such as ergonomic shapes or layouts can vary widely and users can choose what appeals to them based on their particular needs or uses. Specific arrangements and combinations of forces. Various control device arrangements are described below with reference to Figure 5, and it should be appreciated that any particular control device may be utilized interchangeably as a primary or secondary control, depending on user preference. Additionally, additional hardware devices can be paired with the control device for nested communication by using various wireless communication technologies as previously described. For example, fitness trackers (such as including, but not limited to, FITBIT ^™ , MICROSOFT, BAND ^™ , or APPLE WATCH ^™ devices) can be paired via a wireless personal area network protocol such as BLUETOOTH ^™ or ANT ^™ , for example, to provide a user's biometric or health Information and interaction via the control means 400a-b.

5 is a diagram illustrating various exemplary physical configurations of nested communications wireless controllers 500a-c configured for various purposes. According to an exemplary embodiment, the controller 500a may be configured optimized for use as a media playback remote control, such as for connected speakers or playback devices, or for a home theater or media PC setup. According to this arrangement, a number of buttons 501a-n may be provided for navigating the on-screen interface of the media device, for example to select audio or video content from a playback library. A number of additional buttons 502a-n may be provided at secondary locations to provide additional functionality such as play/pause, stop, or menu functions to view additional information on the selected media. Depending on the particular application, the control device 500a may serve as a primary controller, optionally with or without a secondary controller for nested communications, such as a user's fitness tracker for monitoring heartbeat or other biometrics during media viewing or a game controller is used to play interactive content presented through a media server or playback device (such as playing a game on a ROKU ^™ or similar home theater device connected to a television).

According to another exemplary arrangement, the controller 500b may be configured for a specific game use, such as being formed with a suitable physical shape and having control elements (buttons, switches, and other interactive hardware elements) arranged to best suit a simulation-type game. ). According to this configuration, the control device 500b may have a digital or analog "joystick" type control element 510 for controlling the player's character or vehicle in a 360 degree three-dimensional virtual environment, for example in third person action or adventure type games. A number of additional buttons 511a-n may be provided and positioned for easy one-handed use, to access a player's inventory, to perform special in-game actions (such as jumping, running, rolling, or other movement controls), or as needed for a particular game or application. features to perform context-sensitive actions. For example, the emulator-oriented control device 500b can be paired with the media remote control device 500a as a secondary controller, so that the user uses the media remote control device 500a as their primary control device to navigate the media interface, select games from the media content catalog, and then Use the secondary control device 500b to play games. When the game is over, the user can then continue to further navigate the media interface using their primary controller 500a without having to position the remote control or switch controllers for different uses, as the controllers can pair and communicate even when one may be temporarily out of use.

According to another exemplary arrangement, the control device may be configured specifically for use as a secondary controller, for example configured for pairing with simulator controller 500b for a two-handed "hands on throttle" commonly used in flight simulation games. Throttle controller 500c set on the controller and joystick (HOTAS). Depending on the configuration, the control device 500c may include a joystick or hat switch control element 520 for navigating with the user's thumb or for controlling many game or application functions, typically positioned so that the user's thumb naturally interacts with the user's thumb when holding the control device 500c. Control elements 520 are aligned for ease of use. The control device 500c may further include an integrally formed or removably attached ergonomic portion 521, such as a removable or positionable rigid or flexible panel, generally configured to conform to and align with the user's hand or palm. the user's hand for proper manipulation of the various control elements. For example, in one exemplary arrangement, the control device 500c may be configured with a motif made of a rigid plastic material, and with a removable plastic panel 521 shaped to fit the palm of a user for proper grip and alignment during use. This configuration can also be designed to accept a variety of alternative rigid panels 521, allowing users to swap gripped panels to configure the physical shape of the control device 500c to most comfortably fit their particular anatomy (for example, this configuration has been used in the firearms industry to allow the user to configure the pistol's grip for proper posture and aiming). The control device 500c may further include a plurality of additional control elements 522a-n, such as buttons or switches, for example arranged so that when properly aligned (such as by using the ergonomic grip panel, as previously described), the user's Fingers may naturally align in place for ideal actuation of the control elements 522a-n. In this way, it can be seen that the ergonomic portion 521 configuration can be employed in various arrangements, as well as various arrangements of control elements for optimal use when the user's hands are properly aligned and gripped by the control devices 500a-c. , so that any particular control device 500a-c can be adapted to suit a wide range of user anatomy for optimal use, as well as to promote ergonomic comfort and physical fitness. For example, by encouraging users to find the best ergonomic fit, the risk of repetitive stress injuries and other health concerns can be reduced.

FIG. 6 is a flowchart illustrating an exemplary method 600 for nested communication between a control device and an electronic system in accordance with a preferred embodiment of the present invention. In an initial step 601, the control device may be paired with an electronic system (such as, for example, a personal computer, a game console, or a home theater or media center) as the primary control device, for example as an automatic process (such as when the electronic system or controller first at power-on) or manually (such as by directing user action to initiate pairing). In an optional next step 602, a plurality of additional control devices may be paired with the main control device, for example to use a control device in each hand for a user, or to pair one or more control devices each for multiple users. The control devices are arranged for their own use in groups (eg for multiple users playing a video game together). In a next step 603, the master control device may send a number of user interactions to the paired electronic system, for example when the user presses a button or makes a selection. In an optional next step 604, any of the plurality of additional control devices may send a plurality of user inputs to the master device, and the master control device may then send user inputs to the paired electronic system. In a next step 605, the paired electronic system can send a number of feedbacks to the master control device, such as commands for tactile feedback (for example to activate a vibrating motor) or for presentation on a visual display operated by the control device Information. In an optional final step 606, the master control device may send a plurality of feedbacks to the additional control device, for example, if the feedback is sent from an electronic system intentionally designed to connect to the additional control device. In this way, it is known that all communication traffic between the control device and the electronic system passes through and is facilitated by the master control device.

FIG. 8 is a block diagram illustrating components of an applicable fitness system 800 according to one embodiment. The applicable fitness system 800 includes a hardware device 802 , a host device 810 , and a fitness server 818 .

The hardware device 802 includes a motion sensing device 804, a left handheld controller 806A, and a right handheld controller 806B. The motion sensing device 804 is mounted on the user's body and detects the user's body motions, which allows the user to interact with the fitness application 813 running on the host device 810 by performing motions. An exemplary motion sensing device 804 is described in detail with reference to FIGS. 9A and 9B . The slave controls 806A, 806B include buttons that a user can press to interact with the fitness application 813 . For ease of access, one or both of the handheld controls 806A, 806B may be mounted on the handle of a piece of training equipment. An exemplary handheld controller 806 and mounting system are described with reference to FIGS. 9C and 9D , respectively.

In alternative embodiments, additional hardware devices 802 may be used. For example, hardware device 802 may also include athletic shoes with embedded sensors or motion sensors attached directly to the fitness device. In addition, some hardware devices 802 shown in FIG. 8 may be omitted. For example, only one handheld controller may be used, or both handheld controllers 806A, 806B may be replaced by a combined handheld controller containing all of the buttons that would normally be distributed between the two handheld controllers 806A, 806B.

As a whole, the hardware device 802 can be used to fit a fitness device (e.g., stair meter, elliptical machine, treadmill, rowing machine, etc.) into a game controller that receives physical input from the user (both motion and button presses). ) and generate corresponding input data for interacting with the fitness application 813. For example, input data can be used to control virtual characters or objects in a fitness application.

In some embodiments, multiple sets of hardware devices 802 can be added to system 800 to fit additional fitness devices into additional game controllers. For example, multiple stationary bikes may be fitted into a controller for a bike racing application, thereby allowing multiple users to participate in a virtual race against each other.

Incoming data from hardware device 802 is sent to host device API 812 on host device 810 over communication links 808A, 808B, 808C (hereinafter collectively referred to as 808 ). Communication link 808 may be a wired or wireless connection. In one embodiment, communication link 808 is based on the Bluetooth Low Energy (BLE) protocol, and hardware device 802 is a BLE device. If additional sets of hardware devices 802 are included in the system, additional communication links 808 are established to connect the hardware devices 802 to the host device 810 .

In the illustrated embodiment, communication link 808 directly connects each hardware device 802 to host device 810 . In this embodiment, three hardware devices 802 are simultaneously connected to host device 810 over communication links 808A, 808B, 808C.

The hardware devices 802 may all be connected on the same communication protocol (eg, Bluetooth low energy), or on different communication protocols.

In another embodiment, only the motion sensing device 804 is directly connected to the host device 810 via the first communication link 808A. In this embodiment, two handheld controllers 806A, 806B are coupled to the motion sensing device 804 via two other communication links 808B, 808C, and the motion sensing device 804 is configured to The incoming data is relayed to the host device 810 . This embodiment is described in more detail with reference to FIG. 9A.

Host device 810 includes interface software 811 , host device API 812 , and one or more fitness applications 813 . As a whole, host device 810 is a computing device capable of executing program instructions. Host device 810 may be, for example, a smartphone, tablet computer, laptop computer, or desktop computer.

Interface software 811 and host device API 812 cooperate with the interface software to serve as an interface between hardware device 802 , fitness application 813 and fitness server 818 . When no hardware device 802 is connected, the host device API 812 initiates a scan of the hardware device 802 at regular intervals. A button press on the handheld controller 806A, 806B or motion on the motion sensing device 804 will trigger the respective device to broadcast for connection (if not already connected). If a device's broadcast overlaps with one of the host device API's scan intervals, the host device API 812 initiates a connection attempt to the respective device. This process of establishing a connection after detecting a button press or motion allows intuitive connection of the hardware device 802 to the host device 810 . Host device API 812 maintains the connection between fitness application 813 and hardware device 802 as long as application 813 remains active. After establishing a connection with one or more hardware devices 802 , interface software 811 and host device API 812 receive and process input data from hardware devices 802 so that the input data can be used to control applications 813 . For example, input data may be used by application 813 to control virtual characters or objects in a game.

Host device API 812 can distinguish incoming data from different hardware devices because each hardware device's firmware can be configured to include an identifier (eg, a binary value) in the header of any incoming data sent to host device 810 . Accordingly, the motion sensing device 806 may have a different identifier than the handheld controllers 806A, 806B. Interface software 811 may then establish a software state representative of each device based on the device's identifier, and host device API 812 may map input data into its respective software state representation to identify the original hardware device. The data may be used as input to the fitness application 813 after mapping the input data to corresponding hardware devices, eg, via tables or indexes. In embodiments where the motion sensing device 804 is configured to relay input data from the handheld controllers 806A, 806B to the host device 810, the motion sensing device 804 may be configured to read data from both handheld controllers 806A, 806B, identifier in the input stream of 806B and relays the single integrated input stream to host device 810. This process is described in more detail with reference to Figures 9A and 9D.

In some embodiments, the left and right hand held controllers 806A, 806 are manufactured to be identical (eg, to reduce manufacturing costs) and have the same identifier. In these embodiments, it may be difficult to automatically distinguish between input data from the two hand controllers 806A, 806B, so the interface software 811 may prompt the user to manually identify the two hand controllers (e.g., pressing the left hand controller first button on 806A and then press the button on right hand controller 806B). This setting may be stored by the interface software 811 as part of the device's software state representation.

Interface software 811 may use a similar process to distinguish input data from different hardware devices when multiple sets of hardware devices are connected to host device 810 (eg, for a multiplayer game). For example, the interface software 811 may display a prompt to move the first motion-sensing device 804 (e.g., attached to a stationary bicycle), followed by a prompt to move the second motion-sensing device 804 (e.g., attached to an elliptical trainer), and Similar prompts are then displayed to identify the respective handheld controllers on the two exercise devices 806 .

In one embodiment, signals from handheld controllers 806A, 806B and motion sensing device 804 are wirelessly sent to host device 810 as input data using a custom Bluetooth GATT profile. The data sent from the hardware device 802 is then processed by the host device API 812 .

An example of data flow between the hardware device 802 and the host device 810 is shown below:

1. The motion sensing device 804 acts as a server and the host device 810 connects as a client.

2. The host device 810 (client) subscribes to data notifications for special service features of the motion sensing device 804, such as an accelerometer.

3. Acquire readings from the service feature (eg, accelerometer readings) from the motion sensing device 804 .

4. Send readings to host device 810 (client subscriber). For example, a three-axis accelerometer reading could be sent as three bytes.

5. The interface software 811 decodes the reading and makes it available to the application 813 via the API 812 . The readings are interpreted by the application 813 as input data and allow the user to make selections or control/manipulate elements within the application 813 (eg, objects, characters, etc.), or to react to game prompts or other interactive elements within the application 813 .

Fitness application 813 is an application that receives input data from hardware device 802 through host device API 812 and performs some type of user interaction based on the input data. For example, exercise device 813 may be an exercise game or any non-gaming application that can interact with the exercise device in a meaningful way (eg, application 813 interacts with hardware device 802 mounted on a treadmill to simulate a walk through a park).

Applications 813 (including games) can be configured to encourage users to exercise. As the user exercises, data received from the hardware device 802 is recorded on the host device 810 to reflect the progression of the user's experience. The application 813 can also be connected via the network 816 to a fitness server 818 where user data and statistics can be stored and accessed.

One example of how hardware device 802 may interact with software on host device 810 may be a user attaching motion sensing device 804 to the pedals of a stationary bicycle. Accelerometers in the motion sensing device 804 measure the movement of the pedals, allowing realistic control of the “virtual ride” application 813 . Another example of how the handover device 802 interacts with the software is a user attaching a motion sensing device 804 to their hip, which detects the motion for a dance or exercise game.

Host device API 812 may be configured to run on any suitable operating system, such as iOS, Windows/Mac/Linux or Android. Different versions of the host device API 812 can be created for different operating systems. Although shown as a separate entity within host device 810, host device API 812 is typically a separate code library compiled into fitness application 813 at build time. This allows third-party developers to write fitness applications 813 that use functions in the host device API 812 to exchange data with the hardware device 802 and the fitness server 818 .

The fitness server 818 includes a cloud service module 820 , a fitness database 822 , and an application database 824 . The fitness server 818 and its components 820, 822, 824 may be embodied as a single device or as a cluster of networked devices (eg, a network server cluster).

The cloud service module 820 receives historical data such as physical activity and game performance data from the host device API 812 and stores the data in suitable databases 822 , 824 .

The cloud service module 820 also allows users to access their information on any device and at any time. The module also handles user requests to compare and share scores on third party servers 826 such as social networking systems such as FACEBOOK and TWITTER.

The cloud services module 820 can also host multiplayer games. In one embodiment, the game is hosted in an asynchronous context. After the user sets the challenge, the cloud service module 820 can give the user a time window to complete his "turn" or just wait until he completes a given goal. This information is stored in the application database 824 . The cloud services module 820 executes the logic of games, challenges or goals created between players and maintains their state and progress.

In an alternative embodiment, the cloud services module 820 also hosts real-time multiplayer games, where real-time game data is shuttled between players on different host devices 810 .

Databases 822, 824 may be stored across a cluster of several servers for scalability and fault tolerance.

Cloud services module 820 also performs functions associated with processing motor units. Motor units are standard units of measurement across different types of fitness equipment. These motor units are defined by standard movement signatures from the motion sensing device. In one embodiment, the units of motion are normalized to provide consistency across different types and manufacturers of fitness equipment. A normalization table can provide cross-references between devices for later retrieval and analysis to compare based on normalization. In one embodiment, a standard movement signature is achieved when the accelerometer on the motion sensing device 804 crosses a certain gravity threshold. For example, a standard moving signature can be defined as a deviation of 1G from the moving average of 20 accelerometer samples collected over 333ms. In general, motor units can be defined based on any number of parameters, including calories burned or heart rate. Additional software functionality can be added to the host device API 812 or fitness application 813 to improve motor unit accuracy. For example, if calories burned is the primary parameter, an option could be added to select the type of exercise machine being used.

Users can create personal goals to achieve a set number of motor units within a given time period. For example, a goal could be 50,000 motor units in a week. These personal goals can then be shared with friends within the social network.

Users can also create cooperative goals for teams in social networks to achieve a set amount of motor units. For example, a group of five could set a goal to achieve 250k motor units per week.

Motor units can also be used as a benchmark for activity that benefits from normalization of the data for comparison purposes. For example, the motor unit can be used for competitions between teams or friends in a network. Challenges can be initiated and broadcast to the user's friends on social networks.

In one embodiment, the cloud service module 820 searches the user's social graph from the social networking system on the third party server 826 and replicates the social graph on the fitness server 818 . Accordingly, subsequent functions associated with the social game mechanics, such as the creation of challenges and collaborative goals, can be performed on the fitness server 818 . FIG. 9A is a diagram illustrating components of a motion sensing device 804 according to one embodiment. Motion sensing device 804 includes sensor 902 , communication module 908 , and firmware 910 .

The sensor 902 detects the user's body movements. In the illustrated embodiment, sensors 902 include accelerometers 904 and gyroscopes 906 . However, additional or different sensors 902 may be included in motion sensing device 804 .

Communication module 908 interacts with communication link 808A to transfer data between motion sensing device 804 and host device 810 . For example, communication module 908 may receive input data from sensor 902 and send the input data to host device 810 for use by fitness application 813 . In one embodiment, the communication module 908 connects via the Bluetooth Low Energy (BLE) protocol. Firmware 910 contains logic that manages the operation of motion sensing device 804 . For example, firmware 910 may contain logic to support a “shake to connect” functionality that monitors sensor 902 and establishes communication link 808A to host device 810 upon detection that the user has shaken motion sensing device 804 .

In an alternative embodiment described with reference to FIG. 8, two handheld controllers 806A, 806B are coupled to the motion sensing device 804 and the motion sensing device 804 is configured to relay input data from the handheld controllers 806A, 806B to host device 810 . In this alternative embodiment, the firmware 910 is configured to detect and connect to the handheld controller 806A, 806B independently of the host device 810, and the communication module 908 maintains the connection with the handheld controller 806A, 806B after establishing the connection. The connection to the handheld controllers 806A, 806B may be in the same communication protocol as the communication link 808A (eg Bluetooth low energy) or a different communication protocol (eg RF connection). At the same time, the firmware 910 manages the three connections (the connection to the two handheld controllers 806A, 806B, and the communication link 808A to the host device 810) and relays incoming data from the handheld controllers 806A, 806B to the host device 810. . In some embodiments, firmware 910 may be configured to combine input data from sensors 902 and handheld controllers 806A, 806B into a single integrated input stream. Thus, host device API 812 receives a single input stream that includes input from all three hardware devices.

FIG. 9B illustrates motion sensing device 804 , including housing or housing 912 and internal electronics 913 , according to one embodiment. As described above with reference to FIG. 8 , motion sensing device 804 is attached to the user (eg, around the leg) and uses sensor 902 to measure the user's movement, which is recorded as input data and sent to host device 810 . As shown in Figure 9B, the motion sensing device 804 is encased in a small housing 912 so that it can be attached to the user in a variety of ways. In the illustrated embodiment, motion sensing device 804 is wirelessly connected to host device 810 and powered by a battery.

Figure 9C illustrates a handheld controller 806 according to one embodiment. The handheld controller 806 includes a handle 914 and a strap 916 . One or more buttons 918 are mounted on handle 914 . In addition to or instead of buttons 918, handle 914 may also include a directional pad or joystick. As described above with reference to FIG. 8 , handheld controller 806 communicates with host device 810 . In the illustrated embodiment, the handheld controller 806 communicates wirelessly and is battery powered.

Internally, handheld controller 806 includes a radio and a microcontroller. In one embodiment, the radio is an RF radio that connects to the motion sensing device 804 and sends an input signal to the motion sensing device 804 for relay to the host device 810 . In an alternative embodiment, the handheld controller includes a Bluetooth Low Energy (BLE) radio and a system on a chip (SoC). In some embodiments, the radio, microcontroller, and any other electronic components of handheld controller 806 may be placed on a single printed circuit board.

FIG. 9D illustrates an example of a handheld controller 806A, 806B mounted to a handle of an exercise device, according to one embodiment. The strap 916 of the handheld controller 806A, 806B wraps around the handle. In one embodiment, the bottom side of the handle 914 may include an adhesive substance to increase friction between the handheld controller 806A, 806B and the handle.

To reduce manufacturing costs, some or all components of the two handheld controllers 806A, 806B may be identical. In one embodiment, the two handheld controllers 806A, 806B are identical (eg, same mechanical housing, internal electronics, and firmware). In this embodiment, the firmware 910 or host device API 812 may have difficulty distinguishing between the two hand controllers, and the interface software 811 may prompt the user to identify each handset as described above with reference to FIG. 8 . In another embodiment, both handheld controllers have the same internal electronics (e.g., same microcontroller, radio, etc.), but can be placed in different housings to identify one controller as the left handheld controller 806A And identify the other as the right hand controller 806B. For example, the buttons 918 on the two handheld controllers could have different layouts or different colors, or the housings could have symmetrical ergonomic features so that they fit better in the user's hand. Both handheld controllers 806 can also be flashed with the same firmware so that the motion sensing device 804 or host device API 812 can differentiate input data from the two controllers 806 . For example, the firmware on the left hand controller 806A can be configured to send an identifier (such as a binary 0) in the header of its input stream, while the firmware on the second hand controller 806B can be configured to send an identifier (such as a binary 0) in the header of its input stream. Send a different identifier (e.g. binary 1) in .

10 is a flow diagram illustrating an exemplary process for interacting with a fitness application 813 based on a user's interaction with the fitness system, according to an embodiment of the invention. The user begins by attaching the hardware device 802 to the exercise device or the user's body; the handheld controllers 806A, 806B or the motion sensing device 804 may be held in the user's hand or attached to the exercise device, depending on the particular exercise application 813 .

According to an embodiment, the motion sensing device 804 obtains 1005 real-time motion data, including acceleration data from the accelerometer 904 or rotation data from the gyroscope 906, as the user interacts with the fitness system. Fitness systems are not limited to stationary exercise machines such as treadmills, but may also include dynamic exercise regimes, such as running on a road. Data obtained by motion sensing device 804 may be sent to host device 810 over established communication link 808 .

Based on the obtained 1005 data, the interface software 811 on the host device 810 may determine 1010 the activity specific motion signature. The activity-specific motion signature may identify the current activity being performed by the user. According to an embodiment, the motion signature may be used as a baseline for future movements that may be compared by the fitness application 813 .

According to an embodiment, the interface software 811 may generate one or more activity vectors 1015 based on the motion signature. The activity vector represents a metric associated with the user's activity level. For example, the activity vector may represent the number of calories burned by the user, the distance covered over a specific time interval, or the user's intensity level. Activity vectors are a standard unit of measurement across different types of fitness devices.

According to an embodiment, the user interacts 1020 with the fitness application 813 based on the activity vectors generated 1015 by the interface software 811 . By changing the intensity with which the user performs the fitness activity, the user changes the activity vector over a period of time. A change in the activity vector changes the state or behavior of objects in the fitness application 813 , thereby allowing the user to interact with the fitness application 813 on the host device 810 . For example, if the user is playing a running video game in which the user's character runs across an obstacle course on the screen, the intensity with which the user runs on a treadmill or rides a bicycle affects the user's task (object in application 813) in the game Speed across the screen (behavior rendered by the object). Thus by running faster, the user changes the activity vector in real time, corresponding to the user character's increased speed. The user may also interact with the application based on input from the handheld controller 806 . For example, when a user's character approaches an obstacle in a video game, the user may press a button on handheld controller 806 to cause the user's character to jump over or avoid the obstacle.

hardware architecture

In general, the techniques disclosed herein can be implemented in hardware or a combination of software and hardware. For example, they may be implemented in an operating system kernel, in a separate user process, in a library package bound into a network application, on a specially constructed machine, on an application-specific integrated circuit (ASIC), or on a network interface card.

Hybrid software/hardware implementations of at least some of the embodiments disclosed herein may be implemented on programmable network-resident machines (which should be understood to include intermittently connected network-aware machines). The network device may have multiple network interfaces that may be configured or designed to utilize different types of network communication protocols. A general architecture for some of these machines may be described here in order to illustrate one or more exemplary mechanisms by which a given functional unit may be implemented. Depending on the particular embodiment, at least some of the features or functions of the various embodiments disclosed herein may be implemented on one or more general-purpose computers associated with one or more networks, such as, for example, end-user computer systems, client computers, web server or other server system, mobile computing device (such as a tablet computing device, mobile phone, smartphone, laptop computer, or other suitable computing device), consumer electronics device, music player, or any other suitable electronic device, router, switch, or other suitable device, or any combination thereof. In at least some embodiments, at least some of the features or functions of the various embodiments disclosed herein can be implemented in one or more virtualized computing environments (e.g., networked computing clouds, hosted on one or more physical computing machines) virtual machine, or other suitable virtual environment).

Referring now to FIG. 11 , there is shown a block diagram illustrating an example computing device 10 suitable for implementing at least a portion of the features or functions disclosed herein. Computing device 10 may be, for example, any one of the computing machines listed in the preceding paragraph, or even any other electronic device capable of executing software-based or hardware-based instructions according to a program stored in memory. Computing device 10 may be configured to communicate with multiple other computing devices, such as clients or servers, over a communication network, such as a wide area network, a metropolitan area network, a local area network, a wireless network, the Internet, or a network using known protocols for such communications. Any other network, whether wireless or wired.

In one embodiment, computing device 10 includes one or more central processing units (CPUs) 12, one or more interfaces 15, and one or more buses 14, such as a peripheral component interconnect (PCI) bus. When acting under the control of suitable software or firmware, CPU 12 may be responsible for implementing special functions associated with the functions of a specially configured computing device or machine. For example, in at least one embodiment, computing device 10 may be configured or designed to function as a server system utilizing CPU 12 , local memory 11 and/or remote memory 16 , and interface 15 . In at least one embodiment, CPU 12 can be made to perform one or more different types of functions and/or operations under the control of software modules or components, which can include, for example, an operating system and any suitable application software, drivers etc.

CPU 12 may include one or more processors 13, such as, for example, processors from one of the Intel, ARM, Qualcomm, and AMD microprocessor families. In some embodiments, the processor 13 may include specially designed hardware such as Application Specific Integrated Circuits (ASICs), Electrically Erasable Programmable Read-Only Memory (EEPROM), Field Programmable Gate Array (FPGA), etc., for controlling Operation of Computing Device 10 . In particular embodiments, local memory 11 , such as non-volatile Random Access Memory (RAM) and/or Read Only Memory (ROM), including, for example, one or more levels of cache, may also form part of CPU 12 . However, there are many different ways in which memory may be coupled to system 10 . Memory 11 may be used for various purposes such as, for example, caching and/or storing data, programming instructions, and the like. It should further be appreciated that CPU 12 may be one of various system-on-chip (SOC) type hardware which may include additional hardware such as memory or graphics processing chips, such as Qualcomm SNAPDRAGON ^™ or Samsung EXYNOS ^™ , as the art becomes more and more More common, such as for mobile devices or integrated devices.

As used herein, the term "processor" is not limited to those integrated circuits known in the art as processors, mobile processors, or microprocessors, but broadly refers to microcontrollers, microcomputers, programmable logic controllers, , ASICs, and any other programmable circuits.

In one embodiment, interfaces 15 are provided as network interface cards (NICs). Typically, NICs control the sending and receiving of data packets over a computer network; other types of interfaces 15 may, for example, support other peripheral devices for computing device 10 . Among the interfaces that can be provided are Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, graphics interfaces, and the like. Furthermore, various types of interfaces may be provided such as, for example, Universal Serial Bus (USB), Serial Interface, Ethernet Interface, FIREWIRE ^™ Interface, THUNDERBOLT ^™ Interface, PCI Interface, Parallel Interface, Radio Frequency (RF) Interface, BLUETOOTH ^™ interface, near field communication (for example using a near magnetic field) interface, 802.11 (WiFi) interface, Frame Relay interface, TCP/IP interface, ISDN interface, Fast Ethernet interface, Gigabit Ethernet interface, Serial ATA (SATA) or External SATA (ESATA) interface, high-resolution multimedia interface (HDMI), digital visual interface (DVI), analog or digital audio interface, asynchronous transfer mode (ATM) interface, high-speed serial interface (HSSI) interface, point of sale (POS ) interfaces, Fiber Data Distribution Interfaces (FDDIs), and so on. Typically, the interface 15 may comprise a physical port for communicating with a suitable medium. In some cases, they may also include independent processors (such as dedicated audio or video processors, as is common in the art for high-fidelity A/V hardware interfaces), and in some cases, volatile and/or Non-volatile memory (such as RAM).

Although the system shown in FIG. 11 illustrates a particular architecture for a computing device 10 for implementing one or more of the inventions described herein, it is by no means on which at least a portion of the features or techniques described herein can be implemented. Unique device architecture. For example, an architecture with one or any number of processors 13 may be used, and the processors 13 may be present in a single device or distributed among any number of devices. In one embodiment, a single processor 13 handles communications as well as routing calculations, while in other embodiments a separate dedicated communications processor may be provided. In various embodiments, different types of features or functions may be implemented in a system according to the invention, including a client device (such as a tablet device or a smartphone running client software) and a server system (such as the one described in more detail below). server system).

Regardless of network device configuration, the system of the present invention may utilize one or more memories or memory modules (such as, for example, remote memory block 16 and local memory 11) configured to store data, program instructions for general network operations, Or other information (or any combination of the above) related to the functionality of the embodiments described herein. Program instructions may, for example, include or control the execution of an operating system and/or one or more applications. Memory 16 or memories 11, 16 may also be configured to store data structures, configuration data, encrypted data, historical system operating information, or any other special or general non-program information described herein.

Because this information and program instructions may be utilized to implement one or more of the systems or methods described herein, at least some network device embodiments may include a non-transitory machine-readable storage medium that may be configured or designed to store program instructions, Status information and the like are used to perform the various operations described herein. Examples of such non-transitory machine-readable storage media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media such as optical disks; Hardware devices such as read-only memory devices (ROM), flash memory (as is common in mobile devices and integrated systems), solid-state drives (SSD), and "hybrid SSD" storage drives, which can combine solid-state and The physical components of hard disk drives (as is becoming increasingly common in the field of personal computers), memristors, random access memory (RAM), and so on. It should be appreciated that the storage devices may be integral and non-removable (such as RAM hardware modules that may be soldered to the motherboard or otherwise integrated into the electronic device), or they may be removable such as swappable flash memory modules (such as "thumb drives" or other removable media designed for quick exchange of physical storage devices), "hot-swappable" hard disk drives or solid-state drives, removable optical storage drives, or other such removable media, and The integral and removable storage media may be utilized interchangeably. Examples of program instructions include object code such as can be produced by a compiler, machine code such as can be produced by a compiler or linker, byte code such as can be produced by, for example, a JAVA ^™ compiler and can be executed using a Java virtual machine or device , or files that contain higher-level code that can be executed by a computer using an interpreter (such as a script written in Python, Perl, Ruby, Groovy, or any other scripting language).

In some embodiments, systems according to the present invention may be implemented on stand-alone computing systems. Referring now to FIG. 12 , there is shown a block diagram depicting a typical exemplary architecture of one or more embodiments, or components thereof, on a stand-alone computing system. Computing device 20 includes a processor 21 that may run software that implements one or more functions or applications of embodiments of the present invention, such as, for example, client application 24 . The processor 21 can execute calculation instructions under the control of the operating system 22, such as the version of Microsoft's WINDOWS ^TM operating system, Apple's Mac OS/X or iOS operating system, some variants of the Linux operating system, Google's ANDROID ^TM operating system , home game devices such as Microsoft XBOX ^TM , Sony PLAYSTATION ^TM , or virtual reality hardware devices such as Oculus RIFT ^TM , HTC VIVE ^TM , Samsung GEAR VR ^TM , and so on. In many cases, one or more common services 23 may be operational in system 20 and may assist in providing common services to client applications 24 . The service 23 may be, for example, a WINDOWS ^™ service, a user space common service in a Linux environment, or any other type of common service framework for the operating system 21 . Input device 28 may be of any type suitable for receiving user input including, for example, a keyboard, touch screen, microphone (eg, for voice input), mouse, touch pad, trackball, or any combination thereof. Output device 27 may be of any type suitable for providing output to one or more users, whether remote or local to system 20, and may include, for example, one or more screens for visual output, speakers, a printer, or random combination. Memory 25 may be a random access memory having any structure and architecture known in the art for use by processor 21, eg, to run software. Storage device 26 may be any magnetic, optical, mechanical, memristive, or electrical storage device (such as those described above with reference to FIG. 11 ) for storing data in digital form. Examples of storage device 26 include flash memory, magnetic hard drives, CD-ROMs, and/or the like.

In some embodiments, the system of the present invention may be implemented such as on a distributed computing network with any number of clients and/or servers. Referring now to FIG. 13 , there is shown a block diagram depicting an exemplary architecture 30 for implementing at least a portion of a system according to an embodiment of the invention over a distributed computing network. Depending on the embodiment, any number of clients 33 may be provided. Each client 33 may run software for implementing the client-side portion of the invention; a client may comprise a system 20 such as that shown in FIG. 12 . Furthermore, any number of servers 32 may be provided for processing requests received from one or more clients 33 . Client 33 and server 32 may communicate with each other via one or more electronic networks 31, which in various embodiments may be the Internet, a wide area network, a mobile phone network (such as a CDMA or GSM cellular network), a wireless network (such as WiFi, Wimax, LTE, etc.), or a local area network (or indeed any network topology known in the art; the present invention does not prefer either network topology over any other). Network 31 may be implemented using any known network protocol, including, for example, wired and/or wireless protocols.

Furthermore, in some embodiments, server 32 may invoke external service 37 when additional information is required or additional data regarding a particular call is involved. Communication with external services 37 may occur, for example, via one or more networks 31 . In various embodiments, external services 37 may include network-enabled services or functions associated with or installed on the hardware device itself. For example, in embodiments where client application 24 is implemented on a smartphone or other electronic device, client application 24 may be in the cloud or on an external service 37 deployed on one or more properties of a particular business or user. Information stored in server system 32 is obtained.

In some embodiments of the invention, client 33 or server 32 (or both) may utilize one or more dedicated services or applications that may be deployed locally or remotely across one or more networks 31 . For example, one or more databases 34 may be used or referenced by one or more embodiments of the invention. Those skilled in the art will appreciate that database 34 may be arranged in a wide variety of architectures and employ a wide variety of data access and manipulation mechanisms. For example, in various embodiments one or more of databases 34 may comprise a relational database system using Structured Query Language (SQL), while others may comprise alternative data storage technologies such as those known in the art as "NoSQL" (e.g. Hadoop Cassandra, Google BigTable, etc.). In some embodiments, variant database architectures such as column-oriented databases, in-memory data, clustered databases, distributed databases, or even flat data repositories may be used in accordance with the present invention. Those skilled in the art will appreciate that any combination of known or future database technologies may be used as appropriate unless a particular database technology or particular arrangement of components is specified for a particular embodiment herein. Furthermore, it should be appreciated that the term "database" as used herein may refer to a physical database machine, a cluster of machines acting as a single database system, or a logical database within an overall database management system. Unless a special meaning is specified for a given use of the term "database", it should be construed to mean any meaning of these words, all of which are understood by those skilled in the art to be the obvious meaning of the term "database".

Similarly, most embodiments of the invention may utilize one or more security systems 36 and configuration systems 35 . Security and configuration management are common information technology (IT) and network functions, and some quantity of each is typically associated with any IT or network system. Those skilled in the art should understand that any configuration or safety subsystem known in the art now or in the future can be used in conjunction with the embodiments of the present invention without limitation, unless special safety is specifically required by the description of any specific embodiment 35 or Configure the system 35 or scheme.

FIG. 14 shows an exemplary overview of a computer system 40 as may be used at any of various locations throughout the system. It is an example of any computer that can execute code to process data. Various modifications and changes can be made to the computer system 40 without departing from the broader scope of the systems and methods disclosed herein. A central processing unit (CPU) 41 is connected to the bus 42, and a memory 43, a nonvolatile memory 44, a display 47, an input/output (I/O) unit 48, and a network interface card (NIC) 53 are also connected to the bus. I/O unit 48 may typically be connected to keyboard 49 , pointing device 50 , hard disk 52 , and real-time clock 51 . The NIC 53 is connected to a network 54, which may be the Internet or a local area network, where the local area network may or may not have a connection to the Internet. Also shown as part of the system 40 is a power supply unit 45 which in this example is connected to a mains alternating (AC) power supply 46 . Batteries, which may be present, are not shown, nor are the many other devices and modifications that are widely known but are not applicable to the specific innovative functionality of the present systems and methods disclosed herein. It should be appreciated that some or all of the components shown may be combined, such as in various integrated applications, such as Qualcomm or Samsung's system-on-chip (SOC) devices, or as long as it is applicable to combine multiple capabilities or functions into a single hardware device (eg, in a mobile device such as a smartphone, a video game console, an in-vehicle computer system in an automobile such as a navigation or multimedia system, or other integrated hardware device).

In various embodiments, the functionality for implementing the systems or methods of the present invention may be distributed among any number of client and/or server components. For example, various software modules can be implemented to perform various functions in connection with the present invention, and these modules can be variously implemented to run on server and/or client components.

Those skilled in the art will appreciate the range of possible modifications to the various embodiments described above. Accordingly, the invention is defined by the claims and their equivalents.

Claims (6)

1. A system for mixed reality interaction using a plurality of electronic control and tracking devices, comprising:

a first electronic control device comprising at least a first memory, a first processor, and a plurality of first programming instructions stored on the first memory and running on the first processor, the first programming instructions configured to:

communicating with a plurality of electronic devices via a network, each electronic device communicating using one of a plurality of network communication protocols, the network communication protocol of each electronic device selected from the plurality of network communication protocols according to the detected capabilities of the particular electronic device; wherein at least one of the plurality of electronic devices is a second electronic control device, at least one is an electronic system comprising a computing device operating a mixed reality software application, network communications are transmitted in a bi-directional manner between the electronic system and the at least one second electronic control device, and communications are conducted through respective network communications protocols;

Operating a plurality of hardware control elements configured to receive manual interactions from a user;

transmitting a plurality of interaction data to at least a portion of the plurality of electronic devices, the interaction data based at least in part on at least a portion of a manual interaction from the user;

receiving traffic from at least a portion of the plurality of electronic devices, the traffic comprising at least mixed reality data, wherein the mixed reality data comprises at least a plurality of sensor data readings; and

an action is performed within the mixed reality software application based on the received traffic.

2. The system of claim 1, wherein the plurality of electronic devices includes at least a virtual reality device.

3. The system of claim 2, wherein the plurality of electronic devices includes at least an exercise device.

4. The system of claim 3, wherein the fitness device comprises at least a fitness tracking device.

5. The system of claim 4, wherein the fitness tracking system provides at least a plurality of fitness information to at least the virtual reality device.

6. The system of claim 1, wherein the plurality of electronic devices comprises at least a plurality of tethers, each tether comprising a tether line having one end attached to an attachment point at a fixed location away from the body of the human user and the other end attached to an attachment point proximate to the body of the human user, the proximal attachment point configured for wearing or otherwise being attached to the person or clothing of the user, and at least a portion of the distal attachment point comprising a sensor configured for measuring at least strain of the attached tether.

Images