TWI734106B - Stereoscopic visualization camera and integrated robotics platform - Google Patents

Abstract

A robotic imaging apparatus is disclosed. A robotic imaging apparatus includes a robotic arm, a stereoscopic camera, and a sensor positioned between the robotic arm and the stereoscopic camera. The sensor transmits output data that is indicative of translational and rotational force imparted on the stereoscopic camera by an operator. The robotic imaging apparatus also includes a processor that is configured to determine a movement sequence for the robotic arm based on a current position of the robotic arm and the output data from the sensor and to cause at least one of the joints of the robotic arm to rotate based on the determined movement sequence via one or more motor control signals provided to the at least one joint. The rotation of the at least one joint provides power-assisted movement of the robotic arm based on the detected translational and rotational forces imparted by the operator.

Description

Stereo visual camera and integrated robot platform

This application is related to a stereo vision camera and an integrated robot platform.

Department of Surgery Art. Achieving artists create artworks that are far beyond the ability of a normal person. The artist uses a paintbrush to turn the paint can into a vivid image that stimulates the viewer's strong and unique emotions. The artist writes ordinary words on paper and turns them into dramatic and awesome performances. Artists master musical instruments and let them make beautiful music. Similarly, surgeons hold seemingly ordinary scalpels, tweezers and probes and produce life-changing biological miracles.

Like artists, surgeons have their own methods and preferences. To teach aspiring artists the basic principles of their skills. Beginners usually follow the specified method. When they gain experience, confidence and knowledge, they form a unique art that reflects themselves and their personal environment. Similarly, teach medical students the basic principles of surgical procedures. They have been rigorously tested on these methods. As students make progress in their internship and professional practice, they develop basic principles (still within medical standards) based on the way they believe that surgery should be performed best. For example, consider the same medical procedure performed by different famous surgeons. The sequence of events, pace, staff placement, tool placement, and imaging equipment use varies between each of the surgeons based on the surgeon's preference. Even the size and shape of the incision can be unique to the surgeon.

The artistic uniqueness and achievements of surgeons make them tired of changing or changing their methods and surgical tools. The tool should be extended by one of the surgeons so as to operate simultaneously and/or coordinated and synchronized. Instructing the flow of a procedure or changing the rhythm of a surgeon. Surgical tools are usually discarded or modified to meet requirements.

In one example, consider microsurgery visualization, where certain surgical procedures involve patient structures that are too small for a human to be easily visualized with the naked eye. For these microsurgery procedures, magnification is required to fully view the microstructure. Surgeons generally want a visualization tool that is a natural extension of their eyes. In fact, early efforts in microsurgery visualization included attaching magnifying lenses to head-mounted optical eyepieces (called surgical loupes). The first pair was developed in 1876. Today, surgeons are still using a greatly improved version of surgical loupes (some surgical loupes include optical zoom and integrated light sources). FIG. 1 shows a diagram of a pair of surgical loupes 100 having a light source 102 and one of a plurality of magnifying lenses 104. The 150-year endurance of the surgical loupe can be attributed to the fact that it is an extension of a surgeon’s eye.

Despite its longevity, surgical loupes are not perfect. A magnifying glass with a magnifying lens and a light source (such as the magnifying glass 100 in FIG. 1) has a greater weight. Placing even a small amount of weight in front of a surgeon's face can increase discomfort and fatigue, especially during prolonged surgical operations. The surgical loupe 100 also includes a cable 106 connected to a remote power supply. The cable effectively acts as a chain, thus restricting the surgeon's mobility during the surgeon's surgical operation.

Another microsurgery visualization tool is a surgical operating microscope, also known as an operating microscope. The extensive commercial development of surgical microscopes began in the 1950s and was intended to replace surgical loupes. A surgical operating microscope includes an optical path, a lens, and a focusing element that provide greater magnification compared with a surgical loupe. The large array of optical elements (and the resulting weight) means that the surgical operating microscope must be separated from the surgeon. Although this separation gives the surgeon more operating space, the bulkiness of the surgical operating microscope causes it to consume a considerable operating space on a patient, thereby reducing the size of the surgical operating table.

FIG. 2 shows a diagram of a prior art surgical operating microscope 200. As you can imagine, the size and presence of the surgical operating microscope in the operating area make it prone to collisions. To provide stability and rigidity at the lens portion 201, the microscope is connected to relatively large boom arms 202 and 204 or other similar supporting structures. The large boom arms 202 and 204 consume additional surgical space and reduce the operability of surgeons and staff. All in all, the surgical operating microscope 200 shown in FIG. 2 can weigh up to 350 kilograms ("kg").

In order to use the surgical operating microscope 200 to view a target surgical site, a surgeon directly looks at the eyepiece 206. In order to reduce the pressure on the back of a surgeon, the arm 202 is generally used to position the eyepiece 206 along the natural line of sight of a surgeon to adjust the height. However, the surgeon does not just look at a target surgical site and perform it. The eyepiece 206 must be positioned so that the surgeon is within the length of the arm of a working distance from the patient. This precise positioning is critical to ensure that the surgical operating microscope 200 becomes an extension of the surgeon rather than an obstacle, especially when using extended time periods.

Like any complicated instrument, it takes tens to hundreds of hours for surgeons to feel comfortable using a surgical operating microscope. As shown in FIG. 2, the design of the surgical operating microscope 200 requires a substantially 90° optical path from the surgeon to one of the target surgical sites. For example, the optical path from the target surgical site to one of the lens parts 201 needs to be completely vertical. This means that the lens part 201 must be positioned directly above the patient for each microsurgery procedure. In addition, the surgeon must look at the eyepiece 206 almost horizontally (or at a slight angle downward). A surgeon’s natural tendency is to direct his vision to his hands at the surgical site. Some surgeons even want to move their hands closer to the surgical site to have more precise control of their hand movements. Unfortunately, the surgical operating microscope 200 does not give the surgeon this flexibility. Alternatively, the surgical operating microscope 200 ruthlessly instructs the surgeon to place his eyes on the eyepiece 206 and hold his head at the length of his arm during the execution of his surgical operation, and all of this consumes precious surgery on the patient. Surgical space. A surgeon cannot even look down at a patient simply because the lens 201 blocks the surgeon’s field of vision.

To make matters worse, some surgical operating microscopes 200 include a second pair of eyepieces 208 for a co-executor (eg, surgical assistant, nurse, or other clinical staff). The second pair of eyepieces 208 are generally positioned at a right angle to the first eyepiece 206. The tightness between the eyepieces 206 and 208 determines that the assistant must stand (or sit down) in close proximity to the surgeon, thereby further restricting movement. For some surgeons who like to use certain spaces to perform, this can be annoying. Despite its magnification benefits, the surgical operating microscope 200 is not a natural extension of the surgeon. Instead, he is the bossy director in the surgical operating room.

The present invention is directed to a three-dimensional robot system including a three-dimensional visualization camera and a robot arm. The exemplary stereo robotic system is configured to obtain a stereo image of a target surgical site while enabling an operator to use the robotic arm to position the stereo visualization camera. As disclosed herein, the robot arm includes electromechanically operated joints that provide structural stability so that the stereoscopic visual camera can record without jitter or other artifacts (which can be moved by unintentional cameras) Caused) high-resolution images. The robotic arm also provides structural flexibility, which allows an operator to position the stereo visualization camera at different positions and/or orientations to obtain a desired view of a target surgical site. Therefore, the exemplary three-dimensional robot system enables a surgeon to comfortably perform life-changing microsurgery in any position suitable for the surgeon.

The three-dimensional robot system of the present invention can be positioned in any number of orientations relative to the surgical area, which is best suited to the needs of the surgeon or the patient rather than the physical and mechanical limitations of the visualization equipment. The stereo robotic system is configured to provide motorized joint movement assistance that enables a surgeon or other operator to position the stereo visualization camera effortlessly. In some embodiments, the stereo robot system is configured to provide motorized assisted movement of a robot arm based on the force detected by an operator of the self-positioning stereo camera. The stereo robot system can also enable an operator to select a vision lock on a target surgical site and also enable the operator to change the orientation and/or position of the stereo vision camera. Additionally or alternatively, the stereo robotic system is configured to prevent the stereo visualization camera and/or the robotic arm from contacting one or more boundaries of a patient, surgical staff, and/or surgical instrument. In short, the three-dimensional robot system operates as an extension of the eyes of a surgeon while giving the surgeon the freedom to perform a microsurgical procedure without limitation or obstruction.

The aspects of the subject matter described in this article can be used alone or in combination with one or more of the other aspects described in this article. Without limiting the foregoing description, in a first aspect of the present invention, a robotic imaging device includes: a base section configured to be connected to a solid structure or a truck; and a robotic arm It has a first end connected to the base section, a second end including a coupling interface, and a plurality of joints and links connecting the first end to the second end. Each joint includes a motor configured to rotate the joint around an axis and a joint sensor configured to transmit a position of the respective joint. The robot imaging device also includes a stereo camera connected to the robot arm at the coupling interface. The stereo camera is configured to record the left and right images of a target surgical site for generating a three-dimensional image stream of the target surgical site. The robotic imaging device further includes a sensor positioned at the coupling interface and configured to detect the translation and rotation force applied to the stereo camera by an operator and transmit instructions to the translation and rotation force The output data. The robot imaging device additionally includes a memory that stores at least one algorithm defined by one or more commands and/or data structures, the one or more commands and/or data structures are based on at least one of the robot arms The current position and the detected translation and rotation forces define the rotation direction, speed and duration of each of the joints of the robot arm. In addition, the robotic imaging device includes at least one processor communicatively coupled to the sensor and the robotic arm. The at least one processor is configured to receive the output data indicating the translational and rotational forces from the sensor and use the at least one algorithm in the memory based on a current position of the robot arm and from the sensor The output data of the detector determines the movement sequence of the robot arm. The at least one processor is also configured to cause the at least one joint to rotate via one or more motor control signals provided to at least one of the joints of the robot arm based on the determined movement sequence. The rotation of the at least one joint provides power-assisted movement of the robot arm based on the detected translation and rotation forces applied to the stereo camera by the operator.

According to a second aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the at least one processor is configured to be based on the plurality of joints The output data of the joint sensor determines the current position of the robot arm.

According to a third aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the sensor includes a six-degree-of-freedom tactile force sensing device or a torque At least one of the sensors.

According to a fourth aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the stereo camera includes at least one control having a release button to enable the power-assisted movement Arm, and the at least one processor is configured to receive an input message instructing to select one of the release buttons, and use the output data from the sensor to determine the movement after receiving the input message related to the release button order.

According to a fifth aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the device further includes a coupling plate configured to be connected to A first end of the coupling interface of the robot arm and a second end configured to be connected to a second coupling interface of the stereo camera. The coupling plate includes at least one joint, the at least one joint includes a joint sensor configured to transmit a position of the respective joint, and a motor that can be controlled by the at least one processor according to the movement sequence.

According to a sixth aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the sensor is located at the coupling interface or the second coupling interface.

According to a seventh aspect of the present invention, which can be used in combination with any of the other aspects listed herein (unless otherwise stated), the coupling plate includes such that the stereo camera can be oriented and oriented horizontally by an operator. Manually rotate a second joint between a vertical orientation. The second joint includes a joint sensor configured to transmit a position of the second joint.

According to an eighth aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the stereo camera includes a housing that is configured to The coupling interface is connected to a bottom side of the robot arm.

According to a ninth aspect of the present invention, which can be used in combination with any of the other aspects listed herein (unless otherwise stated), a robotic imaging device includes a robot arm that includes a robot arm for connecting to a A first end of the solid structure, a second end that includes a coupling interface, and a plurality of joints and links connecting the first end to the second end, each joint includes a joint configured so that the joint surrounds a A motor that rotates on the axis and a joint sensor that is configured to transmit a position of the respective joint. The robotic imaging equipment also includes: an imaging device connected to the robot arm at the coupling interface, the imaging device configured to record an image of a target surgical site; and a sensor positioned on the coupling The interface is configured to detect the force and/or torque applied to the imaging device by an operator and transmit force and/or torque output data indicating the force and/or torque. The robotic imaging device further includes at least one processor communicatively coupled to the sensor and the robotic arm. The at least one processor is configured to: receive the force and/or torque output data from the sensor; convert the force and/or torque output data into translation and rotation vectors; use kinematics based on the robot arm A current position and the translation and rotation vectors are used to determine a movement sequence of the robot arm. The movement sequence specifies a rotation direction, a speed, and a movement duration of at least some of the joints of the robot arm ; And based on the determined movement sequence, the at least one joint is caused to rotate via at least one or more motor control signals provided to the joints of the robot arm.

According to a tenth aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the processor is configured to be based on the current movement sequence of the robot arm. At least one of the position or a future position of the robot arm is determined to determine at least one scale factor, and the scale factor is applied to at least one joint speed of the movement sequence.

According to an eleventh aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the at least one scale factor is based on the distance from the robot arm or the imaging device to a virtual boundary One distance and configuration. The at least one scale factor decreases to a value of "0" as it approaches the virtual boundary.

According to a twelfth aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the virtual boundary corresponds to a patient, a medical instrument, or an operating room staff member At least one.

According to a thirteenth aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the processor is configured to cause a display device to display indicating the at least one ratio The factor has been applied to one of the icons of the movement sequence.

According to a fourteenth aspect of the present invention, which can be used in combination with any of the other aspects listed herein (unless otherwise stated), the processor is configured to be based on the joints between the joints of the robot arm At least one scale factor is determined based on angle or joint limitation, and the scale factor is applied to at least one joint speed of the movement sequence.

According to a fifteenth aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the processor is configured to provide the force and/or torque output Gravity compensation of the data, and force compensation for the force and/or torque output data is provided to compensate one of the position of the sensor and the imaging device of the force and/or torque applied by the operator An offset between positions.

According to a sixteenth aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the processor is configured to determine or recognize the plurality of robot arms The joint singular points of the joints are used to control hysteresis and backlash, and the movement sequence is determined while avoiding the movement of the robot arm through the joint singular points based on the kinematics.

According to a seventeenth aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the robotic imaging device further includes a coupling plate having a configured To be connected to a first end of the coupling interface of the robot arm and include a second end configured to be connected to a second coupling interface of the stereo camera. The coupling plate includes at least one joint, the at least one joint includes a joint sensor configured to transmit a position of the respective joint, and a motor that can be controlled by the at least one processor according to the movement sequence. The sensor is located at the coupling interface or the second coupling interface.

According to an eighteenth aspect of the present invention, which can be used in combination with any other aspects listed herein (unless otherwise stated), the robot arm includes at least four joints and the coupling plate includes at least two joints .

According to a nineteenth aspect of the present invention, which can be used in combination with any of the other aspects listed herein (unless otherwise stated), the processor is configured to indicate that the movement sequence is specified by One or more command signals of the rotation direction, the speed, and the movement duration are transmitted to the motors of the respective joints to cause at least one of the joints of the robot arm to rotate.

According to a twentieth aspect of the present invention, which can be used in combination with any of the other aspects listed herein (unless otherwise stated), the processor is configured to compare when the robot is activated during the movement sequence The image recorded by the imaging device as the arm moves to confirm that the robot arm moves as specified during the movement sequence.

According to a twenty-first aspect of the present invention, it can be used in combination with any of the other aspects listed herein (unless otherwise stated), and the kinematics include inverse kinematics or Jacobian kinematics At least one of them.

According to a twenty-second aspect of the present invention, any of the structure and functionality illustrated and described in conjunction with FIGS. 3 to 65 can be combined with any of the other figures in FIGS. 3 to 65 And in combination with any one or more of the foregoing aspects, any one of the structure and functionality illustrated and described is used in combination.

In view of the above aspects and the invention herein, one advantage of the invention is to provide a stereo robot system that provides seamless coordination between a stereo camera and a robot arm.

Another advantage of the present invention is to provide a three-dimensional robot system that uses a robot arm to increase a focal range, working distance and/or magnification of a three-dimensional robot system.

An additional advantage of the present invention is to provide a three-dimensional robot system that provides power-assisted movement of the robot arm based on the force/torque applied by an operator to a three-dimensional camera.

The advantages discussed herein may exist in one or some of the embodiments disclosed herein and may not be in all embodiments. Additional features and advantages are described herein, and will be made clear based on the following embodiments and figures.

Priority claim

This application is a non-provisional case of the U.S. Provisional Patent Application No. 62/663,689 filed on April 27, 2018 and claims the priority and rights of the U.S. Provisional Patent Application. The full text of the U.S. Provisional Patent Application Incorporated into this article by reference. This application is also part of the continuation of the U.S. Patent Application No. 15/814,127 filed on November 15, 2017. The U.S. Patent Application claims that the U.S. Provisional No. 62/489,289 filed on April 24, 2017 Priority and rights of patent applications and US provisional patent application No. 62/489,876 filed on April 25, 2017, the full text of these US provisional patent applications are incorporated herein by reference.

The present invention generally relates to a stereo visualization camera and platform. The stereo vision camera may be referred to as a digital stereo microscope ("DSM"). The exemplary camera and platform are configured to integrate microscope optical elements and video sensors to be significantly smaller and smaller than prior art microscopes (such as the surgical loupe 100 of FIG. 1 and the surgical microscope 200 of FIG. 2). Lighter and easier to operate in an independent head unit. The example camera is configured to transmit a stereoscopic video signal to one or more television monitors, projectors, holographic devices, smart glasses, virtual reality devices, or other visual display devices in a surgical environment.

A monitor or other visual display device can be positioned in the surgical environment to be easily within the sight of a surgeon when performing a surgical operation on a patient. This flexibility allows the surgeon to place the display monitor based on personal preference or habits. In addition, the elasticity and slender contour of the stereo vision camera disclosed in this article reduce the area consumed above a patient. In short, compared with the surgical microscope 200 discussed above, the stereo visualization camera and monitor (for example, the stereo visualization platform) enable a surgeon and a surgical team to control without being dominated or restricted by movement. A patient performs a complex microsurgical procedure. The exemplary stereo visualization platform therefore operates as an extension of the surgeon's eyes, allowing the surgeon to perform outstanding microsurgery operations without dealing with the pressure, constraints, and limitations caused by previously known visualization systems.

The present invention generally refers to microsurgery in this context. Can be used in virtually any microsurgery procedure (for example, including cranial surgery, brain surgery, neurosurgery, spinal surgery, ophthalmic surgery, corneal transplantation, orthopedic surgery, ENT surgery, oral surgery Exemplary stereo vision cameras are used in surgery, plastic and reconstructive surgery, or general surgery.

The present invention also refers to the target part, scene or field of view in this article. As used herein, a target part or field of view includes an object (or part of an object) recorded or otherwise imaged by an exemplary stereo visualization camera. Generally speaking, the main objective lens assembly of the target part, scene or field of view is a long working distance and aligned with the example stereo visualization camera. The target site may include a patient's biological tissue, bone, muscle, skin, or a combination thereof. In these cases, the target site may be three-dimensional because it has a depth component corresponding to a progression of a patient's anatomical structure. The target part may also include one or more templates for calibration or verification of the exemplary stereo visualization camera. The templates can be two-dimensional, such as a graphic design on paper (or plastic sheet), or three-dimensional, such as used to approximate the anatomy of a patient in a specific area.

Throughout the text, reference is also made to an x-direction, a y-direction, a z-direction, and an oblique direction. The z direction is along an axis from the exemplary stereo visualization camera to the target site and generally refers to depth. The x direction and the y direction are incident on a plane of the z direction and include a plane of the target part. The x-direction is along an axis that is 90° from an axis in the y-direction. Movement along the x-direction and/or the y-direction refers to in-plane movement and can refer to the movement of the exemplary stereoscopic visualization camera, the movement of optical elements in the exemplary stereoscopic visualization camera, and/or the movement of the target part .

The tilt direction corresponds to movement along Euler angles (for example, a side tilt axis, a pitch axis, and a side roll axis) relative to the x direction, the y direction, and/or the z direction. For example, a perfectly aligned lens has substantially a 0° tilt with respect to the x-direction, the y-direction, and/or the z-direction. In other words, one surface of the lens is 90° or perpendicular to the light along the z direction. In addition, the edges of the lens (if the lens has a rectangular shape) are parallel along the x-direction and the y-direction. The lens and/or the optical image sensor can be tilted through tilting movement, vertical tilting movement and/or rolling movement. For example, a lens and/or optical image sensor can be inclined with respect to the z direction along a pitch axis to face upward or downward. The light along the z-direction contacts one surface of a lens (it tilts upward or downward) at a non-perpendicular angle. The inclination of a lens and/or optical image sensor along a tilt axis, a pitch axis, or a roll axis enables, for example, a focus or ZRP to be adjusted. I. Example Stereo Visualization Camera

3 and 4 show a perspective view of a stereoscopic visualization camera 300 according to an exemplary embodiment of the present invention. The example camera 300 includes a housing 302 configured to enclose optical elements, a lens motor (e.g., an actuator), and a signal processing circuit. The camera 300 has a width (along an x-axis) between 15 centimeters (cm) and 28 centimeters (cm), preferably about 22 cm. In addition, the camera 300 has a length (along a y-axis) between 15 cm and 32 cm, preferably about 25 cm. In addition, the camera 300 has a height (along a z-axis) between 10 cm and 20 cm, preferably about 15 cm. The weight of the camera 300 is between 3 kg and 7 kg, preferably about 3.5 kg.

The camera 300 also includes control arms 304a and 304b (e.g., operating handles) that are configured to control the magnification level, focus, and other microscope features. The control arms 304a and 304b may include respective controls 305a and 305b for activating or selecting specific features. For example, the control arms 304a and 304b may include controls for selecting a fluorescent mode, adjusting the amount/type of light projected on a target site, and controlling a display output signal (for example, in 1080p or 4K and/or stereo Choose between mirrors) controls 305a and 305b. In addition, the controls 305a and/or 305b can be used to initiate and/or execute a calibration procedure and/or move a robot arm connected to the stereo visualization camera 300. In some examples, controls 305a and 305b may include the same buttons and/or features. In other examples, the controls 305a and 305b may include different characteristics. In addition, the control arms 304a and 304b can also be configured as handles so that an operator can position the stereo visualization camera 300.

Each control arm 304 is connected to the housing 302 via a rotatable pillar 306b, as shown in FIG. 3. This connection enables the control arm 304 to rotate relative to the housing 302. This rotation provides flexibility to configure the control arm 304 to a surgeon as needed, thereby further enhancing the adaptability of the stereo vision camera 300 to synchronize with a surgical operation.

Although the example camera 300 shown in FIGS. 3 and 4 includes two control arms 304a and 304b, it should be understood that the camera 300 may include only one control arm or zero control arms. In an example in which the stereo vision camera 300 does not include a control arm, the control can be integrated with the housing 302 and/or provided via a remote control.

FIG. 4 shows a bottom-up perspective view of a rear side of a stereo visualization camera 300 according to an exemplary embodiment of the present invention. The stereo visualization camera 300 includes a mounting bracket 402 configured to be connected to a support. As explained in more detail in Figures 5 and 6, the support may include an arm with one or more joints to provide significant maneuverability. The arm can be connected to a mobile truck or fastened to a wall or ceiling.

The stereoscopic visualization camera 300 also includes a power port 404 configured to receive a power adapter. It can receive power from an AC outlet and/or a battery on a truck. In some cases, the stereo vision camera 300 may include an internal battery to facilitate operation without wires. In these examples, the power port 404 can be used to charge the battery. In an alternative embodiment, the power port 404 may be integrated with the mounting bracket 402, so that the stereo vision camera 300 receives power through wires (or other conductive wiring materials) in the support.

FIG. 4 also shows that the stereo visualization camera 300 may include a data port 406. The example data port 406 can include any type of port, including, for example, an Ethernet interface, a high-definition multimedia interface ("HDMI") interface, and a universal serial bus ("USB") interface , A serial digital interface ("SDI"), a digital optical interface, an RS-232 serial communication interface, etc. The data port 406 is configured to provide a communication connection between the stereo visualization camera 300 and the wires wired to one or more computing devices, servers, recording devices, and/or display devices. The communication connection can transmit a stereoscopic video signal or a two-dimensional video signal for further processing, storage and/or display. The data port 406 can also enable control signals to be sent to the stereo visualization camera 300. For example, once connected to a computer (eg, a laptop, desktop, and/or tablet), an operator can transmit a control signal to the stereo visualization camera 300 to guide the operation, perform calibration, or change an output Display settings.

In some embodiments, the data port 406 can be replaced (and/or supplemented) by a wireless interface. For example, the stereoscopic visualization camera 300 can transmit the stereoscopic display signal to one or more display devices via Wi-Fi. The use of one of the wireless interfaces in combination with an internal battery enables the stereo visualization camera 300 to be wireless, thereby further improving the operability in a surgical environment.

The stereo vision camera 300 shown in FIG. 4 also includes a main objective lens assembly and a front working distance main objective lens 408. The example lens 408 is the beginning of the optical path within the stereo visualization camera 300. The light from a light source inside the stereo visualization camera 300 is transmitted through the lens 408 to reach a target location. In addition, the light reflected from the target site is received in the lens 408 and transmitted to the downstream optical element. II. Illustrative operability of stereo vision camera

5 and 6 show diagrams of a stereo visualization camera 300 used in a microsurgery environment 500 according to an exemplary embodiment of the present invention. As illustrated in the figure, the small footprint and operability of the stereo vision camera 300 (especially when used with a multi-degree-of-freedom arm) achieves elastic positioning relative to a patient 502. A part of the patient 502 in the view of the stereo visualization camera 300 includes a target part 503. A surgeon 504 can position the stereo vision camera 300 in virtually any orientation while leaving more than enough room for surgery on the patient 502 (lying down in a supine position). The stereo vision camera 300 is therefore minimally invasive (or not invasive) so that the surgeon 504 can perform a life-changing microsurgery procedure without being distracted or hindered.

In FIG. 5, the stereo vision camera 300 is connected to a robot arm 506 via a mounting bracket 402. The arm 506 may include one or more rotating or extendable joints with electromechanical brakes to facilitate easy repositioning of the stereoscopic camera 300. To move the stereo vision camera 300, the surgeon 504 or assistant 508 activates the brake on one or more joints of the arm 506 to release. After moving the stereo visualization camera 300 to a desired position, the brakes can be engaged to lock the joints of the arm 506 in place.

One of the salient features of the stereo vision camera 300 is that it does not include eyepieces. This means that the stereo vision camera 300 does not have to be aligned with the eyes of the surgeon 504. This degree of freedom enables the stereo visualization camera 300 to be positioned and oriented in a desirable position that is not practical or possible for previously known surgical operating microscopes. In other words, the surgeon 504 can perform microsurgery using the best view for performing the procedure (not limited to the sufficient view indicated by the eyepiece of a surgical operating microscope).

Returning to FIG. 5, the stereo visualization camera 300 is connected to a truck 510 (collectively referred to as a stereo visualization platform or a stereo robot platform 516) via the robot arm 506 and the display monitors 512 and 514. In the illustrated configuration, the stereo visualization platform 516 is independent and movable to any desired location in the microsurgery environment 500 (including between surgical operating rooms). The integrated platform 516 enables the stereo visualization camera 300 to be moved and used as needed without the time required to configure the system by connecting the display monitors 512 and 514.

Display monitors 512 and 514 can include any type of display, including a high-definition TV, an ultra-high-definition TV, smart goggles, projectors, one or more computer screens, laptops, tablets, and/ Or smart phone. The display monitors 512 and 514 can be connected to the robotic arm to achieve flexible positioning similar to the stereo visual camera 300. In some examples, the display monitors 512 and 514 may include a touch screen to enable an operator to send commands to the stereo visualization camera 300 and/or adjust a setting of a display.

In some embodiments, the truck 510 may include a computer 520. In these embodiments, the computer 520 can control a robotic arm connected to the stereo visualization camera 300. Alternatively or alternatively, the computer 520 can process the video (or stereoscopic video) signal (for example, an image or frame stream) from the stereo visualization camera 300 for display on the display monitors 512 and 514. For example, the computer 520 may combine the left video signal and the right video signal from the stereo visualization camera 300 or interlace the left and right video signals to form a stereo signal for displaying a stereoscopic image of a target part . The computer 520 can also be used to store video and/or stereoscopic video signals in a video file (stored in a memory), so that the surgical operation can be recorded and played back. In addition, the computer 520 may also send a control signal to the stereo visualization camera 300 to select settings and/or perform calibration.

In some embodiments, the microsurgery environment 500 of FIG. 5 includes an ophthalmic surgical procedure. In this embodiment, the robotic arm 506 can be programmed to perform an orbital sweep of a patient's eyes. This saccade allows the surgeon to examine a peripheral retina during the vitreoretinal procedure. In contrast, with conventional optical microscopes, the only way a surgeon can view the peripheral retina is to use a technique called scleral depression to push the side of the eye into the field of view.

Figure 6 shows a diagram of a microsurgery environment 500 with a patient 502 in a sitting position for a posterior approach skull base neurosurgery. In the illustrated embodiment, the stereo visualization camera 300 is placed in a horizontal position to face the back of the patient 502's head. The robotic arm 506 includes joints that enable the stereo visualization camera 300 to be positioned as shown. In addition, the truck 510 includes a monitor 512, which can be aligned with the surgeon's natural viewing direction.

The absence of the eyepiece enables the stereo visualization camera 300 to be positioned horizontally and below the head-up view of the surgeon 504. In addition, the relatively low weight and flexibility enable the stereo visualization camera 300 to be positioned in a way that other known surgical operating microscopes cannot imagine. The stereo visualization camera 300 thus provides a microsurgery view for any desired position and/or orientation of the patient 502 and/or the surgeon 504.

Although FIGS. 5 and 6 show two exemplary embodiments for positioning the stereo visualization camera 300, it should be understood that the stereo visualization camera 300 can be positioned in any number of positions depending on the degree of freedom of the robotic arm 506 . In some embodiments, it is entirely possible to position the stereoscopic visualization camera 300 to face upward (eg, upside down). III. Comparison between an example stereo visualization platform and known surgical operating microscopes

Comparing the stereo vision camera 300 of FIG. 3 to FIG. 6 with the surgical microscope 200 of FIG. 2, the difference is obvious. Including the eyepiece 206 together with the surgical operating microscope requires the surgeon to constantly orient his/her eyes to the eyepieces, which are in a fixed position relative to the lens portion 201 and the patient. In addition, the bulkiness and weight of surgical operating microscopes limit it to only being positioned in a generally vertical orientation relative to a patient. In contrast, the exemplary stereo visualization camera 300 does not include eyepieces and can be positioned in any orientation or position relative to a patient, thereby allowing the surgeon to move freely during the surgical operation.

In order to allow other clinicians to view a target site of microsurgery, the surgical operating microscope 200 needs to add a second eyepiece 208. Generally speaking, most known surgical operating microscopes 200 do not allow the addition of a third eyepiece. In contrast, the example stereo visualization camera 300 can be communicatively coupled to an unlimited number of display monitors. Although FIGS. 5 and 6 show the display monitors 512 and 514 connected to the cart 510 above, a surgical operating room can be surrounded by display monitors, all of which are shown by the stereo visualization camera 300 Recorded view of microsurgery. Therefore, instead of restricting a view to one or two people (or the need to share an eyepiece), an entire surgical team can view an enlarged view of a target surgical site. In addition, people in other rooms, such as training rooms and observation rooms, can be presented with the same magnified view shown to the surgeon.

Compared with the stereo vision camera 300, the binocular surgical operating microscope 200 is more prone to collision or unintentional movement. Since the surgeon places his head on the eyepieces 206 and 208 during the surgical operation to look at the eyepieces, the lens part 201 receives constant force and periodic collisions. Adding a second eyepiece 208 doubles the force from a second angle. In short, the constant force and periodic collision of the surgeon can cause the lens part 201 to move, which requires the lens part 201 to be repositioned. This relocation delays the surgical procedure and annoys the surgeon.

The example stereo visualization camera 300 does not include an eyepiece and does not intend to receive contact from a surgeon once it is locked in place. This corresponds to one of the significantly lower chances of the stereo visualization camera 300 accidentally moving or colliding during the operation of the surgeon.

In order to promote the second eyepiece 208, the surgical operating microscope 200 must be equipped with a beam splitter 210, which may include a glass lens and a mirror contained in a precision metal tube. The use of a beam splitter 210 reduces the light received at the first eyepiece because some of the light is reflected to the second eyepiece 208. In addition, adding the second eyepiece 208 and the beam splitter 210 will increase the weight and bulkiness of the lens portion 201.

Compared with the surgical operating microscope 200, the stereo vision camera 300 only contains the optical path for the sensor, thereby reducing the weight and bulkiness. In addition, the optical sensor receives all incident light, because there is no need for a beam splitter to redirect part of the light. This means that the image received by the optical sensor of the exemplary stereo visualization camera 300 is as bright and clear as possible.

Certain models of surgical microscopes may enable the attachment of a video camera. For example, the surgical operating microscope 200 of FIG. 2 includes a single-image video camera 212 connected to an optical path via a beam splitter 214. The video camera 212 can be mono- or stereo, such as a Leica® TrueVision® 3D visualization system ophthalmology camera. The video camera 212 records an image received from the beam splitter 214 for display on a display monitor. The addition of the video camera 212 and the beam splitter 214 will further increase the weight of the lens portion 201. In addition, the beam splitter 214 consumes extra light destined for the eyepieces 206 and/or 208.

Each beam splitter 210 and 214 divides the incident light into three paths in a fractional manner, thereby removing the light from the surgeon's field of view. The surgeon’s eyes have limited low light sensitivity, so that the light from the surgical site presented to him/her must be sufficient to allow the surgeon to perform the procedure. However, a surgeon cannot always increase the intensity of light applied to a target site on a patient, especially in ophthalmic procedures. The eye of a patient has limited high light sensitivity before it becomes phototoxic. Therefore, the number and division of beam splitters and the amount of light that can be split from the first eyepiece 206 are restricted to enable the use of auxiliary devices 208 and 212.

The exemplary stereo vision camera 300 of FIGS. 3 to 6 does not include a beam splitter, so that the optical imaging sensor receives the full amount of light from a main objective lens assembly. This enables the use of sensors with low light sensitivity or even optical sensors with sensitivity beyond the wavelength of visible light. This is because the post-processing can make the image sufficiently bright and visible (and adjustable) for display On the monitor.

In addition, since the optical elements defining the optical path are independent in the stereo visualization camera 300, the optical elements can be controlled through the camera. This control allows the placement and adjustment of optical components to be optimized for a three-dimensional display rather than for the microscope eyepiece. This configuration of the camera allows control to be provided electronically from the camera control or from a remote computer. In addition, the control can be automatically provided through one or more programs on the camera 300, and the camera 300 is configured to adjust optical elements for maintaining focus while zooming or adjusting optical defects and/or false parallaxes. In contrast, the optical elements of the surgical operating microscope 200 are external to the video camera 212 and controlled only by operator input, which is generally optimized for viewing a target site through the eyepiece 206.

In a final comparison, the surgical operating microscope 200 includes an X-Y horizontal swing device 220 for moving a field of view or target scene. The X-Y horizontal swing device 220 is usually a large, heavy and expensive electromechanical module, because it must rigidly support the surgical lens portion 201 and move the surgical lens portion 201. In addition, moving the lens portion 201 will change the position of the surgeon to the new position of the eyepiece 206.

In contrast, the exemplary stereo visualization camera 300 includes a memory that, when executed, causes a processor to select the pixel data of the optical sensor to achieve XY horizontal swing across a wide pixel grid. . In addition, the exemplary stereo visualization camera 300 may include a small motor or actuator that controls a main objective optical element to change to a target position without moving the camera 300. distance. IV. Example optical components of stereo vision cameras

FIGS. 7 and 8 show diagrams illustrating optical elements in the exemplary stereoscopic visualization camera 300 of FIGS. 3 to 6 according to an exemplary embodiment of the present invention. Obtaining the left and right views of a target part to construct a three-dimensional image may seem relatively simple. However, without careful design and compensation, many stereo images have alignment problems between the left view and the right view. When viewing for a long period of time, alignment problems can cause confusion in the brain of an observer due to the difference between the left and right views. This confusion can cause headaches, fatigue, dizziness and even nausea.

The exemplary stereo vision camera 300 reduces (or eliminates) alignment problems by having a right optical path and a left optical path, in which some optical elements are independently controlled and/or adjusted while other left optical elements and right optical elements Fixed in a common carrier. In an exemplary embodiment, certain left and right zoom lenses can be fixed to a common carrier to ensure that the left and right magnifications are substantially the same. However, the front lens or the rear lens can be adjusted radially, rotationally, and axially and/or the front lens or the rear lens can be tilted to compensate for zoom magnification, visual defects and/or false parallax ( Such as the movement of a zoom repeat point). The compensation provided by the adjustable lens produces an almost perfectly aligned optical path within a full zoom magnification range.

Additionally or alternatively, pixel readout and/or reproduction techniques can be used to reduce (or eliminate) alignment problems. For example, a right image (recorded by a right optical sensor) can be adjusted up or down relative to a left image (recorded by a left optical sensor) to correct the vertical misalignment between the images. Similarly, a right image can be adjusted to the left or right relative to a left image to correct the horizontal misalignment between the images.

Figures 7 and 8 below show an exemplary configuration and positioning of an optical element that provides almost artifacts, false parallaxes, and distortion-free aligned optical paths. As discussed later, specific ones of the optical elements can be moved during calibration and/or use to further align the optical paths and remove any remaining distortion, false parallax, and/or defects. In the illustrated embodiment, the optical elements are positioned in two parallel paths to produce a left view and a right view. Alternative embodiments may include optical paths that are folded, deflected, or otherwise non-parallel.

The illustrated path corresponds to a human visual system, so that the left and right views displayed on a stereoscopic display appear to be separated by a distance forming a condensing angle of approximately 6 degrees, which is the same as that of an adult. The condensing angle of an object approximately 4 inches away from the eyes is equivalent, thus producing stereoscopic vision. In some embodiments, the image data generated from the left view and the right view are combined together on the display monitors 512 and 514 to generate a three-dimensional image of a target part or scene. Alternative embodiments include other stereoscopic displays, in which the left view is presented only to the left eye of a viewer and the corresponding right view is presented only to the right eye. In an exemplary embodiment for adjusting and verifying proper alignment and calibration, the two views are displayed overlappingly to both eyes.

A three-dimensional view is better than a single-image view because it more closely mimics the human visual system. A stereoscopic view provides depth perception, distance perception, and relative size perception to provide a real view of a target surgical site to a surgeon. For procedures such as retinal surgery, the stereoscopic view system is very important, because the surgical movement and the small force system make it impossible for the surgeon to feel it. Provides a three-dimensional view to help a surgeon's brain amplify tactile sensations and perceive depth when the brain senses even small movements.

FIG. 7 shows a side view of an exemplary stereo visualization camera 300 in which the housing 302 is transparent to expose optical elements. FIG. 8 shows a diagram illustrating an optical path provided by the optical element shown in FIG. 7. As shown in FIG. 8, the optical path includes a right optical path and a left optical path. The optical paths in FIG. 8 are shown from a viewing angle of the stereoscopic visualizing camera 300 when facing a forward direction and looking down. From this view, the left optical path appears on the upper right side of Fig. 8 and the right optical path appears on the left side.

The optical element shown in Figure 7 is part of the left optical path. It should be understood that the relationship position and arrangement of the right optical path with respect to the optical elements in FIG. 7 are generally identical to the left optical path. As mentioned above, the interpupillary distance between the centers of one of the optical paths is between 58 mm and 70 mm, which can be scaled to 10 mm to 25 mm. Each of the optical elements includes a lens with a specific diameter (for example, between 2 mm and 29 mm). Therefore, a distance between the optical elements themselves is between 1 mm and 23 mm, preferably about 10 mm.

The exemplary stereo visualization camera 300 is configured to capture images of a target site 700 (also referred to as a scene or field of view ("FOV") or target surgical site). The target site 700 includes an anatomical location on a patient. The target part 700 may also include laboratory biological samples, calibration sliders/templates, and so on. The image from the target area 700 is received at the stereo visualization camera 300 through a main objective lens assembly 702. The main objective lens assembly 702 includes a front working distance lens 408 (shown in FIG. 4) and a rear working distance lens 704. A. Example main objective lens assembly

The exemplary main objective lens assembly 702 may include any type of refraction assembly or reflection assembly. FIG. 7 shows the objective lens assembly 702 as an achromatic refraction assembly, in which the front working distance lens 408 is fixed and the rear working distance lens 704 is movable along the z-axis. The front working distance lens 408 may include a plano-convex ("PCX") lens and/or a meniscus lens. The rear working distance lens 704 may include an achromatic lens. In an example where the main objective lens assembly 702 includes an achromatic refraction assembly, the front working distance lens 408 may include a hemispherical lens and/or a meniscus lens. In addition, the rear working distance lens 704 may include an achromatic double lens, an achromatic double lens group, and/or an achromatic triple lens.

The magnification of the main objective lens assembly 702 is between 6x and 20x. In some cases, the magnification of the main objective lens assembly 702 may vary slightly based on a working distance. For example, the main objective lens assembly 702 may have a magnification of 8.9x for a working distance of 200 mm and a magnification of 8.75x for a working distance of 450 mm.

The exemplary rear working distance lens 704 is configured to be movable relative to the front working distance lens 408 to change a spacing therebetween. The distance between the lens 408 and the lens 704 determines the overall front focal length of the main objective lens assembly 702, and therefore the position of a focal plane. In some embodiments, the focal length is the distance between the lens 408 and the lens 704 plus one-half the thickness of the front working distance lens 408.

In summary, the front working distance lens 408 and the rear working distance lens 704 are configured to provide an infinite conjugate image for providing a best focus for the downstream optical image sensor. In other words, an object located exactly at the focal plane of the target part 700 will project its image at an infinite distance, and thus infinitely couple at a provided working distance. Generally speaking, the optical path of the object along the focal plane appears to be focused up to a specific distance. However, beyond a certain threshold distance, the object begins to appear blurry or out of focus.

FIG. 7 shows the working distance 706, which is the distance between an outer surface of the front working distance lens 408 and the focal plane of the target part 700. The working distance 706 may correspond to a corner field of view, where a longer working distance produces a wider field of view or a larger viewable area. The working distance 706 therefore sets the target part of the focus or a plane of the scene. In the illustrated example, the working distance 706 is adjustable from 200 mm to 450 mm by moving the rear working distance lens 704. In one example, when the working distance is 450 mm, the upstream zoom lens can be used to adjust the field of view between 20 mm×14 mm and 200 mm×140 mm.

The main objective lens assembly 702 shown in FIGS. 7 and 8 provides an image of the target site 700 for both the left optical path and the right optical path. This means that the width of the lenses 408 and 704 should be at least as wide as the left optical path and the right optical path. In an alternative embodiment, the main objective lens assembly 702 may include separate left and right front working distance lenses 408 and separate left and right rear working distance lenses 704. The width of each pair of individual working distance lenses can be between 1/4 and 1/2 of the width of the lenses 408 and 704 shown in FIGS. 7 and 8. In addition, each of the rear working distance lenses 704 may be independently adjustable.

In some embodiments, the main objective lens assembly 702 may be replaceable. For example, different main objective lens assemblies can be added to change a working distance range, a magnification, a numerical aperture, and/or refraction/reflection type. In these embodiments, the stereoscopic visualization camera 300 can change the positioning of the downstream optical element, the properties of the optical image sensor, and/or the image processing parameters based on which main objective lens assembly is installed. An operator can use one of the controls 305 in FIG. 3 and/or a user input device to specify which main objective lens assembly is installed in the stereo visualization camera 300. B. Example light source

To illuminate the target site 700, the exemplary stereo visualization camera 300 includes one or more light sources. Figures 7 and 8 show three light sources, which include a visible light source 708a, a near infrared ("NIR") light source 708b, and a near extreme ultraviolet ("NUV") light source 708c. In other examples, the stereo visualization camera 300 may include additional or fewer light sources (or no light sources). For example, NIR and NUV light sources can be omitted. The example light source 708 is configured to generate light that is projected onto the target scene 700. The generated light interacts with the target scene and reflects away from the target scene, wherein some of the light is reflected to the main objective lens assembly 702. Other examples may include external light sources or ambient light from the environment.

In addition to certain light having wavelengths outside the visible region, the exemplary visible light source 708a is also configured to output light in the human visible portion of the light spectrum. The NIR light source 708b is configured to output light mainly at a wavelength that slightly crosses the red part of the visible spectrum, which is also called "near infrared." The NUV light source 708c is configured to output light with a wavelength mainly in the blue part of the visible spectrum, which is called "near ultra-ultraviolet." The light spectrum output by the light source 708 is controlled by the respective controllers described below. A brightness of the light emitted by the light source 708 can be controlled by a switching rate and/or the applied voltage waveform.

7 and 8 illustrate that the visible light source 708a and the NIR light source 708b are directly provided to the target site 700 through the main objective lens assembly 702. As shown in Figure 8, visible light from visible light source 708a travels along visible path 710a. In addition, the NIR light from the NIR light source 708b propagates along the NIR path 710b. Although the light sources 708a and 708b are shown behind the main objective lens assembly 702 (relative to the target site 700), in other examples, the light sources 708a and 708b may be provided in front of the main objective lens assembly 702. In an embodiment, the light sources 708a and 708b may be provided on an outer side of the housing 302 and face the target site 700. In still other embodiments, for example, the light source 708 may be provided separately from the stereoscopic visualization camera 300 using a Koeher illumination setting and/or a dark field illumination setting.

Compared with the light sources 708a and 708b, a deflection element 712 (eg, a beam splitter) uses an epi-illumination setting to reflect the NUV light from the NUV light source 708c to the main objective lens assembly 702. The deflection element 712 may be coated or otherwise configured to reflect only light outside the NUV wavelength range, thereby filtering the NUV light. NUV light from NUV light source 708c travels along NUV path 710c.

In some embodiments, NIR light source 708b and NUV light source 708c can be used with an excitation filter to further filter light that may not be blocked by the filter (eg, filter 740). The filters may be placed in front of the light sources 708b and 708c before the main objective lens assembly 702 and/or after the main objective lens assembly. The light from the NUV light source 708c and the NIR light source 708b after filtering includes the wavelength that excites the fluorescence in the fluorescent part 914 (shown in FIG. 9) of an anatomical object. In addition, the light from the NUV light source 708c and the NIR light source 708b may include wavelengths that are not in the same range as their light emitted by the fluorescent part 914 after being filtered.

The projection of the light from the light source 708 through the main objective lens assembly provides the benefit of changing the illuminated field of view based on the working distance 706 and/or the focal plane. Since the light passes through the main objective lens assembly 702, the angle of the light projection is changed based on the working distance 706 and corresponds to the angular field of view. This configuration therefore ensures that the field of view is properly illuminated by the light source 708 regardless of working distance or magnification. C. Example deflection element

The exemplary deflection element 712 illustrated in FIGS. 7 and 8 is configured to transmit light of a specific wavelength from the NUV light source 708 c to the target site 700 through the main objective lens assembly 702. The deflection element 712 is also configured to reflect light received from the target site 700 to downstream optical elements, which include a front lens group 714 for zooming and recording. In some embodiments, the deflection element 712 can filter the light received from the target part 700 through the main objective lens assembly 702 so that light of a specific wavelength reaches the front lens group 714.

The deflection element 712 may include any type of mirror or lens to reflect light in a prescribed direction. In one example, the deflection element 712 includes a dichroic mirror or filter, which has different reflection and transmission characteristics at different wavelengths. The stereo vision camera 300 of FIGS. 7 and 8 includes a single deflection element 712 that provides light for both the right optical path and the left optical path. In other examples, the camera 300 may include separate deflection elements for each of the right optical path and the left optical path. In addition, a separate deflection element can be provided for the NUV light source 708c.

FIG. 9 shows a diagram of the deflection element 712 of FIGS. 7 and 8 according to an exemplary embodiment of the present invention. For brevity, the main objective lens assembly 702 is not shown. In this example, the deflection element 712 includes two parallel surfaces 902 and 904 for transmitting and reflecting light of a specific wavelength. The parallel planes 902 and 904 are set at an angle of 45° with respect to the left optical path and the right optical path (denoted as path 906). The 45° angle is selected so that the angle causes the reflected light to propagate from the transmitted light at a 90° angle, thus providing the best separation without causing the separated light to be detected in the downstream front lens group 714. In other embodiments, the angle of the deflection element 712 can be between 10 degrees and 80 degrees without unintentionally propagating light of undesired wavelengths.

An example NUV light source 708c is located behind the deflection element 712 (relative to the target site 700). The light from the light source 708c travels along the path 908 and contacts the deflection element 712. The NUV light around the main wavelength range of the NUV light source 708c is transmitted through the deflecting element 712 to the target site 700 along the path 910. The light from the NUV light source 708c having a wavelength higher (and lower) than the main wavelength range of the NUV light source 708c is reflected along the path 912 to an optical slot or unused area of the housing 302.

When the NUV light reaches the target part 700, one or more fluorescent parts 914 of an anatomical object absorb the NUV light. In some cases, the anatomical object may have been injected with a contrast agent configured to absorb NUV light and emit light with a different dominant wavelength. In other cases, the anatomical object can naturally absorb NUV light and emit light with a different dominant wavelength. At least some of the light reflected or emitted by the fluorescent part 914 travels along the path 916 until it contacts the deflection element 712. Most of the light reflects off the surface 904 along the path 906 to reach the front lens group 714. A part of the light (including the NUV light around the main wavelength range of the NUV light source 708c) is transmitted through the deflecting element 712 along the path 918 to an optical slot or unused area of the housing 302. The deflection element 712 shown in FIG. 9 therefore uses a region of the frequency spectrum to achieve the optical stimulation of a fluorescent agent at the target site 700 while preventing a lot of stimulation light from traveling to the downstream front lens group 714.

It should be understood that the reflectance and transmittance characteristics of the deflection element 712 can be changed to meet other optical spectrum requirements. In some examples, the housing 302 may include a slit that enables the deflection element 712 and/or the NUV light source 708c to be replaced based on the desired light reflectance and transmittance characteristics. It should also be understood that a first path inside the deflection element 712 between the path 908 and the path 910 and a second path inside the deflection element 712 between the path 916 and the path 918 are each angled to schematically It represents the refraction of light, which is because the light travels between the air and the inside of the deflection element 712. The angle shown is not intended to represent the actual reflection angle. D. Example zoom lens

The exemplary stereoscopic visualization camera 300 of FIGS. 7 and 8 includes one or more zoom lenses to change a focal length and angle of view of the target site 700 to provide zoom magnification. In the illustrated example, the zoom lens includes a front lens group 714, a zoom lens assembly 716, and a lens barrel group 718. It should be understood that in other embodiments, the front lens group 714 and/or the lens barrel group 718 may be omitted. Alternatively, the zoom lens may include additional lenses to provide additional magnification and/or image resolution.

The front lens group 714 includes a right front lens 720 for the right optical path and a left front lens 722 for the left optical path. The lenses 720 and 722 may each include a positive condenser lens to guide light from the deflecting element 712 to the respective lenses in the zoom lens assembly 716. A lateral position of the lenses 720 and 722 is therefore defined as a light beam propagating from the main objective lens assembly 702 and the deflection element 712 to the zoom lens assembly 716.

One or both of the lenses 720 and 722 may be radially adjustable to match the optical axis of the left optical path and the right optical path. In other words, one or both of the lenses 720 and 722 can be moved left and right and/or up and down in a plane incident on the optical path. In some embodiments, one or more of the lenses 720 and 722 can be rotated or tilted to reduce or eliminate image optical defects and/or false parallax. Moving either or both of lenses 720 and 722 during zooming can cause the zoom repeat point ("ZRP") of each optical path to appear to remain fixed to a user. In addition to radial movement, one or both of the front lenses 720 and 722 can also be moved axially (along respective optical paths) to match the magnification of the optical paths.

The exemplary zoom lens assembly 716 forms an afocal zoom system for changing the size of a field of view (eg, a linear field of view) by changing the size of one of the light beams propagating to the lens barrel assembly 718. The zoom lens assembly 716 includes a front zoom lens group 724 having a right front zoom lens 726 and a left front zoom lens 728. The zoom lens assembly 716 also includes a rear zoom lens group 730 having a right rear zoom lens 732 and a left rear zoom lens 734. The front zoom lenses 726 and 728 may be positive condenser lenses, and the rear zoom lenses 732 and 734 include negative astigmatism lenses.

The size of one of the image beams for each of the left optical path and the right optical path is determined based on a distance between the front zoom lenses 726 and 728, the rear zoom lenses 732 and 734, and the lens barrel group 718. Generally speaking, the size of the optical paths decreases as the rear zoom lenses 732 and 734 move toward the lens barrel group 718 (along the respective optical paths), thereby reducing the magnification. In addition, when the rear zoom lenses 732 and 734 move toward the lens barrel group 718, the front zoom lenses 726 and 728 can also move toward (or away from) the lens barrel group 718 (such as in a parabolic arc) to maintain the focal plane in The position on the target site 700, thus maintaining the focus.

The front zoom lenses 726 and 728 may be included in a first carrier (for example, the front zoom group 724), and the rear zoom lenses 732 and 734 are included in a second carrier (for example, the rear zoom group 730). Each of the carriers 724 and 730 can be moved on a track (or rail) along an optical path, so that the left magnification and the right magnification are changed at the same time. In this embodiment, any slight difference in magnification between the left optical path and the right optical path can be corrected by moving the right front lens 720 and/or the left front lens 722. Additionally or alternatively, a right lens barrel 736 and/or a left lens barrel 738 of the lens barrel group 718 can be moved axially.

In an alternative embodiment, the front right zoom lens 726 may be moved axially separately from the front left zoom lens 728. In addition, the right rear zoom lens 732 and the left rear zoom lens 734 can be axially moved separately. Moving alone can enable small magnification differences to be corrected by the zoom lens assembly 716, especially when the front lens group 714 and the lens barrel group 718 are fixed along the optical path. In addition, in some embodiments, the right front zoom lens 726 and/or the left front zoom lens 728 may be radially and/or rotationally adjusted (and/or tilted) to maintain a ZRP in the optical path. Location. In addition or alternatively, the right rear zoom lens 732 and/or the left rear zoom lens 734 can be adjusted radially and/or rotationally (and/or tilted) to maintain a ZRP clearly visible in the optical path Location.

The exemplary lens barrel group 718 includes a right lens barrel 736 and a left lens barrel 738. In addition to the zoom lens assembly 716, the right lens barrel 736 and the left lens barrel 738 are also part of an afocal zoom system. Lenses 736 and 738 may include positive condenser lenses configured to straighten or focus a light beam from the zoom lens assembly 716. In other words, the lenses 736 and 738 focus the infinity coupling output of the zoom lens assembly 716.

In some instances, the lens barrel group 718 is fixed in the housing 302 radially and axially. In other examples, the lens barrel group 718 may be axially movable along the optical path to provide increased magnification. Additionally or alternatively, each of the lenses 736 and 738 may be radially and/or rotationally adjustable (and/or tilted) to, for example, correct the front lens group 714, the front zoom lens The optical properties between the left lens and the right lens of the group 724 and/or the rear zoom lens group 730 are poor (depending on manufacturing or natural glass deviation).

In summary, the exemplary front lens group 714, the zoom lens assembly 716, and the lens barrel group 718 are configured to achieve an optical zoom between 5X and about 20X, preferably one of the diffraction-limited resolutions. Zoom level. In some embodiments, if the image quality can be impaired, the front lens group 714, the zoom lens assembly 716, and the lens barrel group 718 can provide a higher zoom range (for example, 25X to 100X). In these embodiments, the stereo vision camera 300 can output a message to an operator indicating that a selected optical range exceeds an optical range and suffers from a degradation of image quality.

In some embodiments, the lenses of the front lens group 714, the zoom lens assembly 716, the lens barrel group 718, and/or the main objective lens assembly 702 can each be made of a plurality of optical sub-elements using materials that balance the optical distortion parameters of each other. The structure is a double lens. This dual structure reduces chromatic aberrations and optical aberrations. For example, the front working distance lens 408 and the rear working distance lens 704 may each be configured as a double lens. In another example, the front lenses 720 and 722, the front zoom lenses 726 and 728, the rear zoom lenses 732 and 734, and the lens barrels 736 and 738 may each include a double lens.

In still additional embodiments, the lenses of the front lens group 714, the zoom lens assembly 716, the lens barrel group 718, and/or the main objective lens assembly 702 may be tuned in different ways and/or have different properties to provide different capabilities. Two parallel optical paths. For example, the right lens of the zoom lens assembly 716 can be selected to provide 5X to 10X optical zoom for the right optical path, and the left lens of the zoom lens assembly 716 can be selected to provide 15X to 20X optical zoom for the left optical path. This configuration allows two different magnifications to be displayed simultaneously and/or on the same screen, albeit in a single image view. E. Example filters

The exemplary stereo visualization camera 300 of FIGS. 7 and 8 includes one or more optical filters 740 (or filter assemblies) to selectively transmit light of desired wavelengths. Figure 8 shows that a single filter 740 can be applied to the right optical path and the left optical path. In other examples, each of the optical paths may have a separate filter. The inclusion of separate filters enables (for example) light of different wavelengths to be filtered simultaneously from the left optical path and the right optical path, which enables (for example) fluorescent images to be displayed along with visible light images.

Figure 7 shows that the filter 740 includes a wheel that rotates about its axis of rotation. In the illustrated embodiment, the filter 740 can accommodate three different optical filter pairs. However, in other embodiments, the filter 740 may include additional or fewer filter pairs. Generally speaking, the light received from the target site 700 at the filter 740 contains a broad wavelength spectrum. The lenses of the main objective lens assembly 702, the front lens group 714, the zoom lens assembly 716, and the lens barrel group 718 are configured to pass light of a relatively wide bandwidth including a wavelength of interest to the operator and an undesirable wavelength. In addition, the downstream optical image sensor is sensitive to specific wavelengths. The example filter 740 therefore passes certain parts of the light spectrum and blocks certain parts of the light spectrum to achieve different desirable characteristics.

As a round, the filter 740 includes a mechanical device capable of changing position at approximately 4 times per second. In other embodiments, the filter 740 may include a digital micro-mirror, which can change the direction of a light path at a video frame rate such as 60 times per second. In these other embodiments, each of the left optical path and the right optical path will include a micro mirror. The left and right micro mirrors can be switched synchronously or simultaneously.

In some embodiments, the filter 740 can be synchronized to the light source 708 to achieve "time-interleaved" multi-spectral imaging. For example, the filter 740 may include an infrared cut filter, a near infrared band pass filter, and a near ultra ultraviolet cut filter. Different filter types are selected to be effective for the different frequency spectrum of the light source 708 and the reflectance and transmittance characteristics of the deflection element 712 to pass light of a specific desired wavelength at a predetermined time.

In one mode, the filter 740 and the light source 708 are configured to provide a visible light mode. In this mode, the visible light source 708a transmits light from the visible region to the target site 700, and some of the light is reflected to the main objective lens assembly 702. The reflected light may include some light beyond the visible spectrum, which may affect the optical image sensor. The visible light is reflected by the deflection element 712 and passes through the front lens group 714, the zoom lens assembly 716, and the lens barrel group 718. In this example, the filter 740 is configured to apply an infrared cut filter or a near ultra-ultraviolet cut filter to the optical path to remove light beyond the visible spectrum, so that only light in the visible spectrum passes through to a final optical path. Group 742 and an optical image sensor 744.

In another mode, the filter 740 and the light source 708 are configured to provide a narrow wavelength of fluorescent light to the optical sensor 744. In this mode, the NUV light source 708c transmits light from the dark blue region of the spectrum to the target site 700. The deflection element 712 allows the desired light in the dark blue area to pass through while reflecting the undesired light. The deep blue light interacts with the target site 700 to cause fluorescence to be emitted. In some instances, delta-aminolevulinic acid ("5ala") and/or protoporphyrin IX are applied to the target site 700 to cause fluorescence to be emitted when deep blue light is received. In addition to the reflected deep blue light and some visible light, the main objective lens assembly 702 also receives fluorescent light. The deep blue light leaves the right optical path and the left optical path through the deflection element 712. Therefore, only visible light and fluorescent light pass through the front lens group 714, the zoom lens assembly 716, and the lens barrel group 718. In this example, the filter 740 is configured to apply a near ultra-ultraviolet cut filter to the optical path to remove light beyond the desired fluorescent spectrum (including visible light and any remaining NUV deep blue light). Therefore, only a narrow wavelength of fluorescent light reaches the optical image sensor 744, which enables the fluorescent light to be more easily detected and distinguished based on the relative intensity.

表 1 In yet another mode, the filter 740 and the light source 708 are configured to provide indigo green ("ICG") fluorescence to the optical sensor 744. In this mode, the NIV light source 708b transmits light in the far red region of the visible spectrum (which is also regarded as near infrared) to the target site 700. In addition, the visible light source 708a transmits visible light to the target scene 700. Visible light and far-red light are absorbed by the material with ICG at the target site, and the target site then emits highly stimulated fluorescence in one of the more distant red regions. In addition to the reflected NIR light and visible light, the main objective lens assembly 702 also receives fluorescent light. The light is reflected by the deflection element 712 to the front lens group 714, the zoom lens assembly 716, and the lens barrel group 718. In this example, the filter 740 is configured to apply a near-infrared bandpass filter to the optical path to remove light (including visible light and at least some NIR light) outside the desired fluorescent spectrum. Therefore, only the fluorescent light in the farther red region reaches the optical image sensor 744, which enables the fluorescent light to be more easily detected and distinguished based on the relative intensity.

Table 1

Table 1 above shows a summary of the different possible combinations of light sources and filters used to cause light of a specific desired wavelength to reach the optical light sensor 744. It should be understood that other types of filters and/or light sources can be used to further increase the different types of light received at the image sensor 744. For example, a bandpass filter configured to pass light of a narrow wavelength can be used to correspond to a specific biological stain or contrast agent applied to the target site 700. In some examples, the filter 740 may include a cascade filter or more than one filter to enable light from two different ranges to be filtered. For example, an infrared cut filter and a near ultra-ultraviolet cut filter can be applied to a first filter 740, so that only visible light in a desired wavelength range can pass through to the optical sensor 744.

In other embodiments, separate filters 740 can be used for the left optical path and the right optical path. For example, a right filter may include an infrared cut filter, and a left filter may include a near infrared pass filter. This configuration makes it possible to view the target site 700 in the visible wavelength at the same time as the IGC green fluorescent wavelength. In another example, a right filter may include an infrared cut filter, and a left filter may include a near ultra-ultraviolet cut filter. In this configuration, the target site 700 can be displayed in visible light simultaneously with 5ALA fluorescence. In these other embodiments, the right image stream and the left image stream can still be combined into a stereo view, which is combined with a view of the target site 700 in visible light to provide a fluorescent view of a specific anatomical structure. F. Example final optical element group

The exemplary stereo visualization camera 300 of FIGS. 7 and 8 includes a final optical element group 742 to focus the light received from the filter 740 onto the optical image sensor 744. The last optical element group 742 includes a right last optical element 745 and a left last optical element 747, which may each include a positive condenser lens. In addition to focusing the light, the optical elements 745 and 747 can also be configured to correct small aberrations in the right and left optical paths before the light reaches the optical image sensor 744. In some examples, the lenses 745 and 747 may be movable radially and/or axially to correct the magnification and/or the magnification caused by the front lens group 714, the zoom lens assembly 716, and the lens barrel group 718. Focus aberration. In one example, the left last optical element 747 can be moved radially, while the right last optical element 745 is fixed to remove ZRP movement during magnification changes. G. Example image sensor

The exemplary stereo visualization camera 300 of FIGS. 7 and 8 includes an image sensor 744 to acquire and/or record the incident light received from the last optical element group 742. The image sensor 744 includes a right optical image sensor 746 to capture and/or record light propagating along the right optical path and a left optical image sensor 748 to capture and/or record the light propagating along the left optical path Light. For example, each of the left optical image sensor 748 and the right optical image sensor 746 includes a complementary metal oxide semiconductor ("CMOS") sensing element, an N-type metal oxide semiconductor ("NMOS") And/or semiconductor charge coupled device ("CCD") sensing element. In some embodiments, the left optical sensor 748 and the right optical sensor 746 are identical and/or have the same properties. In other embodiments, the left optical sensor 748 and the right optical sensor 746 include different sensing elements and/or properties to provide variability. For example, the right optical image sensor 746 (using a first color filter array) can be configured to be more sensitive to blue fluorescence, while the left optical image sensor 748 (using a second color filter) The array) is configured to be more sensitive to visible light.

FIG. 10 shows an example of the right optical image sensor 746 and the left optical image sensor 748 of the image sensor 744 according to an exemplary embodiment of the present invention. The right optical image sensor 746 includes a first two-dimensional grid or matrix 1002 of light sensing elements (eg, pixels). In addition, the left optical image sensor 748 includes a second two-dimensional pixel grid 1004 that is a photo sensor element. Each of the pixels includes a filter that allows only a specific wavelength of light to pass through and thus touch the volt detector. The filters for different colors are spread across the sensors 746 and 748 to provide light detection for all wavelengths across the grid. The light detector can be sensitive to visible light and additional ranges above and below the visible spectrum.

The light sensing elements of the grids 1002 and 1004 are configured to record a range of light wavelengths as one of the target locations 700 in the field of view. Light incident on a light sensing element causes a charge to accumulate. The charge is read to determine the amount of light received at the sensing element. In addition, since the filter characteristics of the sensing element are known to be within manufacturing tolerances, the wavelength range of the received light is known. The representation of the target site 700 is directed to the light sensing element so that the grids 1002 and 1004 for the respective optical image sensors 746 and 748 spatially sample the target site 700. The resolution of spatial sampling is a parameter that affects image quality and parity.

The number of pixels shown in the pixel grids 1002 and 1004 in FIG. 10 does not represent the actual number of pixels in the optical image sensors 746 and 748. Alternatively, the sensors usually have a resolution between 1280×720 pixels and 8500×4500 pixels (preferably about 2048×1560 pixels). However, not all pixels of the grids 1002 and 1004 are selected for image transmission. Alternatively, a subgroup or pixel group of grids 1002 and 1004 is selected for transmission. For example, in FIG. 10, the pixel group 1006 is selected from the pixel grid 1002 for transmission as a right image and the pixel group 1008 is selected from the pixel grid 1004 for transmission as a left image. As illustrated in the figure, regarding the respective pixel grids 1002 and 1004, the pixel group 1006 does not need to be located in the same position as the pixel group 1008. Separate control of the pixel groups 1006 and 1008 enables the left and right images to be aligned and/or corrected for image defects and/or false parallax, such as moving the ZRP.

Selecting a pixel group from a pixel grid enables a portion of the pixel grid to be selected to compensate for image defects/false parallax and/or to make the right optical image and the left optical image more aligned. In other words, the pixel group can be moved or adjusted (in real time) relative to the pixel grid to improve image quality by reducing or eliminating false parallax. Another option is to actually move either or both of the left view and the right view of the stereoscopic image in the image processing pipeline (for example, during the rendering of the view for display) to achieve the same effect. It can also actually correct the rotation misalignment of the sensor. It is also possible to move a pixel group across a pixel grid during use to provide the appearance of horizontal swinging of the field of view. In an example, a pixel group or window of 1920×1080 pixels can be selected from a pixel grid of 2048×1560 pixels. During setup and/or use, the software/firmware can control the position of the pixel window or group and move the position. Therefore, the resolution of the optical image sensors 746 and 748 is specified based on the number of pixels in the length and width directions of the pixel group or window. 1. Use an example image sensor for color sensing

As mentioned above, the optical sensing elements 746 and 748 include pixels with different filters to detect light of a specific color. For example, some pixels are covered with a filter that passes mainly red light, some pixels are covered with a filter that passes mainly green light, and some pixels are covered with a filter that passes mainly blue light. In some embodiments, a Bayer pattern is applied to the pixel grids 1002 and 1004. However, it should be understood that in other embodiments, a different color pattern optimized for a specific wavelength of light may be used. For example, a broadband filter or a near-infrared filter can be used to replace one of the green filters in each sensing area, thereby expanding the sensing spectrum.

The Bayer pattern is implemented by grouping two columns × two rows of pixels and covering one pixel with a red filter, one pixel with a blue filter, and two pixels with a green filter (each in a checkerboard shape) pattern). Therefore, the resolutions of red and blue are each a quarter of the entire sensing area of interest, and the resolution of green is one-half of the entire sensing area of interest.

Green can be assigned to one-half of the sensing area so that the optical image sensors 746 and 748 operate as a brightness sensor and mimic the human visual system. In addition, red and blue mimic the chromaticity sensor of the human visual system, but they are not as critical as green sensing. Once a certain amount of red, green, and blue is determined for a specific area, other colors in the visible spectrum can be determined by averaging the red, green, and blue values, as discussed in conjunction with the Bayer in Figure 16 below. As discussed in program 1580a.

In some embodiments, the optical image sensors 746 and 748 may use stacked components instead of filters to sense colors. For example, the sensing element may include red, green, and blue sensing elements stacked vertically inside the area of a pixel. In another example, the beam splitter is specially coated one or more times (usually at least twice to produce three component colors, called "3 slices") to split the incident light into several components. The sensing element is placed in each of the paths of the split beam. Other sensor types use a different pattern, such as replacing one of the green filters with a broadband filter or a near-infrared filter, thereby expanding the sensing possibilities of digital surgical microscopes. 2. Use an example image sensor to sense light beyond the visible range

The example sensing element filters of the optical image sensors 746 and 748 are configured to also pass near-infrared light in a range detectable by the sensing element. This enables the optical image sensors 746 and 748 to detect at least some light outside the visible range. This sensitivity can reduce the image quality in the visible part of the spectrum because it "washes" the image, which reduces the contrast in many types of scenes and negatively affects the color quality. Therefore, the filter 740 can use an infrared cut filter to block near-infrared wavelengths while allowing visible wavelengths to pass to the optical image sensors 746 and 748.

However, this near-infrared sensitivity may be desirable. For example, a fluorescent agent such as ICG may be introduced to the target site 700. ICG is excited or activated by visible light or light of other wavelengths and emits fluorescence in the near-infrared range. As mentioned above, NIR light source 708b provides NIR light and visible light source 708a provides visible light to excite reagents with ICG. The emitted light is further along the red spectrum, and a near-infrared bandpass or high-pass filter can be used to pass the red spectrum through the filter 740. The light from the red spectrum is then detected by optical image sensors 746 and 748. By matching the spectral characteristics of the filter 740 with the expected behavior of the light source 708 and the fluorescent agent, reagents and biological structures (such as blood containing reagents) can be distinguished at the target site 700 from other structures that do not contain reagents.

It should be noted that in this example, the NIR light source 708b has a dominant wavelength that is different from the near-infrared filter in the filter 740. Specifically, the NIR light source 708b has a dominant wavelength of approximately 780 nanometers ("nm") (most of the light output spectrum exists around it). In contrast, the near-infrared filter of the filter 740 transmits light having a wavelength in a range of approximately 810 nm to 910 nm. Both the light from the NIR light source 708b and the light passing through the filter 740 are "near infrared" wavelengths. However, the wavelengths of the light are separated so that the exemplary stereo visualization camera 300 can use the light source 708 for stimulation and the optical image sensor 744 for detection while filtering the excitation light. This configuration therefore enables the use of fluorescent agents.

In another embodiment, the reagent can be excited in the blue, violet, and near ultra-ultraviolet regions and fluoresce in the red region. An example of such an agent includes the accumulation of porphyrin in malignant glioma caused by the introduction of 5ALA. In this example, it is necessary to filter out the blue light while passing the rest of the spectrum. A near ultra-ultraviolet cut filter is used in this scenario. As in the “near infrared” case discussed above, the NUV light source 708c has a dominant wavelength that is different from the near ultra-ultraviolet cut filter in the filter 740. H. Exemplary lens carrier

Section IV (D) mentioned above that at least some of the lenses of the front lens group 714, the zoom lens assembly 716, and/or the lens barrel group 718 can move along the rails in one or more carriers. For example, the front zoom lens group 724 may include a carrier for axially moving the front zoom lenses 726 and 728 together.

Figures 11 and 12 show diagrams of exemplary carriers according to exemplary embodiments of the present invention. In FIG. 11, the carrier 724 includes a right front zoom lens 726 and a left front zoom lens 728 in a supporting structure 1102. The carrier 724 includes a rail holder 1104 that is configured to be movably connected to the rail 1106. A force “F” is applied to the actuating section 1108 to cause the carrier 724 to move along the rail 1106. The force "F" can be applied by a lead screw or other linear actuation device. As illustrated in FIG. 11, the force “F” is applied at one of the offsets of the carrier 724. The friction between the rail 1106 and the carrier 724 causes the support structure 1102 to slightly move around the Y axis shown in FIG. 11 by a moment _My . This slight movement may cause the right front zoom lens 726 and the left front zoom lens 728 to shift slightly in opposite directions, resulting in false parallax, which is an error of a parallax between views of a stereoscopic image.

FIG. 12 shows another example of the carrier 724. As shown in FIG. In this example, the force “F” is applied symmetrically at the central structure 1202, which is connected to the rail holder 1104 and the support structure 1102. The force "F" is generated to cause the carrier 724 to rotate or slightly move around the X axis shown in FIG. 12 by a moment M _x . This rotational movement causes the right front zoom lens 726 and the left front zoom lens 728 to shift in the same direction by the same degree of movement, thereby reducing (or eliminating) the occurrence of false parallax.

Although FIGS. 11 and 12 show the lenses 726 and 728 in a carrier, in other embodiments the lenses 726 and 728 may each be in a carrier. In these examples, each lens will be on a separate track or rail. A separate lead screw can be provided for each of the lenses to provide independent axial movement along the respective optical path. I. Example flexure

Section IV(D) mentioned above that at least some of the lenses of the front lens group 714, the zoom lens assembly 716, and/or the lens barrel group 718 can be moved, rotated and/or tilted radially. Additionally or alternatively, the optical image sensors 746 and 748 can be axially moved and/or tilted relative to their respective incident optical paths. The axial and/or tilt movement can be provided by one or more flexures. In some examples, the flexures may be cascaded, such that a first flexure provides movement in a first direction and a separate flexure provides independent movement in a second direction. In another example, a first flexure provides tilt along a pitch axis and a single flexure provides tilt along a side tilt axis.

FIG. 13 shows a diagram of an exemplary double flexure 1300 according to an exemplary embodiment of the present invention. The flexure 1300 illustrated in FIG. 13 is used for the optical image sensor 744 and is configured so that the right optical image sensor 746 and the left optical image sensor 748 are independently along their respective optical axes Move for the purpose of final focusing. The flexure 1300 includes a supporting beam 1301 for connecting to the housing 302 of the exemplary stereo visualization camera 300 and providing a rigid base for actuation. The flexure 1300 also includes a beam 1302 for each channel (for example, the sensors 746 and 748), and the beam 1302 is rigid in all directions (except the direction of motion 1310). The beam 1302 is connected to a flexure hinge 1303, and the flexure hinge 1303 enables the beam 1302 to move in a movement direction 1310 (in this example, a parallelogram translation).

The actuator device 1304 deflects the beam 1302 a desired distance in the desired direction. The actuator device 1304 includes a pushing screw 1306 and a pulling screw 1308 for each channel. The pushing screw 1306 and the pulling screw 1308 apply opposite forces to the beam 1302, thereby causing the flexure hinge 1303 to move. For example, the beam 1302 can be moved inward by rotating the pushing screw 1306 to push the beam 1302. The flexure 1300 illustrated in FIG. 13 is configured to move the right optical image sensor 746 and the left optical image sensor 748 independently along their optical axes axially.

After the beam 1302 is flexed into a desired position, a locking mechanism is engaged to prevent further movement, thereby forming a rigid column. The locking mechanism includes a pushing screw 1306 and its respective concentric pulling screw 1308. The pushing screw 1306 and its respective concentric pulling screw 1308 form a large relative force that generates the rigid column of the beam 1302 when tightened.

Although optical image sensors 746 and 748 are shown connected to the same flexure 1300, in other examples, the sensors may be connected to a separate flexure. For example, returning to FIG. 8, the right optical image sensor 746 is connected to the flexure 750 and the left optical image sensor 748 is connected to the flexure 752. The use of separate flexures 750 and 752 enables the optical image sensors 746 and 748 to be individually adjusted to, for example, align the left and right optical views and/or reduce or eliminate false parallax.

In addition, although FIG. 13 shows the image sensors 746 and 748 connected to the flexure 1300, in other examples, the front lens group 714, the zoom lens assembly 716, the lens barrel group 718, and/or the last optical element group The lens of 742 can alternatively be connected to an alternative or additional flexure. In some examples, each of the right lens and the left lens of the front lens group 714, the zoom lens assembly 716, the lens barrel group 718, and/or the last optical element group 742 can be connected to a separate flexure 1300 to provide independent radial, rotation and/or tilt adjustment.

The flexure 1300 can provide a motion resolution of less than one micron. As a result of the very fine motion adjustment, the images from the right optical path and the left optical path may have alignment accuracy of one pixel or even one pixel for a 4K display monitor. This accuracy is viewed on each display 512, 514 by overlaying the left and right views and observing the two views with two eyes instead of a real stereoscope.

In some embodiments, the flexure 1300 may include the flexure disclosed in US Patent No. 5,359,474 entitled "SYSTEM FOR THE SUB-MICRON POSITIONING OF A READ/WRITE TRANSDUCER ", the full text of which is based on The way of citation is incorporated into this article. In still other embodiments, the lenses of the front lens group 714, the zoom lens assembly 716, the lens barrel group 718, and/or the last optical element group 742 may be fixed in a radial direction. Alternatively, a deflection element (for example, a mirror) having an adjustable deflection direction in an optical path can be used to manipulate the right optical path and/or the left optical path to adjust alignment and/or false parallax. Additionally or alternatively, a tilt/shift lens can be provided in the optical path. For example, an adjustable wedge lens can be used to control the tilt of one of the optical axes. In additional embodiments, the lenses of the front lens group 714, the zoom lens assembly 716, the lens barrel group 718, and/or the last optical element group 742 may include dynamic lenses with parameters that can be changed electronically. For example, the lenses may include Varioptic liquid lenses produced by Invenios France SAS. V. Example processor for stereo vision camera

The exemplary stereo visualization camera 300 is configured to record image data from the right optical path and the left optical path and output the image data to the monitor 512 and/or 514 for display as a stereo image. FIG. 14 shows a diagram of a module of an exemplary stereo visualization camera 300 for acquiring and processing image data according to an exemplary embodiment of the present invention. It should be understood that these modules illustrate operations, methods, algorithms, routines, and/or steps performed by specific hardware, controllers, processors, drivers, and/or interfaces. In other embodiments, the modules can be combined, further divided, and/or removed. In addition, one or more of these modules (or parts of a module) may be provided outside the stereo visualization camera 300, such as in a remote server, computer, and/or distributed computing environment.

In the illustrated embodiment of FIG. 14, the components 408, 702 to 750, and 1300 in FIGS. 7 to 13 are collectively referred to as an optical element 1402. The optical element 1402 (specifically, the optical image sensors 746 and 748) are communicatively coupled to an image capturing module 1404 and a motor and illumination module 1406. The image capture module 1404 is communicatively coupled to an information processor module 1408, and the information processor module 1408 can be communicatively coupled to an external user input device 1410 and one or more display monitors 512 and/ Or 514.

The example image capture module 1404 is configured to receive image data from the optical image sensors 746 and 748. In addition, the image capturing module 1404 can define pixel groups 1006 and 1008 in respective pixel grids 1002 and 1004. The image capture module 1404 can also specify image recording properties, such as frame rate and exposure time.

The example motor and lighting module 1406 is configured to control one or more motors (or actuators) to change one or more of the optical elements 1402 in the radial, axial, and/or tilt positions. For example, a motor or actuator can rotate a drive screw to move the carrier 724 along the track 1106, as shown in FIGS. 11 and 12. A motor or actuator can also rotate the pushing screw 1306 and/or pulling the screw 1308 of the flexure 1300 in FIG. 13 to adjust a lens and/or a radial, axial, or tilt position of the optical image sensor. The motor and lighting module 1406 may also include a driver for controlling the light source 708.

The example information processor module 1408 is configured to process image data for display. For example, the information processor module 1408 can provide color correction of the image data, filter defects from the image data, and/or reproduce the image data for stereo display. The information processor module 1408 can also calibrate the stereoscopic camera 300 by executing one or more calibration routines in the following manner: providing instructions for performing prescribed adjustments to optical components to the image capturing module 1404 and/or the motor And the light module 1406. The information processor module 1408 can further determine instructions for improving image alignment and/or reducing false parallax and provide these instructions to the image capturing module 1404 and/or the motor and illumination module 1406 in real time.

The example user input device 1410 may include a computer to provide instructions for changing the operation of the stereo visualization camera 300. The user input device 1410 may also include controls for selecting parameters and/or features of the stereo visualization camera 300. In one embodiment, the user input device 1410 includes the control arm 304 of FIG. 3. The user input device 1410 can be hard-wired to the information processor module 1408. Additionally or alternatively, the user input device 1410 is coupled to the information processor module 1408 in a wireless manner or in an optical communication manner.

For example, example display monitors 512 and 514 include television and/or computer monitors that are configured to provide a three-dimensional viewing experience. For example, the display monitors may include LG® 55LW5600 TVs. Alternatively, the display monitors 512 and 514 may include a laptop computer screen, a tablet computer screen, a smart phone screen, smart goggles, a projector, a holographic display, and so on.

The following sections describe the image capture module 1404, the motor and illumination module 1406, and the information processor module 1408 in more detail. A. Example image capture module

FIG. 15 shows a diagram of an image capturing module 1404 according to an exemplary embodiment of the present invention. The exemplary image capture module 1404 includes an image sensor controller 1502, and the image sensor controller 1502 includes a processor 1504, a memory 1506, and a communication interface 1508. The processor 1504, the memory 1506, and the communication interface 1508 can be communicatively coupled together via an image sensor controller bus 1512.

The processor 1504 can be programmed with one or more programs 1510 that are permanently stored in the memory 1506. The program 1510 includes machine-readable instructions that, when executed, cause the processor 1504 to execute one or more steps, routines, algorithms, etc. In some embodiments, the program 1510 can be transferred from the information processor module 1408 and/or from the user input device 1410 to the memory 1506. In other examples, the program 1510 may be directly transmitted from the information processor module 1408 and/or from the user input device 1410 to the processor 1504.

The example image sensor controller 1502 is communicatively coupled to the right optical image sensor 746 and the left optical image sensor 748 of the optical element 1402. In addition to sending timing control data and/or programming data, the image sensor controller 1502 is also configured to provide power to the optical image sensors 746 and 748. In addition, the image sensor controller 1502 is configured to receive images and/or diagnostic data from the optical image sensors 746 and 748.

Each of the optical image sensors 746 and 748 contains programmable registers to control specific parameters and/or characteristics. One or more of these registers can specify a position of the pixel groups 1006 and 1008 in the respective pixel grids 1002 and 1004 of FIG. 10. The registers can store a value relative to an origin or an edge point of the pixel grids 1002 and 1004. The registers can also specify a width and height of the pixel groups 1006 and 1008 to define a rectangular area of interest. The image sensor controller 1502 is configured to read the pixel data of the pixels in the specified pixel groups 1006 and 1008. In some embodiments, the registers of the optical image sensors 746 and 748 can facilitate the designation of pixel groups of other shapes (such as circles, ovals, triangles, etc.). Additionally or alternatively, the registers of the optical image sensors 746 and 748 can enable multiple pixel groups to be specified for each of the pixel grids 1002 and 1004 at the same time.

One of the light sensing parts of the pixels of the pixel grids 1002 and 1004 is controlled by an embedded circuit (which specifies different light sensing modes). These modes include a reset mode, an integration mode, and a read mode. During the reset mode, a charge storage element of a pixel is reset to a known voltage level. During the integration mode, the pixel switches to an "on" state. The light reaching a sensing area or element of the pixel causes a charge to be accumulated in a charge storage element (for example, a capacitor). The amount of stored charge corresponds to the amount of light incident on the sensing element during the integration mode. During the readout mode, the amount of charge is converted into a digital value and read out from the optical image sensors 746 and 748 via the embedded circuit and transmitted to the image sensor controller 1502. To read each pixel, the charge storage element of each pixel in a given area is sequentially connected to a readout circuit by a switchable internal circuit, and the readout circuit performs the conversion of charge from an analog to digital data. In some embodiments, the pixel analog data is converted to 12-bit digital data. However, it should be understood that the resolution can be smaller or larger based on noise tolerance, stabilization time, frame rate, and data transmission speed. The digital pixel data of each pixel can be stored in a register.

The exemplary processor 1504 of the image sensor controller 1502 of FIG. 15 is configured to receive pixel data from each of the pixels in the pixel groups 1006 and 1008 (for example, indicating an incident on one of the elements of the pixel). The amount of light corresponds to the digital data of a charge stored in the pixel). The processor 1504 forms a right image according to the pixel data received from the right optical image sensor 746. In addition, the processor 1504 forms a left image according to the pixel data received from the left optical image sensor 748. Another option is that the processor 1504 forms only a portion (for example, one row or several rows) of each left image and right image before transmitting the data downstream. In some embodiments, the processor 1504 uses a register location to determine a location of each pixel in an image.

After forming the right image and the left image, the processor 1504 synchronizes the right image with the left image. The processor 1504 then transmits both the right image and the left image to the communication interface 1508, and the communication interface 1508 processes the images into a format for transmission to the information processor module 1408 via a communication channel 1514. In some embodiments, the communication channel 1514 complies with the USB 2.0 or 3.0 standard and may include a copper or fiber optic cable. The communication channel 1514 can enable transmission of up to approximately 60 pairs (or more) of left and right images (with a stereo resolution of 1920×1080 and a data conversion resolution of 12 bits) per second. Using a copper USB cable enables power to be supplied from the information processor module 1408 to the image capture module 1404.

The following sections further describe the features provided by the processor 1504 of the image sensor controller 1502 to execute a specific program 1510 to obtain and/or process the image data from the optical image sensors 746 and 748. 1. Exposure example

The example processor 1504 may control or program the optical image sensors 746 and 748 to be in one of the integrated modes discussed above for an amount of time. The appearance of the integration mode is called a time period of an exposure time. The processor 1504 can set the exposure time by writing a value to an exposure register of the optical image sensors 746 and 748. Alternatively or alternatively, the processor 1504 may transmit instructions to signal the start and end of the exposure time to the optical image sensors 746 and 748. The exposure time can be programmable between a few milliseconds ("ms") and a few seconds. Preferably, the exposure time is roughly the reciprocal of the frame rate.

In some embodiments, the processor 1504 can apply a rolling shutter method to the optical image sensors 746 and 748 to read pixel data. Under this method, the exposure time of a given pixel column of one of the pixel groups 1006 and 1008 starts after the pixels in that column have been read out and then reset. After a short time, the next column (which is usually physically closest to the column just set) is read and therefore reset with its exposure time restarted. The sequential reading of each pixel row continues until the last or bottom row of the pixel groups 1006 and 1008 have been read and reset. The processor 1504 then returns to the top row of the pixel groups 1006 and 1008 to read the pixel data of the next image.

In another embodiment, the processor 1504 applies a global shutter method. Under this method, the processor 1504 implements reading and resetting in a manner similar to the rolling shutter method. However, in this method, integration occurs for all pixels in pixel groups 1006 and 1008 at the same time. The global shutter method has the advantage of reducing defects in an image compared to the rolling shutter method because all pixels are exposed at the same time. In contrast, in the rolling shutter method, there is a small time delay between rows of exposed pixel groups. Small defects may form during the time between row exposures (especially between the top row and the bottom row), where small changes at the target site 700 between readings may occur. 2. Examples of dynamic range

The example processor 1504 can execute one or more programs 1510 to detect light that exceeds one of the dynamic ranges of the optical image sensors 746 and 748. Generally speaking, extremely bright light completely fills a charge storage area of a pixel, thus causing image information about the exact brightness level to be lost. Similarly, extremely low light or lack of light fails to impart a meaningful charge to one of the pixels, which also results in loss of image information. The image formed based on this pixel data therefore does not accurately reflect the light intensity at the target site 700.

To detect light beyond the dynamic range, the processor 1504 can execute one of several high dynamic range ("HDR") programs 1510, including (for example) a multiple exposure program, a multi-slope pixel integration program, and A multi-sensor image fusion program. In one example, the multiple exposure program can utilize HDR features integrated or embedded with optical image sensors 746 and 748. In this method, the pixel groups 1006 and 1008 are placed in the integration mode for a normal exposure time. The rows of the pixel groups 1006 and 1008 are read and stored in one of the optical image sensors 746 and 748 and/or the memory 1506 of the image sensor controller 1502. After reading is performed by the processor 1504, each row in the pixel groups 1006 and 1008 is turned on again for a second exposure time which is less than the normal exposure time. The processor 1504 reads each of the pixel rows after the second exposure time and combines this pixel data with the pixel data from the same row for the normal exposure time. The processor 1504 can apply tone mapping to select (or combine both) pixel data from normal length exposure time and pixel data from short length exposure time and map the resulting pixel data to be compatible with downstream processing and display One range. Using the multiple exposure program, the processor 1504 can expand the dynamic range of the optical image sensors 746 and 748 and compress the pixel data range obtained for display.

The processor 1504 can operate a similar program for relatively low light. However, instead of the second exposure time being less than the normal time, the second exposure time is greater than the normal time, and thus more time is provided for the pixels to accumulate a charge. The processor 1504 can use tone mapping to adjust the read pixel data to compensate for the longer exposure time. 3. Example of frame rate

The example processor 1504 can control or specify a frame rate of the optical image sensors 746 and 748. In some embodiments, the optical image sensors 746 and 748 include on-board timing circuits and programmable control registers to specify that each of the pixels in the pixel groups 1006 and 1008 will cycle through as discussed above The number of imaging modes per second. Each time the pixel group progresses through the three modes, a frame or image is formed. A frame rate is the number of times per second that the pixels in the pixel groups 1006 and 1008 are integrated, read, and reset.

The processor 1504 can be synchronized with the optical image sensors 746 and 748 to enable reading at an appropriate time. In other examples, the processor 1504 is asynchronous with the optical image sensors 746 and 748. In these other examples, the optical image sensors 746 and 748 can store the pixel data in a temporary memory or queue after a local read. The processor 1504 can then periodically read the pixel data to achieve synchronization of the right image and the left image.

Processing frames or images in a chronological manner (for example, creating an image stream) provides the illusion of motion conveyed as a video. The example processor 1504 is configured to programmatically provide the appearance of a smooth video to an observer with a frame rate. A frame rate that is too low will cause any movement to behave inconsistently or unevenly. Film quality above a maximum threshold frame rate is indistinguishable to an observer. The example processor 1504 is configured to generate approximately 20 to 70 frames per second, preferably between 50 frames per second and 60 frames per second for typical surgical visualization. 4. Sensor synchronization example

The example processor 1504 of FIG. 15 is configured to control the synchronization of the optical image sensors 746 and 748. For example, the processor 1504 can provide power to the optical image sensors 746 and 748 at the same time. The processor 1504 can then provide a clock signal to both the optical image sensors 746 and 748. The clock signal enables the optical image sensors 746 and 748 to operate independently but in a synchronized and/or simultaneous manner in a free-running mode. Therefore, the optical image sensors 746 and 748 record pixel data at almost the same time. The example processor 1504 receives pixel data from the optical image sensors 746 and 748, constructs at least one fraction of the image and/or frame, and synchronizes the image and/or frame (or its fraction) to account for any slight timing Mismatch. Generally, the hysteresis between the optical image sensors 746 and 748 is less than 200 microseconds. In other embodiments, the processor 1504 may use a synchronization pin to activate the optical image sensors 746 and 748 at the same time after each reset mode, for example. B. Example motor and lighting module

The exemplary stereo visualization camera 300 of FIG. 15 includes a motor and lighting module 1406 to control one or more motors or actuators for moving the lens of the optical element 1402 and/or controlling the light output from the light source 708. The exemplary motor and lighting module 1406 includes a motor and lighting controller 1520. The motor and lighting controller 1520 include a processor 1522, a memory 1524, and a communication interface 1526 that are communicatively coupled together via a communication bus 1528. . The memory 1524 stores one or more programs 1530, and the one or more programs 1530 can be executed on the processor 1522 to perform the control, adjustment, and/or calibration of the optical element 1402 and/or the lens of the light source 708. In some embodiments, the program 1530 can be transferred from the information processor module 1408 and/or the user input device 1410 to the memory 1524.

The communication interface 1526 is communicatively coupled to the communication interface 1508 of the image capture module 1404 and the communication interface 1532 of the information processor module 1408. The communication interface 1526 is configured to receive command messages, timing signals, status messages, etc. from the image capture module 1404 and the information processor module 1408. For example, the processor 1504 of the image capture module 1404 can send a timing signal to the processor 1522 to synchronize the timing between the light control of the optical image sensors 746 and 748 and the exposure time. In another example, the information processing module 1408 may send a command message indicating that the specific light source 708 will be activated and/or the specific lens of the optical element 1402 will be moved. The commands can be in response to input received from an operator via the user input device 1410, for example. Additionally or alternatively, the commands can be responsive to a calibration routine and/or real-time adjustments to reduce or eliminate image misalignment and/or defects, such as false parallax.

The example motor and lighting module 1406 includes drivers that provide power to control the motors for adjusting the axial and/or radial position of the lens of the optical element 1402 and/or the light output from the light source 708. Specifically, the motor and lighting module 1406 includes a NUV light driver 1534 to transmit a NUV signal to the NUV light source 708c, a NIR light driver 1536 to transmit a NIR signal to the NIR light source 708b, and a visible light driver 1538 In order to transmit a visible light signal to the visible light source 708a.

In addition, the motor and lighting module 1406 includes a filter motor driver 1540 to transmit a filter motor signal to a filter motor 1542, and the filter motor 1542 controls the filter 740 in FIGS. 7 and 8. The motor and lighting module 1406 includes a rear zoom lens motor driver 1544 to transmit a rear zoom lens motor signal to a rear zoom lens motor 1546, and includes a front zoom lens motor driver 1548 to transmit a front zoom lens motor signal to a rear zoom lens motor 1546. The front zoom lens motor 1550 includes a rear working distance lens motor driver 1552 to transmit a working distance lens motor signal to a working distance lens motor 1554. The motor and lighting module 1406 may also include a motor and/or actuator to move and/or tilt the deflection element 712.

The rear zoom lens motor 1546 is configured to rotate a drive screw, which causes the carrier 730 to move axially along a track or rail. The front zoom lens motor 1550 is configured to rotate a drive screw, which causes the carrier 724 to move axially along the track 1106 shown in FIGS. 11 and 12. The working distance lens motor 1554 is configured to rotate a drive screw, which causes the rear working distance lens 704 to move axially along a track or rail.

The drivers 1536, 1538, and 1540 may include any type of lighting drivers, transformers, and/or ballasts. The drivers 1536, 1538, and 1540 are configured to output a pulse width modulation ("PWM") signal to control an intensity of the light output by the light source 708. In some embodiments, the processor 1522 may control the timing of the drivers 1536, 1538, and 1540 to correspond to a timing for applying a specific filter using the filter motor driver 1540.

For example, the example drivers 1540, 1544, 1548, and 1552 may include stepper motor drivers and/or DC motor drivers. Similarly, the motors 1542, 1546, 1550, and/or 1554 may include a stepper motor, a DC motor, or other electric, magnetic, thermal, hydraulic, or pneumatic actuators. For example, the motors 1542, 1546, 1550, and/or 1554 may include a rotary encoder, a slotted optical switch (for example, a photointerrupter), and/or a linear encoder to report a shaft and/or shaft One corner position to achieve feedback reporting and control. Alternative embodiments may include voice coil motors, piezoelectric motors, linear motors with suitable drivers, and their equivalents.

To control the drivers 1534, 1536, 1538, 1540, 1544, 1548, and 1552, the processor 1522 is configured to use a program 1530 to convert a command message into a digital and/or analog signal. The processor 1522 transmits the digital and/or analog signal to an appropriate driver, and the driver outputs an analog power signal, such as a PWM signal corresponding to the received signal. The analog power signal provides power to an appropriate motor or actuator, causing it to rotate (or otherwise move) by a desired amount.

The processor 1522 can receive feedback from the drivers 1534, 1536, 1538, 1540, 1544, 1548, and 1552, the motors 1542, 1546, 1550, and/or 1554, and/or the light source 708. The feedback corresponds to, for example, a light level or light output. Regarding the motors, the feedback corresponds to a position and/or a movement amount of a motor (or other actuator). The processor 1522 uses a program 1530 to convert the received signal into digital feedback to determine (for example) a radial, tilt, and/or axial position of a lens based on an angular position of the corresponding motor or actuator shaft. . The processor 1522 can then transmit a message with position information to the information processor module 1408 for display to a user and/or for tracking a position of the lens of the optical element 1402 for calibration.

In some embodiments, the motor and illumination module 1406 may include additional drivers to change the axial, tilt, and/or radial position of individual lenses in the optical element 1402. For example, the motor and lighting module 1406 may include drivers that control the motors for actuating the flexures 750 and 752 of the optical image sensors 746 and 748 to achieve tilt and/or radial/axial adjust. In addition, the motor and illumination module 1406 may include drivers that control motors (or actuators) for individually driving the front lenses 720 and 722, the front zoom lenses 726 and 728, the rear zoom lenses 732 and 734, and the lens The lens barrels 736 and 738 and/or the final optical elements 745 and 747 are tilted and/or adjusted radially along an x-axis or y-axis and/or axially. The independent adjustment of the lens and/or sensor enables the motor and illumination controller 1520 to, for example, remove image defects and/or align the left and right images.

The following sections describe how the processor 1522 executes one or more programs 1530 to change a working distance, zoom, filter position, lens position, and/or light output. 1. Working distance example

The exemplary processor 1522 of the motor and lighting module 1406 of FIG. 15 is configured to adjust a working distance of the stereo visualization camera 300. The working distance is set by adjusting a distance between the rear working distance lens 704 and the front working distance lens 408. The processor 1522 adjusts the distance by causing the rear working distance lens 704 to move relative to the front working distance lens 408. Specifically, the processor 1522 sends a signal for starting the working distance lens motor 1554 for a predetermined time in proportion to the amount of movement of the rear working distance lens 704 to the rear working distance lens motor driver 1552. The working distance lens motor 1554 drives a lead screw through a thread attached to a sliding track of the fixed working distance lens 704. The working distance lens motor 1554 causes the lens 704 to move a desired distance, thereby adjusting the working distance. The working distance lens motor 1554 can provide a feedback signal to the processor 1522 to determine whether the rear working distance lens 704 is moved by a desired amount. If the movement is less or more than the required amount, the processor 1522 can send an instruction to further refine the position of the working distance lens 704. In some embodiments, the information processor module 1408 can be determined to be used for feedback control of the rear working distance lens 704.

To determine a position of the rear working distance lens 704, the processor 1522 can operate one or more calibration programs 1530. For example, after activation, the processor 1522 can instruct the working distance lens motor 1554 to drive a lead screw to move the rear working distance lens 704 along a track or rail until it triggers a limit switch at one end of the range of motion . The processor 1522 can designate the stop position as a zero point of the encoder of the motor 1554. Knowing the current position of the back working distance lens 704 and the corresponding encoder value, the processor 1522 becomes able to determine the number of shaft rotations that caused the back working distance lens 704 to move to a desired position. The number of rotations of the shaft is transmitted to the working distance lens motor 1554 (via the driver 1552) as an analog signal to thereby move the lens 704 to a prescribed position. 2. Zoom example

The example processor 1522 of FIG. 15 is configured to execute one or more programs 1530 to change a zoom level of the stereo visualization camera 300. As discussed above, zooming (e.g., magnification change) is achieved by changing the positions of the front zoom group 724 and the rear zoom group 730 relative to each other and relative to the front lens group 714 and lens barrel group 718. Similar to the calibration procedure described above for the rear working distance lens 704, the processor 1522 can calibrate the positions of the groups 724 and 730 along the track or rail. Specifically, the processor 1522 sends the rear zoom lens motor 1546 and the front zoom lens motor 1550 to move the groups 724 and 730 (for example, the carrier) along a rail (or track) to a stop position at a limit switch. instruction. The processor 1522 receives encoder feedback from the motors 1546 and 1550 to determine an encoder value associated with the stop positions of the groups 724 and 730. The processor 1522 can then reset the encoder value to zero or use the known encoder value at the stop position to determine the degree of activation of the motors 1546 and 1550 to reach the desired position in one of the groups 724 and 730 along the track.

In addition to the calibration of the stop position, the processor 1522 can also execute a program 1530 that defines the positions of the groups 724 and 730 to achieve a desired zoom level. For example, a known distance setting pattern can be compared to a desired zoom value set and stored as a program 1530 (or a look-up table) during a calibration procedure. The calibration procedure may include placing a template in the target area 700 and instructing the processor 522 to move the groups 724 and 730 until a specific designated mark or character is a specific size in the right image or frame and the left image or frame until. For example, a calibration routine may correspond to when the character "E" on a template at the target site 700 is displayed as having a height of 10 pixels in the right and left images, and determining that the groups 724 and 730 are one The position on the rail.

In some embodiments, the information processor module 1408 can perform visual analysis and send instructions to the processor 1522 to move the groups 724 and 730 to zoom in or out. In addition, the information processor 1408 can send instructions for moving the focal plane to focus the target part 700 at the desired zoom level. For example, the instructions may include instructions to move the rear working distance lens 704 and/or to move the groups 724 and 730 collectively and/or individually. In some alternative embodiments, the processor 1522 may receive the calibration parameters of the track positions of the front zoom group 724 and the rear zoom group 730 at a specific zoom level from the user input device 1410 or another computer.

The example processor 1522 and/or the information processor module 1408 can send commands to keep an image in focus while the magnification changes. For example, the processor 1522 may use a program 1530 and/or a look-up table to determine how a specific lens will move along an optical axis to maintain the focus on the target site 700. The program 1530 and/or the look-up table can specify the magnification level and/or set point on a rail and the corresponding lens adjustment required to prevent the focal plane from moving.

表 2 Table 2 below shows an example program 1530 or look-up table that can be used by the processor 1522 to maintain focus while changing the magnification. The positions of the front zoom lens group 724 and the rear zoom lens group 730 are normalized to the stop positions of the respective groups 724 and 730 based on a length of one rail. To reduce the magnification, the rear zoom lens group is moved toward the lens barrel group 718, thereby increasing one position along a rail. The front zoom lens group 724 is also moved. However, its movement is not necessarily equal to the movement of the rear zoom lens group 730. Alternatively, the movement of the front zoom lens group 724 includes changing a distance between the groups 724 and 730 to maintain the position of the focal plane so as to maintain the focus while changing the magnification. For example, in order to reduce a magnification level from 10X to 9X, the processor 1522 instructs the rear zoom lens group 730 to move from position 10 to position 11 along a rail. In addition, the processor 1522 instructs the front zoom lens group 724 to move from position 5 to position 4 along a rail (or the same rail as the group 730). Not only have the groups 724 and 730 moved to change the magnification, but the groups 724 and 730 have been moved relative to each other to maintain focus.

Table 2

It should be understood that Table 2 provides an example of how groups 724 and 730 can be moved. In other examples, Table 2 may include additional columns to account for more precise magnification and/or location of groups 724 and 730. Additionally or alternatively, Table 2 may include a row for the rear working distance lens 704. For example, the rear working distance lens 704 may be moved in place of or in conjunction with the front zoom lens group 724 to maintain focus. In addition, Table 2 may include a column specifying the positions of the groups 724 and 730 and the rear working distance lens 704 to maintain focus during the changing working distance.

The values in Table 2 can be determined through calibration and/or received from a remote computer or user input device 1410. During the calibration, the information processor module 1408 can operate a calibration program 1560 that progresses through different magnifications and/or working distances. A processor 1562 in the information processor module 1408 can perform image processing of the image itself or the received pixel data to determine when to use (for example) a template with a predetermined shape and/or characters to achieve a desired magnification. The processor 1562 determines whether the received image is in focus. In response to determining that the image is out of focus, the processor 1562 sends an instruction to adjust the front zoom lens group 724 and/or the rear working distance lens group 704 to the processor 1522. The adjustment may include repeated movement in the forward direction and the reverse direction along an optical path until the processor 1562 determines that the image is in focus. In order to determine the focus of an image, the processor 1562 may perform (for example) image analysis to search for images with the least light lack and/or analyze pixel data for the light value difference between adjacent pixel regions (where the larger difference corresponds to More focused images). After determining that an image is in focus at a desired working distance and magnification, the processor 1562 and/or the processor 1522 can then record the positions of the groups 724 and 730 and/or the rear working distance lens 704 and the corresponding magnification level. 3. Example of filter location

The example processor 1522 of the motor and lighting module 1406 of FIG. 15 is configured to move the filter 740 into the right optical path and the left optical path based on the received instruction. In some examples, the filter 740 may include an array of mirrors. In these examples, the processor 1522 sends an instruction to actuate one or more motors 1542 to change the position of the mirror to the filter motor driver 1540. In some examples, the driver 1540 may send a charge to the filter 740 along one or more paths, thereby causing a particular mirror element to switch to an on or off position. In these examples, the filter type selection is generally binary based on which mirrors are activated.

In other examples, the filter 740 may include a round of filters having different types (such as an infrared cut filter, a near infrared band pass filter, and a near ultra ultraviolet cut filter). In these examples, the wheel is rotated by the filter motor 1542. The processor 1522 determines the stop position of the wheel corresponding to the partition between different filters. The processor 1522 also determines the rotary encoder value corresponding to each of the stop positions.

The processor 1522 can operate a calibration program 1530 and/or the processor 1562 can operate a calibration program 1560 to determine the stop positions. For example, the processor 1522 can rotate the filter wheel 740 slowly, where the processor 1562 determines when the light received at the pixel changes (using image analysis or reading the pixel data from the image capture module 1404). A change in one of the light values at the pixel indicates a change in one of the filter types applied to the optical path). In some cases, the processor 1522 may change which light sources 708 are activated to further differentiate at the pixels when a different filter type is applied. 4. Examples of light control and filters

As disclosed above, the processor 1522 together with the filter 740 can control the light source 708 so that light of a desired wavelength reaches the optical image sensors 746 and 748. In some examples, the processor 1522 may control or synchronize the timing between the activation of one or more of the light sources 708 and the activation of one or more of the filters 740. To synchronize timing, a program 1530 can specify a delay time for activating a specific filter. The processor 1522 uses this program 1530 to determine when (for example) to transmit a signal to activate the filter 740 relative to a signal to turn on a light source 708. The scheduled timing ensures that the appropriate filter 740 is applied when the prescribed light source 708 is activated. This configuration enables features highlighted by a light source 708 (such as fluorescent) to be displayed on top of or in conjunction with features displayed under a second light source 708 (such as white light or ambient light).

In some cases, the light source 708 can be switched as fast as the changeable light filter 740, thereby allowing images recorded in different lights to be displayed jointly on top of each other. For example, veins or other anatomical structures that emit fluorescence (due to an application of a dye or contrast agent) can be displayed on top of an image under ambient light. In this example, the veins will be highlighted relative to the background anatomical features shown in visible light. In this example, the processor 1562 of the information processor module 1408 and/or a graphics processing unit 1564 (for example, a video card or a graphics card) enables one or more images to be recorded during the application of a filter with the A combination or overlay of images recorded during the application of a subsequent filter.

In some embodiments, the processor 1522 may activate multiple light sources 708 at the same time. The light source 708 can be activated simultaneously or sequentially to "interleave" light of different wavelengths so that different information can be extracted at the optical image sensors 746 and 748 using appropriate pixels. Starting these light sources at the same time can help illuminate the dark field. For example, some applications use UV light to stimulate fluorescence at a target site 700. However, UV light is perceived by an operator as being very dark. Therefore, the processor 1522 can periodically activate the visible light source 708a to add some visible light to the field of view, so that the surgeon can observe the field of view without covering the pixels sensitive to UV light but can also detect some Visible light. In another example, in some cases, alternating between the light sources 708 avoids flushing the pixels of the optical image sensors 746 and 748 that have overlapping sensitivities at the edges of their ranges. 5. Light intensity control

The example processor 1522 of FIG. 15 is configured to execute one or more programs 1530 to change an intensity or level of illumination provided by the light source 708. It should be understood that the depth of field depends on the irradiation level at the target site 700. Generally speaking, higher illumination provides a greater depth of field. The processor 1522 is configured to ensure that an appropriate amount of exposure is provided for a desired depth of field without flushing the field of view or overheating the field of view.

The visible light source 708a is driven by the visible light driver 1538 and outputs light in the human visible part of the spectrum and some light outside that area. The NIR light source 708b is driven by the NIR light driver 1536 and outputs light mainly at a wavelength called near-infrared. The NUV light source 708c is driven by the NUV light driver 1534 and outputs light whose depth is mainly in one of the wavelengths in the blue part of the visible spectrum, which is called near ultra-ultraviolet. The respective optical drivers 1534, 1536, and 1538 are controlled by commands provided by the processor 1522. The control of the respective output spectrum of the light source 708 is achieved by the PWM signal, wherein a control voltage or current is switched between a minimum value (for example, off) and a maximum value (for example, on). The brightness of the light output from the light source 708 is controlled by changing the switching rate and the percentage of time that the voltage or current is at the maximum level per cycle in the PWM signal.

In some examples, the processor 1522 controls the output of one of the light sources 708 based on one of the field of view or the zoom level. The processor 1522 can execute a program 1530 for specifying the light intensity as a function of the zoom for a specific photosensitive setting. For example, the program 1530 may include a look-up table that correlates a zoom level with a light intensity value. The processor 1522 uses the program 1530 to select the PWM signal for the light source 708 based on the selected amplification level. In some examples, the processor 1522 may reduce the light intensity as the magnification increases to maintain the amount of light provided to the field of view per unit area. C. Example information processor module

The exemplary information processor module 1408 in the stereo visualization camera 300 of FIG. 15 is configured to analyze and process the images/frames received from the image capture module 1404 for display. In addition, the information processor module 1408 is configured to interface with different devices and convert control commands into information for the image capture module 1404 and/or the motor and illumination module 1406. The information processor module 1408 can also provide an interface for manual calibration and/or manage the automatic calibration of the optical element 1402.

As shown in FIG. 15, the information processor module 1408 is communicatively and/or electrically coupled to the image capture module 1404 and the motor and illumination module 1406. For example, in addition to the communication channels 1566 and 1568, the communication channel 1514 may also include USB 2.0 or USB 3.0 connections. In this way, the information processor module 1408 regulates power and provides power to the modules 1404 and 1406. In some embodiments, the information processor module 1408 converts 110 volt alternating current ("AC") power from a wall outlet into 5, 10, 12, and/or 24 volts for one of the modules 1404 and 1406 Direct current ("DC") supply. In addition or alternatively, the information processor module 1408 receives power from a battery inside the housing 302 of the stereo visualization camera 300 and/or a battery at the truck 510.

The example information processor module 1408 includes a communication interface 1532 to communicate with the image capture module 1404 and the motor and illumination module 1406 bidirectionally. The information processor module 1408 also includes a processor 1562, which is configured to execute one or more programs 1560 to process images/frames received from the image capture module 1404. The program 1560 can be stored in a memory 1570. In addition, the processor 1562 may perform calibration of the optical element 1402 and/or adjust the optical element 1402 to align the right and left images and/or remove visual defects.

To process the image and/or frame into a rendered three-dimensional display, the exemplary information processor module 1408 includes a graphics processing unit 1564. Figure 16 shows a diagram of a graphics processing unit 1564 according to an exemplary embodiment of the present invention. During operation, the processor 1562 receives images and/or frames from the image capture module 1404. A decompression routine 1602 converts the image/frame to a format that is useful for transmission across the communication channel 1514 or otherwise changes to a format that is useful for image processing. For example, images and/or frames can be transmitted across the communication channel 1514 in multiple messages. The example decompression routine 1602 combines the data from the multiple messages to reassemble the frame/image. In some embodiments, the decompression routine 1602 may queue frames and/or images until requested by the graphics processing unit 1564. In other examples, the processor 1562 may transmit each right image/frame pair and left image/frame pair after fully receiving and decompressing.

The exemplary graphics processing unit 1564 uses one or more programs 1580 (shown in FIG. 15) to prepare the image for rendering. Examples of the program 1580 are shown in FIGS. 15 and 16. The program 1580 can be executed by a processor of the graphics processing unit 1564. Alternatively, each of the programs 1580 shown in FIG. 16 can be executed by a separate graphics processor, microcontroller, and/or application-specific integrated circuit ("ASIC"). For example, a solution Bayer program 1580a is configured to smooth or average pixel values across adjacent pixels to compensate for the difference between the right optical image sensor 746 and the left optical image sensor 748 of FIGS. 7 and 8 A Bayer pattern of pixel grids 1002 and 1004. The graphics processing unit 1564 may also include programs 1580b, 1580c, and 1580d for color correction and/or white balance adjustment. The graphics processing unit 1564 also includes a renderer program 1580e for preparing the color-corrected image/frame for display on the display monitors 512 and 514. The graphics processing unit 1564 may further interact with a peripheral input unit interface 1574 and/or include a peripheral input unit interface 1574. The peripheral input unit interface 1574 is configured to combine, merge, or otherwise include a stereoscopic image for the target part 700 Display other images and/or graphics presented together. More generally, additional details of the program 1580 and the information processor module 1408 are discussed below.

The example information processor module 1408 can execute one or more programs 1560 to check and improve the delay of the stereo visualization camera 300. Delay refers to the amount of time it takes for an event to appear at the target site 700 and for the same event to be displayed by the display monitors 512 and 514. The low-latency provides the stereo vision camera 300 as an extension of a surgeon’s eye, while the high-latency tends to be distracted from the microsurgery procedure. The example processor 1562 can track how much time has elapsed between reading the image from the optical image sensors 746 and 748 until transmitting the combined stereoscopic image based on the read image for display. High-latency detection can cause the processor 1562 to reduce the queue time, increase the frame rate, and/or skip certain color correction steps. 1. User input example

The example processor 1562 of the information processor module 1408 of FIG. 15 is configured to convert user input commands into messages for the motor and lighting module 1406 and/or the image capture module 1404. The user input command may include changing the optical aspect of the stereo visualization camera 300 (including a magnification level, a working distance, a height of a focal plane (eg, focus), a light source 708 and/or a filter 740 A filter type) request. The user input commands may also include a request to perform calibration, including an image focus instruction and/or an image alignment instruction and/or an alignment ZRP instruction between the left image and the right image. The user input commands may further include adjustment of the parameters of the stereo visualization camera 300 (such as frame rate, exposure time, color correction, image resolution, etc.).

The user input commands may be received from a user input device 1410, and the user input device 1410 may include the control 305 of the control arm 304 of FIG. 3 and/or a remote control. The user input device 1410 may also include a computer, a tablet computer, and so on. In some embodiments, the commands are received via a network interface 1572 and/or a peripheral input unit interface 1574. In other embodiments, the commands can be received from a wired connection and/or an RF interface.

The example processor 1562 includes a program 1560 for determining a command type and determining how to process user input. In one example, a user can press a button of the control 305 to change a zoom level. The button can continue to be pressed until the operator has caused the stereo visualization camera 300 to reach a desired magnification level. In these examples, the user input commands include information instructing to increase, for example, a magnification level. For each command received (or each time period in which a signal indicating the command is received), the processor 1562 sends a control command indicating a change in the magnification to the motor and illumination processor 1406. The processor 1522 determines how much the zoom lens groups 724 and 730 will be moved using (for example) Table 2 according to a formula 1530. The processor 1522 will therefore cause the rear zoom lens motor 1546 and/or the front zoom lens motor 1550 to move the rear zoom lens group 730 and/or the front zoom lens group 724 by an amount specified by the processor 1562 to achieve a desired magnification level. The signal or message is transmitted to the rear zoom lens motor driver 1544 and/or the front zoom lens motor driver 1548.

It should be understood that in the above example, the stereoscopic visualization camera 300 provides a change based on user input, but also makes automatic adjustments to maintain focus and/or a high image quality. For example, instead of merely changing the magnification level, the processor 1522 also determines how to move the zoom lens groups 724 and 730 to also maintain the focus, thereby saving an operator from having to perform this task manually. In addition, the processor 1562 can instantly adjust and/or align the ZRP in the right and left images when a zoom level is changed. For example, this can be accomplished by selecting or changing the positions of the pixel groups 1006 and 1008 in FIG. 10 relative to the pixel grids 1002 and 1004.

In another example, the processor 1562 may receive a command from the user input device 1410 to change a frame rate. The processor 1562 transmits a message to the processor 1504 of the image capturing module 1404. Then, the processor 1504 writes the content indicating the new frame rate to the registers of the right image sensor 746 and the left image sensor 748. The processor 1504 can also update the internal register with the new frame rate to change the speed of reading pixels.

In another example, the processor 1562 may receive a command from the user input device 1410 to start a calibration routine of ZRP. In response, the processor 1562 can execute a program 1560 that specifies how the calibration will be performed. For example, in addition to a routine used to verify image quality, the program 1560 may also include progress or repetitions of magnification level and/or working distance. The routine may specify that for each zoom level, in addition to ZRP, the focus will also be verified. The routine may also specify how to adjust the zoom lens groups 724 and 730 and/or the rear working distance lens 704 to achieve a focused image. This routine may further specify how the ZRP of the right and left images will be centered for the magnification level. Once the image quality has been verified, the program 1560 can store the positions of the zoom lens groups 724 and/or 730 and/or the rear working distance lens 704 in addition to the positions of the pixel groups 1006 and 1008 and the corresponding magnification levels (to a search surface). Therefore, when the same magnification level is requested at a subsequent time, the processor 1562 uses the look-up table to specify the position of the zoom lens group 724 and/or 730 and/or the rear working distance lens 704 to the motor and illumination module 1406 and to capture the image The fetch module 1404 specifies the positions of the pixel groups 1006 and 1008. It should be understood that in some calibration routines, at least some of the lenses of the optical element 1402 can be adjusted radially/rotatably and/or the lenses can be tilted to center the ZRP and/or align the right image And left image. 2. Interface example

In order to facilitate the communication between the stereo visualization camera 300 and external devices, the exemplary information processor module 1408 includes a network interface 1572 and a peripheral input unit interface 1574. The example network interface 1572 is configured to enable a remote device to be communicatively coupled to the information processor module 1408 to, for example, store the recorded video, and control the working distance and zoom position of the stereo visualization camera 300 Alignment, focus, alignment, or other characteristics. In some embodiments, the remote devices can provide values or parameters of a calibration look-up table, or more generally, a program 1530 with calibrated parameters. The network interface 1572 may include an Ethernet interface, a local area network interface, and/or a Wi-Fi interface.

The example peripheral input unit interface 1574 is configured to be communicatively coupled to one or more peripheral devices 1576 and facilitate the integration of stereoscopic image data with peripheral data (such as patient physiological data). The peripheral input unit interface 1574 may include a Bluetooth® interface, a USB interface, an HDMI interface, SDI, and so on. In some embodiments, the peripheral input unit interface 1574 can be combined with the network interface 1572.

For example, the peripheral device 1576 may include a data or video storage unit, a patient's physiological sensor, a medical imaging device, an infusion pump, a dialysis machine, and/or a tablet computer, etc. Peripheral data can include image data from a dedicated two-dimensional infrared professional camera, diagnostic images from a user’s laptop, and/or from an ophthalmic device (such as Alcon Constellation® system and wave technology optical wave refraction analysis (ORA) ^TM ) system) image or patient diagnosis text.

The example peripheral input unit interface 1574 is configured to convert and/or format data from the peripheral device 1576 into an appropriate digital form for use with stereoscopic images. Once in digital form, the graphics processing unit 1564 integrates peripheral data with other system data and/or stereo images/frames. The data is reproduced together with the stereoscopic image for display on the display monitors 512 and/or 514.

In order to configure the included peripheral data and three-dimensional images, the processor 1562 can control an integrated setting. In one example, the processor 1562 may cause the graphics processing unit 1564 to display a configuration panel on the display monitors 512 and/or 514. The configuration panel allows an operator to connect a peripheral device 1576 to the interface 1574 and the processor 1562 to subsequently establish communication with the device 1576. The processor 1562 can then read which data is available or enable the operator to use the configuration panel to select a data directory location. The surrounding data in the directory location is displayed in the configuration panel. The configuration panel can also provide an option for the operator to overlay or display the surrounding data and the three-dimensional image data as a single picture.

The selection of the peripheral data (and overlay format) causes the processor 1562 to read the data and transmit the data to the graphics processing unit 1564. The graphics processing unit 1564 applies the peripheral data to the three-dimensional image data for presentation as an overlay graphic (such as fusing a preoperative image or a graphic with a real-time three-dimensional image), a "picture-in-picture" and/or in the main three-dimensional image A sub-window on the side or top of the window. 3. An example of the Bayer program

The exemplary Bayer solution 1580a of FIG. 16 is configured to generate images and/or frames with values for red, green, and blue colors at each pixel value. As discussed above, the pixels of the right optical image sensor 746 and the left optical image sensor 748 have a filter that passes light in the red wavelength range, blue wavelength range, or green wavelength range. Therefore, each pixel only contains a part of the light data. Therefore, each image and/or frame received from the image capturing module 1404 in the information processor module 1408 has pixels containing red, blue, or green pixel data.

The example Bayer program 1580a is configured to average the red, blue, and green pixel data of adjacent and/or adjacent pixels to determine more complete color data for each pixel. In one example, a pixel with red data and a pixel with blue data are located between two pixels with green data. The green pixel data of the two pixels are averaged and assigned to the pixel with red data and the pixel with blue data. In some examples, the averaged green data may be weighted based on a distance between pixels with red data and pixels with blue data from the respective green pixels. After calculation, pixels that originally had only red or blue data now contain green data. Therefore, after the graphics processing unit 1564 executes the Bayer program 1580a, each pixel contains pixel data for a certain amount of red, blue, and green light. For the pixel data of different colors to be blended to determine the color obtained in one of the color spectrums, the reproducer program 1580e can use the obtained color for display and/or display monitors 512 and 514. In some instances, the DeBayer program 1580a can determine the resulting color and store data or an identifier indicating the color. 4. Examples of color correction

Example color correction programs 1580b, 1580c, and 1580d are configured to adjust pixel color data. The sensor color calibration program 1580b is configured to explain or adjust the color sensing variability of the optical image sensors 746 and 748. The user color calibration program 1580c is configured to adjust pixel color data based on the perception and feedback of an operator. In addition, the display color correction program 1580d is configured to adjust pixel color data based on a display monitor type.

In order to calibrate colors for sensor variability, the example color calibration program 1580b provides for a calibration routine that can be executed by the graphics processing unit 1564 and/or the processor 1562. Sensor calibration includes placing a calibrated color chart (such as ColorChecker® Digital SG from X-Rite) at the target site 700. The processor 1562 and/or the graphics processing unit 1564 executes a program 1580b, and the program 1580b includes instructions to record the right image and the left image of the color chart to the image capture module 1404. The pixel data from the right and left images (after being processed by the Bayer program 1580a) can be compared with the pixel data associated with the color chart (which can be from a peripheral unit 1576 and/or a remote computer via the network interface 1572) Store to memory 1570). The processor 1562 and/or the graphics processing unit 1564 determines the difference between the pixel data. The difference is stored in the memory 1570 as calibration data or parameters. The sensor color calibration program 1580b applies the calibration parameters to the subsequent right and left images.

In some instances, the differences can be averaged over a number of pixel regions so that the program 1580b can find a method that can be applied globally to all pixels of the optical image sensors 746 and 748 to produce colors that are as close as possible to the color chart. One of the color correction data is the best fit. Alternatively or alternatively, the program 1580b can process the user input command received from the user unit device 1410 to correct the color. The commands may include regional and/or global changes to the red, blue, and green pixel data based on the operator's preference.

The example sensor color calibration program 1580b is also configured to calibrate the white balance. Generally speaking, white light should produce red, green, and blue pixels with equal values. However, the difference between the pixels can be due to the color temperature of the light used during imaging, the inherent pattern of the filter and sensing element of each of the pixels, and the (for example) deflection element 712 of FIGS. 7 and 8 The frequency spectrum filtering parameters are generated. The example sensor color calibration program 1580b is configured to specify a calibration routine to correct the light imbalance.

To perform white balance, the processor 1562 (according to the instructions from the program 1580b) can display an instruction on the display monitor 512 and/or 514 for an operator to place a neutral card at the target site 700. The processor 1562 can then instruct the image capture module 1404 to record one or more images of the neutral card. After being processed by the decompression routine 1602 and the Bayer program 1580a, the program 1580b determines the regional and/or global white balance calibration weight values for each of the red, blue, and green data, so that each of the pixels has The red, blue, and green data are substantially equal in value. The white balance calibration weight values are stored in the memory 1570. During operation, the graphics processing unit 1564 uses the program 1580b to apply white balance calibration parameters to provide white balance.

In some examples, the program 1580b determines the white balance calibration parameters for the right optical image sensor 746 and the left optical image sensor 748 individually. In these examples, the program 1580b can store separate calibration parameters for the left and right images. In other examples, the sensor color correction program 1580b determines a weight between the right view and the left view, so that the color pixel data is almost the same for the right optical image sensor 746 and the left optical image sensor 748. The determined weight can be applied to the white balance calibration parameter for subsequent use during the operation of the stereo visualization camera 300.

In some embodiments, the sensor color calibration program 1580b of FIG. 16 specifies that the white balance calibration parameter is applied as a digital gain for the pixels of the right optical image sensor 746 and the left optical image sensor 748. For example, the processor 1504 of the image capture module 1404 applies digital gain to the pixel data read from each of the pixels. In other embodiments, the white balance calibration parameter is applied as an analog gain of the color sensing element for each pixel.

The example sensor color correction program 1580b can perform white balance and/or color correction when the filters 740 of different light sources 708 and/or different filter types are activated. Therefore, the memory 1570 can store different calibration parameters based on which light source 708 is selected. In addition, the sensor color calibration program 1580b can perform white balance and/or color calibration for different types of external light. An operator can use the user input device 1410 to specify the characteristics and/or a type of the external light source. This calibration enables the stereo vision camera 300 to provide color correction and/or white balance for different lighting environments.

The example program 1580b is configured to perform calibration on each of the optical image sensors 746 and 748 individually. Therefore, the program 1580b applies different calibration parameters to the right and left images during operation. However, in some instances, calibration may be performed on only one sensor 746 or 748, where the calibration parameters are used for the other sensor.

The example user color correction program 1580c is configured to request feedback from the operator regarding image quality parameters such as brightness, contrast, gamma, hue, and/or saturation. The feedback can be received as a command from the user input device 1410. The adjustments made by the user are stored in the memory 1570 as user calibration parameters. Then, the user color calibration program 1580c applies these parameters to the right and left optical images after color calibration for the optical image sensors 746 and 748.

The exemplary display color correction program 1580d of Figure 16 is configured to use, for example, the Datacolor ^™ Spyder color checker to correct the image color of a display monitor. Similar to the program 1580b, the program 1580d instructs the image capturing module 1404 to record an image of a display color template at the target scene 700. The display color correction program 1580d operates to adjust a routine of pixel data to match an expected display output in a look-up table stored in the memory 1570. The adjusted pixel data can be stored in the memory 1570 as display calibration parameters. In some instances, a camera or other imaging sensor can be connected to the peripheral input unit interface 1574. The peripheral input unit interface 1574 provides images or other color-related feedback recorded from the display monitors 512 and 514 (which is used to adjust Pixel data). 5. Three-dimensional image display example

The example renderer program 1580e of the graphics processing unit 1564 of FIG. 16 is configured to prepare the right image and/or frame and the left image and/or frame for 3D stereoscopic display. After performing color correction on the pixel data of the right and left images by the programs 1580b, 1580c, and 1580d, the renderer program 1580e is configured to render the left-eye and right-eye data into a format suitable for stereoscopic display and The final reproduced version is placed in an output buffer for transmission to one of the display monitors 512 or 514.

Generally speaking, the renderer program 1580e receives a right image and/or frame and a left image and/or frame. The renderer program 1580e combines the right image and/or frame and the left image and/or frame into a single frame. In some embodiments, the program 1580e operates in a top-down mode and compresses the height of the left image data by one-half. Program 1580e then places the compressed left image data in the top half of one of the combined frames. Similarly, the program 1580e compresses the height of the right image data by one half and places the compressed right image data in the bottom half of the combined frame.

In other embodiments, the renderer program 1580e operates in a side-by-side mode, in which the width of each of the left image and the right image is compressed by one-half and combined in a single image, so that the left image data is provided in one of the images The upper half of the left image data and the right half of the image data are provided on the upper half of the image. In yet another alternative embodiment, the renderer program 1580e operates in a row of interleaved mode, in which every other line in the left and right frames is discarded. The left frame and the right frame are combined to form a complete three-dimensional image.

The example renderer program 1580e is configured to individually render the combined left and right images for each connected display monitor. For example, if the display monitors 512 and 514 are connected, the renderer program 1580e renders a first combined stereoscopic image for the display monitor 512 and a second combined stereoscopic image for the display monitor 514. The renderer program 1580e formats the first combined 3D image and the second combined 3D image so that they are compatible with the type and/or screen size of the display monitor and/or screen.

In some embodiments, the renderer program 1580e selects the image processing mode based on how the display monitor will display the stereo data. A proper interpretation of the three-dimensional image data by the brain of an operator requires that the left-eye data of the three-dimensional image is transmitted to the left eye of the operator and the right-eye data of the three-dimensional image is transmitted to the right eye of the operator. Generally speaking, the display monitor provides a first polarization of left-eye data and a second opposite polarization of right-eye data. Therefore, the combined 3D image must match the polarization of the display monitor.

FIG. 17 shows an example of a display monitor 512 according to an exemplary embodiment of the present invention. For example, the display monitor 512 can be an LG® 55LW5600 3D TV with a screen 1702. The exemplary display monitor 512 uses a polarization film on the screen 1702 so that all odd-numbered columns 1704 have a first polarization and all even-numbered columns 1706 have a pair of polarizations. In order to be compatible with the display monitor 512 shown in FIG. 17, the renderer program 1580e will have to select the row cross mode so that the left image data and the right image data are on alternate lines. In some cases, the renderer program 1580e may request (or otherwise receive) the display characteristics of the display monitor 512 before preparing the stereoscopic image.

To view the three-dimensional image displayed on the screen 1702, the surgeon 504 (remember him from Figure 5) wears glasses 1712 that includes a left lens 1714. The left lens 1714 includes a first polarization that matches the first polarization of column 1704. . In addition, the glasses 1712 include a right lens 1716, and the right lens 1716 includes a second polarization that matches the second polarization of the column 1706. Therefore, the left lens 1714 only allows most of the light from the left image data in the left row 1704 to pass while blocking most of the light from the right image data. In addition, the right lens 1716 allows most of the light from the right image data in the right column 1706 to pass while blocking most of the light from the left image data. The amount of light reaching each individual eye from the "error" view is called "crosstalk" and is generally kept low enough to allow comfortable viewing. Therefore, the surgeon 504 views the left image data recorded by the left optical image sensor 748 in a left eye while viewing the right image data recorded by the right optical image sensor 746 in a right eye. The surgeon’s brain fuses the two views together to form a three-dimensional perception of distance and/or depth. In addition, the use of such a display monitor is advantageous to the accuracy of observing the stereo visualization camera 300. If the surgeon or operator does not wear glasses, he can observe both the left view and the right view with two eyes. If a flat target is placed at the focal plane, the two images will theoretically be aligned. If misalignment is detected, the processor 1562 can initiate a recalibration procedure.

The example renderer program 1580e is configured to render the left and right views for circular polarization. However, in other embodiments, the renderer program 1580e can provide a stereoscopic image compatible with linear polarization. No matter which type of polarization is used, the example processor 1562 can execute a program 1560 to verify or check one of the polarizations of the stereoscopic image output by the renderer program 1580e. To check the polarization, the processor 1562 and/or the peripheral input unit interface 1574 inserts the diagnostic data into the left image and/or the right image. For example, the processor 1562 and/or the peripheral input unit interface 1574 can overlay the "left" text on the left image and the "right" text on the right image. The processor 1562 and/or the peripheral input unit interface 1574 may display a prompt instructing an operator to close one eye at a time while wearing the glasses 1712 to confirm that the left view is received at the left eye and the right view is received at the right eye. The operator can provide confirmation via the user input device 1410 indicating whether the polarization is correct. If the polarization is incorrect, the example renderer program 1580e is configured to reverse the position where the left and right images are inserted into the combined stereoscopic image.

In still other embodiments, the example renderer program 1580e is configured to provide frame sequential projection instead of forming a combined stereo image. Here, the renderer program 1580e intersects the right image and/or frame in time order to reproduce the left image and/or frame. Therefore, the left image and the right image are alternately presented to the surgeon 504. In these other embodiments, the screen 1702 is not polarized. Alternatively, the left lens and the right lens of the glasses 1712 can be electronically or optically synchronized to their respective parts of a frame sequence, which provides a user with the corresponding left and right views to distinguish the depth.

In some instances, the renderer program 1580e can provide specific right and left images for display on a separate display monitor or a separate window on a display monitor. This configuration can be particularly beneficial when the lenses of the right optical path and the left optical path of the optical element 1402 can be adjusted independently. In one example, a right optical path may be set at a first magnification level, and a left optical path may be set at a second magnification level. The example renderer program 1580e can therefore display an image stream from the left view on the display monitor 512 and an image stream from the right view on the display monitor 514. In some examples, the left view may be displayed in a first window on the display monitor 512, and the right view may be displayed in a second window (for example, a picture-in-picture) of the same display monitor 512. Therefore, although it is not three-dimensional, the simultaneous display of the left and right images provides useful information to a surgeon.

In another example, the light source 708 and the filter 740 can be quickly switched to use visible light and fluorescent light to generate alternate images. The example renderer program 1580e can combine the left view and the right view to provide a stereoscopic display under different light sources to highlight (for example) a blood vessel with a dye while displaying the background in visible light.

In yet another example, a digital zoom can be applied to the right optical image sensor 746 and/or the left optical image sensor 748. Digital zoom generally affects the perceived resolution of an image and depends on factors such as display resolution and viewer preference. For example, the processor 1504 of the image capture module 1404 can apply digital zoom by forming interpolated pixels that are synthesized and interspersed between the digitally zoomed pixels. The processor 1504 can operate a program 1510 that coordinates the selection of the optical image sensors 746 and 748 and the interpolation of pixels. The processor 1504 transmits the right image and the left image for subsequent reproduction and display when the digital zoom is applied to the information processor module 1408.

In some embodiments, the processor 1504 receives the following instruction from the processor 1562: a digital zoom image will be recorded between images without digital zoom to provide a digital zoom image of an area of interest of the target site 700 Medium picture (or separate window) display. The processor 1504 therefore applies digital zoom to every other reading from the pixel grids 1002 and 1004. This enables the renderer program 1580e to simultaneously display a stereo full resolution image in addition to a digitally zoomed stereo image. Another option is to copy the digitally zoomed image from the current image, scale it and place it in a proper position overlaid on top of the current image during the reproduction phase. Another option is that this configuration avoids "alternating" recording requirements. 6. Calibration example

The example information processor module 1408 of FIGS. 14-16 can be configured to execute one or more calibration programs 1560 to calibrate, for example, a working distance and/or magnification. For example, the processor 1562 may send an instruction to execute a calibration step to the motor and illumination module 1406 for measuring a working distance (measured in millimeters) from the main objective lens assembly 702 to the target part 700 Map to the known motor position of one of the working distance lens motors 1554. The processor 1562 performs calibration by sequentially moving an object plane along the optical axis in discrete steps and refocusing the left and right images while recording the encoder count and working distance. In some instances, the working distance can be measured by an external device that transmits the measured working distance value to the processor 1562 via the peripheral input unit interface 1574 and/or an interface to the user input device 1410. The processor 1562 can store the position of the rear working distance lens 704 (based on the position of the working distance lens motor 1554) and the corresponding working distance.

The example processor 1562 can also execute a program 1560 to perform magnification calibration. The processor 1562 can use the motor and the light module 1406 to set the optical element 1402 to select the magnification level. The processor 1562 can record the position of the optical element 1402 or the corresponding motor position for each magnification level. The magnification level can be determined by measuring the height of an object and an image of a known size. For example, the processor 1562 can measure an object as having a height of 10 pixels and use a look-up table to determine that a height of 10 pixels corresponds to a 5X magnification.

In order to match the stereo perspective of two different imaging modalities, it is usually desirable to model the two, as if it were a simple pinhole camera. The viewing angle of a 3D computer model such as an MRI brain tumor can be viewed from the user with adjustable direction and distance (for example, as if the image was recorded by a synthetic stereo camera). Adjustability can be used to match the field of view of surgical images (which must therefore be known). The example processor 1562 may calibrate one or more of these pinhole camera model parameters, such as, for example, the center of projection of the right optical image sensor 746 and the left optical image sensor 748 ("COP" ). To determine the projection center, the processor 1562 determines a focal length from the projection center to an object plane. First, the processor 1562 sets the optical element 1402 to a magnification level. The processor 1562 then three different distances along the optical axis (contained in the object plane, one plane at a distance d from one of the planes is smaller than the object distance and the distance d is greater than the object) measuring one of a recorded image height. The processor 1562 uses an algebraic formula of similar triangles at the two most extreme positions to determine the focal length to the center of the projection. The processor 1562 may use the same method or determine the focal length at other magnifications by determining a ratio between the magnifications used for calibration. The processor can use a projection center to match the angle of view of an image of an object to be fused (such as an MRI tumor model) with an on-site stereoscopic surgical image. In addition or alternatively, an existing camera calibration program such as OpenCV calibrateCamera can be used to find the parameters described above and additional camera information, such as a distortion model of the optical element 1402.

The example processor 1562 may further calibrate the left optical axis and the right optical axis. The processor 1562 determines an interpupillary distance between the left optical axis and the right optical axis for calibration. To determine the distance between the pupils, the exemplary processor 1562 records the left and right images, where the pixel groups 1006 and 1008 are centered at the pixel grids 1002 and 1004. The processor 1562 determines the ZRP position (and/or the distance to a displaced object) of the left image and the right image, and these ZRP positions indicate the image misalignment and the degree of parallax. In addition, the processor 1562 scales the parallax and/or distance based on the magnification level. Taking into account the degree of parallax and/or the scaled distance to the object in the display, the processor 1562 then uses a triangulation measurement calculation to determine the interpupillary distance. The processor 1562 then associates the interpupillary distance with the optical axis under a prescribed magnification level as a calibration point. VI. Image alignment and false parallax adjustment embodiment

Similar to human vision, a stereoscopic image includes a right view and a left view that converge at a point of interest. The right view and the left view are recorded at slightly different angles from the point of interest, which causes a parallax between the two views. The items in the scene before or behind the point of interest exhibit parallax, so that the distance or depth of the items from the viewer can be inferred. For example, the accuracy of the perceived distance depends on the clarity of the viewer's vision. The vision of most people exhibits some degree of imperfection, which results in some inaccuracies between the right and left views. However, these views can still achieve stereo vision, where the brain fuses the views with a certain degree of accuracy.

When the left and right images are recorded by a camera instead of being viewed by a human, the parallax between the combined images on a display screen produces stereo vision, which provides an appearance of a three-dimensional image on a two-dimensional display. The error of the parallax can affect the quality of the three-dimensional image. The inaccuracy of the observed parallax compared with a theoretically perfect parallax is called false parallax. Unlike humans, cameras do not have a brain that automatically compensates for inaccuracy.

If the pseudo-parallax becomes significant, the three-dimensional image may be invisible to the extent that it causes dizziness, headache, and nausea. There are many factors that can affect the parallax in a microscope and/or camera. For example, the optical channels of the right view and the left view may not be completely equal. The optical channels may have mismatched focus, magnification, and/or misalignment of the point of interest. These problems can have different severity under different magnifications and/or working distances, thus reducing the effort of correction through calibration.

Known surgical operating microscopes (such as the surgical operating microscope 200 of FIG. 2) are configured to provide a sufficient view through the eyepiece 206. Generally, the image quality of the optical elements of the known surgical operating microscope is insufficient for the stereo camera. The reason for this situation is that the manufacturer of the surgical operating microscope assumes that the main viewing is through the eyepiece. Any camera attachment (such as the camera 212) is mono-image and is not affected by false parallax, or is stereoscopic in the case of low image resolution, where the false parallax is not obvious.

International standards (such as ISO 10936-1:2000, Optics and optical instruments – Operation microscopes – Part 1: Requirements and test methods ) have been developed to provide the specification limits for the image quality of surgical operating microscopes. These specifications are generally set for viewing through the eyepieces of a surgical operating microscope and do not consider the three-dimensional display. For example, regarding false parallax, ISO 10936-1:2000 stipulates that the difference in the vertical axis between the left view and the right view should be less than 15 arc minutes. The small angular deviation of the axis is usually quantified in units of arc minutes (which corresponds to 1/60 of a degree) or arc seconds (which corresponds to 1/60 of an arc minute). For a typical surgical operating microscope with a working distance of 250 mm and a field of view of 35 mm (which has an angular field of view of 8°), the 15 arc minute specification limit corresponds to a 3% between the left view and the right view Difference.

The 3% difference is acceptable for the eyepiece viewing system, in which the brain of a surgeon can overcome a small degree of error. However, when viewed stereoscopically on a display monitor, this 3% difference produces a significant difference between the left view and the right view. For example, when the left view and the right view are displayed together, a 3% difference makes an image look disjointed and difficult to view for an extended period of time.

Another problem is that the known surgical operating microscope can meet the 15 arcmin specification limit under only one or a few magnification levels and/or only individual optical elements can meet a specific specification limit. For example, individual lenses are manufactured to meet certain criteria. However, when individual optical elements are combined in an optical path, small deviations from the standard can be amplified rather than cancelled. This can be particularly significant when five or more optical elements are used in an optical path that includes a common main objective lens. In addition, it is very difficult to completely match the optical elements on the parallel channels. At most during manufacturing, the optical elements of a surgical operating microscope are only calibrated at one or a few specific magnification levels to meet the 15 arcmin specification limit. Therefore, the error between the calibration points can be greater, even though the surgical operating microscope is said to meet the ISO 10936-1:2000 specification.

In addition, when adding additional components, the ISO 10936-1:2000 specification allows greater tolerances. For example, adding a second eyepiece (e.g., eyepiece 208) increases the false parallax by 2 arc minutes. Again, although this error may be acceptable for viewing through the eyepieces 206 and 208, the image misalignment becomes more significant when viewed stereoscopically through a camera.

Compared with known surgical operating microscopes, the exemplary stereo visualization camera 300 disclosed herein is configured to automatically adjust at least some of the optical elements 1402 to reduce or eliminate false parallax. Embedding the optical element in the stereo visualization camera 300 enables automatic (sometimes instantaneous) fine adjustments to be made for three-dimensional stereo display. In some embodiments, the exemplary stereo vision camera 300 can provide an accuracy of 20 arc seconds to 40 arc seconds, which is close to a 97% reduction in optical errors (which is accurate to 15 arc minutes of known surgical operating microscopes). Sexual comparison).

The accuracy improvement allows the exemplary stereo visualization camera 300 to provide features that cannot be performed with known stereo microscopes. For example, many new microsurgery procedures rely on accurate measurement of an on-site surgical site to achieve the best sizing, positioning, matching, guidance, and diagnosis. This includes determining the size of a blood vessel, the placement angle of an toroidal intraocular lens ("IOL"), the matching of the vasculature from a preoperative image to a live view, a tumor at a depth under an artery, etc. . The example stereo visualization camera 300 thus enables the use of, for example, graphic overlays or image analysis to make accurate measurements to determine the size of anatomical structures.

The known surgical operating microscope requires a surgeon to place an object of a known size (such as a micrometer ruler) into the field of view. The surgeon compares the size of the object with the surrounding anatomical structure to determine an approximate size. However, this procedure is relatively slow, because the surgeon must place the object in the proper position and then remove it after performing the measurement. In addition, the measurement only provides an approximation, because the size is based on the subjective comparison and measurement of the surgeon. Some known stereo cameras provide graphic overlays to determine the size. However, if false parallax exists between the left view and the right view, the accuracy of these overlays is reduced. A. ZRP as a source of false parallax

The inaccuracy of ZRP provides a significant source of error between the left and right images, resulting in false parallax. ZRP or zoom repeat point refers to a point in a field of view that remains in the same position when changing a magnification level. Figures 18 and 19 show examples of ZRP in the left view and the right view at one of the different magnification levels. Specifically, FIG. 18 shows a left view area 1800 at a low magnification level and a left view area 1850 at a high magnification level. In addition, FIG. 19 shows a right viewing zone 1900 at a low magnification level and a right viewing zone 1950 at a high magnification level.

It should be noted that FIGS. 18 and 19 show crosshairs 1802 and 1902 to provide an exemplary reference point of the present invention. The reticle 1802 includes a first reticle 1802a located along a y-direction or y-axis and a second reticle 1802b located along an x-direction or x-axis. In addition, the reticle 1902 includes a first reticle 1902a located along a y-direction or y-axis and a second reticle 1902b located along an x-direction or x-axis. In actual implementations, the exemplary stereo visualization camera 300 generally does not include cross-hairs or add cross-hairs to the optical path by default, unless requested by an operator.

Ideally, ZRP should be positioned at a central location or origin. For example, ZRP should be centered in crosshairs 1802 and 1902. However, the inaccuracy of the optical element 1402 and/or the slight misalignment between the optical elements 1402 causes the ZRP to be located away from the center of the crosshairs 1802 and 1902. In addition to the misalignment of the ZRP between the left view and the right view, the degree of false parallax also corresponds to how far each of the ZRP of the left view and the right view is located away from the respective center. In addition, the inaccuracy of the optical element 1402 may cause the ZRP to drift slightly when the magnification is changed, thereby further causing a greater degree of false parallax.

FIG. 18 shows three crescent-shaped objects 1804, 1806, and 1808 in the field of view 1800 and 1850 of the target part 700 of FIG. 7. It should be understood that the viewing zones 1800 and 1850 are related to the linear viewing zones of the optical image sensors 746 and 748. Objects 1804, 1806, and 1808 are placed in the viewing area 1800 to illustrate how to generate false parallax from the misalignment of the left and right images. The object 1804 is positioned on the crosshair 1802b along the crosshair 1802a. The object 1806 is positioned along the crosshair 1802b and on the left side of the crosshair 1802a. The object 1808 is positioned slightly below the reticle 1802b and to the right of the reticle 1802a. A ZRP 1810 of the left view 1800 is positioned in a recess of the object 1808.

The left field of view 1800 is changed to the left field of view 1850 by using the zoom lens assembly 716 of the exemplary stereo visualization camera 300 to increase the magnification level (eg, zoom). Increasing the magnification causes objects 1804, 1806, and 1808 to appear to expand or grow, as shown in view 1850. In the illustrated example, the viewing area 1850 is approximately 3X of the magnification level of the viewing area 1800.

Compared with the low-magnification field of view 1800, the size of the objects 1804, 1806, and 1808 in the high-magnification field of view 1850 have been increased by approximately 3X while also moving relative to the ZRP 1810 to be 3X apart from each other. In addition, the positions of the objects 1804, 1806, and 1808 have moved relative to the crosshair 1802. The object 1804 is now shifted to the left of the reticle 1802a and shifted slightly further away from the reticle 1802b. In addition, the object 1806 is now shifted to the far left of the reticle 1802a and slightly higher than the reticle 1802b. Generally speaking, the object 1808 is located in the same (or almost the same) position relative to the reticle 1802, and the ZRP 1810 is located in the same (or almost the same) position relative to the reticle 1802 and the object 1806. In other words, as the magnification increases, the objects 1804, 1806, and 1808 (and others in the field of view 1850) seem to move away from the ZRP 1810 and outward from the ZRP 1810.

The same objects 1804, 1806, and 1808 are shown in the right views 1900 and 1950 illustrated in FIG. 19. However, the location of ZRP is different. Specifically, the ZRP 1910 is located above the reticle 1902b and to the left of the reticle 1902a in the right viewing zones 1900 and 1950. Therefore, the ZRP 1910 is located at a different position from the ZRP 1810 in the left view zones 1800 and 1850. In the illustrated example, it is assumed that the left optical path and the right optical path are perfectly aligned at the first magnification level. Therefore, the objects 1804, 1806, and 1808 displayed in the right view 1900 are located in the same positions as the objects 1804, 1806, and 1808 in the left view 1800. Since the left view is aligned with the right view, there is no false parallax.

However, in the high-magnification field of view 1950, the objects 1804, 1806, and 1808 expand and move away from the ZRP 1910. Given the position of the ZRP 1910, the object 1804 moves or shifts to the right and the object 1806 moves or shifts downward. In addition, the object 1808 moves down and to the right compared to its position in the viewing area 1900.

FIG. 20 shows a pixel pattern comparing the high-magnification left view area 1850 and the high-magnification right view area. A grid 2000 can represent the positions of objects 1804 (L), 1806 (L) and 1808 (L) on the pixel grid 1004 of the left optical image sensor 748 and the pixel grid 1002 of the right optical image sensor 746 The positions of the above objects 1804(R), 1806(R) and 1808(R) overlap. Figure 20 clearly shows that the objects 1804, 1806, and 1808 are located in different positions for the left view 1850 and the right view 1950. For example, the object 1804(R) is located to the right of the crosshair 1902a and above the crosshair 1902b, while the same object 1804(L) is located to the left of the crosshair 1802a and farther above the crosshair 1802b.

The positional difference of the objects 1804, 1806, and 1808 corresponds to a false parallax, which is formed due to a defect in the optical alignment of the optical element 1402 of the ZRP 1810 and 1910 in different positions. Assuming that there is no distortion or other imaging errors, the false parallax shown in Figure 20 is generally the same for all points in the image. When viewed through the eyepiece of a surgical operating microscope (such as the microscope 200 of FIG. 2), the positional difference of the objects 1804, 1806, and 1808 may not be obvious. However, when viewed in a stereoscopic image on the display monitors 512 and 514, the difference becomes obvious and can cause headaches, nausea, and/or dizziness.

FIG. 21 shows a diagram illustrating the false parallax with respect to the left ZRP and the right ZRP. The figure includes a pixel grid 2100, which includes an overlay of the right pixel grid 1002 and the left pixel grid 1004 in FIG. 10. In the example illustrated here, one of the left optical paths, the left ZRP 2102, is located at +4 along the x-axis and 0 along the y-axis. In addition, one of the right optical paths, the right ZRP 2104, is located at -1 along the x-axis and 0 along the y-axis. An origin 2106 is shown at the intersection of the x-axis and the y-axis.

In this example, the object 2108 is aligned with respect to the left image and the right image at a first low magnification. When the magnification increases by 3X, the object 2108 increases in size and moves away from ZRP 2102 and 2104. The contour object 2110 shows a theoretical position of the object 2108 at the second higher magnification based on the alignment of the ZRP 2102 and 2104 with the origin 2106. Specifically, a notch of the object 2108 at the first magnification level is at a position +2 along the x-axis. In the case of 3X magnification, the notch moves 3X along the x-axis, so that the notch is located at +6 along the x-axis at a higher magnification level. In addition, since the ZRP 2102 and 2104 will be aligned at the origin 2106 in terms of processing, the object 2110 will be aligned between the left view and the right view (in the case of a given overlay, it is shown as a single object in Figure 21). ).

However, in this example, the misalignment of the left ZRP 2102 and the right ZRP 2104 at higher magnification causes the object 2110 to be misaligned between the left view and the right view. Regarding the right optical path, the right ZRP 2104 is located at -1 along the x-axis, making it 3 pixels away from the notch of the object 2108 at low magnification. When zoomed in at 3X, this difference becomes 9 pixels, which is shown as object 2110(R). Similarly, the left ZRP 2102 is located at +4 pixels along the x-axis. At 3X magnification, the object 2108 moves from 2 pixels away to 6 pixels away, which is shown as the object 2110 (L) at -2 along the x-axis.

The position difference between the object 2110 (L) and the object 2110 (R) corresponds to the false parallax between the left view and the right view at a higher magnification. If the right view and the left view are combined into a 3D image for display, the position of the object 2110 will not be aligned at each row (if the renderer program 1580e uses a row interlace mode). Misalignment is detrimental to stereoscopic vision and can produce an image that looks blurry or confuses an operator. B. Other sources of false parallax

Although the ZRP misalignment between the left optical path and the right optical path is a significant source of false parallax, there are other sources of error. For example, false parallax can be caused by unequal magnification changes between the right optical path and the left optical path. The difference in magnification between the parallel optical paths can be caused by slight changes in the optical properties or characteristics of the lens of the optical element 1402. In addition, if each of the left front zoom lens 728 and the right front zoom lens 726 and each of the left rear zoom lens 734 and the right rear zoom lens 732 of FIGS. 7 and 8 are independently controlled, a slight difference can be determined by the positioning cause.

Referring back to FIGS. 18 and 19, the difference in magnification changes produces objects of different sizes and different distances between objects for the left optical path and the right optical path. If, for example, the left optical path has a higher magnification change, the objects 1804, 1806, and 1808 will look larger and more self-contained than the objects 1804, 1806, and 1808 in the right view 1950 in FIG. 19 ZRP 1810 moves a greater distance. Even if the ZRP 1810 is aligned with the ZRP 1910, the difference in the positions of the objects 1804, 1806, and 1808 causes false parallax.

Another source of false parallax is caused by the unequal focus of the left optical path and the right optical path. Generally speaking, any focus difference between the left view and the right view can lead to a perceived reduction in image quality and potential confusion as to whether the left view or the right view should dominate. If the focus difference is significant, it can produce an out-of-focus ("OOF") condition. The OOF condition is especially obvious in stereo images, where the left view and the right view are displayed in the same image. In addition, OOF conditions are not easily correctable, because refocusing an out-of-focus optical path usually causes other optical paths to become out of focus. Generally speaking, it is necessary to determine a point where two optical paths are in focus. This may include changing the positions of the left lens and the right lens along an optical path and/or adjusting a working distance from the target part 700.

Figure 22 shows a diagram illustrating how to form an OOF condition. The diagram correlates the perceived resolution (eg, focus) with a lens position relative to an optimal resolution section 2202. In this example, the zoom lens 734 at the left rear of the position L _1, and the right rear of the zoom lens 732 in the position R _1. L ₁ and at a position at R _1, the zoom lens 732 and 734 in a preferred range of 2202 resolution, so that the left and right optical paths having an optical path through the focusing level matching. However, there is a _{difference between the positions of L 1} and R ₁ , and this difference corresponds to the distance ΔP. At a later time, the working distance 706 is changed so that one point is out of focus. In this example, both the rear zoom lenses 732 and 734 move the same distance to reach the positions L ₂ and R ₂ so that the distance ΔP does not change. However, the position change causes a significant change in one of the resolution ΔR, so that the left rear zoom lens 734 has a higher resolution than the right rear zoom lens 732 (for example, better focus). The resolution ΔR corresponds to the OOF condition, which generates false parallax due to the misalignment between the right optical path and the left optical path.

Another source of false parallax can be caused by imaging an object moving at the target site 700. The false parallax is caused by a small synchronization error between the exposure of the right optical image sensor 746 and the left optical image sensor 748. If the left view and the right view are not recorded at the same time, the object seems to be shifted or misaligned between the two views. The combined 3D image shows the same object at two different positions for the left view and the right view.

In addition, another source of false parallax involves moving the ZRP point during one of the zoom-in periods. The example discussed above in Section IV(A) assumes that the ZRP of the left and right views does not move in the x-direction or the y-direction. However, if the zoom lenses 726, 728, 732, and/or 734 do not move completely parallel to the optical path or optical axis (e.g., in the z-direction), the ZRP may shift during magnification. As discussed above with reference to FIG. 11, the carrier 724 may be slightly displaced or rotated when a force is applied to the actuation section 1108. This rotation can cause the left ZRP and right ZRP to move slightly when changing a magnification level.

In one example, during a magnification change, the carrier 730 moves in a single direction, and the carrier 724 moves in the same direction for a part of the magnification change and moves in the opposite direction for a remaining part of the magnification change. To achieve focus adjustment. If the axis of motion of the carrier 724 is slightly inclined or rotated with respect to the optical axis, the ZRP of the left optical path and/or the right optical path will be shifted in one direction for the first part, and subsequently for the second part of the change in magnification. Shift in the reverse direction. In addition, since the force is not applied equally, the front right zoom lens 726 and the front left zoom lens 728 may experience a ZRP shift of the degree of change between the left optical path and the right optical path. In short, the position change of the ZRP causes misalignment of the optical path, which results in false parallax. C. The reduction of false parallax promotes the integration of digital graphics and images with a stereoscopic view

As surgical operating microscopes become more digital, designers are adding features that overlay graphics, images, and/or other digital effects onto the live view image. For example, guidance overlays, stereo magnetic resonance imaging ("MRI") image fusion, and/or external data can be combined with images recorded by a camera or even displayed in the eyepiece itself. The false parallax reduces the accuracy of the overlay with the basic three-dimensional image. For example, surgeons generally need to use MRI to visualize a tumor as accurately as possible (usually in three dimensions) and place it in a fusion on-site surgical stereoscopic view. Otherwise, preoperative tumor imaging provides little information to the surgeon, thus reducing performance.

For example, a surgical guide can be aligned with a right view image but not aligned with the left view. The misalignment of the surgical guide between the two views is obvious to the operator. In another example, a surgical guide can be individually aligned with the left view and the right view in the information processor module 1408 before the graphics processing unit 1564 forms the combined stereo image. However, the misalignment between the left view and the right view creates a misalignment between the guide plates, thus reducing the effectiveness of the guide plates and creating confusion and delay during the microsurgery procedure.

US Patent No. 9,552,660 (incorporated by reference) entitled " IMAGING SYSTEM AND METHODS DISPLAYING A FUSED MULTIDIMENSIONAL RECONSTRUCTED IMAGE " discloses how preoperative images and/or graphics are visually fused with a stereoscopic image. Figures 23 and 24 show diagrams illustrating how false parallaxes can cause digital graphics and/or images to lose accuracy when fused to a 3D image. FIG. 24 shows a front view of a patient's eye 2402 and FIG. 23 shows a cross-sectional view of the eye along the plane AA of FIG. 24. In FIG. 23, the information processor module 1408 is instructed to determine the distance d from a focal plane 2302 to a tail distance d of an object of interest 2304 on a posterior capsule of the eye 2402, for example. The information processor module 1408 operates a program 1560. The program 1560 specifies (for example) that the distance d is determined by a triangulation calculation of the image data from the left view and the right view of the eye 2402. A view 2306 is shown from a viewing angle of the left optical image sensor 748 and a view 2308 is shown from a viewing angle of the right optical image sensor 746. Assume that the left view 2306 and the right view 2308 coincide with a front center 2310 of the eye 2402. In addition, the left view 2306 and the right view 2308 are two-dimensional views of the object 2304 projected onto a focal plane 2302 by the theoretical right projection 2312 and the theoretical left projection 2314. In this example, the processor 1562 determines the distance d to the object of interest 2304 by calculating an intersection of an extrapolation of the theoretical right projection 2312 and an extrapolation of the theoretical left projection 2314 using a triangulation routine.

However, there is false parallax in this example, which causes an actual left projection 2316 to be located a distance P to the left of the theoretical left projection 2314, as shown in FIGS. 23 and 24. The processor 1562 uses the actual left projection 2316 and the right projection 2312 to determine a distance between an extrapolation of the right projection 2312 and an extrapolation of the actual left projection 2316 and an intersection 2320 using a triangulation measurement routine. The distance of the intersection 2320 is equal to the distance d plus an error distance e . The false parallax therefore uses data obtained from a stereo image to generate an incorrect distance calculation. As shown in Figure 23 and Figure 24, even a small degree of false parallax can produce a significant error. In the context of a fusion image, the wrong distance can lead to inaccurate placement of a tumor three-dimensional visualization used for fusion with a stereo image. This inaccurate placement can delay surgery, hinder the performance of the surgeon, or cause the entire visualization system to be ignored. Worse, a surgeon may rely on inaccurate placement of tumor images and make mistakes during the microsurgery procedure. D. Example stereo visualization camera reduces or eliminates false parallax

The example stereo visualization camera 300 of FIGS. 3-16 is configured to reduce or eliminate visual defects, false parallax, and/or misaligned optical paths that often cause false parallax. In some examples, the stereo visualization camera 300 aligns the ZRP of the left optical path and the right optical path to the pixel groups 1006 and 1008 of the right optical image sensor 746 and the left optical image sensor 748, respectively. Center to reduce or eliminate false parallax. Additionally or alternatively, the stereoscopic visualizing camera 300 can be aimed at the optical path of the left image and the right image. It should be understood that the stereo visualization camera 300 can perform actions to reduce false parallax during calibration. In addition, the stereoscopic visualization camera 300 can instantly reduce the false parallax detected during use.

25 and FIG. 26 illustrate a flowchart of an exemplary procedure 2500 to reduce or eliminate false parallax according to an exemplary embodiment of the present invention. Although the procedure 2500 is described with reference to the flowcharts illustrated in FIGS. 25 and 26, it should be understood that many other methods of performing the steps associated with the procedure 2500 can be used. For example, the order of the blocks in the block can be changed, a specific block can be combined with other blocks, and many of the blocks described are optional. In addition, the actions described in the procedure 2500 can be performed in a plurality of devices including, for example, the optical element 1402 of the exemplary stereo visualization camera 300, the image capturing module 1404, the motor, and the illumination module. Group 1406 and/or information processor module 1408. For example, the program 2500 can be executed by one of the programs 1560 of the information processor module 1408.

The example process 2500 starts when the stereo visualization camera 300 receives an instruction to align the right optical path and the left optical path (block 2502). In response to an operator's request for the stereo visualization camera 300 to execute a calibration routine, the commands can be received from the user input device 1410. In other examples, the commands may be received from the information processor module 1408 after determining that the right image and the left image are not aligned. The information processor module 1408 can determine that the image is misaligned by executing a program 1560 that overlaps the right image and the left image and determines the difference in pixel values, where a larger difference in the large pixel area indicates that the image is not aligned. In some instances, the program 1560 can compare the pixel data of the left and right images without performing an overlay function, where (for example) the left pixel data is subtracted from the right pixel data to determine a serious misalignment sex.

After receiving the instruction to reduce the false parallax, the exemplary stereo visualization camera 300 locates one of the left optical path or the right optical path ZRP. For illustrative purposes, the procedure 2500 includes first determining the ZRP of the left optical path. However, in other embodiments, the procedure 2500 may first determine the ZRP of the right optical path. To determine the left ZRP, the stereo vision camera 300 moves at least one zoom lens (for example, the left front zoom lens 728 and/or the left rear zoom lens 734) to a first magnification level (box 2504). In the example in which the front zoom lenses 726 and 728 are connected to the same carrier 724 and the rear zoom lenses 732 and 734 are connected to the same carrier 730, the movement of the left lens causes the right lens to also move. However, during this section of the process 2500, only the movement of the left lens is considered.

At the first magnification level, the stereo visualization camera 300 causes the left zoom lens to move in the z direction (block 2506). For example, moving may include moving before and after around the first magnification level. For example, if the first magnification level is 5X, the movement can be between 4X and 6X. Movement can also include movement in one direction (such as from 5X to 4X). During this movement, the stereo vision camera 300 can adjust one or more other lenses to maintain the focus of the target part 700. At block 2508, during the movement of the left zoom lens, the stereo visualization camera 300 uses (for example) the left optical image sensor 748 to record a stream or a sequence of images of the target site 700 and/or frame 2509 . The image 2509 is recorded using an oversized pixel group 1008 configured to include an origin of the pixel grid 1004 and one of the possible positions of the left ZRP.

The example processor 1562 of the information processor module 1408 analyzes the image stream to locate a portion of the region that does not move in an x direction or a y direction between the images (block 2510). This part of the zone may contain one or several pixels and correspond to the left ZRP. As discussed above, during a change in magnification, the object moves away from the ZRP or moves toward the ZRP. When the magnification changes, only the object at the ZRP keeps its position relative to the field of view. The processor 1562 can use the pixel data to calculate the difference between the image streams of each pixel. The area with the smallest difference across the video stream corresponds to the left ZRP.

The example processor 1562 of the information processor module 1408 then determines the coordinates of a part of the region that does not move between image streams relative to the pixel grid 1004 (for example, determines a position of the left ZRP) (block 2512). In other examples, the processor 1562 of the information processor module 1408 determines a distance between the origin and the part of the zone (corresponding to the left ZRP). This distance is used to determine a position of the left ZRP on the pixel grid 1004. Once the position of the left ZRP is determined, the processor 1562 of the information processor module 1408 determines a pixel group of the left optical image sensor 748 (for example, the pixel group 1008) so that the left ZRP is located at the center of one of the pixel groups (in the Within one pixel) (block 2514). At this time, the left ZRP is centered in the left optical path.

In some examples, blocks 2504 to 2514 may be repeatedly executed by reselecting pixel groups until the left ZRP is within one pixel of the origin and the false parallax is minimized. After determining the pixel grid, the processor 1562 of the information processor module 1408 stores at least one of the coordinates of the pixel group and/or the coordinates of the left ZRP as a calibration point in the memory 1570 (block 2516). The processor 1562 of the information processor module 1408 can associate the first magnification level with the calibration point, so that the same pixel group is selected when the stereo visualization camera 300 returns to the first magnification level.

FIG. 27 shows a diagram illustrating how to adjust the left ZRP relative to the pixel grid of the left optical image sensor 748. Initially, an initial (for example, too large) pixel group 2702 centered on the origin 2704 is selected. The pixel group 2702 is large enough to record possible ZRP in the video stream. In the example illustrated here, a left ZRP 2706 is located above the origin 2704 and to the right of the origin 2704. The processor 1562 of the information processor module 1408 determines the pixel group 2708 based on a position of the left ZRP 2706, so that the left ZRP 2706 is located or positioned at the center of a pixel group 2708.

After determining the left ZRP in FIG. 25 and aligning the left ZRP with an origin of a pixel group, the example procedure 2500 aligns the left image and the right image in FIG. 26. To align the images, the example processor 1562 compares the pixel data from the left and right images recorded after aligning the left ZRP with the origin. In some embodiments, the processor 1562 overlays the left image and the right image to determine the difference using, for example, a subtraction and/or template method. The processor 1562 selects or determines a pixel group of the right optical path so that the resulting right image is aligned with or coincides with the left image (block 2519).

In the illustrated embodiment, the example processor 1562 determines the right ZRP. The steps are similar to the steps discussed for the left ZRP in blocks 2504-2512. For example, at block 2518, the stereo visualization camera 300 moves a right zoom lens to the first magnification level. In some embodiments, the magnification level of the right lens is different from the magnification level used to determine the left ZRP. The example processor 1562 of the information processor module 1408 then moves the right zoom lens around the magnification level and receives an image stream 2521 from the right optical image sensor 746 during the movement (blocks 2520 and 2522). The example processor 1562 of the information processor module 1408 determines the right ZRP from the right image stream by locating a part of an area that does not move between images (block 2524). The processor 1562 next determines the coordinates of the right ZRP and/or a distance between a center of the aligned pixel group 1006 and the right ZRP (block 2526).

The processor 1562 then instructs the motor and illumination module 1406 to move at least one lens in the right optical path in at least one of an x-direction, a y-direction, and/or an oblique direction to use (for example) the right ZRP The distance or coordinates align the right ZRP with the center of the aligned pixel group 1006 (block 2528). In other words, the right ZRP is moved to coincide with the center of the aligned pixel group 1006. In some examples, the right front lens 720, the right lens barrel 736, the right last optical element 745, and/or the right image sensor 746 are arranged in the x direction, the y direction and/or a direction relative to the z direction of the right optical path. Move in an oblique direction (for example, using a flexure). The degree of movement is proportional to the distance between the right ZRP and the center of the pixel group 1006. In some embodiments, the processor 1562 digitally changes the properties of the right front lens 720, the right lens barrel 736, and/or the right last optical element 745 to have the same effect as moving the lens. The processor 1562 may repeat steps 2520 to 2528 and/or use subsequent right images to confirm that the right ZRP is aligned with the center of the pixel group 1006 and/or repeatedly determine the additional lens movement required to align the right ZRP with the center of the pixel group .

The example processor 1562 stores the coordinates of the right pixel group and/or the right ZRP as a calibration point in the memory 1570 (block 2530). The processor 1562 can also store a position of the right lens moved to align with the right ZRP to the calibration point. In some instances, the calibration points of the right optical path are stored together with the first magnification level and the calibration points of the left optical path. Therefore, when the stereo visualization camera 300 is subsequently set to the first magnification level, the processor 1562 applies the data in the calibration point to the optical image sensors 746 and 748 and/or the radial positioning of the one or more optical elements 1402.

In some instances, the procedure 2500 may be repeated for different magnification levels and/or working distances. Therefore, the processor 1562 determines whether ZRP calibration is required for another magnification level or working distance (block 2532). If another magnification level is to be selected, the process 2500 returns to block 2504 in FIG. 25. However, if another magnification level is not required, the example procedure ends.

Each of the calibration points can be stored in a look-up table. Each column in the table can correspond to a different magnification level and/or working distance. The rows in the lookup table can provide the coordinates of the left ZRP, the right ZRP, the left pixel group, and/or the right pixel group. In addition, one or more rows can specify the relative position of the lens of the optical element 1402 (for example, radial, rotation, tilt, and/or axial position) so as to achieve magnification in addition to aligning the right and left images. Focus on the level.

The process 2500 thus causes the right ZRP and the left ZRP, except for the view of the target site, to be aligned to the pixel grids of the respective optical image sensors 746 and 748 and to each other in a three-dimensional image. In some cases, the left and right images and the corresponding ZRP have an accuracy and alignment within one pixel. By overlaying the left view and the right view (for example, images from the left optical path and the right optical path) and observing the two views with two eyes instead of stereoscopically, this accuracy can be achieved on the display 514 or 514 Observed.

It should be understood that, in some examples, a right pixel group is first selected so that the right ZRP is aligned with or coincides with an origin of the pixel group. Then, the right optical image and the left optical image can be aligned by moving one or more of the right lens and/or the left lens of the optical element 1402. This alternative procedure still provides right ZRP and left ZRP centered and aligned with respect to the optical image sensors 746 and 748 between each other.

The process 2500 finally reduces or eliminates the false parallax in the stereo visualization camera 300 by ensuring that the left ZRP and the right ZRP remain aligned and the right image and the left image remain aligned within a full optical magnification range. In other words, the dual optics of the right optical image sensor 746 and the left optical image sensor 748 are aligned so that the parallax between the left optical path and the right optical path at a center of an image is approximately at the focal plane. zero. In addition, the exemplary stereo visual camera 300 is isofocal across the magnification range, and is isocentric across the magnification and working distance ranges, because the ZRP of each optical path has been aligned to one of the respective pixel groups center. Therefore, only changing the magnification will keep one of the target sites 700 focused in both the optical image sensors 746 and 748, while being trained on the same center point.

The above procedure 2500 may be performed before performing a surgical procedure and/or at the time of calibration based on an operator request. The example procedure 2500 may also be performed before the image registration with a preoperative microsurgery image and/or surgical guidance figure. In addition, the example procedure 2500 can be automatically executed in real time during the operation of the stereo visualization camera 300. 1. Example of template matching

In some embodiments, the example processor 1562 of the information processor module 1408 is configured to use a program 1560 in conjunction with the same or multiple templates to determine one of the right ZRP and/or left ZRP positions. FIG. 28 shows a diagram illustrating how the processor 1562 uses a target template 2802 to determine a position of a left ZRP. In this example, FIG. 28 shows a first left image that includes a template 2802 aligned with an origin 2804 or center of the left pixel grid 1004 of the left optical image sensor 748. The template 2802 can be aligned by moving the stereo visualization camera 300 to an appropriate position. Alternatively, the template 2802 can be moved at the target site 700 until it is aligned. In other examples, the template 2802 may include another pattern that does not need to be centered on one of the pixel grids 1004. For example, the template may include a graphic wave pattern, a graphic pneumograph pattern, a view of a surgical site of a patient, and/or a visually distinguishable feature (with certain characteristics in both the x-direction and the y-direction). A degree of non-periodical) one of the grids. The templates are configured to prevent a subset of a periodic image from being completely aligned on the larger image in a plurality of positions, which makes these templates unsuitable for matching. A template image suitable for template matching is called a "template matchable" template image.

The template 2802 shown in Figure 28 is imaged at a first magnification level. A left ZRP 2806 is shown relative to the template 2802. ZRP 2806 has coordinates of _{L x} , L _y relative to the origin 2804. However, at this point in time, the processor 1562 has not yet recognized the left ZRP 2806.

To position the ZRP 2806, the processor 1562 causes a left zoom lens (for example, the left front zoom lens 728 and/or the left rear zoom lens 734) to change the magnification from a first magnification level to a second magnification level, specifically In this example, change from 1X to 2X. FIG. 29 shows a diagram of a second left image that includes the target 2802 on the pixel grid 1004 when the magnification level is doubled. From the first magnification level to the second magnification level, the size of the part of the target 2802 increases and expands evenly away from the left ZRP 2806, which remains fixed relative to the first image and the second image. In addition, the distance between the origin 2804 of the pixel grid 1004 and the left ZRP 2806 remains the same.

The example processor 1562 synthesizes a digital template image 3000 based on the second image shown in FIG. 29. To form the digital template image, the processor 1562 copies the second image shown in FIG. 29 and scales the copied image by the reciprocal of the magnification change from the first magnification to the second magnification. For example, if the magnification change from the first image to the second image is a factor of 2, the second image is scaled by ½. FIG. 30 shows a diagram of a digital template image 3000 including a template 2802. The template 2802 in the digital template image 3000 of FIG. 30 is scaled to the same size as the template 2802 of the first left image shown in FIG. 28.

The example processor 1562 uses the digital template image 3000 to locate the left ZRP 2806. FIG. 31 shows a diagram showing the digital template image 3000 superimposed on top of the first left image recorded in the pixel grid 1004 (or a subsequent left image recorded at the first magnification level). The combination of the digital template image 3000 and the first left image produces a composite view, as illustrated in FIG. 31. Initially, the digital template image 3000 is centered at the origin 2804 of the pixel grid 1004.

The example processor 1562 compares the digital template image 3000 with the basic template 2802 to determine whether they are aligned or matched. The example processor 1562 then causes the digital template image 3000 to move one or more pixels horizontally or vertically and perform another comparison. The processor 1562 makes the digital template image 3000 move repeatedly, thereby compiling a measurement matrix for how closely the digital template image 3000 matches the basic template 2802 for each position. The processor 1562 selects the position in the matrix corresponding to the best matching metric. In some instances, the processor 1562 uses the OpenCV ^™ template matching function.

FIG. 32 shows a diagram in which the digital template image 3000 and the template 2802 are aligned. The distance to move the digital template image 3000 to achieve the best match is shown as Δx and Δy. Knowing that the digital template image 3000 was synthesized at a ratio of M1/M2 (the first magnification level divided by the second magnification level), the processor 1562 uses the following equations (1) and (2) to determine the coordinates of the left ZRP 2806 (L _x , L _y ). Lx = Δx/ (M1/M2)-equation (1) Ly = Δy/ (M1/M2)-equation (2)

After determining the coordinates (L _x , L _y ) of the left ZRP 2806, the exemplary processor 1562 selects or determines a pixel subgroup having an origin aligned with or coincident with the left ZRP 2806, as described above in conjunction with FIGS. 25 and Procedure 2500 of 26 is discussed. In some embodiments, the processor 1562 may repeatedly use template matching to converge on a highly accurate ZRP position and/or pixel subgroup. In addition, although the above example discusses locating the left ZRP, the same template matching procedure can be used to locate the right ZRP.

In some embodiments, the template matching program 1560 described above can be used to align the left and right images. In these embodiments, the left image and the right image are recorded at a magnification level. For example, the two images may include the target template 2802 of FIG. 28. A part of the right image is selected and overlaps the left image. This part of the right image is then shifted horizontally and/or vertically by one or more pixels around the left image. The example processor 1562 performs a comparison at each position of the portion of the right image to determine how close a match exists with respect to the left image. Once an optimal position is determined, it is determined that a pixel group 1006 of the right pixel grid 1002 makes the right image and the left image substantially coincide. The position of the pixel group 1006 can be determined based on the amount by which the part of the right image is moved to coincide with the left image. Specifically, the processor 1562 uses one of the movement amounts in the x direction, the y direction, and/or the oblique direction to determine the corresponding coordinates of the right pixel group 1006. 2. Example of aligning the right image with the left image

In some embodiments, the example processor 1562 of the information processor module 1408 of FIGS. 14-16 displays an overlay of the right image and the left image on the display monitors 512 and/or 514. The processor 1562 is configured to receive user feedback for aligning the right and left images. In this example, each pixel data of the right image and the left image is accurately mapped to a respective pixel of the display monitor 512 using, for example, the graphics processing unit 1564. The display of the overlaid left and right images makes any false parallax obvious to an operator. Generally speaking, without false parallax, the left and right images should be almost completely aligned.

If an operator detects false parallax, the operator can actuate the control 305 or the user input device 1410 to move the right image or the left image for alignment with the other of the right image and the left image . The instruction from the control 305 can cause the processor 1562 to adjust the position of the left pixel group or the right pixel group accordingly in real time, so that subsequent images reflecting the operator's input are displayed on the display monitor 512. In other examples, the instructions may cause the processor 1562 to change the position of one or more of the optical elements 1402 via radial adjustment, rotation adjustment, axial adjustment, or tilt. The operator continues to provide input via the control 305 and/or the user input device 1410 until the left image and the right image are aligned. After receiving a confirmation command, the processor 1562 is about to store a calibration point reflecting the image alignment under the set magnification level in a look-up table.

Additionally or alternatively, the template matching method described above can be used to perform image alignment while focusing on a plane target that is substantially orthogonal to a stereo optical axis of the stereo visualization camera 300. In addition, whenever a "template can be matched" scene is in both views of the left optical path and the right optical path, the template matching method can be used to instantly align the left view and the right view. In one example, a template image is copied from, for example, a subset of the left view, centered on or near the center of the view. Sampling from the center of a focused image ensures that a similar view of the target site 700 will exist in another view (the right view in this example). For out-of-focus images, this is not the case, so that the alignment method is performed only after a successful autofocus operation in the current embodiment. The selected template is then matched to the current view (or a copy thereof) of another view (the right view in this example) and only a y value is obtained from the result. When the views are aligned vertically, the y value of the template matching is at or near the zero pixel. A non-zero y value indicates the vertical misalignment between the two views and a correction using one of the same values of y is applied to select the pixel readout group of the first view or a correction using one of the negative values of y is applied to the other view The pixel readout group. Another option is to apply split correction in other parts of the visualization pipeline or between the pixel readout group and the pipeline.

In some instances, the operator can also manually align a right ZRP with an origin of the pixel grid 1002. For example, after determining a position of the right ZRP, the processor 1562 (and/or the peripheral input unit interface 1574 or the graphics processing unit 1564) causes the right ZRP to be graphically highlighted on a right image displayed by the display monitor 512 superior. The processor 1562 may also display a graphic indicating the origin of the pixel grid 1002. The operator uses the control 305 and/or the user input device 1410 to manipulate the right ZRP to the origin. The processor 1562 uses commands from the control 305 and/or the user input device 1410 to move one or more of the optical elements 1402 accordingly. In addition to graphically displaying the current position of the right ZRP and the origin, the processor 1562 can also provide a right image stream in real time to provide updated feedback on the positioning of the operator. The operator continues to provide input via the control 305 and/or the user input device 1410 until the right ZRP is aligned. After receiving a confirmation command, the processor 1562 is about to store a calibration point reflecting the position of the optical element 1402 under the set magnification level in a look-up table. 3. Comparison of alignment errors

Compared with a known digital surgical operating microscope with a stereo camera, the exemplary stereo visualization camera 300 produces less alignment error between the right image and the left image. The analysis discussed below is for comparing the false parallax caused by ZRP misalignment with one of the known digital surgical operating microscopes with cameras and the exemplary stereo visualization camera 300. Initially, the two cameras are set at a first magnification level, and one of the focal planes is positioned at a first position of a patient's eye. Equation (3) is used below to determine the working distance ("WD") from each camera to the eye. WD = (IPD / 2) / tan(α)-equation (3)

In this equation, IPD corresponds to the distance between pupils, which is approximately 23 mm. In addition, α is, for example, one-half of an angle between the right optical image sensor 746 and the left optical image sensor 748, which is 2.50° in this example. The condensing angle is twice this angle, which is 5° in this example. The resulting working distance is 263.39 mm.

The camera zooms in to a second zoom level and triangulates on a second position of the patient's eyes. In this example, the second position is at the same physical distance from the camera as the first position, but is presented at the second magnification level. The change in magnification produces false horizontal parallax due to the misalignment of one or both of the ZRP relative to a center of a sensor pixel grid. For the known camera system, it is determined that the false parallax is (for example) 3 arc minutes, which corresponds to 0.05°. In equation (3) above, the 0.05° value is added to α, which produces a working distance of 258.22 mm. The working distance difference is 5.17 mm (263.39 mm – 258.22 mm), which corresponds to the error of known digital surgical operating microscopes with camera attachments.

In contrast, the exemplary stereo visualization camera 300 can automatically align the ZRP to one pixel in the center of a pixel group or grid. If the angular field of view is 5° and is recorded with a 4k image sensor connected to the same 4k display monitor, then one pixel accuracy corresponds to 0.00125° (5°/4000) or 4.5 arc seconds. Using equation (3) above, the value of 0.00125° is added to α, which results in a working distance of 263.25 mm. The working distance difference of the stereo vision camera 300 is 0.14 mm (263.39 mm – 263.25 mm). When compared with the 5.17 mm error of a known digital surgical operating microscope, the exemplary stereo vision camera 300 reduces the alignment error by 97.5%.

In some embodiments, the stereoscopic visualization camera 300 may be more accurate at a higher resolution. In the above example, the resolution is approximately 4.5 arc seconds for a 5° field of view. For an 8K ultra-high-definition system with a field of view of 2° (in which 8000 pixels are in each of 4000 columns), the resolution of the stereo visualization camera 300 is approximately 1 arc second. This means that the ZRP of the left and right views can be aligned to one pixel or one arc second. This is significantly more accurate than known digital microscope systems with pseudo parallaxes of approximately several arc minutes. 4. Reduction of other sources of false parallax

The above example discusses how the exemplary stereo visualization camera 300 reduces the false parallax caused by the misalignment of the ZRP and/or the left and right images themselves. The stereo visualization camera 300 can also be configured to reduce other sources of false parallax. For example, the stereo visualization camera 300 can reduce the false parallax caused by motion by simultaneously timing the right optical image sensor 746 and the left optical image sensor 748 that record images at substantially the same time.

The exemplary stereo visualization camera 300 can also reduce false parallax caused by the dissimilar magnification between the left optical path and the right optical path. For example, the stereo visualization camera 300 can set the magnification level based on the left optical path. The stereoscopic visualization camera 300 can then automatically adjust so that the magnification of the right image matches the magnification of the left image. For example, the processor 1562 may, for example, use the image data to calculate the control parameter by measuring the number of pixels between a specific feature that is common in the left image and the right image. The processor 1562 can then equalize the magnification levels of the left and right images by digital scaling, inserting interpolated pixels, and/or deleting extraneous pixels. The example processor 1562 and/or the graphics processing unit 1564 can reproduce the right image so that the magnification is matched to the left image. Additionally or alternatively, the stereo vision camera 300 may include independent adjustments of the left optical element and the right optical element 1402. The processor 1562 can individually control the left optical element and the right optical element 1402 to achieve the same magnification. In some instances, the processor 1562 may first set (for example) the left magnification level, and then individually adjust the right optical element 1402 to achieve the same magnification level.

The exemplary stereo visualization camera 300 can further reduce false parallax caused by dissimilar focal points. In one example, the processor 1562 can execute a program 1560 for determining the best focus of each optical path for a given magnification and/or working distance. The processor 1562 first executes the optical element 1402 to focus at one of the best resolution points. The processor 1562 can then check the OOF condition at a suitable non-object plane position and match the focus of the left and right images. The processor 1562 then rechecks the focus at the best resolution and repeatedly adjusts the focus until both the left optical element and the right optical element 1402 are equally well focused on an object plane and away from the object plane.

The example processor 1562 can measure and verify the best focus by monitoring a signal related to the focus of one or both of the right and left images. For example, the graphics processing unit 1564 simultaneously and/or synchronously generates a "resolution" signal for the left image and the right image. The signal changes as the focus changes and can be determined based on an image analysis program, an edge detection analysis program, a pattern intensity Fourier transform bandwidth program, and/or a modulation transfer function ("MTF") measurement program . The processor 1562 adjusts a focus of the optical element 1402 while monitoring a maximum signal indicating a clear image.

To optimize OOF conditions, the processor 1562 can monitor the sharpness signals of both the left and right images. If the focus is moved away from the object plane and the signal related to, for example, the left image increases but the signal related to the right image decreases, the processor 1562 is configured to determine that the optical element 1402 moves away from the focus. However, if the signals related to both the right image and the left image are relatively high and approximately equal, the processor 1562 is configured to determine that the optical element 1402 is properly positioned for focusing. 5. The benefits of low false parallax

The exemplary stereo visualization camera 300 has several advantages over known digital surgical operating microscopes due to the low false parallax between the right image and the left image. For example, aligning the left image and the right image almost completely creates an almost perfect stereoscopic display for a surgeon, thereby reducing eye fatigue. This allows the stereo visualization camera 300 to be used as an extension of a surgeon's eye rather than a heavy tool.

In another example, the precise alignment of the left and right images allows accurate measurement of the surgical site in a digital manner. For example, the size of a lens capsule of a patient's eyepiece can be measured so that an appropriately sized IOL can be implanted judiciously and accurately. In another example, the movement of a moving blood vessel can be measured so that an infrared fluorescein overlay can be accurately placed in a fusion image. Here, the actual movement speed is generally not of concern to the surgeon, but it is critical for the placement and real-time adjustment of the overlay image. To provide an accurately fused combined live 3D image and an alternative mode image, the proper proportion, registration, and viewing angle of the overlay image are all important.

In some instances, the processor 1562 may enable an operator to plot the measurement parameters on the display monitor 512. The processor 1562 receives the drawn coordinates on a screen and thus transforms these coordinates into a stereoscopic image. The processor 1562 can determine the measurement value by scaling the scale on the display monitor 512 to one of the magnification levels shown in the stereo image. The measurement performed by the processor 1562 includes point-to-point measurement, point-to-surface measurement, surface characterization measurement, volume determination measurement, speed verification measurement, and coordinates of two or three positions displayed in the stereo display Transformation, instrumentation and/or organization tracking, etc. VII. Example Robot System of Stereo Visual Camera

As discussed in conjunction with FIGS. 5 and 6, an exemplary stereo visualization camera 300 can be connected to a robotic arm or robot arm 506 as a part of a stereo visualization platform or stereo robot platform 516. The example robotic arm 506 is configured to enable an operator to position and/or orient the stereo visualization camera 300 on and/or next to a patient during one or more procedures. Therefore, the robot arm 506 enables an operator to move the stereo visualization camera 300 to a desired field of view ("FOV") of a target surgical site. Surgeons generally prefer to position and/or orient the camera in a FOV similar to its own FOV in order to achieve easier visual orientation and correspondence between the image displayed on a screen and the FOV of the surgeon. The exemplary robotic arm 506 disclosed herein provides structural flexibility and auxiliary control to enable positioning to coincide or coincide with a surgeon's FOV without blocking the surgeon's own FOV.

Compared with the three-dimensional robot platform 516 disclosed herein, the known three-dimensional microscope holding device includes a simple mechanical arm that is manually moved by an operator. These devices include multiple rotary joints equipped with electromechanical brakes that allow manual repositioning. In addition, in order to allow an operator to easily change a view without interrupting a program, some known holding devices have motorized joints. These motorized joints have levels of complexity ranging from (for example) simple X-Y positioning up to devices including the manipulation of multiple independent rotary joints connected to rigid arms. During most procedures, it is desirable to quickly and easily obtain views from various directions. However, known stereo microscope holding devices suffer from one or more problems.

The known stereo microscope holding device has a limited position, direction, and/or orientation accuracy that is generally limited by the manual ability of the surgeon to manipulate the microscope to view images in a desirable manner. A holding device with multiple joints can be particularly cumbersome to handle, because device manipulation usually causes all joints to move at the same time. From time to time, an operator watches how one arm moves. After the arm is positioned in a desired position, the operator checks whether the FOV of the imaging device is aligned in the desired position. In many cases, even if the device is properly aligned, one of the focal points of the device must be adjusted. It is additionally known that the stereo microscope holding device cannot provide a consistent FOV or focal plane with respect to other objects in a target surgical site. This is because the device does not have arm position memory, or these memories move a patient during a procedure Or it is inaccurate at the time of displacement.

The known stereo microscope holding device generally has a positioning system in which the control is independent of microscope parameters (such as the focal length of the object plane, magnification, and illumination). For these devices, the coordination of positioning and (for example) zooming must be performed manually. In one example, an operator can reach a lens limit for focusing or changing a working distance. The operator must manually change the position of one of the holding devices and then refocus the stereo microscope.

The known stereo microscope holding device is only intended to be used for observation of a surgical site. These known devices do not determine the position or distance from the tissue within a FOV to another object outside the FOV. These known devices also do not provide an alternative viewing modality such as combining an MRI image and a live view by comparing tissues in an on-site surgical site with other objects to form an alternative viewing modality. Alternatively, the view from the known device is displayed separately and is not aligned with other medical images or templates.

In addition, the known stereo microscope holding device has parameters that may be inaccurate, because accuracy (except observation) is rarely emphasized. The requirements in ISO Standard 10936-1:2000(E) " Optics and optical instruments – Operation microscopes – Part 1: Requirements and test methods " are largely derived to allow a normal human operator to use eyepieces to achieve reasonable stereoscopic optical images quality. The operator's brain combines the views into inner images to achieve stereo vision. Generally, these views are not combined or compared together in other ways. As long as the operator sees an acceptable image and does not suffer from harmful effects such as headaches, his needs are met. The same holds true for stereo microscope holding devices, where certain instabilities, arm sag and inaccurate movement control are permitted. However, when using high-resolution digital cameras with known holding devices, structural inaccuracies can be easily observed and their use can be reduced, especially for microsurgery procedures.

As mentioned above, the known stereo microscope holding device can sag due to the weight of a camera. Generally speaking, known robot positioning systems are calibrated to determine compliance or inaccuracy only for the system itself. These stereo microscope holding devices do not consider any inaccuracy between the camera or a camera mount and the holding device. Sag is generally compensated by an operator manually positioning the camera while observing an image on a display. In a system that provides motorized motion, for example, when a camera's center of gravity ("CG") is repositioned on an opposite side of an arm joint on an opposite side of an axis of rotation, the sag changes, in which the rotation around the axis is restored Moment will reverse direction. Subsequently, any compliance or sag in the mechanism (which is compensated by an operator by adjusting the position, direction, and/or orientation of the camera) now adds position, direction, and/or orientation errors. In some cases, for example, when the camera is moved through a robot singularity, the torque reversal occurs quickly and the error of the resulting camera image is quickly and excessively shifted. This error limits the ability of known stereo microscope holding devices (for example) to accurately follow or track tissues or instruments in a site.

The known stereo microscope holding device includes features for spatially positioning and tracking surgical instruments and providing their subsequent representative display on a monitor. However, these known systems need to highlight and locate an additional stereotactic camera or triangulation measurement device and a conspicuous reference device on the instrument. The added devices add complexity, cost, and operational rashness.

The exemplary stereo robotic platform 516 disclosed herein includes an exemplary stereo visualization camera 300 connected to a robotic arm or robotic arm 506. 5 and 6 illustrate an example of the three-dimensional robot platform 516. The three-dimensional images recorded by the camera 300 are displayed via one or more display monitors 512 and 514. The robot arm 506 is mechanically connected to a truck 510, and the truck 510 can also support one or more of the display monitors 512 and 514. For example, the robot arm may include an articulated robot arm that is generally anthropomorphic in terms of size, nature, function, and operation.

FIG. 33 shows a side view of the microsurgery environment 500 of FIG. 5 according to an exemplary embodiment of the present invention. In the illustrated example, the display monitor 512 may be connected to the truck 510 via a robot arm 3302 having one or more joints to achieve elastic positioning. In some embodiments, the robotic arm 3302 may be long enough to extend over a patient during surgery to provide a relatively close view of a surgeon.

FIG. 33 also illustrates a side view of the stereo robot platform 516 including the stereo visualization camera 300 and the robot arm 506. The camera 300 is mechanically coupled to the robot arm 506 via a coupling board 3304. In some embodiments, the coupling plate 3304 may include one or more joints that provide an additional degree of positioning and/or orientation of the camera 300. In some embodiments, the coupling plate 3304 must be manually moved or rotated by an operator. For example, the coupling plate 3304 may have a configuration that enables the camera 300 to be quickly positioned with an optical axis along a z-axis (that is, pointing downward toward a patient) and an optical axis along an x-axis or y-axis. A joint between the shafts (that is, pointing laterally toward a patient).

The example coupling plate 3304 may include a sensor 3306 configured to detect the force and/or torque applied by an operator to move the camera 300. In some embodiments, an operator can position the camera 300 by grasping the control arms 304a and 304b (shown in FIG. 3). After the operator has used his hands to control the control arms 304a and 304b, the user can position and/or orient the camera 300 with the assistance of the robot arm 306. The sensor 3306 detects a force vector or torque angle provided by the operator. The exemplary platform 516 disclosed herein uses the sensed force/torque to determine which joints of the robot arm 506 should be rotated (and how fast the joints should be rotated) to provide a corresponding force/torque provided by the operator Auxiliary movement of the camera 300. The sensor 3306 may be located at an interface between the coupling plate 3304 and the camera 300 for detecting the force and/or torque applied by an operator through the control arm 304.

In some embodiments, the sensor 3306 may include, for example, a six-degree-of-freedom tactile force sensing module. In these embodiments, the sensor 3306 can detect the translational force or movement on the x-axis, y-axis, and z-axis. The sensor 3306 can also separately detect the rotational force or movement around a pitch axis, a pitch axis, and a roll axis. The decoupling of the translational force and the rotational force allows the three-dimensional robot platform 516 to more easily calculate the direct and/or inverse kinematics for controlling the robot arm 506.

The example sensor 3306 can be configured to detect force because the robot arm 506 cannot be moved by a user alone. Alternatively, the sensor 3306 detects the translation and rotation force applied by a user, and the three-dimensional robot platform 516 uses the translation and rotation force to determine which joints rotate to provide auxiliary movement control of the robot arm 506. In other examples, the robotic arm 506 may permit the operator to move without assistance or at least initial assistance. In these other examples, the sensor 3306 detects the movement imparted by the user, and the three-dimensional robot platform 516 uses the movement to subsequently cause one or more joints to rotate, thereby providing auxiliary movement. The time between the initial detection of movement or force generating movement until the rotation of the joints caused by the three-dimensional robot platform 516 can be less than 200 milliseconds ("ms"), 100 ms, 50 ms, or as little as 10 ms, where the user does not pay attention to the robot The initial time of the unassisted movement of the arm 506.

The example sensor 3306 can output digital data indicating rotation force/motion and digital data indicating translation force/motion. In this example, the digital data can have a resolution of 8, 16, 32, or 64 bits for the detected force/motion on each axis. Alternatively, the sensor 3306 can transmit an analog signal proportional to the sensed force and/or motion. The example sensor 3306 can (for example) transmit data at a periodic sampling rate of 1 ms, 5 ms, 10 ms, 20 ms, 50 ms, 100 ms, etc. Alternatively, the sensor 3306 can provide a nearly continuous stream of force/motion data.

In some embodiments, the instance sensor 3306 may alternatively be located in one or more of the control arms 304a and 304b or between the control arms 304a and 304b and the housing 302. In an example where each of the control arms 304a and 304b includes a sensor 3306, the example three-dimensional robot platform 516 can receive two sets of translational and rotational forces or movements. In these examples, the three-dimensional robot platform 516 can average the values from the sensors 3306.

In the illustrated embodiment, a first end of the robotic arm 506 is mounted to the truck 510, and a second opposite end of the robotic arm is mechanically connected to the stereo visualization camera 300 (eg, a robotic end effector). Figure 33 shows the robotic arm 506 holding the stereo visualization camera 300 in an extended position (such as positioning the stereo visualization camera 300 over a surgical site while keeping the rest of the platform 516 out of the way of a surgeon). The truck 510 is configured to firmly hold the three-dimensional robot platform 516 and is weighted and balanced to prevent tipping at the designated operating position.

The exemplary three-dimensional robotic platform 516 is configured to provide the following benefits. 1. Enhanced visualization. The communication between the robotic arm 506 and the stereo visualization camera 300 enables the platform 516 to point the camera 300 and manipulate the camera 300 to quickly and more accurately visualize the surgical site. For example, the robot arm 506 can move the camera 300 along its optical axis to extend the focus and zoom range beyond the range contained only in the camera. The relatively small size of the platform 516 provides Heads-Up Surgery® in a wider variety of surgical procedures and orientations, thereby improving surgical efficiency and surgeon ergonomics. 2. Enhanced dimensional performance. The exemplary stereo visualization camera 300 is configured to transmit the measurement information to the robot arm 506 when it has its ability to accurately measure all points in the stereo image. The robot arm 506 also includes accurate position, direction, and/or orientation determination capabilities and is registered to the camera 300, so that the dimensions in and between images can be accurately transformed to the stereo robot platform 516 and an anatomical structure of a patient. A standard system. 3. The quality and accuracy of the stereo image data from the stereo visualization camera 300 enables it to be combined with images or diagnostic data from external sources of various modalities to construct a fusion image. These fused images can be used by surgeons to perform procedures more accurately and efficiently and achieve better patient outcomes. 4. The stereo vision camera 300, the robot arm 506, and/or the image and motion processor (for example, the processor 1408 of FIG. 14) can be programmed to achieve beneficial program applications. For example, the location, direction, and/or orientation of a specific visualization site can be saved, and then later returned in the program. The precise movement path can be programmed to, for example, follow a specific length or line of tissue. In other examples, pre-programmed path points may be set, thereby allowing an operator to change a position and/or orientation of the robot arm 506, based on which steps are performed during a medical procedure. 5. The three-dimensional robot platform 516 provides essentially guided surgery by using and analyzing accurate image location information. This guidance can be transferred to other devices, such as another robotic system that performs at least portions of a surgical procedure. The components of the three-dimensional robot platform 516 that share functionality with the components of these other devices can be integrated into a package to achieve performance, accuracy, and cost efficiency. A. Robot arm embodiment

Figure 34 illustrates an embodiment of an exemplary robotic arm 506 in accordance with an exemplary embodiment of the present invention. In some embodiments, the robot arm 506 is similar to or includes the model UR5 from Universal Robots S/A. The outer surface of the robot arm 506 includes aluminum and plastic materials, which are compatible for use in an operating room and easy to clean.

Although the robotic arm 506 is described herein as being electromechanical, in other examples, the robotic arm 506 may be mechanical, hydraulic, or pneumatic. In some embodiments, the robot arm 506 may have, for example, a hybrid actuation mechanism that uses a pinhole suction cup together with a control valve to hold and manipulate the camera 300. In addition, although the robot arm 506 is described below as including a specific number of joints and links, it should be understood that the robot arm 506 may include any number of joints, any length of links, and/or include any type of joints or sensing. Device.

As described herein, the robotic arm 506 is seated and the joints are oriented to provide an unrestricted view of an operating area while providing an operator with a 3D stereoscopic display for any surgical procedure of a patient. The movement of the robot arm 506 in non-critical motions is provided to be fast enough to make it convenient and safe for an operator. The movement of the robotic arm 506 is controlled carefully and accurately during the surgical operation. In addition, the movement of the robot arm is controlled to be smooth and predictable in the entire range of motion required for a surgical procedure. As explained herein, the movement of the robotic arm 506 can be controlled by remote controls or via manual manipulation of the arm itself. In certain embodiments, the robotic arm 506 is configured to be positioned with minimal force (e.g., via an auxiliary guidance feature) using only (for example) a single little finger.

In some embodiments, the robotic arm 506 may include mechanically or electronically locking brakes on the joints. Once the target or "posture" of the camera 300 is set by an operator, the brakes can be engaged. The robot arm 506 may include a lock or unlock switch or other input device to prevent undesired manual or accidental movement. When locked, the exemplary robotic arm provides sufficient stability to enable the stereo visualization camera 300 to provide a stable and clear image. Additionally or alternatively, the robot arm 506 may include one or more damping devices to absorb or dampen vibrations after the stereo visualization camera 300 moves to a new posture. For example, the damping devices may include fluid-filled linear or rotary dampers, rubber-based vibration isolation mounted dampers, and/or tuned mass spring dampers. Alternatively or additionally, for example, the arm 506 may include electromechanical damping through the use of a proportional integral derivative ("PID") servo system.

The exemplary robotic arm 506 may be configured with a loading position to which one or more linkages are returned for transportation and storage. A loading position allows the robotic arm to be transported and stored in a compact footprint, but is deployed with a long reach that is required in certain surgical procedures. Cables (such as those wired for the stereo visualization camera 300) are provided along the robotic arm 506 in order to avoid interference with a surgical procedure.

In the illustrated embodiment of FIG. 34, the robot arm 506 includes six joints labeled R1, R2, R3, R4, R5, and R6. In other embodiments, the robotic arm 506 may include fewer or additional joints. In addition, in some embodiments, at least some of the joints R1 to R6 have a rotational movement capability of +/-360°. Rotational motion can be provided by an electromechanical subsystem that includes an electric motor for each joint that is configured to drive a mechanical rotary joint through one or more anti-backlash joint gearboxes. Each of the joints R1 to R6 may include one or more rotation sensors to detect the joint position. In addition, each joint may include a slip clutch and/or an electromechanical brake.

Each of the joints R1 to R6 may have an overall motion repeatability of approximately +/-1/10 of one millimeter ("mm") (with the camera 300 attached). The joints may have a variable rotation speed controllable between 0.5°/sec and 180°/sec. In short, this translates to camera movement between 1 mm/sec and 1 m/sec. In certain embodiments, the three-dimensional robotic platform 516 may have a speed governor for one or more of the joints R1 to R6 that are in place during the surgical procedure. Each of the joints R1 to R6 may be electrically connected to a power supply and/or command line in a controller of the robot arm 506. The wires used for power and command signals can be routed inside the joints and connecting rods. In addition, one or more of the joints may include a damper, such as an o-ring for connecting to a connecting rod. For example, the dampers can reduce or absorb vibrations in the robot arm 506, vibrations from the truck 510, and/or vibrations imparted by the stereo visualization camera 300.

The joint R1 includes a base joint mechanically coupled to a flange 3402 which is fastened to a fixing structure 3404. The flange 3402 can include any type of mechanical connector. For example, the fixed structure 3404 may include the truck 510 in FIG. 5, a wall, a ceiling, a table, and so on. The joint R1 is configured to rotate about a first axis 3410 (which may include the z-axis).

The joint R1 is connected to the joint R2 via a link 3430. The example link 3430 includes a cylinder or other tubular structure configured to provide structural support for the downstream section of the robotic arm 506. The link 3430 is configured to provide a rotationally secure connection with one of the joints R2 so that the joint R2 can rotate, while the link 3430 is held in place by its connection to the joint R1. For example, the joint R2 may include a shoulder joint configured to rotate about an axis 3412. The example axis 3412 is configured to be perpendicular (or substantially perpendicular) to the axis 3410. Given the rotation of the joint R1 around the z-axis, the axis 3412 is configured to be in an x-y plane.

The joint R2 is mechanically coupled to the joint R3 via a link 3432. The link 3432 is configured to have a length greater than that of the link 3430 and is configured to provide structural support for the downstream portion of the robot arm 506. For example, the joint R3 may include an elbow joint. The joint R3 and the joint R2 together provide the extendable positioning and/or orientation of the robot arm 506. The joint R3 is configured to rotate about an axis 3414, which is perpendicular or orthogonal to the axis 3410 and parallel to the axis 3412.

The joint R3 is connected to the joint R4 via a link 3434, and the link 3434 provides structural support for the downstream part of the robot arm 506. For example, the example joint R4 may be a first wrist joint configured to provide rotation about axis 3416, which may be orthogonal to axes 3412 and 3414. The joint R4 is mechanically connected to the joint R5 via a link 3436. The joint R5 may be a second wrist joint configured to provide rotation about an axis 3418 that is orthogonal to the axis 3416. The joint R5 is mechanically connected to the joint R6 via a link 3438. The joint R6 may be a third wrist joint configured to rotate about the axis 3420, which is orthogonal to the axis 3418. In short, the wrist joints R4 to R6 provide precise flexibility in positioning the stereo visualization camera 300 described herein.

The example robotic arm 506 includes a connector 3450. The example connector 3450 is connected to the joint R6 via a link 3440. In certain embodiments, the example link 3440 may include a sleeve that enables the joint R6 to rotate the connector 3450. As discussed herein, the connector 3450 can be configured to be mechanically coupled to the coupling plate 3304 or directly to the stereo visualization camera 300 when a coupling plate is not used. The connector 3450 may include one or more screws to fasten the robot arm 506 to the coupling plate 3304 and/or the stereo visualization camera 300.

In certain embodiments, the robotic arm 506 of the illustrated example may have a maximum reach of 85 mm in an orientation substantially similar to that of a human arm. The arm 506 may have a payload capacity of 5 kilograms. In addition, the arm 506 can be configured as a "cooperative" device to achieve safe operation in the vicinity of humans. For example, control the maximum force that the robot arm 506 can apply to the external surface. If a part of the robot arm unexpectedly touches another object, it detects a collision and immediately stops moving. During an emergency stop scenario (for example, where power is lost), the joints R1 to R6 can be reverse-driven or manually rotated so that an operator can grasp a part of the robot system and swing the part so that it does not get in the way. For example, a slip clutch within the joint limits the maximum torque that the joint motor can rotationally apply to the arm 506 during operation. When power is off, the sliding clutch of the joint slides when manually manipulated to allow an operator to quickly move the robot arm 506 out of the way.

35-40 illustrate an example configuration of the robot arm 506 and the stereo visualization camera 300 according to an example embodiment of the present invention. 35 shows a diagram of the robot arm 506 connected to the truck 510 via the flange 3402. In this example, the stereo visualization camera 300 is directly connected to the connector 3540. In this embodiment, the connector 3540 and/or the stereo visualization camera 300 may include the sensor 3306 of FIG. 33 for sensing the translation and/or rotation force applied to the stereo visualization camera 300 by an operator/ sports. If the connector 3540 includes the sensor 3306, the output force/motion data can be transmitted to a controller through the robot arm 506. If, for example, the sensor 3306 is located on the stereo visualization camera 300, the output data can be transmitted to a separate controller along with the control data. In some embodiments, a controller may be provided in the truck 510 or separately provided at a server.

FIG. 36 shows an embodiment in which the robot arm 506 is mounted to a ceiling 3404 via a flange 3402. The robot arm can be hung from the ceiling of an operating room to reduce floor space clutter. The robot arm 506 including joints can be positioned on the area where surgical activities are performed and pass through the area, without blocking the way for surgeons and operating room staff, but still providing the functional positioning and/or orientation of the camera 300 while providing display monitoring One of the devices 512 and 514 has a clear view.

FIG. 37 shows an embodiment of the coupling plate 3304. In the illustrated example, the first end 3702 of one of the coupling plates 3304 is connected to the connector 3450 of the robot arm 506. A second end 3704 of the coupling plate 3304 is connected to the stereo visualization camera 300. The example coupling plate 3304 is configured to provide additional degrees of freedom for moving the stereo visualization camera 300. The coupling plate 3304 also extends the maximum reach of the robot arm 506. The coupling plate 3304 may have a length between 10 cm and 100 cm.

The coupling plate 3304 may include one or more joints. In the illustrated example, the coupling plate 3304 includes joints R7, R8, and R9. The exemplary joint system provides mechanical joints that rotate around respective axes. The joints R7 to R9 may include a rotatable latch mechanism that can be moved after an operator activates a release button or lever. Each joint R7 to R9 may have its own release button, or a single button may release each of the joints R7 to R9.

The joints R7 to R9 can be connected together via individual links. In addition, a link 3718 is provided for connecting to the connector 3450 of the robot arm 506. The joint R7 is configured to rotate about the axis 3710, the joint R8 is configured to rotate about the axis 3712, and the joint R9 is configured to rotate about the axis 3714. The axes 3710 and 3714 are parallel to each other and orthogonal to the axis 3712. Joints R7 and R9 can be configured to provide +/- 360° rotation. In other examples, the joints R7 and R9 may provide +/- 90°, +/- 180° rotation, or +/- 270° rotation about the respective axes 3710 and 3714. The joint R8 can provide +/- 90° rotation around the axis 3712. In some examples, the joint R8 may only be set at +90°, 0°, and -90°.

In some embodiments, the joints R7 to R9 may include motors that provide continuous movement. The joints R7 to R9 may also include control devices, such as switches or position sensors that transmit or provide data indicating a rotational position. In this way, the joints R7 to R9 can be similar to the joints R1 to R6 of the robot arm 506 and provide auxiliary movement and positioning sensing for feedback control. The power and control for the joints R7 to R9 can be provided by wiring through the wires of the robot arm 506, the power/wire connectors in the connector 3450, and/or the wires outside the robot arm 506.

FIG. 37 shows an example in which the stereo visualization camera 300 is positioned in a horizontal orientation such that an optical axis 3720 is along a z axis. This horizontal orientation can be used to image a patient lying down. In contrast, FIG. 38 shows an embodiment where the joint R8 is rotated by 90° to position the camera 300 in a vertical orientation such that the optical axis 3720 is along an x-axis or a y-axis orthogonal to the x-axis. This vertical orientation can be used to image a sitting patient. It should be understood that the joint R8 enables the stereo visualization camera 300 to quickly reorient between the horizontal position and the vertical position based on the program.

In the illustrated examples of FIGS. 36 and 37, the example sensor 3306 may be located, for example, at the connector 3450 of the robot arm (wherein the coupling plate 3304 is connected) and/or the first end 3702 of the coupling plate (At the connection with connector 3450). Alternatively or additionally, the example sensor 3306 may be located at the second end 3704 of the coupling plate (at the connection with the camera 300) and/or located at the second end 3704 of the coupling plate 3304 and/or the camera 300 at the first end of the coupling plate 3304. The connection point of the two ends 3704.

Figures 39 and 40 show the stereo visualization camera 300 rotated +90° around the axis 3714 of the joint R9 in a horizontal orientation. FIG. 40 shows an example of a stereo vision camera 300 in a horizontal orientation and rotated -90° around the axis 3714 of the joint R9.

As illustrated in Figures 34-40, an example robotic arm 506 is configured to provide support for the stereo visualization camera 300 and allow precise positioning and/or orientation and aiming of the optical axis of the camera. Since the stereo vision camera 300 does not have eyepieces and does not need to be oriented for the eyes of a surgeon, there are many desirable positions and/or orientations for achievable imaging (which were previously impractical). A surgeon can perform it with a view that is best for a procedure, not for its orientation with the eyepiece.

The exemplary robotic arm 506 when used with the stereo visualization camera 300 enables a surgeon to look around corners and other locations that are not easily seen. The robot arm 506 also enables the patient to be placed in different positions including supine, prone, sitting, semi-sitting and the like. Therefore, the robot arm 506 enables the patient to be placed in the best position for a specific procedure. The exemplary robotic arm 506 can be installed to achieve the least protruding position when used with the stereo visualization camera 300. The arm 506 and the camera 300 therefore provide a surgeon with numerous possibilities of visual position and orientation while being conveniently positioned and oriented so as not to get in the way.

The configuration of the links and joints of the robot arm 506 and/or the coupling plate 3304 together with the motorized six (or nine) degrees of freedom generally allows the camera 300 to be positioned as required, and the configuration of links and joints is not unique to the camera. posture. As discussed in more detail below, the joints and links of the arm 506 and/or the plate 3304 can be manually repositioned and/or redirected without changing the pose or FOV of the camera 300. For example, this configuration allows an elbow joint to move out of a blocked view without changing the view of the surgical site through the camera 300. In addition, a control system can determine the position and posture of the camera 300 and calculate and display the alternate position and/or orientation of the robot arm 506 to, for example, avoid staff or display obstruction. Using the various positions and/or orientations of the coupling plate 3304 together with the ability of the same image processor to flip, invert, or otherwise redirect the displayed image allows even more robot arm 506 positions and/or orientations.

The robot arm 506 and/or the coupling plate 3304 are generally seated, and the joints are positioned so as to avoid joint singularities during any general movement. Avoiding joint singularities will provide better robot control over hysteresis and backlash. In addition, the length and configuration of the links and joints of the robot arm 506 and/or the coupling plate 3304 provide smooth movement along most any desired motion path. For example, the repositioning and/or reorientation of the robotic arm 506 enables it to change the direction of the camera 300 view of a target point in a surgical site without changing a focal point, thereby allowing a surgeon to move from different directions/orientations Watch the same target point. In another example, the robot arm 506 can change a working distance to a target point without changing a focus by moving the camera 300 along the line of sight toward or away from a target point. If necessary, the robot arm 506 and/or the coupling plate 3304 can be used together with the stereo visualization camera 300 of the stereo robot platform 516 to obtain many similar motion paths. B. Robot control example

The example robot arm 506 and/or the coupling plate 3304 of FIGS. 34-40 can be controlled by one or more controllers. FIG. 41 illustrates an embodiment of the three-dimensional robot platform 516 of FIGS. 3 to 40 according to an exemplary embodiment of the present invention. The exemplary three-dimensional robot platform 516 includes the three-dimensional visualization camera 300 and the corresponding image capturing module 1404 and the motor and lighting module 1406 described in conjunction with FIGS. 14 and 15.

In the illustrated embodiment, the stereo robotic platform 516 includes a server or processor 4102 located remote from the stereo visualization camera 300. For example, the processor 4102 may include a laptop computer, a workstation, a desktop computer, a tablet computer, a laptop computer, a workstation, a desktop computer, a tablet computer, a Smart phones, etc., when executed by the processor 4102, these instructions cause the processor 4102 to perform the operations described here. In this example, the example processor 4102 is configured to include the information processor module 1408, image sensor controller 1502, and/or motor and illumination controller 1520 of FIGS. 14-16 (or execute a combination of these The operations described in each item).

In some instances, at least some of the operations of the image sensor controller 1502 and/or the motor and illumination controller 1520 can be shared with the image capture module 1404 and the motor and illumination module 1406, respectively. For example, the processor 4102 can generate commands for changing the focus, magnification and/or working distance, and through the first part of the motor and the light controller 1520 in the motor and light module 1406 and the motor and light controller A second part of 1520 controls drivers 1534 to 1552. Alternatively or alternatively, a first part of the information processor module 1408 operatively located in the processor 4102 is configured to receive from a second part of the information processor module 1408 in the image capture module 1404 Individual left/right images and/or stereo images. The first part of the information processor module 1408 can be configured to process images for display on one or more display monitors 512 and/or 514, including visually fused images with graphical guides/texts, Image overlay and/or fluorescent image from an MRI machine, X-ray or other imaging device.

The processor 4102 is electrically coupled and/or communicatively coupled to the image capturing module 1404 and the motor and illumination module 1406 of the stereo visualization camera 300 via a wire harness 4102. In some embodiments, the wire harness 4102 may be external to the robot arm 506. In other embodiments, the wire harness 4102 can be internal or routed through the robot arm. In still other embodiments, the image capture module 1404 and the motor and illumination module 1406 can communicate with the processor 4102 wirelessly via Bluetooth®, for example.

The example processor 4102 is also electrically and/or communicatively coupled to the sensor 3306 via a wire harness 4102. The processor 4102 is configured to receive, for example, rotation and/or translation output data from the sensor 3306. The data may include digital data and/or analog signals. In some embodiments, the processor 4102 receives an indication of the detected force and/or motion from the sensor 3306 to output a data stream almost continuously. In other examples, the processor 4102 receives output data at periodic sampling intervals. In still other examples, the processor 4102 periodically transmits a request message for requesting the output data.

In the illustrated example, the processor 4102 is further communicatively coupled to a display monitor 512, input devices 1410a, 1410b, and other devices/systems 4104 (e.g., medical imaging devices such as an X-ray machine, a computer tomography ("CT") machine, a magnetic resonance imaging ("MRI") machine, a camera, a workstation for storing images or surgical guides, etc.). The input device 1410a may include a touch screen device and the input device 1410b may include a foot switch. The touch screen input device 1410a can be integrated with the display monitor 512 and/or provided as a separate device on the truck 510 of FIG. 5, for example. The exemplary display monitor 512 is configured to display one or more user interfaces, the one or more user interfaces including a stereoscopic video (or a separate two-dimensional video) recorded by the stereo visualization camera 300 of a target surgical site Left video and right video).

The touch screen input device 1410a is configured to provide one or more user interfaces for receiving user input related to the control of the stereo visualization camera 300, the coupling board 3304, and/or the robot arm 506. The input device 1410a may include a working distance, focus, magnification, illumination source and level, filter configured to enable an operator to specify, set, or otherwise provide instructions for controlling a working distance, focus, magnification, illumination source and level, and filter of the stereo visualization camera 300 And/or one or more graphic control buttons, sliders, etc. for digital zoom. The input device 1410a may also include one or more control buttons to enable an operator to select surgical guidance graphics/text, a video and/or an image for fusion and/or other ways to superimpose the display on the display monitor On the 3D video displayed on the device 512. The input device 1410a may also include a user interface configured to enable an operator to input or create a surgical procedure visualization template. The input device 1410a may further include one or more control buttons for controlling the robot arm 506 and/or the coupling board 3304, including operation parameters such as speed, movement, deployment/loading, calibration, target lock, and storage of a view The position and/or the option of changing or entering a new orientation of the camera 300. The user interface controls for the robot arm 506 and/or the coupling board 3304 may include controls for moving the camera 300, which are converted into commands for the individual joints R1 to R9. Additionally or alternatively, the user interface controls for the robot arm 506 and/or the coupling plate 3304 may include controls for individually moving each of the joints R1 to R9. The input received via the input device 1410a is transmitted to the processor 4102 for processing.

For example, the example foot switch input device 1410 may include a foot pedal configured to receive input for controlling a position of the stereo visualization camera 300, the coupling plate 3304, and/or the robot arm 506. For example, the foot pedal input device 1410b may include controls for moving the camera 300 along the x-axis, y-axis, and/or z-axis. The foot pedal input device 1410b may also include controls for storing a position of the camera 300 and/or returning to a previously stored position. The foot pedal input device 1410b may further include controls for changing a focus, zoom, magnification, etc. of the camera 300.

In other embodiments, the three-dimensional robot platform 516 may include additional and/or alternative input devices 1410, such as a joystick, a mouse, or other similar 2D or 3D manual input devices. The input device 1410 is configured to provide input similar to an X-Y horizontal oscillating device, where the extra degrees of freedom cause flexibility in the movement of the system. An input device with 3D capability (such as a 3D mouse or a six-degree-of-freedom controller) is well suited for flexible and convenient command input. One of the main benefits of these user control devices is that they can easily view surgical images while moving. In addition, a surgeon can watch what is happening around the entire surgical operation and nearby parts to avoid (for example) causing the camera 300 to collide with the surgical staff and/or nearby equipment.

Optionally, the input device 1410 may include a head-mounted, eye- or glasses-mounted tracking device, a voice recognition device, and/or a gesture input device. These types of input devices 1410 promote "hands-free" operability, so that an operator does not need to touch anything with his sterile gloves. A gesture recognition control can be used, in which specific operator movements are recognized and converted into control signals for the camera 300, the coupling plate 3304, and/or the robot arm 506. A similar function is provided by a voice recognition device, where a microphone senses a command from an operator, such as "move the camera to the left", recognizes the voice as a command, and converts it into an appropriate camera and/or robot control signal. Alternative embodiments include an eye tracking device that is configured to determine a position of an operator's eyes relative to a 3D display, and can adjust the view depending on where the operator looks in the displayed scene.

Other embodiments include a device configured to track an operator's head (for example, a trackable target or group of targets installed on an operator's 3D glasses) in a position in a reference frame And a foot switch used to activate the "head tracking". The exemplary tracking input device is configured to store a starting position of an operator's head at the start time and then continuously detect the head position at a certain time interval. The tracking input device together with the processor 4102 can calculate a movement difference vector between a current position and a starting position and convert the vector into a corresponding robot arm or camera lens movement. For example, a tracking input device 1410 and processor 4102 can convert left/right head movement into robot arm movement, so that an image on the screen moves left/right. The tracking input device 1410 and the processor 4102 can also convert the up/down head movement into a robot arm or camera lens movement, so that an image on the screen moves up/down, and can convert the front/back head movement into a robot arm or camera The lens moves to enlarge/reduce an image on the screen. Other movement transformations are possible, for example, by transforming the head rotation into a "lock to target" movement of the robot arm 506. As explained here, the lock to target is configured to maintain a focus of the robotic platform 516 at the same point in a scene or FOV within a certain tolerance and cause the robotic arm 506 (and therefore the view) to mimic an operator The head moves in one direction and pivots.

Before a specific surgical procedure, a surgical plan is formed for instrumentation and visualization to establish one of the necessary paths. In some embodiments, the input device 1410 is configured to follow this predetermined path with little additional input from an operator. In this way, the operator can continue the operation as planned in advance while the view of the surgical site is automatically changed. In some embodiments, the surgical operation plan may include a set of pre-planned path points corresponding to the camera position, magnification, focus, and so on. An operator can actuate the input device 1410 to progress through waypoints as the surgical procedure progresses (causing the processor 4102 to move the robot arm 506, coupling plate 3304, and/or camera 300 as planned).

In the illustrated embodiment, the example sensor 3306 is an input device. The sensor 3306 is configured to detect the movement or force of an operator on the stereo visualization camera 300 and convert the detected force/movement into rotation and/or translation data. The sensor 3306 may include a motion expectation input device, such as a six-degree-of-freedom tactile force sensing module or a photoelectric sensor (for example, a force/torque sensor), which enables the robot arm 506 to operate on a The reader responds electromechanically to the gentle push of the camera 300. The photoelectric sensor may include an electro-optical device configured to convert the applied force and/or torque into an electrical signal, thereby enabling a desired force/torque input by an operator to be sensed And the transformation is to provide a motion request in the sensed linear and/or rotational direction. In other embodiments, other sensor types may be used for the sensor 3306. For example, the sensor 3306 may include a strain gauge or piezoelectric device configured to sense a tactile request from an operator.

In one embodiment, a surgeon holds one or more of the control arms 304 and actuates or pushes a release button (which may be located on one or both of the control arms 304). The actuation of the release button causes the camera 300 to transmit to the processor 4102 a message indicating that an operator desires to start an “assisted movement” mode. The processor 4102 configures the robotic arm 506 and/or the coupling plate 3304 to enable the surgeon to gently manipulate the camera 300 in a desired direction. During this movement, the processor 4102 causes the robot arm 506 and/or the coupling plate to move the camera 300 in a "power steering" manner, thereby safely supporting its weight and automatically determining which joints should be activated and which joints should be braked in a coordinated manner In order to achieve what the surgeon wants to move.

In the illustrated example, the three-dimensional robot platform 516 of FIG. 41 includes a robot arm controller 4106 that is configured to control the robot arm 506 and/or the coupling plate 3304. The robot arm controller 4106 may include a process configured to convert one or more messages or instructions from the processor 4102 into one or more messages and/or signals that cause any one of the joints R1 to R9 to rotate Server, a server, a microcontroller, a workstation, etc. The robot arm controller 4106 is also configured to receive sensor information (such as the joint position and/or speed from the robot arm 506 and/or the coupling plate 3304) and convert the sensor information to the processor 4102 One or more messages.

In some embodiments, the robot arm controller 4106 is configured as an independent module located between the processor 4102 and the robot arm 506. In other embodiments, the robot arm controller 4106 may be included in the robot arm 506. In still other embodiments, the robot arm controller 4106 may be included with the processor 4102.

The example robot arm controller 4106 includes one or more instructions stored in a memory 4120 that can be executed by a robot processor 4122. These instructions can be configured into one or more software programs, algorithms and/or routines. The memory 4120 may include any type of volatile or non-volatile memory. The example robotic processor 4122 is communicatively coupled to the processor 4102 and is configured to receive one or more messages related to the operation of the robotic arm 506 and/or the coupling board 3304. The example robot processor 4122 is also configured to transmit one or more messages indicating the position and/or velocity of the joints R1 to R9 to the processor 4102. The one or more messages may also indicate that a joint has reached a terminal or is prevented from moving.

The example processor 4102 is configured to determine which joints R1 to R9 are powered in a coordinated manner so that the sum of all the motions of all the joints produces the desired image motion at the camera 300. In an example of "move the camera to the left", there may be complex motions of several joints that cause the surgical image of the camera to appear to be simply and smoothly translated to the left from a relative viewpoint of a surgeon. It should be noted that in the example of "move the camera to the left", depending on how the camera 300 is connected to the robot arm 506 through the coupling plate 3304, the control signal to a specific joint may be significantly different depending on the position/orientation.

The memory 4120 may contain one or more instructions specifying how to move the joints R1 to R9 based on the known position of one of the joints. The robot arm controller 4106 is configured to execute one or more instructions to determine how to convert the instructed camera movement into joint movement. In one example, the robot arm controller 4106 may receive a message from the processor 4102 indicating that the stereoscopic visualization camera 300 will move down along a z-axis and move sideways in an x-y plane. In other words, the processor 4102 transmits the content indicating the input received via the input device 1410 regarding the desired movement of the camera 300. The exemplary robot arm controller 4106 is configured to convert the movement vector in three-dimensional coordinates into joint position movement information to achieve the desired position/orientation. The robot arm controller 4106 can determine or consider the current positions of the links and joints of the robot arm 506 and/or the coupling plate 3304 (and/or a position/orientation of the camera 300) together with the desired movement to determine a movement difference vector. In addition, the robot arm controller 4106 may perform one or more checks to ensure that the desired movement does not cause the camera 300 to enter or advance close to a restricted area, such as defined in the same coordinate system as the arm 506 and the coupling plate 3304. Defined by one or more three-dimensional boundaries. The area close to a boundary may specify that the robot arm controller 4106 applies a reduced scale factor when sending a movement signal to the joint. The reduced scale factor causes the joint to move more as the robot arm 506 approaches a boundary. Slow, and do not move further across a boundary.

After performing the boundary check, the robot arm controller 4106 uses the movement difference and the current position/orientation of each of the joints R1 to R9 to determine which one is the best or the best for rotating one or more of the joints. The sequence of movement is nearly optimal so that the robot arm 506 moves the camera 300 to a prescribed position. The robot arm controller 4106 can use, for example, an optimization routine that determines a minimum joint movement required to meet the movement difference vector. After determining the amount of joint movement, the example robot arm controller 4106 is configured to send one or more messages (indicating a rotation amount and rotation speed, thereby taking into account any scaling factors) to a motor controller 4124. The robot arm controller 4106 can transmit a message sequence to cause the robot arm 506 and/or the coupling board 3304 to move in a defined or coordinated sequence. The message sequence can also cause one of the joint speeds to change as the robot arm 506 approaches a virtual or physical boundary, for example.

The example motor controller 4124 is configured to convert or convert received information into analog signals, such as pulse width modulation ("PWM") signals that cause one or more of the joints R1 to R9 to rotate. For example, the motor controller 4124 can select the input line to the appropriate joint motor, where a pulse duration is used to control the motor rotation for a duration and the pulse frequency, duty cycle and/or amplitude are used to control the rotation speed . The motor controller 4124 can also provide power for the joint motors and corresponding joint sensors.

In some embodiments, the robot arm controller 4106 combined with the motor controller 4124 is configured to receive or read the joint sensor position information and determine the position and orientation of the robot joints and the camera 300 through kinematics. Each joint R1 to R9 may include at least one sensor that detects and transmits data indicating joint position, joint rotation speed, and/or joint rotation direction. In some embodiments, the sensors only transmit position information, and the robot arm controller 4106 determines the speed/direction based on the difference in position information over time. The robot arm controller 4106 can transmit sensor data to the processor 4102 for determining movement information.

The robot arm controller 4106 receives the movement command from the processor 4102 and determines which motors and joints should be activated, how fast and how far, and in what direction through sub-comparable forward and/or inverse kinematics. The robot arm controller 4106 then sends appropriate command signals to the motor power amplifier in the motor controller 4124 to drive the joint motors in the robot arm 506.

The example robotic arm 506 receives the appropriate motor power signal and moves accordingly. The sensors and actuators in the arm 506 respond to various operations and feedback information from the robot arm controller 4106. In some embodiments, the robot arm 506 is mechanically and communicatively connected to the coupling board 3304, and the coupling board 3304 transmits the coupler status and orientation information to the robot arm controller 4106.

In some embodiments, the example robotic arm 506 of FIG. 41 includes a coupler controller 4130. The example coupler controller 4130 is configured to bypass the robot processor 4122 and relay control information between the processor 4102 and the coupling board 3304. The coupler controller 4130 can receive messages from the processor 4102 and thus cause the joints R7 to R9 to rotate on the coupling plate 3304. The coupler controller 4130 can also receive sensor information about the joint position and/or speed and transmit one or more messages indicating the joint position and/or speed to the processor 4102. In these embodiments, the processor 4102 can transmit a message for controlling the robot arm 506 and a separate message for the coupling board 3304.

In some embodiments, the robot arm controller 4106 is configured to determine how the joints R7 to R9 will move. However, if the coupling board 3304 is not directly coupled to the robot arm 506 in a communication manner, the robot processor 4122 may transmit the movement signal to the coupler controller 4130 via the processor 4102. In the example in which at least some operators of the robot processor 4122 are positioned together with the processor 4102, together with the robot arm 506 receiving movement commands or signals from the processor 4102, the coupler controller 4130 receives movement commands or signals from the processor 4102. Signal.

In the illustrated embodiment of FIG. 41, the exemplary stereo visualization camera 300, the processor 4102, the coupling board 3304, the robot arm 506, the robot arm controller 4106, and/or the input device 1410 receive via an input power module 4140 electricity. The example module 4140 includes a power supply (such as power from a wall outlet) and/or an isolation transformer to prevent abnormal power lines from disrupting system performance. In some examples, the power supply may include a battery power supply.

The stereo visualization platform 516 may also include an emergency stop switch 4142 that is configured to immediately cut off the power. The switch 4142 can only cut off the power to the robot arm 506 and/or the coupling plate 3304. The processor 4102 can detect the activation of the emergency stop switch 4142 and cause the joint brake to engage to prevent the robot arm 506 from descending. In some examples, the robot arm 506 is configured to activate the joint brake after detecting a power loss. In some embodiments, the joints R1 to R6 of the robot arm 506 are configured to slide when a force above a threshold is applied, thereby enabling an operator to have or not in an emergency situation. Move the arm quickly when it has power to keep it out of the way.

In some embodiments, the example processor 4102 is configured to display one or more graphical representations of the robotic arm 506, the coupling board 3304, and/or the stereo visualization camera 300. The processor 4102 may cause the graphical representation to be displayed in one or more user interfaces that provide control of the robot arm 506, the coupling board 3304, and/or the camera 300. The processor 4102 can position and orient the graphical representation to reflect the current position of the robot arm 506, the coupling board 3304, and/or the camera. For example, the processor 4102 uses feedback from the robot arm controller 4106 to determine which joints in the graphical representation will be rotated, thereby changing the orientation and/or position of the display device. In some examples, the processor 4102 is configured to receive user input via a graphical representation by, for example, the following: an operator moves a link, joint or camera 300 in the graphical representation to a desired Location. In the case of the movement of the stereo visualization camera 300, the processor 4102 may transmit a new coordinate corresponding to where the camera moves. In the case of a moved joint or link, the processor 4102 may transmit a message indicating the rotation of the joint and/or the position of the link to the robot arm controller 4106.

In some embodiments, the processor 4102 operates in conjunction with the robot arm controller 4106 to adjust one of the cameras based on the movement of the robot arm 506 and/or the coupling plate 3304 or in cooperation with the movement of the robot arm 506 and/or the coupling plate 3304 Or multiple lenses. For example, if the robot arm 506 is moved toward a surgical site, the processor 4102 operates in conjunction with the robot arm controller 4106 to change a job by moving one or more of the lenses of the stereo visualization camera 300 Distance or focus to maintain focus. The processor 4102 operates in conjunction with the robot arm controller 4106 to determine, for example, that the movement of the robot arm 506 causes a working distance to decrease. The EPU processor 4102 operates in conjunction with the robot arm controller 4106 to determine a new position of the lens based on a new working distance set by moving the robot arm 506. This can include moving one or more lenses for adjusting focus. In some embodiments, the processor 4102 may instruct the camera 300 to operate a calibration routine for the new position of the robot arm 506 to eliminate, for example, false parallax.

In some cases, an operator can change the position of one or more lenses of the stereoscopic visualization camera 300 and reach a lens travel limit. The position of the lens is sent from the camera 300 to the processor 4102, which is used to determine that a limit has been reached. After detecting that a limit has been reached, the processor 4102 can cause the robot arm 506 to move (via the controller 4106) based on the input from the operator, thereby causing its command to extend from the lens movement to the arm movement to reach a desired magnification Or target area. In this way, the processor 4102 operating in conjunction with the robot arm controller 4106 enables an operator to use only one user interface instead of changing between one interface for the robot arm and the camera. It should be appreciated that the processor 4102 and/or the controller 4106 can check the desired movement against any predetermined movement restrictions to ensure that the movement will not cause the camera 300 or the robotic arm 506 to enter the restricted patient or operator space. If a restriction violation is detected, the processor 4102 in conjunction with the robot arm controller 4106 can display a warning indicating one of the restrictions to the operator (using one of the displayed on the touch screen input device 1410a and/or the display monitor 512) User interface) to indicate one of the reasons for stopping the robot arm 506. C. Examples of calibration of robot arms and stereo cameras

As discussed above, the exemplary stereo visualization camera 300 is configured to provide a high-resolution stereoscopic video image of a target surgical site at different magnifications. As part of the stereo visualization platform 516, the stereo visualization camera 300 is operated in conjunction with the robot arm 506 and/or the coupling board 3304 to achieve precise and clear changes to the image focus, working distance, magnification, etc. In order to achieve flexibility in image acquisition, the stereo visualization platform 516 is configured to operate one or more calibration, initialization and/or reset routines. In some embodiments, the stereo visualization camera 300, the robotic arm 506, the coupling plate 3304, or more generally the stereo visualization platform 516 are calibrated during manufacturing and/or after installation. Calibrating the camera 300 with the robot arm 506 will provide positioning information of the camera 300 relative to the robot arm 506 and the operator space. After powering on the stereo visualization platform 516, in some embodiments, the camera 300 and/or the robot arm 506 are configured to perform additional calibration/initialization to measure and verify a position and orientation of the camera 300 at that time .

The example processor 4102 is configured to store the results from the calibration (eg, calibration data) in, for example, the memory 1570 and/or the memory 4120 of FIG. 41. The calibration results can be stored in the calibration register and/or the look-up table ("LUT") in the memory 1570 and/or 4120. The stored calibration data relates the optical, functional and/or performance characteristics to the attributes of the camera 300, the robot arm 506 and/or the coupling board 3304 (adjustable, measured and/or verified by an operator or by the processor 4102) or the These optical, functional and/or performance characteristics are mapped to these attributes. For example, the position of a working distance actuator motor encoder of the main objective lens assembly 702 (of FIG. 7) is mapped to a working distance in a LUT. In another example, the axial position of a zoom lens along a linear encoder of the zoom lens assembly 716 is mapped to the magnification level in an LUT. For each of these examples, the example processor 4102 is configured to determine the appropriate level of an encoder characteristic, adjust and verify that the characteristic provides the specified or desired working distance and/or magnification. In some embodiments, the LUT may be composite, where multiple performance characteristics are mapped to multiple platform 516 attributes to achieve overall control of all relevant aspects of the camera 300, the robot arm 506, and/or the coupling board 3304.

The combination of a robot arm 506 and the exemplary stereo visualization camera 300 provides highly accurate position, direction and/or orientation information of the target view relative to a reference frame of the robot arm 506. The following sections describe how to calibrate the stereo visualization camera 300 to define a visual tip. After determining a visual tip, the stereo vision camera 300 is registered to a reference system of the robot arm 506 and/or the coupling board 3306. Therefore, after calibration and registration, a stereo view of a surgical site is unified with the integrated control of the stereo visualization camera 300 (combined with the position, direction and orientation control of the robot arm 506 and/or the coupling plate 3304) .

In some embodiments, the example processor 4102 of FIG. 41 is configured to accurately integrate a registration of the stereo visualization camera 300 (including its visual tip) with a position, direction, and/or orientation of the robotic arm 506 Calibration is to define a unified position, direction and/or orientation perception of the acquired three-dimensional image and all points within it relative to a specified coordinate system. The example processor 4102 is configured to use inherent visual imaging data from the stereo visualization camera 300 to coordinate or direct the physical positioning and/or orientation of the robotic arm 506 to provide visualization as required by an operator. In addition, the direction and coordination provided by the processor 4102 are provided to maintain better visual characteristics, such as focus, working distance, predefined positioning, orientation, and so on.

In some embodiments, the calibration of the stereo visualization camera 300, the robot arm 506, and/or the coupling plate 3304 generally includes (i) determining and/or measuring the inaccuracy of the functional parameters of the stereo visualization platform 516 that affect the stereo image (Ii) Calibrate or adjust the stereo visualization platform 516 to minimize the inaccuracy of the stereo image to a desired level or below the desired level; (iii) Through the dual channels of the stereo image, align with each other or with each other The templates are compared at the same time to verify that the adjustment has been made within a desired calibration accuracy level; and (iv) the stereo visualization platform 516 is used when performing its task, where one level of the calibration accuracy is detectable And maintained.

In an alternative embodiment, the robot arm 506 has a physical relationship for accurately calibrating the joints and linkages of the robot arm 506 and for calibrating the vision tip of the camera 300 and the robot arm 506 and/or an initial posture. Configure one or more fixed calibration standards for a relationship. The fixed calibration datum of the robot platform can be used to register or integrate the robot arm 506 with an external environment (such as an operating room) or a patient or target space in an external environment. The fixed calibration standards may include a dedicated attachment in combination with an external part of a main body of the robot arm 506 or one of the known external features of the robot arm 506 (such as mounting points, joints, corners, or the like). 1. Calibration of the stereo vision camera embodiment

To match a stereo view of a surgical site, the example processor 4102 and/or stereo visualization camera 300 are configured to execute one or more calibration routines. The example routine may be specified by one or more instructions stored in the memory 1570, which, when executed by the processor 4102, cause the processor 4102 to determine that it is related to a specific working distance, magnification (e.g., zoom level), and focus The position of the lens corresponding to the level. The instructions may also cause the processor 4102 to operate one or more routines for determining a projection center, a stereo optical axis, and/or a ZRP of the stereo visualization camera 300 at different working distances and/or magnifications. Calibration enables, for example, the processor 4102 to maintain focus on a target surgical site when changing magnification and/or working distance.

FIG. 42 illustrates an example procedure 4200 or routine for calibrating the stereo visualization camera 300 according to an example embodiment of the present invention. Although the procedure 4200 is described with reference to the flowchart illustrated in FIG. 42, it should be understood that many other methods of performing the steps associated with the procedure 4200 can be used. For example, the order of the blocks in the block can be changed, a specific block can be combined with other blocks, and many of the blocks described are optional. In addition, the actions described in the procedure 4200 can be executed in multiple devices including, for example, the optical element 1402 of the exemplary stereo visualization camera 300, the image capturing module 1404, the motor, and the illumination module. Group 1406 and/or information processor module 1408. For example, the program 4200 can be executed by one of the programs 1560 of the information processor module 1408.

The example process 4200 starts when the stereo visualization camera 300 is powered or otherwise initialized (block 4202). The camera 300 can be mounted to the robot arm 506. Alternatively, the program 4200 can be executed when the stereo visualization camera 300 is connected to a fixed mount. The example procedure 4200 next performs ZRP alignment, as discussed above in connection with FIGS. 25 and 26 (block 4204). The example processor 4102 may automatically align the ZRP, as discussed above, and/or operate in conjunction with an operator to provide alignment of the left optical path and the right optical path on the image sensors 746, 748. In some instances, the processor 4102 and/or an operator can make very small adjustments to a tilt of a lens assembly via a motor to make a ZRP move aligned with the origin of a pixel grid. The accuracy causes a small movement or deflection of a flexure (for example, the flexure 1300 of FIG. 13). During the semi-manual alignment, the processor 4102 may cause the left and right images from the image sensors 746 and 748 to be overlaid on the display monitor 512. An operator can use the input device 1410 to adjust the image, causing the pixel groups of the sensors 746 and 748 to move accordingly until the ZRP is properly aligned.

During the alignment, the ZRP is set to align at the center of an image to avoid false parallax. Alignment to approximately one single pixel on a display is possible. The degree of alignment from the left and right views to the center of an image is visible in the overlay image (including during the zoom operation). In an example of an 8° FOV, using a 4K image sensor 746, 748 and a display monitor 512 corresponding to a 4K display resolution (including approximately 4000 pixels by approximately 2000 columns) will produce 8°/ 4000 pixels = one system resolution of 7 arc seconds. However, ZRP alignment can be performed at most any magnification, where the resolution (the same number of pixels (for example, 4000)) is divided by a reduced (or increased) angle FOV. For example, an exemplary embodiment of the camera 300 produces an FOV of approximately 2° at a high magnification. An 8K UHD display monitor 512 and sensors 746 and 748 have about 8000 pixels in about 4000 columns. The resolution of this system is 2°/ 8000 pixels = 1 arc second. This system better than the systems known in the art of about one order of magnitude or more, wherein the assembled individually by the accuracy of the measuring assembly having tolerances, these tolerances having arc minutes measured in units of Resolution. Since the density of image sensors and display monitors generally becomes higher as the pixels in the same physical sensor or display space become smaller, the accuracy of the adjustability of the stereoscopic camera 300 changes with the pixel size. Small and scaled. The enhanced high-accuracy alignment of the stereo vision camera 300 provides better and more accurate digital effects.

When the ZRP of the left image and the right image remain within a desired tolerance range at the center of an image, the alignment of the ZRP is complete, and the image of the target surgical site remains accurate when cycling from low magnification to high magnification of. After the ZRP is aligned through the magnification capability of the stereo visualization camera 300, the pixel group position and/or lens position for each of the magnification levels are stored in, for example, a LUT 4203 or other data structure. In other examples, the processor 4102 writes the pixel group position and/or lens position for each of the magnification levels to the calibration register.

After ZRP alignment, the example processor 4102 is configured to calibrate the working distance and/or magnification (e.g., zoom) (block 4206). As discussed above in conjunction with the working distance and zoom example of FIG. 15, accurate knowledge of the working distance is important in the camera 300 so that the robot arm 506 can accurately position the camera with respect to the desired coordinates. In some cases, an accurate datum is used together with the mechanical dimensions of the robot arm 506 to transform the object plane data from an image to a respective coordinate system of the stereo visualization platform 516 (referred to as the robot space in this article). )middle.

The exemplary calibration procedure 4200 is executed to map the working distance of the optical system of the stereo vision camera 300, which can be calculated or measured from a common mode objective ("CMO") lens assembly (for example, FIGS. 4 and 7). Previous working distance The working distance from the front of one of the main objective lens 408) to an object plane. Mapping the working distance from a known ``original'' position (such as a physical stop or limit switch trigger position) to a known measurable parameter (such as (for example) measured in the count of a motor shaft encoder device) Measure one of the focus motor position).

The calibration at block 4206 is performed by the processor 4102. The processor 4102 sequentially moves the object plane along the optical axis in discrete steps and refocuses the image while recording the encoder count and working distance, as discussed in more detail in connection with FIG. 43 . The processor 4102 measures the working distance externally before the CMO. The mapping of encoder count and working distance is stored in LUT 4203 or a different LUT and/or calibration register. Given a desired working distance, this calibration enables the processor 4102 to output an encoder count position to the motor controller. The exemplary embodiment of the stereo visualization camera 300 uses a high-count shaft encoding device per revolution, where the resolution of the working distance is about 1 micron/count per encoder. Alternative embodiments may include different encoder resolutions to provide higher or lower resolutions of the working distance as needed.

FIG. 43 shows an embodiment of an exemplary stereoscopic visualization camera 300 that moves an object plane in a discrete step according to an exemplary embodiment of the present invention. The example stereo visualization camera 300 includes the main objective assembly 702 of FIG. 7 (eg, a single CMO) configured to provide left and right views of a target surgical site. In the illustrated example, the main objective lens assembly 702 is shown as an achromatic refraction assembly with a fixed front working distance lens 408 in a housing 4302 and can be along the z-axis (or The other optical axis) moves the movable rear working distance lens 704. The movement of the rear working distance lens 704 is changed to the distance of the front working distance lens 408. The distance between the lens 408 and the lens 704 determines the overall front focal length 4304 of the main objective lens assembly 702 and therefore the position of a front focal plane (or just the "focal plane") 4306. The front focal plane 4306 is located at a distance equal to the focal length 4304 of a main plane 4308 of the main objective lens assembly 702. It can be difficult to measure the position of the main plane 4308, so the distance from the bottom surface of the housing 4302 to the front focal plane is defined as the working distance 4310. The working distance 4310 therefore accurately sets the target part of the focus or a plane of the scene.

Imaging an object at the front focal plane 4306 will result in a conjugate image located at an infinite distance from or behind one of the main objective lens assemblies 702. Two parallel optical paths including the optics 714, 716, 718 of the camera 300 and the sensors 744R, 744L (which are laterally separated by an interpupillary distance ("IPD") 4312) are along the respective left optical axis 4320 and right The optical axis 4322 produces a left view and a right view in a direction slightly different from an optical axis 4324 of the main objective lens assembly 702. The two optical paths are adjusted so that the left axis and the right axis of their respective external focus are set to coincide at the center of the FOV of an image 4330. This point 4330 is referred to herein as the "tip" of the stereo visualization camera 300 at the front focal plane 4306.

The adjustment of one position of the rear working distance lens 704 causes one of the front focal lengths 4304 of the main objective lens assembly 702 to change. As illustrated in the figure, one of the positions of the rear working distance lens 704 is changed to form a new working distance 4310' at the position of a new front focal plane 4306'. The movement of the rear working distance lens 704 also causes the left optical axis 4320' to realign with one of the right optical axis 4322', resulting in a repositioned tip 4330' of one of the cameras 300. When the camera 300 is located above or below the focal plane 4306, the visualization of an object will reduce the focus on one of the objects.

In a manner similar to the working distance calibration, the processor 4102 can change the magnification while measuring the height of an image of an object of known size to construct a similar LUT or an extra row in the working distance LUT 4203. The magnification can be quantified by determining the count of the motor shaft encoder device based on a known "original" position (such as a physical stop or limit switch trigger position). The processor 4102 may, for example, use the number of sensor pixels at each magnification position to relatively measure the image height. The magnification can be characterized by, for example, dividing the height in pixels by the height in millimeters.

Returning to Figure 42, after calibrating the working distance and magnification of the stereo visualization camera 300, the example processor 4102 is configured to determine a projection center (block 4208). One or more routines that model the stereo visualization camera 300 may be used to determine the projection center (for example, COP), as discussed above in conjunction with FIG. 15. In order to match the left and right stereoscopic views of a surgical site, it is generally desirable to use a mathematical model implemented in the software, firmware, hardware, and/or GPU code for the processor 4102 to model the physical camera 300. Usually, the user can adjust the direction and distance to reproduce and view a perspective of a 3D computer model (such as an MRI image of a brain tumor) (for example, as if the image was captured by a synthetic stereo camera). The adjustability of the model can be used by the processor 4102 to match a viewing angle of a live surgical image, which must therefore be known.

An exemplary embodiment of the stereo visualization camera 300 and/or the processor 4102 is configured to accurately measure and calculate camera model parameters for each value of magnification and working distance. These equivalent values are controlled by a separate optical device contained in the stereo visualization camera 300. The dual optics are aligned so that the parallax between the left channel/view and the right channel/view at the center of an image is approximately zero at the focal plane 4330. In addition, the stereo vision camera 300 is isofocal across the magnification range, and is isocentric across the magnification and working distance range, because the ZRP of each left and right channel has been aligned to its respective pixel network The center of the grid (described above in block 4202). In other words, just changing the magnification will keep the image in focus in both channels and be trained on the same center point. Similarly, if the target and the stereoscopic visualization camera 300 remain fixed, only changing a working distance should not cause vertical parallax in the image, but only increase the horizontal parallax between the left view and the right view.

FIG. 44 illustrates a chart 4400 according to an exemplary embodiment of the present invention. The chart 4400 illustrates a routine that can be executed by the processor 4102 for determining a COP of the stereoscopic visualization camera 300. In the illustrated example, a pinhole or a COP of the modeled camera 300 is along an optical axis 4402 at the plane of the pinhole (O). To determine the COP of the camera model, a virtual pinhole camera model is used, in which the processor 4102 is configured to determine the actual focal length 4404 from the COP to an object plane. During the calibration routine, the processor 4102 keeps the magnification of the camera 300 fixed while (for example) recording a measurement of the image height 4406 with the number of pixels on a plane of the optical image sensor 744, where the height 4408 is in 4402 an object along the optical axis of three different distances: the object plane, and "d" in one plane at a distance less than the distance of the object, and one plane from a distance "d" is greater than at the object. The processor 4102 determines the focal length 4404 to a COP 4410 using a routine containing algebras based on similar triangles at the two most extreme positions. The processor 4102 may determine the focal length at the alternative magnification based on the ratio of the alternative magnification and the magnification used for calibration.

Returning to Figure 42, the example processor 4102 is configured to determine the COP of varying working distance and magnification. The processor 4102 makes the count of the motor shaft encoder of the lens and the COP of varying working distance and magnification correlated in the LUT 4203, a different LUT, or one or more calibration registers. In some embodiments, the processor 4102 may only store one COP relationship of one magnification and/or working distance and use a known COP relationship to calculate other magnifications and/or working distances.

After calibrating a COP, the example processor 4102 is configured to calibrate an interpupillary distance ("IPD") between the stereoscopic left and right optical axis and the axis of the stereo visualization camera 300 (block 4210) . In order to characterize the optical components of the stereo vision camera 300, the IPD between the left channel/view and the right channel/view should be known. In an embodiment, the IPD can be designed into the mechanical components that hold the sensors and optics shown in FIG. 7. Therefore, the IPD is set mechanically. However, the actual optical axis may be different from the mechanical axis of the optical element and its mount. Other embodiments enable the IPD to be changed within the stereo visualization camera 300.

In some applications, it is desirable to accurately know the direction of the stereoscopic optical axis relative to a reference or mechanical axis on a coordinate system of the stereoscopic visualization camera 300. This enables, for example, the processor 4102 to accurately aim the stereoscopic visualization camera 300 through mechanical components. The aiming can be characterized by a geometrically defined viewing vector that coincides with the stereo optical axis and looks outward with respect to a reference system of the stereo visualization camera 300. In addition, the timing of the left channel and the right channel of the optical sensor 744 is included in a viewing vector (including the orientation or posture of the stereo visualization camera 300).

FIG. 45 shows a plan view of an optical schematic diagram illustrating how to measure and calibrate the IPD of the stereo visualization camera 300 according to an exemplary embodiment of the present invention. In the illustrated example, an optical axis 4502 is fully aligned with a mechanical axis 4504. The right image sensor 746 and the left image sensor 748 (eg, approximated by one or more camera models) are separated by an IPD 4506. The sensors 746 and 748 are aligned and focused on an object 4508 (in a target surgical site). The object 4508 is placed at a focal length 4510 from the sensors 746 and 748, so that the parallax at the plane of the object is theoretically zero, as shown in the display of the left or right view of the object at the focal plane 4512. In this illustrative example, for clarity, the object 4508 is a disc, and the front view of the disc is shown as an item 4514.

FIG. 45 also illustrates another example in which the object 4508 is displaced a distance " d " along the mechanical axis and is shown as the item 4508'. Parallax displacement of the object 4508, the disparity in the performance of a left side view of the display 4520 as the P _L and a right side view of the display 4522 appears as a P _R. In this example, the mechanical axis coincides with the optical axis and the parallax magnitude is the same. The parallax can be measured, for example, by counting the number of pixels of the difference between the left view 4520 and the right view 4522 and multiplying it by a magnification factor (pixels/mm) determined in the COP calibration step. The processor 4102 can use triangulation to calculate the IPD. The accuracy of the measurement of the displacement distance d and the parallax of each view helps to accurately know the IPD of the stereo visualization camera 300.

46 shows a plan view of an optical schematic diagram illustrating how to measure and calibrate the optical axis of the stereo visualization camera 300 according to an exemplary embodiment of the present invention. In this example, the optical axis 4502 and the mechanical axis 4504 are not aligned, and the difference is shown as an angle (α) of 4602. The right image sensor 746 and the left image sensor 748 (as approximated by one or more camera models) are aligned and focused on an object 4508 (in a target surgical site), and the object 4508 is placed on a The focal length makes the parallax at the plane of the object 4508 theoretically zero, as shown in the display of the left or right view of the object 4508 at the focal plane 4604.

FIG. 46 also illustrates another example in which the object 4508 is displaced a distance " d " along the mechanical axis and is shown as the object 4508'. The displacement of the object 4508 produces a parallax, which is represented as P _{L 'in the display of the left view 4610 and P R} ' in the display of the right view 4612. In this example where the mechanical axis 4504 and the optical axis 4502 do not coincide, the parallax magnitudes are not equal. The example processor 4102 is configured to calculate IPD via triangulation and the misalignment angle α (e.g., stereo optical axis). The accuracy of the measurement of the displacement distance d and the parallax of each view enables the processor 4102 to accurately determine the IPD and optical axis of the stereo visualization camera 300.

A similar procedure can be used to measure (for example) the misalignment of the mechanical axis and the optical axis in the vertical plane. Misaligned combinations in, for example, the horizontal plane or the vertical plane can be combined so that a viewing vector can be accurately derived with respect to the mechanical axis. In some embodiments, the IPD and optical axis parameters can be measured at varying levels of working distance and/or magnification. The relationship between the IPD, optical axis, working distance, and/or magnification can be stored by the processor 4102 in the LUT 4203, another LUT and/or calibration register.

Returning to FIG. 42, after calibrating the optical axis and/or IPD of the exemplary stereo visualization camera 300, the exemplary processor 4102 is configured to complete the calibration process so that the camera 300 can be connected to the robot arm 506 (block 4212). Procedure 4200 can then end. In some embodiments, if the camera 300 is reinitialized and/or if any of the calibrations cannot be verified or verified, at least portions of the example procedure 4200 are repeated.

It should be understood that, in some embodiments, the above steps of the procedure 4200 can be performed manually or semi-manually. In other embodiments, the above steps may be executed automatically and continuously by the processor 4102. In some embodiments, it is possible to measure the image recognition of a target or any target with one of a sufficient number of objects (including sufficient contrast to enable recognition in both the left view and the right view). In addition, the processor 4102 can determine or calculate a parallax measurement for estimating the accurate relative position of the optical elements of the stereo visualization camera 300. The processor 4102 can perform optical measurements on a real-time basis.

In some embodiments, using automated iterative techniques to perform these or equivalent methods of calibration and measurement can increase accuracy and reduce the time and/or effort required for calibration and measurement. For example, the working distance (and therefore the displacement d ) is accurately known by the number of encoders and the LUT 4203, as explained previously. The magnification rate and its (for example) pixel/mm conversion factor are also accurately known by the number of encoders and the LUT 4203, as described previously. Counting the pixels in the image for difference or object size determination can be accurately performed manually or automatically, for example, as previously explained using template matching. The measurement and storage of these equivalent values can be combined so that the example processor 4102 can accurately deduce the stereo camera model parameters and viewing vectors in near real time.

FIG. 47 illustrates a diagram of a calibrated stereo visualization camera 300 in which one of the optical parameters is sufficiently characterized. In the illustrated embodiment, the left and right optical axes leading to the imaginary left and right image sensor positions 4700 are shown, as determined by a camera model. Figure 47 also shows the central stereoscopic optical axis or viewing vector 4702. The imaginary component of the left view and a right side view of the camera model component is positioned at a focal distance Z. In addition, the left view component and the right view component of the camera model are separated from the measured effective IPD. In the illustrated example, an object at the focal plane is viewed in a similar stereoscopic perspective by imaginary left view components and right view components as recorded by the image sensors 746 and 748 in the stereo visualization camera 300. 2. The calibration of stereo visualization is achieved and the fusion of additional images

The example processor 4102 is configured to use calibration parameters/information to not only provide high-resolution clear images, but also to align the live stereo images and receive one or more images/models from the external device 4104. The mapping of the calibration data related to the camera model parameters in the LUT 4203 and/or the calibration register enables the processor 4102 to create the stereo visualization camera 300 implemented in software, firmware, hardware, and/or computer code. A mathematical model. In one example, the processor 4102 is configured to use, for example, the procedure 4200 discussed in conjunction with FIG. 42 to receive, determine, or access camera model parameters. If a calibration has been performed, the processor 4102 accesses camera model parameters from one or more memories 1570 and/or 4120. The processor 4102 also receives an alternative modality of image data from the device 4104, such as pre-surgical images, MRI images, a 3D model of a surgical site derived from MRI or CT data, X-ray images and/or surgical guides. Board/template. The processor 4102 is configured to use the camera model parameters to reproduce one of the alternative modal data to synthesize a stereoscopic image. The example processor 4102 is also configured to provide synthetic stereo images for display via the monitor 512. In some instances, the processor 4102 is configured to fuse the composite 3D image with the current 3D visualization, where the desired shape of each modality is visible and/or overlapped in the exact same viewing angle (as if by Acquired by a single visualization device).

In some embodiments, the parameters illustrated in FIG. 47 are used by the processor 4102 to match the synthetic stereo image (for example, MRI image data) of one of the alternative modalities with the stereo viewing angle of the stereo visualization camera 300. Therefore, the exemplary processor 4102 uses the stored optical calibration parameters to perform stereoscopic image synthesis. In one example, the processor 4102 uses optical calibration parameters to fuse the live stereo image with a three-dimensional model of a brain tumor imaged preoperatively using an MRI device. The example processor 4102 uses optical calibration parameters to select the corresponding position, size, and/or orientation of the three-dimensional model of the brain tumor that matches the stereo image. In other words, the processor 4102 selects a part of the three-dimensional model corresponding to the view recorded by the stereo visualization camera 300. The processor 4102 may also change which part of the model is displayed based on detecting how the working distance, magnification, and/or orientation of the camera 300 have changed.

The processor 4102 may cause a graphical representation of the model to overlap with the stereoscopic image and/or cause the graphical representation of the model to appear to be visually fused with the stereoscopic image. The image processing performed by the processor 4102 may include smoothing the boundary between the graphical representation of the model and the live stereoscopic view. Image processing may also include causing at least a portion of the graphical representation of the model to have an increased transparency so that the basic stereoscopic view of the scene can also be visible to a surgeon.

In some examples, the processor 4102 is configured to generate and/or render a depth map for each pixel in a stereoscopic image. The processor 4102 can use the calibration parameters to determine, for example, the depth of tissue in an image. The processor 4102 can use the depth information to perform image recognition to record the tissue of interest and/or to identify the position of the instrument so as to avoid unintentional contact when the camera 300 is mated with the robot arm 506. The in-depth information can be output by the processor 4102 to, for example, robotic suture devices, diagnostic equipment, procedure monitoring and recording systems, etc. to perform a coordinated and at least semi-automated surgical procedure. 3. Calibration of the robot arm embodiment

After the stereo visualization camera 300 is calibrated as discussed above, it can be connected to the robot arm 506 and/or the coupling board 3304. As explained below, the accurately known working distance relative to the focal length Z (provided by the stored calibration parameters) is used by the example processor 4102 and/or the robotic processor 4122 to determine the position and position of the stereo visualization camera 300 /Or orientation. The combination of the stereo vision camera 300 and the robotic arm is configured to provide seamless transitions of various working distances while maintaining a focus or view of a target surgical site.

The calibration procedure of the robot arm 506 described below can be executed regardless of the type of the robot arm. For example, a calibration procedure can be performed for an articulated robot system that includes mechanical linkages connected to each other via rotary joints (from simple one or two linkages and joints numbered to six or six The joint structure of the above joints). The calibration procedure can also be executed for a Cartesian robot system, which includes a bracket with linear joints, and the bracket uses a coordinate system with X, Y, and Z directions. One of the last joints of a Cartesian robot system may include a wrist-type revolute joint. The calibration procedure can be further performed on a cylindrical robot system that includes a rotary joint at its base and one or more additional rotary and/or linear joints for a cylindrical working space. In addition, a calibration procedure can be performed for a pole robot system that includes an arm connected to a base via a joint that can operate on more than one axis of rotation and further includes one or more linear or wrist joints. The calibration procedure can additionally be performed for a selective compliant assembly robotic arm ("SCARA") system, which includes operating one of the selective compliant arms in a predominantly cylindrical manner (which is used for assembly applications) .

FIG. 48 illustrates an example program 4800 or routine for calibrating the robot arm 506 according to an example embodiment of the present invention. Although the procedure 4800 is described with reference to the flowchart illustrated in FIG. 48, it should be understood that many other methods of performing the steps associated with the procedure 4800 can be used. For example, the order of the blocks in the block can be changed, a specific block can be combined with other blocks, and many of the blocks described are optional. In addition, the actions described in the procedure 4800 can be executed in multiple devices, including, for example, the optical element 1402 of the exemplary stereoscopic visualization camera 300 of FIG. 14, the image capturing module 1404, and the motor And the lighting module 1406, the information processor module 1408 and/or the joints R1 to R9 and the robot arm controller 4106 of FIG. 41. For example, the program 4800 can be executed by a program stored in the memory 4120 of the robot arm controller 4106.

In some embodiments, the coupling plate 3304 is connected to the robot arm 506 (block 4802). If a coupling board 3304 is not used, the stereo visualization camera 300 is directly connected to the connection or coupling interface 3450 of the robot arm 506. If the coupling board 3304 is used, the stereo visualization camera 300 is connected to the coupling board (block 4804). As discussed above, the first end 3702 of the coupling plate 3304 is connected to the robot arm 506 and the second end 3704 of the coupling plate 3304 is connected to the stereo visualization camera 300.

After the example stereo visualization camera 300 is connected to the robot arm 506, the example processor 4102 and/or the robot arm controller 4106 are configured to calibrate the camera and its viewing vector around the fixed base 3404 originating from the robot arm 506 One of the coordinate systems (block 4806). This coordinate system is called "robot space" or "robotic space" in this article. During this calibration step, the processor 4102 and/or the robotic arm controller 4106 uses the known movement of the robotic arm 506 to determine the orientation of a viewing vector of the camera 300 and an orientation of the object plane during the visualization of a target surgical site And a position.

In some embodiments, there are mechanical features of the camera 300, the coupling plate 3304, and the robot arm 506 so that the relationship between the camera 300, the coupling plate 3304, and the robot arm 506 is uniquely determined and known when they are mechanically connected together. In these embodiments, the processor 4102 and/or the robot arm controller 4106 determines the position, direction, and/or orientation of the viewing vector based on the known mechanical geometry of the camera 300, the coupling plate 3304, and the robot arm 506.

In other embodiments where there are no mechanical features, the example processor 4102 and/or the robot arm controller 4106 are configured to execute a routine to accurately determine the difference between the camera 300 and the robot arm 506 in the robot space. A spatial relationship. The processor 4102 and/or the robot arm controller 4106 move the stereo visualization camera 300 to a starting position, which may include a loading position, a reorientation position, or a surgical operation position. The stereo visualization camera 300 then moves the camera from the starting position to a position of a calibration target that is roughly visualized on the base 3404 of the fixed robot arm 506. The calibration target may be located, for example, in a convenient area of the truck 510 in a position in the motion sphere of the robot arm 506. For example, some examples of calibration targets include small spheres or other uniquely identifiable objects that can be located in a unique known orientation relative to each other (in two-dimensional or three-dimensional images). The coordinates of the sphere are fixed and known relative to the truck 510 and the fixed base 3404, and therefore are known in the robot space. The processor 4102 and/or the robot arm controller 4106 are configured to store the coordinates in the memory 4120, for example.

During the calibration, the processor 4102 and/or the robot arm controller 4106 receive viewing vector data 4807 about the working distance, magnification, stereo optical axis, and/or IPD. The stereo visualization camera 300 is set to simultaneously visualize the sphere at the calibration target and determine its position by using the parallax in the stereo image. The processor 4102 and/or the robot arm controller 4106 records the position of a sphere in an initial coordinate system (for example, X, Y, and Z) relative to a reference in the camera 300 (ie, "camera space"). The X, Y, and Z positions can correspond to an origin position, and are defined as the origin in a file or LUT or have other known coordinate values. The processor 4102 and/or the robot arm controller 4106 also use the output data from the joint sensors to determine the position and orientation of the joints and links in the robot arm 506. The processor 4102 and/or the robot arm controller 4106 also receive position information to determine a position and orientation of the coupling device 3304. In short, the position and orientation of the robot arm 506 and the coupling device 3304 enable the processor 4102 and/or the robot arm controller 4106 to determine a posture of the camera 300. The processor 4102 and/or the robot arm controller 4106 are configured to perform a conversion between the camera space and the robot space based on the position of the sphere of the calibration target and the position of the robot arm 506 and/or the coupling plate 3304 as recorded by the camera. Coordinate transformation. The processor 4102 and/or the robot arm controller 4106 can store the coordinate transformation in one of the LUT 4203, the robot arm 506, and/or one or more calibration registers.

In some embodiments, the camera 300 is moved to record images of multiple calibration targets located in a cart 510, a ceiling, a wall, and/or a surgical area. Each of the calibration targets may have a unique orientation that enables one of its physical X, Y, Z positions to be identified. The processor 4102 and/or the robot arm controller 4106 perform additional coordinate transformations for each of the calibration targets and store the transformations in one or more LUTs and/or registers.

In other embodiments, the processor 4102 and/or the robot arm controller 4106 may use alternative methods to calibrate the camera 300 to the robot space. In this context, "calibration" is regarded as meaning "registration", in which the processor 4102 and/or the robot arm controller 4106 are configured to calculate the registration in a wide space in which the registration can vary. For example, a single stereo camera can be used to observe and locate a calibration target on the cart 510 and a system that is installed on the camera 300 and/or a similar calibration target on a patient or surgical bed. The processor 4102 and/or the robot arm controller 4106 are configured to model and track the camera 300, which is modeled and tracked as a surgical instrument with a viewing vector and working distance. The viewing vector and working distance definition are used to accurately visualize the parameters of a target surgical site. In these other embodiments, another camera determines and reports the position and orientation information of each such instrument in a frame of reference (such as the stereo camera 300). Then, using linear algebra, the processor 4102 and/or the robot arm controller 4106 calculate the posture and/or position of the instruments relative to each other, thereby causing the camera 300 to be calibrated to one of the robot spaces.

In some embodiments, the processor 4102 and/or the robot arm controller 4106 are also configured to calibrate the coupling board 3304. In some cases, the coupling plate 3304 includes one or more switches that are activated depending on the position of one of the joints R7 to R9. The known position of the switch is used by the processor 4102 and/or the robot arm controller 4106 as part of the coordinate transformation. Alternatively or alternatively, the coupling plate 3304 is calibrated by causing the robot arm 506 to move while monitoring the image from the camera 300 to determine the orientation. In one example of the orientation coupling plate 3304 as shown in FIG. 37, the robot arm 506 is commanded to move in a direction relative to an assumed orientation (for example, to move the camera 300 along the z axis). If the hypothetical orientation is as shown in Figure 37, where the camera 300 is aimed downwards, then one of the robot arms 506 moving downwards should cause an object in the image to become larger and larger as the camera 300 gets closer. . If, for example, the object in the image instead moves sideways or up and down, the processor 4102 and/or the robot arm controller 4106 are configured to detect movement and determine that the assumed orientation is incorrect. The processor 4102 and/or the robot arm controller 4106 can generate an error and prompt an operator to be correctly orientated and/or determine the correct orientation based on the detected movement in the image. By using the image matching template algorithm as described previously (for example), the image changes caused by the movement of the camera 300 are automatically decrypted. In some embodiments, the processor 4102 and/or the robot arm controller 4106 uses a matching template algorithm to determine the joint orientation at the coupling plate 3304 (which is stored in a LUT for calibration).

FIG. 49 shows a diagram illustrating how to calibrate the stereo visualization camera 300 and/or the robot arm 506 to the robot space according to an exemplary embodiment of the present invention. In the illustrated embodiment, each of the joints R1 to R9 and the corresponding links are modeled based on the rotational ability and/or length. The memory 4120 can store mathematical parameters associated with the model. In addition, the processor 4102 and/or the robot arm controller 4106 can use a mathematical model to determine (for example) the current position of one of the robot arm 506 and/or the camera 300, and the current position can be used to calculate how the joint will be based on an operation The given movement and rotation provided by the person.

In the illustrated example, the joint R1 is provided at one coordinate position (0,0,0). The length between the joints R1 to R9 corresponds to the length of one of the links. In the illustrated example, the stereo visualization camera 300 is modeled as a robotic end effector connected to one of nine couplers. A sequence of ten homogeneous transformations (which may include matrix multiplication) is used to model the three-dimensional space shown in FIG. 49. The first six coordinate systems or joints R1 to R6 represent the forward kinematics of the robot arm 506, and can be calculated using the Denavit–Hartenberg parameters of a robot arm. The next three coordinate systems or joints R7 to R9 represent the transformation from the tip of the tool of the robot arm 506 to the tip of the coupling plate 3304. The last coordinate system R10 represents the transformation from the tool tip of the coupling plate 3304 to the control point of the stereo visualization camera 300.

The coordinate system or joint R7 represents the pitch joint of the coupling plate 3304 that can be changed between 0° and 90°. The coordinate system or joint R8 represents the roll joint of the coupling plate 3304, and can be changed between -90°, 0°, and 90° depending on the roll configuration. The joints R7 to R9 of the coupling plate can include a voltage source and a potentiometer. The connector 3450 and/or the coupler controller 4130 of the robot arm 506 may include an I/O tool tip connector configured to receive a voltage output from the potentiometer. The processor 4102 and/or the robot arm controller 4106 are configured to receive the output voltage and thus determine the pitch angle and the roll angle of the coupling plate 3304. The processor 4102 and/or the robot arm controller 4106 combine the pitch and roll information of the coupling plate with the sensor output data from the joints R1 to R6 of the robot arm 506 to calculate the position of the coordinate system R1 to R10 to determine the robot arm 506. The three-dimensional position of the coupling board 3304 and/or the camera 300.

The control point represents the coordinate system 10 at the end of the kinematic chain, and is fully programmable in terms of position (based on which features are selected). For example, if an operator selects an auxiliary driving feature, the processor 4102 and/or the robot arm controller 4106 are configured to set the control point representing the camera 300 along a rotation axis of the control arm 304 to be at the camera Inside. In another example, if an operator selects a lock to target feature, the processor 4102 and/or the robot arm controller 4106 are configured to set the control point of the camera 300 to the origin of an optical axis viewing vector .

Returning to FIG. 48, after calibrating the camera 300 to the robot space, the processor 4102 and/or the robot arm controller 4106 are configured to calibrate the robot space to the patient space (block 4808). The calibration of the patient space needs to enable the stereoscopic visualization platform 516 to accurately visualize a patient, which requires the orientation between the robot system and the patient. In some embodiments, this orientation is fixed. In other embodiments, the orientation (if changing) is sensed and known. In some embodiments, one or more fiducials 4809 are used to place a patient on an operating room bed and register to the bed. For example, if a patient is undergoing brain surgery, he is fastened to a bed and an external frame is fixed to his skull. In a configuration in which two or more non-collinear objects with known positions are visible at the same time (such as the configuration of a calibration target), the frame can be observed by the stereo visualization camera 300 and can include the reference 4809, so that the frame and therefore The position and orientation of the patient's skull can be determined. Other embodiments may use fiducial 4809 implanted in a patient and visible in MRI or similar images. These benchmarks 4809 can be used to accurately track a patient's skull and MRI images and register the patient's skull and MRI images to a coordinate system representing the patient's space. In addition, other embodiments may use image recognition based on the characteristics of the patient. For example, facial or similar recognition using biometric data, in situ x-rays or similar alternative modal imaging can be used to accurately locate the position and orientation of one of the patients. In another example, one or more depth map calculations as described above and the surface matching function performed by the processor 4102 and/or the robot arm controller 4106 can be used to determine a model of a surface of a patient's face .

In one embodiment, the position and orientation of an operating room bed relative to the robot space are fixed and determined. Certain embodiments include mechanically registering the bed to, for example, a rigid frame of an accessory on the truck 510 in a known position and orientation. Alternatively, the bed can be fixed relative to the robot arm 506 and the reference can be used to determine the position and orientation. For example, the robot truck 510 and the bed can be anchored to the floor and fixed for the duration of the procedure.

After the patient's fiducial 4809 is visualized by the camera 300, the processor 4102 and/or the robot arm controller 4106 can decrypt and store its position and orientation in the robot space, in which the coordinate system transformation from the robot space to the patient space is achieved. . It should be noted that the transformation of the coordinate system from one space to another is generally selectable and reversible. For example, transforming the desired camera motion or posture into the robot space so that the processor 4102 and/or the robot arm controller 4106 can determine the discrete joint motion and orientation can be more efficient. Alternatively, presenting information to a surgeon on the display monitor 512 in the patient space can be easier and more efficient. The processor 4102 and/or the robot arm controller 4106 can optionally transform the positions of points and vectors to any coordinate system (for example, a cart origin, a patient reference system, GPS and/or other coordinate systems). ) Of each.

In some embodiments, the processor 4102 and/or the robotic arm controller 4106 are configured to use automated iteration techniques to perform these or equivalent methods of robot/patient space calibration and measurement to increase accuracy and reduce calibration time. In an exemplary embodiment, the processor 4102 and/or the robot arm controller 4106 accurately know the displacement and orientation of the stereo visualization camera 300 relative to the reference. The movement of the robot arm 506 can be accurately executed, and the subsequent images of the benchmark can be accurately analyzed. The processor 4102 and/or the robot arm controller 4106 can combine the visualization and knowledge of the calibration parameters, so that the measurement and thus the calibration can be accurately performed in an automated manner. This (for example) is important to maintain accurate calibration from one surgical procedure and one patient to the next surgical procedure and the next patient.

In some instances, the processor 4102 and/or the robotic arm controller 4106 are configured to determine the boundary of the robotic arm 506 and/or the camera 300 with respect to the patient space and/or the robot space. These boundaries represent virtual restrictions implemented in software to prevent the robot arm 506 and/or the camera 300 from contacting or escaping from the defined area or space. In some instances, the limits are defined in one or more LUTs or registers stored in the memory 4120 as a scale factor applied by the processor 4102 and/or the robot arm controller 4106 to the joint movement signal. The magnitude of the scale factor decreases to zero as it approaches the limit of each individual boundary. For example, the joint rotation amount and speed can be determined based on the operator's input. However, the processor 4102 and/or the robot arm controller 4106 scales the joint rotation speed by a scaling factor before sending the signal to the appropriate joint. In addition, the processor 4102 and/or the robot arm controller 4106 can maintain the amount of rotation, so that the joint moves by the required amount but at a reduced speed until the joint reaches the boundary. It should be understood that a scale factor may not be applied to a joint in a rotation region in which a scale factor is applied when the moving system is far away from the boundary. Therefore, the processor 4102 and/or the robot arm controller 4106 can apply a scale factor to a specific joint while applying a scale factor of "1" to other joints based on a current position and estimated movement from an operator. .

The scale factors are strictly between zero and one, which makes it possible to link them together and enables the software to support an unlimited number of possible boundaries. The scale factors can linearly decrease as they approach a boundary, which causes the rotation of the joints R1 to R9 to gradually slow down as the robot arm 506 approaches a boundary. In other examples, the scale factors may decrease exponentially as they approach a boundary.

Generally speaking, the operator usually focuses his attention on the surgical field or the 3D image on the display monitor 512. As such, the operator usually does not know the positions of the individual links of the robot arm 506 and/or the coupling plate 3304. Therefore, it is not always intuitive when the robot arm 506 is about to reach a limit or hit another part of the robot arm 506. The joint restriction can therefore always be effective and prevent any part of the robot arm 506 from hitting itself or placing the joint in a single configuration (such as an elbow lock). The example processor 4102 and/or the robot arm controller 4106 are configured to determine the scale factor based on the current position of one of the robot arms 506. The processor 4102 and/or the robot arm controller 4106 may also consider a predetermined movement command provided by an operator to determine which scale factor will be applied. Based on the current and/or expected movement, the processor 4102 and/or the robot arm controller 4106 uses, for example, one or more LUTs to calculate a scale factor based on the distance in the joint angle space. The joint angle spacing can define a specific combination of joint angles, which are known to cause the joint to lock or cause the robot arm 506 to hit itself. In this way, the joint angular distance determination is based on determining the current (and/or expected) joint movement and comparing the current (and/or expected) joint movement with respect to each other.

In addition to the boundary of the robot arm 506, the memory 4120 can also store Cartesian restrictions for preventing the robot arm 506 from hitting the truck 510, preventing the robot arm from hitting the display monitor 512, and/or preventing the camera 300 from hitting the robot arm 506 The relevant boundary. The processor 4102 and/or the robot arm controller 4106 may use, for example, the coordinate system discussed in conjunction with FIG. 49 to determine and/or apply Cartesian constraints. In some instances, the restrictions may be relative or anchored to a specific link. In this way, when the connecting rod is moved in the 3D space, its surrounding boundary moves accordingly. In other examples, the constraints are static and fixed to specific coordinate planes or lines in the 3D space shown in FIG. 49. The processor 4102 and/or the robot arm controller 4106 can apply these restrictions by calculating or determining the scale factor in Cartesian space and applying the forward motion transformation.

The example processor 4102 and/or the robot arm controller 4106 can also determine a patient boundary, which defines a virtual place where any point of the robot arm 506 and/or camera 300 is inviolable. The patient boundary can be determined by calculating the scale factor in Cartesian space for a distance from each position joint on the robot arm 506 and/or the coupling plate 3304 to a position of a boundary plane. The boundary plane shown in the orientation 5002 of FIG. 50 is implemented as an X, Y plane at a certain vertical Z position to achieve a non-trim configuration. For a trim configuration, such as the patient half-sitting shown in the orientation 5004 of FIG. 50, the boundary plane is set to be the Y, Z plane at one of the positive or negative X values depending on the direction the camera 300 faces.

The exemplary boundaries discussed above can be stored in the memory 4120 as a preset boundary and/or determined by the processor 4102 and/or the robotic arm controller 4106 before a surgical procedure. In some embodiments, a specific boundary can be accessed or determined based on the input type of one of the surgical procedures to be performed. For example, the patient boundary can be determined by imaging the patient with the camera 300 and using calibration information/parameters to determine the depth of the patient. The processor 4102 and/or the robotic arm controller 4106 may then create a boundary and apply the boundary to a designated location on or close to the patient. A similar boundary can be created after a monitor, surgical staff, or surgical instrument is detected.

For example, the boundary can be determined around the use of a specific surgical tool (such as a tool of a larger size or a tool that poses a specific risk if touched). The example processor 4102 and/or the robot arm controller 4106 may receive input of one of the tool types and/or use image analysis to detect the tool in the stereo image. In other examples, the processor 4102 and/or the robotic arm controller 4106 calculate depth information about a surgical instrument to determine its size, orientation, and/or position. The example processor 4102 and/or the robotic arm controller 4106 convert the image of the surgical instrument into a coordinate system, such as the coordinate system discussed in conjunction with FIG. 49. The processor 4102 and/or the robot arm controller 4106 also apply a scale factor with a value less than "1" to the area corresponding to a position of the surgical instrument, thereby preventing the robot arm 506 and/or the camera 300 from inadvertently contacting Surgical tools. In some instances, the processor 4102 and/or the robotic arm controller 4106 may track the movement of one of the surgical tools during a procedure and thus change the boundary.

FIG. 51 illustrates an example of how to scale the rotational joint speed of the robot arm 506 and/or the coupling plate 3304 based on the distance to a boundary according to an exemplary embodiment of the present invention. The graph 5102 shows a rotation speed of the joint R1 and the graph 5104 shows a first zone 5112 (where a scale factor decreases from a value of "1") relative to a region close to a boundary and the boundary A shoulder angle (for example, a rotation position) 5110 corresponding to a second zone 5114 (where the scale factor is reduced to a value of "0").

FIG. 51 shows that the rotation speed is dynamically scaled relative to the distance to the second zone 5114 when the robot arm 506 and, in particular, the joint R1 causes the at least one link and/or the stereo visualization camera 300 to approach the first zone 5112. Then, when the at least one link and/or stereo visualization camera 300 reaches the second zone 5114, the scale factor is reduced to a value of "0" and all the rotation joints moving toward the boundary are stopped. In other words, when the robot arm 506 and the stereo visualization camera 300 approach a limit or boundary, the processor 4102 and/or the robot arm controller 4106 causes the rotation speed of at least some of the joints R1 to R9 to decrease and reach the first The second zone finally reaches a speed of "0" degrees/second at 5114 (as shown in charts 5102 and 5104 as being between 20 seconds and 30 seconds). The graph also shows that when at least one linkage and/or stereo visualization camera 300 moves away from the second zone 5114, the processor 4102 and/or the robot arm controller 4106 uses a scale factor value of "1". Close to the second zone 5114.

In some embodiments, the processor 4102 and/or the robot arm controller 4106 are configured to cause the display monitor 512 or other user interface to display one or more graphical icons representing a state of the robot arm 506. For example, when the robot arm 506 and/or the camera 300 are located in a zone or zone where the scale factor has a value of "1", a green icon may be displayed. In addition, when the robot arm 506 and/or the camera 300 are located in the first zone 5112, a yellow icon may be displayed to indicate that the joint rotation speed is slowing down. In addition, when the robot arm 506 reaches the second zone 5114 or a boundary/limit, a red icon may be displayed to indicate that additional movement beyond the boundary is impossible.

Returning to FIG. 48, after determining the robot space boundary, the example processor 4102 and/or the robot arm controller 4106 are configured to enable the robot arm 506 to operate with the stereo visualization camera 300 (block 4812). This may include enabling the robotic arm 506 and the stereo visualization camera 300 to be used during a surgical procedure. This may also include enabling features, such as assisted drive and/or lock to target. Additionally or alternatively, this may include enabling one or more user controls at one or more of the input devices 1410 of FIG. 41. The example procedure 4800 ends after the robotic arm 506 and the stereo visualization camera 300 are activated together. If the stereo visualization platform 516 is reinitialized, undergoes a detected failure, or cannot verify the calibration, the example procedure 4800 can be repeated. D. Example of operation of stereo visual camera and robot arm

The example stereo visualization camera 300 is configured to operate in conjunction with the robotic arm 506 and/or the coupling plate 3304 to provide enhanced visualization features. As discussed in more detail below, enhanced features include an extended focus, automatic focus tip positioning, providing a measurement of the distance between objects in an image, providing robot motion with joint visualization, sag compensation, and image fusion And visualized location/orientation storage. The enhanced visualization features may also include auxiliary driving capabilities of the robotic arm 506 and a lock-to-target capability that enables the camera to be locked to a specific view while enabling one of the orientations of the robotic arm 506 and/or the coupling plate 3304 to be changed. 1. Example of extended focus

In some embodiments, the robotic arm 506 and/or the coupling plate 3304 may provide an extended focus of one of the cameras 300. As discussed above in conjunction with FIG. 43, the stereo visualization camera 300 includes the main objective lens assembly 702 for changing a working distance. In order to focus on an object in the surgical site, the main objective lens assembly 702 is changed from just in front of the object to just behind the object by a focal length. However, in some cases, the main objective assembly 702 reaches one of the mechanical limits of the front working distance lens 408 before reaching the best focus.

The example processor 4102 and/or the robot arm controller 4106 are configured to detect when a mechanical limit is reached and/or determine that a mechanical limit of the lens 408 is about to be reached and therefore adjust a position of the robot arm 506 instead. The processor 4102 and/or the robot arm controller 4106 are configured to use the robot arm 506 to view the vector by calculating one of the cameras 300 and cause the robot arm 506 to be actuated along the optical axis to extend the focus. The processor 4102 and/or the robot arm controller 4106 use the calibration parameters described above of the stereo visualization camera 300 to determine a distance required to achieve the focus. For example, as discussed above, a position of the front working distance lens 408 is mapped to a physical working distance of the main objective lens assembly 702 to a target object. This distance provides an estimate of how far the center of one of the cameras 300 is from the target object. In addition, the calibration parameters may include a mapping between the motor or encoder pitch of the front working distance lens 408 and the working distance to provide an estimate of the distance required to achieve a specific working distance or focus. Therefore, the processor 4102 and/or the robot arm controller 4106 can read the current encoder value of the front working distance lens 408 and determine that it represents a vertical distance of one meter from the camera 300 to the target object. In other words, the processor 4102 and/or the robot arm controller 4106 convert the lens movement (in the encoder count) into a physical distance in the robot space. The processor 4102 and/or the robot arm controller 4106 then determine the joint rotation speed, direction and/or duration (for example, a movement sequence) that will cause the robot arm 506 to move along the optical axis by the determined distance. The processor 4102 and/or the robot arm controller 4106 then transmits one or more signals corresponding to the movement sequence to the appropriate joints to cause the robot arm 506 to provide an extended focus. In some cases, the processor 4102 and/or the robot arm controller 4106 may apply a scale factor before the signal is transmitted to the robot arm 506 and/or the joints R1 to R9 of the coupling plate 3304.

It should be understood that the focus extension causes one of the robotic arms 506 to move automatically. In other words, the robot arm 506 can continue the motion of the camera 300 through the best focus point. In addition, the movement of the robot arm 506 occurs without an input from an operator to move the robot arm (to be precise, regarding an operator image that changes a focal length). In some cases, the processor 4102 and/or the robot arm controller 4106 can automatically adjust the focus to maintain a clear image.

In some embodiments, the processor 4102 and/or the robot arm controller 4106 are configured to move the robot arm 506 along the working distance of the camera in response to a single button press via the input device 1410. This feature enables an operator to fix a motor position of the main objective lens assembly 702 and obtain focus by moving the robot arm 506 and/or the coupling plate 3304. The "robot auto-focusing" feature or procedure is completed by the processor 4102 and/or the robot arm controller 4106 to estimate or determine the distance from the front of the main objective lens assembly 702 to a target, as discussed above in conjunction with FIG. 43 . The processor 4102 and/or the robot arm controller 4106 are configured to use the determined distance with a feedback law to command a vertical speed of the robot arm 506 (or the speed along an optical axis of the camera 300) to the determined distance Until it reaches a value of "0". The processor 4102 and/or the robot arm controller 4106 can use the auto-focus algorithm at any time during a program to focus a target object. In some embodiments, the processor 4102 and/or the robot arm controller 4106 may use the movement of the robot arm 506 and/or the coupling plate 3304 since the last time the auto focus was used as a kind of sub or starting point when searching for an auto focus direction. , Thus improving the speed and accuracy of focusing a target object.

It should be understood that, in addition to or instead of making the front lens group 714, lens barrel group 718 and/or last optical group 742 (each of which can be mapped to position, focus, working distance and/or magnification) The encoder counts the movement of one of the individual motors. The example processor 4102 and/or the robot arm controller 4106 can also be configured to cause the robot arm 506 and/or the coupling plate 3304 to move. For example, when any one of the current lens group 714, the lens barrel group 718, and/or the last optical group 742 is about to approach a movement limit, the processor 4102 and/or the robot arm controller 4106 may cause the robot arm 506 to move along Move along an optical axis. In some examples, the processor 4102 and/or the robot arm controller 4106 may cause the robot arm 506 to first move to a position that is roughly in focus or close to the focus, and then adjust the front lens group 714, lens barrel group 718, and/or Finally, the optical group 742 makes the target image enter close to the ideal focus. 2. Example of automatic focusing tip positioning

In some embodiments, the robotic arm 506 and/or the coupling plate 3304 may operate in conjunction with the stereo visualization camera 300 to provide automated focus tip positioning. In these embodiments, the processor 4102 and/or the robotic arm controller 4106 are configured to position the camera 300 for visualizing a target surgical procedure without information or feedback of a specific image and its content Location. The processor 4102 and/or the robot arm controller 4106 may use the calibrated camera model parameters discussed above in connection with FIGS. 42 and 49 to perform open loop camera positioning. The processor 4102 and/or the robot arm controller 4106 can cause the robot arm 506 to position the stereo visualization camera 300 so that a focal point or tip of the camera is in a scene. The stereo visualization camera 300 determines the aiming direction of the camera 300 relative to a standard system based on the calibration information about the posture of the robot arm 506 and/or the coupling plate and the optical calibration parameters of the camera 300. The processor 4102 and/or the robot arm controller 4106 can characterize the aiming with respect to the coordinate system of the robot arm 506 by aligning a geometrically defined viewing vector coincident with the stereo optical axis of the camera 300.

In some embodiments, the processor 4102 and/or the robot arm controller 4106 are configured to execute an initialization routine to align the calibration parameters and/or other memory data to an actual physical reference position, which can be used for Tip positioning. For example, the processor 4102 and/or the robot arm controller 4106 can cause the robot arm 506 and/or the coupling plate to move to a hard stop at "position 0", where all position data fields are set to 0 (or in a three-dimensional 0,0,0 in space). Perform additional movement relative to this point and update the position data based on, for example, the encoder count of the robot arm 506 and/or the various joint motors of the coupling plate 3304.

In other embodiments, the processor 4102 and/or the robot arm controller 4106 may determine or set a tip position of the camera 300 based on one or more visualization parameters. For example, the processor 4102 and/or the robot arm controller 4106 may use a projection center position as a near end of a viewing vector (for example, for aiming the camera 300 at a "starting point"). In some surgical systems, this point on a surgical instrument is called the "post" point and can be provided relative to the tip. The processor 4102 and/or the robot arm controller 4106 calculates a viewing vector direction from the tip and back points to determine whether the camera 300 is aimed at one of the coordinate systems of the robot arm 506.

Alternatively or alternatively, the processor 4102 and/or the robot arm controller 4106 may determine a focal length for calculating a range from the projection center of a focal plane of a stereoscopic image. The center of the image at the focal plane is the "tip" point. The processor 4102 and/or the robot arm controller 4106 can use a calibrated working distance to determine the actual, spatial, and physical distance from the camera 300 to the tip point. In addition, the processor 4102 and/or the robot arm controller 4106 may determine the magnification, as discussed above regarding magnification calibration. 3. Examples of distance measurement

In some embodiments, the robot arm 506 and/or the coupling plate 3304 can be operated in conjunction with the stereo visualization camera 300 to provide a distance measurement and/or depth measurement between objects in a stereo image. For example, the processor 4102 may use the optical calibration parameters transformed into the robot space to determine a focal point or a center of a tip of the camera 300 relative to the coordinate system of the robot arm 506 in dimensionality. As discussed above in conjunction with FIGS. 45 and 46, one of the viewing vectors and left/right disparity information at any point in an image can be used by the processor 4102 to measure the three points by triangulation about the tip or about any other point in the image. Calculate its position on three dimensions. This triangulation measurement enables the processor 4102 to map any point in an image to the robot coordinate system. In this way, the processor 4102 can calculate the position and/or depth of multiple objects and/or the positions of different parts of an object with respect to the same coordinate space of the robot arm 506, which makes it possible to determine a distance measurement and/or between objects Or depth measurement.

The processor 4102 can cause the distance and/or depth measurement information to be visually displayed above the stereoscopic image and/or displayed together with the stereoscopic image. In some cases, an operator can use the input device 1410 to select two or more objects by using a finger or a surgical instrument to select these objects on a screen or point directly to one of the patients. The actual object. The processor 4102 receives the indication of selection and therefore determines the coordinates of the object and the distance between the objects. The processor 4102 can then display a linear graph and/or value (and/or an indication of the selected object) indicating the distance along with the stereo image.

In addition, tracking of objects allows the locations of other objects that have been previously imaged (or provided in other images) to be stored and compared later. For example, the camera 300 can be moved to a position where at least some of the objects are outside the current FOV. However, an operator can instruct the processor 4102 to determine a distance between an object in the FOV and a previously imaged object currently outside the FOV.

In some embodiments, the processor 4102 may use the coordinates of the object to fuse digital images or models from alternative modal visualization, such as MRI images, X-ray images, surgical templates or guides, preoperative images, and the like. The example processor 4102 is configured to use the object position and depth information in the coordinate plane to appropriately scale, orient, and position the alternative modality visualization. The processor 4102 may select at least a part of the alternative modal visualization that has exactly the same feature (for example, an object) in a displayed 3D image. For example, the processor 4102 can use an image analysis routine to locate a blood vessel pattern, a scar, a deformity, or other visible physical structure or object in a three-dimensional image. The processor 4102 then locates the exact same features in the alternative modality visualization. The processor 4102 selects a part of the alternative modal visualization that contains exactly the same characteristics. The processor 4102 may then use the coordinates, depth, and/or distance between features in the stereoscopic image to scale, rotate, and/or orient selected portions of the alternative modality visualization. The processor can then fuse the adjusted part of the alternative modal visualization with the stereo image. The processor 4102 can track how the identifiable objects move relative to each other and/or relative to the FOV to determine how to update the fused image accordingly. For example, moving the camera 300 to another surgical location may cause the processor 4102 to select another part of the pre-surgical image for fusion with the stereoscopic image of the other surgical location.

In some examples, the processor 4102 and/or the robot arm controller 4106 may cause the robot arm 506 to move to track the movement of one of the objects in the FOV. The processor 4102 uses the coordinate position of the object to detect movement or confusion. In response to the detected movement or chaos, the processor 4102 and/or the robot arm controller 4106 are configured to determine how to move the robot arm 506 and/or the coupling plate 3304 to track the movement of the object or overcome the chaos. For example, the processor 4102 and/or the robot arm controller 4106 can move the robot arm 506 in a circular path to visualize a point on a patient's retina from multiple directions to avoid reflections or confusion from the tool. 4. Example of image fusion

As discussed above, the processor 4102 is configured to merge an image from an alternative modality into a live stereo image. For example, if a surgeon is operating on a patient with a deep brain tumor, the surgeon can instruct the processor 4102 to take an MRI image of the brain tumor in the proper position and at the proper depth and perspective. It is visualized to display the live image from the camera 300 on the monitor 512. In some embodiments, the processor 4102 is configured to use distance and/or depth measurement information of one or more objects in the FOV to merge with the alternative modal view. The processor 4102 may also use stereoscopic optical axis (eg, viewing vector), IPD, and/or camera model parameters calculated in the calibration step discussed in conjunction with FIG. 42 and stored in one or more LUTs to provide imaging fusion. The use of these optical calibration parameters enables the processor 4102 to display an alternative modal image, as if the image was acquired by the stereo visualization camera 300. The processor 4102 can use the optical calibration parameters of the camera to model, scale or modify the alternative modal image based on an effective IPD of the camera 300, so that the distance Z from a focal point in the surgical site View alternate modal images (given the working distance and magnification of the camera 300).

FIG. 52 shows a diagram of an example program 5200 for fusing an image and a stereo image from an alternative modal visualization according to an exemplary embodiment of the present invention. Although the procedure 5200 is described with reference to the flowchart illustrated in FIG. 52, it should be understood that many other methods of performing the steps associated with the procedure 5200 can be used. For example, the order of the blocks in the block can be changed, a specific block can be combined with other blocks, and many of the blocks described are optional. In addition, the actions described in the procedure 5200 can be executed in multiple devices, including, for example, the optical element 1402 of the exemplary stereo visualization camera 300 of FIG. 14, the image capturing module 1404, The motor and lighting module 1406, the information processor module 1408 and/or the joints R1 to R9 and the robot arm controller 4106 of FIG. 41. For example, the program 5200 can be executed by a program stored in the memory 1570 of the processor 4102.

The example processor 4102 of the program 5200 is configured to use optical calibration parameters to reproduce, for example, the previously generated three-dimensional MRI data of a patient as a stereoscopic image with a proper angle of view as a stereoscopic image recorded by the camera 300 image. The processor 4102 may receive (for example) an alternative modal visualization, such as MRI data, from the device 4104 of FIG. 41 (block 5202). The processor 4102 may also receive an input 5203 which indicates that the alternative modal visualization will be merged with the stereoscopic image recorded by the stereoscopic visualization camera 300 via an input device 1410 (block 5204).

During the procedure 5200, when a surgeon positions the camera 300 at a desired orientation and position for a surgical procedure, the processor 4102 obtains the posture data 5205 (block 5206). The posture data 5201 may include the position of the robot arm 506, the coupling board 3304, and/or the stereo visualization camera 300. The processor 4102 also accesses the magnification and working distance optical calibration parameters 5207 related to the camera 300 from one or more LUTs (such as the LUT 4203 of FIG. 42) (block 5208). The processor 4102 uses the posture data 5205 together with the magnification and working distance optical calibration parameters 5207 to determine a stereo axis and IPD of the camera 300 (block 5210). The processor 4102 applies posture data, three-dimensional axis data, IPD data, and/or optical calibration parameters to select at least a part of the MRI data and/or modify, scale, orient, and segment the selected part of the MRI data, so that A perspective of a view of the brain of the patient viewed by the stereoscopic visualizing camera 300 provides the selected portion (block 5212). The processor 4102 is configured to apply the stereoscopic optical axis viewing vector and IPD for reproducing the selected portion of the MRI data as a stereoscopic image corresponding to the current scene view of the camera 300 (block 5114). The processor 4102 may then fuse the stereo MRI image with the live stereo image from the camera 300, as discussed herein (block 5216) .

As discussed above, the processor 4102 can use an object or feature to locate or fuse the reproduced MRI data with the stereo image from the stereo visualization camera 300. For example, the processor 4102 may use one or more image analysis routines to identify distinguishing features or objects in a stereo image, locate the same distinguishing features in the reproduced stereo MRI data, and place the reproduced stereo MRI data on the camera Above the appropriate part of the 3D image, the features or objects are aligned and have the same scale, size, depth, orientation, etc. The processor 4102 can make the reproduced stereo MRI data at least partially transparent so that the live images can also be viewed. Additionally or alternatively, the processor 4102 can adjust a shadow at a frame of the reproduced stereo MRI data to reduce the visual contrast between the reproduced stereo MRI data and the camera stereo image. The example procedure 5200 of FIG. 52 may then end.

The example procedure 5200 enables the brain tumor to be visualized by the surgeon in an accurate position relative to the camera 300 in one of the stereoscopic images. Surgeons can use this fusion visualization especially in the middle of a surgical procedure. For example, the surgeon can see the tumor that is still exposed as one of the "x-ray vision" methods best described as being below a current anatomical level. The input device 1410 can be used to adjust the transparency control of the live or reproduced 3D MRI image to optimize the clarity of the fused image. The exemplary procedure thus achieves a safer, more accurate and efficient resection of one of the tumors.

In some embodiments, if a FOV, focus, working distance, and/or magnification change, the procedure 5200 can be repeated. In these embodiments, the processor 4102 is configured to use the updated posture information and extract the corresponding stereo axis and IPD from a lookup table to reproduce the MRI data as an updated accurate stereo image. The processor 4102 is configured to fuse the newly reproduced MRI data into the current three-dimensional image, so that the scene view and the corresponding MRI data are located in the proper position, depth, and orientation.

In some embodiments, the example processor 4102 is configured to operate with the stereo visualization camera 300, the robotic arm 506, and/or the coupling plate 3304 to produce a live profile fusion visualization. The visualization of a cross-section of a surgical site provides a surgeon with a significantly improved perspective that cannot be obtained in other ways. Figure 53 shows a diagram of a patient 5300 with a glioblastoma 5302. The glioblastoma 5302 is illustrated as a phantom on the inside of the head of a patient. Specifically, the glioblastoma 5302 can be located in the patient's brain 5304, which is displayed as a light phantom line. The diagram in FIG. 53 is, for example, a typical preoperative diagnostic image from an MRI device, in which a number of image slices are stacked and reproduce and visualize a 3D model of an interior of a patient's skull 5306.

In the illustrated example, the glioblastoma 5302 will be removed through brain surgery. FIG. 54 shows a diagram of a perspective view of a patient 5300 undergoing a craniotomy procedure 5400 to provide access to the skull 5306. Procedure 5400 also includes brain dissection and retraction using surgical instruments 5402. Generally speaking, a surgical access site 5404 is formed in a deep cone shape to access the glioblastoma 5302.

55 shows a diagram of a stereo visualization platform 516 according to an exemplary embodiment of the present invention. The stereo visualization platform 516 includes a stereo visualization camera 300 and a robotic arm 506 to visualize the craniotomy procedure 5400. As illustrated, the craniotomy procedure 5400 is configured such that the robotic arm 506 is positioned so that the stereo visualization camera 300 is aimed along the visualization axis 5500 of the conical surgical site 5404 through the top of the skull 5306. One of the operating surgeons looks generally through the top of the skull 5306, as shown in FIG. 57. As can be understood from FIG. 7, the depth of the surgical operation and (for example) the tip of the surgical instrument 5402 are difficult to see.

The exemplary stereo visualization camera 300 shown in FIG. 55 provides a highly accurate stereo image that is viewed downward along the axis 5500 of the accessible part of the conical surgery. As discussed above, the disparity information between the left view and the right view of the camera 300 is used by the processor 4102 for all points in the accessible part that are common to the two views according to a known reference depth (such as, for example, the object plane). ) Determine the depth of each point. In the illustrated example, the disparity between the left view and the right view is equal to a value of "0", which enables the processor 4102 to determine a depth map for each point in the image. The processor 4102 can reproduce the depth map as if viewing the map from a different angle. In addition, the processor 4102 is configured to make at least a portion of the depth map transparent based on receiving an instruction from a surgeon and/or an operator. In the illustrated example, the processor 4102 can make a part of the depth map below the cross-sectional plane AA of FIG. 57 transparent, thereby enabling the processor 4102 to generate a cross-sectional view of the site 5404 accessible to surgery.

Figure 56 shows a schematic diagram of a phantom image of the accessible part 5404 of conical surgery. The illustrated surgical access site 5404 includes stepped conical segments for clarity in this discussion. In this example, one of the scan cone angles of part 5404 is specified by the angle "α".

Figure 58 shows a diagram of the conical surgical access site 5404 used in the craniotomy procedure 5400. The prior knowledge of the size and shape of the surgical instrument 5402 together with the image recognition of its position, direction, and/or orientation enables the processor 4102 to generate the image data of the cross-sectional view shown in FIG. 58. The identification of the instrument 5402 in the three-dimensional view shown in FIG. 57 achieves its precise placement in the cross-sectional view of FIG. 58 and (for example) the bottom side of the instrument that is not visible to the surgeon during surgery on the brain 5304 of the patient Of visualization.

In some embodiments, the processor 4102 is configured to fuse an image of a glioblastoma 5302 with an approaching scene or a three-dimensional image of the scene. As discussed above, the combination of the robot arm 506 and the camera 300 provides highly accurate position, direction, and/or orientation information of a view relative to the robot reference frame or robot space. After the robot arm 506 and the camera 300 are registered or calibrated to the reference frame of the patient 5300, the processor 4102 generates the accurate position, direction and/or orientation information of the surgical accessible part 5404 and its respective positions to the patient. The processor 4102 uses image fusion to superimpose a selected part of the MRI image of the glioblastoma 5302 onto a cross-sectional view, as shown in FIG. 59. In addition, the image fusion achieves the visualization of other related MRI image data, including, for example, the cerebral vasculature or other structures expected to be included in the image. The exemplary surgical procedure continues with the surgeon being able to see and understand the depth location of the glioblastoma 5302 (except for a safe distance or positioning of the instrument 5402).

Fig. 59 shows a schematic diagram of a cross-sectional view of a surgical access site 5404. In this example, a portion 5302' of the glioblastoma 5302 is visible to the stereo visualization camera 300 series. FIG. 60 shows a diagram of a cross-sectional view orthogonal to the plane AA of FIG. 57. This drawing can illustrate a cross-sectional view generated by the processor 4102 based on the MRI data fusion with the live view of the surgical access site 5404. The processor 4102 uses the depth map to make it possible to reproduce the surgically accessible parts 5404 at various desired segment planes and segment plane combinations, as shown in FIG. 61. The rendering enables the processor 4102 to display the intact glioblastoma 5302 including the visible part 5402' and the remaining part based on the MRI data. The processor 4102 can display the visualization from a perspective of the camera 300 or display the visualization as a cross-sectional view, as shown in FIG. 61. 5. Robot motion embodiment with joint visualization

In some embodiments, the example processor 4102 operates in conjunction with the robot arm controller 4106, the stereo visualization camera 300, the robot arm 506, and/or the coupling board 3304 to combine visualization and robot motion. In some instances, the processor 4102 and/or the robot arm controller 4106 operate in a closed loop to provide joint visualization based on robot motion. In these examples, the processor 4102 and/or the robot arm controller 4106 are configured to position the robot arm 506, the coupling plate 3304, and/or the robot arm 506, the coupling plate 3304, and/or the robot arm 506 based on a specific image and its content (eg, object, recognizable feature, etc.) The camera 300 is used for visualization of a surgical site. As discussed above, the processor 4102 and/or the robot arm controller 4106 know the positions of the robot arm 506 and the camera 300. In addition, the image data recorded by the camera is three-dimensional, which provides depth data. Therefore, the processor 4102 and/or the robot arm controller 4106 can determine a position in the robot three-dimensional space on a patient or each visualized point. Therefore, when the robot arm 506 moves the camera 300 in a desired direction from an initial position with an initial image, it is expected to see the desired image change in the image after a second movement.

Alternatively, the processor 4102 and/or the robot arm controller 4106 can be configured to apply an equation representing the desired movement to the initial image data to calculate the expected post-movement image, which generates a calculated second image. The processor 4102 and/or the robot arm controller 4106 use a matching template routine or function to compare the actual image after the movement with the calculated image, as described above. If errors are detected, the processor 4102 and/or the robot arm controller 4106 can correct the errors by moving the robot arm 506 and/or the camera 300 accordingly. For example, given an initial image received from an operator and a "100 pixels to the right" to be moved, the processor 4102 and/or the robot arm controller 4106 can calculate the theoretical image data of the moved image as the direction One of the 100 pixels to the right is shifted. Then, physical movement is performed by executing commands on various coordinated robot joints as disclosed, so as to reposition the robot arm 506 and/or the camera 300 to the theoretically desired position. The camera 300 records a second image, and the processor 4102 and/or the robot arm controller 4106 uses, for example, a matching template function or its equivalent form to compare the second image with the calculated image data. If the movement is accurate, the data will indicate a 100% relationship at a tip of the camera 300, where the two images are perfectly aligned. However, if the actual image data shows the best correlation at another position (for example, 101 pixels to the right and 5 pixels to the top), the processor 4102 and/or the robot arm controller 4106 can modify the movement by The camera 300 is physically moved 1 pixel to the left and 5 pixels to the down via the robot arm 506 to correct errors. 6. Sag compensation embodiment

In some embodiments, at least some of the joints R1 to R9 of the robot arm 506 and/or the coupling plate 3304 may experience some sag. The processor 4102 and/or the robot arm controller 4106 can be configured to provide correction for the sag of the robot arm. In some examples, the processor 4102 and/or the robot arm controller 4106 are configured to perform sag compensation for a series of small movements so that the accuracy of the motion is maintained within one of the ranges of motion of the robot arm 506. For example, in order to characterize and eliminate sag, sag compensation is performed in the direction of movement of a specific robot joint to isolate the error that varies with the actual robot joint rotation position. By comparing the error and torque (calculated by multiplying the load weight of the camera 300 by the length of the moment arm (or link)), the compliance of the joint can be determined. Alternatively, analytical techniques (for example, finite element analysis ("FEA")) can be used to calculate joint compliance.

Using and storing the above compliance representations of all joints in all rotational positions, the processor 4102 and/or the robot arm controller 4106 can calculate the overall sag of a specific camera position. The processor 4102 and/or the robot arm controller 4106 can determine a LUT and/or a sag correction factor of the calibration register for each camera position. In addition, the processor 4102 and/or the robot arm controller 4106 can apply the sag correction factor to the robot arm movement command or a movement sequence (before or after applying the scale factor), so that the sag compensation is incorporated into the movement command/signal middle. The correction factor can be calculated in a continuous motion program, thus achieving accurate tracking and following of the camera 300. This correction factor further eliminates the need for one of the second cameras for the calibration/positioning of the stereo visualization platform 516, and eliminates the need to have a reference target on the camera 300, and therefore eliminates the problem of surgical abdominal cloth interference. 7. Example of Visualized Location /Orientation Storage

In some embodiments, the example processor 4102 is configured to save the visualization parameters used to return to a specific orientation and/or position of the stereo visualization camera 300. The visualization parameters may include a viewing vector, position, magnification, working distance, focus, position, and/or orientation of the stereo visualization camera 300, the robot arm 506, and/or the coupling board 3304.

In one example, a surgeon may wish to have a small wiring and a highly magnified visualization during an anastomosis of a part of a blood vessel under visual illumination. The surgeon can then zoom out to a wider view of the entire blood vessel under infrared irradiation to check patency. The surgeon can then return to the magnified visualization to complete the wiring. In this example, the processor 4102 is configured to save the visualization parameters at each of the locations. The processor 4102 may store a position corresponding to a position that has been continuously viewed for a period of time (such as two seconds, five seconds, thirty seconds, etc.). The processor 4102 may also store a position after receiving an instruction from the surgeon via the input device 1410.

The processor 4102 can display a list of stored locations and/or waypoints. The selection of a stored position causes the processor 4102 and/or the robot arm controller 4106 to move the robot arm and/or the coupling plate 3304 to the previous position and adjust the optical parameters as previously set, including light irradiation and filtering. This configuration allows a surgeon to seamlessly view all stored positions in the sequence without removing his eyes or removing his hands and instruments from the image displayed in one of the programs.

In certain embodiments, the processor 4102 may be configured to enable an operator to form waypoints or positions/orientations before a surgical procedure. The waypoints may be provided in a sequence, which enables the processor 4102 to progress through prescribed waypoints during the procedure after receiving an input of progress from an operator. The processor 4102 can provide a three-dimensional representation of the robot arm 506, the coupling board 3304, and/or the camera 300 via the touch screen input device 1410a so that an operator can actually position the stereo visualization platform 516. This may include providing a magnification, working distance, and/or focus relative to a virtualized patient and/or patient-based alternative modality visualization. The processor 4102 is configured to store the visualization parameters to, for example, the memory 1570 and/or the memory 4120 for each waypoint.

In some embodiments, the processor 4102 is configured to perform specific visualizations unique to specific programs. For example, the image recognition functionality in the processor 4102 is used to automatically align the camera 300 with an object of interest. The processor 4102 compares the image of the surgical site with a previous image or image of the target object to provide identification of a desired object and its position and orientation in a three-dimensional image. The processor 4102 and/or the robot arm controller 4106 are configured to, for example, move the robot arm 506 toward the object and zoom the camera 300 toward the object and set desired image view attributes for specific objects and programs. For example, in ophthalmology, a live retinal image can be compared with a saved image so that, for example, the optic nerve head of the patient's retina can be located based on image recognition. The processor 4102 and/or the robot arm controller 4106 then make the robot arm 506 and/or the coupling plate 3304 and the focus automatically move and/or change the magnification of the camera 300 so that the tip of the camera 300 points to the nerve head for diagnosis . The processor 4102 can then set the camera 300 and/or the monitor 512 to perform image display without the red coloration so that the characteristics of the retina can be more easily distinguished from the surrounding tissues.

In addition to saving and returning to the stored visualization, the example processor can also save the movement path from one view to another. In the anastomosis example discussed above, the processor 4102 and/or the robotic arm controller 4106 can cause the robotic arm 506 and/or the camera 300 to follow a full length of a blood vessel at high magnification to check for aneurysms or other conditions . The processor 4102 can be configured to recognize and follow continuous blood vessels as needed. The processor 4102 can execute a matching template routine on a finite pixel group to actively determine the movement direction of the robot arm 506 and/or the camera 300.

The example processor 4102 can also be programmed and stored in a movement path within a visualization of an object from different viewing angles. For example, the programmable processor 4102 and/or the robot arm controller 4106 can be used to pivot around a point inside the eye to perform an ophthalmic gonioscopy of the eye of a patient. In this example, the robotic arm 506 causes the camera 300 to scan in a generally conical motion so that the patient's eyes can be viewed from a large number of viewing angles. This movement of visualizing the surgical site can be used to select the best angle for self-illumination to eliminate false reflections or to look around obstructions in alternative viewing angles.

In some embodiments, the processor 4102 is configured to reduce occlusion in the calculation of the depth map. Occlusion is inherent in the calculation of the depth map due to the parallax between the two views of a stereoscopic image, where a first view sees a part of a part that is different from the other views. Therefore, each view does not see a certain part of the other views. By moving the robot arm 506 among various positions and recalculating the depth map while using the knowledge of the three-dimensional position of the image pixel, the occlusion is reduced. The map can be made more accurate by repeatedly calculating the depth map after performing the known motion steps, calculating the expected map change, determining the error by the difference, and constructing an average map. E. Auxiliary drive embodiment

In some embodiments, the processor 4102 and/or the robot arm controller 4106 are configured to execute one or more algorithms, routines, etc. defined by instructions stored in the memory 1570 and/or 4120 to make The robot arm 506 and/or the coupling plate 3304 can provide powered joint movement based on a detected force applied by an operator to move the stereo visualization camera 300. In these embodiments, the auxiliary driving feature enables the robotic arm 506 to operate as an extension of a surgeon by moving the stereo visualization camera 300 to a desired position and/or orientation. As explained below, the processor 4102 and/or the robot arm controller 4106 are configured to monitor the force/torque/movement applied by an operator and the position of the arm joints to infer the intention of an operator and thus enable the robot The arm 506 and/or the coupling plate 3304 move.

FIG. 62 shows a diagram illustrating an algorithm, routine, or program 6200 for providing auxiliary driving of the stereo visualization camera 300 according to an exemplary embodiment of the present invention. Although the procedure 6200 is described with reference to the flowchart illustrated in FIG. 62, it should be understood that many other methods of performing the steps associated with the procedure 6200 can be used. For example, the order of the blocks in the block can be changed, a specific block can be combined with other blocks, and many of the blocks described are optional. In addition, the actions described in the procedure 6200 can be executed in multiple devices including, for example, the information processor module 1408 of the exemplary stereo visualization camera 300 of FIG. 14 and/or FIG. 41 The joints R1 to R9 and the robot arm controller 4106. In some instances, the program 6200 can be executed by a program stored in the memory 4120 of the robot arm controller 4106. The example procedure 6200 may be executed periodically as force is applied to the camera 300. For example, the procedure 6200 may sample the force/torque data every update cycle (which may be 1 ("ms"), 5 ms, 8 ms, 20 ms, 50 ms, etc.).

In the illustrated embodiment, the processor 4102 and/or the robot arm controller 4106 receives from the sensor 3306 force/torque output data 6201 related to the force applied to the camera 300 by an operator. The processor 4102 and/or the robot arm controller 4106 are configured to filter the received output data 6201 (block 6202). The output data may include a force and/or torque vector. The filtering may include applying a first low-pass filter, a second low-pass filter, and/or a notch filter that targets vehicle vibration. In other examples, the processor 4102 and/or the robot arm controller 4106 may use a single low-pass filter and a notch filter.

The example processor 4102 and/or the robot arm controller 4106 also receive joint position data 6203 from one or more joint sensors in the robot arm 506 and/or the coupling plate 3304. The processor 4102 and/or the robot arm controller 4106 use the joint position data 6203 to provide compensation for the filtered force/torque output data (block 6204). The compensation may include gravity compensation and/or force point compensation. For gravity compensation, the effect of earth's gravity is removed from the filtered data. For force point compensation, the processor 4102 and/or the robot arm controller 4106 provide compensation for the filtered data (and/or gravity compensated data) based on a point in which the force is applied to the camera 300 (for example, the control arm 304) . As discussed above in conjunction with FIG. 35, the sensor 3306 is located at an offset distance from the control arm 304 at an angle. The offset distance and angle cause the force applied at the control arm 304 to be slightly shifted in direction and angle when detected in the sensor 3306. The force compensation adjusts the force value as if the force is directly applied to the sensor 3306 instead of the control arm 304. The force compensation can be predetermined based on a known angle and/or distance between the sensor 3306 and the control arm 304. In summary, gravity compensation and force point compensation modify the filtered force/torque data to form a force/torque vector proportional to the force/torque provided by an operator at the control arm 304 of the camera.

The example processor 4102 and/or the robot arm controller 4106 also uses the joint position data 6203 along with the compensated filtered force/torque output data to perform a relationship between the force/torque coordinate system and a global coordinate system or robot space. Coordinate transformation (block 6206). The transformation may include one or more predefined equations or relationships based on the known robot space and the orientation of the sensor 3306. The example processor 4102 and/or the robot arm controller 4106 also use the joint position data 6203 to perform a coordinate transformation between a camera coordinate system of the stereo visualization camera 300 and the global coordinate system or robot space (block 6208). The coordinate transformation of the camera coordinate system can be based on the optical calibration parameters mapped to the robot space of the robot arm 506, as described above.

After performing the coordinate transformations, the example processor 4102 and/or the robot arm controller 4106 are configured to use at least one sigmoid function to convert the force/torque vector into one or more translation/rotation vectors (block 6210 ). The formation of this (etc.) translation/rotation vector generates an inference in one of the predetermined directions of the operator. The translation and rotation information is used to determine how to rotate the joint of the robot arm 506 to reflect, match, and/or approximate the predetermined movement of the operator.

In some instances, the example processor 4102 and/or the robot arm controller 4106 are configured to apply robot speed scaling to the translation/rotation vector (block 6212). The speed scaling may be based on the operating conditions of the robot arm 506, for example. For example, once a surgical procedure has been started to prevent the arm from accidentally hitting the operating room staff, instruments, and/or patient at a relatively high speed ratio, for example, speed scaling can be applied. When a program has not yet started, the example processor 4102 and/or the robotic arm controller 4106 can apply less speed scaling for calibration or setting of the robotic arm 506 when there is no patient.

The example processor 4102 and/or the robot arm controller 4106 determine the possible movement sequence of the joints of the robot arm 506 based on the scaled translation/rotation vectors. When evaluating possible sequences, the processor 4102 and/or the robot arm controller 4106 identify joint singularities for avoidance, thereby excluding the corresponding movement operation of the robot arm 506 (block 6214). As discussed above, singularities can include toggle locks or other locations that can be prone to hysteresis and backlash. The processor 4102 and/or the robot arm controller 4106 are configured to select a movement sequence after removing the moving singularities using (for example) sub-comparable kinematics (e.g., an inversion of a sub-comparable matrix) (block 6216). ). The sub-comparable kinematics equations define how the robot arm 506 and/or the specific joint of the coupling plate 3304 will be moved based on the scaled translation/rotation vector. Yacobi kinematics provides speed control, while inverse kinematics discussed below provides position control. In some embodiments, the processor 4102 and/or the robot arm controller 4106 may alternatively use inverse kinematics or other robot arm control routines. The processor 4102 and/or the robot arm controller 4106 determine how the specified joints of the robot arm and/or the coupling plate 3304 will move in a coordinated manner and specify (for example) joint rotation speed, joint rotation direction and/or joint rotation One of the duration of the movement sequence. The movement sequence may also specify a sequence in which the joints of the robot arm 506 and/or the coupling plate 3304 will be rotated. The robot arm and/or any one of the joints R1 to R9 of the coupling plate 3304 may individually rotate or have overlapping movements depending on the movement sequence.

After determining a movement sequence, the processor 4102 and/or the robot arm controller 4106 are configured to use joint speed scaling and/or boundaries to perform collision avoidance. For example, the processor 4102 and/or the robot arm controller 4106 are configured to determine whether the movement sequence will cause one or more joints and/or links of the robot arm 506 and/or the coupling plate 3304 to approach a boundary or other A defined Cartesian limit, such as the space around a patient or instrument. As discussed above in connection with FIG. 49, the processor 4102 and/or the robot arm controller 4106 may compare the estimates of the positions of the links and/or joints in the robot space based on the movement sequence with one or more defined boundaries and/or Angle limit. Based on a distance from a boundary, the processor 4102 and/or the robot arm controller 4106 applies one or more joint speed limits via a proportional value (block 6218). The processor 4102 and/or the robot arm controller 4106 may also be used to prevent (for example) the linkages of the robot arm 506 from hitting each other and/or prevent the robot arm 506, the coupling plate 3304, and/or the camera 300 from extending across a boundary. Or multiple joint position restrictions (block 6220). The position just before the position limit (for example, 1 cm ("cm"), 2 cm, 10 cm, etc.) in front of the position limit and/or the position in the position limit can correspond to the position in the Cartesian robot space, where One of the scale factors is "0".

In some instances, the processor 4102 and/or the robot arm controller 4106 can perform sub-comparable kinematics with the boundary provided as one of the inputs to the equation, where a motion is provided for movement through a region close to a boundary. Higher cost factor. When determining a movement sequence, the boundary cost factor is used to cause the processor 4102 and/or the robot arm controller 4106 to avoid locations close to the boundary (if possible). The cost factor may include a reduction in inverse proportion to one of the scale factors associated with a specific location in the robot space. The scale factor can be applied to each joint/link, or there can be a separate scale factor for each joint at the same position in the robot space.

After providing collision avoidance, the example processor 4102 and/or the robotic arm controller 4106 are configured to provide corrections for the relatively fast reversal of the robotic arm 506 (block 6222). The processor 4102 and/or the robot arm controller 4106 may implement a zero phase delay algorithm to reject direction pulses that quickly cause one or more joints to change the direction of rotation. The zero phase delay algorithm can be implemented by a filter that prevents (for example) the robot arm from suddenly moving or swinging if the operator reverses the direction too quickly.

As illustrated in Figure 62, the example processor 4102 and/or the robotic arm controller 4106 are configured to validate the order of movement (block 6224). The processor 4102 and/or the robot arm controller 4106 may verify a command to ensure that a command (or a signal indicating a command) is within the operating parameters (eg, duration, rotation speed, etc.) of a joint motor. The processor 4102 and/or the robot arm controller 4106 can also verify a command by comparing the command with the current threshold to ensure that the robot arm 506 will not draw too much current during any stage of the movement sequence.

The example processor 4102 and/or the robot arm controller 4106 may also apply one or more anti-noise filters to the movement command or the signal indicating the movement command (block 6226). The filter may include a high-frequency low-pass filter that removes high-frequency noise components (which can induce transient signals in an articulated motor). After any filtering, the processor 4102 and/or the robot arm controller 4106 transmits one or more commands to the robot arm 506 and/or the appropriate joint motor of the coupling board 3304 via one or more signals or messages according to the movement sequence (block 6228). The transmitted commands cause the motors at the respective joints to move the robot arm 506 and/or the coupling plate 3304, thereby causing the camera 300 to move as expected by the operator. The example procedure 6200 can be repeated as long as an operator applies force to the camera 300.

FIG. 63 shows a diagram of an example program 6300 for using an input device 1410 to move the example visualization camera 300 according to an example embodiment of the present invention. The example procedure 6300 is almost identical to the procedure 6200 of FIG. 62, except that the blocks 6202 to 6206 related to the sensor 3306 are removed. In the illustrated example, a control input 6301 is received from an input device 1410 (such as a button on the control arm 304, a foot pedal, a joystick, a touch screen interface, etc.). The control input 6301 instructs the directional movement of the camera in the Cartesian robot space of the robot arm 506.

As illustrated in Figure 63, the control input 6301 is combined with joint position data 6203 from one or more joint sensors in the robot arm 506 and/or the coupling plate 3304 for execution from a camera coordinate system to a global coordinate system System and/or one coordinate transformation in robot space (block 6208). The example procedure 6300 then continues in the same manner as discussed for the procedure 6200. The processor 4102 and/or the robot arm controller 4106 therefore cause the robot arm 506, the coupling plate 3304 and/or the camera 300 to move to a desired position and/or orientation based on the control input 6301 received from the input device 1410. F. Lock to the target embodiment

In some embodiments, the processor 4102 and/or the robot arm controller 4106 are configured to execute one or more algorithms, routines, etc. defined by instructions stored in the memory 1570 and/or 4120 to make The robot arm 506 and/or the coupling plate 3304 can provide a lock-to-target feature. In these embodiments, the lock-to-target feature enables the robotic arm 506 to operate as an extension of a surgeon by enabling the stereo visualization camera 300 to be redirected and locked to a target surgical site. As explained below, the processor 4102 and/or the robot arm controller 4106 are configured to monitor the force/torque/movement applied by an operator and the position of the arm joints to infer the intention of an operator and thus redirect The robot arm 506 and/or the coupling board 3304 keep the focus of the camera 300 locked or fixed.

Locking to the target feature allows the camera 300 to be redirected by causing all motion to be constrained to the surface of a virtual sphere. The tip of the camera 300 is located at an outer surface of the virtual sphere (for example, a top hemisphere of the virtual sphere) and a focal point of the camera 300 or the target surgical site constitutes a center of the virtual sphere. The example processor 4102 and/or robotic arm controller 4106 enables an operator to move the camera 300 over an outer surface of the virtual sphere while pointing the camera 300 at the center of the sphere, thereby keeping the target surgical site in focus during the movement . Locking to the target feature allows an operator to easily and quickly obtain significantly different views of the same target site.

FIG. 64 shows a diagram illustrating an algorithm, routine, or program 6400 for providing one of the stereoscopic visualization cameras 300 to be locked to the target according to an exemplary embodiment of the present invention. Although the procedure 6400 is described with reference to the flowchart illustrated in FIG. 64, it should be understood that many other methods of performing the steps associated with the procedure 6400 can be used. For example, the order of the blocks in the block can be changed, a specific block can be combined with other blocks, and many of the blocks described are optional. In addition, the actions described in the procedure 6400 can be executed in a plurality of devices including, for example, the information processor module 1408 of the exemplary stereoscopic visualization camera 300 in FIG. 14 and/or the information processor module 1408 in FIG. 41 Joints R1 to R9 and robot arm controller 4106. In some instances, the program 6400 can be executed by a program stored in the memory 4120 of the robot arm controller 4106.

The example program 6400 is similar to the auxiliary driver 6200. However, the program 6400 is ready to command the joint position to maintain one of the focus of the camera 300, and the example program 6200 is ready to calculate the joint speed. The example program 6400 determines a required force/movement vector input by an operator and calculates a rotation transformation so that the focus of the camera 300 remains fixed while the robot arm 506 and/or one or more joints of the coupling plate 3304 are moved for weight Directional camera 300. The reorientation of the camera 300 enables a target surgical site to be imaged from different angles. Redirection may be required when (for example) an instrument blocks a first viewing path and the surgeon wishes to maintain the current focus.

The example procedure 6400 starts when the operator selects the lock to target button on the input device 1410 (which causes a command message or signal to be transmitted to the processor 4102 and/or the robot arm controller 4106). After receiving the message, the processor 4102 and/or the robot arm controller 4106 maintains a fixed working distance and/or focus therein while enabling an operator to change the orientation of the camera 300 (this causes the robot arm and/or the coupling plate 3304 One or more joints provide auxiliary movement) One of the locks to the target mode for operation. When an instruction is received, the example processor 4102 and/or the robot arm controller 4106 can record the current working distance, magnification, focus, and/or other optical parameters of the camera 300. The processor 4102 and/or the robot arm controller 4106 can also record a current image of the FOV.

After the program 6400 is started, the processor 4102 and/or the robot arm controller 4106 receives from the sensor 3306 force/torque output data 6201 related to the force applied to the camera 300 by an operator. As discussed in connection with FIG. 62, the processor 4102 and/or the robot arm controller 4106 filters the data 6102 and provides gravity/force compensation to the data 6102 (blocks 6202 and 6204). Also similar to Fig. 62, the processor 4102 and/or the robot arm controller 4106 together with the compensated filtered force/torque output data uses the joint position data 6203 to execute the force/torque coordinate system and a global coordinate system or robot space Transformation between one icon (block 6206). The example processor 4102 and/or the robot arm controller 4106 also use the joint position data 6203 to perform a coordinate transformation between a camera coordinate system of the stereo visualization camera 300 and the global coordinate system or robot space (block 6208). The example processor 4102 and/or the robot arm controller 4106 also perform a transformation from the global coordinate system or robot space to a spherical coordinate corresponding to a virtual sphere (block 6410).

After the coordinate transformation, the example processor 4102 and/or the robot arm controller 4106 are configured to scale the trajectory speed based on, for example, one of the operating modes of the camera 300 (block 6412). The scaling may be similar to the scaling performed at block 6212 in FIG. 62. The example program 6400 of FIG. 64 continues by the processor 4102 and/or the robot arm controller 4106 calculating the end points of a sphere (block 6414). The calculation of the end of the sphere provides an inference about one of the operator's desired moving directions and determines how far the camera 300 will move over the virtual sphere without rotating the sphere.

FIG. 65 shows a diagram illustrating a virtual sphere 6500 for locking to a target feature according to an exemplary embodiment of the present invention. As shown in FIG. 65, the stereoscopic visualization camera 300 is virtually placed on the sphere 6500 based on a current position as determined from the joint position data 6203. One of the viewing vectors of the camera 300 points to a tip (designated as an xyz target) located in one of the centers of the sphere 6500. The processor 4102 and/or the robot arm controller 4106 are configured to use the transformed force/torque data to determine how the camera 300 on the sphere will move along a surface of the sphere 6500 while keeping the viewing vector pointing to the xyz target, where Any given point on the sphere is given by an equation of one of the functions of the rotating spherical angle " v " and "u". When using force/torque data, the processor 4102 and/or the robot arm controller 4106 use one of the "x" and "y" components corresponding to the translational force to directly determine how the camera 300 will move on the virtual sphere 6500 to determine The end of the sphere.

The processor 4102 and/or the robot arm controller 4106 can determine the end of the sphere in different ways for different inputs. For example, if an input is received via the input device 1410, as shown in FIG. 63, the processor 4102 and/or the robot arm controller 4106 will "up", "down", "left" and "right" ( It is provided as x vector, y vector) from the camera coordinates to the robot space coordinates. Similar to the force/torque data, the processor 4102 and/or the robot arm controller 4106 use the x vector and the y vector to directly determine how the camera 300 will move on the virtual sphere 6500 to determine the end of the sphere. It should be understood that in the example in which input is received via the input device, the operations discussed in connection with blocks 6202 to 6206 may be omitted, as shown in FIG. 63.

In some instances, the processor 4102 and/or the robot arm controller 4106 are configured to receive track input data. In these examples, the processor 4102 and/or the robot arm controller 4106 keeps the spherical angle " v " constant while causing the movement to be repeated along the spherical angle "u " of the virtual sphere 6500. The repeated movement along the spherical angle " u " enables the end of the sphere to be judged for the orbital input. It should be understood that although the input is applied to the virtual sphere 6500, in other examples, the input may be applied to other shapes. For example, the virtual sphere 6500 may alternatively be defined as a virtual cylinder, an ellipsoid, an egg shape, a pyramid/frustum, etc.

In other examples, the processor 4102 and/or the robot arm controller 4106 are configured to receive level category input data. In these examples, the processor 4102 and/or the robot arm controller 4106 keeps the spherical angle " u " constant while causing the movement to be repeated along the spherical angle "v " of the virtual sphere 6500. The repeated movement along the spherical angle " v " causes the camera 300 to move to the top of one of the virtual spheres 6500.

Returning to Figure 64, after determining the end of the sphere, the processor 4102 and/or the robot arm controller 4106 are configured to calculate that the camera 300 remains locked after the camera 300 has been moved along the virtual sphere to the determined end point. The amount of rotation required for the x, y, and z target (block 6416). The processor 4102 and/or the robot arm controller 4106 may also provide anti-roll correction during this calculation (block 6418). In other words, the processor 4102 and/or the robot arm controller 4106 are configured to determine how the camera 300 will be oriented (given its new position on the virtual sphere 6500) so that the viewing vector or tip of the camera 300 is provided to point to the virtual sphere 6500 A center (which corresponds to a target surgical site or focus) at the same x, y, z target.

During this step, the processor 4102 and/or the robot arm controller 4106 determine the joint angles of the robot arm 506 and/or the coupling plate 3304 required to achieve the desired orientation. After calculating the x, y, z sphere endpoints in block 6414, the processor 4102 and/or the robot arm controller 4106 determine the roll and pitch of the camera 300. In some embodiments, the calculation is a two-step process. First, the processor 4102 and/or the robot arm controller 4106 calculates an initial 4×4 transformation matrix T that provides the movement of the camera 300 (given x, y, z sphere endpoints) without rotation. Then, the processor 4102 and/or the robot arm controller 4106 calculates the amount of roll and pitch of the area so that the camera 300 remains locked at x, y, z (and/or at the end of the x, y, z sphere) One of the targets is used for subsequent joint rotation cycles. The processor 4102 and/or the robot arm controller 4106 may use equations (4) and (5) below to calculate the amount of roll and pitch, where T _next corresponds to a 4×4 transformation matrix. The calculation can be performed every update cycle (for example, 8 ms).

In the above equation (4), x _{taregt_next} , y _{taregt_next} and z _{taregt_next} are constraints on the T _next matrix. The above constraints stipulate that the roll angle and pitch angle are selected to make the above equations of x, y, z valid. In other words, the x, y, z position of a target in the next update cycle of one of the joint rotations must be equal to the x, y, z position of the target in the current cycle. These constraints enable the camera 300 to rotate through the roll angle and pitch angle but remain locked relative to the x, y, and z positions.

In addition, -sin θ on the bottom column of the first matrix of equation (5) corresponds to a pitch angle, and sin θ on the bottom column of the second matrix corresponds to a roll angle. Given the function cos(roll), there may be a closed expression of the pitch. The processor 4102 and/or the robot arm controller 4106 can use an iterative method to estimate the roll calculated as a function cos(roll), where the trim is equal to fn(cos(roll)) to generate a correct roll for the above equation /The vertical tilt solution is right.

After calculating the amount of roll and pitch according to the operations described in conjunction with blocks 6416 and 6418, the example processor 4102 and/or the robot arm controller 4106 are configured to provide singularity avoidance and calculate inverse kinematics to determine joint rotation Thus, in addition to the new x, y, and z positions of the camera 300 along the virtual sphere 6500, the amount of roll and pitch is also achieved (blocks 6214 and 6420). The calculation of inverse kinematics enables the processor 4102 and/or the robot arm controller 4106 to determine the movement sequence of one of the joints of the robot arm 506 and/or the coupling plate 3304.

In addition to joint speed limits and/or position limits, the example processor 4102 and/or robot arm controller 4106 may apply error correction for the movement sequence (blocks 6418, 6218, 6220). As discussed above in connection with FIG. 62, these limitations and error corrections can prevent the robot arm 506, camera 300, and/or coupling plate 3304 from hitting itself, exceeding one or more boundaries, and/or being within acceptable joint positions. The processor 4102 and/or the robot arm controller 4106 can also verify that the commands for the joints of the movement sequence provide anti-noise filtering, and then send the commands (or signals indicating the commands) to the robot arm 506 based on the movement sequence And/or one or more joints R1 to R9 of the coupling plate 3304 (blocks 6224, 6226, 6228). If no other movement is detected, the example procedure 6400 can then end. Otherwise, the procedure 6400 is repeated at periodic intervals (for example, 10 ms, 20 ms, etc.) when the operator input is received.

In some embodiments, the processor 4102 and/or the robot arm controller 4106 can provide the instrument with lock-to-target tracking. In these examples, the xyz target at a center of the virtual sphere 6500 is replaced with a dynamic trajectory corresponding to a moving target. For example, this feature can be used to track one of the spine tools. In these embodiments, an instrument may include one or more fiducials and/or other markings. The exemplary stereo visualization camera 300 records images that include these references. The processor 4102 and/or the robot arm controller 4106 can perform a coordinate transformation from the camera coordinate system space to the robot space to determine how to move the instrument along the x-axis, y-axis, and z-axis. The example processor 4102 and/or the robot arm controller 4106 tracks how the fiducial moves in the image and determines the corresponding x, y, z movement vectors. In some cases, the x, y, and z vectors can be input to the calculation of the sphere endpoints in block 6414 of FIG. 64 to change the position of one of the centers of the virtual sphere 6500. In response to one of the spheres 6500 moving, the processor 4102 and/or the robot arm controller 4106 determines how the robot arm 506 will be positioned to maintain the same working distance and/or orientation as the new target position. The processor 4102 and/or the robot arm controller 4106 may then apply inverse kinematics to determine the joint rotation of the robot arm 506 and/or the coupling plate to track the movement of the target. Similar to procedures 6200 and 6400, the processor 4102 and/or the robot arm controller 4106 may apply error correction, joint restriction, filters, and/or verification before sending commands to the joints as specified in a determined movement sequence. Summarize

It will be appreciated that one or more computer programs or components can be used to implement each of the systems, structures, methods, and procedures described herein. These programs and components can be provided as a series of computer instructions on any conventional computer-readable media (including random access memory ("RAM"), read-only memory ("ROM"), flash memory, magnetic disks, or CD, optical memory or other storage media and their combinations and derivatives). The instructions can be configured to be executed by a processor that, when executing the series of computer instructions, executes or promotes all or part of the performance of the disclosed methods and procedures.

It should be understood that those skilled in the art will be aware of various changes and modifications to the exemplary embodiments described herein. Such changes and modifications can be made without departing from the spirit and scope of the subject matter of the present invention and without diminishing its established advantages. Therefore, these changes and modifications are intended to be covered by the scope of the attached patent application. In addition, consistent with current U.S. law, it should be understood that it is not intended to call 35 U.S.C. 112(f) or the former AIA 35 U.S.C. 112 (paragraph 6) unless the term "component" or "step" is clearly stated in the scope of the patent application. Therefore, the scope of patent application is not intended to be limited to the corresponding structures, materials or actions described in the specification or its equivalent content.

100. Lever arm/arm 204‧‧‧Boom arm/arm 206‧‧‧Eyepiece/First eyepiece 208‧‧‧Eyepiece/Second eyepiece/Auxiliary device 210‧‧‧Beam splitter 212‧‧‧Single image video camera/ Video camera/auxiliary device/camera 214‧‧‧Beam splitter 220‧‧‧XY horizontal swing device 300‧‧‧Stereoscopic visual camera/camera 302‧‧‧Shell 304a‧‧‧Control arm 304b‧‧‧Control arm 305a‧‧‧Control 305b‧‧‧Control 306b‧‧‧Rotating pillar 402‧‧‧Mounting bracket 404‧‧‧Power port 406‧‧‧Data port 408‧‧‧Front working distance Main objective lens/lens/front Working distance lens/Left front working distance lens/Right front working distance lens/Assembly Robot arm 508‧‧‧Assistant 510‧‧‧Truck 512‧‧‧Display monitor/monitor/display 514‧‧‧Display monitor/monitor/display 516 Integrated platform/platform/robot platform 520‧‧‧computer 700‧‧‧target part/target scene 702‧‧‧objective lens assembly/main objective lens assembly/component 704‧‧‧rear working distance lens/lens/left rear working Distance lens/Right rear working distance lens 706‧‧‧Working distance 708A‧‧‧Visible light source/light source 708B‧‧‧Near infrared light source/Light source 708C‧‧ Near ultra-ultraviolet light source/Light source 710a‧‧‧Visible path 710b‧‧ ‧Near infrared path 710c‧‧‧Near ultra-ultraviolet path 712‧‧‧Deflection element 714‧‧‧Front lens group/downstream front lens group/Optics 716‧‧‧Zoom lens assembly/Optics 718‧‧‧Lens mirror Tube group/Optics 720‧‧‧Right front lens/lens/front lens 722‧‧‧Left front lens/lens/front lens 724‧‧‧Front zoom lens group/front zoom group/group/carrier/zoom lens group 726‧‧ ‧Front right zoom lens/front zoom lens/lens/zoom lens 728‧‧‧Left front zoom lens/front zoom lens/lens/zoom lens 730 ‧‧‧Right rear zoom lens/rear zoom lens/zoom lens 734‧‧‧Left rear zoom lens/rear zoom lens/zoom lens 736‧‧‧Right lens barrel/lens/lens barrel 738‧‧‧Left lens lens Tube/lens/lens tube 740‧‧‧filter/optical filter/first filter/filter wheel/optical filter 742‧‧‧last optical group/most Rear optical element group 744‧‧‧Optical image sensor/Optical sensor/Optical light sensor/Image sensor 744L‧‧‧Sensor 744R‧‧‧Sensor 745‧‧‧Right last optics Element/optical element/lens/final optical element 746‧‧‧right optical image sensor/right optical sensor/sensor/optical image sensor/optical sensor element/image sensor/right image sensor 747‧‧‧left last optical element/optical element/lens/last optical element 748‧‧‧left optical image sensor/left optical sensor/sensor/optical image sensor/optical sensor element /Image sensor/Left image sensor 750‧‧‧Flexible part/Assembly 752‧‧‧Flexible part 902‧‧‧Surface 904‧‧‧Surface/Surface 906‧‧‧Path 908‧‧‧Path 910 ‧‧‧Path 912‧‧‧Path 914‧‧‧Fluorescent parts 916 1004‧‧‧Second two-dimensional pixel grid/grid/pixel grid/left pixel grid 1006‧‧‧pixel group/right pixel group 1008‧‧‧pixel group 1102‧‧‧support structure 1104‧‧‧rail Bar holder 1106‧‧‧rail/track 1108‧‧‧actuating section 1202‧‧‧center structure 1300‧‧‧double flexure/flexure/component 1301‧‧‧support beam 1302‧‧‧beam 1303. ‧Image capture module/module 1406‧‧‧Motor and lighting module 1408‧‧‧Information processor module/Information processing module/Information processor 1410‧‧‧User input device/User unit device/ Foot switch input device/input device/tracking input device 1410a‧‧‧Input device/touch screen input device 1410b‧‧‧Foot pedal input device 1502‧‧‧Image sensor controller 1504‧‧‧Processor 1506 ‧‧‧Memory 1508‧‧‧Communication interface 1510‧‧‧Program 1512‧‧‧Image sensor controller bus 1514‧‧‧Communication channel 1520 ‧‧‧Memory 1526‧‧‧Communication interface 1528‧‧‧Communication bus 1530‧‧‧Program/calibration program 1532‧‧‧Communication interface 1534 Near-infrared light driver/driver/optical driver 1538‧‧‧visible light driver/driver/light driver 1540‧‧‧filter motor driver/drive 1542‧‧‧filter motor/motor 1544‧‧‧rear zoom lens motor driver/drive 1546‧‧‧Rear zoom lens motor/motor 1548‧‧‧Front zoom lens motor driver/drive 1550 Lens motor/motor 1560‧‧‧calibration program/program/template matching program 1562‧‧‧processor 1564‧‧‧graphic processing unit 1566‧‧‧communication channel 1568‧‧‧communication channel 1570‧‧‧memory 1572‧‧ ‧Network Interface 1574‧‧‧Peripheral Input Unit Interface/Interface 1576‧‧‧Peripheral Device/Device/Peripheral Unit 1580‧‧‧Program 1580a‧‧‧DeBayer Program 1580b‧‧‧Program/Color Calibration Program/Sensor Color calibration program 1580c‧‧‧program/color calibration program/user color calibration program 1580d‧‧‧program/color calibration program/display color calibration program 1580e‧‧‧reproducer program/program 1602‧‧‧decompression routine 1702 Screen 1712 ‧‧Second crosshair/crosshair 1804‧‧‧Crescent object/object 1806‧‧‧Crescent object/object 1808‧‧‧Crescent object/object 1810‧‧‧Zoom repeat point 1850‧‧‧ Left field of view / field of view / high magnification field of view / high magnification of left field of view Silk 1910‧‧‧Zoom repeat point 1950‧‧‧Right field of view/High magnification field of view 2000‧‧‧Grid 2100‧‧‧Pixel grid 2102‧‧‧Left zoom repeat point/Zoom repeat point 2104‧‧‧ Right zoom repeat point/Zoom repeat point 2106‧‧‧origin 2108‧‧‧object 2110‧‧‧contour object/object 2202‧‧‧best resolution section/best resolution range 2302‧‧‧focus plane/ Focal plane 2304‧‧‧Object of interest/object 2306‧‧‧View/Left view 2308‧‧‧View/Right view 2310‧‧‧Front center 2312‧‧‧Theoretical right projection/Right projection 2314‧‧‧Theoretical left projection /Extrapolation and Theoretical Left Projection 2316‧‧‧Actual Left Projection/Extrapolation and Actual Left Projection 2320‧‧‧Intersection Point 2402‧‧‧Eyes 2500‧‧‧Program 2502‧‧‧Block 2504‧‧‧Block 2506‧‧ ‧Block 2508‧‧‧Block 2509‧‧‧Image/Frame 2510‧‧‧Block 2512‧‧‧Block 2514‧‧‧Block 2516‧‧‧Block 2518 /Step 2521‧‧‧Video streaming 2522‧‧ ‧Block/Step 2524‧‧‧Block/Step 2526‧‧‧Block/Step 2528‧‧‧Block/Step 2530 Point 2706‧‧‧Left zoom repeat point 2708‧‧‧Pixel group 2802‧‧‧Target template/template/target/basic template 2804‧‧‧Origin 2806‧‧‧Zoom repeat point/Left zoom repeat point 3000‧‧‧ Digital template image 3302. Sensor 3410‧‧‧Axis 3412‧‧‧Axis 3414‧‧‧Axis 3416‧‧‧Axis 3418‧‧‧Axis 3420‧‧‧Axis 3430‧‧‧Connecting rod 3432‧‧‧Connecting rod 3434‧‧‧Connecting Rod 3436‧‧‧Connecting rod 3438‧‧‧Connecting rod 3440 ‧‧‧Axis 3714‧‧‧Axis 3718‧‧‧Connecting rod 3720‧‧‧Optical axis 4102 Arm Controller/Controller 4120‧‧‧Memory 4122‧‧‧Robot Processor 4124‧‧‧Motor Controller 4130‧‧‧Coupler Controller 4140‧‧‧Input Power Module/Module 4142‧‧‧Emergency Stop switch/switch 4200‧‧‧Program/calibration program 4202‧‧‧Block 4203‧‧‧Look-up table 4204‧‧‧Block 4206‧‧‧Block 4208‧‧‧Block 4210‧‧‧Block 4212‧‧‧Block 4302‧ ‧‧Housing 4304‧‧‧Overall front focal length/focal length/front focal length 4306‧‧‧Front focal plane/focal plane 4306'‧‧‧New front focal plane 4308‧‧‧Main plane 4310 ‧‧New working distance 4312‧‧‧Pupillary distance 4320‧‧‧Left optical axis 4320'‧‧‧Left optical axis 4322‧‧‧Right optical axis 4322'‧‧‧Right optical axis 4324‧‧‧Optical axis 4330‧ ‧‧Image/Point/Focal Plane 4330'‧‧‧Repositioned Tip 4400‧‧‧Chart 4402‧‧‧Optical Axis 4404‧‧‧Actual Focal Length/Focal Length 4406‧‧‧Image Height 4408‧‧‧Height 4410‧‧ ‧Projection Center 4502‧‧‧Optical Axis 4504‧‧‧Mechanical Axis 4506‧‧‧Pupillary Distance 4508‧‧‧Object 4508'‧‧Object/Item 4510‧‧‧Focal Length 4512‧‧‧Focal Plane 4514‧‧ ‧Item 4520‧‧‧Left view 4522‧‧ ‧Right view 4602‧‧‧Angle 4604‧‧‧Focal plane 4610‧‧‧Left view 4612‧‧‧Right view 4700‧‧‧Imaginary left image sensor position and right image sensor position 4702‧‧‧Viewing vector 4800‧‧‧Program 4802‧‧‧Block 4804‧‧‧Block 4806‧‧‧Block 4807 ‧Orientation 5102‧‧‧Chart 5104‧‧‧Chart 5110 ‧Block 5205‧‧‧Pose data 5206‧‧‧Block 5207‧‧‧Magnification and working distance Optical calibration parameters 5208‧‧‧Block 5210‧‧‧Block 5212‧‧‧Block 5216‧‧‧Block 5300‧‧‧ Patient 5302‧‧‧Glioblastoma 5302'‧‧‧Part 5304‧‧‧Brain 5306‧‧‧Cranial 5400‧‧‧Craniotomy procedure/procedure 5402‧‧‧Surgical instrument/instrument 5402'‧‧‧ Visible part 5404‧‧‧Surgery accessible site/conical surgical site/site surgical site accessible/conical surgical site accessible/site 5500‧‧‧visualized axis/axis 6200‧‧‧algorithm/routine/ Program/auxiliary driver 6201‧‧‧Force/torque output data/Received output data 6202‧‧‧Block 6203‧‧‧Joint position data 6204‧‧‧Block 6206‧‧‧Block 6208‧‧‧Block 6210‧‧ ‧Block 6212‧‧‧Block 6214‧‧‧Block 6216‧‧‧Block 6218‧‧‧Block 6220 ‧Program 6301‧‧‧Control input 6400‧‧‧Program 6410‧‧‧Block 6412‧‧‧Block 6414‧‧‧Block 6416‧‧‧Block 6418‧‧‧Block 6420‧‧‧Block 6500‧‧‧Virtual sphere Sphere α‧‧‧angle AA‧‧‧plane d‧‧‧tail distance/distance/displacement distance/displacement e‧‧‧error distance F‧‧‧force L1‧‧‧position L2‧‧‧position M _x ‧‧‧ Moment M _y ‧‧‧Moment O‧‧‧Pinhole P‧‧‧Distance P _L ‧‧‧Parallax P _R ‧‧‧Distance ΔP‧‧‧Distance R1‧‧‧Position R2‧‧‧Position R1 Coordinate system/joint R2‧‧‧coordinate system/joint R3‧‧‧coordinate system/joint R4‧‧‧coordinate system/joint/wrist joint R5 Wrist joint R7‧‧‧Coordinate system/joint R8 ‧‧‧Coordinate system/joint R9‧‧‧Coordinate system/joint R10 ‧‧‧Distance ΔY‧‧‧Distance Z‧‧‧Focal length

Figure 1 shows a diagram of a pair of prior art surgical loupes.

Figure 2 shows a diagram of a prior art surgical operating microscope.

3 and 4 show a perspective view of a stereo vision camera according to an exemplary embodiment of the present invention.

FIGS. 5 and 6 show diagrams of a microsurgery environment including the stereo visualization camera of FIGS. 3 and 4 according to an exemplary embodiment of the present invention.

FIGS. 7 and 8 show diagrams illustrating optical elements in the exemplary stereo visualization camera of FIGS. 3 to 6 according to an exemplary embodiment of the present invention.

FIG. 9 shows a diagram of a deflection element of the exemplary stereo visualization camera of FIGS. 7 and 8 according to an exemplary embodiment of the present invention.

10 shows a diagram of an example of a right optical image sensor and a left optical image sensor of the exemplary stereoscopic visualization cameras of FIGS. 7 and 8 according to an exemplary embodiment of the present invention.

11 and 12 show diagrams of an exemplary carrier of the optical element of the exemplary stereoscopic visualization camera of FIGS. 7 and 8 according to an exemplary embodiment of the present invention.

FIG. 13 shows a diagram of an exemplary flexure of the exemplary stereoscopic visualization camera of FIGS. 7 and 8 according to an exemplary embodiment of the present invention.

FIG. 14 shows a diagram of a module of an exemplary stereo visualization camera for acquiring and processing image data according to an exemplary embodiment of the present invention.

FIG. 15 shows a diagram of the internal components of the module of FIG. 14 according to an exemplary embodiment of the present invention.

FIG. 16 shows a diagram of the information processor module of FIGS. 14 and 15 according to an exemplary embodiment of the present invention.

Fig. 17 shows an example of a display monitor according to an exemplary embodiment of the present invention.

18-21 show diagrams illustrating the false parallax between the right optical path and the left optical path.

FIG. 22 shows a diagram illustrating a defocus condition related to a position of two parallel lenses for respective right and left optical paths.

Figures 23 and 24 show diagrams illustrating how false parallax causes digital graphics and/or images to lose accuracy when fused into a stereoscopic image.

25 and FIG. 26 illustrate a flowchart showing an exemplary procedure for reducing or eliminating false parallax according to an exemplary embodiment of the present invention.

FIG. 27 shows a diagram illustrating how to adjust a zoom repeat point relative to a pixel grid of an optical image sensor according to an exemplary embodiment of the present invention.

28 to 32 show diagrams illustrating a template matching program used to locate a zoom repeat point according to an exemplary embodiment of the present invention.

Fig. 33 shows a side view of the microsurgery environment of Fig. 5 according to an exemplary embodiment of the present invention.

Fig. 34 shows an embodiment of the exemplary robotic arm of Fig. 5 according to an exemplary embodiment of the present invention.

35 shows a diagram of the robot arm of FIGS. 33 and 34 connected to a truck according to an exemplary embodiment of the present invention.

Fig. 36 shows a diagram in which the robot arm of Figs. 33 and 34 is installed on a ceiling according to an exemplary embodiment of the present invention.

Fig. 37 shows an embodiment of a coupling plate for a robot arm according to an exemplary embodiment of the present invention.

Figures 38-40 show diagrams of coupling plates in different rotational positions according to an exemplary embodiment of the present invention.

FIG. 41 illustrates an embodiment of the three-dimensional robot platform of FIGS. 3 to 40 according to an exemplary embodiment of the present invention.

FIG. 42 illustrates an example procedure or routine for calibrating the stereoscopic visualization camera of FIGS. 3 to 33 according to an example embodiment of the present invention.

FIG. 43 shows an embodiment of the exemplary stereoscopic visualization camera of FIGS. 3 to 33 and FIG. 42 for moving an object plane in a discrete step according to an exemplary embodiment of the present invention.

FIG. 44 illustrates a graph illustrating a routine that can be executed by a processor for determining a projection center of the stereo vision camera of FIGS. 3 to 33 and 42 according to an exemplary embodiment of the present invention.

FIG. 45 shows a plan view of an optical schematic diagram illustrating how to measure and calibrate the distance between pupils of the stereo vision camera of FIGS. 3 to 33 according to an exemplary embodiment of the present invention.

46 shows a plan view illustrating how to measure and calibrate an optical axis and an optical schematic diagram of the stereo vision camera of FIGS. 3 to 33 according to an exemplary embodiment of the present invention.

FIG. 47 illustrates a diagram of a calibrated stereo visualization camera in which one of the optical parameters is sufficiently characterized according to an exemplary embodiment of the present invention.

FIG. 48 illustrates an example program or routine for calibrating the robot arm of FIGS. 5 and 33 to 41 according to an example embodiment of the present invention.

FIG. 49 shows a diagram illustrating how to calibrate the stereo visualization camera and/or the robot arm to the robot space according to an exemplary embodiment of the present invention.

FIG. 50 shows a diagram illustrating the horizontal and vertical boundary planes used to define the movement of the stereo visualization camera and/or the robot arm according to an exemplary embodiment of the present invention.

FIG. 51 illustrates an example of how to scale the rotational joint speed of the robot arm and/or the coupling plate based on the distance to a boundary according to an exemplary embodiment of the present invention.

FIG. 52 shows a schematic diagram of an example procedure for fusing an image from an alternative modal visualization with (several) stereoscopic images according to an exemplary embodiment of the present invention.

FIGS. 53 to 61 show diagrams illustrating the scene cross-sectional fusion visualization generated by the combination of the stereo visualization camera and/or robot arm of FIGS. 3 to 52 according to an exemplary embodiment of the present invention.

FIG. 62 shows a diagram illustrating a program for providing auxiliary driving of the stereo vision camera of FIGS. 3 to 52 according to an exemplary embodiment of the present invention.

FIG. 63 shows a diagram of an example program for using an input device to move the example visualization camera of FIGS. 3 to 52 according to an example embodiment of the present invention.

FIG. 64 shows a diagram illustrating an algorithm, routine, or program for providing a stereoscopic visualization camera locked to a target according to an exemplary embodiment of the present invention.

FIG. 65 shows a diagram illustrating a virtual sphere used in the lock-to-target feature of FIG. 64 according to an exemplary embodiment of the present invention.

300‧‧‧Stereoscopic Visualization Camera/Camera

500‧‧‧Microsurgery environment

502‧‧‧Patient

503‧‧‧Target part

504‧‧‧Surgeon

506‧‧‧Robot arm/arm/robot arm

508‧‧‧Assistant

510‧‧‧Truck

512‧‧‧Display monitor/monitor/display

514‧‧‧Display monitor/monitor/display

516‧‧‧Three-dimensional visualization platform/three-dimensional robot platform/integrated platform/platform/robot platform

520‧‧‧Computer

Claims (21)

A robotic imaging device includes: a base section configured to be connected to a solid structure or a truck; a robot arm including a first end connected to the base section, and a second End, which includes a coupling interface, and a plurality of joints and links, which connect the first end to the second end, and each joint includes a motor and a warp group configured to rotate the joint around an axis State to transmit a joint sensor at a position of the respective joint; a stereo camera connected to the robot arm at the coupling interface, the stereo camera configured to record the left image of a target surgical site and The right image is used to generate a stream of stereo images of the target surgical site; a sensor positioned at the coupling interface and configured to detect the translation and rotation of the stereo camera applied by an operator Force and transmit output data indicating the translation and rotation force; a memory that stores at least one algorithm defined by one or more commands and/or data structures, the one or more commands and/or data structures are based on at least The current position of the robot arm and the detected translational and rotational forces define the rotation direction, speed, and duration of each of the joints of the robot arm; and at least one processor, which communicates Is coupled to the sensor and the robot arm, the at least one processor is configured to: receive the output data indicating the translation and rotation forces from the sensor, and convert the output data into translation and rotation vector, Use the at least one algorithm in the memory to determine a movement sequence of the robot arm based on a current position of the robot arm and the output data from the sensor, and provide the robot arm based on the determined movement sequence by providing One or more of the motor control signals to at least one of the joints of the robot arm causes the at least one joint to rotate, wherein the rotation of the at least one joint is based on the positions given to the stereo camera by the operator The detected translation and rotation forces provide the power assisted movement of the robot arm. Such as the device of claim 1, wherein the at least one processor is configured to determine the current position of the robot arm based on output data from the joint sensors of the plurality of joints. Such as the device of claim 1, wherein the sensor includes at least one of a six-degree-of-freedom tactile force sensor or a torque sensor. Such as the device of claim 1, wherein the stereo camera includes at least one control arm having a release button to enable the power-assisted movement, and wherein the at least one processor is configured to: receive an input message indicating that one of the release buttons is selected ; And after receiving the input message related to the release button, the output data from the sensor is used to determine the movement sequence. Such as the device of claim 1, which further includes a coupling plate configured to A first end of the coupling interface connected to the robot arm and a second end including a second coupling interface configured to be connected to the stereo camera, wherein the coupling plate includes at least one joint, the at least one joint It includes a joint sensor configured to transmit a position of the respective joint and a motor that can be controlled by the at least one processor according to the movement sequence, and wherein the coupling plate includes one or more latches with rotatable The mechanism is rotated by the joint, and the rotatable latch mechanisms can be selectively moved by the operator. Such as the device of claim 5, wherein the sensor is located at the coupling interface or the second coupling interface. The device of claim 5, wherein the coupling plate includes a second joint that enables the stereo camera to be manually rotated by the operator between a horizontal orientation and a vertical orientation, wherein the second joint includes a second joint configured to A joint sensor that transmits a position of the second joint. The device of claim 1, wherein the stereo camera includes a housing, the housing including a bottom side configured to be connected to the robot arm at the coupling interface. A robot imaging device includes: a robot arm, which includes a first end for connecting to a firm structure, a second end, which includes a coupling interface, and a plurality of joints and connecting rods. The first end is connected to the second end, and each joint pack Containing a motor configured to rotate the joint around an axis and a joint sensor configured to transmit a position of the respective joint; an imaging device connected to the robot arm at the coupling interface , The imaging device is configured to record an image of a target surgical site; a sensor is positioned at the coupling interface and is configured to detect the force applied to the imaging device by an operator and/or Torque and transmit force and/or torque output data indicating the force and/or torque; and at least one processor, which is communicatively coupled to the sensor and the robot arm, and the at least one processor is grouped State: Receive the force and/or torque output data from the sensor, convert the force and/or torque output data into translation and rotation vectors, and use kinematics based on a current position of the robot arm and the The translation and rotation vectors determine a movement sequence of the robot arm. The movement sequence specifies a rotation direction, a speed, and a movement duration of at least some of the joints of the robot arm, and is based on the determined movement Sequentially, the rotation of the at least one joint is caused by one or more motor control signals provided to at least one of the joints of the robot arm. The device of claim 9, wherein the processor is configured to: based on the movement sequence, determine at least one scale factor based on at least one of the current position of the robot arm or a future position of the robot arm; and The scale factor is applied to at least one joint speed of the movement sequence. Such as the device of claim 10, wherein the at least one scale factor is configured based on a distance of the robot arm or the imaging device from a virtual boundary, and the at least one scale factor decreases to A value of "0". Such as the equipment of claim 11, wherein the virtual boundary corresponds to at least one of a patient, a medical instrument, or an operating room staff. Such as the device of claim 11, wherein the processor is configured to cause a display device to display an icon indicating that the at least one scale factor has been applied to the movement sequence. Such as the device of claim 9, wherein the processor is configured to: determine at least one scale factor based on the joint angle or joint limit between the joints of the robot arm; and apply the scale factor to at least one of the movement sequence Joint speed. Such as the device of claim 9, wherein the processor is configured to: provide gravity compensation for the force and/or torque output data; and provide force compensation for the force and/or torque output data to compensate An offset between a position of the sensor and a position of the imaging device where the force and/or torque is applied by the operator. Such as the device of claim 9, in which the processor is configured to: Determine or identify the joint singular points of the plurality of joints of the robot arm for controlling hysteresis and backlash; and determine the movement sequence while avoiding the movement of the robot arm through the joint singular points based on the kinematics. For example, the device of claim 9, which further includes a coupling plate having a first end of the coupling interface configured to be connected to the robot arm and a first end configured to be connected to the stereo camera One of the second ends of the two coupling interfaces, wherein the coupling plate includes at least one joint, the at least one joint includes a joint sensor configured to transmit a position of the respective joint, and the at least one processor can respond according to the The movement sequence controls a motor, and the sensor is located at the coupling interface or the second coupling interface. The device of claim 17, wherein the robot arm includes at least four joints and the coupling plate includes at least two joints. Such as the device of claim 9, wherein the processor is configured to transmit one or more command signals indicating the rotation direction, the speed, and the movement duration as specified by the movement sequence to the respective joint The motor causes at least one of the joints of the robot arm to rotate. The device of claim 9, wherein the processor is configured to compare images recorded by the imaging device when the robot arm is moved during the movement sequence to confirm that the robot arm moves as determined during the movement sequence . Such as the device of claim 9, wherein the kinematics includes at least one of inverse kinematics or Jacobian kinematics.

Images