With Ixion becoming somewhat known for medical simulators, we displayed our ERCP (Endoscopic Retrograde CholangioPancreatography) simulator — which we had recently shown in Sydney, Australia — at a St. Louis show for medical technology. At our booth, we were asked to take a break to talk with someone from Ethicon, the largest division of Johnson & Johnson. That someone turned out to be the CEO. He had played with the ERCP simulator. He was looking for a company to build a laparoscopic simulator for the innovative new “keyhole” surgeries that were being used by surgeons to remove gall bladders by watching their inserted tools on video monitors.
We told him what we do is not easy, and that we would require a major commitment from Ethicon to follow through the tangled pathways of building a simulator. He said Ethicon had made 11 billion that year and that he could follow this through. I said I would like to work with Ethicon on this, and we agreed that I would contact him in the next week. All very professional, very business-like, while I was shaking in my boots!
We discovered that Ethicon was also talking with another company, a leader in making flight simulators which had excellent engineers and a significant track record in successful projects. Luckily, I had recently been granted a patent in exactly what Ethicon needed, and we agreed to do a feasibility project to demonstrate how the simulator might work. This gave us a small infusion of cash, and another impossible project to complete in a few months.
I had adopted this feasibility project approach for a couple of reasons. I did not want to do a 30 page proposal and have the ideas stolen while giving the project to a brother in law who said he could do the same thing. Other companies were doing this, spending godawful hours and ruining families generating proposal after proposal which went nowhere. If someone was interested in our participation, I suggested a short meeting to understand their needs, and then I gave them a price for a feasibility study. It would include a complete development plan with costs and time. I often built-in a 5-minute video “demo” envisionment…Managers could usually get five minutes with other decision makers to get more buy in for a final, large, package. And the deal was, when they had paid for this feasibility study, they could take the information and the demo to whomever they wanted. However, because of the time invested and the tool they could show about to get concensus on the project, we were never thrown out on the street. We created added value in this process. Some of the other companies which generated 30 page proposals got a lot more projects and grew much larger than mine. But only some of them. There were a lot of burnouts and heart attacks in those companies as well.
The question of how to create a vision of a future reality is much like advertising agencies or movie makers doing a creative pitch. The difference is that, with projects like Ethicon wanted, we did it without knowing exactly how we would make the technology, or exactly how it would all work together. Movie people and ad people know how they will get the concept completed. We didn’t. We were only vaguely familiar with software and hardware that could possibly achieve a realistic simulation. Therefore the exclusive invitation was nice, and the bit of cash helped our tiny company survive, but the next step was when we proved to Ethicon not only that this realistic laparoscopic simulation could be done, but that we could do it. Scary.
Ethicon had some urgent need, however, because they were selling surgical equipment for this new laparoscopic procedure, which was minimally invasive. This meant it left only a few small scars, and the healing time was within a few weeks. People with their gall bladders removed could be back to work the next week. Everybody wanted this procedure, but no one knew how to do it. It required operating with tools inserted in trocars in the abdomen, and the surgeon had to operate remotely with long sticks holding needles and scissors and graspers. Some of the finest surgeons who operated in the open body could never get the hang of operating so remotely, both because of their limited vision the inserted television camera, and in the limited “chopsticks” feel they could get through the special instruments.
For this reason, Ethicon had to be responsible for teaching the surgeons. And short of operating on humans for the first time with no prior experience, the surgeons could only gain their first experiences in pig labs, operating on live pigs which were anesthetized for the procedure. Ethicon had a history, selling tools and thread and other operating room necessities to surgeons since the Civil War. Now Ethicon and U.S. Surgical were selling the laparoscopic tools, grippers and scissors and curved needles, plus the trocars, the tubes that went through the abdominal wall, usually three of them. The trocars would also accommodate a video camera, which gave the surgeons their remote visualizations.
Ethicon’s real problem, in this booming new minimally invasive surgical business, was the animal labs. More than one animal rights group had real issues with surgeons practicing on pigs. It was the idea of it, being unnecessarily cruel to animals – even though they were anesthetized as well as any human in surgery, and their remains were disposed of in highly hygenic circumstances. These same people seemed to have no problem with bacon, and within limits, to animal experimentation for scientific purposes. But the surgical pigs seemed to these protesting groups to fall in an inhumane category of wasteful killing for the profits of big medicine. The problem came to a head when one of the U.S. Surgical pig labs was bombed. It occured after hours and no one was killed, but it was a dynamic protest that made headlines, and a dilemma for Ethicon. They had to teach doctors in order to sell equipment. The public wanted more minimally invasive procedures, and a whole market area was wide open if they could train without pig labs.
So, once again we had a project with psychic benefits of training for lifesaving, and one which was highly important to Ethicon to waylay criticism for the pig labs. Even developing a simulator showed the public that they were responding in a responsible way to a major social problem.
I decided to pull out all the stops with the feasibility study, and create the model for it for the actual demonstration. This would make our final proposal far more understandable and believable. It was not the first time I proceeded as if I already had contract, and put in far more effort than we were being paid for. This was one of those projects that could be worth it. We were going to have to put half a spherical background in virtual reality, and that abdominal cavity had to look highly realistic. We had Dr. Noar, who had worked with us on the ERCP do the shooting in precise circles, so that our videodisc demo could skip down up and down video “rings” when the camera went in and out and pan in real time if the camera was swiveled. The final simulator would have to have angular moves by the camera, but these rings were good enough for a demo.
To get pig footage in these somewhat precise rings, I devised metal school protractor, and soldered it to a screen which would be sewn onto the pig. My son Galen actually soldered the protractor to the screening. This was a bit slap dash, but we did get the footage we needed to index on a videodisc, and at the demo, change their parameters of perception. The surgeons they had at the demo could actually maneuver and search within the 180 degree sphere we had created.
That the people at Ethicon saw something they had not imagined could be done, even in our simple demo, moved them to offer us the contract. They were sure that another company could not do it, and I had the patent which described it. It was my first 7 digit contract…well into 7 digits in fact. Now we had to figure out how to do it. I mean really figure it out…Showing a realistic background that can be maneuvered within wasn’t doing half the job. Now we had to create virtual organs to place inside the anatomical cavity, and we had to invent our own digital instruments that a surgeon could hold and believe it, and a torso with trocars which made them believe what was inside. Moreover, we found that they unanimimously wanted the organs to react to the instruments not only visually, in real time, but tactilely.
TACTILE? GRAPHICS? 3D PICTURES THAT YOU CAN PULL AND STRETCH AND SEW AND CUT? Sometimes promises get out of hand. It was 1991 and no one in the world had done this. Not games, not the space program, no one…gulp. Visual realism can be achieved with 30 frames a second, although the 3D models in real time required a “dynamics engine” to achieve something that looked like face-morphing in the movie “The Terminator” but the Silicon Graphics computers that created them worked all night on one movement. Ours had to be done in real time. But that turned out to be the easy part. To feel what you grip and cut, requires 1,500 frames per second because our sense of feel is so refined.
Our programmers began simply, trying to create simple shapes in space, so that a stick could feel the virtual outlines and ascertain what shape it was. The only problem was, there were no experts in feeling virtual shapes. But there were experts in feeling. The blind were experts in telling shapes by feeling.
Somehow we found a computer engineer attending the University of Washington…who was blind from birth. When we brought him in to test our shapes by touching them with a stick, we had been woeful failures, taking swipes in space and trying to logically space the figure in our mind from where our sticks hit. We couldn’t tell a square from a circle from a triangle. And then the blind engineer took the stick. Tap tap. “Square.” Another tap, tap. “Triangle.” And tap, tap, tap. “Circle.” Nothing quite like expertise.
So we had a team of about twenty people, mechanical engineers, software engineers, electrical engineers to design small printed circuit boards for each instruments, plastics engineers building a torso with mechanical switching when different instruments were used…everyone trying to design an experience more realistic that anyone had seen in simulation.
And then there was Russia. Yes, Russia – but not the Soviets. Eight years later I was headed back to a new and different Russia. It turned out that our realistic anatomy of a human would be a little harder to get than that from a pig. We would have to have a human body open long enough to take a sequence of individual still pictures, high resolution, that were located exactly against each other so that the virtual camera moving in an infinite combination of angles and depths seem a totally realistic and seamless experience to the surgeon. Problem was, American lawyers could not find a way to justify having the abdominal cavity left open for even an hour, and we though we would need two or three hours, photographing hundreds of positions in a circular grid, to achieve even one virtual patient. And we aimed for 4 or 5. Luckily, Ethicon had some business going with the newly democratic Russia in 1993, and they enlisted the ex-surgeon general of the army to guide our activities. It was discovered that under certain circumstances, Russian medicine would allow up to 8 hours for a medicinal drip, which theoretically left an incision open. That’s all we needed, we would have a camera on a shaft with special lenses, and shoot through that tiny incision.
Of course, the last problem was precision. The surgeon had to be able to return to exactly the same spot with his trocar camera, and for that we had to invent a stand for the 3D circular grid. I got a film crew I knew that built innovative frames for shooting, and in the spirit of true groping we went to the butcher shop and got about twenty steaks. We lined a shooting box with meat, so we could try to design a holder for the camera that allow precise locations to return to, especially during shooting, where the hundreds of pictures would be digitally sewn into the realistic background. Ethicon saw that precision mechanical engineering of such a stand to create a 3D circular grid might be right in their wheelhouse. Free of charge, their manufacturing engineers designed and built a very precise protractor frame, in effect a sort of reverse sextant, a world of sophisticated engineering away from the little protractor my son Galen soldered to the wire screen which was sewn to the pigs abdomen a year before.
We practiced on a pig and got excellent registration of each inside photo. Having now seen robotic prostate surgery, wherein the surgeon in a 3 hr operation can stop and rest and return to exactly the same spot, and knowing Ethicon worked later with the Da Vinci folks, I strongly suspect the connection. I’m actually proud of it. I’d developed the shooting frame for one kind of precision, and if it now serves another, that’s great. The world moves forward.
So we packed up for Russia and I studied my little 30 Days to Learn Russian book, and we were off in a film crew caravan, through Frankfurt airport and on to St. Petersburg. We were taken to the top hospital in St. Petersburg, which was dimly lit and overall, fairly humble by US standards. The operating room had two stories of glass windows to make use of natural light, and the vital signs consisted of a nurse holding each patients pulse and reporting on it every few minutes. Our film-shooting structures were built to surround the patient and rigidly support our futuristic camera holder, which was lowered over the patient. However, It turned out that film processing was not that dependable in Russia at the time, and with that amount of precision involved, I decided to fly the first film photo rolls to Helsinki, process them overnight, and then call the team in St. Petersburg so they could adjust the photos for the best resolution.
The Ethicon rep in Helsinki arranged for to meet me at the airport, take the film to the processor, and then we could look at it in the early morning before I caught the next flight back to St. Petersburg. I looked at the photos and made the assessment that the grid should start about 2 centimeters closer to the tissue for best effect. We could waste a whole week of shooting if I was wrong. Then, the Russians refused to let me back into the country that morning but I made an impassioned plea with my 30 days of Russian (which also allowed me later to read the names of Russian ships in Seattle from bona fide Cyrillic). Another wait of a few hours and perhaps a few Russian medical favors cashed in, and I was free to go back to the hospital.
We did get the footage in an ardous 4 days of shooting hundreds of tightly registered photos. When I was about to go through Russian customs with my photos in lead packs, the border guard said we would have to open them up and have them x-rayed. I swear this was the big square jawed Russian in a heavy overcoat that eight years before had denied me the hotel I reserved, and sent me to another for the good of the Soviet state. Anyway this time I said Nyet, Nyet, Nyet and the Ethicon people got the ex-surgeon general and they kept this from being an international incident.
And it worked. After another year’s effort, we turned the project over to Ethicon and they showed it around. Apparently the Ethicon CEO who enlisted us and who had maintained faith in a group of ragtag innovators from Seattle (though probably no one should have) presented the system to a group of Japanese surgeons visiting Ethicon, and got a standing ovation from people who know their technology. Also, our programmers wowed the top-notch SIGGRAPH conference in San Francisco with what we called Tactile Graphics. Echoes of cool cool cool buzzed through the whole Valley for a day or so. That’s the way you want these things to end.
