Tactile Graphics…and a Russian Redux

With Ixion becoming somewhat known for medical simulators, we displayed our ERCP (Endoscopic Retrograde CholangioPancreatography) simulator — which we had recently shown in Sydney, Australia — at a St. Louis show for medical technology. At our booth, we were asked to take a break to talk with someone from Ethicon, the largest division of Johnson & Johnson. That someone turned out to be the CEO. He had played with the ERCP simulator. He was looking for a company to build a laparoscopic simulator for the innovative new “keyhole” surgeries that were being used by surgeons to remove gall bladders by watching their inserted tools on video monitors.

We told him what we do is not easy, and that we would require a major commitment from Ethicon to follow through the tangled pathways of building a simulator. He said Ethicon had made 11 billion that year and that he could follow this through. I said I would like to work with Ethicon on this, and we agreed that I would contact him in the next week.  All very professional, very business-like, while I was shaking in my boots!

We discovered that Ethicon was also talking with another company, a leader in making flight simulators which had excellent engineers and a significant track record in successful projects. Luckily, I had recently been granted a patent in exactly what Ethicon needed, and we agreed to do a feasibility project to demonstrate how the simulator might work. This gave us a small infusion of cash, and another impossible project to complete in a few months.

I had adopted this feasibility project approach for a couple of reasons. I did not want to do a 30 page proposal and have the ideas stolen while giving the project to a brother in law who said he could do the same thing. Other companies were doing this, spending godawful hours and ruining families generating proposal after proposal which went nowhere. If someone was interested in our participation, I suggested a short meeting to understand their needs, and then I gave them a price for a feasibility study. It would include a complete development plan with costs and time. I often built-in a 5-minute video “demo” envisionment…Managers could usually get five minutes with other decision makers to get more buy in for a final, large, package. And the deal was, when they had paid for this feasibility study, they could take the information and the demo to whomever they wanted. However, because of the time invested and the tool they could show about to get concensus on the project, we were never thrown out on the street. We created added value in this process. Some of the other companies which generated 30 page proposals got a lot more projects and grew much larger than mine. But only some of them. There were a lot of burnouts and heart attacks in those companies as well.

The question of how to create a vision of a future reality is much like advertising agencies or movie makers doing a creative pitch. The difference is that, with projects like Ethicon wanted, we did it without knowing exactly how we would make the technology, or exactly how it would all work together. Movie people and ad people know how they will get the concept completed. We didn’t. We were only vaguely familiar with software and hardware that could possibly achieve a realistic simulation. Therefore the exclusive invitation was nice, and the bit of cash helped our tiny company survive, but the next step was when we proved to Ethicon not only that this realistic laparoscopic simulation could be done, but that we could do it. Scary.

Ethicon had some urgent need, however, because they were selling surgical equipment for this new laparoscopic procedure, which was minimally invasive. This meant it left only a few small scars, and the healing time was within a few weeks. People with their gall bladders removed could be back to work the next week. Everybody wanted this procedure, but no one knew how to do it. It required operating with tools inserted in trocars in the abdomen, and the surgeon had to operate remotely with long sticks holding needles and scissors and graspers. Some of the finest surgeons who operated in the open body could never get the hang of operating so remotely, both because of their limited vision the inserted television camera, and in the limited “chopsticks” feel they could get through the special instruments.

For this reason, Ethicon had to be responsible for teaching the surgeons. And short of operating on humans for the first time with no prior experience, the surgeons could only gain their first experiences in pig labs, operating on live pigs which were anesthetized for the procedure. Ethicon had a history, selling tools and thread and other operating room necessities to surgeons since the Civil War. Now Ethicon and U.S. Surgical were selling the laparoscopic tools, grippers and scissors and curved needles, plus the trocars, the tubes that went through the abdominal wall, usually three of them. The trocars would also accommodate a video camera, which gave the surgeons their remote visualizations.

Ethicon’s real problem, in this booming new minimally invasive surgical business, was the animal labs. More than one animal rights group had real issues with surgeons practicing on pigs. It was the idea of it, being unnecessarily cruel to animals – even though they were anesthetized as well as any human in surgery, and their remains were disposed of in highly hygenic circumstances. These same people seemed to have no problem with bacon, and within limits, to animal experimentation for scientific purposes. But the surgical pigs seemed to these protesting groups to fall in an inhumane category of wasteful killing for the profits of big medicine. The problem came to a head when one of the U.S. Surgical pig labs was bombed. It occured after hours and no one was killed, but it was a dynamic protest that made headlines, and a dilemma for Ethicon. They had to teach doctors in order to sell equipment. The public wanted more minimally invasive procedures, and a whole market area was wide open if they could train without pig labs.

So, once again we had a project with psychic benefits of training for lifesaving, and one which was highly important to Ethicon to waylay criticism for the pig labs. Even developing a simulator showed the public that they were responding in a responsible way to a major social problem.

I decided to pull out all the stops with the feasibility study, and create the model for it for the actual demonstration. This would make our final proposal far more understandable and believable. It was not the first time I proceeded as if I already had contract, and put in far more effort than we were being paid for. This was one of those projects that could be worth it. We were going to have to put half a spherical background in virtual reality, and that abdominal cavity had to look highly realistic. We had Dr. Noar, who had worked with us on the ERCP do the shooting in precise circles, so that our videodisc demo could skip down up and down video “rings” when the camera went in and out and pan in real time if the camera was swiveled. The final simulator would have to have angular moves by the camera, but these rings were good enough for a demo.

To get pig footage in these somewhat precise rings, I devised metal school protractor, and soldered it to a screen which would be sewn onto the pig. My son Galen actually soldered the protractor to the screening. This was a bit slap dash, but we did get the footage we needed to index on a videodisc, and at the demo, change their parameters of perception. The surgeons they had at the demo could actually maneuver and search within the 180 degree sphere we had created.

That the people at Ethicon saw something they had not imagined could be done, even in our simple demo, moved them to offer us the contract. They were sure that another company could not do it, and I had the patent which described it. It was my first 7 digit contract…well into 7 digits in fact. Now we had to figure out how to do it. I mean really figure it out…Showing a realistic background that can be maneuvered within wasn’t doing half the job. Now we had to create virtual organs to place inside the anatomical cavity, and we had to invent our own digital instruments that a surgeon could hold and believe it, and a torso with trocars which made them believe what was inside. Moreover, we found that they unanimimously wanted the organs to react to the instruments not only visually, in real time, but tactilely.

TACTILE? GRAPHICS? 3D PICTURES THAT YOU CAN PULL AND STRETCH AND SEW AND CUT? Sometimes promises get out of hand. It was 1991 and no one in the world had done this. Not games, not the space program, no one…gulp. Visual realism can be achieved with 30 frames a second, although the 3D models in real time required a “dynamics engine” to achieve something that looked like face-morphing in the movie “The Terminator” but the Silicon Graphics computers that created them worked all night on one movement. Ours had to be done in real time. But that turned out to be the easy part. To feel what you grip and cut, requires 1,500 frames per second because our sense of feel is so refined.

Our programmers began simply, trying to create simple shapes in space, so that a stick could feel the virtual outlines and ascertain what shape it was. The only problem was, there were no experts in feeling virtual shapes. But there were experts in feeling. The blind were experts in telling shapes by feeling.

Somehow we found a computer engineer attending the University of Washington…who was blind from birth. When we brought him in to test our shapes by touching them with a stick, we had been woeful failures, taking swipes in space and trying to logically space the figure in our mind from where our sticks hit. We couldn’t tell a square from a circle from a triangle. And then the blind engineer took the stick. Tap tap. “Square.” Another tap, tap. “Triangle.” And tap, tap, tap. “Circle.” Nothing quite like expertise.

So we had a team of about twenty people, mechanical engineers, software engineers, electrical engineers to design small printed circuit boards for each instruments, plastics engineers building a torso with mechanical switching when different instruments were used…everyone trying to design an experience more realistic that anyone had seen in simulation.

And then there was Russia. Yes, Russia – but not the Soviets. Eight years later I was headed back to a new and different Russia. It turned out that our realistic anatomy of a human would be a little harder to get than that from a pig. We would have to have a human body open long enough to take a sequence of individual still pictures, high resolution, that were located exactly against each other so that the virtual camera moving in an infinite combination of angles and depths seem a totally realistic and seamless experience to the surgeon. Problem was, American lawyers could not find a way to justify having the abdominal cavity left open for even an hour, and we though we would need two or three hours, photographing hundreds of positions in a circular grid, to achieve even one virtual patient. And we aimed for 4 or 5. Luckily, Ethicon had some business going with the newly democratic Russia in 1993, and they enlisted the ex-surgeon general of the army to guide our activities. It was discovered that under certain circumstances, Russian medicine would allow up to 8 hours for a medicinal drip, which theoretically left an incision open. That’s all we needed, we would have a camera on a shaft with special lenses, and shoot through that tiny incision.

Of course, the last problem was precision. The surgeon had to be able to return to exactly the same spot with his trocar camera, and for that we had to invent a stand for the 3D circular grid. I got a film crew I knew that built innovative frames for shooting, and in the spirit of true groping we went to the butcher shop and got about twenty steaks. We lined a shooting box with meat, so we could try to design a holder for the camera that allow precise locations to return to, especially during shooting, where the hundreds of pictures would be digitally sewn into the realistic background. Ethicon saw that precision mechanical engineering of such a stand to create a 3D circular grid might be right in their wheelhouse. Free of charge, their manufacturing engineers designed and built a very precise protractor frame, in effect a sort of reverse sextant, a world of sophisticated engineering away from the little protractor my son Galen soldered to the wire screen which was sewn to the pigs abdomen a year before.

We practiced on a pig and got excellent registration of each inside photo. Having now seen robotic prostate surgery, wherein the surgeon in a 3 hr operation can stop and rest and return to exactly the same spot, and knowing Ethicon worked later with the Da Vinci folks, I strongly suspect the connection. I’m actually proud of it. I’d developed the shooting frame for one kind of precision, and if it now serves another, that’s great. The world moves forward.

So we packed up for Russia and I studied my little 30 Days to Learn Russian book, and we were off in a film crew caravan, through Frankfurt airport and on to St. Petersburg. We were taken to the top hospital in St. Petersburg, which was dimly lit and overall, fairly humble by US standards. The operating room had two stories of glass windows to make use of natural light, and the vital signs consisted of a nurse holding each patients pulse and reporting on it every few minutes. Our film-shooting structures were built to surround the patient and rigidly support our futuristic camera holder, which was lowered over the patient. However, It turned out that film processing was not that dependable in Russia at the time, and with that amount of precision involved, I decided to fly the first film photo rolls to Helsinki, process them overnight, and then call the team in St. Petersburg so they could adjust the photos for the best resolution.

The Ethicon rep in Helsinki arranged for to meet me at the airport, take the film to the processor, and then we could look at it in the early morning before I caught the next flight back to St. Petersburg. I looked at the photos and made the assessment that the grid should start about 2 centimeters closer to the tissue for best effect. We could waste a whole week of shooting if I was wrong. Then, the Russians refused to let me back into the country that morning but I made an impassioned plea with my 30 days of Russian (which also allowed me later to read the names of Russian ships in Seattle from bona fide Cyrillic). Another wait of a few hours and perhaps a few Russian medical favors cashed in, and I was free to go back to the hospital.

We did get the footage in an ardous 4 days of shooting hundreds of tightly registered photos. When I was about to go through Russian customs with my photos in lead packs, the border guard said we would have to open them up and have them x-rayed. I swear this was the big square jawed Russian in a heavy overcoat that eight years before had denied me the hotel I reserved, and sent me to another for the good of the Soviet state. Anyway this time I said Nyet, Nyet, Nyet and the Ethicon people got the ex-surgeon general and they kept this from being an international incident.

And it worked. After another year’s effort, we turned the project over to Ethicon and they showed it around. Apparently the Ethicon CEO who enlisted us and who had maintained faith in a group of ragtag innovators from Seattle (though probably no one should have) presented the system to a group of Japanese surgeons visiting Ethicon, and got a standing ovation from people who know their technology. Also, our programmers wowed the top-notch SIGGRAPH conference in San Francisco with what we called Tactile Graphics. Echoes of cool cool cool buzzed through the whole Valley for a day or so. That’s the way you want these things to end.

Copyright 2018 David Hon – All rights reserved

A Day in the Clutches of Communism

Why I thought of visiting Soviet Russia escapes me now. I’d read Dostoyevsky and Tolstoy but there was no fascination left over from the 1800’s. In 1982, I tried for about a year to get a visa with application to their embassy through the mails and had quite a collection of the necessary documents I had sent with no answer. No one in Russia had asked me to visit, of course, though I did have some things they might have wanted to see.

As I was doing quite a number of tricks with the early videodiscs in the 1980s, including the CPR Learning System, the word got around and I found myself speaking at a computer show in Helsinki, Finland. I carried a videodisc to show the crowd — as most had never even imagined one, large and round and exceptionally mirror like. I also took the baby manikin and an Apple 3 computer along with the Sony Videodisc player, to give a small approximation of the program to audiences in Europe. I had a couple of blank videodiscs just to hold up and show. In a fit of good Karma, the audience of about 500 from all over Europe was treated to a pass-around of my blank videodisc, being carefully transferred like a collection plate from hand to hand. About two-thirds through the rows, one of the audience members took opposite edges of the videodisc in his two hands, and snapped it in two. He looked up with some wonderment, as if anything so shining and solid could have a brittle character as well.

The managers of the show were beside themselves, and hurried the troubled soul out of the room blubbering. I wasn’t too worried, as it was a blank videodisc provided by Sony to show people what a videodisc looked like, but my Finnish hosts apologized over and over and gave me his current troubled story. He’d been a respected professor but his wife left him and his mind crumbled. Even though he could not teach classes, his university considered him part of the family, and let him go to events such as mine.

   I finished my presentation , including showing the baby manikin demonstration. While the crowd was still intact, the show managers announced that their special Russia trip by train needed its registered participants to check in this afternoon for the trip tomorrow. On the stage, I expressed my regret I could not go with them for the two-day trip. The travel agent, who had been selling those trips from one of the vendor booths, said she was sorry as well. She mentioned that if I had kept the application papers with the proper notaries etc, which I had sent to the Russian Embassy, she could have done something. I quickly ran to my bag, and pulled out copies of all the documents had I had been instructed to send the past year. They were the ones which the Soviet Embassy had never acknowledged receipt of. The travel agent looked over my documents, surprised.

“I can work with these.” She said. “I know the right people.”

“You mean I can get on the train trip with everyone else?”

“Well, no, but I can get you a visa to fly to Leningrad for an overnight stay.”

“On a Soviet plane?”

“Right, and where would you like to stay?”

Ah, I had the answer. When things were going well for Hitler and he felt Russia was almost sewn up, he boldly sent out gold engraved invitations to all the world leaders, friend and foe, to meet him for breakfast on a day in June at the Astoria Hotel. “I want to stay at the Astoria.”

“Done. I will have the tickets and papers delivered to your hotel room tonight, and your flight will be at 10 am tomorrow.”

That night a wildly sodden Finnish bus driver drove our group of speakers in a Greyhound-sized bus at 50 miles per hour over narrow dirt roads that must have been used for logging trucks. Luckily we had a free cocktail hour before, so this dangerous excursion was actually entertaining. The bus bounded along weaving through ruts and knocking off branches which overhung each side of the road. Finally after about ten miles into the deep dark forest he stopped at a lake. There were sauna cabins with smoke pouring out. We were given towels and a little bag for our clothes and this true Finish sauna included a jump in the icy lake afterward. Unfortunately, the icy plunge left us stone sober for the harrowing trip back with the daredevil driver.

The next morning I boarded one of those Aeroflot twin-engine passenger planes that left from the sparkling English speaking airport at Helsinki, managed to squeeze in a black bread snack, and landed half an hour later at the drab, totally Russian speaking airport in Leningrad. I had left the Apple 3 computer and the the baby manikin in storage at the Helsinki hotel, and was glad of it. In the Leningrad terminal I was interviewed by one of those 6 foot 5 inch Russians whose milkman hat made him look taller, and his wool felt overcoat with epaulets made his shoulders look much wider. He squinted at my Astoria hotel reservation with cool, bland impatience. And said “Nyet.”

“But I’ve paid for the room.”


“You’re sending me back?”


“So did I do something wrong?” Geopolitical etiquette not being my forte, perhaps I did not burp appropriately after my little airplane meal. The Soviets, I had heard, were always out to get Americans.


And then with an officiously loud stamp on my Astoria reservation, he said, “You will stay at the tourist hotel,” and he pointed toward a line of similarly confused and dismayed travelers.

Our bus full of confused and dismayed travelers rolled out of the drab Leningrad Airport, rife with rumor and speculation. “They’re rounding up all foreigners…” “Where is this bus going…We’re headed outside the city, not into it.” “Why won’t they tell us anything?” The same words were likely being spoken in French, German, Chinese, and probably even Uzbek.

We crossed a drawbridge onto an island. These islands were not unusual. The Leningrad area is laced with rivers and canals leading to the ocean. And then, out of the mist, sitting on 1000 unkempt acres, rose a sort of futuristic marble palace. We gasped in 10 languages (Sacre Bleu, etc.). The uniformed drivers and monitors guided us into this ten-story palace. Inside were many reception desks that recalled the Hilton in New York. Apparently some modern Swedish architect made this creation for Soviet Russia’s latest tourism campaign.

“Welcome to the Lenigrad Intourist Hotel. Our concierges will assist you in your visit to Leningrad, and assure your every comfort.” This pleasant loudspeaker repeated itself in several languages. There were attractive young Russians in Hotel uniforms everywhere as we checked in, and we found we had clean modern rooms, in a brand new full-function hotel. But I wanted to go into the city, to see St. Peter’s Square and The Hermitage Art museum, where so many of the paintings of art history books are stored in reality. My flight back to Helsinki was at noon the next day, so I knew I would have to see something now. Luckily I had a little note pad, and the helpful concierge wrote down The Hermitage in Russian, and said they would have a Limosine for me outside, for which I would pay $40 here inside the hotel. Ah, capitalism smells the same everywhere.

“Hey, come with me. I go here before.” I guess he was Italian, but I went with him past the rows of Mercedes limos and a few blocks away into the surrounding village where most of the staff and their families lived. There were a few old Russian taxis languishing near the bus terminal. The taxi to The Hermitage cost me 75 cents. The Italian friend had changed a ten dollar bill into god knows how many rubles for me, and they seemed to spend just fine.
St. Peter’s Square was immense, and empty midday, but The Hermitage Museum was open. The lobby was a bit disheveled, the guides questioned to about 40 of us standing there as to who needed English. 10 of us did, and were herded up a stairway to a second floor. Herded is really what we were at the InTourist Hotel as well, though they were extremely organized and pleasant about it. Here less so. We were given a precise amount of time in each room and then moved on unceremoniously. The guides seem bothered by our questions as we stood in absolute awe looking at the walls.

The Hermitage was like a gigantic warehouse of impossibly great painting. They were up there with no extravagant frames and just the even, though adequate, room lighting. Most were hung edge to edge and corner to corner, almost  like a huge quilt hung on each wall. Each room of The Hermitage was jammed on all four walls with the most famous paintings from the most famous painters in the world. Apparently Peter the Great dipped deep into the national treasury of Russia and went on a buying sprees for art in Paris. (This while serfs were freezing and starving through all of his reign.) There were walls of Rembrandts and Gaugins and everyone back to the cave painters who had ever made an impact on Europe and the world with their paintings. The Hermitage was an overload at warp speed and we had finished in two hours, breathless, our eyesight assaulted with excellence.

And then there was getting back. I tried to ask the guide in English how to say InTourist Hotel in Russian, but she was off with another crowd at once. As people filed out to buses and cabs and limos I tried to find the way to say my hotel in Russian. I had not thought even to get a card there. It was as if I was in outer space. This Russian language dominated in a way I had not been exposed to before, because they allowed so few English or Americans into Russia, I suspect. Finally a kind Algerian offered to let me pay for the cab because he needed to go near the InTourist hotel himself. It was a bargain, assuming I would not be kidnapped.

Back to Helsinki, and off to other escapades around Europe with my baby manikin, but this was the last of Russia for a while. I was so shocked at the outer-space feeling of having no way to communicate, I swore that I would learn a bit of Russian if I ever came back. I did come back ten years later, and I did arm myself with a few months study that time.

That first time, in 1983 under the Soviets, we were warned not to take any rubles out of Russia. So I slipped a few bills in between pages a book. That way I could plausibly offer apologies if they searched that far.  (With inflation that hit a few years later, they would probably become virtually worthless.)  I gave those paper rubles to my two young sons, Liam and Galen,  saying I got them at great risk and could have been years in a Russian jail just for having them. Probably the boys showed them at their grade school. Probably for years I should have been watching around dark corners in case some communist-sympathizing elementary teacher in Dallas tipped them off, and — all that way — the Soviets came after me.

Probably I am safe now.

Copyright 2017 David Hon – All rights reserved