BIOM 5402/SYSC5303 (BMG 5402/ELGY 6131)

Interactive Networked Systems and Telemedicine

Prof. Peter X. LIU Office: MC 7040 Tel:520-2600 ext. 1774 Email: xpliu@sce.carleton.ca

Course Description
Telemanipulator, networked mobile robot,
human motoring and sensory capability, typical interface device, visual and haptic rendering, remote control scheme, predictive technique, delay compensation, force feedback, stability and transparency, fusion of human and machine intelligence, real-time protocols and networking, data transmission and compression, history and challenges of telemedicine, and telemedicine applications: telesurgery, tele-monitoring, telediagnosis and tele-homecare .

Lecture Schedule/Office-hour
Lecture Schedule
Monday 6:05 pm-8:55 pm ME 4236

Office-Hours
TR 12:00-13:00 Or by appointment Random drop-by is very welcome, but availability is not guaranteed.

Website and Textbook

Website
http://www.sce.carleton.ca/faculty/liu/sysc5303/win13 Some lecture notes and materials, research papers to be lectured and presented by students, descriptions and requirements of course project, announcements concerning the course, etc. will be posted on the course website. Course materials will be password-protected.

Login: sysc5303 Password: BIOM5402

Textbook

Students are required to check the course website frequently. There is no assigned textbook. (Most topics lectured are still open and not well-defined. The research in this area is very active.)

Performance Assessment
In-class mid-term Exam Paper Presentation
Monday March 4th 6:05pm- 8:55 pm (Week 8) Each student is required to present one research paper selected from the paper list. Monday March 11th(Week 9) The list of papers will be posted on the course website Individually or in a group of two Presentation: Monday April 1st (Week 12) Report (IEEE conf. paper style, 6 pages) : to be received by Friday April 5th 11:59pm (xpliu@sce.carleton.ca) Take-home Exam to be distributed to students in class: Monday March 25th (Week 11) Due: to be received by Friday April 5th 11:59pm (xpliu@sce.carleton.ca)

Course Project

Final Exam

No Assignments

Grading Scheme
In-class Test 30% Paper Presentation 10% Project 20% Final Exam (take-home) 40%

The final examination and project report are for

evaluation purposes only and will not be returned to students.

Notes
Students who miss the midterm test must contact the
professor with a valid medical report or an equivalent paper within one week; otherwise, they will get zero from the missed test. Students who have missed the in-class midterm test for reasons that have been deemed valid by the professor will have the portion of the mark that was allocated to the in-class test, reallocated to the course project (not final exam). There is no make-up midterm exam.

Academic Accommodation
You may need special arrangements to meet your academic obligations
during the term because of disability, pregnancy or religious obligations. Please review the course outline promptly and write to me with any requests for academic accommodation during the first two weeks of class, or as soon as possible after the need for accommodation is known to exist. Students with disabilities requiring academic accommodations in this course are encouraged to contact a coordinator at the Paul Menton Centre for Students with Disabilities to complete the necessary letters of accommodation. After registering with the PMC, make an appointment to meet and discuss your needs with me at least two weeks prior to the final exam. This is necessary in order to ensure sufficient time to make the necessary arrangements. Please note the following deadlines for submitting completed forms to the Paul Menton Centre: March 7, 2012 for the Winter Term.

Academic Integrity
Plagiarism and cheating at the graduate level are viewed as being
particularly serious and the sanctions imposed are accordingly severe. Students are expected to familiarize themselves with and follow the Carleton University Student Academic Integrity Policy (See http://www2.carleton.ca/graduate-studies/policies-and-guidelines). The Policy is strictly enforced and is binding on all students. Plagiarism and cheating presenting anothers ideas, arguments, words or images as your own, using unauthorized material, misrepresentation, fabricating or misrepresenting research data, unauthorized co-operation or collaboration or completing work for another student weaken the quality of the graduate degree. Academic dishonesty in any form will not be tolerated. Students who infringe the Policy may be subject to one of several penalties including: expulsion; suspension from all studies at Carleton; suspension from full-time studies; and/or a reprimand; a refusal of permission to continue or to register in a specific degree program; academic probation; or a grade of Failure in the course.

Course Outline
Introduction (Week 0)
History of interactive networked and telemedicine systems; teleoperators and telehaptics, current research status and challenges Human-Machine Interface (Week 1 and Week 2) Human factors: perception and motoring capabilities; command generation techniques; cross-modal interactions and sensory substitution; visual and haptic rendering; force-reflection and haptic interface devices; requirements on haptic interfaces and design challenges Kinematics of Haptic Interfaces and Haptic Rendering (Week 3 and Week 4) Spatial descriptions; mappings; translation, rotation and transformations; Static force and moments propagation from link to link; haptic rendering through manipulator

Course Outline
Master-Slave Teleoperator and Haptic Feedback (Week 5 to
Week 7) Two-port network modelling; force and tactile feedback; time delay and discretization issues; wave-variables and passivity theorem; stability and transparency; impedance control; optimization for fidelity; control schemes and intelligence integration for telerobotics Midterm Exam (Week 8) Paper Presentation (Week 9) Data Transmission and Networking (Week 10) QoS issues; modeling of time delays; real-time and TCP-friendly protocols; the trinomial protocol; synchronization of multiplestreams; IntServ and DiffServ networks; next generation Internet. Telemedicine Applications (Week 11 ) Origins and development; technologies, challenges and obstacles; on-going telemedicine projects; robotic surgery; minimally invasive surgery; and telesurgery. Project Presentation (Week 12)

Chapter I-Introduction to Interactive Networked System (INS)

Yesterdays Internet

Initialized by USA in part for nuclear attack Matured in the 70's and commercialized in 1994 Static data transmission: email, Telnet, FTP, Interconnect mainframe computers. Limited users
Thousands Research, education and government Computer experts

Complex system In 1986, 56 Kbps network backbone in USA

Todays Internet
WWW Real-time applications (VoIP):
Facebook; MySpace; LinkedIn;

Interconnect mainframe computers, servers, PCs, iPAD,

Voice and Video over IP: e.g. Skype, iPhone, i

cell phones, PDAs, TV and Hundreds of millions of users Easy to use and access (everybody is able to use the Internet, available everywhere) An integral part of our daily life ~ Gbps network backbone (1G=106K) in most countries

Illustration of Network Speed Evolution

Download of The
Matrix DVD

The Fastest Internet

The Fastest data transfer speed: 186 Gbps In 2012, the fastest end-user download
speed is 593.20 Mbps The fastest countries

Tomorrows Internet
Size ? Accessibility? Capacity ? Speed? Application ?

We are not sure, but it is certain that:

The Internet will become faster and faster. More and more people will use the Internet QoS will be more likely to be ensured or even guaranteed. More and more devices and services ( if not everything) will be networked and available online.

Tomorrows Internet
Internet of Things: anytime, anywhere, by anyone
and anything. IoT: is envisioned to be the third wave in the global
information industry after computer and network.

Tomorrows Network Bandwidth

Third-generation fiber optics was tested separately
by both NEC and Alcatel in 2007 Speed: 10 trillion (1012) 10,000Gbps for one strand of fiber That is 150 million simultaneous phone calls. It is currently tripling about every 6 months and is expected to do so for at least the next 20 years. The fiber is already there. They are just improving the switches on the ends.

Applications on the Internet

Human interacts with the environment through five
fundamental sensory and motoring channels:
Eye (seeing) interactive, the only one can change the environment Ear (hearing) Hand (touching, feeling and manipulating) Nose (smelling)

We already realized on the network (e.g. Internet) What is expected next on the Internet?
Hand (Haptics) over IP (HoIP): Telehaptics, Teleoperation or Telerobots ---- INS Systems Seeing (video conferencing, VoIP) Hearing (IP telephone, VoIP)

Tongue (tasting)

10

The Power of Touch

A little evidence can tell you the whole story!

New Publications on Haptics

IEEE Transactions on Haptics
IEEE Robotics and Automation Society IEEE Computer Society

The first issue was published in 2008 A haptic system is essentially an INS
system.

11

A Space Application Example

Retrieve satellites and place them in the cargo bay Maintenance work

A Space Application Example

12

Some Applications Demonstrated by the Sarcos Company

Master-slave Teleoperator (MST)

Slave robot Master robot Users hand Remote object/environment

Communication Channel: e.g., the Internet or other media

13

Distinctions of INS (HoIP) from VoIP

We interact with the external world in two different ways:
Informatic

Seeing and hearing (e.g. converse with someone, read a book, etc.) A negligible amount of energy is exchanged. Interaction and energy flows are unidirectional
What is important is the information. Sensory

Energetic

Touch and manipulation (lift, touch, manipulate an object, shake hands or Significant amounts of mechanical energy pass back and forth between our
body and the external world.
Sensory and motoring

embrace other people)

Interaction is bi-directional and energy flows in two directions

VoIP (Voice and Video over IP) INS (Haptics/Hand over IP)
interactive)

In addition to information, mechanical energy exchange is involved.

Informatic, unidirectional, sensory Both informatic and energetic, bi-directional, sensory and motoring ( truly

What is this course about?

It is about interactive networked system (INS). The focus of applications is in the biomedical domain, in particular in It is mainly about: It is relevant to
Extension of human haptic capability (touch, feel, and manipulation) over a distance (or no distance) through networks or other communication media.

telemedicine (such as remote telesurgery, MIS surgery, cell surgery )

Bilateral master-slave teleoperator (MST) systems

Tele-haptics (more about sensing) Teleoperation (more about manipulation)

It is not about

Telerobots Remote control of vehicles Networked robots (relevant to sensor networks) Telepresence systems

Since we dont regard these systems as interactive ones from the energy exchange point of view.

Voice over networks (e.g., IP phone, VoIP) Video over networks (e.g., video conferencing)

14

Course Objectives
To provide an introduction to the important aspects of
human-machine interfacing haptic (touching) rending control and system integration

interactive networked systems (INS systems) including

transparency and stability; time delay compensation

To train students to do research in the relevant areas

through
Attending lectures Reading research papers Making presentation Writing paper-style project reports

networking and data transmission; applications to telemedicine.

Definition
Haptic
Relevant the sense of touch

Haptics
The science that deals with the sense of touch.

Telehaptics
Related to the transmission of haptic information (tactile and force) over a distance (such as a network) to the human operator from the remote explored object/environments. Extension of human touching sensation

15

Definition
Teleoperation
The extension of a persons manipulative capability to a remote location.

Telehaptics vs. Teleoperation

In common Both are realized through human using of a MST system Difference: Telehaptics: the tasks are exploratory, such as shape

detection, and dominated by the sensorial part of the sensory-motoring loop. Conversely, teleoperation tasks aim at manipulating and modifying the environment and are thus dominated by the motoring aspect of the sensory-motoring loop

Definition
Teleoperator/telemanipulator
handler, joystick, data-glove)

Human operator Master/human-machine interface (e. g. master robotic arm, Communication channel (e.g., wireless or wired networks, Slave/ remote manipulator and sensors (slave robotic arm, Remote objects or environments
Slave robot

direct cables, radio or satellite, private and public) force sensors)

Master robot Users hand

Remote object/environment

Communication Channel: Network

16

Definition
Telerobot
Is one form of teleoperation in which a human operator acts as a supervisor, intermittently communicating to the remote robot. Communication is on a higher level of abstraction in which the human communicates goals and the slave robot synthesizes a trajectory or plan to meet that goal. Primarily supports information interaction because of the higher level of abstraction.

In the literature, telerobot, teleoperation and

teleoperator are usually synonymous. Ether one may mean all or anyone. Telepresence

Is the situation of sensing sufficient information about the remote task environment and communicating this to the human operator in a sufficiently natural way, that the operator feels physically present at the remote site.

History
There are always situations that we need to extend our arm
and hands

17

History
Before 1500, arm extensions, such as fire tongs and animal prods,
were used and they are still used today.

History
In November 8, 1898, N. Tesla used radio waves to move a robotic N. Tesla: Father of Robotics
boat in a small pool of water in New York City.

18

History
Around 1948, the first modern
master-slave teleoperator was developed by Ray Goertz at a nuclear lab of USA to handle radiactive materilas. The master and slave are exactly the same. The master was used to send the slave commands and the slave followed the masters movements exactly. Goertzs teleoperators are still used in nuclear plants and labs today. Ray Goertz: Father of Modern Teleoperator

Separation

History
In 1950, the very first wired
television remote control, Lazy Bone, was invented. It could turn a television on and off and change channels The first wireless TV remote control was invented in 1955.
Four photocells, one in each corner of the TV screen. The viewer used a directional flashlight to activate the four control functions,

change channels turn sound on and off. The improved remote control went into commercial production in 1956.

19

History
In 1960s, GE developed the GE Handy-man which had two electrohydraulic arms each with ten DOFs (Two DOFs for each of five fingers)

History
1960s, the teleoperator technology was applied to
human limb prostheses.
For example, a lower-arm prosthesis was driven by myoelectric signals picked up from the muscles in the upper arm.

20

History
From the 1960s, telemanipulators were being attached to submarines
by the US, USSR, and French navies and used experimentally.

History
In 1990s, "Hot-Line Telerobot System" was developed
and put into the market This system allows a human operator to safely repair highvoltage electrical power lines.

21

History
In 1995, almost at the same time, Ken Goldberg et al.
at the University of Southern California and Ken Taylor et al. at University of Western Australia connected robots to the Internet

Plant in the garden

Play chess

History
On July 4, 1997 the NASA Explorer-Robot landed on Mars and
became the first radio-guided vehicle to roam the surface of the Mars. The rover sent visual and other sensor data to operators back on Earth who, in turn, sent instructions for the rover to carry out. The time-lag inherent in Earth to Mars communications required the rover to act semi-independently

22

History
April 18th 2001 Canadarm2, a Space Station
Remote Manipulator System (SSRMS), was successfully launched into space

It is used to assemble the

17.6 meters (57.7 feet) long when fully extended

International Space Station (large payload)

to attach modules weighing tons to extend solar panels longer than a bus to haul equipment to and from the space shuttle. to help dock the space shuttle

Built by Canadian Space Agency

and MD robotics

History
da Vinci MST minimally
July 11, 2000

invasive surgery system approved by the U.S. FDA on

Revenue up approximately 60% year by year Thousands of da Vinci system bases sold worldwide Over 75% sold to community hospitals Primary markets:
Urology, Cardiac, General
Surgery, Gynecology

Developed by Intuitive Surgical

A MST Configuration

A console (master) A surgical arm unit (slave) with 3 or 4 arms

23

 2001, the first

History

trans-Atlantic remote telesurgical operation between US and France. Doctors in the United States removed a gall bladder from a patient in eastern France by remotely operating a surgical robot arm. The patient, a 68year-old woman, was discharged two days after the operation.

History
Dec 22nd, 2009
Pioneering stem-cell surgery restores sight in England Now however his sight has been almost fully restored thanks to a

new technique where doctors regrow the outside membrane of his cornea from stem cells taken from his healthy eye.

The new operation involves cutting away a millimetre squared section of his left eye complete with stem cells and growing it to 400 times that size in the laboratory. The new outer skin of the eye is then stitched onto the badly damaged cornea in place of the damaged membrane.

24

Applications

Nuclear labs and power plants Space Underwater Mining Military applications Policing Housing Many not human-friendly environments Entertainment Manufacturing Agriculture and forestry Telemedicine

Nuclear

Hot cells: radioactive

environment MST with mobile platforms Tasks

Used in many nuclear power

plants and labs

Dismantling, Decontamination, Inspection (e.g. leaking pipes), Repair (e.g. tube-repair on steam boilers) Maintenance tasks Radioactive materials collection and handling.

25

Nuclear
2011 Fukushima nuclear power plant
disaster: Autonomous robots impossible

26

Space
In addition to Remotely controlling
Telemanipulator Robonauts operations:
Robonaut is a man-like robot designed to handle dexterous tasks Telepresence technology Visual and haptic feedback Human operator in spaceship or even on the ground Satellite communications Robonaut functions like a human

Now in development in
NASA USA

Underwater
In addition to robotic arms
attached to submarines ROVs (Remotely Operated Vehicles)

Unmanned cabled or wireless networked vehicles Equipped with video cameras, manipulators and other tools Used by offshore oil and gas industry to monitor of pipelines, placing of nodes, inspection of welds etc. Undersea scientific investigations Monitor earthquakes in the deep sea Navy

27

Neptune Networks
Neptune:
A Canada-US jointed, the worlds largest fiber optical network for seafloor observation ROVs are being used to perform tasks such as

Exploration Routine inspection Maintenance Investment

Hundreds millions dollars on the Canadian side USA, even bigger

Underground Mining - Today

Drilling Blasting

Mucking

Bring material to surface

In 2003, 17,315 mining workers died in mine related accidents in China 5000 died as a result of gas explosion at underground coal mines

28

Solution to U/G Mining

Operations Center

Start of Shift

Communications Infrastructure (e.g. fiber, coax) Local Wireless Ethernet

Work site

Remotely Controlled mining system

Tele-mining
Work site Drilling

Blasting

Mucking

29

E-manufacturing
Operator1

Network

Operator2

Robotic Manufacturing Workcell

Construction
Earth removal Assembly of building
components Repair power lines Cut branches from trees Perform inspection or maintenance on bridges Wash windows of high buildings Flooded tunnel inspections Pipe-man

30

Agriculture and Forestry

The Finnish Company: Six-legged walking robot Suitable for any of the
Plustech Oy

various kinds of ground conditions Weight distributed on the forest floor User to handle and move tree trunks Can be remotely controlled by using joysticks

Housing Assistance
Home surveillance and
security Home assistance
Remote control of heating systems. Remotely water the flowers Remotely feed the cats

(Samsung VC_RP30W) Samsung has developed an internet controlled vacuum with an advanced video system

31

Tele-Immersion Communication
In the future:
You sit comfortably, eyes closed, wearing a device which senses your breathing, the beat of your heart, and the exact tilt of your head. Your partner is similarly equipped. You too, intimately hear and feel each other.

Scale up/down operation

Scale down: manipulate micro- or nano- worlds Scale up: assemble or move objects weighing tons

32

Tele-Immersion into 3D Virtual Environments

Many others
Fire-fighting Policing
Dismantling a bomb Handling dangerous materials Remotely controlled inspection and finding Remotely controlled robotic fighters Unmanned Aerial Vehicle (UAV) Games Toys

Military operations Entertainment

.. Of course, telemedicine

USA Air Force Un-manned Predator

33

 Traditional surgery

Minimally Invasive Surgery

Excessive fees Postoperative pain Extended recovery periods Risk of infection Risk of complication Physician fatigue

MIS surgery is expected to

replace traditional surgery

Many surgeries now can be done by robotic devices Work is going on to standardize robotic surgery It is estimated that within the next 10 years, 90% of today's open operations will then be performed via a minimal access procedure.

Remote Telesurgery
Telesurgery
Underserved areas Specialists in remote locations to assist, train and participate directly in operations Where qualified medical care may not be readily accessible (i.e. longlasting space missions, voyage, geometrically isolated people).

34

Cellular Surgery and Fine Surgery

Stem cell surgery DNA repair Change DNA

Surgical Simulator

35

Tele-diagnosis/Tele-palpation
Tele-diagnostics of Breast Cancer

Internet

Treatment of Epidemic Diseases

H5N1 strain of bird flu N1H1 flu

Ottawa, 1918 flu pandemic

36

Aids for the Handicapped

A human-assisting telemanipulator
teleoperated by electromyographic (EMG) signals and arm motions. The system can realize a masterslave manipulator system that uses no mechanical master controller. A person whose forearm has been amputated can use this manipulator as a personal assistant for desktop work. 2003 IEEE Trans RA Best Paper Award

Prosthesis

Rehabilitation and Muscle Enhancement

37

Issues, Problems and Challenges

Human Haptics
Compared with visual and auditory
chancels, we know very little about our haptic channel

38

Haptic Inference Devices

Maximum force output: application dependent Dynamic force range: application dependent Workspace: application dependent Force output resolution: (e.g. 12bits) Spatial resolution: (e.g. a tactile display needs more than 1000 stimulator elements for the hand) Friction: (e.g. 1% of the maximum force output) Back-drivability (minimal static friction and low actuator inertia) Bandwidth: Minimum sampling rate Latency Dexterity Number of degrees of freedom Power consumption Safety Maneuvering capability Dual arms

Control and System Integration

Free-space tracking Contact stability
Master and slave are energetically coupled Time delay, delay jitter and packet loss Very difficult and complicated, still open

step down a staircase or stop a swinging pendulum

Transparency (fidelity of haptic feedback)

A teleoperator is transparent if the operator feels that he/she is directly interacting with the remote environments Why important

imagine the problems we will encounter if the same numb

Trade-off between stability and transparency Human intent and autonomy synthesis

sensation is applied to our hands when we are attempting to screw the lid onto a food jar.

39

Networking and Data Transmission

Time delay and jitter
Loss of causality Instability In-transparency Inefficiency

Bandwidth

Packet loss Sensor fusion and

synchronization

Insufficiency and variation

Mediation Security Authentication

Various signals Various data rate Various time delay

Degree of Telepresence
Sense of space
Insufficiency of information and the current technology is not ready to enable full telepresence Lack of in-depth information Lose track of the relative position and orientation of the remote manipulator with the master Dont know how fast the manipulator and end effector are moving and in what direction Scaling Time Delay

Sense of time

40

Societal Issues

Societal acceptance Privacy Is an INS system right? Potential crime

Where is your privacy ?

Hunting via the Internet Remote control rifle range debuts Nov 2004

Why INS Systems?

Now, the Internet is accessible to everybody
everywhere . Through the Internet (or a network), an INS system is able to extend our haptic capabilities to a remote environment.
Be there without physically being there. Do things there without physically being there.

INS also can extend ourselves into micro-nanoand virtual worlds

41

The Future

42