VIRTUAL HAND AS DIRECT INTERACTION MODEL FOR

AR - RESIDENTIAL AREA DESIGN

Yuri Yudhaswana Joefrie
Email : yu_ri79@yahoo.com
Abstract: This thesis promoted the topic concerning the interaction technique using virtual hand in Augmented
Reality environment (AR). AR Residential Area Design (ARAD) uses tracker library that use marker
to know the human gesture. With the kind of interaction i.e. manipulation of the model, manipulation
of the street, button pressing, and duplicate the model, ARAD could become alternative in designing
residential area. At the end of the thesis, experiment was conducted concerning level of comfort of
ARAD. The result were almost all the respondents did not experience difficulty in interacting and all
the respondents agreed that AR technology will bring the new paradigm in the residential are design.
Keyword: augmented reality, virtual hand, natural gesture
1.

INTRODUCTION

AR is synthetic image added to the real world.

Despite already many developments from AR good
from the side of technology and tracking system,
but just a little equipment that enabled the well-off
user to interact with the virtual object that was seen,
as moving the object, carried out the change in the
form so as the user felt in accordance with the new
form, et cetera. The interaction ought to in AR be
intuitive so as to be able to be accepted by the user.
At this time, there were several researches that
focus towards the movement of the human body,
especially the hands. This could be understood as
an effort to humankind in developing the
interaction technique that in accordance with the
hands of humankind because of the hands of
humankind was the perfect interaction implement
that could move free in accordance with the wish
and interact with the real object.
Many application scenarios that were suitable for
model of virtual hand. One of them is residential
area design. In this field, several individuals, like
the architect, drafter, and urban planner gathered
and colaborate to draft a dwelling region.
2.

ARAD

In this part, will be explained more in concerning

the system of ARAD. Explanation covered ARAD
on the whole, the ARAD design, and the interaction
in the ARAD system.
2.1 Overview of ARAD
ARAD is the application for designing the
settlement region by using AR. ARAD uses virtual
hand as basic interaction model and sole input. The

AR library that was used is ARToolKitPlus that was

developed by Daniel Wagner.
2.2 Design of ARAD
ARAD uses 3 marker sets to accommodate the
master's model, results of the design process, and
drafting virtual hand. Marker to accommodate the
master's model has the measurement of 70cm
70cm with the height 100cm. When ARAD is
running, the virtual model will be drawn exact in
the middle of marker. In the left corner bottom
there are 5 virtual buttons that respectively
functioned for the change of interaction mode, save
results of the design, loads results of the design that
was saved beforehand, navigation to the previous
page, and navigation to the next page. Whereas
marker for the design of the model has the
measurement of 180cm 180cm with the height
100cm. When ARAD runs, will be drawn virtual
landscape above markers. In the right corner
bottom draws garbage icon. Finally, marker for
drafting virtual hand uses 3 markers that was glued
at human hand. It had the measurement of 2cm
2cm.
2.3 Interaction in ARAD
Interaction in ARAD is divided into 4 kinds that is:
manipulation of the model, button pressing,
manipulation of the street, and duplication of the
model. The manipulation of model is the
interaction kind that enable user to take, moved,
removed, heighten/shorten the virtual model. This
interaction was aimed for model that has big in
size, like the building and the house. The button
pressing is the interaction kind that enable user to
interact with the virtual button. The manipulation
of the street is interaction kind that enable user to
interact with the virtual street. The user could draw

the virtual street in the area of the design and

stretch out or shorten the virtual street. Duplication
of the model is the interaction kind that enable user
to interact with the virtual model that small in size,
like the tree and the drum of the waste.

b. find transformation matrix at the point in the

middle of J1 and J2 with design marker,
c. find how much model has been rotate in z-axis,
d. set attribute of virtual model with those result
calculated before.

3.

3. Heighten or shorten virtual model.

The system will translate the user's wish whether
the user want to heighten/shorten the model or not
based on high the grabbed point against the
building. If the point took smaller than 3cm from
on top of the building, then the system will say
scaling, moreover not scaling. Along with this
pseudo code to heighten or shorten the model:
a. for every marker J1 and J2, check whether J1
and J2 is in the virtual model,
b. continue against all the models if the status of
the system not yet skala,
c. check whether the grab point between J1 and the
virtual model smaller than 3cm from the top of
the building,
d. set system status equals to skala,
e. find the distance between J1 and J2,
f. find height J1 relative to the design field of the
model,
g. set the value of the scale in z-axis from the
model virtual with result of the calculation
above,
h. continue until the status of the system equals to
normal, that is if the distance J1 and J2 bigger
than the distance J1 and J2 beforehand.

INTERACTION ALGORITHM

The interaction in ARAD in basically involved 3

measurements, that is the distance of marker at the
hand with the virtual model, the distance of marker
at the hand with the area of master or design, and
the distance of 2 markers on the hand. For each
interaction kind is explained as follows.
3.1 Model Manipulation
Several activities in the model manipulation are
grabbing, dropping, scaling and deleting. This kind
of interaction require marker that was on the end of
the index finger and the tip of the thumb is detected
by ARToolKitPlus. This kind of interaction is
necessarily because this is the part of the design
process. Respectively the interaction kind will be
explained as follows.
1. Grabbing.
The system will enter the status ambil if and only
if marker that was located on the end of the index
finger and on the end of the thumb is in the virtual
model. Along with this pseudo code to take the
model:
a. for every marker J1 and J2, check whether
marker is in the virtual model,
b. keep continue against all the models if the status
of the system not yet ambil,
c. find point in the middle of J1 and J2 in world
coordinate,
d. find the value of z-axis of the model,
e. find the distance between J1 and J2,
f. find grabbing point J1 from the top of model.
For the status of the system still ambil then the
system continued rendering virtual model in the
middle between J1 and J2.
2. Dropping.
This activity was done only if the system at this
time in the status ambil. The system will state that
the model was dropped or released only if the
distance between markers that was on the end of
the index finger and marker that was on the end of
the thumb was bigger than the distance that was
kept beforehand. Along with this pseudo code to
drop or release the model:
a. if the distance between J1 and J2 now bigger
than the distance between J1 and J2 beforehand,
then set the status of the system as normal,

4. Deleting.
When the user want to eliminate the model virtual
from the design table, then the user was kept taking
this model and aiming the model at the area of the
waste. Pseudo code from this activity was clarified
in the point (1) with the addition as follows:
a. find point in the middle of J1 and J2 with the
design field of the model,
b. find the distance between waste area and the
middle point of J1 J2,
c. if result of calculation stated that the middle
point of J1 J2 entered the area of the waste,
then the system rearrange array of design
variable,
d. set system status that beforehand ambil to
normal.
3.2 Pressing.
This kind of interaction that enable user to interact
with the menus that were available in the program
of ARAD. Only need 1 marker that was detected by
ARToolKitPlus, that is marker located on the end of
the index finger. The interaction of this kind is
simplest in the matter of the algorithm from all

over the interaction kind in the program of ARAD.

Here is the pseudo code for the pressing:
a. for marker J1, checking whether J1 is in the
virtual button,
b. continue checking until the status of the system
be the same as tekan,
c. set height of the virtual button in order to create
animation: the button descended and afterwards
rose again,
d. set system status equals to normal.
3.3 Street manipulation.
This kind of interaction enables the user to take the
virtual street and placed in the certain location in
accordance with the user's wish. Needed by just
one marker that is detected by ARToolKitPlus for
this kind of interaction, that is marker located on
the end of the index finger (same like the pressing).
Along with this pseudo code for the street
manipulation:
a. for marker J1, checking whether J1 is in the
street virtual,
b. continue checking until the status of the system
be the same as tunjuk,
c. find height J1 relative against the field of the
model master,
d. if the height J1 less than 2cm from the top of the
field of the model master and J1 entered the area
of the road virtual, then set the status of the
system be the same as tunjuk and set state
equals to 0;
If marker on the design field of the model was
detected, then continue:
e. find the height J1 relative to design field of the
model,
f. if height of J1 less than 2cm from the top of
design field then system status still 'tunjuk' and
set 'state' equals to 1,
g. find in z-axis,
h. find d distance J1 relative to design field,
i. do rotate virtual street in z-axis equals to ,
j. do scale virtual street in y-axis equals to d.
If 'state' equals to 1 then continue to the following
pseudo code:
k. find the distance J1 with design field,
l. if the distance less than 2cm then set system
status equals to 'normal',
m. set all of street's attribut with the result
calculated above.
3.4 Duplicating
This kind of interaction duplicate model with small
dimension. This kind of interaction only need
marker that was located on the end of the index
finger (same as button pressing and street
manipulation). To activate this mode, user could
press the button that had special texture. Afterwards

by using the index finger, marker was brought

closer with the model that wanted to be taken. After
succeeding in being taken, the user aimed this
model at the place that user wanted and tilted
marker against the x-axis or y-axis as big as 180
so that this model falls. Along with this pseudo
code to copy the model:
a. for marker J1, checking whether J1 is in the
virtual model,
b. continue checking until system status is salin,
If marker on the design field was detected, then
continue to this pseudo code to drop the model:
c. find angle between J1 and design field,
d. if > 180 then model is dropped,
e. find transformation matrix J1 relative to design
field,
f. save translation matrix as translation matrixs
new model,
g. set system status equals to normal.
4.

TESTING AND ANALYSIS

Taken by as many as 12 respondents, and

respectively the respondent was given eight dual
choice questions and three questions essai. From
the questionnaire for the dual choice question,
results show that the respondent did not experience
the significant difficulty when interacting except for
two kinds of interaction, that is the placement of the
virtual model and the orientation of the virtual
model. Two problems were believed that these are
result by the production of the source code that was
not yet perfect, and not was caused by the
instability or inaccuracy ARToolKitPlus. Whereas
for the question essai, heterogenous opinion
emerged regarding the program of ARAD. Some
respondents felt the difficulty when manipulating
the model virtual. However after the light was
matched, the difficulty when taking the object did
not happen again.
5.

CONCLUSION

ARAD was built by using direct and natural kind of

interaction. The interaction that was introduced in
this system was the manipulation of the model, the
manipulation of the street, pressing button, and
duplicate the model. ARAD used only two virtual
fingers to interact with the system. The scenario of
residential area design was applied in ARAD. The
ability of ARAD in manipulating the model enabled
supported the work of the group's collaboration in
internal developer. ARAD was also introduced in a
limited way and public to test the level of comfort
of interaction. The results were almost all the
respondents agreed that ARAD found it easy to be
used and for the future become alternative

technology in the process of residential area design.

However, the respondent admitted to having
difficulty in determining the orientation of the
model
when
interacting.
Therefore,
the
improvement of the code also became the case that
will be researched for the future work. Must be paid
attention to concerning the light factor that could
influence the stability and the continuity of the
marker detection. If the intensity of the light was
too strong or too weak then the possibility marker
to be not detected increasingly big.
REFERENCES
Aliakseyeu, D., Martens, J., and Rauterberg, M.,
(2006), A Computer Support Tool for the
Early Stages of Architectural Design,
Interacting with Computers, 18(4), 507-868.
ARToolKitPlus,
Documentation,
http://studierstube.icg.tugraz.ac.at/handheld_ar/artoolkitplus.php,
diakses tanggal 14 Agustus 2008, 12:13 WIB.
Joseph, Ben, Chak, D., Ishii, H., Kangi, Z., Piper,
B., Underkoffler, J., Yeung, L., (2002),
Augmented Urban planning Workbench:
Overlaying Drawings, Physical Models and
Digital Simulation, Proceedings of ISMAR
2002, 203- 211.
Buchmann, Volkert et al., (2004), FingARtips
Gesture Based Direct Manipulation in
Augmented Reality, HitLabNZ, New Zealand.
Dnser, Andreas et al., (2004), Applying HCI
Principles to AR Systems Design, HIT Lab
NZ, University of Canterbury, New Zealand.
Gibet, S., et al., (2006), Gesture in Human
Computer Interaction and Simulation, Berlin,
Heidelberg , and New York: Springer.

Hillebrandt, Patricia M., (1974), Economic

Theory and the Construction Industry,
Macmillan Press Ltd, London.
Kato, Hirokazu, (2000), ARToolKit Version 2.33,
http://www.tinmith.net/lca2004/ARToolkit/AR
ToolKit2.33doc.pdf, diakses tanggal 05 Juli
2008, 13:00 WIB.
Kieferl J., and Wossner U., (2003), Mixed
Realities: Improving the Planning Process by
Using Augmented and Virtual Reality,
Proceedings of the 4th Conference on new
technologies in landscape architecture, Dessau.
Ledermann, F., (2004), An Authoring
Framework
for
Augmented
Reality
Presentations, Masters thesis, Vienna
University of Technology.
Lowton, M., (2005), Visual Location Tracking
using Artificial Landmarks, Thesis.
Malbezin, P., et al., (2004), Measuring
ARToolKit Accuracy in Long Distance
Tracking Experiments, Wearable Computer
Laboratory, University of South Australia.
Malik, Shahzad (2002), Robust Registration of
Virtual Objects for Real-Time Augmented
Reality, Thesis, The Ottawa-Carleton Institute
for Computer Science, Canada.
Piekarski, W., Backpacking Scientist Rearranges
the World Using Augmented Reality, 2003,
url:http://www.sciencenow.org.au/sciencenow/
2003/wayne _piekarksi.htm, diakses tanggal 12
Juni 2008, 11:00 WIB.
Wang, Xiangyu et al., (2008), Experimental
Study on Augmented Reality Potentials in
Urban Design, University of Sydney, Sydney
NSW 2008, Australia.