FYP17011

Tsai, Yi Ting (3035124520) Bui Thanh Tang (3035124491) Supervisor: Dr. Dirk Schnieders
Introduction
Learning to play the piano is a difficult process that requires constant and rigorous practice. While it is believed that 65% percent of human beings Objectives
are visual learners, traditional methods of teaching and learning to play the piano have not been focusing on exploiting the potential of human vision.
Meanwhile, in helping new learners picking up the instrument faster, the idea of using falling notes and colored keys as visual hints has proven to be • Pianow is reasonably accurate and effective in aligning its virtual
highly promising by many applications. In this project, this idea will be brought into Mixed Reality, a combination that is expected to improve the content with a standard 88-key piano keyboard.
learning experience, as learners can follow the tutorials without looking away from their own physical piano keyboard. The new application, named
Pianow, aims to offer beginners a new way to practice playing the piano by themselves, which is easy, fun, and interactive. Pianow is developed for • Pianow offers at least 2 working tutorials with intuitive user
the HoloLens. It can effectively align virtual assets to the physical piano keyboard and display interactive piano tutorials consisting of falling notes and interface and good user experience.
colored keys.

Methodology
A. Manual alignment:
The 3D model of keyboard is oriented perpendicular
to the physical keyboard. The model is attached
with HandDraggable script from HoloToolkit,
allowing user to drag and drop the virtual keyboard.
B. Marker detection with Vuforia:
Upon detecting the image target, Vuforia
performs feature-based pose estimation and
generates the attached content based on the
pose of the target detected.
Result
A. Manual
C. Keyboard recognition with OpenCV : Step 4. Obtaining the keyboard’s position and orientation
D. Texture mapping:
The keyboard position is determined by its center of gravity
Step 1. Binarization extract only relevant
(point A in left figure below), which can be calculate using Render premade
information from the image to reduce computational
OpenCV Moments class. The center of gravity is chosen because video into Unity
cost in later steps, which takes advantages of the
it proves to be more stable. The the eigenvector of the contour is texture and material.
color characteristics of white and black.
calculated using Principal Component Analysis (PCA) (vector AB Texture mapping the
in left figure below). This eigenvector indicates the orientation of video on an L-shaped
the keyboard. model.
Step 2. Morphological closing operation with kernel
of size 5x5 is performed on the binary image to
eliminate the gaps between different keys, which A B E. User testing:
ensures that the entire keyboard area is considered Subject: 10 university students with and
as a connected component. without musical background
Setup: Camera recording
Step 5. Pose estimation Procedure:
Step 3. Contour extraction and analysis are aimed at In the main program, the coordinates of A and B are first translated Gesture learning→Alignment
identifying the keyboard area. The contour is filtered in from the camera coordinate to world coordinates. This function testing→Tutorial testing→survey
first size and dimension of min-size rectangle and second
requires the knowledge of the camera-to-world translation matrix Evaluation: Survey, posture observation,
Convex hull and Douglas-Peucker approximation.
and projection matrix of the camera. (Figure right) video recording
C. Alignment method comparison:
D. Tutorial
Marker with Vuforia method is considered the
D. User satisfaction:
Do you think the falling notes/colored
best solution among three in user testing. keys help you to learn?
Preliminary test result summary
(unit: number of users)

12 Yes
No
10

8
B. Marker Do you consider Pianow as a tool
6 of learning piano?

4
Yes
E. Control menu F. Instruction for aligning 2 No

(Marker method) 0
Manual OpenCV Vuforia

App from 1-10 average rate (10 is the highest): 7.5

Satisfied Encounter difficulties Succeed at first try

C. OpenCV E. Posture/Gesture observation:

Hand placement could be added in the future to provide a better learning experience.

Uncomfortable sitting posture Subject with musical background Subject without musical background
Center view (left), Side view(right) due to screen view limitation. with correct fingering. with awkward fingering.

Conclusion
• There are two major challenges to be addressed in this project: virtual asset alignment and
design for effectiveness.
• Three methods have been implemented, including manual alignment, keyboard
recognition with OpenCV and marker detection with Vuforia.
• Marker detection with Vuforia method achieved the highest accuracy of alignment and
has the simplicity for user to conduct alignment.
• Pianow has the flexibility to add in more functionality for example hand
placement and practice evaluation system.
• In the long term, the application is anticipated to change the way self-taught
pianists learn and be a good foundation for future projects which aspire to
exploit the potential of Mixed Reality in musical education.