Evaluation of Learning Game Application

“Memrise” Using RETAIN Model

Written by:

NAFRIANTI

1212817023

MAGISTER OF ENGLISH DEPARTMENT

FACULTY OF LANGUAGE AND ART

STATE UNIVERSITY OF JAKARTA

2018
 TABLE OF CONTENTS

CHAPTER 1
1.1 Background of Research ...............................................................................
1.2 Problem of Research .....................................................................................
1.3 Purpose of Research ......................................................................................
1.4 Significance of Research...............................................................................
CHAPTER 2 Literature Review
2.1 Mobile Learning Apps ..................................................................................
2.2 Learning Games in Apps...............................................................................
2.3 English Learning Apps..................................................................................
2.3.1Top 5 Popular English Learning Apps in Google Play Store .....................
2.3.2Criteria for English-Learning Apps ............................................................
2.4 Language Acquisition Theories and Learning-games Apps .........................
2.5 Evaluating Online Learning ..........................................................................
2.6 RETAIN Model .............................................................................................
2.7 Previous Studies ............................................................................................
CHAPTER 3
3.1 Method ..........................................................................................................
3.2 Design ...........................................................................................................
3.3 Source of Data ...............................................................................................
3.4 Data ...............................................................................................................
3.5 Instrument .....................................................................................................
3.6 Technique of Data Collection .......................................................................
REFERENCES
 CHAPTER 1

INTRODUCTION

1.1 Background of Research

Nowdays, technology has played as important role in educational field. Some
technologies have been gradually integrated into teaching and learning process. There are some
kinds of technologies that used for educational purpose such computer, laptop, television, and
even a device that can be used anywhere and anytime, smartphones. The wide use of
smartphones and other portable and wireless devices has been significantly changing ways of
learning in many contexts, including language learning (Kukulska-hulme cited in Guo:2014)
Those technologies can bring user to experience learning process just trough their fingertips.
This type of web-based technology and powerful internet connections provide various
new possibilities for the development of educational technology like for English
learning(Cavus&Sarica:2009). Since the smartphone and any mobile devices are designed for
access everything that users need, it affects learning English in this current decade. English has
become easier to learn more than ever before with the availability many sources to help people
learn very easily and enjoyable. Through the use of internet and multimedia, learners can engage
in individualized instruction where they can investigate and learn concepts and content to meet
their needs (Kazanci&Okan:2009).
The component which can make learner learning by theirselve in any mobile devices is
called by APPs. APPs refers to the applications on mobile intelligent terminal equipment
(Liu&He:2014). Mobile learning APP, which is one kind of APPs, mainly involves the content
of study, reading, examination, etc. Mobile learning APP makes it possible for users to study
anytime and anywhere via mobile phone which greatly improve the convenience of learning and
users’ learning enthusiasm. Because some developers try to create learning app is more
interesting, they combine it with game element in it. Combining games with mobile devices can
promote learning opportunities at the learners’ fingertips and enable global learning experiences
(Gunter, et all:2016)
However, it is rare to find in-depth studies which have been carried out to investigate
the features of current English learning games mobile apps in relation to existing learning
theories and educational features. The great deal of work has usually centred upon how ICTs are
use, not talking why into account. (Kazanci&Okan:2009). As Okan(2003) cited in Kazanci
&Okan(2009), a critical perspective on computers and any mobile devices are lacking among
educators. Also, when the learning app is combine with game element inside it in order to attract
the user to use it, sometimes the content can be questioned, it is fulfil educational element when
it becomes learning-games app or not.
Gunter et all (2016) have already done evaluation toward two learning-games APPs in
android, Busee and Duolingo in their research. In their research, they used RETAIN model
rubric as their reference to evaluate those apps. The finding indicated although a stand-alone
educational game scored low on the RETAIN model, these language learning applications do in
fact have a place in learning environments. Their contributions to introducing and practicing
vocabulary and grammar contribute to language learning. However, there is no explanation what
learning theories or paradigm that used by developer in creating those apps.
Besides two of application above, one of APPs for learning English which is also
popular among learners is memrise APP. From the number of download from user, it reaches 10
million downloads. Besides that, this APP got award twice for the best educational APP in 2016
and 2017(google play store). The accumulation score which are given by the user is 4,7. It seems
that the most users are satisfied which the content of APP when it comes for learning language,
especially English. It reflects that the content is good in users’ perspective.
Since recent studies worldwide have recognized that students actively play game or
learning apps to learn English, indicating that games must be taken into account to strengthen the
student and provide students the opportunity to appropriate digital tools for learning that they
have already used daily activities (Gunter et all:2016). With the language teacher perspective and
taking into consideration that mobile devices are the fastest growing technology platform in the
developing world, this study includes the evaluation and discussion of memrise app used for
language learning. Besides that , this study find out whether contain language learning theories
or second language theories.

1.2 Problem of Research

1. How effective the memrise app in helping students learn the content or develop the skills
the games are reinforcing?
2. Which learning theories are most relevant to the app?
1.3 Purpose of Research
1. To know effective the memrise app in helping students learn the content or develop the
skills the games are reinforcing?
2. to know learning theories are most relevant to the app?

1.4 Significance of Research

This research will be expected to give several benefits and new insight for every
participant in educational field, teachers, students, and the research.
- For teacher
This research will give teacher to decide whether application is appropriate as
supplementary material for teaching and learning English.
- For student
This research will be as students’ guide to choose the appropriate application for learning
English
- For researcher
This research will give a new sight about learning English application.
 CHAPTER 2
LITERATURE REVIEW

2.1 Mobile learning apps

The development of mobile technology provides users more chance to receive education.
The availability of mobile learning app helped bridge inequities around access and uses of
technology for academic growth and personal empowerment( Mouzat&Greenly:2015). The rapid
development internet connection and system in mobile phones triggers to increase the number of
APP downloaded by users. The users are more interested to download the APPs because of this
situation. Mobile learning could enhance the learning interest and improve the grades
(Mouzart&Greenly:2015). In Indonesia, some studies indicated that teachers had positive
perception to mobile learning and had high enthusiasm to participate in it. Eventhough there are
also gave bad impression toward usage of technology in the class.
The developers paid attention to how to improve users’ learning efficiency by using
mobile learning APP [E. VÁZquez-Cano as cited in Liu&He:2014], by means of the method of
quantitative to analyze the effect of the use of smartphones and APPs by college students, found
that mobile learning could bring students more practice chances. As the perceived
responsiveness is a important influence factor of satisfaction, the mobile learning APP
developers could take the interaction into consider.
Firstly, using wireless network can help users to share at anytime and anywhere.
Secondly, especially with the use of social networks, a lot of social APPs have been embedded in
mobile learning environment. Using the public platform, pushing message and other functions
can set up the interaction platform between students and students, teachers and students. It will
increase the interaction efficiency and promote meaningful learning. Lastly, establishing a
comprehensive, dynamic and diversified evaluation mechanism is beneficial to the result
evaluation between the users. Users can evaluate other people’ learning proces, sharing
resources, uploading task and completing result.
Users can mutually evaluate by comment and instant communication, which makes it
possible for users to timely correct problems and find the solutions to further improve the
learning process and enhance the learning effect. Perceived content could enhance satisfaction if
it can meet the users’ needs and even above their expectation. The fun element should be added
to attract users to participate in study. The developers need to decide learning content according
to the users’ learning rule and characteristics. The design of learning activity should combine the
teaching content, for example, preview before class and review after class could be put into
mobile learning APP, which can let users independently study. To attract more users, developers
make learning apps integrated with games features.

2.2 Learning games in apps

Games allow players to be producers and not just consumers. Along with the designer,
the players actions co-create the game world( Gee:2003) Good games confront players in the
initial game levels with problems that are specifically designed to allow players to form good
generalizations about what will work well later when they face more complex problems. Often,
in fact, the initial levels of a game are in actuality hidden tutorials. Work in cognitive science has
shown that people need to be presented with problems in a fruitful order, getting initial problems
that set up good generalizations for later problems.
Motivation is the most important factor that drives learning. When motivation dies,
learning dies and playing stops. Cognitive science has had a hard time defining motivation,
though one definition is a learner.s willingness to make an extended commitment to engage in a
new area of learning (diSessa as cited in Gee: 2003). Since good games are highly motivating to
a great many people, we can learn from them how motivation is created and sustained.
Cognitive research suggests that such fine-grained action at a distance actually causes humans to
feel as if their bodies and minds have stretched into a new space [Clark as cited on Gee 2003], a
highly motivating state. Books and movies, for all their virtues, cannot do this. The more a player
can manipulate a game character and make decisions that impact on the character, the more the
player invests in the character and the game at a deep level. This investment appears to be the
deepest foundation of a players with the end create them to be master game.
Finally, we can state that when players play in massive multiplayer games, they often
collaborate in teams, each using a different, but overlapping, set of skills, and share knowledge,
skills, and values with others both inside the game and on various Internet sites. In the process,
they create distributed and spread knowledge within a community [Wenger et al, 2002 as cited in
Gee:2003].
 When dealing with mobile applications, however, the reusability of their different parts is
not something essential to the quality of the app, even if it is always convenient for app designers
to be able to reuse already created materials. Besides that, the value content related with
pedagogical must considered by the developer. If the learning objective does not exist in app, or
the objective does not match with the apps’ content, it can be questioned about the quality of
apps in educational view. The quality from pedagogical view can look through some criterion.
This criterion assesses if the teaching aim(s), the target users and the skills to be developed are
clearly set from the beginning. According Whitake (2016) it must be assessed whether:
a. The app’s download page, or the app itself, includes a description or a demo clearly specifying
and/or showing the pedagogic aims, the skills to be developed and the users’ type/level/needs.
b. There is coherence between the aims, the skills, and the target users.
c. There is coherence between the aims/users/skills and the app’s contents, resulting in a
potential high cognitive value

2.3 English Learning apps

2.3.1 Top 5 popular English-learning apps in Google play store
According to top apps ranking on Google playstore, these are the 5 most popular apps
relating English learning in the world, based on the editor choice from Google.

1. Memrise app
Memrise is a user-generated learning platform which uses flashcards as memory
aids. It specialises in language learning, but also offers content on a wide range of other
subjects. Memrise has more than 150 language courses across 25 languages. The app has
over 20 million registered users. Memrise has been profitable since late
2016(Wikipedia.org).

The google play has been listed some information about memrise app:

a. 15 million people are using the app.

b. There are 2 million words and phrases from 14 languages on the app.
c. Chatbots and native speaker videos, along with traditional flash cards can be
used to learn languages.
d. The app is rated at 4.7 with most users leaving five stars.
 Memrise operates on a freemium model. In the free mode you can:

a. Learn new words: The app will prompt you to learn new words which you
haven’t covered in the course yet.
b. Review words: You can review previously learned words through two modes:
a timed review mode called classic review and a gamified mode called speed
review.
c. Listen to pronunciation: Every time you pick a word, an automated voice will
play the correct pronunciation (this might not apply for all courses).

In the Pro version you can have access to other features like learning difficult
words, watching videos of other users talking in their native languages, and other features
like in-depth lessons using chatbots. On May 2017, Memrise was named as the overall
Best App winner of the second edition of the Google Play awards. The Awards celebrate
achievements of the developer community across the globe over the past year and
recognise the best apps and games covering twelve categories. Selected by a panel of
experts, the winners are evaluated on quality, innovation, and major launches or updates
over the past year. Not only in 2017, this app also was chosen as the best app for learning
language in 2016. There's a limited free version and a Pro version for $8.99/month
($59.99/year).
To begin using MEMRISE: LEARN LANGUAGES, choose a language and learn
about the background storyline of "protecting the diversity of languages in the
'Multiverse' from the evil Uniformis Front." After listening to a few words, you'll see a
multiple-choice screen on which you must choose the correct word in the language you're
learning to match the English word. Tap on the audio symbol to hear the word again on
any page. Continue this process through other screens that reinforce the words, including
video questions. Once you learn a certain number of words, you'll see your progress page.
Then you'll see the message "Next Up, Learn New Words." Tap on the right arrow to
begin the new words in the same learning process as the first group. Set the alarm so it
will remind you of your scheduled practice time daily.
This app uses a repetitive training method to build vocabulary in a new language
(or many new languages) in fun, easy ways. Memrise: learn languages uses audio and
 video of native language speakers to help users learn basic vocabulary. Some of the first
words and phrases that users learn are "hello," "let's go," "bottom's up!," and "yes."
Though the background narrative isn't central to the overall learning, it assists in tracking
levels in a fun way and keeps a theme going. The Pro version may be useful for app users
who have moved beyond the basic levels and are seriously committed to learning the
chosen language(s).

2. Learn 33 languages

Learn 33 languages or Mondly is an EdTech company that develops a fermium

language learning platform which includes a website and apps with free and paid courses
for 33 languages. The company also offers lessons in virtual and augmented realities of
2017, Mondly made available courses for 33 languages using localized content in 33
native languages. The available levels of language proficiency are beginner, intermediate
and advanced.(Wikipedia.org)

According to Google Play Store listing:

a. 1 million people are using the app.

b. There are so many language exercise for reading, listening, writing, and spekiang,
enhanced with a dictionary,
c. The app is rated at 4.7 with most users leaving five stars

3. Tandem language exchange: speak any language

The worldwide Tandem language partner community is made up of friendly,

enthusiastic language learners from all across the globe. No matter what language you
speak, or what language you’re trying to learn, there’s someone internationally who
wants to help you learn their native language. More than a billion people worldwide are
learning a foreign language - but miss out on speaking practice. Enter Tandem, the
mobile community where you can practice with native speakers and language learners
anytime, anywhere. On Tandem, you are matched with language exchange partners so
you can teach each other your native languages and commonly used slang.
 For example, you can learn Spanish from someone in Mexico City while they learn
English from you - without you even having to leave your bedroom.
With 150+ languages, more than 2500 language combinations and 1M + members,
there’s never been a better way to a learn a new language with native speakers.
The learn exchange community is completely free, and is based on the Tandem learning
method - a scientifically proven technique where two people meet in person to teach each
other their native languages, and of course, to allow a cultural exchange that you won't
find in any classroom. Tandem app is the first to bring this experience to everyone with a
smartphone.

4. Busuu

The Busuu app forms a part of a website (busuu.com) which is an extensive

online language community of over 25.000.000 users that offers language courses in 12
langugaes (busuu.com mobile apps). According to busuu.com the founders were tired of
the traditional way of learning languages which they thought was too expensive and
boring and so they decided to create their own learning material based on image and
sound and allowing learners to learn directly from native speakers and to offer parts of
the material for free.

The app does not offer all the functions and material that can be found on the
website but the material is none the less substantial and covers 150 topics related to
everyday life. The material is designed for beginners to advanced intermediate learners
(A1-B2) and includes 3000 words and key phrases.

5. Duolinggo

Duolingo: Learn Languages Free5 (henceforth Duolingo) provides “free language

education for the world” where learning is gamified. Players advance in the game by
unlocking bite-sized activities as they try to finish each lesson without losing their hearts,
the game’s progress indicator. The languages available depend on the players’ mother
tongue: if Portuguese, they can choose from English, Spanish, and French (and soon,
German), compared with more than 10 languages if the players speak English. During the
setup, students must “pick a daily goal” to plan their studies from a list that includes:
 ‘Casual’, 5 minutes a day; ‘Regular’, 10 minutes a day; ‘Serious’, 15 minutes a day; and
‘Insane’, 20 minutes a day. The game uses this setting to track the progress and send
occasional reminders if players stay out for too long. Players finish the initial settings,
defining whether they are beginners and will start at the ‘Basics’ level or they will take
the ‘Placement Test’. When players start a level, like ‘Basics 1’, they know how many
lessons it has and what the main content is.
The game’s content is organized in topics, which include Basics, Food, Objects,
Adjectives, Questions, Present, etc., which are further divided into lessons that work with
grammar and vocabulary, both as isolated language items and as parts of phrases and
sentences.
Each lesson offers a set of activities, including: unscrambling a sentence from oral
input, translating a phrase with scrambled prompts, translating a sentence from written
input, translating a sentence from oral input, translating a sentence from both written and
oral input, typing what you hear from slow or natural speed input, choosing the picture
and word for a vocabulary item, choosing all the right translations from a list, and
repeating a sentence from both written and oral input.

2.3.2 Criteria for English-learning apps

The easy availability of apps on mobile devices means that students are increasingly
turning to online resources for learning(Liu&He:2014) When encouraging learner on how
students how to the best use mobile apps for their English learning, it is important to emphasize
critical thinking. Unlike resources from the books, newspapers or television broadcasts,
information available on mobile apps is not regulated for quality or accuracy. When using
information online as an English resource, there are some criteria should be considered.
Practical criteria for English resources were designed by Tomlinson (2010), Some of
them are as follows:
1. English materials should contain enough spoken and written text.
2. Language input in materials should be contextualized.
3. Learners should be exposed sufficient samples of language in authentic use.
 4. English materials should include activities that provide critical thinking and encourage
learners to visualize.
5. English materials should include interesting and engaging tasks.
According to Howard and Major (2004), a series of specific guidelines was listed for
online English resources.
1. Online English resources should be stimulate interaction and be generative.
2. Learners are encouraged to develop learning skills and strategies when they use online
English resources.
3. Online English resources should link to each other to develop a progression of skills,
understandings and language item.
4. Online English resources have appropriate instructions.
5. Online English resources should be attractive and flexible.
Furthermore, updatability of apps is one of the most important criteria. In terms of
updatability, if students think there is a need for a change in the material depending on the needs
and level of what they have learned, they can easily adapt the material. It means that high
updatability can save their time, money and energy in the progress of learning English.

2.4 Language acquisition theories and learning-game apps

Since there is number of increasing internet in mobile learning, a large amount of projects
in mobile learning applications have been started. The investigation of those projects is to
identify the different learning theories and paradigms that underpinned in the application. The
purpose is will give educators and technical developers a deeper understanding of mobile
learning from a pedagogical perspective. The existing learning theories in relation to mobile
learning include behaviourism, cognitivism, constructivism, situated learning, problem-based
learning, context awareness learning, collaborative learning, lifelong learning, and informal
learning, etc. (Keskin & Metcalf, 2011). Naismith et al (2004) as cited in Guo have succinctly
identified six main theories and areas of learning relevant to learning with mobile technologies.
They are behaviourist, constructivist, situated, collaborative, informal and lifelong learning, and
learning and teaching support.
The previous paragraphs reviewed the main learning theories and paradigms applied in
mobile learning. They are behaviourist learning, constructivist learning, situated learning,
collaborative learning, informal and lifelong learning, and learning and teaching support as
summarised in Table 1. These selected learning theories and paradigms are by no means
mutually exclusive, but could provide a general theoretical background for analysing and
evaluating the current mobile apps for English language learning in the following chapters.
Learning Theories
Behaviourist learning
Constructivist learning
Situated learning
Collaborative learning
Informal and lifelong learning
Learning and teaching support
Guo(2014)
Definition
Learning occurs and
facilitated through the
appropriate reinforcement of
an association between a
particular stimulus and a
response
Learners actively construct
new concepts or ideas based
on their previous and current
knowledge.
Learning occurs within an
authentic context and culture,
and through a process of
social participation.
Learning is promoted,
facilitated, and enhanced
through social interaction and
collaboration between
students.
Learning occurs outside a
dedicated learning
environment and formal
curriculum.
Support for coordinating
learners and resources for
learning activities and
assisting administration duties
more generally
Activities
Information and content
delivery: drill and feedback,
test, practices, quiz, etc.
Questions for exploration,
problem solved and decision
making applications,
interaction between learners,
communication via mobile
phones, etc.
Authentic domain activity,
situated mentoring, workplace
learning, etc.
Active participation in social
context, communication
between peers via mobile
phones, etc.
Mobile information resources,
mobile website,
etc.
Effective personal
organization, attendance
reporting, reviewing student
marks, etc.
2.5 Evaluating Online Learning

Evaluating online learning is not different from assessing any other type of education
program. However, there are some challenges that may be face by evaluator when do assessing
an online learning (Horn:2008). Some unanticipated challenges come up when evaluators try to
investigate more about application. The problems may encounter like to the distance between
program sites and students, participants’ unfamiliarity with the technology being used, and lack
of relevant evaluation tools.
Online learning resources sometimes offer participants a wide range of learning
experiences. The applications offer some features for students or teachers with diverse interest.
The implication from diversity is evaluators can be dilemma finding effectiveness. Several
recurring themes can be found in the featured evaluations, as well as in the research and
comments from experts in the field of online learning (Horn:2008)
• Begin with a clear vision for the evaluation. Determine what you want the
evaluation to accomplish and what questions you hope to answer.
• Determine the most appropriate evaluation methods for meeting your goals.
Consider the different types of evaluations and the costs and benefits of each. What type
of evaluation is appropriate given your stated purpose? What research methods will best
capture program effects?
• Budget to meet evaluation needs. Limited budgets are a common barrier to
evaluators. When designing evaluations, consider whether there are available funds to
cover all planned data collection and analysis activities, plus the costs of any needed
background research, internal communications and reporting, and incentives for study
participants.
• Develop a program culture that supports evaluation. Discuss evaluation with
staff members, clearly explaining its value and their roles in collecting and analyzing
data.
• Communicate early and often with anyone who will be affected by the
evaluation
2.6 RETAIN model

Gunter, Kenny, and Vick(2007) proposed the design and evaluation model of educational
games, RETAIN. The design consists of six elements relevance, embedding, transfer, adaptation,
immersion, and naturalization. The model based on game and instructional design. This design
reflect Bloom’s concept of the nature of knowledge acquisition and question construction and
also the rubric combines Piagets’s thoughts. Each of element of RETAIN provides A frame of a
learning game’s strengths and weaknesses(Gunter et all:2016).

a. Relevance
Learning activities should be related to learner’s existing knowledge (Gunter,et
all:2007). Learning materials presented should be relevant to the needs of learning
style of learners. Educational games should create realistic learning contexts for
learners. The context should be familiar with it. It should be able to stimulate
learner’s interest, motivation and emotional responses
b. Embedding
The learning games consist of gameplay and learning strategies, in which
knowledge exploration and game are going at the same time (Gunter et all:2007).
The fantasy concept is a core element when designing a game. Learning content
should be integration of content and story. Learning goals should be divided into
sub goals at all levels, going from from simple to complex throughout the game
c. Transfer
The game should cause a circumstance in which the targeted knowledge is
transferred to similar or new situations and players can assimilate and
accommodate it. The learning of the targeted content should make the players
effective users of knowledge. In learning cycle, transfer and adaptation are closely
coupled. In game situations, learners may encounter similar problems to solve in
new contexts, which require them to reapply learned knowledge in variety of
forms.
d. Adaptation
Adaptation is a follow-on concept to transfer, which is from Piaget's ideas on
cognitive structural development. Learners develop their cognitive structure and
 skills through the process of assimilation and accommodation. In the usual sense,
the individual's cognitive processes are the balance of both. However, in real
world, individuals need to learn and adapt continuously, in which the both are
imbalance. Games can be used to balance them. Game is an adaptation activity
that an individual is doing in order to achieve the cognitive balance between
assimilation and accommodation.
e. Immersion
In the context of the game, that is, learning situation, immersion can be measured
hierarchically from a simple interaction to being fully engaged to an intellectual
investing(Gunter et all:2007). Skills and challenges are two important factors,
which must be in balance and drive oneself towards higher and more complex
level. If one can meet the following conditions: skills with the appropriate
difficulty; clear task objectives; fully concentrate and timely feedback, learners
can filter out all irrelevant perception and be fully engaged to an intellectual
investing when complete the task. The immersion experience will be helpful to
follow-on study.
f. Naturalization
Naturalization is from Bloom's psychomotor taxonomy and related to the concept
of automaticity and spontaneous knowledge. Learners can use existing knowledge
habitually and don't need to devote mental resources. So learners can devote
cognitive effort to high level thinking skills. The design of educational games
should be truly engaging, motivating, and immersing so learners can be immersed
in the fantasy and willing to play game repeatedly(Gunter et all:2007). As a result,
automaticity is developed. According to Vygotsky's "the Zone of Proximal
Development" and John Sweller's "Cognitive Load Theory", the more naturalized
the knowledge, the easier learners use knowledge in new situations.
Category 1: Relevance
Explanation: Learning material and its presentation should be relevant to the learner and
instructional units must be relevant to each other
Level 0 Little stimulus for learning
Level 1 Limited educational focus, some irrelevant content
Level 2 Learning objectives are defined, interest is created
Level 3 Game is relevant to learners and challenges or adequate for learning
Category 2: Embedding
Explanation: how close the academic content is coupled with the fantasy/story
Level 0 Learning content disrupts play
Level 1 Learning is exogenous to fantasy context (learning is “outside” the fantasy
context)
Level 2 Includes intellectual challenge and problems
Level 3 Content is endogenous to fantasy and fully involves learner
Category 3: Transfer
Explanation: The transfer of knowledge from one level to another
Level 0 No levels of challenge mapped to objectives
Level 1 Levels of challenge are too similar, some useful content
Level 2 Easy progress through levels through active problem solving. Higher level
knowledge should be
transferable
Level 3 Authentic real life situations and after action reviews
Category 4: Adaptation
Explanation: Refers to learners being forced to change or create new knowledge to deal with
something that does not fit existing ideas and understanding
Level 0 Fails to engage in interactive, unstructured information
Level 1 Builds upon existing cognitive structures, engages in cognitive conflict
Level 2 Learners are encouraged to go beyond given information. Old schemas are
identified and adapted
to new situations
Level 3 Learning becomes an active process that integrates prior knowledge
Category 5: Immersion
Explanation: the learners intellectual investment in the context of the game
Level 0 No formative feedback, little active participation
Level 1 Elements of play are not in sync with learning objectives, players do not feel
fully interactive
Level 2 Learners are involved cognitively, physically and emotionally
Level 3 Favours belief creation and includes opportunities for reciprocal action
Category 6: Naturalisation
Explanation: how well learners develop automated or spontaneous use of information
Level 0 Little opportunity for mastery of facts and skills
Level 1 Replay is encouraged to improve speed of processing
Level 2 Encourages synthesis of elements and judgments
Level 3 Learners become efficient content users and spontaneously use acquired
knowledge
Table 2. Summary of RETAIN rubric (Gunter et al.,2007)

For the second weighting, the aspects are ordered from the least to the most important.
These aspects are relevance, immersion, embedding, adaption, transfer and naturalisation. In this
case, points are awarded as more proof of that element’s construct is present, according to the
specifications provided in the specific cell of Table 2. The weighting chart is shown in Table 3.
Order of importance Level 0 Level 1 Level 2 Level 3
Relevance 1 0 1 2 3
Embedding 2 0 3 6 9
Transfer 3 0 5 10 15
Adaptation 4 0 4 8 12
Immersion 5 0 2 4 6
Naturalization 6 0 6 12 18
Table 3: RETAIN weighting chart (Gunter et all:2007)

2.7 Previous Studies

In Gunter (et all:2016), the finding indicate that although educational game Busuu and
Duolingo scored low on the RETAIN model, these language learning applications have a place
in learning environments. Their contributions to introducing and practicing vocabulary and
grammar contribute to language learning. According to the RETAIN model, Busuu and Duolingo
strengths and included Transfer, Adaptation, and Naturalization. Learning levels are sequenced
and integrated, allowing learners to use the knowledge developed. Busuu has a social aspect. It
can practice what they have learned through questions and answers and then receive peer
feedback in the writing. The challenges are organized not only to allow players to apply previous
content knowledge.
 The other references, the RETAIN model has been used to evaluate console games and
mobile learning apps (Campbell et all:2015) because it explains how well a game includes
academic knowledge and transfer as a standalone approach to learning. Educators and computer
science experts alike find the model to be useful for selecting serious games as instructional tools
(prinslo&Jordaan:2014) The RETAIN model weighs the strengths and weaknesses of a game for
a teacher to become well informed as how to best use game in learning.
 CHAPTER 3
METHODOLOGY

3.1 Method
This research will be conducted by using quantitative method. As Harwood (2010)
explains, quantitative analysis of materials involves identifying content categories and
calculating the percentage of space devoted to each category. Features of app were qualitatively
analysed and noted down.

3.2 Design
This research will be used content analysis as the research design for analyzing because it
could most effectively review the result of evaluation and also learning theories inside of
memrise application.Weber (1990) indicates that “Content analysis is a research method that
uses a set of procedures to make valid inferences from text”. As simple as that may sound, it
must be noted that content analysis includes a wide range of techniques and applications in the
social sciences

3.3 Source of Data

The data that will be used for this research will be taken from memrise application in
google play store.

3.4 Data
The data from this research is the content from the application itself and also theories
which are underpinned that used in presenting materials in app.

3.5 Instrument
The instrument for this research will be taken a rubric which proposed by Gunter et all
(2007), RETAIN rubric model. The validity and reliability have already checked by previous
researchers since the instrument will be adopted by the current researcher. The scale that will be
used is weighting scale that has already decided by Gunter et all(2007).
 3.6 Technique of Data Collection
The data in this research will be collected by reading, observing and identifying directly the
memrise application. The process of data collection will be conducted through direct observation
and identification of the content because they were directly visible and objectively identifiable.
All sample data in descriptive study are mainly collected by reading, analyzing, identifying and
presenting systematically
An analysing data of memrise application the writer used the data collected from memrise
application, and the rubric assessment from Gunter et all(2007) RETAIN model, the analysis will
be done using these following steps:
1) Comparing the material presented in the application with the category of required
characteristic of an application by rubric assessment from RETAIN model,
2) Evaluating material in the application “Memrise” by giving score in respected category listed
in the rubric assessment and also evaluate based on some learning theories
3) Interpreted the data gained from the process of evaluation.
 References

Anderson, I. (2013). mobile apps for learning english (Master's thesis, sigillum universitatis
islandiae, Reykjavík, iceland).
Arancon, P. R., Arus, J., & Calle, C. (2013). the use of current mobile learning applications in
efl. social and behavioral science, 103, 1189-1196. doi:10.1016/j.sbspro.2013.10.446
Gee, J. P. (2003). What video games have to teach us about learning and literacy. Computers in
Entertainment, 1(1), 20. doi:10.1145/950566.950595
Gunter, G. A., Campbell, L. O., Braga, J., Racilan, M., & Souza, V. V. (2016). Language
learning apps or games: an investigation utilizing the RETAIN model. Revista Brasileira
de Linguística Aplicada, 16(2), 209-235. doi:10.1590/1984-639820168543
Gunter, G. A., Kenny, R. F., & Vick, E. H. (2007). Taking educational games seriously: using
the RETAIN model to design endogenous fantasy into standalone educational
games. Educational Technology Research and Development, 56(5-6), 511-537.
doi:10.1007/s11423-007-9073-2
Guo, H. (2014). analysing and evauating current mobileapplications for learning english
speaking (Master's thesis, Birkbeck, London, England).
Horn, S. K. (2008). Evaluating online learning: challenges and strategies for success (ED-01-
CO-0012). Retrieved from WestEd website:
http://www.ed.gov/admins/lead/academic/evalonline
Howard, J. & J. Major. 2004, Guidelines for designing effective English language teaching
materials, Available from
http://www.paaljapan.org/resources/proceedings/PAAL9/pdf/Howard.pdf
Liu, Q. (2014). usingmobile apps to facilitate englishlearning for college students in
chine(Master's thesis, unversity of boras, boras, swedish).
Prinsloo, J. W., & Jordaan, D. B. (2014). Selecting serious games for computer science
class. mediterranean journal of social sciences, 5(21), 391-398.
doi:10.5901/mjss.2014.v5n21p391
Tomlinson, B. 2010, Principles of effective materials development, In N. Harwood (ed.), English
language teaching materials, pp. 81-108, NY: CUP
Weber, R.P. (1990), Basic Content Analysis, Newbury Park, CA: Sage.

Whitaker, J. T. (2016). evaluating the effectiveness of computer applications in developing

english learning. international journal of environmental & science education, 11(18),
12947-12958.
Zhang, H. F., Fan, X. Y., & Xing, H. F. (2010). Reserach on the design and evaluation of
educational games based on the RETAIN model. international symposium on knowledge
acquisition and modeling, 10, 375-378.