IGUAL Project - State of The Art

Analysis of State-of-the-Art Solutions for Personalised Learning Support
IGUAL Innovation for Equality in Latin American University (Innovacin para la iGualdad en la Universidad de America Latina) DCI-ALA/19.09.01/10/21526/245-315/ALFAHI (2010)123) Alfa III programme
TABLE OF CONTENTS 2 3 12 22 32 From the Editors Learning/teaching a Computer Programming Course. Camilo Jimnez, Jorge Villalobos Localization of Learning Objects. Cristian Cechinel, Sandro da Silva Camargo. Sharing and Recommending Learning Materials. Xavier Ochoa. Mental Models, Strategies and Metacognition in Programming, and its relation to Verbal Protocols as a learning mechanism. Carlos Argelio Arvalo Mercado, Lizeth Itziguery Solano Romo Learning Designs: Concepts, Standards and Software Tools. Luis A. lvarez-Gonzlez, Erick A. Araya Araya and Jorge Morales Vilugrn
42
Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support
From the Editors

This report is part of the project IGUAL Innovation for Equality in Latin American University (Innovacin para la iGualdad en la Universidad de America Latina), financial supported by EuropeAid Cooperation Office Latin America. The main objective of this project is to create and validate innovative and contextualised solutions to reduce the knowledge and skill gap between private- and public-educated students. In this report, each Latin American partner, with the help of European partners, analyzes the State-of-the-Art for personalised solutions support. The universities partners in this project are: Tampere University of Applied Sciences - Finland (coordinator), Fundao Universidade Federal Do Pampa - Brazil, Universidad Austral de Chile Chile, Universidad de los Andes - Colombia, Escuela Superior Politecnica del Litoral Ecuador, Universidad Autnoma de Aguascalientes - Mexico and Universitatea Tehnica de Constructii Bucuresti Romania. The project IGUAL is focused to computer programming education in higher education. The programming skills is not only relevant for computer sciences and engineering, also is relevant for other disciplines where the teaching strategies could be different. Jimnez, et al., review some holistic approaches to support the teaching/learning of computer programming. If the Learning Technologies will be used to contextualised/personalised solutions, then the Learning Objects will be important elements of this solutions. Given the large dissemination of Learning Objects and its reusability, Cechinel & da Silva analyze the localisation and internationalisation of these. To complement the localisation and internationalisation, Ochoa presents a survey of current technologies to publish, social share and recommend learning material. The social sharing is through Learning Objects Repositories and the recommenders for learning material are presented. Every student is different to each others; however, the most of the teaching strategies do not consider these differences, or uses strategies from others contexts. In otherwise, we need to know the students to find the best personalised solutions. Argelio & Itziguery analyzes the most recently studies for Mental Models, Strategies, and Metacognition in basic programming in order to establish a link with Verbal Protocols to improve in the students the programming skills. Finally, personalised learning process will be one of the solutions for the project, so lvarez-Gonzlez, et al., describes the concepts of Learning Designs as a way to formalize the learning process, analyze the standard IMS LD and the three main software tools that implements this concept are also presented. Editors Esa Kujansuu Tampere University of Applied Sciences Luis A. lvarez-Gonzlez Universidad Austral de Chile Ion Mierlus-Mazilu Universitatea Tehnica de Constructii Bucuresti
IGUAL Innovation for Equality in Latin American University
Page 2
Learning/teaching a Computer Programming Course

Camilo Jimnez, Jorge Villalobos
Department of Computer and Systems Engineering, Universidad de los Andes Bogot, Colombia. camil-ji@uniandes.edu.co, jvillalo@uniandes.edu.co Abstract. It is commonly agreed that Computer Programming Education is a challenge for higher education programs worldwide. Although this is especially true for Computer Science (CS) programs, the development of programming skills has become relevant for many other disciplines such as engineering, mathematics, physics and cognitive science. Additionally, the programming education challenge is made even more complex by its dynamic nature (e.g. the constantly evolving technology) and the necessity to face it from the perspective of the different stakeholders involved, each of them with their own requests. Important efforts have been made around the work on students motivation, the kinds of teaching strategies as well as diverse plans for knowledge management; however treating all these as isolated strategies has not shown permanent improvements. This way, some researchers have designed and implemented holistic approaches to support the teaching and learning of computer programming. In this paper, we review some of them and conclude in a set of principles that a holistic learning approach should fulfill.
Key words: Programming, Active learning, Course design.
INTRODUCTION
For the last twenty years, several challenges in Computer Programming Education (CPE) have been identified (Woodley, 2007), most of them grounded in indicators such as high drop-out rates, students low motivational sense, and low course grade point averages (Biggers, 2008) (Denning, 2005) (Jenkins, 2001) (Ma, 2008) (McCracken, 2001) (Robins, 2003). This situation has conducted Computer Science (CS) institutions to consider alternative pedagogical approaches (Gearailt, 2002) (BlueJ, 2008) (Robocode, 2008) (Alonso, 2009) (Conole, 2009) (Scheard, 2009). Unfortunately , rather than finding a sustainable solution that satisfies students, teachers, and CS institutions in the same way, these approaches have focused in solving specific requirements of isolated challenges to each of these stakeholders. This becomes a problem itself since their role in CPE is deeply related. As an example, neither it is sufficient to design and implement a tool for increasing the students motivation if it is not aligned with the teachers personal strategies, nor designing a way to improve teaching strategies without considering the students attitude to those strategies. Every decision towards solving specific challenges in Computer Programming Education is extremely related with other kind of challenge. This way, solutions to CPE challenges must first consider and understand all of them in an integrated way. On this behalf, it is firstly essential to comprehend that Computer Programming Education involves different stakeholders each one with different needs. Contrary to conventional teacher-centered approaches, which have been found to be not motivating and not effective for students (Lea, 2003), a good educational model should not focus exclusively on teachers and their process of teaching (Govindasamy, 2002). It should also concentrate on students and their learning process (Alonso, 2010), as well as on CS institutions and their process of managing knowledge around computer programming courses (Govender, 2009). It is important to recognize that Computer Programming Education is also highly dynamic. Along with technology, computer programming languages, tools, techniques, and even computer programming paradigms are constantly changing and evolving (Robins, 2003). Since conventional approaches are focused on covering programming fundamentals (Woodley, 2007), this changing situation has generated several difficulties due to the significant and continuous amount of effort required from teachers to keep up-to-date with the latest available technology (Villalobos 2009a, 2009b). Computer Programming Education is also massive. Just at our university, basic computer programming courses (CS1, CS2 and CS3) receive approximately 1,200 students per semester hiring more than 40 teachers including
Page 3

graduate assistants that practice teaching as part of their professional growing. This situation makes knowledge management around computer programming courses a very challenging task (Villalobos, 2009a). In conventional approaches, this complex task is left to teachers, often with different academic profiles, who have typically developed their own personal strategies (Villalobos 2009a). This way, different classroom lectures and laboratory practices that may fit the approaches adopted in the CS courses are prepared individually in a context characterized by a poor culture around networking practices between teachers and a lack of implanted reflection mechanisms to encourage them to share and validate results of their implemented strategies or practices, or even learning objects and materials. In consequence, CS institutions have shown difficulties when managing and supporting knowledge generated in these courses, which generally depend heavily on the instructor abilities rather than a clear pedagogical model (Villalobos, 2009a). Last but not least, it is necessary to comprehend that Computer Programming Education has turned to be fundamental and universal. Due to the widespread use of technology across the world, computer programming constitutes one of the core competencies of a graduate from the engineering discipline (Law, 2010) and it is currently included in several curriculums other than Computer Science, such as Math, Physics, other Engineering programs, and even Arts and Humanities. Accordingly, current approaches must understand that students in computer programming courses have different and heterogeneous backgrounds as well as particular learning styles and a variety of motivations when learning computer programming (Govender, 2009). Unfortunately, conventional approaches have been primarily focused on covering programming fundamentals rather than on generating abilities relevant to graduates for their different professional domains. This way, the development of abilities to understand and abstract problems, analyze abstractions, model solutions and to build those solutions using technological tools in different contexts have been left aside, expecting that writing many programs will lead to their development (Woodley, 2007).
LEARNING SOLUTIONS IN COMPUTER PROGRAMMING

There are thousands of articles that explain different learning solutions that claim to overcome different challenges in learning and teaching computer programming. While it is very difficult to summarize them completely in this literature review, it is possible to discern patterns in the way each author designed each learning solution. For instance, a great number of authors ground their learning solutions in a deep examination of the factors that affect the learning process of computer programming students. Articles following this trend discuss why and how the learning process occurs by analyzing distinct variables extracted from the perspective of students. Authors claim that the way students acquire knowledge has to be previously understood in order to propose adequate solutions that can make their learning process effective and overcome challenges around teaching and learning programming. In this manner, an outstanding work is described in (Law, 2010) where the authors state that computer programming learning is mostly affected by one important factor: Motivation. Even more, they discern that learning is not likely to happen unless students are adequately motivated. To support this statement, a preliminary study was conducted in order to analyze internal and external factors that determine motivation among students that take computer programming courses. The study suggests that these factors indeed have a very strong correlation with students efficacy to learn. With this in mind, the authors created an e-learning framework called PASS (Programming Assignment Assessment System) which objective is to facilitate students learning by enhancing key motivating factors. To do this, the system provides an infrastructure that allows students to learn computer programming through a series of steps involving automatic feedback mechanisms while they build a program. Though it is shown that PASS enhances learning motivation and self-efficacy, a deeper study on how to integrate this tool with pedagogical objectives would strengthen their approach. As it is mentioned within the article, it is not clear how the tool can accommodate to the various teaching strategies or the different learning contexts of students. Nonetheless, the exhaustive examination of motivation factors give educators insights on the advantages of well facilitated e-learning settings. A more extensive study on factors that affect the students learning process is described in (Govender, 2009). In this article, the authors expand their analysis from just motivation to the general learning context of students. They not only consider key motivation factors, but also factors that derive from the institutional context of students (i.e. the type of class: face-to-face learning, or distance learning) as well as their personal context (i.e. their aptitude, learning style, motivation, or programming experience). Authors argue that students experience of these contexts have important implications for teaching and learning. To support this, they performed a quantitative and qualitative study that highlights the influence of different learning contexts among students learning. Valuable conclusions are reported in this article suggesting that the learning context has to be highly considered in order to design effective
Page 4

courses on computer programming. In addition, pragmatic opinions from instructors supporting their argument are documented. Again, though this study validates the impact of external factors in students learning processes, a deeper investigation is needed to design a learning solution that enhances students efficiency to learn based on its results. Although authors of (Law, 2010) and (Govender, 2009) do not focus in technology, it is commonly agreed that it represents a very important factor that affects the students learning process. New innovative and engaging elements such as multimedia animations, distributed communication tools, and online curricula have been adopted by several institutions trying to make teaching and learning more effective. Considering this, authors in (Li-Ping Tang, 2009) performed a rigorous study on the students perceptions to technology. Particularly, they aimed at indicating how to adequately use technology in the classroom based on the attitude of students towards different teaching technologies. They contrast four objectives, namely enjoyment, learning, motivation, and career application, across five teaching technologies, namely projector, power point, video, Internet, and lectures. Overall, they found that technologies using video delivered the highest amount of enjoyment in class, while using power point yielded in an increase of learning and motivation. Since this conclusion partly contradicts former research in active learning where lectures and the inappropriate use of power point are discouraged, the authors attempt to conclude in a more general way, that professors may use a different mix of technologies in the classroom and use them creatively to promote the most learning for students. In other words, the authors indeed agree that technology enhances learning in students. Nonetheless, they do not analyze how to use technology in order to accomplish it. In this manner, the term visualization software recalls attention. It has emerged in the 1980s as algorithm animations to later become a technology used to illustrate graphical representations of several computer science concepts (Naps, 2002). Instructors have found these tools to have a valuable impact in their class sessions, leading to the development of several analysis and studies to both classify them and find their effectiveness within the students learning process (Myller, 2007). The educational impact behind visualization technology (VT) has allowed its classification after several criteria. One of the first approaches to classify VT within a taxonomy is found in (Crescenzi, 2007), where the authors identified eleven elements as standard tools present in algorithm animations. A later approach, and one of the most prominent, is presented in (Naps, 2002). In this work, Naps et al. propose what they call the Engagement Taxonomy (ET). This taxonomy classifies visualization technology according to different forms of interaction between this technology and learners. Six levels are identified: No Viewing, Viewing, Responding, Changing, Constructing and Presenting. In addition, the ET provides an overview of how these different forms of interaction result in different understanding levels according to Blooms taxonomy. Additionally, a framework for experimental studies of visualization effectiveness is also proposed in (Naps, 2002). This framework allows computer science educators to find and measure the relation between a visualization element and the consequent understanding levels (considerer in Blooms taxonomy) that should be reached. Several authors have carried studies within this framework (Crescenzi, 2007) (Myller, 2007) and their findings have ended in positive results that evidence the necessity of an appropriate learning environment for both students and instructors. Regarding the use of technology and its relation to how students acquire knowledge, several online social networks have appeared in recent learning environments, both from academia and the private sector. In academia, one of the most prominent is the Cupi2 Community, an online social network developed to promote networking practices among computer programming teachers and to support relevant processes related to teaching programming under active methodologies: the teachers adaptation process to the new methodologies introduced by Cupi2; the process of planning the instruction of CS courses; the design of the required learning resources and the process of supporting the instruction of these courses; and a structured process of capturing feedback from teachers experiences (Villalobos, 2009b). The community provides support, policies, and tools for each of these processes (see FIGURE 1). The community also provides mechanisms that enable members of the CS community to share and reuse learning resources and to exchange knowledge and experiences around the teaching of computer programming. It also empowers departments with the ability to manage knowledge and to gain experience and facilitates the tracking and evaluation of dynamic results. On the other hand, private social networks often focus on sharing and connecting rather than directly supporting the process of learning. An example of such online networks is the TeachingToday website (TeachingToday, 2008). As the developers of the website are aware that online communities provide a wealth of opportunities for educators to share teaching strategies, ideas, and best practices, they developed an online warehouse of free tips, tools, and resources that are easy-to-use and pedagogically sound. These elements can also be found in the National Geographic Education Network (NatGeo, 2008). This community includes information about professional development opportunities such as workshops and online events that members can attend.
Page 5
FIGURE 1. Cupi2 Community structure (Villalobos, 2009a).
The interrogative of how to manage these factors adequately in a learning solution has motivated the emergence of various learning theories. These theories are usually the base of the solutions we call holistic in Computer Programming Education. The most popular of these theories is called Active Learning. According to (Robins, 2003), active learning suggests focusing not on the instructor teaching, but on both the student learning and the effective communication between the two of them. On the one hand, the main goal is to foster deep learning of principles and skills, and to create independent, reflective, life-long learners. On the other hand, the methods involve clearly stated course goals and objectives, stimulating the students interest and involvement with the course, actively engaging students with the course material, and appropriate assessment and feedback. Another popular learning theory is Problem-based Learning (PBL). PBL requires students to become responsible for their own learning. The PBL teacher is a facilitator of student learning, and his/her interventions diminish as students progressively take on responsibility for their own learning processes. This method is characteristically carried out in small, facilitated groups and takes advantage of the social aspect of learning through discussion, problem solving, and study with peers (Hmelo-Silver, 2004). The facilitator guides students in the learning process, pushing them to think deeply, and models the kinds of questions that students need to be asking themselves, thus forming a cognitive apprenticeship (Collins et al., 1989). In PBL this facilitator must be able to model good strategies for learning and thinking, rather than providing expertise in specic content. This role is critical, as the facilitator must continually monitor the discussion, selecting and implementing appropriate strategies as needed. As students become more experienced with PBL, facilitators can fade their scaffolding until nally the learners adopt much of their questioning role. Student learning occurs as students collaboratively engage in constructive processing. Both active learning and PBL are widely used in approaches that aim to support the teaching and learning of computer programming holistically. Particularly, authors in (Villalobos, 2006) propose a learning solution called Cupi2, that is based in an active learning pedagogical model that is grounded in four main components that allow the construction of a balanced solution (FIGURE 2). These components (see FIGURE 2) engage students as the main role in the learning process. Firstly, by working as an Active Learning approach, teachers in this approach act less as instructors and more as promoters of activities that ensure the generation of relevant skills to analyze problems and model solutions. Secondly, by using Problem Based Learning, students continually face problems that reflect real world challenges. In this way, concepts on a subject are explained through their relation with specific parts of these problems. The third component in the model is Incremental Learning, whereby students are able to generate skills and acquire knowledge distributed in separated levels. Cupi2 structures 18 levels from the beginning of CS1 to the end of CS3. The levels allow the instruction of CS concepts from the fundamentals to gradually more complex structures. In this way, students continuously reinforce knowledge and skills developed incrementally through each level. The last component is the Learning by Example approach in which students have access to examples of best practices and common solutions.
Page 6
FIGURE 2. Cupi2 pedagogical model (Villalobos, 2009b).
As an active learning approach, there are 3 relevant topics that must be considered in order to perform learning. Firstly a learning environment is established incrementally following 18 levels of knowledge. Thus, students are confronted with a frequent reinforcement of previous subjects. Secondly, activities of different kind such as: classroom activities, extraclass activities, collaborative and individual laboratory practices, and homework asignments, are defined to guide students in a constructivist path to learning programming. Thirdly, since each of these activities needs to be setup differently, an appropriate set of learning objects (artifacts built to support activities) was developed. Learning objects inside Cupi2 were classified considering the kind of work they support. In this way teachers are able to design lecture classes or labs using the most appropriated resources. Cupi2 was successfully evaluated (Villalobos, 2006) (Villalobos, 2009a) (Villalobos, 2009b) and it has conducted the project to be recognized by important regional institutions in two different occasions. In the first occasion, Cupi2 was awarded the 2007 Colombian Informatics Award by the Association of Colombian Computer Engineers (ACIS), based on the quality of its learning objects and its academic impact in more than 30 universities in Colombia. In the second occasion, Cupi2 obtained the first place in the 10th prize of Educational Informatics 2009 by the Iberoamerican Network of Educational Informatics (RIBIE), based on its academic and research quality, its social incidence, and the number of students and faculty members benefited from it. In (Alonso, 2006) an instructional model that combines three learning theories is proposed. These learning theories, namely behaviorism (Good & Brophy, 1990), cognitivism (Anderson, 1996), and constructivism (Jonassen,1991), form the base of what they call a moderate constructivist e-learning pragmatic model that is based in six cognitive principles (Alonso, 2009). Firstly, if a visual structure is provided and maintained throughout the courses, students will be able to differentiate and identify the content type and the perception processes will be automated. Secondly, when designing instructional contents, methods that optimize the attention process by easing the selection of important material should be included. Formulating the right questions, clearly establishing learning objectives, and addressing students personally and informally improves their conscious attention and, consequently, the effectiveness of the working memory. Thirdly, methods should be used that reduce the foreign cognitive load not related to the content type. For example, incrusting text in pictures (principle of contiguity) reduces the effort required to retain the information presented in the text and then locate it in the picture. Fourthly, information presented as text and illustrations can be better recalled than information that is presented as text only. The use of conceptual maps and process diagrams is recommended for the adequate construction of mental models. Fifthly, to improve learning, contexts with which students are familiar should be provided. As coding specificity determines the retrieval process, the examples should be as close as possible to the context to which the learning is to be transferred. Finally, it is important to further the knowledge that students have of their own thought processes by means of strategies of reflection. Checklists can be used, as well as forums and dialogues with tutors to check what they have learned and correct mistakes. The results of applying the model in (Alonso, 2006) indicate that, while it is true that the effort required on the part of the learner in terms of learning time is higher for distance learning with the proposed instructional model, the overall performance and level of learner satisfaction is similar to courses taught face-to-face in a classroom, whereas it has the advantage of cutting costs considerably (Alonso, 2009). Similarly to (Alonso, 2009), Moreno et al propose a constructivist and collaborative methodological learning approach in (Moreno, 2005). The methodology is supported by both a Moodle platform and a simulator called SIMDE that provides an interactive environment for students to learn. Above all, their proposal is based on a theoretical framework that states some procedures to follow when adopting their model. Firstly, as an alternative to memorization, learning contexts should be based on constructivism. This permits to build knowledge, doing
Page 7

activities closer to the real world and generally involves discussion groups. Secondly, the learning environments must be flexible and are characterized by the fact that the same knowledge can be represented in different ways. Thirdly, instead of having a passive role in learning, computers must act as supporter tools for the experimentation and building of knowledge in students. Fourthly, application to specific instructive situations of constructivism and mediation of learning should be done through computers and people. Through the individual exploration, the student should acquire determined general schemes of knowledge. Fifthly, instructors should act as a guide besides computers. It is essential to define the type of intervention of other people: Teacher and students. Finally, the learning environment should encourage a communitarian or collaborative learning, where the students work together helping each other, reinforcing the social dimension of the education. Through the activity and the experience, the best results are obtained. This learning approach has been tested by students from last year of a Computer Engineering degree in University of La Laguna. With respect to the learning experience with the proposed methodology, the students have shown a deeper knowledge as well as a higher level of motivation, difficult to find in this type of courses.
CONCLUDING REMARKS
From the previous state of the art, we conclude that current challenges in Computer programming Education (CPE) has led traditional educational methods to change towards a pedagogical culture more focused on learning instead on teaching, where the students are taking more responsibility for their own learning. In general, there is a growing interest in reflective teaching and learning, moving the focus from a procedural approach, to thoughts, reflections and conversation for gaining understanding and perception of the real world. Instead of stimulus and response, there should be insight and action. With this in mind, as a concluding remark we describe a set of principles we think must be accomplished in a current learning solution. Principle 1. Due to the fundamental and universal nature of CPE, programmers need to understand that problem solving is not necessarily limited to technology troubleshooting, which relies only on knowledge and experience. There is also a need for them to understand that complex problems are usually systemic and chronic in nature and may require a logical organized and systematic approach to their solution. (Felder, 2000) and (Rugarcia, 2000) noted that it is becoming apparent that there is an increasing need to improve the quality and focus of engineering education. Apart from acquiring technical content knowledge and skills, students require competence in the applications of engineering design and operations and vital skills such as systems thinking and problem solving. Our first principle states that a learning solution should be focused on developing high level learning skills (See FIGURE 3) rather than on specific contents around some particular knowledge. In fact, these thinking skills help students to construct knowledge more effectively in the end.
FIGURE 3. Cognitive aspects of programming and their relation (Villalobos, 2009b).
Principle 2. (Felder, 2003) noted that the only way high-level skills are developed in students is practice. But to make students practice, learning must shift its focus from the teacher to the student. Active learning enable the student to
Page 8

take responsibility for learning and acknowledge the constructivist views of learning, emphasizing that the student is the only one who can be responsible for learning. Teachers can do a lot to facilitate learning, but they cannot take over student learning. The more involved students get in their own learning process, the more likely they are to achieve their goals and succeed. In other words, teachers should be responsible of organizing what is to be learned in the hands of the learners themselves (Dodge, 2003), thus challenging students to engage in high level thinking skills, instead of passively absorbing material. Principle 3. Bonwell and Sutherland (Bonwell, 1996) identified that one major barrier to implementing active learning in the classroom is the coverage problem. Teachers are concerned about how to cover all or most of the syllabus in an adequate way and considering not only covering a great set of contents, but ensuring the best retention of knowledge and the optimal practice of high level skills. Regarding this barrier, learning solutions should use problem-based learning (PBL) environments so that learners are presented with opportunities to engage in complex tasks that would otherwise be beyond their current abilities. Together with scaffolding techniques, PBL makes the learning more tractable for students by changing complex and difficult tasks in ways that make these tasks accessible and manageable. Particularly, scaffolding has been conceived as a key element of cognitive apprenticeship, whereby students become increasingly accomplished problem-solvers. In fact, an important feature of these environments is that they support students learning of both how to do the task as well as why the task should be done that way (Hmelo-Silver, 2006). Scaffolding depends in a clearly defined structure of the problem, and the guidance from teachers who, without explicitly giving students the nal answers, coach them and organize their learning process to reach a given solution. Principle 4. In active learning environments, students need to be actively involved in learning activities. Actively means freedom to explore materials based on personal interest and motivation, as well as freedom to make choices with regard to methods and assessment. Learning environments must be exible and characterized by the fact that the same knowledge can be represented in dierent ways. This way, students can learn through a variety of proposals that could vary from giving students a task such as a question or problem to solve, an original example to develop, or to work in collaborative groups, reviewing sessions, games, videos, debates, or case studies. Several strategies that promote active learning have been similarly shown to influence favorably students' attitudes and achievement. Visual-based instruction, for example, can provide a helpful focal point for other interactive techniques. Two popular instructional strategies based on problem-solving model include the case study method of instruction and Guided Design. Other active learning pedagogies worthy of instructors' use include cooperative learning, debates, drama, role playing and simulation, and peer teaching. In short, the published literature on alternatives to traditional classroom presentations provides a rich menu of different approaches faculty can readily add to their repertoire of instructional skills. Principle 5. Teachers agree to find that they incur in increased preparation time and some of them posses limited resources to implement active learning in their classrooms. Defining the appropriate materials to support active learning activities considering the vast and rapidly changing environment around programming represents a very challenging issue for several reasons. First, reference books around programming are very diverse in scope and complexity levels. For this reason, it is very unlikely to find one book that fits within the complete set of concepts defined in a course. On the other side, finding several books, each one specialized in certain group of concepts, would confuse students since methodologies and learning activities defined in each one of them are different and various. Second, public tutorials and learning objects available on the Web usually lack of an academic validation and most times they are not adequately aligned with the development of programming skills. Additionally, some of them are subject to copyright norms and others are simply not continuously updated. Considering this barrier, in such an active learning environment CS departments must support the process of designing, defining, planning, identifying, searching, implementing, validating and delivering learning objects as well as facilitating the learning activities that faculty plans in a course. Principle 6. Though freedom to make choices with regard to methods are some elements of active learning, students learning assessment has to be similar between them regardless the course section in which they are enrolled. Instructors should have a similar evaluation model so that students can accomplish their learning objectives in a consistent manner. Additionally, the mechanisms to implement this evaluation model must be scalable to them as well.
Page 9
Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support ACKNOWLEDGMENTS

The work presented here has been funded by the European Commission through the project entitled IGUAL Innovation for Equality in Latin American University (code DCI-ALA/19.09.01/10/21526/245-315/ALFAHI (2010)123) of the ALFA III programme.
REFERENCES
Alonso, F., Manrique, D., Vies, J. A moderate constructivist e-learning instructional model evaluated on computer specialists Computers and Education 53 (2009) 57 65. Alonso, F., Lopez, G., Font, J. M., Manrique, D. Learner satisfaction when applying an instructional model in elearning. Proceedings of the 2nd International Conference on Computer Supported Education (CSEDU), Spain, April 2010. Anderson, N. H. (1996). A functional theory of cognition. New Jersey: Lawrence Erlbaum Associates. BlueJ. The Interactive Java Environment, available at: http://www.bluej.org/. (Accessed May 28th 2008). Bonwell, C. C., and Sutherland, T. E., The Active Learning Continuum: Choosing Activities to Engage Students in the Classroom, in Sutherland, T. E. and Bonwell, C. C., (Eds.), Using Active Learning in College Classes: A Range of Options for Faculty, New Directions for Teaching and Learning, No. 67, Jossey-Bass, San Francisco, CA, 1996 Felder, R. & Brent, R. (2003). Learning By Doing. Chemical Engineering Education, 37(4), 282 283. Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction:Essays in honor of Robert Glaser (pp. 453-494). Hillsdale NJ: Erlbaum. Conole G., Culver, J. The design of Cloudworks: Applying social networking practice to foster the exchange of learning and teaching ideas and designs Computers and Education 54 (2009) 679 - 692. Crescenzi, P., and Nocentini, C. Fully Integrating Algorithm Visualization into a CS2 Course. A two-year experience. ITiCSE07 Proceedings, Dundee, Scotland, (June 23-27) Denning, P. J., and McGettrick, A. 2005. Recentering computer science. Communications of the ACM, 48, 11, pp1519. Dodge B., Active Learning on the Web, San Diego State University, URL: ttp://edweb.sdsu.edu/people/bdodge/active/ActiveLearningk-12.html, accessed August 7, 2003 Felder, R.M., Woods, D.R., Stice, J.E. & Rugarcia, A. (2000). The future of Engineering Education Teaching Methods That Work. Chemical Engineering Education, 34(1), 2639 Gearailt, A., 2002. Teaching with Java: Using Java to increase active learning in programming courses. Proceedings of the inaugural conference on the Principles and Practice of programming, Dublin, Ireland 2002. Good, T. L., & Brophy, J. E. (1990). Educational psychology: A realistic approach (4th ed.). New York: White Plains. Govindasamy, T., 2002. Successful Implementation of elearning Pedagogical Considerations. Internet and Higher Education, 4, 287-299. Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 235-266 Hmelo-Silver, C. E., & Barrows, H. S. (2006). Goals and strategies of a problem-based learning facilitator. Interdisciplinary Journal of Problem based Learning, 1, 2139. Jenkins T., 2001. The motivation of students of programming. Proceedings of the 6th annual conference on Innovation and technology in computer science education, Canterbury, United Kingdom, 2001. Jonassen, D. H. (1991). Objectivism versus constructivism: Do we need a new philosophical paradigm? Educational Technology and Research and Development, 39, 514. Law, K. M. Y., Lee, V. C. S., Yu, Y. T., Learning Motivation in E-learning Facilitated Computer Programming courses, Computers & Education (2010), doi: 10.1016/j.compedu.2010.01.007. Lea, Susan J. , Stephenson, David and Troy, Juliette (2003) 'Higher Education Students' Attitudes to Student-centred Learning: beyond 'educational bulimia'?', Studies in Higher Education, 28: 3, 321 334. Li-Ping, T., Jill Austin, M. (2009) 'Students perceptiones of teaching technologies, application of technologies, and academic performance', Computers & Education (53), 1241 1255.
Page 10

Ma, L., Ferguson, J., Roper, M., Ross, I., and Wood, M. 2008. Using cognitive conflict and visualization to improve mental models held by novice programmers. In 39th ACM Technical Symposium on Computer Science Education (SIGCSE), Oregon, USA, pp342-346. McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., and Wilusz, T. 2001. A multi-national, multi-institutional study of assessment of programming skills of first-year CS students. SIGCSE Bulletin, 33(4), 125180. Moreno, L., Gonzlez, C., Castilla, I., Gonzlez, E., Sigut, J. 2005. Applying a constructivist and collaborative methodological approach in engineering education. Computer & Education 49 (2007) 891-915 Myller, N., Laakso, M., and Korhonen, A. Analyzing Engagement Taxonomy in Collaborative Algorithm Visualization. ITiCSE07 Proceedings, Dundee, Scotland, (June 23-27) Naps, T. L., Robling, G., Alstrum, V., Dann, W., Fleischer, R., Hundhausen, C. , Korhonen, A., Malmi, L., McNally, M., Rodger, S., and Velsquez-Iturbide. Exploring the role of Visualization and Engagement in Computer Science Education. In Working Group Reports from ITiCSE on Innovation and Technology in Computer Science Education, pages 131-152. New York, NY, USA, 2002. ACM Press. NatGeo. National Geographic Education Network Workshops, Forums and more, available at: http://www.ngsednet.org/. (Accessed November 5th 2008). O' Kelly, J. and S. Bergin and S. Dunne and P. Gaughran and J. Ghent and A. Mooney, Initial findings on the impact of an alternative approach to Problem Based Learning in Computer Science, Problem-Based Learning International. Conference 2004: Pleasure by Learning, 2004, Cancun, Mexico. Robins, Anthony, Rountree, Janet and Rountree, Nathan (2003) Learning and Teaching Programming: A review and discussion, Computer Science Education, 13:2, 137 - 172. Robocode. The open source educational game, available at http://robocode.sourceforge.net/. (Accessed May 28th 2008). Rugarcia, A., Felder, R.M., Woods, D.R., & Stice, J.E (2000). The Future of Engineering Education A Vision for a New Century. Chemical Engineering Education, 34(1), 1625. Scheard, J., Simon, Hamilton, M., Lonnberg, J. Analysis of Research into the Teaching and Learning of Programming. TeachingToday. Teaching Tips, lesson plans and more, available at: http://teachingtoday.glencoe.com/. (Accessed November 05th 2008). Villalobos, J., Casallas, R. "Teaching/Learning a First Object-Oriented Programming Course outside the CS Curriculum", 10th Workshop on Pedagogies and Tools for the Teaching and Learning of Object Oriented Concepts - ECOOP (European Conference on Object-Oriented Programming), 2006. Villalobos J., N. Caldern, C. Jimnez. Cupi2 Community: Promoting a Networking Culture that Supports the Teaching of Computer Programming. Proceedings of the 1st International Conference on Computer Supported Education (CSEDU), Portugal, March 2009. Villalobos J., N. Caldern, C. Jimnez. Developing Programming Skills by Using Interactive Learning Objects, Proceedings of the 14th Conference on Innovation and Technology in Computer Science Education (ITICSE), France, July 2009. Woodley, M., Kamin, S., 2007. Programming Studio: A course for improving programming skills in undergraduates. In Proceedings of the 38th technical symposium on computer science education. USA.
Page 11
Localization of Learning Objects

Cristian Cechinel, Sandro da Silva Camargo
Computer Engineering Course, Federal University of Pampa, Caixa Postal 07 CEP 96400-970, Bag (RS), Brazil contato@cristiancechinel.pro.br, sandro.camargo@unipampa.edu.br
Abstract. Most of the issues that arise when we deal with the concept of learning objects (metadata, granularity, interoperability, and standards) are in one way or another related to some ideal level of reusability which community wishes to achieve when using these resources. The large dissemination of learning objects over the internet has increased the diversity among the possible learners of these resources (different countries, contexts and languages), and has brought to attention the need of considering aspects of localization and internationalization during the process of development. Localizing and internationalizing learning objects will foster reusability by allowing their further adaptation and reuse in cultures and contexts different from the ones for those they were initially designed for. This paper presents a short overview of the state-of-art of some of the main issues that must be considered to proper localize and internationalize learning objects. The aspects studied here will be further used to create a proper methodology for learning objects adaptation to that will guide the development of new adaptable and reusable learning objects under the context of the IGUAL Project (Innovation for Equality in Latin American University). Keywords: Localization, Internationalization, Learning Objects, Openness, Cultural Adaptation
INTRODUCTION
The Learning Object (LO) concept has emerged as a new paradigm of computer technologies for teaching and learning in the middle of the 90s and can be considered as the cornerstone for the widespread development of elearning initiatives over the globe. According to (L'Allier, 1997), a Learning Object can be defined as the smallest independent instructional structure containing an objective, a learning activity and an evaluation. In this sense, a LO consists of a component basis of a course or a class which can be used and reused to create new unique instructional structures. Other authors present different definitions for Learning Objects, for instance, (Polsani, 2003) defines LO as an independent and self-standing unit of learning that is predisposed to reuse in multiple instructional contexts, McGreal(2004) characterizes LO as any reusable digital resource that is encapsulated in a lesson or assemblage of lessons grouped in units, modules, courses, and even programmes; and (Churchill, 2007) stands that a LO is a representation designed to afford uses in different educational contexts. As pointed by David Wiley, it seems that the number of definitions for learning objects is as huge as the number of people employing it (Wiley, 2000); in fact, even the terminologies used to describe the field vary according to the context in which they are inserted. Just to mention some of them, it is possible to find in the literature references to learning objects as: reusable learning objects (Polsani, 2003), knowledge objects (Merrill, 1999), educational objects (Friesen, 2001), reusable information objects (Barritt, Lewis, & Wieseler, 1999), and learning resources, among others. According to (Saum, 2007), sometimes some of the different terms are used interchangeably and other times they are independent from each other, but the term learning objects remains as the most widely recognized one. In any case, and despite all the disagreement around the theme, it seems clear that the concept of learning objects always brings together the implicit expectation of reusability (Duncan, 2009). This property, defined by (Sicilia & Garcia, 2003) as the possibility and adequacy for the object to be usable in prospective educational settings, is the cornerstone of learning objects concept and one of the main responsible for the widespread success of such technologies. In fact, most of the issues that arise when one deals with the concept of learning objects (metadata, granularity,
Page 12

interoperability, and standards) are in one way or another related to some ideal level of reusability which community wishes to achieve when using learning objects. So, in order to acquire the status of a LO, the media must fundamentally respect the principle of reusability, which normally demands that the LO must be developed independently of instructional methodologies and usage, thus facilitating its exchange among developers, organizations and institutions. According to (Longmire, 2000), to achieve reusability, features such as flexibility and personalization must be provided by LOs. These needs are reinforced by David Wiley in its presentation Openness, Localization and the Future of Learning Objects(Wiley, 2010). In his lecture, Wiley defends that the community involved with the production of digital learning materials should shift directions concerning the way the materials are being produced, focusing more on Openness and Localization, where Openness stands for digitized materials offered freely and openly for educators, students and self-learners to use and reuse for teaching, learning and research (OECD, 2007) and Localization stands for the process of taking educational resources developed for one context and adapting them for other contexts (OER COMMONS Wiki, 2010). The claim here is that resources should not only be decomposable and assemblable, but also changeable in a manner that developers and users are allowed to adapt existing resources according to their needs. These needs of adaptation could be related to a huge spectrum of issues, such as technical requirements, pedagogical approaches, learning personalization and cultural aspects, for instance. The IGUAL Project (Innovation for Equality in Latin American University) is an initiative funded by the European Commission focused on the development of innovative learning solutions for helping the teaching and learning process of the Introductory Programming Courses in Latin American Universities. In addition to the development of new learning solutions, the project will also adapt to the context of Latin American Universities current technologies already successfully developed and tested by the European partners of the project. The main goal of this paper is reviewing the state-of-art of some of the main issues that must be considered on the process of adaptation and construction of Learning Objects for different contexts and cultures. This short overview will be further used to create a proper methodology for LO adaptation to guide the development of new adaptable and reusable learning objects that will be openly distributed and disseminated over the internet. The rest of the paper is organized as follows. Section 2 differentiates the processes of Localization and Internationalization and presents some of the main tasks involved in the process of Localization. Section 3 describes four approaches for the culturalization of LOs, and section 4 points out the importance of openness for the sake of Localization. Section 5 briefly shows an initial proposal of cultural adaptation that will be implemented in the context of the IGUAL Project, and section 6 points out the final remarks about the present work.
LOCALIZATION AND INTERNATIONALIZATION

Localization (also known as L10n) is the process of modifying and adapting software so that it can be used in specific locales, different from those to which the software was primarily designed for. According to (Esselink, 2003), localization normally deals with the combination of technology and language in order to produce a software that can cross cultural and language barriers. It is important to highlight that localization is not just a matter of translating the application components (menus, dialogs) into a specific target community language, but also to adapt such application so that it respects the cultural preferences and conventions of this community. In the beginning of the software industry, this adaptation was separated from the software development process, i.e., it was carried out just after the whole cycle of development was concluded. However, with the large dissemination of desktop computers for international markets and the growth of the demand for localized software overseas, it became unfeasible to sustain such approach, and the industry started to provide features for supporting software localization during the process of development. This is referred as Internationalization (i18n), also defined by the Localization Industry Standards Association (LISA) as "the process of generalizing a product so that it can handle multiple languages and cultural conventions without the need for re-design (Esselink, 2000). As it can be seen, Internationalization and Localization are complementary processes, where the former intends to better enable the application of the latter. Localization is already largely used in education, such as for instance, in the adaptation of books developed elsewhere to local language and culture (Muegge, Mora, Hassin, & Pullin, 2008). According to
Page 13

(Muegge, et al., 2008), research has shown that the learning effectiveness of localized material is much higher than that of unlocalized material. (Kurilovas & Dagiene, 2009) claims that cross-border reuse of LOs is more likely if the material offers language customization (i.e., is offered in more than one language). Software Localization normally involves the following tasks (Dagiene & Zilinskiene, 2009): 1) adaptation to target locale; 2) translation and adaptation of interface; and 3) translation and adaptation of documentation (see figure 1).
FIGURE 1. Tasks of Localization (Adapted from (Dagiene & Zilinskiene, 2009)).
In computing, locale refers to a subset definition of environment variables that is stored at a separate and external file and that depends on language and culture conventions (IEEE & Group, 2004 ). Among the categories that are normally addressed by locale, one can mention (IEEE & Group, 2004 ).: 1) character classification definition of some attributes of the character, such as for instance, if it is a lowercase or an uppercase letter, a numeric digit, a punctuation character, a printable character; 2) collation order defines the sequence of collating elements so that can be used for the OS in other tasks, such as, for instance in the task of sorting character string data; 3) monetary formatting - describes the local currency as well as the rules and symbols to format monetary information, e.g., the character that delimiters decimal numbers, the character that represents the local currency symbol; 4) numeric (non-monetary) formatting describes the format of numeric information such as the character that delimitates the decimals and thousands; 5) date and time definition of the format of date and time, such as the use of 24 or 12 hours, the name of the week days and the order and format of appearance of day, month and year. (Dagiene & Jevsikova, 2009) point out that there are other cultural aspects which are human-sensitive elements and that are not necessarily described in locale. These elements (such as for instance, color scheme, hand signals, symbols, usage of sounds and videos) are often ambiguous, difficult to identify, and not formally defined by normative documents. The next section will describe some proposed approach for the Localization of learning objects that specifically address these kinds of elements. Translation and adaptation of the interface refers to the localization of the interface and components texts such as menus and buttons. For instance, it is known that the same sentences in different languages can vary significantly in size thus requiring different screen spaces. In order to ease localization of the interface, (EPIC, 2010) recommends to minimize the text on the screen (thus leaving room for longer languages), and avoid longer words and phrases on menus and dialog boxes. It is also recommended to provide options for the user dynamically configure the system interface settings (such as the font size and type, the colors of the interface components, or even the size of the windows, buttons and dialogs). According to (Dagiene & Zilinskiene, 2009), the translation of the documentation is the last step of the Localization. In here, there are three main components to localize: the help of the system, the license and the metadata. The authors highlight that the translation of the help (along with the interface) is the task that normally most consume time and effort due to the huge amount of text. The license refers to the special copyright conditions
Page 14

imposed for the use of the software. Learning Objects that are distributed under Creative Commons 1 license, for instance, can have their licenses (languages and laws) automatically localized for approximately 50 jurisdictions. Metadata can be defined as structured data about other data, created and managed to describe it in some way that can be used for a function different from the functions of the original data object (Lytras, Sicilia, & Cechinel, Forthcoming). In the context of learning objects, metadata could be defined as records that describe the most important features of the resources. These descriptions may consider several aspects of the technology, such as authorship, technical and educational information (Sicilia & Garcia, 2003) with the purpose of supporting indexation and search of the materials inside retrieval systems. Providing descriptive metadata is key to succeed on discovering and selecting desirable and relevant material on any retrieval system, and as stated by (Currier, Barton, O'Beirne, & Ryan, 2004), "poor quality metadata can mean that a resource is essentially invisible within the repository and remains unused". This statement makes clear the important role metadata plays on the reusability of learning objects and the consequently importance of its localization. Learning Object metadata can be localized when the translation of the language used to describe the information is required, or when the Learning Object is going to be adapted to different educational and technical contexts of usage than those described by the metadata, for instance.
CULTURAL ASPECTS
Different countries, regions and localities can make distinct associations with and can take distinct messages and understandings from certain words, symbols, colors, gestures, behavioral patterns, phrases and sentences. Adapting a LO to provide some intended effect may require, beyond internationalization and localization, the selection and adaptation of other elements, such as symbols, images or colors. In this scope, the translation of LO language is not sufficient, and a deeper level of culture must be addressed (Yeo, 2003), i.e., the LO must be culturalized. Considering that the scope of culture is very complex and the term has many different meanings, for the context of our work we deal with the following definition: culture is learned behavior consisting of thoughts, feelings and actions (Hoft, 1996). In order to illustrate cultural distinctions, a well known example can be mentioned: the MS-Windows folder icon. This icon shows a yellow rectangular object with a tab which suggest, at least to western users, that apparently it is a file folder used to organize and protect loose papers. This object is rarely seen in some other countries, like China. Folders used in China are usually made from thick brown paper without tabs. In this scope, Chinese users are forced to associate this unfamiliar object to folders. Surely, associating unfamiliar objects should not be the intention of using icons and symbols (Marcus, 1996).
Culturalization of Learning Objects

There are two ways of thinking about culturalization of LOs: one is to use the resources to learn about other culture, and another is to suppose that users just can learn with a culturalized LO. Regarding that, (Amiel, Squires, & Orey, 2009) proposed the following four classes to classify culturalization approaches in the context of LO, which are: LO, LOMA, n-Culture, and LOCA. LO: It is the conventional Learning Object. In this approach, developers create LOs considering just your own perception of learner culture. Afterwards, LOs are distributed and can be used in another cultural context, which can be significantly different from the original one. This situation is twofold: the learners can know another culture or some concepts explored in LO can be misunderstood. This approach is low time consuming as a consequence of analyzing an unique cultural context during LO development. LOMA: This approach is an improvement of LO forasmuch as it tries to lead learner to cultural context contemplated by developers. This strategy of teaching the original cultural background is called Learning Object with Multicultural Affordances. This approach is more time consuming than LO whereas some cultural elements must be explained by developers in order to embed the context in built artifact. However, developers must identify terms which will not be understood by users of other cultures. This task is completely imprecise. n-Culture: Supposing one specific learning objective, developing one LO customized for each existing culture is an impractical task. However, customized LO can be developed for cultures where artifact is planned to be used. This approach advocates for using the experience of users and developers in development process. The term nCulture denotes that n different cultures can be represented during development process. Time-consumption and
1
http://creativecommons.org/
Page 15

amount of labor increase in function of n. Moreover, LO also can be used by users of cultures which was unrepresented in development process. LOCA: It is the acronym for Learning Objects with Cultural Adaptability. This approach is the most complex and time consuming one. It involves creating an open structure which allows local developers to collaborate in culturalization process. Local developers are immersed on local culture and are supposed to be able to achieve a higher level of culturalization in LO. Other positive aspect is the potential division of work with interested learners and tutors around the world. While LO can be freely distributed over the internet, the diversity among their learners increases, and the idea of developing learning resources that fit the needs of all of them at the same time becomes less likely. Each learner is immersed in its culture and is surrounded by a set of traditions that filter the way their members understand and behave into the world. Because of that, learning resources may become inefficient if applied to other cultures and contexts different from the ones for those they were designed for. As stated before, some of the elements that need localization can be automatically made through the use of locale files, or by offering interface facilities, however, other elements require a closer human evaluation in order to be localized or adapted. Although there is no consensus about which items must be analyzed in culturalization process, these items commonly encompass cultural, demographical, political, religious, and historical aspects. In order to ease the adaptation process of such elements (Richter & Pawlowski, 2007) propose to store all this information in the so-called context metadata. These metadata would describe the influence factors that may impact in learning situations, including here the cultural aspects.
OPENNESS AND ADAPTATION

According to (Wiley, 2009) the initial metaphors about learning objects have limited the ways we understand the concept of reuse, which is normally associated to the idea of assembling and decomposing existing resources in order to form new ones. In here, resources are just combined, but not changed and adapted. The author claims that such vision has prevented us from seeing reuse as the possibility of changing the learning object itself in substantive ways and calls the community to shift directions towards a new way of approaching and defining reuse, where developers and users of learning objects are allowed to adapt existing resources according to their needs. Such paradigm has emerged in the last few years named as open educational resources (OER) which are understood as resources that comprise content for teaching and learning, software-based tools and services, and licenses that allow for open development and re-use of content, tools and services (Geser, 2007). The first step towards this new paradigm consists on overcome current restrictions of copyright and intellectual property that are intrinsically attached to the developed learning objects. Learning object producers are invited to free their creations through the use of copyright licenses that allow usage rights to others without any cost and without the need to ask for permission when they want to change or to adapt these resources. The most known license covering these aspects is the Creative Commons which allows developers to share their resources according to some predefined conditions such as authorizations for the commercial use of the work and for the modification of the work. David Wiley (Wiley, 2007) describes four distinct ways open content can be used for free, called by him as the 4Rs, which are: 1) Reuse (using and copying the same way the learning object is put available); 2) Revise (altering the resource according to our needs); 3) Remix (mixing the learning resource with other available resources according to our needs); and 4) Redistribute (sharing with others the work derived from the three previous ways of use). This level of usage will depend on the license attached to the resource and also on the type (or format) of the resource. For instance, learning objects in pdf format could be put available together with a license allowing revision and remixing, however, the task of adapting and localizing it would be difficult to perform once the format is closed and do not allow modifications. (Amiel, Orey, & West, 2011) reinforce this by stating that besides the legal aspects (copyright licenses) there is another essential issue regarding openness: the technical aspects. The authors claim that even though some resources easily allow the combination with other resources, they are technically difficult to adapt (sometimes impossible) for anyone different from the original author. This situation could be softened by giving total free access to the source code of the resource, thus allowing anyone to change, adapt and repurpose the resource according to his specific needs and requirements.
Page 16

One successful example of open educational resources initiative is the Connexions 2 repository. Connexions is a repository which allows users to collaborative create and share learning materials and that has presented an exponential growth of contributors in the last years. According to (Ochoa, 2010), such success can be attributed to the fact that, differently from the traditional learning objects repositories, Connexions functions through the social interaction for the creation of materials. In Connexions all materials are created by its own community of members that can develop materials in two formats: modules (small pieces of knowledge) and collections (groups of modules structured into course notes). In Connexions every material available is free to use, reuse and share with others (under a Creative Commons license), and, as the community of members has total access to the core of the materials, the 4Rs of reuse are fully put in practice. However, whilst openness is something to be desired and encouraged, it does not guarantee the software will be adapted and localized. According to (Jevsikova, 2006), the level of internationalization of open software is not as high as the level of proprietary software. This is because open source software is normally developed by a community of volunteers who not necessarily possess the proper knowledge about localization issues, while proprietary software is developed for market, thus aiming profit accross the border. The author has evaluated three open source virtual learning environments (Moodle, ATutor, Claroline) according to localization needs and aspects and concluded that these software internationalization are not sufficient for effectivelly localize them. To put internationalization efforts during the process of development is essential to ease proper Localization.
THE INITIAL PROPOSAL

The initial framework we are willing to implement during the Internationalization and Localization of learning resources developed under the scope of the IGUAL Project is shown in Figure 2.
FIGURE 2. Framework for the Adaptation of Learning Objects.
As it can be seen in the figure, Learning Objects shall be localized according to the main dimensions previously mentioned in the paper. In here, we are considering the Internationalization/Localization of the Locale and the Interface as the first layers of adaptation that must be observed. Information related to locale dimensions (language, collation, date and time format), as well as textual information displayed on the interface will be stored in external files (for instance, XML files, or regular locale text files). This is consistent with the recommendation given by (McBrien), who stands that all localizable text should be put in external resources files. A second layer of Internationalization/Localization will deal with information related to the adaptation of cultural aspects, more precisely those that are not possible to automatically adapt through the use of regular locale files. If possible, cultural information will also be put available in external files in a similar manner that was implemented by (Altuner & Turker, 2006). In that work, the authors described how localization can be applied for the benefit of personalized adaptation of e-learning by providing ways of adapting the face of the character used in the animation of a given learning object. The architecture of each LO will vary depending on the chosen
2
http://cnx.org/
Page 17

culturalization approach. For instance, considering we are implementing a n-Culture LO, the architecture could be similar to the one proposed by (Muoz-Arteaga, Vanderdonckt, Gonzlez-Calleros, & Orey, 2011) where the LO is developed under a PAC (Presentation-Abstraction-Control) model architecture. In that work, different possible cultural aspects of the resources content are represented by distinct PAC agents that could be selected by the students according to their learning needs. Metadata about the LO could be considered the third layer of our Framework, and is divided into two main classes: the outer-metadata and the inner-metadata. The outer-metadata is related to all information that is recommended by the standards (IEEE LOM, SCORM) and addresses issues such as: general information about the LO (name, location, and author), educational aspects (target user and level of difficulty) and technical (size and format), among others. The inner-metadata describes the inner structure of the resource and can contain information about the content of the LO, the graphical interface, and the text, for instance. For our case, for instance, innermetadata could also contain specific information related to cultural usage of the LO given a certain context. The utilization of inner-metadata is proposed by (Rodriguez, Chen, Shi, & Shang, 2002; Shi, Rodriguez, Chen, & Shang, 2004) who claim that such organization of inner information facilitates further content adaptation. The task of providing inner-metadata must be done in parallel with the development of the resource since that all this information will reflect the structure and architecture of the LO as well as will have to communicate with it. The last layer of the framework would be of attaching a Creative Commons license to the resource, as well as freely distributing the source code for those who want to adapt it.
An Example
Learning Objects that will be developed and culturally localized under the IGUAL Project scope will focus on helping students on basic computer programming topics. Such resources normally contain information related to the source code of programs that are developed in specific languages, such as, C, Python, or JAVA. Let us consider a small LO containing a C code that calculates the average grade of a given student and informs if this student was approved or has failed in the course. For this given scenario the average grade for approval is 6.0. The main goal of the LO would be of putting in order the given code in C so that the program could perform the proposed task. Figure 3 presents the interface of this LO with the lines of the code in C all mixed up. The user has basically the following options: 1) shuffling the lines of the code, 2) checking if the order of the code is correct, 3) resetting the code (i.e., putting its lines in the correct order), and 4) changing the order of a given line (up and down). In order to change the order of some line, the user has to click over it and click in the desired direction (up or down).
FIGURE 3. LO interface.
The adaptation of this LO could be done in terms of the language by putting all text information inside external XML locale files. For instance, for the case of the program code, the items that could change from one language to another would be the names of the
Page 18

variables and the text printed by the program. These items could be identified and translated (for instance from English to Brazilian Portuguese) into a XML file named as content.pt_br.xml. For the case of the interface, the labels of each button and the messages that are showed on the screen could also be stored on an external XML locale file named as interface.pt_br.xml. Figure 4 shows a possible structure for these files.
<?xml version="1.0" encoding="ISO-8859-1" ?> <content> <code> <variable> <id>averageGrade</id> <translation>media</translation> </variable> <variable> <id>grade1</id> <translation>nota1</translation> </variable> <variable> <id>grade2</id> <translation>nota2</translation> </variable> <printf> <id>Inform two grades:</id> <translation>Informe duas notas:</translation> </printf> <printf> <id>Average grade is %f</id> <translation>A media %f</translation> </printf> </code> </content> <?xml version="1.0" encoding="ISO-8859-1" ?> <interface> <button> <id>Shuffle</id> <translation>Embaralhar</translation> </button> <button> <id>Check order</id> <translation>Verificar a ordem</translation> </button> <button> <id>Reset</id> <translation>Organizar</translation> </button> <button> <id>Up</id> <translation>Cima</translation> </button> <button> <id>Down</id> <translation>Baixo</translation> </button> <message> <id> The order is correct </id> <translation> A ordem est correta</translation> </message> <message> <id> The order is not correct </id> <translation>A ordem no est correta</translation> </message> </interface>
FIGURE 4. Internal structure of content.pt_br.xml (left side) and interface.pt_br.xml files (right side)
As mentioned before, languages occupy differently the space on the screen, so, it is possible that the text translated to the target language does not fit in the current objects of the interface. This would be the case of allowing the user to change the interface settings accordingly to the current language needs. Moreover, at a first glance, it may seem that the cultural adaptation of this resource would be restricted to the language and interface, however, if we think about the context of the problem that this simple C program is trying to solve, it is possible to raise some other aspects that could be culturally adapted, if desired. For instance, it is known that there are differences in the way students are evaluated in different institutions and countries. The final grades can be calculated through different formulas, and the minimum grade required for approval can significantly vary. A LOCA strategy for this LO would allow for the modification of the way the final grade is calculated and how this grade should be evaluated to know if the student is approved. In addition, a LOMA strategy would provide some basic explanations about the giving context where the current evaluation technique is taking place.
FINAL REMARKS
Considering aspects of localization and internationalization during the process of development of learning objects is crucial to ease further adaptation of these resources to different cultures and contexts of usage, and thus to foster reusability. The present work has shown some of the main aspects of localization that must be considered in order to
Page 19

allow proper cultural adaptation of learning objects. These aspects were also organized under an initial framework divided into the following layers of localization: Locale and Interface, Culture (LOMA, LOCA, n-Culture), Metadata (inner-metadata, outer-metadata) and Openness. Along with this initial framework, we have also shown a simple proposal of cultural adaptation of a learning resource focused on basic computer programming skills, where different target users will be able to use the same resource, but adapted according to their language. Moreover, we have also briefly tackled how the given example could support cultural adaptations according two presented strategies, precisely LOMA and LOCA. This study will serve as the basis for the creation of a methodology for the localization and internationalization of learning objects that will be implemented in the context of the IGUAL Project.
ACKNOWLEDGMENTS
The work presented here has been funded by the European Commission through the project entitled IGUAL Innovation for Equality in Latin American University (code DCI-ALA/19.09.01/10/21526/245-315/ALFAHI (2010)123) of the ALFA III programme.
REFERENCES
Altuner, E. M., & Turker, M. A. (2006). Learning Designs Supporting Localisation Personalised and Adaptive ELearning. In Proceedings of Adaptive Hypermedia. Dublin, Ireland. Amiel, T., Orey, M., & West, R. (2011). Open educational resources (OER): models for adaptation and localization. ETD - Educao Temtica Digital, 12, 112-125. Amiel, T., Squires, J., & Orey, M. (2009). Four Strategies for Designing Instruction for Diverse Cultures: Context and Localization of Learning Objects. Educational Technology Magazine: The Magazine for Managers of Change in Education, 49(6), 28-34. Barritt, C., Lewis, D., & Wieseler, W. (1999). Cisco systems reusable information object strategy: Definition, creation overview, and guidelines. A CISCO Whitepaper. Churchill, D. (2007). Towards a useful classification of learning objects. Educational Technology Research and Development, 55(5), 479-497. Currier, S., Barton, J., O'Beirne, R., & Ryan, B. (2004). Quality assurance for digital learning object repositories: issues for the metadata creation process. ALTJ: Research in Learning Technology, 12(1), 5-20 Dagiene, V., & Jevsikova, T. (2009). Cultural elements in internet software localization. Revue d'Intelligence Artificielle, 23(4), 485-501. Dagiene, V., & Zilinskiene, I. (2009). Localization of Learning Objects in Mathematics. In L. P. A. Rogerson (Ed.), 10th International Conference on Models in Developing Mathematics Education (pp. 129-133). Dresden, Saxony, Germany. Duncan, S. M. (2009). Patterns of Learning Object Reuse in the Connexions Repository. EPIC. (2010). EPIC white paper on the Localisation and e-learning [White paper]. Esselink, B. (2000). A Practical Guide to Localization: John Benjamins Pub Co. Esselink, B. (2003). The Evolution of Localization. In Multilingual Computing & Technology: (Jul/Aug ed., pp. 47). Guide to Localization 2003: MultiLingual Computing, Inc. Friesen, N. (2001). What are educational objects? Interactive Learning Environments, 9(3), 219230. Geser, G. (2007). Open educational practices and resources: OLCOS Roadmap 2012. Revista de Universidad y Sociedad del Conocimiento, 4(1), 1-9. Hoft, N. L. (1996). Developing a cultural model. In International users interface (pp. 41-73): John Wiley and Sons, Inc. IEEE, & Group, O. (2004 ). IEEE Std 1003.1. In: The Open Group Base Specifications Issue 6. Jevsikova, T. (2006). Localization and Internationalization of Web-Based Learning Environment. In R. Mittermeir (Ed.), Informatics Education The Bridge between Using and Understanding Computers (Vol. 4226, pp. 310-318): Springer Berlin / Heidelberg.
Page 20

Kurilovas, E., & Dagiene, V. (2009). Learning Objects and Virtual Learning Environments Technical Evaluation Criteria. Electronic Journal of eLearning, 7(2), 127-136. L'Allier, J. J. (1997). Frame of Reference: NETg's Map to the Products, Their Structure and Core Beliefs. NetG. Longmire, W. (2000). A Primer on Learning Objects. In ASTD Learning Circuits. Lytras, M. D., Sicilia, M.-A., & Cechinel, C. (Forthcoming). The value and cost of metadata. In M.-. Sicilia (Ed.), Handbook of Metadata, Semantics and Ontologies: World Scientific Publishing Company. Marcus, A. (1996). Icon and symbol design issues for graphical user interfaces. In International users interface (pp. 257-270): John Wiley \& Sons, Inc. McBrien, K. Developing Localization Friendly E-Learning. In http://www.learningcircuits.org/2005/may2005/mcbrien.htm. Merrill, M. D. (1999). Instructional Transaction Theory (ITT): Instructional Design Based on Knowledge Objects. In C. M. Reigeluth (Ed.), Instructional Design Theories and Models: A New Paradigm of Instructional Technology (pp. 397-424): Lawrence Erlbaum Associates. Muegge, S., Mora, M., Hassin, K., & Pullin, A. (2008). A Flat Network for the Unflat World: Open Educational Resources in Developing Countries. Open Source Business Resource. Muoz-Arteaga, J., Vanderdonckt, J., Gonzlez-Calleros, J. M., & Orey, M. (2011). An Architectural Model for Designing Multicultural Learning Objects. In ACHI 2011: The Fourth International Conference on Advances in Computer-Human Interactions (pp. 249 - 253). Gosier, Guadeloupe, France: IARIA. Ochoa, X. (2010). Connexions: a Social and Successful Anomaly among Learning Object Repositories. Journal of Emerging Technologies in Web Intelligence, 2(1). OECD. (2007). Giving Knowledge for Free: The Emergence of Open Educational Resources: OECD Publishing. OER COMMONS Wiki (2010). In http://wiki.oercommons.org/mediawiki/index.php/Tutorial. Polsani, P. R. (2003). Use and Abuse of Reusable Learning Objects. Journal of Digital Information, 3(4). Richter, T., & Pawlowski, J. M. (2007). The Need for Standardization of Context Metadata for e-Learning Environments. In e-ASEM. Seoul. Rodriguez, O., Chen, S., Shi, H., & Shang, Y. (2002). Open Learning Objects: the case for inner metadata. In Proceedings of WWW2002 Education Track. Honolulu, Hawaii, USA. Saum, R. R. (2007). An Abridged History of Learning Objects. In P. T. Northrup (Ed.), Learning Objects for Instruction: Design and Evaluation (pp. 1-16). New York: Idea Group Publishing Shi, H., Rodriguez, O., Chen, S., & Shang, Y. (2004). Open Learning Objects as an Intelligent Way of Organizing Educational Material. International Journal on e-Learning, 3(2), 51-63. Sicilia, M.-A., & Garcia, E. (2003). On the Concepts of Usability and Reusability of Learning Objects (Vol. 4). Wiley, D. A. (2000). Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy. In D. A. Wiley (Ed.), The Instructional Use of Learning Objects: Online Version. . Wiley, D. A. (2007). Open education license draft. In http://opencontent.org/blog/archives/355. [Last Access 25th October 2010] Wiley, D. A. (2009). Impediments to Learning Object Reuse and Openness as a Potential Solution. Revista Brasileira de Informtica na Educao, 17(3). Wiley, D. A. (2010). Openness and Analytics: The Future of Learning Objects. Presentation In Fifth Latin American Conference on Learning Objects and Technology Enhanced Learning LACLO 2010. So Paulo. Yeo, A. (2003). Translation is not enough. Localisation Focus, 2(1).
Page 21
Sharing and Recommending Lea rning Materials

Xavier Ochoa
Escuela Superior Politcnica del Litoral
Abstract. This work presents a survey of the current technologies to publish, socially share and recommend learning materials. First, the Social Sharing is examined throught the current concepts of Learning Object Repositories (LOR), the state-of-the-art of the understanding about the publication/sharing process. Social LORs are presented and analyzed as the new and most sucessful form of LORs and Connexions, the best example of this new kind of LORs is taken as a case-study. Second, the state-of-the-art in Recommenders for Learning Materials is presented through the analysis of traditional concepts of recommender systems and how those concepts have been applied in the field of learning material recommendation. Context- aware recommenders are presented as the current trend in the development of this kind of systems.
Keywords: learning object repositories, social sharing, context-based recommenders PACS: Technology Enhanced Learning
SOCIAL SHARING OF LEARNING RESOURCES Cur rent Definition of Learning Object Repositories
Learning Objects can be shared in several ways. They can be just published on the web, made available in online forums or even pass personally from user to user. This study, concentrate in the most formal way of learning object sharing: Learning Object Repositories. To share an object in this way, the object is indexed in what is called a Learning Object Repository (LOR). In their most common form, LORs usually store the learning object itself and the metadata instances associated with it. These LORs provide some sort of indexation facility, where users can add new learning objects together with their metadata. Also, some sort of search or browsing facility is provided to provide access to the content of the repository. An important sub-type of LORs are the Learning Object Referatories. These only store the metadata, while the object itself is stored elsewhere, usually on a server on the Web. The most popular examples of Learning Object Repositories are: ARIADNE Knowledge Pool System1 [? ]: It originated from a European project to create a repository of learning materials in the region. It is based on a distributed architecture that enables each node to keep control of its own materials. With almost 12 years of existence, it can be considered one of the oldest still operative repositories. Connexions2 [? ]: A repository born from the need to share materials for Digital Signal Processing that has expanded to other fields. It can be considered to be one of the newest and, currently, most successful repositories. It is based on a Creative Commons [? ] license. This license enables the free sharing and adaptation of the material. Maricopa Learning Exchange 3 : A small size repository belonging to a small group of institutions. The focus of this repository is to provide packaged objects that can be easily reused in Learning Management Systems. The contributor base is restricted only to faculty of Maricopa Community Colleges.
1 2 3
ARIADNE Foundation. http://www.ariadne-eu.org Connexions. http://www.cnx.org Maricopa Learning Exchange. http://www.mcli.dist.maricopa.edu/mlx
The most popular examples of Learning Object Referatories are:
MERLOT4 [? ]: A USA initiative to catalog learning material on the web. It is one of the oldest referatories.
Page 22

It is open for contribution, but it has a system where experts review the on-line material and provide extensive reviews and ratings. This model is unique among the community of LORs. This referatory currently is still growing strongly. INTUTE5 : An initiative based on UK where a group of experts also catalog on-line materials. It is closed to external contribution and tries to keep an uniform level of quality. Due to its age and continuous funding, INTUTE is one of the biggest referatories.
These LORs describe themselves as Learning Object repositories or referatories. However, due to the fuzziness of the concept of Learning Object, what can be considered a Learning Object Repository is also not clear-cut. Current Open Courseware (OCW) initiatives provide digital material that can be reused in a learning setting. However OCW sites do not identify themselves as Learning Object Repositories, even if they comply with the definition given at the beginning of this subsection. Learning Management Systems also store a great amount of learning material that is shared in a small community of the teacher and the students of a course. Even if they are not open, these systems can also be considered as LORs. Finally any type of digital library, such as Institutional Repositories, where digital learning material can be stored, could also be considered under this definition. During this dissertation LORs, OCWs, LMSs and IRs are considered as Learning Object Repositories. As conclusion, in the context of this study, a LOR is consider in its widest definition. Any system that stores digital learning material and that provides some sort of indexing and searching or browsing interface for those materials.
PUBLICATION OF LEARNING OBJECTS

Learning Object publication can be defined as the act of making a learning object available to a certain community. Strijker and Collis call this process Offering in their Learning Object Life cycle model [? ]. The publication process can take several forms. A professor can publish lectures notes for students in a Learning Management System (LMS). The same professor can decide to share objects with a broader community and publish them in a Learning Object Repository (LOR), such as ARIADNE [? ] or Connexions [? ]. The University where this professor works can decide to start an Open Courseware (OCW) initiative [? ] and put the learning material of its courses freely available on the Web. Moreover, material already available online, can be discovered and re-published for other communities. For example, a student that found an interesting Web site to learn about basic Physics, could publish a link to that Web site on a Learning Object Referatory (LORF), such as Merlot [? ] or SMETE [? ]. In all its different forms, Learning Object publication is the most important enabler of the Learning Object Economy [? ], because making the objects available is the first step to fuel the share, reuse, improve and share again philosophy behind this economy. The publication of learning objects has been an important research issue since the definition of the field 15 years ago. These efforts can be summarized into three different research lines: Publishing Infrastructure [? ] [? ], Interoperability [? ] and Copyright and DRM [? ] [? ] . However, one area of research that is practically unexplored is the study of the actual process and results of learning object publication. The research on technical and legal aspects lays the ground on which publication can take place. However, it does not provide any information about simple questions, such as how many learning objects are actually published, how they are distributed among different repositories or how repositories grow. Moreover, answers to these questions are not only relevant to measure the progress of the Learning Object Economy, but also to provide information on which decisions about architecture, interoperability strategies and planning for growth should be based. The most prominent attempts to characterize learning object repositories and measure their characteristics are made by McGreal in [? ]. He provides a comprehensive survey of existing LORs and classifies them in various typologies. Unfortunately, his analysis is mostly qualitative and cannot be used to answer the questions mentioned above. Other relevant studies are [? ], [? ], [? ], [? ] and [? ]. In a previous work of the author of this paper [? ], several systems for learning content publication were quantitatively examined in order to find common characteristics and behavioral patterns. The main answers found in this work are: What is the typical size of a repository? Is it related to its type? In general, individual learning object repositories seems to vary from hundreds to million of objects. Their average size depends of the type of repository. LORPs can be considered to have few thousand of objects. LORFs are in the order of the tens of thousands. However, those numbers are small compared with multi-institutional IRs that can count hundreds of thousands and even millions of objects. OCWs and LMSs can have from hundreds to thousands of courses.
Page 23

4 5
MERLOT. http://www.merlot.org INTUTE. http://www.intute.ac.uk
However, the answer to this question is not that simple. The size is not Normally distributed, meaning that the average value cannot be used to gain understanding of the whole population. It is not strange to find repositories several orders of magnitude bigger or smaller than the average. Sampling biases aside, the distribution of learning objects among repositories seems to follow a Lotka or Power Law distribution with an exponent of 1.75. The main implication of this finding is that most of the content is stored in few big repositories, with a long, but not significant tail. Administrators of a big repository would want to federate [? ] their searches with other big repositories in order to gain access to a big proportion of the available content. On the other hand, it makes more sense for small repositories to publish their metadata [? ] for a big repository to harvest it in exchange for the access to their federated search. It seems, through an initial reading of this finding, that a two (or three) tiered approach mixing federation and metadata harvesting is the most efficient way to make most of the content available to the wider audience possible using the current infrastructure. How many learning objects are typically used in a course? The simplest answer is 20. That is the amount of learning objects that, in average, are present in common aggregation of learning objects (courses) in LORs, OCWs and LMSs. However, a heavy-tail distribution, Weibull, reduces the meaning of this value. An instructor normally strives to have between 15 to 35 objects in her course. This number can be related to the number of lessons or sections of the course (Probably 1 or 2 objects per lesson). There is a considerable amount of courses that have from 1 to 20 objects and small fraction of the courses that can have more than 200 objects. The main implication of this finding is that if OCWs and LMSs are decomposed and converted into repositories, they can be considered very large LORPs. The fact that LMSs are a widely deploy technology [? ] and that these systems are not accessible for external visitors make us think of the learning objects present in LORs as just the "tip of the iceberg". The bigger part of learning resources is hidden behind login pages. This finding validates the effort of the OCW Consortium and OER Commons [? ]. If we want to create a really functioning Learning Object Economy, we must start opening the door of our LMSs. How repositories grow over time? Linearly. This is a discouraging finding. Even popular and currently active repositories grow linearly. Even if we add them all together, we will still have a faster linear, but no exponential. The main reason for this behavior is the contributor desertion. Even if the repository is able to attract contributors exponentially, it is not able to retain them long enough to feel the effect. The value equation, how the contributor benefits from contributing to the repository, is still an unsolved issue in most repositories. Several researches have suggested incentive mechanism [? ] [? ] comparable to scientific publication, in order to provide the professor with some type of reward for their contribution. Another interesting result in the growth analysis was to find that all repositories went through an initialization with usually a very low growth rate. The length of this stage varied from 1 to 3 years (shortening for more recent repositories). After this period, a more rapid expansion begins caused by (or that cause) an increase in the number of contributors joining the repository. Having knowledge of these phases could help repository administrators to not discard slow growing repositories too soon. What is the typical number of contributors a repository has? Is it related to its type? We can estimate, from the analysis in section 3, that medium LORs have a base of 500 to 1500 contributors. This number is similar also for OCWs and LMSs contributor bases. On the other hand, IRs, being targeted also to students, have contributor bases one order of magnitude bigger. The size of the contributor base, however, is not always related to the size of the repository. Merlot contributors, being outnumbered 1 to 10, produce a comparable amount of objects as MIT IR contributors. Moreover, the title to the most productive contributors in the study goes to the OCWs and LMSs professors (Table ??) with around 40 objects in average. This results also support the idea that LMSs are the most effective type of repository, given that they provide a clear value into the publishing step (students not asking for copies of the material, for example). Given the relatively small size of the communities that build repositories, it would be an interesting experiment to measure the impact that the introduction of social networks could have in the sharing of material. For example, users would be interested in knowing when a colleague in his same field has published new learning objects [? ]. This social networks can be created explicitly (a l Facebook) or implicitly (relationship mining) [? ]. The deployment of these types of networks could also help to solve the lack of engagement problem. How the number of contributors grows over time? Most of them linearly, but surprisingly three of them Connexions,
Page 24

MIT OCW and SIDWeb, exponentially. This unexpected result, specially in SIDWeb, a run-of-the-mill LMS, is very encouraging for the future of the Learning Object Economy, because it can give rise, with the right environment, to exponential growth of content available. However, we also found that at this stage, the growth in those repositories continue linear. However, this observation can be due to the recent kick-off of exponential contributor base growth in those repositories. A follow-up study in a year period would help us to have a better perspective. Again, the finding of exponential growth in course-based repositories confirms the idea that we should strive to connect LMS as the main source of learning material. How many learning objects a contributor publishes in average? As mentioned before, the average productivity of users depends on the type of repository. For LORP it can be around 10 objects per contributors. IRs present the lowest production per contributor with 1 or 2 in average. However, heavy tail distributions, Lotka and Weibull, makes this answer a little more complicated. The problem with the average values given previously in the current situation, is that in heavy tailed distributions there is not such thing as an average user. As mentioned in section 4, the best way to describe the production of different contributors is to cluster them in classes similar to socioeconomic strata. If we adopt this approach we gain a new way to look at our results. In LORP and LORF, the repository is dominated by the high-class. Most of the material is created by a few hyperproductive contributors. the 10% of the users could easily have produced more than half of the content of the repository. In the case of OCWs and LMS, the Weibull distribution determines that the middle-class is the real motor of the repository. The low- and high-class are comparatively small. Finally, University IRs, with Lotka with high al pha are dominated by the lower-class as more than 98% of the population produces just one object. From our analysis on publishing rate and lifetime, we can conclude that these different distributions are caused not by an inherent difference in the talent or capacity among the different communities, but by the difference in contributor engagement with the repository. It seems that the distribution of lifetime, the time that the contributor remains active, is different for this three observed repository types. In LORP and LORF, there is some time of novelty engagement that keep the contributor active at the beginning, but the chances of ceasing publication increases as more time is spent in the repository. For OCWs and LMSs, there is a goal-oriented engagement that keeps the contributor productive until her task is finished (course is fully published). In the case of IRs, there is no engagement at all. The norm is just discrete contributions. Changes on the type of engagement should have an effect not only in the distribution of publications among users, but also in the growth and size of the repository. In conclusion, it is very important for a repository administrator to know the composition and characteristics of her contributor base. Having a clear view of what and who need to be incentivized is the first step before building any type of incentive plan [? ].
SOCIAL LEARNING OBJECT REPOSITORIES

While the qualitative and quantitive analysis of Social Learning Object Repositories is a not well researched area in the field of Technology Enhanced Learning, there are some works that explore the factors that contribute Social LORs success. The most detailed description of the desirable characteristics of Connexions, one of the most representative Social LORs, is presented by Dholakia et al. in [? ]. They present Connexions as a exemplary case of a sustainable open educational program. The most important aspect of this work is the description of the characteristics that the authors considered are key factors behind the success of Connexions: 1) Increase the equity of the Connexions brand, 2) High quality, ample, modular, continually updated, personalized on assembly, published on demand content, 3) An engaged and involved user community and 4) Site usability. In a more general approach, Monge et al. in [? ], analyze what they considered are the characteristics of the so called Repository 2.0". They perform a Qualitative study of repositories that have a community aspect as part of its constitution (Connexions among them). From the analysis of the characteristics of these repositories and inspired by Web 2.0 technologies, they recommend several strategies to create a community backed repository: 1) Clear authorship and use license attribution, 2) Rapid content creation, 3) Indexable content for search engines, 4) Social tagging, 5) Reputation systems for contents and 6) Social recommendation systems. The present work finds evidence to back strategy number two and five. Petrides et al. in [? ], present the first quantitive analysis of the production and reuse of content in Connexions. While it is not presented as conclusion of that work their results provide indication of exponential growth in the number of modules, as well as the inequality in the production of content. The main focus of their research is
Page 25

the analysis of the reuse behavior of the users based on the commentaries made when a new version of the module is published. The authors complement the quantitive study with a series of interviews made to selected contributors.
CONNEXIONS AS AN EXEMPLARY SOCIAL LOR

A previous work from the author ?? was inspired by slight anomalies found in the behavior of Connexions compared to traditional Learning Object Repositories [? ]. A re-analysis of the anomalies with updated data show that they have not disappeared, but increased with time and expanded to previously normally behaving characteristics. Given that this characteristics are deeply related with the kind of repository, this work hypothesizes that Connexions is the first in a new class of LOR, the Social LOR. To provide initial support for this hypothesis, the intrinsic characteristics of Connexions were analyzed to find possible explanations for these new behaviors, namely exponential growth and very unequal engagement with the repository. In the analysis of these intrinsic characteristics, It was found that the materials in Connexions are edited or updated several times and at a rapid rate. This goes in line with the open source community dictum: release early, release often". This strategy is recommended in order to attract other programmers to experiment with the code and improve it. This strategy seems to be working in Connexions, where the number of contributors that edit or publish material is growing exponentially and is causing an exponential growth in the number of available material. The analysis of the social network that emerge from the collaborative creation of materials help us to understand the unequal engagement of contributors with the repository and why it is not affecting negatively to the growth of Connexions. The main feature found in the community formed by Connexion contributors is its openness to accept sporadic and new contributors. A backbone of committed members seems to provide help and support to non-expert contributors and in exchange receiving recognition and community status. The unequal distribution of the lifetime is a natural effect of the unequal distribution of interest in the repository. The negative effect of the inequality is mitigated by the fluid social interactions between experts and novices. It seems that the same forces that contribute to the success of Wikipedia and other wikis are also pushing the success of Connexions, and differentiating it from traditional LORs.
RECOMMENDER SYSTEMS FOR LEARNING MATERIALS Recommender Systems

Recommender systems have been researched and deployed extensively over the last decade in various application areas, including e-commerce and e-health. Several recommendation algorithms, such as content-based filtering [88], collaborative filtering [54], knowledge-based filtering [24] and their hybridizations [27] are widely discussed in the literature and in several surveys of the state-of-the-art [25], [4]. Also in the Technology Enhanced Learning (TEL) domain, the deployment of recommender systems has attracted increased interest during the past years [29]. By identifying suitable learning resources from a potentially overwhelming variety of choices, recommender systems offer a promising approach to facilitate both learning and teaching tasks. A recent survey of recommender systems in TEL has been elaborated by Manouselis et al. [72]. The authors presented an extensive overview of existing recommender systems in TEL. In addition, evaluation perspectives on current research in this area and future challenges with respect to the evaluation of TEL recommenders were discussed. The major interest in this area is also reflected by a series of workshops, handbooks and special issues on Social Information Retrieval [121][42] and Recommender Systems for TEL [71] that have been organized in recent years. Contributions to a recent special issue on context- aware recommender systems [118] indicate that there is an increasing interest to the role of contextual data in TEL recommenders. Among others, advancements of network and mobile services, the growing tool and device landscape and the pervasive computing vision provides many new opportunities for the TEL domain to adjust itself to this landscape appropriately [108]. Several examples of contextual recommendation scenarios have been presented by researchers in this field to demonstrate its potential. A first example considers the location of the user and the noise level at this location as a basis to suggest learning resources [36]. If the learner is in a cafeteria, the noise level associated to this location might have an impact on her level of concentration and likelihood of interruption. Therefore, a contextual
Page 26

recommender would in such a context suggest learning activities to assess her knowledge on previously learned topics, e.g. through simple questionnaires that she can resume easily at a later point in time. On the other hand, if she is in a quiet study room, more elaborate learning resources that introduce new topics might be suggested. A second example considers proximity to support collaborative learning. If a contextual recommender is able to detect people nearby who are working on similar learning activities, the system can suggest suitable peer learners to help with a learning activity. A third example also takes into account the device type that the learner is currently using [118]. For example, if a learner is being recommended material to study the theory of relativity while commuting from work to school using a smartphone, short, audiovisual material that fits the screen of the smartphone should be more relevant than a long, text- only document. Accordingly, the recommender system should rank the short videos higher than the long documents. On the other hand, if the learner is reviewing the same subject at night, at home, more in-depth material, including long texts and formulas, should be more useful and, consequently, recommended first. This re-ranking of the recommended resources is not possible if the system does not know about the context of the user. In this perspective, new challenges emerge for capturing and understanding the context of the user and exploiting such contextual information for creating intelligent recommendations adapted to the current needs of the user.
RECOMMENDER SYSTEMS TAILORED FOR LEARNING MATERIALS

Recommender systems are a well studied and well established field of research and application. Most prominent examples of recommender systems are those used by Amazon.com and Google. Unfortunately, the algorithms underlying regular recommender systems are not directly transferable to the area of TEL. The TEL area offers some specific characteristics that are not met by todayA Z s general purpose recommendation approaches. The main difference is, of course, that learning is an individual process per human. Each learner uses her own tools, methods, paths and processes. Consequently, guidance within the learning process must be tailored to the individual needs and, hence, must be personalized to an extreme extent. For example, rather than recommending resources that other users with similar interests have used, the recommendation must also respect the actual learning situation of the learner, including the learning history, environment, timing and accessible resources. Furthermore, learning activities take place in learning environments that are composed of numerous tools and systems. For example, learning management systems (LMS) [86] as a notion of learning environments provide access to learning resources and collaboration facilities, but do not ensure that teachers or students of a course use them only. Often, learners use additional tools to collaborate or find resources, for instance, in case that the learning material offered in the LMS is not sufficient. Personal learning environments (PLEs) [114] try to address this issue by enabling learners to compile the tools they want in order to address a learning challenge. PLEs also offer the advantage of capturing the individual learning activity to a greater extent than an LMS can [47]. Learning situations become even more complex due to the fact that pedagogical approaches differentiate between formal and informal learning processes. Both have different requirements for the learning environment and, as such, for the recommendation within the environment. Often, it is not possible to draw a clear line between formal and informal learning scenarios. For example, recommender systems need to deal with the tension of recommendations for activities liked by the learner and those required by the teacher [112]. Consequently, the need for massive amounts of data about the user and her activities within all of her learning environments is necessary to facilitate precise recommendations. This leads to the problem of usage data available. Many recommendation algorithms rely on massive amounts of usage data from numerous users to make precise recommendations. In TEL, this situation often does not occur. Instead, most learning activities take place with only a few learners participating. This situation occurs in both formal and informal learning settings, e.g. in courses [1] or in learning networks [41] that assemble and dissolve continuously. In addition, the flexible way of learning makes it problematic to establish automatically when a learning process has more or less successfully terminated. This information is needed in recommender algorithms to identify commonalities between learners, e.g. the closeness of learners based on their way of completing a learning process. Therefore, recommendation algorithms need to deal with this issue by being able to handle incomplete learning processes through feedback by learners. Fee back of successful recommendations is rather implicit in TEL [99], so that further information about the learner and her learning environment are often necessary.
Page 27

At least some tailoring of the respective approach is necessary. In this article, we outline how such tailoring is possible by taking the context of the learner into account in a much more specific way than applied in todayA Z s recommendation approaches. First, we present a brief overview of existing TEL recommender systems that rely on traditional recommendation algorithms to generate recommendations. Then, we discuss context definitions and existing research on the incorporation of contextual information in the recommendation process.
EXAMPLES OF RECOMMENDER SYSTEMS FOR LEARNING MATERIALS

In this section, we present existing recommender systems in TEL that rely on standard collaborative filtering, content-based filtering, knowledge-based filtering or hybrid recommendation algorithms. Collaborative filtering recog- nizes commonalities between users or between items on the basis of explicit (ratings, tags, etc.) or implicit (actions like reading, downloading, etc.) relevance indicators and generates recommendations based on preferences of other users [28]. Content-based filtering matches descriptions of items to descriptions of users [87]. They base their predictions on information about individual users and items, and ignore contributions from other users. Knowledge-based filtering attempts to suggest items based on inferences about user needs and preferences. In some sense, all recommendation techniques could be described as doing some kind of inference [24]. Knowledgebased approaches are distinguished in that they have have knowledge about how a particular item meets a particular user need, and can therefore reason about the relationship between a need and a possible recommendation. Hybrids combine recommendation techniques, to gain better performance with fewer drawbacks [26]. A recent discussion of the advantages and drawbacks of the various techniques has been presented in [40]. Recommender systems in TEL are quite diverse. Most systems suggest learning resources, learning activities or peer learners to users. Examples include pioneering work in this area from Tang and McCalla [112], InLinx [20], Papyres [77], QSIA [89], the exam question recommender [51], ALOCOM [119], Altered Vista [91], PEL-IRT [34], Lu et al. [69], RACOFI [65], PHelpS [49], CYCLADES [7] and prominent work of Sicilia et al. [107] on recommendation of MERLOT [100] learning resources. Course recommenders typically provide advice on courses to enroll in. Examples in this area are AACORN [97], CourseAgent [45], RARE [14] and CourseRank [48]. Manouselis et al. [72] also surveyed social navigation systems that rely on recommendation techniques to suggest resources within adaptive hypermedia systems. Prominent examples are CoFIND [21], Knowledge Sea II [22] and research of Romero et al. [94]. While not traditionally considered as recommender systems, Intelligent Tutoring Systems (ITS) use information about the learner to suggest personalized hints, advice and/or resources while the learner is solving a problem. Therefore, we also include two prominent examples, Algebra Cognitive Tutor [61] and SQL Tutor [76], in this analysis. As outlined by Manouselis et al. [72], there has been much advancement in this field. However, one of the challenges that research in this field is currently facing is thorough evaluations of the impact of recommendation on learning. Whereas trials with human users often provide useful insight into the usefulness of a system, experimental investigation and parametrization of recommendation algorithms with data that has been captured in real-life settings is conducted rarely. In addition, current TEL recommender systems mainly use information about learners and resources. Although multiple attributes are often considered, additional context dimensions (such as the current location or device) are not incorporated in most of these systems. The systems do in many cases use implicit data collection methods and often combine different recommendation techniques to incorporate multiple data elements in the recommendation process. Therefore, many of these systems have potential for future extensions that incorporate contextual data in the recommendation process.
CONTEXT-AWA RE RECOMMENDERS
Traditionally, collaborative, content-based, knowledge-based and hybrid recommender systems deal with two types of entities, users and items. As discussed in Section 2.1.1, TEL applications have inherent additional complexities and may not fit well into the traditional two-dimensional user/item approach based on ratings only. Of interest for TEL recommenders is the incorporation of additional information about learners and teachers and
Page 28

their context in the recommendation process. Such data can be used to adapt recommendations based on individual learner characteristics, such as learning goals and knowledge levels, and additional contextual information such as time, location, people nearby, etc. Pioneering work on context-aware recommender systems (CARS) has been done by Adomavicius and Tuzhilin [5]. The authors researched approaches where the traditional user/item paradigm was extended to support additional dimensions capturing the context in which recommendations are made. Contextual information can be obtained in a number of ways:

Explicit context capturing relies on manual input by users. Registration modules are often used to capture information of users or rating modules are used to retrieve interests and preferences. Implicit context capturing methods capture contextual information automatically from the environment, for instance by obtaining the current location or device type. Contextual information can also be inferred by analyzing user interactions with tools and resources, for instance to estimate the current task of the user.
Several paradigms have been proposed to incorporate contextual information in the recommendation process. A first recommendation via context-driven querying and search approach uses contextual information to query or search a certain repository of resources (e.g., restaurants) and presents the best matching resources (e.g., nearby restaurants that are currently open) to the user. A second contextual preference elicitation and estimation approach is a more recent trend in context-aware recommender system research. This approach attempts to model and learn contextual user preferences. These recommender systems are built on knowledge of partial contextual user preferences and typically deal with data records of the form <user, item, context, rating>. Each record therefore captures how much a user liked a particular item in a specific context (e.g. weekend or weekday). Three approaches have been identified to deal with such contextual preferences. In a contextual pre-filtering approach, contextual information is used to filter the dataset before applying a traditional recommendation algorithm. In a contextual post-filtering approach, recommendations are generated on the entire dataset. The resulting set of recommendations is adjusted using the contextual information. Contextual modeling approaches use contextual information directly in the recommendation function as an explicit predictor of a rating for an item. Whereas contextual pre-filtering and post-filtering approaches can use traditional recommendation algorithms, the contextual modeling approach uses multi-dimensional recommendation algorithms. Examples of heuristic-based and model-based approaches have been described in [4]. Several contextual recommender systems have been developed that use these paradigms in various application domains. Examples include context-aware recommender systems that suggest gas stations to a driver of a car [8] and contextualized media delivery systems such as CoMeR (COntext-aware Media Recommendations) [130] and CA-MRS (Context-Aware Music Recommendation System) [84]. Context-aware recommendation technologies are also researched for building intelligent tourist guides [5]. For example, COMPASS (COntextaware Mobile Personal ASSistant) [116] is a context-aware recommendation system that uses a context-driven querying and search approach to provide a tourist with relevant information, such as nearby monuments, hotels and nearby people. In an evaluation experiment, time and location were used to contextualize recommendations. Interestingly, the authors report that last time visited had a negative influence on the perceived usefulness of the recommender system. These results illustrate that careful analysis of data that is taken into account is necessary when deploying contextualization algorithms. The influence of various parameters on the recommendation process is therefore currently of major interest. This challenge has been identified by several authors. Yujie and Licai [131] outline that is difficult to describe clearly and uniformly what types of contexts are truly needed in CARS because of the variety of application scenarios and user needs. Discovering valid context types and instances and then implementing them are therefore serious challenges that CARS should face and resolve. Adomavicius et al. [5] identify the development of highperforming context-aware recommender systems and testing them on practical applications as an important challenge. They argue that most work on context-aware recommender systems has been conceptual, where a certain method has been developed, tested on some (often limited) data, and shown to perform well in comparison to certain benchmarks. Among others, they argue that there has been little work done on developing novel data structures and new systems architectures for CARS that incorporate context sensors and various filters and converters in a modular fashion. A third important challenge is the evaluation and lack of publicly available datasets of CARS. In order to assess the impact of various contextual parameters, datasets are needed that contain contextual data. The interest in this area is reflected by the organization of several workshops and challenges
Page 29

related to contextual movie recommendation that have been organized in recent years [3][6]. Overall, the field of contextual recommender systems is promising, but much work is needed to explore it compre- hensively. In this article, we explore the challenges outlined above for the development of CARS for learning. As a first step in this direction, we define a context framework for TEL that identifies relevant context categories for the analysis and development of CARS for learning. Then, we present an analysis of CARS that have been deployed in recent years within the TEL domain. In particular, we focus on context dimensions that are used to contextualize recommendations, context sensors that are used to capture relevant contextual information, contextual recommendation algorithms that are used and evaluation methods and results.
Page 30
Page 31
Page 32
Mental Models, Strategies and Metacognition in Programming, and its relation to Verbal Protocols as a learning mechanism3.
Carlos Argelio Arvalo Mercado, Lizeth Itziguery Solano Romo
Department of Information Systems, Autonomous University of Aguascalientes Av. Universidad 940, CP 20131, Mxico. carevalo@correo.uaa.mx, lisolano@correo.uaa.mx Abstract. Learning to program is difficult for many first year undergraduate students. Instructional strategies of traditional programming courses tend to focus on syntactic issues and assigning practice exercises using the presentation-examples-practice formula and by showing the verbal and visual explanation of a teacher during the step by step process of writing a computer program. Cognitive literature regarding the mental processes involved in programming suggests that the explicit teaching of aspects such as mental models, strategic knowledge and metacognitive abilities, are critical issues of how to write and assemble the pieces of a computer program. Related to these three aspects, also from cognitive sciences, Verbal Protocols are often used as a technique to record the short term cognitive process of an expert in problem solving scenarios. With this background, this paper reports the seminal papers, as well as the most recent studies regarding Mental Models, Strategies, and Metacognition in basic programming and in others contexts, and tries to establish a link between these and Verbal Protocols as a mean to effectively transfer basic programming skills.
Key words: programming, mental models, strategies, Metacognition, verbal protocols.
INTRODUCTION
The programming context The difficulties many undergraduate students face when learning programming are still a common topic in cognitive, educational and technological research literature. The problem has been approached from many angles, such as the study of the cognitive behavior of novices and experts (Fixx et al. 1993; Ma et al. 2007; George 2000) some creative pedagogical strategies (Dagdilelis 2004; Hagan 1998; Jenkins 1998), the cultural environment of the student (Bruce 2004; Booth 2001), and of course the use of software tools (McIver 1999; Kujansuu 2006; Conway 1997; McKeown 2004). In developing countries such as Mexico, programming skills are relevant for undergraduate students, given the increasing trends of first world economies to outsource programming, information technology and software related jobs (de Raadt et al. 2003; Koong et al. 2002; Litecky et al. 2006).
3
The themes discussed in this paper will be extended and explored with support of a three year European Commission ALFA III project grant (DCI-ALA/19.09.01/21526/245-315/ALFAHI(2010)/123), IGUAL Innovation for Equality in Latin American University beginning January 2011.
Page 33

But the most visible aspect of the problem is the almost universal pattern of high failure rates among first year computer science programming students. Depending on the source, it can be found that this failure rates range from 30% to 60% (Boyle 2003; Guzdial 2002; V. Aleven & K. Koedinger 2002; UAA 2007). The Traditional teaching model. Teaching programming is often based on the pedagogical pattern of: 1) presenting the topic, 2) showing a few examples, and 3) assigning practice exercises; that is, the presentation-examples-practice formula (Shaffer 2005). And so, a traditional programming course is mainly based on theoretical lecture sessions and practical work on computer laboratories, where most of the content is focused on the characteristics of the computer language being taught. Researchers (Linn & Clancy 1992; Linn 1985; McGill & Volet 1997; Robins et al. 2003) agree that most introductory programming courses are reasonably good to emphasize syntax comprehension of programs, but that they do not reinforce the strategic kind of knowledge also required to write programs. A common perception of computer programming educators is the assumption that this strategic knowledge will develop itself as a byproduct of curricular design (Mead et al. 2006), while literature suggests that a more effective approach is that this knowledge has to be explicitly taught (Robins et al. 2003; Volet & Lund 1994). Software Tools. During the last four decades, researchers and designers have been trying to make programming more appealing to students and to the public. They have developed a wide variety of software applications, to make programming skills easier to acquire. From Logo (Papert 1980), to Alice (Conway 1997) a diversity of learning goals have been pursued: to develop problem resolution abilities, to develop logical thinking through games, or to facilitate the transition to general purpose programming languages by way of alternative and easier to use interfaces, among other goals. Kelleher & Pausch (2005) did a survey of approximately 80 software tools designed to teach programming or to foster the interest in programming by way of games, animations or puzzles. Other recent categories of software tools designed to be an aid in teaching programming are: a) program visualization; by using graphics that enable the student to visualize the behavior of algorithms and data structures (Pollack 2004), b) learning objects, small instructional components that can be reused in several contexts (Kujansuu 2006), c) concept maps, that work like big knowledge scaffolds to represent the main concepts of programming, and to combine them with other teaching strategies and tools (Maries & A. Kumar 2007; A. N. Kumar 2006), and d) cognitive tutors, that use declarative and procedural knowledge in the form of production rules to give guided feedback to the student (J. R. Anderson et al. 1995; Vincent Aleven et al. 2006). All these types of tools have reported positive results, and have had various degrees of success in real life contexts in the goal of teaching programming4, but we argue that while some of them have been adopted by programming educators, most have focused only on a limited subset of the cognitive aspects (e.g. the transfer of mental models through graphics or interactive feedback) that literature has hinted as critical.
MENTAL MODELS
Norman (1983) introduced the term Mental Model and defined it as the conceptualization that a user has of a target system. A mental model can be seen as the internal representation of a task or complex system (George 2000). Its construction allows the subject to predict and comprehend the function of that system or task. Factors that affect the development of a mental model includes the preceding technical knowledge of the user, previous experience with similar systems and the persons own knowledge structure patterns. Also, it is explained that in problem solving, in the absence of a previous mental model, the human brain has a tendency to use general problem solving strategies (for example, means-ends analysis) that seem to apply in principle to most systems or problems, but that demand for more cognitive effort (J. Sweller et al. 1998). So, the mental model serves as a quick reference to the user to predict the state of a system.
For example, Cognitive Tutors have been very successful at teaching Algebra to High School students in the United States.
Page 34

In programming a mental model is the image that the programmer has about the internal processing that occurs inside the machine, between an input and an output. In this context, writing a program involves having many and very diverse mental models, separated from the syntax of the programming language (Johnson-Laird 1983). The existence of correct mental models in novice programmers has been seen as critical in the first stages of the process of learning to program (Ma et al. 2007; Bornat & Dehnadi, Simon 2008; Ramalingam 2004; Winslow 1996), and in some studies, it has been presented as a predictor of success or failure (Wiedenbeck et al. 2004; Dehnadi et al. 2009). In cognitive studies about programming, evaluation and measurement of a subjects mental model is usually done with code comprehension tests. For example, one question of a designed test would have the following form (Dehnadi & Bornat 2006): int a = 10; int b = 20; a = b; If the student selects a correct value of a and b, then it is implied that he or she possesses a valid mental model 5. In this context, (Dehnadi et al. 2009) identified, for example, eleven possible mental models associated with the value assignment operation of variables of students in their first programming course (see Table 1).
Table 1. Value assigment mental models. (Taken from Dehnadi et al. 2009)
So, it is understood that to write a program a person has to have many and very diverse mental models, referring, for example, as to how a loop, a data structure or decision structure behaves (Johnson-Laird 1983). Winslow (1996) notes that the existence of a wide range of valid mental models is critical (but not enough) for the novice to acquire the ability to write programs, and if these mental models are not explicitly taught, the student will anyway create its own, of dubious quality and effectiveness. Another line of research regarding mental models in programming, that started in the 1990s and has continued in the 2000s, has to do with comparing the performance of experts an novices, an thus, their mental models. And so it has been found that in the subject of mental models, experts: (Fixx et al. 1993; Hegarty 1993; Ma et al. 2007; Ramalingam 2004) a) Have well organized schemas or structures of specialized knowledge. b) Organize their knowledge according to functional characteristics (such as the nature of the underlying algorithm, instead of the low level details of syntax). c) Combine general as well as specific problem solving strategies. d) Are flexible in the use of strategies and can abandon hypothesis that turn out to be false during problem solving. e) Use bottom up strategies to decompose and comprehend programs. As it can be seen, the concept of mental model, and the notion of strategy are closely linked during the process of programming, as it will be discussed in the following section.
In Table 1, the valid mental model is M2.
Page 35
Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support STRATEGIES

It has been found that even though a student is in fact able to acquire valid mental models, and knows the correct syntax of a programming language, another key cognitive element is still necessary for him to write effective programs. This component is called strategy (Rist 2004), also known as schemas, plans or clichs. Strategies are predefined solutions to stereotyped kinds of problems. In one seminal work about the subject, Gilmore (1990) presented a set of studies to demonstrate that expert programmers have a much wider repertoire of strategies available than novices. The lack of a minimum amount of these strategies restricts the student ability to recognize certain types of problems, and therefore their solution. Brooks (1990) indicates that an important aspect of strategies is that they cannot be deducted from the final form of the program. This means that a novice can study the final shape of a program, but unless explicitly taught by a teacher, he or she cannot see the process and strategies involved in its writing. The final form of a program can give the student information about the concepts and syntactic structures used, but not about the strategies and decisions applied during the writing process. Spohrer & Soloway (1989), hinted on the fact that students were not given enough instruction about how to put the pieces of a program together and that a focus on strategies could help to revert that situation. In 2003, Robins et.al. (p.166) suggested that strategies in programming have to receive more explicit attention in introductory courses, and that one way of achieving this is by introducing many examples of programs as they are being written, and discuss the strategies being used. Indeed, these strategies are difficult to teach in the classroom and laboratories, but (Rist 1995) notes that in programming, there is considerable empirical evidence that suggests that strategies are the main basic cognitive component used in design and program comprehension. In recent years, (Skubch & Thielscher 2005) proposed a method for knowledge-based agents to learn strategies. They take the approach of using techniques of inductive logic programming, so that strategies are learned in two steps: A given example set is first generalized into an overly general theory, which then gets refined. They present this study as a first step toward the long term goal of adaptive, reasoning agents capable of changing their behavior when appropriate. The notion of strategy is still been explored in contexts such as mathematics. Heeren et al. (2010) proposed a language for specifying strategies for solving exercises, to automatically calculate feedback, for example when a user makes an erroneous step in a calculation. They propose to automatically generate worked-out examples, track the progress of a student by inspecting submitted intermediate answers, and report back suggestions in case the student deviates from the strategy. They see that a strategy in mathematics describes valid sequences of rewrite rules, which turns tracking intermediate steps into a parsing (that is, computational) problem. Davies (1993) clarifies another important issue related to strategies: the process of writing a program does not have to be understood as a literal transcription of a previously stored and typified solution, but rather as an iterative, exploratory, and incremental process determined by minor episodes of problem solving and constant reevaluation of the effectiveness of applied strategies. That is, the effectiveness of a set of strategies is constantly monitored and evaluated by the programmer, in the process of writing the program. These finding leads directly to another important cognitive aspect of programming (and of problem solving in general) called Metacognition.
METACOGNITION
(Flavell 1979) described Metacognition as awareness of a persons own cognitive process. While strategies allow a programmer to solve problems, Metacognition allows him to monitor its progress, apply his knowledge to new situations, and identify its own limitations. (Gourgey 1998) indicates that through Metacognition a student can define the nature of a problem or task, select useful mental models, use the most pertinent strategy to implement them and put attention to feedback as to how he or she is making progress towards the solution. In the context of programming, favorable results have been reported through the use of instructional strategies that explicitly focus on Metacognition, such as pair programming (Williams & Upchurch 2001), that is, a pair of novice programmers monitor each others progress, with constant feedback; and the use of think-alouds (Arshad 2009), that consists of instructing the students to verbally reproduce their thought process when writing a program, thus explicitly making such students aware of the decisions, and problem solving strategies they are applying. These are clear examples of Metacognition in the programming context. Wilson & Bai (2010, p. 271) state that since metacognition is key to comprehension, it must be a valued component in literacy instruction. Such instruction must address student background knowledge, knowledge and
Page 36

practice of metacognitive strategies, and knowledge regarding implementation of the strategies. They clarify that metacognitive focused instruction includes long term direct explanation and modeling of strategies followed by guided student practice with the strategies. That is, students must understand the what, how, and when of the strategies to be applied. A student needs to know what the metacognitive strategies are, how to implement them, and under what conditions to implement them. They give a clear example in the context of reading: a student who finds a confusing phrase needs to do something. If he simply keeps reading despite not understanding it, he is not being metacognitive; but if the student stops to ask a question and/or backtrack and reread, he is indeed applying a particular strategy for a specific purpose when it is needed. To recognize the cognitive behaviors involved in the process of metacognition, and to design instructional strategies that include them, it is necessary to understand its basic components (Flavell 1979), which are: Planning. Has to do with selecting the appropriate strategies for a certain problem and the allocation of resources that could affect performance. Monitoring. Involves paying attention to feedback on the current state of a task and translating this feedback into improved performance, either during execution or in a plan for the future. Monitoring (also called regulation) pertains to the revision of progress toward goals. Evaluation. Refers to the student ongoing assessment of his own scope of knowledge or understanding, resources, tasks, and goals. Other researchers have added to Flavells original components of Metacognition. For example, in the context of reading, (Meijer et al. 2006) differentiate between planning and orientation. They add that orientation activities concern goal setting, activating prior knowledge, predicting content before acting6 and establishing task demands. Also as a subcomponent of planning, Butler (1998) identified task-analysis, as a form of interpreting task requirements. She emphasized that successful task interpretation is critical to focused engagement in tasks. During task interpretation, students decipher the requirements of a given task and are aware of the task demands. Then, students self-regulate .further learning Given the importance of metacognition in education, researchers such as (Roll et al. 2007), have proposed a set of pedagogical and technical principles to design effective software solutions that apply metacognition in learning, mainly based on the model of Computational Psychology developed in Carnegie Mellon University (J. Anderson et al. 2004). This model has strong literature background and empirical support, and has derived in software artifacts know as cognitive tutors (J. R. Anderson et al. 1995; Keneth Koedinger & Vincent Aleven 2004; V. Aleven et al. 2006), but from our experience, they have the drawback of being complex and costly to develop. To summarize, and given the empirical evidence shown in cognitive literature, we argue that these three cognitive components (functional Mental Models, Strategies and Metacognition) are at the same time critical to acquire the ability to program and also tightly interlinked. The absence of one of these components could be an impediment to acquire the ability to program. This argument is synthesized in Figure 1.
This, in the programming domain, can be interpreted as the concept of mental model, which has to do with predicting how a system or operation will work.
Page 37
Figure 1. Cognitive aspects of programming and their relation.
VERBAL PROTOCOLS
Verbal protocols, as a method of representation and analysis of a persons thought processes, have a solid tradition in the context of cognitive psychology (Ericsson & Simon 1993; Newell & Simon 1972; Russo et al. 1989). As a technique, verbal protocols were initially developed to analyze a sample of a persons short term memory processes (to what things they are paying attention to, and in what order, when given a certain task?), and find patterns of behavior. In time, this technique has been extensively used in other disciplines such as usability studies (Knox et al. 1989; Nielsen et al. 2002), software task analysis (Vessey & Conger 1994), and more recently in programming teaching (Arshad 2009). According to the classic work of Ericsson & Simon (1993), the premise behind Verbal Protocols is that of viewing humans as information processors, in which information enters the mind, it is processed in a series of ordered stages in short term memory, additional information is retrieved from long term memory, and the results of cognitive processing are outputs and/or saved in long term memory. A central characteristic of this approach is knowing that human cognitive resources are limited (Hungerford et al. 2004); therefore, complex tasks may overload these resources and decrease the quality and/or quantity of outputs. Aspects such as Attention, corresponds to the elements of human cognition that the subject can process simultaneously at a given time, and it is a limited resource, whether viewed as a singular or multiple-channel. Focalization, is concentration of consciousness to one particular discrete cognitive element. It implies withdrawal (not paying attention to) from some things in order to deal more effectively with others. Therefore, problem solvers selectively direct their attention to stimuli and their processing, and the patterns of that focalization reveal systematic and unique ways of working on tasks. And so, Verbal protocols document (by means of transcribed audio and/or video recordings) the mental behavior of a subject, when he or she is asked to think aloud, while performing a previously designed task. Later, the transcribed documents that is, the protocols are analyzed to be used in whatever research objective is needed (traditionally, to find patterns and common structures of behavior). A brief summary of the process of producing a verbal protocol can be as follows (adapted from Nielsen et al. 2002). a) Transcription. With the advent of new technology, came the possibility of transcribing a recorded audio (and visual) session with a series of subject. But Ericsson & Simon (1993) warn that data do not speak for itself, especially in the human memory system where it is difficult (if not impossible) to have first
Page 38

hand observations. Data must always be coded and interpreted within a theoretical framework7. A special situation has to be taken into account during recording: if the subject tends to be silent, the interviewer has to remind him to keep talking because, according to Ericsson & Simon, thinking in already encoded verbal forms is fast, the activation of old thoughts is somewhat slower and the generation of new thoughts is really slow. That is why it is important to slow down the thought processes (and speed up the verbalization) and why they suggest the subject to keep talking. b) Segmentation. Protocols should be fragmented into segments that correspond to individual verbalizations, and these in turn, correspond to an instance of a general process. Although there is no universal criteria as to how to clearly identify these segments, some clues can be noted from pauses, intonations and grammatical markers of phrases and sentences. c) Coding. A coding scheme has to be developed a-priori, and a vocabulary has to be elaborated in order to a) perform an individual task analysis and b) to perform an general protocol analysis. A coding scheme could help to identify elements, dictated by theory or other predefined criteria. As it can be inferred, verbal protocols and metacognition are closely related, in the sense that most metacognitive studies involve some use of think-aloud methods for analysis. Pugalee (2004) states that the internal thought process of a person is not altered when verbalizing information not normally attended while performing a task. They claim that there are no differences in the specific steps taken by people who either solved a problem silently or while talking aloud.
In Software Engineering this may not be the case: the focus point is a software artifact or diagram being tested or analyzed. But still, when the objective of the research is interpretative or predictive, verbal data needs something to be contrasted against.
Page 39
FINAL REMARKS
As previously seen, verbal protocols can be used as a mechanism to record the cognitive process of a programming instructor, when he or she is writing a program. Therefore we propose that a significantly varied set of verbal protocols can be used to explicitly show a student: 1. What mental models a programming expert is using. 2. What type of problem the expert has recognized (and why), and the strategies that he or she has applied (the interlinked nature of metacognition and strategy). 3. How the programming expert identifies when he or she has made a mistake and has to backtrack to correct it (monitoring). These three components are the elementary cognitive aspects discussed in previous sections of this paper, and schematized in Figure 1. With this in mind, an audio-visual form of experts verbal protocols could be recorded, transcribed, edited, produced and later shown to students, in the form of worked examples (Gerjets et al. 2004; Gerjets et al. 2004; Ward & J. Sweller 1990; J. Sweller & Cooper 1985; Moreno 2006). A recording session would consist of asking an expert programmer to write a program to solve a basic and typified problem, while verbalizing his or her thought process. The actual process of recording can be done using common available video capture software, to not only record the verbal (think-aloud) data, but also the behavior of the expert programmer in the act of writing code. Taking the idea one step further, the resulting set of protocols could be loaded into a software tool designed to visualize them, in order to reduce cognitive load (J. Sweller et al. 1998; J. Sweller 1988; John Sweller 1994), even though the pedagogical strategy of worked examples must be applied carefully, for its limitations (Kalyuga et al. 2001; Moreno 2006; van Merrienboer & Krammer 1990). The effectiveness of this tool can be tested in controlled environments, contrasting the performance of students in contexts of traditional teaching methods, against students using this protocol visualizing tool. It is intended that this proposed tool will be designed, tested developed and integrated with a set of learning solutions, in the context of the Alfa III funded IGUAL project8
REFERENCES
Aleven, V. & Koedinger, K., 2002. An effective metacognitive strategy: learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26, p.147179. Aleven, V. et al., 2006. Rapid authoring of intelligent tutors for real-world and experimental use. En International Conference on Intelligent Tutoring Systems (ITS 2006). Berlin, pp 61-70. Aleven, Vincent et al., 2006. The Cognitive Tutor Authoring Tools (CTAT): Preliminary evaluation of efficiency gains. En Intelligent Tutoring Systems Conference. Maceio, Brazil. Anderson, J.R., Corbet, A.T. & Koedinger, K.R., 1995. Cognitive Tutors, Lessons Learned. The Journal of Learning Sciences, 4(2), pp.167-207. Anderson, J., Bothell, D. & Byrne, M., 2004. An Integrated Theory of the mind. Psychological Review, 111(4), pp.1036-1060. Arshad, N., 2009. Teaching Programming and Problem Solving to CS2 Students using Think-Alouds. ACM SIGCSE Bulletin, 41(1), pp.372-376. Bayman, P., 1983. A diagnosis of beginning programmers misconceptions of BASIC programming statements. Communications of the ACM, 26(9), pp.677 - 679. Booth, S., 2001. Learning to program as entering the datalogical culture: a phenomenographic exploration. En Fribourg Switzerland, August 2001. Bornat, R. & Dehnadi, S., Simon, 2008. Mental models, Consistency and Programming Aptitude. En 20th Annual Workshop. Psychology of Programming interested Group (PPIG). Boyle, T., 2003. Improved success rates for students studying Programming. Investigations in university teaching and learning, 1(1), pp.52-54.
8
(DCI-ALA/19.09.01/21526/245-315/ALFAHI(2010)/123), IGUAL Innovation for Equality in Latin American University
Page 40

Brooks, R.E., 1990. Categories of programming knowledge and their application. International Journal of ManMachine Studies, 18, pp.543-554. Bruce, C., 2004. Ways of experiencing the act of learning to program: A phenomenographic study of introductory programming students at university. Journal of Information Technology Education, 3, p.144. Conway, M.J., 1997. Alice:Easy-to-Learn 3D Scripting for Novices. PhD. Thesis. University of Virginia. Dagdilelis, V.S., 2004. Introducing Secondary Education Students to Algorithms and Programming. Education and Information Technologies, 9(2), p.159173. Davies, S.P., 1993. Models and Theories of programming strategy. International Journal of Man-Machine Studies, 39, pp.237-267. Dehnadi, S. & Bornat, R., 2006. The camel has two humps, Middlesex University. Available at: http://www.cs.mdx.ac.uk/research/PhDArea/saeed. Dehnadi, S., Bornat, R. & Adams, R., 2009. Meta-analysis of the e_ffect of consistency on success in early learning of programming. En 21st Annual Workshop. Psychology of Programming interested Group (PPIG). University of Limerick, Ireland. Ericsson, K. & Simon, H.A., 1993. Protocol Analysis. Verbal reports as data (rev. ed)., Cambridge Massachusets.: MIT Press. Fixx, V., Wiedenbeck, S. & Scholtz, J., 1993. Mental Representations of Programs by Novices and Experts. En Proceedings of the SIGCHI conference on Human factors in computing systems. Amsterdam, The Netherlands: ACM Press New York, NY, USA, pp 74-79. Flavell, J.H., 1979. Metacognition and cognitive monitoring: A new area of cognitive developmental inquiry. American Psychologist, 34(10), p.906911. George, C.E., 2000. Experiences with Novices: The Importance of Graphical Representations in Supporting Mental Models. En Cozenza Italy. A.F. Blackwell & E. Bilotta. Gerjets, P., Scheiter, K. & Catrambone, R., 2004. Designing Instructional Examples to Reduce Intrinsic Cognitive Load: Molar versus Modular Presentation of Solution Procedures. Instructional Science, 32(1-2), pp.33-58. Gilmore, D.J., 1990. Expert Programming Knowledge: a strategic Approach. En Psychology of Programming. European Association of Cognitive Ergonomics and Academic Press, pp 223-234. Available at: http://www.cl.cam.ac.uk/teaching/1011/R201/. Gourgey, A.F., 1998. Metacognition in basic skills instruction. Instructional Science, 26(1-2), pp.81-96. Guzdial, M.S., 2002. Teaching the Nintendo generation how to program. Communications of the ACM, 45(4), pp.17-21. Hagan, D.S., 1998. The value of discussion classes for teaching introductory programming. En Dublin City Univ., Ireland: ACM Press New York, NY, USA, pp 108 - 111. Heeren, B., Jeuring, J. & Gerdes, A., 2010. Specifying Rewrite Strategies for Interactive Exercises. Mathematics in computer science, 3(3), pp.349-370. Hegarty, M.J., 1993. Constructing mental models from text and diagrams. J. Mem. Lang., 32, p.717742. Hungerford, B.C., Hevner, A.R. & Collins, R.W., 2004. Reviewing Software Diagrams: A Cognitive Study. IEEE Transactions on software engineering, 30(2), pp.82-96. Jenkins, T., 1998. A participative approach to teaching programming. En Dublin City Univ., Ireland: ACM Press New York, NY, USA, pp 125 - 129. Johnson-Laird, P.N., 1983. Mental Models, Cambridge University Press. Kalyuga, S. et al., 2001. When problem solving is superior to studying worked examples. Journal of Educational Psychology, 93, p.579588. Kelleher, C.P. & Pausch, R., 2005. Lowering the Barriers to Programming: a survey of programming environments and languages for novice programmers. ACM Computing surveys (CSUR), 37(2), pp.83 - 137. Knox, S.T., Bailey, W.A. & Lynch, E.F., 1989. Directed dialogue protocols: verbal data for user interface design. En Conference on Human Factors in Computing Systems. ACM New York, NY, USA, pp 283 - 287. Koedinger, Keneth & Aleven, Vincent, 2004. Toward a Rapid Development Environment for Cognitive Tutors. En Engineering Advanced Web Applications: Proceedings of Workshops in Connection with the 4th International Conference on Web Engineering. Princeton: Rinton Press., pp 167-179. Koong, K.S., Liu, L.C. & Liu, X., 2002. A Study of the Demand for Information TechnologyProfessionals in Selected Internet Job Portals. Journal of Information Systems Education, 13(1), pp.21-28. Kujansuu, E., 2006. Using program visualisation learning objects with non-major students with different study background. En Tampere, Finland, pp 21-26. Kumar, A.N., 2006. Using Enhanced Concept Map for Student Modeling in Programming Tutors. En FLAIRS
Page 41

Conference. Melbourne Beach, Florida, pp 527-532. Linn, M.C., 1985. The Cognitive Consequences of Programming Instruction in Classrooms. Educational Researcher, 14. Linn, M.C. & Clancy, M.J., 1992. The Case for Case Studies of Programming Problems. Communications of the ACM, 35(3). Litecky, C., Prabhakatar, B. & Arnett, K., 2006. The IT/IS job market: a longitudinal perspective. En Special Interest Group on Computer Personnel Research Annual Conference. Claremont, California, USA, pp 5052. Ma, L. et al., 2007. Investigating the viability of mental models held by novice programmers. ACM SIGCSE Bulletin, 39(1), pp.499-503. Maries, A. & Kumar, A., 2007. Concept maps in intelligent tutors for programming. Journal of Computing Sciences in Colleges, 22(3), p.54. McGill, T.J. & Volet, S.E., 1997. A Conceptual Framework for Analyzing Students Knowledge of Programming. Journal of Research on Computing in Education, 29. McIver, L., 1999. GRAIL: A Zeroth Programming Language. En Chiba, Japan, pp 43-50. McKeown, J., 2004. The use of a multimedia lesson to increase novice programmers understanding of programming array concepts. Journal of Computing Sciences in Colleges, Volume 19(4), pp.39 - 50. Mead, J. et al., 2006. A cognitive approach to identifying measurable milestones for programming skill acquisition. En Annual Joint Conference Integrating Technology into Computer Science Education. Bologna, Italy: ACM New York, NY, USA, pp 182 - 194. Meijer, J., Veenman, M. & Van Hout, B., 2006. Metacognitive Activities in Text-Studying and ProblemSolving:Development of a taxonomy. Educational Research and Evaluation, 12(3), pp.209-237. van Merrienboer, J. & Krammer, H., 1990. The completion strategy in programming instruction: Theoretical and empirical support. En Research on Instruction. Englewood Cliffs, NJ: Educational Technology Publications, pp 45-61. Moreno, R., 2006. When worked examples dont work: Is cognitive loadtheory at an Impasse? Learning and Instruction, 16, pp.170-81. Newell, A. & Simon, H.A., 1972. Human Problem Solving, Englewood Cliffs, NJ: Prentice Hall. Nielsen, J., Clemmensen, T. & Carsten, Y., 2002. Getting access to what goes on in peoples heads?: reflections on the think-aloud technique. En Nordic Conference on Human-Computer Interaction. Aarhus, Denmark: ACM New York, NY, USA, pp 101 - 110. Norman, D.A., 1983. Some observations on mental models., Erlbaum, Hillsdale, NJ. Papert, S., 1980. Mindstorms, children, computers and powerful ideas, Basic Books, New York. Pollack, S.B.-A., 2004. Selecting a Visualization System. En Warwick, UK. Pugalee, D.K., 2004. A Comparison Of Verbal And Written Descriptions Of Students Problem Solving Processes. Educational Studies in Mathematics, 55, p.2747. de Raadt, M., Watson, R. & Toleman, M., 2003. Language tug-of-war: industry demand and academic choice. En Conferences in Research and Practice in Information Technology. Adelaide, Australia, pp 137 - 142. Ramalingam, V.L., 2004. Self-Efficacy and mental models in learning to program. En Leeds, United Kingdom: ACM Press New York, NY, USA, pp 171 - 175. Rist, R.S., 2004. Learning to program: schema creation, application and evaluation. En Computer Science Education and Research. Netherlands: Taylor & Francis Group, pp 175-197. Rist, R.S., 1995. Program Structure and Design. Cognitive Science, 19(4), pp.507-562. Robins, A., Rountree, J. & Rountree, N., 2003. Learning and Teaching Programming: A Review. Computer Science Education, 13(2), p.137172. Roll, I. et al., 2007. Designing for metacognitionapplying cognitive tutorprinciples to the tutoring of help seeking. Metacognition Learning, 2, p.125140. Russo, J.E., Jonson, E.J. & Stephens, D.L., 1989. The validity of verbal methods. Memory Cognition, 17(6), pp.759769. Shaffer, S., 2005. Ludwig: An online programming tutoring and assesment system. Inroads - The SIGCSE Bulletin, 37, pp.56-60. Skubch, H. & Thielscher, M., 2005. Strategy Learning for Reasoning Agents. Lecture notes in computer science, 3720/2005, pp.733-740. Spohrer, J.C. & Soloway, E., 1989. Novice mistakes: Are the folks wisdoms correct? En Studying the novice programmer. New Jersey: Lawrence Erlbaum Associates, Hillsdale., pp 401-416.
Page 42

Sweller, J., 1988. Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, pp.257-285. Sweller, J. & Cooper, G., 1985. The use of worked examples as a substitute for problemsolving in learning algebra. Cognition and Instruction, 2, pp.59-89. Sweller, J., van Merrienboer, J. & Paas, F., 1998. Cognitive Architecture and Instructional Design. Educational Psychology Review, 10(3), pp.251-295. Sweller, John, 1994. Cognitive Load Theory, Learning Difficulty and Instructional Design. Learning and Instruction, 4, pp.295-312. UAA, D. de E.I., 2007. Porcentajes de reprobacin en la materia de Programacin I, en las carreras de LI-LTI/ISC durante los aos 2004 y 2007, Aguascalientes: Universidad Autnoma de Aguascalientes. Vessey, I. & Conger, S., 1994. Requirements Specification: Learning Object, Process, and Data Methodologies. Communications of the ACM, 37(5). Volet, S.E. & Lund, C.P., 1994. Metacognitive instruction in introductory computer programming: A better explanatory construct for performance than traditional factors. Journal of educational computing research, 10. Ward, M. & Sweller, J., 1990. Structuring effective worked examples. Cognition and Instruction, 7(1), pp.1-39. Wiedenbeck, S., LaBelle, D. & Kain, V., 2004. Factors affecting course outcomes in introductory programming. En Institute of Technology, Carlow, Ireland. Williams, L. & Upchurch, R.L., 2001. In support of student pair-programming. ACM SIGCSE Bulletin, 33(1), pp.327 - 331. Wilson, N.S. & Bai, H., 2010. The relationships and impact of teachers metacognitive knowledge and pedagogical understandings of metacognition. Metacognition and Learning, 5(3), pp.269-288. Winslow, L.E., 1996. Programming Pedagogy - A psychological Overview. SIGCSE Bulletin, 28, pp.17-22.
Page 43
Learning Designs: Concepts, Standards and Software Tools

Luis A. lvarez-Gonzlez, Erick A. Araya Araya and Jorge Morales Vilugrn
Grupo de Investigacin en Tecnologas de Aprendizaje Instituto de Informtica, Facultad de Ciencias de la Ingeniera. Universidad Austral de Chile, Valdivia, Chile.
Abstract. The paper describes the concepts of Learning Design, the IMS LD specification and the three most used tools to build learning sequences. The concept of Learning Design is explained under the equation if learning situation S then the method M is applied. The metaphor of theatrical script for learning method M is used. The content package of IMS and how the learning flow follows the IMS LD specification is also explained. Finally Reload tool (Learning Design Editor and Learning Design Player), CompendiumLD and LAMS are showed. Keywords: Learning Design, IMD LD, Learning Designs Tools.
INTRODUCTION
The main challenge in education is how to help students to learn in an effective and efficient way. This question is so difficult, because the answers depend on specific situation. The solution X will work best for student Y. In other word, is necessary to have different learning designs for different situation. According to this, a leaning design can defined as an application of a pedagogical model for a specific learning objective, target group, and a specific context or knowledge domain [10]. From ICT point of view, as a response to this definition, there is a standard given by IMS, and several software tools as RELOAD, CompendiumLD and LAMS. To build learning designs, there are basically three ways: 1. Instructional Design. Based on the three main categories of learning theories: behaviorism, cognitivism and constructivism. 2. Best Practices. Basically used by lecturers without pedagogical background, but with good pedagogical strategies. 3. Pedagogical Patterns (see http://www.pedagogicalpatterns.org). Oriented to capture experts knowledge in the practice of teaching and learning. All these forms of knowledge are based on capturing a greater or lesser extent on the Vygotsky's social constructivist theory [15]. The general principles of constructivist theory suggests that learning is an active process, motivation is key, experience plays a critical role, must be contextualized and is a social activity. On the other hand from the student perspective, they shall become responsible for their own learning, and the teacher is the facilitator and guide. FIGURE 1 shows graphically the general principles, the roles of teacher and learner and the goals of social constructivism. In other words, to create a good learning design, some knowledge about of Social Constructivism Theory is required. In the other hand, the IMS Global Consortium Inc. (IMS) works in several specifications, one of them is the IMS Learning Design Specification [9], originally developed at the Open University of Netherland (OUNL). There some software tools to implement learning design under this specification.
LEARNING DESIGNS
A learning design is defined as applying of learning design knowledge, when developing a concrete unit of learning, e.g. a course, a lesson, a curriculum, a learning event. To do this, it is assumed that the quality of learning depends on the quality of design [10]. The vision is broader than a single learning object, because it contemplates
Page 44

the use of tools or processes such as communication between teachers and students. For example, the role of a student may change from time to time to monitor the work of other students.
FIGURE 4. Constructivist learning: Goals, general principles, role of students and role of lecturers. The building of a learning design is made by a designer, from a set of learning outcomes. The designer, does not necessarily teach. A simple example is a design for introductory course on computer programming languages, in this course the students will learn to convert binary numbers to digital numbers. To develop an effective course, the designer must consider the context, e.g. should make different designs for teaching of programming to different careers. To do this you must follow design rules. The construction of learning designs consists of a set of rules, with the following structure [10]:
Page 45

If learning occurs in situation S then use the learning design method M These rules are probabilistic (not deterministic), ie, If learning occurs in situation S then use the learning design method M with probability P The probability P can be improved with practice and it is likely that for the same situation, many other alternative methods could be applied. The learning situation S is a requirement or a description of a situation, ie, they depend on the so-called Learning Results and Learning Conditions. The Learning Results correspond to: 1. Effectiveness: What percentage of students passed the test? 2. Efficiency: What is the cost of achieving effectiveness? 3. Attractiveness: are the students motivated to learn? 4. Accessibility: Do you have online access to course content? The Learning Conditions correspond to: 1. Learning objectives: Knowledge, skills, attitude or competence. 2. Student characteristics: prior knowledge, motivation, situational circumstances. 3. Characteristics of learning: individual or group work at home or classroom. 4. Technical characteristics: bandwidth, synchronous or asynchronous, linear or interactive media types. Finally, the rules can be implemented in a learning designs authoring tool or in a learning design management system. The FIGURE 2 shows the construction rules for learning designs and some software tools.
FIGURE 2. : The construction rules for learning designs The method M describes the learning-teaching process, which is carried on an environment, where teachers and students interact. To describe the learning-teaching process (method M) and environment, a lesson plans is used. The metaphor of a script of a theatrical play is used to write a lesson plan. So, the lesson plan has basically metadata and acts, (see FIGURE 2). TABLE.1 shows a lesson plan to teach how to convert a binary number to its decimal representation. The learning design, like the theatrical script can be repeated many times as needed in different scenarios, with different actors. In other words, it is reusable. The designer as well as the theatrical script writer, is not necessary the director of the theatrical play or one of the actors.
Page 46
TABLE 1. A Lesson Plan to teach how to convert a binary number to its decimal representation Title: Converting a Binary to Decimal Number. Author: Jennifer Alejandra Flaig Seplveda University/Faculty: Universidad Austral de Chile / Facultad de Cs. De la Ingeniera. Fecha: 27- April - 2011 Level: First Semester Subject: Introduction to Programming / Computer Architecture -----------------------------------------------------------------------------First Act The lecturer explains the subject of the activity, also the importance and the properties of the binary system. The lecturer explains step by step the converting process, doing basic examples. The lecturers apply a diagnostic test to the students. Second Act The lecturer forms working group of three students. The lecturer gives four binary numbers to each group. The number must be of three digits or less. The students have 15 minutes to complete the task. To verify the solution, the students use a learning objet. Third Act The lecturer asks the students to write a program in Java, to convert a binary number (with five digits or less) to their decimal representation. When all the groups are finished, each program is tested with any binary number. Forth Act The lecturer applies a final assessment to verify that all the students can convert binary number to their decimal representation.
THE IMS LD SPECIFICATION

In the absence of standardised way to specify learning process, the IMD LD specification is a proposition for a designer to use a language to code the sequences of activities to be performed by students in order to store de results of interactions and for communications facilities in general. IMS LD provides a level of abstraction in the learning process, and it gives the facilities for different pedagogical approaches. The IMS-LD specification is a language for modeling units of learning, where a unit of learning can be a course, a lesson, a curriculum, a learning event. Using this language, designers are able to talk in terms of pedagogical rather than technology. The IMS LD is a language to describe a learning process, and it is described using an XML document. The XML document instance is loaded in a IMS LD application and then played. The IMD LD languages and an IMS LD player are analogous to the relationship between HTML and a browser. The IMS LD specification is sometimes discussed in terms of pedagogical neutrality, because no pedagogical theory is implied in the specification. However, is very important for designers to have a good pedagogical background. The IMS LD is a pedagogic meta-model, which allow pedagogical model (e.g. Problem Based Learning, Competency Based Learning) to be described. For lesson plan, basically two components (metadata and acts) are sufficient, however, the IMS LD specification defines more components for a learning design. The components of a learning design, according to IMS LD specification are: Metadata: Describes title, author, outcomes, etc. The metadata is not part of the theatrical play. Role: Describes the role played, such as teachers or students. Act: The acts are sequential and can be a new act, when there is a change of scenario o just a break, e.g. first act, individual work and second act, group work. Environment: e.g. background music, chairs in a circle, etc. Role-part: Describes the activities of an actor. E.g. Juan seeks the algorithm, on Internet. Activity-structure: You can specify tasks to be performed (lines of text or time).
Page 47

Condition: tells the actor how to adapt to a specific situations. For example, if the students solve the exercises too soon, then they can move to the next activity.
The IMS LD 1.0 specification can be found in http://www.imsglobal.org/learningdesign/index.cfm, where three documents can be found. 1. IMS Learning Design Best Practice Guide. It is the most narrative of the three Specification documents which describe how to implement an IMS Specification. 2. IMS Learning Design Information Binding. Provides detailed descriptions of each of the elements in the Specifications (XML) binding. 3. IMS Learning Design Information Model. Describes the data structures of the Specification. The specification is divided in three parts, knows as Level A, Level B and Level C. Level A contains the bulk of the IMS LD constructs, including activities, environments, plays, acts, roles, services etc; Level B adds Properties and Conditions to level A, which enable personalization and more elaborate sequencing and interactions based on learner portfolios. Properties can be used to direct learning activities as well as record outcomes. Level C adds notifications to Level B. A notification is triggered by an outcome and can make a new activity available for a role to perform. In the FIGURE 3, you can see graphically the three levels of the IMS LD specifications. IMS Learning Design level C
Notifications
IMS Learning Design level B

Properties Conditions Monitor Global
IMS Learning Design level A

Roles Activities Environments Learning Prerequisites
Method
FIGURE 3. : The levels of the IMS Learning Design specification [14]. A Unit of Learning is a complete and independent unit, such as a course, a lesson or module. Creating a Unit of Learning involves the creation of a Learning Design as well as the inclusion of all needed resources such as files, references, websites, evaluation or educational material. Given this, it is necessary to have a packaging mechanism to condense everything into a single element, so it is recommended to use the IMS Content Packaging.
Content Package
A content package is a single package file that contains a file called manifest and content files. The manifest file includes the content package structure, which is described in the organizations section, also includes the list of files contained in the resources section. In the Metadata section the package is described. This is shown in
Page 48

FIGURE 4 a). The package contains only a main manifest, can contain one or more (sub) manifest nested. The LD specification is constructed so that it can fit into a Content Package (now Unit of Learning), replacing only the organization element. See FIGURE 4 b).
FIGURE 4. a) The structure of a Content Package. b) The location of the learning-design element in a Content Package c) The basic structure of the learning-design element (source http://www.imsglobal.org/learningdesign/ldv1p0/imsld_infov1p0.html)
The learning-design has a title, objectives, prerequisites, components and metadata (see FIGURE 3 c).) . The Component section includes roles, activities and learning environment. In the Environment section there are links to resources and services used by activities. In the Method section is the learning flow. A play has one or more acts and each act contains one or more role-parts.
Running a Learning Design

To use an IMS-LD means to be it available for students. To do that, a software player must be used. As in the play acts are performed in sequence, and the play ends when all acts have been completed. The transition from one act to another serves as a focal point for students, ensuring that everyone can move to the next event at the same time (see FIGURE 5). You can perform one or more roles (role-part), which contains a reference to a role and another activity. Every activity has a description and a reference to an environment and the environment may include learning objects (Web pages, content packages, etc.), and the learning services (email, conferences, forums, etc.). The software player provide to each student learning objects and services. IMS LD allows to build learning designs for one student or multiples students. In other words, can be used to individual learning or collaborative learning and also, can be used in multiuser environments.
Page 49
FIGURE 5. A learning flow following the IMS LD specification [10].
SOFTWARE TOOLS
There are several project regarding with learning design, however, there are not many software tools to implement learning designs. The most used are: Reload, is a project focused to develop tools based on IMS LD specification. They develop the Learning Design Editor and Learning Design Player. This project is managed by the University of Bolton. CompendiumLD is a tool developed by the Open University in London and is a form of mind mapping, with a set of icons to create maps to describe argumentations, communications, etc. LAMS is a tool for designing, managing and delivering online collaborative learning activities. Its developed by LAMS International.
Reload
Reload is an acronym for Reusable eLearning Object Authoring & Delivery. Reload is a project that focuses on the development of tools that are based on interoperability specifications for learning technologies. It aims to facilitate the creation, distribution and reuse of learning objects. Is an editor of learning designs, learning content packaging and metadata, learning design player IMS LD and SCORM packages player [http://www.reload.ac.uk]. The learning design editor is based on the IMS LD and lets you create reusable templates where you can define the objectives, activities and learning environments. Reload supports all levels of IMS-LD. With the editor you can import and export content packages according to the IMS Content Package. The runtime environment, also based on IMS-LD allows you to run a unit of learning by selecting and implementing any activity in accordance to the sequence of performances, events, activities and environments associated with that role in the learning design [13]. The FIGURE 6 a) shows a screen of the Reload Learning Design Editor, you can see the tabs Roles, Environments, Activities, Method, and Resources, according the IMS LD specification. FIGURE 6 b) shows the Reload Learning Design Payer, used by student to follow the LD. The LD shown in the FIGURE 6., is the implementation of the lesson plan of the TABLE 1.
Page 50
FIGURE 6 a) Reload Learning Design Editor
FIGURE 6 b) Reload Learning Design Player
LAMS
Page 51

LAMS (Learning Activity Management System) it is an Open Source Web system for creating, managing and delivering learning activities sequences collaboratively. Provides an authoring environment for building learning sequences, an environment of real-time monitoring to track student progress, and an environment for students where they can make a sequence. Activities may include both, individual and group work. LAMS implements the concept of learning design, and is IMS LD specification level A compliant [http://www.lamsinternational.com/]. It focuses more on creating the structure of the learning process than content [7]. FIGURE 7 show the implementation in LAMS of the lesson plan (see TABLA 1). LAMS is like a LMS in the sence of users management. LAMS has four types of users: administrator (creates and edits classes, users, etc.), author (designer of learning sequences), monitor (to guide students in a learning sequence) and learner. LAMS is developed by LAMS Foundation (http://lamsfoundation.org), this organisation also research into concept of Learning Design. LAMS Foundation from 2006, organice every year, the LAMS Conferences, where teachers and developers share experiences, some examples of experiences using LAMS to different students and different matters, are in [2], [3], [4], [5]. Also, there is a community (http://lamscommunity.org) for teachers and developers who use the LAMS.
CompendiumLD
CompendiumLD is developed by Compendium Institute (http://compendium.open.ac.uk/institute/index.htm), its a software tool to builds learning design as mind mapping [6], to do this it provides a set of icons for creating maps to describe argumentation, and for communication issues. (see e.g. Shum & Okada, 2008). A user can drag and drop the icons onto CompendiumLDs user interface so creating nodes which may be linked and labelled. The three classes of nodes objects within CompendiumLD that are of interest with respect to learning design are: Node Summary - this holds the information about the node, including its label, image, and descriptive textual information added by a user . Link node - this holds information about a link from a source node to a destination node.
FIGURE 7. A sequence of activities in LAMS.
Page 52

View node - this is a subclass of NodeSummary i.e. it holds all the information that a NodeSummary does, but has additional attributes (e.g. positional information) which enables the CompendiumLD application to create lists views or map views (Bachler, 2004).
FIGURE 8. A sequence of activities in CompendiumLD.
CONCLUSIONS
The most of the teachers design a lesson based on knowledge transmission [12]. When preparing a lesson, they think in content (text, figures, tools, etc.). In this way, the Learning Design are oriented to the process, the same way that the learning objects are oriented to contents. The Learning Designs dont follow any learning style, neither learning theories. In other words, you can build learning designs to any group of student or you build personalized learning design. The teachers are using learning design for ever and ever, according to their experience or following some learning theory. To do that, IMS-LD offers a semantic notation to describe an educational scenario in a formal way The IMD LD specification is a formalization of the traditional learning design in the sense of use the theatrical script, ie., acts, stakeholder, enviroments, resources, etc. The authoring tools, follows in this sense the the learning design specification. IMS-LD specification come from 2003, several authoring tools, to implement the specification are developed, some of this tools are CoSMoS (Collaboration Script Modelling System) [11], ALFANET [1], CopperAuthor [8] and ReLoad. However the authoring tools under IMD-LD specification are still at an immature stage. Reload Learning Design Editor / Player is the most development, but are not sufficient friendly to teachers. The use of Learning Design by teachers, depends of how much easy to use are the authoring tools. In this sense CompendiumLD is easy to use, is based in mind maps, but do not give any information to teachers about the progress of the students. LAMS is the most mature of the authoring tools; there is a community of teachers and developers, every year organise a conference to share experiences and results, etc.;, in fact, is a platform to management learning sequences, users, lessons, classes, etc. The teachers can ask the students porfolio, at any instant of the lesson, to know the progress of each one or everyone, and finaly is very easy to use.
ACKNOWLEDGMENTS
Financial support for this work gratefully acknowledge from EuropeAid Cooperation Office Latin America, European Commission, trough project CRIS: 245-315 Alfa III 2nd call, entitled IGUAL: Innovation for Equality in Latin American University, and also thanks to Research and Development Office of the Universidad Austral de Chile. Additional thanks to the valuable comments of all members of the Research Group on Learning Technologies of the Universidad Austral de Chile (www.gita.cl)
REFERENCES
1. ALFANET home page. Retrieved from http://alfanet.ia.uned.es/
Page 53

2. Luis A. lvarez-Gonzlez, Sergio A. Trivios-Villanueva, Sandra Bucarey-Arriagada (2008), Teaching of Human 3. 4. 5.
Liver Anatomy with Learning Designs. 2008 European LAMS Conference (http://lams2008.lamsfoundation.org) a Cadiz, Espaa, 25 al 27 de junio del 2008. Luis A. lvarez-Gonzlez, Luis R. Ojeda-Gallegos, Cristian A. Cifuentes-Salazar (2008), Diseos de Aprendizaje en LAMS y la Enseanza de Mecnica Bsica. 2008 European LAMS Conference (http://lams2008.lamsfoundation.org) Cadiz, Espaa, 25 al 27 de junio del 2008. Erick Araya, Miguel Velsquez, Carolina Flores (2008), Puede LAMS ayudar a reducir el alto ndice de fracaso de los estudiantes de 1 ao de Ingeniera? 2008 European LAMS Conference (http://lams2008.lamsfoundation.org) Cadiz, Espaa, 25 al 27 de junio del 2008. Erick Araya, Miguel Velsquez, Carolina Flores (2008), Incorporacin de Expresiones Matemticas en Secuencias de Aprendizaje Usando Mtodos Alternativos a jsMath, 2008 European LAMS Conference (http://lams2008.lamsfoundation.org) Cadiz, Espaa, 25 al 27 de junio del 2008.
6. 6.Andrew Brasher, Grinne Conole, Simon Cross, Martin Weller, Paul Clark, Juliette White (2008). CompendiumLD a tool for effective, efficient and creative learning design. Institute of Educational Technology, The Open University. Available in http://lams2008.lamsfoundation.org/proceedings.htm.
7. Britain, S. (2004). A review of Learning Design: Concept, Specifications and Tools. A report for the JISC Elearning Pedagogy Programme.
8. CopperCore home page, Retrieved from http://coppercore.org/
9. IMS Learning Design Specification http://www.imsglobal.org/learningdesign/ 10. Koper, Rob, Learning Design: A Handbook on Modelling and Delivering Networked Education and Training. Chapter
1.An Introduction to Learning Design. Colin Tattersal (Eds.) Spinger.
11. Yongwu Miao (2006). CoSMoS: Facilitating Learning Designers to Author Units of Learning Using IMSLD. Institute of Computer Science and Interactive Systems, University of Duisburg-Essen, Germany. Availaible in http://www.collide.info/Members/admin/publications/ICCE05.260.pdf. 12. Ton Mooij (2007) Education and ICT-based self-regulation in learning: Theory, design and implementation. Heerlen: Open Universiteit Nederland 13. Baltasar Fernndez Manjn (2006) Editor de Learning Design. Facultad de Informtica. Universidad Complutence de Madrid. Reload. Available in http://eprints.ucm.es/9010/1/TC2006-28.pdf. 14. 14.Van Rosmalen, P. (2008). Supporting the tutor in the design and support of adaptive e-learning, Heerlen: Open Universiteit Nederland. ISBN 978-90-79447-01-5
15. Vigotsky, L. (1978). Mind in Society: The Development of Higher sychological Processes. Ed. M. Cole, et al. Cambridge, MA: Harvard University Press.
Page 54

IGUAL Project - State of The Art

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

IGUAL Project - State of The Art

Enviado por

Direitos autorais:

Formatos disponíveis

Analysis of State-of-the-Art Solutions for Personalised Learning Support

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

From the Editors

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

Learning/teaching a Computer Programming Course

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

LEARNING SOLUTIONS IN COMPUTER PROGRAMMING

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

FIGURE 1. Cupi2 Community structure (Villalobos, 2009a).

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

FIGURE 2. Cupi2 pedagogical model (Villalobos, 2009b).

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

FIGURE 3. Cognitive aspects of programming and their relation (Villalobos, 2009b).

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support ACKNOWLEDGMENTS

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

Localization of Learning Objects

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

LOCALIZATION AND INTERNATIONALIZATION

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

FIGURE 1. Tasks of Localization (Adapted from (Dagiene & Zilinskiene, 2009)).

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

Culturalization of Learning Objects

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

OPENNESS AND ADAPTATION

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

THE INITIAL PROPOSAL

FIGURE 2. Framework for the Adaptation of Learning Objects.

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

Sharing and Recommending Lea rning Materials

ARIADNE Foundation. http://www.ariadne-eu.org Connexions. http://www.cnx.org Maricopa Learning Exchange. http://www.mcli.dist.maricopa.edu/mlx

The most popular examples of Learning Object Referatories are:

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support

PUBLICATION OF LEARNING OBJECTS

IGUAL Innovation for Equality in Latin American University

Analysis of State-of-the-Art Solutions for Personalised Learning 2011 Support