ICLS2012 Proceedings Vol 1 2012

10th International Conference of the Learning Sciences
The Future of Learning
Proceedings
Volume 1 - Full papers
July 2-6, 2012 July 2 nd -Australia 6 th, 2012 Sydney,
Sydney, Australia
www.isls.org/icls2012
SPONSORED BY: HOSTED BY:
Centre for Research on Computer Supported Learning & Cognition (CoCo)
ICLS 2012 Conference Proceedings Volume 1 Full Papers

10th International Conference of the Learning Sciences The University of Sydney, Sydney, NSW, AUSTRALIA July 2nd 6th, 2012, Sydney
THE FUTURE OF LEARNING
Editors: Jan van Aalst, Kate Thompson, Michael J. Jacobson, and Peter Reimann
Title:
Cite as:
Editors:
ISBN: 978-0-578-10641-0
2012, INTERNATIONAL SOCIETY OF THE LEARNING SCIENCES [ISLS] Copyright 2012 International Society of the Learning Sciences, Inc. Rights reserved
van Aalst, J., Thompson, K., Jacobson, M. J., & Reimann, P. (Eds.) (2012). The Future of Learning: Proceedings of the 10th International Conference of the Learning Sciences (ICLS 2012) Volume 1, Full papers. International Society of the Learning Sciences: Sydney, NSW, AUSTRALIA
The Future of Learning: Proceedings of the 10th International Conference of the Learning Sciences (ICLS 2012) Volume 1, Full Papers Jan van Aalst, Kate Thompson, Michael J. Jacobson, and Peter Reimann
Published by: International Society of the Learning Sciences (ISLS) Proceedings printed by: LuLu http://www.lulu.com Proceedings distributed by: LuLu and Amazon Book Cover Design: Dorian Peters Book Cover Photo: Christine Bree
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the International Society of the Learning Sciences. The International Society of the Learning Sciences is not responsible for the use which might be made of the information contained in this book. http://www.isls.org
OUR SPONSORS
ICLS2012 would like to thank our sponsors:

NSW Trade & Investment The Centre for Research on Computer Supported Learning and Cognition (CoCo) Smart Services Collaborative Research Centre
The Early Career workshop and Doctoral Consortium would particularly like to thank the following for their support:
National Science Foundation The International Society of the Learning Sciences Asia-Pacific Society of Computers in Education
We would also like to thank the following for their generosity in donating their time, skills and information to the conference.
Inquirium
ICLS 2012
Volume 1: Full Papers
ORGANISERS AND COMMITTEES CONFERENCE CHAIRS

Michael J Jacobson | The University of Wouter Van Joolingen | University of
Twente, the Netherlands

Anindito Aditomo | The University of
Sydney, Australia Peter Reimann | The University of Sydney, Australia CONFERENCE ADVISORY BOARD
Shaaron
Sydney, Australia EARLY CAREER WORKSHOP

Tom Moher | University of Illinois, United
States
Anne Newstead | The University of Sydney,
Ainsworth | University of Nottingham, UK Michael Baker | Telecom ParisTech, Paris, France Katerine Bielaczyc | Nanyang Technological University, Singapore Paul Chandler | Wollongong University, Australia Susan Goldman | University of Illinois, United States Kai Hakkarainnen | University of Helsinki, Finland Yasmin Kafai | University of Pennsylvania, United States Paul Kirschner | Open University of the Netherlands, the Netherlands Marcia Linn | University of California, United States Ric Lowe | Curtin University, Australia Naomi Miyake | University of Tokyo, Japan Stella Vosniadou | University of Athens (Greece) and University of Adelaide (Australia) Uri Wilensky | Northwestern University, United States James Pellegrino | University of Illinois, United States
Australia
Tak-Wai Chan | National Central Taiwan
University, Taiwan PRE-CONFERENCE WORKSHOPS AND SPECIAL SESSIONS

Lina Markauskaite | The University of
Sydney, Australia
Dan Suthers | University of Hawaii, United
States
Carol Chan | The University of Hong Kong,
Hong Kong
Nikol Rummel | Ruhr-Universitat Bochum,
Germany PAPER REVIEW COORDINATION

Peter Reimann | The University of Sydney,
Australia
Peter Freebody | The University of Sydney,
Australia Kyza | Cyprus University of Technology, Cyprus Ton de Jong | University of Twente, the Netherlands Nick Kelly | The University of Sydney, Australia
Eleni
PROCEEDINGS
Jan van Aalst | The University of Hong
Kong, Hong Kong

Brian J. Reiser | Northwestern University,
United States
Cindy Hmelo-Silver | Rutgers University,
United States
Kate Thompson | The University of Sydney,
LOGISTICS AND ADMINISTRATION SUPPORT Sadhbh Warren | The University of Sydney, Australia Sadhana Puntambekar | The International Society of the Learning Sciences Dana Gnesdilow | The International Society of the Learning Sciences
Australia
Stella Wen Tian | The University of Hong
Kong, Hong Kong DOCTORAL CONSORTIUM

Susan Yoon | University of Pennsylvania,
United States
ISLS
Page vi
ICLS 2012
PAPER REVIEWERS Abrahamson, Dor, University of California, United States Aditomo, Anindito,, The University of Sydney, Australia Ahn, June, University of Maryland, United States Ainsworth, Shaaron, University of Nottingham, United Kingdom Alac, Morana, University of California, United States Alonzo, Alicia, Michigan State University, United States Alterman, Rick, Brandeis University, United States Andriessen, Jerry, Utrecht University, The Netherlands Ares, Nancy, University of Rochester, United States Arnseth, Hans Christian, University of Oslo, Norway Arthur-Kelly, Michael, University of Newcastle, Australia Asterhan, Christa, Hebrew University of Jerusalem, Israel Avramides, Katerina, London Knowledge Lab, United Kingdom Bagley, Elizabeth, University of Illinois, United States Bairral, Marcelo, Federal Rural University of Rio de Janeiro, Brazil Baker, Michael, CNRS - Telecom ParisTech, France Baker, Ryan, Worcester Polytechnic Institute, United Kingdom Barnes, Jacqueline, Indiana University, United States Barth-Cohen, Lauren, University of California, United States Belland, Brian, Utah State University, United States Berland, Leema, University of Texas, United States Bers, Marina, Tufts University, United States Bielaczyc, Katerine, Singapore National Institute of Education, Singapore Booker, Angela, University of California, United States Brett, Clare, University of Toronto, Canada Brian, Reiser, Northwestern University, United States Buckingham, Brandy, Northwestern University, United States Burke, Quinn, University of Pennsylvania, United States Calabrese Barton, Angela, Michigan State University, United States Carmela, Aprea, Swiss Federal Institute for Vocational Education and Training, Switzerland Carpendale, Jeremy, Simon Fraser University, Canada Castro-Alonso, Cris, University of New South Wales, Australia Cesar, Collazos, Universidad del Cauca, Colombia Chan, Carol, The University of Hong Kong, Hong Kong Chang, Yi-Hsing, Southern Taiwan University, Taiwan Charoenying, Timothy, University of California, United States Chase, Catherine, Carnegie Mellon University, United States Chen, Ying-Chih, University of Minnesota, United States Childs, Joshua, University of Pittsburgh, United States Ching, Cynthia Carter, University of California, United States Chinn, Clark, Rutgers University, United States Christophe, Reffay, ENS Cachan, France Chye, Stefanie, Singapore National Institute of Education, Singapore Clegg, Tamara, University of Maryland, United States Cooper, Benny, University of California, United States Correia, Ana-Paula, Iowa State University, United States Crain, Rhiannon, Cornell University, United States Cress, Ulrike, Knowledge Media Research Center, Germany D'Angelo, Cynthia, University of Wisconsin, United States Damsa, Crina, Utrecht University, The Netherlands Davis, Pryce, Northwestern University, United States De Jong, Ton, University of Twente, The Netherlands de Vries, Erica, University of Grenoble, France Dillenbourg, Pierre, Swiss Federal Institute of Technology, Lausanne, Switzerland
ISLS
Page vii
ICLS 2012
Drago, Kathryn, University of Michigan, United States Dugan, Therese, University of Washington, United States Duncan, Ravit, Rutgers University, United States Durga, Shree, University of Wisconsin, United States Dyke, Gregory, Carnegie Mellon University, United States Earnest, Darrell, University of California, United States Ehret, Christian, Vanderbilt University, United States Eisenberg, Michael, University of Colorado, United States Elen, Jan, K.U.Leuven, Belgium Engelmann, Tanja, Knowledge Media Research Center, Germany Engle, Randi A., University of California, United States Enyedy, Noel, University of California, United States Erkens, Gijsbert, Utrecht University, The Netherlands Erlandson, Benjamin, Arizona State University, United States Ertl, Bernhard, Universitt der Bundeswehr, Germany Fatos, Xhafa, Universitat Politecnica de Catalunya, Spain Femke, Kirschner, Utrecht University, The Netherlands Fields, Deborah, University of California, United States Fischer, Frank, University of Munich, Germany Fishman, Barry, University of Michigan, United States Forest, Dominique, Universitat de Brest, France Freebody, Peter, The University of Sydney, Australia Frode, Guriybe, University of Bergen, Germany Furtak, Erin Marie, University of Colorado, United States Gegenfurtner, Andreas, Technical University of Munich, Germany Gerofsky, Susan, University of British Columbia, Canada Gillen, Julia, Lancaster University, United Kingdom Ginns, Paul, The University of Sydney, Australia Goldman, Susan R, University of Illinois, United States
Goldman, Ricki, New York University, United States Gomez, Kimberley, University of Pittsburgh, United States Gomez, Kimberley, University of California, United States Gonzalez, Carlos, Pontificia Universidad Catolica, Chile Gresalfi, Melissa, Indiana University, United States Groza, Gabriela, University of Illinois, United States Gruson, Brigitte, IUFM de Bretagne, France Gutierrez, Jose, University of California, United States Hajime, Shirouzu, Chukyo University, Japan Hakkarainen, Kai, University of Turku, Finland Halverson, Erica, University Of Wisconsin, United States Halverson, Richard, University of Wisconsin, United States Hawi, Nazir, Notre Dame University, Lebanon Herman, Phillip, University of Pittsburgh, United States Herold, David, Hong Kong Polytechnic University, Hong Kong Herrmann, Thomas, University of Bochum, Germany Hershkovitz, Arnon, Worcester Polytechnic Institute, United Kingdom Hesse, Friedrich, Knowledge Media Research Center, Germany Hickey, Daniel, Indiana University, United States Hmelo-Silver, Cindy, Rutgers University, United States Hoadley, Christopher, Penn State University, United States Holbert, Nathan, Northwestern University, United States Hong, Huang-Yao, National Chengchi University, Taiwan Hora, Matthew, University of Wisconsin, United States Horn, Ilana, Vanderbilt University, United States Howard, Sarah, University of Wollongong, Australia Hulshof, Casper, University of Utrecht, The Netherlands Inge, Molenaar, University of Amsterdam, The Netherlands Ingerman, ke, University of Gothenburg, Sweden Jackson, Kara, McGill University, Canada
Page viii
ISLS
ICLS 2012
Jacobson, Michael, The University of Sydney, Australia Jamaludin, Azilawati, National Institute of Education, Singapore Jay, Tim, University of Bristol, United Kingdom Jochen, Rick, Saarland University, Germany Joiner, Richard, University of Bath, United Kingdom Jurow, A. Susan, University of Colorado, United States Kafai, Yasmin, University of Pennsylvania, United States Kanselaar, Gellof, Utrecht University, The Netherlands Kapur, Manu, National Institute of Education, Singapore Keifert, Danielle, Northwestern University, United States Kimmerle, Joachim, University of Tuebingen, Germany King Chen, Jennifer, University of California, United States Kirschner, Paul, Open University, The Netherlands Kollffel, Bas, University of Twente, The Netherlands Kollar, Ingo, University of Munich, Germany Koschmann, Timothy, Southern Illinois University, United States Kraemer, Nicole, University Duisburg-Essen, Germany Krauskopf, Karsten, Knowledge Media Research Center, Germany Kwon, Samuel, Learning Point Associates, United States Kyza, Eleni, Cyprus University of Technology, Cyprus Lai, Jiang, Katholieke University of Leuven, Belgium Lakkala, Minna, University of Helsinki, Finland Langer-Osuna, Jennifer, University of Miami, United States Law, Nancy, University of Hong Kong, Hong Kong Lazonder, Ard, University of Twente, The Netherlands Leary, Heather, University of Colorado, United States Lee, Kyungmee, University of Toronto, Canada Lee, Chien Sing, National Central University, China Lee, Tiffany, University of Washington, United States Lee, Victor, Utah State University, United States
Lehrer, Rich, Vanderbilt University, United States Lei, Jing, Syracuse University, United States Liao, Chang-Yen, National Central University, China Liesbeth, Kester, Open University of the Netherlands, The Netherlands Limon, Margarita, University Autonoma of Madrid, Spain Lin, Hsien-Ta, National Chengchi University, Taiwan Lindgren, Robb, University of Central Florida, United States Lindwall, Oskar, University of Gotheburg, Sweden Liu, Shiyu, University of Minnesota, United States Liu, Ru-De, Beijing Normal University, China Lone, Dirckinck-Holmfeld, Aalborg University, Denmark Lonn, Steven, University of Michigan, United States Looi, Chee Kit, National Institute of Education, Singapore Louca, Loucas T., European University, Cyprus Lowe, Ric, Curtin University, Australia Lozano, Maritza, University of California, United States Luckin, Rose, London Knowledge Lab, United Kingdom Ludvigsen, Sten, University of Oslo, Norway Lund, Kristine, CNRS, France Magnifico, Alecia Marie, University of Illinois United States Magno, Carlo, De La Salle University, Philippines Manches, Andrew, University of Nottingham, United Kingdom Martell, Sandra, National Science Foundation, United States Martin, Crystle, University of Wisconsin, United States McElhaney, Kevin, University of California, United States McGee, Steven, The Learning Partnership, Canada McKenney, Susan, Open University of the Netherlands, The Netherlands Medina, Richard, University of Hawaii, United States Mercier, Emma, Durham University, United Kingdom Miyake, Naomi, the University of Tokyo, Japan Moher, Tom, University of Illinois, United States
Page ix
ISLS
ICLS 2012
Moore, Joyce, University of Iowa, United States Mor, Yishay, Open University, United Kingdom Morch, Anders, University of Oslo, Norway Moskaliuk, Johannes, Knowledge Media Research Center, Germany Munoz-Baell, Irma M, University of Alicante, Spain Myint Swe, Khine, Bahrain Teachers College, Bahrain Nckles, Matthias, University of Freiburg, Germany Naidoo, Jayaluxmi, University of KwaZuluNatal, South Africa Nicolau, Juan L., University of Alicante, Spain Noroozi, Omid, Wageningen University, The Netherlands O'Malley, Claire, University of Nottingham, United Kingdom O'Neill, Kevin, Simon Fraser University, Canada Oeberst, Aileen, Knowledge Media Research Center, Germany Olimpo, Jeffrey, University of Maryland, United States Opfermann, Maria, University of DuisburgEssen, Germany Orrill, Chandra, University of Massachusetts, United States Oshima, Jun, Shizuoka University, Japan Oztok, Murat, University of Toronto, Canada Paavola, Sami, University of Helsinki, Finland Papademetri-Kachrimani, Chrystalla, European Univesrity Cyprus, Cyprus Parisio, Martin, The University of Sydney, Australia Pea, Roy, Stanford University, United States Pedaste, Margus, University of Tartu, Estonia Penuel, William, University of Colorado, United States Pluta, William, Rutgers, The State University of New Jersey, United States Polman, Joe, University of Missouri, United States Potgieter, Marietjie, University of Pretoria, South Africa Prins, Frans, Utrecht University, The Netherlands Radinsky, Josh, University of Illinois, United States Raes, Annelies, Ghent University, Belgium Ranney, Michael, University of California, United States Reber, Rolf, University of Bergen, Norway Reimann, Peter, The University of Sydney, Australia
Renninger, K. Ann, Swarthmore College, United States Rivet, Ann, Columbia University, United States Rogotneva, Elena, Tomsk Polytechnic University, Russia Rose, Carolyn, Carnegie Mellon University, United States Rummel, Nikol, University of Freiburg, Germany Rusman, Ellen, Open University of the Netherlands, The Netherlands Ryan, Stephanie, University of Illinois, United States Saleh, Asmalina, Indiana University, United States Sandoval, William, University of California, United States Sayre, Eleanor, Kansas State University, United States Schnotz, Wolfgang, University of KoblenzLandau, Germany Schoerning, Emily, University of Iowa, United States Schwarz, Baruch, Hebrew University, Israel Schwendimann, Beat, The University of Sydney, Australia Scott Curwood, Jen, University of Wisconsin, United States Seifert, Colleen, Univ. of Michigan, United States Sensevy, Grard, University of Western Brittany, France Shapiro, Amy, University of Massachusetts, United States Shelton, Brett, Utah State University, United States Singer, Susan, Carleton College, United States Slof, Bert, Utrecht University, The Netherlands Smith, Debbie, Clemson University, United States Solomou, Maria, Indiana University, United States Spada, Hans, University of Freiburg, Germany Specht, Marcus, Open University, The Netherlands Stager, Sarah, Pennsylvania State University, United States Stevens, Reed, Northwestern University, United States Stieff, Mike, University of Illinois, United States Strijbos, Jan-Willem, Ludwig-MaximiliansUniversity Mnchen, Germany Sudol-DeLyser, Leigh Ann, Carnegie Mellon University, United States
ISLS
Page x
ICLS 2012
Suthers, Daniel, University of Hawaii, United States Tan, Seng-Chee, National Institute of Education, Singapore Taylor, Martin, University of Texas, United States Tchounikine, Pierre, University of Grenoble, France Tee, Meng Yew, University of Malaya, Malaysia Thayer, Alexander, University of Washington, United States Thompson, Kate, The University of Sydney, Australia Tok, kran, Pamukkale University, Turkey Trausan-Matu, Stefan, University "Politehnica" of Bucharest, Romania Trninic, Dragan, University of California, United States Tung, I-Pei (Vicky), McGill University, Canada Turns, Jennifer, University of Washington, United States Tzialli, Dora, European University Cyprus, Cyprus van Aalst, Jan, The University of Hong Kong, Hong Kong van Amelsvoort, Marije, Tilburg University, The Netherlands van Es, Beth, University of California, United States van Oers, Bert, University Amsterdam, The Netherlands van Oostendorp, Herre, Utrecht University, The Netherlands Vandewaetere, Mieke, K.U. Leuven, Belgium Vanover, Charles, University of South Florida, United States Vatrapu, Ravi, Copenhagen Business School, Denmark Venkataswamy, Arjun, Community Links High School, United States Virnes, Marjo, University of Eastern Finland, Finland Volker, Wulf, University of Siegen, Germany Walker, Richard, The University of Sydney, Australia Walkington, Candace, University of Wisconsin, United States
Wang, Tsungjuang, National Taipei University of Technology, Taiwan Wardrip, Peter, Northwestern University, United States Warren, Scott, Indiana University, United States Wecker, Christof, University of Munich, Germany Wee, Juan Dee, National Institute of Education, Singapore Weinberger, Armin, Saarland University, Germany Weinstock, Michael, Ben-Gurion University of the Negev, Israel Wessel, Daniel, Knowledge Media Research Center, Germany Wessner, Martin, Fraunhofer IESE, Germany White, Tobin, University of California, United States Wichmann, Astrid, Ruhr University Bochum, Germany Williams, Robert, Lawrence University, United States Wodzicki, Katrin, Knowledge Media Research Center, Germany Wong, Lung-Hsiang, National Institute of Education, Singapore Wouters, Pieter, Utrecht University, The Netherlands Wu, Hsin-Kai, National Taiwan Normal University, Taiwan Yael, Kali, University of Haifa, Israel Yeo, Ai Choo Jennifer, National Institute of Education, Singapore Yilmaz, Kaya, Marmara University, Turkey Young, Michael, University of Connecticut, United States Zagal, Jose, DePaul University, United States Zahn, Carmen, Knowledge Media Research Center, Germany Zannetou, Marilyn, EUC European University of Cyprus, Cyprus Zhang, Jianwei, University at Albany, United States Zingaro, Daniel, University of Toronto, Canada Zywica, Jolene, University of Pittsburgh, United States
ISLS
Page xi
ICLS 2012
ABOUT ISLS
ISLS is a professional society dedicated to the interdisciplinary empirical investigation of learning as it exists in real-world settings and how learning may be facilitated both with and without technology. ISLS sponsors two professional conferences, held in alternate years. The International Conference of the Learning Sciences (ICLS), first held in 1991 and held biannually since 1996, covers the entire field of the learning sciences. http://www.isls.org/
ABOUT "COCO" AT THE UNIVERSITY OF SYDNEY
ICLS 2012 is hosted by the Centre for Research on Computer Supported Learning and Cognition (CoCo) at the University of Sydney. CoCo's mission is to contribute to theory and research in the field of the learning sciences in order to discover how innovative learning technologies and pedagogical approaches can enhance formal and informal learning. CoCo is a University of Sydney Research Centre operating within the Faculty of Education and Social Work. http://sydney.edu.au/edsw/coco
ISLS
Page xii
ICLS 2012
PREFACE
Michael J. Jacobson, Peter Reimann, and Jan van Aalst The international and interdisciplinary field of the learning sciences brings together researchers from the fields of cognitive science, educational research, psychology, computer science, artificial intelligence, anthropology, neuroscience, and other fields to study learning in a wide variety of formal and informal contexts. This field emerged in the late 1980s and early 1990s, with the first International Conference of the Learning Sciences (ICLS) being held in 1991 at Northwestern University in Evanston, Illinois, USA. Subsequent meetings of ICLS were held again in Evanston, USA (1996), and then in Atlanta, USA (1998), Ann Arbor, USA (2000), Seattle, USA (2004), Santa Monica, USA (2004), and Bloomington, USA (2006). The first ICLS to be held outside of North America was in Utrecht, the Netherlands (2008), and then back to the USA in Chicago (2010). The ICLS 2012 in Sydney is the first hosting of the conference in the Asia-Pacific region. Papers for this conference were submitted in November 2011, and then went through a process of peer review. Full papers and symposia submissions received three anonymous reviews with a member of the Program Committee summarizing the reviews and making a recommendation. Short paper poster submissions went through the same process, except with two anonymous reviews. See Table 1 for a summary of the conference paper statistics. Table 1. Paper submissions, acceptance, and rejection rates.
Full papers 264 Accepted as Full Papers: 65 (25%) Accepted as Short Paper: 54 (20%) Accepted as Poster: 42 (16%) Rejected: 103 (39%) Short Papers 79 Accepted: 15 (19%) Accepted as poster: 11 (14%) Rejected: 53 (67%) Posters 76 Accepted: 37 (49%) Rejected: 39 (51%) Symposia 27 Accepted: 18 (67%) Rejected: 9 (33%) Total Submissions: 446
The final papers included in the ICLS 2012 Proceedings are for 18 symposia, 60 full papers, 61 short papers, and 62 poster papers, as well as abstracts for the four keynote talks and three special sessions, including the invited Presidential Session on The Future of Learning, and nine workshops. The themes reflected in the papers and presentations at ICLS 2012 cover a wide range of issues and research areas. Some papers deal with long standing theoretical issues, such as conceptual change and knowledge transfer, whereas other papers report on new learning research in conventional subject areas such as science, mathematics, and literacy. Research is also reported on newer knowledge areas such as complex systems, as well as more recent
ISLS Page xiii
ICLS 2012
perspectives on pedagogy and learning such as knowledge building, inquiry, and productive failure. A number of learning sciences research projects internationally are also exploring ways in which innovative environments for learning may be designed with new technologies such virtual and game environments, modeling and visualization systems, robotics, collaboration technologies, mobile and hand held devices, and educational data mining and learning analytics. Emerging areas in learning sciences research that perhaps may lead to newer perspectives and directions in our understanding of learning include creatively, identity, and embodied cognition. The field continues to engage the broad issues of contributing to and impacting policy and practice more generally. Overall, the research presented in the proceedings of ICSL 2012 contributes numerous perspectives on the conference theme of the future of learning, both as trajectories of research that have been maturing over a number of years and bold new perspectives that promise to shape new trajectories for the future. Making this conference possible, we thank the hard work and the countless hours put in by our international Conference Advisory Board and the various conference subcommittees. We would like to thank all those who assisted with the review process. The Conference is also fortunate to have a number of financial sponsors whose support contributes to conference events: NSW Trade & Investment, Smart Services Collaborative Research Centre, United States National Science Foundation, The International Society of the Learning Sciences, Asia-Pacific Society of Computers in Education, Inquirium, LLC, and The Centre for Research on Computer Supported Learning and Cognition. Any conference is but a snapshot in the dynamic process of articulating and vetting scientifically principled ideas and approaches. That the International Conference of the Learning Sciences has now had its 10th meeting and has entered its third decade are exciting milestones that this proceedings helps to document. We close by reflecting that just as a great movie is about the journey of discovery and development that the characters experience, so might our research field be a journey of discovery and development to more deeply understand how people learn now and as the future unfolds, knowing learning itself as a core essence of life and our humanity.
ISLS
Page xiv
ICLS2012
CONTENTS - VOLUME 1 Using Controversies for Knowledge Construction: Thinking and Writing about Alternative Medicine
Joachim Kimmerle, Johannes Moskaliuk, Martina Bientzle, Ansgar Thiel, Ulrike Cress
Understanding How Learners Grapple with Wicked Problems in Environmental Science

Brian Slattery, Chandan Dasgupta, Tia Shelley, Leilah Lyons, Emily Minor, Moira Zellner
9 17
Semiosis Process in Joint Didactic Action

Grard Sensevy , Dominique Forest
Agent-Based Computer Models for Learning About Climate Change and Process Analysis Techniques
Nick Kelly, Michael Jacobson, Lina Markauskaite, Vilaythong Southavilay
25
Learning How to Create: Toward A Learning Sciences of Art and Design

Keith Sawyer
33
A Microgenetic Analysis of How Expansive Framing Led to Transfer with One Struggling Student
Diane Lam, Xenia Meyer, Randi A. Engle, Lloyd Goldwasser, Kathleen Zheng, Erica Naves, Danny Tan, Richard Hsu, Hernan Rosas, Sarah Perez
40
How Metacognitive Awareness Caused A Domino Effect in Learning

Kristin Schmidt, Andreas Lachner, Bjrn Stucke, Sabine Rey, Cornelius Frmmel, Matthias Nckles
48
Initial Validation of Listening Behavior Typologies for Online Discussions Using Microanalytic Case Studies
Alyssa Friend Wise, Ying-Ting Hsiao, Farshid Marbouti, Jennifer Speer, Nishan Perera
56
How to Schedule Multiple Graphical Representations? A Classroom Experiment With an Intelligent Tutoring System for Fractions
Martina Rau, Nikol Rummel, Vincent Aleven, Laura Pacilio, Zelha Tunc-Pekkan
64
Investigating the Relative Difficulty of Various Complex Systems Ideas in Biology

Sao-Ee Goh, Susan Yoon, Joyce Wang, Zhitong Yang, Eric Klopfer
72
Using Adaptive Learning Technologies to Personalize Instruction: The Impact of InterestBased Scenarios on Performance in Algebra
Candace Walkington, Milan Sherman
80
The Impact of Students Exploration Strategies in Discovery-Based Instructional Software

Barney Dalgarno, Gregor Kennedy, Sue Bennett
88 96 103
Design Research in Early Literacy within the Zone of Proximal Implementation

Susan McKenney, Paul Kirschner, Joke Voogt
Kitchen Chemistry: Supporting Learners Decisions in Science

Jason Yip, Tamara Clegg, Elizabeth Bonsignore, Helene Gelderblom, Becky Lewittes, Mona Leigh Guha, Allison Druin
Students Intuitive Understanding of Promisingness and Promisingness Judgments to Facilitate Knowledge Advancement
Bodong Chen, Marlene Scardamalia, Monica Resendes, Maria Chuy, Carl Bereiter
111
Designing Video Games that Encourage Players to Integrate Formal Representations with Informal Play
Nathan Holbert, Uri Wilensky
119
Dilemmas of Promoting Expansive Educational Transformation through Information and Communication Technologies (ICTs) in Botswana
127
ISLS
Page xv
ICLS2012
Tshepo Batane, Ritva Engestrm, Kai Hakkarainen, Denise Newnham, Paul Nleya, Jaakko Virkkunen
Visible Spending: Information Visualization of Spending Behavior and the Thought-Act Gap
Emily S. Lin, Holly A. Taylor
135
Beyond the Screen: Game-Based Learning As A Nexus of Identification

Ben DeVane
143
Working With Teenagers to Design Technology That Supports Learning About Energy in Informal Contexts
Katerina Avramides, Brock Craft, Rosemary Luckin, Janet Read
151
Unpacking the Use of Talk and Writing in Argument-based Inquiry: Instruction and Cognition
Ying-Chih Chen, Soonhye Park, Brian Hand
159
A Data-driven Path Model of Student Attributes, Affect, and Engagement in a Computerbased Science Inquiry Microworld
Arnon Hershkovitz, Ryan Baker, Janice Gobert, Adam Nakama
167
Exploring Connectedness: Applying ENA to Teacher Knowledge

Chandra Hawley Orrill, David Williamson Shaffer
175 180
Investigating the Effects of Varying Labels as Scaffolds for Visitor Learning

Joyce Wang, Susan Yoon, Karen Elinich, Jackie Van Schooneveld
Making Technology Visible: Connecting the Learning of Crafts, Circuitry and Coding in E188 textiles by Youth Designers
yasmin kafai, Deborah Fields, Kristin Searle
Functional Aesthetics for Learning: Creative Tensions in Youth e-Textile Designs

Deborah A. Fields, Yasmin B. Kafai, Kristin Searle
196 204 212
Learning Progressions, Learning Trajectories, and Equity

Cesar Delgado, Karisma Morton
A Teacher's Journey in Knowledge Building Pedagogy

Nancy Law, Johnny Yuen, Hidy Tse
Is Computer Support More Significant than Collaboration in Promoting Self-Efficacy and Transfer?
Andreas Gegenfurtner, Marja Vauras, Koen Veermans
220
Teacher Paradigm Shifts for 21st Practice Skills: The Role of Scaffolded Reflection Within 227 A Peer Community
Cheryl-Ann Madeira, Jim Slotta
The Impacts of Flexible Grouping in a Mobile-Assisted Game-based Chinese Character Learning Approach
Lung-Hsiang Wong, Ching-Kun Hsu, Jizhen Sun, Ivica Boticki
235
Exploratory Study on the Physical Tool-based Conceptions of Learning of Young Students 243 in a Technology-Rich Primary School
Lung-Hsiang Wong, Mingfong Jan, Yancy Toh, Ching-Sing Chai
Alternate Reality Games: Platforms for Collaborative Learning

Elizabeth Bonsignore, Derek Hansen, Kari Kraus, June Ahn, Amanda Visconti, Ann Fraistat, Allison Druin
251
Learning Innovation Diffusion as Complex Adaptive Systems through Model Building, Simulation, Game Play and Reflections
Junsong Huang, Manu Kapur
259
ISLS
Page xvi
ICLS2012
Peer Collaboration and Mediation in Elementary Students Lamp Designing Process

Kaiju Kangas, Pirita Seitamaa-Hakkarainen
267 275 283
Dynabook: Supporting Teacher Learning about Mathematical Thinking

Jeremy Roschelle, Charles Patton, Elizabeth Murray
Embodied Artifacts and Conceptual Performances

Dragan Trninic, Dor Abrahamson
The Impact of Structural Characteristics of Concept Maps on Automatic Quality Measurement

H. Ulrich Hoppe, Jan Engler, Stefan Weinbrenner
291
Toward a Cognitive Framework of Interdisciplinary Understanding

Shannon Sung, Ji Shen, Dongmei Zhang
299 307
Using Innovation with Contrasting Cases to Scaffold Collaborative Learning and Transfer
Seungyon Ha, David A. Sears
Young Children's Everyday Inquiry: A Field Study of a Young Girl's Play Across Contexts 315
Danielle Keifert
Using the Idea Manager to Promote Coherent Understanding of Inquiry Investigations

Kevin McElhaney, Camillia Matuk, David Miller, Marcia Linn
323
Using Heuristic Worked Examples and Collaboration Scripts to Help Learners Acquire Mathematical Argumentation Skills
Ingo Kollar, Stefan Ufer, Elisabeth Lorenz, Freydis Vogel, Kristina Reiss, Frank Fischer
331
Technology Supports in CSCL

Heisawn Jeong, Cindy Hmelo-Silver
339 347 355
Situating epistemological development

William Sandoval
Improving Revision in Wiki-based Writing: Coordination Pays off

Astrid Wichmann, Marina Becker, Nikol Rummel
Challenging Assumptions: using sliding window visualizations to reveal time-based irregularities in CSCL processes
Gregory Dyke, Rohit Kumar, Hua Ai, Carolyn Rose
363
Distributing Practice: Challenges and Opportunities for Inquiry Learning

Vanessa Svihla, Marcia Linn
371 379
Technology for Learning: Moving from the Cognitive to the Anthropological Stance
Michael Eisenberg
Re-presenting Complex Scientific Phenomena Using Agent-Based Modeling in Engineering 387 Education
Paulo Blikstein
Metadiscourse to Foster Student Collective Responsibility for Deepening Inquiry

Jianwei Zhang, Jiyeon Lee, Jane Wilde
395
Consequential Feedback as a Means of Supporting Student Engagement and Understanding 403

Melissa Gresalfi, Jacqueline Barnes
Multiple Trajectories for Understanding Ecosystems

Catherine Eberbach, Cindy Hmelo-Silver, Rebecca Jordan, Suparna Sinha, Ashok Goel
411
Supporting Learners to Conceptualize the Vast Range of Imperceptible Smallness with Temporal-aural-visual Representations
Minyoung Song, Chris Quintana
419
ISLS
Page xvii
ICLS2012
Inter-Identity Technologies for Learning

Robb Lindgren, Roy Pea
427 435 443 451 459
Predicting Idea Co-Construction in Speech Data using Insights from Sociolinguistics

Gahgene Gweon, Mahaveer Jain, John McDonough, Bhiksha Raj, Carolyn Rose
Automatically extract interpretable topics from online discussion

Yonghe Zhang, Nancy Law, Yanyan Li, Ronghuai Huang
Finding Voices and Emerging Agency in Classroom Learning

Tuck Leong Lee, Beaumie Kim, Mi Song Kim, Jason Wen Yau Lee
Processes of decision-making with adaptive combinations of wiki and chat tools

Kate Thompson, Nick Kelly
Examining the Adequacy of Students' Priors and Teacher's Role in Attention to Critical Features in Designing for Productive Failure
Pee Li Leslie Toh, Manu Kapur
467
Author Index
475
ISLS
Page xviii
ICLS2012
Using Controversies for Knowledge Construction: Thinking and Writing about Alternative Medicine
Joachim Kimmerle, University of Tuebingen, Schleichstr. 4, 72076 Tuebingen, j.kimmerle@iwm-kmrc.de Johannes Moskaliuk, University of Tuebingen, Schleichstr. 4, 72076 Tuebingen, j.moskaliuk@iwm-kmrc.de Martina Bientzle, KMRC, Schleichstr. 6, 72076 Tuebingen, m.bientzle@iwm-kmrc.de Ansgar Thiel, University of Tuebingen, Wilhelmstrae 124, 72072 Tuebingen, ansgar.thiel@uni-tuebingen.de Ulrike Cress, KMRC, Schleichstr. 6, 72076 Tuebingen, u.cress@iwm-kmrc.de Abstract: This paper presents a case study in which two participants with opposing points of view used a shared document for purposes of dealing with a controversy on the effectiveness of homeopathic medicine. The participants were working by turns on a common presentation that should outline their opinions on the topic. During this alternating process they had to think aloud to make individual processes observable. In addition, we analyzed the development of the shared text. We combined a content analysis and the think-aloud data to measure internal and external processes of knowledge construction, and we describe how these processes influenced each other. This procedure led to insights into the development of perception shared by the participants and into the resulting knowledge construction with respect to quantitative and qualitative processes. In particular, we analyzed the interplay between the participants thinking and writing processes as well as the co-evolution of their thinking and writing as cognitive and social processes. In our conclusion we discuss the implications of our findings for understanding processes of knowledge construction.
Introduction
Collaboratively developed shared digital artifacts may facilitate collaborative learning and the interplay between individual and collective knowledge processes (Kimmerle, Cress, & Held, 2010; Nussbaum, Winsor, Aqui, & Poliquin, 2007). In previous publications we have outlined a co-evolution model of cognitive and social systems, which is a theoretical approach that takes into account both the social processes facilitated by collaboratively developed digital artifacts and the cognitive processes of the individuals involved (Cress & Kimmerle, 2007, 2008). The aim of the current paper is to demonstrate in a detailed analysis how these social and cognitive processes may mutually influence each other in a process of collaboration and construction of knowledge. In an empirical study, we took into consideration both interdependent processes simultaneously: One persons trains of thought during collaboration as well as concurrent contributions this person made to a shared digital artifact. We related these processes to the corresponding reactions of another person with an opposing point of view, revealing this second persons trains of thought and contributions to the artifact, and so on. The basic idea is that it is a conflictual situation that initiates such an interdependency and interplay of social and cognitive processes. Piaget (1977) pointed out the significance of socio-cognitive conflicts for the development of knowledge. The utilization of cognitive conflicts for facilitating learning processes has been debated in the Learning Sciences for decades (Johnson & Johnson, 1987; Wadsworth, 1978; cf. also ODonnell & OKelly, 1994). We are convinced that conflicts also play an important role when it comes to collaboration and knowledge construction with the use of shared digital artifacts (for the role of confronting cognitions in computer-supported collaborative learning settings cf. Andriessen, Baker, & Suthers, 2003). The controversy between two or more individuals is supposed to trigger the construction of emergent knowledge, both in the cognitive system of a user and in the social systems represented by the shared digital artifact. Tools that particularly suggest themselves for this purpose are wikis, as they are established as teaching tools and learning environments (Forte & Bruckman, 2010; Parker & Chao, 2007; cf. also Glassman & Kang, 2011; Pifarr & Kleine Staarman, 2011) and greatly allow dealing with conflicts creatively and constructively (Moskaliuk & Kimmerle, 2009): Wikis provide a space in which ideas can be formulated, revised, or rejected. The people involved can contribute to the advancement of knowledge in the social system. Here, advancing knowledge is supposed to take place in the form of an improvement of ideas, not as the development of a true or perfect solution (cf. Bereiter & Scardamalia, 1996; Scardamalia, 2002). In the following section we will briefly describe the underlying theoretical considerations with regard to the relationship of cognitive and social systems, as they apply to the interplay between individual and collective knowledge. Subsequently, we will describe the method that was applied in the empirical study reported here and present its main results regarding the processes of knowledge construction. Concluding, we will discuss the implications of our analysis for further research on knowledge construction.
ISLS
ICLS2012
The Interplay between Cognitive and Social Systems

The collaborative construction of new knowledge may be described as an interplay between cognitive systems of individuals and a social system (Kimmerle, Moskaliuk, & Cress, 2009; Moskaliuk, Kimmerle, & Cress, 2008). We argue that research should focus equally on both systems by considering individual cognitive processes (such as perception, attention, learning, thinking, decision-making) as well as the collective development of knowledge. In this context we consider shared artifacts not just as a means to an end, but as the space where the collective development of knowledge takes place or, in other words, the space where collective knowledge and social structures manifest themselves. Our main assumption is that knowledge construction can only be understood if we take a very detailed look at the interplay between individuals and the collective. From a systems-theoretical point of view (Luhmann, 1995) the cognitive systems of individuals need to be distinguished from a social systemwhich is here represented by the interaction that takes place with the help of a shared digital artifact (cf. Kimmerle, Moskaliuk, & Cress, 2011). Cognitive systems and social systems, however, can affect each other in their development by providing irritating information to the other system (Moskaliuk, Kimmerle, & Cress, 2012). This is the process we want to take into account in our analysis. We regard irritations, along with Piaget (1977), as cognitive conflicts. We assume that cognitive systems will develop when individuals solve cognitive conflicts. Cognitive conflicts occur when individuals realize that their prior knowledge and the information from the environment which they have to deal with are contradictory to some extent. People may solve these conflicts by means of two sorts of equilibration processes: They may either assimilate information (basically attach new information to their prior knowledge, i.e., a quantitative process) or accommodate prior knowledge to novel information, which comes along with qualitative improvements. Both processes contribute to the development of a cognitive system, that is, they represent processes of individual learning. We argue that those processes that occur in cognitive systems through internalization may happen in social systems in an analogous way through externalization of an individuals own knowledge into a shared artifact. So, social systems can be shaped by integrating relevant information or by separating inappropriate informationin either an assimilative or accommodating way: They can develop by merely adding new content or by removing contradictory information (external assimilation). Or, social systems may emerge by modifying their own structure (external accommodation). Such external accommodation processes are supposed to lead to a more pronounced complexity of the shared digital artifact and, potentially, as a consequence, to new cognitive conflicts in other people. We emphasize that internalization and externalization, that is, individual learning and collaborative knowledge construction, are strongly interdependent. Thus, we need to examine internalization and externalization as interdependent processes that take place at the same time in order to understand this continuous exchange process between cognitive and social systems. This continuous exchange process is exactly what we want to trace and illustrate here. In the past years we have conducted a variety of empirical investigations into the interplay between cognitive and social systems, applying a range of methods, such as laboratory experiments (Moskaliuk, Kimmerle, & Cress, 2009) or Social Network Analysis (Kimmerle, Moskaliuk, Harrer, & Cress, 2010). Based on our experiences from previous studies, we have conducted an empirical examination in which we considered processes of individual thinking and related processes of social exchange at the same time. We will explicate the processes of internalization and externalization by examining how two people collaboratively developed a shared text on a controversial issue in a sequence of individual sessions. In doing so, we simultaneously collected thinking-aloud data and contributions to a text.
Method
The wiki-approach can be applied well to situations in which people tend to have controversial opinions on a certain subject. They can introduce their own positions on equal terms (Moskaliuk & Kimmerle, 2009): Participants may express opposing opinions, addressing each other, and they incorporate their own perspective into a coherent text [and] Individuals may acquire new knowledge when they internalize information from the wiki (Kimmerle, Cress, Held, & Moskaliuk, 2010, pp. 13-14). Therefore, we applied the same wiki principle here and in this case used a collaborative writing tool in order to support participants in writing a shared text on a controversial topic. We provided them with a document that allowed them to revise all parts of a text, add, change, or delete anything (like with a wiki). We were selectively looking for participants who were interested in naturopathy and homeopathy and had an unambiguous opinion either for or against homeopathy. We chose this domain, as there was recently a controversial debate on the role of homeopathy in the public health sector in Germany, and much information on this topic was discussed in the media coverage. This allowed us to recruit participants with extensive prior knowledge on this topic who were motivated to take part in this discussion. We advertised the participation in this study and offered 60 EUR for a three-hour session. The advertisement pointed out that the participants task would be to collaboratively write a text on the issue Should homeopathy still be allowed in Germany. The
ISLS
ICLS2012
advertisement explained that participants were supposed to advance their own point of view, but be willing to engage in a discussion with others. The advertisement outlined as the goal of the collaborative writing process the preparation of an article that expressed the opinions of the people involved and summarized the basic arguments. Candidates had to fill in a short questionnaire to apply for participation in this study. We asked them about their prior knowledge on the topic and for their personal opinion. In addition, they had to indicate their agreement or disagreement with two statements (pro and contra homeopathy) using a 7-point Likert Scale. We used the data of this questionnaire to select two university students as participants in our study: One woman (age: 25; German education and English education student) and one man (age: 32; chemistry student). The two participants had different opinions about the given issue. Their applications showed that they had detailed knowledge about homeopathy, were able to outline their opinions in an adequate way (for the role of argumentation in computer-supported collaboration cf. Stegmann, Weinberger, Fischer, 2007; Weinberger, Stegmann, Fischer, 2010), and were highly motivated to participate in this study. We used eduPad (http://edupad.ch/) as a collaborative writing tool. eduPad provides a simple text editor and allows participants to change the text online. The text written by the two participants was marked with two different colors during the writing process to identify the author of specific parts of the text. This supported the quick identification of ongoing changes in the text. All changes of the text made by the two participants were stored as different versions of the text. The two participants had to write in turns on the shared text: There were eight phases, with four writing phases for each of the two participants. Their goal was to develop a text that outlined their combined reasoning on the given question (Should homeopathy still be allowed in Germany?). They were instructed that the resulting text should represent both opinions and integrate the different lines of argument. Each of the two participants had ten minutes available per round to write on the text (however, if participants asked for more time in order to finish their current entry, we permitted up to four additional minutes). One participant had to wait in a separate room while the other one was working on the text. Participant 1 (P1), who advanced a pro homeopathy view, started with an almost empty file that contained no content except for a short headline: Homeopathy in crisis. After the first phase, participant 2 (P2), who advanced a contra homeopathy view, proceeded, followed again by P1 and so on. We instructed the participants to think aloud (cf. Ericsson & Simon, 1980) during their work with eduPad and recorded their comments digitally (for a discussion on the suitability of the think-aloud method cf. Ericsson & Fox, 2011; Schooler, 2011). In addition, a screen-capturing tool was used to log the writing process in detail. The verbal comments were transcribed (think-aloud protocols) and matched with the data from the screen-capturing tool and the stored text versions. This led to a rich data base that combined internal processes (think-aloud protocols) and external processes (text development).
Analysis
In this section we will present the core findings of our analysis. As described above, both individual learning and collaborative knowledge construction may either take place in the form of assimilation or accommodation. The former is chiefly a quantitative process while the latter is supposed to be a process that leads to qualitative improvement. This section is organized accordingly. First, we will briefly describe the development of the wiki text over time in quantitative terms, that is, the development of the absolute length of the text and the increments of text in the eight particular episodes. In doing so, we will illustrate the alternations between phases of quantitative and qualitative development. Subsequently, we will provide a qualitative analysis in which we will examine critical incidents in the knowledge construction process. In this process, we will analyze situations in which the participants decided to deal with controversies by either writing new text or deleting existing text. We will describe the interplay of thinking and writing and examine situations in which the participants decided not to introduce their opinions or knowledge into the text. In addition, we will analyze how participants compromised in the collaboration process and discuss our findings regarding this process as a co-evolution of thinking and writing. Finally, we will describe how norms of collaboration developed during the writing process.
Alternations between Quantitative and Qualitative Development

As was mentioned above P1 started with an almost empty document that contained just a short headline. In the beginning of the writing process there was, naturally, mainly a quantitative increase. After the first two rounds (one for each participant) the document contained already 40 % of the final text (311 of 780 words). In the last two versions (rounds 7 and 8) the addition of text was only about 5 % (42 of 780 words). In these final phases of the writing process, however, qualitative development instead of quantitative expansion of the text became more relevant. This increase of the text length can be seen in Figure 1. Such alternations between quantitative and qualitative writing activities did not merely occur between the first and the last versions, but over the entire writing process. Alternations between quantitative and qualitative phases of
ISLS
ICLS2012
text-production are also known from previous research on individual writing processes (cf. Hayes & Flower, 1986).
Figure 1. Development of text length (cumulative) across the eight versions of the text. In the beginning of the collaborative writing process P1 merely added initial information (version 1), because there was no text that this participant could revise in any way. Afterward, there was an alternation from mainly quantity-orientated and to rather quality-orientated periods. We found a first shift from quantitative to qualitative writing activities in text version 3: In version 1, P1 had started with a short definition of homeopathy and tried to outline the controversy of the topic concerning the efficacy issue; in version 2, P2 complemented the efficacy discussion with an additional aspect, the placebo effect. In her second turn (version 3) P1 wrote a kind of conclusion trying to integrate the different points of view: P1, wrote text: Its a fact that homeopathy and other alternative therapeutic methods are controversial topics. There is no clear answer as to whether they are a true alternative, because their advocates just focus on the effects, whereas the opponents focus only on the facts which are not measurable. Everybody should decide for himself whether he is for or against it. The think aloud protocols (TA) supported the assumption that P1 tried to integrate the conflicting approaches here: P1, TA: I dont want to contrast both perspectives, but bring them together somehow. In version 4, in turn, we found a shift from qualitative to quantitative writing activities. In this version P2 contributed two new perspectives on the homeopathy controversy, a financial and an ethical aspect. In version 5 P1 complemented in turn the financial aspect. Then, in version 6 the next shift from quantitative to qualitative activities took place: Here, P2 tried to merge the different aspects of the text after having recognized: P2, TA: Its really noticeable that this whole thing doesnt fit together. Altogether, we may conclude that taking up a new perspective of the topic (by one of the writing partners) triggered quantitative processes in a first step which was, however, closely followed by qualitative processes, such as structuring the text or merging different points of view. So a quantitative focus alternates with a qualitative focus across the whole collaborative writing process. This is in line with our theoretical differentiation of external assimilation and external accommodation. The co-evolution model describes these processes as two interrelated aspects of knowledge construction in a social system.
The Interplay of Thinking and Writing

When people want to contribute to a shared document, they have to externalize particular aspects of their own knowledge. For this purpose they have to transform their thoughts into written language. In doing so, they have to reflect on pre-existing information in the text to ensure that they integrate their cognitive concepts adequately. In this section we focus on the process of externalization, considering, on the one hand, the situations and social
ISLS
ICLS2012
conditions that seem to foster externalization and, on the other hand, situations that seem to inhibit externalization processes. In our assessment we take into account not only writing but also deleting activities. As we have captured thinking-aloud data simultaneously while participants edited the text, we have the opportunity to match internal and external processes in this analysis. So we can see that in many cases the process of externalization was clearly associated with the respective mindset of the particular participant, for example: P1, TA: Homeopathy wasno, umoh, how should I start Homeopathy Yes, right! I will start with a definition, namely with a translation of the term... Accordingly, this is what P1 was writing directly thereafter: P1, wrote text: Homeopathy means literally: To treat the same with the same. A consistency of thinking and acting may also find expression in deleting text as can be seen from the following example: P2, TA [after reading the text]: I dont think thats so great, for example. And it colors everything, as well. P2, deleted text: One should first give homeopathy a try, and if it doesnt have any effect, then fall back on allopathic medicine. This strong association of internal and external processes is in line with the idea of the co-evolution model that externalization and internalization depend on each other. If the participant had a deviating opinion about the information introduced by the other participant or wanted to introduce new knowledge, s/he added, edited, or deleted the shared text. There is, so to speak, a direct interrelation between the internal processes recorded using the think-aloud protocols and the resulting external processes, that is, writing text. At the same time, we observed episodes in which externalization seemed to be influenced by additional processes. We argue that individual perception of the shared text influenced externalization. The previous interactions between the two participants, the current status of the collaboration, and the occurring socio-cognitive conflict became manifest in the shared document. This led to a mutual development of internal and external processes. In the following, however, we will report episodes in which internal processes differed from resulting external processes.
Co-Evolution of Thinking and Writing

A peculiar gap between the writing process and the present thinking of a participant could be observed several times during the knowledge construction process. In some cases, the participants decided to keep their opinions or their current thoughts out of the collaborative situation: P1, TA: You cant pigeon-hole people who use homeopathy. Like saying they are some kind of strange people. But I am not going to add any more about that now. Anyway, Ill leave it like that, otherwise he will change it again. In other situations we observed great differences or a missing relationship respectively between thinking and writing: P1, TA: No, no. I do not agree with one thing he wrote there. With anything. Anything at all. I could delete all of his changes now and, and... Oh, never mind. Ok, what should I do with that? But, instead, P1 addressed a completely different topic in the next moment: P1, wrote text: Still, one should consider also that allopathic drugs are much more expensive, because In this example P1 decided not to add her own opinion, as she assumed that the other participant would delete her additions anyway. P1 seemed to look for an agreement, as the goal of the collaboration was to come to a common statement on the given topic. Instead of resuming the confrontation she tried to integrate the different perspectives. The observed gap between thinking and writing resulted from a kind of filter that decided which part of ones own knowledge should be introduced to the shared text. Even if ones own opinion or knowledge was different from that of the other participant, the written text was adjusted to accommodate the common goalthat is, coming to an agreement. In the first example, P1 perceived the socio-cognitive conflict between her own opinion and that of her counterpart but decided to ignore the difference. In the second example, she decided to elaborate more on the topic and introduced an additional counter-argument. Based on the co-evolution model, we might conclude that the social system (the shared text, the current collaboration, and previous experience during the interaction) influenced how thinking was related to writing. Individual perception of the social system filtered if and how the knowledge from the cognitive system was externalized into the shared document. We argue that the observed gap between thinking and writing is in line with the co-evolution model, as there was a mutual influence of internal and external processes. The externalization process depended on the current status of the social system. The collaborative writing processes were strongly influenced by the social conditions implemented here. Thus, we also observed situations in which internal processes and external
ISLS
ICLS2012
process strongly influenced each other. In the following we present two additional episodes where this coevolution of thinking and writing became evident. P2 made clear his skeptical attitude towards homeopathy in the text: P2, wrote text: [] effect does not measurably differ from placebos [] [] the most radical advocates of homeopathy do not accept this and often use outrageous pseudo-scientific facts to sell their often expansive preparations [] [] so-called alternative medicine. When P1 returned to the computer and read the revised text for the first time (in phase 3), the conflict of opinion became evident (<> means that a participant is reading the text aloud): P1, TA: <What is indisputable is the placebo effect most radical advocates> What!? What?... so <his self-healing powers> yes, okay, well. If he thinks so. I dont think so, but if he thinks so, mmh that is so well, but, so, what strikes me is he is obviously against homeopathy yes, that is obvious. Well! Im gonna write something into it. P1, wrote text: [] effect does also occur with allopathic drugs. P1, deleted: so-called in so-called alternative medicine and deleted: outrageous in outrageous pseudo-scientific facts. Even if P1 advanced a pro homeopathy view, she did only delete outrageous and did not edit the attribution pseudo-scientific. Again this seemed to be an attempt to find a compromise between the two contrary positions. P1 subordinated her own opinion to the goal of writing a common statement. We observed similar processes for P2. Even if he had a contra homeopathy view, he did try to find formulations with which he could anticipate the agreement of P1. In another episode P2 highlighted the empowerment of patients and the obligation of physicians and politics to extensively inform patients about homeopathy: P2, TA: Enlightenment is an important point P2, wrote text: [] a way that enlightens patients [] nonsensical and harmful [] P2, TA: Alas! Its no use. P2, deleted: that enlightens patients [] nonsensical and harmful [] P2, wrote instead: In any case the patient should be the priority. Probably, as result of the anticipation that P1 would not accept the description of homoeopathy as nonsensical or harmful, P2 deleted this part of the text and searched for more moderate wording. This, again, points out that internal processes are influenced by external, that is, (anticipated) social processes. P2 started to write down his opinion about homeopathy as a nonsensical and harmful method, but then deleted the written text and attenuated his point. Filtering his own knowledge before externalizing it to the shared document alleviated the perceived socio-cognitive conflict between his own knowledge and the information in the wiki. Again, the thinking processes seemed to be influenced by the perception of the social system. In this way, the previous collaboration, the externalized knowledge of both participants, and their common goals led to a more moderate formulation. In the presented data we observed that the participants tried to come to an agreement and made compromises. As the goal of their collaboration was to write a common text, the participants subordinated their individual goals. They solved occurring conflicts by filtering their own knowledge, or even by going so far as to not externalize specific arguments. Thus, internal processes and the resulting process of externalization were strongly interdependent. Both processes took place simultaneously as a continuous exchange between cognitive and social systems.
Conclusion
In this paper, we have elaborated the interdependence between internal and external processes of knowledge construction. We examined the internal process by asking the participants to think aloud and the external processes by recording the writing process closely. This allowed us to observe internalization and externalization as interdependent processes that take place at the same time. Our data highlight the continuous exchange processes between cognitive and social systems and the resulting process of knowledge construction. We interpret the observed processes as a co-evolution of cognitive and social systems. We created a specific setting in which two people with extensive knowledge on a given topic, but with contrary opinions, had to collaborate and construct a common statement. This setting provided ideal conditions to study the postulated process of knowledge construction. The socio-cognitive conflict was a key incitement factor of the collaborative construction of knowledge and had to be resolved by the two participants. Thus, the participants made compromises and integrated their different arguments. This led to the reported gap between thinking and writing as an indicator for the influence of the social system on individual processes. In the described collaboration of the two participants we observed a development of collaboration norms during the writing process. Even if there was no direct interaction via chat or face to face, the collaboration using the shared text was strongly influenced by the perceived or anticipated reactions of the other person and by the knowledge which was presented. The current status of the
ISLS
ICLS2012
shared document as a social system strongly influenced how an individual chose to introduce his/her own knowledge into a shared artifact. We also observed a behavior that might be called a strategic component of knowledge construction: A controversial idea with an anticipated low chance of survival in the knowledge construction process was not even externalized. To sum up, new knowledge may develop when people interact with each other via a shared digital artifact such as a wiki text. The integration of different arguments is triggered by the perceived cognitive conflict. We could describe this process of knowledge construction as co-evolution of internal and external processes. The method of combining a think-aloud protocol with a close observation of the writing process emphasizes that research on knowledge construction should consider internal and external process at the same time. Future studies should aim at extending and modifying our research procedure in two respects. First, further analyses might consider examining a collaborative situation in which the participants do not have such a strong commitment to their respective positions. In many situations in research on computer-supported collaborative learning this is rather unusual, for example in scripted scenarios where one person is given different information than another. Second, in our research setting knowledge construction was more constricted than in typical collaborative interactions with wikis. Usually people have more freedom to organize their collaboration than with our predefined time periods. Future studies might apply our approach to examine collaboration scenarios with higher degrees of ecological validityfor example with wikis in educational contexts or on the Internet.
References
Andriessen, J., Baker, M., & Suthers, D. (2003). Arguing to learn: Confronting cognitions in computersupported collaborative learning environments. Dordrecht, Netherlands: Kluwer Academic Publishers. Bereiter, C., & Scardamalia, M. (1996). Rethinking learning. In D. Olson, & N. Torrance (Eds.), Handbook of education and human development: New models of learning, teaching and schooling (pp. 485513). Cambridge, MA: Basil Blackwell. Cress, U., & Kimmerle, J. (2007). A theoretical framework of collaborative knowledge building with wikisa systemic and cognitive perspective. In C. A. Chinn, G. Erkens, & S. Puntambekar. (Eds.), Proceedings of the 7th Computer Supported Collaborative Learning Conference (pp. 153161). New Brunswick, NJ: International Society of the Learning Sciences. Cress, U., & Kimmerle, J. (2008). A systemic and cognitive view on collaborative knowledge building with wikis. International Journal of Computer-Supported Collaborative Learning, 3, 105122. Ericsson, K. A., & Fox, M. C. (2011). Thinking aloud is not a form of introspection but a qualitatively different methodology: Reply to Schooler (2011). Psychological Bulletin, 137, 351354. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87, 215251. Forte, A., & Bruckman, A. (2010). Citing, writing and participatory media: Wikis as learning environments in the high school classroom. International Journal of Learning and Media, 1(4), 2344. Glassman, M., & Kang, M. J. (2011). The logic of wikis: The possibilities of the Web 2.0 classroom. International Journal of Computer-Supported Collaborative Learning, 6, 93112. Hayes, J. R., & Flower, L. S. (1986). Writing research and the writer. American Psychologist, 41, 11061113. Johnson, D. W., & Johnson, R. T. (1987). Creative conflict. Edina, MN: Interaction Book Co. Kimmerle, J., Cress, U., & Held, C. (2010). The interplay between individual and collective knowledge: Technologies for organisational learning and knowledge building. Knowledge Management Research and Practice, 8, 3344. Kimmerle, J., Cress, U., Held, C., & Moskaliuk, J. (2010). Social software and knowledge building: Supporting co-evolution of individual and collective knowledge. In K. Gomez, L. Lyons, & J. Radinsky (Eds.), Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences (Vol. 1, pp. 916). Chicago, IL: International Society of the Learning Sciences. Kimmerle, J., Moskaliuk, J., & Cress, U. (2009). Learning and knowledge building with social software. In C. O'Malley, D. Suthers, P. Reimann, & A. Dimitracopoulou (Eds.), Computer Supported Collaborative Learning Practices: CSCL2009 Conference Proceedings (Vol. 1, pp. 459468). Rhodes, Greece: International Society of the Learning Sciences. Kimmerle, J., Moskaliuk, J., & Cress, U. (2011). Using wikis for learning and knowledge building: Results of an experimental study. Educational Technology & Society, 14(4), 138148. Kimmerle, J., Moskaliuk, J., Harrer, A., & Cress, U. (2010). Visualizing co-evolution of individual and collective knowledge. Information, Communication and Society, 13, 10991121. Luhmann, N. (1995). Social systems. Stanford: Stanford University Press. Moskaliuk, J., & Kimmerle, J. (2009). Using wikis for organizational learning: Functional and psycho-social principles. Development and Learning in Organizations, 23(4), 2124.
ISLS
ICLS2012
Moskaliuk, J., Kimmerle, J., & Cress, U. (2008). Learning and knowledge building with wikis: The impact of incongruity between peoples knowledge and a wikis information. Proceedings of the International Conference for the Learning Sciences 2008, Vol. 2 (pp. 99106). Utrecht, The Netherlands: International Society of the Learning Sciences. Moskaliuk, J., Kimmerle, J., & Cress (2009). Wiki-supported learning and knowledge building: Effects of incongruity between knowledge and information. Journal of Computer Assisted Learning, 25, 549 561. Moskaliuk, J., Kimmerle, J., & Cress, U. (2012). Collaborative knowledge building with wikis: The impact of redundancy and polarity. Computers & Education, 58, 10491057. Nussbaum, E. M., Winsor, D. L., Aqui, Y. M., & Poliquin, A. M. (2007). Putting the pieces together: Online argumentation vee diagrams enhance thinking during discussions. International Journal of ComputerSupported Collaborative Learning, 2, 479500. ODonnell, A. M. & OKelly, J. (1994). Learning from peers: Beyond the rhetoric of positive results. Educational Psychology Review, 6, 321349. Parker, K. R., & Chao, J. T. (2007). Wiki as a teaching tool. Interdisciplinary Journal of Knowledge and Learning Objects, 3, 5772. Piaget, J. (1977). The development of thought: Equilibration of cognitive structures. New York: Viking Press. Pifarr, M., & Kleine Staarman, J. (2011). Wiki-supported collaborative learning in primary education: How a dialogic space is created for thinking together. International Journal of Computer-Supported Collaborative Learning, 6, 187205. Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith (Ed.), Liberal education in a knowledge society (pp. 6798). Chicago: Open Court. Schooler, J. W. (2011). Introspecting in the spirit of William James: Comment on Fox, Ericsson, and Best (2011). Psychological Bulletin, 137, 345350. Stegmann, K., Weinberger, A., & Fischer, F. (2007). Facilitating argumentative knowledge construction with computer-supported collaboration scripts International Journal of Computer-Supported Collaborative Learning, 2, 421447. Wadsworth, B. (1978). Piaget for the classroom teacher. New York: Longman. Weinberger, A., Stegmann, K., & Fischer, F. (2010). Learning to argue online: Scripted groups surpass individuals (unscripted groups do not). Computers in Human Behavior, 26, 506515.
ISLS
ICLS2012
Understanding How Learners Grapple with Wicked Problems in Environmental Science

Brian Slattery, Chandan Dasgupta, Tia Shelley, Leilah Lyons, Emily Minor, Moira Zellner University of Illinois at Chicago, 851 S. Morgan, Chicago, IL 60607 bslatt2@uic.edu, cdasgu2@uic.edu, tshell2@uic.edu, llyons@uic.edu, eminor@uic.edu, mzellner@uic.edu Abstract: Environmental science standards are calling for a perspective that highlights how social and natural systems interact. In order to properly deal with the wicked problems arising from this interaction, learners must recognize that there is no right answer, since solutions require compromise. They must also use spatial concepts instrumentally to reason about these systems. We propose to address these challenges by adapting authentic complex human-natural systems models into collaborative learning experiences. To do so, we need to better understand the challenges learners face as they use simulations to link spatial reasoning with dynamic processes. This paper presents two cases where we examine learners spatial and problem-solving strategies as they interact with a modified stormwater management model. We show that learners require support for core spatial reasoning skills and for problem solving around wicked problems. We then recommend forms of scaffolding and further development.
Introduction
Over the past decade there has been increased acknowledgement that mile wide, inch deep coverage of largely factual content is insufficient to prepare future scientists and scientifically literate citizens (NAE, 2008), causing the College Board to embark on the redesign of four AP (Advanced Placement) science courses (Biology, Chemistry, Environmental Science, and Physics) in June 2006. The changes made to the AP Environmental Science standards have placed increased recognition on the importance of taking a complex systems perspective on scientific phenomena such as ecosystems (College Board, 2009). Agent-Based Models (ABMs), one approach to complex systems modeling, can facilitate multidisciplinary learning about the science and policy issues arising from human-environment interactions, but research is needed on how to make these models intellectually and pragmatically accessible to learners. This project addresses that need by developing and testing new tools and pedagogical strategies using an iterative design-based research approach. Our first exploration of these issues engages learners in Green Infrastructure (GI) planning, an authentic problem-based context incorporating Urban Planning (UP) and Environmental Science (ES) disciplines. GI is defined as "an interconnected network of green spaces that conserves natural ecosystem values and functions and provides associated benefits to human populations" (Schilling & Logan, 2008). GI often takes the form of vegetated swales or green roofs that aim to minimize urban stormwater runoff and associated pollution. Since communities do not have infinite resources, they must make strategic decisions about where GI elements will make the most impact. In our collaboration with disciplinary experts from both UP and ES, we have identified certain key reasoning skills required to grapple with challenges at the intersection of human and natural systems. While others have investigated challenges of learning about complex systems (Hmelo-Silver, Marathe, & Liu, 2007), two things set our learning challenges apart. First, the complex systems present in UP and ES are fundamentally defined by their spatial properties and relations. Second, in order to understand how human (UP) and natural (ES) systems interrelate, one must study how their spatial properties and relations intersect. Similarly, the integration of policy or economics transforms these complex problem spaces into wicked problems, a class of ill-structured problems originally defined in UP literature (described further below). This exploratory study focused on a subset of spatial reasoning skills, and on one critical skill for grappling with wicked problems: learning to compromise across multiple competing demands. The ABM we used in this study was adapted from a GI planning ABM, L-GrID, developed for the Illinois Environmental Protection Agency by the projects disciplinary expert co-PIs (Zellner et al., in progress). We recruited pairs of undergraduates (to act as proxies for high school seniors who might be in AP Environmental Science courses), and asked them to engage in an authentic GI problem-solving task with the ABM. Because this was a lab-based study, to encourage them to take the problem solving task seriously, we borrowed from methods used in experimental economics. The ABM was structured to contain two competing reward functions, and participants received a bonus cash payout proportional to their optimization of these two functions. We wanted to observe how the participants strategized as they tried to optimize, to better understand how we can support the reasoning skills we identified as being key to the domain. This paper describes how we derived those key skills, then illustrates via case studies how participants used these skills to address a wicked problem, and concludes with recommendations for further development of both ABM tools and the supporting curriculum for multidisciplinary UP and ES learning.
ISLS 9
ICLS2012
Background
Content Domain: The Intersection of Environmental Science and Urban Planning
To introduce students to the fields of ES and UP, we chose to work with a complex systems model that represents rare stormwater events. This model incorporates elevation, land usage, roads, and sewers, and demonstrates how the placement of green infrastructure (in the form of rain gardens) affects stormwater movement. Our study allowed students to place these gardens on a map of 20 city blocks with various land uses (residential, business, and industrial) that can support differing numbers of rain gardens. The simulation shows the effects of a hundred year storm on the area, illustrating how the arrangement of gardens leads to more or less effective stormwater management. Effectiveness is defined by the two competing reward functions: one stresses a social priority (cost), and the other stresses a natural priority (time for water to drain into the soil). While this problem is important to practicing urban planners and ecologists, it also provides a valuable context for learners new to those fields. Key reasoning practices found in these disciplines are described below.
Spatial Reasoning in Environmental Science and Urban Planning

Planners argue for the importance of spatiotemporal context and place-based analysis in their research. These tasks draw on planners ability to think spatially. Spatial thinking in the social sciences generally demands a simultaneous integration of multiple spatial concepts. In Table 1 we build on five general spatial skills identified by Janelle & Goodchild (2011), who situated these skills in the context of practice. Column 1 lists the general skills, while Column 2 provides examples of areas where those skills would be used in UP and ES. A host of other researchers (Golledge, 1995; Janelle & Goodchild, 2011; Jo & Bednarz, 2009; Kaufman, 2004), however, have also produced categorizations and ontologies of spatial skills, which we use to identify component skills in Column 3. Column 4 then illustrates where those component skills would arise in UP and ES. Table 1: Illustration of how general spatial skills can be broken into component skills
General spatial skill (Janelle & Goodchild, 2011) Example skill application areas, for UP and ES
Generalized component skills
Examples of component skills in use in UP and ES
Differentiating: 1. Detect changes in ES: habitats the uses of, and UP: land use, regional e.g., differentiation of, agricultural, spaces residential 2. Measure the physical arrangement and clustering of phenomena to identify spatial patterns
Satellite imagery and maps are often used to determine boundaries, but one must: The ability to define a set of salient ES: select habitat factors and cutoff points variables, based on the phenomena (e.g. % tree coverage that counts as habitat) of interest, with which to relevant to species of interest differentiate space into different UP: define properties (e.g., types of areas residences) and proportions that distinguish urban from exurban land use The ability to differentiate areas (see above) and to perceive their relative position; adapt concepts of distance and connectivity to context; qualitative language for patterns A distance that permits two areas to be connected can be influenced by: ES: the foraging range of species of interest UP: the type of a road (highway, surface street)
Identifying: ES: nest patterns UP: settlement patterns
The ability to recognize and distinguish among different spatial Documenting: patterns (see prior general skill), to 3. Document spatial ES: invasive associate different patterns with patterns over time to species spread aspects of dynamic process, and to infer processes UP: urban design consistent measurement sprawl schemes to track patterns of interest; integrate analysis of both relative and absolute location 4. Study flows between specific locations as indicators of spatiotemporal interactions 5. Measure spatial Spatial dependence: the ability to integrate understanding of a Studying: dynamic process with an ES: genetic understanding of how spatial drift locations attenuate that process; the UP: traffic flow ability to map key steps in process to key locations Investigating: A sense of time scale and spatial
Both UP and ES have been hampered by inconsistent schemes for measuring patterns. For example, over the last 30 years: ES: National Land Cover Data set (NLCD) has changed land cover definitions UP: Land use definitions (especially urban) have changed ES: Studying migration of genes across landscape as an indication of how connectivity of habitat may have changed UP: Studying road congestions as an indication of the interaction between signal light timing and vehicle acceleration ES: Test if connectivity affects rate of genetic
ISLS
10
ICLS2012 associations (and space-time associations) to test hypotheses ES: effect of habitat features on organisms UP: effect of zoning on land use scale magnitudes, and quantitative metrics for measuring spatial properties and patterns
Volume 1: Full Papers drift by measuring genetic diversity over time as habitat pattern changes UP: Test if zoning affects land use by measuring density of development in zoned and unzoned areas over span of 10 years
Engaging in the spatial reasoning processes outlined above entails integrating a number of component spatial skills, many of which involve making nuanced judgements about spatial concepts. For example, although the concept of a habitat patch may be simple, operationalizing it in a spatial sense requires judgements about spatial cut off points, among other distinctions. While these judgements are certainly guided by the topic of interest (e.g., a specific species of animal, or certain impacts of land use), even experts recognize that these decisions are not objectively certain. It is just this sort of practice with making critical, qualitative judgements (reasoned guesses) that has been lacking in traditional STEM curricula, and whose lack often renders recent STEM graduates paralyzed when confronted with real-world ambiguity. Learning how to approach a problem space that requires such judgements, learning to not be paralyzed by uncertainty, but also recognizing that other judgements could have resulted in alternate results and remaining open to other problem formulations are part of authentic practice in these domains. While this is impossible to fully address in our study without a structured curriculum, extended time, and other supports, we do wish to discover how learners differentiate spatial regions based on the salience of observed characteristics (general skill 1), perceive and describe patterns (general skill 2), and test hypotheses by measuring spatiotemporal associations (general skill 3).
Wicked Problems
An important component of the planning and policy practices described above is the necessity to make consequential decisions when there is inadequate, conflicting, or changing information. This particularly complex quandary is known as a wicked problem (Rittel & Webber, 1973). Wicked problemsin contrast to "tame" or "benign" problems that may be difficult, but are definitely solvablecannot be clearly defined or solved, resist objective judgment, and are unique and nested in larger issues, among other qualities. In this way, wicked problems share some similarities to ill-structured problems. Ill-structured problems dont have a positive definition, however, but are instead defined in relation to well-structured problems, which have single solutions, optimal solution paths, and structured goals (Sinott, 1989). By contrast, ill-structured problems have unclear (and possibly multiple) solutions, although the mechanisms used to solve both classes of problems are thought to be the same (Simon, 1973). Wicked problems similarly have multiple possible solutions, no clear optimal solution path, and ill-defined goals. Wicked problems differ from ill-structured problems in three main ways, however: (1) wicked problems emphasize the importance of context in shaping the goals, solutions, and solution paths, meaning that solution strategies developed for one context may not be applicable to another context; (2) wicked problems stress the consequentiality of solutions on the real world, meaning that problem solvers are asked to take stances on values and morality in order to define goals; (3) wicked problems are collective in that the values and morality incorporated into goal definitions are informed by a range of diverse stakeholders (Munneke et al., 2007). These differences suggest that what is known about ill-structured problem solving strategies may not be sufficient to understand how to support learners as they attempt to solve wicked problems. When planners attempt to solve wicked problems, they must have the ability to frame and define problem spaces, identify possible goals and solutions in those spaces, enact procedures to reach those goals, and do all of these in a way that fits real-world constraints. With the short, 2-year professional training planners receive, they cannot practice exercising these skills enough to adopt them as part of their standard practice (Zellner & Campbell, 2011). Clearly, planners need earlier and more frequent exposure to these problem spaces, but others would benefit as well. Incorporating wicked problems into public education would better prepare future stakeholders for participating in the planning process, because stakeholders must ultimately buy into the tradeoffs inherent in plans for those plans to succeed. Thus, the wicked problems selected for incorporation into curricula should have relevance for both future planners and future stakeholders. For our study, we chose the domain of stormwater management, which is a planning problem that urban and suburban residents find relatable. This problem domain also has relevance to environmental science. Stormwater management (especially in urban areas) requires spatial specificity incorporating both human and natural systems, and has direct consequences on city residents and other stakeholders. Working with this rich domain, we decided to focus on how it could be used to highlight a subset of the reasoning required for dealing with wicked problems. The accessibility of the stormwater management domain allows us to gradually introduce novice learners to reasoning about wicked problems, with the goal of starting learners on the progression towards developing the skill of compromising across competing definitions and demands. This is similar to Songers (2006) BioKids curriculum, which similarly repackages professional-level concepts into a learning progression more suitable to novice learners.
ISLS 11
ICLS2012
Complex Systems Models as Tools for Exploring Wicked Problems

Complex systems simulations are an existing approach for presenting domain-specific and real-world phenomena to learners (Goldstone, 2006). These simulations are dynamic, centrally rely on iteration and feedback, and are made up of many parts organized across multiple levels of scale. As such, these tools provide a new opportunity for confronting aspects of wicked problems otherwise considered intractible. For example, when Rittel & Webber (1973) conceptualized wicked problems, they assumed that any exploration of the problem space would take place in the real world. With complex systems simulations, practices that Rittel & Webber advocated against (e.g. experimenting with different strategies, examining multi-level interactions) are instead sources of explanatory power (Zellner & Campbell, 2011). Learners are being provided with new tools that represent the spatial and dynamic interactions that underlie wicked problems in urban planning and environmental science, allowing learners to reason about these problems in a different way than they could without modeling and simulation software. This contrasts with common school practices, which prioritize correct answers over exploratory processes, often fail to link classroom activities to real-world practices, and mainly deal with well-defined problems and procedures. This exacerbates students inability to deal with wicked problems by giving them practice in exactly the wrong skills. One value of complex systems simulations lies in their ability to help users recognize how spatial and dynamic interactions lead to the emergence of wicked problems, which allows them to explore a richer range of solutions. The tools also allow learners to actively explore the different tradeoffs involved in solving wicked problems. This exploration forces learners to voice and test their hidden assumptions, leading to the collective definition of values that is necessary when grappling with wicked problems.
Prior Work
Learning Technologies Applicable to Wicked Problems in Urban Planning and Environmental Science
Technological capabilities for scaffolding spatial thinking have been developing faster than our understanding of the acquisition of skills in fundamental spatial thinking (Janelle & Goodchild, 2011). Geographic information systems (GIS) are heralded as a potentially effective tool for teaching basic spatial concepts in K12 classrooms (NRC, 2006). However, when implemented in their current state in classrooms the risk may become one of teaching buttonology, or point-and-click procedures, to obtain a specified outcome (Marsh, Golledge & Battersby, 2007). With the buttonology approach, learning how to use the software program often supercedes conceptual and procedural understanding of the spatial analysis the software program is performing. Some packages, like MyWorld (Brown & Edelson, 1998) intentionally simplify the interface to re-align focus on the critical concepts and content (e.g., global earthquake data). We are taking a similar approach by adapting complex system models designed for use by environmental scientists and urban planners. Researchers have studied introducing learners to complex systems perspectives, which typically addresses how complex science content should be structured (e.g., Liu, Marathe, & Hmelo-Silver, 2005) or how to regulate students learning about complex systems (e.g., Azevedo et al., 2004). Such ideas will influence us as we structure our curriculum in the future, but perhaps more relevant to this study is work exploring how ABMs can be used to teach complex systems principles (e.g., Wilensky & Resnick, 1999; Goldstone, 2006). What distinguishes our work from theirs is that we are not trying to induce students to understand basic principles like emergence rather, we are using these tools to help students confront compromise.
Activity and Study Design

The original stormwater management model (L-GrID) was designed to allow researchers to investigate the impact of different types of ground cover on groundwater infiltration. To adapt it for use with novices in fifteenminute sessions we had to scale back the detail made apparent to the user. We reduced the types of ground cover to three: impermeable (road surfaces), highly-permeable (swales) and semi-permeable (non-road patches that were not yet converted to swales). Additional simulation elements included sewers (which drain water from surrounding patches), elevation (the map sloped from a high at the top right corner to a low at the bottom left), an output sink (at the lowest corner of the map), and rain (set to always reproduce a devastating hundred year storm). We presented different contexts by varying sewer placement and ground cover across three maps encountered by learners. We reduced the outputs of the simulation to two: infiltration speed (time taken to clear all of the storms water from the map) and cost (each swale cost $10,000). Users could move and place swales on patches of the map, hit go, and witness how their configuration affected both the score-based outputs (cost and infiltration) as well as view a visualization of the depth of the water as it flowed across the map. We recruited fourteen pairs of undergraduates (as proxies for high school seniors). Because this was a lab-based pilot study, to encourage them to take the problem solving task seriously we borrowed from methods used in experimental economics. The ABM was structured to use the two simulation outputs (cost and infiltration) as two competing reward functions, and the cash payout participants would receive could be
ISLS 12
ICLS2012
improved upon by optimizing across these two reward functions. To avoid discouraging them from exploring the problem space, the final reward was based on the best score they were able to attain within the allotted time of 15 minutes. We recorded video, transcribed their conversations, and collected data on the scores they attained with each trial. This stormwater management task gives learners the opportunity to engage with the aspects of wicked problems that are not always addressed by ill-structured problem solving tasks (context, consequentiality, collectivity). The two monetary reward functions are simplistic placeholders for the values of different stakeholders. Learners must determine how to reconcile the scores produced by these functions, which drives further exploration of the problem space. This reconciliation process is an entry point for novices to confront the demands of consequentiality and collectivity involved in solving wicked problems. Furthermore, by exposing learners to multiple maps whichdespite surface similaritiesdemand different approaches to swale placement, we highlight that the context shapes the selection of heuristics for solving wicked problems.
Figure 1. Screenshot of the GI ABM used in this study, adapted from a more extensive NetLogo-based model built by the co-PIs for the Illinois EPA-funded Green Infrastructure Plan for Illinois project.
Coding Schemes
The literature on problem solving has codified several common strategies, labeled weak problem solving strategies (Simon, 1978), that learners use when confronting ill-structured problems (e.g. analogy, hill climbing, generate-and-test, means-ends analysis, problem decomposition, problem conversion). However, these strategies were deduced from studies with individual learners, and thus may not encompass the full range of strategies we expect to see during wicked problem solution. In particular, the properties that distinguish wicked problems (context, consequentiality, collectivity) place special emphasis on evolving a collectively acceptable goal definition that fits a particular set of circumstances. This change in emphasis necessitated an emergent coding approach (Strauss & Corbin, 2008) to identify the kinds of strategies used by our participants. The strategy kinds we identified each describe a particular form of shared, sustained discussion or work on the model. These included budgeting the total amount of gardens used (formal budgeting), adding or removing a nonspecified number of gardens (informal budgeting), focusing on particular large (map-wide) or small (block- and road-level) scales (large- and small-scale spatial), reacting to the spatial dynamics of the last run of the simulation (reactive spatial), and exploring garden-placement maxima and minima (determining limits). Three strategies we identified correlated with higher scores, or were used more often (see Table 2). A formal budgeting strategy was defined as the pair of learners setting a shared cap on the number of gardens used. This strategy was unsurprisingly associated with better performance on our cost score metric, although the budgets we observed were seemingly arbitrary. For instance, they tended not to stray from a restricted band of values (a cap of around 20 gardens). Despite a surface similarity to a means-end strategy (Simon, 1978), we did not observe learners justifying their budget. This indicates that learners need to be scaffolded to explore a wider range of budgets which would have a more principled effect on cost score. Table 2: Average scores per trial and frequency of use for observed strategies.
Reactive Spatial 0.97 0.50 1.47 26.31% Formal Budgeting 1.03 0.34 1.37 6.88% Informal Budgeting 0.94 0.53 1.46 14.55% Determining Limits 0.88 0.56 1.43 3.33% Large-Scale Spatial 1.01 0.42 1.43 30.86% Small-Scale Spatial 1.01 0.42 1.43 13.78%
Avg. Cost Score Avg. Infiltration Score Avg. Overall Score % Freq. of strategy use
With regards to the infiltration score, learners were most effective using a reactive spatial strategy, so named because it involved incremental changes based on the visualization. Users would point out spots of the map where water flow slowed or stopped, and would accordingly shore those up with gardens or attempt to restructure their arrangement to guide water in different ways. An exemplar of hill-climbing (Simon, 1978),
ISLS 13
ICLS2012
using a reactive spatial strategy was effective in incrementally improving the infiltration score, but not as effective as the determining limits strategy. This suggests that learners need to better identify which parts of the map are critical, which is tied to detection of potential changes in the use of space (Janelle & Goodchild, 2011). Unlike the prior strategies we identified, the small- and large-scale spatial strategies were not so much related to problem solving heuristics as they were descriptive of the spatial reasoning of learners. It is also significant that the majority of learners began their interaction with the model by using a large-scale spatial strategy (see Table 2), which involves learners selecting and working on large regions of the map (usually described by learners in cardinal or relative directions, e.g., north/south, up/down, left/right). This stands in contrast to the relative paucity of small-scale spatial strategies, which would involve attending to small-scale features (e.g., adjacency and relative distance of map elements). Adjacency and distance can in fact strongly affect flooding, so learners require support to notice partitions of space (general skill 1) and their relations to one another (general skill 2), and to transition between different granularities of scale (general skill 5) (see Table 1). This could be accomplished through instructional support that identifies domain-meaningful distinctions amongst regions that users could choose to adopt.
Case Studies
Case 1: Learning to compromise takes more than heuristics
In this section, we highlight group conversations that show engagement with compromises between competing reward systems i.e, showing how they responded to the characteristic lack of true or false answers to wicked problems. The two reward systems highlighted in our experimental trial were cost reduction (the players were penalized for the cost of installing each green infrastructure garden), and rate of rainwater infiltration (the players were rewarded for reducing the time taken for rainwater to drain). These two reward systems combine to form an overall score although it should be emphasized that these reward functions are often competing: more gardens improve infiltration, but also increase the overall cost, and vice-versa. Moreover, the specific spatial arrangements of the gardens can also affect their infiltration efficacy. The following conversations have been selected from two groups of students to illustrate the types of discussion about compromise that occurred. In one group, S17 and S18 have just completed a garden arrangement and are watching the simulated outcome. They discuss their next strategy, balancing an observation that a larger number of gardens was helping them do well with infiltration against the observation that the cost was prohibitive: S17: We're doing well. Is the answer more gardens? S18: Oooh, arright. Where's the drainage seem to be caught, though? S17: Its going a lot faster, but its the garden cost, is, uh, kinda screwin us over S18: Well, you can removeI wanna remove this one. Maybe we could put two down here, maybe S17s observation of the decrease in cost score shapes the subsequent strategy. They attempt to balance these competing rewards and achieve an alternative solution in the problem space as they have constructed it. Their approach is incremental (this one, we could put two), suggesting that it did not occur to them that they could use a means-end strategy, working backwards from the desired cost score increase to determine the number of gardens to target for removal. Another group, however, showed a much more sophisticated approach. Initially, students S19 and S20 are trying to figure out their first garden-placement move, and seem to only consider the rate of infiltration (which they refer to as the time score): S20: In theory you should probably want to have more, uh, more sinks for the water up at the top, because that's where the source of them S19: Right S20: But, if you have them towards the bottom, then it would drain out the entire area probably more effectively S19: But in order to keep the time score low, maybe we should have some at the top and some at the bottom? S20: Um, let's see S19: To prevent all of the water from rushing down S20: Lets see, so the top drains out first Their initial strategy is to consider the infiltration pattern of the given map, by study[ing] flows between specific locations, (general spatial skill 4, see Table 1). They discuss whether it would be more effective to place gardens near the high elevation region (they mistake it as the source of the water) or near the main outflow at the bottom of the map. Later, after watching the simulation run their arrangement, they incorporate the reward structure of garden cost: S20: I think we're, uh, seeing cost score cominggoinggo down pretty quickly, so S19: /Yeah S20: that may mean we'rewe're, uhwe're adding few S19: We're getting a good return on, uh, on the gardens. Um, but we're still not anywhere near 2.25 S20: Yeah, lets see
ISLS 14
ICLS2012
S19: If we haveif we're getting a good return on the gardens we have, but we're ideally looking for a score that's S20: /Probably around 2? S19: Yeah, I mean 2.25 is the ideal S20: Yeah S19: I think we could probably have20 gardenson there, and still get a fairly good overall score. And a good return on gardens S19 and S20 not only identify the competing reward structure and factor it into their decision making process, but they actually generate an entirely new construct: good return on the gardens. The language they use here indicates that they are embedding cost and infiltration not as independent factors, but as a unified ratio a tradeoff. Unsurprisingly, it is after this discussion that S19 and S20 achieve their highest combined score. In their reasoning they also use the ideal score (what one could earn with maximum infiltration and no gardens, an impossibility) as an anchor point to work backwards from, in order to make judgements about how many gardens they can add without a significant hit in cost score. From these cases it is evident that while students are thinking about the competing reward structures without any explicit external prompts, this alone is not enough to guarantee success. The pattern exemplified by the first group (S17 and S18) shows a tendency to vacillate between first trying to satisfy one constraint, then the other. While this generate-and-test strategy can result in improved scores, these improvements are gradual, and may not help the learners to understand the essential relationship between the reward structures. The second group, on the other hand, demonstrates what is possible when means-end reasoning can be brought to bear on the problem, and when intermediate score evaluation representations (such as ratios) can be constructed.
Case 2: Lack of consideration of distance and adjacency

The pattern of sewers on the city map is an important small-scale spatial feature of the problem space. If learners place the gardens relative to sewer locations, they have the opportunity to maximize infiltration with a reduced number of gardens. However, we seldom saw this occur. While sewers were spoken of in 16 out of 28 (57%) conditions, mentions of sewers were present in only 1.19% of all discussion turns, a very low rate of occurrence around the second spatial component skill of distance and adjacency (see Table 1). In groups that do talk about distance and adjacency of gardens to sewers, the idea is very shortlived. For example: S20: What do you thinkedgesor towards the, um, towards the middle? I think, um, keep em towards the streets, thenthat's also where the sewers are S19: Yeah, I was thinking that perhaps we should orient the gardens near the sewers, um S20: Well then you have two sinks at theat the same locationversus S19: Spreading them out S20: Yeah S19: Yeah. Ok. Well what if we added four more gardens, umm, in this general area but not near the sewers This conversation lasts for less than a minute, and sewers are only briefly mentioned once more towards the end of their activity. This is typical of the conversations happening in other groups, where sewers are not recurrent strategic elements. This is an example of where scaffolding to help learners attend to smallscale spatial features could assist learners to bolster their spatial strategizing.
Discussion and Conclusion

Overall, learners required more support to engage in the spatial component tasks we identified as valuable for spatial reasoning and dealing with wicked problems. In particular, we found a need to support learners as they decompose space into regions, notice patterns across regions, and apply strategies that operate across multiple spatial granularities. To afford better reasoning about wicked problems, we need to encourage learners to engage in boundary testing, as this allows them to bring other problem-solving strategies (like means-ends analysis) to bear on the problem space. Since this study reports on a pilot of the simulation, we are in the process of developing a larger curriculum to complement this models impact on the development of spatial reasoning and systems thinking skills. It is very difficult for learners to incorporate all elements of the model into their judgments and plans, so a curricular scaffold could allow them to better accomplish this task. By balancing the diverse inputs, outputs, and processes that make up the simulation, learners can make more intentional and justified choices to affect the model, instead of taking actions that are abitrary or needlessly restricted.
References
Azevedo, R., Winters, F.I., & Moos, D.C., (2004). Can students collaboratively use hypermedia to learn about science? The dynamics of self- and other-regulatory processes in an ecology classroom. Journal of Educational Computing Research, 31(3), 215245.
ISLS 15
ICLS2012
Brown, M., & Edelson, D. C. (1998). Software in context: Designing for students, teachers, and classroom enactment. In A. S. Bruckman, M. Guzdial, J. L. Kolodner, & A. Ram (Eds.), Proceedings of ICLS 98: International Conference on the Learning Sciences, Atlanta, GA (pp. 63-69). AACE. College Board (2009). College Board Standards for College Success, from http://professionals.collegeboard.com/profdownload/cbscs-science-standards-2009.pdf Goldstone, R. L., (2006). The Complex Systems See-Change in Education. Journal of the Learning Sciences, 15(1), 3543. Golledge, R. G. (1995). Primitives of Spatial Knowledge. In Nyerges T.L., D.M. Mark, R. Laurini, & M.J. Egenhofer (Eds.), Cognitive Aspects of Human-Computer Interaction for Geographic Information Systems (pp. 29-44), Dordrecht: Kluwer Academic Publishers. Hmelo-Silver, C. E., Marathe, S., & Liu, L. (2007). Fish Swim, Rocks Sit, and Lungs Breathe: Expert-Novice Understanding of Complex Systems. Journal of the Learning Sciences, 16(3), 307 - 331. Janelle, D. G. and M. F. Goodchild (2011). Concepts, Principles, Tools, and Challenges in Spatially Integrated Social Science. In Nyerges, T.L., H. Couclelis, and R. McMaster (Eds.) The Sage Handbook of GIS & Society. Sage Publications. 27-45. Jo, I. & Bednarz, S. W. (2009). Evaluating Geography Textbook Questions from a Spatial Perspective: Using Concepts of Space, Tools of Representation, and Cognitive Processes to Evaluate Spatiality. Journal of Geography, 108(1), 4-13 Kaufman, M. M. (2004). Using Spatial-Temporal Primitives to Improve Geographic Skills for Preservice Teachers. Journal of Geography, 103(4), 171-181 Liu, L., Marathe, S., & Hmelo-Silver, C. E. (2005). Function before form: An alternative approach to learning about complex systems. Paper presented at AERA 2005: The 2005 Annual Meeting of the American Educational Research Association. Marsh, M., Golledge, R., & Battersby, S. E. (2007). Geospatial Concept Understanding and Recognition in G6 College Students: A Preliminary Argument for Minimal GIS. Annals of the Association of American Geographers, 97(4), 696-712. Munneke, L., Andriessen, J., Kanselaar, G., & Kirschner, P. (2007). Supporting interactive argumentation: Influence of representational tools on discussing a wicked problem. Computers in Human Behavior 23, 1072-1088. National Academy of Education (2008). Science and Mathematics Education: Education Policy White Paper. Washington, D.C.: National Academies Press. National Research Council (NRC) (2006). Learning to Think Spatially: GIS as a Support System in the K-12 Curriculum. Washington, DC: The National Academies Press. Rittel, H. & Webber, M. (1973). Dilemmas in a General Theory of Planning. Policy Sciences, 4, 155-169. Schilling, J., Logan, J. (2008). Greening the rust belt: A green infrastructure model for right sizing America's shrinking cities. Journal of the American Planning Association, 74, 451-466. Simon, H. A. (1973). The structure of ill-structured problems. Artificial Intelligence, 4(3), 181-201. Simon, H. A. (1978). Information processing theory of human problem solving. In W.K. Estes (Ed.), Handbook of Learning and Cognitive Processes (Vol 5, pp. 271-295). Hillsdale: Erlbaum. Sinnott, J.D. (1989). A model for solution of ill-structured problems: Implications for everyday and abstract problem solving. In J.D. Sinnott (Ed.), Everyday problem solving: Theory and applications (pp. 72-99). New York: Praeger. Songer, N. (2006). BioKIDS: An animated conversation on the development of curricular activity structures in inquiry science. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 355371). New York: Cambridge University Press. Strauss, A. & Corbin. (2008). Basics of qualitative research: techniques and procedures for developing grounded theory. Los Angeles: Sage. Wilensky, U., & Resnick, M. (1999). Thinking in levels: A dynamic systems approach to making sense of the world. Journal of Science Education and Technology, 8, 3-19. Zellner, M. & Campbell, S. D. (2011). New Tools for Deep-Rooted Problems: Using Complex Systems to Decode Wicked Problems. Paper presented at ACSP: The Association of Collegiate Schools of Planning Conference, Salt Lake City, UT. Zellner, M., Massey, D., Cotner, L., Minor, E., Gonzalez-Meler, M. (in progress). How Effective Is Green Infrastructure for Stormwater Management? Landscape and Urban Planning.
Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant No. 1020065. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
ISLS
16
ICLS2012
Semiosis Process in Instructional Practice

G rard Sensevy, Dominique Forest, Brittany Institute of Education, University of Western Brittany, Rennes, France Email: gerard.sensevy@bretagne.iufm.fr, dominique.forest@bretagne.iufm.fr Abstract: In this paper, we aim to show how the Joint Action Theory in Didactics (JATD) may contribute to construct a science of instructional practice. From this perspective, we propose the two concepts of students twofold semiosis and teachers equilibration work, which allow us to understand the instructional practice and the learning process in the same conceptualization. We illustrate these concepts and our video research methodology within two case studies. The first case study concerns tactile reading with a visually impaired student at primary school. The second case study refers to a learning situation proposed by Brousseau (2004), the Treasure Game, which one can see as a fundamental situation for the representational process. By contrasting the two cases, we attempt to identify two different kinds of equilibration works. We show how these two different equilibration works involve different ways of articulating the twofold semiosis in the didactic transactions.
Introduction
In this paper, we address two major issues. First, we concentrate our analysis on practice, and particularly on the instructional practice. Sharing Koschmanns question How do we begin to construct a real science of instructional practice? (Koschmann, 2011, p. 6), we try to provide some elements to contribute to this construction. From that perspective, we sketch a theory, the Joint Action Theory in Didactics (Sensevy & Mercier, 2007; Sensevy, 2011a; 2011b; 2011c), stemmed from French Didactics (Brousseau, 1997; Chevallard, 2007; Sensevy et al., 2005; Laborde et al., 2005 ; Sensevy et al., 2008), a theory which aims at linking teaching and learning by postulating that one cannot understand learning practices without understanding the related teaching practices. In this theoretical frame, didactic has the general meaning of relating to the teaching/learning process concerning a specific knowledge domain. The second issue of our paper concerns the way the teaching/learning relationship can be seen as a semiosis process - that we define, in Peirces sense (Peirce, 1998), as the way of producing and deciphering signs - in which the teachers action plays a prominent role. Within the JATD theoretical frame, we argue that learning needs teaching, and we postulate that the teaching learning process is above all a communicative process. In order to learn, a student needs to understand the learning situation in a right way, i.e. in the same way as the teacher does. Under certain conditions, the teacher and the student have to play the same game. How can the teacher and the students acquire a common understanding of a given situation? We argue that students have to become able to see the signs in/of the situation in the same way the teacher does. They have to build an accurate seeing-as (Wittgenstein, 1997) in a specific thought style (Fleck, 1979).
The theoretical approach: the semiosis process in the joint action theory
The JATD is based on the notions of didactic game and learning game. The didactic game (Sensevy et Mercier, 2007; Sensevy, 2011a) describes the fundamental structure of the teaching/learning process. We consider the didactic game as a game in which one player (the teacher) wins if and only if the other player (the student) wins. The teacher cannot make the winning moves (i.e. cannot learn) in place of the student, even though she has to enact a specific guidance (Kirschner et al., 2006) to orient the students. All that he or she can do is play in an indirect way in order to enable the student to learn. In this regard, the student has to act in a corresponding way, and enact a first-hand relationship to the piece of knowledge involved. We have described (Sensevy, 2011b) this fundamental constraint by arguing that the teacher has to be reticent the teacher must not tell the student all that he or she knows in order to enable the student to learn on her own a necessity we term the proprio motu clause. This fundamental property of the didactic game implies some other necessities that we have no space here to mention, but the main point is that the teachers reticence and the proprio motu clause strongly shape the kind of semiosis inherent to the didactic game. A learning game, which refers to the teachers game on the students game, can be described by relying on two elements. The first element is the didactic contract, an expectations system between the student and the teacher, which gathers the current habits (rules, norms, capacities) relating to the knowledge at stake, and which constitutes the current students strategic system. The didactic contract can be seen as a system of social and socio-epistemic norms (Yackel & Cobb, 1996; McClain & Cobb, 2001) relating to the didactic institution in which the didactic process unfolds. The second related element is the didactic milieu, which refers to the virtual
ISLS
17
ICLS2012
knowledge to be acquired, as it is displayed in the learning game. This knowledge is embedded in the material and symbolic objects of the milieu, to the extent that this milieu enacts a problem. A main feature of the didactic transactions lies in the process of reaching a kind of equilibrium between contract and milieu in the learning games. We called this process didactic equilibration. The didactic equilibration can be seen as a semiosis process, in which, by relying on the didactic contract, the students become able to solve the problem, i.e. to interpret the milieu in a relevant way. In the semiosis process, the student has to produce a first semiosis, the deciphering of the signs of the milieu. These signs are non-intentional signs. By non-intentional, we mean that the signs are not directly produced within in intentional process. But this first kind of semiosis is not produced independently of the social structure in which the didactic process unfolds. It is enacted against the background of the previously taught knowledge as it has been structured in the preceding didactic joint action, the didactic contract background. This didactic contract may be seen as an epistemic system, which enables the students to cope with the milieu of the problem at stake, and a transactional system, in that the teacher may orient the students action in the milieu by relying on the content of the epistemic system provided by the contract. This transactional process refers to what we call the second semiosis. In the first semiosis, the student has to decipher the non-intentional signs of the milieu. In the second semiosis, she has to decipher the intentional signs that the teacher provides in order to orient her in the milieu. In the following, we document this twofold semiosis and the related teachers equilibration work in two empirical fields. The first case study refers to a situation of tactile reading, involving visually impaired students. The second case study elaborates on a didactic situation at Kindergarten proposed by Brousseau (2004).
The methodological approach: some elements

Our theoretical frame rests upon the analysis of the joint action between the teacher and the students. As we have seen, this theoretical viewpoint gives a major importance to the production and the deciphering of signs, in the semiosis process, which is partly a semiosis of others. The teacher has to identify the signs provided by the students, and, in what we have called the second semiosis, students have to recognize the teachers intentional signs. Moreover, the knowledge as stake can be conceived of a symbolic system in itself, that the students have to acknowledge and practice. For these reasons, we ground our inquiries in the video recording of teaching/learning practices, and we include our approach (Forest & Mercier, 2011) in the methodological paradigm of the Video Research (Goldman et al., 2007). As we have argued elsewhere (Sensevy, 2011a; Tiberghien & Sensevy, in press), video recordings enable to keep the analogical character of the situations with its specificity and its density of information (Dretske, 1981). Above all, as Goldman & McDermott (2007) state, it makes communication visible, and allows accounting for the embodied instructional communication. Our methodological endeavor consists of building what we call hybrid texts-images systems, in which is fostered a reciprocal annotation process between texts and images.
Case study 1: Visually impaired students

General description of the situation
The first situation we present involves tactile reading. Visually impaired students (5 years old), are trained by a special teacher to read a tactile book, which name is Little point draws a man, and which is structured as presented below.
Figure 1: 3/9 pages showing the structure of the tactile book little point draws a man. The teacher decided to give the book to the visually impaired student without reading it to her. The student had to recognize the shapes without any information. She first recognized the head of the man and the studied episode take place when the student has to identify what she first called the sweater of the man.
Solving the problem of representation: an oval for the belly

The student has just turned the page and she is exploring the tactile representation, i.e. a new piece of fabric below the head that she has previously identified.
ISLS
18
ICLS2012
General location phase: the student Specific exploration phase (1): the uses the left index to locate the student enacts a vertical exploration ear element, while the right hand of the new element. recognizes the whole element head. In the same time the thumb starts to explore the new element (the belly). Table 1: first tactile exploration, in a silent way
Specific exploration phase (2): hemispheric dissociative exploration. In particular, she focuses on the hinge between the head element and the new element.
Specific exploration phase (3): the Specific exploration phase (4): the student focuses on the new element. student verifies the specific location She feels the texture of the new of the belly element. T: What can it represent, according to you? S: The belly T: what is the shape of the belly? + can you tell me the shape? Table 2: exploration with joint verbal action S (Student): The sweater+ T (Teacher): This is the sweater? S: Ups! no
Specific exploration phase (5): the students explores the new element, in search of new differential properties (other than texture) S: round, mm T: Its a round? I do not totally agree S: an oval
In this episode, from the students point of view, the twofold semiosis means a milieu semiosis (S M), the first semiosis, and a teacher semiosis (ST ), the second one. The learning game at stake is a form recognition game, i.e. a game where the student must identify the forms she touches on the book. The epistemic elements of the didactic contract on which the student relies are the prototypic body image, her textures knowledge, a common geometric knowledge, and the currently taught geometric knowledge. The student explores the didactic milieu constituted by the different elements of the man. The fact that the student is visually impaired put forward the necessity she meets to explore the didactic milieu, as the annotated photograms show. The relationship to the milieu is a necessary condition of her learning. Nevertheless, as we will see, the didactic contract plays a prominent role in this episode. We can describe the didactic transactions in this excerpt according to four phases. Phase 1: (SM1/T1). The semiosis is oriented to the milieu (First Semiosis), following the teachers general indication. Its the meaning of the symbolic notation S M1/T1: a semiosis of the milieu (first letter M), following the teachers orientation (second letter T). The student explores the specific location of the new element in the body image and relies on its texture. She finds a first answer (The sweater), by focusing on one may call a non-intentional sign (the fabric on the tactile book) Phase 2: Then the teacher asks a question (T: This is the sweater? What does it can represent, according to you?) (ST1). Here the students semiosis is of the second kind. The student deciphers the teachers signs, against the background of the didactic contract. One can make the hypothesis that she has understood that the game is about the parts of the body, she identifies an intentional sign, and she answers by guessing the likely teachers expectations (The Belly) Phase 3: Then the teacher asks a new question (T: What is the shape of the belly?)(ST2). Again, the students semiosis is of the second kind. The teachers word shape probably allows her to identify the geometric didactic contract, which rule this transaction. She understands she has to focus on the geometric property of the belly, and she answers with this respect (Round).
Tactile reading: the twofold semiosis process and the teachers equilibration work
ISLS
19
ICLS2012
Phase 4: Now the teacher tries to obtain a better geometric response from the student (T: Its a round? I do not totally agree)(S T2/M2). The students semiosis continues to unfold in the deciphering of the teachers signs (her utterances). She explores the form of the belly and produces a new answer (Oval). This case study enables us to make a clear distinction between the two forms of semiosis. At the beginning of the episode, the student really faces the milieu, and she has to decipher the tactile signs by relying on her knowledge of the body image and the texture she recognizes. But after this first kind of semiosis, she has to take into account the teachers questions and assertions. In doing that, she orients her deciphering towards the expectations of the teacher, that she interprets against the background of the didactic contract. From the teachers action viewpoint, one can recognize an equilibration work in which the milieu is an auxiliary to the contract. Everything occurs as if the main intention of the teacher was to elicit some words (belly, oval) that the student already knows. The milieu, and the exploratory work of the student in this milieu, has the main function of producing the right predefined seeing-as: seeing the sweater as a body part; seeing the round as an oval.
Case study 2: The Treasures Game

General description of the situation
Brousseau and his team designed the treasures game for kindergarten students at the beginning of the nineteen eighty. This learning sequence takes place over several months and it aims to have students build a system of graphical representations. Brousseau (2004) has presented a strong theorization of this research design, which he considered as a fundamental situation for the notion of a representation. The situation was recently reimplemented in some classes in Switzerland (Leutenegger & Ligozat, 2009) and in France. It consists of producing a list of objects to be remembered and communicated. The game is organized in four stages, the rules changing as the game progresses. In the first stage, the teacher presents two or three small new objects each day. These objects belong to the world of children and are passed from hand to hand. The teacher asks the students to name them, and she puts them in a box (the treasure chest). Then she asks: what's in my box? A student then calls out the name of an object, the teacher pulls the object out of the box and places it in full view. Then she asks Is my box empty? and if not, the game continues, and so on. Every two or three days, two or three new objects are presented by the teacher and are added to the previous ones. At the end of one month, the whole class can empty a box of 40 objects (thus being able to collectively memorize 40 objects). These objects can seem disparate but they were carefully chosen. One can look at some examples of objects on the figure 1, including the lens and the pan which will support the problem of representation in the episode we will study in this section.
Figure 2: some examples from the reference collection of 40 objects. This stage is played out with the entire group of students and focuses on the creation of a verbal system of reference for the objects inside the treasure chest. The stage 2 proceeds in two memory games. Firstly the students have to play an individual memory game, as each pupil must individually remember 3 objects that are daily hidden in the treasure chest. But when all the pupils have understood this memory game with 3 objects, the teacher introduces an important change: she asks each pupil to remember ten (10) hidden objects. This informational leap makes it impossible to win only with an internal memory. The jump from 3 to 10 objects is intended to oblige pupils to produce individual graphical lists, in order to remember the specific objects and win the game. The stage 3 aimed at communicating with lists. This new part of the game takes place in small groups of 5 pupils. 4 object representations are written by a pupil, who is the designer, and these objects are hidden in the box. The other 4 pupils have to read the graphic representations of the designer, and name each object to get it out of the box. This third stage gives pupils opportunities to debate, firstly in their small group, and secondly in the whole classroom during the 4th stage of the game. In stage 4, the pupils are building a common code for the whole class. In the studied carrying on of this engineering, stage 3 and 4 are intertwined. Some changes in stage 3 prepare the debates in stage 4.
Solving the problem of pan's representation

The small episode we study takes place in stage 3, when pupils have to read the representation of the designer. On this episode, the teacher gave the designer four quite different objects; one of them is the pan (cf. fig.2, above). When students have to read the designer's
ISLS
20
ICLS2012
production (cf. fig.3, shown against), they fail to recognize the pan (the designed object, fig. 2, above), confusing it with the lens, another object of the collection of reference, which have a similar form (cf. fig.2, above). In the short moment of joint didactic action we present, the teacher has first Figure 3: the representation shared with the students the failure of the designing and recognizing the pan. of the pan by the designer She then asks students to find some ideas to draw and recognize the pan better. student Like in the previous example, our description and analysis use a hybrid system with photograms, utterances and commentaries on twofold semiosis. The English discourse is a translation from French. The commentaries are informed by proxemic analysis (Hall, 1966; Forest, 2009) of non-verbal behaviors. Table 3: giving the problem Iman: we have to make a small line, we have to do this... The teacher shows the object in her left hand, and its representation (cf. fig.3 above) by pointing it with her finger, in her right hand, and asks the students for a comparison. A student, Iman, points a part of the object, which is characteristic of it (a hole on the handle). In that way, the student links a sensory feature to a specific characteristic of the object, which is potentially relevant to solve the problem. One can notice the direction of the student's eyes as an indication of a joint semiosis between the teacher and the student. At this moment, the local learning game consists in representing in a better way the pan, in order to distinguish it from the lens. By showing the object and its representation, the teacher gives the student the specific problem of the distinction lens-pan in the graphic representation. According to the teachers expectations, the student (Iman) acts in the milieu, and suggests by pointing on the object that it would be accurate to represent the hole (we have to do this). We can argue that the student plays a relevant move in the representation game by deciphering the signs of the milieu (semiosis 1, she indicates a possible characteristic to design). Even though she responds to the teachers expectation (semiosis 2), she positions herself as a potential provider of relevant design features. The semiosis is of the form (SM/T ). Table 4: another strategy is proposed Iman: a big ring, and for the lens a small ring The teacher looks herself at the representation drawn by the designer (cf. fig.3 above). She points it by her left hand finger. Iman writes with her finger on the table by evoking a possible representation of the difference between the pan and the lens (big ring-small ring). By showing the representation to the students, and looking herself at it, the teacher drives student's attention onto the design. Iman tries to draw on the table, and a generic sign emerges (a ring). But the representation big-small is not a distinctive sign (which may allow to recognize the object in itself) and it will not really work if only one object (the lens or the pan) is to be recognized among others: in this case only the pan is represented, and the comparisons of representations (between the lens and the pan) cannot work. Interpreting in a general way the teachers expectations about the drawings, the student produces a kind of strategy which can be useful, but which is not a winning one in this specific learning game. Even if it is relatively ineffective, the semiosis is always of the form (SM/T ), in that it enables an explorative and imaginative inquiry into the milieu, not oriented to pre-conceived meanings. Table 5: the less relevant strategy starts spreading Another student: ...a small ring, and a stick As in table 3, the teacher shows the object and its representation. Again, she asks the students for a comparison. Another student writes with his finger on the table, as Iman did. He repeats Imans proposition (a ring), adding the possibility of drawing a stick (for the handle). Its worth noticing that the attention of the students focuses on Imans action. The teacher attention is oriented to the last student who had made a proposal.
ISLS
21
ICLS2012
The teacher goes on showing the object and its representation in order to have the students comparing them, but Iman's proposal is taken up and meliorated by others. The new proposal (a small ring and a stick) possibly solves the problem of a generic representation for the pan and the lens. From this point of view, it is relevant. But it doesn't work as a distinctive representation between the two objects. So it is less relevant in this representation game. The enacted semiosis is of the same kind as the previously analyzed (SM/T ). The students are oriented by the strong teachers guidance, but in an open way, which allows them to propose different representations. Table 6: the teacher do precise the problem Showing the pan and its Staying in her posture in front of Moving on her right side, near the representation, the teacher reminds the students, the teacher moves the designer, she formulates the the students the current difficulty object and its representation on her problem for the pan. (we had confused with the lens). left side. When talking, the teacher directs One can notice the central posture In the same time, she reformulates her eyes toward different children, of the teacher, and the convergence the problem, relating to the lens (so finishing toward the designer. of the children's eyes. that one guesses every time that it's the lens)
The teacher: It's true that we had We should try to find a drawing for ...and a design for the pan, so that confused with the lens, because it the lens, so that one guesses every one guesses every time that it's the was like the drawing of the pan. time that it's the lens pan One can see, at this moment, how the teacher works in the semiosis 2 (the students deciphering of teachers signs) to take on the issues of the learning game (we had confused... we should try to find a design... so that one guesses every time). However, the teacher remains reticent. To support her reticence, she uses the resources of the proxemics, joining gesture to her talk: first, she shows the object and its representation like in the photogram 1, showing them to all the students; second, she evokes the lens (which is absent from the package) putting the pan and its representation on the left side; third, she moves on the other side to finally show the representation which failed, to the designer. The learning game is strongly restated at this time (a drawing for the lens/the pan, so that one guesses every time that it's the lens/the pan): the students have to produce by themselves a strategy to design the pan, while taking into account its similarity with the lens. One can analyze this episode by relying on the contract-milieu dialectics within the learning game. In order to explore the milieu (to inquire into the different ways of representing the objects) in a relevant way, the students have to play the right game. The students have to become able to differentiate some very similar objects (from the students representational viewpoint), as the pan and the lens. In this perspective, the teachers monitoring consists of reminding these contract rules, which embed the end-in-view of their game. Table 7: Joint action provides a new relevant move Tilio: Ah yes! so one can make a small hole here. The teacher continues showing the object and its representation. A student (Tilio) points a part of the pan, the same that Iman had pointed on table 3. We can perceive the convergence of students gaze on Tilios gesture. According to the teacher expectation, Tilio acts directly into the milieu, by pointing the hole on the object and by naming it (a small hole for the handle). In a more theoretical language, we can say that Tilio deciphered the teachers expectation (semiosis 2) about the playing of the right game. He then plays a relevant move in the representation game, by deciphering a sign of the milieu (semiosis 1), and categorizing it in an efficient way. In effect, if the representation of the pan includes a small hole on the handle, it will allow to
ISLS 22
ICLS2012
differentiate the pan from the lens, even though if the general form of the two representations is similar (a small ring and a stick). Tilios move will be followed by other students and his proposal will be institutionalized. Once again, the semiosis process is of the form (SM/T ). The students produce imaginative and accurate ways of seeing the milieu, but one has to keep in mind that such performance is made possible by the enduring teachers work.
The Treasure Game: the twofold semiosis process and the teachers equilibration work
According to us, the above analysis is in line with a transactional perspective (Clancey, 1997, 2001; Koschmann, 2001, 2011), which looks at the teaching-learning process from a Deweyan viewpoint. In that way, we agree with Koschmanns contention according to which Deweyan inquiry, when successfully carried out not only effects a change in the problem solver (what psychologists treat as learning) but also leads to a reconstruction of the problematic situation that led to the inquiry in the first place (Koschmann, 2011, p. 12). In the Treasure Game situation we have presented in this paper, one may appreciate an actual form of inquiry endorsed by young students (5 years old). The theoretical categories of the Joint Action Theory in Didactics enable us to understand what could be the sources of such an inquiry. Grounded in a representation game, this situation entitles the students to propose signs as means of representing objects. But this game does obey to strict rules, as we have seen above. If a sign does not provide the collective with a successful way to recognize an object, it has to be rejected, not by the teacher, not even by the students, but in the unfolding of the actual game (at is was the case at the beginning of the studied episode). Its worth noting that such a situation does not achieve its goal without a specific teachers equilibration work. One can imagine two opposed counterfactuals in order to figure out this point. In the first counterfactual, one can think about a teacher who would abandon her reticence, and try to directly influence the students, for example by immediately rejecting the ineffective signs, or by proposing one particular good sign to the students. In the second counterfactual, one can envisage a teacher who would let the students explore the milieu, for example without reminding them the overarching goal of the learning game. In contrast with these both counterfactuals, in the actual situation, the teacher enacts a third way of doing. She provides the students with a system of meanings, embedded in the signs she uses, while keeping a strong reticence about the representational knowledge in question. In her equilibration work, the contract can be seen as an auxiliary to the milieu, which means that in the twofold semiosis that the students enact, the deciphering of the teachers signs gains its sense firstly as a way of carrying out the inquiry (Dewey, 1938/2008).
Discussion: an outline of a comparative analysis

The comparison of practices in the two case studies could provide an exemplar, in Kuhns sense (Kuhn, 1979), of two possible relationships between the students deciphering of signs and the structure of the instructional process. In effect, in the first case study (Tactile Reading by a visually impaired student), the twofold semiosis is mainly of the form (ST/M), even (ST ). It is to say that the student, when she explores the milieu, does it by enabling herself to recognize the didactic contract meanings that stand as a background against which she acts. Her agency is constrained by the predefined meanings that the teacher has in mind (i.e. belly rather than sweater, or oval rather than round). In this perspective, the teachers equilibration work is contractdriven, in that her main intention consists of making the student more familiar with a specific seeing-as (see the round as an oval) in a particular though style (the geometric though style), that has been previously studied. By contrast, in the second case study (The Treasure Game), the twofold semiosis is mainly of the form (SM/T ). The deciphering of the teachers signs, in the second semiosis, is never an end in itself. The main function of this second semiosis consists of making the student able to carry on the inquiry process. The didactic contract stands as a background for the students activity, in that it orients in a decisive way the students action (the weakness of the contract would be very detrimental to the joint action), but it is not sufficient to reach the learning games goal. The more important feature of the students activity lies in their inquiry process into the milieu (i.e. becoming able to invent effective representations), and rests on the way they endorse the first semiosis. In that way, the structure of the learning game fosters students agency. In this perspective, the teachers equilibration work is milieu-driven, in that her main intention consists of making the students able to enter the representational process by directly understanding the nature and properties of signs through the consequences of their decisions.
References
Brousseau, G. (1997). The theory of didactic situations in mathematics. Dordrecht: Kluwer. Brousseau, G. (2004). Les repr sentations: tude en th orie des situations didactiques. Revue des sciences de lducation, 2(30), 241-277. Chevallard, Y. (2007). Readjusting didactics to a changing epistemology. European Educational Research Journal, 6(2), 131134.
ISLS
23
ICLS2012
Clancey, W. J. (1997). Situated Cognition: On Human Knowledge and Computer Representations. Cambridge: Cambridge University Press. Clancey, W. J. (2011). A Transactional Perspective on the Practice-Based Science of Teaching and Learning. In T. D. Koschmann (Ed.), Theories of Learning and Studies of Instructional Practice (pp. 247-278). New York: Springer. Dewey, J. (2008). John Dewey The Later Works, 1925-1953: 1938: Logic: The Theory of Inquiry. Chicago: Southern Illinois University Press. Fleck, L. (1981). Genesis and development of a scientific fact. Chicago: University of Chicago press. Forest, D. (2009). Agencements didactiques, pour une analyse fonctionnelle du comportement non-verbal du professeur. Revue fran aise de p dagogie, 165, 77-89. Forest, D., & Mercier, A. (2011). Classroom Video Data and Ressources for Teaching. In G. Gueudet, B. Pepin & L. Trouche (Eds.), From Text to "Lived" Ressources : Mathematics Curriculum Materials and Teachers Development. Berlin : Springer. Goldman, R., Pea, R., Barron, B., & Derry, S. J. (2007). Video Research in the Learning Sciences. Mahwah, NJ: Lawrence Erlbaum Associates Inc. Goldman, S., & McDermott, R. (2007). Staying the course with video analysis. In R. Goldman & R. Pea & B. Barron & S. Derry (Eds.), Video research in the learning sciences (pp. 101-113). Mahwah New Jersey: Lawrence Erlbaum Associates. Hall, E. (1966). The Hidden Dimension. New York: Anchor Books Doubleday Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and InquiryBased Teaching. Educational Psychologist, 41, 75-86. doi:10.1207/s15326985ep4102_1 Koschmann, T. D. (2001). A third metaphor for learning: Toward a Deweyan form of transactional inquiry. In S. M. Carver & D. Klahr (Eds.), Cognition and Instruction: 25 Years of Progress (pp. 402-415). Mahwah, NJ: Lawrence Erlbaum Associates Inc. Koschmann, T. D. (2011). Theories of Learning and Studies of Instructional Practice. New York: Springer. Kuhn, T. (1979). The Essential Tension. Chicago: University of Chicago Press. Laborde, C., Perrin-Glorian, M.-J., & Sierpinska, A. (2005). Beyond the Apparent Banality of the Mathematics Classroom. New-York: Springer. Leutenegger, F & Ligozat, F. (2009). The Treasure Game : grasping the premises of the subject matter norms in pre-school classes. Communication, Network 27, European Congress in Educational Research (ECER) "Theory and evidence in European educational research?", [28-30 september] University of Vienna, Austria. McClain, K., & Cobb, P. (2001). An Analysis of Development of Sociomathematical Norms in One First-Grade Classroom. Journal for Research in Mathematics Education, 32(3), 236-266. Mauss, M. (2006). Techniques, Technology And Civilization. London: Berghahn Books. Peirce, C. S. (1998). The essential Peirce: (1893-1913). Bloomington: Indiana University Press. Sensevy, G., Mercier, A., Schubauer-Leoni, M-L., Ligozat, F. Perrot, G (2005). An attempt to model the teachers action in mathematics. Educational Studies in mathematics, 59(1), 153-181. Sensevy, G. & Mercier, A. (2007). Agir ensemble. Rennes : Presses Universitaires de Rennes. Sensevy, G., Tiberghien, A., Santini, J., Laub S. & Griggs, P. (2008). Modelling, an epistemological approach : , cases studies and implications for science teaching. Science Education, 92(3), 424-446. Sensevy, G. (2011a). Le sens du savoir. lments pour une thorie de laction conjointe en didactique. Bruxelles : De Boeck. Sensevy, G. (2011b). Overcoming fragmentation : Towards a joint action theory in didactics. In B. Hudson & M. Meyer (Eds.), Beyond Fragmentation : Didactics, Learning and Teaching in Europe (pp. 60-76). Opladen and Farmington Hills : Barbara Budrich. Sensevy, G. (2011c). Patterns of Didactic Intentions. Thought Collective and Documentation Work. In G. Gueudet, B. Pepin, & L. Trouche (Eds.), From Text to Lived Resources : Mathematics Curriculum Materials and Teacher Development (pp. 43-57). New-York : Springer. Tiberghien, A., & Sensevy, G. (in press). Video studies : Time and duration in the teaching-learning processes In J. Dillon & D. Jorde (Eds.), Handbook "The World of Science Education volume 4. Rotterdam/Boston/Taipei : Sense Publishers. Yackel, E., & Cobb, P. (1996). Sociomathematical Norms, Argumentation, and Autonomy in Mathematics. Journal for Research in Mathematics Education, 27(4), 458-477. Wittgenstein, L. (1997). Philosophical Investigations. Oxford: Blackwell.
ISLS
24
ICLS2012
Agent-based computer models for learning about climate change and process analysis techniques
Nick Kelly, Michael J Jacobson, Lina Markauskaite, Vilaythong Southavilay The University of Sydney, Australia Email: nick.kelly.mail@gmail.edu.au, michael.jacobson@sydney.edu.au, lina.markauskaite@sydney.edu.au, vstoto@it.usyd.edu.au Abstract: This paper describes a design-based research project that investigates the learning of scientific knowledge about climate change through computational models. The design experiment used two NetLogo models and problem-based learning materials developed in partnership with a high school science teacher. In the study, three classes of year nine science students were divided into two groups based upon different levels of structure that was provided during learning activities with the models. The results indicate that there was significant learning of concepts about greenhouse gases and the carbon cycle through engagement with the models. We also describe the process analysis techniques being developed to analyze the log files of the interactions the students had with the computer models. This paper presents results from the first iteration of a design-based research project that is exploring how students may learn scientific knowledge about climate systems using computer models as part of a four year federally funded project in Australia. There are three main research areas of this project. The first area is investigating if different patterns for the sequencing and structure of pedagogical guidance during learning activities can enhance learning with computer models. Whereas the notion of strict discovery-learning has been largely discredited (Mayer, 2004) there is recent evidence that a sequence of low-structured activities followed by high structured activities can result in enhanced learning outcomes when compared to purely high-structured activities only (Kapur, 2008, 2010; Schwartz & Bransford, 1998; VanLehn, Siler, & Murray, 2003). Second, there are increasing calls for students to learn important scientific knowledge about systems, including topics related to complex climate systems and global warming (ACARA, 2011; National Research Council, 2012). An important recent approach for helping students to successfully learn more standard science education topics involves model-based learning (MBL) (Clement, 2000; Zhang, Liu, & Krajcik, 2005). In general, MBL approaches engage the learners with computer simulation and visualization systems and learning the targeted science concepts interactively via problem solving. There has also been research involving the use of agent-based models (ABM), which computationally represent phenomena as a number of agents or elements that each has particular rules they follow, and for which the apparent complexity of the system being modeled emerges from the interactions of the agents in the systems (Railsback & Grimm, 2011). A number of recent studies have documented significant learning with ABMs, especially for learning about various types of complex physical and biological systems (Goldstone & Wilensky, 2008; Jacobson, Kapur, So, & Lee, 2011; Wilensky & Reisman, 2006). In this study, we investigate how ABMs can help students learn important concepts about climate systems. Finally, when students work with computer models, a large stream of fine-grained process data can be collected. Recently, researchers have begun to explore various process analysis techniques for analyzing data of human-computer interactions, such as data mining, process mining, and learning analytics (Reimann, 2009; Romero, Ventura, Pechenizkiy, Ryan, & Baker, 2010; Trka, Pechenizkiy, & van der Aalst, 2011; van der Aalst, 2011). These techniques have demonstrated potential for discovering student learning patterns and predicting unproductive behaviors where early intervention may be possible. However, research is needed to investigate how these techniques might be used to assist teachers and students during MBL. Therefore, the third area of our research is to provide a proof of concept for specific data mining techniques that might be useful for analysis of MBL. This study involved MBL of scientific knowledge about climate change to explore these three areas. In particular, the research design involved different sequences of learning activities as students worked with agent-based computer models of the carbon cycle and the greenhouse effect. In addition to the pre-post test assessments of conceptual understanding of climate change knowledge, our analysis of students learning makes use of process discovery techniques involving log file data of student dyads using the computer models.
Method
A classroom experiment was devised in which students from three year nine science classes were randomly assigned to two treatment groups: Challenge and Guided Learning (CGL) (i.e., low-to-high sequence of pedagogical structure) and Guided Learning (GL) (i.e., high-to-high sequence of pedagogical structure). Both
ISLS
25
ICLS2012
groups experienced two classroom periods of learning with two computational models relating to climate science (described below). For each of the two models, three problem-based activities were prepared. The two experimental groups (CGL and GL) differed only in the structure provided in the first of two activities with each model, so that the second activity and assessment were identical across both groups. For the CGL group, this involved the posing of an initial challenging question with no instructional support, e.g., What is the difference between the behavior of visible light (yellow rays) and infrared light (red rays)? The GL group had the same question but was provided additional instructional support in the form of small guiding tasks that structured their exploration of the models in a way intended to help them answer the question successfully. This experimental design was used in previous studies in which significant learning outcomes were found (Kapur, 2006; Pathak, Kim, Jacobson, & Zhang, 2009). Students individually completed a pre-test and a post-test in which their knowledge about climate science was assessed, and then all learning activities with the computer models were carried out in dyads.
Figure 1: A NetLogo model of the greenhouse effect The agent-based models of the carbon cycle and the greenhouse effect were developed with NetLogo (Wilensky, 1999). The models were co-created by the first author and a collaborating schoolteacher with a PhD in physics, and were then verified by an earth sciences graduate. The carbon cycle model represents a fixed number of carbon molecules within a closed system. Carbon molecules exchange at the boundaries between atmosphere, land, and ocean. Students can alter the rates of transfer at these boundaries as well as control the rate of release of deeply stored carbon into the atmosphere. Carbon moves in different regions as well as in different life forms, represented as trees (plants) and sheep (animals). The greenhouse effect model allows students to explore the interaction between the suns energy (i.e., electromagnetic radiation) and the carbon cycle. This model builds on the simpler carbon cycle model that the students use first. Students can release carbon dioxide into the atmosphere and observe the effects over time. The output shows atmospheric gases and surface temperatures. The model includes parameters for the quantity of the greenhouse gasses of CO2 and H2O and includes a parameter for the Earths albedo or reflectiveness. Rules are specified for the interaction between electromagnetic radiation from the sun and the Earths surface (reflection, absorption) as well as for heat released from the Earths surface (infrared radiation [IR]) and greenhouse gases (CO2 and H2O). The resulting complex system has the emergent property of global heat that changes based upon the proportions of solar radiation/greenhouse gases/reflection in the system. The simulation generates output that was calibrated to be an approximation of real data for carbon and temperature (Intergovernmental Panel on Climate Change, 2007). Students interact with the models by changing the parameters and observing the effects. For example, in the
ISLS
26
ICLS2012
model of the greenhouse effect shown in Figure 1, releasing CO2 into the atmosphere (by changing a slider, left side) leads to real-time visual feedback (in the visualization window, right side) as well as more CO2 molecules being present and an increased incidence of re-radiation of IR light leaving Earths surface. This is displayed in the graphs of temperature and carbon (lower left) that show an increase in the former and a shift in the equilibrium in the latter over time. The study was conducted in 2011 at an all girls high school in Australia, with three year nine classes (third year of secondary school) and a total of 90 students (30 students each). Students completed individual pre-tests and post-tests of their knowledge of climate science and complex systems. The pre- and post-tests consisted of 20 questionssix multiple choice and 14 open-ended short answer questionsthat were intended to assess both declarative and conceptual knowledge of climate science. Students were given 20 minutes to complete each test. During the experiment, students carried out three activities with each of the two models in pairs (45 dyads in total). Each pair was randomly assigned to one of two conditions: (a) CGL where dyads had no structure for the first activity of each model, and (b) GL where dyads had a structured task for the first activity with each model. Students were allocated 80 minutes for working with each of the two models and completing the given activities. Computer screen recordings were made and log files of NetLogo interactions were maintained for all dyads. Additional webcam audio and video recordings were made for six dyad groups (three in each of the treatment groups).
Results
This section presents the main results related to our three research areas from the first year of the project. First we discuss the pre- and post-test assessment results. Second, we illustrate the potential of process analysis techniques to detect student behavior patterns from the NetLogo log files in our analysis of one high achieving and one low achieving dyad from CGL group. Of the initial 90 students, one of the classes was used as a pilot to tune data collection processes, and therefore was not included in the final analysis. Of the remaining 60 students, only 33 completed all tasks due to absences and technical difficulties (e.g., computer failures resulting in missing data). Ten short answer questions related to scientific knowledge of climate change phenomena were coded using an adapted five-point knowledge integration rubric (Gerard, Spitulnik, & Linn, 2010). This coding scheme elucidates students capabilities to make connections between different elements of the phenomenon, thus was appropriate for investigating the depth of students understanding. One question was removed as we determined it had unclear language, leaving nine questions. Initially, two individuals each coded 60% of the results with a 20% overlap for reliability. Cohens Kappa on this initial 20% was 0.64. The coders then met to discuss discrepancies and coded the remaining results so that each had rated all of the responses. Following discussion of discrepancies, 100% agreement was reached. These scores were scaled to a maximum of 100 and Table 1 summarizes these results. Table 1: Pre- and posttest results N Challenge & Guided Learning Guided Learning All students 15 18 33 Mean 31.73 34.67 33.33 Pre-Test SD (15.81) (12.27) (13.84) Mean 41.07 38.89 39.88 Post-Test SD (14.38) (13.22) (13.58)
Overall, there was a significant increase in the mean post-test scores of all the students, F(1,31)=9.367, p=.005, 2 =.232. Repeated measures ANOVA on the summed short answer responses with a between-subjects factor of the CGL/GL group found no significant differences, F(1,31) = 1.532, p = .225, 2 =.047, although the CGL group did have a higher score on the post-test than the GL group. The problem-based activity and assessment questions the dyads worked on were similarly coded and scaled (Table 2). Repeated measures ANOVA on the activity and assessment responses with a between-subjects factor of the CGL/GL group found no significant group differences, F(1,12) = 0.300, p = .594, 2 =.024, although again the CGL group had a higher mean scores than the GL group on both of the assessment tasks for each models.
ISLS
27
ICLS2012
Table 2: Student scores for activity and assessment questions N (Dyads) Challenge & Guided Learning Guided Learning All dyads 6 8 14 Model 1 (Carbon Cycle) Mean SD 61.11 (16.39) 55.21 (9.90) 57.74 (12.85) Model 2 (Greenhouse Effect) Mean SD 38.89 (18.26) 37.50 (18.72) 38.10 (17.82)
Process Analysis
In this section we outline an approach that we are developing for analyzing the stream of data that is available from students engaged in learning activities with NetLogo agent-based models. The main goal of this early phase of our work is to explore possible ways that data mining techniques may provide insights into the students interactions with the NetLogo models. Educational data mining employs various algorithmic techniques on large amounts of institutional data records or student learning log files that are associated with online learning activities in order to discover student learning patterns (Romero & Ventura, 2006; Romero et al., 2010). This information then could be used to improve the design of software and learning materials or provide individualized feedback and support for learners during MBL activities. As a part of our initial analysis we are exploring several possible techniques to analyze students interaction patterns: Hidden Markov Model (HMM), process mining, and various statistical visualization techniques. In our initial analysis, we used the HMM constructing algorithm adapted from Jeong, Biswas, Johnson, & Howard, (2010). On the basis of sequences of student interactions with NetLogo models recorded in log files, the algorithm extracted HMMs consisting of hidden states that depict how students interact with the models. Student activities or actions are equivalent to the use of controls on the model interface: Setup button, Go button (that initiates Start and Stop of the simulation), Speed control, Fossil Fuel Use control, and Tracking CO2 Molecule control. The HMM shows the composition and the percentage of activities in each state as well as transitioning probabilities between these states. Students activities within individual states and likelihoods with which they move from one state to another together depict students overall behavior patterns. Each HMM is made up of a set of states and transitions: the behavior probabilities associated with each state and the transition probabilities between the states. For example, the HMM in Figure 2 shows that a high achieving dyad in State D (Fossil Fuel and Speed) explored the model by controlling the simulation in two distinct ways: by changing the amount of fossil fuel use 61% of the times (14 actions in total) and by changing modeling speed 39% of the time (9 actions in total). The link from one state to another and the percentage associated with the link indicate how likely it is that a certain state will be followed by another. For example, the HMM in Figure 2 shows the probabilities that the dyad would transition from State A (in which the they set up the simulation) to states B, C, and D. Specifically, the HMM for the high achieving dyad was composed from five states (Figure 2). Three simple states were associated with model control: Setup (State A), Stop (State C), and Stop Tracing of a CO2 molecule action (State E). Two composite states were associated with the model exploration: (a) start simulation and initiate tracing of a CO2 molecule actions (State B), and (b) control the fossil-fuel use and speed control (State D). The HMM shows that the high achieving pair often moved from the initial model setup (State A) to the control of the main model parameters and speed (State D), with the transition probability as high as 70%. The transition probability between the model manipulation (State D) and Start (State B) was also relatively high (48%). In contrast, the likelihoods of transition between Setup (State A) and Start (State B) and Setup (State A) and Stop (State C) were as low as 20% and 10% respectively. This suggests that after resetting the model (State A), the dyad often initially configured the main model parameters and only then started exploration of the model by switching on CO2 tracking or pressing Start (State B). The HMM also indicated that these students often explored the model by running and pausing it, with the transition probabilities between Start (State B) and Stop (State C), and vice versa, ranging between 47% and 50%. The HMM for the low achieving pair indicates that these students interacted with the model differently than did the high achieving dyad. The model was composed of four states only (Figure 3). Two simple states and one composite state were similar to the states in the HMM for high achieving students: Setup (Sate A); Stop (State C), and model manipulation (State D). One further simple state was formed from Start action (State B) only. The transitions indicated that the low achieving dyads behavior patterns could be characterized by two dominant moves between the three basic control buttons: Setup, Start, and Stop. The transition probabilities between Setup (State A) and Start (State B), and between Start (State B) and Stop (State C) were as high as 65% and 70%, respectively. All other students transitions among a range of states had medium or low probabilities, suggesting that the exploration was less systematic. In contrast to the high achievers HMM, the transition probability from the initial Setup (State A) to the model manipulation (State D) was only 10%. This result suggests that these students rarely setup the main parameters after resetting the model, but rushed to start the simulation (State B). Nevertheless, this dyad sometimes also manipulated some model parameters (State D). The students often stopped the simulation (State C), changed the parameters (State D), and then resumed (State
ISLS 28
ICLS2012
B). It is also interesting to note that log file analysis revealed this dyad never attempted to track an individual CO2 molecule, unlike the high achieving dyad.
Figure 2: Hidden Markov Model of the high achieving dyad These results indicated that the model exploration strategies adopted by two dyads were different. The HMM for the high achieving dyad indicates that their explorations can be characterized by the interaction with the model, systematic control of the parameters, and model tracking behaviors. It also suggests that these students combined qualitative and quantitative explorations depicted by State D and State B, respectively. Their dominant interaction behavior can be broadly called a Setup-Configure-Start/Follow pattern. The HMM for the less successful dyad depicts more passive student interactions with the model and more surface exploration behaviors. Their dominant behavior pattern can be broadly called Setup-Start-Stop, which was then followed by the manipulation of model parameters. Overall, the HMM for the low achieving dyad indicates that these students did not engage in systematic explorations of the agents depicted in the carbon cycle model.
Figure 3: Hidden Markov Model of the low achieving dyad In order to triangulate the HMM results, we used other core process visualization and statistical techniques to explore student interactions by constructing interaction graphs (Figure 4 and Figure 5). These graphs depict dyads interactions with the carbon cycle models during the experimental period. The vertical axis represents the time line, and the horizontal axis shows the total number of various model control actions taken from the beginning of the session. The interaction graph for the high achieving dyad confirms that the students actively explored the model during the entire experimental sequence by using three main simulation controls and parameters: fossil fuel use, speed, and tracking CO2 molecule (Figure 4). It also depicts that during the first half of the session
ISLS
29
ICLS2012
students changed the fossil fuel rates and speed, which suggests simultaneous quantitative and qualitative explorations for tracking of a CO2 molecule. However, it also reveals that there were noticeable changes in the student behaviors over time. For example, these students changed the CO2 tracking setting during the first half of the session only. This was followed by a shorter, but a noticeable period, when students actively controlled the fossil fuel use. At the end of the session, they interacted with the model simply by pausing and resuming the simulation without changing its settings.
Figure 4: Interaction graph of the high achieving dyad Similarly the interaction graph for the low achieving dyad generally reaffirmed the pattern depicted by the HMM (Figure 5). It showed that these students interacted with the model much longer, but their interaction was often limited to Setup, Start, and Stop actions. The interaction graph further revealed noticeable changes in the student behavior during the session. For example, in more than half of the session, these two students changed only the fossil fuel use values. This was followed by a short interval when the students started to change and manipulate only speed, and they then proceeded to changing the two parameters simultaneously.
Figure 5: Interaction of the low achieving dyad

In terms of the implications of these findings for the research interest related to sequences of pedagogical structure, we note that participants in both the Challenge and Guided Learning (CGL) and Guided Learning (GL) sequences made significant learning gains. Although there were not significant learning differences between the two treatment groups, the main findings were actually in the hypothesized directions, with the CGL group scoring higher on the model problem solutions (especially Model 1) and on the posttest than did the GL group. We note that this study involved a relatively small number of students, and we are planning future studies that will involve larger numbers by collecting data in additional schools, which should provide greater statistical power to further investigate aspects of the sequencing of pedagogical guidance for agent-based
ISLS 30
ICLS2012
models and learning. Still, given there have been few reported empirical studies of learning in this area, an important finding in this study is that on average all participants significantly enhanced their understanding of important scientific knowledge about greenhouse gases and the carbon cycle. The process analysis is an ongoing research theme in this project. We aim to find key patterns of students interactions with the models, and find ways to identify on-the-fly effective and inefficient behaviors for learning with agent-based computer models. A current limitation is that the algorithm that we used to construct the HMMs has a local maxima problem and the log data files are relatively small. In our study, we executed the algorithm one hundred times with random initializations by sampling the initial parameter values from uniform distributions (Jeong et al., 2010; Southavilay, Yacef, & Callvo., 2010). The HMMs were constructed when these executions converged to the same configuration. A better solution is needed to execute this algorithm if we are to use the HMM for providing feedback to students and teachers in real time. Further, the HMM algorithm applied to the student model generated log files identifies patterns that are directly rooted in students technical fine-grained actions and constructs only one model for the entire interaction sequence. Two key challenges need further exploration and solution. First, as we illustrated above, students behavior patterns may change during their interaction with the model. For example, we expect that one of possible reasons for the change in student interactions depicted in the timelines (Figure 4 and 5) was the change between the three activities associated with each model. Other research (Levy & Wilensky, 2010) has that found students tend to apply similar strategies when they interact with different models. In addition, their interaction patterns are not fixed and students, particularly successful learners, may flexibly adapt their interaction strategies as their goals change (i.e., when students complete one activity or answer a question and move to the other). The analytical techniques should be flexible enough to identify such macro level changes (i.e., shifts between stages or goals) in student exploration strategies and then construct interaction patterns for each stage separately. Second, when modeling is based on the raw log data, it directly links students activities to the students technical fine-grained interactions with the model via the interface (i.e., button click is equal to student activity). These technical actions are likely not identical to the meaning actions that are associated with the students intentions and conscious behavior. Drawing on evidence from cognitive research, we can expect that meaning actions are likely to be more abstract chunks of common sequences of student behavior (i.e., clusters of common strings of technical actions). Mining techniques are needed for identifying meaning actions in student logs initially, which then could be used for constructing further student interaction models. To address the first challenge, we are currently working on two goals. First, we will put a time/date stamp into the NetLogo log files to mark the beginning and ending of the specific problem solving activities the students are working on. This time/date stamp will allow us to identify possible changes in students strategies as they are engaged in the different model-based problem activities; in particular, different model use behaviors in sequences of low and high structured tasks associated with Challenge and Guided Learning and Guided Learning. Second, in order to gain deeper insights into how students performed their activities, we are also exploring possibilities to incorporate process mining techniques, particularly in the initial exploratory stage of the analysis (Trka et al., 2011; van der Aalst, 2011). Addressing the second challenge, we plan to utilize a Hidden Markov Model clustering algorithm (Shih, Koedinger, & Scheines, 2010) and explore other similar techniques. By clustering and learning HMMs rather than just one for each student group, we can discover different strategies that students employed and to use those strategies to understand and, perhaps in the future, to predict learning outcomes. This combination of techniques should provide a good fit between automatically identified behavior patterns to students conscious behavior. In terms of more general research issues highlighted at the start of this paper, these results are encouraging for the continuation of this design-based research, and we are hopeful that the process mining approaches we are developing may be useful in providing further insights into the dynamics of learning scientific knowledge with computer model based learning activities.
References
ACARA. (2011). The Australian Curriculum: Science (Version 1.2). Sydney: Australian Curriculum Assessment and Reporting Authority. Clement, J. (2000). Model based learning as a key research area for science education. International Journal of Science Education, 22(9), 1041-1053. Gerard, L. F., Spitulnik, M., & Linn, M. C. (2010). Teacher use of evidence to customize inquiry science instruction. Journal of Research in Science Teaching, 47(9), 1037-1063. Goldstone, R. L., & Wilensky, U. (2008). Promoting transfer through complex systems principles. Journal of the Learning Sciences, 17(4), 465-516. Intergovernmental Panel on Climate Change. (2007). Fourth Assessment Report of the Intergovernmental Panel on Climate Change, .
ISLS
31
ICLS2012
Jacobson, M. J., Kapur, M., So, H.-J., & Lee, J. (2011). The ontologies of complexity and learning about complex systems. Instructional Science, 39, 763-783. doi: 10.1007/s11251-010-9147-0 Jeong, H., Biswas, G., Johnson, J., & Howard, L. (2010). Analysis of productive learning behaviors in a structured inquiry cycle using hidden markov models. Paper presented at the 3rd International Conference on Educational Data Mining, Pittsburgh, PA, USA. Kapur, M. (2006). Productive failure. In S. Barab, K. Hay & D. Hickey (Eds.), Proceedings of the International Conference of the Learning Sciences (pp. 307-313). Mahwah, NJ: Erlbaum. Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3), 379 - 424. Kapur, M. (2010). A further study of productive failure in mathematical problem solving: Unpacking the design components. Instructional Science, DOI: 10.1007/s11251-11010-19144-11253. Levy, S. T., & Wilensky, U. (2010). Mining students inquiry actions for understanding of complex systems. Computers and Education, 56(3), 556-573. doi: 10.1016/j.compedu.2010.09.015 Mayer, R. (2004). Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. American Psychologist, 59(1), 14-19. National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Committee on a Conceptual Framework for New K-12 Science Education Standards. Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. Pathak, S., Kim, B., Jacobson, M. J., & Zhang, B. (2009). Failures and successes in collaborative inquiry: Learning the physics of electricity with agent-based models. In C. OMalley, D. Suthers, P. Reimann & A. Dimitracopoulou (Eds.), Computer-supported collaborative learning practices: CSCL2009 conference proceedings (pp. 199-203). Rhodes, Greece: International Society of the Learning Sciences. Railsback, S. F., & Grimm, V. (2011). Agent-based and individual-based modeling: A Practical Introduction. New Jersey: Princeton University Press. Reimann, P. (2009). Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning, 4(3), 239-257. Romero, C., & Ventura, S. (Eds.). (2006). Data mining in e-learning. Southampton: WITpress. Romero, C., Ventura, S., Pechenizkiy, M., Ryan, S., & Baker, R. (Eds.). (2010). Handbook of Educational Data Mining. Boca Raton: CRC Press. Schwartz, D. L., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4), 475522. Shih, B., Koedinger, K. R., & Scheines, R. (2010). Discovery of learning tactics using Hidden Markov Model clustering. Paper presented at the 3rd International Conference on Educational Data Mining, 2010, Pittsburg, PA, USA. Southavilay, V., Yacef, K., & Callvo., R. A. (2010, June 11-13). Process Mining to Support Students' Collaborative Writing. Paper presented at the International Conference on Educational Data Mining, Pittsburgh, PA, USA. Trka, N., Pechenizkiy, M., & van der Aalst, W. (2011). Process Mining from Educational Data. In C. Romero, S. Ventura, M. Pechenizkiy & R. S. J. d. Baker (Eds.), Handbook of Educational Data Mining (pp. 123-142). Boca Raton: CRC Press. van der Aalst, W. M. P. (2011). Process Mining: Discovery, Conformance and Enhancement of Business Processes: Springer. VanLehn, K., Siler, S., & Murray, C. (2003). Why do only some events cause learning during human tutoring? Cognition and Instruction, 2(3), 209-249. Wilensky, U. (1999). NetLogo. Evanston, IL: Center for Connected Learning and Computer-Based Modeling. Northwestern University (http://ccl.northwestern.edu/netlogo). Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep or a firefly: Learning biology through constructing and testing computational theories -- an embodied modeling approach. Cognition & Instruction, 24(2), 171-209. Zhang, B., Liu, X., & Krajcik, J. S. (2005). Expert models and modeling processes associated with a computermodeling tool. Science Education, 90, 579-604.
Acknowledgments
The research reported in this paper was funded by an Australia Research Council Linkage Grant No. LP100100594 to the second and third authors. We thank Dr. Paul Stokes, Kashmira Dave, and the participating teachers and students for their invaluable assistance in this study.
ISLS
32
ICLS2012
Learning how to create: Toward a learning sciences of art and design

R. Keith Sawyer, Washington University in St. Louis Abstract: This paper is the first report on an extensive ethnographic study of two professional schools of art and design in the United States. The overall purpose of the study is to identify general principles for how to design learning environments that prepare learners to be creative. First, I document the cultural model of teaching and learning held by the faculty and students, and analyze the pedagogical practices used. This studio model is of interest because it emerged naturally in a community of educational practice. I argue that it is distinct from the two cultural models most familiar to learning scientists: instructionism and apprenticeship. Second, I argue that the studio model is more closely aligned with learning sciences principles than either instructionism or apprenticeship. Third, I draw lessons from this studio model for designing learning environments in all school subjects, in particular STEM subjects.
Introduction
How can we teach students to be creative? How can we teach them disciplinary knowledge in a way that prepares them to build on it and to go beyond? These questions are increasingly important in education due to a widely acknowledged global transformation to a knowledge-based society and a creative economy (Florida, 2002; Sawyer, 2008). In the United States, many learning scientists have joined with policy makers to advocate that schools should teach 21st century skills including creativity (Trilling & Fadel, 2011). In the United Kingdom, national policy since the late 1990s has emphasized the creative industries as an economic strategy, and creativity is a part of the national curriculum (NACCCE, 1999). Internationally, the OECD has generated a series of reports about the need for schools to become creative learning environments (e.g., OECD, 2006, 2008). The future of learning must centrally involve creativity. This paper is the first report on an extensive ethnographic study of two professional schools of art and design in the United States. The goal of the study was to document the cultural model of teaching and learning held by the faculty and students, and to document and analyze the pedagogical practices used. Two institutions were chosen: the Savannah College of Art & Design (SCAD), a free-standing art and design college dedicated to art and design with a career focus; and Washington University (WU), where the art and design school is embedded within a comprehensive research university, with a more theoretical and research focus. These very different sites were chosen to ensure that the identified cultural model would be more likely to be general to all art and design schools rather than specific to a certain type of institution. Data include videotapes of studio classes and critiques, audiotaped interviews with professors, audiotaped interviews with students, photos of student work, and copies of curricular materials provided by professors. As of Fall 2011, the data-gathering phase is complete and transcription has begun. Although the transcription and coding is expected to continue through 2012, it is already possible to identify a core set of features of the cultural model of teaching and learning, and this paper presents the first preliminary findings from the study. This study is of importance to learning scientists for three reasons. First, it documents a naturally occurring cultural model, one that emerged over time in a professional community of practice. Thus it occupies an analogous position to the various ethnographies of apprenticeship that so strongly influenced learning sciences in the 1980s and 1990s (e.g., Collins, Brown, & Newman, 1989; Lave & Wenger, 1991; Rogoff, 1990). Second, I will argue that this cultural model is strongly consistent with learning sciences findings, and yet that it presents an intriguingly different vision of teaching and learning. Third, I will argue that this cultural model can help us to resolve a fundamental paradox facing learning sciences theory and research: How can students learn the basic facts, concepts, and skills of a discipline, and yet learn them in such a way as to be prepared to be creative and to go beyond them? To date, learning scientists have rarely addressed creativity explicitly. However, many of the reforms advocated by learning scientists are designed to result in learning outcomes that researchers believe result in higher creative potential (Sawyer, 2008). For example, when one learns deeper conceptual understanding rather than superficial knowledge, one is better prepared to transfer that understanding to a new situation, resulting in adaptive expertise (Hatano & Inagaki, 1986; Schwartz, Bransford, & Sears, 2005) or adaptive competence (de Corte, 2010). Several learning scientists have been inspired by studio practices in design education and engineering education, and have adopted studio practices in K-12 math and science classes (e.g., Cossentino & Shaffer, 1999; Cox, Harrison, & Hoadley, 2009; Greeno, 1997; Jacobson & Lehrer, 2000; Hmelo, Holton, & Kolodner, 2000), but without empirically documenting the range of practices found in art and design education, and without fully theorizing and articulating how such environments might foster greater creativity. This project intends to build on and extend this previous work.
ISLS
33
ICLS2012
Learning scientists have argued that the traditional cultural model of teaching and learning, referred to as instructionism (Papert, 1980), transmission and acquisition (Rogoff, 1990), or the standard model (OECD, 2008) is not grounded in the science of learning. Beginning in the 1980s, education researchers searched in the wild for other cultural models of teaching and learning, and were drawn to apprenticeship (Collins, Brown, & Newman, 1989; Lave & Wenger, 1991; Rogoff, 1990). These two cultural models have often presented as two poles of an axis from formal learning to informal learning. I argue that the studio model represents a third cultural model of teaching and learning, distinct from both instructionism and apprenticeship. Documenting this cultural model is the first goal of this paper. The second goal is to demonstrate that this cultural model is closely aligned with learning sciences findings, and thus to demonstrate that those findings are not restricted to STEM disciplines. Learning scientists have argued that their findings reflect general properties of human cognition that underlie all learning. And yet, learning sciences research has occurred almost exclusively in STEM disciplines. This study serves as an important validation test of the generality of learning sciences findings. The third goal is to argue that the studio model has implications beyond art and design education; that it provides lessons for how to resolve longstanding tensions faced by learning scientists and educators in STEM disciplines. Most importantly, I argue that studio learning environments resolve a fundamental paradox of constructivist teaching: how to ensure that learners master core facts and skills, while at the same time learning how to use that knowledge creatively? Sawyer (2011) argued that such teaching is best thought of as an improvisational activity, and that the best teaching is disciplined improvisation: teaching that provides space for the flexibility required for constructivist learning, but guided within structures and frameworksin a similar fashion to professionally performed improvisations found in jazz and improv theater. In this paper, I show that the teaching practices of professional schools of art and design represent a form of disciplined improvisation, and thus show us how to teach for creativity.
Related studies
Hetland et al. (2007) studied the teaching practices of five teachers at two secondary schools, and identified three practices: demonstration/lecture, student work, and critique. Their analysis was focused on studio art, and did not draw connections to the learning sciences. As in most other academic disciplines, university instructors often engage in reflection and writing about their own teaching practice. Design educators have written about design education (e.g., Design Studies 1999 special issue); and architecture educators have written about architecture education (e.g., Dutton, 1991; Nicol & Pilling, 2000). Although these writings are helpful, they tend to be focused on practitioner issues and rarely explore underlying models of learning. Two exceptions are (Schn, 1985) and (Shaffer, 2003) which report on ethnographies of educational practice in architecture studios. There have been several studies of design expertise (Cross, 2011; Lawson & Dorst, 2009) and design practice (Hall & Stevens, 1996). Many of these studies identify various stages of the design process and the iterative cycling through them; a typical model includes stages such as (1) explore the problem, (2) generate a range of concepts, (3) select the most promising concept, (4) develop the concept into a detailed design, (5) communicate the final proposed design. This study builds on previous work in several ways: (1) although prior studies have described specific practices, none document the complete cultural model of teaching and learning shared in this community of practice; (2) for the most part, each study tends to focus on one discipline (art, architecture, or engineering) whereas this study attempts to identify what is common across all art and design disciplines; (3) previous studies tend to focus on one class, teacher, or project, and thus are not able to identify the way that learning trajectories are designed for the learner, nor the place of any one practice in it; (4) finally, this study aims to identify general principles, grounded in the learning sciences, that can be applied to all academic disciplines, particularly STEM disciplines. The most closely related prior studies are (Shaffer, 2003) and (Hetland et al., 2007); this study is inspired by those studies, and attempts to be both broader and deeper.
Methodology
The author conducted an ethnographic study of two professional schools of art and design. Two very different types of institutions were chosen to ensure that the cultural model identified would be more likely to be general to all art and design schools rather than specific to a certain type of institution. The Savannah College of Art & Design (SCAD) is a freestanding institution dedicated to art and design. It is proud of its focus on creative careers and its ability to place students in creative industry jobs after graduation. It is the largest art and design school in the United States, with over 9,000 students and more than 40 different majors offered, ranging from painting and sculpture to architecture, industrial design, furniture design, advertising, videogame design, and movie production.
ISLS
34
ICLS2012
The Sam Fox School of Design & Visual Arts is embedded within Washington University (WU), a comprehensive research university. Students are required to take one course each semester outside of the Sam Fox School. Most students take classes in the College of Arts and Sciences, but some take classes in the School of Business or even the School of Engineering. Compared to SCAD, there is more of a focus on theoretical foundations, conceptual underpinnings, and a notion of arts practice as research. WU is substantially smaller, with under 1,000 students and just over ten undergraduate and graduate degrees. At SCAD the author served as a Visiting Professor for two academic quarters (a total of six months) and conducted three other shorter site visits. At WU the author has been a professor since 1996, in a different academic unit but with an office about mile distant. Observations and interviews occurred during two academic semesters. Both sites combined resulted in a total of approximately 75 hours of video observations and audiotape interviews, as follows: Table 1: Data gathered. Classes videotapes 6 (2.5 hours each) 6 Instructors interviewed 18 16 Students interviewed 6 4
SCAD WU
This conference paper is a preliminary report; not all materials have been transcribed and coded. However, based on initial analyses of these materials, several preliminary findings have emerged. By the time of the conference in Summer 2012, specific quotations and video episodes will be available for presentation.
Findings
Based on a preliminary review of the ethnographic materials, the studio model can be characterized by the following features.
Complex, authentic, real-world projects

In all of the classes observed, instructors carefully design projects to initiate and guide student work. A project poses a problem that is open ended, so that there is no single obvious way to resolve the problem, and so that each student must identify their own path to a solution. In design terms, these are ill defined or wicked problems. In the first two years of the curriculum, the projects typically take two to three weeks to complete, and over the course of a quarter or semester term, four to five projects will be completed. In the latter two years of the curriculum, projects become progressively more open ended and involve longer periods of work, fewer constraints, and greater student discretion. In these later classes, it is common for one project to absorb the entire term. The projects are designed so that students will not be able to complete them unless they master specific knowledge and skills that have been identified by the instructor as the desired learning outcomes. Instructors believe that students will learn the required knowledge and skills more effectively if they learn them while engaged in a process of authentic problem solving. Instructors justify this method of instruction by pointing out that in the art and design professions, problems are complex wholes, and cannot be reduced to individual component skills. Real world projects are complex and all the parts are interrelated. Its not linear (Ella, SCAD interior design).1 They believe that teaching individual skills apart from authentic projects would not properly prepare students for professional disciplinary practice. In many classes in design disciplines, instructors work with external industry partners as clients. This is particularly common in disciplines like industrial design and graphic design. Executives from the client firm will attend a class early in the term to present the problem to be solved, and will visit again at the end of the term, when student teams will present their final work to the client. A common theme in learning sciences based curricula is that students are encouraged to engage in activities that are similar to those practiced by professionals in the discipline (e.g., Edelson, 2006; Krajcik & Blumenfeld, 2006). Because this study analyzes professional schools with the goal of preparing students for professional practice, it is not surprising that authentic practice is heavily emphasized.
Guided problem solving

Although projects are open ended so that each student must find their own way to a solution, projects also typically specify a set of constraints or parameters that restrict how the student can proceed. Here are two representative examples of project assignments. In a Communication Design class: The attached population data compares populations in 2000 and 2010 for all countries and regions with more than 5000 people. Using at least the first two columns, design two different
ISLS
35
ICLS2012
approaches to visualize this data. One of these approaches must make some use of some aspect of a world map (shape of countries, relative position in the world, etc.) Each of your two solutions must be able to fit within a 20 x 16 format, but can go smaller. You may use a maximum of two colors, black and a second color of your choice. (Heather Corcoran, WU, Spring 2011) In a first year 2D design class: Create a project that conveys (1) something that just happened; (2) something happening; or (3) something about to happen. You can only use a quarter or less of a figure, you must use these fifteen objects provided, and it must be in an architecturally significant interior. (Kathy, SCAD, Winter 2010) When I asked professors why they provided such specific constraints, they provided a variety of intriguing answers that invoke many learning sciences principles and warrant further exploration. First, the goal of project parameters is to limit students creative options, so that their creative effort is focused on the specific learning outcomes desirable at that point in their learning trajectory. John Hendrix (WU, illustration) said Project constraints set you free. It focuses the learning because it removes the number of variables they have to think about. Second, instructors often said that without such constraints, students would fall into familiar patterns. Julie Varland (SCAD, architecture) said We are peeling away their assumptions and forcing them into discomfort without structure, theyd do what they know already. Arne Nadler (WU, core) said You find these things through doing, not while you are thinking about it. Other instructors said Students need the structure. It gives them something to push against (SCAD reading group). This practice is consistent with learning sciences studies of misconceptions (diSessa, 2006): The constraints are thought to be necessary to help the students to deeply challenge their own misconceptions, and then to guide them to a more advanced conception. Third, instructors said that the constraints were valuable because they would lead students to fail early, thus more deeply realizing the need to move beyond their existing misconceptions. John Hendrix said, Students should come in with an expectation and then have it proven wrong. Then, we redirect them down the right path. Scott Thorpe (SCAD, sculpture) said, People dont change their model of thinking unless theres a consequence. I wanted them to fail. Rhonda Arntsen (SCAD, graphic design) said I encourage failure. This practice is consistent with several cognitive studies showing that learner failure results in more effective learning (impasse driven learning of Van Lehn, 1988; desirable difficulties of Bjork & Linn, 2006; productive failure of Kapur, 2008). Fourth, instructors often said that constraints were necessary to force students to slow down. Professors report that beginning art and design students attempt to move too quickly toward a solution. I want students to stop and think. Dont start working right away (Rhonda Arntsen). An important outcome of art and design education is that students master a deliberate process that will result in consistently effective work. Prior research (see Related studies) has documented that expert designers engage in an early exploratory phase, which students tend to rush through. One painting instructor at SCAD, Susan, assigned a project that required students to use the lost medieval technique of glazesnot only because glazes were a valuable technique for a painter, but also because glazes take a long time to dry, and cannot be applied until the under painting has dried, thus forcing students to slow down. The constraints of a project assignment are the subject of substantial thought and explicit discussion among instructors. Project assignments are continually revised, a process familiar to learning scientists as design based research (Barab, 2006). Instructors often talked about projects that didnt work and required revision. The most common problem was that not enough constraints were specified. In this situation, what typically happens is that all students generate a too-obvious solution; their misconceptions are not successfully challenged. A less common problem is that too many constraints are specified. In this situation, all students tend to generate very similar solutions. The studio model is similar to project-based learning, a learning-sciences based science curriculum (Krajcik & Blumenfeld, 2006). According to Krajcik and Blumenfeld, learning environments that are project based have five key features: 1. 2. They start with a driving question, a problem to be solved. Learners explore the driving question by participating in authentic, situated inquiryprocesses of problem solving that are central to expert performance in the discipline. As students explore the driving question, they learn and apply important ideas in the discipline. Students, teachers, and community members engage in collaborative activities to find solutions to the driving question.
36
3.
ISLS
ICLS2012
4. 5.
While engaged in the inquiry process, students are scaffolded with learning technologies that help them participate in activities normally beyond student ability. Students create a set of tangible products that address the driving question. These are shared artifacts, publicly accessible external representations of the classs learning.
Only feature (4), technology scaffolding, is not found in art and design education. Feature (3), collaboration, was often observed in art and design classes, but collaboration is beyond the scope of this paper. Feature (5) is at the core of art and design education and will be discussed further below.
Curriculum design
Instructors design open-ended projects so that as students develop their own creative solutions, they attain the desired learning outcomes. Douglas Dowd (WU, illustration) said, The learning is embedded in the act. Instructors believe that simply warning students about common problems, or describing how to use standard techniques, will not work. Students have to learn through problem solving. Each term, a class begins with high scaffolding and progressively fades to less scaffolding. Over the four year undergraduate curriculum, fading occurs on a broader scale. Project assignments are carefully sequenced so that desired techniques and domain knowledge are acquired in the process of engaging in authentic problem solving activity, and so that each projects knowledge prepares the learner to begin the next project. Amy Thompson (WU, printmaking) said Every project builds on the project before it. In many design disciplines, there is a standard sequential practice in the discipline (see Related studies above) that students must learn. In the course of a two-week project, a studio class will typically meet for four timestwice each week for 2.5 hoursand for each of these four classes, the instructor will specify an assignment that moves the student to the next stage of the design process. This practice scaffolds and externalizes the design process used by professionals in the discipline. For example, in a 3D graphics studio in SCADs advertising department (Joe diGioia), a term-length project required students to work through all of the stages of a real project: identify a target market, create a logo, develop prototypes, provide preliminary sketches, and at the end, present a final model. As students engaged in this project, at each stage they learned specific techniques and skills; yet these were contextualized within an ongoing authentic activity. Instructors comments frequently remind students of what they were working on in previous weeks, how their work has evolved, and mention potential next steps for the work. In a painting class, Susan Russell asked students to estimate what percentage of the work they had completed so far, and then asked What comes next in your painting? One student was asked, What will you do in your next hour of work on this painting? Instructors constantly reinforce the embedded, authentic nature of creative work. Learning sciences research has demonstrated that constructivist learning processes must be carefully guided to result in maximum effectiveness. One group of studies has shown that discovery learning, a relatively unguided version of constructivism, is less effective than instructionism (Mayer, 2004). The consensus emerging from learning sciences research is that the most effective learning environments are fairly highly constrained, while nonetheless providing students with the opportunity to engage in authentic, situated inquiry practices. Art and design students are substantially scaffolded by the parameters of the project assignments, and by the sequencing of projects over the course of a school term.
Externalization and reflection

One of the most fundamental characteristics of art and design learning is that a learners developing understandings are externally represented in a visible artifact. Ideas come from making things in design, making is thinking (Ken Botnick, WU, printmaking). Catalina Freixas (WU, architecture) had an explicit rule in her class, If its not on your drawing then we cant discuss it, because of her frequent experience of students wanting to talk about what they thought they were doing (but had not actually done). Studio conversation is always focused on the external artifact generated by the student. Students have to explain their choices and back them up (Amy Thompson). In one architecture critique I observed, a visiting architects first comment to a student was I dont understand what we are looking at and later, Youre not saying what you are doing and giving it a name (Freixas studio 11 May 2011). Such comments refer to one of the most common problems with student work: that the students intention for the work is not realized in the actual work. When this happens, instructors ask questions like: Can anyone help him figure out what this is about? (Susan); Why do you want to pursue a solution like this? (Ella). In the studio, reflection is constantly required. Why this choice? Why this action? What are you trying to do? Why are you using this technique? Learning sciences has found that environments that foster reflection about learning are more effective (e.g., Bransford et al., 2000). These project-based learning environments are designed to guide students toward an increasing ability to link intention with making.
ISLS
37
ICLS2012
Balancing domain skill learning and creative abilities

Expert performance in most domains requires a great deal of repetitive practice, and most knowledge intensive professions require that a large body of facts and skills be mastered (Ericsson, 2002). In professional schools of art and designwhere creativity is more valued than in any other disciplinebasic skills are viewed as essential and are constantly emphasized. Yet their learning is embedded in the authentic professional practices of the discipline. Skills are often taught opportunistically, as a question or problem arises in the course of a students project work. For example, in Susans painting class (22 Feb 2010), a review of one students painted portrait gave the instructor an opportunity to tell the entire class about classic techniques for shadow and background. Responding to these teachable moments requires a high degree of pedagogical content knowledge; because the syllabus is not structured around specific facts and skills, instructors must keep in mind their own set of desired learning outcomes and wait for good opportunities to introduce them. The studio model demonstrates that teaching creativity is not opposed to repetition, craft, and skill that in fact, creative performance depends on their mastery. Thus the studio model can help STEM educators to resolve an ever-present tension: how to design scaffolded constructivist learning environments, such as projectbased and inquiry-based approaches, that also ensure that students master basic facts and skills.
Conclusion
This study contributes to the learning sciences in the three ways identified in the introduction. First, the learning sciences have almost exclusively studied learning in science, technology, engineering, and math (STEM). Learning scientists believe that their findings apply to learning in all domains, and yet this belief has not been empirically validated because there has been so little learning sciences research in other disciplines, particularly in the arts. This study confirms that the principles of effective learning that have been identified in the learning sciences, primarily through the study of STEM disciplines, apply equally to art and design. Second, I have argued that the studio model is closely aligned with learning sciences findingsmore so than either instructionism or apprenticeship. This is particularly intriguing because the studio model emerged over historical time in a community of practice. Instructors all reported that they learned how to teach on their own, drawing on memories of their own favorite instructors, or on conversation with colleagues. Each instructor develops their own project assignments (they do not use textbooks or online repositories of good project ideas). Third, the learning environments that are found in schools of art and design are well articulated and carefully designed, to an extent that has been difficult to attain in STEM curricular reforms based on learning sciences findings. Projects are linked together sequentially through each term, and the classes are linked together across the four year undergraduate curriculum. A sustained study of the nature of this sequencing could provide valuable insights into how to redesign STEM curricula. Most importantly, these learning environments are designed with the goal of resolving a fundamental paradox of constructivist teaching: how to ensure that learners master core facts and skills, while at the same time learning how to use that knowledge creatively? In the future of learning, in the 21st centurys creative age, these are the same goals that we have for the reform of STEM education.
Endnotes
(1) Many participants requested that I use their actual name, wanting to be credited for their pedagogical practices, so for those participants I provide both first and last name. Other participants requested a pseudonym, and for those participants I use only a first name pseudonym.
References
Barab, S. (2006). Design-based research: A methodological toolkit for the learning scientist. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 153-169). New York: Cambridge University Press. Bjork, R. A., & Linn, M. C. (2006). The science of learning and the learning of science: Introducing desirable difficulties. APS Observer, 19(3), 29,39. Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 453-494). Mahwah, NJ: Erlbaum. Cossentino, J., & Shaffer, D. W. (1999). The math studio: Harnessing the power of the arts to teach across disciplines. Journal of Aesthetic Education, 33(2), 99-109. Cox, C., Harrison, S., & Hoadley, C. (2009). Applying the "studio model" to learning technology design. In C. DiGiano, S. Goldman & M. Chorost (Eds.), Educating learning technology designers (pp. 145-164). New York: Routledge. Cross, N. (2011). Design thinking: Understanding how designers think and work. Oxford: Berg.
ISLS 38
ICLS2012
de Corte, E. (2010). Historical understandings in the understanding of learning. In H. Dumont, F. Benavides & D. Istance (Eds.), The nature of learning: Using research to inspire practice (pp. 35-67). Paris: OECD. diSessa, A. A. (2006). A history of conceptual change research. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 265-281). New York: Cambridge University Press. Dutton, T. A. (Ed.). (1991). Voices in architectural education: Cultural politics and pedagogy. New York: Bergin & Garvey. Edelson, D. C., & Reiser, B. J. (2006). Making authentic practices accessible to learners: Design challenges and strategies. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 335-354). New York: Cambridge University Press. Ericsson, K. A. (2002). Attaining excellence through deliberate practice: Insights from the study of expert performance. In M. Ferrari (Ed.), The pursuit of excellence in education (pp. 21-55). Hillsdale, NJ: Erlbaum. Florida, R. (2002). The rise of the creative class and how it's transforming work, life, community and everyday life. New York: Basic Books. Greeno, J. G., & The Middle School Mathematics Through Applications Project. (1997). Theories and practices of thinking and learning to think. American Journal of Education, 106, 85-126. Hall, R., & Stevens, R. (1996). Teaching/learning events in the workplace: A comparative analysis of their organizational and interactional structure. In G. W. Cottrell (Ed.), Proceedings of the Eighteenth Annual Conference of the Cognitive Science Society (pp. 160-165). Hillsdale, NJ: Erlbaum. Hatano, G., & Inagaki, K. (1986). Two courses of expertise. Child Development and Education in Japan, 262272. Hmelo, C. E., Holton, D. L., & Kolodner, J. L. (2000). Designing to learn about complex systems. Journal of the Learning Sciences, 9(3), 247-298. Jacobson, C., & Lehrer, R. (2000). Teacher appropriation and student learning of geometry through design. Journal for Research in Mathematics Education, 31(1), 71-88. Kapur, M. (2008). Productive failure. Curriculum and Instruction, 26(3), 379-424. Kolodner, J. L. (2006). Case-based reasoning. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 225-242). New York: Cambridge University Press. Krajcik, J. S., & Blumenfeld, P. (2006). Project based learning. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 317-333). New York: Cambridge University Press. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York: Cambridge University Press. Lawson, B., & Dorst, K. (2009). Design expertise. Burlington, MA: Architectural Press. Mayer, R. E. (2004). Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. American Psychologist, 59(1), 14-19. NACCCE (National Advisory Committee on Creative and Cultural Education). (1999). All our futures: Creativity, culture and education. London, UK: DFEE. Nicol, D., & Pilling, S. (Eds.). (2000). Changing architectural education: Towards a new professionalism. New York: Spon Press. OECD. (2006). 21st century learning environments. Paris: OECD. OECD. (2008). Innovating to learn, learning to innovate. Paris, France: OECD. Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. New York: Basic Books. Rogoff, B. (1990). Apprenticeship in thinking: Cognitive development in social context. New York: Oxford University Press. Sawyer, R. K. (2008). Optimizing learning: Implications of learning sciences research. In Innovating to learn, learning to innovate (pp. 45-65). Paris: OECD. Sawyer, R. K. (Ed.). (2011). Structure and improvisation in creative teaching. Cambridge, UK: Cambridge University Press. Schn, D. (1985). The design studio: An exploration of its traditions and potentials. London, UK: RIBA Publications Limited. Schwartz, D. L., Bransford, J. D., & Sears, D. A. (2006). Efficiency and innovation in transfer. In J. P. Mestre (Ed.), Transfer of learning from a modern multidisciplinary perspective (pp. 1-51). Greenwich, CT: Information Age Publishing. Shaffer, D. W. (2003). Portrait of the Oxford design studio: An ethnography of design pedagogy (Working paper No. 2003-11). Madison, WI: Wisconsin Center for Education Research. Trilling, B., & Fadel, C. (2009). 21st century skills: Learning for life in our times. San Francisco, CA: JosseyBass. VanLehn, K. A. (1988). Towards a theory of impasse-driven learning. In H. Mandl & A. Lesgold (Eds.), Learning Issues for Intelligent Tutoring Systems (pp. 19-41). New York, NY: Springer.
ISLS
39
ICLS2012
A Microgenetic Analysis of How Expansive Framing Led to Transfer with One Struggling Student
Diane P. Lam, Xenia S. Meyer, Randi A. Engle, Lloyd Goldwasser, Sarah Perez, Kathleen Zheng, Jim Clark, Erica Naves, Hernan Rosas, Danny Tan Graduate School of Education, UC Berkeley, 4646 Tolman Hall Berkeley, CA. 94720 dianelam@berkeley.edu, xenia.meyer@berkeley.edu, raengle@berkeley.edu Abstract: Research has shown that it is uncommon for students to transfer what they learn about science to their other classes, let alone their everyday lives. However, a set of recent studies provides empirical evidence that the expansive framing of learning contexts can foster transfer. In this paper, we further our investigations of the idea that transfer is promoted by the instructional practice of framing learning contexts expansively. We do so by offering a microgenetic analysis of a relatively surprising case of transfer by a student in a high school biology class. In particular, we use her case to evaluate five recently proposed explanations for exactly how expansive framing may foster transfer, and close with additional hypotheses for how transfer may have been supported with this particular student.
Introduction
If students are to be successful and if schooling is to have a significant impact on their lives, it is essential that students regularly transfer what they learn (Renkl, Mandl & Gruber, 1996; Schwartz, Bransford & Sears, 2005). Specifically, this research is based on the premise that it is not just the nature of the particular science content that students learn that matters for transfer, but also how learning and transfer contexts are framed. In this paper, we examine the idea that transfer can be promoted through an expansive framing of learning contexts by conducting a microgenetic analysis of a case of one struggling student in a high school biology class (Maxwell, 2004; Siegler, 2006). First, we explain and describe what we mean by expansive framing, and then briefly present recent studies that provide evidence that framing affects transfer. In the core of this paper, we evaluate how expansive framing may have promoted transfer for this particular student by evaluating five recently proposed explanations for how it may do so (Engle, Lam, Meyer & Nix, in press). We present relatively strong evidence for four out of the five explanations with data that we have collected in this target classroom. Finally, we close by proposing additional explanations for this students unexpectedly high rate of transfer that may be less related to the expansive framing of her instruction.
What is Expansive Framing?

Framing is the meta-communicative act of characterizing what is happening and how different people are participating in a given context (Bateson, 1972; Goffman, 1974; Tannen, 1993). For example, a teacher can frame a lesson as a one-time event of learning something that students are unlikely to ever use again, or as beginning to discuss an issue that students are likely to be actively engaging with throughout their lives. Our claim is that the first kind of framing, which we refer to as bounded, will tend to discourage students from later transferring what they learn, while the second, which we refer to as expansive, will tend to encourage students to transfer what they have learned. This claim is consistent with several prior proposals by situative and sociocultural theorists (Greeno, Smith & Moore, 1993; Laboratory for Comparative Human Cognition, 1983; Pea, 1987) who argue that it is not just the physical aspects of contexts that matter for transfer (Catrambone & Holyoak, 1989; Spencer & Weisberg, 1986), but also how social interactions frame contexts as particular kinds of social realities (Gee & Green, 1998; Searle, 1995). There are several aspects of learning contexts that can be framed in ways that can affect transfer (see Figure 1). First, a learning setting may be framed in an expansive manner by extending it to include past and future times, different places, and additional participants. In contrast, a bounded framing of a learning setting may constrain it to the present time, place, and just a few participants. Furthermore, in an expansive framing of roles, learners may be positioned as active contributors and authors of their own ideas. In contrast, in a bounded framing of roles, learners may be positioned on the periphery of a learning context, where they are expected to report on the ideas of others. In the biology class we studied, the teacher exhibited expansive framing of all of these aspects with great frequency (Meyer, Mendelson, Engle & Clark, 2011). With regards to settings, the teacher regularly referenced other times like the beginning of the school year and later in life; other places like college and afterschool environments; and other participants like friends, family members, and scientists. For example, during a lesson on cells, the teacher explained the following:
ISLS
40
ICLS2012
Now all this, all of this cell stuff, is kind of described on its own as the cell theory. And every biology student in the world, okay, maybe not the whole world but [inaudible] learns the cell theory. So whether you're going to be in this class, or you're moving to Texas, or you're going to be going to Berkeley, eventually someday you're gonna walk in, and they're gonna say, "Hey, tell me about the cell theory," so here you go. Here, the teacher explicitly mentioned times other than the present by saying eventually someday, he also referred to places outside of the classroom such as the world, Texas, and Berkeley, and involved other participants by including biology students and they. This type of expansive framing of settings was frequent in his instruction. Furthermore, with respect to roles, the teacher routinely expected students to author their own ideas, credited students for ideas that they shared with the class, and held them accountable for their contributions. A common activity he had students engage in was a class discussion of double multiple-choice questions that he projected on the board. During these activities, students were asked to individually choose an answer before the class was polled for their choices by a raise of hands. If discrepancies arose from the poll, students were expected to respectfully argue with each other for their answer choices until all students understood why one of the answers was the best choice. While these discussions took place, the teacher would stand quietly to the side unless he believed his input was needed to guide the students conversation. Typically, the teachers only contributions to the discussion were to ensure that the students directed their conversations at each other and not at him, credit a student for a statement he/she made and ask others to respond to it, and encourage individual students to argue for the answers they chose.
Evidence that Expansive Framing Promotes Transfer

A growing series of studies has empirically investigated connections between the framing of contexts and transfer. Several classroom case studies have provided evidence that transfer may be affected by framing in realworld learning environments (Engle, 2006; Engle Meyer, Clark, White & Mendelson, 2010; Hammer et al., 2005; Mendelson, 2010). Additionally, two experiments have systematically tested one or more aspects of this hypothesis, with results consistent with a causal connection between expansive framing and transfer (Engle, Nguyen & Mendelson, 2011; Hart and Albarracin, 2009).
Figure 1. How different aspects of learning environments can be framed expansively.
Five Proposed Hypotheses for How Expansive Framing May Foster Transfer
Although prior research has shown that framing contexts differently can affect transfer, it has not explained how this occurs. In Engle et al. (in press), we proposed five explanations--not mutually exclusive--for how expansive framing may promote transfer: 1) Connecting settings with each other encourages transfer during learning when students expect they will later need to transfer what they are learning and in response may be more likely to prepare for this possibility (e.g., Bereiter, 1995; Brown, 1989). 2) Connecting settings also encourages transfer during potential transfer contexts, when students view prior content as continuing to be relevant and so are reminded of it (e.g., Leander, 2001; Pea, 1987; Ross, 1984). 3) By connecting to prior settings and by positioning students as authors, students are likely to view their own prior knowledge as relevant to current learning, encouraging them to transfer-in more of their prior knowledge during learning, as they construct new understandings, which can enhance the quality of learning and therefore transfer (e.g., Bransford, Brown & Cocking, 1999; Schwartz, Bransford & Sears, 2005).
ISLS
41
ICLS2012
4) Framing students as authors of particular content may foster student accountability to that content, making students more likely to use this content in transfer contexts (e.g., Greeno, 2006; Jacoby & Gonzales, 1991). 5) If authorship becomes a general practice that students regularly participate in, it may promote the practices of generating new knowledge and engaging in adaptive problem solving (e.g., Boaler, 2002; Greeno, 2006; Hatano & Oura, 2003; Schwartz et al., 2005). In this paper, we investigate which of these five potential explanations are helpful in understanding why and how one particular student had an unexpectedly high rate of transfer.
A Somewhat Surprising Case of Transfer

Elaine was a Hispanic 9th grader who qualified for the free and reduced lunch program, and whose parents had a high school level education. Her 8th grade standardized test score in science was not particularly strong (basic, as opposed to proficient or advanced). For the first month of school she often did not attend class consistently nor do her biology homework, earning a D the first semester. She was also a student who had a long history of getting into trouble in school. She was identified by the high school as a student at risk and also reported everybody assumes that I have straight Fs. Thus, Elaine did not fit the typical profile of a student that would be expected to demonstrate strong learning, let alone transfer. We assessed Elaine and her classmates transfer of the concept of concentration gradient, as it was central to the course. In particular, we had students explain scenarios about osmosis and diffusion (Odom & Barrow, 1995) that were designed to be more or less analogous to what students had learned in class (Gick & Holyoak, 1983). Scenarios that were analogous were included in a knowledge assessment given after the class unit on osmosis and diffusion a month into the second semester. Scenarios that were less analogous were included in the transfer assessment administered 3 months later. Elaine scored surprisingly high on the transfer test in comparison to the rest of the class. She transferred 78% (z=+1.22) of the knowledge elements she had known on the knowledge assessment while the class as a whole transferred a mean of 57% of what they knew. In addition, her score on the transfer test was 67% higher than her score on the knowledge assessment, suggesting she used more knowledge on the more challenging transfer questions than on the more straightforward questions in the knowledge assessment.
Data Sources for Explaining Elaines Transfer

1. Student framing survey: Conducted midway through the school year, it used open-ended and Likert-style questions to measure the degree to which students were aware of their teachers expansive framing of their biology class and the degree to which they were responsive to his expansive framing in their own actions. 2. Video and audio recordings: One video camera captured a view of the students and the other, a view of the instructor. The audio recorders captured conversations among pairs and groups of students during class activities. 3. Instructional materials and graded student work: These were collected to supplement the video and audio recordings, as well as to track individual students performances. 4. Student interviews: Conducted at the end of the school year, Elaine and other students were asked to update and elaborate on their responses to the framing survey and share their perspectives on two recorded excerpts from class in which the teacher explained aspects of his expansive framing of settings and roles. 5. Teacher interviews: We draw on excerpts in which he discussed Elaine.
Explaining How Expansive Framing May Have Led to Elaines Transfer

We first show that Elaines higher-than-expected transfer was paralleled by high awareness and responsiveness to the teachers expansive framing. We then use her case to investigate evidence relevant to each of the five proposed hypotheses for how framing can foster transfer.
Elaine was especially aware of and responsive to the teachers expansive framing
Survey results indicated that Elaine perceived and responded to the teachers expansive framing to a greater extent than her classmates (Figure 2). On average, students in this biology class scored 3.66 when asked questions aimed at the degree to which they were aware of their teachers expansive framing, and 3.33 when asked about how they responded to this framing (on a five point scale from 1 [never] to 5 [all the time]). However, Elaine was aware of the teachers expansive framing with an average score of 4.42 and responded to the framing with an average score of 3.99, both about one standard deviation unit above the class means. Thus, Elaine appears to have been more aware of and responsive to the teachers expansive framing than the average student in her biology class. Specifically, many of Elaines responses to survey questions about her awareness of particular aspects of framing were more than a full standard deviation above the average score of her peers: the teachers framing of place (4.17, z=+1.07), participants (4.40, z=+1.56), and roles (5.00, z=+1.32). She also indicated particularly higher responsiveness to expansive framing related to time (4.88, z=+1.50) and place
ISLS
42
ICLS2012
(4.33, z=+1.12). According to the prior studies cited above, these results predict that Elaine would have a higher rate transfer of than her classmates.
Figure 2. Student awareness of and responsiveness to the teacher's expansive framing: the class vs. Elaine
Explaining Elaines case by examining evidence relevant to each of the five hypotheses 1. The student expects future transfer, so prepares for it by studying more or better
There is only partial evidence that this explanation helps explain Elaines enhanced transfer as we have strong evidence that Elaine expected she would need to transfer what she had learned in the future, but weak evidence that she prepared for transfer by studying more or better. When asked in the survey about the frequency with which the teacher told the students what they are learning may be useful to them at various times in the future (the next day, the next few weeks, the next year, and beyond the next year), Elaine frequently reported that this happened often or all the time (4.50, z = +0.97). Furthermore, she reported thinking all the time about how to apply what she learned at all of these times in the future (5.00, z = +1.17). On the other hand, evidence that Elaine strongly prepared for future transfer is weak. When asked about her studying habits she stated Im lucky if I look over my notes like, once, like I never study for anythingcause Im lazy or Im doing something else. However, evidence from the teacher and grade reports shows that Elaine completed more biology homework as the year progressed, gradually improving her grade. When asked specifically about turning in assignments, Elaine said, I kinda caught up now, cause um how I said [the teacher] kinda pushed me to do betterIm doing better in his class, in specific, so everythings kinda working out. Moreover, she also appeared to recognize the value in studying and doing homework, even though she did not regularly do these things herself. For example, when asked if she thought that completing the pre-lab assignments were helpful she responded like me, if you kinda wait til the last minute or try to copy it it doesnt help you at all, suggesting that she was aware of the fact that completing the assignments would have helped her had she prepared better. In summary, while Elaine may not have exhibited strong preparation for future transfer to the extent that we would have expected her to be, considering her successful rate of transfer, her class performance improved over the course of the school year and she appeared to be aware of the value of preparation for class.
2. Viewing what was learned before as having continued relevance in transfer situations
There is strong evidence that this explanation is significant to Elaines case as we found she viewed ideas from the past as continuing to be relevant. Based on survey and interview data, we found that Elaine noticed the teacher regularly referred to what the students had learned at various prior times and recognized that topics in class built upon each other: Its not always something brand new we always elaborate on what we know. So like the only time its really brand new is in the beginning of the year [Otherwise,] we always refer back to certain stuff.Like um, with mitosis and meiosis, we had to know that for evolution we always just kinda adds -- add something new, but it builds on what we already know. In this example, Elaine showed that she was aware that the content in class built upon prior lessons and that what she previously learned had continued relevance. Elaine also reported that she frequently drew on ideas from previous times in class. For example, Elaine once asked: I have a question....It just relates to sorta what we were doing last week, but you know how...we said that...an animal will blend in with like its environment? So, um, does that kinda happen with
ISLS
43
ICLS2012
plants and stuff too? Elaine drew on what she had learned before about animals to pose a new transfer question to the class about whether the same idea extends to plants, and then began engaging the class in answering her question.
3. Prior knowledge as author is relevant, so transfer-in more prior knowledge, including examples or generalizations that can enhance learning and therefore later transfer.
We also have evidence that this explanation is relevant to explaining Elaines transfer. Elaine reported that she appreciated participating in class, which in this particular case meant drawing on her prior knowledge to author her own ideas. As evidence, we saw Elaine bring in knowledge from outside of school to help her learn biology, which is necessary for future transfer. For example, when the teacher was introducing different cell parts and displayed a picture of a centriole (a long, tube-shaped structure), Elaine exclaimed it looks like a churro!, a similarly long and tube-shaped Hispanic pastry. By comparing the shape of centrioles to churros, Elaine brought in her cultural knowledge in ways that likely enhanced her memory of the structure of centrioles, allowing her to draw on it later. In one lesson, Elaine was also able to transfer-in her cultural knowledge to interpret the species name of a cougar, concolor: In Spanish, its with color, except that there is a space. Potentially, Elaine may generalize the idea that her cultural knowledge is applicable to learning contexts to help her remember concepts and interpret Latin words. We also have evidence of Elaine transferring-in prior knowledge that is not based on culture. During a lesson that began with the topic of animal traits, students became engaged in a discussion about plant characteristics that may deter predators from injuring them. Elaine contributed to the discussion by saying, so a cactus has thorns to stop animals from eating them, bringing in a specific example of cacti characteristics to help understand the general idea that plants have traits that help prevent them from being eaten by predators. This act of transferring-in additional examples and comparing them to each other can also directly enhance transfer (e.g., Gentner, Loewenstein & Thompson, 2003; Gick & Holyoak, 1983).
4. Students become publicly recognized as the author of particular content that they then become accountable for sharing
Elaines ability to transfer was likely affected by being regularly positioned as an author of ideas and questions during discussions. As a result, she was regularly held accountable for the biology content that she was authoring in class. For example, during a particular class discussion about chromosomes, Elaine shared with the class that she did not think human eggs and sperm have 46 chromosomes. The teacher followed her contribution to the discussion by asking the class what do you think about what Elaine said? In doing so, he publicly credited Elaine for her ideas and made them a part of the instruction. Later in the discussion, the teacher even asks Elaine to repeat what she had shared, re-emphasizing her ideas. Our survey results indicated that Elaine noticed that the teacher regularly credited her for coming up with important ideas in class and that it came with an expectation that she would continue to comment intelligently on what we are learning in class...all the time. Our data also reveal evidence that Elaine was recognized as an author and accountable for sharing her knowledge outside of class. During her interview, when she was asked for details about her survey response where she indicated that she talked about biology to those around her at home, Elaine elaborated: Just last night, my step-dad said that he was sick or something, and that -- something about amino acids and then I asked him Well, you know what that is? Like, how do you even know what that is? And um, he started laughing and he -- he told me, um, well how do you not know what that is... And he says, um, do YOU know what it is? And I was like yeah, we just learned that this year. And I kinda explained it him, like what I -- MY understanding of what they were and all that, and he just kinda stayed quiet and was like, mm hmm and shook his head like, yeah. In this example, Elaine was authoring her own ideas about amino acids, emphasizing MY understanding of what they were while her step-father held her accountable for sharing her understanding. This accountability appears and is likely to increase opportunities and the social expectation for students to continue contributing to conversations about the ideas that they have and the ideas that they have previously authored.
5. Authorship becomes a practice so students regularly adapt what has been learned to earnestly attempt or propose novel problems.
There is strong evidence that Elaines practice of being an author may have contributed to her high transfer results. First, it is clear that over time Elaine strongly adopted authorship as a practice. By the end of the year, she said I participate as much as I can. With respect to being generative, Elaine always made a genuine attempt to answer all of the questions on the transfer assessment while other classmates occasionally left questions blank or responded with I dont know.
ISLS
44
ICLS2012
Elaines role as an author is best summarized by an excerpt from the teacher during a class discussion several months into the school year when Elaine proposed an incorrect answer to a question: Teacher: I think without question I can say the reason Elaine is doing so much better in here is because she is willing to say No, I think its A because its what a gamete is and that takes some guts because shes not sure but its what she thinks and thats really good. Is that the right thing to say though? I mean, is that a correct statement? Students [in unison]: No Teacher: No, its not a correct statement, ok, but shes not over there crying because she was wrong, but what it did then was give ... everybody else who thought it and get a chance to correct her which reinforces the right answers for them, allows Elaine to rethink through her response and change her mind. Shes more likely to remember that now and everybody, now everybody benefits because Elaine had, you know, was able to say something. From this transcript, it is clear that the teacher recognized that Elaine earnestly attempted to answer questions and shared with the class even when she was uncertain of her answers. In our survey, Elaine also stated that she shared what she thought in class, even though she was not sure if she was right, all of the time (5, z= 1.5). She further reported being able to say what I feel and always shared her ideas in class even if she did not know if her answer was correct. These survey data suggest that Elaine attempted problems presented in class and was always willing to share her ideas. In addition to earnestly attempting novel problems in class, video data captured many instances of Elaine proposing her own novel problems by adapting her prior knowledge to the current learning context. For example, during a lesson when students discussed a visualization of equilibration, Elaine posed the question, Does it stay at equilibrium after it reaches equilibrium? In presenting this question to the class, Elaine extended the discussion beyond initial attempts to understand the definition of equilibrium, to the process of how equilibrium is reached and what occurs after that point. The teacher subsequently supported Elaines proposal and encouraged students to propose similar questions by responding to Elaines question saying That, ladies and gentlemen, is a ten thousand dollar question. Smart kids dont give the best answers, they ask the best questions. So in groups, after equilibrium is reached, does it stay that way? The teacher credited Elaine as the author of a smart question, and used her question as a point for further discussion and instruction. Occasions such as this, where Elaine proposed novel problems built from her understanding of the content, occurred often throughout the school year. We believe that this practice of being an author, and attempting or proposing novel problems in class, ultimately enhanced her ability to transfer.
Other hypotheses for how Elaines transfer may have been supported
In addition to these 5 hypotheses, we also considered how data from Elaines case might suggest other hypotheses for how her transfer was supported that may be less related to the effects of the teachers expansive framing. The first hypothesis suggested by Elaines data was that she learned more and more deeply in the class because of the quality of the teaching she received, which then allowed her to transfer more of what she learned (Bransford, Brown & Cocking, 1999). Elaine specifically referred to the quality of the teaching in her biology class in an open-ended survey question that asked her to describe her biology class and teacher to a new student: [He] always finds fun ways for us to learn all the material.. Everything he teaches us is a pretty hard subject and yet this is the class Ive learned the most in. Its complicated, but [the teacher] does a great job of explaining and breaking things down until we understand. Consistent with Elaines comments, we can report that classes taught by this teacher, including this section, generally received higher scores both on standardized tests and in-class conceptual tests as compared to the other biology teachers in this same school. However, while it is clear that the teachers success at helping Elaine learn the material was a necessary prerequisite for her transfer, this does not help explain why we observed more transfer with her than other students, as other students also presumably benefited from the high quality of in-class instruction. So by itself, this hypothesis is not sufficient for explaining Elaines higher than usual rate of transfer. We then investigated whether there were any aspects of support that Elaine received that were different other students and found that her engagement in class was supported more strongly, and eventually led to greater engagement when it was originally much weaker than most students. In particular, Elaine had been identified as a potential at-risk student, leading to her biology teachers decision to visit her home and better understand her out-of-school constraints and resources. This led to him regularly taking her aside in school to encourage and push her, which Elaine said helped her do better in class as we illustrated in discussing Elaines study practices around Hypothesis 1. When we asked exactly how her biology teacher helped her improve, Elaine explained:
ISLS
45
ICLS2012
[By] him always reminding me. Having somebody tell me that I have to do this, that I have to do it for myself. Reminding me of the consequences, um, just telling me, like, that Im capable of doing better. [So I] get home and dont forgetthat I actually have homework to do. So the fact that Elaine began doing more schoolwork and therefore learning more that she could then transfer, we attribute to the teachers extra efforts to support her engagement with school. However, while this hypothesis would explain why Elaine had learned more material to transfer, it does not fully explain why she transferred a higher proportion of material than other students. Our final hypothesis centers on the often tricky and subtle issue of student motivation. From Elaines interview we found out that she was particularly interested in taking this class as shed heard from older students about its culminating forensics unit, a career she was seriously considering after a school project in the prior year. In addition, she expressed a particular interest in animals and noted that this also made her more interested in biology. Finally, she reported particularly enjoying this biology class because of the teachers sense of humor: I really enjoy being in [the teachers] class.[He] always finds fun ways for us to learn all the material Hes very funny too. I come in this class knowing that by the end of the one-hour period, Ill have a huge grin on my face. Both Elaines prior individual and current situational interests (Hidi & Renninger, 2006) as well as the future utility value she perceived with the content for her future (Hulleman & Harackiewicz, 2009) could be expected to support Elaines engagement with the class as well as the kinds of re-engagement with the content at later times (Renninger, 2000) that become transfer. However, at first glance this hypothesis of Elaines strong motivation helping to foster transfer seems inconsistent with the fact that Elaine needed to be reminded to do homework and reported never studying for tests. In the end, though, we do not think this is the case as we learned through the teachers home visit that (a) doing schoolwork at home often conflicted with Elaines family responsibilities, and (b) completing out-of-class assignments takes more time and effort than the kinds of transfer we observed Elaine evidencing that involved using more of her knowledge on required in-class transfer assessments, making comments in class, or bringing up what she had learned about biology with her friends or family when relevant to ongoing activity. So we believe that it may have been Elaines individual and situationally supported interests that interacted positively with the teachers expansive framing and other support that fostered her transfer.
Conclusion
Based on our analyses of Elaines case of transfer, we found relatively strong support for four of the five proposed explanations for how expansive framing fostered transfer for a student that was not expected to demonstrate strong transfer. We have also developed additional hypotheses that may further contribute to her high rate of transfer such as the quality of instruction she received, teacher support, and personal motivation. In our presentation, we will further describe how the processes suggested by each explanation unfolded throughout the school year, drawing on data to describe how Elaines student activities changed over time. Moreover, we believe that the explanations are not mutually exclusive. Many of the examples we have provided under each of the five explanations are closely related to and even demonstrate one or more of the other four explanations. Therefore, we aim to further develop our theories about how these five explanations are related.
References
Bateson, G. (1972). A theory of play and fantasy. In Steps to an ecology of mind: Collected essays in anthropology, psychiatry, evolution, and epistemology (pp. 177193). New York: Ballantine. Bereiter, C. (1995). A dispositional view of transfer. In A. McKeough, J. Lupart, & A. Marini (Eds.), Teaching for transfer: Fostering generalization in learning (pp. 2134). Mahwah, NJ: Erlbaum. Boaler, J. (2002). Experiencing school mathematics: Traditional and reform approaches to teaching and their effect on student learning. Mahwah, NJ: Erlbaum. Bransford, J., Brown, A. L., & Cocking, R. (Eds.). (1999). Learning and transfer. In How people learn: Brain, mind, experience, and school (pp. 3966). Washington, DC: National Academy Press. Brown, A. L. (1989). Analogical learning and transfer: What develops? In S. Vosniadou & A. Ortony (Eds.), Similarity and analogical reasoning (pp. 369-412). Cambridge, UK: Cambridge U. Press. Engle, R. A. (2006). Framing interactions to foster generative learning: A situative account of transfer in a community of learners classroom. J. of the Learning Sciences, 15(4), 451-498. Engle, R. A., Lam, D. P., Meyer, X. S., & Nix, S. (in press). How does expansive framing promote transfer? Several proposed explanations. Educational Psychologist. Engle, R. A., Meyer, X., Clark, J., White, J. & Mendelson, A. (2010). Expansive framing and transfer in a high school biology class: Hybridizing settings and promoting connections within a larger learning community. In R. A. Engle (organizer), Applying new mechanisms and conceptualizations of the
ISLS
46
ICLS2012
transfer-of-learning to science classrooms: The dynamic role of contexts and interactions. Symposium presented at the National Association for Research in Science Teaching Annual International Conference, Philadelphia, PA. Engle, R. A., Nguyen, P. D. & Mendelson, A. (2011). The influence of framing on transfer: Initial evidence from a tutoring experiment. Instructional Science, 39(5), 603-628. Gentner, D., Loewenstein, J. & Thompson, L. (2003). Learning and transfer: A general role for analogical encoding. J. of Educational Psychology, 95(2), 393-408. Gick, M. L. & Holyoak, K. (1983). Schema induction and analogical transfer. Cognitive Psychology, 15, 138. Goffman, E. (1974). Frame analysis: An essay on the organization of experience. New York: Harper & Row. Greeno, J. G. (2006). Authoritative, accountable positioning and connected, general knowing: Progressive themes in understanding transfer. J. of the Learning Sciences, 15(4), 537-547. Greeno, J. G., Smith, D. R., & Moore, J. L. (1993). Transfer of situated learning. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 99127). Norwood, NJ: Ablex. Hammer, D., Elby, A., Scherr, R. E., & Redish, E. F. (2005). Resources, framing, and transfer. In J. Mestre (Ed.), Transfer of learning: Research and perspectives. Greenwich, CT: Information Age Publishing. Hart, W. & Albarracin, D. (2009). What was doing vs. what I did: Verb aspect influences memory and future actions. Psychological Science, 20(2), 238-244. Hatano, G. & Oura, Y. (2003). Commentary: Reconceptualizing school learning using insight from expertise research. Educational Researcher, 32(8), 26-29. Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Ed. Psychologist, 41, 111 127. Hulleman, C. S. & Harackiewicz, J. M. (2009). Promoting interest and performance in high school science classes. Science, 326, 1410-1412. Jacoby, S. & Gonzales, P. (1991). The construction of expertnovice in scientific discourse. Issues in Applied Linguistics, 2(2), 149181. Laboratory of Comparative Human Cognition. (1983). Culture and cognitive development. In P. H. Mussen (Ed.), Handbook of child psychology: Vol. 1. History, theory and methods (pp. 295356). New York: Wiley. Leander, K.M. (2001). This is our freedom bus going home right now: Producing and hybridizing space-time contexts in pedagogical discourse. J. of Literacy Research, 33(4), 637-679. Maxwell, J. A. (2004). Using qualitative methods for causal explanation. Field Methods, 16(3), 243-264. Mendelson, A. (2010). Using online forums to scaffold oral participation in foreign language instruction. L2 Journal, 2(1), 23-44. Meyer, X., Mendelson, A., Engle, R.A. & Clark, J (2011). Expansive framing and transfer in a high school biology class: Promoting connections within and beyond the classroom. Manuscript submitted for publication. Pea, R. D. (1987). Socializing the knowledge transfer problem. Intl. J. of Educational Research, 11, 639663. Renninger, K. A. (2000). Individual interest and its implications for understanding intrinsic motivation. In C. Sansone & J. M. Harackiewicz (Eds.), Intrinsic and extrinsic motivation: The search for optimal motivation and performance (pp. 373-404). San Diego, CA: Academic Press. Ross, B. H. (1984). Reminding and their effects in learning a cognitive skill. Cognitive Psychology, 16(3), 371 416. Schwartz, D. L., Bransford, J. D., & Sears, D. (2005). Efficiency and innovation in transfer. In J. Mestre (Ed.), Transfer of learning: Research and perspectives. Greenwich, CT: Information Age Publishing. Siegler, R. S. (2006). Microgenetic analyses of learning. In W. Damon & R.M. Lerner (Series Eds.) & D. Kuhn & R.S. Siegler (Vol. Eds.), Handbook of child psychology: Volume 2: Cognition, perception, and language (6th ed., pp. 464-510). Hoboken, NJ: Wiley. Tannen, D. (1993). Introduction. In D. Tannen (Ed.), Framing in discourse (pp.313). New York: Oxford U. Press.
ISLS
47
ICLS2012
How metacognitive awareness caused a domino effect in learning

Kristin Schmidt, University of Freiburg, kristin.schmidt@ezw.uni-freiburg.de Andreas Lachner, University of Gttingen, andreas.lachner@med.uni-goettingen.de Bjrn Stucke, University of Gttingen, bjoern.stucke@med.uni-goettingen.de Sabine Rey, University of Gttingen, srey@med.uni-goettingen.de Cornelius Frmmel, University of Gttingen, cfroemmel@med.uni-goettingen.de Matthias Nckles, University of Freiburg, matthias.nueckles@ezw.uni-freiburg.de Abstract: The scope of this study was to investigate effects of a metacognitive awareness tool on self-regulated learning. In a series of web-based self-monitoring protocols, participants (N=49) recorded their class preparation. In the experimental condition, they received individual scores of dilatory behavior mirrored in a line chart. We were interested in whether potential advantages of the line chart can be traced back to non-specific signaling of dilatory behavior or to specific individual feedback. Therefore, we compared the experimental group with two control groups: without visualization or with random visualization. The random visualization was expected to have a signaling but no feedback effect. Results supported the effects of signaling and individual feedback. Participants with random or veridical visualizations were more likely to engage in self-reflection, but only those who received individual feedback were more likely to reduce their dilatory behavior and improved selfregulated learning.
Introduction
Although there is an increasing demand for self-regulation strategies in learning, many students have problems in regulating themselves (Zimmerman, 2002). Consider the case of Tony, a university student of medicine, who is in the last year of his clinical studies. An important exam on neurology is three weeks away, and Tony has not started studying yet. To calm himself, he tells himself that enough time is left and studying too long before the exam would cause him to forget important facts. In this way, he keeps on postponing his tasks, although he experiences anxiety due to the growing pressure. Two days before the exam, Tony starts repeating the facts without understanding anything because there is no time for deeper understanding. Anyway, he argues, he learns best under pressure. However, he does not set any learning goals except of studying, and does not reflect his preparation. In some cases, students like Tony truly receive satisfying grades for their exam performance, but in the majority of cases, these students suffer from dilatory behavior and often fail to achieve their goals. Taken together, Tony shows deficient self-regulation. One possibility to improve self-regulation is to guide students metacognitive awareness to processes that need to be improved (Schraw, 1998), for example by using time management techniques. Metacognitive awareness is expected to facilitate self-reactivity, which is an improvement of behavior without further treatments (Zimmerman, 2002). In the present article, we want to discuss the role of metacognitive awareness in self-regulated learning and present an empirical study on the effectiveness of a metacognitive awareness tool.
Dilatory Behavior Failure of Self-Regulation

A ubiquitous problem of self-regulation is students irrational postponing of important tasks, which is known as dilatory behavior (Lay & Silverman, 1996; Steel, 2007). It often results from ineffective metacognitive strategy use when high standards of self-regulation are required (Wolters, 2003). Many learners delay the beginning of unpleasant but important learning tasks to an unspecified later date. It seems to be difficult for them to motivate themselves and shield motivation against interruptions. Thus, they seem to lack volitional control (Boekaerts, 1996). Several studies showed that dilatory behavior is negatively related to self-regulated learning. It interferes with goal-setting (Lay & Schouwenburg, 1993), self-efficacy (Wolters, 2003; Ferrari, Parker, & Ware, 1992), strategy use (Wolters, 2003), and learning outcomes (Lay & Schouwenburg, 1993; Klassen, Krawchuk, & Rajani, 2008). Dilatory behavior is influenced by a personal tendency to delay tasks (procrastination as a personal trait), task characteristics and a learners repertoire of self-regulation strategies (Steel, 2007). Thus, the tendency to procrastinate needs not necessarily result in dilatory behavior, because strategies for self-regulation are expected to mediate personal and contextual characteristics and behavior (Pintrich, 2004). The correlation between trait procrastination and dilatory behavior (Stainton, Lay, & Flett, 2000: r = .51) indicates that trait procrastination and dilatory behavior are related but not the same. Thus, trait procrastination could be compensated by strategies for self-regulation.
Metacognitive Strategies
ISLS
48
ICLS2012
To self-regulate learning processes successfully and to avoid problems in motivation and understanding, sophisticated metacognitive strategies are needed (Schraw, 1998; Zimmerman, 2000). These strategies encompass planning, monitoring and self-reflection of the learning process (Schraw, 1998; Zimmerman, 2002). To explain the relations between metacognitive, motivational, and behavioral aspects of self-regulated learning in a more systematic manner, different researchers draw on prescriptive models of self-regulated learning (Pintrich, 2004; Winne & Hadwin, 1998; Zimmerman, 2000). According to Zimmerman (2002), self-regulated learning involves the selective and adaptive use of specific metacognitive strategies to monitor and regulate cognitive processes. These cognitive processes ensure understanding and recall of knowledge as well as high degrees of motivation. Some of these metacognitive strategies are goal setting, self-motivation, strategy selection and adaptation, comprehension monitoring, and self-reflection of products and efficiency (Schraw, 1998; Zimmerman, 2002). Each of these prescriptive models of self-regulated learning includes a planning phase, an implementation phase, and a reflection phase, each depending on each other. Planning refers to the selection of personal standards that should be achieved during the learning episode and the selection of appropriate cognitive strategies in relation to the standards. During the implementation of cognitive strategies, self-monitoring of performance and understanding is essential (Zimmerman, 2000). Therefore, learners deliberate observation of the learning process is necessary in order to recognize discrepancies between the observed learning process and the personal ideal of the learning process. Afterwards, the learners reflect upon these discrepancies (Winne & Hadwin, 1998). Self-reflection includes an attribution of success and failure to internal and external responsibilities (Zimmerman, 2000). In doing so, learners can deduce consequences for later learning episodes to avoid making the same mistakes. Thus, via metacognitive regulation, learners have the possibility to improve their learning self-reactively (Zimmerman, 2002).
Volitional Control
Strongly related to metacognitive regulation is the regulation of motivation. This regulation is also called volitional control (Boekaerts, 1996; Corno, 1986). Volitional control is expected to prevent postponements and to shield studying against interruptions. The relation between metacognitive strategies and volitional control during the learning process can be illustrated for goal-setting. Goal setting is a metacognitive strategy that is built around a reference to decide whether the learning behavior is appropriate (Zimmerman, Bandura, & Martinez-Pons, 1992). There is considerable evidence for increased academic success related to specific learning goals (Locke & Latham, 2002; Schunk, 2001). Specific, proximal goals facilitate the selection of appropriate cognitive strategies, self-monitoring of performance and strategy regulation, and the assessment of goal achievement. Additionally, goal setting is important for motivation. Personally relevant learning goals contribute to high motivation to achieve the task. Specific, challenging but obtainable learning goals support self-efficacy, which is a central source for motivation (Locke & Latham, 2002; Schunk, 2001). Vice versa, a higher motivation to deal with a particular subject could lead to more challenging learning goals, and in this way to more learning effort, deep-learning strategies (Sankaran & Bui, 2001) and better learning outcomes (Zimmerman et al., 1992; Boekaerts, 1996).
Reducing Dilatory Behavior by Supporting Self-Monitoring

Self-monitoring provides learners with data about their learning process (e.g., What goals are appropriate?), and the learner themselves (e.g., What motivates me?) (Zimmerman, 2002). Hence, self-monitoring establishes the base for increased metacognitive knowledge that may help to improve future studying (Zohar & Peled, 2008). Via self-monitoring, students become aware of ineffective or inappropriate behavior and try to better fit their personal standards by changing their behavior. Several studies impressively showed that improved self-monitoring can improve various learning tasks, like reading (Joseph & Eveleigh, 2011), writing (Cho, Cho, & Hacker, 2010), and academic performance (Chang, 2007). Additionally, self-monitoring can improve motivational beliefs (Chang, 2007). Thus, accurate, deliberate and critical self-monitoring is essential for self-reactive improvements of learning behavior (Boekaerts, 1996). Self-monitoring of dilatory behavior could be improved by guiding students awareness to dilatory behavior and by supporting the accuracy of self-monitoring. This self-monitoring can be improved by asking learners to record their learning behavior in self-monitoring protocols (Lan, 1996; Schmitz & Wiese, 2006). The self-monitoring protocols require students to monitor their learning behavior in order to answer questions about it. These questions can guide a learners attention to aspects that should be improved. Monitoring dilatory behavior provides students with data that conflict with the normative ideal of studying that students are expected to have. The discrepancies between observed learning behavior and personal standards could lead to a selfreactive improvement of studying. However, self-regulated learning is a very complex activity that involves numerous processes. Therefore, students have the possibility to choose data concordant with their self-image and monitor and reflect primarily on positive aspects of their studying (Lewicki, 1983). This allows for a protection of their self-image, but inhibits improvements in learning behavior. To avoid biased self-monitoring,
ISLS
49
ICLS2012
a tool would be helpful that guides students awareness to rather problematic aspects of learning, for instance to dilatory behavior. To support students deliberate self- monitoring and metacognitive awareness of their dilatory behavior, we investigated the effectiveness of a graphical visualization that provided learners with data about their own dilatory behavior. Awareness tools are often used to facilitate computer-supported collaborative learning (CSCL). They sensitize students to the collaborative setting. Furthermore, specific information about the collaborative partners activities helps students to calibrate their own learning behavior in accordance with their collaboration partners behavior (Dehler, Bodemer, Buder, & Hesse, 2011).
Research questions of the current study

However, we were interested in whether web-based metacognitive awareness tools can also be used to improve ones own learning processes by decreasing dilatory behavior (1) and increasing self-reflection (2). Additionally, we were interested in whether such improvements are related to motivation and cognitive strategy use (3) and whether this affects learning outcomes (4). More precisely, we investigated the effects of systematic feedback by an individual visualization of dilatory behavior within self-monitoring protocols. Every week, students recorded their class preparations in a web-based self-monitoring protocol. To improve the effects of self-monitoring protocols, we included a graphical visualization of dilatory behavior. We expected the visualization to boost deliberate self-monitoring for two reasons. The signaling effect (or eye catching effect) of the traffic light colors could sensitize participants for dilatory behavior and thus cause increased self-reflection. Additionally, individual feedback in a veridical line chart could help students to estimate their behavior more accurately and thus facilitate selfreactivity, which helps to decrease the degree of dilatory behavior. To investigate whether the effect of the veridical visualization could be traced back to general awareness or to individual feedback, we compared the experimental condition with a no-visualization and a random condition. This random group received the same type of visualization with traffic light colors but with a random line chart. In this way, we could separate out the effects of the signaling effect of the traffic light colors, irrespective of the veridical values in the line chart from individual feedback.
Method
Participants, Design and Procedure
49 medical students (freshmen) with an average age of 20.9 years (SD = 2.17) participated in the study. The proportion of females was 67%. During an anatomy course (12 weeks, grades were not relevant for the preliminary medical examination), participants were asked to record their class preparation and homework in a web-based self-monitoring protocol once a week. The participants were randomly assigned to one of three experimental conditions: a veridical condition, a random condition, or a no-visualization condition. Students in the veridical condition received their average score of dilatory behavior ratings mirrored back in an individual line chart. The random group received the same visualization but with random values. The no-visualization group received no chart. During the self-monitoring period, students wrote 12 protocols at most. Before and after the self-monitoring period, students took part in a knowledge test that included declarative and conceptual knowledge tasks. In the study, dependent variables encompassed learning goals, motivation, dilatory behavior, strategy use, and learning outcomes.
Self-monitoring protocols
The self-monitoring protocols were realized in a web-based learning management system. Thus, to make their entries, students only needed Internet access. The self-monitoring protocols consisted of different sections related to models of self-regulated learning. Except for the learning goals, participants had to estimate their agreement with the items on a 5-point Likert scale ranging from low to high agreement. First, participants assessed whether they achieved their learning goals in the last week. Furthermore, they estimated the degree of their dilatory behavior based on four items (e.g. I often delayed important tasks to a later date.), which were adapted and translated from a dilatory behavior questionnaire (Lay & Silverman, 1996). Second, participants estimated their cognitive and metacognitive strategy use. The items were slightly adapted and translated from the MSLQ (Pintrich, Smith, Garcia, & McKeachie, 1993). As cognitive strategies, we collected self-reported use of elaboration strategies (e.g., I thought about examples that illustrate abstract learning content.), organization strategies (e.g., I tried to structure the learning material.), and rehearsal strategies (e.g., I repeated the learning materials more than once.). The items of the metacognitive strategy scale encompassed planning (e.g., Before studying, I thought about how to organize the materials.), monitoring (e.g., While studying, I monitored whether my learning strategy use is efficient to achieve my goals.), and self-reflection (e.g., I thought about possibilities to optimize my studying.).
ISLS
50
ICLS2012
Third, participants defined up to five learning goals for the following week and estimated their motivation to achieve these goals (e.g., The topic of learning content is relevant for me.). Learning goals were categorized in terms of their specificity level either as non-specific or specific. Inter-rater reliability was good ( =.78.). Non-specific goals neither included concrete topics nor strategic planning (e.g., studying anatomy). Accordingly, students might have problems in deciding whether they had reached these goals. It was presumably easier for students to assess whether they had actually reached specific goals. Specific goals included a concrete topic and strategic planning (e.g., repeating the organs of the abdomen). Fourth, to compare knowledge gains, a knowledge test was administered as part of the first and the last self-monitoring protocols. It contained several declarative knowledge and conceptual knowledge tasks. In declarative knowledge tasks, students were asked to mention facts about a particular topic (e.g., Name the muscles of the human face.). In conceptual knowledge tasks, students were asked to explain complex medical processes (e.g., Explain the follow-ups of a pancreas excision for patients.).
The graphical visualization

Participants in the veridical condition received their averaged responses of dilatory behavior items mirrored back in an individual line chart that expanded on a weekly basis. Thus, when students filled out the protocol they were confronted with their scores of dilatory behavior since the beginning of the study. The visualizations included the course of dilatory behavior since the beginning of the study, and in accordance with traffic lights, a classification into low (green), medium (yellow), and critical (red) values of dilatory behavior (see Figure 1). Participants in the random condition received computer-generated random values that could take values between 1 and 5. The participants in that condition were debriefed after the end of the study.
You tend to postpone your tasks.
Your time management is fine.
Figure 1. Example of a visualization of dilatory behavior
Results
To obtain mean scores, we averaged for each student across her/his self-monitoring protocols of the first and second week, the third and fourth week, the fifth and sixth week, and so on. By using average scores, we improved the reliability of a participants individual scores and we avoided an unnecessary loss of data. As we analyzed the protocol data with repeated measures analyses of variance, students who had one or more missing journal entries otherwise would have been completely excluded as cases.
Reduction of Dilatory Behavior

To investigate the course of dilatory behavior, we conducted an analysis of variance with repeated measurement. The individual mean scores of dilatory behavior items were treated as within subject factor, and experimental condition (veridical, random, no-visualization) as between subject factors. The main effect of measurement was significant, F(1, 30) = 5.12, p < .05, partial = .15. Hence, dilatory behavior changed during the surveyed time span. The main effect of experimental condition failed to reach statistical significance F(2, 30) = 0.40, n.s.. Nevertheless, the interaction effect between measurements and experimental condition was significant, F(2, 30) = 3.62, p < .05, partial = .19, indicating that the courses of dilatory behavior differed between the experimental conditions. The courses of dilatory behavior indicated that students in the veridical condition learned more consistently (see Figure 2, left). To investigate this assumption, we computed the intra-individual variance of each participants dilatory behavior that were subjected as dependent variable to a one-factorial analysis of variance with experimental condition (veridical, random, no-visualization) as independent variable. The results indicated that the variance of dilatory behavior could be reduced in the conditions with graphical support, especially in the veridical condition, F(2, 43) = 3.93, p < .05, partial = .16. A Tukey post hoc test revealed that the difference between the no-visualization and the veridical condition was significant. Thus, mirroring students their self-reported dilatory behavior resulted in more consistent studying.
Improvement of Self-Reflection
To investigate whether individual feedback provided by a graphical visualization of dilatory behavior improved self-reflection, we conducted an analysis of variance with repeated measurement. We included the mean degrees of strategy use as within subject factor and the experimental condition (veridical, random, no-visualization) as between subject factor. The main effect of measurement time failed statistical significance, F(1, 30) = 3.99, p =
ISLS 51
ICLS2012
.06. The main effect of experimental condition, however, was significant F(2, 30) = 3.84, p < .05, partial = .20, as well as the interaction effect of experimental condition and measurement time, F(2, 30) = 4.24, p < .05, partial = .22. A Tukey post hoc test revealed that participants in the no-visualization condition reported to use less self-reflection strategies than participants in the veridical or random condition. Thus, participants with a visualization tool (veridical and random) recorded a more constant and higher level of self-reflection than the no-visualization group (see Figure 2, right).
2,7
4,5
Degree of dilatory behavior
2,6 2,5 2,4 2,3 2,2 2,1 2 1,9 1,8 1,7
control condition
Degree of self-reflection
random condition 4,0 feedback condition 3,5
random condition
3,0
control condition
feedback condition
2,5
week 1/2
week 3/4
week 5/6
week 7/8
week 9/10
week 11/12
week 1/2
week 3/4
week 5/6
week 7/8
week 9/10
week 11/12
Figure 2. Course of dilatory behavior (left) and self-reflection (right) during the semester in the experimental conditions (rated on a 5-point scale from 1 to 5)
Relations between components of self-regulated learning

First, we conducted an exploratory analysis of correlations in which we investigated the relations between components of self-regulated learning. For this purpose, we averaged the weekly collected individual scores of the variables of interest and computed Pearson correlations. Table 1 provides an overview of the intercorrelations. Non-specific goals were negatively related to both the average motivation (r = -.48, p < .01) and learning outcomes (declarative knowledge: r = -.39, p < .05; conceptual knowledge: r = -.43, p < .05). In contrast, specific goals were related to better learning outcomes (declarative knowledge: r = .47, p < .01; conceptual knowledge: r = .59, p < .01), higher motivation (r = .29, p < .05), and less intra-individual variance in dilatory behavior (r = -.37, p < .05), and thus contributed to a consistent learning progress. Dilatory behavior was related to low motivation (r = -.42, p< .01). Related to self-reported strategy use, we found few correlations. However, the results of the correlation analyses indicated different relations between motivational, cognitive, metacognitive and behavioral aspects in self-regulated learning. Thus, the results offered the opportunity to improve the whole process by supporting one particular aspect of self-regulated learning, for example dilatory behavior. Table 1. Summary of inter-correlations among variables of self-regulated learning and learning outcomes Measure N 1 2 3 4 5 6 7 8 9 10 1. Declarative knowledge 35 -.55** .47** -.39* -.33 .12 -05 .00 -.09 .11 2. Conceptual knowledge 35 -.59** -.43* -.18 .10 .08 07 -.22 .25 3. Specific goals 46 --.49** -.03 .29* -.37* -.07 -.05 .13 4. Non-specific goals 46 -.25 -.48** -.06 -.21 -.21 -.22 5. Dilatory behavior DB 49 --.42** -.02 -14 .05 -.23 6. Motivation 49 --.16 .24 .40** .01 7. Variance of DB 49 -.08 .04 .00 8. Metacognitive Strat. 49 -.37* .03 9. Elaboration 49 --.16 10. Organization 49 -11. Rehearsal strategies 49 Note. * p < .05; ** p < .01
11 -.03 .13 -.08 .05 -.23 .04 -.10 .27 .00 .37** --
Improvement of learning outcomes

In the next step, we investigated improvements of learning outcomes due to the experimental conditions. Table 2 provides an overview of means and standard deviations in the knowledge tests separately for the experimental conditions. To investigate whether prior knowledge was comparable, we conducted a multivariate analysis of variance with declarative knowledge and conceptual knowledge as dependent variables and experimental condition (veridical, random, no-visualization) as independent variable. The results showed that the experimental conditions were comparable with regard to this important prerequisite, Pillays trace = .07, F(4, 90) = 0.84, p = .50. Second, we conducted a multivariate analysis of variance with posttest results (declarative and conceptual knowledge) as dependent variables and experimental condition (veridical, random, no-
ISLS
52
ICLS2012
visualization) as independent variable. The difference between the conditions failed statistical significance, Pillays trace = .15, F(4, 64) = 1.30, p = .28. Thus, no statistically significant, direct effect of experimental conditions on learning outcomes could be found. Table 2. Means and standard deviations of the knowledge test results Experimental condition No-visualization Random visualization Knowledge tests N M (SD) N M (SD) Conceptual Pre 14 7.29(4.68) 18 6.83(3.43) knowledge Post 14 6.81(2.75) 18 7.13(3.18) Declarative Pre 11 4.00(5.14) 15 6.11(6.10) knowledge Post 11 16.80(7.97) 15 17.09(5.91)
Veridical visualization N M (SD) 16 6.44(3.16) 16 6.67(1.87) 9 5.94(5.34) 9 12.33(4.74)
As we expected that metacognitive strategies and volitional control influenced learning outcomes, we first conducted two separate multiple regression analyses (backward) with indicators of metacognition and volitional control (goal categories, metacognitive strategies, motivation, intra-individual variance of dilatory behavior) as potential predictors of conceptual and declarative knowledge which were the dependent variables. Additionally we included the pretest as predictor. Only specific goals were included in the final regression model for conceptual knowledge, t = 4.09, p < .01, = .59, R = .34. For declarative knowledge, specific goals, t = 3.43, p < .01, = .47, and prior knowledge t = 2.73, p < .05, = .38, R = .50, were included in the final regression model. The more often students set up specific goals, the higher were learning outcomes in conceptual and declarative knowledge tasks as measured by the knowledge test. To investigate aspects that are positively related to specific goal setting, we conducted another multiple regression analysis. We included motivation, variance of dilatory behavior and prior knowledge as potential predictors and specific goals as dependent variable. We identified variance of dilatory behavior as having a negative impact on specific goals, t = -3.05, p< .01, = .37, and motivation having a positive impact, t = 2.66, p< .05, = .37, R = .26. Thus, students who reported to learn more consistently and assessed the topic as personally relevant set a higher number of specific learning goals. As reported above, the graphical visualization with veridical feedback improved consistent studying and reduced the variance of dilatory behavior. Thus, one can speculate that the tool had an indirect influence on goal setting and on conceptual and declarative knowledge tasks, although the direct effect of the visualization on learning outcomes did not become significant, probably because of the low number of participants. However, supporting self-monitoring of dilatory behavior with a graphical visualization reduced ups and downs in postponing that supported specific goal setting that improved learning outcomes. Thus, providing a graphical visualization initiated a domino effect in selfregulated learning. To investigate this assumption, we conducted a path analysis with a linear contrast coding for experimental condition (feedback: 1; random: 0; no-visualization: -1) to account for both the signaling effect and the individual feedback. The results are illustrated in Figure 3. The model fit was satisfactory, CFI =.97; CMIN/df = 1.20, indicating that the relationships in these data could be well represented by this model.
Figure 3. Path model illustrating the domino effect initiated by a metacognitive awareness tool (all regression weights are significant at least at the .05--level)
Discussion
The aim of the current study was to evaluate a tool for self-regulated learning that should help students to reduce their dilatory behavior and improve their self-regulated learning. Deliberately monitoring of ones own problematic behavior was expected to induce a reactivity effect that can promote changes in behavior without further instructional support (Zimmerman, 2002). A graphical visualization of dilatory behavior should help students to become aware of their dilatory behavior and improve it (Schraw, 1998). In web-based selfmonitoring protocols, students recorded their self-regulated learning behavior once a week. To support such monitoring and increase its effects, we used a line chart and reflected back to students their self-reported degree of dilatory behavior and categorized their scores of dilatory behavior in low, medium and critical levels using traffic light colors. To evaluate this metacognitive awareness tool, we conducted an experimental field study with three different groups: The veridical feedback group received their dilatory behavior scores in an individual
ISLS
53
ICLS2012
line-chart. The random feedback group received random scores in the same chart. The no-visualization group did not receive any visualization. The results of the experimental field-study can be summarized as follows. First, directing students attention to dilatory behavior using traffic light colors (signaling effect) caused a greater degree of self-reported self-reflection in both groups with visualizations (random and veridical). Thus, students in these conditions were more likely to think about their comprehension and effectiveness of learning. This is a precondition for developing self-regulated learning and motivational beliefs (Pintrich, 2004; Zimmerman, 2002). Visualizations seem to have signaling effects that helped students to become aware of their learning processes that were reflected more intensively. Second, by directing students attention to individual dilatory behavior, and thus supporting a realistic self-observation, the veridical feedback tool reduced ups and downs in learning. Students in that condition avoided postponing to a substantial extent. In this way, the veridical visualization contributed to more continuous studying. Third, a low variance of dilatory behavior was related to a greater number of specific goals. Specific goals included both a concrete topic and strategic planning. Such goals are expected to be related to better learning outcomes (Schunk, 2001). A path analysis showed that providing a metacognitive awareness tool that mirrors students dilatory behavior and thus supports self-monitoring of ones behavior could have caused a domino effect in the learning process. In the current study, a visualization of dilatory behavior increased regular studying and improved goal setting, which improved learning outcomes. This domino effect is an indicator for the strong connectivity of behavioral (postponing), motivational (goal setting), and cognitive processes (learning outcomes) in self-regulated learning. As a consequence, students who engaged in learning activities regularly might have experienced a lower stress level before their exams (Tice & Baumeister, 1997). Furthermore, studying regularly could have eliminated the pressure to learn and understand high amounts of learning contents in a very short time, because it enhanced time-on-task (Tabak, Nguyen, Basuray & Darrow, 2009). As mentioned above, a lower stress level and the will to study regularly might enable students to take responsibility for their studying, and thus plan and reflect on it. These potential effects need to be proved in future studies. It seems to be important for future research to examine self-regulated learning as a complex system of different interacting aspects. Despite the promising results, there are some limitations in the current study. First, the low number of participants diminishes the statistical power of complex analyses and should therefore be interpreted carefully. The low number of participants can be traced back to the field-experimental design in demanding courses of study. Nevertheless, a field study raises the ecological validity of the data. Second, we partially referred to selfreported strategy use. As noted in prior research and supported by the missing relations between strategy use and learning outcomes in the present study, self-reported strategy use is not always reliable and thus of limited interpretability (Boekaerts & Corno, 2005).Therefore, we added alternative methods to investigate self-regulated learning, like the categorization of students learning goals. Third, our baseline - students in the no-visualization group - also experienced an intervention. As they recorded their class preparation and homework, selfmonitoring was supported concerning important aspects of learning that were included in the self-monitoring protocols. This might lead to an underestimation of the effect of metacognitive awareness tools. Nevertheless, though self-monitoring protocols are indeed an intervention tool, they are at the same time a sophisticated tool to diagnose self-regulated learning efficiently (Schmitz & Wiese, 2006). In summary, the metacognitive awareness tool used in the present study had beneficial effects on selfregulated learning. The study showed that metacognitive awareness tools, used for individual learning, had two main effects: raising students metacognitive awareness and supporting the accuracy of self-monitoring. More precisely, signaling dilatory behavior caused stronger self-reflection, and supporting realistic self-observation by veridical feedback induced behavior calibration in regard to dilatory behavior. Therefore, graphical visualizations of dilatory behavior, which can be easily integrated in online management systems, seem to be promising to improve the effects of self-monitoring protocols. They guide students awareness to problematic behavior that is ubiquitous among university students.
References
Boekaerts, M. (1996). Self-regulated learning at the junction of cognition and motivation. European Psychologist, 1(2), 100-112. Boekaerts, M., & Corno, L. (2005). Selfregulation in the classroom: A perspective on assessment and intervention. Applied Psychology, 54(2), 199-231. Chang, M.-M. (2007). Enhancing web-based language learning through self-monitoring. Journal of Computer Assisted Learning, 23(3), 187-196. Cho, K., Cho, M.-H., & Hacker, D. J. (2010). Self-monitoring support for learning to write. Interactive Learning Environments, 18(2), 101-113. Corno, L. (1986). The metacognitive control components of self-regulated learning. Contemporary Educational Psychology, 11(4), 333-346. Dehler, J., Bodemer, D., Buder, J., & Hesse, F. W. (2011). Guiding knowledge communication in CSCL via group knowledge awareness. Computers in Human Behavior, 27(3), 1068-1078.
ISLS
54
ICLS2012
Ferrari, J., Parker, J., & Ware, C. (1992). Academic procrastination: Personality correlates with Myers-Briggs types, self-efficacy, and academic locus of control. Journal of Social Behavior and Personality, 7, 495502. Joseph, L. M., & Eveleigh, E. L. (2011). A review of the effects of self-monitoring on reading performance of students with disabilities. The Journal of Special Education, 45(1), 43-53. Klassen, R. M., Krawchuk, L. L., & Rajani, S. (2008). Academic procrastination of undergraduates: Low selfefficacy to self-regulate predicts higher levels of procrastination. Contemporary Educational Psychology, 33(4), 915-931. Lan, W. Y. (1996). The effects of self-monitoring on students course performance, use of learning strategies, attitude, self-judgment ability, and knowledge representation. The Journal of Experimental Education, 64, 101-115. Lay, C., & Schouwenburg, H. (1993). Trait procrastination, time management, and academic behavior. Journal of Social Behavior and Personality, 8, 647-662. Lay, C., & Silverman, S. (1996). Trait procrastination, anxiety, and dilatory behavior. Personality and Individual Differences, 21(1), 61-67. Lewicki, P. (1983). Self-image bias in person perception. Journal of Personality and Social Psychology, 45(2), 384-393. Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57(9), 705-717. Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16, 385-407. Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1993). Reliability and predictive validity of the motivated strategies for learning questionnaire (MSLQ). Educational and Psychological Measurement, 53(3), 801-813. Sankaran, S. R., &Bui, T. (2001). Impact of learning strategies and motivation on performance: A study in webbased instruction. Journal of Instructional Psychology, 28(3), 191-198. Schmitz, B., & Wiese, B. S. (2006). New perspectives for the evaluation of training sessions in self-regulated learning: Time-series analyses of diary data. Contemporary Educational Psychology, 31(1), 64-96. Schraw, G. (1998). Promoting general metacognitive awareness. Instructional Science, 26, 113-125. Schunk, D.H. (2001). Social cognitive theory and self-regulated learning. In B.J. Zimmerman & D.H. Schunk (Eds.), Self-regulated learning and academic achievement: Theoretical perspectives (2nd ed., pp. 125152). Mahwah, NJ: Erlbaum. Stainton, M., Lay, C. H., & Flett, G. L. (2000). Trait procrastinators and behavior/ trait-specific cognitions. Journal of Social Behavior & Personality, 15(5), 297-312. Steel, P. (2007). The nature of procrastination: A meta-analytic and theoretical review of quintessential selfregulatory failure. Psychological Bulletin, 133(1), 65-94. Tabak, F., Nguyen, N., Basuray, T., & Darrow, W. (2009). Exploring the impact of personality on performance: How time-on-task moderates the mediation by self-efficacy. Personality and Individual Differences, 47(8), 823-828. Tice, D. M., & Baumeister, R. F. (1997). Longitudinal study of procrastination, performance, stress, and health: The costs and benefits of dawdling. Psychological Science, 8, 454-458. Winne, P. H., & Hadwin, A. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277-304). Hillsdale, NJ: Erlbaum. Wolters, C. A. (2003). Understanding procrastination from a self-regulated learning perspective. Journal of Educational Psychology, 95(1), 179-187. Zimmerman, B.J. (2000). Attainment of self-regulation: A social cognitive perspective. In M. Boekaerts, P.R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13-39). San Diego, CA: Academic Press. Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory Into Practice, 41(2), 64-70. Zimmerman, B. J., Bandura, A., &Martinez-Pons, M. (1992). Self-motivation for academic attainment: The role of self-efficacy beliefs and personal goal setting. American Educational Research Journal, 29(3), 663676. Zohar, A., & Peled, B. (2008). The effects of explicit teaching of metastrategic knowledge on low- and highachieving students. Learning and Instruction, 18(4), 337-353.
Acknowledgments
The research reported in this paper was supported by the German Federal Ministry of Education and Research (BMBF) under contract 01PH08031A.
ISLS
55
ICLS2012
Initial Validation of Listening Behavior Typologies for Online Discussions Using Microanalytic Case Studies
Alyssa Friend Wise, Ying-Ting Hsiao, Farshid Marbouti, Jennifer Speer, Nishan Perera Simon Fraser University, 250-13450 102 Avenue, Surrey BC, Canada, V3T 0A3 Email: afw3@sfu.ca, yha73@sfu.ca, fmarbout@sfu.ca, jspeer@sfu.ca, ncp3@sfu.ca Abstract: Collaborative participation in online discussions requires interacting with the ideas posted by others, yet many students do so ineffectually. This study conceptualizes these actions as online listening behaviors and proposes six prototypical behavioral patterns. To test these prototypes, microanalytic case studies were conducted. Five cases with distinct listening profiles are described: Gigi, showed a content coverage approach, viewing all her classmates posts but not drawing on them in her own; Stephanie, showed an interactive approach, focusing on a smaller number of posts, but building on them in her contributions. Ron showed a targeted approach, focusing on particular kinds of posts in the discussion and disregarding the rest; Darren showed a social coverage approach to listening, briefly scanning many posts and acknowledging his classmates contributions in a conversational tone; and Julie, showed a disregard for listening, ignoring the majority of the posts in the discussion. Theoretical and practical implications of the findings are discussed.
Introduction
In previous work we have argued for the importance of students listening in online discussions and the need for research to understand different listening approaches (Wise, Marbouti et al., 2011). For the context of online discussions, we define listening behaviors as the ways learners engage with others posts (e.g. which posts they attend to, at what point(s) in time, and in what ways). It is through these behaviors that learners become aware of others ideas, a pre-requisite for taking-up these ideas and creating comments that respond to them (Suthers, 2006). As attending to others ideas is necessary as a first step towards interactivity, a thorough notion of discussion participation must include not only the activity of making posts (i.e. speaking), but also that of viewing posts of others (i.e. listening). Prior work has suggested that in general learners in online discussions attend to each others ideas superficially (e.g. Hewitt, 2003; Palmer et al., 2008; Thomas, 2002) and that there is a relationship between the amount of engagement with others posts and course performance (Hamann et al., 2009; Morris et al., 2005). However, this work is based on examinations of student behaviors in aggregate. As we have argued previously, there is no reason to assume that students all listen in similar ways; in fact, initial evidence suggests the contrary (Wise, Perera et al., 2012). Even for individual students, participation is unlikely to be uniform over time; such temporal variations are artificially smoothed (and thus analytically eliminated) in the creation of aggregate measures. Thus, patterns of student listening in online discussions and their relationships to learning may be much more complex than previously suggested. To take full advantage of the analytical opportunities afforded by rich log-file data we need methods that unpack both the individual (Peters & Hewitt, 2010) and temporal (Dringus & Ellis, 2010) aspects of discussion participation. In past work, we found that different students interacted with prior messages in distinct ways (Wise, Perera et al., 2012) and documented three particular approaches to online listening (Wise, Marbouti et al., 2011). While our initial work on listening behaviors focused on empirically-derived patterns across multiple students, the current study moves this line of research forward by generating a set of theoretical listening typologies and comparing them with in-depth case studies of a small number of students individual behaviors. In this way we work towards a taxonomy of online listening behaviors that can be used to analyze and support online discussions.
Conceptualizing Online Listening Behaviors as Part of Discussion Participation

To conceptualize listening as an aspect of participation in online discussions, we draw on Knowltons (2005) taxonomy of five online discussion participation types characterized in terms of their perspectives on collaborative knowledge construction and its enactment in online discussions. Knowlton argues that based on different conceptualization of the meaning and purpose of online discussions, learners can participate differently and advance to more sophisticated participation types over time. Expanding on this general theorization of participation types, we propose specific listening behaviors that correspond with each category in the taxonomy (see Table 1). Knowltons (2005) model begins with the category of passive participation: those students who do not actively contribute posts. He suggests two explanations for this behavior. First, students inactivity may be due to a lack of interest in learning, a lack of knowledge in engaging in a collaborative learning environment, or a
ISLS
56
ICLS2012
poor ability to manage their time. We would expect these students to view few posts and spend little time on them. Second, these students may be engaging in initial legitimate peripheral participation (Lave & Wenger, 1991) as they familiarize themselves with the discussion. This second class of students would be expected to actively listen to the discussion as they prepare to eventually contribute, thus we consider them as a separate listening type. The second level of participation is developmental, in which students regard discussions as a social space for building community (rather than a place for learning the course material). Consequently, they are responsive in acknowledging their peers and their efforts in a conversational tone but the content of their comments lacks depth of thinking. In terms of listening, we would expect them to orient socially based on relationships, reading posts made by their friends and those that reply directly to them. The third level of participation is generative. Students at this level consider discussions as an individual channel to demonstrate their understanding and thus are more likely to contribute lecture-like posts in an academic writing style. Concentrating on articulating their own ideas to the instructor, we would expect them to listen primarily to the posts of the instructor rather than those of their peers. The two highest levels of participation are dialogical and metacognitive. At these levels learners participate with a collaborative mindset. While focused on their peers, they craft their comments with the goal of more than just community-building, recognizing a need to connect with others in a collective process of building ideas. For example, learners exhibiting dialogical participation are expected to be interactive in their listening, attending in depth to the contents of others posts. This enables them to build on others ideas in their own posts, and even synthesize disparate viewpoints to pull the groups conversation together. Learners at the metacognitive level exhibit dialogical participation plus have the additional characteristic of reflecting on their own thinking. Their goal of participation is not only to understand the topic discussed, but also recognize the influence of others ideas in changes in their own thinking. As metacognition often occurs internally, it can be difficult to detect (Knowlton, 2005); in terms of listening behaviors it might be reflected in substantial time spent re-reading posts made by others and oneself. Table 1: Knowltons (2005) participation types and proposed listening behaviors (Knowlton, 2005) Description Sees the discussion as a channel to receive information; does not contribute posts Sees the discussion as a social space; posts to acknowledge classmates Posts to construct their own ideas and report them to the instructor Posts to clarify their understanding through interacting with others Dialogical participation plus reflects on the process of knowledge development Proposed Listening Behaviors Disregardful. Minimal attention to others ideas. OR Preparatory. Listens to others ideas in preparation to contribute Social. Attends to peers posts in a social sense, superficially based on reciprocity Targeted. Attends primarily to the instructors posts, not those of their peers Interactive. Actively attends to content of peers posts (draws on them to contribute) Reflective. Dialogical plus reviews own and others posts throughout the discussion
Participation Type Passive
Developmental Generative Dialogical Metacognitive
Purpose of the Research

As a first step to empirically ground the proposed typologies of listening behaviors, we constructed a series of microanalytic case studies of student interactions in online discussions as part of a university class. Extreme case sampling (Patton, 2002) was used to identify patterns of interaction that were clearly distinct and would thus allow for well-defined characterization and potential alignment with the theoretical-generated typologies. Specifically, we attempted to answer two research questions: 1. What distinct listening patterns do individual students enact over time as they participate in asynchronous discussions as part of an online course? 2. How do these patterns align with theoretically-predicted typologies of listening behaviors?
Methods
Learning Environment and Discussion Task
Students in a fully-online undergraduate course on Educational Psychology participated in six week-long discussions, in groups of 10-13 students, worth 20% of their grade. Discussions were conducted in Phorum, a basic asynchronous threaded discussion forum. Students were required to discuss contrasting perspectives on authentic, contested questions in educational psychology. Discussions ran from Monday to Friday and required
ISLS 57
ICLS2012
students to contribute three posts on different days. Each week two students were assigned roles: Synthesizer (summarize the groups early ideas) and Wrapper (pull the conversation together at the end).
Data Extraction and Processing

Thirty-three consenting students log-file data was collected based on their activity in the discussion forum. The system automatically logged every action students performed in the discussions, noting the identity of the post read or created and applying a time-date stamp. Three types of actions were processed: view, post, and review (for details see Wise, Marbouti et al., 2011). Views were subdivided into reads or scans based on a threshold reading speed of 6.5 words per second (see Hewitt et al., 2007). Action duration was calculated by subtracting the time between actions for each student. One limitation of this calculation is that it does not account for the fact that students may engage in off-task activities during some portion of this time. Adjustments were made for unlogged system exits and actions were divided into sessions (a series of consecutive actions in the discussions) following the procedure described in Wise, Perera et al. (2012). Sixteen interaction variables (e.g. number and duration of sessions, total time spent reading, percentage of total posts viewed) were calculated for all students based on the log-file data.
Case Selection and Framing

Potential cases were selected for analysis by ranking students based on each interaction variable. Extreme sampling was enacted by identifying students who appeared most frequently in the top or bottom 10% for each variable (indicating extremely high or low participation on that indicator). Eight cases were selected for further examination of listening patterns. Preliminary investigation compared these cases with respect to the selection criteria of being extreme and unique; three cases were eliminated and five were retained (three high-volume and two low-volume participants). The fourth discussion week (on the topic of critical thinking) was chosen as the focus week for case study; none of the students studied had assigned discussion roles this week. The analytic frame included all actions performed by a student during this week; to capture related early/late activity, any actions in the critical thinking discussion up to two days before or after were also included.
Microanalytic Case Analysis Process

To conduct the case studies, temporal microanalysis of log-file data (Wise, Perera et al., 2012) was employed. This is an analytic approach that provides a methodology for constructing a meaningful account of students participation in an online discussion by providing a detailed on-the-ground account of student behaviors. The first analytic step was to get a broad sense of each students behaviors in the discussions globally by looking at their overall data patterns (see Table 2). The number and duration of each kind of action were also represented with pie charts to examine their relative fraction of total activity (see Figure 1). The second analytic step narrowed in on the focal discussion. A session-by-session table of activity was built for each student (see Tables 3-7) and this data was used to make a preliminary characterization of the students participation and listening behaviors. This interpretation would be tested and further developed in the third stage of the analysis. The third analytic step was to reconstruct learners experiences participating in the discussion during the focus week, with a focus on their listening behaviors. To do this, we created a dynamic discussion map: a spreadsheet of all posts organized by date, time and threading structure that could be filtered to reproduce the state of the forum at any point in the discussion. The log-file data for each case was examined using this map to provide context; this allowed us to make sense of the actions students performed within the historical setting of the discussion as it appeared at that time. (For example, we could distinguish when a student read posts scattered across a discussion versus those in sequence, even if by the end of the discussion the posts were no longer adjacent due to interspersed replies.) Analyzing students behaviors action-by-action in this way, we produced a narrative for each student reconstructing their participation in the discussion. Reference to the content of the students posts was used to contextualize actions in the discussion as needed.
Results
The five students studied were Gigi, Stephanie, Ron, Darren, and Julie (pseudonyms). The first three were high participation students; all spent relatively long time in the discussions, though activity patterns were distinct (see Table 2 and Figure 1). The latter two were low participation students; specific activity patterns also differed.
Gigi: A content coverage approach to listening

Gigi visited the discussions often, logging-in for 80 sessions over 23 hours (see Table 2). A large percentage of her sessions involved only reading actions (no posts made). Gigi viewed all posts in each discussion; however, many of these views were superficial scans. She made 24 posts in total, six more than required, and reviewed them multiple times during the discussions. In the Critical Thinking discussion, Gigi visited the forum every day, for a total of 13 sessions (see Table 3). She began early Monday morning, quickly viewing several posts from the preceding discussion. She then read the current weeks discussion prompt before returning to the previous forum to review her final post
ISLS
58
ICLS2012
and the last post in the discussion. Later Monday, she viewed 11 classmates posts in a linear sequence of scans and quick reads. Prior to making a reply, Gigi reviewed her final post from the previous discussion (about overlearning in math) one more time. She then contributed her first post, building on this concept and its relationship to transfer, while integrating academic language from the textbook. Finally, she quickly viewed three posts and logged-off. Table 2: Aggregate measures of listening and speaking behaviors in the discussions Behavior Time in the system (hrs) Number of sessions Percent of sessions with posts made Percent of time spent reading Number of posts viewed Percent of views that were reads (not scans) Av. time reading a post (min) Percent of posts viewed at least once Percent of posts read at least once Number posts made Av. length of post (words) Av. time making a post (min) Av. number of reads before posting Av. number of reads after posting Number of reviews Final course grade Gigi 22.8 80 28% 64% 413 56% 3.8 100% 80% 24 258 18.0 2.6 1 30 92 Stephanie 30.7 62 29% 97% 330 55% 9.8 100% 62% 18 247 0.8 4.8 0.4 5 87 Ron 24.1 28 68% 71% 217 65% 7.2 38% 31% 19 174 18.2 5.7 0.5 9 72 Darren 5.5 33 61% 48% 304 33% 1.6 89% 36% 26 134 4.9 2.2 0.6 2 67 Julie 6.6 17 82% 47% 158 25% 4.7 39% 14% 14 288 12.4 1.6 0 3 87
The next day, Gigi worked through the forum methodically and efficiently: she quickly read all four new posts in order, scanning the discussion prompt a couple of times between reads. She also reviewed her post from the previous evening. Later that night, she logged-in again and replied to the final post in the discussion, picking up on the notion of transfer she mentioned earlier. She then read one post and left. On Wednesday, Gigi had two quick sessions checking the discussion for new posts. In two subsequent sessions, she viewed a small number of new posts in a distinctive pattern, toggling between them and the one she had made. Very late that night (1am Thursday morning), Gigi read one new post, created her third post for the week (emphasizing the importance of overlearning), and left. Although she had met her posting requirement for the week, Gigi returned to the discussion daily, engaging in short sessions to briefly view all new posts linearly. Table 3: Gigis activities in the critical thinking discussion by session 1 Session 2 3 4 5 6 7 8 9 10 11 12 13 Day Mon Mon Tue Tue Wed Wed Wed Wed Thur Thur Fri Sat Sun Time 2:32 21:59 19:55 22:31 1:30 17:13 18:27 21:42 1:41 21:19 1:02 2:35 4:30 Started Length 17 42 10 22 2 2 6 27 28 3 3 6 1 (min) Scans 3 7 2 1 0 0 2 5 1 2 1 4 1 (# of) Reads 4 4 5 1 1 1 3 5 0 1 4 7 1 (# of) Posts 0 1 0 1 0 0 0 0 1 0 0 0 0 (# of) Reviews 2 1 1 0 0 0 2 2 0 0 0 0 0 (# of) Several actions in this session took place in the forum for the previous discussion, see text for details. Total Time
167 min 5 min 71 min 83 min 8 min
Stephanie: An interactive approach to listening

Stephanie spent the most time in the discussions overall, logging in 62 times for a total of 31 hours (see Table 2). Her primary activity in the forum was reading (see Figure 1) and she spent more time per post than each of the other four students. However, she only spent 15 minutes to make her 18 required posts, thus it is likely that she composed her posts in an outside tool during some of her long reading times.
ISLS
59
ICLS2012
Figure 1. Proportion of total number of events and amount of time reading, scanning, reviewing and posting for (a) Gigi (b) Stephanie (c) Ron (d) Darren and (e) Julie. Pie size is proportional to total amount of activity. In the Critical Thinking discussion, Stephanie visited the discussion seven times on three different days, with most of her time spent on reading (see Table 4). She viewed the entire discussion, though she focused her attention on a small number of posts. On Monday afternoon, she began by scanning the discussion prompt and several posts from the prior discussion. She then spent a few minutes reading the Critical Thinking prompt and one of the existing replies. A few hours later, she logged in again and read another reply. That evening she spent a long time on the discussion prompt and the two posts she viewed earlier; she then viewed the rest of the posts in the forum. After, she replied to the prompt stating her position that critical thinking should be taught as part of a coordinated approach with other subjects. She then scanned the remaining two posts and logged-out. Stephanie returned to the discussion on Thursday evening for another session. During this time, she read two posts for almost an hour and then viewed other posts in a linear fashion. At last she made her second post trying to present a group position and respond to a question posed by a classmate. On Friday, Stephanie briefly viewed the discussion prompt again, but did not read any new posts. Later that evening, she spent almost two hours viewing all of the posts, three of which she spent substantial time on. She then made her final post, which built on the wrappers conclusion post, drawing on her own relevant personal experiences. Table 4: Stephanies activities in the critical thinking discussion by session Session Day Time Started Length (min) 1 Mon 14:07 6 2 Mon 16:15 3 3 Mon 21:05 129 4 Thur 20:38 125 5 Fri 15:52 4 6 Fri 19:53 164 7 Mon 20:38 <1 Total Time
430 min
Scans (# of) 2 1 6 2 0 3 1 4 min Reads (# of) 2 1 6 11 1 11 1 425 min Posts (# of) 0 0 1 1 0 1 0 1 min Reviews (# of) 0 0 0 0 0 0 0 0 min Several actions in this session took place in the forum for the previous discussion, see text for details.
ISLS
60
ICLS2012
Ron: A targeted approach to listening

Ron spent a similarly long amount of time in the discussions as Gigi (see Table 2), with 24 hours across 28 sessions. While he only viewed 38% of his classmates posts, when he did read, he took his time. Most sessions included posting actions, and he contributed one post more than the assignment required. In the Critical Thinking discussion, Ron logged-in on three days, always in the evening. The first day he had three separate sessions; the other two days he only had one session each (see Table 5). Early Monday Ron visited the prompts for the current and previous discussions. Later Monday, he spent substantial time reading all five existing posts, cycling between them and the prompt. He was then inactive for two hours before creating his own reply to the prompt. Because of the extended time between actions, this was considered a new session. Ron then left the forum. On Thursday Ron quickly viewed three new posts and then read a post by the Synthesizer, viewed its first reply, and reread the synthesis again. He then scanned his previous post, and replied to the synthesis, explicitly responding to a question raised. Finally, he spent 30 minutes revisiting the synthesis post, and 5 minutes on its first reply before exiting the discussion leaving 14 posts unviewed. On Friday Ron scanned a post midway through the discussion, before reading the final four new posts in reverse chronological order. He returned to the synthesis post, quickly viewed two of the replies to this post and posted, again addressing other questions raised by the Synthesizer. In total, Ron viewed only 18 of the 39 posts available this week. Table 5: Rons activities in the critical thinking discussion by session Session Day Time Started Length (min) Scans (# of) Reads (# of) Posts (# of) Reviews (# of) * Estimated value 1 Mon 2:03 4 0 2 0 0 2 Mon 18:44 96 2 5 0 0 3 Mon 20:21 13* 0 0 1 0 4 Thur 23:14 106 1 7 1 1 5 Fri 19:23 25 3 6 1 0 Total Time
244 min 3 min 190 min 51 min <1 min
Darren: A social coverage approach to listening

Darren spent little time in the discussions (33 sessions, 5.5 hours, see Table 2). He viewed most posts at least once, but the majority of his views were scans. He was active in making many short posts (26 in total). In the Critical Thinking discussion Darren had several brief sessions and spent most of his time on a small number of posts (see Table 6). On Monday evening he finished up the prior discussion, viewing its remaining new posts. In the current weeks discussion he read the discussion prompt, scanned four of the seven posts and read one post extensively. Later he viewed the prompt, the two skipped posts, and a previously scanned post; he then replied to the post he read earlier. On Wednesday, Darren first spent a few minutes reading a new post in the prior discussion. In the Critical Thinking discussion, he read the first reply to his post, scanned 11 new posts, and read the last one (contributed by a classmate, Mike). Later, Darren revisited nine of these posts, and posted a reply to Mike explicitly responding to his question. He also acknowledged another classmate (Sundeep) for sharing an example. Thursday, Darren continued the conversation by viewing the replies to his post (one by Mike) and contributing a reply to Mikes reply. He then re-viewed several posts in the same thread and made another post replying to the last one; this time he acknowledged a third classmate (Melissa). Darren returned to the discussion twice more, both times viewing all new posts in under a minute. Table 6: Darrens activities in the critical thinking discussion by session Session Day Time Started Length (min) Scans (# of) Reads (# of) Posts (# of) Reviews (# of)
ISLS
1 Mon 19:27 12 7 4 0 0
2 Mon 21:09 14 2 3 1 0
3 Wed 18:19 4 13 3 0 0
4 Wed 22:13 7 8 2 1 0
5 Thur 21:17 8 4 3 2 1
6 Fri 16:36 1 4 1 0 0
7 Mon 11:59 1 4 0 0 0
Total Time
47 min 8 min 27 min 12 min <1 min

61
ICLS2012
Several actions in this session took place in the forum for the previous discussion, see text for details.
Julie: A disregard for listening

Julie had the fewest sessions and least amount of time in the discussions (Figure 1). She viewed only 39% of others posts and read less than half of those. She was also inactive in speaking, contributing only 14 posts total. During the Critical Thinking Discussion, Julie logged-in three times, creating a single post each time (see Table 7). On Monday, she quickly viewed the discussion prompt and the four existing posts in order. While she spent 54 minutes on the last post she read before making a post, it is likely that some of this time indicates off-task behavior because the post only consisted of 190 words and she did not refer to it in her own. Instead, her post (made in the final action of the session), replied directly to the discussion prompt and shared her own ideas on the subject. On Wednesday, Julie had a short session in which she viewed the synthesizer post, skipped its several replies, and replied directly to it. This post also did not refer to any other posts from the discussion. In her final session on Friday, Julie viewed only five posts. These posts were distributed across the forum and half of them were only scanned. Julie created a post as a reply to one of these, but its content repeated the same ideas as her previous posts. At the end of the discussion Julie had left two-thirds of the posts (22 of 33) unread. Table 7: Julies activities in the critical thinking discussion by session Session Day Time Started Length (min) Scans (# of) Reads (# of) Posts (# of) Reviews (# of) 1 Mon 13:48 64 3 3 1 0 2 Wed 22:32 6 0 1 1 0 3 Fri 12:45 18 3 3 1 0 Total Time
89 min 1 min 60 min 28 min 0 min
Discussion
The case studies displayed unique interaction styles, showing approaches to listening that emphasized different aspects of the conversation. Gigi, Stephanie and Darren viewed a large percentage of their classmates posts, but in different ways for different apparent purposes. Gigis listening activity was broad yet superficial; she seemed to cover the content by glancing over all the posts without spending much time on any one post and briefly revisiting the forum to check for new posts after fulfilling her posting requirement. She also seemed to compare her classmates posts to her own but never drew on their ideas. Her posts often used academic language. Darren also covered the discussion by opening a high percentage of posts for short periods of time; however his use of casual language and focus on acknowledging classmates by name suggests a more social purpose to his behavior. The inverse levels of interactivity and formality of language used by Darren and Gigi has also been seen in previous research (Thomas, 2002), suggesting that the goals of having a high level of academic discourse and building trust and relationships between students may at times be at cross purposes. While Stephanie also viewed a large percentage of the discussion, she is distinguished from Gigi and Darren in that she not only attended to all of her classmates posts, but also built substantively on their ideas. In addition, Stephanie tended to concentrate her reading activity on only a portion of the posts made, spending a substantial amount of time reading (and re-reading) them. Her posts drew on these posts, at times synthesizing them together, and built her ideas on top of them to move the conversation forward. In contrast to these three, Ron and Julie did not seem to worry about covering the conversation. Instead, Ron focused on strategically reading certain posts (particularly by the Synthesizer) to respond to the prompt while Julie made no effort to read deeply or thoroughly. While Julies behaviors echo common findings of minimal student effort (Palmer et al., 2008; Webb et al., 2004), the other students behaviors extend the knowledge base by presenting evidence of several new, distinct, and seemingly purposeful strategies for listening. Comparing the cases with the proposed listening typologies (see Table 1), Julies actions align with the predicted behaviors for disregardful listening, indicating that she did not value, or understand how to engage in collaborative processes in online discussions. No student seemed to exhibit a preparatory listening style; this is unsurprising in a formal discussion context requiring students to post. Social listening appeared present in Darrens behaviors of reading posts by particular classmates and acknowledging people without engaging deeply with their ideas. Targeted listening was evidenced in Rons focus on the authoritative voices of the prompt and synthesizer, and Interactive listening was clearly enacted by Stephanie as she spent extensive time reading and responding to others ideas. Stephanies focus on a subset of the posts was not predicted as part of Interactive listening and thus is an interesting phenomenon for further investigation. It may be that this selective attention make the tasks of interactivity and synthesis more manageable, or simply that these were the posts she
ISLS
62
ICLS2012
found most valuable. One students behavior did not clearly align with a single typology. Gigi paid superficial attention to many of her peers comments, but did not respond to them socially. The posts she made herself focused on her own ideas, but her listening was not targeted towards the instructor or authoritative others. Finally, despite her lack of interactivity in posting, she spent considerable time comparing her posts with those of others. Further work is needed to examine if Gigis behaviors can be explained as a hybrid of existing categories, if an additional listening typology is needed, or if these patterns simply reflect one persons idiosyncratic behaviors.
Conclusion
Past research has indicated that student interactions with prior messages in online discussions are often suboptimal; however, little empirical work has examined the different approaches students take to listen to others ideas. The general fit between the theoretically proposed typologies and the empirical cases studied here presents some initial validation for a taxonomy of listening in online discussion, showing that students listening is more varied and nuanced than previously known. The hybrid of behaviors seen for Gigi also indicates that categories may not be mutually exclusive. Further work is needed to probe the motivation for and usefulness of Gigis eclectic behaviors, as well as Stephanies concentration on subsets of the discussion. This work lays the foundation for building a more detailed understanding of different learner participation approaches that can be used to analyze and support online discussions. Specifically, this can provide useful information for educators in designing early detection and support (e.g. feedback, listening strategies and interaction guidelines) tailored for different participation types. Additional work is ongoing to test, validate, and expand these listening prototypes for different discussion contexts using additional measures and larger data sets. Future research will extend this work by examining motivations behind different behaviors as well as their impact on the quality of individual and group learning using additional data and measures such as interviews and content analysis.
References
Dringus, L. P., & Ellis, T. (2010). Temporal transitions in participation flow in an asynchronous discussion forum. Computers in Education, 54(2), 340349. Hamann, K., Pollock, P. H., & Wilson, B. M. (2009). Learning from listening to peers in online political science classes. Journal of Political Science Education, 5(1), 111. Hewitt, J. (2003). How habitual online practices affect the development of asynchronous discussion threads. Journal Educational Computing Research, 28(1), 31-45. Hewitt, J., Brett, C., & Peters, V. (2007). Scan Rate: A new metric for the analysis of reading behaviors in asynchronous computer conferencing environments. AJDE, 21(4), 1-17. Knowlton, D.S. (2005). A taxonomy of learning through asynchronous discussion. Journal of Interactive Learning Research, 16(2), 155-177. Lave, J., & Wenger, E. (1991). Situated learning: legitimate peripheral participation. New York: Cambridge. Morris, L. V., Finnegan, C., & Wu, S. (2005). Tracking student behavior, persistence, and achievement in online courses. The Internet and Higher Education, 8(3), 221231. Palmer, S., Holt, D., & Bray, S. (2008). Does the discussion help? The impact of a formally assessed online discussion on final student results. British Journal of Educational Technology, 39(5), 847-858. Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage. Peters, V., & Hewitt, J. (2010). An investigation of student practices in asynchronous computer conferencing courses. Computers in Education, 54(4), 951961. Suthers, D. (2006). Technology affordances for intersubjective meaning making: A research agenda for CSCL. International Journal of Computer-Supported Collaborative Learning, 1(3), 315-337. Thomas, M. J. W. (2002). Learning within incoherent structures: the space of online discussion forums. Journal of Computer Assisted Learning, 18, 351-366. Webb, E., Jones, A., Barker, P., & van Schaik, P. (2004). Using e-learning dialogues in higher education. Innovations in Education and Teaching International, 41(1), 93-103. Wise, A. F., Marbouti, F. Speer, J., & Hsiao, Y. T., (2011). Towards an understanding of listening in online discussions: A cluster analysis of learners interaction patterns. In H. Spada, G. Stahl, N. Miyake & N. Law (Eds.) Proceedings of the International Conference on Computer Supported Learning 2011, Volume I (pp. 88-95), Hong Kong, China: International Society of the Learning Sciences. Wise, A. F., Perera, N., Hsiao, Y. Speer, J., & Marbouti, F. (2012). Microanalytic case studies of individual participation patterns in an asynchronous online discussion in an undergraduate blended course. Internet and Higher Education, 15(2), 108117.
Acknowledgments
This work was supported by the Social Sciences and Humanities Council of Canada.
ISLS
63
ICLS2012
How to schedule multiple graphical representations? A classroom experiment with an intelligent tutoring system for fractions
Martina A. Rau, Carnegie Mellon University, Human-Computer Interaction Institute, 5000 Forbes Ave, Pittsburgh, PA 15213, USA, marau@cs.cmu.edu, Nikol Rummel, Ruhr-Universitt Bochum, Institute of Educational Research, Universittsstrae 150, 44801 Bochum, Germany, nikol.rummel@rub.de Vincent Aleven, Laura Pacilio, Zelha Tunc-Pekkan, Carnegie Mellon University, Human-Computer Interaction Institute, 5000 Forbes Ave, Pittsburgh, PA 15213, USA, aleven@cs.cmu.edu, lpacilio@andrew.cmu.edu, zelha@cs.cmu.edu Abstract: Providing learners with multiple representations of the learning content has been shown to enhance learning outcomes. When designing problem sequences with multiple representations, designers of intelligent tutoring systems must decide how to schedule the representations. Prior research on contextual interference has demonstrated that interleaving different types of learning tasks can foster a deep understanding of the underlying concepts. Do the same advantages apply to interleaving representations? In a classroom experiment, we compared four conditions that varied the practice schedules of multiple graphical representations between interleaving and blocking. The multiple-representation conditions were compared to three single-representation control conditions. During their regular classroom instruction, 290 4th and 5th-grade students worked for five hours with versions of an intelligent tutoring system for fractions. On several dependent measures, interleaving multiple graphical representations led to better learning results than blocking multiple graphical representations. Findings from a think-aloud study give insight into the underlying cognitive processes.
Introduction
Graphical representations of learning contents are often used for instruction (Ainsworth, 2006). When used in learning technology, graphical representations can be especially useful since they allow for interactions that are physically impossible or very difficult to realize, for instance by dragging and dropping symbolic statements into a chart that automatically updates to display the information (Moyer, Bolyard, & Spikell, 2002). However, learning with multiple graphical representations is challenging. An important prerequisite for benefiting from multiple representations is the acquisition of representational fluency: students need to conceptually understand each of the representations, and they need to be able to use them to solve problems (Ainsworth, 2006). Furthermore, students need to develop representational flexibility: they need to understand the differences and similarities between the representations, they need to learn to relate the different representations to one another, and to use the different representations interchangeably to solve problems (Ainsworth, 2006; de Jong et al., 1998). Fractions are one of many areas in mathematics where multiple graphical representations are used extensively (National Mathematics Advisory Panel, 2008). There are different conceptual interpretations of fractions, such as the measurement concept and the part-whole interpretation (Charalambous & Pitta-Pantazi, 2007). Each conceptual interpretation can be illustrated using a different graphical representation, such as number lines for measurement, or area models (e.g., circles or rectangles) to support part-whole interpretations. Multiple graphical representations may help students understand different conceptual aspects of fractions and thus gain a robust understanding of fractions. In a prior study, we found experimental evidence that students working with multiple graphical representation of fractions (e.g., circles, rectangles, and number lines) outperform students who work with a single graphical representation (a number line), although only when prompted to explain how the graphical representations (e.g., half a circle) relate to the symbolic representation (e.g., 1/2) (Rau, Aleven, & Rummel, 2009). These results demonstrate that understanding individual graphical representations (i.e., by relating graphically displayed information to the concepts of numerator and denominator) is essential in order for learners to benefit from multiple graphical representations. When designing instruction that uses multiple graphical representations, curriculum designers must decide how to temporally sequence the different graphical representations. How frequently should the curriculum alternate between graphical representations? Practice schedules are likely to impact how students understand each graphical representation and, consequently, how well they learn the underlying mathematical concepts. In particular, it may matter whether the different representations are practiced in a blocked manner (e.g., A A B B) or are interleaved with practice of other representations (e.g., A B A B). Research on contextual interference has investigated scheduling effects of different task types. Results show that interleaving task types leads to better learning results than blocked practice (Battig, 1972; de Croock, van Merrienboer, &
ISLS
64
ICLS2012
Paas, 1998). A common interpretation of this finding is that interleaved practice encourages deep processing (de Croock et al., 1998). Since students cannot hold all relevant knowledge in working memory, they must reactivate task-specific knowledge as it comes up again in the task sequence. Another interpretation is that when frequently switching between task types, students will be more likely to abstract a common principle from the tasks than when switching infrequently between them (de Croock et al., 1998). They can do so, for instance, by comparing different task types to one another. Against the background of research on blocking versus interleaving different task types, one could hypothesize that interleaving practice with different graphical representations (i.e. switching frequently between them) may lead to deeper processing of conceptual fractions knowledge and may encourage students to abstract a robust conceptual understanding from the multiplicity of graphical representations. These processes may help students acquire representational flexibility. On the other hand, our own previous work (Rau et al., 2009) leads to the hypothesis that developing representational fluency (i.e., coming to understand each single representation) may be a prerequisite for developing representational flexibility. Blocked practice with the representations may be successful in promoting representational fluency because it provides students with the opportunity to become fluent with one graphical representation before starting to work with a second graphical representation. This could be achieved by switching infrequently or with moderate frequency between the representations. If representation flexibility builds on representational fluency, as just argued, then students may benefit most from a condition that gradually moves from blocking to an increasingly interleaved schedule of multiple graphical representations. To test the hypotheses mentioned above, we compared four conditions that varied the practice schedules of multiple graphical representations in a classroom experiment. Students in all conditions worked on the same problems, but multiple graphical representations were presented either in a blocked, moderately interleaved, fully interleaved, or increasingly interleaved manner. We also employed three single graphical representation control conditions in which students worked either with only a circle, a rectangle, or a number line. These control conditions allow us to replicate the results from our first study (which, as mentioned showed advantages for learning with multiple graphical representations over learning with a number line; see Rau, et al., 2009) and to extend this finding to working with only area models. In order to gain insights into the cognitive processes underlying the most successful practice schedule, we additionally conducted a small think-aloud study. We report the results of the think-aloud study after discussing the results from the classroom experiment. We investigated the effects of interleaving multiple graphical representations in the context of a proven intelligent tutoring system technology, namely Cognitive Tutors (Koedinger & Corbett, 2006). Cognitive Tutors have a proven track record in improving students mathematics achievement (Koedinger & Corbett, 2006). We developed several versions of an example-tracing tutor for fractions learning, using Cognitive Tutor Authoring Tools (Aleven et al., 2009). This type of tutor behaves like a Cognitive Tutor but relies on examples of correct and incorrect solution paths rather than on a cognitive model. The design of the fractions tutor was informed by results from our previous studies (Rau, Aleven, & Rummel, 2009; 2010), and by small-scale user studies.
Experimental Study
We investigated the effects of blocking versus interleaving multiple graphical representations in a classroom experiment. We expected that interleaving multiple graphical representations of fractions would enhance the acquisition of representational flexibility whereas blocking and moderately interleaving representations would enhance the acquisition of representational fluency. We expected that increasingly interleaving representations would be most successful if representational flexibility builds on representational fluency.
Methods
The tutors used in the study included three interactive graphical representations of fractions: circles, rectangles, and number lines (Figure 1). Our fractions tutor curriculum covered six task types: identifying fractions from multiple graphical representations, making multiple graphical representations of symbolic fractions, reconstructing the unit from unit fractions, reconstructing the unit from proper fractions, identifying improper fractions from multiple graphical representations, and making multiple graphical representations of improper fractions. The tutoring system takes a conceptually-focused approach in introducing fractions. A common theme throughout the fractions tutor was the unit of the fraction (i.e., what the fraction is taken of). The concept of the unit is being introduced in the first task types, and revisited during the later task types as students learn about improper fractions. Figure 2 shows an example of a problem in which students make circle representations for two given symbolic fractions and are then prompted to reflect on the relative size of the two fractions. Students solved each problem by interacting both with symbols and with interactive graphical representations. Students manipulated the graphical representations in various ways: by clicking on fraction pieces to highlight them, by dragging and dropping fraction pieces, and through buttons to change the partitioning of the graphical representations. The tutor interfaces updated interactively after each step to show
ISLS
65
ICLS2012
the next step to work on, and to emphasize parts of the graphical representations that were conceptually relevant for the subtask at hand through color-highlighting. Students received error feedback and hints on all steps. Error feedback messages were designed to make students reconsider their answer using the multiple graphical representations, or by reminding them of a previously introduced principle. Hint messages provided conceptually oriented help in relation to the graphical representation. Each problem included conceptually oriented prompts to help students relate the multiple graphical representations to the symbolic notation of fractions. We found these prompts to be effective in an earlier experimental study (Rau et al., 2009).
Figure 1. Interactive representations used in fractions tutor: circle, rectangle, and number line.
Figure 2. Making a circle given a symbolic fraction, combined with prompts to compare the two fractions. Reflection prompts are implemented with drop-down menus shown in the bottom half of each problem. A total of 587 4th- and 5th-grade students from six different schools (31 classes) participated in the study during their regular mathematics instruction. We excluded students who missed at least one test day, and who completed less than 67% of all tutor problems (to ensure that students in the multiple graphical representations conditions encountered all three graphical representations). This results in a total of N = 290 (n = 63 in blocked, n = 53 in moderate, n = 52 in fully interleaved, n = 62 in increased, n = 21 in single-circle, n = 20 in single-rectangle, n = 19 in single-number-line). Prior to working on the fractions tutor, students completed a pretest. The pretest took about 30 minutes. On the following day, all students started working with the fractions tutor. Students accessed the tutoring system from the computer lab at their schools and worked on the tutor for fractions for about five hours as part of their regular math instruction for five to six consecutive school days (depending on the length of the respective schools class periods). All students worked on the fractions tutor at their own pace, but the time students spent with the system was held constant across classrooms and across experimental conditions. On the day following the tutoring sessions, students took the immediate posttest which took about 30 minutes to complete. Seven days after the posttest, students completed an equivalent delayed posttest. Figure 3 illustrates the practice schedules of task types and graphical representations for the four multiple graphical representations conditions. In all conditions, students worked through the same sequence of task types and fraction problems, and switched task types after every 9 of a total of 108 problems. Each task type was revisited three times. This procedure corresponds to the most successful level of interleaving task types in our prior experiment (Rau et al., 2010). We randomly assigned students to one of seven conditions. In the blocked condition, students switched graphical representations after 36 problems. In the moderate condition,
ISLS 66
ICLS2012
students switched representations after every six problems. In the fully interleaved condition, students switched representations after each problem. In the increased condition, the length of the blocks was gradually reduced from twelve problems at the beginning to a single problem at the end. To account for possible effects of the order of graphical representations, we randomized the order in which students encountered the graphical representations. Finally, students in the three single graphical representation conditions worked on all tutor problems with only the circle, the rectangle, or the number line, respectively.
Figure 3: Practice schedule for multiple graphical representations conditions for all six task types. Each task type was revisited three times. Task types are indicated by numbers 1-6, representations are indicated by the different shapes. We assessed students knowledge of fractions at three test times. Three equivalent test forms were created, and we randomized the order in which they were administered. The tests included four knowledge types: fluency with area models (i.e., circles and rectangles), fluency with the number line, conceptual transfer and procedural transfer. The fluency items included identifying fractions given a graphical representation, making a graphical representation given a symbolic fraction, and recreating the unit given a graphical representation of both unit fractions and proper fractions. Conceptual transfer items included proportional reasoning questions with and without graphical representations. Procedural transfer items included comparison questions with and without graphical representations. The theoretical structure of the test (i.e., the four knowledge types just mentioned) resulted from a factor analysis performed on the pretest data. Test items including the number line seemed to be more challenging for students than area models.
Analysis
As mentioned, we analyzed the data of N = 290 students. There was no significant difference between conditions with respect to the number of students excluded ( < 1). There were no significant differences between conditions at pretest for any dependent measure, ps > .10. There was no significant effect for order of multiple graphical representations for any dependent measure, F(5, 285) = 1.56, ps > .10. We used a hierarchical linear model (HLM, see Raudenbush & Bryk, 2002) with four nested levels to analyze the data. We modeled performance on the tests for each student (level 1), differences between students
ISLS
67
ICLS2012
nested within classes (level 2), differences between classes nested within schools (level 3), and differences between schools (level 4). More specifically, the following HLM model was fitted to the data: scoreij = testj + conditioni + testj*conditioni + preScorei + preScorei*conditioni + numProblemsi + 1) student(class)i+ class(school)i + schooli, with the dependent variable scoreij being studentis score on the dependent measures at testj (i.e., immediate or delayed posttest). In order to analyze whether students with different levels of prior knowledge benefit differently from our conditions, we included students pretest scores as a covariate (preScorei), and modeled the interaction of pretest score with condition (preScorei*conditioni). Since the HLM described in (1) uses students pretest scores as a covariate, it does not allow us to analyze whether students in the various conditions improved from pretest to immediate and delayed posttest. To analyze learning gains, we included pretest score in the dependent variable, yielding: scoreij = testj + conditioni + testj*conditioni + numProblemsi + student(class)i+ class(school)i + 2) schooli, with the dependent variable scoreij being studentis score on the dependent measures at testj (i.e., pretest, immediate posttest, or delayed posttest). We used planned contrasts and post-hoc comparisons to clarify results from the HLM analysis. All reported p-values were adjusted using the Bonferroni correction for multiple comparisons. Table 1: Improvement of test scores at immediate posttest (post) over pretest (pre) and delayed posttest (delayed) over pretest by knowledge types and conditions. Condition
blocked moderately interleaved fully interleaved increasingly interleaved singlecircle singlerectangle singlenumber-line
Effect
post > pre delayed > pre post > pre delayed > pre post > pre delayed > pre post > pre delayed > pre post > pre delayed > pre post > pre delayed > pre post > pre delayed > pre
Fluency with area models n.s. p < .05, d = .52 n.s. n.s. p < .05, d = .45 p < .05, d = .38 p < .05, d = .38 p < .05, d = .55 n.s. n.s. n.s. n.s. n.s. n.s.
Fluency with the number line n.s. p < .01, d = .39 n.s. p < .01, d = .50 p < .01, d = .51 p < .01, d = .75 p < .01, d = .43 p < .01, d = .46 n.s. n.s. n.s. n.s. n.s. n.s.
Conceptual transfer p < .05, d = .42 p < .05, d = .39 p < .05, d = .29 p < .05, d = .30 p < .01, d = .34 p < .01, d = .60 n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s.
Procedural transfer n.s. n.s. n.s. p < .05, d = .45 n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s.
Table 2: Differences between conditions at immediate posttest (post) and delayed posttest (delayed) by type of knowledge. n.s. indicates non-significant results. - indicates that no post-hoc comparisons were computed.
Effect fully interleaved > blocked, moderately interleaved, increasingly interleaved increasingly interleaved > blocked, moderately interleaved, fully interleaved moderately interleaved > blocked, fully interleaved, increasingly interleaved Test post delayed post delayed post delayed Fluency with area models p < .10, d = .30 p < .10, d = .30 Fluency with number line n.s. n.s. Conceptual transfer n.s. p < .05, d = .33 Procedural transfer n.s. n.s.
Results
To investigate whether students learned from the fractions tutor, we analyzed learning gains using the simpler HLM described in formula (2). The main effect of test time was significant for fluency with the number line, F(2, 867) = 20.09, p < .01, partial = .03, for fluency with area models, F(2, 867) = 17.54, p < .01, = .02, conceptual transfer, F(2, 867) = 38.78, p < .01, partial = .03, and marginally significant for procedural transfer, F(2, 867) = 2.84, p < .10, partial = .01. The interaction between test time and condition was significant for fluency with area models F(12, 862) = 2.06, p < .05, partial = .01. These results show that students (regardless of condition) improved on fluency with the number line, area models, procedural and conceptual transfer. On fluency with area models, students learning gains also depended on the condition. To further clarify these results, we computed post-hoc comparisons that compared students scores at the immediate posttest and the delayed posttest compared to the pretest, respectively. Table 1 provides a
ISLS
68
ICLS2012
summary of these post-hoc comparisons. Generally, we found significant learning gains at the delayed posttest for most of the multiple graphical representations conditions on fluency with area models, fluency with the number line, and conceptual transfer. On procedural transfer, only the moderate condition showed significant learning gains at the delayed posttest. Finally, we found no significant learning gains for the single graphical representation conditions except for the single-circle condition at the delayed posttest on conceptual transfer. To analyze the effect of practice schedules of multiple graphical representations, we computed the HLM presented in formula (1) for only the multiple graphical representations conditions. There was no significant main effect of condition on any posttest scale, indicating that there was no global effect of practice schedules of multiple graphical representations across immediate and delayed posttests. An interaction between test time and condition was marginally significant for fluency with area models, F(3, 867) = 2.57, p < .10, = .01, indicating that the effect of practice schedules depends on test time. The interaction between pretest score and condition was marginally significant for conceptual transfer, F(3, 219) = 2.52, p < .10, = .02, demonstrating that students with different pretest scores benefit from different practice schedules. To clarify the interaction between test time and condition, we used post-hoc comparisons separately for the immediate and the delayed posttest. To limit the number of comparisons, we only compared the most successful multiple graphical representations condition against the other three multiple graphical representations conditions, as summarized in Table 2. We found some support for a benefit of interleaving multiple graphical representations: the fully interleaved condition significantly outperformed the blocked, the moderately interleaved, and the increasingly interleaved conditions on conceptual transfer at the delayed posttest. Furthermore, we found a marginally significant advantage for the increasingly interleaved condition over the blocked, moderately interleaved, and fully interleaved conditions on fluency with the number line at the immediate and the delayed posttests. To clarify the interaction between pretest score and condition on proportional reasoning items, we computed post-hoc comparisons for students with extremely low or high pretest scores. For students with a pretest score of 15%, 20%, and 25%, we found a significant advantage for the interleaved over the blocked condition (ps < .05). We found no differences for high prior knowledge students. To analyze the difference between multiple graphical representations conditions and the single graphical representation conditions, we computed the HLM described in formula (1). We used planned contrasts to compare the multiple graphical representations conditions to the single graphical representation conditions. We found a significant advantage for the multiple graphical representations conditions over the single graphical representation conditions for number line test items at delayed posttest (p < .05, d =.29).
Think-aloud study
In order to gain further insight into the cognitive processes underlying the benefits of the interleaved practice schedules, we conducted a small-scale think-aloud study with six students who worked on the fully interleaved version of the tutoring system. As argued above, one hypothesized mechanism is that interleaved practice leads students to abstract across multiple graphical representations, for instance by comparing them to one another. Alternatively, it has been hypothesized that interleaved practice leads to the reactivation of representationspecific knowledge. Specifically, we were interested in what kinds of spontaneous comparisons students are making between the graphical representations, at what points in the curriculum they make comparisons, and whether students who fail to make spontaneous comparisons can be prompted to do so.
Methods
Six 5th-grade students participated in the think-aloud study. The think-aloud study was conducted in our laboratory and included three sessions. During the first session, students took the same pretest that was used in our experimental study. The pretest took about 30 minutes to complete. During the second session, students worked for one hour on a subset of problems taken from the interleaved version of the tutoring system while being prompted to think aloud, following the procedure described in Ericsson and Simon (1984). In the third session, students worked with similar tutor problems for one hour while being prompted to relate the different graphical representations to one another. We varied the type of prompts based on a within-subjects design: the prompt questions were either implicit (i.e., without directly prompting comparisons between the representations; e.g. How is this problem the same as the last two you did? or How is this problem different from the last one you did?), or explicit (i.e., directly referring to aspects that the different representations share; e.g., What is the unit in the circle / rectangle / number line? or How are the rectangle and the circle and the number line the same / different?). All students received two implicit prompts and four explicit prompts, in a fixed sequence. Students utterances were recorded and transcribed. We combined top-down and bottom-up approaches in developing a coding scheme: the experimenters identified types of comparisons that students might make prior to the think-aloud study, and then refined the coding scheme after viewing the transcripts from the thinkaloud study. Comparisons between graphical representations were coded as surface comparisons if they either referred to the color of the representation, the shape of the representation, or the action performed on the
ISLS
69
ICLS2012
representation (e.g., dragging and dropping). Comparisons were coded as conceptual if they referred to the corresponding features of the representations (i.e., numerator, denominator, unit), or the magnitude represented.
Results and Discussion

The results from the pretest indicate that all students had a good understanding of fractions. During the spontaneous comparison phase of the think-aloud study, we found only five instances of comparisons. These five comparisons were uttered by five of the six students. All five comparisons were surface comparisons. When prompted to compare the different representations, students generated 138 instances of comparisons overall. Table 3 summarizes the average number of comparisons coded as surface and conceptual comparisons per implicit and explicit prompt. Given the small number of students, a statistical test on the types of comparisons in response to implicit and explicit prompts is not warranted. Table 3 shows that students generated substantially more surface than conceptual comparisons per prompt. We can also see that the implicit prompts yielded most of the surface comparisons, but almost none of the conceptual comparisons. In contrast, explicit prompts seem to have yielded more of the conceptual comparisons and fewer of the surface comparisons, compared to the implicit prompts. Table 3: Average number of surface and conceptual comparisons per implicit and explicit prompts. Implicit prompts 4.17 0.58 2.38 Explicit prompts 2.33 1.63 1.98
Surface Conceptual
2.94 1.28
Conclusions
Taken together, the results from the experiment demonstrate significant learning gains for students who worked with a tutoring system that supports learning with multiple graphical representations of fractions, but not for those students who worked with only a single graphical representation. The gains persist until one week after the study when we administered the delayed posttest. Learning gains were found for students in all multiple graphical representations conditions on all posttest scales except procedural transfer. The fact that students performance on procedural transfer does not improve may be due to the fact that comparing fractions was not the focus of the tutor, and that the comparison tasks in the procedural transfer items were too difficult for students. We did not find evidence for learning in the single representation groups. The lack of learning gains in the single representation control conditions demonstrates that the learning gains in the other conditions are not due to practice effects with the test format. Furthermore, the finding that the multiple graphical representations conditions outperform the single graphical representation conditions on number line items at the delayed posttest suggests an advantage of learning with multiple graphical representations over learning with a single graphical representation. Taken together with the lack of learning gains in the single representation conditions, these results replicate our earlier finding that multiple graphical representations lead to better learning of fractions than a single graphical representation (Rau et al., 2009), and extend this finding from working with a number line only to working with circles or rectangles only. We argued that interleaving multiple graphical representations may help students acquire representational flexibility, whereas blocking graphical representations will help students acquire fluency with each representation. Our results confirm that the practice schedule for learning with multiple graphical representations matters: interleaving multiple graphical representations enhances the acquisition of representational flexibility as assessed by the conceptual transfer scale of the test in particular for students with low prior knowledge. On the other hand, we did not find differences between experimental conditions for representational fluency (i.e., on area model and number line items) with the exception of a marginally significant advantage of the increasingly interleaved condition on fluency with area models. Our results therefore suggest that students in all multiple representations conditions were able to acquire representational fluency from the tutoring system; and that contrary to our hypothesis blocking graphical representations does not contribute to the development of representational fluency. We hypothesized that interleaving graphical representations will promote the acquisition of representational flexibility. The significant advantage of the fully interleaved condition on conceptual transfer supports this hypothesis. Reasoning that representational flexibility may build on representational fluency, we had hypothesized that increasingly interleaving representations is the best option. Our findings do not support this hypothesis. Rather, the finding that especially low prior knowledge students benefit from fully interleaved representations may suggest that developing representational flexibility is particularly important for novice learners and that, perhaps, representational flexibility is a prerequisite for the development of representational fluency. Although more research is needed to investigate whether our findings generalize to other domains, based on our findings we can carefully conclude that designers of intelligent
ISLS
70
ICLS2012
tutoring systems should employ an interleaved practice schedule of multiple graphical representations in order to enhance robust conceptual understanding of fractions. What might be the mechanisms leading to the advantage of interleaving multiple graphical representations over blocking multiple graphical representations? Are students actively abstracting across graphical representations by explicitly comparing them to one another? Or are they, as argued above, reactivating knowledge that is specific to the graphical representations? Findings from our think-aloud study with the fully interleaved version of the tutoring system suggest that students did not spontaneously relate the different graphical representations to one another. The benefit from interleaving multiple graphical representations does not seem to stem from conscious abstraction across the different representations, but from more subliminal processes, such as repeatedly reactivating knowledge about the specific representations. On the other hand, the finding that students generate a good number of conceptual comparisons between the graphical representations when explicitly prompted to do so, suggests that students might benefit from explicit support in relating the different representations to one another. Although these considerations are based on the findings from only a small-scale think-aloud study, we believe that they provide interesting insights into the processes that may underlie the benefits of interleaving multiple graphical representations. Future work should investigate whether indeed the advantage of interleaving representations results from repeated reactivation of representational knowledge. Furthermore, future work should explore the benefit of explicitly supporting students in conceptually relating multiple graphical representations and in making sense of their differences and similarities in how they depict fractions.
References
Ainsworth, S. (2006). DeFT: A conceptual framework for considering learning with multiple representations. Learning and Instruction, 16(3), 183-198. Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. R. (2009). Example-tracing tutors: A new paradigm for intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 19(2), 105154. Battig, W. F. (1972). Intratask interference as a source of facilitation in transfer and retention. In R. F. Thompson & J. F. Vos (Eds.), Topics in learning and performance (pp. 131-159). New York: Academic Press. Charalambous, C. Y., & Pitta-Pantazi, D. (2007). Drawing on a Theoretical Model to Study Students' Understandings of Fractions. Educational Studies in Mathematics, 64(3), 293-316. de Croock, M. B. M., Van Merrienboer, J. J. G., & Paas, F. (1998). High versus low contextual interference in simulation-based training of troubleshooting skills: Effects on transfer performance and invested mental effort. Computers in Human Behavior, 14(2), 249-267. de Jong, T., Ainsworth, S. E., Dobson, M., Van der Meij, J., Levonen, J., Reimann, P., et al. (1998). Acquiring knowledge in science and mathematics: The use of multiple representations in technology-based learning environments. In M. W. Van Someren, W. Reimers, H. P. A. Boshuizen & T. de Jong (Eds.), Learning with Multiple Representations. Oxford. Ericsson, & Simon. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press. Koedinger, K., & Corbett, A. (2006). Cognitive Tutors: Technology bringing learning science to the classroom. In K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (pp. 61-78): Cambridge University Press. Moyer, P., Bolyard, J., & Spikell, M. A. (2002). What are virtual manipulatives? Teaching children mathematics, 8, 372-377. National Mathematics Advisory Panel (2008). Foundations for Success: Report of the National Mathematics Advisory Board Panel: U.S. Government Printing Office. Rau, M. A., Aleven, V., & Rummel, N. (2009). Intelligent tutoring systems with multiple representations and self-explanation prompts support learning of fractions. In Dimibrova, V., Mizoguchi, R., du Boulay, B. (Eds.). 14th International Conference on Artificial Intelligence in Education, pp. 441-448. IOS Press, Amsterdam. Rau, M. A., Aleven, V., & Rummel, N. (2010). Blocked versus Interleaved Practice with Multiple Representations in an Intelligent Tutoring System for Fractions In Aleven, V., Kay, J., Mostow, J. (Eds.) 10th International Conference of Intelligent Tutoring Systems, pp. 412-422, Springer, Heidelberg. Raudenbush, S.W., Bryk, A.S. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods. Sage Publications, Newbury Park.
Acknowledgments
This work was supported by the National Science Foundation, REESE-21851-1-1121307. We thank Ken Koedinger, Mitchell Nathan, Kathy Cramer, Peg Smith, Jay Raspat, Michael Ringenberg, the Datashop and CTAT teams, Brian Junker, Howard Seltman, Cassandra Studer, the students, teachers, and school principals.
ISLS 71
ICLS2012
Investigating the Relative Difficulty of Complex Systems Ideas in Biology

Sao-Ee Goha, Susan A. Yoona, Joyce Wanga, Zhitong Yanga, Eric Klopferb Graduate School of Education, University of Pennsylvania, 3700 Walnut Street, Philadelphia, PA 19104 goh@dolphin.upenn.edu, yoonsa@gse.upenn.edu, joyce.s.wang@gmail.com, zhitongy@hotmail.com b Scheller MIT Teacher Education Program, 77 Mass. Ave., MIT Building 10-337, Cambridge, MA 02139 klopfer@mit.edu
a
Abstract: A number of students biology misconceptions can be attributed to their lack of understanding in complex systems. Comprehending complex systems can be counterintuitive and difficult. A literature review reveals that we have yet to systematically determine a learning approach to address these learning challenges. In this paper, we propose that learning progressions research offers a methodical approach to organize the learning pathways students take to improve conceptual competence in complex systems. As a first step, we articulate a sequence of complex systems ideas - from the least to most difficult - by analyzing students written responses. Using an Item Response Theory model, we found that the easiest ideas to grasp are those that relate to the interconnected nature of these systems whereas the most difficult ideas are those concerning the decentralized organization of the system, and the predictability of the system effects.
Introduction
Studies in science education have revealed robust misconceptions about concepts and processes in school biology that directly impact students abilities to understand scientific topics. Misconceptions have been found across scales, from atoms (Taber & Garcia-Franco, 2010), to cells and genes (Garvin-Doxas & Klymkowsky, 2008), to organisms and ecology (Gotwals & Songer, 2010). Difficulties in understanding the relationships between these various scales have also been documented (Sewell, 2002). Researchers have speculated that these problems may exist due to a lack of understanding of the complex systems realms in which these scientific phenomena reside and interact (Charles & dApollonia, 2004; Chi, 2005). Thus, educational agencies in the US have urged science curriculum and instruction to emphasize systems content (AAAS, 2009; NRC, 2011). A complex system can be defined as an organization of interconnected components that as a whole, exhibits patterns and properties not obvious from those of the individual components (Mitchell, 2009). The way in which herds of ungulates roam across the savannah provides one of the most vivid examples of how individual actions and interactions lead to large scale patterns. To understand complex systems is to be able to reason that the individual components interact with one another in multiple and nonlinear ways, and recognize that the patterns observed at the system level emerge from these interactions at the component level (Sweeney & Sterman, 2007; Yoon, 2008). Research has documented that grasping complex systems ideas can be counterintuitive and difficult (Chi, 2005; Jacobson & Wilensky, 2006). While there have been several studies that promote the learning of particular complex systems or specific complexity ideas (e.g., Klopfer et al., 2009; Levy & Wilensky, 2009), the literature also suggests that we have yet to systematically determine a learning approach to address the challenges associated with a comprehensive understanding of the various ideas (HmeloSilver & Azevedo, 2006). We believe that the learning progressions methodology offers one such systematic approach in structuring the learning of various complex systems ideas. Learning progressions are defined as sequences of ordered descriptions that illustrate the learning pathways students can take to improve conceptual competence in science (Alonso & Steedle, 2006; CPRE, 2009). For example, Mohan and her colleagues (2009) identified levels of increasing sophistication in students perception of carbon-transforming events in complex socioecological systems. These ordered descriptions represent a research-informed framework for structuring the learning of core scientific ideas (NRC, 2007). Curriculum and instructional activities can in turn be mapped onto the learning progressions so as to better facilitate the learning of the core ideas (Gotwals & Songer, 2010; Songer et al., 2009). Our study extends the research on learning progressions by focusing on complex systems ideas grounded in high school biology content. Our broad goal is to eventually design valid curricular and instructional activities based on a learning progression of complex systems ideas. As a first step, in this paper we report on an exploratory work that investigates a conceptual sequence of complex systems ideas, from the least to the most difficult, by analyzing a diverse group of students written responses on the effects of geese on a park ecosystem.
Theoretical Considerations
ISLS 72
ICLS2012
Our research lies at the intersection of two fields understanding complex systems and learning progressions. We first discuss known challenges in understanding complex systems, before illustrating how learning progressions as a methodology can help address these challenges.
Understanding complex systems

As previously mentioned students typically possess misconceptions about complex systems (Ben-Zvi Assaraf & Orion, 2005; Hmelo, Holton, & Kolodner, 2000; Jacobson & Wilensky, 2006). For example, students tend to adopt a linear approach when thinking about the relationships among system components (Gotwals & Songer, 2010; Hogan, 2000; Riess & Mischo, 2010; Sweeney & Sterman, 2007). That is, they perceive single cause-andeffect relationships, where small actions lead to small effects. However, a complex system with its multiple connections among the components can often result in an action that gives rise to widespread effects throughout the system. Students also generally conceive of systems being controlled by certain components, and systemic patterns as designed with specific functions in mind (Penner, 2000; Resnick, 1996; Taber & Garcia-Franco, 2010). In other words, students are unable to interpret the decentralized nature and emergent patterns of complex systems. Wilensky and Resnick (1999) argue that this attribution of centralized control and intentionality to system processes is due to students confusion about the various levels of representation (e.g., microscopic and macroscopic) of complex systems. Despite the valuable information these studies provide, they examine single aspects of complex systems understanding. Given the current overall systems thrust in science education (AAAS, 2009; NRC, 2011) it is prudent to turn our attention to developing a more comprehensive view of complex systems content. Some studies have already articulated analytical frameworks that encompass multiple complex systems ideas and beliefs (Hmelo-Silver et al., 2004; 2007; Jacobson, 2001; 2011). For example Jacobson and his colleagues (2001) developed a framework that delineates distinct categories of complex systems ideas. Using this framework in an expert-novice study, he discovered that pre-college students tended to believe that systems operate in reductive, centralized and predictable way, whereas, science experts described phenomena as nonreductive, decentralized, and non-linear. We adapt this framework and evolutions of it (Jacobson, 2011; Yoon, 2008; 2011) to analyze levels of understanding for various ideas of complex systems. In doing so, we unpack and reveal the relative levels of difficulties that can eventually inform the development of a learning progression.
Learning Progressions
Learning progressions provide a methodical approach to organize curriculum and instructional activities to learn and teach about complex systems ideas, as they illustrate the cognitive pathways and skills students are likely to follow over a period of time in mastering the scientific concepts (Alonso & Steedle, 2006; NRC, 2007). Learning progressions structure the sequences based on what cognitive researchers know about science learning from empirical research findings (CPRE, 2009). To date, there have been three published works on learning progressions on biology topics related to complex systems concepts (Gotwals & Songer, 2010; Mohan et al., 2009; Songer et al., 2009). These studies have collectively ascertained that students learning of biological systems, or the development in the way they think about systems, follows some trajectory. However, these progressions address particular science content such as ecosystems and biodiversity rather than a more generalized complexity perspective. Although they have not characterized their work in terms of learning progressions, Ben-Zvi Assaraf and Orion (2005; 2010a; 2010b) examine students development of systems thinking abilities, and construct a hierarchical model of the stages by which students reason about complex systems. They explain that at the most rudimentary level students are able to identify the components of a system and processes within the system. At the intermediate level, students were able to identify relationships between system components, and organize the systems components, processes, and their interactions within a framework of relationships. At the most sophisticated level students can perceive hidden system components, explain the history of current patterns, and make predictions on the evolution of the system. However, their model comprises only a subset of essential complex systems components as identified in Jacobson (2001). A sequence or markers for learning about other equally salient ideas such as decentralization and emergence are also necessary to ensure that students acquire an adequate overall understanding of complex systems. As an initial development of a learning progression on complex systems ideas, a comprehensive set of ideas is first differentiated in order of difficulty levels. Such delineation can help organize the sequence of complexity ideas to be learned. There are generally two broad approaches in constructing the initial learning sequences (CPRE, 2009). The first approach begins by conjecturing a possible sequence from existing literature and then validating it. For example, Songer and her team (2009) first hypothesized their learning progression on biodiversity from literature reviews, and then tested the validity of the progression by comparing student learning outcomes in a control-treatment study. The second approach starts by analyzing a cross-sectional sample of students responses to provide indicators of their understanding and then derives the levels of
ISLS 73
ICLS2012
sophistication from the analysis. For instance, Mohan and her colleagues (2009) analyzed almost 300 elementary through high school students written accounts of the biogeochemical carbon cycling processes in socio-ecological systems for distinct levels of sophistication, and constructed their progression based on these levels. The latter approach presents an advantage when existing literature does not inform much about which ideas should be learned before others. In our exploratory study, we adopt this more grounds-up approach in drawing up the initial sequence of our learning progression.
Methods
Context and Participants
This study is part of a larger National Science Foundation-funded research project in which we design and develop curriculum and instructional activities using computational modeling tools in four high school biology units Chemistry of Life, Population Ecology, Community Ecology, and Evolution that promote student learning of complex systems. While the activities are being constructed and tested, we focus concurrently on building the learning progression that informs them. To begin, we developed eight open-ended short-answer questions, two for each biology unit. These were selected or derived from the OECD Programme for International Student Assessment (PISA, 2006) international science test and Ph.D. level biology content experts. We then administered this test to Grades 8 to 12 students in order to determine the range of conceptual difficulties. To recruit students to participate in our study, we enlisted the support of science teachers with whom we had previously worked in another study. We targeted different kinds of schools (including 3 magnet and 4 public) to account for differences in science learning experience. All teachers and students were recruited from a large urban school district in the northeast part of the US. Two days prior to the implementation, a researcher went to the teachers classes and gave a short presentation regarding the goals of the study and the roles of the students (should they participate). Those who participated were self-selected and were offered a $10 gift card in exchange for their participation. Their involvement lasted about one hour after school in which they answered questions on the open-ended biology test. The students were given as much time as they required to answer the questions. In total 44 students of various ethnicities from 7 urban schools participated. There were 20 males and 24 females in grade 8 (5), grade 9 (13), grade 10 (14), and grade 12 (12). The recruitment and data collection occurred over 3 months.
Data Sources, Data Coding and Analyses

For this paper, only the responses to one question were analyzed: Imagine a flock of geese arriving in a park in [your city], where geese havent lived before. Describe how the addition of these geese to the park affects the ecosystem over time. Consider both the living and non-living parts of the ecosystem. This question, written by an expert in biology, sought to solicit students understanding of biology and complex systems in an ecological context. A content analysis of the students responses was performed using six categories of complex systems understanding - Agent Effects, Action Effects, Networked Interactions, Multiple Causes, Order, and Processes derived from Jacobsons (2001; 2011) and Yoons (2008; 2011) studies. Table 1 provides an abridged description of each category. To account for variation in students understanding of the complexity ideas within each category, each response was coded six times - once for each category - for four levels of increasing sophistication: Completely Clockwork (Level 1), Somewhat Clockwork (Level 2), Somewhat Complex (Level 3), and Completely Complex (Level 4). Clockwork responses encompassed those that showed linear, deterministic, isolated, centralized, single-cause, and predictable system interactions or states, whereas complex responses included those that demonstrated non-linear, non-deterministic, networked, decentralized, multiplecauses, and random system interactions or states. After the coding manual was constructed and vetted, its reliability was assessed with two independent doctoral student raters coding 20% of the written responses. An inter-rater agreement of 0.8 was achieved collectively across categories using the Cronbach alpha reliability test. The remainder of the responses was subsequently coded by two of the authors using the coding scheme, with any discrepancies discussed. Below we provide an example of a response that achieved mostly Levels 3 or 4 for all categories of complexity ideas to show how the coding was done: Well if geese arrived it would probably help the ecosystem. The bird droppings might make the soil fertile. It would start to look a lot greener. The problem with that is erosion. The increase of plants and root size might cause paths or walkways to be damaged or destroyed. Statues might start to fall apart from the constant weight of birds. Plus the increase of plants in amount and size make O2 levels higher. Which could cause a warm and wet ecosystem, much similar to a swamp. Over a long period of time of course. (Student response, March 2011)
ISLS
74
ICLS2012
Table 1: Complex Systems Category Code Descriptions

Category Agent Effects The emphasis is the predictability of the effects caused by the part in question. Completely Clockwork (Level 1) Response shows that the way in which a part operates or affects other parts is completely predictable. No alternative is offered in the response. Somewhat Clockwork (Level 2) Response shows that the way in which a part operates or affects other parts is somewhat predictable. There are 1-2 possibilities suggested in the response. Response contains one complex component (out of three) of action effects. (See Level 4.) Somewhat Complex (Level 3) Response shows that the way in which a part operates or affects other parts is somewhat unpredictable. There are 3-4 possibilities suggested in the response. Response contains two complex components (out of three) of action effects. (See Level 4.) Completely Complex (Level 4) Response shows that the way in which a part operates or affects other parts is completely unpredictable. There are many possibilities suggested in the response. Response indicates (i) small actions can lead to large effects; (ii) the action can produce both localized changes and cascading effects; and (iii) the changes can take place both immediately and over a long period of time. Response attributes the outcome(s) of an event to four or more causes/factors.
Action Effects Three components are considered: (i) the relative scale of outcomes caused by action; (ii) the cascading effects of the action; and (iii) the time scale at which changes happen. Multiple Causes The focus is on the number of causes that may/will attribute to the outcome(s) of an event. Networked Interactions Three components are assessed: (i) interdependency among parts in the system; (ii) nonlinearity in reasoning; and (iii) emergent patterns over scale. Order The focus is the organization of the system or phenomenon centralized or decentralized.
Response indicates (i) small actions only lead to small effects; (ii) there is a sense that the action causes localized changes only; and (iii) the changes are immediate and do not sustain for a long time.
Response attributes the outcome(s) of an event to one cause/factor.
Response attributes the outcome(s) of an event to two causes/factors.
Response attributes the outcome(s) of an event to three causes/factors.
Response indicates that (i) the parts of a system are isolated with no interdependency among them; (ii) the interactions between parts are linear with no feedback; and (iii) the patterns at the system level are the same from those at the component level. Response indicates that the system is controlled by one central agent, that is, all action is dictated by a leader. Order in the system is established top-down or determined with a specific purpose in mind. Response indicates that the system is composed of static events. While perturbations in the system cause change to occur, the change terminates once an outcome is achieved (i.e., a definite end).
Response contains one complex component (out of three) of networked interactions. (See Level 4.)
Response contains two complex components (out of three) of networked interactions. (See Level 4.)
Response indicates that (i) the parts are interdependent; (ii) the interactions between parts are non-linear with feedback; and (iii) the patterns at the system level are emergent.
Processes Processes refer to the dynamism of the mechanisms that underlie the phenomena; in other words, how the system works or is thought to work.
Response indicates that the system is largely controlled by 2-3 central agents, i.e., there are other parts that may dictate how the system behaves. Order in the system is established top-down. Response indicates that the system is somewhat composed of static events with suggestions that these events take time to reach the outcome(s).
Response indicates that the system is largely decentralized and the control lies with 4-5 components. However, there is little evidence to show that the order in the system is selforganized. Response indicates that the system is somewhat of an ongoing process. Perturbations take a long time to reach the final outcomes, which are at larger scale than the initial event(s).
Response indicates that the system is decentralized and control lies with a myriad (more than 5) of parts. Order in the system is self-organized or bottom-up, and emerges spontaneously. Response indicates that the system is an ongoing, dynamic process. System continues to be in a state of flux. The parts adapt or evolve, and continue to do so accordingly.
In this exemplar response, the student repeatedly used non-deterministic words, such as might and could, and suggested three possibilities of the effects due to the geeses arrival, which indicates his uncertainty in the effects of the geeses arrival (Level 3 Agent Effects). There are suggestions that the geeses
ISLS 75
ICLS2012
arrival can cause cascading effects to the soil, the plant population, the oxygen levels, may lead to large-scale and long-term effects such as causing a warm and wet ecosystem (Level 4 Action Effects). More than four factors (e.g., droppings, soil, erosion, plants, oxygen) have been identified as attributing to possible outcomes (Level 4 Multiple Causes). The response also hints of an understanding of the interdependence of the various components in the park ecosystem, the feedback mechanisms present in the system (i.e., paths or walkways damaged by more plants), and the possible emergence of a systemic pattern (i.e., warm and wet environment) (Level 4 Networked Interactions). In addition, the idea of decentralization is clearly demonstrated as four actors (i.e., geese, plants, soil, and oxygen) are said to be involved (Level 3 Order). Furthermore, the response implies that the perturbations in the park ecosystem are somewhat on-going, and may take a long time to arrive at a final state (Level 3 Processes). As we are interested in determining the differences in conceptual difficulties among the six categories of complex systems ideas, and given that we have employed successively ordered rating scales (i.e., 4 levels of understanding), the Generalized Partial Credit Model (GPCM: Muraki, 1992), which is an item response theory model, was deemed appropriate for the analysis of the responses. The items, or categories of ideas in our case, are conceptualized as a series of hierarchical levels of performances where respondent receives partial credit for successfully performing at a particular level. Since our coding included four levels of understanding for six categories, the GPCM was an appropriate means to provide the information that we wanted. Students raw scores were first standardized on a continuum between -3 to +3 to reveal the difficulty level of each category. On this continuum of logit scale, 0 is set as the mean of the item difficulty parameter. On the positive direction toward +3, each increase indicates that the item is becoming more difficult; conversely, on the negative direction toward -3, each decrease signals that the item is turning less difficult. In addition, the model is able to show us how well each item can distinguish students with different abilities by the discrimination parameter. The discrimination parameter typically ranges between 0.5 and 2.5 in value. The larger this parameter, the more effective the item can distinguish students with varying levels of understanding.
Results
We rated the written responses for their understanding in each category of complex systems ideas. A breakdown of how they scored is given in Table 2. A simple frequency count shows most of the responses were at a level 2 (somewhat clockwork) or level 3 (somewhat complex) understanding of various complex systems ideas, with very few showing a level 4 (completely complex) understanding. We expected this because as described earlier, the literature indicated that most students face difficulty in grasping complex systems ideas. Table 2: Breakdown of scoring for each category of complex systems ideas
Category Agent Effects Action Effects Multiple Causes Networked Interactions Processes Order Total: Level 1 16 8 11 6 11 15 67 Level 2 14 11 19 12 23 16 95 Level 3 12 14 9 22 7 12 76 Level 4 2 11 5 4 3 1 26
We then ran the GPCM on the 44 sets of ratings, using PARSCLE 4.1. The model converged at critical value = 0.005, and had no items fit statistical significance, indicating a good model fit. The mean difficulty of the six items or categories is 0.49 (SD = 0.45), and mean discrimination parameter was 1.10 (SD =0.39). The categories were found reliable to measure students understanding of complex system (composite reliability = 0.87). The total test information peaked between 0 and 1, indicating the categories could measure students with slightly higher than average ability with most precision. Table 3 presents the six categories, their difficulty parameters, and their discrimination parameters. As indicated by the difficulty parameters, Agent Effects, Order, and Processes categories of ideas were found to be the most difficult among these students; Multiple Causes was found to be at the intermediate level; and Networked Interaction and Action Effects categories were the easiest. All six categories can also distinguish students based on their abilities, as their discrimination parameters ranged from 0.69 to 1.64. That is, the question is suitable for measuring the categories theorized. In addition, an examination of the test characteristic curves for this model indicated that the six items collectively provide adequate information for students with ability levels both below and above average. Table 3: Generalized Partial Credit Model
Category Difficulty (SE) Slope (SE)
ISLS
76
ICLS2012 Action Effects Networked Interaction Multiple Causes Processes Order Agent Effects N = 44. -0.17 (.018) 0.04 (0.22) 0.54 (0.21) 0.77 (0.26) 0.86 (0.22) 0.87 (0.29) 1.42 (0.50) 1.64 (0.58) 1.24 (0.51) 0.69 (0.39) 0.72 (0.98) 0.91 (0.34)
Discussion and Implications

In the introduction, we highlighted that a number of students misconceptions in biology can be attributed to their lack of understanding in complex systems. Comprehending complex systems ideas can be counterintuitive and difficult, and a review of the literature has revealed that we have yet to systematically determine a learning approach to address these learning challenges. We have proposed that learning progressions research can help to systematize the learning pathways students take to improve conceptual competence in complex systems. These learning pathways can in turn guide the curriculum and instructional activities in promoting complex systems learning, which is our broad goal. This paper articulates the initial development of a learning progression of complex systems ideas. A sequence of a comprehensive set of these ideas, from the least to the most difficult to understand, has been determined. As there is no prior literature to guide us in hypothesizing such a sequence, we have adopted a grounds-up approach in constructing the progression similar to Mohan et al.s study (2009). Using the GPCM, we analyzed 44 students written responses to a biology question on the effects of geese arrival on a park ecosystem for their understanding of various complexity ideas. We acknowledge the small sample size in this exploratory study and the implication this may have in our analyses, but we believe we have taken sufficient steps to ensure that different grade levels and types of schools are represented so as to make our findings as generalizable as possible. We found that two of the easiest categories of ideas for students to comprehend are those that relate to the effects of actions in complex systems (Action Effects), and the interconnected and emergent nature of these systems (Networked Interactions). A completely complex understanding of these ideas includes the ability to reason that: small actions can lead to large effects; these actions can produce both localized and cascading changes; the changes can occur immediately and over a long period of time; the parts in the system are interdependent; the relationships among the parts in the system are nonlinear with feedback mechanisms; and the parts interact to produce emergent patterns that are not obvious at the component level. Students seem to experience more difficulty in interpreting that there are multiple causes that may attribute to the outcome(s) of a change (Multiple Causes), and perceiving that the system continues to undergo adaptation or evolution (Processes). The categories of complex systems ideas that apparently are the most complicated to understand are Order and Agent Effects; these complexity ideas concern the decentralized organization of the system, and the predictability of the effects caused by the parts of the system. We believe this ordered sequence of complexity ideas is valid at least in the context of our test question. In a question that frames the complexity within a familiar park ecosystem, the students might have found it easier to observe the manifestation of complexity ideas associated with the interconnectedness and interdependency of the various plant and animal species in the park, the cascading, emergent, and non-linear effects the arrival of the geese can have on the rest of the ecosystem (i.e., the Action Effects and Networked Interactions categories). In contrast, complexity ideas in the Order and Agent Effects categories are less visible; the ideas that there is a myriad of other components (e.g., plants, predators and prey to geese, climate) that may contribute to how the system is organized, and that it is not possible to predict with precision the effects of geeses arrival at the park could well fall outside of students perceptual abilities. Chi (2005) also proposes that some scientific concepts and ideas may be easier to comprehend because of the ontological categories they belong to. Chi explains that ontological categories refer to the basic categories of realities or the kinds of existence in the world, such as concrete objects, events, and abstractions (p. 163). The more visible ideas of complex systems may have represented ontologically-easier categories for students to understand. In sum, our findings align well with current literature on complex systems understanding, and extend it by establishing an initial progression of the various ideas. Resnick (1996) and other researchers (e.g., Wilensky & Resnick, 1999; Wilensky & Reisman, 2006; Jacobson et al., 2011) have repeatedly iterated the deterministic and centralized mindset students typically have. Likewise, Charles and dApollonia (2004) and others (Ben-Zvi Assaraf & Orion, 2010a; Hogan, 2000; Lin & Hu, 2003; Perkins & Grozter, 2005) have observed that students tend to think linearly about relationships among the components or parts of complex systems. Collectively, these researchers have shown that when students are exposed to appropriate instruction, the learning challenges can be overcome. However, as we have argued, most of the interventions are targeted at improving student understanding of particular systems or specific complexity concepts rather than promoting a more holistic understanding of complex systems. A learning sequence of complexity ideas that specifies levels of difficulty
ISLS
77
ICLS2012
which we have found seeks to address this gap. This initial conception of difficulty levels is only the first, but important, step to developing a systematic approach in designing curriculum and instructional activities for the learning of complex systems ideas. In subsequent studies we aim to further validate the sequence with a larger sample size and use questions that involve other types of complex systems in order to determine how robust it is as a general complex systems heuristic.
Reference
Alonzo, A. & Steedle, J. (2008). Developing and assessing a force and motion learning progression. Science Education, 93, 389-421. doi: 10.1002/sce.20303. American Association for the Advancement of Science, AAAS (2009). Benchmarks for scientific literacy. Retrieved from http://www.project2061.org/publications/bsl/online/index.php on Mar 1, 2010. Ben-Zvi Assaraf, O., & Orion, N. (2005). Development of system thinking skills in the context of earth system education. Journal of Research in Science Teaching, 42(5), 518-560. doi: 10.1002/tea.20061. Ben-Zvi Assaraf, O., & Orion, N. (2010a). System thinking skills at the elementary school level. Journal of Research in Science Teaching, 47(5), 540-563. doi: 10.1002/tea.20351. Ben-Zvi Assaraf, O., & Orion, N. (2010b). Four case studies, six years later: Developing system thinking skills in junior high school and sustaining them over time. Journal of Research in Science Teaching, 47(10), 1253-1280. doi: 10.1002/tea.20383. Charles, E. S., & dApollonia, S. (2004). Developing a conceptual framework to explain emergent causality: Overcoming ontological beliefs to achieve conceptual change. In K. Forbus, D. Gentner, & T. Reiger (Eds.), Proceedings of the 26th annual cognitive science society. Mahwah, NJ: Lawrence Erlbaum Associates. Chi, M.T.H. (2005). Commonsense conceptions of emergent processes: Why some misconceptions are robust. The Journal of the Learning Sciences, 14(2), 161-199. doi: 10.1207/s15327809jls1402_1. Consortium for Policy Research in Education, CPRE (2009). Learning Progressions in Science: An EvidenceBased Approach to Reform. CPRE Research Report # RR-63. New York: Teachers College Columbia University. Retrieved from http://www.cpre.org/images/stories/cpre_pdfs/lp_science_rr63.pdf on August 20th, 2010. Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum. Garvin-Doxas, K.,& Klymkowsky, M.W. (2008). Understanding randomness and its impact on student learning: Lessons learned from building the biology concept inventory (BCI). Life Science Education, 7, 227233. doi: 10.1187/cbe.07-08-0063. Gotwals, A.W., & Songer, N.B. (2010). Reasoning and down a food chain: Using an assessment framework to investigate students middle knowledge. Science Education, 94, 259-281. doi: 10.1002/sce.20368. Hmelo-Silver, C.E., & Azevedo, R. (2006). Understanding complex systems: some core challenges. The Journal of the Learning Sciences, 15(1), 53-61. doi: 10.1207/s15327809jls1501_7. Hmelo, C. E., Holton, D., & Kolodner, J. L. (2000). Designing to learn about complex systems. The Journal of the Learning Sciences, 9, 247298. doi: 10.1207/S15327809JLS0903_2. Hmelo-Silver, C.E., Marathe, S., & Liu, L. (2007). Fish swim, rocks sit, and lungs breathe: Expert-novice understanding of complex systems. The Journal of the Learning Sciences, 16(3), 307-331. doi: 10.1080/10508400901413401. Hmelo-Silver, C.E., & Pfeffer, M.G. (2004). Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions. Cognitive Science, 28, 127-138. doi: 10.1207/s15516709cog2801_7. Hogan, K. (2000). Assessing students systems reasoning in ecology. Journal of Biological Education, 35(1), 22-28. Jacobson, M. (2001). Problem solving, cognition, and complex systems: Differences between experts and novices. Complexity, 6(3), 41-49. doi: 10.1002/cplx.1027. Jacobson, M. J., Kapur, M., So, H.J., & Lee, J. (2011). The ontologies of complexity and learning about complex systems. Instructional Science, 39, 763-783. doi: 10.1007/s11251-010-9147-0. Jacobson, M.J., & Wilensky, U. (2006). Complex systems in education: Scientific and educational importance and implications for the learning sciences. The Journal of the Learning Sciences, 15(1), 11-34. doi: 10.1207/s15327809jls1501_4. Johnson, S. (2002). Emergence. New York: Touchstone. Klopfer, E., Scheintaub, H., Huang, W., Wendel, D., & Roque, R. (2009). The simulation cycle: combining games, simulations, engineering and science using StarLogo TNG. E-Learning, 6(1), 71-96. Liu, L., & Hmelo-Silver, C.E. (2009). Promoting complex systems learning through the use of conceptual representations in hypermedia. Journal of Research in Science Teaching, 46(9), 1023-1040. doi: 10.1002/tea.20297.
ISLS
78
ICLS2012
Levy, S.T., & Wilensky, U. (2009). Students learning with the connected chemistry (CCI) curriculum: Navigating the complexities of the particulate world. Journal of Science Education and Technology, 18, 243-254. doi: 10.1007/s10956-009-9145-7. Levy, S.T., & Wilensky, U. (2008). Inventing a mid level to make ends meet: Reasoning between the levels of complexity. Cognition and Instruction, 26(1), 1-47. doi: 10.1080/07370000701798479. Levy, S.T., & Wilensky, U. (2011). Mining students inquiry actions for understanding of complex systems. Computers & Education, 56, 556-573. doi: 10.1016/j.compedu.2010.09.015. Mitchell, M. (2009). Complexity: A Guided Tour. New York: Oxford University Press. Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176. doi: 10.1177/014662169201600206. National Research Council, NRC (2007). Generating and evaluating scientific evidence and explanation (Chapter 5). In, R. Duschl, H. Schweingruber, and A. Shouse (Eds.), Taking Science to School: Learning and Teaching Science in Grades K-8 (pp. 129-167). Washington D.C.: National Academies Press. National Science Foundation, NSF. (2011). Empowering the Nation through Discovery and Innovation NSF Strategic Plan for Fiscal Years (FY) 2011-2016. Arlington, VA: National Science Foundation. Mohan, L., Chen, J., & Anderson, C.W. (2009). Developing a multi-year learning progression for carbon cycling in socio-ecological systems. Journal of Research in Science Teaching, 46(6), 675-698. doi: 10.1002/tea.20314 Program for International Student Assessment, PISA. (2006). PISA Released Items Science. Paris: OECD. Retrieved from http://www.oecd.org/dataoecd/13/33/38709385.pdf on Oct 14, 2011. Penner, D.E. (2000). Explaining systems: Investigating middle school students understanding of emergent phenomena. Journal of Research in Science Teaching, 37(8), 784-806. doi: 10.1002/10982736(200010)37:8<784::AID-TEA3>3.0.CO;2-E. Resnick, M. (1996). Beyond the centralized mindset. Journal of the Learning Sciences, 5, 122. doi: 10.1207/s15327809jls0501_1. Riess, W., & Mischo, C. (2010). Promoting systems thinking through biology lessons. International Journal of Science Education, 32(6), 705-725. doi: 10.1080/09500690902769946. Sewell, A. (2002). Constructivism and student misconceptions. Why every teacher needs to know about them. Australian Science Teachers Journal, 48(4), 24-28. Songer, N.B., Kelcey, B., & Gotwals, A.W. (2009). How and when does complex reasoning occur? Empirically driven development of a learning progression focused on complex reasoning about biodiversity. Journal of Research in Science Teaching, 46(6), 610-631. doi: 10.1002/tea.20313. Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theories-an embodied modeling approach. Cognition and Instruction, 24(2), 171-209. doi: 10.1207/s1532690xci2402_1. Wilensky, U., & Resnick, M. (1999). Thinking in levels: A dynamic systems approach to making sense of the world. Journal of Science Education and Technology, 8(1), 3-19. doi: 10.1023/A:1009421303064. Yoon, S.A. (2008). An evolutionary approach to harnessing complex systems thinking in the science and technology classroom. International Journal of Science Education, 30(1), 1-32. doi: 10.1080/09500690601101672. Yoon, S. A. (2011). Using social network graphs as visualization tools to influence peer selection decisionmaking strategies to access information about complex socioscientific issues. Journal of the Learning Sciences, 20(4), 549-588. doi: 10.1080/10508406.2011.563655.
Acknowledgement
This material is based upon work supported by the National Science Foundation under Grant No. 1019228. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
ISLS
79
ICLS2012
Using Adaptive Learning Technologies to Personalize Instruction: The Impact of Interest-Based Scenarios on Performance in Algebra
Candace Walkington, University of Wisconsin-Madison, 1025 W. Johnson St., Madison, WI, cwalkington@wisc.edu Milan Sherman, Portland State University, P.O. Box 751, Portland, OR, milan3@pdx.edu Abstract: Context personalization refers to the idea of adapting learning activities based on students interests and experiences. While new learning technologies make such innovations feasible, little research supports whether and how context personalization may mediate important learner outcomes, especially in mathematics. Here, we present results of a experimental study of Algebra I students (N = 145) receiving context personalization within an intelligent tutoring system, Cognitive Tutor Algebra. Results suggest that personalization allowed students to form more accurate and meaningful situation models of story problems, facilitating algebraic symbolization. Interest-based interventions may allow learners to make critical connections between personally relevant scenarios and abstract representational systems. Adaptive learning environments that personalize instruction by leveraging interests are an important future direction for advanced learning technologies.
Introduction
Collins and Halverson (2009) in the book Rethinking Education in the Age of Technology describe a compelling vision for how technology could change the nature of schooling. Among the key strengths that technologybased interactive learning environments could bring to educational practice is customization of learning. New technology innovations allow learning to be personalized to students interests, abilities, and preferences, in order to provide assistance when needed, and present instruction that is understandable, engaging, and situated in the context of what is important to leaners. Such technologies are timely, given the pressing problems with motivation that face schools today (Hidi & Harackiewicz, 2000), especially in secondary mathematics (Mitchell, 1993; Loveless, Fennel, Williams, Ball, & Banfield, 2008). One powerful way to adapt instruction using technology that is relatively widespread in mathematics education is intelligent tutoring systems (ITS; Koedinger & Corbett, 2006). These systems build a cognitive model of a learners current knowledge, and use these models to adapt problem selection, track mastery, and provide just-in-time feedback and hints. However, beyond adapting instruction to students knowledge, recently these systems have begun to consider a new and potentially important form of personalized learning presenting instruction in the context of students interests (Hidi & Renninger, 2006; Renninger, Ewen, & Lasher, 2002). Carnegie Learnings new ITS MATHia personalizes middle school mathematics problems to different topics of student interest, such as sports, art, and music (Carnegie Learning, 2011). Here, we refer to this type of adaption based on topic interest as context personalization, or, for simplicity, personalization. While many stakeholders in education, including teachers (Fives & Manning, 2005), believe such interventions support learning, there is little evidence of their effectiveness, especially in secondary mathematics. Further, little research exists that confirms or provides an explanation as to why personalization could support learning in more abstract domains such as high school algebra, which has been framed as a gatekeeper to higher-level mathematics (Kaput, 2000) with significant implications for equity and access (Moses & Cobb, 2001). Here, we present the results of a context personalization intervention within the Cognitive Tutor software that was carried out in Algebra I classes. We explore at how such technology-based innovations have the potential to mediate learning outcomes by matching instruction to student interests.
Literature Review
Interest
Personalization, as conceptualized here, has the potential to enable productive pathways to learning by activating topic interest (Ainley, Hidi, & Berndorff, 2002). Interest is defined as the psychological state of engaging and the predisposition to re-engage with particular classes of objects, events, or ideas over time (Hidi & Renninger, 2006, p. 112). Topic interest has components of both situational interest, activated by surprising, salient, or evocative features of the environment, and individual interest, a learners enduring preferences towards particular topics, activities, or objects (Mitchell, 1993; Hidi & Renninger, 2006). Individual interest is assumed to have both knowledge-related and value-related components, meaning that it is associated with both learners feelings of value for and knowledge of the topic (Renninger et al., 2002). In some cases, when learners are presented with a topic they are interested in, their familiarity with and connection to the particular scenario may allow for the knowledge and value components of individual interest to be activated. In other cases, even
ISLS
80
ICLS2012
though a problem is personalized to a general topic that interests the student, their connection with the particular story might not be very strong. In this case, an affective reaction from the situational interest spurred by topic familiarity may be more likely (Ainley et al., 2002). Thus topic interest is influenced by both individual and situational factors, and may be an important support for learning, as research has demonstrated that interest mediates attention, persistence, and engagement (Ainley, Hidi, & Berndorff, 2002; Durik & Harackiewicz, 2007; Hidi, 2001; McDaniel et al., 2000; Renninger & Wozinak, 1985). Prior studies on personalization in mathematics have shown mixed results, suggesting that there is a need to better understand how personalization can impact learner outcomes. Several studies have found learning gains for personalization (Anand and Ross, 1987; Cordova & Lepper, 1996; Chen & Liu, 2007), while others have not (Bates & Weist, 2004; Cakir & Simsek, 2010; Ku & Sullivan, 2000). In previous work (Walkington & Maull, 2011) we hypothesized that these mixed results are due to the fact that in order for personalization to be effective, the difficulty of the problems being personalized must be at the edge of learners abilities. This makes the ITS system used in the present study an especially interesting platform to study personalization, as it intelligently selects problems based on a cognitive model of performance. In addition to the mixed results in the literature, all of the previous studies have been conducted with elementary school students, and few look at personalization during the actual course of instruction where issues of interest and motivation would be most salient. Here, we present an ITS-based study of personalization during the course of algebra instruction.
Algebraic Reasoning
The present study focuses on algebra story problems (Figure 1), specifically the skill of writing of symbolic expressions from story problems. Koedinger and Nathan (2004) found that algebra story problems were easier to solve than matched symbolic equations, and attributed this to difficulties students have learning the language of algebra. Indeed, a number of studies have shown that algebraic expression-writing is a challenging skill (Walkington, Sherman, & Petrosino, 2012; Bardini, Pierce, & Stacey, 2004; Koedginer & McLaughlin, 2010; Clement, 1982; Stacey & MacGregor, 1999). Students may have a tendency to conceptualize a symbolic equation as a string of calculations rather than a statement about equality, and may assign variables multiple or shifting values (Clement, 1982; Stacey & MacGregor, 1999). Students may also have difficulty combining different parts of symbolic expressions (Koedinger & McLaughlin, 2010; Heffernan & Koedinger, 1997). Nathan, Kintsch, and Young (1992) proposed a model of algebra story problem solving where learners negotiate three levels of representation: (1) a propositional textbase containing the information given in the story in propositional form, (2) a situation model representing a qualitative understanding of the actions and relationships in the story, and (3) a problem model which includes formal mathematical operands, variables, and equations. They found that when students were given support in forming situation models through computer animations, they were better able to write equations from stories. Similarly, in a pilot study of 24 students solving normal and personalized algebra story problems, we found that personalization supported students in forming more detailed and accurate situation models of problem scenarios. Students were more likely to attempt personalized problems, reported that personalized problems easier to solve and more related to their lives, and were more likely to use informal strategies on that closely mirrored the problems action (Walkington et al., 2012). Here, we investigate this idea of supporting learners situation models on a larger scale, during the course of actual classroom instruction with an intelligent tutoring system.
Research Question
In previous work (Walkington, 2012) we found that personalization of algebra story problems promotes performance and learning gains for algebraic symbolization. However, it is not clear from our previous studies precisely why this result occurred. Here, we engage in a deeper investigation of the positive performance effect for personalization, in order to try to uncover how personalization supports learning. Thus our research question is: How and under what circumstances does personalization of algebra story problems support learning pathways? To investigate this question, we will examine the interaction between the context of a story problem and the impact of personalization, the effect of the readability level of the problem on the impact of personalization, and the impact of personalization on student mistake patterns.
Methods
This study took place during instruction as 145 9th grade Algebra I students used the Cognitive Tutor Algebra software. Cognitive Tutor is an intelligent tutoring system for Algebra I that individualizes instruction through adaptive problem selection and feedback (Koedinger & Corbett, 2006). Participants used Cognitive Tutor two days per week, most students spent several sessions working through the unit examined in this study, Unit 6. Participants were from a suburban school in the Northeastern United States that was majority Caucasian (96%), with 18% of students eligible for free/reduced lunch and 71% proficient in math on the 11th grade assessment. Participants were randomly assigned to two conditions when they entered Unit 6, which contained algebra story problems on linear functions. The control group received the standard problems for the unit, while
ISLS
81
ICLS2012
the experimental group received problems matched to their topic interests through a computer survey administered to both conditions. The survey asked students to rate their level of interest in (i.e., liking of) 9 topic areas: sports, music, movies, games, TV, computers, food, art, and shopping. Personalized problems corresponding to each topic were written based on prior surveys (N = 60) and interviews (N = 29). Although the topic of each problem was matched to students interests at the individual level, there is no measure in this study of whether the specific story in each problem was relevant to individual students experiences with that topic. However, our prior survey, interview, and observational work revealed a great deal of overlap in the individual experiences and preferences high school students typically had in relation to each of the 9 topics. There were 27 problems in the original unit, and 4 variations on each problem were written to correspond to the interest topics (Table 1). For each problem, students filled in different cells in a spreadsheet as they solved result and start unknowns. In result unknowns (questions 1-2, Figure 1), students solved for the y variable in a linear function given a specific x value, which involves directly applying the described operations. For start unknowns (questions 3-4, Figure 1), students solved for the x variable in a linear function given a y value, which may require working backwards. Students also wrote algebraic expressions from story scenarios. Table 1: Example of four personalized variations on original algebra story problem from Unit 6 Original Problem Sports Music Art Games One method for estimating the cost of new home construction is based on the proposed square footage of the home. Locally, the average cost per square foot is estimated to be $46.50. You are working at the ticket office for a college football team. Each ticket to the first home football game costs $46.50. You are helping to organize a concert where some local R&B artists will be performing. Each ticket to the concert costs $46.50. You have been working for the school yearbook, taking pictures and designing pages, and now its time for the school to sell the yearbooks for $46.50 each. You work for a Best Buy store that is selling the newest Rock Band game for $46.50.
Figure 1. Example of normal story problem scenario - upper text shows result and start unknown questions posed, while lower table displays correct responses to each problem part The data were analyzed using hierarchical logistic regression (Snijders & Bosker, 1999), where problem parts were nested within students who were nested within teachers. Random intercept terms were included for which story problem the student was working on, and what linear function (i.e., y=-1550 7x) or item was being described in the story problem. The dependent variable was whether the student got the problem part correct on the first attempt. Fixed effects included which condition the student was in (experimental group - personalized problems or control group - normal problems), the difficulty of the knowledge component being assessed in the problem part, and the interaction between condition and knowledge component difficulty. A knowledge component (KC; Koedinger & Aleven, 2007) is a skill that is tracked for mastery by Cognitive Tutor, and KCs were classified as easy, medium, or hard based on student performance data. Hard KCs included writing algebraic expressions with slope and intercept terms, while medium KCs included solving result and start unknowns, working with different types of numbers, and writing expressions with only a slope. Easy KCs included identifying independent and dependent units and entering given values.
ISLS 82
ICLS2012

The output from the regression model of performance in Unit 6 is shown in Table 2. Personalization increased performance when students were solving problem parts with hard KCs (odds = 1.5228, p < .001) and easy KCs (odds = 1.5228*.9368 = 1.4266, p < .001). The effect of personalization on medium KCs did not quite reach significance (odds = 1.5228*.7689 = 1.1709, p = 0.0601). Personalization supported students algebraic expression-writing for linear functions that had both a slope and intercept term (hard KCs), and also facilitated the solving of less mathematically-relevant portions of the problem, like identifying units (easy KCs). Table 2: Output for hierarchical logistic regression model of performance within Unit 6 Raw Coeff Std. Error Exp(Coeff) (Intercept) -0.27378 0.22707 0.7605 Condition-Control Ref. Condition-Experimental 0.42053 0.11134 1.5228 KC-Hard Ref. KC-Medium 1.32951 0.05986 3.7792 KC-Easy 2.17917 0.06105 8.8390 Condition-Exp:KC-Medium -0.26276 0.08255 0.7689 Condition-Exp:KC-Easy -0.06528 0.08484 0.9368 Note. Significance codes: *** 0.001, ** 0.01, * 0.05 z value -1.21 3.78 22.21 35.69 -3.18 -0.77 Sig
*** *** *** **
Although it is not included here for brevity, analyses of students work in a future unit of Cognitive Tutor showed that personalization not only increased performance on hard KCs within Unit 6, it significantly increased performance on hard KCs in the subsequent unit, after the personalization intervention had been removed (Walkington, 2012). This suggests that the performance boost associated with personalization in Unit 6 facilitated students learning of the concept of algebraic expression-writing. Thus we determined that it is critical to engage in a deeper investigation of what promoted or caused the performance differences seen in Unit 6, in order to better understand why personalization ultimately impacted students long-term learning.
Problem Cover Stories

We begin by examining which of the story problems in Unit 6 benefitted most from being personalized. This may indicate classes of story problems for which personalization is particularity helpful. To do this, we added a condition by item random slope to the model shown in Table 3. This term essentially shows how the condition (personalized versus non-personalized) interacted with each base story problem to impact student performance. We concentrated on two groups of problems for this analysis. First, we looked at problems with random slopes indicating that personalization reduced the odds of getting the problem correct, corresponding to odds values of 0.7 to 0.9. Second, we looked at problems with random slopes indicating that personalization increased the odds of getting the problem correct, corresponding to odds values of 1.1 to 1.4. The results are shown in Table 3. Table 3: The impact of personalization on performance for selected story problems in Unit 6 Topic of Original Problem Cell phone service Getting raise at work Pay from work Selling paintings Mail-order shopping Sending cards to friends Paying cell phone bill Making wages at work Nitrogen in an asteroid Home square footage Taking pictures with film Installing carpet Trading wampum shells Taking vitamins Breaking crayons Linear Function y = 60-5x y = .04x y = .72x y = 64-6x y = x -.025x+10 y = 55-.5x y = -.23x+7.87 y = 10.50x y = 90-2x y = 46.50x y = 36-6x y = 12.95x y = 80-6x y = 50-2x y = 24-2x Personalizations Impact on Odds 0.73 0.74 0.75 0.82 0.85 0.86 0.86 0.87 1.13 1.13 1.15 1.19 1.22 1.22 1.39
Personalization Reduced performance Reduced performance Reduced performance Reduced performance Reduced performance Reduced performance Reduced performance Reduced performance Increased performance Increased performance Increased performance Increased performance Increased performance Increased performance Increased performance
ISLS
83
ICLS2012
Several trends are apparent from Table 3 that provide insight into how personalization may support learning. First, personalization was not particularly helpful when the problems topic was already relevant to students lives and interests like paying for their cell phone or getting paid at work. In these cases, the original problem could actually be slightly easier for students to solve than the personalized versions. Most of the problems where personalization was not beneficial dealt with money. High school students already have strong familiarity with doing calculations with money, especially in the context of work and shopping (Walkington et al, 2012), thus it makes sense that personalizing such a problem is unlikely to further improve performance. The instances in Table 3 where personalization increased performance are also revealing high school students are not likely to have direct experience with calculations relating to nitrogen in an asteroid or installing carpet, thus personalization of these problems was particularly beneficial. Taken together, these results suggest that personalization supports performance when irrelevant story contexts become more situated within students experiences and interests, and the ways in which they encounter numbers in their everyday life. Thus personalization may be effective in supporting learners in forming detailed and comprehensible situation models (Nathan et al., 1992) of problem scenarios, allowing them to leverage their prior knowledge of familiar contexts.
Readability Level
One reason why personalization may increase performance is that personalized problems are easier for students to read and interpret. This is supported by the data shown in the previous section, which suggested that personalization had the largest impact when it was used on problems with contexts that were not familiar or relevant to students experiences. To examine this hypothesis, we added the readability level (Flesch, 1948) of the story scenario to the hierarchical model as a predictor. Readability level is an index used to measure comprehension difficulty, and uses indicators such as number of words, number of sentences, and syllables. Each story problem was classified as below grade level if the Flesch-Kincaid grade level test predicted its readability was appropriate for grades 4-7, at grade level if it was appropriate for grades 8-10, and above grade level if it was appropriate for grades 11-15. This addition also allowed us to check whether differences in performance for personalized problems were being driven solely by systematic differences in readability level. Results show that personalization still had a significant positive effect on performance on easy (odds = 1.397, p < .001) and hard (odds = 1.489, p < .001) KCs when controlling for readability level. The model also showed a significant interaction between readability level and condition (2(2) = 6.364, p < .05), and readability level and knowledge component difficulty (2 (4) = 39.532, p < .001). The results are summarized in Figure 2. For both easy and hard KCs, when a problem was easy to read (below grade level) the personalization condition had higher performance. However, the advantage of the personalization condition became even greater as the problems became more difficult to read. So, for example, if a student was presented with normal problem that was difficult to read, and a personalized problem that was also difficult to read, their performance would be considerably higher on the personalized problem. Whereas different readability levels did not seem to impact performance on personalized problems, increasing grade level readability had a considerable negative effect on performance on normal problems. The model showed that personalization had significantly greater positive impact on performance when the text of the problem was above grade level, for both hard and easy KCs (odds = 1.424, p < .05), compared to performance on these KCs on below grade level texts.
Figure 2. The impact of readability level on problem-solving performance for control (normal problems) and experimental (personalized problems) groups, separated by Easy KCs (left) and Hard KCs (right) These results suggest that personalization allows students to better interpret difficult verbal scenarios of complex mathematical situations, and that this effect is especially large when students are asked to write algebraic expressions from story scenarios. The students familiarity with personalized story contexts, due to the
ISLS
84
ICLS2012
connection with their interests, may allow students to better understand the relationships within a story scenario with a complicated verbal structure, facilitating the formation of detailed and accurate situation models that support expression-writing. Thus personalization may allow learners to better move from their propositional textbase to an elaborated understanding of the storys actions and relationships.
Analysis of Mistakes
Personalization had an important positive impact on performance and learning for hard KCs, when students were writing symbolic equations from story scenarios. The analyses above suggest that successfully writing a symbolic equation from a story is tied to the students verbal comprehension of the text and their ability to draw on their prior knowledge to form a situation model of the actions and relationships being described. We conducted an analysis of students interactions with the tutor when solving hard KCs in order to see if the situation model support provided by personalization allowed students to avoid certain kinds of mistakes. As can be seen from Table 4, when students were presented with personalized problems, they were more likely to write an algebraic expression successfully on their first attempt, were less likely to initially ask for a hint, and were less likely to make a mistake and write the expression incorrectly. Results of the HLM analysis presented at the beginning of the section show that the difference in correct answers for the control group (42.6%) versus the experimental group (51.6%) is a statistically significant (p < .001) trend. Table 4: Student performance on Hard KCs Outcome First Attempt Correct First Attempt Hint First Attempt Incorrect Control 42.6% 8.2% 49.2% Experimental 51.6% 4.9% 43.4%
As can be seen from Table 5, there are also differences in mistake patterns that may account for the greater success of the experimental group in writing algebraic expressions. The experimental group was equally likely to make simple sign reversal errors on either the slope or the intercept term. However, the experimental group was less likely to mix up the slope and intercept parameters or use incorrect signs for both the slope and intercept. The experimental group was also slightly less likely to not include a slope coefficient (i.e., use a slope of 1), not include an intercept term, or leave out both the slope term and the independent variable. Generally, this seems to imply that personalization assists students in remembering to include the appropriate parameters and variables in their expression, while also helping them make sense of how these parameters should be combined. This suggests that personalization reduces the likelihood of large conceptual misspecifications of the functional relationship by supporting students situation models. In this way, personalization may allow students to meaningfully create a mathematical model for a situation, allowing them to coordinate their understanding of the story with a formal symbolic representation. This is contrasted with a purely procedural or syntactic construction of an algebraic expression, where situational understanding is not well connected to symbolic reasoning. Such approaches may be more prone to major conceptual errors, as shown by research on direct translation strategies when solving arithmetic word problems (Hegarty, Mayer, & Monk, 1995). Table 5: Mistakes made by students when writing symbolic expressions from story scenarios Write Equation Mistake Entered slope of 1 Mixed up slope and intercept No intercept Only intercept Intercept wrong sign Slope wrong sign Wrong sign, slope and intercept Control 111 57 146 16 166 116 104 Experimental 103 39 136 9 165 114 118
Summary and Conclusion

In previous work, we found that personalization to students interests mediates performance and learning in potentially powerful ways, supporting students in navigating the abstract representational systems in algebra. Here, we took a closer look at the data from our studies in order to investigate why such an effect occurs. We conclude that personalization may support students formation of more detailed, accurate, and meaningful situation models of problem scenarios. This is evidenced by findings that the effect of personalization is largest when it makes an irrelevant problem relevant to students experiences and when it makes the actions and relationships in a complex verbal scenario more easily read and interpreted. Personalization also seemed to
ISLS 85
ICLS2012
allow students to make fewer large conceptual mistakes when specifying the relationships in a story, suggesting that personalization supports coordination of situation and problem models. Rather than creating a general motivational effect that was seen throughout learners problem-solving, the impact of topic interest on performance was targeted. Interest facilitated performance when students were working directly with the scenario, naming quantities and units and formulating a symbolic equation of the relationships. When students wrote equations, interest seemed to support understanding of the deeper, quantitative structure of the story. This suggests that for algebra story problems, the effectiveness of interestbased interventions lies in their ability to allow productive connections between situational contexts and formal models of situations. As the context becomes more comprehensible and connected to prior experience, students are better able to coordinate their situation and problem models in the process of problem-solving. We hypothesize that personalization could be considered a form of grounding (Goldstone & Son, 2005; Koedinger, Alibali, & Nathan, 2008) that allows for abstract formalisms to be coordinated with concrete everyday experiences. Here, we presented evidence that the grounding provided by personalization becomes more effective as the story scenarios become more related to the interests, language, and activities the students are familiar with. Further, we described how in related work (Walkington, 2012) providing such grounding not only facilitated performance on personalized problems, but also allowed for robust learning of underlying algebraic ideas. This suggests that grounding interventions based on topic interest are not simply a crutch for learners to lean on when they struggle or become disengaged they are a scaffold that has the potential to allow learners to come to understand powerful systems of representation. With the rise of advanced learning technologies, the potential to connect instruction to both students interests and to their current level of understanding has become a reality. It is critical that we understand how this technology impacts students cognition and problem-solving, such that it can be leveraged in schools to enable productive pathways to learning. Here, we show how interest-based interventions can mediate students situational understanding of mathematical relationships in story problems, fostering critical connections between personalized scenarios and abstract representational systems. It is both surprising and promising that such a simple alteration to instructional materials to make them more adaptive had an important impact on both performance and long-term learning of difficult concepts in secondary mathematics. As learning technologies advance, we thus propose that an important future direction is designing environments that can adapt to student background and preferences in a deeper and more meaningful manner, by leveraging the interests that drive and motivate students in authentic and powerful ways.
References
Ainley, M., Hidi, S., & Berndorff, D. (2002). Interest, learning, and the psychological processes that mediate their relationship. Journal of Educational Psychology, 94(3), 545-561. Anand, P., & Ross, S. (1987). Using computer-assisted instruction to personalize arithmetic materials for elementary school children. Journal of Educational Psychology, 79(1), 72-78. Bardini, C., Pierce, R., & Stacey, K. (2004) Teaching linear functions in context with graphics calculators: students' responses and the impact of the approach on their use of algebraic symbols. International Journal of Science and Mathematics Education, 2, 353-376. Bates, E., & Wiest, L. (2004). The impact of personalization of mathematical word problems on student performance. The Mathematics Educator, 14(2), 17-26. Caker, O., & Simsek, N. (2010). A comparative analysis of computer and paper-based personalization on student achievement. Computers & Education, 55, 1524-1531. Carnegie Learning (2011). Carnegie Learning Math Series: Carnegie Learning MATHia Software. Retrieved May 24, 2011 from http://mathseries.carnegielearning.com/product-info/software Chen, C., & Liu, P. (2007). Personalized computer-assisted mathematics problem-solving program and its impact on Taiwanese students. Journal of Computers in Mathematics and Science Teaching, 26(2), 105-121. Clement, J. (1982). Algebra word problem solutions: Thought processes underlying a common misconception. Journal for Research in Mathematics Education, 13(1), 16-30. Collins, A., & Halverson, R. (2009). Rethinking Education in the Age of Technology: The Digital Revolution and Schooling in America. New York: Teachers College Press. Cordova, D., & Lepper, M. (1996). Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice. Journal of Educational Psychology, 88(4), 715-730. Durik, A., & Harackiewicz, J. (2007). Different strokes for different folks: How individual interest moderates effects of situational factors on task interest. Journal of Educational Psychology, 99(3), 597-610. Fives, H., & Manning, D. (2005). Teachers' strategies for student engagement: Comparing research to demonstrated knowledge. Paper presented at 2005 Annual Meeting of American Psychological Association, Washington DC. Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32(3), 221-233.
ISLS
86
ICLS2012
Goldstone, R., & Son, J. (2005). The transfer of scientific principles using concrete and idealized simulations. Journal of the Learning Sciences, 14(1), 69-110. Heffernan, N. T., & Koedinger, K. R. (1997).The composition effect in symbolizing: The role of symbol production vs. text comprehension. In Proceedings of the nineteenth annual meeting of the Cognitive Science Society (pp. 307-312). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Hegarty, M., Mayer, R., & Monk, C. (1995). Comprehension of arithmetic word problems: A comparison of successful and unsuccessful problem solvers. Journal of Educational Psychology, 87(1), 18-32. Hidi, S. (2001). Interest, reading and learning: Theoretical and practical considerations. Educational Psychology Review, 13, 191210. Hidi, S., & Harackiewicz, J. (2000). Motivating the academically unmotivated: A critical issue for the 21st century. Review of Educational Research, 70, 151179. Kaput, J. J. (2000). Teaching and learning a new algebra with understanding. U.S.; Massachusetts: National Center for Improving Student Learning and Achievement. Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19, 239-264. Koedinger, K., Alibali, M., & Nathan, M. (2008). Trade-offs between grounded and abstract representations: Evidence from algebra problem solving. Cognitive Science, 32, 366-397. Koedinger, K. R., & Corbett, A. (2006). Cognitive Tutors - Technology Bringing Learning Sciences to the Classroom. In R. K. Sawyer (ed.) The Cambridge Handbook of the Learning Sciences. St. Louis, Cambridge University Press: 61-77. Koedinger, K., & McLaughlin, E. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society. (pp. 471-476.) Austin, TX: Cognitive Science Society. Koedinger, K., & Nathan, M. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. Journal of the Learning Sciences, 13(2), 129-164. Ku, H., & Sullivan, H. (2000). Personalization of mathematics word problems in Taiwan. Educational Technology Research and Development, 48(3), 49-59. Loveless, T, Fennel, F., Williams, V., Ball, D., & Banfield, M. (2008). Chapter 9: Report of the Subcommittee on the National Survey of Algebra I Teachers. In Foundations for Success: Report of the National Mathematics Advisory Panel. Retrieved 14 October 2010 from http://www2.ed.gov/about/bdscomm/list/mathpanel/report/nsat.pdf. McDaniel, M., Waddill, P., Finstad, K., & Bourg, T. (2000). The effects of text-based interest on attention and recall. Journal of Educational Psychology, 92(3), 492-502. Mitchell, M. (1993). Situational interest: Its multifaceted structure in the secondary school mathematics classroom. Journal of Educational Psychology, 85, 424436. Moses, R., & Cobb, C. (2001). Radical Equations: Math Literacy and Civil Rights. Boston: Beacon Press. Nathan, M., Kintsch, W., & Young, E. (1992). A theory of algebra-word-problem comprehension and its implications for the design of learning environments. Cognition and Instruction, 9(4), 329-389. Renninger, K., Ewen, L., & Lasher, A. (2002). Individual interest as context in expository text and mathematical word problems. Learning and Instruction, 12(4), 467-490. Renninger, K., & Wozinak, R. (1985). Effect of interest on attentional shift, recognition, and recall in young children. Developmental Psychology, 21(4), 624-632. Snijders, T. & Bosker, R. (1999). Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. Sage Publications. Stacey, K., & MacGregor, M. (1999). Learning the algebraic method of solving problems. Journal of Mathematical Behavior, 18(2), 149-167. Walkington, C. (April, 2012). Context personalization in algebra: Supporting connections between relevant stories and symbolic representations. Paper presented at the 2012 Annual Meeting of the American Educational Research Association. Vancouver, Canada. Walkington, C., & Maull, K. (2011). Exploring the assistance dilemma: The case of context personalization. In L. Carlson, C. H lscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 90-95). Boston, MA: Cognitive Science Society. Walkington, C., Sherman, M., & Petrosino, A. (2012). Playing the game of story problems: Coordinating situation-based reasoning with algebraic representation. Journal of Mathematical Behavior, 31(2), 174195.
Acknowledgments
This work was conducted in partnership with Carnegie Learning, and was supported by the Pittsburgh Science of Learning Center which is funded by the National Science Foundation award # SBE-0354420.
ISLS
87
ICLS2012
The Impact of Students Exploration Strategies in Discovery-Based Instructional Software

Associate Professor Barney Dalgarno, Charles Sturt University, bdalgarno@csu.edu.au Associate Professor Gregor Kennedy, University of Melbourne, gek@unimelb.edu.au Associate Professor Sue Bennett, University of Wollongong, sbennett@uow.edu.au Abstract: Active learning is a key element of constructivist learning theory and has been used as an argument for employing discovery-based designs with instructional software. On the other hand, researchers have highlighted empirical evidence showing that pure discoverybased learning is of limited value. This suggests that how learners interact is important in predicting whether learning occurs. This paper reports on a study of 158 university students who each used two instructional simulations one with a discovery-based design and the other with a tutorial-based design. Students learning outcomes were assessed via pre-tests and post-tests of conceptual understanding. Students interactions using the discovery-based program were recorded and coded as either systematic or unsystematic. The results showed that when compared with the tutorial-based learning program systematic exploration resulted in learning benefits, while unsystematic exploration did not. These results have implications for the design of instructional software if such resources are to be used effectively.
Introduction
A key element of constructivist theories of learning is the idea that each person forms their own representation of knowledge, building on their individual experiences an idea attributed primarily to Piaget (1973). This knowledge representation is constantly reviewed and revised as inconsistencies between the current knowledge representation and experience are encountered through active exploration (Bruner, 1962; Piaget, 1973). Piaget (1973) explains the learning process in terms of equilibration. Equilibration begins with the construction by the individual of their own internal knowledge representation, or in Piagets terms they accommodate their knowledge representation to fit with their experience. Subsequent experiences that are consistent with this knowledge representation are then assimilated into their knowledge representation. New experiences that do not fit with their current knowledge representation result in a further accommodation of their knowledge representation to fit with this new experience. This idea that learning involves active knowledge construction has been used in support of various learning design frameworks, including inquiry-based learning (particularly in the sciences), and discovery-based learning using educational multimedia and computer-based simulations. On the other hand, researchers such as Mayer (2004) and Kirschner, Sweller and Clarke (2006) have highlighted empirical evidence showing that pure discovery-based learning is of limited value. This suggests that how learners interact is an important avenue for further research. Specifically, the relationship between a learners interaction with an instructional software program and his or her cognition has been identified as an important factor in predicting whether learning occurs (see Kennedy, 2004). This paper reports on research investigating how learners different exploration strategies impacted on their cognitive processes and consequently on their learning outcomes.
Background
Interest in the potential learning benefits of interactive multimedia over the past 15 to 20 years was driven initially by the advent of the personal computer and its evolving multimedia capabilities, and subsequently by the advent of the Internet, making resources more readily located and acquired. Early multimedia took the form of programmed instruction or tutorials in which the learner would work through a linear sequence of instructional screens or drill-and-practice activities which provided an opportunity for learners to perfect their responses with immediate feedback. These designs were typically based on behaviourist theories of learning and so limited choice was available to learners. More recently the range of instructional software available has expanded but this type is still widely available (eg. Mathletics). Alternative types of multimedia software began to emerge in the 1990s, with resources like Investigating Lake Iluka (Harper, Hedberg & Brown, 1995) and the Jasper Woodbury series (Cognition and Technology Group At Vanderbilt, 1992), which allowed non-sequential inquiry-based exploration of a computer-based learning environment scaffolded by the provision of holistic problem scenarios. The design of these types of interactive learning environment has been underpinned by the principles of active, inquiry-based learning that occurs in context and promotes knowledge construction and articulation (see Duffy & Cunningham, 1996). Early research questions included: how multiple media should be combined to present content; how multimedia information might be structured; how learners might control the pace or the route by which they
ISLS 88
ICLS2012
moved through the data; and how the interface might be designed so learners would know where to click. This research agenda led to numerous experimental studies drawing on cognitive theories through which principles for designers and developers were derived (eg. Chandler and Sweller, 1991; Mayer; 2001; Sweller, 2005). Another body of research in the field emerged in the form of small-scale naturalistic studies of the use of multimedia resources by learners in particular contexts resulting in the derivation of design guidelines incorporating active exploratory learning underpinned by constructivist approaches to teaching (eg. Harper & Hedberg, 1997; Herrington & Oliver, 2000). These two research agendas have developed in parallel and at times at odds with one another. Recently, Mayers (2004) conclusion that unscaffolded discovery learning is of limited or no value has re-ignited debate over how much direction is optimal in the design and use of instructional software (see also Kirschner, Sweller & Clarke, 2006; Kapur, 2008; Hmelo-Silver, Duncan & Chinn, 2007) In summary, despite changes in technology over the past decade, interactive multimedia remains relevant, while researchers understanding of the role interactivity plays in learning is limited. It is well accepted from a theoretical standpoint that there is great potential in exploratory resources that allow learners to actively construct their own knowledge representation through interaction with the environment, however the empirical results are mixed. Consequently there is a need for further research that explores the relationship between the design of the resource, the interactive tasks undertaken by the learner, and the cognition and learning that occurs as a result. The study described in this paper was intended to contribute to knowledge about the relationship between interactive learning and cognition through an exploration of the learning performance of students using a multimedia resource with a discovery-based design compared to students using a resource with a tutorialbased design. As well as allowing for a direct comparison of the two groups the study also collected data on student prior knowledge, engagement and cognitive load as well as logging student interactions in order to characterise their exploration strategies. The focus of this paper is on the way in which exploration strategy affects learning performance. Future articles will report on the relationship between prior knowledge, engagement, cognitive load and learning performance.
Methodology
The study compared learning performance using a discovery-based design with performance using a tutorialbased design in each of two content domains, as shown in Table 1. The study was conducted with 158 University of Wollongong teacher education students. Each student completed a discovery-based learning condition using a multimedia resource focussing on one content domain and a tutorial-based learning condition using a multimedia resource focussing on the other content domain. The order was varied so that approximately half of the students undertook their tutorial learning condition first while the other half undertook their discovery learning condition first. Prior to each learning condition the students completed a knowledge pre-test. After each learning condition the students completed a knowledge post-test and a questionnaire with engagement and cognitive load items. Student actions within the learning resources were also logged to allow later analysis of their exploration strategies. For analysis purposes the two content domains were treated as distinct experiments. In each case the independent variable was learning condition, with two levels, Tutorial and Discovery, and the dependent variable was learning performance. The decision to essentially undertake two concurrent experiments using two distinct content domains and two distinct sets of multimedia resources was made because, being mindful of the potential for content or resource specific factors to impact on the results, we were keen to have two distinct data sets to draw upon in making conclusions from the data. The content domains, blood alcohol concentration and global warming, were chosen because of their widespread interest within the likely participant community. Table 1: Experimental Design Content Domain Blood Alcohol Concentration N=73 N=85
Condition Tutorial Discovery
Global Warming N=85 N=73
To address issues encountered in earlier studies (see, for example, Dalgarno, Bennett & Harper, 2010), resources and learning tasks were designed to ensure alignment with the intended learning outcomes. The discovery resources were designed around a predict-observe-explain learning design (White & Gunstone, 1992), with students encouraged to mentally predict the effect of their simulation parameter changes, observe the results of the change and mentally try to explain the observed results. Figures 1 and 2 provide excerpts from the discovery-based resources showing the screens on which participants could change variables on the lefthand side of the screen before running the simulation to see the results. In the case of global warming, the four graphs on the right-hand side of the screen, showing values for ozone layer thickness, CO 2 concentration,
ISLS 89
ICLS2012
greenhouse insulation and surface temperature, changed if the variables were altered. In the blood alcohol concentration simulation the graph on the right-hand side showing values for blood alcohol concentration over time, changed if the input variables were altered. As far as possible the user interface was consistent across the two topics to maximise comparability between the resources.
Figure 1. Excerpt from the Global Warming Discovery Resource
Figure 2. Excerpt from the Blood Alcohol Concentration Discovery Resource The tutorial-based resources for each topic area were created from the discovery resource, but instead of allowing students to change the variables a series of pre-defined values and the relevant graphical outputs were shown. Students were provided with a single option, a continue button, which resulted in the next set of predefined values being loaded and the corresponding output displayed. This is illustrated in Figures 3 and 4.
ISLS
90
ICLS2012
Figure 3. Excerpt from the Global Warming Tutorial Resource
Figure 4. Excerpt from the Blood Alcohol Concentration Tutorial Resource The knowledge pre-tests and post-tests contained a series of identical items designed to measure understanding of key concepts within the content area. For example, within the Blood Alcohol Concentration content area, the following item was included within the pre-test and post-test to explore the participants understanding of the relationship between of a persons weight and their blood alcohol concentration: A person with greater body weight: a) Will have a higher Blood Alcohol Concentration (BAC) than a lighter person. b) Will have a lower BAC than a lighter person. c) Will have the same BAC as a lighter person. d) Will have their BAC increase at a greater rate than a lighter person
ISLS
91
ICLS2012
The following item was included within the Global Warming pre-test and post-test to test the participants on their understanding of the way various environmental factors affect the global temperature: Which of the following environmental factors have a direct or an indirect effect on the Global Average Surface Temperature (GAST)? a) The amount of Carbon Dioxide (CO2) absorbed by plants b) The thickness of the ozone layer c) The percentage of CO2 in the atmosphere d) The Greenhouse insulation effect Some questions were specifically designed to test whether participants held certain common misconceptions within each content area. Some of the questions, such as the Global Warming example question above, had multiple correct stems while some, such as the Blood Alcohol Concentration example question above, had a single correct stem. Students scored one point for each question in which only the correct stems were chosen. There were seven questions in the Global Warming test and nine questions in the Blood Alcohol Concentration test.
Results
The first analysis considered whether participants who had completed the program with a discovery-based or tutorial-based learning design showed improved understanding. As shown in Table 2, an initial analysis of the learning which occurred in each condition was undertaken using paired T tests which compared the Pre-Test and Post-Test scores. The results of this analysis indicated that for the Global Warming content area, which participants found quite challenging, there was a slight decrease in test performance from the Pre-Test to the Post-Test by Tutorial participants and no difference in performance for Discovery participants. For the Blood Alcohol Concentration content area there was no difference in test performance for the Tutorial participants and an increase for Discovery participants. For each content domain (Global Warming and Blood Alcohol Concentration) an analysis of covariance (ANCOVA) test was completed which included post-test score as the dependent variable, learning design condition as the independent variable (discovery, tutorial), and pre-test score as a covariate. For the Global Warming content domain there was no effect for condition (F (1,155) = 2.40; p = .124) but there was a significant effect for participants pre-test scores on their post-test understanding (F (1,155) = 27.50; p < .001). For the Blood Alcohol Concentration, there was an effect for both condition (F (1, 155) = 5.52; p = .02) and the participants pre-test scores on their post test understanding (F (1,155) = 16.40; p < .001). It can be seen from the mean scores and standard deviations presented in Table 2, that despite these significant result, particularly in the content area of Blood Alcohol Concentration, in general participants showed very little improvement in performance from pre- to post-test, regardless of content domain or condition. Given the maximum scores that could be obtained by students, performance scores were well below the mid point of the scale for both content domains. Table 2: Mean pre- and post-test scores for both conditions across the two content domains Content Domain Global Warming+ Blood Alcohol^
+max = 7; ^max = 9
Condition Tutorial (n=85) Discovery (n=73) Tutorial (n=73) Discovery (n=85)
Pre-Test M (SD) 1.82 (1.51) 1.68 (1.42) 3.55 (1.25) 3.60 (1.24)
Post Test M (SD) 1.42 (1.29) 1.72 (1.85) 3.42 (1.31) 3.93 (1.40)
T 2.26 (p=0.027) 0.20 (p=0.841) 0.60 (p=0.552) 2.33 (p=0.022)
ISLS
92
ICLS2012
We also noticed that the variance in post-test scores for participants discovery-based condition, particularly for the global warming content domain, was quite high; and we wondered whether there may be variation within the discovery-based group that reflected differences in how the simulation was used and explored by participants. A preliminary analysis showed that some participants seemed to have explored the simulation systematically and others were more unsystematic in their approach. Given this, we characterised participants strategies while interacting with the simulation-based learning activity as Systematic Discovery if they investigated the content area by running the simulation by changing only one variable from the provided example (e.g. Bills values or 2006 values) or from previous simulation outputs on four or more occasions. All other discovery participants strategies were classified as Non Systematic Discovery. Table 3 shows the implicit experimental design once the discovery participants were separated into two groups using this characterisation. The new design again has learning condition as the independent variable, but with three levels, Tutorial, Non-Systematic Discovery and Systematic Discovery. The criteria and thresholds for distinguishing between non-systematic and systematic discovery participants could have been arrived at in a number of ways. We experimented with cluster analysis using a number of the indicators of exploration approach (time spent, iterations undertaken, variables changed etc) but ultimately decided to use the number of variables changed as a simple, intuitively meaningful criteria, and chose thresholds that ensured sufficiently sized and meaningful sub-groups. In order to investigate whether this characterisation of participants exploration strategies in the simulator was better able to account for variance in post-test performance scores, an ANCOVA for each content area was again calculated but this time with a three-level independent variable (tutorial, non-systematic discovery, and systematic discovery), as shown in Table 4. For both content domains there were again significant covariate effects for the pre-test scores on the post-test scores (Global Warming (F (1,154) = 23.72; p < .001); Blood Alcohol (F (1,154) = 23.72; p < .001)). Main effects for condition were also recorded for both content domains. In both content domains the pattern of results was the same. After accounting for pre-test knowledge, post-test comparisons indicated that participants who were in the systematic discovery group recorded significantly higher post-test scores than participants in both the non-systematic discovery and the tutorial groups. Interestingly, there were no differences between these latter two groups in terms of their understanding (post-test) for either content domain. Table 3: Experimental Design after Characterising Discovery Strategies Content Domain Blood Alcohol Concentration N=73 N=51 N=34
Condition Tutorial Non-Systematic Discovery Systematic Discovery
Global Warming N=85 N=48 N=25
Table 4: Significant main effects for condition (three levels) across the two content domains Content Domain Post-Test Tutorial M (SD) 1.42 (1.29) a 3.42 (1.31) a Post-Test Non Systematic Discovery M (SD) 1.33 (1.52) a 3.51 (1.30) a Post-Test Systematic Discovery M (SD) 2.48 (2.20) b 4.56 (1.33) b F p
Global Warming+ Blood Alcohol^

ab
4.17 8.69
.017 <.001
different superscript across rows indicate between group differences (p < .001).
Discussion
Take Home Messages and Implications
The comparison of the post-test performance of students who undertook their learning using a learning resource with a tutorial-based design with those who used a resource with a discovery-based design found a small but significant difference in test performance in one content domain and no significant difference in the other. This absence of a consistent benefit for pure discovery learning is consistent with Mayers (2004) argument against the use of such learning designs. However, when the exploration strategies of participants using the discoverybased resources were analysed it became clear that some students explored in a systematic fashion and some did not, and that there was a clear difference in test performance of systematic and unsystematic explorers within
ISLS
93
ICLS2012
each content area. Additionally, systematic explorers in each content area performed significantly better than tutorial participants. This suggests that, contrary to Mayers (2004) arguments, for some students, an active discovery-based design may be ideal. However, for others, the benefits of active exploration are countered by the confusion caused by unsystematic exploration strategies and consequently they do no better than tutorial participants. An important question, then, is what led some students to explore the simulation systematically and others nonsystematically. Kirschner, Sweller and Clarke (2006) suggest that the value of instructional guidance only begins to diminish when learners have sufficient prior knowledge to allow for internal guidance. On the other hand, it may be that some learners have a natural aptitude for systematic exploration or through prior experience of more structured inquiry-based learning designs, such as problem-based learning (see Hmelo-Silver, Duncan & Chinn, 2007), they have developed more systematic exploratory approaches. The concept of scaffolding (Wood, Bruner, & Ross, 1976) is helpful in thinking about learning designs which capitalise on Piagets ideas about active learning while providing sufficient guidance to help prevent the learner becoming confused. Scaffolding could be provided by a teacher, with the simulation explored by a student on an interactive whiteboard while the teacher directs students attention to the emergent concepts. Alternatively, the tutorial resource could be enhanced by the inclusion of explanatory text or audio, similarly directing the students attention to the key concepts (see also Lee & Dalgarno, 2011, for a more detailed discussion of scaffolding in discovery-based learning environments).
Follow up Research
We are currently analysing the results of the engagement and cognitive load questionnaires to determine whether there is a relationship between resource type and these factors, or between these factors and learning outcomes. The relationship between prior knowledge and engagement and between prior knowledge and exploration strategy is another avenue being explored, which will help to inform on Kirschner, Sweller and Clarkes (2006) proposition that learners with higher prior knowledge have less need for instructional guidance. We have also undertaken a pilot study using the same experimental design but with participants in an MRI scanner to determine whether brain activation differences are evident between the two conditions (see Dalgarno, Kennedy & Bennett, 2010) and we are also currently analysing these results.
References
Bruner, J.S. (1962). On knowing: Essays for the left hand. Cambridge, MA: Harvard University Press. Chandler, P. & Sweller, J. (1991). Cognitive Load Theory and the Format of Instruction. Cognition and Instruction, 8(4), 293-332. Cognition and Technology Group at Vanderbilt University (1992). The Jasper experiment: An exploration of issues in learning and instructional design. Educational Technology Research and Development, 40(1), 66-80. Dalgarno, B., Kennedy, G. & Bennett, S. (2010). Can functional brain imaging be used to explore interactivity and cognition in multimedia learning environments? Journal of Interactive Learning Research, 21(3), 317-342. Dalgarno, B., Bennett, S. & Harper, B. (2010). The Importance of Active Exploration, Optical Flow and Task Alignment for Spatial Learning in Desktop 3D Environments. Human Computer Interaction, 25(1), 2566. Duffy, T. M., & Cunningham, D. J. (1996). Constructivism: Implications for the design and delivery of instruction. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 170-198). New York: Macmillan Library Reference. Harper, B., Hedberg, J. & Brown, C. (1995). Investigating Lake Iluka [computer software]. University of Wollongong NSW: Interactive Multimedia Unit. Harper, B., & Hedberg, J. (1997). Creating motivating interactive learning environments: a constructivist view. In R. Kevill, R. Oliver, & R. Phillips, (Eds.), What works and why, proceedings of the 14th annual conference of the Australasian Society for Computers in Learning in Tertiary Education Perth: Curtin University of Technology. Hawkins, D. (1994). Constructivism: Some history. In P.J. Fensham, R.F. Gunstone & R.T. White (Eds), The content of science: A constructivist approach to its teaching and learning (pp. 9-13). London: Falmer. Hmelo-Silver, C.E., Duncan, R.G., & Chinn, C.A. (2007). Scaffolding and Achievement in Problem-Based and Inquiry Learning: A Response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42(2), 99-107. Kapur, M. (2008). Productive Failure. Cognition and Instruction, 26(3), 379-424. Kennedy, G. E. (2004). Promoting cognition in multimedia interactivity research. Journal of Interactive Learning Research, 15(1), 43-61.
ISLS
94
ICLS2012
Kirschner, P.A., Sweller, J. & Clarke, R.E. (2006). Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and InquiryBased Teaching. Educational Psychologist, 41(2), 75-86. Lee, M.J.W. & Dalgarno, B. (2011). Scaffolding discovery learning in 3D virtual environments: Challenges and considerations for instructional design. In S. Hai-Jew (Ed) Virtual Immersive and 3D Learning Spaces: Emerging Technologies and Trends, IGI-Global. Mayer, R. E. (2001). Multimedia Learning. New York: Cambridge University Press. Mayer, R. E. (2004). Should there be a three-strikes rule against pure discovery learning? American Psychologist, 59(1), 14-19. Piaget, J. (1973). To understand is to invent: The future of education. New York: Grossman. Sweller, J. (2005). Implications of Cognitive Load Theory for Multimedia Learning, In R.E. Mayer (Ed), The Cambridge Handbook of Multimedia Learning, von Glasserfeld, E. (1984). An introduction to radical constructivism. In P.W. Watzlawick (Ed.), The invented reality: How do we know what we believe we know? (pp. 17-40). New York: Norton. Vygotsky, L.S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89-100. White, R. T., & Gunstone, R. F. (1992). Probing Understanding. Great Britain: Falmer Press
Acknowledgements
Support for this project has been provided through a Charles Sturt University Small Grant and a University of Wollongong Centre for Research in Interactive Learning Environments Seed Grant. The authors would also like to acknowledge the contributions of Dr Terry Judd and Dr Michael Lew of the University of Melbourne to the development of the multimedia resources.
ISLS
95
ICLS2012
Design Research in Early Literacy within the Zone of Proximal Implementation

Susan McKenney, Open University of the Netherlands & Twente University, PO Box 2960, 6401DL Heerlen, susan.mckenney@ou.nl Paul Kirschner, Open University of the Netherlands, PO Box 2960, 6401DL Heerlen, paul.kirschner@ou.nl Joke Voogt, Twente University, PO Box 217, 7500 AE, Enschede, j.m.voogt@utwente.nl
Abstract: Despite intentions to the contrary, insights on pedagogically appropriate innovations with representative teachers in everyday school settings are severely limited. In part, this is because (design) research is often conducted at the bleeding edge of what is possible, exploring innovative uses of new technologies and/or emerging theories, while insufficient research and development work focuses on what is practical, today. This leaves a problematic gap between what could be useful research in theory, and what can be useful research in practice. This paper calls for (design) researchers to attend to factors that determine if and how innovations are understood, adopted and used by teachers and schools, and gives one example of how this was tackled in the domain of early literacy. Across ten studies, researchers collected data that helped shape an intervention that can be implemented by representative teachers, for diverse learners, in varied school settings.
Purpose
Educational (design) research strives to develop theoretical understanding that will ultimately improve the quality of teaching and learning. But even when research contributes to programs that are robustly designed and promising in specific settings, most prove difficult to scale up (cf. Penuel, Fishman, Cheng, & Sabelli, 2011). In part, this is because research is conducted at the bleeding edge of what is possible (i.e., exploring new technologies and/or emerging theories). There is no disputing that such work is greatly needed to seek out new ways to potentially enhance the quality of teaching and learning. However, in the excitement of exploring what is possible tomorrow, there is insufficient research and development work focusing on what is practical today. This leaves a problematic gap between what could be useful research in theory, and what can be useful research in practice. With the aim of generating usable knowledge (cf. Lagemann, 2002) and creating innovations that truly serve teaching and learning in practice, this contribution calls for researchers to devote attention to not only fine-grained issues of pupil learning and instruction, but also to broader factors that determine if and how innovations are understood, adopted and used by teachers and schools. Allowing these issues to steer design is necessary to yield innovations that can feasibly be implemented outside of (often highly enabling) research and development trajectories. We argue for more research conducted from the perspective of actual implementation and provide an example of how this was accomplished across nearly a decade of design research related to early literacy.
Toward Relevant and Useful (Design) Research

Revisiting a Familiar Challenge
A hundred years ago, psychologists Hugo Musterburg and John Dewey called for a linking science that would connect theoretical and practical work. In the last 20 years, researchers in the field of curriculum have increasingly called for the development of theoretical understanding that can guide the design of educational innovation (van den Akker, 1999; Walker, 1992). At the same time, researchers in the learning sciences have emphasized the need for research to be situated in authentic contexts to understand learning as it naturally occurs and specifically, what that means for how we shape education (Bransford, Brown, & Cocking, 2000; Brown, 1992). In addition, researchers have stressed the need for innovations to fit into the complex and diverse systems in which they are implemented (Hall & Hord, 2010). Doing so requires developing theories and models to underpin such innovations (Penuel, et al., 2011). Such work must be fed by research and development investigations that are tightly connected to practical realities such as dominant curricula; school cultures; high stakes assessments; interest and expertise of teachers (McKenney, Nieveen, & van den Akker, 2006). They must also be sufficiently problematized to focus efforts not on quasi-problems (e.g., our teachers need ideas for how to use the iPads we gave them), but on problems that are urgent enough to warrant scientific investigation to yield solutions and/or theoretical understanding that can help solve them (McKenney & Reeves, 2012).
The Zone of Proximal Implementation (ZPI)

For research to be relevant and useful, it must reflect an understanding of practical realities and concerns of the domain studied. Studying the status quo of teaching, learning and settings, and designing innovations such that
ISLS
96
ICLS2012
they gradually bridge from the current situation to the desired situation, is essential to developing both the knowledge and the tools required to address real needs in todays classrooms. This perspective has been referred to as the Zone of Proximal Implementation (ZPI; McKenney, 2011). Vygotskys concept of the zone of proximal development the distance between what learners can accomplish independently and what they can accomplish through guidance or collaboration has previously been applied to large scale reform (Rogan, 2007; Rogan & Grayson, 2003); school leadership (McGivney & Moynihan, 1972); and the mediation of educational partnerships (Oakes, Welner, Yonezawa, & Allen, 1998). Similarly, others have referred to the need to pursue certain innovation goals in stepwise fashion, gradually moving from the current situation toward what is desired (Sullivan, 2004). A similar basic concept is applied to the design of educational innovations; but rather than focusing on what can be achieved by learners, it focuses on what can be implemented by teachers and/or schools. The ZPI refers to the distance between what teachers and schools can implement independently and what they can implement through guidance or collaboration (McKenney, 2011). Designing for the ZPI means explicitly tailoring products and processes to fit the capabilities, opportunities and limitations present among teachers and in schools. It also means planning for implementation scaffolding (e.g., honoraria or researcher co-teaching) to fade away in a timely fashion, while simultaneously developing the ownership and expertise among practitioners that will engender the desire and ability to sustain innovation. This is done, in part, through responsive (and sometimes participatory) design, fed by insights concerning learners, practitioners and context.
Implementation-prone Designs
In their book on conducting educational design research, McKenney and Reeves (2012) identify four characteristics of innovations that are prone to successful implementation, namely innovations that are valueadded, clear, compatible and tolerant. During the inception, creation and testing of educational innovations at the ZPI, these characteristics may be considered criteria to be met. Value-added innovations offer something better than what is already in place. Similar to Rogers (Rogers, 2003) notion of the relative advantage, the potential benefits of value-added innovations visibly outweigh the investments required to yield them. Clear innovations enable participants to easily envision their involvement. Innovations may be clear through high levels of explicitness via a priori specifications of procedures and/or interactive mechanisms whereby developers and users co-define the innovation or elements thereof. Compatible innovations are congruent with existing values, cultures, practices and beliefs. They are innovative, but the innovations and/or their underlying assumptions do not violate or reject fundamental concerns and principles of those involved. Compatible innovations are also aligned with non-changeable aspects of the educational system, such as assessment frameworks or policies. Finally, tolerant innovations are those that degrade gracefully (cf. Walker, 2006) as opposed to yielding lethal mutations (cf. Brown & Campione, 1996) during the natural variation in enactment that inevitably comes along with differing contexts, resources, expertise, acceptance levels and so on. Tolerance refers to how precisely core components must be enacted for the innovation to be true to its goals, and how well an innovation withstands local adaptations. The remainder of this contribution describes how a multi-year (design) research series of studies was conducted within the zone of proximal implementation. Attention is given to how data were collected pertaining to the characteristics described above. The innovation was PictoPal: a technology-supported intervention designed to foster the development of emergent reading and writing skills in four and five year old children. Two main strands of inquiry have been attached to the PictoPal endeavor: technology for early literacy; and the influential role of teachers designing the PictoPal materials.
The PictoPal Studies

Technology for Early Literacy
Clay, (1966) emphasized that literacy begins long before school entry, calling this phenomenon emergent literacy. Underpinning this notion, which involves synergistic development of listening, speaking, reading, writing and viewing from birth, are several assertions which have been stressed by other experts as well. First, well-known theorists have long claimed that children play active roles in their own development (Bruner, 1983; Piaget, 1952; Piaget & Inhelder, 1969; Vygotsky, 1962). Clays position that children are active learners about print long before they can read or write is consistent with this view. Second, Macnamara (1972) argued that language learning is driven by and dependent on the capacity to understand and participate in social situations. This is well-aligned with Clays view that social interaction is the basis of emergent literacy. The computers potential to promulgate discourse and thereby knowledge creation has been examined across various age ranges (McLoughlin & Oliver, 1998; Scardmalia, Bereiter, & Lamon, 1994). In terms of early childhood literacy, studies have shown that properly shaped collaborative use of the computer can contribute to pro-social behaviours, including: lively interactions, shared vocabularies, mutual enjoyment and spontaneous, active off-computer play (Brooker & Siraj-Blatchford, 2002; Van Scoter, 2008). In such ways, technology can serve as a catalyst for social interaction and contribute positively to fostering early literacy (Van Scoter, 2008).
ISLS
97
ICLS2012
Yet teachers struggle to integrate technology with their classroom cultures (Labbo et al., 2003; Olson, 2000). This situation is exacerbated by a lack of high-quality emergent literacy materials (de Jong & Bus, 2003; Segers & Verhoeven, 2002). Appropriate software for fostering literacy skills in young children should be created in such a way that the learners previous knowledge is taken into account, involve learners actively and encourage the use of language and the explorative nature and curiosity of young children (Brooker & Siraj-Blatchford; Plowman & Stephen, 2005). In addition, computer activities of young learners should be integrated with related classroom activities (Van Scoter, 2008) and embedded in appropriate pedagogical models for technology applications for young children (Plowman & Stephen, 2005). The PictoPal studies examine how the computer can contribute to an understanding of the nature and function of written language. Doing so requires that attention be given to the relationship between spoken and written language and the purposes for which reading and writing are used. As in oral language learning, childrens concept formation regarding written language is driven by its use, in an environment rich with meaningful messages and functional print (Warash, Strong, & Donoho, 1999; Van Scoter, 2008). The main question shaping this strand of inquiry is How can the technology-supported learning environment, PictoPal, contribute to helping kindergarteners understand the nature and function of written language?
Teachers as Designers
Teacher involvement in the development of classroom curricula often fosters a sense of ownership, which increases the chances of actual curriculum use (Fullan, 2003). When teachers are supported during the design of innovative curricula, they can learn more about the innovation (Crow & Pounder, 2000), which also increases the chances of the implementation being successful. This is partly because teachers are then better informed, and able to visualize how curriculum enactment could look. Being able to see a curriculum in action is an important factor considered by teachers as they weigh off the amount of effort they invest and the potential benefits of the innovative curriculum (cf. Doyle & Ponder, 1978). Designing requires teacher time and effort, but also has the potential to improve implementation of an innovative curriculum. There are various ways to involve teachers in design and support that involvement. This strand of inquiry is concerned with answering the research question, What forms and levels of involvement are feasible, and still yield the benefits of ownership, and understanding a new curriculum?
Research approach The Designed Intervention

PictoPal is an ICT-rich learning environment with two main components: (a) on-computer activities through which pre-readers use words, sound and images to construct written texts; and (b) off-computer activities that prompt children to use their printed documents for authentic purposes. For example, children create grocery lists using the computer and then shop for the items on the printed list in the store corner of the kindergarten classroom. Alternatively, they prepare a weather forecast with the aid of the computer, and then deliver the forecast to their class from the television corner (from inside a television fashioned by the children from a large cardboard box). Figure 1 shows children, composing a recipe for vegetable soup (left) and then following the cooking instructions on their printed recipe (right). For more information about the PictoPal learning environment, please refer to McKenney and Voogt (2009).
Figure 1. Children Creating the Recipe on the Computer (left) and then Following it in the Classroom (right).
The Methods Used

The PictoPal studies have been underway for nearly a decade. Evolving through five different prototypes, ten different studies have been completed relating to the two strands of inquiry described. Comprehensive portrayal
ISLS 98
ICLS2012
of the studies is therefore not possible here. Table 1 provides an overview of the studies including the prototype concerned, focus of the data collection in each study, methods used and the report reference for additional details. Table 1: Research overview of focus and methods in the 10 studies to date Study 1 2 3 4 5 Teacher/school habits, beliefs, values, resources Childrens habits, attitudes Pupil learning gains Pupil engagement Integration Teacher experiences Teacher and child questionnaires Site visits, field notes Document analysis Pre/post-test On-pc observation Off-pc observation Teacher interviews (S. McKenney & Voogt, 2005) (S. McKenney & Voogt, 2010) (S. McKenney & Voogt, 2009) (S. McKenney & Voogt, in press) (Cviko, McKenney, & Voogt, 2012) Cviko, McKenney Voogt, 2010) Cviko, McKenney, Voogt, 2011) A B C D E F G H I J
Prototype
Focus
Methods
Reporting
Selected Results Practical Innovation: How the Research Informed Intervention Design
Throughout the series of studies, data were collected that informed the design of both PictoPal and its accompanying supportive mechanisms (teacher guides, professional development workshops). In hindsight, it is clear that the data contributed to developing an intervention that was, as characterized earlier, value-added, clear, compatible and tolerant. Table 2 provides an overview of how the research helped to develop PictoPal within the zone of proximal implementation. Table 2. Research methods used to feed design within the zone of proximal implementation Pre-design (needs/context analysis) Value-added (better than status quo) Clear (participants can envision their involvement) Site visits to see learning practices and ask about problems, in the baseline situation Teacher interviews to explore mindsets, habits and conventions within the classroom/school in the baseline situation Teacher interviews and child During development (prototyping and formative evaluation) Pre/post-tests of pupil learning during use With stable design (used as means to study teacher design practices) Pre/post-tests of pupil learning with implementation scaffolds removed Teacher interviews to explore mindsets, habits and conventions within the classroom/school that are sustained or changed after the innovation Teacher interviews to
Teacher interviews to explore mindsets, habits and conventions within the classroom/school during use Field notes concerning
Compatible
ISLS
99
ICLS2012
(compatible with values, beliefs, surrounding educational context/system) Tolerant (withstands the natural variation of actual use)
questionnaire to explore values, cultures, beliefs, priorities, and contextual /system factors in the baseline situation Site visits and field notes of actual behaviors of teachers and learners and reasons for them in the baseline situation
values, cultures, beliefs, priorities, and contextual /system factors that help or hinder implementation
Observation of teachers and learners (and conjectured reasons for their conduct) during use
explore values, cultures, beliefs, priorities, and contextual /system factors that are sustained or changed after the innovation Observation of teachers and learners, and document analysis of both teacher products and learner products resulting from use with implementation scaffolds removed
Relevant Theoretical Understanding: Insights on Developing Technology for Early Literacy

This decade of design studies has yielded multiple reports and different kinds of insights. The papers cited in Table 1 offer detailed descriptions of findings and the data supporting them. In this section, examples of theoretical understanding emerging from the PictoPal studies are given: - Related to shaping technology for early literacy, several important understandings include: o Combining on- and off-computer activities can significantly influence childrens understanding of functional literacy. o Children engage in dialogue about their work on and off the computer with peers or adults, though the quality of the conversation differs. o Once the software has been mastered (about two turns), new elements can be introduced into the learning environment (e.g., contrary to initial hypotheses, there is no need to teach the image-based vocabulary). o Children can use PictoPal semi-independently; the limited guidance they need is best given by an adult, but can be handled by an older child (e.g. Grade 6). - Related to understanding the influence of teachers as designers, several important understandings include: o Curricular ownership is positively related to the level of technology integration. o Kindergarten teachers tend to limit new initiatives in the classroom - even if they support them until a safe, trusting, routine and predictable classroom climate has been firmly established. o Well-structured, even modest design involvement fosters curricular ownership, facilitating implementation. o Providing tailored support can build teacher understanding and endorsement of core ideas, while freedom and creativity should be encouraged to develop different manifestations of those core ideas.
Reflections
The PictoPal studies were carried out over nearly a decade. After an initial needs and context analysis, the evaluation of early prototypes could be characterized as proof-of-concept work, focusing especially on the soundness and feasibility of the envisioned intervention. Thereafter, additional prototypes were developed and iteratively tested. During this work, implementation concerns received particular attention. Once the design stabilized, a more extensive look was taken at issues that could potentially increase effectiveness. Earlier work thus fed the research on shaping technology for early literacy. After one study (Study D in Table 1) showed substantial learning gains using materials that had been created by teachers themselves, research turned more explicitly toward exploring teachers as designers, how ownership is created through the design process, and its resulting influences in technology integration. Pupil learning was still examined, but mostly in light of these processes. The PictoPal intervention was designed to be usable in a wide range of schools, by different kinds of kindergarten teachers, with diverse groups of learners. It has also been tested under varying conditions. The fact that teachers in several schools have continued using PictoPal, even outside of the research activities and with no external support suggests that it did fall within their ZPI. This is likely due, in part, to being value-added, clear, compatible and tolerant. The research findings indicate that PictoPal grew to become value-added because, in addition to regularly resulting in significant pupil learning gains, teachers appreciated how it addressed a known and disconcerting gap between the existing language curriculum used in schools and the national interim targets for early literacy. Use of PictoPal became clear to teachers through their involvement in designing content, and through examples, passed along from previous teachers (e.g. through video), of how PictoPal came to life in the classroom. While PictoPal was extremely innovative, being joined by very few other technologies for early literacy that support the development of discursive prowess (Lankshear and Knoebel, 2003), it was also
ISLS
100
ICLS2012
designed to be compatible with existing values, cultures, beliefs and priorities that are shared by most kindergarten teachers, e.g. that children learn through play (dramatic play features frequently in the off-computer activities); or that children shape understanding about the world through first-hand experiences (using printed texts for authentic purposes). The studies also explored different scenarios that would help teachers orchestrate the implementation of PictoPal, especially providing guidance during on-computer activities (e.g. adding an agent to the software, peer tutors, adult guides) in ways that are compatible with the options available in representative settings. Finally, PictoPal grew tolerant over time. As the intervention matured, it became clear which features had to be adhered to more strictly to achieve pupil learning (e.g. duration of intervention; structure and layout of words in the software grid; and a gradual increase in difficulty level); and which ones could be tailored by teachers to their own preferences, themes or inspiration. The need for researchers to seek understanding about where teachers and schools are, how they perceive the problem(s) to be addressed, and to and frame innovations within a reachable distance, has been described in literature (e.g. Bielaczyc, 2006; Blumenfeld, 2000; McKenney & Voogt, in press). This paper further emphasizes the importance of explicitly collecting data that can attune both interventions and theoretical insights accordingly. It offers some examples of how that was accomplished in one case relating to technology for early literacy. Toward increasing the relevance and usefulness of educational (design) research, less examples are needed of what might be potentially possible, and more examples are needed of how to understand and design for what is realistically feasible: in the Zone of Proximal Implementation.
References
Bielaczyc, K. (2006). Designing social infrastructure: Critical issues in creating learning environments with technology. Journal of the Learning Sciences, 15(3), 301-329. Blumenfeld, P., Fishman, B. J., Krajcik, J., Marx, R. W., & Soloway, E. (2000). Creating Usable Innovations in Systemic Reform: Scaling Up Technology-Embedded Project-Based Science in Urban Schools. Educational Psychologist, 35(3), 149-164. Bransford, J., Brown, A., & Cocking, R. (2000). How people learn: Brain, mind, experience and school. Washington, DC.: National Academy Press. Brooker, L., & Siraj-Blatchford, J. (2002). Click on Miaow!: how children of three and four years experience the nursery computer. Contemporary Issues in Early Childhood, 3(2), 251-273. Brown, A. L. (1992). Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings. Journal of the Learning Sciences, 2(2), 141-178. Brown, A. L., & Campione, J. (1996). Psychological theory and the design of innovative learning environments: On procedures, principles, and systems. In L. Schauble & R. Glaser (Eds.), Innovations in learning (pp. 289-325). Mahwah, NJ: Earlbaum. Clay, M. (1966). Emergent Reading Behaviour. Unpublished doctoral dissertation, University of Auckland, New Zealand. Crow, G.M., & Pounder, D. G. (2000). Interdisciplinary Teacher Teams: Context, Design, and Process. Educational Administration Quarterly, 36, 216-254. Cviko, A., McKenney, S. & Voogt, J. (2010). Docenten als ontwerpers en uitvoerders van een ICT-rijk curriculum voor beginnende geletterdheid. Round table presentation at the ORD annual meeting, June 23-25: Enschede. Cviko, A., McKenney, S. & Voogt, J. (2011). Teachers as (re-)designers of an ICT-rich learning environment for early literacy. Paper presentation at the ECER annual meeting, September13-16: Berlin. Cviko, A., McKenney, S., & Voogt, J. (2012). Teachers enacting a technology-rich curriculum for emergent literacy Educational Technology Research and Development, 60(1), 31-54. doi: I 10.1007/s11423-0119208-3. de Jong, M. T., & Bus, A. G. (2003). How Well Suited are Electronic Books to Supporting Literacy? Journal of Early Childhood Literacy, 3(2), 147-164. doi: 10.1177/14687984030032002 Doyle, W., & Ponder, G. (1978). The practicality ethic in teacher decision-making. Interchange, 8(3), 1-12. Fullan, M. (2003). Change forces with a vengeance. London: RoutledgeFalmer. Hall, G., & Hord, S. (2010). Implementing change: Patterns, principles and potholes. Upper Saddle River, NJ: Prentice Hall. Labbo, L., Leu, D., Kinzer, C., Teale, W., Cammack, D., Kara-Soteriou, J., & Sanny, R. (2003). Teacher wisdom stories: Cautions and recommentaions for using computer-related technologies for literacy instruction. The Reading Teacher, 57(3), 300-304. Lagemann, E. (2002). An elusive science: The troubling history of education research. Chicago: University of Chicago Press. Lankshear, C., & Knoebel, M. (2003). New technologies in early childhood literacy research: A review of research. Journal of Early Childhood Literacy, 3(1), 59-82. Macnamara, J. (1972). Cognitive basis of lanaugage learning in infants. Psychological Review, 79, 1-12.
ISLS
101
ICLS2012
McGivney, J., & Moynihan, W. (1972). School and community. Teachers College Record, 74, 209-224. McKenney, S. (2011). Designing and researching technology enhanced learning fort he zone of proximal implementation. Invited paper presentation at the Art and Science of Learning Design workshop. September 13-14: London Knowledge Lab, London. McKenney, S., & Reeves, T. (2012). Conducting Educational Design Research. London: Routledge. McKenney, S., & Voogt, J. (2005). Using ICT to foster (pre) reading and writing skills in young children. Paper presented at the American Educational Research Association, Montreal. McKenney, S., & Voogt, J. (2009). Designing technology for emergent literacy: the PictoPal initiative. Computers & Education, 52, 719-729. McKenney, S., & Voogt, J. (2010). Technology and young children: How 4-7 year olds perceive their own use of computers. Computers in Human Behavior, 26, 656-664. McKenney, S., & Voogt, J. (in press). Teacher design of technology for emergent literacy: An explorative feasibility study. Australasian Journal of Early Childhood. McKenney, S., Nieveen, N., & van den Akker, J. (2006). Design research from the curriculum perspective. In J. van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. 67-90). London: Routledge. McLoughlin, C., & Oliver, R. (1998). Maximising the language and learning link in computer learning environments. British Journal of Educational Technology, 29(2), 125-136. doi: doi:10.1111/14678535.00054 Oakes, J., Welner, K., Yonezawa, S., & Allen, R. (1998). Norms and policies of enquiry-minded change: Reaching the 'zone of mediation'. In A. Hargreaves, A. Liebermann, M. Fullan & D. Hopkins (Eds.), International handbook of educational change (pp. 952-975). Dordrecht: Kluwer. Olson, J. (2000). Trojan horse or teachers pet? Computer and the culture of the school. Journal of Curriculum Studies, 32, 1-8. Penuel, W., Fishman, B., Cheng, B., & Sabelli, N. (2011). Organizing research and development ad the intersection of learning, implementation and design. Educational Researcher, 40(7), 331-337. Piaget, J. (1952). The Origins of Intelligence in Children. New York: W. W. Norton. Piaget, J., & Inhelder, B. (1969). The psychology of the child. New York: Basic Books. Plowman, L., & Stephen, C. (2005). Children, play and computers in pre-school education. British Journal of Educational Technology, 36(2), 145-157. Reeves, T. (2000). Enhancing the worth of instructional technology research through design experiments and other development research strategies. Paper presented at the Annual AERA Meeting, New Orleans. Rogan, J. (2007). How much curriculum change is appropriate? Defining a zone of feasible innovation. Science Education, 91, 439-460. Rogan, J., & Grayson, D. (2003). Towards a theory of curriculum implementation with particular referenc to science education in developing countries. International Journal of Science Education, 25, 1171-1204. Rogers, E. (2003). Diffusion of innovations (5th ed.). New York: Free Press. Scardmalia, M., Bereiter, C., & Lamon, M. (1994). The CSILE Project: Trying to bring the classroom into World 3. ). : . In K. McGilly (Ed.), Classroom lessons: Integrating cognitive theory and classroom practice (pp. 201-228). Cambridge, MA: MIT Press/Bradford Books. Segers, E., & Verhoeven, L. (2002). Multimedia support of early literacy learning. Computers in Education, 39(3), 207-221. Sullivan, M. (2004). The reconceptualisation of learner-centred approaches: a Namibian case study. International Journal of Educational Development, 24, 585-602. Tebbutt, M. (2000). ICT in science: problems, possibilities and principles. School Science Revie, 81(297), 57-64. van den Akker, J. (1999). Principles and methods of development research. In J. van den Akker, R. Branch, K. Gustafson, N. Nieveen & T. Plomp (Eds.), Design approaches and tools in education and training (pp. 1-14). Dordrecht: Kluwer Academic Publishers. Van Scoter, J. (2008). The Potential of IT to Foster Literacy Development in Kindergarten. In J. V. Knezek (Ed.), International Handbook of Information Technology in Education (pp. 149-161). London: Springer. Vygotsky, L. (1962). Thought and language. Cambridge, MA: MIT Press. Walker, D. (1992). Methodological Issues in Curriculum Research. In P. W. Jackson (Ed.), Handbook of Research on Curriculum (pp. 98-118). New York: Macmillan. Walker, D. (2006). Toward productive design studies. In J. Van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. ZZ-XX). London: Routledge. Warash, B., Strong, M., & Donoho, R. (1999). Approaches to environmental print with young children. . In O. G. Nelson & W. M. Linek (Eds.), Practical classroom applications of language experience: Looking back, looking forward (pp. 53-58). Boston, MA: Allyn & Bacon.
ISLS
102
ICLS2012
Kitchen Chemistry: Supporting Learners Decisions in Science

Jason C. Yip, Tamara Clegg, Elizabeth Bonsignore, Becky Lewittes, Mona Leigh Guha, and Allison Druin, Univeristy of Maryland College Park, Human-Computer Interaction Lab, 2117 Hornbake Bldg, South Wing, College Park, MD 20742 Helene Gelderblom, University of South Africa, School of Computing, PO Box 392, Pretoria, South Africa, 0003 jasonyip@umd.edu, tclegg@umd.edu, ebonsign@umd.edu, geldejh@unisa.ac.za, charley@cs.umd.edu, mona@cs.umd.edu, allisond@umiacs.umd.edu Abstract: Students often find science to be disconnected from their everyday lives. One reason for this disengagement is that learners are often not given the chance to choose how to pursue their personal goals using science reasoning. Therefore, we are creating science programs that emphasize life-relevant learning - the ability to engage science learners in the context of achieving their own goals. We developed Kitchen Chemistry to engage and support children in the design of their own personal investigations. In this paper, we use a case study analysis to examine three groups of learners in Kitchen Chemistry. We analyze the decisions that learners make, how learners make these decisions, and the supports needed to make informed choices. We examine how the use of semi-structured activities, whole group discussions, adult facilitation, and mobile technologies interact and support learners in their decision-making practices.
Introduction
Researchers have documented that some youth find aspects of traditional school science to be disconnected from learners daily lives and interests (e.g., Atwater, 1996). Indeed, within the didactic instructional model, particularly within US schools, abstract knowledge is often presented without making direct connections to learners lives outside the classroom. When science learning does not make connections to learners interests and experiences, students may have pervasive negative views of science (e.g., Basu & Barton, 2007). We believe that one reason for this disengagement is that learners are often not given the chance to make choices and develop agency in how to pursue their personal goals using science reasoning. A challenge now facing educators is the development of learning activities that connect learners personal goals to science knowledge and processes. This is important because all citizens must be scientifically literate and be able to reason well about complex evidence to make educated decisions about important issues, such as health and environmental policies (Chinn & Malhotra, 2002). We aim to address this issue of science disengagement through the development a life-relevant learning (LRL) environment called Kitchen Chemistry (KC). LRL environments are learning environments that engage learners in science through the pursuit of personally meaningful and relevant goals, in either formal (e.g., Bouillion & Gomez, 2001) or informal learning contexts (e.g., Fusco, 2001). One everyday context we believe can support science learning is the kitchen. Therefore, we developed KC as a guided-inquiry LRL program that allows participants to learn science and engage in scientific practices within the context of cooking (e.g., Clegg, Gardner, & Kolodner, 2010). Through semi-structured activities and discussions, learners use their science knowledge to later develop their own personal investigations into cooking. We assert that learners are capable of playing an active role in designing their own avenues of scientific inquiry. However, having learners design scientific investigations based on their own personal goals is no easy task. Learners often need support as inquiry tasks become more complex (e.g., Reiser, 2004). We argue that giving learners agency in how they make decisions in inquiry practices is an important means of getting learners engaged in science. We believe there is much importance in supporting learners choices in science, but that we need to be mindful of what supports are needed to allow for appropriate science choices. Therefore, in this paper, we ask what supports are needed for learners to make informed personal choices about the designs of their investigations within a guided-inquiry based science-learning environment. We seek to find out 1) what decisions learners make in the development of their own investigations; 2) how learners come to make such decisions; and 3) what aspects of the learning environment support these choices.
Theoretical Framework
To help engage learners in science, we must bridge the gap between science knowledge and learners lifeworlds (Aikenhead, 1996). We have to provide a means for learners to engage in an authentic scientific experience in the context of their own personal goals (Clegg et al., 2010) and give learners a chance to make decisions in what they want to pursue. We recognize that while we need learners to make personal choices to accomplish their goals, they need help developing the skill set to make decisions to achieve those goals. Indeed, learners come
ISLS
103
ICLS2012
with diverse funds of knowledge that are rooted in their everyday-life experience and participation in their community (e.g., Barton & Tan, 2009). However, much of the science experience in schools does not acknowledge learners own personal goals and resources. Often, reforms in school science push learners into the unfamiliar world of scientists, as opposed to engaging learners in their own culture (Aikenhead, 1996). In particular, the lack of relevance of science knowledge to learners lives leads to these higher levels of disinterest in later years of schooling (e.g., Aikenhead, 1996). Much of school science focuses on simple experiments and tasks that bear little resemblance to professional and authentic science. Often these simple science tasks do not connect to learners own funds of knowledge and can promote an unscientific epistemology (Chinn & Malhotra, 2002). More authentic science learning gives learners a chance to solve problems through designing investigations and complex experiments, collaborating with others, considering multiple variables for testing, and establishing rigor through checks and balances. Because authentic science learning requires learners to socially construct knowledge, we wanted to promote a participatory culture in KC. Jenkins, Clinton, Purushotma, Robinson, and Weigel (2006) define a participatory culture as a culture with relatively low barriers to artistic expression and civic engagement, strong support for creating and sharing ones creations, and some type of informal mentorship whereby what is known by the most experienced is passed along to novices (p. 3). In a participatory culture, learners believe that their contributions matter and they develop social connections between each other. To help support this participatory culture, we developed cultural practices based on our experiences with two frameworks for design: Cooperative Inquiry (e.g., Druin, 2005) and Learning By Design (Kolodner et al., 2003). In Cooperative Inquiry, children are partners with adults in the design process. From our previous work with Cooperative Inquiry, we established customs that attempted to minimize existing power structures. Our team insisted on three basic practices: sitting together in a circle with children in discussion, calling each other by our first names, and speaking in ordered turn without raising our hands. From Learning by Design, we adhered to two principles. First, we adhered to the culture of iteration: members of the community develop the responsibility for helping each other learn through critique of one another and iteration on their designs. Second, we supported scientific reasoning: members of the community must adhere to principles of science and causality in explanations and refer to evidence to back up claims.
Design of Kitchen Chemistry

KC is an after-school or summer camp program where learners engage in scientific practices within the context of cooking. Holding the program outside of school enables learners to choose the directions of their scientific inquiry without being bound to a particular curriculum. To provide an environment in which learners can participate in scientific practices to design investigations that are personally meaningful to them, we developed two activity sequences. First, learners engage in semi-structured activities that help familiarize them with cooking and science practices. In the cooking practices, learners are given the tasks such as observing what eggs do in brownies and what leaveners (e.g., baking powder, baking soda) do in cookies. In these cooking experiments, learners vary the amounts of eggs and types of leaveners in the recipes and examine the results. The semi-structured activities also include non-cooking experiments in which the ingredient variations in their cooking experiments are highlighted specifically to help learners think about the underlying scientific phenomena. For example, learners mix different amounts of eggs into fixed amounts of oil and water to observe how eggs act as emulsifiers in the mixture. They also compare the heights and amount of foam that are generated from shaking the egg, water, and oil mixture. For the experiment on leaveners, participants mix different amounts of baking soda, baking powder, and cream of tartar into water and lemon juice and observe the amount of bubbles generated from the mixture. These experiments help learners to see how eggs act as emulsifiers to bind together water and oils in the brownies and how different leaveners incorporate gas bubbles into dough. Learners participate in these semi-structured activities to prepare for the second activity sequence flexible exploratory activities that we called Choice Days. Learners are given the opportunity to use what they have learned to prepare an investigation into a recipe of their choice. Here, learners make decisions about their investigations, such as which recipes to explore, what modifications to make to ingredient amounts, and what observations they will make. Embedded in both the semi-structured activities and Choice Days are whole group conversations in which learners further discuss and reason about the observed phenomena. To develop a community that invites participation, both adults and learners sit together in a circle. We discuss what we observed, our thoughts on the results, and what we think about the outcomes. We also discuss authentic scientific practices (Chinn & Malhotra, 2002), such as making qualitative and quantitative observations, tying evidence to claims and reasoning, thinking about the underlying mechanism that causes the observed phenomenon, and considering how independent variables may change multiple dependent variables. All participants work together to think about the underlying phenomenon and build on each others scientific arguments. Learners also use mobile technologies on the iPad to conduct their investigations and reflect on their observations. In particular, learners use StoryKit (e.g., Quinn, Bederson, Bonsignore, & Druin, 2009) and Zydeco (e.g., Cahill, Kuhn,
ISLS
104
ICLS2012
Schmoll, Pompe, & Quintana, 2010). StoryKit is an iPhone application for creating and sharing audio-visual stories through text, photos, drawings, and audio recordings. Zydeco is also an iPhone application that can be used to photograph, tag, and annotate information within different contexts. From this collective information, users can develop claims, evidence and reasoning. We use both StoryKit and Zydeco together to support learners scientific practice in the context of choice-based activities and to help them reflect during and after the investigations on the scientific aspects of the activities (Clegg, Gardner, & Kolodner, 2011).
Methods
For this study, we used the standards of a comparative case study (Yin, 2009) of a one-week implementation of KC. We employed a multiple case study design, that is, within the single implementation there are multiple units of analysis. In this exploratory case study, we are comparing three different groups of learners within the same context. We examined the decisions that learners made, how learners made these decisions, and what factors supported their decision-making practices.
Context and Data Collection

KC was implemented as an all day, one-week summer program at a local private school. Six to nine learners between the ages of 9 to 13 participated in the program each day. The participants were from public and private schools in the area. Each day, we collected video recordings of all semi-structured activities and whole group discussions. Facilitators also recorded post-observation field notes that described what we thought were the most significant aspects of the day. We collected learners entries in the Zydeco and Storykit software systems. We conducted interviews with four learners who agreed to do interviews to understand their perspectives on their experiences with the program, activities, and technology.
Criteria for Case Selection and Data Analysis

Since we are interested in the decision-making processes of the learners, we selected three of the four Choice Day groups as cases. These are three cases for which we had sufficient data recordings (we were unable to fully record the fourth case). All learner names are pseudonyms. We began the data analysis through an initial examination of the interviews, videos, software artifacts, and facilitators field notes to find evidence that links to our questions on decision-making. During this time, we wrote content logs of the videos and transcribed salient quotes. As particular segments emerged as significant, we expanded the content logs into transcriptions. Since the unit of analysis was the small group, we coded instances when the groups or interviewees talked about group-based choices, how the groups made a decision, and the outcomes of their decisions. We triangulated the data to make sure that all pieces of evidence supported each other (Merriam, 2009). From our answers, we developed a formal assembly of the evidence into a case study database in which field notes, transcripts, and software artifacts were put online for each author to independently examine (Yin, 2009). Using this database, we maintained a chain of evidence for each embedded case that builds connections between the variety of evidence, the questions we had on decision-making and our initial conclusions. To establish validity, we presented each case to the corresponding facilitator to ensure that the cases were representative of the learners and their decision-making practices. Once each case was thoroughly developed, we conducted a cross case analysis between the three embedded cases to see if similarities and patterns were present in the data.
Key Findings
We begin each case with a description of the activity that the learners and facilitators designed for Choice Day. We then analyze each case to understand what decisions learners made and how they came to make them.
Case 1: Vegan Brownies

For this Choice Day activity, Ari and Clyde, two 10-year-old boys, wanted to make vegan brownies. Ari was motivated to try this recipe because he had tasted a vegan brownie from a prior activity and found the taste to be displeasing. Ari decided to take up the challenge to make a vegan brownie that would taste better and be gooey and fudgy. Clyde decided to join Ari in this endeavor. Becky (facilitator) was given the task of supporting this group. What sets Becky apart from the other facilitators is that this session was her first time at KC. Therefore, Becky was learning about the prior activities and discussions as she was helping to facilitate. The lead facilitators provided the group with a vegan brownie recipe to try. However, the boys did not want to blindly follow a recipe whose outcome might not meet their gooey and fudgy goals. Instead, Ari and Clyde wanted to figure out if they should use the vegan brownie recipe or modify a non-vegan recipe. To help them decide, Ari and Clyde immediately chose to run an egg experiment using an egg substitute powder. To test the emulsifying properties, the learners made a mixture of oil and water and added the egg substitute powder. They added different amounts of egg substitute powder as their new independent variable. Their dependent variable was the amount of foam generated in the new mixture. They added the egg substitute in one-tablespoon
ISLS
105
ICLS2012
increments, shook the bottle, and observed the height of the foam. Using two tablespoons of the egg substitute powder, Ari and Clyde found that the foam did not rise as much as when they used the real eggs. Based on these observations, the boys concluded that the egg substitute could be used as an emulsifier. Meanwhile, Becky observed them, asking questions about why they were engaged in this process. Ari and Clyde also referred to the prior StoryKit photographs to make comparisons between the new experimental foam and the prior egg experiments. The learners explained to Becky, who was not present for the original egg experiment, that the lower amount of bubbles from two tablespoons of egg substitute would create a more gooey brownie that was less leavened. Because they wanted that outcome, Ari and Clyde decided to go ahead and make modifications to the recipe to include two tablespoons of egg substitute.
Case 1: Analysis
We observed that the learners immediately chose to use the egg experiment with the new variable. Becky stated, "They knew to set up the experiment. And then they added the oil, the water, and the fake eggs." Here, the learners took aspects of the semi-structured activities and were able to tailor them to help them make their decisions. Second, although Becky was a first time visitor, she was able facilitate their decisions. Being new to the activity, Becky did not know all the answers, which she reported may have helped the whole group make decisions together. Although Becky spent a lot of time trying to keep Clyde on task, she also made sure to spend time in discussion with both learners to help them make their investigation decisions. For example, the vegan brownie group spent the most time of any group discussing their decisions. While Becky was not involved in the prior activities, she knew what questions they were asking and what decision points they were heading toward. Becky recalls prompting them on making the decision between the two [original and new] recipes and thinking that they needed me to prompt a bit in terms of what those two things [leaveners and emulsifiers] were and like, making the decision. While Becky was able to prompt Ari and Clyde on the decision points, the two learners also used the mobile technologies to help guide their choice. Ari and Clyde chose to use StoryKit as a means to reflect on the process of making their decisions. During the setup, Ari and Clyde ran their egg-oil-water experiment using the egg substitute and observed the foam generated from shaking the mixture. Once a slight bit of foam was created, Becky noticed that Ari and Clyde were flipping through prior photos and recordings from StoryKit of the foam from the prior egg-oil-water experiment. They referred back to these photos of the foam results to compare with the current foam results they had with the egg substitute to decide if the egg substitute would be a good emulsifier. From these comparisons between the prior data and their new data, the learners made the decision to use their modified investigation.
Case 2: Togo Cakes

For the entire week, Lisa, a 10-year-old girl, consistently reminded us that she had a brilliant idea to make Togo Cakes (a name she coined herself, derived from to go). Instead of normal pancakes with the syrup drenched all over, Lisa wanted to develop a pancake that could enclose the syrup in the middle. This way, she could toast a frozen Togo Cake that would have a pocket of syrup in the middle so that she could eat it on the go. Lisa easily decided to choose Togo Cakes as the focus of her investigation. The more difficult decision was how to bring this project to fruition. Working with Tammy (facilitator), Lisa first had to determine what she wanted from a pancake recipe. She writes in StoryKit, We want to find out: how much baking powder we need for it not to be as fluffy and still taste right. We don't want it to be fluffy because we don't want the syrup to absorb into the pancakes. Lisa wanted to design a pancake with two features. First, the pancake could not be too fluffy or the syrup would absorb into the pancake. Second, the pancake had to have a crispy top so the syrup would not leak out. For the crispy top, Lisa chose to use three eggs because in their brownie experiment, she observed that the three-egg variation had a harder crust. Next, Lisa and Tammy ran a series of experiments to determine the amount of baking powder to use in their pancakes. They heated varying amounts of baking powder in water, observing how much foam the cooked solution generated before deciding that 1 teaspoon produced the desired amount of foam and fluff. Lisa also came across a new problem in her investigation. As part of her engineering task, she needed to figure out a way to put the syrup into the middle of the pancake. Tammy and Lisa tried to freeze the syrup on top of a small plate. They found, however, that maple syrup does not freeze very well. Lisa decided to try another viscous and sweet liquid that would be similar to maple syrup. She designed another set of experiments to test out whether honey, corn syrup or maple syrup would freeze enough so that she could lift it off the plate and place into the middle of the pancake. Placing the three liquids side by side in the freezer, Lisa determined that honey had the consistency and solidity that worked for her Togo Cakes. Lisa and Tammy concluded their Storykit story, Success at last, we got honey in the pancakes without absorbing. In the end, Lisa found the Togo Cakes were successful; the final product had the right crispiness and fluffiness and was able to hold the honey in the middle.
ISLS
106
ICLS2012
Case 2: Analysis
Based on the tasks, Lisa had to make several key decisions in her investigation. She did not make haphazard guesses, but instead, had a very methodical approach to her design. First, Lisa depended on the semi-structured activities to develop her series of experiments. In particular, she used the leavener experiment in which learners had to make observations about different combinations of leaveners (e.g., baking soda, baking powder, cream of tartar) and whether they reacted with lemon juice, vinegar or water to produce bubbles. Learners also made observations of how warmer temperatures affected these reactions. Without hesitation, Lisa gravitated towards conducting an experiment on baking powder alone, to determine how much she would use in her recipe. The choice of using baking powder, as opposed to baking soda, may have also come from her experience from baking cookies using baking soda. Her use of baking soda in her cookies resulted in a very flat and unleavened product. Lisa stated in one of the discussions, the baking soda needs an acid to react because it was a base and that the baking soda didnt react until we added vinegar to the cup experiment. Her choice in using baking powder for this design is reflected in her thoughts that baking powder added, a little more acid to the bicarbonate to produce the bubbles. Many of these statements of Lisas understanding of the ingredients emerged during our discussion time. Lisa also used StoryKit as a way to organize her investigation. For example, in Lisas story, there are multiple pictures and descriptions of the three sets of experiments she performed. Each photo showed the foam coming out. Lisa wrote about her decision, This one [referring to 2 teaspoons] boiled over (not quite as high as the 3-1/2 tsp) too! So were going to use 1-3/4 tsp. of baking powder. We saw that Lisas decision to use 1 teaspoon of baking powder was done carefully and methodically. Although Lisa was very organized, we observed that Tammy, the facilitator, still helped her sort through her decisions. Tammy recalls that she prompted Lisa to think about what goals she had, It was sort of like thinking about what goals she had for it. She knew she wanted it (pancake) to be thicker, thicker than normal, and that was why she thought about it more, the leaveners." Lisa told Tammy the issues and the goals that she had. Tammy felt that she wanted to help her find solutions to meet that goal. This meant that as a facilitator, Tammy helped to plan out ways to adapt the semi-structured activities to help meet Lisas goals.
Case 3: Red Velvet Cake

Denise, a 10-year-old girl, choose to make red velvet cakes, because her aunt made very good red velvet cakes. Lily and Meg (10- and 13-year-old respectively) decided to join Denise in her endeavor. The girls and their facilitators, Helene and Beth, discussed how they were going to make this into a science investigation. Denise initially did not want to think about the science behind baking this cake; she just wanted to make a cake as delicious as her aunts. Compared with Cases 1 and 2, Denise talked more about liking cake and icing, as opposed to wanting to conduct an experiment to determine ways to make her cake better. Based on this motivation, Beth and Helene prompted the group with questions about the choices they could make in the cake. They asked the learners questions about how the recipe could be modified to achieve their desired results in terms of texture and taste. The adults recalled structuring the development of the red velvet investigation using a goals chart that prompted learners to think about what choices they want to make and what outcomes they predicted would occur. Learners recorded information onto the goals chart, such as taste, texture, mouthfeel, handfeel, smell, and look, and what leavener proportion they thought would achieve these outcomes. Using StoryKit, Meg took a photo of the goal sheet, and the group worked together using the iPad to complete the goals chart. Based on these prompts and the structure of the goals chart, the learners chose to compare a recipe they found online and a modified recipe by adding eggs and substituting baking powder for vinegar and baking soda. The group realized that they only had enough resources to make two smaller comparison cakes, so they decided to cut the recipe in half. To make these modifications, the learners relied on their knowledge from the prior experiments. For example, Denise decided that since the ingredients in the brownies recipe were similar to the red velvet, she chose two eggs (doubling the original recipe) for the modification because she predicted this would yield a more moist cake (one of her recipe goals). The learners and facilitators made an additional modification to their experimental recipe by replacing the original vinegar and baking soda ingredients with baking powder. Since baking powder is both dry acid and base combined, the group wanted to see if this ingredient would produce the same leavening effects as vinegar and baking soda. As the group made progress through their Choice Day efforts, each learner raised more scientific questions. For example, Denise began to inquire why different icing mixes exhibit different consistencies (e.g., the cream cheese icing is thicker and not as melty as the butter cream). Lily also made more objective observations about the outcome of the cake. Instead of making subjective statements about whether the cake was yummy or tasty, Lily noticed that, one (the experimental) has more bubbles than the other, well not more bubbles, but theyre bigger than the other one (original outcome). In their StoryKit story, the group concluded that the experimental cake with more eggs make cake-ier, heavier, more packed cake. The group recorded comparative height measurements of each baked cake, noting that the original recipe rose to 3 cm, while the experimental one rose to 4.9 cm.
ISLS
107
ICLS2012
Case 3: Analysis
We observed that the learners prediction of the outcome of these modifications was tied to their participation in the semi-structured activities. The learners used past knowledge from these activities to make their choices. Even though the group did not conduct an experiment, they determined to use two eggs in the recipe because of the connection to the prior brownie activity. For example, in a prior StoryKit story, the learners wrote that the addition of more eggs produced brownies that were, Spongy and sticky. A lot more like cake than like brownie. Because they wanted a cake that is spongier, the learners made the modification to double the amount of eggs that was originally called for. However, for the replacement of baking soda and vinegar with baking powder, the adults worked carefully with the learners using the goals chart as a catalyst that helped them to structure their initial discussion so that they could ask the learners what they wanted in the cake. We gave each group the goals sheet to help them work through their decision-making process. The goal charts helped the adults to structure the discussion. For example, the adults prompted the learners with questions, such as, what do we want it (the cake) to taste like?, do cakey brownies have more eggs or less eggs?, and did it (the cookies) have more baking powder or less?. For each of the outcomes on the goals chart, the learners wanted a product that was cakey, moist, and soft. From this, they made as many modifications as they could to make sure they would have a moist outcome.
Discussion
One of the primary goals of KC was to support learners agency and personal investment into inquiry based scientific practices. We wanted learners to walk away not just with content knowledge and better understanding of the social practices of science, but with the notion that learners can make informed choices in science. Four common themes from the cross-case analysis emerged that identified aspects of the learning environment that support learners personal decisions: 1) Semi-structured activities helped learners to think about how they might structure their own investigations. 2) Facilitation from adults allowed learners to think scientifically about what decisions needed to be made. 3) Participation in whole group discussions helped learners to reflect and build knowledge for their choices. 4) The practice of using mobile technology allowed learners to build personalized narratives for reflection and discussion to help make informed choices for their personal investigations. Although we have parsed out these four practices, we recognize that each of them interacts with one another to help learners make informed decisions to support their personal goals. Each of the themes is an extension of our vision of Cooperative Inquiry (e.g., Druin, 2005), that is, learners can be design partners within the development of a learning environment. Our four cultural practices helped to support learners choice in design because we attempted to take seriously the challenge that children can make viable decisions in their science learning. This culture of learner agency and partnership permeated throughout the week. Lisa describes KC as a place where the kids can be adults, and the adults can be kids and that I was really excited, I was excited because I dont get to do a lot of this in school. You dont get to pick anything you want. I mean, they [teachers at school] give you a choice between one and two and half of the time, and like one and two and sometimes thats not what you want at all. Each of these four practices manifested and intertwined in three aspects of KC. First, the sequence of the activities gives learners a chance to develop ways to make decisions. Informed decision-making practices are not the sole responsibility of the learners, the facilitators, or the technology. Instead, our four cultural practices worked together to support these learning goals. We sequenced KC so that during the semi-structured activities and whole group conversations, learners would familiarize themselves with scientific and cooking practices in preparation for the decisions they would make in Choice Day. However, this sequence of activities would not have prepared the learners for decision-making if learners were not already engaged in this practice. For instance, using the mobile technologies in the activities, learners chose when to take photos, when to interview each other, and what belonged in their digital artifacts. During discussions, facilitators routinely worked with learners to help them make informed decisions about their arguments and claims. In the cases we have presented, the four practices work in tandem to help guide learners and their choices. Second, guidance and organization helps learners recognize what decisions they can make. In each of the three cases, while the sequence of the activities helped learners engage in the practices of inquiry to build independent investigations, we found that learners still needed help to use scientific reasoning to justify all choices. They often worked together with the adults and used the mobile technologies to guide them. During the activities, facilitators helped to break down their larger goals into smaller decision points. Facilitators prompted and reminded the learners of their goals and asked them what they wanted. Learners still had to make decisions, but justify those choices with evidence and reasoning. We observed from Cases 1 and 2 that mobile technologies helped to organize learners decisions to structure their investigations. From Case 1, Ari and Clyde
ISLS 108
ICLS2012
used the digital artifacts from the prior activities to guide their decisions. In Case 2, Lisa used the technology to record and make comparisons between her leavening experiments to select which amount of baking powder to use. Lisa states, I like StoryKit because it lets you use sequence. And Im a big fan of sequence because it helps you get in order. We recognize, however, that there is a tension between guidance and learners choices. Our description of Case 3 illustrates this point. Initially, Denises group was just interested in baking the cake, as opposed to engaging in scientific practices and understanding the phenomenon. We argue that choice in science can be an entry point for engagement and participation in science learning. However, Furman and Barton (2006) caution that giving learners choice means that facilitators need to work with learners in how to make choices in an informed way. The role of the facilitator is not just to prompt for decision points, but also to challenge learners to provide evidence or reasons in a way that makes their choices stronger. In our case, Beth and Helene attempted to work with the girls to bake their cake and help the learners make well-informed modifications and comparisons to the recipe. Lastly, opportunities for reflection are an important means for learners to make informed decisions. We argue reflection supports learners decision practices in two ways. First, learners needed to work together and listen to each others arguments. Part of the practices we put forth in KC are intended to help learners to focus more attention on group deliberation to ensure all members listen and learn from each others ideas and arguments. Learners had to feel secure enough to ask questions, make claims, and provide evidence for their reasoning. During whole group discussion time, adults, and learners worked together to come up with viable explanations for the observed phenomena. Both children and adults comfortably built on each others arguments. For example, we noticed that during discussions, Lisa jumped into discussions with ease and elaborated on an adults argument: Tammy: So you think less eggs, makes it more chewy? Lisa: Yeah, less eggs makes it more chewy because theres not enough egg particles to hold everything together. Jason: What do you mean by hold everything together? Lisa: Because you dont, I hear Tammy say that it [eggs] like water and oil, so the only reason that the oil and the water mix is that the egg particles forces to mix because it hung on to both. In the prior discussion, Tammy referred to eggs as emulsifiers that help oil and water mix together. Here, we see Lisa further adding to Tammys definition, giving a possible causal reason for why the oil and water mix. Our argument is that during these discussions, Lisa developed a comfort level with the adults, which later allowed her to develop in her decision-making practices. Second, in-the-moment captures of learners experiences are useful for helping them to stop and reflect on the activities. In the semi-structured activities, learners took photos and recorded their thoughts in situ on the iPads. We believe that in so doing, the learners slowed down from the activities and began to reflect. In discussions, learners would refer back to their stories and present them to the group as evidence for their claims. For instance, the participants engaged in a whole group discussion on bubbles and their role in the outcome of their semi-structured experiments on eggs and leaveners. During this time, learners shared their stories with the group. For example, Ari shared this portion of his StoryKit story with the group, The bubbles probably make the brownies rise. Within the whole group setting both learners and adults attempted to determine what bubbles are composed of, how bubbles might be formed (both from a macroscopic and submicroscopic perspective), and what role did bubbles have in producing the outcomes they observed in the brownies and cookies. There is a chance that Ari and Clydes group utilized the ideas from the whole group discussion and their story presentation to connect the foam to the future outcome of the vegan brownies.
Conclusions
This work is not the prescriptive, singular manual for how to help learners make choices in science learning. Instead, we suggest that supporting learners decisions in science learning is a dynamic and complex challenge that requires practitioners and researchers to consider many issues. We argue that the cultural practices are important for supporting decision-making practices in science, not because they are formal scaffolds, but because they interact to support a participatory culture in which informed choices are a form of empowerment for learners. As mentioned before, learners lack of interest in science may be in part due to their perception of disempowerment, including the lack of opportunities to use science in meaningful real life contexts. Based on the work presented here, we suggest that empowerment can come from helping learners make personal decisions in science with contexts and goals that support their interests.
References
ISLS
109
ICLS2012
Aikenhead, G. S. (1996). Science education: Border crossing into the subculture of science. Studies in Science Education, 27, 1-52. Atwater, M. M. (1996). Social constructivism: Infusion into the multicultural science education research agenda. Journal of Research in Science Teaching, 33(8), 821837. Barton, A. C., & Tan, E. (2009). Funds of knowledge and discourses and hybrid space. Journal of Research in Science Teaching, 46(1), 5073. Basu, S. J., & Barton, A. C. (2007). Developing a sustained interest in science among urban minority youth. Journal of Research in Science Teaching, 44(3), 466489. Bouillion, L. M., & Gomez, L. M. (2001). Connecting school and community with science learning: Real world problems and schoolcommunity partnerships as contextual scaffolds. Journal of Research in Science Teaching, 38(8), 878898. Cahill, C., Kuhn, A., Schmoll, S., Pompe, A., & Quintana, C. (2010). Zydeco: Using mobile and web technologies to support seamless inquiry between museum and school contexts. Proceedings of the Ninth International Conference on Interaction Design and Children (pp. 174177). Chinn, C. A., & Malhotra, B. A. (2002). Epistemologically authentic inquiry in schools: A theoretical framework for evaluating inquiry tasks. Science Education, 86(2), 175218. Clegg, T. L., Gardner, C., & Kolodner, J. (2011). Technology for supporting learners in out-of-school learning environments. In Proceedings of the Ninth International CSCL Conference. Hong Kong, China. Clegg, T. L., Gardner, C., & Kolodner, J. L. (2010). Playing with food: Moving from interests and goals into scientifically meaningful experiences. In Proceedings of the Ninth International Conference of the Learning Sciences (ICLS) (Vol. 1, pp. 1135-1142). Druin, A. (2005). What children can teach us: Developing digital libraries for children with children. The Library Quarterly, 75(1), 2041. Furman, M., & Barton, A. C. (2006). Capturing urban student voices in the creation of a science minidocumentary. Journal of Research in Science Teaching, 43(7), 667694. Fusco, D. (2001). Creating relevant science through urban planning and gardening. Journal of Research in Science Teaching, 38(8), 860877. Jenkins, H., Clinton, K., Purushotma, R., Robinson, A., & Weigel, M. (2006). Confronting the challenges of participatory culture: Media education for the 21st century. Chicago: MacArthur Foundation. Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., Puntambekar, S., et al. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting Learning by DesignTM into practice. Journal of the Learning Sciences, 12(4), 495547. Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. San Francisco, CA: John Wiley and Sons. Quinn, A., Bederson, B., Bonsignore, E., & Druin, A. (2009). StoryKit: Designing a mobile application for story creation by children and older adults (No. HCIL-2009-22) (pp. 1-10). College Park, MD: Human Computer Interaction Lab, University of Maryland. Reiser, B. J. (2004). Scaffolding complex learning: The mechanisms of structuring and problematizing student work. Journal of the Learning Sciences, 13, 273304. Yin, R. K. (2009). Case study research: Design and methods (4th ed., Vol. 5). Thousand Oaks, CA: Sage Publications, Inc.
Acknowledgements
We want to thank the CI Fellows program for funding this work. We thank David Cavallo for his advice, Alex Quinn and Ben Bederson for the use of StoryKit, and Alex Kuhn and Chris Quintana for the use of Zydeco. Finally, we acknowledge our participants and the local school community that partnered with us for this work.
ISLS
110
ICLS2012
Students Intuitive Understanding of Promisingness and Promisingness Judgments to Facilitate Knowledge Advancement
Bodong Chen, Marlene Scardamalia, Monica Resendes, Maria Chuy, Carl Bereiter, University of Toronto, 252 Bloor Street West, IKIT 9th floor, Toronto, ON, M5S 1V6, Canada Email: bodong.chen@utoronto.ca, marlene.scardamalia@utoronto.ca, monica.resendes@utoronto.ca, maria.chuy@gmail.com, carl.bereiter@utoronto.ca Abstract: The ability to identify promising ideas is an important but obscure and undeveloped aspect of knowledge building. The goal of this research was to examine the extent to which young students can make promisingness judgments and, as a result, engage in more effective knowledge building. Toward this end we embedded a design experiment in a Grade 3 classroom. In this experiment students were engaged in discussion and reflection of the concept of promisingness and used a Promising Ideas tool to identify promising ideas in their written online discourse. They used the tool for two refinements of idea selections to focus ongoing community dialogue. Results suggest that students as young as 8 years of age can make promisingness judgments that facilitate knowledge advancement in their work. These results inform future work in classroom interventions and tool development to promote promisingness judgments in collaborative knowledge building. Like scientists in research laboratories (Dunbar, 1995), students engaged in knowledge building participate in constructive and progressive knowledge-building discourse, in which they contribute to group dialogue in distinctive ways, including proposing theories, synthesizing ideas, and making analogies (Chuy, Zhang, Resendes, Scardamalia, & Bereiter, 2011). A knowledge building principle that frames such discourse is collective cognitive responsibility (Scardamalia, 2002), according to which students share responsibility for advancing the knowledge of their community. To advance community knowledge, knowledge building calls for risk taking in pursing novel solutions to problems, with an ongoing commitment to continual idea improvement. Knowledge Forum (Scardamalia, 2004) is an online, community space specially designed to support Knowledge Building pedagogy. This pedagogy and technology has been adopted in international contexts and across the school curriculum (Scardamalia & Egnatoff, eds., 2010). The ability to identify promising ideasideas that with development might grow to something of consequenceis essential for creative work with ideas, but remains unexplored (Bereiter & Scardamalia, 1993). According to Bereiter (2002), knowledge of promisingness is crucial in every kind of creative work at all levels, and accordingly should be recognized as an important component in the knowledge-building process. Knowledge of promisingness is acquired over time as people engage in creative practices, by taking risks and learning from the successes and failures that are integral to the creative process. This type of knowledge increases with creative expertise, helping people improve their ability to take successful risks. Thus, to prepare young generations to be future creative achievers in various fields, we should encourage them to make promisingness judgments during their knowledge building work. This study serves as a first step in a program of research that seeks to explore the extent to which young students can make promisingness judgments and how educators can support them. The study addresses two central research questions: Can young students assess promisingness, or the knowledge building potential of their own ideas? Do their selections of promising ideas, and further community discourse on these selections, facilitate knowledge advancement in the community?
Promisingness and Knowledge Building Discourse

Substantial work is normally needed to develop ideas to address knowledge creation goals. Single ideas seldom constitute problem solutions; neither do simple combinations of ideas. When the goal is knowledge creation, the problem space tends to be complex and the extent to which an idea will prove valuable cannot be known at the beginning. As Johnson (2010) points out, an idea is not a single thing, but more like a swarm (p. 43); to better grow, an idea should live in a liquid network where it can more easily connect with its adjacent possible (Kauffman, 2002, p.47). However, an abundance of possible connections to an idea does not guarantee the fulfillment of a knowledge creation goal. A significant challenge in all creative work, in both the fine and broad grain, is to identify promising directions and to avoid wasting time or becoming entrapped by unpromising ones (Bereiter & Scardamalia, 1993). In the knowledge-creating context, promising ideas provide indication of future success or good results in a specific problem domain. Evaluation of risk and promisingness is a natural component of the knowledge creation process. For example, in his study of scientific reasoning in real-world laboratories, Dunbar (1995) found that scientists tend to categorize their projects into different levels of risk: a high risk project has a low probability of working out but may lead to breakthroughs, whereas a low risk project has a high probability of
ISLS
111
ICLS2012
success but does not promise important discoveries. Assessments of promisingness help scientists determine how and when to invest their resources and time. Promisingness guides action of the moment as well as overall approach. To become a creative achiever in any field, one has to take risks at many levels and learn from the results (Bereiter & Scardamalia, 1993). The current study focuses on knowledge-building discourse by young students, exploring the extent to which promisingness judgments might facilitate their knowledge advancement.
Method
Participants
Participants were students from a grade 3 class at the Dr. Eric Jackman Institute of Child Study in downtown Toronto. There were 11 boys and 9 girls taught by a novice Knowledge Building teacher in 2011. They were studying earth sciences, with focus on soil and worms, for eight weeks. In this elementary school, Knowledge Building pedagogy and technology are used from grade 1 on, so students each year had roughly equivalent experience in the pedagogy and technology.
Promising Ideas Tool

A Promising Ideas tool was added to Knowledge Forum to facilitate this research. It includes two components: (a) a highlight feature to allow students to tag an idea within a note using a customizable categorization scheme (see Figure 1, left side) and (b) a component aggregates all selected ideas within a same view, merges overlapping ideas, and visualizes them in a list (see Figure 1, right side). An early version of this tool was implemented and tested in a small-scale pilot study (Chen, Chuy, Resendes, & Scardamalia, 2010). The goal of that study was not to enhance students ability to make judgments on idea promisingness, but to identify whether they are able to make these judgments at all. Results indicated that without any explanation of the concept of promisingness, grade 5/6 students tended to identify important-sounding facts and questions as promising. Thus, we had reason to doubt that students were making promisingness judgments based on the potential for deepening their understanding or identifying new directions in their research and development (Chen, Chuy, Resendes, Scardamalia, & Bereiter, 2011). Based on results from the first pilot study, we reengineered this tool to emphasize the concept of promisingness as the essence of this tool. The new version maintained functionalities from the previous version, including highlighting, tagging, and filtering. To make the endeavor of identifying promising ideas more meaningful, we implemented a third component that allows students to export a subset of identified promising ideas from the aggregate list to a new workspace for further inquiry. By reviewing the promising idea list together, a class could collectively determine which ideas or questions they wish to pursue for next steps. These ideas can be selected by clicking on checkboxes corresponding to each chosen idea on the list (see Figure 1, right side). Then, notes which contain those promising ideas can be exported to a new Knowledge Forum view by clicking on the Export Notes button. After that, students can work to develop those exported promising ideas to a deeper level. This function could engage students in identifying promising ideas with a commitment to define new directions for their work.
Figure 1. The Promising Ideas tool currently has three components: (a) tag an ideaon the left, a student can identify an idea with a customizable scheme; (b) list tagged ideasin the background window on the right side, all identified promising ideas from a view are listed; and (c) export selected ideas to a new viewusers can then select a subset of ideas from this list and export them to a new view (the foreground window on the right).
ISLS
112
ICLS2012
Procedure
Beside the grade 3 class which participated in this study, another grade 3 class from the previous year in the same school was included to provide baseline data. Both grade 3 classes studied Soil in the Environment for around 8 weeks, a unit in the Ontario Science and Technology Curriculum. In studying this unit, both classes were following a knowledge-building approach guided by twelve knowledge building principles (Scardamalia, 2002; Zhang, Hong, Scardamalia, Teo, & Morley, 2011). Knowledge building practice in this school usually contains four components: Knowledge Building talk (KB Talk), online discourse in Knowledge Forum, study on authoritative sources, and experiments or observations. There is no routine that defines how these four components should proceed. Rather, these components are organized in a classroom in a flexible and organic manner that facilitates students collective knowledge building. Both classes started with a KB talk, with students sitting in a circle sharing their initial authentic questions and ideas about soil. After the first KB talk, what is soil made of? and how to make soil? emerged as the focus of students study of soil in both classes. To keep students ideas alive and to extend student discussion to a broader scale, students then moved to Knowledge Forum to record their ideas and start online dialogue. They contributed to their discourse in various ways, including proposing their own theories, building on each others theories, providing observation results to defend their claims, designing and conducting experiments to test hypotheses, and reading and introducing authoritative sources such as books or online materials. Although two classes were led by two different teachers, dynamics in both classrooms followed a similar knowledge building process. In the experimental class we implemented a promisingness intervention into regular knowledgebuilding discourse. This intervention mainly comprised two components. The first component was a 30 minute pedagogical intervention targeting students understanding of promisingness. As the first step, we asked grade 3 students to brainstorm with their partners the meaning of a promising idea and a promising question, and recorded their group thoughts on paper. Student notes were collected for later analysis. Ten minutes later, students brought their thoughts back to a class discussion, advancing different but plausible definitions of the concept and supporting them with meaningful examples. In this discussion, the teacher specifically invited students to ponder the difference between promising ideas and verified facts, to help them distinguish promisingness from truthfulness. By the end of the 30-minute discussion, the class arrived at a shared understanding that promising ideas are ideas that they wish to spend time on, may change in further inquiry, and would deepen their shared understanding. The second component of this intervention engaged students in progressive use of the Promising Ideas tool. The whole process involved two iterations of note writing and promisingness intervention. During the first two weeks, students worked on a view named Grade 3 Soil 2010/11 in Knowledge Forum. After that, students were directed to conduct promisingness judgments on this view, picking ideas that they thought were promising using the Promising Ideas tool. Then the whole class had a discussion on identified ideas, and decided to export several promising ideas to a new view called Where does soil come from? The second iteration of note writing and promisingness evaluation then followed. During the next three weeks students kept working on the exported view. After that, students did a second round of promisingness evaluation on that view, and exported their promising ideas to another new view named Worms and Soil, on which students worked for another two weeks. In the whole process, promisingness evaluation was integrated into regular knowledge-building discourse as a component to determine future steps of student inquiry. Student notes and selections of ideas were collected for data analysis.
Data Analysis
Students notes of their intuitive understanding of promisingness were collected from the brainstorm session in the experimental class. Overall, 8 groups of students produced 35 distinctive notes. We coded data based on a grounded theory approach (Glaser & Strauss, 1967), with a goal to identify popular conceptions, especially misleading ones, of promisingness among this group of students. In work on this soil unit, students from the experimental group created 125 notes in three views (respectively 38, 47, and 40 notes). During two sessions of promisingness judgments, students identified 57 and 94 promising ideas from the first two views. Each single highlighting action was counted as an idea. Since several students could independently select a same idea, there were a number of repetitions of ideas. In the baseline group, students produced 130 notes in five views. They were not making promisingness judgments in any form, so no idea was identified. Two independent raters used the following criteria to code the notes and ideas. They agreed on 82.5% of the coding. Discrepancies were discussed to reach a final agreement. Contribution types of identified promising ideas. Because promisingness judgments in knowledge building focus on potential for growth, we expect ideas identified by students as promising to be of a theoretical nature rather than pure facts (Chen et al., 2011). In order to understand the nature of identified ideas and assess whether the pedagogical intervention was effective in transforming students understanding of
ISLS
113
ICLS2012
promisingness, we applied an extended list of ways of contributing scheme previously developed by Chuy and colleagues (2011). This scheme includes six major categories, including: formulating thought-provoking questions (e.g., explanatory, factual, and design questions), theorizing (e.g., proposing, supporting, and improving a theory), obtaining evidence (e.g., seeking evidence, reporting experiment results, introducing facts from sources or experience), working with evidence (e.g., using evidence to support or discard a theory), synthesizing and comparing (e.g., synthesizing discussion, creating rise-aboves, and making analogies), and supporting discussion (e.g., facilitate discussion, give an opinion). Measurements of individual and community knowledge advancement. Knowledge advancement by students in both the experimental and baseline groups was assessed through content analysis of student notes in Knowledge Forum. First of all, scientificness of student ideas was assessed using a four-point scale developed by Zhang and colleagues (2007): 1pre-scientific (containing a misconception while applying a naive conceptual framework), 2hybrid (containing misconceptions that have incorporated scientific information), 3basically scientific (containing ideas based on a scientific framework, but not precise), or 4scientific (containing explanations that are consistent with scientific knowledge). To evaluate whether the experimental class has made any progress in terms of scientificness of their ideas, we analyzed all student notes that fell into the theorizing category in the ways of contributing analysis, and compared across three views produced by three phases of student work. The same analysis was performed for the baseline group, and results were compared between two groups. Second, latent semantic similarities between student notes and authoritative sources were calculated through Latent Semantic Analysis (LSA) to approximately represent students level of collective understanding of the subject matter. LSA is a theory and model for extracting and inferring contextualized, latent meaning of words from statistical analysis of a corpus of text (Landauer, Foltz, & Laham, 1998). It has been widely applied to compare semantic meaning of different documents, and its adequacy in reflecting human knowledge has been established in extensive educational settings (Foltz, Kintsch, & Landauer, 1998; Forster & Dunbar, 2009; Landauer & Dumais, 1997; Rehder et al., 1998; Teplovs & Fujita, 2009). In LSA, each document is treated as a bag of words; comparison between documents is based on latent connections between their words in a predefined semantic space trained through complex mathematical computations. To train a semantic space about soil, we first selected four pieces of authoritative sources that represent varied levels of complexity, including Soil in the Environment unit in the Ontario curriculum standards (2007), Wikepedia article about Soil (2011), an entry level article about soil from US Environmental Protection Agency (2011), and a more sophisticated education article from US Department of Agriculture (2011). Then we exported student discourse in three views to three individual documents, and compared them with the list of authoritative sources for similarity. Similarity scores between student discourse and authoritative sources represented the advancement of community knowledge at different knowledge building phases.

Student Understanding of Promisingness Students Intuitive Understanding of Promisingness
As shown in previous studies, when asked to select promising ideas, young students often select important facts of the sort they address in their school work (Chen et al., 2011). In this investigation we considered students intuitive understanding of promisingness. Analysis indicated that some students thought a promising idea was a true idea, an idea that promises you something or actually happened; for a promising question, some students thought this could be explained as a question you promise to answer, a question that has multiple answers, a question that has not many answers, or a question that is difficult and should be solved. Overall, students revealed different understandings of promisingness in different contexts. However, three themes emerged from students notes about the meaning of promisingness. First of all, consistent with the results from previous studies (Chen et al., 2011), the feature of being true or truthfulness was a popular notion present in notes of all groups. For example, some students thought a promising idea was: Group 1: A true idea or an idea that is not incorrect. Group 2: True. Group 3: You know that an idea is true, or you promise that it is right. Group 5: An idea that you are pretty sure is right. These thoughts could explain why students in our previous study were identifying facts such as The universe is 13,000,000,000 years old! and the grand canyon could have 912,456 layers of rock (Chen et al., 2011). The second cluster of thoughts focused on the notion of likelihood. For example, a number of groups mentioned that a promising idea was: Group 4: An idea that is very good and probably be right.
ISLS
114
ICLS2012
Group 5: An idea that might work. Group 7: This is probably a right idea. Group 8: Idea that is most likely to be right, or 90% sure its right. This notion of likelihood went further than emphasizing truthfulness, in that there is a recognition that we cannot tell whether a promising idea is true for sure, but a promising idea has high odds of being true. According to dictionaries such as Merriam-Webster, promising means showing signs of future success or excellence, likely to turn out well, or likely to succeed or yield good results. Bereiter (2002, p.337) explained promisingness by the metaphor of hill climbing, highlighting that a promisingness evaluation is like to find the way up the hill although you cannot see a path to the summit. In this sense, the notion of likelihood students captured is an essential component of promisingness. However, it is important to note that students were thinking of likelihood in terms of an ideas current status of being true. Apparently, they were not thinking of ideas with a trajectory for growth. The third notion that emerged from students notes was about future actions. When thinking about what is a promising question, a few groups wrote: Group 1: A question you can spend time on. Group 2: A question you need to know. Group 5: A question that can help you do something. The notion that a promising idea or question is something that you can spend time on or can help you better know or do something fit the notion that a promising idea could be flawed but worth laboring to improve it (Bereiter & Scardamalia, 1993; Bereiter, 2002). This notion provided a good basis to work from to prepare students with a favorable understanding for promisingness judgments. To summarize, without intervention the concept of promisingness proved to have different meanings under different conditions for grade 3 students.
Was promisingness talk effective?

Previous research shows young students tend to select important facts of the sort they address in their school work as being promising (Chen et al., 2011). In the current study we introduced a pedagogical intervention module to engage grade 3 students in thinking and discussing about the meaning of promisingness as a group. Given this was our first attempt to engage them actively in the concept of promisingness, it is important to evaluate whether this intervention was effective in preparing students to consider promisingness as extending beyond factual content to ideas that might advance work in the field. To answer this question, we applied the ways of contributing scheme described above. Results showed that grade 3 students identified a large portion of theorizing (68.9%) and much less obtaining evidence contributions (6.7%) as being promising. Students also identified other contribution types, including questioning (9.8%), working with evidence (4.9%), synthesizing and comparing (1.8%), and supporting discussion (8.0%). Overall, the proportion of fact/evidence has decreased significantly, comparing to selections by grade 5/6 students reported in previous research (Chen et al., 2011). This finding indicated that the concept of promisingness could be easily extended, so that rather than a focus on true ideas students have more of a sense of ideas with a promising growth trajectory. To further understand whether evidence picked by grade 3 students was introduced with a goal to move discourse forward, we analyzed the dialogue context of the contributions related to obtaining evidence. Results indicated most these contributions fell into the introducing new information sub-category, mostly containing new facts or new pieces of evidence; this situation was similar to that in two grade 5/6 classes (Chen et al., 2011). However, further analysis on the grade 3 discourse revealed that obtaining evidence contributions identified by students as promising usually co-occurred with the contribution type of using evidence or reference to support a theory (which falls within the working with evidence contribution category). Apparently, students from this grade 3 class were identifying those obtaining evidence contributions which were made to drive discourse forward. In contrast, as reported in previous study, fact/evidence contributions were identified by grade 5/6 students, who did not spend time brainstorming or discussing the concept of promisingness, were more likely to be standalone. The shift in understanding can be seen in the following contrast. In an idea identified by one grade 3 student, the new information when its night its cooler in the day the sun is shining and its warmer was introduced to support her theory that worms sense light by temperature. However, grade 5/6 students were more likely to identify standalone facts, such as the solar system formed 4570 million years ago. Notably, grade 3 students who went through the promisingness discussion were more likely to identify facts or evidence that contributed to their knowledge-building discourse, while grade 5/6 students tended to identify pure facts that had some distinctive attractiveness (Chen et al., 2011).
Promisingness Judgments to Facilitate Knowledge Advancement

Promisingness judgments are to boost community knowledge by identification of promising directions that are refocusing in ongoing discourse. Evaluating the promisingness of their own ideas engages students in deep reflection on the state of their knowledge work and helps them recognize ideas that they deem worth extended work. In this way, students can denote their limited time and energy to more promising ideas. As a result, their
ISLS
115
ICLS2012
individual and collective understanding may get more opportunities to grow. In this study, our hypothesis was that students in the experimental class, who performed promisingness judgments, would achieve greater knowledge advancement than students in the baseline class. To test this hypothesis, in this section we examine idea improvement and knowledge advancement in two classes by two means: rating students theorizing contributions for scientificness, and comparing semantic space of student discourse with expert content.
Scientificness of notes and conceptual change

First of all, we rated students theorizing contributions in two classes on the continuum from pre-scientific to scientific understanding. In the experimental class, we identified and analyzed 91 theorizing notes, 26 notes from Grade 3 Soil 2010/11 (view 1), 42 notes from Where does soil come from (view 2), and the rest 23 from Worms and Soil (view 3). In the baseline class, we identified a total number of 68 theorizing notes. Because the baseline group did not integrate promisingness judgments into their discourse, there was no natural distinction of different phases; so we sequenced student notes according to the time of creation and manually divided them into three phases with equivalent number of notes. Mean scores and standard deviations for three discourse phases in both classes are presented in Table 1. A 2 (Group) 3 (Discourse Phase) factorial ANOVA was performed to assess whether scientificness scores of student ideas could be predicted from student group (experimental vs. baseline), phases of discourse (phase 1, 2 and 3), and the interaction between these two factors. This analysis of variance revealed a significant main effect for discourse phases, F (2, 30) = 13.51, p < .001, 2 = .15, indicating the mean scientificness scores were different among three phases. The main effect for group difference was not significant, F (1, 30) = .47, p = .49, 2 = .00. The interaction between discourse phases and group difference was significant, F (2, 30) = 3.81, p < .05, 2 = .05. Table 1: Mean and Standard Deviations for Three Discourse Phases in Experimental and Baseline Groups. Phase 1 Experimental Group M SD n Baseline Group M SD n 1.29 .38 26 1.50 .58 23 Phase 2 1.45 .55 42 1.50 .62 23 Phase 3 2.24 .89 23 1.77 .67 22
Figure 2. Mean scores of scientificness of theories across three discourse phases. The graph of mean scores (in Figure 2) indicated that scientificness scores in the experimental class have been substantially improved across three phases, while scientificness scores in the baseline group were
ISLS 116
ICLS2012
only slightly improved. Planned contrasts were done with L-Matrix subcommand to assess whether these differences (between phase 1, 2, and 3) were significant within the experimental and baseline groups. For the experimental group, the improvement from phase 1 to 3 was significant, F (1, 153) = 28.70, p < .001, 2 = .16; so was the improvement from phase 2 and 3, F (1, 153) = 23.94, p < .05, 2 = .14. In contrast, none of the differences in the baseline group was significant. Notably, the experimental group started with less scientific ideas than the baseline group, and finished with more scientific understanding about soil and worms. In conclusion, students in the experimental class, who performed promisingness judgments, achieved greater knowledge advancement than students in the baseline class.
Growth of subject domain knowledge

To further explore the changes of community knowledge in the experimental group, we applied Latent Semantic Analysis (LSA) as another distinctive analysis technique. LSA is able to accurately identify semantic similarity between two documents within a semantic space. In this analysis, student notes in three different views were exported to three individual documents, and then respectively compared with authoritative sources about soil. LSA semantic similarity scores between student written dialogues and authoritative sources were used to infer students levels of understanding in different phases. Because view 3 (Worms and Soil) in the experimental group focused on worms rather than soil, we only included view 1 (Grade 3 Soil 2010/11) and view 2 (Where does soil come from?) into this analysis. We also included a view (Grade Three Soil 2009/10) produced by the baseline group for comparison. Figure 3 presents LSA cosine similarity scores of three views when they are respectively compared to four pieces of authoritative sources, i.e. Ontario curriculum standards, Wikipedia article, US Environmental Protection Agency (US EPA) article, and US Department of Agriculture (US DA) article.
Figure 3. LSA cosine similarity scores between views and different authoritative sources. The first two views, i.e. Grade 3 Soil 2010/11 and Where does soil come from?, were produced by the experimental class, and the third view, Grade Three Soil 2009/10, was from the baseline group. From this analysis a notable pattern emerged that the similarity scores of view 2 (M = .24) were substantially higher than view 1 (M = .08) in comparing with those different pieces of authoritative sources. The similarity scores of view 2 were even higher than the baseline group (M = .18) taught by a more experienced Knowledge Building teacher. These results from LSA indicated that students in the experimental group had improved their understanding on soil across the first two dialogue phases, to an equivalent or higher level of the baseline group.

The goal of this study was to examine the extent to which young students can make promisingness judgments on their own ideas, and accordingly achieve more advancement in their knowledge-building discourse. Toward this end, a promisingness intervention was employed in a grade 3 class, which engaged students in discussing the concept of promisingness and then using the Promising Ideas tool to identify promising ideas during knowledge-building discourse in a progressive manner. Analysis of results indicated that with discussion and analysis of the concept of promisingness, students as young as 8 to 9 years old could make promisingness judgments which, compared to those of grade 5/6 students (Chen et al., 2011), advanced beyond the idea that factual content defines promisingness. Scientificness and knowledge coverage of student notes were further analyzed across three discourse phases divided by two sessions of promisingness judgments. Results showed a significant growth of scientificness as well as domain knowledge across three phases in the experimental group,
ISLS
117
ICLS2012
while the baseline group, who did not engage in making promisingness judgments on their work, did not reveal such growth. Promisingness evaluation is an important, but obscure component in knowledge-building discourse (Bereiter & Scardamalia, 1993). Substantial further research is needed to examine the role promisingness evaluations play in different contexts. New studies will tackle fundamental research issues such as how people make promisingness judgments, a developmental trajectory for making promisingness judgments, and means to help students gain deeper understanding through making more promisingness judgments.
References
Bereiter, C. (2002). Education and Mind in the Knowledge Age (p. 544). Lawrence Erlbaum. Bereiter, C., & Scardamalia, M. (1993). Surpassing Ourselves: An Inquiry Into the Nature and Implications of Expertise (p. 296). Open Court Publishing Company. Chen, B., Chuy, M., Resendes, M., & Scardamalia, M. (2010, August 4). Big Ideas Tool as a New Feature of Knowledge Forum. Poser presented at 2010 Knowledge Building Summer Institute. Toronto, Ontario, Canada. Chen, B., Chuy, M., Resendes, M., Scardamalia, M., & Bereiter, C. (2011). Evaluation by Grade 5 and 6 Students of the Promisingness of Ideas in Knowledge-Building Discourse. In H. Spada, G. Stahl, N. Miyake, & N. Law (Eds.), Connecting Computer-Supported Collaborative Learning to Policy and Practice: CSCL2011 Conference Proceedings. Volume II - Short Papers and Posters (pp. 571-575). International Society of the Learning Sciences. Chuy, M., Zhang, J., Resendes, M., Scardamalia, M., & Bereiter, C. (2011). Does Contributing to a Knowledge Building Dialogue lead to Individual Advancement of Knowledge? In H. Spada, G. Stahl, N. Miyake, & N. Law (Eds.), Connecting Computer-Supported Collaborative Learning to Policy and Practice: CSCL2011 Conference Proceedings. Volume I - Long Papers (pp. 57-63). International Society of the Learning Sciences. Dunbar, K. N. (1995). How scientists really reason: Scientific reasoning in real-world laboratories. In R. J. Sternberg & J. Davidson (Eds.), The Nature of Insight (pp. 365-395). MIT Press. Foltz, P., Kintsch, W., & Landauer, T. (1998). The measurement of textual coherence with latent semantic analysis. Discourse Processes, 25(2), 285-307. doi:10.1080/01638539809545029 Forster, E. A., & Dunbar, K. N. (2009). Creativity Evaluation through Latent Semantic Analysis. Proceedings of the Annual Conference of the Cognitive Science Society. Glaser, B. G., & Strauss, A. L. (1967). The Discovery of Grounded Theory. Discovery (p. 271). Aldine. Johnson, S. (2010). Where Good Ideas Come From: The Natural History of Innovation. Riverhead Hardcover. Landauer, T. K., & Dumais, S. T. (1997). A solution to Platos problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211-240. doi:10.1037//0033-295X.104.2.211 Landauer, T., Foltz, P., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25(2), 259-284. doi:10.1080/01638539809545028 Rehder, B., Schreiner, M. E., Wolfe, M. B. W., Laham, D., Landauer, T. K., & Kintsch, W. (1998). Using latent semantic analysis to assess knowledge: Some technical considerations. Discourse Processes, 25(2), 337 354 ST - Using latent semantic analysis to. doi:10.1080/01638539809545031 Scardamalia, M. (2004). CSILE/Knowledge Forum. In A. Kovalchick & K. Dawson (Eds.), Education and technology: An encyclopedia (pp. 183-192). Santa Barbara, CA: ABC-CLIO. Scardamalia, M., & Egnatoff, W. (Eds.). (2010). Canadian Journal of Learning and Technology, Special Issue on Knowledge Building, 36(1). Teplovs, C., & Fujita, N. (2009). Determining curricular coverage of student contributions to an online discourse environment: Using latent semantic analysis to construct differential term clouds. Proceedings of the 9th International Conference of Computer Supported Collaborative Learning 2009 - Volume 2 (pp. 165-167). Rhodes, Greece: International Society of the Learning Sciences. Zhang, J., Scardamalia, M., Lamon, M., Messina, R., & Reeve, R. (2007). Socio-cognitive dynamics of knowledge building in the work of 9- and 10-year-olds. Educational Technology Research and Development, 55(2), 117-145. doi:10.1007/s11423-006-9019-0
Acknowledgments
This research was funded by a Research Grant from the Social Sciences and Humanities Research Council of Canada titled Ways of contributing to dialogue in elementary school science and history. We would like to express our gratitude to the students, teachers and principals of the Dr. Eric Jackman Institute of Child Study, University of Toronto for the insights and the opportunities enabled by their involvement.
ISLS
118
ICLS2012
Designing Video Games that Encourage Players to Integrate Formal Representations with Informal Play
Nathan R. Holbert, Uri Wilensky, Northwestern University, Evanston, IL Email: nholbert@u.northwestern.edu, uri@northwestern.edu Abstract: In this paper we present a case study of a player of the game FormulaT Racinga prototype racing video game we created to explore design principles that could transform popular games into powerful learning environmentsshifting between and ultimately integrating video game and school-based epistemological framings. We show a player shifting from describing acceleration and velocity in a purely game-based narrative during a pre-game interview to descriptions that seamlessly integrated both game and formal representations in a post-game interview. We contend that this change is due to novel interactions with key representations introduced throughout the game and discuss design implications for future games.
Introduction
As technology continues to improve, video games are becoming a preferred way many American children spend their free time. The PEW Internet and American Life Project claims that as many as 97% of all American teens (regardless of gender, age, or socioeconomic status) play video games in some way and 50% play games daily for an hour or more (Lenhart et al., 2008). A more recent report by the Kaiser Family Foundation suggests young tweens and Black and Hispanic children (Rideout, Foehr, & Roberts, 2010) spend even more time playing video games daily. With such a time investment by all of our children it is important to not only try to understand the cognitive implications of this common experience, but also to consider ways in which we might enhance this activity that is so central to youth culture. A number of video games have recently been designed to teach science content in the classroom (Barab, Thomas, Dodge, Carteaux, & Tuzun, 2005; Clark, Nelson, D'Angelo, Slack, & Menekse, 2010; Squire, Barnett, Grant, & Higginbotham, 2004). Like our colleagues, we believe video games are excellent vehicles for experimenting with science phenomena, but rather than design science video games for the classroom, we have chosen to explore design principles that could transform popular commercial video games, games that are typically played outside of school settings, into powerful educational experiences. With this in mind, we have designed a prototype video game, FormulaT Racing (Holbert & Wilensky, 2010a) to be an archetype of the racing genre while providing a platform for experimentation with new representations and interaction mechanisms. Instead of trying to explicitly teach players embedded physics concepts, FormulaT Racing (FTR) is designed to tap into childrens intuitive notions of motion and to connect these intuitions to formal representations. Consequently, while players may not become experts in kinematics by playing FTR, they should be left with a sense that their experiences in the game are relevant to non-game motion experiences and likewise, that formal definitions and representations of kinematic content are relevant to game experiences. Many video games designed with educational outcomes in mind have shown moderate success on related assessment measures, however, gains in related but not directly equivalent concepts are often less pronounced (Clark et al., 2010). In this paper we propose this failure to transfer may be due to incongruent epistemological framing by players and explore how the design of FTR helps one player overcome this dilemma. In particular, we show Brian, a fourteen year old gamer, shifting from describing acceleration and velocity in a purely game-based narrative during a pre-game interview to descriptions that seamlessly integrated both game and formal representations in a post-game interview. We contend that this epistemological integration is due to interactions with key representational changes introduced throughout the game and discuss design implications for future games.
FormulaT Racing's Design

FormulaT Racing allows players to explore kinematics by putting the player in the shoes of a new driver as they train to join the FormulaT Racing Circuit. The design principles of FTR specifically target relevant phenomenological causal schema (diSessa, 1993) bringing typically intuitive interpretations of in-game motion to the foreground through motion-sensitive and body-syntonic controls (Papert 1980) so that they can be explored and manipulated. FTR is designed to look and feel like a traditional commercial racing game though it includes a number of key design changes. One important design feature is the inclusion of intuitive and bodysyntonic controls. Using a Nintendo Wiimote players manipulate their vehicles acceleration by rolling the controller forward or backwards for positive and negative acceleration respectively. Track features and level challenges force players to frequently adjust the roll applied to the controller encouraging them to reflect on acceleration as a changing quantity that directly impacts velocity.
ISLS 119
ICLS2012
Traditionally racing video games have provided kinematic feedback to the player through visual cues such as a passing background, haptic cues such as a controller, and numerical cues such as a speedometer. FormulaT Racing retains the traditional passing background visual cue, but adds a new color-trails representation. In some of the games levels, velocity is represented by a color-trail left by the player vehicle that changes with the player cars velocity. The color-trail representation encourages the players to make connections between the motion sensitive controls, changes in acceleration and velocity, and the varying structure and features of the track. Interviews with children playing popular commercial racing games suggested players rarely attended to provided speedometers (Holbert, 2009). Rather than continue the tradition of providing discrete quantitative indications of the player cars speed, we chose to utilize a velocity versus time graph. This representation serves two important goals. First, as a widely used formal representation, the velocity versus time graph acts as an anchor to formal knowledge within the informal game-space. As we will argue later, we believe this mixing of formal representations and gameplay encourages the player to see her game-knowledge as relevant across contexts. Since the player often identifies as an expert gamer, the application of this game-knowledge to nongame spaces is extremely powerful. Second, rather than a speedometer, which highlights instantaneous speed, a velocity versus time graph foregrounds changing velocity, or acceleration. Though acceleration has been shown to be a challenging concept for children to understand (Tasar, 2010; Trowbridge & McDermott, 1981), we believe that it is a powerful idea that provides an entry point to many other difficult and useful concepts (Papert, 1980). FTR also includes important construction tools that fundamentally change the way the player causes motion. These designs, inspired by constructionist methods (Papert, 1980; Papert & Harel, 1991) as well as work examining expert and novice embodied interactions with formal graphic representations (diSessa, Hammer, Sherin, & Kolpakowski, 1991; Nemirovsky, Tierney, & Wright, 1998; Roschelle, Kaput, & Stroup, 2000; Sherin, 2000; Wilensky & Reisman, 2006) are intimately connected to previously discussed controls and visual cues but are not explicitly introduced until the third phase of the game. In this phase, rather than drive the car directly, the player constructs personal notions of motion by interacting with the representations of motion rather than the car itself. Specifically this occurs in one of two ways, either by painting the track using the palette presented by the color-trails, or by constructing a velocity versus time graph. In the painting mode of the construction phase the player uses colors corresponding to the previously seen color-trails to paint various parts of the track. Once the player is done painting the entire track, the car begins the race, changing velocity based on the color being driven over. The player can paint the track in any way she prefers, however, because each color corresponds to a particular velocity, and the cars ability to turn is impacted by its current velocity, the choices made in painting the track determine whether or not the car will successfully complete the race. The player may also interact with the car by constructing a velocity versus position graph. In the graphing mode, the player constructs a velocity versus position graph using the motion sensitive controller. To construct the graph, the player accelerates points added to the graph by rolling the controller forward and backward. Once the graph is constructed, the car downloads the data and drives around the track according to the player-constructed graph. In this way the player directly connects the intuitive feeling of acceleration to formal graphic representations and can also explore how varying graphic features, such as sharp drops or plateaus in velocity correspond to particular track features.
While the design of FTR draws upon many important cognitive theories including embodied cognition (Lakoff & Nez, 2000; Wilson, 2002) and knowledge-in-pieces (diSessa, 1993; Hammer, 1996; Sherin, 2006), in this paper we draw on two key lines of research to understand how FTR's design impacts player knowledge: restructurations (Wilensky & Papert, 2010) and epistemological framing (Hammer, Eleby, Scherr, & Redish, 2005). Representations in the world are often created with the intent to store, embody, or alter knowledge and processes. In this way, external representations become in a very real sense part of our thinking, remembering, and communicating (diSessa, 2000 p. 6). The term restructuration was proposed by Wilensky and Papert (2006, 2010) to describe the dramatic cognitive shift that can occur due to changes in representational infrastructure. In the same way the processes of multiplication and divisionskills only practiced by the priestly fewwere democratized after the shift from Roman to Hindu-Arabic numerals, so too do we believe advanced topics such as electricity (Sengupta & Wilensky, 2009), differential equations (Wilkerson-Jerde & Wilensky, 2010), and kinematics (Holbert & Wilensky, 2011a, 2011b) can be taught to younger children. While we have designed FTR to include important representations for kinematic restructuration, how a player interprets and interacts with these representations in a video game is not always straightforward. In particular, getting players to connect experiences in video games to formal concepts and representations has proved challenging. We believe player epistemological framing might explain much of this difficulty (Hammer et al., 2005). In education research the term epistemology is often used to describe an individuals beliefs
ISLS
120
ICLS2012
about the form and type of knowledge in a domain or context. Drawing on the theory of knowledge-in-pieces (diSessa, 1993), which suggests that cognition is emergent from the activation of a large number of fine-grained, highly distributed, context-sensitive primitive elements, Hammer and colleagues (2005) suggests that individual epistemologies change in-the-moment as various cognitive resources are activated. They go on to propose the idea of epistemological framing and define it as, Phenomenologically, a set of expectations an individual has about the situation in which she finds herself that affect what she notices and how she thinks to act (Hammer et al., 2005 p. 98). In this model, the ability to recall or apply knowledge in a given context then is affected by both the epistemological framing of the learning situation as well as the framing of the recall situation. Designers interested in creating educationally relevant video games must wrestle with the tradeoff of making the learning goals explicit, versus foregrounding the game narrative and fantasy (Collins, 1995). An overemphasis on learning goals often results in an experience that feels more like interactive flashcards than a game school work dressed up with animation and sound (Ito, 2007). Here the content remains clearly schoollike and as such, likely activates a knowledge as propagated stuff epistemology (Hammer & Eleby, 2002). In contrast, by focusing solely on a game narrative, the embedded content that the researcher hopes to share with the player, may become clouded. Here the player sees the game as just that, a game, and not something that is relevant in a traditional school setting (Ito, 2007). The challenge then is trying to find a way to keep players firmly embedded in the game narrative, while making the embedded content flexible enough to be utilized in non-game contexts. To achieve this delicate balance, we have created FTR to be what Clark et al. (2011) call a conceptually integrated game (Habgood and Ainsworth, 2010, discuss a similar idea they refer to as intrinsic integration). Such a design ensures the learning outcomes match the game mechanics and are tightly integrated into the game narrative. Furthermore, FTR is representationally integrated, in that domain relevant representations become highly tied to the player interface and game-mechanics. As we will show in the discussion section, we believe this encourages players to connect formal representations to informal epistemological framings (Hammer et al., 2005) leading to a kinematic restructuration.
Method
To explore the impact of specific design features incorporated into FormulaT Racing we have conducted a number of studies of children playing the prototype game. These studies are conducted exclusively in informal settings to remain consistent with the typical video game playing contexts and employ multiple methodologies. We have written elsewhere about how increasingly complex player strategies in the painting phase match common programming strategies (Holbert & Wilensky, 2011a) and how FTR players learn to construct velocity versus time graphs (2011b, 2011c). For this paper we discuss semi-clinical interviews conducted with participants before and after they play FTR to identify changes in player knowledge about ideas and representations of motion. In these interviews, participants were asked a series of questions focused on real-world examples of acceleration and velocity using toy cars, adjustable speedometers, and graphing paper. Using methods described by Russ, Lee, & Sherin (in press) we analyzed video data and transcripts of the interviews looking for moments where participant behavior and language indicates a shift in how he frames the interview activity. While we currently have data on 12 participants, in this paper we focus on one participant, Brian, who serves as an exemplar of epistemological integration.
Results
In this paper, the focus of our analysis is on instances of pre- and post-game epistemological shifts. In particular we present a case study of one participant, Brian, shifting from an epistemological stance where game knowledge is separate from formal mathematic and physics knowledge, to a stance that seamlessly integrates the two. While we noticed other interesting shifts, such as players that shifted from a game to a real world focused epistemology, we believe the integrated epistemology discussed here highlights the importance of specific design features and offers important lessons for future iterations of FormulaT Racing and other such games. Brian is a very confident 14 year old boy from a low income neighborhood in a large Midwestern city who prides himself in his ability to not only beat nearly every game he plays, but also to create the hardest games. Most of Brians game building experience has involved personal constructions hes made in Scratch at a computer clubhouse he attends near his home. During interviews with the first author, Brian asked many offthe-script questions about the programming environment used to create FTR as well as how he could hack the Nintendo Wiimote to work with his own Scratch games. There was no doubt that Brian considers himself an expert gamer.
Pre-game interview: Non-fluid epistemological shifting

This sense of himself as an expert gamer impacted how Brian perceived the activity and relevant knowledge for the pre-game interview. Brian approached nearly every question in the pre-game interview from
ISLS 121
ICLS2012
an epistemological frame that privileged video game knowledge that is, Brian answers all questions about motion as if they are about how motion occurs in the context of a video game despite the fact that the representations and questions asked in the interview are agnostic about context. When asked to describe phases of acceleration, or situations involving constant velocity, Brian assumed the appropriate epistemological activity was one of story telling. In particular, Brian answered as if he was describing action occurring in a racing video game. In the exchange below Brian was asked, Let's imagine this car is at a red light. I want you to sort of show me, by moving the car and also describing with words, what happens when the light turns green. Before speaking, Brian pushes the toy car used in the interview quickly so that it speeds across the table into the interviewer's arm. When asked to describe the motion he states: Well I tried, I just kept moving as fast as I could to make sure no one else is coming after me. Maybe I may have used a turbo boost, like a mushroom. And then I bumped into the obstacle, which was your arm. And then I got hurt, and then the other racers passed me. Brians description of motion here directly invokes experiences with the commercial racing video games Mario Kart (indicated by his reference to a mushroom turbo boost). Its important to note again that the interview questions do not reference video games, and the toy cars used by the participants to answer questions more closely resemble real cars than the cartoon-like ones of games like Mario Kart. Despite this, Brian sees the activity of answering the questions of the interviewer to be about describing events that might occur in the video game world what we might call a game story framing. In this framing, relevant knowledge comes from video game experiences and is about recalling or creating action-filled stories of game play. It is not surprising that participants might associate the interviewer with video games. While participants have not yet played our racing game at this point in the study, they have been told that we have designed a game that we want them to test. Because we are interested in studying how kids interact with games in informal environments, associating the interviewer with video games is preferred to an association with the classroom. In addition, such a focus on video games in the pre-game interview, like we saw with Brian, increases our confidence in the external validity of the experience. If we want to explore the possibility of designs in real video games impacting the cognition of players, we want to be sure the games are being tested in real gaming settings by individuals that consider themselves gamers. Furthermore, as we will show later in this section, the video game knowledge that Brian draws from throughout the interview seems to be incompatible with formal mathematics knowledge. Moments where Brian is forced to shift into a different epistemological frame are particularly enlightening for theorizing about game design. The most apparent epistemological shift occurs when the researcher introduces the graphing task. In this task participants are given a sheet of graphing paper with the x-axis labeled as time and the y-axis labeled as velocity. The interviewer uses a paper speedometer with a movable needle to indicate different actions of a hypothetical car, and the participant is asked to graph what you think is happening. Brian, who has talked freely and playfully during the interviewer while enacting a game story framing, becomes uncomfortable and stumbles a bit. Brian pauses, and for the next ten seconds mumbles, Uh, wait. Im trying to think which way to go. Lets see, the faster...the faster it moves, which means timealright, ok, Im ready. In this moment Brians physical demeanor changes, as does his playful attitude. Both his verbal wait and visible behavior shift indicates Brian is likely experiencing a frame mismatch between his previous game story frame and a school knowledge frame he interprets the question to be about. Brian sees this school knowledge frame as being about providing a formal representation that involves certain rules and vocabulary. This mismatch means abandoning the previous game-story frame in favor of the alternate school knowledge frame. As he discusses his graph with the researcher, Brian doesnt invoke a fictional car as it might act in a video game as he has done throughout the interview when confronted with other non-game representations (such as the speedometer). Instead, Brian describes the action of plotted points. He states: Brian: Let's see...so, since the y is the speed velocity, like miles per hour, uh...cars, the car increased as the plot, as the plotted points are moving more up. As long...the steeper the slope is the speed of the car. Interviewer: The, the steeper the slope is the speed? Brian: Yep. Interviewer: Okay. Brian: The speed rate. At this point of the interview Brian struggles to think of the correct word and the result is statements that seem initiated to use science vocabulary like slope as well as mixed terms such as speed velocity and speed rate. This remarkable shift continues for a few moments until he once again relaxes, and then returns to describing his graph as how it might be formed by a car in a video game sharp phases of acceleration are described as getting a mushroom or hitting a turbo boost and sharp phases of negative acceleration are described as bumping into something. Of interest in this analysis is how his earlier game story
ISLS
122
ICLS2012
epistemological framing of the activity shifts to a very different school knowledge framing. This school knowledge framing assumes knowledge comes from expert sources (books, teachers, etc), requires specialized representations and vocabulary, and is about recitation, rather than knowledge creation. While in this frame Brian does not see his expert gaming knowledge as relevant.
Post-game interview: An integrated epistemological framing

While there is still some residue of his earlier framing in the post-game interview, Brian seems to have engaged a highly flexible epistemological frame that privileges both game and formal knowledge. While segments of the post-game interview show Brian continues to answer some motion questions using video game stories, such as describing phases of acceleration as boosts and occasionally giving the interviewer an instant replay of his movements with the toy car, when describing the motion of two different toy cars being pushed by the interviewer across the table (one is being pushed with constant velocity, the other constant acceleration), he casually states Well the other one was just this constant rate, but what you did just now, be where...where the slope would increase. Brians use of the term slope, invoking a graph representation, is particularly interesting as no graphs have been used yet at this point in the post-game interview. When pushed by the interviewer to explain what he means by slope, Brian states, well thats what I learned in algebra. Later, when describing his post-game graph, Brian frequently uses terms like constant rate, intermingling this formal vocabulary with more informal game-like terms like boost. Of note is that Brians use of this formal vocabulary is not marked by visible shifts in behavior or language patterns, as was the case in the pre-game interview. In the post-game interview, Brian freely uses terms like slope, constant rate, and turbo boost apparently without shifting between epistemological frames. In the pre-game interview, Brian almost exclusively utilized a game story frame when answering motion questions using the toy cars. It was only when given the graphing task that Brian attempted to use formal school-like knowledge after a visible shift from his previous epistemological framing. While further investigation shows Brians explanation of slope is somewhat faulty, the fact that Brian spontaneously and smoothly integrates this formal knowledge from his algebra class into his discussion of the motion of toy cars indicates he is engaged in a very different epistemological framing, one that draws on both game and formal knowledge and is more about knowledge creation, rather than knowledge regurgitation.
Discussion
In this section, we propose a possible mechanism provided by the design of FTR that might lead to the integrated epistemology enacted by Brian in the post-game interview. We also discuss the implication of this case study for future iterations of FTR and other games intending to take advantage of informal game play.
Figure 1. Some representations are better at foregrounding kinematic concepts than others. Before playing the game, video game and formal representations cue different epistemological framings. In the pre-game interview, Brians answers to questions and the language used to describe various representations suggest he is framing the epistemological activity as being about drawing on video game experiences. This is particularly true when interview questions utilize a speedometer or toy car and track representations. While it is possible that these representations prompt Brian to engage this gaming epistemological framing, we think it is more likely that the association of the interviewer with games prompted this framing, and since these representations are common in typical commercial racing games, they did not prompt an epistemological shift. Utilizing the toy car and track and speedometer representations, Brian was able
ISLS
123
ICLS2012
to reason about velocity and acceleration with little trouble, though answers to questions typically focused on one concept at a time, with little discussion of how these concepts intertwine (Figure 1). In contrast, the formal representation of a velocity versus time graph is particularly suited to the integration of velocity and acceleration. While Brian is familiar with this representation, in the interview, the simple introduction of graphing papers causes Brian to shift into a new epistemological framing. Now Brian sees the task as being about formal school-like knowledge. While in this framing Brian ceases to draw on video game knowledge, something he has done throughout the interview, and instead focuses on answering questions using sanctioned science vocabulary. While his explanation is not entirely correct, we see evidence of Brian attempting to use the graphing representation to reason about both speed and acceleration as he not only discusses the height of the plotted points but also the steepness of the slope. During the post-game interview Brian engages a new epistemological framing that allows him to freely use both formal and gaming knowledge. Where did this epistemological framing come from? We propose that specific designs of FormulaT Racing particularly the intuitive and embodied control interface and the programming phases of the game encourage players to tie representations that highlight individual concepts of velocity and acceleration to powerful formal representations that integrate the two (Figure 2). Once these formal and informal representations are effectively linked, the player no longer sees the interview task as privileging either game or formal knowledge and instead sees the task as being about constructing new knowledge by drawing on game, real-world, and formal experiences.
Figure 2. Programming by painting and graphing in the construction phase, and the use of the motion sensitive controller in the racing and construction phases effectively ties the various representations in FTR together. This results in a new epistemological framing for Brian that privileges knowledge construction. As discussed in the game design section of this paper, FTR includes a number of important representations meant to foreground various kinematic concepts. Of the new representations, most important to this discussion is the color-trail representation. This representation is particularly good at highlighting the player vehicles current velocity. Moreover, this representation allows players to visually see how the vehicle velocity relates to various track features. The track itself is a representation that is good at highlighting acceleration. The animation of traditional racing video games, the controlling scheme used, as well as real-world experiences shared by American youth provide the player with enough knowledge to know he should slow down around turns and speed up when the track is straight. Finally, a formal graph, which is included instead of a more traditional speedometer, is very effective at foregrounding both acceleration and velocity. However, like the speedometer in traditional racing video games, players rarely consider this representation during a race. While the new representations in FTR are important, the construction phase of the game effectively ties these representations together in a new way, encouraging players to integrate game and formal representations. This happens first by pairing the painting and graphing constructions within the same game level. While the interactions in the painting and graphing constructions are somewhat different, because they share a common narrative and goal in the game (that of directing other cars as the pit boss) the representations become highly intertwined. Finally, the two phases share the activity of programming in some form. Whether painting or graphing we see evidence of players systematically enacting computational strategies (Holbert & Wilensky, 2010b). Video evidence suggests players rarely reference the velocity versus time graph during the racing phases, however, the graphing representation utilized in the construction phase becomes tied to specific track features through the accelerometer-based interface. When players drive the car in the racing phases, they do this using a motion-sensitive controller. This interface foregrounds acceleration as it allows players to feel moments of speeding up and slowing down when they approach straightaways and turns. Weve adopted this same
ISLS 124
ICLS2012
interface for the graph construction phase. Here, as weve suggested before, players construct a graph that the car will use to drive around the track. Players construct these graphs by accelerating points up and down the y-axis in the same way they accelerated their car around the trackthough here, instead of dealing with acceleration in the moment, players consider acceleration over the course of an entire race. By tying these two representations, the track features and velocity versus time graph, though embodied controller use, we believe players more easily see the graphs as being highly relevant to specific race action. As the graph, track, and color-trail representations begin to be tied together by programming in the construction phases and the embodied controller use throughout the game, Brian more easily integrates previously incongruent epistemological frames. While we did see epistemological shifts with many participants, Brians merger of game and formal epistemologies was unique. We believe this might be due to Brians very salient identity as an expert gamer. Furthermore, as our oldest participant (14), Brian likely has more formal experience with the mathematics embedded in FTR. His ability to integrate previously learned knowledge from the classroom with his FTR video game experience gives us hope that younger players might utilize FTR experiences in the classroom. This case study highlights the importance of both 1) designing new representations that provide infrastructure for complex cognitive tasks and 2) providing mechanisms for connecting representations to alternate contexts and epistemological frames. The very best representation will only be effective if it is seen as relevant in various contexts. As we move forward, we should continue to strive to make connections to more casual gamers as well as those that consider themselves experts. This may mean making certain relevant game features less nuanced. For example, the forced braking is only noticed by expert gamers and as such, may be missed entirely by novice gamers. In addition, some features that are large breaks from gaming tradition, such as the painting and graphing phases, may remain unconnected to other commercial racing video games (or even the more traditional phases in FTR) for novice and expert gamers. In this case more care should be taken to gradually introduce these important phases so they can continue to be seen as connected to the rest of the racing genre and typical gaming culture.
Conclusion
Video games permeate the lives of young people in America and continue to be highly relevant in youth culture. Video games are a rich space in which children can experiment with scientific phenomena. Whether we design games for the classroom or beanbag chair, it will be important for designers and researchers to consider players personal epistemologies both in and out of the gaming context. If neglected, we run the risk of designing isolated experiences that only matter in the moment, rather than providing a rich space for integrating the virtual with the real and the informal with the formal.
References
Barab, S., Thomas, M., Dodge, T., Carteaux, R., & Tuzun, H. (2005). Making learning fun: Quest Atlantis, a game without guns. Educational Technology Research and Development, 53(1), 86-107. Clark, D. B., Nelson, B., Chang, H., Martinez-Garza, M., Slack, K., & D'Angelo, C. M. (2011). Exploring Newtonian Mechanics in a conceptually-integrated digital game: Comparisons of learning and affective outcomes for students in Taiwan and the United States. Computers and Education, 57, 2178-2195. Clark, D. B., Nelson, B., D'Angelo, C. M., Slack, K., & Menekse, M. (2010). SURGE: Assessing students' intuitive and formalized understandings about kinematics and Newtonian mechanics through immersive game play. Paper presented at the Annual American Educational Research Association 2010 meeting, Denver, CO. Collins, A. (1995). Design issues for learning environments. In S. Vosniadou, E. d. Corte & H. Mandle (Eds.), International perspectives on the psychological foundations of technology-based learning environments (pp. 347-361). Hillsdale, NJ: Lawrence Erlebaum. diSessa, A. A. (1993). Toward an epistemology of physics. Cognition and instruction, 10(2 & 3), 105-225. diSessa, A. A. (2000). Changing Minds: Computers, Learning, and Literacy: The MIT Press. diSessa, A. A., Hammer, D., Sherin, B., & Kolpakowski, T. (1991). Inventing graphing; Meta-representational expertise in children. Journal of Mathematical Behavior, 10(2), 117-160. Habgood, J. M. P., & Ainsworth, S. E. (2011). Motivating children to learn effectively: Exploring the value of intrinsic integration in educational games. Journal of the Learning Sciences, 20(2), 169-206. Hammer, D. (1996). Misconceptions or p-prims: How may alternative perspectives of cognitive structure influence instructional perceptions and intentions? Journal of the Learning Sciences, 5(2), 97-127. Hammer, D., & Eleby, A. (2002). On the form of personal epistemology. In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs and knowledge and knowing (pp. 169-190). Mahwah, NJ: Erlbaum.
ISLS
125
ICLS2012
Hammer, D., Eleby, A., Scherr, R. E., & Redish, E. F. (2005). Resources, framing, and transfer. In J. Mestre (Ed.), Transfer of learning from a modern multidisciplinary perspective. Greenwich, CT: Information Age Publishing. Holbert, N. (2009). Learning Newton while crashing cars. Games, Learning, and Society 2009: Madison, WI. Holbert, N., & Wilensky, U. (2010a). FormulaT Racing. Evanston, IL: Center for Connected Learning and Computer-based Modeling. Holbert, N., & Wilensky, U. (2010b). FormulaT Racing: Brining kinematics to the bean bag chair. Paper presented at the Games, Learning, and Society 6.0, Madison, WI. Holbert, N., & Wilensky, U. (2011a). FormulaT Racing: Designing a game for kinematic exploration and computational thinking. Proceedings of the 7th Annual Games, Learning, and Society Conference, Madison, WI. Holbert, N., & Wilensky, U. (2011b). Putting the turtle on the racetrack: Investigating a constructionist racing game for exploring kinematics. Paper presented at the NARST, Orlando, FL. Holbert, N., & Wilensky, U. (2011c). Racing games for exploring kinematics: A computational thinking approach. Paper presented at the American Educational Research Association 2011 meeting, New Orleans, Louisiana. Ito, M. (2007). Education vs. Entertainment: A Cultural History of Children's Software. The John D. and Catherine T. MacArthur Foundation Series on Digital Media and Learning, 89-116. Lakoff, G., & Nez, R. E. (2000). Where mathematics comes from: How the embodied mind brings mathematics into being. New York: Basic Books. Lenhart, A., Kahne, J., Middaugh, E., Macgill, A. R., Evans, C., & Vitak, J. (2008). Teens, Video Games, and Civics: PEW Internet & American Life Project. Nemirovsky, R., Tierney, C., & Wright, T. (1998). Body motion and graphing. Cognition and Instruction, 16(2), 199-172. Papert, S. (1980). Mindstorms. New York: Basic Books. Papert, S., & Harel, I. (1991). Situating constructionism. In S. Papert & I. Harel (Eds.), Constructionism. New York: Ablex Publishing. Rideout, V. J., Foehr, U. G., & Roberts, D. F. (2010). Generation M2: Media in the lives of 8- to 18-year olds: The Henry J. Kaiser Family Foundation. Roschelle, J., Kaput, J. J., & Stroup, W. (2000). SimCalc: Accelerating students' engagement with the mathematics of change. In M. J. Jacobson & R. B. Kozma (Eds.), Innovations in science and mathematics education: Advanced designs for technologies of learning (pp. 47-76). Hillsdale, NJ: Earlbaum. Sengupta, P., & Wilensky, U. (2009). Learning electricity with NIELS: Thinking with electrons and thinking in levels. International Journal of Computers for Mathematical Learning, 14(1), 21-50. Sherin, B. (2000). How students invent representations of motion: A genetic account. Journal of Mathematical Behavior, 19(4), 399-441. Sherin, B. (2006). Common sense clarified: The role of intuitive knowledge in physics problem solving. Journal of Research in Science Teaching, 43(6), 535-555. Squire, K., Barnett, M., Grant, J. M., & Higginbotham, T. (2004). Electromagnetism supercharged!: Learning physics with digital simulation games. Proceedings of the 6th International conference on Learning Sciences, Santa Monica, California. Tasar, M. H. (2010). What part of the concept of acceleration is difficult to understand: the mathematics, the physics, or both? ZDM Mathematics Education, 42, 469-482. Trowbridge, D. E., & McDermott, L. C. (1981). Investigation of student understanding of the concept of acceleration in one dimension. American Journal of Physics, 49(3), 242-253. Wilensky, U. (2001). Embodied learning: Students enacting complex dynamic phenomena with the HubNet architecture. Paper presented at the annual meeting of the American Educational Research Association, Seattle, Washington. Wilensky, U., & Papert, S. (2006). Restructurations: Reformulations of knowledge disciplines through a change in representational forms. [Unpublished working paper]. Center for Connected Learning and Computer-Based Modeling. Northwestern University. Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theories--and embodied modeling approach. Cognition and Instruction, 24(2), 171-209. Wilensky, U., & Papert, S. (2010). Restructurations: Reformulating knowledge disciplines through new representational forms. Proceedings of Constructionism 2010, Paris, France. Wilkerson-Jerde, M., & Wilensky, U. (2010). Restructuring change, interpreting changes: The deltatick modeling and analysis toolkit. Proceedings of Constructionism 2010, Paris, France. Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9(4), 625.
ISLS
126
ICLS2012
Dilemmas of Promoting Expansive Educational Transformation through Information and Communication Technologies (ICTs) in Botswana
Batane, Tsepo, University of Botswana, Private Bag 0022 Gaborone, Botswana, email: batane@mopipi.ub.bw Engestr Ritva, Institute of Behavioural Sciences, P.O Box 9, 00014 University of Helsinki, Finland, m, email: ritva.engestrom@helsinki.fi Hakkarainen, Kai, Department of Education, Assistentinkatu 5, 20014 University of Turku, Finland, email: kai.hakkarainen@helsinki.fi Newnham, Denise Shelley, Route de Chailly 14 1815, Clarens, Switzerland, email: deniseshelleynewnham1@gmail.com Nleya, Paul, University of Bostwana, Private Bag 0022 Gaborone, Botswana, email: NLEYAPT@mopipi.ub.bw. Virkkunen, Jaakko, Institute of Behavioural Sciences, P.O Box 9, 00014 University of Helsinki, Finland, email: jaakko.virkkunen@helsinki.fi Abstract: The purpose of the present paper is to address challenges and obstacles that educators and researchers face while promoting expansive educational transformation through technology-mediated learning in developing countries. The paper reports challenges and possibilities encountered in Botswana Expansive School transformation (BeST) project by describing preliminary results of creating cross-functional networks, organizing ICT workshops and Change-Laboratory interventions in collaboration between European and African researchers.
Introduction
The Revised National Policy on Education in Botswana is designed to prepare the country for the transition from an agro-based economy to an industrial economy driven by information technology that the country aspires to. In light of the rapid technological changes that are taking place both within the country and in the rest of the world, the policy has called for the introduction of computer use in schools. The purpose of the present investigation is to use methods of the Cultural-Historical Activity Theory (CHAT), such as Change-Laboratory (CL) interventions, to promote educational transformations in the context of ICT-mediated learning in Botswana. Rather than seeing computers as mere technological tools, CHAT guides one to examine technology-mediated learning as an integral part of the contradiction-laden historical development of educational activity calling for profound transformation of social practices prevailing at school and well as renegotiation of relations between schools and the surrounding community. Accordingly, there appears to be an complex interplay between technological and social innovations (Perez, 2002). When a revolutionary combination of technological innovations emerges, it is usually followed by intensive capital investment in technology implementation and creation of the required novel infrastructure. This phase is characterized by a contradictory combination of overoptimistic technology hype and a propensity to apply new technologies within the existing institutional practices. After the economic bubble based on such illusory hopes explodes, a qualitatively different phase emerges that is characterized by broad application of the new technology across society as well as transformation of prevailing institutional practices according to the possibilities of the new technological infrastructure. Venkatraman (1991) argued that the gains achieved when ICTs are used to support local needs for internal integration of activities in business organizations are relatively meager compared to their use for business-processes engineering, for transforming enterprise networks, and for redefining the scope of the business activity. A corresponding potential of facilitating revolutionary development of educational activities has emerged. Initially, ICT were used to elicit traditional learning tasks. Gradually integration of compartmentalized subject domains has become possible in project-based learning activities of students and teachers. A more revolutionary development would involve re-conceptualizing learning processes in terms of technology-mediated investigative learning and knowledge-building projects, transforming and creating a new collaborative network, and redefining the scope and object of the schools educational activity. The first wave of implementing and using this technology in education in developed countries in the 80s and 90s was characterized by the computerization of traditional forms of teaching and school activities. Movement toward the later, transformative phase of educational use of ICTs has been gradually taking place in developed countries since the turn of the millennium. However, an on-going phase involving the re-definition and recreation of educational functions of the school with the help of ICTs is predominantly lacking in developing countries. The introduction of ICTs has somewhat remained at the level of eliciting traditional learning tasks and teaching styles. Such a historical point of stagnation can only be overcome through the introduction of new ways of perceiving the school activity. Rather than taking the school and its functioning as a given object of
ISLS 127
ICLS2012
development, the challenge is to radically re-conceptualize what school and learning are all about, redefine the boundaries of school and community, and explore expansive possibilities in practice. Prevailing social practices in schools channel teachers and students activities in a way that is very hard to change without extended deliberate efforts and support from outside intervenors. In order to elicit expansive educational transformation, it appears necessary to involve a heterogeneous network of actors with expanded mandates and perspectives of development. Both vertical dialogue between different levels of authority and horizontal dialogue between various actors involved in transformation are needed. Rather than directly transferring models of using ICTs, it might be advantageous for developing countries to utilize the historical experiences of developed countries as resources for reflecting critically on their own current practices and trajectories of development of school. In order to avoid typical pitfalls of interventions carried out between developed and developing countries sophisticated interventional methodology is needed that allows the participants to make visible, reflect on and deliberately master various aspects of their collective school transformation efforts (Long, 2001), Change Laboratory (CL), based on and the methodology of Developmental Work Research (DRW), Engestr 1987), is an already well-defined intervention method to support m, organizational transformations (Engestr et al., 1996). In CL interventions, workplace communities reflect m upon their prevailing practices on the basis of data; they analyze the historical development of their current activity system and its inner contradictions, as well as envision and implement, through practical experimentation with a new model of the activity. These collaborative reflections are triggered and guided by the researchers, who collect and prepare ethnographic mirror data for assisting participants to reflect on their work. CL comprises a series of workgroup meetings to reflect on current practices and envision future activities. Essential aspects of CLs are; a) using videotaped practices as a mirror for assessing current activity; b) generating ideas and tools (e.g., charts) that help to assess past, present and future activity; and c) modeling present practices by using an activity-system analysis. Sharing participants memories of the past assist the workplace community in analyzing the historical development of its activities. Within a CL process, a facilitator in the initial organizes, provokes and guides participants learning activity through discourse, and expansive learning actions aimed at questioning, analyzing, modeling and transforming the activitys system. Learning activity is a process, in which the participants ascend from the perspective of their individual work to the joint analysis and development of the system of their actual productive activity in order to radically transform the former into a new and prolific future form which has surpassed its initial contradictions. As a result a new form of productive activity evolves in which the participants increasingly cohere toward a shared object of transformation which is less determined by pre-planned tasks set by the facilitator but rather increasingly by the evolving object. For example, a dialogue between people who examine the school and childrens everyday life from varying perspectives is complemented with a dialogue focused on analyzing the historical development of prevailing practices of schooling, examining educational practices in detail, and deliberately experimenting with various new solutions. The participants are themselves engaged in analyzing disturbances that hinder pupils learning and the development of the school community in its extended social and cultural context. They are provided with conceptual tools for analyzing the data and ways of modeling the dynamic development of the activity system are suggested.
Figure 1. Elements of the setting of the Change Laboratory (Engestr et al., 1996). m The introduction of ICTs into school activities opens up new possibilities for teachers educational work and teaching as well as childrens learning. In Figure 2, four ideal typical conceptions of school have been depicted by crossing two dimensions of expansion of the traditional school discussed in the literature (Engestr et al., 2002). One of these depicts the type of problems school children are working on and m identifies the developmental trend from learning by solving given tasks for which the right answer is known to complex real-life problems that can have varying creative solutions. The other dimension describes a schools relationship to the surrounding community and goes from an isolated school to an open school, where teaching
ISLS 128
ICLS2012
and education is realized in collaboration with the surrounding community. Taking the school texts as objects that have to be remembered in their own right is strengthened in a relatively isolated school that has little contact with the surrounding community. Opening the school up to the community and taking the vital problems of the community, and more broadly of mankind as objects to be solved with the help of culturally developed intellectual tools, can strengthen the pupils agency in mastering their lives as members of a community. This model introduces the concept of ICTs as a tool for expanding the school ethos. For historical reasons, the western model of school brought to developing countries carries many of the characteristics of the old-time isolated school (Sector 1) in which teachers work has been defined by subject-by-subject curriculum packages, stage-by-stage teaching models, standardized tests, and corresponding ICTs. The present trend of introducing test-based control systems in schools have strengthened the tendency to highlight memorizing isolated pieces of knowledge and skill on the cost of understanding and creative use knowledge and education (Ravitch, 2010). The increasing emphasis on test scores has broadened the gulf between school knowledge and students creative interaction with and learning from real life problems and utilizing the new possibilities of digital networks of open innovation (Tapscott & Williams, 2006) and collaborative building of knowledge (Bereiter, 2002). Sector 2 represents various reform-pedagogical ideas of a school that follows ideas of technology-enhanced projectand problem-based learning. Sector 3 represents approaches that highlight the importance of creating students and teachers classroom learning communities focused on collaborative building of knowledge with the help of the new ICTs (Scardamalia & Bereiter, 2006); there is a tendency to examine the school community as relatively isolated from the surrounding society. Sector 4 combines the two dimensions of expansion for examining the collective zone of proximal development of school. In such a school, learning is to a great extent carried out by solving real-world problems in contexts of various hybrid forms of activities and networks of learning; these are processes, also, in which representatives of local productive work are involved in and in which the rules are different from those in the classroom (Hakkarainen et al. 2004; Yamazumi, 2006; Roth & Lee, 2007). Various ICTs from word-processing and spreadsheets to collaborative web publishing provide support for such activities. There is, however currently an aggravating contradiction between these new possibilities and the tendency to strengthen the traditional school through increasing testing.
Figure 2. Two dimensions of expansive school transformation.
Research Aims
The purpose of this paper is to describe experiences from the Botswana Expansive School Transformation (BeST) project by addressing the following questions: 1) How is expansive school transformation facilitated through Change-Laboratory (CL) interventions; 2) What kinds of actors and agencies are needed for facilitating ICT-mediated educational interventions in Botswana; and 3) What kinds of challenges and constraints emerge?
Methods, Data Collection, Data Analysis

In order to elicit sustained education transformation of developing countries, based on their needs and developmental potentials, one has to employ sophisticated intervention methodology beyond standard academic research methods. Toward that end, the present developmental intervention relies on multiple methods and associated data sources. Two five-day workshops of 10 pilot schools (2005; 2008) and several smaller meetings have been documented by audio and video recording. Beyond within-school activities, ethnographic data were collected, by interviewing parents and community members and observing local activities, such as Gotla (municipal) gatherings, animal husbandry and vegetable gardening, child care as well as health and social services. These observations gave an insight into how the school activity and the existing way of life of the community related one to the other. Such understanding is paramount, acting as a springboard to unearthing the contradictions existing within the activity. Multiple voices have been recorded by interviewing principals, teachers, students, parents, and other relevant peripheral or distant actors. Change-Laboratory, that allows local communities to deliberately make visible, reflect on, and transform their practices with the help of researchers, has had a central role in the present developmental intervention. Relevant parts of the ethnographic material were used as mirror material in CL interventions. The material collected provides a rich body of data concerning the prevailing practices of learning and instruction at the intervention schools and school-community relations. Qualitative content analysis has been used to assess ethnographic data, such as interviews and videotaped participant observations as well as discourse taking place in Change-Laboratory sessions.
ISLS 129
ICLS2012
Preliminary Results of the Project

In the following, preliminary results of the present project will be summarized with a focus on the creation of new developmental collaboration. In order to achieve this end, the structure of the project and the first two CL interventions and data illustrating discussions between collective actors shall be presented and discussed.
Actors and Structure of the Project

The present project relies on collaboration between the universities of Botswana and Helsinki for supporting the educational use of ICTs in schools in Botswana and the educational administrators from Botswana. In the initial stages of the project, The University of Mauritus had been part of the project at its initial stage and some staff members attended the first workshops held but they later dropped out. As a result of negotiations and workshops between organizations involved a new unofficial boundary-crossing network of actors was created. The University of Helsinki held a one month Activity Theory Workshop for AS-TIG during 2008. Ten schools, representing five different areas of the country and different types of schools were selected as pilots in the educational use of ICTs and in each school a group of interested teachers, that represented the activity vertically and horizontally, were selected to be responsible for the measures of developing ICT use in the school. The workshops arranged for the ICT-groups of the ten schools provided opportunities for hands on learning of ICTs as well as planning and modeling transformation with the help of the activity system model. In order to have a bigger impact on education, it was essential to involve all ten schools rather than only three CL schools as well as a network of mandated change stakeholders: There are drivers, the people who are mandated with providing infrastructure or guidelines so I suppose if they understand our needs and they can re-align our needs towards the vision , they will ensure that whatever we are thinking of, they will provide a platform for us to be going in that line" (a ministry official). Teachers from the ten pilot schools (two teachers from each) and varying groups of local school officials, their supervisors, representatives of the universities of Botswana and Helsinki, Ministry of Education as well as other organizations took part in various workshops. In these workshops, the basics of activity theory was explained, its application in school development as well as the Change Laboratory method. Later on, an unofficial Activity Theory Special Interest Group (AT-SIG) that involved a heterogeneous group of researchers and educational administrators to discuss how ICT can be used to develop schools, was formed. The idea was that the group would function as a cross-functional and cross-organizational forum for developing a pedagogically grounded concept for ICT use in schools (Figure 3).
Figure 3. Cross-functional network for developing methods of using ICTs at schools The research team is currently analyzing results concerning ICT development workshops organized during the project; the workshops were video recorded and there are tens of hours of video data. The local participants contributions to joint discussion as well as their narrative descriptions of situations at school have been transcribed. Researcher Lauri Hietaj is analyzing those contributions in which the participants addressed rvi challenges of implementing ICTs in the school activity. Preliminary results of the study indicate that at the discourse level, there was some improvement in the integration of ICTs in the pilot schools from 2005 to 2008. Discussions that focused on arising awareness of ICTs transformed toward discourse addressing possibilities of transforming educational practices.
Intervention at School A
The first CL-intervention took place at a senior secondary-level school A (with a population of 1800 students and about 150 teachers) in fall 2008. The school is situated in a village close to a major city. Ethnographic data were collected in school A across ten days by interviewing teachers, principals, community institutions, students, ICTs community centers, and parents in order to gain insight into the activity and to prepare models and mirror data for the CL. The audio and video recorded data were transcribed and field notes analyzed. The school had a very limited ICT infrastructure about 20 computers in a computer class and there was only a minimal ICTrelated transformation in educational practices in terms of teaching ICTs as a new school subject. While
ISLS
130
ICLS2012
government policy required teaching ICT literacy to all students, it was hard to implement such expansion of ICT education due to lack of computers, frequent malfunctioning, constantly occurring viruses, slow or nonexistent computer network, and limited teachers ICT competence. Ethnographic data indicated a strong tradition of categorizing students into different curricular groups on the basis of tests in sciences. Accordingly, computer classes were reserved for high achieving double or triple science students whereas automatically promoted low achievers (single science students) were stigmatized and their consequent tarnished feeling of self-worth aggravated by a lack of access to ICTs and consequently a developing world: R: What about access to the computer lab T: It is a bit of a problem because the computer lab there is only one and there are something like 20 computers and there are 80 students doing ComputerStudies in the mornings they are attending their lessons and in the afternoons they are doing their projects so it is a bit of a problem to have access to a computer R: How many students do you have in the school? T: 1800 R: What is the nature of computer literacy across the board, do all of the students have computer awareness and what access do they have? T: Well like I said, there are only about 80 students, 40 form 4 and 40 form 5, in our committee, one of the responsibilities is that we would like to start computer awareness that is just the basic of ... or introduction to computers. And if we can start computer awareness, then we can get somewhere. R: So other students who are not doing computer studies are not having...dont have access to computers; T: Ya they do not have... An interview with a Computer Science student revealed that the problem was not only due to a limited amount of technology but also instruction as well: It is not like I am blaming any one but we have a very big problem. They just give those many textbooks and then they say go and read them but some of us are not conversant with the computer terms which makes it very difficult to understand and we do not get enough end of month tests. I am not going after anybody, I am just trying to remedy the situation. Eight (8) to thirteen (13) teachers, representing multiple subject domains and levels of authority, formed the schools ICT group and, consequently, participated in the CL process. Two members of the local media and teacher training and development centre participated in all the sessions together with the moderating intervenors. The process consisted of eight (8) two-hour sessions at a rate of two, two hourly session per week for 4 weeks. All sessions were video and audio recorded and transcribed. The sessions followed the cycle of expansive learning in terms of analyzing and questioning the prevailing practices, examining the historical development of the activity, planning some concrete changes, implementing the changes, and reflecting on the process. The moderator set tasks in order to stimulate questioning of their activity such as: causes of joys and sorrows for teachers, students and parents; examining the historical development of the school activity in order to track the central contradiction, and envisioned changes that would make educational activity more meaningful for the participants. In order to stimulate the construction of new ideas, each task was supported by a second stimulus such as mirror data or theoretical models of schooling and learning. In one of the first CL sessions, the researchers projected a video clip, onto the mirror board, of a student testifying how being categorized as a single science student engendered stigmatization by teachers and fellow pupils and that some teachers refused to answer questions and provide added support. This piece of mirror data elicited a lively discussion with tones of self-critique among the participants, the consequence of which was a request for further data on the problem. Later on, the same theme came up in another form when the teachers reflected on methods of Science education that involve asking the students to plan and carry out experimental projects; it was pointed out that the single-science students would not have capability to do it and should, therefore, not be given such tasks. A historical analysis revealed that a former established government policy, requiring automatic promotion of all students to junior and senior secondary schools regardless of their educational achievements, has made the student population more heterogeneous. In order to cope with the prevailing heterogeneity of student learning abilities, categorization was installed. As a consequence, there appeared to be a tension-laden contradiction between the object of teachers instructional activity for all and prevailing instructional methods and practices that did not tailor and personal instructional efforts according to students competencies. The tendency to categorize and segregate students according to different levels of accomplishments created some visible contradiction in the school system. Participants envisioned two solutions for overcoming the double binds together with the intervenors: 1) In order to facilitate students motivation, the participants designed a new tool (personal study plan) for addressing a students studying conditions, problems, and study objectives. The design aimed at bringing student weaknesses and strengths to the fore and subsequently to create a peer support network across the categories. 2) the teachers initiated, moreover, experiments with co-teaching so as to provide more time for individual student support. Neither of these solutions was based on the immediate use of ICTs, which is
ISLS
131
ICLS2012
understandable in view of the limited ICT resources and lack of ICT competencies of the teachers. Yet, the new ideas of a more personalized way of teaching could later be sustained with the use of ICTs as a tool to lead to a more enabling learning environment. Furthermore, the teachers organized collaboration with the local Audiovisual Center for support and teaching in creating ICT based teaching aids. Related to the model in Figure 1, these initiatives would lead to changes on levels 2 and 3. An analysis of transcribed CL data indicates that while ICT-related discourse surfaced frequently, presumably because of limited experience, the participants were not able to sustain concrete ICT-related discussions, their focus tended to gravitate toward other themes. Nevertheless, making the computer into an instrument in teaching other subjects would, however, mean a major qualitative step in the development that could only have been taken with an expanded ICT infrastructure. The results of the first CL intervention in the AT-SIG surfaced tensions between ICT implementation and school development. Firstly, the teachers, due to new measures, had difficulties in finding time for the further development and experimentation of the new solutions. Secondly, after leaving the country, the foreign intervenors could not adequately continue supporting the educational transformations needed after the 8 CL sessions and the internal change agents were displaced to new geographical areas. Thirdly, as a consequence of the centralized management of Botswana teaching service , which enables teachers to move from one school to another across their careers, (in pursuance for career progression), some key CL teachers participating in developing the new solutions were transferred to other schools before they could elaborate the new tools and practices. Accordingly, the teachers did not have a sufficient support network to expand on the new practices despite adequate analyses of the developmental challenges having occurred through the CL. The envisaged virtual support platform that was planned to be provided by the University of Mauritius as a collaborating partner did not come forth and thus much was lost in this domain.
The CL Intervention at School B.

A recurrent problem in the development of ICT-based learning environments has been the lack of student motivation related to the lack of authenticity of the problems dealt with. The second CL intervention specifically addressed this problem at school B. A school located in a remote and rather deprived rural area in the Kalahari Desert in which the livelihood was mainly based on animal husbandry. School B had 800 students 80% of which are boarders and 42 relatively inexperienced teachers inexperienced in what?sent to the school from elsewhere for a 5-year-period and who maintained a strict separation from the village population of about 1590. The ethnography based on interviews and discussions with teachers, principals, community institutions, students, ICT centers and parents revealed a cleavage between school, parents and the village officials each blaming the other of problems in child rearing in the village. Moreover, teachers are far away from their families which as the following testimony indicates, impacts on their teaching: Every time I come back here I have my body here but my heart is with my wife and girlsthey need my affection and supportwhat is a dad that is not at homewe cannot be good teachers with our hearts broken like this (field-notes, 2007). Teachers generally blamed parents for a lack of interest in the childrens education which they said was manifested by a lack of school function attendance and alcohol abuse. Because parents felt that they had little control over their children and little to do with their school going activity, they could not find productive solutions. The intervenors hypothesized that the problems lay in the school-community relationships rather than within the school. Instead of inviting only the teachers to develop the educational practices within the school, village officials, teachers from the primary and secondary school and parents were invited to take part in a boundary crossing CL process. Participants of CL were composed of eight (8) members of the Junior secondary school, five (5) members of the primary school, two (2) members of the vocational school, two (2) members of the non-formal sector, several parents (later this number was around 8), and odd members of the different institutions, a total of twenty-five (25) people. The sessions were organized as two hourly- twice per week sessions for 5 weeks; most of them involving both school and community. Introductory sessions were held separately with each school. All session were video-taped and voice recorded and transcribed. The CL experimental space was held at the local Kgotla (court house), a thatched roofed open- to- the-environment building situated half way between the three educational institutions. The boundary crossing CL began by showing part of the ethnographic data including the interview with a parent welder who testified to the estrangement between the school culture and arts and crafts culture prevalent in this area. The participants then discussed the historical changes that had taken place in child rearing practices between home and school in the village. They came to a collective decision that the children performed better when school lessons were given under the tree and child rearing had been a shared responsibility. They began to question what had led to this transformation: Head teacher: It is something like teacher 2 said, mind your own business. It is like parents are not very much supportive to the program for the school, so the parents live their own life there, teachers live their own life, other people in other departments live their own life and people do not come together and support one another it is like there are some barriers in between if the school ask for assistance from the community maybe to say come for a board collection or come to address the issues related to peoples performance and so forth they
ISLS
132
ICLS2012
come with a different mind to be against what the school is trying to do or to try to be on the side of the kids even when they have done something wrong. So I think that it is a very serious limiting factor because when they come they come to say no even when their kids are wrong they say that they are not wrong. As if we are not good enough to the kids we always say they have done ABCD which is not good and so I think that there is a problem there. Childrens voices were as well part of the mirror data and used to maintain focus on the object. A metaphor of a school under the tree arose and grew until it became the pivoting point where parents and teachers began to reconstruct models for a learning environment that was both for the wider world and the everyday. In the extract above the school head suggested a new way to understanding the problem of teacher parent collaboration. This insight was strengthened through the introduction of further ethnographic data in which parents discussed a bad child and led to discussion in which it was agreed that parents, teachers and community are all ultimately responsible for outcomes of children under their care; all should work to promote cases of the good child and to minimize cases of the the bad child. The computer room of the Junior Secondary school contains twenty-five (25) a computers and all children have access at different periods of the week. All teachers are encouraged to plan their lessons and write reports as well as produce exam and test results on the computers. During the first field study period, this laboratory was relatively under exploited but two years later, it was always busy and the level of computer literacy amongst the teaching staff appeared much higher. However, teachers were not using ICTs as a tool to stimulate the learning and development of their students. It appeared clear that creating novel ways of using ICTs as instructional tools and overcoming contradictions embedded within educational the institution and its relation to surrounding society were needed for bringing pedagogical benefits about. Their new designs included the cultivation of vegetables gardens, an arts and crafts center and a bakery projects. These work groups consisted of teachers of the Junior secondary and primary schools as well as the vocational schools, principals of the schools, members of all of the village institutions and parents. These projects planned to make use of all the subjects offered within the curriculum such as Mathematics, Geography, Science and the integration of computers. The concept is that of a school-community type of learning where children are actively involved in the creation of their own knowledge. ICTs are to be used in order to measure, monitor, and design implementation and progress of the projects. Such a leap requires careful planning and nurturing as it is far easier once the contradictions between the old activity and the new design surface to return to the former way of working. Spirits were high in the community after the conclusion of the CL. This was expressed in the words of a member of the Village extension team: I never thought that I would see the day when parents and teachers came together to work on their childrens future. Table 1. A Summary of focuses on the two Change-Laboratory interventions School A School B School Senior secondary school Junior secondary school and Primary school Location Relatively urban Rural Ethnography Interviews of students, teachers, principals, Interviews of students, teachers, parents, parents, educational facilitators principal and community members Teachers, an intervenor, media centre, AS-TIG Community stakeholders, teachers, AT-SIG Participants members members and an intervenor Process of Nine sessions.(3 with the two schools and 6 Seven sessions. intervention with the schools and community) Developmental Collaborative study planning to enhance student Joint pursuit of an agricultural project and motivation; learning to create audiovisual an arts and crafts project experiments teaching aids to enhance teacher motivation; coteaching, system of controlling the progress in the implementation of new policies.
Discussion
The present investigation focused on examining how CL-intervention is used to elicit expansive school transformation in the context of technology-mediated learning. We considered future possibilities of such usage. The analysis revealed how teachers and educational institutions supported by interventionists explored ways to facilitate educational transformations in difficult conditions. Although there was not an adequate infrastructure, a high shortage of computers, and limited human resources, many critical issues were addressed regarding student motivation, collaborative teaching, or transformed school-community relations. Research on the use of ICTs in schools has shown that the greatest gains from new technology are achieved by redefining the content of the educational activity and using the new artifacts for realizing something that was previously not possible.
ISLS
133
ICLS2012
Regarding teaching, this means, that the teachers have to rethink the pedagogical principles of their teaching and to use ICTs to realize a new kind of pedagogy. The present experiences of using CL for promoting such educational transformation and experiences are encouraging, in spite of the challenges involved in working through various linguistic, social, and cultural boundaries. It appears, however, that without personal and collaborative experiences of using ICTs, discussion in CL sessions tended to drift to issues other than ICTs (such as how to deal with diversity, overcome prevailing strong categorization practices and cope with common place corporate punishment). As far as CL sessions involve teachers only talking about technology without appropriating ICTs as a part of their everyday activity the interventions may not have sustained effect. Thus far, interventions have only hinted as to expansive possibilities provided by ICTs, and there is more talk about technology-mediated learning and instruction than its actual concrete exploration. The slow progress reported in the African context does not, however, appear to be an exception but a rule of technology-mediated learning. Our experiences from ICT in education projects from Finland and Europe (Lehtinen et al., 2001; Sinko & Lehtinen, 1999) indicate that educational transformations take years to consolidate. The present investigation aimed also at analyzing challenges and constraints of ICT-mediated educational intervention. Much of the research on the use of ICTs in schools has focused on analyzing the use and reasons of lack of use of ICTs, assuming uncritically that these technologies help the schools to carry out their societal task. Fewer studies have been carried out on what inner developmental contradictions in the current school education could be overcome with the introduction of these technologies. The expansive use of ICTs cannot, however, derive sufficient motivation from the new technology as such but must be based on well analyzed developmental challenges of the school that the teachers recognize and are ready to work on. According to the previously quoted theory of Perez (2002) the broad societal utilization of new technologies only started after the implementation of the technology has been carried out by actors representing the new technology. A turn from technology push to need pull takes place as the agency in the development moves incrementally from the providers to the users. In this research we have used the Change Laboratory method to help the practitioners to analyze the central developmental challenges and possibilities of their educational activity. One can argue that this need-based approach came to the pilot schools too early in view of the implementation of the new technologies. On the other hand, the new technologies can never be the only and the whole solution to a major historical challenge of development in an activity. Rather the needs and possibilities of the activities are more aptly recognized by a group of actors involved in an ongoing process of developing their educational activity, rather than by individuals carrying out their traditional work routines.
References
Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah, NJ: Erlbaum. Engestr Y. (1987). Learning by Expanding. Helsinki: Orienta-Konsultit Oy m, Engestr Y, Engestr R., & Suntio, A. (2002). Can a school community learn to master its own future? In m, m, G. Wells & G. Claxton (Eds.), Learning for life in the 21st century (pp.211-224). Cambridge, MA: Blackwell. Engestr Y., Virkkunen, J., Helle, M., Pihlaja, J., & Poikela, R. (1996). The Change Laboratory as a tool for m, transforming work. Lifelong Learning in Europe, 2/1996. Pp. 10-17. Hakkarainen, K. Palonen, T. Paavola, S. & Lehtinen, E. (2004). Communities of networked expertise: Educational and professional perspectives. Amsterdan: Elsevier Lehtinen, E., Sinko, M. & Hakkarainen, K. (2001). ICT in Finnish education: How to scale up best practices. International Journal of Educational Policy, Research and Practice, 2 (1), 77-89. Long, N. (2001). Development sociology: Actor perspectives. London: Routledge. Perez, C. (2002) Technological revolution and financial capital. Cheltenham, UK: Edward Elgar. Ravitch, D.(2010). The death and life of the great American school system: How testing and choice are undermining education. New York: Basic Books. Roth, W.-M. & Lee, Y. J. (2007) Vygotskys neglected legacy. Review of Educational Research, 77, 2, 186232. Scardamalia, M. & Bereiter, C. (2006). Knowledge building. In K. Sawyer (Ed.) The Cambridge handbook of the learning sciences (pp. 97-115). Cambridge, MA: Cambridge University Press. Sinko, M. & Lehtinen, E. (1999). The challenges of ICT in Finnish education. Juva: Arena. Venkatraman, N. (1994). It-enable business transformation. Sloan Management Review, 35, 73-87. Yamazumi, K. (2006) Learning for critical and creative agency. In K. Yamazumi (Ed.) Building activity theory in practice (pp. 73-97). Osaka: Center for Human Activity Theory. Kansai University.
Acknowledgments
The present project was supported by grant 1127019 from the Academy of Finland. The study relies on collaboration between Universities of Helsinki and Botswana.
ISLS
134
ICLS2012
Visible Spending: Information Visualization of Spending Behavior and the Thought-Act Gap
Emily S. Lin, Tufts University, 105 College Ave, Medford, MA 02155, es.lin@tufts.edu Holly A. Taylor, Tufts University, 490 Boston Ave, Medford, MA 02155, holly.taylor@tufts.edu Abstract: Most people know they ought not to spend beyond their means. Yet, many do so anyway, often without being aware of it. To explore how to narrow this thought-act gap, we conducted a three-month mixed-methods pilot study (N=25) of the effects of graphical vs. textual feedback on the relationship between estimated and actual spending. Results of a 2 x 2 x 3 repeated measures, doubly multivariate ANOVA indicated that high-debt participants experienced improvement in their estimations more than low-debt participants, and that the estimations of participants who received graphical visualizations became more accurate. Excerpts from interview data shed light on the learning processes underlying such changes.
Introduction
Individuals make private financial decisions daily; sometimes these decisions result in taking unwise financial risks or simply spending beyond ones means. The consequences of these decisions can be damaging on the individual and family levels, leading to such consequences as emotional stress, an inability to access needed resources, or bankruptcy (Drentea, 2000). Individual financial choices can also play a role in larger-scale crises. While multiple factors contributed to the recent global economic recession, a massive wave of consumer-level defaults on credit card debt and mortgages counts among them. Even at the bull economys peak, economists warned that individual consumer behavior in the form of rising credit card and housing debt could negatively impact the nation's economic future (Greenspan, 2004). Given that most people would agree that they ought not to spend beyond their means, we posit that some people simply may not be aware of the extent to which their day-to-day actions lead them to spend more than they take in. An individuals financial profile thus calls on three different types of representations a mental measure of intended spending, a mental measure of estimated spending, and an external measure of actual spending. For brevity's sake, we call these representations oughts, thoughts, and acts, respectively. Clearly, there are practical implications to understanding how we might nudge individuals toward aligning their oughts, thoughts, and acts. The present study builds on research on visual perspective, cognition, and behavior to explore how thoughts might play a role in affecting that alignment. In particular, we studied whether participants who received interactive graphical reports of their spending behavior modified their thoughts to produce more desirable financial outcomes than people who only received text-based feedback.
Thoughts and Failures of Self-Knowledge

Thoughts are usually discussed using an analytic frame. When I think that I am in control of my spending, I am able to give evidence e.g., I was tempted to go out to a pub with friends but instead opted to stay in and cook, which was much cheaper to frame my analysis of the situation. Because good thinking is understood to be tied to externally available evidence as well as a subjective analysis of that evidence, it is reasonable that thoughts would mediate the alignment between oughts and acts. However, when thoughts falter in their analytic rigor, oughts and acts while appearing to be aligned may in fact be at odds. Revisiting the earlier example, I might think I am in control of my spending because I opted to cook dinner at home rather than going out with friends, but I may be overlooking the sandwich and chips I bought every day for lunch, which add up to more than I would have spent going out to the pub that one night. People are predictably more likely to make these kinds of analytic slips of thought and subsequently construct inaccurate mental models of their own behavior when viewing situations from particular perspectives or through particular cognitive frames. For example, research has consistently shown that using a first-person rather than a third-person perspective yields less accurate perceptions of phenomena ranging from navigational orientation (Bruny, Mahoney, Gardony, & Taylor, 2010) to judgment of one's causal understanding (Keil, 2008). Another common heuristic in these cases is confirmation bias (e.g., Nickerson, Perkins, & Smith, 1985), a framing that predisposes us to seek information and adopt interpretations consistent with pre-existing beliefs. To return to the prior example, if I believe I am good at resisting temptation, I am more likely to remember the confirming incident when I resisted going to the pub, and also to neglect the disconfirming incidents of buying convenience food. In addition to selective memory, confirmation bias can manifest in the face of disconfirming representations of self by making more salient contextual information that explains or excuses one's behavior in a way consistent with self-concept. Jones and Nisbett (1973) present a classic example of this attribution error, where others' behavior is seen as having dispositional causes but our own behavior is explained by situational causes.
ISLS
135
ICLS2012
The question that naturally arises is how to enable perspectives and frames of mind that promote the development of more productive mental models of one's own behavior. Integrating the lines of research mentioned above, we suggest that the differentially accurate mental models produced by taking first- and thirdperson perspectives may be a result not of simply changing perspective, but of what changing perspectives gives you a chance to see yourself through a different, and disconfirming, mental frame. Indeed, in their study of voting behavior, Libby, Shaeffer, Eibach, and Slemmer (2007) showed that the perspective changes that affected behavioral changes were mediated by changes in dispositional understandings of self. With this insight into the mechanism by which perspective shifts influence behavior change, we can propose the role of thoughts in connecting oughts and acts. Assume I have an ought (e.g., I should not spend too much money eating out,) and want my act a to be in line with that . My acts appear to be motivated by my thoughts about my disposition. That is, given thought A about my own disposition (e.g., I am thrifty,), I am motivated to commit act a (e.g., not going to a pub), which in concordance with that disposition. Additionally, though, my acts are also interpreted through my thoughts. After committing a, if I hold dispositional thought A about myself, I may overlook (via confirmation bias) or discount (via situational attribution) an act b (e.g., buying sandwiches for lunch) that does not appear to be in concordance with A. In this case, if I have ought and dispositional thought A, but have actually committed act b, then I have an ought-act mismatch and am either not aware of it, or at least not likely to act differently should act b present itself again in the future because I did not incorporate b into a revision of A. Therefore, while A motivates a, it also pushes us to neglect instances of b.
Getting to Good Spending

From this analysis we ask, what next? Is our goal to change act b into a? To change ought into a new ought ? Change thought A into a thought B that more accurately describes my disposition? We have arrived at a basic question what kind of ought-thought-act relationship constitutes good spending? The typical accounting conceptualization is to hold oughts constant and get acts to match them that is, good spending is spending within my budget. If my spending does not match my budget, then I will change my spending to match my budget, or change my budget to match my spending. Any dispositional thinking is fairly trivial in this view. However, there is another way to operationalize good spending, and that is to say that good spending is spending of which I'm aware. In this view, my judgment of myself as a good or bad spender is not contingent on achieving some pre-determined goal, but on whether I know what I'm doing. A focus on thoughtact gaps, rather than ought-act gaps, encourages us to think of spenders not as accountants, but as scientists, whose investigations are driven not on quick judgment but on testing and generating hypotheses, a process in which paying attention to unexpected findings (e.g., a thought-act gap) is a critical strategy (Dunbar, 2000). While the scientists framing of good spending may seem counterintuitive by appearing to value, for example, overspending of which I am aware more than under-spending of which I am not aware, it also takes a longer, more global view of spending than the accountants view. Indeed, the accountant may be more likely to stay within a month-to-month budget, but the scientist may learn how to shift resources adaptively, overspending when it is strategically worthwhile (e.g., taking financial risks to start a small business, or buying a higher-quality car that will last, rather than a more affordable one that will yield years of costly repairs). Motivated by the idea that good spending of the scientific sort can be learned with the appropriate cognitive scaffolds, the present work comprised a three-month mixed-methods pilot study to examine whether participants who received graphical representations of spending behavior achieved greater alignment between estimated and actual spending as compared to participants who received text-only representations. We propose that graphical representation of data (also known as information visualization) can help overcome confirmation biases in a way similar to the mechanism by which dispositional thinking circumvents those biases. Evidence for differential analytic results based on types of visual representations is found in decisionmaking. People select different options when the same information is presented differently or when decisions are made through different means (e.g., Kahneman & Tversky, 1984, 2000; Redelmeier & Tversky, 1992). In addition, the inability to make rational decisions is exacerbated by too much information (Lurie, 2004). Lurie and Mason (2007) have suggested that visualizations can address these analytic slips, and the growing field of information visualization (Card, MacKinlay, & Shneiderman, 1999) is producing relevant tools. We suggest that presenting information on personal spending through visualizations can take advantage of humans' consummate pattern seeking ability (Wolford, Newman, Miller, & Wig, 2004), therefore generating, in a single view, alternative but plausible dispositional interpretations of their spending pattern. Thus, in the present study, we predicted that participants who received graphical feedback would show a greater narrowing of the gap between estimated and actual spending that is, between their spending thoughts and their spending acts.
Method Participants and Procedures
ISLS
136
ICLS2012
The present analysis was conducted using data from a pilot study examining the spending behavior of 29 adults. Using e-mail, electronic advertising, and posters, we recruited people with either high credit card debt (>$2000 carried over from month-to-month) or low credit card debt (<$200 carried over from month-to-month) to participate in a study that would track their spending over three months. Of these participants, 25 contributed valid data for this analysis; 4 were removed from the final sample for failing to follow directions to accurately report their data. Participants who completed any part of the study were financially compensated. The participants mean age was 29 years (SD=7.59), with 23 participants self-identifying as White, 1 as Asian, 1 as Hispanic, and 1 as both Asian and Hispanic. The data were collected over three months and included visits to a campus laboratory as well as electronic communications. Participants were required to have on-line access to checking and credit card accounts that comprised at least 80% of their daily financial transactions and to link them to a commercially available financial tracking website. High-debt and low-debt participants were differentiated, and then within each group were separated into pairs, with the first person in the pair being randomly assigned to either a visualization (VIS) group or a no-visualization (NOVIS) group, and the second person assigned to the other group. The group assignments can be seen in Table 1. VIS participants had online access to real-time graphical representations (bar and pie charts) of their spending by category, and NOVIS participants received weekly spreadsheets with text-only lists of their transactions, as well as text-only reports of spending by category. Table 1: Participant Group Assignments. Low-Debt 5 8 High-Debt 7 5
NOVIS VIS
All participants set their own initial budgets by category, then estimated and monitored their actual spending through weekly on-line surveys and monthly in-person questionnaires. Weekly surveys included monitoring and knowledge questions (e.g., According to your spending summary, how much have you spent in each of the following categories so far this month?) to ensure that participants regularly processed their spending reports. To check that participants had roughly the same exposure to their spending information, weekly surveys asked participants to report how many times they had looked at any kind of spending report (i.e., transaction lists not provided by the experimenters, credit card statements, bank statements, and, for the VIS group, their real-time online reports). Because personal spending is frequently budgeted and evaluated on a monthly timeframe, more extensive monthly questionnaires were completed in the laboratory, requiring participants to estimate their monthly spending in every budgeted category, then to compare their estimates with their actual spending and respond. All values were checked against automated reports and participants' selfreports of spending, and were corrected for notable errors (e.g., the mortgage payment for both the current and the following month being deducted in one month). Some participants were interviewed at monthly visits and were videotaped using the on-line financial tracking tool to elicit information about the psychological processes that might explain spending oughts, thoughts, and acts, as well as the relationship between them.
Measures
Both the signed and unsigned gap between estimated and actual spending were calculated as outcome measures for each participant at three time points after one month, two months, and three months of tracking. The signed estimation gap was designed to give a standardized index of how well participants were able to estimate their actual spending. This measure captures accuracy of estimation but also gives information about whether participants are underestimating or overestimating their spending, a valuable differentiation in the accounting perspective, despite the lack of budgetary anchors. The signed gap was calculated by subtracting the total amount of actual spending for each participant each month from the total amount of estimated spending for that month, then it was normalized by dividing that difference by the total amount of estimated spending. Thus, negative values indicate underestimation (i.e., spending more than one thinks), positive values indicate overestimation (i.e., spending less than one thinks), and the distance of the values from zero indicates the accuracy of the estimation. The absolute value of the signed estimation measure each month was used as a standardized index of unsigned estimation accuracy, independent of whether the individual over- or underestimated his or her actual spending. This measure is useful for the scientific perspective on good spending and allows for the calculation of meaningful change scores to compare accuracy from month-to-month. For example, if Participant 1 overestimated her spending in Month 1 by 20%, then underestimated in Month 2 by 20%, and Participant 2 overestimated her spending in Month 1 by 5%, then underestimated in Month 2 by 35%, a signed measure would give them both the same change score of 40 percentage points. However, the unsigned accuracy score shows that Participant 2's accuracy became worse over time, while Participant 1 remained roughly on target.
ISLS
137
ICLS2012
Results
A 2 x 2 x 3 repeated measures, doubly multivariate analysis of variance (ANOVA), with debt status and visualization condition as the between-subjects factors and three points of measurement as the within-subjects factor was conducted with both the unsigned accuracy gap and the signed estimation gap as outcomes.
Unsigned accuracy gap

A marginally significant main effect of visualization condition on unsigned accuracy was found, with the VIS group (M=16.78, SD=12.53) being more accurate, on average, than the NOVIS group (M=27.85, SD=22.25), F(1, 21)=4.34, p=.05. No significant main effect for time was found, but tests of within-subjects contrasts reveal significant effects on unsigned accuracy between months 1 and 2 for the two-way interaction of time and visualization condition (F(1, 21)=5.32, p<.05). Calculation of the accuracy change scores from Month 1 to Month 2 showed that, on average, the VIS group became more accurate in their estimates (M=-4.03), whereas the NOVIS group became less accurate (M=11.35). While both the VIS and NOVIS groups had mean accuracy gaps of about 20% at Month 1, by Month 2 alone the VIS group had, on average, improved to a 15% gap, whereas the NOVIS group had actually gotten worse, with an accuracy gap of about 30%, as shown in Figure 1.
Figure 1. Mean unsigned accuracy gap by visualization condition over three months (N=25). In addition, the NOVIS group had greater variability (SD=28.85) in their individual change scores between Month 1 and Month 2 than did the VIS group (SD=13.40), F(11, 12)=4.64, p<.05, as seen in Figure 2.
Figure 2. Individual unsigned accuracy gap change scores between Month 1 and Month 2, grouped by visualization condition (N=25).
Signed estimation gap

No significant main effect of visualization condition or time on the signed estimation gap was found. However, as shown in Figure 3, there was a significant interaction between time and debt status, F(2, 42)=5.14, p=.01. Within-subjects contrasts indicate significant interactions between time and debt status between Months 1 and 2 (F(1, 21)=4.51, p<.05) and between Months 2 and 3 (F(1, 21)=10.17, p<.01).
ISLS
138
ICLS2012
Figure 3. Mean signed estimation gap by debt group over three months (N=25). High-debt participants began the study underestimating their actual spending by about 20% (mean gap at month 1 was -21.65, SD=26.38) but by the end of 3 months of participation, their estimates were higher than their actual spending by about 6% (M=5.68, SD=33.01). On the other hand, low-debt participants began the study underestimating their spending by about 4% (M=-3.93, SD=13.9) and ended doing slightly worse, underestimating their spending by about 9% (M=-8.79, SD=10.29). A marginally significant three-way interaction between time, debt status, and visualization condition was also found on the signed estimation gap between Months 1 and 2, F(1, 21)=4.06, p=.057. A visual representation of the complete 2 x 2 x 3 analysis for the signed gap is shown in Figure 4.
Figure 4. Mean signed estimation gap by debt and visualization group over three months (N=25).
Explaining Process: Preliminary Qualitative Results

Given the significant interaction between time and debt status, as well as the marginally significant interactions with visualization condition, we examined interview data from two high-debt VIS participants to shed light on processes that might underlie their improved spending awareness. While a deep qualitative analysis is beyond the scope of the current paper, the following excerpts are nevertheless illuminating and point toward potentially fruitful paths for further investigating these processes. Participant A: Beginning: I don't make lots of frivolous purchases; I don't eat out much and I don't buy clothes or things so when I think, where can I skimp, because I'm making less than I spend, often, so I think, what's my biggest extravagance, or the thing I could live without? Month 1: When I click onto each individual pie [piece of the chart], that's where I really noticed that I was eating out too much for lunch, or buying too many little treats ... 'Cause in my head I'm like, all I spend every month are the big ticket things, like, oh, I spend my car payment, I
ISLS
139
ICLS2012
spend on this, so that's how this is helpful for me, looking at these little tiny ways I could have saved, almost a hundred dollars this month. Month 2: The fees are just stupid. Whereas before I was like, 'Well, I'll get to that bill...' now, I've been calling my creditors, my student loan people... So it's sort of that, just seeing it again, that big chunk, altogether, seeing it sort of in its sum total, because I can write off 5 dollars here and there, but I can't write off 200 dollars. It's made me feel like I had more, sort of, agency in what's going on with it I don't even know that I could have told you that I spent more than a hundred dollars, or even a hundred dollars on fees and charges, back in May. So that wasn't really part of how I was thinking about my spending. Month 3: I got really excited about the different parts of the graph I knew I wanted to see whittled down and be able to click through and see them get smaller and smaller. I went from hundreds and hundreds [in fees] the first month, down to $87 this month, which is good. [Its] definitely something that I saw it, and I made some changes ... when bills are getting paid and ... moving things to different credit cards with lower fees and stuff like that ... it's been nice to know that I have a little bit of control over that. Looking at it every time, I was like, 'Oh gosh, what invisible thing is going to pop up ' Luckily that all went down over time because I made those changes, but ... I guess awareness equals stress, in this situation. I think I need to be more honest about the way I spend with myself. Participant A describes an arc that begins with a dispositional thought about himself I dont make lots of frivolous purchases. By the end of the study, he acknowledges that something about his experience has allowed him to overcome confirmation bias of his initial dispositional assessment of his spending. The monitoring effect is evident in repeated references to becoming more aware of tiny ways to cut spending that he had previously overlooked in favor of larger, more salient expenditures. However, this effect appears to be notably influenced by the fact that feedback was being provided graphically. The participant describes his interaction with his data using visual metaphors, naming the act of clicking on pieces of a pie graph as a key strategy for understanding his spending better, as well as expressing excitement about seeing undesirable pieces of that graph get smaller and smaller. Though these descriptions could be applied to numerical representations of spending, this participant appears to have used the information visualizations not only to improve his mental awareness of his spending (a scientific success) but to actually alter his real-life spending, with the effect of significantly reducing the amount of money he was paying toward fees (an accounting success). Participant B: Month 3: My narrative about money, all along, has been that there's never enough, I'm always overspending so everything we do financially is by definition wrong so there's no point paying attention or trying to figure out how to do it right. Being in the study has been really interesting because it's really drawn my attention to the ways that's not true anymore I spend a lot of time and emotional energy fussing over things and it's like, really? Thirty dollars is not where I should be spending hours fussing. So it really made me realize that, you know, I do have enough money, and I have a lot of personal tools to manage it wisely, and I don't always do it right... but it's never this, like, crisis that it would have been in the past. Can I just relax about this and like put my energy back into other things in my life? It's like right now I'm spending an evening a week fussing over that stuff and that's time I could spend hopefully doing really creative work, you know? Maybe I'll write a book. Participant B also experienced improved accounting success, but through circumstances external to the study (a new, more lucrative job). Even in this case, though, she experienced improved scientific success, reducing her unsigned accuracy gap from about 25% in Month 1 to about 2% in Month 3. She repeatedly cited the graphs and charts as more useful than transaction lists to her improved understanding, which not only helped her circumvent pre-existing notions of herself as an irresponsible spender, but that helped her alleviate anxiety and learn how to act more strategically.
Discussion
This study examined whether participants who received graphical representations of their spending behavior achieved greater alignment over three months between their estimated and actual spending behavior, as
ISLS
140
ICLS2012
compared to participants who received text-only representations. Additionally, we investigated whether any effects were moderated by participants' pre-existing consumer debt status. We found one significant effect, the interaction of time and debt status on the signed estimation gap. On average, high-debt participants moved from underestimating to overestimating their spending and improved their estimation accuracy, whereas low-debt participants underestimated their spending with relatively consistent accuracy over the entire study duration. This finding might be indicative of a monitoring effect, whereby the act of tracking one's finances, regardless of feedback format, itself results in a change in awareness or behavior. A possible explanation is that the intense tracking required for participation in the study introduced a significant change from the prior habits of the high-debt participants, whereas the low-debt participants entered the study already doing a reasonable level of monitoring of their spending and so experienced a less dramatic change. Though the effects involving visualization condition were not found to be statistically significant, we discuss here the marginally significant and contrast effects as indicative of interesting trends to be investigated more rigorously. The main effect of visualization condition on the unsigned accuracy gap aligns with the hypothesis that graphical representations of spending support more salient and persistent mental models of actual spending than do textual representations, such that VIS participants were able to narrow their initial thought-act gaps by the second month of participation. NOVIS participants, on the other hand, increased their thought-act gaps, on average, regardless of direction (i.e., whether over- or underestimating). In addition, the significantly larger variance in the change scores of the NOVIS group can be interpreted to mean that while graphical feedback presented a perspective on spending to VIS participants that was salient enough to overcome other, non-systematic biases around estimation of spending, textual feedback presented a relatively weak mental model whose effect was washed out by whatever other factors normally influence the estimated models of NOVIS participants, independent of feedback format. We did not have a measure of dispositional factors in this change, but at the least we can propose that if graphical feedback can produce a predictable effect on estimation accuracy, dispositional biases toward overlooking overspending violations can be circumvented through perspective manipulation, if not directly changed. This proposal helps frame the finding of a significant contrast between Months 1 and 2 on the signed estimation gap for the three-way interaction of time, debt, and visualization condition. Figure 4 shows that highdebt participants had quite different experiences between Months 1 and 2, depending on their visualization condition. High-debt VIS participants became slightly less underestimating of their spending, whereas high-debt NOVIS participants went from overestimating to underestimating their spending. For low-debt participants, VIS participants remained fairly accurate in their estimations (though slightly underestimating), but NOVIS participants moved from being fairly accurate to highly overestimating their spending. The paths of these four groups lend support to the proposal that while textual feedback on spending appears to have no systematic effect on mental models of spending, graphical feedback produces a systematic and desirable effect on improving one's awareness of one's spending, and that this effect holds across types of spenders. In fact, the group that showed the most improved awareness (i.e., both an increase in accuracy and a move from underestimating to overestimating spending) was the high-debt VIS group, a finding that has implications for policies and interventions to help consumers remediate overspending and suggests that providing high-debt consumers with salient, visual feedback on their spending habits can circumvent the mental and behavioral habits that have put them into debt in the first place. While more systematic analyses of the interview data will be necessary to make interpretive claims about the processes that may lead to improved spending from both the accounting and scientific perspectives, the interview excerpts we have included above suggest the importance of future research on feedback format as it relates to the role of developing agency in changing the alignment between oughts, thoughts, and acts, and to surfacing and challenging pre-existing notions of ones self as a spender. Such processes may be favorable to both improving ones estimations of spending and becoming more strategic about how one uses valuable resources like time, mental energy, and, of course, money. The reflectiveness shown by the participants also underscores the importance of observing processes that relate oughts, thoughts, and acts in context. Follow-up work could, for example, observe participants in the moment of exploring their spending data and more deeply probe the processes by which these awareness shifts happen and actual changes in day-to-day spending occur. As noted above, all results, interpretations, and recommendations in this pilot study are made with caution, due to the small sample size and therefore low statistical power, as well as methodological limitations of this pilot study. For example, while we checked the validity of actual spending measures against both selfand automated reports, we did not test the reliability of our measures of estimated spending, nor did we rigorously control for the feedback dosage participants received, as measured by the number of times they reported looking at their data. A baseline measure of the estimated-actual spending gap before study participation begins would also be useful in future, more rigorous studies, in order to capture any changes in the gap from pre-intervention to post-intervention, as would a more theoretically predicated cutoff between highand low-debt participants.
ISLS
141
ICLS2012
In addition, our framing theory for the study proposes taking a scientific perspective on spending in which thought-act misalignments can produce investigations that result in better spending that is, spending in which thoughts and acts become more aligned. However, while we can look for evidence of these investigative processes in interview data, we have no reliable measures of their existence, nor of whether changes in situational or dispositional attributions occurred. Our evidence is therefore consistent with but not confirming of our theory that the mental framing of spending presented by graphic representations can help overcome confirmation biases or situational attributions. More systematic qualitative analyses could trace the relevance of these processes and explore the associations between aspects of those processes and increased thought-act alignment. Finally, a stronger connection between this work and the traditional accounting view of good spending, where acts and oughts align, should be made. We did include oughts in our study by having participants set a budget at the beginning of their participation, but we did not have a reliable way of assessing whether that initial budget accurately represented participants' intended spending goals, especially as those goals evolved in response to participants' changing awareness of their actual spending patterns. Though these limitations present major constraints on the generalizability of these results and any related recommendations on mitigating the negative consequences of overspending, the pilot study has served its purpose in producing data and results that point toward the value of future research on the role of information visualization in changing the relationships between oughts, thoughts, and acts. The present study may therefore be seen as a preliminary empirical step toward bringing the body of work on visual perspective, mental representation, and behavior to bear on the question of how to nudge people toward bringing their intentions, estimations, and actions into better, more productive alignment.
References
Bruny, T.T., Mahoney, C.R., Gardony, A.L., & Taylor, H.A. (2010). North is up(hill): Route planning heuristics in real-world environments. Memory & Cognition, 38(6), 700-712. Card, S., MacKinlay, J., & Shneiderman, B. (1999). Readings in information visualization: Using vision to think. San Francisco, CA: Morgan Kaufmann. Drentea, P. (2000). Age, debt, and anxiety. Journal of Health and Social Behavior, 41(4), 437-350. Dunbar, K. (2000). How scientists think in the real world: Implications for science education. Journal of Applied Developmental Psychology, 21(1), 49-58. Frank, M.G., & Gilovich, T. (1989). Effect of memory perspective on retrospective causal attributions. Journal of Personality and Social Psychology, 57, 399403. Greenspan, A. (2004). The mortgage market and consumer debt. Americas Community Bankers Annual Convention, Washington, D.C. 19 October 2004. http://www.federalreserve.gov/boarddocs/speeches/2004/20041019/default.htm Jones, E.E., & Nisbett, R.E. (1971). The actor and the observer: Divergent perceptions of the causes of behavior. In E.E. Jones, D.E. Kanouse, H.H. Kelley, R.E. Nisbett, S. Valins, & B. Weiner (Eds.), Attribution: Perceiving the causes of behavior (pp. 7994). New York: General Learning Press. Kahneman, D., & Tversky, A. (1984). Choices, values, and frames. American Psychologist, 39, 341-350. Kahneman, D., & Tversky, A. (2000). Choices, values, and frames. New York: Cambridge University Press. Keil, F.C. (2008). Getting to the truth: Grounding incomplete knowledge. Brooklyn Law Review, 73 (3), 10351052. Libby, L.K., Shaeffer, E.M., Eibach, R.P., & Slemmer, J.A. (2007). Picture yourself at the polls: Visual perspective in mental imagery affects self-perception and behavior. Psychological Science, 18(3), 199203. Lurie, N.H. (2004). Decision making in information-rich environments: The role of information structure. Journal of Consumer Research, 30, 473-486. Lurie, N.H., & Mason, C.H. (2007). Visual representation: Implications for decision making. Journal of Marketing, 71, 160-177. Nickerson, R., Perkins, D., & Smith, E. (1985). The teaching of thinking. Hillsdale, NJ: Lawrence Erlbaum. Redelmeier, D.A., & Tversky, A. (1992). On the framing of multiple prospects. Psychological Science, 3(3), 191-193. Wolford, G., Newman, S.E., Miller, M.B., & Wig, G.S. (2004). Searching for patterns in random sequences. Canadian Journal of Experimental Psychology, 58(4), 221-228.
Acknowledgments
This study was supported in part by a grant from the Tufts University Faculty Research Awards Committee. The authors thank Thomas C. Mann, Jason Tsai, and Vivien Lim for their assistance in conducting this research, and Remco Chang, Sara Su, Fernanda Vigas, and Martin Wattenberg for their early advice and introduction to the field of information visualization.
ISLS
142
ICLS2012
Beyond the screen: game-based learning as a nexus of identification

Ben DeVane, University of Florida, P.O.Box 115810, 101 Norman Gym, Gainesville, Florida 32611-5810, ben@digitalworlds.ufl.edu Abstract: Drawing on data from a four-year ethnographic research study of CivWorld, a game-based learning community, this paper argues that moments of identity enactment (e.g. a conversational performance of a students knowledge) result in part from the nexus of longterm projects of identity formation (e.g. the project of forming an identity as a student, a teen, a brother). It uses positioning analysis to examine the relationship between moments of identity enactment in social learning communities and temporally longer trajectories of identification. It does so by examining the trajectory of identification (and by extension the arc of participation) of an eight year-old CivWorld participant named Salim as he moves from peripheral towards more full participation in the community, moving from a novice in history and game-play toward becoming an expert over a six month period.
Introduction
It is hard to overstate the extent to which research in the learning sciences, specifically in the field of games and learning, have increased its focus on identity as an important mediator of learning and cognitive processes. Identity has become an increasingly central construct in learning sciences that ties learning in context to longterm knowledge and practices (Gee, 2000; Nasir, 2002; Wortham, 2006; Enyedy et al., 2006). ). Within the learning sciences, scholarship on games and learning has embraced this new analytic direction, emphasizing the ability of games and virtual worlds to recruit players into new identities and knowledgeable practices (Gee, 2003; Squire & Barab, 2004; Steinkuehler, 2004; Barab et al., 2005; Shaffer, 2005). Yet these still exists no clear consensus regarding which descriptive model of identity is widely accepted, or even what the major frameworks for understanding identity. In particular, there exists no clear understanding of how to connect moments of identity enactment to long-term trajectories of identity formation as learner in a given knowledge domain. In the context of a four-year ethnographic research study of CivWorld, a game-based learning community, this paper argues that moments of identity enactment (e.g. a conversational performance of a students knowledge) result in part from the nexus of long-term projects of identity formation (e.g. the project of forming an identity as a student, a teen, a brother). Furthermore, it contends that these moments of identity enactment in learning communities must be understood as being in dialogue with temporally longer trajectories of identification. It does so by examining the trajectory of identification (and by extension the arc of participation) of a young CivWorld participant named Salim (a pseudonym) as he moves from peripheral towards more full participation in the community, moving from a novice in history and game-play toward becoming an expert over a six month period.
Literature Review
A major family of frameworks understands identity to be driven by social interaction the socially-contingent roles, contexts and macro-structures in which a person exists in their day to day life (see Figure 1). Influenced by linguistic anthropology (Sapir, 1921; Hymes, 1964), pragmatism (Mead, 1934) and interactionist sociology (Goffman, 1956, Blumer, 1962), sociolinguistic research in the post-war period increasingly showed that peoples social, ethnic and gender identities to greatly influence how they speak, act and think in the world (Labov, 1972, Gumperz & Hymes, 1972; Scollon & Scollon, 1981; Gee, 1989). While previous research sociological and anthropological research had previously taken gender, ethnicity and class as given pramaters and boundaries within which we [humans] create our own social identities, new research found language as interactional discourse demonstrates that these parameters are not constants that can be taken for granted but are communicatively produced (Gumperz & Cook-Gumperz, 1982, pg. 1). Language and identity, in this understanding, are emergent phenomena that mediate each other to some extent, while maintaining some continuity across contexts (Royce, 1982; Le Page & Tabouret-Keller, 1985).
ISLS
143
ICLS2012
Figure 1 - Major frameworks of identity While applied, design-oriented games and learning research has for the most part eschewed a focus on identity, some research has emphasized the importance of identity in learning with games. This research has adopted two general ways of understanding identity relative to the design of a game or game-based learning community. First, it has sought to understand how the design of a learning environment shapes, but does not determine, the way enact different identities within game-related spaces (Gee, 2003; Steinkuehler, 2006). Gees (2003) theory of projective identity has been influential in this regard, but it explicitly restricts its focus to the identity work that occurs between a single player and the semiotic-activity space of a single-player role-playing game. Second, this research has examined the long-term, overarching identity transformations that long-term participants in game-based learning communities experience over the course of moving from periphery to full participation in the community (Squire et al., 2005; 2008; see also Barab et al., 2007). These two strands of research have cumulatively generated a tremendous amount of insight into a) how identity works in different genres of commercial video games, and b) how game-based learning environments transform players identities (and attendant knowledge and skills) into those of experts. However, there still exist significant problems yet to be resolved on the topic. For an analytic construct that remains at the heart of analyses of games and learning, however, the theory of identity in games and learning scholarship has not been explicated well in much of the research literature.. If a major affordance of games and other new media learning environments is the ability to perform significant identity work, then the successful design of game-based learning communities should theoretically depend on explicit understandings of how to engineer both day-to-day engagement via identity resources and overarching transformations of how a person sees themself in terms of the relevant knowledge or activities around which the community is centered. For a game-based learning community to continually spark a young persons interest and engage them in a long-term process of identity transformation, it must have a certain compelling local alignment of social identity resources at the proximate level of activity.
Research Background
This paper is part of a larger study of a four-year long research project focused on creating, sustaining and studying a game-based learning community, CivWorld, that centered on a curriculum created using researchermodified versions of the Civilization historical strategy game series. The program located at an after-school center in a medium-sized Midwestern city, which had both weekly after-school and daily summer programs, of which CivWorld was one, for young people who were mostly aged nine to fourteen. Attendees almost entirely came from working-class or working-poor families and a eighty-percent of were African-American. Participants both learned about history and geography through custom designed game modules, and learned about information technology and game design by redesigning the game themselves This paper focuses on the twin trajectories of participation and identification of young man named Salim, who was Salim was an eight year-old African American male when he started in the CivWorld program in the fall of 2005. With one older brother and one older sister, he is the youngest child in his family, and also
ISLS
144
ICLS2012
the only child in his immigrant family that was born in the United States. Salim was introduced to the CivWorld program through his brother Malik, who at age 12 was his oldest sibling. It draws on some 70 hours of audiovideo data over the course of 6 months of game play to examine changes in Salims patterns of participation as he moved from novice to expert.
Methodology
But what methods do we use to understand the interaction of identity processes across and at multiple timescales both long-term and short-term and through multiple contexts? This study does not presume that social identities and practices are entirely incontiguous over time and context. Rather, it understands them to be continuous to a degree, while retaining a varying degree of specificity, dynamism and fluidity. As such, the analysis proposed here will not simply consist of a description of interaction (through language and practice) in these bounded spatial-temporal contexts, but rather attempt to understand it in terms of the overall arc of activity and identity work in the community. Methodologically, it will attempt to accomplish this task by employing an ethnographic perspective on positioning analysis (Gee & Green, 1998; Davies & Harre, 1990; Harre & van Langenhove, 1991). Using the framework of positioning (Davies & Harre, 1990; Harre & van Langenhove, 1991) it examines how one participants socially-situated identity is jointly constructed and negotiated in a game-based learning community. However, it extends positioning methods by looking at home the patterns of positioning for one individual change across time, and how modes of positioning appear in one context from others. Positioning analysis examines how a persons attributes their character, status and role and their location within the structure of rights of social participation are continually established and negotiated in social interaction (Davies & Harre, 1990; Harre & van Langenhove, 1991). This dialogic negotiation of personhood through interaction speaks volumes about their situated social identity in an affinity space. Identity, then, is repeatedly negotiated as people are re-positioned within different modes of social interaction like conversation and narrative. The specifications of ones position, and their social identity, are not unilateral or static. Rather a persons positioning and identity are both fluid and dynamic across social contexts, in conversations with different people, and even within conversations as dynamics change and shift. These ways of locating or specifying a kind of person exist in reciprocal relationship with the ongoing dynamic events of an interaction and longer social customs. Wortham (2006) examines broader, sociohistorical models of identification in terms of how they manifest in specific events of identification. Wortham does so by studying how broader identity tropes (e.g. poor student, rowdy student) more precisely called metapragmatic models of identity are assigned to students over the course of one full school year. Such an event-driven analysis provides Wortham with the analytic power that Erickson located in close empirical descriptions of events while maintaining an emphasis on changes in trajectories across time. However, Wortham, perhaps wisely, mostly emphasizes the events and trajectories of identification that occur within the classroom in his analysis. How might such an event-driven analysis be extended to consider the trajectories of identification that students bring with them from outside the walls of their learning environment? This paper considers precisely this question, as it attempt to understand how learners everyday trajectories of identification, many of which originate outside the learning environment, inform the events of identification the enactments of identity that occur within the learning environment. For the purposes of this data analysis, there are two important classes of positioning that are relevant to the social identity work that takes place in the interaction below: first-order and second-order positioning. These two types of positioning refer directly to the moral and power assumptions that are embedded in language. First-order positioning, for instance, refers to the assumptions of status in conversation using imperatives. In conversation, commands and other imperatives signal that the addresser feels that the addressee is obliged to him or her in some way. When a person can order another to do a thing without yielding a rationale, such an utterance suggests a certain manner or positioning between the two parties. However, such first-order positioning breaks down when an addressee refuses a command or requests an explanation. Once one conversant starts questioning why they should perform the requested task, the interaction moves into the domain of second-order positioning, and the moral order has to be justified by the conversant who initiated it. Second-order positioning connotes a more egalitarian status relationship between two conversants, as one must compel the other through rhetoric rather than decree.
Results
Maliks enthusiasm for the Civilization game series, and the CivWorld program, had gotten Salim interested in the program. Salim faced a two major obstacles related to identification upon entering the program. First, he was by far the youngest player among the participants, and he was entering a social cohort that had mostly coalesced during the summer months. The vast majority of participants in the fall program were working-class African-American young men, who seemed to enjoy exchanging competitive, but mostly good-natured, taunts and boasts. Salim, shy and deferential, was socially ostracized by the other older (ages ten to twelve) male
ISLS
145
ICLS2012
participants in the program, which often resulted in him being subjected to severe punitive competition, rather than supportive collaboration, by these older youth as he sought to learn the game.. Second, the program organized mostly competitive and multiplayer game sessions, and Salim had to learn the basics of how to play a very hard game that can take hundreds of hours to master while almost all the other participants his potential in-game rivals had learned or learn basic game mechanics and strategies over the summer.
Interaction 1: Salim as novice and outsider

Salims brother, Malik, entered the computer lab in which the program was housed. Malik was friends with both Michael and Jason, and regularly hung out with them outside of the program. He was also friendly with Darius and Betti, who were both in his grade-level, from school. Tacitly recognized as one of the more advanced players, Malik held significant social status in the CivWorld community. While Malik exchanged hellos with Darius, Jason and Michael, he ignored Salim in line 25, who was clearly anticipating being greeted. He did not seem intentionally or purposely ignoring Salim, but rather just excited to see his friends. Regardless, Salim, who had twisted around entirely in his seat and watched Malik as he entered, presumably in an attempt to catch Maliks eye and say hello, seemed disappointed that he had been ignored. Salim continued to negotiate with Malik and Salim in an effort to get Malik to join the multiplayer game he was playing, and specifically to determine if Malik could play as America. Malik persisted in declining to join the game, and Lawrence had to inform Salim twice that it was impossible for anyone to join the game as America. Salim quickly became the center of attention, however, when Malik asked who was winning the multiplayer game. 43/ (0:22:15.5) L: [No:::o 44/ M: I'm playing Egypt. 45/ Ma: Who's - who's WINNING?] 46/ (0:22:17.6) M: [U::uh let's see who's winning? 47/ Ma: ((walks to Salim and leans over him to look at screen))] 48/ (0:22:18.9) Ma: >Who are you?< 49/ (0:22:19.6) Ma: You're GREECE? 50/ (0:22:21.3) Ma: [DA:::AANG! see Salim? ((pats Salim on the top of his head affectionately)) 51/ M: ((turns and watches Malik and Salim)) 52/ (0:22:23.5) Ma: Good job Salim! Good job! ((continues patting Salim on the head)) 53/ (0:22:25.2) Ma: You're Greece. 54/ (0:22:26.5) Ma: ((grinning, walks to an adjacent and jostles it excitedly)) 55/ S: ((alternates between grinning and stiffling a grin)) After Malik became interested in who was winning the game, he glanced at the scoredboard on one of the computer screens. Seeing that Greece was in the lead, he asked who was playing as Greece, then looked over at Salims screen and saw that it was him. Despite previously having minimal interaction with his brother, and then only interactions that had been initiated by Salim, Malik quickly shifted his attention to Salim and Salims game. Getting very excited that Salim was at the top of the scoreboard, Malik excitedly congratulated him on his performance thus far in the game. As Salim grinned widely and Jason and Marcus turned to watch, Malik repeatedly and animatedly praised Salim for being in the lead. Malik emphasized how impressed he was with Salim by raising the volume of his voice, over-stressing words, and over-extending exclamations. While his attempts to enter into conversation earlier were not embraced by his brother, Salims in-game performance attracted significant notice and affection. Immediately after Malik congratulated Salim, Jason and Michael, both friends of Malik, conspired to attack Salim. Perhaps they were threatened by the attention Salim was receiving from Malik, or perhaps they were worried that Salims in-game successes might translate into broader recognition within the CivWorld community, but the immediacy of their response after Maliks loud praise of Salim suggests a causal connection between the two. Over the remaining hour and fifteen minutes left in the program session that day, Michael and Jason focused their in-game energies on attacking Salim. During this time, they periodically erupted with another episode of loud taunts directed toward Salim. Salim, conversely, for the most part did not respond to Michael and Jason, but directed his attention toward fending off their in-game attacks. Eventually, with five minutes left in the game, Salim did experience a significant loss his capital city was captured by Jason. While Salim seemed disappointed by this development, he did not seem frustrated or despondent, and actually appeared to be proud that he had defended his civilization against both Michael and Jason for such a long time.
After interaction: enforcing Salims positioning
ISLS
146
ICLS2012
Maliks arrival set off a series of changes in the discourse of the space that are important for understanding Salims identity relative to the CivWorld program in terms of both his relationship towards the activity of game play in the space as well as the social space around it. Two themes emerged in the continuities and discontinuities of the groups discursive practices that took place immediately before and immediately after Malik arrived at the program. First, Salim was recognized by his brother in a way that he had not been prior to his arrival. Second, the way in which Michael and Jason positioned Salim changed significantly and swiftly as a result of this recognition. 1) Maliks recognition of Salim - Sometime, but not immediately, after his arrival, Malik excitedly recognized Salims in-game achievements, which had the implicit effect of elevating, or opening up the possibility of elevation, of Salims social status role in the group. After ignoring Salim when he entered the program room and began greeting all his friends, Malik later changed course and began excitedly praising Salim and drawing attention to his in-game achievements. Far from the illocutionary marginal status that was implicit in Maliks greeting ritual that ignored Salim, this sincerely excited and loud recognition of Salims achievements put Salim at the center of the groups attention. In fact, Malik, who was generally recognized as one of the most central members of the group and one its best Civilization players, suddenly appeared to be able to barely restrain his excitement and affection for his brother because of his in-game achievements. He animatedly rubbed Salims head, patted his shoulders, and jostled neighboring chairs as he expressed his astonishment and pride that his brother was doing so well in his game play. His praise had the effect of according Salim a significant measure of recognition in this context for his nascent identities as a Civilization player specifically and as a member of the CivWorld community generally. It also had the effect of raising the possibility that Salims temporarily elevated positiong in these two local status roles Salims reflexive and context-dependent status of membership in the group might translate into more permanently raised social status roles more lasting recognition and status in the group (Kiesling, 2001). 2) Enforcing Salims positioning. Immediately following Maliks recognition of Salims in-game achievements, Michael and Jason undertook a campaign to re-enforce Salims status as a newer, peripheral member of the group through both in-game attacks and social positioning work outside the game. As Jason and Michael attempted to re-stabilize Salims peripheral status in the group, their speech acts and choices in language evidenced their underlying goal. As they gloated to Salim that they were about to attack him, Michael and Jason fell into patterns of discourse that were simultaneously similar and dissimilar from those they had used earlier. As he urged Jason to attack Salim, Michael again adopted an imperative discursive frame, again asserting his authority, confidence and certainty within the social group. Through this positioning work, Michaels implicit message was that Salims identity was that of a social outsider, and a novice to game play.
Interaction 2: Salim and membership

Salim became very involved in the CivWorld program in the four and a half months between Interaction 1 and Interaction 2. Accompanied by his brother, he attended the program every single week and became progressively better and better at playing the game. During the five program weekly sessions after Interaction 1, he chose to play the Greek civilization that he had played during Interaction 1 each time. Fascinated by Greece, he began bringing to the program childrens history books that he checked out from his school library, which kept in his lap or on the desk beside him while playing. He talked regularly to Lawrence, an adult facilitator in the program, about Greece in antiquity. He became particularly interested in hoplites, the citizen-soldiers of ancient Greece, because they were represented in the game as a defensive special unit for Greece. He reported that at home, he and Malik took turns watching each other play the game, and Salim often talked to his brother while Malik was making game mods. Salim, in short, became a very engaged and interested participant in the CivWorld club, and took that interest into other parts of his life. The data in Interaction 1 comes from a program session that ran four and a half months after the data in Interaction 2. In this interaction, Salim was waiting with Darius, Malik and Jason to again start playing a multiplayer Civilization III game. Adult facilitators Lawrence, Karl and Brian worked to overcome technical problems to load a saved game session from the previous week that the young participants wanted to play. During a past multiplayer game, Salim, playing as Rome, had suddenly taken the lead. Salim, followed by Malik, walked toward the corner where Darius and Jason were sitting: 24/ (0:16:50.0) M: Salim! ((puts hand on Salim's shoulder)) 25/ (0:16:52.1) M: Rome is gonna die today. 26/ (0:16:54.1) S: No all you [guys are gonna die 27/ (0:16:54.8) J: Yes it is.] 28/ (0:16:55.4) M: Rome is gonna die today. 29/ (0:16:57.0) S: [No:::o? 30/ (0:16:57.3) D: It'll be all of] us put [together versus you, and you're gonna win? 31/ (0:16:58.5) M: Rome- (.) Rome- (.)] ((pats Salim to get attention))
ISLS
147
ICLS2012
32/ (0:17:00.7) K: WHO's- WHO's winning right now? 33/ (0:17:02.4) M: Right now he's [winning 34/ (0:17:02.8) S: I'm] winning. 35/ (0:17:03.2) M: He's the one who cheats. 36/ (0:17:05.5) D: Yup! 37/ (0:17:07.1) S: NO I'm winning. Again in this interaction, Salim became the target of game-related threats and taunts from his brothers circle of friends, but this time it was instigated by Malik himself. Salim, however, seemed to become engaged by the taunt in line 26, treating it as a social game itself. As such, he returned the taunt and declared that it was his three interlocutors that were going to lose in the game. Jason and Darius join Malik in taunting Salim, with all three predicting his doom. Darius, however, instead of declaring by fiat that Salim was going to be beaten, he justified his prediction and challenged Salim to provide a rational. In joining the chorus predicting Salims doom in line 30, he interrogated Salims self-confidence by asking him how Salim was going to defeat his three opponents at the same time. Still, Salim did not seem to mind one bit, and seemed very engaged commanding a war against Brian alongside his brother. Jason, who was presumably allied against Malik, seemed to follow a different trajectory through the game even though he was nominally part of the Malik and Salims alliance. While he provided material support to Malik and Salim in-game, and verbal support out-of-game, Jason never directly attacked Brian it was a rare occurrence that Jason did not participate in a war. Salim and Maliks war against Brian devolved into a statemate, but Salim seemed to be enjoying himself thoroughly the entire time as he and Malik sought to find a way to use the numerical superiority of their armies to overcome Brians defenses.
Analysis 2:
In this social warm-up to the actual multiplayer game that was characterized by braggadocio and smack talk, Salim performed a very different version of himself than he had in prior interactions that speaks to the changes in identification toward the social space surrounding CivWorl that he had undergone in the previous four months. Salim had become much more active and willing to try to define his own positioning in an interaction with older youth, even as they contested just as much as they had in the previous interaction. Throughout the interaction above, Salim contested the way in which Jason, Malik and Darius tried to negative position him in conversation. While Salim showed a growing competency in his ability to reposition himself, Malik, Darius and Jason demonstrated that he was far from mastery beginning in line 24. As they anticipated playing the game, Malik let Salim know that his civilization was going to perish and implied that Malik would be responsible at least in part. While Salim protested in earnest, Jason added his voice in agreement to Maliks in line 27, declaring that Salims civilization would indeed die. In these first lines, both Malik and Jason adopted a declarative frame toward Salim his civilization was inevitably going to die. Salim, in turn, responded in the declarative frame by telling them that it would not in lines 26 and 29. Darius then shifted the interaction into an explanatory frame in line 30 by asking Salim to justify why he thought he could defeat three opponents at once. Perhaps because he thought the older youth were ganging up on Salim unfairly, or perhaps just because he was curious, Karl, an adult facilitator, seized upon this explanatory frame to ask who was currently winning the game in terms of points in line 32. When both Salim and Malik answered that Salim was winning, Salim was provided with an argument to respond to Darius rhetorical question he was winning already, so why could not he maintain that advantage. Salims ways of identifying, and negotiating how others identified him, himself socially in the space had changed significantly over the course of a four and a half month period. As the data presented in Interaction 2 demonstrates, Salim had developed some basic competencies in new ways of interacting and positioning himself as well as new forms of talk and identification through language. These new identity-related competencies were linked to larger ongoing project of identification with and identity formation in the CivWorld social space. Salim, in other words, was developing the competencies in order to be considered a full member of the highly social community of practice in the CivWorld program.
Epilogue: Salims trajectory of participation

Over the next six months, Salim continued his trajectory toward more full participation in the CivWorld community. Over this time, he became more and more proficient at the social and game-based activities that were valued in the Discourse of the community. Salim became much more knowledgeable and proficient in all aspects of the game, but he became recognized for his ability to achieve a high score in the game without resorting to war or conquest. Specifically, Salim became noted for his ability to achieve cultural and wonder-based victories in which he advanced his civilization through the production of culture and cultural institutions, or through the construction of World Wonders for which the game accords point based on the civilizations resulting increase in stature. During this time, Salim, following his brothers lead, began designing rudimentary game scenarios and modifications. Salim was notable for designing game scenarios that
ISLS
148
ICLS2012
did not revolve around conflict, but rather around diplomatic and cultural acumen. Both Salim and his brother continued their practice of taking turns at home playing the game and spectating as the other played. They began taking part in ad hoc multiplayer games organized on the Internet, and took pride in boasting of their (mostly Maliks) online victories during the weekly meetings at the CivWorld program.
Discussion
As he moved from peripheral to full participation, Salim developed intermediate identities as a competent (and then expert) Civilization game player and a recognized member of the clubs social community. These intermediate identities were in constant dialogue with both the emergent and reflexive identity positioning work (Davies & Harre, 1990) that occurred at the local level of interaction and the longitudinal identification projects (as a brother, as a masculine pre-adolescent, as a student) that Salim carried with him. Identity at the local level was negotiated between Salims positioning at different moments in game play and his concomitant social positioning in the group. At the same time, the social positioning work, in particular, was heavily influenced by long-term identification projects. The form that Salims positioning work took changed quite a bit depending upon the long-term positioning project that was foregrounded by the situated and reflexive circumstances of interaction. When Salim was foregrounding his nascent long-term identity as a masculine adolescent, the social work he performed to position himself was far more aggressive, just as the way Jason, Michael, Malik and others tried to position him was far more forceful when they were foregrounding that particular project. Similarly, when Salim placed his identity-as-a-brother long-term project at the forefront of his identity performance in Interactions 1 and 2, the way he positioned himself was again very distinct. When acting first as a brother, Salim oriented his actions around his perception of his brothers needs trying to help him join the multiplayer game in Interaction 1, and trying to prove his acumen to him in Interaction 2. In those moments, his self-positioning work became supportive and/or inquisitive, instead of the positioning continuum between quiet and aggressive that he inhabited at other times. As such, although the intermediate level was the primary level of identification processes in dialogue with the local positioning work, larger and longitudinal processes were influential as well. To be sure, he underwent powerful identity transformations that were immediately related to the relevant learning game and game-based learning community as he transformed from a user to a designer, from a novice to an expert, and from an outside to a old-time participant. However, these more immediate paths of identity formations were inextricably and dialogically connected to both long-term identity trajectories that were not an immediate result of the game and influential short-term identity positioning work that occurred on a local timescale. These different identity processes and trajectories intersected in the nexus of identification that was the CivWorld program. For Salim, they were very tightly bound up in a mesh of identification that resulted in the change in his CivWorld-related identities - his dispositions, capacities and evaluative stances related to his social membership and expertise in the gaming club. There was no one identity that was transferred, unfettered, to Salim by his participation in the program, but rather a number of identities that were enmeshed in Salims long-term trajectories of identification.
Conclusion
The central issue in question is the temporal, spatial and social scale at which learning scientists should examine processes of socialization and identification. Researchers favoring a broad, long-term scale argue that research focused only on the short-term, situated contexts fails to relate what happens in those contexts back to historical trajectories and overarching social structures which in their view is what matters. Conversely, scholars favoring a focus on short-term and situated social contexts argue that a focus on large scales deprive individuals of agency and fail to understand the nuances and fluidity in everyday social life. As an alternative to this bifurcation of both the short-term from the long-term, and enactment from formation, a practical science of identity and learning must not, in an effort to describe the relationship between enactment and formation, fall into the same totalizing trap underlying the bifurcation in the first place. It must eschew the implicit claim that its methods and analysis can describe and represent the totality of a social person, community or ecology, and instead explicitly acknowledge that its methods provide a lens to investigate that relationship that magnifies some parts and occludes others. As Bourdieu (1977) advised, it should embrace the limits of its objective exploration as a means of making clear what it is investigating.
ISLS
149
ICLS2012
References
Barab, S., Thomas, M., Dodge, T., Carteaux, R., & Tuzun, H. (2005). Making learning fun: Quest Atlantis, a game without guns. Educational Technology Research and Development, 53(1), 86-107. Barab, S., Dodge, T., Thomas, M., Jackson, C., & Tuzun, H. (2007). Our designs and the social agendas they carry. Journal of the Learning Sciences, 16(2), 263-305. Bauman, R., & Sherzer, J. (1975). The ethnography of speaking. Annual Review of Anthropology, 4(1), 95-119. Bourdieu, P. (1977). Outline of a Theory of Practice. Cambridge: Cambridge University Press. Ct, J., & Levine, C. (2002). Identity formation, agency, and culture: A social psychological synthesis. New York: Lawrence Erlbaum. Duranti, A. (1985). Sociocultural dimensions of discourse. Handbook of discourse analysis, 1, 193-230. Erickson, F. (1979). Mere ethnography: Some problems in its use in educational practice. Anthropology & Education Quarterly, 10(3), 182-188. Erickson, F., & Schultz, J. (1981). When Is a Context? Some Issues and Methods in the Analysis of Social Competence. Ethnography and language in educational settings, 147. Fairclough, N. (1995). Critical discourse analysis: the critical study of language. London: Longman. Gee, J.P. (2000). Identity as an Analytic Lens for Research in Education. Review of Research in Education, 25, 99-125. Gee, J.P. & Green, J. (1998). Discourse analysis, learning, and social practice: A methodological study. Review of research in education, 23(1), 119-63. Goffman, E. (1959). The presentation of self in everyday life. New York: Doubleday Anchor. Gumperz, J., & Hymes, D. (Eds.). (1972). Directions in Sociolinguistics: The Ethnography of Communication. New York & London: Holt, Rinehart and Winston. Gumperz, J., & Cook-Gumperz, J. (1982). Introduction: Language and the communication of social identity. In J. Gumperz (Ed.), Language and social identity (pp. 1-21). Cambridge: Cambridge University Press. Harr, R., & Van Langenhove, L. (1991). Varieties of positioning. Journal for the Theory of Social Behaviour, 21(4), 393-407. Hymes, D. (1964). Directions in (Ethno') Linguistic Theory. American Anthropologist, 66(3), 6-56. Labov, W. (1972). Language in the Inner City: Studies in the Black English Vernacular. Philadelphia: University of Pennsylvania Press. Mead, G. (1934). Mind, self and society. Chicago: University of Chicago Press. Nasir, N. S., & Hand, V. (2008). From the Court to the Classroom: Opportunities for Engagement, Learning, and Identity in Basketball and Classroom Mathematics. Journal of the Learning Sciences, 17(2), 143179. Scollon, R. (2001). Mediated discourse: The nexus of practice. New York: Routledge. Van Langenhove, L., & Harr, R. (1999). Introducing positioning theory. Positioning theory: Moral contexts of intentional action, 14-31. Wierzbicka, A., & Goddard, C. (1997). Discourse and culture. In T. Van Dijk (Ed.), Discourse as social interaction (pp. 231-57). London: Sage. Wodak, R. (2001). The discourse-historical approach. In R. Wodak & M. Meyer (Eds.), Methods of critical discourse analysis (pp. 63-94). London: Sage. Wortham, S. (2006). Learning identity: The joint emergence of social identification and academic learning. New York: Cambridge University Press.
ISLS
150
ICLS2012
Working with Teenagers to Design Technology that Supports Learning about Energy in Informal Contexts
Katerina Avramides, Brock Craft, Rosemary Luckin, London Knowledge Lab, University of London, 23-29 Emerald Street, London WC1N 3QS, UK K.Avramides@ioe.ac.uk, B.Craft@ioe.ac.uk, R.Luckin@ioe.ac.uk Janet C Read, University of Central Lancashire, Preston, Lancashire, PR1 2HE, UK, JCRead@uclan.ac.uk Abstract: Energy sustainability is prevalent in political and popular rhetoric and yet energy consumption is rising. Teenagers are an important category of future energy consumers, but little is known about their conceptions. We report on research to design learning technologies that support teenagers learning about personal consumption of energy. In this paper we describe our analysis and methodology, which are shaped by the Ecology of Resources design framework (Luckin, 2010). Findings suggest teenagers are aware of energy issues at an abstract level, but do not apply this information in the context of their behaviour. We identify indirect energy use and the relative energy intensity of different behaviours as important areas for learning. For example, behaviours of particular relevance to teenagers are use of electronic devices, and choice of food and personal care products. These findings have implications for the design of technology to support learning about energy in informal contexts.
Introduction
Climate change and resource scarcity have brought energy sustainability to the forefront of environmental concerns. However, contrary to the need for adoption of energy sustainable lifestyles, energy consumption is rising and few people take measures to save energy (for UK trends see Whitmarsh, 2009). Indeed, few studies have found that consumers adopt energy saving behaviours over the long-term even when specifically targeted by intervention studies (see review by Abrahamse, Steg, Vlek & Rothengatter, 2005). Evidence also suggests people are unwilling to adopt behaviour changes that have a high impact on energy demand (Whitmarsh, 2009; Gatersleben, Steg & Vlek, 2002). An important factor in shaping future patterns of energy consumption is the attitudes and willingness to address the problem by younger generations. Little is known about teenagers conceptions of the issues around energy use and their attitudes towards the adoption of energy sustainable lifestyles (Wray-Lake, Flanagan & Osgood, 2010). However, the pattern of increase in energy consumption coupled with evidence implicating teenagers in increased energy household bills (Thgersen & Grnhj, 2010; BBC, 2006) suggests few are consciously acting to save energy. The reasons for this are likely to be complex. Behaviour is influenced by many factors, including knowledge, attitudes and concerns as well as contextual factors, such as habit and availability of alternatives (Stern, 2000; Whitmarsh, 2009; Gatersleben et al., 2002). Therefore, the lack of conservation behaviour may not necessarily reflect a lack of awareness or understanding. That said, there is evidence that teenagers concern about the environment and adoption of conservation behaviours has been declining (Wray-Lake et al., 2010). Also, studies with adults suggest there is both misunderstanding of the energy intensity of different behaviours as well as a more general lack of awareness regarding the connection between environmental issues that arise from energy use and individual consumer choices (Lorenzoni, Nicholson-Cole & Whitmarsh, 2007; Crompton & Thgersen, 2009; Steg, 2008; Chauhan, Rama das, Haigh & Rita, 2010). This disconnect between individual actions and larger scale problems, and the lack of awareness of how one might act to make a difference suggests the need to link learning about energy and energy related environmental problems with everyday concerns, motivations, and choices. For teenagers to change their energy behaviours they must be motivated to engage with energy related problems in the context of their everyday lives and also helped to develop the skills to explore the impact of their behaviour. In this paper, we report on ongoing work to design learning technologies that support learning about energy in the informal contexts in which teenagers might make energy conscious choices. Our work contributes to research within the learning sciences community to design technologies that support scientific learning in informal contexts (Zimmerman, et al. 2010; Clegg, Gardner & Kolodner, 2011). Specifically, we discuss the application of a methodology for designing learning technology that identifies the learners context and draws connections between their circumstances, motivations, attitudes and knowledge.
Consumer Awareness of Energy Consumption

Energy consumption has been increasing over recent years (Abrahamse et al., 2005). It is estimated that households are responsible for between 15 and 20% of national energy consumption in western countries (Steg, 2008; Abrahamse et al., 2005). Hillman (2004, reported in Lorenzoni et al., 2007) reports individuals contribution to energy demand in the UK, including household and transport, at 51% of national energy use. Of
ISLS
151
ICLS2012
this, estimates show around half is due to direct energy consumption, such as house heating, water heating, cooking and lighting, and the other half to indirect energy demand through consumption of products and services (Reinger, Vringer & Blok, 2003). Supporting individuals to adopt energy saving behaviour, therefore, can potentially have a significant impact on national energy demand. The majority of research on attitudes towards energy and adoption of energy saving behaviour has focused on adult consumers. In terms of awareness, these studies suggest people are concerned about the environment, but do not see climate change as an immediate threat and if they do save energy it is mainly for financial reasons (Lorenzoni et al., 2007). There is widespread lack of awareness of the behaviours that impact on energy demand. For example, many people report they adopt recycling to tackle climate change, when in fact recycling has a low impact (Whitmarsh, 2009). Moreover, there is evidence that people are not aware of the energy intensity of different behaviours, such as heating water (Steg, 2008), and tend to attribute greater impact to those behaviours they have adopted over those they have not (Crompton & Thgersen, 2009). More generally, there appears to be a disconnect in understanding the source of environmental problems in individuals behaviour (Lorenzoni et al., 2007). Of greater concern are the findings that people are more willing to adopt changes in their behaviour that do not require much effort and change in lifestyle, such as recycling, but which have small impacts, and much less willing to adopt behaviours that have high impacts but are inconvenient, such as choosing alternatives to car travel (Lorenzoni et al., 2007; Crompton & Thgersen, 2009; Lindenberg & Steg, 2007). It is also reported that intervention programs that target behaviour changes have been less successful in changing high impact behaviours (Chauhan et al., 2010). We know little about teenagers attitudes and behaviours, we can however observe that current trends in energy consumption would suggest that few teenagers are engaging in high impact energy saving behaviour. It is also likely that their understanding of the energy intensity of different behaviours mirrors that of the adult participants as reported in the aforementioned research. Some studies that have involved teenagers have presented a negative picture of their contribution to energy consumption (Thgersen & Grnhj, 2010; BBC, 2006), and reported evidence of decline in environmental concerns (Wray-Lake et al., 2010).
Teenagers Role in Energy Saving

Teenagers are not the principal decision-makers within the household but they are well placed to have an impact, both by changing their individual behaviour and by acting to influence others within the family, their social network and the public sphere (Larsson, Andersson & Osbeck, 2010). Public campaigning has the potential for greater change, because the adoption of many energy saving behaviours is hindered by contextual factors that must be addressed by organizations and government (Stern, 2000). For example, the availability of alternatives to car travel will facilitate or inhibit their uptake. It is important that teenagers behaviour change related to energy use is coupled with knowledge and awareness and not achieved solely through other motivations. Knowledge is an essential addition to other factors, such as the financial and contextual influences that have been found to be strong determinants of energy saving behaviour (Stern, 2000; Abrahamse & Steg, 2011). Change that is based on knowledge, awareness and concern is likely to be more robust than changes in circumstances alone (Lindernber & Steg, 2007). By contrast, if change is based on reward, the moment the reward is removed or is no longer of significance, energy use will return to higher levels. Knowledge and awareness are also important because they are essential for influencing others. In order for teenagers to become effective agents for change, they must be informed about why energy is a relevant issue and about how people can act to mitigate the environmental problems that arise from current energy consumption. However, for the learning experience to be effective, information must be tailored to personal circumstances and made relevant to individuals behaviour and choices. The adaptability, flexibility, and portability of learning technologies makes them well placed to create learning experiences that are personalised and that embed information about energy and energy saving into teenagers contexts.
Framework for Learner Centred Design

Technology has been used to good effect to support science learning, particularly in collaborative settings and when the learners broader circumstances are taken into account (see for example, Pea, 2002; Puntambekar and Kolodner, 2005). A methodology that identifies the multiple influences upon teenage decisions and behaviours, and that draws connections between their circumstances, motivations, attitudes and knowledge is required for the design and use of technology to support learning about energy. This methodology needs to be participatory and to engage the teenagers in the design process in order to ensure that a clear and accurate understanding of the complex influences within their personal contexts is integrated within any resultant design. The Ecology of Resources design framework (EoR, Luckin, 2010) offers a process for working with participants that models and takes account of their context. The framework is inspired by a sociocultural philosophy to understanding and supporting learning (Vygotsky, 1978, 1986) and the notion of scaffolding, which defines the process of providing support that is closely contingent on the learners current understanding and skills (Wood, 1976, Luckin at al, 2011; Pea, 2004). The EoR provides a method for designing learning technology and/or learning
ISLS 152
ICLS2012
technology use that considers the important relationship between the learner in their context and the learning that arises out of their interactions with their context. The EoR conceptualizes a persons context in terms of their interactions with the multiple resources of their world, for example, with the people, places, books, knowledge concepts and technologies they encounter; and in terms of their personal resources, such as their motivation and existing understanding. In order to support learning the design process seeks to effectively link the world resources that are most appropriate for a specific learning goal and to scaffold interactions between these resources and the learner, taking into account the learners personal resources. The EoR provides a method through which we first identify the world resources available to the learner and the processes and relationships that shape the learners access to these. We also build an understanding of the learner and what they bring to the learning experience: their personal resources. The EoR also introduces the notion of filters to describe the artefacts that constrain a learners access to resources, such as rules, regulations or physical boundaries. Having mapped out the learners context we begin an iterative participatory process of design with the aim of developing technology that facilitates access to appropriate resources at appropriate times during the learning process. The EoR has been used with learners and teachers across a range of subjects, including science and language learning to design technology rich learning activities and technology applications, such as smart phone applications (Underwood et al, 2011). The EoR design process offers a 3 Phase structure through which educators and technologists can work together to develop technologies and technology-rich learning activities that take a learners wider context into account. In this paper our focus is on Phase 1 (see Luckin, 2010 for framework detail):: Create an Ecology of Resources Model to identify and organize potential resources for learning. Six iterative steps support the modeling process in which some steps will require completion multiple times. Step 2 is of particular importance and can require several iterations through other steps: Step 1 Brainstorming Potential World Resources Step 2 Specifying the Focus of Attention Step 3 Categorizing World Resource Elements into: Knowledge and Skills, People and Tools, Environment. Step 4 Identify potential Resource Filters Step 5 Identify the Learners Personal Resources Step 6 Identify potential More Able Partners (MAPs). Phase 2: Identify the relationships within and between the resources produced in Phase 1. Identify the extent to which these relationships meet a learners needs and how they might be optimized. Phase 3: Develop the Scaffolds and Adjustments to support learning and enable the negotiation of a Zone of Proximal Adjustment (ZPA) for a learner. Phase 3 of the framework is about identifying the possible ways in which the relationships identified in Phase 2 might best be supported or scaffolded. This support might for example be offered through the manner in which technology is introduced, used or designed.
Research Questions
The study reported here is part of ongoing research to design technologies that will support learning about energy issues and the energy intensity of individual behaviour and choices. The design of these learning technologies will be based on identification of: appropriate areas of focus for learning, an understanding of how to motivate teenagers to engage in learning about energy, and an understanding of how the technology can fit in with the learners context in order to make the link between the learning material and the (everyday) context in which it applies. To this end we need to identify and understand the resources available to teenagers: 1. Teenagers personal resources: conceptions of energy issues (what are the priority areas for learning?) a. How do they understand energy use and saving in their everyday life? b. How aware are they of their energy use? c. How much knowledge do they have about the energy intensity of their behaviour and choices? 2. Teenagers personal resources: concerns and motivations (how do we motivate teenagers to engage?) a. Are they concerned about energy issues? b. What are their attitudes towards energy saving? c. What motivates them to save energy? 3. World resources available to teenagers (how will the technology fit into teenagers ecologies of resources?) a. What energy uses are most relevant to them? b. What are the circumstances in which they use energy? c. What are their sources of information about energy and energy saving?
Study Design
Participants
Our participants were a group of 14 teenagers (aged 14-17; 5 female and 9 male) who were taking an IT course at an international school in the UK. Our sample is not representative of the teenage population. However, their
ISLS 153
ICLS2012
conceptions of energy use and motivations are very relevant. They come from families with higher incomes who are therefore likely to be greater consumers of energy (Abrahamse & Steg, 2011), both now and in the future. In subsequent studies we plan to compare our current findings with those from teenagers from state schools.
Method
The course teacher integrated our research on energy with a course project in which students documented and presented how they use energy and what they could do to save energy. The project was completed shortly after they participated in our study. This allowed us to work with the students over 3 sessions and also to gather data from a presentation of their formally assessed work. We used a combination of photo diaries and focus groups to explore the research questions. Photo diaries In our initial exploration of teenagers learning contexts we designed an activity that would be open-ended and not explicitly related to energy. We did not want to prescribe the kinds of things they identified as relevant to energy. Therefore, we gave participants the activity of creating a photo diary of a day in their lives. In preparation for this activity, we dedicated a separate session to the discussion of anonymity in taking photos in line with ethical constraints. We gave the opportunity to participants to practice taking photos that captured an activity but did not identify people. We did not provide reminders, for example to take a photo at set time intervals, because we were interested in what activities the participants would choose to document. Instead we asked them to document any change in activity in order to create a diary of a day in their life with the guideline to aim for 24 photos in a day. Although by necessity the consent forms did mention the energy focus of our research only two participants mentioned energy when discussing their photos and in fact the activities they documented in their diary were not limited to what activities they identified as being relevant to energy use in the focus groups. The session in which we discussed the photo diary activity and gave the participants the digital cameras was on a Friday. They were asked to complete the diary starting from Friday afternoon to Saturday in order to capture activities both in school and at the weekend. Focus groups The photos from the diaries were revisited in a subsequent session in order to translate them into a narrative. This session took place 2 days after the photo diaries were created. Participants worked in groups of 3-4 and took turns in creating their narrative by laying out the photos as a storyboard. Within the group each teenager was assigned a role of either interviewee (the person whose diary was being discussed), interviewer(s) or notetaker. The interviewer(s) were given the following questions to ask the interviewee in order to help create the storyboard: a) What were you doing when you took the photo? b) What else was going on around you? c) How long were you doing this activity for? and d) Why did this activity end? Once the narrative was complete we discussed the activities that were captured in the photos through the lens of energy consumption. In these discussions we explored teenagers awareness of energy-related issues, their attitudes and their behaviour. As before, each teenagers within the group was assigned roles of interviewee, interviewer and note-taker. For this discussion we provided the following questions: a) Is what you were doing related to energy use? If yes, then in what way is it related to energy? Where you conscious of how much energy you were using? Could you use less energy? Would you choose to use less energy? b) Are there times when you intentionally use less energy? If yes, when? c) Are you concerned about energy use? Formal presentation and small group discussion After the 3 sessions the students were required to complete a formally assessed project on ways in which they could reduce their energy use. They visited the university department to present their work to our colleagues. After the general post-presentation discussion we divided the participants into groups of 4-5, with one of the researchers leading the discussion in each group. In these small group discussions we used some of their photos from the photo diary activity as prompts to probe into their conceptions further. Specifically we focused on indirect use of energy, which we had identified from our analysis of the photo diaries and focus groups that they appeared to have less awareness of (photos of a bottle of water, filtered water, a zero-sugar soft drink, food (a burger), and clothes). The focus groups, presentation and small group discussions were audio recorded for later analysis.
Findings
We report on the first iteration of Phase 1 of the EoR process of design, which involves identifying the resources that make up the learners contexts in relation to energy use and forming an understanding of the learner and their personal resources. We have structured our findings based on the research questions. In the subsequent discussion we reflect on how these findings inform the identification of a focus for learning and motivation for teenagers to engage in learning. We analysed the data by an iterative process of identifying themes. We extracted those conceptions, motivations and context elements that recurred across participants and across data sources (photo diaries, focus group discussions, formal presentation, and small group discussion).
ISLS
154
ICLS2012
Teenagers personal resources: conceptions of energy issues

How do teenagers understand energy use and saving in their everyday life? The discussions of energy mostly concerned electricity and the use of electronic devices. This focus also came through in the participants formal presentation, where most focused exclusively on electricity. Only two participants mentioned other uses, that is, amount of hot water used to shower and the energy required to produce plastic water bottles and paper. This focus on direct energy use and lack of awareness of indirect energy use is interesting principally because when prompted about indirect energy use explicitly participants engaged in discussion about it. In the small group discussions (which we engaged in after the formal presentation), we prompted participants with selected photos of food, drink, and clothes, with the aim to elicit their conceptions around indirect energy use (as we had identified this as an area of low awareness from the photo diaries and focus groups). During the discussion participants discussed several issues including production, transportation, refrigeration (in the case of food and drink), packaging, and waste processing of products. When prompted further they also discussed the relative energy intensity of different foods, for example meat versus vegetarian products. This apparent distinction between the energy they use directly and the energy used indirectly in the products they consume also emerged in their focus on energy use that is solely due to their behaviour. In other words, shared use of electricity, for example, in the familys use of the dishwasher, or the use of classroom computers was only considered after prompting. For example, in the discussion of each photo diary we came across several pictures of food. The participants were prompted to consider what uses of energy are associated with, for example, their dinner, which led to a discussion about the use involved in dishwashers, microwave ovens, and refrigeration. Their ideas for saving energy were also limited to using less but not consuming different products or engaging in different activities. For example, switching off devices when not in use, or using their laptop less. It appears the concept of energy and energy saving is closely tied with not wasting more generally, and it is closely linked with not wasting other resources. For example, during the photo diary discussions many participants mentioned water (though not heating water) and compared not wasting electricity with not wasting water. When asked specifically about food in the small group discussions, the focus for many was again on waste. Similarly with clothes, the focus was on not throwing them away and handing them down to others. From the focus groups it emerged they are prompted by their parents to not waste electricity and water, and many are given their siblings clothes, and so it seems natural these are closely linked ideas. How aware are teenagers of their energy use? Almost all participants reported they were generally not aware of their energy use prior to engaging with us. Their teacher also confirmed the difference in their awareness over the course of the study. The only instance they discussed of being aware of energy use and acting to save energy was switching off the lights and music players when they left the room, as prompted to by their parents. We note that while they were able to discuss the different uses of energy, even indirect uses when asked explicitly about it, this information was not translated into awareness of energy use in their everyday life until we prompted them to reflect on it. How much knowledge do teenagers have about the energy intensity of their behaviour? The participants discussions of how they could save the most energy indicated little awareness of the energy intensity of different behaviours. They appeared to base their calculations of the amount of energy used by how long a device had been switched on. For example, because a fridge or phone is continuously in use it was assumed to use more energy. Discussions around indirect energy use suggested that, while they were aware of indirect energy requirements, they considered it negligible in comparison with energy used by lights or electronic devices.
Teenagers personal resources: concerns and motivations

Are teenagers concerned about energy issues? There was general agreement that energy related problems are important and relatively urgent issues. The problems discussed included climate change, resource scarcity, and animal habitat destruction. However, the participants did not express great concern. Some suggested problems would be resolved through improved technology, while others simply did not appear to have reflected on the issues and did not elaborate on their concern. What are teenagers attitudes towards energy saving? For the majority of participants the responsibility of alleviating the problems around energy use does not lie with them. Some were willing to take responsibility for individual actions and stated that each should lead by example. However, their perception was that this is not something they should be expected to take responsibility for. This finding is not surprising but it highlights an interesting issue in perception of responsibility more generally. For these teenagers the question of whether they should act appeared to be about it being unreasonable for us to expect them to act. It was not seen as a problem that affects them and, consequently, they might choose to do something about.
ISLS
155
ICLS2012
When we discussed their willingness to change their behaviour in order to save energy, they seemed very willing to act to prevent waste but not to modify their lifestyle. They also doubted that anyone else would be willing to change. Their focus was very much on limiting amount of use rather than finding alternatives. In relation to indirect energy use, as weve discussed they did not consider indirect energy use to have as great an impact, so did not discuss purchasing alternative products or services. When queried about what factors would affect their purchases their focus was overwhelmingly on money, but also on the impact on people. For example, they would not buy clothes that involved child labour. What motivates teenagers to save energy? None of the participants reported that they consciously change behaviour to save energy. The only motivation was their parents telling them to switch off the lights and electronic devices when not in use.
Teenagers contexts
Our focus in addressing this question was on the kind of activities the participants engage in and the choices they make. We analysed all the data (from the photo diaries, focus sessions, formal presentation, and small group discussions) by firstly identifying all the activities, contexts of use, people and products, secondly selecting the ones that occurred with the greatest frequency, and finally grouping these into the EoR categories: Knowledge and Skills, People and Tools, Environment. People and Tools: What energy artefacts are most relevant to teenagers? We identified three groups of energy artefacts that are most relevant to our participants: electronic devices, food, and personal care products. Although there was individual variation within each category, there were some common products across participants, such as laptops, phones, music players, drinks (energy drinks), and water (bottled, filtered, tap water). Environment: What are the environments in which teenagers use energy? The main environment that appeared in the photo diaries and formal presentation was the home. School and restaurants also featured in the photo diaries but were exceptions. Knowledge and Skills: What are teenagers sources of information about energy and energy saving? The main source of information the participants mentioned explicitly, and that was brought up in discussions of energy, were their parents. However, this was only in relation to not wasting energy. In terms of facts about energy use and energy intensity of behaviours some mentioned TV programs. For example, one participant mentioned a program about the amount of energy that can be saved through recycling. It is worth noting when information from TV programs was mentioned it was presented with uncertainty as something that they had heard but couldnt quite recall the details of.
Discussion
The study we report in this paper is part of ongoing research to support teenagers in learning about energy issues and the impact of their individual behaviour and choices. We engaged with teenagers using qualitative methods (photo diary, focus groups and formal project work) to identify important areas of focus for learning, to understand how to motivate teenagers to engage in learning about energy, and to explore how the technology can fit in with the learners context in order to make the link between the learning material and the (everyday) context in which it applies. Our work contributes to research within the ICLS community that is concerned with the design of learning technologies to support science learning in informal contexts (for example, Zimmerman et al., 2010). The design of learning technologies must tap into learners motivations and identify the multiple influences and resources within the learners context. The methodology needs to be participatory to ensure that a clear and accurate understanding of the learners personal contexts is integrated within any resultant design. The EoR methodology that we applied offers a process for working with participants that models and takes account of their context. We purposely used open ended tasks that did not focus on energy from the beginning. This was important in understanding the participants informal contexts as filtered by what they considered relevant to their lives and also acceptable for us to see. The data also confirmed the importance of an open ended method for exploring teenagers awareness in the context of their everyday life and differentiating awareness and knowledge in this context from the more formal knowledge they have about energy. For example, when we discussed their photo diaries indirect energy use was hardly touched upon. However, when we used some selected photos from the diaries to prompt them explicitly about indirect energy use related to food and water we generated a discussion around energy requirements for production, transportation, storage, and disposal. Had we only conducted a focus group around examples of indirect energy use we might have assumed this information would be readily available at all times. Our focus on the photo diaries also allowed us to centre the discussion around instances of energy use that were personally relevant and, therefore, probably more engaging. The combination of multiple sources of data was also important to our method. The photo diary and subsequent focus sessions allowed us to probe into the learners informal contexts and the nature of their conceptions within those contexts. The formal project work around participants energy use and ways they
ISLS
156
ICLS2012
could save energy gave us an insight into their formalized notions of energy use, especially as it was conducted after the diaries and focus groups. When we started the study they had not yet explored their energy use so were still at their pre-study awareness levels. When they began the formal project work they had already engaged in discussion around energy use with us. Therefore, it is reasonable to assume that what was distilled in their formal work represented core ideas (for example, the focus on direct energy over indirect, and their perception of the relative energy intensity of different activities). Motivating teenagers to engage with energy issues will be the biggest challenge. Although they report concerns, it seems the problems around energy use are too abstract for them to make a connection with their lives and the people and things they care about (as appears to be the case with adults, Lorenzoni et al., 2007). It will be critical to foster understanding of the impact of energy issues on people, both those close to them and those further away. Our data suggests that they are concerned about the impact of their choices on people. For example, it is possible the concern they reported about child labour in buying clothes might extend to concern about populations who are more affected by energy related problems. In terms of the knowledge the teenagers would bring to the learning experience, our findings on teenagers conceptions of and attitudes to energy use and energy saving largely echo those reported from research with adults (Lorenzoni et al., 2007; Crompton & Thgersen, 2009; Steg, 2008). Our participants were aware of energy issues at an abstract level, but this did not translate into an awareness of energy use in the context of their individual behaviour. Their conception of the relative energy intensity of different behaviours was not very accurate (for example, the assumption that the fridge uses a lot of energy because it is on all the time), and indirect energy use was largely invisible to them until prompted. It, therefore, appears that what is of critical importance for this group of teenagers is to raise their awareness of the energy intensity of their behaviour and choices. Especially the lack of awareness with regards to indirect energy use which is important given the fact that indirect energy use accounts for around half household energy use. Another reason to focus on indirect energy is that in our analysis we identified it as one of the most relevant to this group of teenagers, specifically in the consumption of food, water and personal care products. Also their focus on personal energy use suggests we need to make their contribution to shared (within the family) and public energy consumption more salient. More generally, the issues around energy might be best conveyed in conjunction with other environmental issues. The participants thinking about energy was not clearly differentiated from other resources, such as water. Moreover, the idea of saving was generally interpreted in terms of not wasting rather than changing behaviour to adopt less energy intensive alternatives. This is coupled with a willingness to cut back on waste but a reluctance to make changes that require sacrifice, such as not flying as frequently (which is consistent with adult research, Lorenzoni et al., 2007; Crompton & Thgersen, 2009; Lindenberg & Steg, 2007; Chauhan et al., 2010). This again suggests we must raise awareness of the impact of behaviours, for teenagers to be in a position to judge whether cutting down on waste is enough. It also suggests that it might be productive to frame energy reduction in general in terms of wastage. In terms of the sources of information in teenagers lives that learning technologies could tap into, parents seemed to have the greatest influence but this was limited to not wasting energy by switching off lights and electronic devices. In this group of teenagers at least, there appears to be scope for them to educate their parents. Other information from media was occasionally mentioned, but the uncertainty with which it was presented suggests that the learning technologies we design need to help teenagers capture this information and then actively try to reflect on the implications it has for their actions.
Study Limitations and Ongoing Work

We worked with a small number of participants, which allowed us to collect detailed data but limits the generalisability of our findings. However, the fact that our results largely replicate research reports with adults, suggests that the issues we have identified as those which are an important focus are applicable beyond our sample. We are continuing our work with the same group and engaging with a second group of teenagers. In the second cycle of Phase 1 of the EoR we are presenting and discussing our analysis of the data with the participants. Through this participatory process of involving the teenagers in the design, we will ensure that the learning technologies we develop will focus on the resources that are most relevant and engaging to the learners.
References
Abrahamse, W., & Steg, L. (2011). Factors Related to Household Energy Use and Intention to Reduce It: The Role of Psychological and Socio-Demographic Variables. Human Ecology Review, 18(1), 30-40. Abrahamse, W., Steg, L., Vlek, C., & Rothengatter, T. (2005). A review of intervention studies aimed a household energy conservation. Journal of Environmental Psychology, 25(3), 273-291. BBC (2006). Teenagers are standby villains. Available online at: http://news.bbc.co.uk/1/hi/scotland/6219862.stm (Accessed 1 November 2011).
ISLS
157
ICLS2012
Chauhan, S., Rama das, S., Haigh, M., & Rita, N. (2010). Awareness vs intentionality: exploring education for sustainable development in a British Hindu community. Sustainable Development. Available online: http://onlinelibrary.wiley.com/doi/10.1002/sd.494/abstract (accessed 7 October 2011). Clegg, T.L., Gardner, C.M. & Kolodner, J.L (2011). Technology for supporting learning in out-of-school learning environments. Proceedings of the International Conference of Computer Supported Collaborative Learning, Hong Kong. pp 248 255. Crompton, T., & Thgersen, J. (2009). Simple and painless? The limitations of spillover in environmental campaigning. Journal of Consumer Policy, 32, 141-163. Gardner, G. T., & Stern, P. C. (2002). Environmental Problems and Human Behavior. Boston: Pearson. Gatersleben, B., Steg, L., & Vlek, C. (2002). Measurement and Determinants of Environmentally Significant Consumer Behavior. Environment and Behavior, 34(3), 335-362. Larsson, B., Andersson, M., & Osbeck, C. (2010). Bringing Environmentalism Home: Childrens influence on family consumption in the Nordic countries and beyond. Childhood, 17(1), 129-147. Lindenberg, S., & Steg, L. (2007). Normative, Gain and Hedonic Goal Frames Guiding Environmental Behavior. Journal of Social Issues, 63(1), 117-137. Lorenzoni, I., Nicholsoncole, S., & Whitmarsh, L. (2007). Barriers perceived to engaging with climate change among the UK public and their policy implications. Global Environmental Change, 17(3-4), 445-459. Luckin, R. (2010). Re-designing Learning Contexts: Technology-Rich, Learner-Centred Ecologies. London and New York: Routledge. Luckin, R., Looi, C.K., Puntambekar, S., Stanton Fraser, D. Tabak, I. Underwood, J. & Chen W. (2011) Contextualizing the changing face of Scaffolding Research: Are we driving pedagogical theory development or avoiding it? Proceedings of the International Conference of Computer Supported Collaborative Learning, Hong Kong. pp 1037 - 1045. Pea, R.D. (2002) Learning science through collaborative visualization over the Internet. In Proceedings of the Nobel Symposium NS 120, pp. 1-13. Pea, R.D. (2004) The social and technological dimensions of scaffolding and related theoretical concepts for learning, education, and human activity Journal of the Learning Sciences, 13, 423-51. Puntambekar, S., & Kolodner, J. L. (2005). Distributed scaffolding: Helping students learn science by design. Journal of Research in Science Teaching, 42, (2), 185-217 Reinders, A. H. M. E., Vringer, K., & Blok, K. (2003). The direct and indirect energy requirement of households in the European Union. Energy Policy, 31(2), 139-153. Steg, L. (2008). Promoting household energy conservation. Energy Policy, 36, 4449-4453. Stern, P. C. (2000). Toward a Coherent Theory of Environmentally Significant Behavior. Journal of Social Issues, 56(3), 407-424. Thgersen, J., & Grnhj, A. (2010). Electricity saving in households: A social cognitive approach. Energy Policy, 38(12), 7732-7743 Underwood, J., Luckin, R. & Winters, N. Retelling Stories: Setting Learner Narratives in Resource Ecologies. In Proceedings of the International Conference of Computer Supported Collaborative Learning, Hong Long. pp 611 - 616. Vygotsky, L. S. (1978) Mind in Society: The Development of Higher Psychological Processes. Trans. Cole, M., John-Steiner, V., Scribner, S. & Souberman, E. Cambridge, MA, Harvard University Press. Vygotsky, L. S. (1986) Thought and Language. Cambridge, MA, MIT Press. Wood, D. J., Bruner, J. S. & Ross, G. (1976) The Role of Tutoring in Problem Solving. Journal of Child Psychology and Psychiatry, 17(2), 89-100. Whitmarsh, L. (2009). Behavioural responses to climate change: Asymmetry of intentions and impacts. Journal of Environmental Psychology, 29(1), 13-23. Wray-Lake, L., Flanagan, C. A., & Osgood, D. W. (2010). Examining Trends in Adolescent Environmental Attitudes, Beliefs, and Behaviors Across Three Decades. Environment and behavior, 42(1), 61-85. Zimmerman, H.T., Kanter, D.E., Ellenbogen, K., Lyons, L., Zuiker, S.J., Satwicz, T., Martell, S.T., Brown, M., Hsi, S., Smith, B., Phipps, M., Jordan, R., Weible, J., Gamrat, C., Loh, B. & Joyce, M. (2010). Technologies and tools to support informal science learning. Proceedings of the 9th International Conference of the Learning Sciences Volume 2, Chicago. pp 260 266.
Acknowledgements
We thank the students and staff who have worked with us on the Taking in the Teenagers project funded by the Engineering and Physical Sciences Research Council in the UK: Grant reference number EP/I000550/1
ISLS
158
ICLS2012
Unpacking the Use of Talk and Writing in Argument-based Inquiry: Instruction and Cognition
Ying-Chih Chen, University of Minnesota, 320 LES 1954 Buford Ave, St Paul, MN 55108, chen2719@umn.edu Soonhye Park, University of Iowa, N278 Lindquist Center, Iowa City, IA 52242, soonhye-park@uiowa.edu Brian Hand, University of Iowa, N297A Lindquist Center, Iowa City, IA 52242, brian-hand@uiowa.edu Abstract: The purpose of this study was to unpack the use of talk and writing to support students construction of scientific knowledge in argument-based inquiry. Grounded in interactive constructivism, this sixteen-week study utilized qualitative design and was conducted in one argument-based inquiry classroom with participation of 22 fifth-grade students. The results indicated (a) as students had more opportunities to practice, they developed ore sophisticated understanding for oral argumentation, (b) students ability to craft a written argument improved over time, (c) when both talk and writing were used, student knowledge construction occurred more than when only one learning tool was used, (d) students higher cognitive processes were facilitated more than when talk or writing were used alone, and (e) the more talk and writing were used together, the more student-centered the classroom was.
Introduction
Argumentation is a core practice of scientific inquiry in which students engage in reasoning processes of constructing knowledge claims through interpreting data as sound evidence and debating those claims (Ford, 2008; Hand, 2008). Current research indicates that learning how to engage in productive scientific argumentation to build and propose knowledge is difficult for students (Sampson & Clark, 2009). Many scholars advocate strongly the need to create environments where students can talk in order to construct explanations, models, and theories just as scientists use arguments to relate the evidence they select to their claims (e.g., McNeill, & Pimentel, 2010; Mercer, 2008). Nevertheless, Yore and Treagust (2006) argued that talking is necessary for argumentation, but not sufficient to do and learn science (p. 296). They noted that writing also play an important role during argumentative processes. Wallace (2007) succinctly concluded about the effect of talking and writing on argumentation: talk is most important for distributing knowledge, while writing is important for manipulating, consolidating, and integrating knowledge (p. 11). Talking and writing are therefore complementary tools for argumentative practice. However, Studies in the field of argumentation have investigated talk and writing separately and thus have tended to disconnect the relationship between the two in terms of the argumentative process due to different theoretical frameworks or methodologies adopted by the researchers. For example, some researchers emphasize the impact of engaging students in talking within groups or as a whole class (e.g. McNeill & Pimentel, 2010), while others focus on the important mechanisms for individually scaffolding the construction of students written arguments (e.g. Takao & Kelly, 2003). Those studies that point out the challenges, difficulties, and effects of using talk or writing in argumentation have contributed greatly to the field of science education. However, to learn better how to support students in this argumentative practice and to overcome the challenges, an examination of investigating how talk and writing combined can best support students construction of scientific knowledge through argumentation and what kinds of cognitive processes might be facilitated by the combination of talk and writing is needed. So far, only two quasi-experimental studies, conducted by Rivard and Straw (2000) and Rivard (2004), have directly reported interesting findings regarding the impact of both talk and writing on students conceptual learning. Rivard and Straw (2000) compared the effects of talk alone, writing alone, the combination with talk and writing, and control condition at a posttest and delayed posttest. They found that talk and writing together were more effective than other three groups in contributing to simple and integrated knowledge at a post-test and delayed posttest. They suggested that science teachers should endeavor to include more writing tasks in the classroom after students have had sufficient opportunities for collaborative exploratory talk while being guided by cognitive engagement in argumentative processes. Yet, their research was quantitative and still focused on the outcome of knowledge acquisition. The patterns of the combination of talk and writing that support students knowledge construction and cognitive processes still remain unknown. In response to those research deficits, we try to unpack the use of talk and writing in an argument-based inquiry classroom and set up four research questions to guide this qualitative study: (1) How do students develop an understanding of using talk as a learning tool when participating in an argument-based inquiry classroom during two units over sixteen weeks? (2) How do students craft their written arguments over time in an argument-based inquiry classroom? (3) In what ways do talk and writing support scientific knowledge
ISLS
159
ICLS2012
construction in an argument-based inquiry classroom? (4) What are the characteristics of the cognitive processes associated with the use of talk and writing in the context?
Grounded in interactive constructivism (Yore, 2001) and using language as a learning tool (Wallace, 2007), argumentation is defined in the current study as both an individual cognitive activity and a negotiated social act for knowledge construction (Hand, 2008). Both the individual and social aspects of argumentation are essential for classroom practice because they enhance students ability to reason and justify claims as well as to interact with their teacher and peers in the process of constructing and critiquing their ideas. When students move quickly and effectively between the two landscapes, their knowledge construction can best be facilitated. What makes their moves successful for knowledge construction is the use of language. Stated differently, knowledge construction involves an iterated process of negotiating meaning between a cognitive dynamic within individuals and social interaction in small group and whole class settings while simultaneously utilizing a variety of language forms and modes for representing those meanings. Although other forms of language are used to construct knowledge, talk and writing have been considered two critical and powerful learning tools of knowledge construction in science classrooms by a number of scholars (e.g. Yore & Treagust, 2006). In this study, writing includes not only texts, but also all representations which have the potential to improve students conceptual understanding such as graphs, pictures, diagrams, tables, etc. To guide data analysis and interpretation, this study identifies the structure of scientific argument as consisting of three interrelated components: question, claim, and evidence (see Chen, 2011, for descriptions of the framework and the three components).
Research Design
To answer the first and second research question, a basic qualitative approach (Merriam, 1998) was conducted in a fifth grade science classroom taught by one white male teacher in the Midwestern United States. The teacher participating in the study had 10 years of teaching experience. He had been involved in a professional development project for using an argument-based inquiry approachthe Science Writing Heuristic (SWH) approach (Hand, 2008)for the past three years. This teacher was selected using a purposeful sampling technique because he had incorporated the SWH approach into his classroom at a high level of implementation based upon his score on the Reformed Teaching Observation Protocol (RTOP). In the summer of 2010, he was recruited as an instructor for a SWH professional development project. To understand in depth elementary school students perceptions, reasons, and thoughts about their actions and participation in the classroom, six students (two girls, four boys; three high achievement, three medium achievement; three initially talkative, three initially quiet) were selected for targeted interview from the 22 students in the class. To answer the third and fourth research question, a multiple-case study was conducted. Three students were selected from the six target students for careful observation while they discussed, investigated, and wrote in whole class and small group settings over sixteen weeks. The purpose of only choosing three students was to examine the trajectory of students use of talk and writing for knowledge construction in a great deal.
Data Collection
To triangulate the findings, data were collected through a variety of sources during two classroom units (Ecosystem & Human Body System), including daily non-participant observations, semi-structured interviews, students writing samples, and researchers field notes. Interviews about individual students writing samples and about classroom observations were conducted in a semi-structured fashion to elucidate the reasons they wrote, drew, and organized datasets in particular ways. The average number of interviews with each student was 11 over sixteen weeks. Students had science class every day. Seventy four classroom videotapes were collected and transcribed.
Data Analysis
Research Question 1&2: Two different analytical approaches were used: (1) the constant comparative method and (2) the enumerative approach (LeCompte & Preissle, 1993). As a result of analysis via the constant comparative method, six core components of oral argumentation related to public negotiation were identified from five rounds of public negotiations over the course of the two learning units: 1) Information Seeking, 2) Elaborating, 3) Challenging (testing, relationship between question and claim, relationship between claim and evidence, the value of a claim, and the value of evidence--sufficiency, validity, and reasoning), 4) Defending (simple, evidence-based), 5) Supporting (simple, evidencebased), and 6) Rejecting (simple, evidence-based). Five core components related to students written arguments were also identified: 1) Accuracy of Claim, 2) Sufficiency of Evidence, 3) Reasoning of Evidence, 4) Relationship between Claim and Question, and 5) Relationship between Claim and Evidence. To gain a better
ISLS 160
ICLS2012
understanding of the change in the quality of students written arguments over time, a writing rubric was created based upon the core written argument components. Each component was rated using a five-point scale from 0= Lack to 4= Proficient for a possible total of 20 points for the instrument as a whole. In addition, six aspects related to students perspectives of argumentation emerged from interview and field notes: 1) Meaning of Negotiation, 2) Function of Talk, 3) Function of Writing, 4) Argument Structure, 5) Reason to Shift Ideas, and 6) View of Feedback. Research Question 3&4: To capture individual students knowledge construction over time, and the way students use talk and/or writing for the knowledge construction, a purposeful approach was designed, called indepth analysis of Knowledge Construction Trajectory (KCT). This approach consisted of four stages: a) dividing units into classes and identifying activities within a class, b) identifying events within an activity, c) synthesizing events into KCTs, and d) analyzing data related to the KCT by the constant comparative method and the enumerative approach (see Chen, 2011, for descriptions of KCT and the analysis process). To answer research question 3, the data related to each KCT were first analyzed to identify the patterns in the use of talk and writing. As a result of the analysis, each of the 21 KCTs revealed one of the five patterns in the use of talk and writing :a) talk only, b) writing only , c) use of talk and writing in sequence , d) use of both talk and writing simultaneously , and e) a combination of sequential and simultaneous use of talk and writing. Table 1 shows examples of the five patterns. Table 1: Examples of Five Patterns of KCTs Pattern 1: Talk Only (Blair, KCT 1) Pattern 2: Writing Only (Kurt, KCT 1)
Pattern 3: Use of Talk and Writing in Sequence (Blair, KCT 3)
Pattern 4: Use of Talk and Writing Simultaneously (Nolan, KCT 5)
Pattern 5: A Combination of Sequential and Simultaneous Use of Talk and Writing (Kurt, KCT 3)
Writing only Talk Only Talk and Writing
Note: Broken line boxes indicate that a student only used writing in the given event; solid line boxes indicate that a student only used talk in the event; double line boxes indicate that a student used both talk and writing in the event.
ISLS
161
ICLS2012
To answer research question 4, all data related to the KCT were analyzed using the constant comparative method to identify cognitive processed associated with the use of talk and writing in each KCT. Codes developed through the analysis include express, report, record, describe, elaborate, organize, challenge, compare, reflect, integrate, stimulate alternative ideas, defend, multi-model representation, audience awareness, and analogize. Then those codes were categorized using Blooms taxonomy to determine the level of a cognitive process involved in writing, talk, or both (Bloom, Englehart, Furst, Hill, & Krathwohl, 1956). As a result, express, report, record, and describe were categorized in low level (knowledge/comprehension); elaborate, compare, and organize were categorized in medium level (application/analysis); challenge, defend, reflect, stimulate alternative ideas, multi-model representation, audience awareness, and analogize were categorized in high level (synthesis/evaluation). (see Chen, 2011, for descriptions of the coding schemes and procedures).
Findings Finding 1: Increased Understanding of the Nature of Argumentation

First, the frequency of student utterances for each argumentation component increased from the first unit to the second unit (see Figure 1). This result suggests that student participation in argumentative discussions was increasingly encouraged and elicited the more they engaged in negotiation. When students oral participation increased and their argumentative practice was encouraged, they were observed to challenge others ideas, use evidence to back up their claims, and evaluate explanatory claims in terms of evidence to construct more complete scientific knowledge. Importantly, students gradually linked the use of argumentation components with the negotiated processes. Argumentative skills were not eliminated due to the change of the topic context, as students quickly adapted their argumentative responses to a new situation.
Figure 1. The Frequency of Argumentative Utterances Contributed by Students in Discussing Claim and Evidence in a Whole Class Setting through Sixteen Weeks Second, three main dimensions through which students challenged each others ideas were distinguished in the different stages of negotiation. These dimensions were (1) a focus on the test and the accuracy of the claim (beginning stage), (2) a focus on the argument structure (middle stage), and (3) a focus on the quality of the evidence (late stage) (see Figure 2). These results confirmed those from other studies on scientific argumentation that suggest that the ability to understand the structure of argument and the quality of evidence does not come naturally to most individuals, but rather is grown through practice (Hand, 2008).
Figure 2. Frequency, Proportion, and Types of Challenging Responses Contributed by Students over Five Rounds of Negotiations
ISLS 162
ICLS2012
Third, these students developed a better understanding of how to adopt more evidence to defend, support, and reject an argument after they participated in more rounds of negotiation. The data revealed that the proportion of utterances made by students using evidence to defend, support, and reject an argument were 13%, 0%, and 0% in the first-round negotiation of the first unit. In the second round of the second unit, in contrast, the proportion of students who relied on evidence to defend, support, and reject was substantially increased to 88%, 74%, and 59%. This trend resulted in two characteristics: (1) students were more willing to accept other arguments or shift their original ideas, and (2) the discussion was more effective in terms of core concepts related to that topic.
Finding 2: Increased Quality of Written Arguments

The overall quality of students written arguments and their understanding of argument structure and components were improved when they were provided with more opportunities to craft their writing over time (see Figure 3). Students also incorporated multi-model representations to explain their ideas and back themselves up. This improvement in the quality of argument seemed to be due, in large part, to two factors: (1) the engagement in scientific writing embedded within argumentative practice, and (2) an awareness of the usefulness of feedback from peers.
Figure 3. The Average Score for Written Scientific Arguments on Each Component over Two Units Taking the results from Finding 1 and Finding 2 together, the quality of writing was dependent on the way students participated orally in the argumentative practice, and vice versa. In other words, the result, with a parallel shift between oral argumentative practices and the quality of written arguments, indicates that talk and writing are interdependent. Building on these results, Finding 3 will discuss how talk and writing interact with each other and how the different interactions support student knowledge construction in argument-based inquiry. Finding 4 will discuss the characteristics of the cognitive processes associated with the use of talk and writing in the context.
Finding 3: Increased Integration of Using Talk and Writing as Learning Tools

An in-depth analysis of KCTs identified twenty-one KCTs in which students built scientific knowledge using talk and/or writing in an argument-based inquiry classroom during two units. As the semester proceeded, with increasing opportunities to engage in argument-based inquiry, the students came to use different combinations of talk and writing in activities to construct their understanding of core concepts. The number of KCTs in which students used talk and writing simultaneously increased from zero in the first unit to 7 in the second unit, whereas the frequency of the use of talk and writing in sequence decreased from 4 to zero (see Table 2). The use of talk and writing was embedded in the students investigations and negotiations and became an integral part of inquiry as students became capable of using the two tools to represent their arguments, analyze data, and debate their ideas. During the interview, students confessed that talk associated with writing helped them to clarify, visualize, and integrate their ideas which brought them to discuss concepts at a deeper level.
Finding 4: Increased Higher Cognitive Processes by Using Talk and Writing in Sequence or Simultaneously
When students perceived and used talk and writing as separated learning tools, their cognitive processes were constrained to lower level, such as reporting, sharing, and recording their ideas. In contrast, when students used those two learning tools together, students usually became involved in higher and more complex cognitive processes, such as integrating, defending, using multi-model representations, and analogizing, etc. The function of writing extended from an individual learning tool supporting personal cognitive activities when used alone to a socially negotiated learning tool through combination with talk. Additionally, writing became both a product-based and process-based approach when students use it with talk simultaneously.
ISLS
163
ICLS2012
Writing is not simply a matter of translating preconceived ideas into text, but it is a knowledge-constituting process (Galbraith, 1999) in which students synthesize their ideas in written forms to communicate with their peers and external their mental model by using multi-model representation (drawing, tables, figures). The quality of talk was promoted to more evidence-based when writing was used in combination. This trend of using talk and writing together in constructing scientific knowledge over time might be the result of two reasons: (1) students came to take ownership of their learning, and (2) students came to understand the meaning of negotiation. Table 2: Number of KCTs Identified in Each Unit for Each Student Combination of Talk & Writing Writing Combination of Use of Talk and Only Sequential Use Sequence and Writing Simultaneity Simultaneously 1 1 2 0 0 1 1 0 0 2 2 0 1 4 5 0 0 0 1 2 0 0 1 3 0 0 1 2 0 0 3 7 1 4 8 7
Topics
Students Calvin Betty John Calvin Betty John
Talk Only 0 1 0 1 0 0 0 0 1
Unit 1: Ecosystem Sub Total Unit 2: Human Body System Sub Total Total
Summary of Results
Table 3 summarizes the main findings addressing this studys four research questions in terms of the following five aspects: (1) oral argumentation, (2) ability to craft written arguments, (3) use of talk and/or writing, (4) cognitive processes, and (5) meaning of negotiation. In examining these five major conceptual outcomes arising from this study, the researchers would suggest that the common critical element across these five areas is time. As students encountered more opportunities to obtain a grasp of the practice (Ford, 2008), they were observed to develop a more sophisticated understanding of argumentation, use talk and writing as learning tools to construct and negotiate their ideas with peers, engage in more complex and higher-order cognitive processes, and take ownership of their learning in science.
Discussion and Implication for Science Education

One of the major findings of this study indicates that there is a relationship between the way these students participated in oral argumentation and their abilities in crafting written arguments. Examining Finding 1 and Finding 2 together, at the beginning of the semester students focused on challenging the test procedure and accuracy of claims, which was reflected in the improved quality of their written arguments about the accuracy of claims. In the middle of the semester, students spoken challenges focused on the structure of the argument, which was also reflected in their improved quality of writing about the relationship between question and claim and the relationship between claim and evidence. At the end of the semester, students moved to challenge the quality of evidence, which was reflected in their improved quality of written arguments regarding reasoning and sufficiency of evidence. These observations, when taken together, indicate that there seems to be a positive relationship between students participation in oral argumentation and writing as outcome measures. As such this study would suggest that talk and writing are interdependent in this context. However, it is believed that improved performance in one practice does not necessarily lead to better performance in the other; instead the students seemed to develop better awareness of audience and understanding of claim/ evidence, which guided how they engaged in both practices. This epistemic shift requires a different classroom culture and discourse environment (Cavagnetto, 2010). It is speculated that such a shift requires two conditions to occur: (1) students must be introduced to new criteria or norms for what counts as a claim and what counts as evidence in an explicit fashion in an appropriate context and time, and (2) students need to be encouraged by others to use these new criteria and norms in an appropriate context and time in which they are fruitful and make sense. Berland and Reised (2011) have indicated that students rarely revise their ideas in light of the challenges and questions posed to them. However, this study did find that students at the end of the semester were more willing to revise their ideas if their peers provided evidence to support their opinions. This suggests that a challenge component is necessary for argumentative practice, but it is not enough to generate complete and convincing scientific arguments. Evidence-based defending, supporting, and rejecting components may also
ISLS
164
ICLS2012
help students to more carefully rethink, reflect on, and compare their ideas to those of others with evidentiary support. The significance here is that only using talk as a learning tool may not produce evidence-based discussion. When students engaged in evidence-based discussion, they usually used writing associated with talk as negotiation tools to help them explain and elaborate upon their ideas. In addition, asking or encouraging students to use multi-model representations may foster their ability to clarify and revisit their ideas instead of relying only on talk or one model (text) for reasoning concepts. Table 3: Matrix of Argumentative Practices in Five Aspects Time Line Week 1-3 Oral Argumentation --Information seeking (great proportion) --Elaborating --Challenging (focus on the procedure, process, and the accuracy of a claim) --Rejecting, supporting, and defending
(simple)
Written Arguments --Focus on the accuracy of a claim --Audience is the teacher --The reason to change ideas is because of teachers hint
Use of Talk and/or Writing --Talk or writing alone (writing is a product) --Use talk and writing in sequence (writing is a product, then becomes a process)
Cognitive Processes --Expressing, reporting, recording, describing, and elaborating
Meaning of Negotiation --Teacherdirected --Negotiation is to talk out ideas --Students usually talked over each other
Week 4-12
Week 13-16
--Information seeking (small proportion) --Elaborating --Challenging (focus on the structure of an argument: question, claims, and evidence) --Rejecting, supporting, and defending (more evidence-based) --Information seeking (small proportion) --Elaborating --Challenging (focus on the quality of evidence: sufficiency, validity, and reasoning) --Rejecting, supporting, and defending (evidence-based)
--Focus on the relationship between question, claims and evidence --Awareness of audience shifting from teacher to peers
--Combination of sequence and simultaneity
--Elaborating, organizing, challenging, comparing, reflecting, stimulating alternative ideas, defending, multimodel representing, and audience awareness
--Transition from teacher-directed to studentdirected --Negotiation is to explain ideas and to revise them --Students began to realize the value of critique and listening --Studentdirected --Negotiation is to explain ideas and reach a consensus; provide evidence to support claim --Effective dialogue: students realized the importance of critiquing, constructing, and listening
--Focus on the sufficiency and reasoning of evidence --Audience is both teacher and peers --Willing to change or shift ideas based on peers feedback
--Use talk and writing simultaneously in a whole class setting or a small group (writing is both product and process in this context)
--Elaborating, organizing, challenging, comparing, reflecting, stimulating alternative ideas, defending, multimodel representing, audience awareness, and analogizing
The analysis of this study led us to rethink the value of the combination of talk and writing to promote students construction of science, as it appears to help students engage in more productive arguments than when using talk or writing alone. This kind of talk associated with writing, as Galbraith noted (1999), required an oscillation between disposition (targeted topic) and linguistic (writing task) knowledge, which led to clarification of conceptual understanding and may lead to the formation of new knowledge. Talk alone simply
ISLS
165
ICLS2012
deposits ideas that are not recorded. It is difficult to go back to check an idea generated 10 minutes ago. By contrast, the form of talk associated with writing for sharing, challenging and defending is an interaction between students and texts which is represented by someone. Talk associated with writing encourages students to freeze their ideas on paper, which records them. Students can frequently collaboratively reflect on their ideas based upon their writing. The role of writing in this case serves learning uniquely because writing as process-and-product possesses a cluster of attributes that correspond uniquely to certain powerful learning strategies (Emig, 1977, p. 122). Importantly, this study also found that students engaged in higher-order and more complex cognitive processes when talk and writing were used together either in sequence or simultaneously, rather than when they were used alone. It is clear that teachers have a responsibility to design [a] sequence of instruction that provides opportunities for student growth (Duschl & Ellenbogen, 2002, p.3). More importantly, the teacher should create a discourse space in which students can construct an argument as part of an investigation with their peers, clarify their thinking by using the six core argumentation components, monitor their conceptual understandings, as well as learn and use the criteria by which these arguments will be judged or evaluated. Talk and writing can be powerful means by which to foster students to engage in higher cognitive reasoning. Especially, the simultaneous use of talk and writing can be an effective way to help students develop ownership of their learning, participate in more productive argumentative practices, and advance their conceptual growth.
References
Bloom, B. S., Englehart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956) Taxonomy of educational objectives. Handbook 1: Cognitive domain. New York: Longmans, Green. Cavagnetto, A. R. (2010). Argument to foster scientific literacy: A review of argument interventions in K12 science contexts. Review of Educational Research, 80(3), 336-371. Chen, Y.-C. (2011). Examining the integration of talk and writing for student knowledge construction through argumentation. Unpublished doctoral dissertation, Iowa City, IA. Duschl, R.A., & Ellenbogen, K. (2002). Argumentation processes in science learning. Paper presented at the conference on Philosophical, Psychological, and Linguistic Foundations for Language and Science Literacy Research, University of Victoria, BC, Canada. Emig, J. (1977). Writing as a mode of learning. College Composition and Communication, 28(2), 122-128. Ford, M. J. (2008). Disciplinary authority and accountability in scientific practice and learning. Science Education, 92(3), 404-423. Galbraith, D. (1999). Writing as a knowledge-constituting process. In D. Galbraith, & Torrance, M. (Ed.), Knowing what to write: Conceptual processes in text production (pp. 139-159). Amsterdam: Amsterdam University Press. Hand, B. (2008). Introducing the science writing heuristic approach. In B. Hand (Ed.), Science inquiry, argument and language: A case for the science writing heuristic. Rotterdam, The Netherlands: Sense. LeCompte, M. D., & Preissle, J. (Eds.). (1993). Ethnography and qualitative design in educational research (2nd ed.). San Diego, CA: Academic Press. Mercer, N. (2008). The seeds of time: Why classroom dialogue needs a temporal analysis. The Journal of the Learning Sciences, 17(1), 33-59. Merriam, S. (1998). Qualitative research and case study applications in education. San Francisco: Jossey-Bass. McNeill, K. L., & Pimentel, D. S. (2010). Scientific discourse in three urban classrooms: The role of the teacher in engaging high school students in argumentation. Science Education, 94(2), 203-229. Rivard, L. P. (2004). Are language-based activities in science effective for all students, including low achievers? Science Education, 88(3), 420-442. Rivard, L. P., & Straw, S. B. (2000). The effect of talk and writing on learning science: An exploratory study. Science Education, 84(5), 566-593. Sampson, V., & Clark, D. (2009). The impact of collaboration on the outcomes of scientific argumentation. Science Education, 93(3), 448-484. Takao, A., & Kelly, G. (2003). Assessment of evidence in university students' scientific writing. Science and Education, 12, 341-363. Wallace, C. S. (2007). Evidence from the literature for writing as a mode of science learning. In C. S. Wallace., B. Hand & V. Prain (Ed.), Writing and learning in the science classroom (pp. 9-19). The Netherlands: Springer. Yore, L. D. (2001). What is meant by constructivist science teaching and will the science education community stay the course for meaningful reform? Electronic Journal of Science Education, 5(4), 1-7. Yore, L. D., & Treagust, D. F. (2006). Current Realities and future possibilities: Language and science literacyempowering research and informing instruction. International Journal of Science Education, 28, 291314.
ISLS
166
ICLS2012
A Data-driven Path Model of Student Attributes, Affect, and Engagement in a Computer-based Science Inquiry Microworld
Arnon Hershkovitz, Ryan S.J.d. Baker, Janice Gobert, Adam Nakama Worcester Polytechnic Institute, Worcester, MA, USA {arnonh,rsbaker,jgobert,nakama}@wpi.edu Abstract: Work in recent years has shown that a students goals, attitudes, and beliefs towards learning impact their level of engagement during learning, and that engagement during learning plays a key role in learning outcomes. In this paper, we investigate the mechanisms through which a students goals, attitudes, and beliefs impact the students engaged and disengaged behaviors. In particular, we study whether affect is a mediating variable between learner attributes and engagement/disengagement during learning. To this end, we conduct exploratory path analysis on data from middle school learners who were conducting inquiry in science a microworld. We find weak but significant relationships between variables related to attitudes and beliefs and variables related to affective states and engagement. We present a path model that highlights boredom as an important mediator between a tendency towards avoiding novelty and off-task behavior during learning.
Introduction
Engagement is a critical factor in accomplishing learning tasks, in both traditional classroom and computerbased learning (Corno & Mandinach, 1983; Pintrich & Schrauben, 1992). Engagement, the extent to which students are involved in learning tasks, is positively linked with desirable learning outcomes such as deep learning and grades (Skinner, Wellborn & Connell, 1990; Wigfield et al., 2008). Individual differences in engagement are often due to differences in students goals, beliefs, and attitudes (e.g., Zimmerman & Bandura, 1994). Affect has been hypothesized to be a key potential mediator between these relationships (e.g., Linnenbrink, 2007). In this paper, we investigate how learners goals, attitudes, and beliefs influence their degree of engagement during learning, and how this process is mediated by affective states, using data from middle school students completing a computer-based science inquiry activity in the domain of physical science. Although in recent years, research has revealed relationships between student attributes, affective states, and engagement behaviors while using computer-based learning environments, past studies have typically focused only on small subsets of these constructs. We, on the other hand, examine the relationships between a larger set of variables, and we do it using an exploratory path analysis, a method for describing the direct and indirect dependencies among a set of variables (Land, 1969).
Engaged and Disengaged Behaviors

Engagement and disengagement have been characterized as elusive variables (Corno & Mandinach, 1993), due to the difficulty of defining these constructs precisely. Our approach towards operationalizing engagement and disengagement is to identify and measure specific behaviors that indicate engagement and/or disengagement. One such disengaged behavior is off-task behavior (Karweit & Slavin, 1982), where the student ceases to engage in the learning task, and instead engages in an unrelated activity. Off-task behavior has been shown to lead to poorer learning outcomes in educational software (Baker et al., 2004; Rowe et al., 2009). Another form of disengaged behavior frequently seen in educational software is gaming the system, a behavior in which students engage in systematic guessing or rapid help requests in order to obtain answers rather than thinking through the learning material (Baker et al., 2004). Gaming the system also has been shown to be negatively correlated to learning (Baker et al., 2004).
Student Attributes and Disengagement

There is increasing evidence that semi-stable student attributes such as poor goal orientation (including the goal of avoiding failure Elliot, McGregor, & Gable, 1999, and the goal of avoiding work Harackiewicz et al 2002), low self-efficacy (Zimmerman & Bandura, 1994), and low grit (perseverance and passion for long-term goals Duckworth et al., 2007) are associated with disengagement (cf. Elliot, McGregor, & Gable, 1999; Harackiewicz et al., 2000; 2002; Baker et al., 2008). At the same time, there is increasing evidence that student behaviors of various types mediate the relationships between student attributes and learning (cf. Skinner, Wellborn & Connell, 1990; Middleton & Midgley, 1997; Elliot, McGregor, & Gable, 1999; Blackwell, Trzesniewski & Dweck, 2007). For instance, goal orientation was found to be related to cognitive engagement:
ISLS
167
ICLS2012
students with learning goals tend to be more engaged, and tend to achieve better learning outcomes than students with performance goals (e.g., Wolters & Pintrich, 1996; Dupeyart & Marin 2005). Another example , is grit, which predicts gaming the system on material the student does not know, in turn leading to poorer learning outcomes (Baker et al., 2008).
Affect and Disengagement

Another factor which plays an important role in student learning and engagement is affect (Craig et al., 2004), the subjective perception of emotional states (Corno, 1986) in context. The affective state of boredom has been shown to be negatively correlated with learning outcomes (Craig et al., 2004), while the affective state of engaged concentration (cf. Baker et al., 2010), also termed flow, has been shown to be positively correlated with learning outcomes (Craig et al., 2004). The relationship between confusion and learning appears to be complex, with one study suggesting that the context of confusion determines its relationship to learning (Lee et al., 2011). In addition, affect has been found to be related to students achievement goals and self-efficacy beliefs (e.g., Dweck & Leggett, 1988; Linnenbrink & Pintrich, 2002). Furthermore, it was shown that affect mediates the relationships between student goals and engagement, with pleasant affect having a positive mediating function and negative affect having a negative effect (Linnenbrink, 2007). To give another example, gaming the system is associated both with high levels of boredom (Baker et al., 2010) and low levels of grit (Baker et al., 2008), and boredom is positively associated with work avoidance (Duda & Nichols, 1992), a construct related to grit. This leads to the question: Does boredom potentially mediate the relationship between grit and gaming the system? And which affective states mediate the relationships between student attributes and other disengaged behaviors?
Methods
Participants
The study participants were 48 sixth-grade students (from 3 different classes), ranging in age from 10-12 years old, from a public middle school in a small city in Central Massachusetts. This school is majority White (68%), but has a substantial Hispanic minority (15%), and over 20% of residents in the city speak a language other than English at home. Overall, the city where the school is located has a median income only moderately lower than the U.S. median, but significantly lower than surrounding communities in the region, with almost 20% of individuals below the poverty level. The relationship between carelessness and student goals was studied in the context of students scientific inquiry within a physical science microworld. This microworld, for the domain of phase change, enabled students to explore how a particular substance, such as water, changes phases from solid to liquid to gas as it is heated. The goal of this environment and its associated activities was for students to engage in inquiry in order to develop an understanding that a substance has a melting point and a boiling point, which are both independent of the amount of sample. The phase change environment used, hosted by Science Assistments (www.scienceassistments.org; Gobert et al., 2009; Sao Pedro et al., in press) enabled students to engage in authentic inquiry using a microworld and inquiry support tools. Each task in the learning environment required students to conduct experiments to determine if a particular independent variable (container size, heat level, substance amount, or cover status) affected various outcomes (melting point, boiling point, time to melt, or time to boil).
Variables and Procedure

Within the study, we collected data relative to three categories of research variables, conceptualized as a 3-tier model. The three tiers studied are student attributes, affective states, and engaged and disengaged behaviors. We study the relationships between these tiers under the modeling assumption that the effect of student attributes on engagement is mediated by affective states. After receiving a short introduction to the activity, the students first completed the student attribute surveys online, and then engaged in the phase change learning activities for 20 minutes. Student Attributes (11 variables): The variables are: learning goal orientation, the goal of developing skill or learning; performance-approach goal orientation, the goal of demonstrating competence; performance-avoidance goal orientation, the goal of avoiding demonstrating incompetence; academic efficacy; avoiding novelty; disruptive behavior; self-presentation of low achievement, the desire to prevent peers from knowing how well the student is performing; skepticism about the relevance of school for future success; grit, the perservance and passion for long-term goals; work-avoidance; and self-efficacy for self-regulated learning.
ISLS
168
ICLS2012
These variables are measured using surveys administered to students online. The Patterns of Adaptive Learning Scales (PALS) survey (Midgley et al., 1997) was used to assess semi-stable student attributes. In specific, two scales were utilized: PALS 1, Personal Achievement Goal Orientation (14 items), and PALS 4, Academic-related Perceptions, Beliefs, and Strategies (28 items), with all items in these scales rated on a 5-point Likert scale. Also, surveys were used to measure grit on a 5-point Likert scale (Duckworth et al., 2007), workavoidance on a 7-point Likert scale (Harackiewicz et al., 2000), and self-efficacy for self-regulated learning on a 7-point Likert scale (Bandura, 1990). Engaged and Disengaged Behaviors (4 variables): The following engaged and disengaged behaviors were coded: on task, on-task conversation, off-task, and gaming the system. These variables were assessed using quantitative field observations during student use of the learning environment (cf. Baker et al. 2004, 2010). For each student and for each observed variable, the percentage of the instances in which each behavior was observed was computed. Observed were conducted by trained observers (in specific, the second and fourth authors), who had previously achieved inter-rater reliability over 0.6 for the coding protocol and both of the coding schemes used in this study, with a different population. This level of inter-rater reliability, while imperfect, is substantially higher than that seen for expert coding of academic emotions and behaviors of this nature in video data (cf. Graesser et al., 2006). Each observation lasted twenty seconds, and was conducted using peripheral vision. That is, the observers stood diagonally behind or in front of the student being observed and avoided looking at the student directly, in order to make it less clear when an observation was occurring (cf. Baker et al. 2004, 2010). If two distinct behaviors were seen during an observation, only the first behavior observed was coded; similarly, if two distinct affective states were seen during an observation, only the first was coded. Only the student currently being observed was coded during each observation period, to avoid bias towards more salient behaviors and affective states. Observations were recorded using an app for the Google Android platform, which enforced the coding procedure and coding schemes. For each student and for each variable, the percentage of the instances in which this variable was observed was computed. Affective States (4 variables): The following affective states were measured: confusion, engaged concentration, boredom, and frustration. These observations were conducted concurrently to the observations of engaged and disengaged behaviors, using the same procedure and coding schemes as in Baker et al. (2010). For each student and for each observed variable, the percentage of the instances in which each affective state was observed were computed. During analysis, the affective state of engaged concentration was treated as representing a different state when students were off-task as opposed to on-task, as being deeply engaged in offtask behavior should not be considered evidence for engagement with the learning material.
Analysis Approach: Path Models

Path models are an extension of multiple regression, in which the relationships between multiple variables are evaluated simultaneously. Path models are a graphical representation of a Structural Equation Model (SEM) describing linear relationships between the variables under consideration (Land, 1969). In recent years, path models have been used to shed light on the relationships between multiple learning variables. In most cases, path models have served as a means to test existing theoretical models of the relationships between variables of different types, e.g., goals, metacognitive, study strategies, and academic achievement (Dupeyart & Marine, 2005; Blackwell, Trzesniewski, & Dweck, 2007). However, in other cases, path models have been built bottomup, to find the best possible model for a given dataset without assuming the model a-priori (cf. Fouad & Smith, 1996; Iverach & Fisher, 2008). In this paper, given the lack of a prior theoretical model linking all of the constructs under investigation, we adopt a bottom-up method, essentially using path models as an educational data mining method (cf. Baker & Yacef, 2009). To this end, we follow a heuristic inspired by Cohen et al. (1993) for building path models in this fashion: potential links are chosen to be added to the path model according to the degree of improvement in the models fit to the data. We impose only one type of structure: a 3-tier model, consisting of the following sets of variables: student attributes (tier one), affective states (tier two), and behaviors indicating engagement or disengagement (tier three). Attributes are assumed to influence later affective states, and both attributes and affective states are assumed to influence behavior. The model creation process was initiated using a set of models, each of which consisted of only two variables and one link, based on the set of correlations which were statistically significant under post-hoc correction (listed below). Then, we assessed the impact on model significance from adding additional variables and links to the model. Models were built and assessed using the AMOS 19 software for path modeling (Arbuckle, 2010).
Results
First, in order to understand the frequencies of the research variables, we computed means and standard deviations for each of the research variables. A full list of proportions, means and standard deviations of the variables used is given in Table 1. The average proportion of off-task behavior (13%) is within the range previously observed in studies of students using intelligent tutor software in the U.S. (cf. Baker et al., 2004;
ISLS
169
ICLS2012
Baker, 2009; Cetintas et al., 2010), and is similar or slightly lower than proportions of off-task behavior seen in traditional classrooms (cf. Karweit & Slavin, 1982). The proportion of gaming the system (1%) is substantially lower than seen in previous studies of intelligent tutoring systems and educational games (Baker et al., 2004, 2010; Walonoski & Heffernan, 2006). Students were observed to be in a state of engaged concentration 59% of the time, a proportion consistent with past research on students using intelligent tutoring systems (Craig et al., 2004; Baker et al., 2010), though higher than seen in some studies (Sabourin et al., 2011). Boredom was substantially more common than in several of these past studies, occurring 20% of the time, while confusion was relatively less common, occurring 8% of the time. Table 1: Mean and standard deviation for the research variables. Student Attributes (all but the last two are on a 5-Likert scale; the last two are on a 7-Likert scale) Variable Mean SD learning goal orientation (N=46) 4.20 0.59 performance-approach goal 3.00 1.19 orientation (N=46) performance-avoid goal 3.09 1.24 orientation (N=46) academic efficacy (N=46) 3.57 0.90 avoiding novelty (N=46) 2.90 1.10 disruptive behavior (N=46) 2.77 1.21 self-presentation of low 2.14 1.00 achievement (N=45) skepticism about the relevance of 2.65 1.12 school for future success (N=45) grit (N=43) 2.86 0.65 work-avoidance (7-Likert scale) 4.06 1.19 (N=47) self-efficacy for self-regulated 4.52 0.88 learning (7-Likert scale) (N=47) Affective States (proportion of observed instances to total number of observations), N=48 Variable frustration confusion engaged concentration Mean 0.06 0.08 0.59 SD 0.06 0.07 0.18
boredom 0.20 0.13 Engagement (proportion of observed instances to total number of observations), N=48 Variable Mean SD on task on-task conversation off-task gaming the system 0.75 0.11 0.13 0.01 0.16 0.10 0.11 0.04
Relationships between the Research Variables

In order to explore the relationships between the research variables, we examined the inter-correlations between the variables of the different tiers. Results are summarized in Table 2. As we ran an overall of 104 bivariate correlations tests, we corrected for multiple comparisons using the post-hoc False Discovery Rate (FDR) method, which produces a q-value that can be interpreted the same way as a p-value. Between student attributes and affective states, only one pair of variables (out of 44) was found to be significantly correlated when controlling for multiple comparisons: disruptive behavior and confusion (r=0.45, q<0.05). This relationship might be explained as the following: as disruptive behavior increases, less learning opportunities are available for the student, hence she or he is more likely to be confused when required to demonstrate their skills. It is surprising that none of the three main goal orientation scales (learning, performance-approach, and performance-avoid) were found to be significantly correlated with any of the affective states, considering previous studies (e.g., Linnenbrink & Pintrich, 2002). Between student attributes and engagement tiers, no significant correlations were found. Again, no correlations were found between the three main goal orientation scales with any of the engagement variables. The strongest correlation (in terms of q-value) was between performance-approach goal orientation and on task conversation (r=0.20, q=0.73). This is consistent with previous studies which have found no relationships between goal orientation and disengaged behaviors within students using educational software (e.g. Baker, 2007; Baker et al., 2008, Rowe et al., 2009). Finally, between the affective states and engaged/disengaged behaviors, four significant correlations were found. Engaged concentration was found to be significantly correlated with on-task (r=0.65, q<0.01) and off-task (-0.54, q<0.01); also, boredom was found to be significantly correlated with on-task (r=0.57, q<0.01) and off-task (r=0.65, q<0.01). One complication in analyzing these relationships is that the affective states and engagement variables are not normally distributed. Testing for multivariate normality, we used Smalls test for skewness (Small, 1980) and Anscome & Glynns (1983) variant of Smalls test for kurtosis, and found significant evidence for skew,
ISLS 170
ICLS2012
Q1=71.28 (df=19), p<0.0001, and for multivariate kurtosis, Q2=56.74 (df=19), at p<0.0001. Hence, nonparametric bootstrapping and Maximum Likelihood estimation were used to generate model parameters and estimates (cf. Kline, 2010), with removal of cases with missing values during bootstrapping. Table 2: Inter-correlations between the research variables of different tiers (significant values after post-hoc corrections bolded). Variables frust. Affective States conf. concent. boredom on task
-0.02 -0.01 0.07 0.15 0.31 -0.16 0.11 0.12 0.24 -0.08 -0.06 -0.19 -0.02 0.65** -0.57**
Engagement ontask conv.

0.00 0.20 0.09 0.00 0.13 0.23 0.00 0.06 0.00 0.24 0.04 0.24 0.14 -0.34 0.18
offtask
0.03 -0.12 -0.15 -0.11 -0.34 0.03 -0.15 -0.23 -0.28 -0.14 -0.01 -0.04 -0.09 -0.54** 0.65**
gaming
learning goal (N=46) perf-approach.goal (N=46) perf.-avoid goal (N=46) academic. efficacy (N=46) avoiding. novelty (N=46) disruptive behavior (N=46) self-presnt. of low achv. (N=46) skepticism (N=45) grit (N=43) work-avoidance (N=47) self-efficacy (N=47) frustration (N=46) confusion (N=46) concentration (N=46) boredom (N=46) * q<0.05, ** q<0.01 Student Attributes Affective States
-0.16 -0.07 -0.16 -0.38 0.01 0.00 -0.13 -0.09 0.17 0.05 -0.16
0.04 0.26 -0.01 -0.15 0.13 0.45* 0.22 0.34 -0.03 0.11 -0.12
0.06 0.12 0.11 0.32 0.21 -0.12 0.07 0.07 0.07 -0.05 0.19
0.09 -0.25 -0.15 -0.17 -0.35 -0.10 -0.19 -0.14 -0.04 -0.02 -0.14
0.00 -0.13 -0.09 -0.24 0.10 -0.05 -0.03 0.01 -0.14 -0.01 0.00 0.23 -0.01 -0.09 -0.11
Path Model
As was previously mentioned, we applied a bottom-up construction heuristic for the path model. Figure 1 presents the best fitting model found, involving only three variables, one of each tier: avoiding novelty, boredom, and off-task. The standardized regression coefficient between avoiding novelty and boredom is -0.28, which is marginally statistically significant, with p=0.06; the coefficient between boredom and off-task is 0.48, which is statistically significant at p<0.01. This model was validated using chi-square, obtaining a value of 1.24 (df=1), p=0.27 (This test compares the best model to a full (saturated) model; a non-significant chi-square is therefore desirable). For this test, a ratio of chi-square to df which is equal to or smaller than 3 is considered acceptable (Schreiber et al., 2006). Three measure of goodness of fit, used for small samples that are not multivariate normal, CFI (comparative fit index), GFI (goodness of fit index), and IFI (incremental fit index), got values of 0.98 each; the agreed cutoff for these measures is 0.95.
ISLS
171
ICLS2012
Figure 1. The final path model: Rectangles represent observed variables, circles latent variables This model suggests that the negative relationship between avoiding novelty and off-task behavior is mediated by boredom. In other words, students who avoid novelty are less likely to be bored during learning, and students who are less bored are less likely to become off-task. We can also understand this model as saying that students who desire novelty are more likely to be bored and therefore are more likely to be off-task. The relationship between avoiding novelty and off-task behavior is much more understandable when the mediating factor of boredom is taken into account.
Discussion & Conclusions

Within this paper we have analyzed the relationships between three categories of variables related to students characteristics and behavior: a) student attributes, including goal orientation, beliefs regarding learning, and attitudes towards learning; b) affective states during learning; and c) engagement and disengagement during learning. Overall, there were only a few statistically significant correlations in between the different categories. Interestingly, this study failed to replicate previous results that establish relationships between goal orientation and engagement (e.g., Dweck & Leggett, 1988; Elliot & Harackiewicz, 1996), as well as between goal orientation and affective states (e.g., Pekrun, Elliot, & Maier, 2006). A possible explanation is that the domain plays a major role in the lack of these relationships, as inquiry-based learning environments foster better engagement and may lead to a greater probability of mastery goal orientation during learning (cf. Hmelo-Silver, Duncan & Chinn, 2007), hence changing the nature of interactions between variables. Indeed, previously there were no associations found between carelessness (a different measure of disengagement) and goal orientation (Hershkovitz et al., 2011), and no relationships between goal orientation and achievement (Shimoda, White & Frederiksen, 2002) in science inquiry tasks. In general, goal orientation has been a poor predictor of engagement in educational software (Baker, 2007; Baker et al., 2008; Rowe et al., 2009), despite its general importance in other contexts (Wolters & Pintrich, 1996; Dupeyart & Marin 2005; Blackwell et al., 2007). , We applied a data-driven approach for constructing a path model to describe the relationships between the research variables. Avoiding novelty was found to be related to boredom, which is in turn related to off-task behavior. The model suggests that it will be valuable to proactively scaffold students who tend to avoid novelty in ways that promote interest. It is important to emphasize that the path model by itself does not imply causality, as is generally the case for post-hoc correlational analyses. In general, it will be valuable to replicate this analysis using different datasets from the same domain, and datasets from additional domains, in order to determine how general the patterns are. One other surprising finding was the relative rarity of gaming the system compared to prior research in intelligent tutoring systems and educational games. One explanation is that gaming the system is more difficult in the current learning environment, given the relatively lower degree of feedback than seen in these types of environments. Interestingly, carelessness another behavior indicating disengagement has also been found to be less common in Science Assistments than in mathematics tutoring systems (Hershkovitz et al., 2011). In summary, the research presented here suggests that in the context of computer-based science inquiry, there is a substantial relationship between avoiding novelty (a student attribute) and off-task behavior (a disengaged behavior), mediated by boredom (an affective state). In general, it will be valuable to study further how and why boredom mediates this relationship. One method for doing so will be to apply temporal analysis, exploring how engagement and affect change during the learning task.
References
Anscombe, F.J. & Glynn, W.J. (1983). Distribution of kurtosis statistic b2 for normal statistics. Biometrika, 70, 227234. Arbuckle, J.L. (2010). IBM SPSS Amos 19 Users Guide. Chicago, IL: IBM. Baker, R.S.J.d. (2007). Modeling and understanding students' off-task behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068. Baker, R.S.J.d. (2009) Differences between intelligent tutor lessons, and the choice to go off-task. Proceedings of the 2nd International Conference on Educational Data Mining, 11-20.
ISLS
172
ICLS2012
Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-task behavior in the cognitive tutor classroom: When students "game the system". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390. Baker, R.S.J.d., D'Mello, S.K., Rodrigo, M.M.T., Graesser, A.C. (2010) Better to Be Frustrated than Bored: The Incidence, Persistence, and Impact of Learners' Cognitive-Affective States during Interactions with Three Different Computer-Based Learning Environments. International Journal of Human-Computer Studies, 68(4), 223-241. Baker, R.S.J.d., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., & Koedinger, K. (2008). Why students engage in gaming the system behavior in interactive learning environments. Journal of Interactive Learning Research, 19(2), 185-224. Baker, R.S.J.d., Yacef, K. (2009) The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3-17. Bandura, A. (1990). Multidimensional Scales of Perceived Self-efficacy. Stanford, CA: Stanford University. Blackwell, L.S., Trzesniewski, K.H., & Dweck, C.S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78(1), 246-263. Cetintas, S., Si, L., Xin, Y.P., & Hord, C. (2010). Automatic detection of off-task behaviors in intelligent tutoring systems with machine learning techniques. IEEE Transactions on Learning Technologies, 3(3), 228-236. Cohen, P.R., Ballesteros, L., Carlson, A., & Amant, R.St. (1993). Automating path analysis for building causal models from data: First results and open problems. In Proceedings of the Tenth International Conference on Machine Learning, 57-64. Corno, L. (1986). The metacognition control components of self-regulated learning. Contemporary Educational Psychology, 11, 333-346. Corno, L. & Mandinach, E.B. (1983). The role of cognitive engagement in classroom learning and motivation. Educational Psychologist, 18(2), 88-108. Craig, S.D., Graesser, A.C., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3), 241-250. Duckworth, A.L., Peterson, C., Matthews, M.D., & Kelly, D.R. (2007). Grit: Perseverance and passion for longterm goals. Journal of Personality and social Psychology, 92(6), 1087-1101. Duda, J., & Nicholls, J. (1992). Dimensions of achievement motivation in school work and sport. Journal of Educational Psychology, 84, 290-299. Dupeyrat, C. and Marin C. (2005). Implicit theories of intelligence, goal orientation, cognitive engagement, , and achievement: A test of Dwecks model with returning to school adults. Contemporary Educational Psychology, 30(1), 43-59. Dweck, C.S. and Leggett, E.L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95, 256273. Elliot, A.J., & Harackiewicz, J.M. (1996). Approach and avoidance achievement goals and intrinsic motivation: A mediational analysis. Journal of Personality and Social Psychology, 70(3), 461475 Elliot, A.J., McGregor, H.A., & Gable, S.L. (1999). Achievement goals, study strategies, and exam performance: A mediational analysis. Journal of Educational Psychology, 76, 628644. Fouad, N.A. & Smith, P.L. (1996). A test of a social cognitive model for middle school students: Math and science. Journal of Counseling Psychology, 43(3), 338-346. Gobert, J., Heffernan, N., Koedinger, K.., & Beck, J. (2009). ASSISTments Meets Science Learning (AMSL). Proposal (R305A090170) funded by the U.S. Department of Education. Graesser, A.C., McDaniel, B., Chipman, P., Witherspoon, A., DMello, S., & Gholson, B. (2006). Detection of emotions during learning with AutoTutor. Proceedings of the 28th Annual Meeting of the Cognitive Science Society, 285-290. Harackiewicz, J.M., Barron, K.E., Tauer, J.M., Carter, S.M., & Elliott, A.J. (2000). Short-term and long-term consequences of achievement goals: Predicting interest and performance over time. Journal of Educational Psychology, 92(2), 316-330. Harackiewicz, J.M., Barron, K.E., Tauer, J.M., & Elliot, A.J. (2002). Predicting success in college: A longitudinal study of achievement goals and ability measures as predictors of Interest and performance from freshman year through graduation. Journal of Educational Psychology, 94, 562575. Hershkovitz, A., Wixon, M., Baker, R.S.J.d., Gobert, J., Sao Pedro, M. (2011) Carelessness and goal orientation in a science microworld. Poster paper. Proceedings of the 15th International Conference on Artificial Intelligence in Education, 462-465. Hmelo-Silver, C.E., Duncan, R.G., & Chinn, C.A. (2007). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42(2), 99-107.
ISLS
173
ICLS2012
Iverach, M.R. & Fisher, D.L. (2008). An interdisciplinary investigation of high schools approaches to learning sciences: The relations amongst achievement goals, constructivist pedagogical dimensions, motivational beliefs and self-regulated learning. Proceedings of the Fifth International Conference on Science, Mathematics and Technology Education, 233-255. Karweit, N. & Slavin, R.E. (1982). Time-on-task: Issues of timing, sampling and definition. Journal of Educational Psychology, 74, 844-851. Kline, R.B. (2010). Principles and Practice of Structural Equation Modeling (3 rd edition). New York, NY: The Guilford Press. Land, K.C., (1969). Principles of path analysis. Sociological Methodology, 1, 3-37. Lee, D.M.C., Rodrigo, Ma.M.T., Baker, R.S.J.d., Sugay, J.O., & Coronel, A. (2011). Exploring the relationship between novice programmer confusion and achievement. In Proceedings of the 4th Bi-annual International Conference on Affective Computing and Intelligent Interaction. Linnenbrink, E.A. (2007). The role of affect in student learning: A multi-dimensional approach to considering the interaction of affect, motivation, and engagement. In P.A. & P. Reinhard (eds.), Emotion in Education (pp. 107-124). San Diego, CA: Elsevier Academic Press. Linnenbrink, E.A. & Pintrich, P.R. (2002). Achievement goals theory and affect: An asymmetrical bidirectional model. Educational Psychologist, 27(2), 69-78. Middleton, M., & Midgley, C. (1997). Avoiding the demonstration of lack of ability: An underexplored aspect of goal theory. Journal of Educational Psychology, 89, 710718 Midgley, C., Maehr, M., Hicks, L., Roeser, R., Urdan, T., Anderman, E., & Kaplan, A., Arun-Kumar, R., Middleton, M. (1997). Patterns of Adaptive Learning Survey (PALS). Ann Arbor, MI: University of Michigan. Pekrun, R., Elliot, A.J., & Maier, M.A. (2006). Acheivement goals and discrete achievement emotions: A theoretical model and prospective tests. Journal of Educational Psychology, 98(3), 583-597. Pintrich, P.R. & Schrauben, B. (1992). Students motivational beliefs and their cognitive engagement in classroom academic tasks. In D.H. Schunk & Meece, J.L. (eds.), Student Perceptions in the Classroom (pp. 149-183). Hillsdale, NJ: Lawrence Erlbaum Associates. Rowe, J.P., McQuiggan, S.W., Robison, J.L., & Lester, J.C. (2009). Off-task behavior in narrative-centered learning environments. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Education, 99-106. Sabourin, J., Mott, B., Lester, J. (2011) Modeling Learner Affect with Theoretically Grounded Dynamic Bayesian Network. Proceedings of 4th International Conference on Affective Computing and Intelligent Interaction. Sao Pedro, M., Baker, R., Gobert, J., Montalvo, O., & Nakama, A. (in press). Using Machine-Learned Detectors of Systematic Inquiry Behavior to Predict Gains in Inquiry Skills. To appear in User Modeling and UserAdapted Interaction. Schreiber, J.B., Stage, F.K., King, J., Nora, A., & Barlow, E.A. (2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. The Journal of Educational Research, 99(6), 323-337. Shimoda, T.A., White, B.Y., & Frederiksen, J.R. (2002). Student goal orientation in learning inquiry skills with modifiable software advisors. Science Education, 86(2), 244-263. Skinner, E.A., Wellborn, J.G., & Connell, J.P. (1990). What it takes to do well in school and whether I've got it: A process model of perceived control and children's engagement and achievement in school. Journal of Educational Psychology, 82(1), 22-32. Small, N.J.H. (1980). Marginal skewness and kurtosis in testing multivariate normality. Applied Statistics, 29, 8587. Walonoski, J. & Heffernan, N.T. (2006). Detection and analysis of off-task gaming behavior in intelligent tutoring systems. Proceedings of the Eight International Conference on Intelligent Tutoring Systems, 382-391. Wigfield, A., Guthrie, J.T., Perencevich, K.C., Taboada, A., Klauda, S.L., McRae, A., & Barbosa, P. (2008). Role of reading engagement in mediating effects of reading comprehension instruction on reading outcomes. Psychology in the Schools, 45(5), 432-445. Wolters, C.W., Yu, S.L., Pintrich, P.R. (1996). The relation between goal orientation and students' motivational beliefs and self-regulated learning. Learning and Individual Differences, 8(3), 211-238. Zimmerman, B.J. & Bandura, A. (1994). Impact of self-regulatory influences on writing course attainment. American Educational Research Journal, 31(4), 845-862.
Acknowledgments
This research was supported by NSF grant DRL-1008649, awarded to Janice Gobert and Ryan Baker. We would also like to thank Michael Sao Pedro, Ermal Toto, Orlando Montalvo, and Juelaila Raziuddin for their role in the development of the physical science inquiry microworld.
ISLS
174
ICLS2012
Exploring Connectedness: Applying ENA to Teacher Knowledge

Chandra Hawley Orrill, University of Massachusetts Dartmouth, Fairhaven, MA, corrill@umassd.edu David Williamson Shaffer, University of Wisconsin, Madison, WI, dws@education.wisc.edu Abstract: In this study, we consider teacher knowledge of mathematics from the perspective of connectedness. To accomplish this, we adapted Epistemic Network Analysis techniques to characterize the connections between and among pieces of teacher knowledge related to one aspect of proportional reasoning. We discuss the value of this approach as well as directions for further research.
Introduction & Significance

Over the past several years, there has been a growing interest in mathematics teachers knowledge (e.g., Ball, Thames, & Phelps, 2008; Hill, Ball, & Schilling, 2008). Despite researchers efforts to better understand both the nature of teacher knowledge and how that knowledge might shape learning experiences for students, research connecting teacher knowledge to student learning remains inconclusive. For example, Hill and colleagues have found a correlation between teacher knowledge and student growth (Hill, Rowan, & Ball, 2005). At the same time, Baumert and colleagues (Baumert, et al., 2010) were able to statistically distinguish between pedagogical content knowledge and content knowledge in measurements as well as to determine that teachers pedagogical content knowledge seemed more closely related to student learning than did content knowledge. While these studies suggest progress in understanding specialized knowledge teachers need, other studies present less promising results. For example, Shechtman and colleagues (Shechtman, Roschelle, Hertel, & Knudsen, 2010) did not find differences among student learning accounted for by teacher knowledge of mathematics for teaching (the same construct measured in the Ball et al. studies). Similarly, Kersting and colleagues (Kersting, Giviin, Sotelo, & Stigler, 2002) only found one subconstruct of several to be related to student learning despite their efforts to measure the knowledge that teachers use. In short, more research is needed to understand the nature of teacher knowledge. To this end, researchers have begun considering how different statistical models, paired with different conceptions of the domain, might help us better understand teacher knowledge (e.g., Izsk, Orrill, Cohen, & Brown, 2010). To further this line of work, in this exploratory study, we use an emerging understanding of teacher knowledge based on research suggesting that the development of expertise is, in part, related to ones organization of knowledge (e.g., Bdard & Chi, 2006; Berliner, 1986; Hiebert & Carpenter, 1992; Leinhardt & Greeno, 1991). This is a departure from studies that consider teacher knowledge from the perspective of measuring amounts of information teachers know. We combine this emerging understanding with a relatively new analysis technique designed to highlight connections among actors in a systemwhether those actors are people, ideas, or concepts. This approach, Epistemic Network Analysis (ENA; Shaffer et al., 2009), allows the discovery of the interconnected nature of expertise. ENA has been used to understand the processes by which students become more expert-like in game situations. In the current study, we consider whether ENA can be used to understand the connections among finegrained ideas about mathematics that teachers use in reasoning about a single proportional reasoning task. We undertake this as a first step toward refining a theory of teacher knowledge grounded in research on expert/novice differences (e.g., Bdard & Chi, 2006; Bransford, Brown, & Cocking, 1999) and related to the knowledge in pieces conceptions of learning (e.g., diSessa, 2006). Our findings highlight the potential benefits of using this approach and we include directions for further investigation.
Expertise & Knowledge in Pieces
Our theoretical framework is grounded in the research on expertise, then introduces Epistemic Frame Theory as the framework of analysis for our study. As mentioned above, we situate this research in prior research on the development of expertise (e.g., Bdard & Chi, 2006) as well as knowledge in pieces (diSessa, 2006; diSessa, et al., 2004). Research on expertise has shown that while experts possess more knowledge of a domain than novices, the quantity of knowledge is not the factor that differentiates them from novices (Bdard & Chi, 2006). Rather, it is the organization of the knowledge that makes them different. For example, experts consider principles rather than surface features in solving problems (Bransford, Brown, & Cocking, 1999). Teachers, for example, differ in their ability to comprehend complex teaching situations in video with the experts more able to attend to important instructional aspects of the class and novices tending to become overwhelmed with all the details of the classroom videos (Sabers, Cushing, & Berliner, 1991).
ISLS
175
ICLS2012
Developing expertise can be paired with the knowledge in pieces theory (diSessa, 2006; diSessa, et al., 2004), which posits that learners have many fine-grained understandings that are drawn together in different configurations when the learner is presented with a problem. We posit that middle grades mathematics teachers, who are often underprepared in mathematics, may have pieces of knowledge that are present, but not wellconnected to other mathematical understandings. For example, in our work, we have seen teachers fail to invoke particular understandings when posed with mathematics problems or student learning situations despite the fact that they have understanding of particular content. As an example, we interviewed one teacher who lamented the inclusion of a rate problem about free throws (e.g., which student would you select to make the next free throw based on given data of the students recent free throw attempts) in her fractions unit. Despite the teachers guide explaining its alignment to the curriculum in terms of equivalent fractions and decimal comparisons, the teacher was unable to reconcile the presence of this task. She knew about fractions and equivalence and she knew about rate, but she could not use those pieces of knowledge together. Certainly, we recognize that teachers need some amount of particular kinds of knowledge. But, we assert that learning more mathematics is unlikely to be helpful to teachers if they lack connectedness in their understanding (e.g., Hiebert & Carpenter, 1992). If teachers are not building connected networks of understanding, they only have a pool of individual ideas, disconnected, and unable to support them (and their students) in the classroom.
Epistemic Frame Theory

Building from the research on expertise, Epistemic Frame Theory (EFT) suggests that complex thinking can be understood in terms of the connections between frame elements: different skills, knowledge, values, identities, and epistemological rules from a particular domain. In our case, we are considering particular skills and knowledge related to proportional reasoning. EFT argues that expertise can be modeled as a network of connections between specific understandings, which are articulated through discourse. Assessing the development of such expertise, however, is a significant challenge. EFT allows us a means, ENA, to quantify the relationships between epistemic frame elements, and thus assess them, using network analysis (Shaffer et al., 2009; Shaffer & Graesser, 2010). Epistemic network analysis (ENA) extends social network analysis by focusing on the patterns of relations among discourse elements rather than individual ideas within that discourse. By treating epistemic frames as cognitive networks, different frame elementsthe particular skills, understandings, identities, values and epistemologies of a professionbecome nodes while the patterns of connections constitute the links between these nodes. By quantifying the patterns of connections between the different elements that combine to form epistemic frames, ENA methods provide a new alternative for measuring complex thinking and problem solving. Thus, we wanted to use them to determine whether they provide insight into the organization of teachers mathematical knowledge.
Figure 1: Task presented to interview participants
Methods
The data for this study were collected as part of a larger project aimed at understanding how teachers make sense of proportional reasoning. The data were drawn from a single question (Figure 1) that was included in a clinical interview protocol. The question was from a series of prompts considering the relationship of oil to
ISLS
176
ICLS2012
vinegar in a salad dressing recipe. The videotaped interviews were conducted with middle grades teachers and transcribed verbatim. Those transcripts were coded for this analysis. Note that the researchers had previously coded these data as part of a grounded theory analysis. Therefore, the researchers were familiar with the teachers and their ideas at the outset of this exploratory study allowing coding to be based on the prior coding. We chose three teachers of varying mathematical ability based on their overall interview (not just the task of interest to this study). The strongest teacher, Lynda (all names are pseudonyms), had been a high school teacher while the other two teachers, Susan and Kate, were middle grades trained. Susan was in the middle in terms of overall performance on the interview tasks and Kate was the weakest. However, all three teachers were clear about their thinking making them ideal for exploring the proposed analysis techniques. Table 1: Codes used in the data analysis. Ratio Concepts: R1: Iteration R2: Ideas of Amount/Quantity/ Growth R3: Invariance R4: Iteration without Calculation R5: Ratio-as-Measure R6: Ratio Calculation Fraction Concepts: F1: Fractions (generic) F2: Rules for Operations F3: Equivalence F4: Comparison Representation (Diagram): D1: Interpretation of Diagram D2: Validate Representations Problem Solving: P1: Problem Interpretation P2: Verify Mathematics is Correct Other Mathematical Ideas: M1: Equality M2: Discussion of Fraction/Ratio Relationship M3: Attending to Context
Epistemic Network Analysis

We coded the interviews for important ideas related to connections between fraction and ratio ideas as they arose (Table 1). These codes captured ideas about fractions, ratios, problem solving, reasoning with drawings, and other mathematical topics. For this exploratory analysis, we treated each box in Table 1 as a node and used ENA to help us find the linkages between those nodes. Formally, an epistemic frame can be characterized by individual frame elements, fi, where i = a particular coded element of an epistemic frame. For any participant, p, in any given interview, s, each segment of discourse, Dp,s, provides evidence of whether participant p was using one or more of the epistemic frame elements (nodes). Each segment of coded data was represented as a vector of ones or zeroes representing the presence or absence, respectively, of each of the codes. Links, or relations, between epistemic frame elements were defined as co-occurrences of qualitative codes within the same segment. To calculate these links, each coded vector was then converted into an adjacency matrix, Ap,s, for participant p in a given story assignment, s (1). Ap,s i,j = 1 if fi and fj are both in Dp,s
p,s
(1)
Each coded segments adjacency matrix, A i,j, was then converted into an adjacency vector and summed into a single cumulative adjacency vector for each participant p, and each story assignment, Up,s, i.e., for each unit of analysis (2). (2) Up,s = Ap,s For each participant, p, and each story assignment, s, the cumulative adjacency vector, Up,s , was used to define the location of the segments in a high dimensional vector space defined by the intersections of each the codes. Cumulative adjacency vectors were then normalized to a unit hypersphere to control for the variation in vector length, representing frequencies of co-occurring code pairs, by dividing each value by the square root of the sum of squares of the vector (3). (3) nUp,s = Up,s / (Up,s)2 This was necessary to emphasize similarities in the types of co-occurring code pairs rather than similarities based primarily on the frequencies of co-occurring code pairs. A singular value decomposition (SVD) was then performed to explore the structure of the qualitative code co-occurrences in the dataset. SVD was used to first project the normalized cumulative adjacency vectors into a high dimensional space such that similar patterns of co-occurrences between epistemic frame element codes would be positioned proximately. The SVD analysis then decomposed the structure of the data in this high dimensional space into a set of uncorrelated components, fewer in number than the number of dimensions that still account for as much of the variance in the data as possible. The resulting networks were then visualized by locating the original frame elements in a two dimensional space as follows: For each frame element ei we compute the mean coordinates x! and y! of all of the n2 loading vectors Vi,j for j=1 to n:
ISLS
177
ICLS2012
!,! ! !!! V! !,! ! !!! V!
x! = y! =
This produces a distribution of nodes in the network graph determined by the loading vectors that contain them in the space of adjacency vectors. Nodes are then positioned in the space, and links are constructed according to the adjacency matrix.
Findings
We found substantial differences between the teachers network graphs. An initial analysis determined that fraction concepts largely fell to the left of the y-axis while ratio concepts fell more to the right. For example, in Kate (Figure 4) we see more attention to thinking about rules for operating on fractions (f2) and finding equivalence (f3). In contrast, Lyndas map (Figure 2) shows more attention to ratio concepts, like ratio as measure (r5; meaning the measure, taste, is the fixed relationship between the vinegar and the oil) and the idea of invariance (r4; meaning the relationship between the two values remains constant). The x-axis appears to divide ideas related to procedural and conceptual understandings. For example, Susans graph (Figure 3) shows the relationship of fractions and ratios (m2) as well as problem interpretation (p1) below the x-axis.
Figure 2: Lyndas map.
Figure 3: Susans map.
Figure 4: Kates map. The maps also show the teachers connectedness of concepts in response to our prompt. To appear as a point, a concept has to have been connected to another concept at least one time. Concepts with lines connecting are related by at least two connections. For example, in Lyndas graph, ideas of quantity (r2) and attending to the context (m3) co-occurred at least twice, indicating some kind of relationship between those ideas for Lynda. Analysis of these maps suggests to us that Lynda relies on ratio reasoning and problem solving skills (e.g., understanding the problem) to make sense of ratios. In contrast, Kate relies on fraction rules and operations to make sense of these concepts. Susan, interestingly, leaned more toward ratio understandings, but introduced fewer key ideas together as evidenced by the limited number of points in her graph. We posit that this may be a natural step in the progression toward expertise. Kate, our least expert teacher, introduced many ideas that were not connected, while Lynda, the most expert-like teacher, introduced many ideas that were more closely connected. Lynda may have a high number of conceptual resources from which to draw and more connections among those ideas than Kate. Susan did not invoke as many ideas, but the ideas she did invoke were largely connected, suggesting that connectedness of ideas and number of ideas are not necessarily related
ISLS
178
ICLS2012
in active problem-solving activities, supporting the findings of expertise research that suggest expertise is more than an accumulation of knowledge.
Next Steps
This exploratory study first confirms that fine-grained analysis of teachers cognition can lead to interesting and informative network maps. Next steps will include the analysis of larger datasets as well as refinements to the coding scheme to disentangle conceptual and procedural understandings. In short, we see ENA as having promise for making sense of teacher understanding, but more work is needed.
References
Ball, D. L., Thames, M. H., & Phelps, G. (2008). Content knowledge for teaching: What makes it special? Journal of Teacher Education, 59(5), 389-407. Baumert, J., Kunter, M., Blum, W., Brunner, M., Voss, T., Jordan, A., & Tsai, Y. M. (2010). Teachers mathematical knowledge, cognitive activation in the classroom, and student progress. American Educational Research Journal, 47(1), 133-180. Bdard, J., & Chi, M. T. H. (1992). Expertise. Current Directions in Psychological Science, 1(4), 135-139. Berliner, D. C. (1986). In pursuit of the expert pedagogue. Educational Researcher, 15(7), 5-13. Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999). How people learn. Washington, DC: National Academy Press. diSessa, A. A. (2006). A history of conceptual change research: Threads and fault lines. In R. K. Sawyer (Ed.). The Cambridge handbook of the learning sciences (pp. 265-282). New York: Cambridge University Press. diSessa, A. A., Gillespie, N. M., & Esterly, (2004). Coherence versus fragmentation in the development of the concept of force. Cognitive Science, 28, 843-900. Hiebert, J., & Carpenter, T. (1992). Learning and teaching with understanding. In D. A. Grouws (Ed.) Handbook of research on mathematics teaching and learning (pp. 65-97). Reston, VA: National Council of Teachers of Mathematics. Hill, H. C., Ball, D. L., & Schilling, S. G. (2008). Unpacking pedagogical content knowledge: Conceptualizing and measuring teachers topic-specific knowledge of students. Journal for Research in Mathematics Education, 39(4), 372-400. Hill, H. C., Rowan, B, & Ball, D. L. (2005). Effects of teachers mathematical knowledge for teaching on student achievement. American Educational Research Journal, 42(2), 371-406. Izsk, A., Orrill, C. H., Cohen, A., & Brown, R. E. (2010). Measuring middle grades teachers understanding of rational numbers with the mixture Rasch model. Elementary School Journal, 110(3), 279-300. Kersting, N. B., Giviin, K. B., Sotelo, F. L., & Stigler, J. W. (2002). Teachers analyses of classroom video predict student learning of mathematics: Further explorations of a novel measure of teacher knowledge. Journal of Teacher Education, 61(1-2), 172-181. Leinhardt, G., & Greeno, J. G. (1991). The cognitive skill of teaching. In P. Goodyear (Ed.) Teaching knowledge and intelligent tutoring (pp. 233-268). Norwood, NJ: Ablex Publishing Company. Sabers, D. S., Cushing, K. S., & Berliner, D. C. (1991). Differences among teachers in a task characterized by simultaneity, multidimensionality, and immediacy. American Educational Research Journal, 28(1), 6388. Shaffer, D.W. & Graesser, A. (2010). Using Quantitative Model of Participation in a Community of Practice to Direct Automated Mentoring in an Ill-Formed Domain. Paper presented at Intelligent Tutoring Systems (ITS), Pittsburgh, PA. Shaffer, D. W., Hatfield, D., Svarovsky, G. N., Nash, P., Nulty, A., Bagley, E., Frank, K., Rupp, A. A., & Mislevy, R. (2009). Epistemic network analysis: A prototype for 21st century assessment of learning. International Journal of Learning and Media, 1(2), 33-53. Shechtman, N., Roschelle, J., Haertel, G., Knudsen, J. (2010). Investigating links from teacher knowledge, to classroom practice, to student learning in the instructional system of the middle-school mathematics classroom. Cognition and Instruction, 28(3), 317-359.
Acknowledgments
This work was supported by the National Science Foundation through grants REC-034700, DUE-0919347, DRL-0918409, DRL-0946372, and EEC-0938517, DRL-0903411, DRL-1125621, and DRL-1054170 and the MacArthur Foundation. The views expressed here are those of the authors and do not necessarily reflect those of the funding agencies. The authors thank Joanne Lobato and Erik Jacobson for their assistance in collecting the data analyzed in this paper.
ISLS
179
ICLS2012
Investigating the Effects of Varying Labels as Scaffolds for Visitor Learning

Joyce Wang a, Susan Yoon a, Karen Elinich b, and Jacqueline Van Schooneveld a a University of Pennsylvania, Philadelphia, PA b The Franklin Institute, Philadelphia, PA joycew@gse.upenn.edu, yoonsa@gse.upenn.edu, kelinich@fi.edu, jflicker@dolphin.upenn.edu Abstract: In museum literature, labels have been found to increase visitor learning and contribute to greater cognitive gains. In this study, we seek to understand how various labels support the visitors learning experience, and specifically in regards to conceptual and cognitive learning. We investigated the increasing use of three labels (visual digital augmentations and text-based questions and instructions) in four treatment groups of middle school students. Our analyses indicated that when labels were used, there were significant increases in both conceptual and cognitive understanding. Furthermore, observation notes demonstrated that in the highest treatment group, there were increased instances of problem solving behavior as well as association with prior knowledge.
Introduction
Informal environments, such as science museums, have the potential to offer rich learning experiences (NRC, 2009). Though many studies have determined that museum visits increase visitors curiosity, motivation, and interest in science, less have presented empirical evidence of visitors cognitive or conceptual gains (Allen, 2007; Anderson et al., 2003; Falk et al., 2004). As such, we resonate with this years ICLS conference theme, The Future of Learning, to investigate how museums can improve conceptual and cognitive learning through innovative technological advances, as well as through promising research of new approaches to learning. Most commonly, museums support visitors learning through scaffolds incorporated into exhibit designs. Thus, we examine how various scaffolds affect museum visitors learning experiences (NRC, 2009). This study follows a series of studies that investigate how designed interactives enhance learning in the museum setting (Yoon et al., 2011a, 2011b). In this study, we seek to understand how different forms of labels support visitors learning. We conceive of labels as a type of scaffold that can vary in their representational nature to reveal information visitors would otherwise be unaware of or unable to acquire by themselves. Although labels are traditionally presented as printed text (typically explanatory in nature), we expand this boundary to include any form of a guide that scaffolds a learning experience. We compare the learning outcomes of middle school students placed in several treatment groups that vary in the amount and type of labels as they interact with an exhibit aimed at teaching circuits and electrical conductivity. The research question investigated was How does the increasing use of various representations of labels affect students ability to make sense of the information with which they are being presented?
Conceptual Framework
In museum literature, labels, originally defined as written words used alone or with illustrations in museum exhibitions to provide information for visitors, presented as text on exhibit graphic panels or computer screens (Serrell, 1996, p.239), have been found to impact visitors experiences at the exhibits in various ways. When they are written clearly to convey the goals of the exhibit, more visitors will understand, find meaning in, and enjoy museum exhibitions. Some studies have documented how different types of labels change the way visitors interact with exhibits (e.g. Atkins et al., 2009) while other studies have investigated how labels affect the type of conversations that ensue between group members (e.g., Hohenstein & Tran, 2007). Ultimately, labels seek to increase visitors learning and to contribute to greater cognitive gains (Falk, 1997; NRC 2009). In this study, we want to build upon these claims and examine different ways in which labels can be constructed to elicit deeper understanding. We investigated two representations of three labels: visual digital augmentations and text-based questions and directions. We hypothesized that these labels, when used together, would collectively target and enable students to construct their own knowledge, and would reflect an increase in their level of thinking and learning. In reframing the conception of a museum label more generally as a scaffold, we expand upon the conventional consideration of labels as solely offering descriptive information to support the individual visitor, to include ways in which labels purposefully change visitors behaviors to support each others learning and consequently, their own learning as well.
ISLS
180
ICLS2012
Digital Augmentation Labels (through Augmented Reality)

At its simplest, Augmented Reality (AR) describes systems that integrate computer-generated virtual elements (objects or environments) or information with the real world environment (Mou et al., 2004). Specifically by superimposing virtual elements and images (which we define as digital augmentations) onto the real world environments, AR allows users to experience and perceive the newly incorporated information as part of their present world, thereby enhancing their perception and interaction with the real world (Kirkley & Kirkley, 2004). In this way, AR serves as a scaffold by supporting the user with additional (virtual) information, which might not be directly detected by their senses, to aid in their performance of specific tasks. It is precisely in this scaffolding role that AR offers the unique potential to transform learning at multiple levels. For this reason, we consider these digital augmentations as a type of label in our study. In classrooms, AR has been employed in the form of handheld games to teach mathematical and scientific concepts (e.g. Dunleavy et al., 2009; Klopfer & Squire, 2008) and to cultivate the development of inquiry skills, argumentation skills, and collaborative learning (e.g. Rosenbaum et al., 2006). For example, Klopfer and Squire (2008) developed a Pocket PC game, enabled with augmented reality, to investigate a chemical spill in a watershed that could potentially contaminate ground and surface water. One of the outcomes of the study indicated that students became active problem solvers who learned to critically analyze strategies and negotiate with each other. They were often observed arguing about different investigative strategies that they believed would give them the most reliable information to formulate a solution (Klopfer & Squire, 2008). Although AR has been used in school settings to scaffold learning, in museums, it is still a fairly new phenomenon. While a few studies have investigated the impact of AR devices on visitors experiences (e.g. Szymanski et al., 2008), more have evaluated the design and creation of an AR system and the feasibility of incorporating AR into the museum environment (e.g., Damala et al., 2008). A significant finding of several of these studies reveals that AR increases visitors engagement and interest in museum settings when using these devices (e.g., Hall & Bannon, 2006). While this is an important component of museum experiences, this focuses primarily on engagement and access rather than on learning. In terms of conceptual and cognitive learning, there has been a dearth of research that investigates the impact that AR has on visitors.
Question Labels
Text-based question labels were also incorporated in our investigation as a method to scaffold learning. Questions, as a pedagogical strategy, have the potential to stimulate visitors to think, look, get involved with, and converse with others to ultimately learn about a particular exhibit (Serrell, 1996). In interactive exhibits, question labels can invite visitors to actively participate in an exhibit and thereby integrate their experience with the exhibits main message (Serrell). In some cases, they invite and challenge the visitor to use the exhibit to accomplish a certain task (Gutwill, 2006). Regardless of the method of invitation, questions in labels act to encourage physical engagement of visitors, thereby helping them to anticipate and formulate new meanings (Serrell, 1996). A few studies have evaluated the effectiveness of incorporating questions in exhibit labels to stimulate participation and learning. For example, Hohenstein and Tran (2007) examined the use of guiding questions to stimulate conversations in an object-based science museum. Their findings indicated that in the presence of these questions, visitors were more likely to not only offer explanatory answers to the questions, but to also ask their own open-ended questions. However, it should be noted that the addition of these questions did not have the same effect on all exhibits. Thus, the context of the exhibit must also be considered when creating labels to assist visitors experiences. A different study also investigated the impact that questions had on visitor behavior (their learning and their conversations) in their interaction with objects at a science exhibit (Atkins et al., 2008). After including a hybrid label, a label that involved both questions and suggestions for improved visitor engagement, the exhibit was more likely to be perceived as a lesson with an intentional goal and a particular learning agenda. Specifically, in the presence of the labels, parents were observed to explain the phenomena to their children more often than when the label was not present (Atkins et al., 2008). As such, these studies indicate that text-based questions have the potential to encourage visitors to become active learners and thus engage in deeper levels of critical thinking that develop increases in conceptual knowledge.
Instructional Labels to Promote Collaboration

The last label that our study sought to investigate are labels that instruct visitors on collaboration. Instructional labels have been used to guide visitors in using exhibit devices to produce a particular outcome that illustrates a scientific phenomenon (Gutwill, 2006). In instances like these, labels provided directions that signal to the visitor whether the device worked or not depending on how the visitor operated the device. Unfortunately, the disadvantage of these types of instructional labels is that they may communicate an endpoint or finality that would inhibit further exploration by the visitor (Gutwill, 2006). While these studies have acknowledged how museums have commonly used labels to inform visitors of specific ways to interact with exhibits, there have
ISLS
181
ICLS2012
been a lack of studies that not only investigate the effects of labels that instruct visitors to collaborate with one another to build knowledge, but that indicate that museums are designing these types of labels. In formal education, collaborative learning has been found to have a positive impact upon student achievement (ODonnell, 2006). Regardless of how the groupings are formed, whether homogenous or heterogenous for ability, when students collaborated together, they exhibited increased levels of achievement and motivation (Saleh et al., 2007). A critical factor that impacts students learning in these groups, however, is the extent to which structures are in place to support effective collaboration skills. Many studies have investigated the effect of various scaffolds that support collaboration, including the assignment of student roles, cues, scripts, and rules (Kester & Paas, 2005; Saleh et al., 2007). For example, it was found that when students were verbally cued to summarize, explain, identify errors, or ask for help from one another, they demonstrated higher learning gains. These studies indicate the impact and potential that collaboration, when designed for, can have on learning in the formal environment. Consequently, these results can also offer important considerations for informal environments. As labels are an elemental component within the museum space, we suggest modifying them in a way to encourage visitors to collaborate more. Unfortunately, there has been a dearth of studies that investigate how labels have been designed to encourage visitor collaboration. Thus, we attempt to address this gap in the literature by investigating how instructions for collaborative interaction, incorporated in text-based labels, affect visitors learning and behaviors.
Methods Context & Participants

This study leveraged a currently funded large scale National Science Foundation informal science education project in which the central goal is to design, integrate, and increase the use of educational technologiesand to study their impactwithin the science museum learning experience (Yoon et al., 2011a, 2011b). While the larger project focuses on the viability of using augmented reality technologies in informal science settings, the present study extended this focus by investigating the effects of various labels to scaffold understanding. In total, 164 students, from nine diverse schools (including public, charter, urban and suburban) participated. The participants were 6th, 7th, and 8th grade students, chosen because the device addresses electricity and magnetism, topics which are normally covered in grade 4 of the local school curriculum. Thus, all students would already have some prior knowledge. In order to recruit student participants, we enlisted the help of science teachers who were associated with either the research institution or the museum. Students participation in our study was embedded in a larger school field trip. Prior to the day of implementation, researchers went to schools to administer presurveys to the students. On the day of implementation, students were randomly assigned to visit a particular exhibit from which researchers would pull students in pairs or groups of 3 and randomly assign them to one of four conditions (C1C4) to test the study device. Refer to Table 1 for participant breakdown and a description of the conditions. (The variance in participant numbers for C4 is due to a larger study, consisting of 6 conditions that required greater participant numbers (Yoon et al., 2012). Whereas the larger study sought to investigate the effects of certain structured knowledge building scaffolds on visitor learning, this study analyzes the knowledge building scaffolds as prompts and labels that merely served to complement the device.) The device Be The Path consisted of a table with two metal spheres (located at opposite ends of the table), connected in a half loop with wires to a battery and a light bulb. As is, the loop is not fully closed (and is referred to as an open circuit) so the bulb does not light. When individuals place their hands on the metal spheres, the bulb would light up because their body closed the loop to complete the circuit. In the testing room, students were told to play with the device as if they had come across it on the museum floor while two researchers recorded observation notes of students behavior. After their engagement with the device, the students were led to a different location to answer some questions regarding their interaction and to reflect upon what they learned. (These reflection questions were the same questions on the question labels in conditions 3 and 4.) Following the implementation, researchers administered a post-survey (same as the pre-survey) to the students. Table 1: Description of each condition
Cond. 1 2 N 30 31 Description Control: Device without any additional scaffolds and representative of the current devices out on the exhibit floor. Addition of visual digital augmentation label (DA): When students completed the circuit and made the bulb light up, a projected image (of moving electrons) would appear from overhead and map onto the device and the students bodies. Label s None DA
ISLS
182
ICLS2012 3 31
Volume 1: Full Papers Addition of question labels (QL): 1. What happened when you touched both metal spheres? 2. What happened when you only touched 1 metal sphere? 3. What happened to make the bulb light up? 4. What does the projection show? 5. What are you supposed to learn by using this device? Addition of instruction labels (IL): Write down your answer only after your group has come to agreement on it. Be clear about the difference between what you actually observe happening and what you think about it. It is important that you listen to each other. For each question: 1. Brainstorm possible answers. 2. Give reasons to support your answers. 3. Compare your answers and the reasons behind them. 4. Decide on the best possible answer. DA + QL
72
DA + QL + IL
Data Sources & Analyses

1. Pre- and post-surveys were gathered to measure changes in students conceptual understanding before and after the implementation. These surveys consisted of 5 multiple choice questions on electricity and magnetism as well as 1 open-ended question (Think about an electric circuit that supplies electricity to a light bulb. What parts make it work so the bulb lights up? Use words and pictures in your answers.) Although we included multiple choice questions (compiled from district curricular materials) to collect some baseline data of students general understanding of electricity, the concept that our device was demonstrating was most related to the open-ended question. A categorization manual was constructed to evaluate this last question and interrater reliability was obtained on 20% of the responses by two independent researchers ( = 0.94). The open-ended responses were coded on a scale from No Understanding (Level 1) to Complete Understanding (Level 5). Paired samples t-tests were conducted for each condition to determine whether there were significant differences in students understandings before and after the intervention. An ANOVA was also conducted to determine which particular label had the greatest impact. Student written reflections were solicited regarding their understanding of the science phenomenon being presented. Although there were five questions asked, only one question (What were you supposed to learn by using this device?) was coded, as it most clearly illustrated their ability to think critically, theorize, and make sense of the phenomenon. Thus, this question intended to evaluate changes in cognitive learning. (These reflection questions were also the same questions on the question labels in C3 and C4). A separate categorization manual was also constructed to code these responses (on a scale from No Understanding (Level 1) to Complete Understanding (Level 5)) and interrater reliability was obtained on 20% of the responses by two independent researchers ( = 0.948). An ANOVA was conducted to determine whether the differences were statistically significant. Researcher observation notes were collected and coded to register different types of critical thinking skills that the students were seen engaging in. The categories, adapted from the Critical Thinking Skills Checklist described in Luke et al. (2007) and refined to reflect the typical experiences of a science museum, were: Participating, Interpreting, Evaluating, Associating with Prior Science Knowledge, Problem Solving, Associating with Other Knowledge, and Engaging in Teamwork. These seven categories were further divided into more refined subcategories (for example, Interpreting included behaviors such as asking questions and exploring how the device operated) and were then coded as 1 or 0 depending on whether the students exhibited these components. The frequencies of these exhibited skills were then compared across the 4 conditions and a chi-square test was administered to determine if the results were statistically significant.
2.
3.
Results Pre-/Post-Surveys (Measuring Conceptual Learning): Aggregated & Disaggregated for Multiple Choice and Open-Ended Questions
A paired-samples t-test indicated that significant learning gains were made between the pre-surveys (M = 4.22, SD = 1.59) and post-surveys (M = 4.81, SD = 1.60), t(163) = -5.04, p = .01. Specifically, as our device targeted the conceptual understanding in the open-ended response question, we then disaggregated the data to analyze these scores independent of the entire survey score (which was comprised of the multiple choice and openended questions). As shown in Table 2, conceptual knowledge gains were also reflected in students open ISLS 183
ICLS2012
ended responses between the pre- and post-surveys. Between the conditions on just the post-survey scores, there was a general increase in understanding on the open-ended questions, which were reflected in increases on the entire test as well (see Table 3 for means). However, an ANOVA conducted between the four conditions revealed that these learning gains as reflected in the open-ended question, F(3, 160) = 1.82, p = 0.15, and in the test as a whole, F(3, 160) = 1.25, p = 0.29, were not a direct result of the incorporation of labels. Table 2: Paired-samples t- test results of open-ended responses on pre-/post-surveys
Condition Cond. 1 Cond. 2 Cond. 3 Cond. 4 Test Pre (OE) Post (OE) Pre (OE) Post (OE) Pre (OE) Post (OE) Pre (OE) Post (OE) Mean 2.17 2. 27 2.19 2.65 2.03 2.48 2.25 2.72 SD 1.15 0.78 0.95 0.84 1.08 1.09 0.92 0.97 t value (df) 0.47(29) -2.72(30) -2.04(30) -4.6(71) p 0.64 0.01* 0.05* 0.00*
Table 3: ANOVA results of post-survey scores (aggregated and disaggregated) between conditions
Condition Cond. 1 Cond. 2 Cond. 3 Cond. 4 Post MC Scores (Mean, SD) 2.37, 1.16 2.16, 0.82 2.10, 0.94 2.29, 1.20 Post OE Scores (Mean, SD) 2.27, 0.78 2.65, 0.84 2.48, 1.09 2.72, 0.97 Post-Survey Scores (Mean, SD) 4.63, 1.50 4.74, 1.29 4.58, 1.59 5.01, 1.60
Student Reflections (Measuring Cognitive Learning)

Although there seemed to be a generally increased ability to think and reason at a higher level as more scaffolds were added between C1 (M = 2.23, SD = 0.50), C2 (M = 2.39, SD = 0.76), C3 (M = 2.48, SD = 0.72), and C4 (M = 2.43, SD = 0.80), the main effect of the labels was not significant, F(3, 160) = 1.93, p = 0.13. However, an ANOVA performed between C1 and all other conditions aggregated together, indicated that there was a significant difference F(1, 162) = 5.84, p = 0.02. The observation notes reflected some differences in the students interactions between the four conditions. Although there were seven skills being observed, a chi-square test revealed that there were significant differences in the frequencies of only two categories: Problem Solving (x2 = 9.166, df = 3, p < .027) and Association with Prior Science Content Knowledge (x2 = 12.731, df = 3, p < .005). C4 and C2 students exhibited the greatest instances of demonstrating Problem Solving processes (represented by behaviors that indicated investigation of various configurations in lighting the bulb). Specifically, 96% and 94% of C4 and C2 students respectively reflected these behaviors, as compared to students in C3 (78%) and C1 (84%). Likewise, students in C4 tended to associate their experience with other science knowledge (usually involving recall of similar activities performed in science classes) (26%) more often than students in C3 (9.7%), C2 (13%) and C1 (0%).
Observation Notes
Discussion & Implications

Museum studies have consistently acknowledged the importance of exhibit labels on visitors learning and behaviors (Atkins et al., 2009; Falk, 1997; Gutwill, 2006; Hohenstein & Tran, 2007). In accordance with this, the goal of our study was to evaluate, not only the effects of various types of labels on students learning, but also the effects of increasing the use of these labels on students conceptual and cogntivie understanding of electricity. Through pre- and post-surveys (to evaluate conceptual understanding), written reflections (to evaluate cognitive skills) and observation notes (to evaluate behaviors that would indicate changes in either conceptual or cognitive understanding), we wanted to understand how students made sense of the phenomenon in both a performative and generative context. To this end, we found significant increases in students conceptual understanding after the museum visit, consistent with prior research (NRC, 2009), for students in C2, C3, and C4. Looking within each condition, we found that Condition 2, 3, and 4 students, collectively, after our intervention, significantly understood better the components and operation of an electric circuit. On average, students understanding increased by half a point (which is reflective of half a level of understanding) a significant result considering
ISLS 184
ICLS2012
that their interaction with the device was, like most interactions with museum exhibit devices, brief. This positive finding suggests that our device, when scaffolded with various labels, supports students learning of electricity and ciruits. Additionally, at first glance, we noticed that the increase in students conceptual understanding was in proportion to the number of labels they received. For example, it appeared that students in C4, who experienced the most labels, exhibited the most understanding as they had the highest scores. However, an ANOVA conducted across the four conditions indicated that these differences in knowledge growth were not significant. Thus the condition, and subsequently, the labels that comprised each condition, had no significant effect on students understanding. A C4 student would increase their knowledge as much as a C2 student. However, as C1 students who didnt receive any labels also did not exhibit any significant knowledge growth, this would suggest that at least one label is needed to significantly impact learning. Thus, this implies that the amount and types of labels that one can use to scaffold a visitors experience does not make a difference. In regards to the impact of labels on cognitive learning, we evaluated students responses on their reflections. Specifically, we hoped that the digital augmentation labels, question labels, and instruction labels on collaborative work, would to varying degrees, affect students ability to theorize about how our bodies can complete a circuit and conduct electricity. Our results indicated that initially, it appeared that as we increased the amount of labels to scaffold their learning, they exhibited greater abilities to think and theorize at a higher level. However, similar to their survey results, an ANOVA on their reflections determined that the increases in their ability to theorize were not significant across the four conditions. Interestingly though, when we aggregated Conditions 2, 3, and 4 together to represent Labeled Conditions and compared this to Condition 1, the differences in their cognitive scores were significant. Thus, this confirms our earlier speculation that the amount and type of labels are not as significant in learning as the presence of just one of these labels. In addition to conceptual and cognitive learning, we were interested in understanding how these three labels affected students behaviors and interactions, that would help us to understand their changes in learning. A chi-square test revealed that there were significant differences in students behaviors in two categories: Problem Solving and Association with Prior Knowledge. Specifically, we saw a marked difference in C4 students interactions; as this label pointedly instructed students on ways to collaborate together, it was encouraging to see more students investigating and attempting additional configurations to light the bulb. For example, a few groups discovered that when the students all linked hands, the bulb would light up. Other students would experiment using different parts of their body including their elbows, fingertips, and the backs of their hands. The discovery of these phenomena often led to expressions of excitement and wonder. Other studies have confirmed that museums can stimulate joy, delight, awe, wonder, appreciation, surprise, etc. (NRC, 2009) and that these affeective reponses are linked to other forms of learning (e.g., Steidl et al., 2006). It was interesting to note that C2 students also demonstrated very high instances of Problem Solving behaviors. This would suggest to us that the novelty of the digital augmentation label by itself, was just as impactful as the instructional label in encouraging this kind of behavior. When the digital augmentation was combined with another label, specifically the question label, however, this exploratory behavior decreased. Likewise, C4 and C2 students were observed making associations with other knowledge in more instances than C3 or C1 students (though the frequency for this behavior is not as similar as that of Problem Solving). The labels in these conditions encouraged students to connect their interaction between our device and other knowledge gained from prior experiences. Some recalled similarities to other school science class activities while others remembered that they had learned about this content previously. In C4, because our instructional labels strongly influenced students to collaborate together, we observed greater amounts of dialogue relating to electricity (though this was not formally measured). Thus, it is not surprising to us that as students are talking about these questions and sharing their observations and hypotheses, their conversations would include more instances of connections to prior knowledge. Though we did not find as many significant results as we had anticipated, we would not rush to conclude that these labels are ineffective. Instead, we suggest revisitng the context and conditions in which our intervention was administered as well as the design of the instruments themselves. First, we posit that the survey question may not have been sensitive enough to measure the smaller differences in students thinking and learning. We worded the question broadly so that students would not feel constrained to answer in a particular manner. However, as a number of responses were too general (i.e., We learned about electricity.) to offer any significant information, we were not able to accurately assess what students learned and thus, had to code their responses as No Understanding. Second, the variation in learning results could be explained by the different forms of assessments that were used to measure thinking (hands-on versus paper-pencil) and to the different environments in which the students were assessed. We suggest there may have been a failure of transfer between what was observed and written. Educational researchers have long since theorized that a learning experience may look differently depending on the testing context, as the manner in which the information is learned affects transfer (Bransford & Schwartz, 1999). Thus, while it appeared that students were able to think at a higher level when they were engaging with the device (especially in C4), this aptitude did not transfer once students were removed from the original context and asked to reflect upon it in writing. Lastly,
ISLS
185
ICLS2012
although we tried to control for this, it was apparent that many students rushed through both the reflections and the post-surveys in an attempt to hurry on to the next activity. For example, some students remarked impatiently that they wanted to finish the reflections quickly so they could return to the main exhibits and continue their exploration. Similarly, during the administration of the post-surveys, many students objected that this was exactly the same thing that they answered previously. Although the researchers explained the rationale for requiring both a pre- and post-survey to the students, informal comparisons (acquired during the collection of the post-surveys) reflected that students had not put as much effort and thought into the post-surveys as they did in the pre-surveys. In the next iteration of this study, we suggest collecting this information in a different format, perhaps embedded in interviews This study sought to investigate the effects of layering three types of scaffolds, in the form of labels, on students conceptual and cognitive learning. While the results didnt support all of our hypotheses, we did learn that the addition of at least one of our labels significantly increases conceptual learning. Furthermore, we were encouraged by changes in students interactions to reflect more scientific inquiry behaviors. As we begin to plan for future phases of this study, the findings weve discovered here will guide our decisions as we seek to revisit the context, methods and instruments used to evaluate museum learning.
References
Allen, S. (2007). Secrets of circles summative evaluation report. Report prepared for the Childrens Discovery Museum of San Jose. Available: http://www.informalscience.org/evaluation/show/115 Anderson, D., Lucas, K., & Ginns, I. (2003). Theoretical perspectives on learning in an informal setting. Journal of Research in Science Teaching, 40(2), 177-199. doi: 10.1002/tea.10071 Atkins, L., Velez, L., Goudy, D., & Dunbar, K. (2009). The unintended effects of interactive objects and labels in the science museum. Science Education, 93, 161-184. doi: 10.1002/sce.20291 Bransford, J. & Schwartz, D. (1999). Rethinking transfer: A simple proposal with multiple implications. Review of Research in Education, 24, 61-100. doi: 10.2307/1167267 Damala, A., Cubaud, P., Bationo, A., Houlier, P., & Marchal, I. (2008). Bridging the gap between the digital and the physical: Design and evaluation of a mobile augmented reality guide for the museum visit. In 3rd ACM International Conference on Digital and Interactive Media in Entertainment and Arts, 120-128. New York: ACM Press. Dunleavy, M., Dede, C., Mitchell, R. (2009). Affordances and limitations of immersive participatory augmented reality simulations for teaching and learning. Journal of Science Educational Technology, 18, 7-22. doi:10.1007/s10956-008-9119-1 Falk, J. H. (1997). Testing a museum exhibition design assumption: effect of explicit labeling of exhibit clusters on visitor concept development. Science Education, 81, 679-687. doi: 10.1002/(SICI)1098237X(199711)81:6<679::AID-SCES>3.3.CO;2-7 Falk, J., Scott, C., Dierking, L., Rennie, L., & Jones, M. (2004). Interactives and visitor learning. Curator, 47(2), 171-198. doi: 10.1111/j.2151-6952.2004.tb00116.x Gutwill, J. (2006). Labels for open-ended exhibits: Using questions and suggestions to motivate physical activity. Visitor Studies Today, 9(1), 1-9. Hall, T., & Bannon, L. (2006). Designing ubiquitous computing to enhance childrens learning in museums. Journal of Computer Assisted Learning, 22(4), 231-243. Hohenstein, J., & Tran, L. U. (2007). Use of questions in exhibit labels to generate explanatory conversation among science museum visitors. International Journal of Science Education, 29(12), 1557-1580. doi: 10.1080/09500690701494068 Kester, L., & Paas, F. (2005). Instructional interventions to enhance collaboration in powerful learning environments. Computers in Human Behavior, 21, 689-696. doi:10.1016/j.chb.2004.11.008 Kirkley, S., & Kirkley, J. (2004). Creating next generation blended learning environments using mixed reality, video games and simulations. TechTrends, 49(3), 42-89. doi: 10.1007/BF02763646 Klopfer, E., & Squire, K. (2008). Environmental Detectives the development of an augmented reality platform for environmental simulations. Educational Technology Research and Development, 56, 203-228. doi: 10.1007/s11423-007-9037-6 Luke, J., Stein, J., Foutz, S., & Adams, M. (2007). Research to practice: Testing a tool for assessing critical thinking in art museum programs. Journal of Museum Education, 32(2), 123-136. Mou, W., Biocca, F., Owen, C., Tang, A., Xiao, F., & Lim, L. (2004). Frames of reference in mobile augmented reality displays. Journal of Experimental Psychology: Applied, 10(4), 238-244. doi: 10.1037/1076898X.10.4.238 National Research Council (NRC). (2009). Learning Science in Informal Environments: People, Places, and Pursuits. Washington, DC: National Academies Press. ODonnell, A. (2006). The role of peers and group learning. In P. H. Winne & P. A. Alexander (Eds.), Handbook of educational psychology (pp. 781-802). Erlbaum: Mahwah, NJ.
ISLS 186
ICLS2012
Rosenbaum, E., Klopfer, E., & Perry, J. (2006). On location learning: Authentic applied science with networked augmented realities. Journal of Science Education and Technology, 16(1), 31-45. doi:10.1007/s10956006-9036-0 Saleh, M., Lazonder, A., & de Jong, T. (2007). Structuring collaboration in mixed-ability groups to promote verbal interaction, learning, and motivation of average-ability students. Contemporary Educational Psychology, 32, 314-331. doi:10.1016/j.cedpsych.2006.05.001 Serrell, B. (1996). Exhibit labels: An interpretive approach. Walnut Creek, CA: AltaMira Press. Steidl, Sl., Mohi-uddin, S., & Anderson, A. (2006). Effects of emotional arousal on multiple memory systems: Evidence from declarative and procedural learning. Learning & Memory, 13(5), 650-658. Available: http://www.ncbi.nlm.nih.gov/pubmed/17015860 [accessed October 2011]. Szymanski, M., Aoki, P., Grinter, R., Hurst, A., Thomton, J. & Woodruff, A. (2008). Sotto Voce: Facilitating social learning in a historic house. Computer Supported Cooperative Work, 17(1), 5-34. Yoon, S., Elinich, K., Steinmeier, C., Wang, J., & Tucker, S. (2011a). Learning science through knowledgebuilding and augmented reality in museums. In the proceedings of the annual conference of the International Conference on Computer Supported Collaborative Learning, Hong Kong, China. Yoon, S., Elinich, K., Wang, J., Steinmeier, C., & Van Schooneveld, J. (2011b). Fostering critical thinking in science museums through digital augmentations. In the proceedings of the annual conference of the International Conference on Computer Supported Collaborative Learning, Hong Kong, China. Yoon, S., Elinich, K., Wang, J., & Van Schooneveld, J. (2012, April). Augmented reality and knowledge building: To what extent can they enhance learning in a science museum? Paper to be presented at the annual meeting of the American Educational Research Association, Vancouver, British Columbia, Canada.
Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant No. 0741659. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
ISLS
187
ICLS2012
Making Technology Visible: Connecting the Learning of Crafts, Circuitry and Coding in Youth e-Textile Designs
Yasmin B. Kafai, Deborah A. Fields, & Kristin A. Searle University of Pennsylvania, 3700 Walnut Street kafai@upenn.edu, stareyes@gmail.com, searle@dolphin.upenn.edu Abstract: We examine high school students designs with the LilyPad Arduino, an electronic textile (e-textile) construction kit used for designing programmable garments. Each kit contains a microcontroller, sensors and LED and other actuators that can be embedded in textiles. We conducted three workshops with 35 high school youth between 14-15 years in a science museum to document, describe and develop a framework for analyzing student learning. The analyses of workshop interactions, students artifacts and reflections indicate how the fabrication of stitches, circuits and code reveals the underlying structures and processes of craft, engineering and programming in tangible and observable ways and renders visible how technology is designed and built. We discuss how this situated nature of e-textiles artifact production provides a promising context for student learning and generates new instructional designs of workshops and computational construction kits.
Introduction
This paper focuses on learning with a new type of tangible, programmable media called electronic textiles (etextiles hereafter). E-textiles include young people's designs of programmable garments, accessories, and costumes. Such designs incorporate elements of embedded computing for controlling the behavior of fabric artifacts, novel materials such as conductive fibers or Velcro, sensors for light and sound, and actuators such as LEDs and speakers, in addition to traditional aspects of fabric crafts using needles, thread, and cloth among others. The development of the LilyPad Arduino (Buechley, 2006) has made e-textiles accessible to novice designers in the same way the design of Lego/Logo (Resnick & Ocko, 1991) did for robotic constructions. Etextiles introduce aesthetic elements that can enrich youth's expressive and intellectual lives and materials that make programming accessible to new groups (Eisenberg, Eisenberg, Buechley & Elumeze, 2006). To date, most of the work around e-textiles has focused on technical developments to test usability and design of materials and activities. We take the next step in documenting, describing and analyzing the learning that takes place as high school youth engage in designing e-textile artifacts. In understanding students design processes and products, we focus on how the craft activities situate and connect engineering and programming in e-textiles, making abstract concepts concrete for learners and making the learning that is (or is not) taking place visible to researchers. This is not just an effort to emphasize interdisciplinary connections present in e-textile designs but also to highlight what Eisenberg et al. (2006; see also Buechley, 2010) called the invisibility principle that when technology is invisible, it is deliberately placed outside the users awareness; thus there is little reason to communicate how the technology works, and how the user might extend or control it. (p. 2). It might be productive for expert programmers to know that their black-boxed programs may be taken up exactly and used as they are (Turkle & Papert, 1992), likewise average users might find it appropriate and functional not knowing how computer or software works. However, for learners this invisibility might be counterproductive in some instances because students are unable to grapple with the messiness of the technology: to take things apart and put them back together in a multitude of ways (Resnick, Berg & Eisenberg, 2000). In e-textiles, the fabrication of stitches, circuits and code can reveal the underlying structures and processes of craft, engineering and programming in tangible and observable ways and renders visible how technology is designed and built. Our analyses focus on a series of workshops that took place in a science museum partnering with an urban public high school during the 2010-11 school year. A total of 35 female and male high school freshmen participated in designing e-textiles. While learning how to sew, how to make functional circuits, and how to write code to light up LEDs and control sensors are each important and valuable practices in their own right, learning how to see the connections between these parts is what makes learning to design e-textiles greater than its individual parts. How do young designers see the connections between the stitches they make and the functioning of the circuits they have drawn? How do they understand how writing the code relates to the circuits they have laid out? How can they craft pieces that allow changes in code and vice versa? Finally, what do students conceptualizations of the relationships between craft, circuitry, and code tell us about what they are learning through their engagement with the LilyPad Arduino construction kit? These research questions are at the heart of the analytical framework we developed to understand students learning with and through e-textiles. In the next sections, we review what we know about learning with tangible programmable artifacts at large.
ISLS
188
ICLS2012
Background
E-textiles are a relatively new development (Post, Orth, Russo, & Gershenfeld, 2000) and only in recent years have e-textile construction kits become available to novice programmers (Buechley, 2006; Ngai, Chan, Cheung, & Lau, 2009; Katterfeldt, Dittmar, & Schelhowe, 2009). Most of what we know about learning with such tangible construction kits comes from research on robotics activities developed and implemented across K-16 settings (Bers, 2009; Druin & Hendler, 2000). In general, these studies find robotics to be engaging activities for students of all ages (and even participating parents) but rarely provide much detail on how and what students learn in constructing a robot. A notable exception is a study by Sullivan (2008) that aligned learning robotics with practices and themes relevant for science literacy. She found that building robotics engages middle school students in (1) the development of certain thinking skills such as computation and estimation, manipulation and observation, communication, and critical-response skills; (2) the ability to engage in the activity of inquiry and (3) the development of an understanding of the common themes of science such as systems, models, constancy and change and scale (p. 374). Robotics activities are now a prominent part of afterschool programs and have developed over the last 10 years into a nation-wide network of competitions. These findings are relevant for our work because e-textile activities share many common elements with robotics activities but are also distinct in some aspects. Like robotics, e-textiles involve designing and constructing physical artifacts that operate with different sensors and actuators and coordinating interactions via writing of program code. Unlike robotics, the purpose of these artifacts is not to compete but rather is aligned with more personal, decorative goals. Both construction activities are situated in different domains: in the case of robotics the artifacts are associated with the engineering of motors while in the case of e-textiles the artifacts are associated with the crafts of sewing and embroidery. Designing e-textile and robotics artifacts involve an array of different activities, each complex in its own right, that need to be coordinated. Understanding the learning of crafts, circuitry and code then draws on learning in multiple disciplinary contexts that have a longstanding disciplinary tradition, but with e-textiles they venture into newer territory. The crafting involved in e-textiles is the most unusual practice and also the one that has a minimal foothold in traditional K-12 curricula, at least in the United States. Initially part of home economics, crafting involved sewing, embroidery, and knitting for girls and the wood and metal shop and for boys. Many of these crafts activities have been relegated to vocational schooling. Nowadays, the crafts are experiencing a renaissance outside of school in DIY communities as well as in fabrication labs that allow for personal manufacturing (Frauenfelder, 2010; Gauntlett, 2011). On the other hand, learning about circuitry, and by extension about electricity, is an established part of the K-12 science curriculum. Numerous studies have documented students difficulties in understanding the concept of circuits (Engelhardt & Beichner, 2004; Perkins & Grotzer, 2005). Likewise, learning of programming has established the multiple challenges that beginning learners face in writing and implementing code (Guzdial, 2004; Palumbo, 1990; Soloway & Spohrer, 1989). While it is possible to approach the learning with e-textiles from these respective disciplinary foundations, it is also clear that designing and making e-textile artifacts is more than learning discretely about crafts, circuitry or code. In fact, it is the intersection of these three disciplines that makes learning with e-textiles challenging while also providing an authentic context. In particular we examine some of the ways in which the materiality of e-textiles makes aspects of circuitry and programming visible to learners. Previous research has focused on students alternative conceptions and how instructional activities can progressively enrich students models of science (Perkins & Grotzer, 2005). Our research examines in more detail the situated nature of students understanding of electrical circuits and program control flow that overlay and support each other. For instance due to the unusual nature of materials such as the conductive thread, students are challenged to develop understandings of conductivity that draw on some of their informal knowledge about circuits and electricity. Likewise, the less tangible aspect of control flow in programming overlays the coding of sensors and actuators that are connected via circuits. In the production of their e-textile designs, students negotiate and link representations of graphical, textual, symbolic, and material nature and create linkages by literally threading and tying them together. These connections are reminiscent of Bliksteins (2010) bifocal modeling that provides explicit links between physical artifacts and their computational counterparts that allow students to design and experiment with both representations. However, in the context of e-textile design, students are the ones constructing the links. Our analyses are a first effort to build a framework to captures the processes involved.
Methods
Participants were 35 freshmen, 14-15 years old, from a public magnet high school focused on science and technology in a large urban school district. The students' self-identified demographic make up was 23% African American, 29% Caucasian, 14% Asian, and 17% mixed race/ethnicity. Five students chose not to identify their race/ethnicity in survey responses. Just under half of the participants were girls (n=15). Overall, the demographics of the workshops reflected the diversity found in the school and district at large. In spite of students interests in science and technology, only a few of our participants had prior programming experience
ISLS
189
ICLS2012
and none had ever worked with electronic textiles when they elected to participate in the e-textiles workshops. The workshops were conducted as part of a partnership with a local science museum where students spent one afternoon a week at the museum learning about a topic of their choosing. For this study, we organized a series of three workshops, ranging in length from 4 to 6 weeks. Workshop sessions were held once a week and lasted for two hours. During this time period, students learned how to design and create their own e-textiles projects, beginning with aesthetic drawings, followed by circuit schematics, sewing/crafting of designs, and programming. The first workshop had all novices and therefore the concepts built more slowly upon one another; most students finished very simple e-textiles projects with 1-2 LEDs. In contrast, the second workshop had roughly half returning students and all but one of the students in the third workshop had prior experience with etextiles. As students gained knowledge, their projects tended to increase in complexity and they experienced greater success. We also became better instructors and began to set tangible goals for each workshop session, like sew two LEDs and program them, so the construction process became more recursive, with students identifying errors as they went rather than just at the end when their completed project did not function as they expected it would.
Materials
All students had access to the LilyPad Arduino construction kit (see Figure 1) that enables novice engineers/designers to embed electronic hardware into textiles (Buechley & Eisenberg, 2008; Buechley, Eisenberg, Catchen & Crockett, 2008). In addition, we provided various caps, t-shirts, gloves, cotton bags, fabric and felt pieces on which students could sew their projects in addition clothes or objects student brought themselves. The LilyPad kit is a set of sewable electronic components, including a programmable microcontroller and an assortment of sensors and actuators that allows users to build their own soft wearable computers. Users sew LilyPad modules together with conductive thread instead of traditional tools like insulated wire and soldering irons. To define the behaviors of the project, users employ the popular Arduino or ModKit development environments, enabling them to program the LilyPad microcontroller to manage sensor and output modules (like LEDs) employed in their designs.
The LilyPad Arduino kit and components. The microcontroller is in the center, and the other components, clockwise from the top, are: accelerometer, light sensor, tri-color LED, power supply (requires a AAA battery), speaker, and vibrating motor. An FTDI board and USB cable are also part of the introductory kit.
Figure 1. LilyPad Arduino.
Data Collection and Analyses

The data collection of all workshops included video recordings (focused on groups working at individual tables), field notes by two independent researchers, photographs of students projects, and final interviews with students. The writings of field notes and interviews were refined in each successive workshop to increasingly focus on the process of design. For instance, in the final workshop two researchers wrote field notes focusing on the individual design decisions made by students with accompanying photographs of their designs at different stages of production (photographs were taken roughly every 60 minutes). This allowed us deeper insight into the ideas, challenges, and decisions that arose in the design process than was available on videos or in interviews about students final projects. We also conducted casual ethnographic interviews with students every week in the final workshop, asking questions like, What are your goals for this workshop?, What parts of e-textiles do you still need help understanding?, and the more open-ended tell me about your project. This allowed us to cull reflections from students in process rather than just at the end. Thus, like students projects, the final workshop was the richest in details about students design decisions and processes. For analysis of workshop activities and case studies, we conducted a two-step open coding (Charmaz, 2000) of all our data (field notes, logged videos and interviews). To wit, we first began by reading a third of the data and listing some of the challenges of learning to design with e-textiles. We then created an initial coding scheme of the learning challenges, categorized by the overlap of crafting, circuitry, and coding, coded one section of the data together to build consensus, and proceeded to independently code two workshops. We then refined our coding scheme to reflect insights from this analysis of the data and re-coded all workshops. We created a thesaurus of codes with definitions and examples and indexed (i.e., counted) all codes, listing them by date with a 1-sentence summary. This allowed us to see which learning challenges were most prominent in
ISLS
190
ICLS2012
individual workshops or across workshops. Below we discuss the most prevalent codes found across all workshops in regard to students learning in e-textiles.
Results
In analyzing where students faced the greatest challenges in learning how to design e-textiles, we found some of the most significant learning occurred where the material and conceptual aspects of the projects overlapped. First we explain the ways that the physical aspects of crafting illuminated the inner workings of circuits. Then we describe how the materially embodied circuitry shed light on coding. Both of these findings stretch across the breadth of our data from all three workshops.
How Sewing Reveals the Intricacies of Circuits

When designing and creating their own electronic textiles projects, novices must not only sew (a new experience in and of itself for many students) but do so in a way that takes into account the conductivity of the thread they are working with and the fact that what they are sewing must ultimately result in functional electronic circuitry. In our analysis of students design of e-textiles, we found three areas that were particularly non-intuitive: sewing for conductivity, tying knots to end a conductive line, and tidying up loose threads that created accidental short circuits. Each of these areas made qualities of circuits visible to the students in ways that they learned about basic aspects of circuitry.
Sewing for Conductivity

At the most elementary level, students learn that they must sew an electronic component to ensure conductivity as well as stability. If an electronic component like an LED is sewn too loosely or the stitches are too large, the connection between the component and the conductive thread may be lost when the textile bends. Unlike using alligator clips or soldering wires, where electricity is easily conducted on a slight touch between metal and metal, when attaching electrical parts to something more malleable like fabric, one must work (sew) to make sure there is a consistent physical connection. This is the most basic element of conductive sewing but it is not intuitive to novices. Nathaniel described this as a very frustrating part of making his project: RESEARCHER: NATHANIEL: RESEARCHER: NATHANIEL: What was the hardest part about the project? The sewing. The sewing. Why? Like, I know I went through it once, and I had like finished the thing. But it got loose. So I had to unthread it all, go back again. RESEARCHER: Because you hadn't sewn it tight enough? NATHANIEL: Yeah. (Dec. 15, 2010, Video)
Though Nathaniel thought his project was sewn adequately, when he tried to turn his lights on, the connection between the light and the conductive thread was too loose and he had to re-do the project. This happened for many students. We found that despite our instructions from the very beginning to sew each light with three stitches, most students did not do this until they understood the reason for this was to create a conductive connection. Thus students understanding (or lack thereof) of this introductory concept of electrical circuits was made visible in their sewing for conductivity.
The Importance of Tying Knots

Whereas sewing electrical components on fabric made visible the need for conductive connections, tying knots revealed the way electricity takes the path of least resistance in a circuit. For instance, in order to direct electricity to flow through an LED one must consciously tie a knot to end the positive connection on one side of the LED, cut the thread, and start a completely new line of thread on the negative side. This separation between the sides is less visible when one uses alligator clips to connect an LED to a power source because the alligator clips have to be connected at each end anyway. Thus, even after using alligator clips to make and test circuits, in moving to thread many students made the common mistake of using one uncut thread to sew through the sides of an LED, in effect bypassing the LED in the circuit. Mallory described this mistake below. MALLORY: I just now learned like, when I sew here like. When I first, I was sewing through the positive part of [the LED] [then] through the negative side and then so, it was kind of hard for me cause I was doin' it incorrectly? But now, I know not to. RESEARCHER: Do you understand why that doesn't work? (March 2, 2010, Video) MALLORY: Yeah Well it's gonna go over it [the LED].
ISLS
191
ICLS2012
Here Mallory described how she had sewn through both ends of an LED with the same thread, not realizing that she had to cut and separate the sides in order for electricity to flow through the LED rather than thread. As students began to understand conductive pathways, they came up with the phrase sew like alligator clips to formalize the concept. Because of the non-trivial effort it took to tie a knot at the end of each positive and negative sewn line, especially for novices sewers, students were anxious to find ways to minimize the number of knots they needed to tie. They were keen to simplify their circuits, which could be done by using one conductive line to connect the negative (or positive) sides of multiple LEDs without the need to tie knots at every end (for an example see the diagram on the right of Figure 3, knots are shown in stars). So, it is okay to sew continuously through the negative ends of multiple LEDs but it is not okay to sew through the negative and positive ends of a single LED without tying knots and cutting the thread to direct electricity through the LED. To our surprise, we found the best way to convey this concept was to describe it in terms of how many knots they needed to tie. We showed this by comparing multiple electric diagrams and having students re-draw the electrical connections for the fewest knots. Silas reported proudly about learning to rearrange circuits by connecting multiple sewn lines (i.e., wires): My learning experience with the lights has definitely changed--that's one thing. I can like, now, now that I know, um, the electric, um, what is that called? What am I trying to say? Um, how to hook up one wire [thread] from one end to another with positives and negatives, how to connect multiple wires [threads], and create multiple lights on em. (March, 2011, Video) Many students like Silas said their favorite learning experience was figuring out what could be sewn together and what had to be sewn individually. This learning changed their designs substantially, with almost all students re-orienting their circuits so that either the positive or negative sides of their LEDs could be connected in one line, allowing for many more LEDs to be sewn on with fewer knots and less sewing effort.
Loose Threads Mean Short Circuits

One final circuitry concept that the element of crafting made visible was short circuits. In conductive sewing, crossed threads (which work like non-insulated wires) amount to crossed negative and positive lines, resulting in short circuits. This is non-intuitive from a crafting perspective where we are taught that the back doesnt matter because no one will see it when the project is finished. Yet in e-textiles, loose threads or too much sewing in a small space can cause the project to fail. Amari and Marcela learned to identify this problem in a project where they were trying to figure out why the LEDs would not turn on: AMARI: Kay, I think I found the problem with this. MARCELA: Okay, what is it? AMARI: Okay, I was looking at the back of it, and I think the problem is that if you look at it closely, these [threads] are all overlapping each other and that could affect the conductivity. ((Pointing to overlapping threads on the back)) MARCELA: "Oh yeah, 'cause negative can't overlap the positive. (May 20, 2011, Video) When Amari pointed to the back of the project where many long ends of threads were touching, Marcela correctly identified this as a problem with short circuits where the negative and positive lines were overlapping. The girls solved the problem by trimming the threads and tying the knots closer to the surface of the fabric where they couldnt get loose and touch each other. All students struggled with the issue of short circuits, coming to terms with the idea that the threads conducted electricity even if they were just the loose ends of knots. Yet though it took time to learn this idea, by the end of each workshop, loose threads became one of the first things students looked to when trying to make their LEDs turn on.
How Embodied Circuitry Reveals the Intricacies of Coding

Not only did the materiality of crafting circuits bring out some conceptual issues about circuitry and electronics, but the crafted electronics revealed some concepts about coding and vice versa. At the most basic level, students learned that connecting LEDs to the LilyPad microcomputer was not the same as connecting them to a battery. First, the LilyPad had to be programmed. Kyra demonstrated her partial understanding of this idea when, having finished sewing her project with two LEDs connected to pins 8, 10, and a negative ground, she programmed pin 8 and then wondered why only one LED flashed on. A researcher pointed out to her that she had not yet programmed pin 10 perhaps an obvious idea to people with programming experience but a novel idea to beginners. A week later Kyra summarized this episode as one of her most important learning experiences when she said It helped me understand that things that most things cant work without a code (Dec. 15, 2010, Video). Below we describe other ways that students learned about coding through circuitry, and circuitry
ISLS
192
ICLS2012
through coding. We note that in each case, it was not until designs were actually material (sewn circuits) that the relationship between the two became clear to the students.
Why Parallel Circuitry Might Not Be the Best for Blinking Patterns
One challenge that students faced was to learn the affordances of different kinds of circuits. As we mentioned above, students had a vested interest in tying as few knots as possible and for this reason many were interested in making parallel circuits, with one continuous negative line connecting the negative pins of several LEDs and one continuous positive line connecting the positive pins of the LEDs. However, when it came to coding their circuits, they began to realize that parallel circuits did not afford them any opportunities to have their LEDs blink at different times the shared positive and negative lines meant that all of the lights worked together. For instance, when Aaliyah was drawing out the design of her five-point star with an LED on each point, she decided that some of the LEDs should be sewn in parallel (see left diagram in Figure 2). However, two weeks later after her project was partially sewn and she had made some of the LEDs turn on, she changed her mind as she realized that she would not be able to blink each light independently (see right diagram of Figure 3 for final design). Her old design would not allow her to turn each LED on one by one, rather they would go on in groups that she could never change. This was a common dilemma students faced as they began to understand the relationship between the programming effects they desired and the implications of the design/crafting of their circuitry.
Figure 2: Aaliyahs original design (left) with two parallel circuits & one additional circuit and final design (right). with five independent circuits with a shared negative line. Negative lines are solid, positive lines are dashed, knots are shown with small stars (*).
On Hardwired and Programmable Ports

Another challenge that students faced at the intersection of circuitry and programming was learning about the affordances of the LilyPad. Two pins on the LilyPad (the + and the pins) cannot be programmed and have constant polarization. The remaining (numbered or lettered) pins can be programmed to be either positive or negative. The first realization that students came to was the difference between these two types of pins. One type (the positive and the negative) was unalterable, while the other was completely programmable. Marcela described this as one of her biggest realizations: I didnt get in the beginning why, you know how you connected to negative and then positive? I didnt get what does the number represent... And I didnt know you could make it anything you want. (March 2, 2011, Video). Once students realized that they could program any numbered or lettered pin on the LilyPad to be either positive or negative, they had a great deal of freedom to design their circuits. Still, the implications that some pins were hardwired while others were programmable did not come easily to everyone. Below we share how Amari became conscious of this relationship between circuitry design and programming: Amari had finished putting her 5 LEDs on the points of her five point star with the negative ends of the LEDs connected together around the outside (in a circle), securing them to the fabric and connecting the negative line to the pin on the LilyPad. When she was sewing her first LED to the LilyPad (the outside part already finished) she asked where to connect it. I directed her to the #3, the closest numbered pin to that particular light. But what about the positive? she asked, pointing out that the + pin was even closer. Well if you do that, the light will always be on - you wont be able to control it on and off, I said. Hm. I wont mind. Im going to do it anyway, Amari said. (May 18, 2011, Field note excerpt) Amari programmed her lights to turn on, making them blink. The LED that she had connected to the + and the pins of the LilyPad stayed on continuously. I know that you told me it
ISLS 193
ICLS2012
would always be on and I did it anyway, but want to re-do it, Amari said. Then she cut off the positive line of that LED and re-sewed it to a different, programmable pin on the LilyPad all on her own. (May 25, 201, Field note excerpt) In this example, Amari did not seem to realize the full implications of connecting an LED to the hardwired pins until she had sewn her entire project and programmed the LEDs to blink. Then the implications of her circuitry design on her programming capabilities registered and she went to the trouble of undoing and redoing part of her project so that there were greater affordances for programming. Though it was frustrating to us to watch students make decisions that would limit their designs despite our admonitions, we were surprised at how many students were willing to re-design their projects, some going to great trouble to do so, when they finally understood why certain circuit designs would not allow for the programmed lighting effects they desired. It was truly a process of learning through design.
Discussion
One goal of our investigation was to understand how making technology visible could be beneficial in students learning with e-textiles designs. In order to examine this aspect, we needed to understand first the particulars of how students approached the connections between circuit and code. Most of their problems fell into two areas: creating circuits with fabric and thread and programming physically laid out circuits. Remarkably, both of these broad categories involve the juncture of the conceptual and the physical, such that the material qualities of etextiles made visible the conceptual issues of electronics and programming. In other words, the infusion of crafting into the domains of electronics and code made visible the inner workings of circuits and programming. Thus it was the initial move from circuit diagrams to actual sewn circuits and the subsequent move from abstract concepts of code to coding physical circuits embedded in cloth that held both the greatest challenge and, as we argue, some of the greatest opportunities for learning. We see these findings as evidence that in certain circumstances it might be helpful to make technology visible for learning. Far too often technology designs are hidden away, and that purposefully so. Indeed, intentional invisibility or blackboxing has always been part of educational design. But even computational construction kits blackbox certain aspects of computation and processing so that novice programmers can focus on essential aspects of programming such as understanding control structures or variable inputs while not having to worry about syntax. In the end, these are always educational decisions on what to render visible or what to leave invisible (Resnick & Silverman, 2005). One can question whether students actually would be better served in not dealing with the knotty aspects of sewing in order to focus more on the functional aspects of designing circuits and programming sensors and actuators. Some e-textile construction kits indeed have provided a shortcut to the sewing by using snaps rather than thread and needle (Ngai et al, 2009). As such any computational construction kits makes certain aspects of e-textile design visible while hiding others, but the involvement with the design and functionality of technology is still miles away from the limited point-and-click interactions available in many commercial textile construction kits (Kafai et al., 2010). Finally, using textiles and crafts in the digital age invariably brings the gender issue to the fore an aspect that we have dealt with in a different paper in more detail but which does belong to any discussion about technology and learning. E-textiles, of course, by design brings together sewing and fabrics that have been historically more associated with females together with engineering and computing that have been historically more associated with males (Searle, Kafai & Fields, in preparation). In this paper we examined the complicating of learning by making connections between crafts, circuitry, and code visible while a focus on gender reveals a complicated relationship in how cultural norms define who see themselves as technologists in the making or not. As such e-textiles occupy fertile territory for thought. Along the same lines, the domains of crafts and fabric also push to into the foreground issues of aesthetics and learning that are rarely discussed in the context of science and engineering (Fields, Kafai, & Searle, 2012). Functionalities present a rich context to think how personal uses can become motivations for functional aesthetics. Our study gave us rich food for thought on how to create and expand already popular hands-on activities for learning science into the computational realm.
References
Bers, M. (2009). Blocks to Robots. New York, NY: Teachers College Press. Blikstein, P. (2010). Connecting the science classroom and tangible interfaces: the Bifocal Modeling framework. In K. Gomez, L. Lyons, & J. Radinsky (Eds.), Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010). University of Illinois at Chicago: Chicago, IL. Buechley, L. (2010). Questioning Invisibility. IEEE Computer, 43(4), 84-86. Buechley, L. (2006). A Construction Kit for Electronic Textiles. In Proceedings of IEEE International Symposium on Wearable Computers (ISWC), Montreux, Switzerland, pp. 83-92.
ISLS
194
ICLS2012
Buechley, L., and Eisenberg, M. (2008). The LilyPad Arduino: Toward wearable engineering for everyone. Wearable Computing Column in IEEE Pervasive, 7(2), 12-15. Buechley, L., Eisenberg, M., Catchen, J. and Crockett, A. (2008). The LilyPad Arduino: Using Computational Textiles to Investigate Engagement, Aesthetics, and Diversity in Computer Science Education. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI), Florence, Italy, April 2008, pp. 423-432. Charmaz, K. (2000). Grounded theory: Objectivist and constructivist methods. In N.K. Denzin & Y.S. Lincoln (Eds.), Handbook of qualitative research (pp. 509535). Thousand Oaks, CA: Sage Publications. Druin & Hendler, (2000). Robots for kids: Exploring new technologies for learning. San Francisco, CA: Morgan Kaufman Eisenberg, M., Eisenberg, A., Buechley, L. & Elumeze, N. (2006). Invisibility considered harmful: revisiting traditional principles of ubiquitous computing in the context of education. Proceedings of Fourth IEEE International Workshop on Wireless, Mobile and Ubiquitous Technology in Education (WMTE'06) (pp. 102-110). New York: IEEE. Engelhardt, P. V. & Beichner, R. J. (2004). Students understanding of direct current resistive electrical circuits. American Journal of Physics, 72(1), 98. Fields, D. F. Kafai, Y. B. & Searle, K. A. (2012). Functional aesthetics for learning: Creative and Productive Tensions in Youth e-Textile Designs. Paper to be presented at the 10th International Conference of the Learning Sciences, Sydney, Australia. Frauenfelder, M. (2010). Made by Hand: Searching for Meaning in a Throwaway World. New York, NY: Penguin. Gauntlett, D. (2011). Making is Connecting. Cambridge, UK: Polity Press. Guzdial, M. (2004). Programming Environments for Novices. In S. Fincher and M. Petre (Eds.), Computer Science Education Research (pp. 127-154). London, UK: Taylor & Francis. Kafai, Y. B., Peppler, K. A., Burke, W. Q., Moore, M., & Glosson, D. (2010, June). Froebels forgotten gift: electronic textile construction kits as pathways into design and computation. Proceedings of the Interaction Design for Children Conference (IDC10), Barcelona, Spain. Katterfeldt, E.-S., Dittert, N., & Schelhowe, H. (2009). EduWear: Smart Textiles as Ways of Relating Computing. Proceedings of the 8th International Conference on Interaction Design and Children. DOI: 10.1145/1551788.1551791 Ngai, G., Chan, S.C.F., Cheung, J.C., & Lau, W. W. (2009). The TeeBoard: an education-friendly construction platform for e-textiles and wearable computing. In Proceedings of CHI09, 249-258 Palumbo, D. (1990). Programming Language/Problem-Solving Research: A Review of Relevant Issues. Review of Educational Research, 60(1), 65-89. Perkins, D.N., & Grotzer, T.A. (2005). Dimensions of causal understanding: The role of complex causal models in students' understanding of science. Studies in Science Education, 41, 117-166. Post, E. R., Orth, M., Russo, P. R., & Gershenfeld, N. (2000). E-broidery: design and fabrication of textilebased computing. IBM Systems Journal, 39(3-4), 840-860. Resnick, M., Berg, R, & Eisenberg, M. (2000). Beyond black boxes: bringing transparency and aesthetics back to scientific investigation. Journal of the Learning Sciences, 9(1), 7-30. Resnick, M. & Ocko, C. (1991). Lego/Logo: Learning through and about design. In I. Harel & S. Papert (Eds.), Constructionism, Norwood, NJ: Ablex Publishing. Resnick, M., & Silverman, B. (2005). Some Reflections on Designing Construction Kits for Kids. Proceedings of Interaction Design and Children Conference, Boulder, CO. Searle, K. A., Kafai, Y. B. & Fields, D. A. (in preparation). Crafting High-Low Tech Identities with Electronic Textiles: Examining Gender and Technology in Youth Designs. Soloway, E. & Spohrer, J.C. (Eds.) (1989). Studying the Novice Programmer. Hillsdale, N.J.: Lawrence Erlbaum Associates. Sullivan, F.R., (2008). Robotics and science literacy: Thinking skills, science process skills, and systems understanding. Journal of Research in Science Teaching, 45(3), 373-394. Turkle. S. & Papert, S. (1992). Epistemological pluralism and the revaluation of the concrete. Journal of Mathematical Behavior, 11(1), 3-33.
Acknowledgments
This work was supported by a collaborative grant (0855868) from the National Science Foundation to Yasmin Kafai, Leah Buechley and Kylie Peppler. Any opinions, findings, and conclusions or recommendations expressed in this chapter are those of the authors and do not necessarily reflect the views of the National Science Foundation or the University of Pennsylvania. We thank Leah Buechley and members from Penn Research Apprenticeship Class for comments on earlier versions of the paper.
ISLS
195
ICLS2012
Functional Aesthetics for Learning: Creative Tensions in Youth e-Textile Designs

Deborah A. Fields, Yasmin B. Kafai, & Kristin A. Searle University of Pennsylvania, 3700 Walnut Street, Philadelphia, PA 19104 defields@upenn.edu, kafai@upenn.edu, searle@dolphin.upenn.edu Abstract: Most research in programming and engineering focuses on students understanding of functionality as a way to gage their learning, leaving aside aesthetic dimensions. In our work with the LilyPad Arduino, an e-textile construction kit with controller, sensors and actuators that can be embedded via conductive thread and programmed in fabric and garments, we examine how functional aesthetics can play a productive or sometimes unproductive role in learning. Drawing from observations and interviews with 35 high school youth that created e-textile artifacts, we identified three different approaches ranging from giving up on desired designs to making something functional or not finishing or getting a design to work because of unwillingness to give up on aesthetics. We see the third approach, finding a new design that both meets aesthetic desires and matches affordances of the technologies, as particularly promising approach and discuss how aesthetic dimensions can provide important connections in learning.
Introduction
Learning sciences research has made great strides in understanding learning and teaching science, mathematics, social sciences and language arts inside and outside of schools, developing modeling tools and learning environments, and promoting new methods for research (for overview see Sawyer, 2006). There are, however, a few fields such as arts education and aesthetics that have received little, if any, consideration, especially in regard to how they relate to STEM subjects. As Lemke (2010) recently noted, by and large the use of emotionally and aesthetically appealing imagery, video, or simulations and games has been excluded from the teaching of science owing to a misplaced desire to portray science as a body of theory and fact, rather than as a human activity. These observations can easily be extended from science into other disciplines and to activities beyond the screen: namely, those aspects of digital media construction and design that dovetail with hands-on crafts, physical construction and design, as well as material play. In this paper we focus on learning with a new type of tangible, programmable media called electronic textiles (e-textiles hereafter). E-textiles include designs of programmable garments, accessories, and costumes (Post, Orth, Russo, & Gershenfeld, 2000). Such designs can incorporate elements of embedded computing for controlling the behavior of fabric artifacts, novel materials such conductive fibers or conductive Velcro, sensors for light and sound, and actuators such as LEDs and speakers, in addition to traditional aspects of fabric crafts. The making of e-textiles engages both functional and aesthetic aspects that have been described as the concept of merging a fashionable technology object deemed aesthetically pleasant with technically enhanced functionalities (Seymour, 2010, p.10). While Seymour also focused on the production of new fabrics, techniques and technologies, we were particularly interested in the role of functional aesthetics in learning to work with an e-textile construction kit called the LilyPad Arduino (Buechley, 2006). The LilyPad Arduino belongs in the family of material computational construction kits that makes these technologies accessible to novice designers as the Lego Mindstorms did for robotics constructions in school programs. But e-textiles do so within context of soft computation where the design of digital and electronic technology is composed of soft materials such as textiles and yarns, as well as predicated on traditional construction methods to create interactive physical designs (Berzowska, 2005, p. 67). This work is part of a larger research project aimed at understanding creativity in computation (Kafai, Peppler, & Buechley, 2009) in which we conducted a series of e-textile workshops with high school youth during 2010-2011. While our main interest was in understanding how youth learn about electrical circuit design and programming, we also wanted to know how the aesthetic features of their e-textile projects influenced their design decisions and progress. Our intention was to attend to the process of creating aesthetic and functional artifacts and to develop a framework for how aesthetics related to learning (or not) while also serving to promote personal relevance through connecting attractiveness to technology. Our data comes from observations, field notes, video records of workshop interactions, analyses of e-textile artifacts designs and realizations, and interviews with high school youth about their experiences and artifacts. We analyzed design decisions by 35 high school freshmen and women creating e-textiles over four weeks, attending to the times when their aesthetic ideas interacted with functional requirements in shaping their designs. We address the following research questions: How do aesthetics influence the design of e-textile artifacts? How do students aesthetic visions relate
ISLS
196
ICLS2012
to their learning to make e-textile projects? In the discussion we will consider how functional aesthetics for learning can also be expanded into other digital media projects.
Background
The arts and sciences have long been connected in the scientific (if not the public) imagination. In the past, scientific tools were not intended simply as measurement devices, but also as aesthetic objects: as occasion for artistic ingenuity and whimsy possibilities for blending art and instrumentation design (Resnick, Berg & Eisenberg, 2000, p. 24,). Science professionals often derive aesthetic pleasure in their work: Mathematicians describe math as poetry, computer scientists attend to beautiful and simple solutions, and physicists pronounce theories and equations to be beautiful (Girod, 2007). Chamberlain (1987) argues that in both art and the science, deep observation reveals new ideas and suggests that aesthetics is an integral part of discovery in the sciences (Chamberlain, 1987). Furthermore, Lemke (2010) goes so far as to consider aesthetics part of a scientific habitus that includes positive, affective dispositions toward conventional scientific technologies, media genres, and styles of representation within specific science disciplines. In her decades long studies of physicists, Traweek (1995) describes an elaborate aesthetic and moral discourse among high energy physicists where machines, software, data, and laboratories are described with adjectives like sexy, cute, beautiful, constant, and ordered. She argues that scientists have aesthetic values in the images and tools that they use, recounting a time when, [O]nce I watched a group of physicists gazing at a book of images of fractals; with each turn of the page came a soft chorus of pleasured sounds (Traweek, 1995, p. 212). Appreciating the aesthetics of scientific phenomena and representations is an important part of what it means to be enculturated into science. In K-12 science and technology education, aesthetics is most often viewed as extraneous to core academic learning, a misplaced desire to portray science as a body of theory and fact, rather than as a human activity (Lemke, 2010). At best it is seen as a motivating function, something that promotes interest and identification in science and technology. For instance, several studies have found that both positive and negative aesthetics can reveal engagement and promote identification with science topics inside and outside of school (Barton, Tan & Rivet, 2008; Brickhouse, Schultz, & Lowery, 2000; Jacobson & Wickman, 2008). There is a small body of research that has examined aesthetics in relation to learning and teaching of science (Flannery, 2006; Wickman, 2006) focusing on social, cultural and personal aspects. A few studies of classroom teaching where aesthetic ideas about science were conveyed have also shown promise for furthering learning. Hadzigeorgiou (2011) found that students retained more ideas, asked more questions, and wrote more in voluntarily kept science journals when aesthetics was consciously included in the science curriculum. Girod, Twyman, and Wojckiewicz (2010) also found that a focus on aesthetics promoted better retention in later assessments. In these studies, aesthetics is seen as having a sense of wonder (Hadzigeorgiou, 2011), becoming aware of powerful science ideas (Girod et al., 2010), or having transformative experiences akin to Deweys writing on the power of aesthetics (Pugh, 2011). Our approach in examining aesthetics in the context of e-textile designs builds on these studies but does so paired with learning to design with a technology, particularly in the areas of engineering (designing and testing electronic circuits) and programming (turning circuits on and off e.g., making lights blink). While some studies have begun to look at the relationship between aesthetics and functionality in students technological designs, they have largely focused on the final product (is it attractive?) rather than the process, and have attended mostly to the motivational aspects of making something aesthetic. For instance, researchers found that when students pay attention not only to functionality but also to aesthetics of robotics artifacts, they have a much stronger connection to their creations (Resnick, Berg, & Eisenberg, 2000) as well as recruit and retain a larger group of students (Rusk, Resnick, Berg & Pezella-Granlund, 2007). In this study we focus on aesthetics as more than a tangent: students e-textile designs are driven by functionality as well as aesthetics in their personal choices and thus become a context for learning. In particular, we want to include the material or physical side as scientists such as Nobel prize winner Barbara McClintock have argued for physical touch as an important aspect of inquiry (Keller, 1983). Here we suggest that the tension between designing something aesthetically pleasing that is also functional is a creative tension, one that promotes new learning because the aesthetics can push the design and vice versa.
Context & Methods

The primary context of the study took place in e-textiles workshops led with 9th grade (14-15 year-old) high school students at a local science museum. We led a series of three workshops, each roughly one month-long with weekly two-hour meetings. In the workshops, a total of 35 students learned how to design and create their own e-textiles projects, beginning with aesthetic drawings, followed by circuit schematics, sewing/crafting of designs, and programming. The series culminated in the third workshop where 16 students, 15 of which had prior experience making e-textiles from another workshop, completed finished e-textiles that had at least 2 blinking lights that they had sewn and programmed. Since students finished the most complex work in this final
ISLS
197
ICLS2012
workshop, it serves as the focal point for our analysis of the relationship of functional aesthetics to learning in etextiles.
Materials
All students had access to the LilyPad Arduino construction kit (Buechley & Eisenberg, 2008) that enables novice engineers/designers to embed electronic hardware into textiles (see: http://web.media.mit.edu/~leah/LilyPad/). In addition, we had various caps, t-shirts, gloves, cotton bags, fabric and felt pieces. Some students also brought their own clothes or objects for e-textile designs. The LilyPad is a set of sewable electronic components, including a programmable microcontroller and an assortment of sensors and actuators that allows users to build their own soft wearable computers. Users sew LilyPad modules together with conductive thread instead of traditional tools like insulated wire and soldering techniques. To define the behaviors of the project, users employ the popular Arduino or ModKit development environments, enabling them to program the LilyPad microcontroller to manage sensor and output modules (like LEDs) employed in their designs.
Data Collection and Analyses

Across all workshops we gathered video recordings of students working together, field notes by at least two independent researchers, final interviews with students, and weekly photographs of students designs. As our expertise in how to help students create e-textiles grew, our data collection became better defined. We conducted iterative analyses after each workshop, reflecting on what students struggled with and what strategies helped them succeed. This reflection led us to focus our data collection in the third and final workshop on students processes of design, from their original ideas through their learning about the affordances of the LilyPad, LEDs, and conductive thread, and to their final designs. To gather this data two researchers collaborated to write detailed, descriptively rich field notes on each students design progress each week accompanied by multiple photographs per student. This allowed us to follow all students consistently through their design process, supplemented by final interviews where we asked students to explain how their project had changed from their original ideas to finished product, what they were most proud of, what they struggled with, and what they felt they had learned. Analysis was completed in two steps. First, we conducted a two-step open coding across all of our data (videos, field notes, interviews, photographs) based on grounded theory (Charmaz, 2000) focused on what students struggled to learn. From this coding we noticed two categories of learning related to aesthetics that were present across all workshops: working out the polarized orientation of LEDs for efficient circuitry and using the LilyPad to greatest effect in both placement and choices of circuits. With this insight we sought to understand how students growing knowledge of polarized circuitry and the affordances of the LilyPad related to the aesthetics of their designs. To do this we conducted a second phase of analysis, developing case studies for all 16 students in the third workshop that detailed their original ideas, the challenges they faced, their decisions at these challenges, and the next cycles of ideas-challenges-decisions until the design process ended with their final product at the end of the workshop. We also took into account interviews and reflective statements about what they were most proud of as well as how others responded to their projects. Across the case studies we looked for common issues related to aesthetics and learning.
Findings
In this section, we first present one of the 16 case studies for which we mapped out the design process of students projects, including their original ideas, challenges, decisions and resolutions, and final products. The case study focuses on one students design process and highlights in detail how aesthetics played a role in her design decisions and learning. Then we report on common relationships between aesthetics and circuitry and aesthetics and programming found across all cases.
The Unfolding of Amaris Shining Star

We begin by describing the design process of one girl, Amari, whose project changed significantly in scheme if not in essence from the beginning to the end of her two-workshop design process. Amari told us that when she first came to the workshop, she thought it would be really confusing and that she wouldnt like it. However, she grabbed onto the crafting aspect and was allowed to start by simply cutting a star out of white felt and sewing it onto a pink rectangle of felt: I just sewed it onto felt and then I was like, It's time for lights and techie stuff, and then slowly I started to understand it (May 27, 2011, Video). Her initial idea for her project was to make something simple, a five-point star with LEDs on each point. Okay, well I decided to do this because, well, I wanted to make something that was kind of bright so I just chose the pink color and I didn't want to do something really complicated like other people did. I wanted to do something simple, so I just did the star and, um, I just put a
ISLS 198
ICLS2012
star on the pink and I decided to make each of the points light up 'cause that'd be really, what do you call it, bright I guess.
(May 27, 2011, Video)
Here we see some of her original aesthetic desires a white star on a bright pink felt background, with each LED lighting up. However, actualizing these ideas was not trivial for Amari; even though she felt her design was simple, she had to re-design and even re-sew parts of it several times. One challenge that she and many students faced at the beginning was how to layer the positive and negative connections between the LEDs over their aesthetic drawings, which usually had no considerations of electrical connections. Amari started with a diagram of a star with five LEDs where the positives and negatives of the LEDs connected with each other but nothing else, not a LilyPad or a battery (see the large pencil star on the left of Figure 1). With coaching, she drew another, smaller diagram with a circular LilyPad inside the star and the positive ends of the LEDs connecting to different pins on the LilyPad (see smaller star on the left of Figure 1 and our schematic of this diagram on the right of Figure 1). This was a significant improvement on her earlier design, but it still lacked a connection between most of the negative sides of the LEDs and the LilyPad: only two negative sides were connected to each other and to the negative pin of the LilyPad in her drawing.
Figure 1. Amaris initial designs (left), with the large penciled star her first design and the small penciled star her second design, and our schematic diagram of her second design (right). Amaris circuitry design became more sophisticated through the process of sewing on the LEDs. These design changes were influenced both by her sense of aesthetic and by her growing understanding of circuitry and the affordances of the LilyPad. First, Amari realized that she could connect all of the negative sides of the LEDs together by sewing a circle around the points of the star. She described this unplanned change to us as both a convenient and cool looking solution: My original design? I didn't know that I'd be putting the negatives all in a circle which is a really cool effect but I didn't think that I'd be doing that first. Um, I just thought I'd do each of the negatives going into a different thing [pin] I thought it'd save me more work so I decided to do that [the circle] and, it did. And then I decided to make the positives go around in like little swirls or something and that also made it fun and easy.
(May 27, 2011, Video)
Compared to Amaris original design with, as she put it, random lines going to random places, or her second design where only some of the negative sides were connected to anything, this third version was a completely functional solution (see left side of Figure 2). Connecting all of the negative sides of the LEDs in a single line to the hardwired negative pin on the LilyPad was a simpler circuit that was also more convenient to sew as Amari did not need to change, cut, or tie the thread in the continuous negative line. Yet while it was a functional solution, it was also an aesthetic one as well. Amari saw the circle around the star as cool effect and later magnified this effect by cutting a larger circle of pink about two inches outside of the conductive thread circle.
ISLS
199
ICLS2012
Figure 2. Changes in Amaris sewn designs. Left, she connected the top LED to the hardwired positive and negative pins of the LilyPad. Right, she re-sewed the top LED to the #3 pin to computationally control it. After her project was sewn, computational considerations led Amari to alter her project. First, once she programmed the LEDs to turn on and off, she realized the downside of connecting an LED to both the hardwired negative and positive pins on the LilyPad: she had no computational control over that LED (the top LED in the schematics in Figure 2) and could not make it blink at all. She told us, I know that you told me it would always be on and I did it anyway, but I want to re-do it, (May 25, 2011, Field Notes). Amari ripped out the stitches of one of her conductive lines and re-sewed the positive side of the LED to a number pin on the LilyPad (#3) that she could manipulate through programming. This design change, shown between the two schematic diagrams in Figure 2, while minor, shows Amaris developing understanding of the affordances of the hardwired and programmable pins on the LilyPad. Second, Amari altered her initial computer program that made the LEDs blink to a much higher level of program where the LEDs faded. Motivated to produce a particular aesthetic fading effect in the LEDs of her star, she came in for a special session during her lunch hour to learn how to use variables and embedded loops in code in order to create a fade effect. This was a big step in learning for Amari, who said that before the workshops she had no idea how a computer worked and was intimidated by the technology. We have illustrated in detail the design changes in Amaris e-textile project to show how they reflected her increasing understanding of circuitry and programming while being largely motivated by her aesthetic goals. The conductive circle connecting the negative sides of the LEDs in her star, suggests at least one instance where aesthetic desires worked alongside growing understanding of circuitry to make both an attractive and simple solution. In the instances of her physically changing the circuit of one LED from a hardwired to a programmable pin and her learning to create a fading effect through code, we can see how her aesthetic vision for her project motivated deeper learning of circuitry and programming. Here we have provided in detail the changes in design of Amaris e-textile star, but below we summarize the themes we found across all 16 case studies of students designs, for which we conducted equally detailed analyses.
Figure 3. Completed E-textiles. Left: Amaris final star; Middle: Jacksons rocket, Right: Jaylees Jack Skellington.
Aesthetics and Deepened Circuitry Learning

Amaris design process illustrates a common relationship between aesthetics and learning to design sewn circuits that occurred throughout the data. Across 15 of our 16 cases students re-designed circuits with aesthetic reasons driving their revisions. This usually happened for a few reasons. First, as students learned about the polarization of LEDs and thus the polarization of circuits, they modified their circuitry designs in ways that, like Amaris star, took into account an eye-pleasing form as well as ease of creation. This includes learning about the
ISLS 200
ICLS2012
affordances of parallel and independent circuits. Parallel circuits allow for much easier crafting and lay out because both the positives and the negatives of the LEDs can be connected in two continuous polarized lines. However, they do not allow for computational flexibility. Depending on the students desires, they altered their designs. Second, as they realized the differences between hardwired and programmable pins on the LilyPad, they often changed their design, as Amari did, to provide the opportunity for more computational flexibility. Sometimes this extended to understanding the differences of analog and digital pins on the LilyPad, which had divergent affordances for reading data from sensors (analog) or fading (digital). Finally, as circuits were realized in physical form on fabric, students amended designs to take into account practicalities like the location of the LilyPad relative to the LEDs or other electrical elements like sensors and overlapping circuit lines. More often than not, multiple circuitry issues intersected to create tensions between functional and aesthetic designs. For instance, Jaylee wanted to create the head of Jack Skellington (see Figure 3, right), a lead character in Tim Burtons The Nightmare Before Christmas, with two LEDs in each of Jacks two eyes. In working through her design, she decided that a single parallel circuit would be best because she wanted the eyes to light up the same way every time. However, she also wanted the head to be mobile not attached to a backing of felt so that it could be pinned to a backpack, a jacket or other accessories. To do this, she decided the LilyPad needed to be on the back of Jacks head, and developed a complex set of layers of felt to insulate the positive and negative lines of her otherwise simple 4-LED circuit. In another case, Silas began with a complex design of twisting ribbons that had more than 30 LEDs attached. As he tried to map out the positives and negatives of the polarized lines to the LilyPad, he eventually settled on 8 sets of parallel circuits with 2 LEDs, each on its own ribbon length. However, he had to re-design his e-textile project again when he realized that there were only 9 programmable pins on the LilyPad, so he developed a similar solution to Amari in connecting the negative lines of multiple circuits together and sewed that to the hardwired, negative pin. These cases illustrate how students aesthetic goals interacted with the practicalities of the technology they were using (LilyPad, LEDs, conductive thread) and their growing understanding of these technologies and circuits in general in the realization of their designs.
Aesthetics & Coded Lighting Effects

In addition to affecting students circuitry designs, aesthetics also played a role in most students learning to program. The primary reason for this is because programming was the way to achieve particular lighting effects in the e-textiles students designed. In the 8-hour workshop, there was relatively little time at the end to do more complex lighting effects, and the 10 of 16 students (of 16 total students in the workshop) who wanted different effects had to spend extra time outside of the workshop learning more programming than was required simply to turn their lights on and off. Some students wrestled to create multi-step lighting effects for projects with several circuits. Consider Jackson who created a rocket with six LEDs in the tail that he eventually programmed to blink in timed stages: two middle lights on [delay] next two lights on [delay] outside lights on [delay] all off [long delay] (Figure 3, middle). Jackson and others spent extra time experimenting with delays and order of lights to design this effect. For another instance consider Corbin who had two circuits with four LEDs each that he wanted to make blink on and off in turn (left side on then off, right side on then off). However, because one of the circuits was connected to two programmable pins, he struggled with the issue of control flow in his programming. He eventually learned to turn the negative sides of both circuits on at the beginning of his program, then in a forever loop to turn the positive sides on and off. Though these may seem like simple concepts to experienced programmers, it was a very challenging concept for nearly all of the novice students in our workshops to learn as they had to coordinate the design of code and circuits. Further, five students went significantly deeper into programming in order to use sensors or make fading effects in their projects. Using sensors in a project required learning to use conditionals in programming. For instance, Darryl created a bag with a light sensor on the handle such that when someone picked up the bag (making the sensor completely dark), the lights turned on, and otherwise they were off. To create a fading effect students had to learn to use variables and embedded loops in their code. Interestingly, three girls who had expressed that programming intimidated them came in during lunch one day to learn to do fading effects (including Jaylee and Amari whose projects are shown in Figure 3). In sum, the desire for an attractive lighting effect motivated many students to learn more complex coding.
Discussion
While many who have striven to connect aesthetics to science or technology education have focused on the importance of aesthetics in the final product, in this paper we have attended to the process of creating an aesthetic and functional product. We analyzed design decisions by students creating e-textiles over four weeks, concentrating on the times when their aesthetic ideas interacted with functional requirements in shaping their designs. We found that aesthetics played a role in almost every step of design, from the students original ideas to their development of explicit circuitry designs, from their physical construction of the project to their final coding of lighting effects. In each step the students aesthetic vision interacted with the practicalities of
ISLS 201
ICLS2012
construction, the necessities for functional circuits, and the affordances of the LilyPad microcontroller and the polarized LEDs. In every case, students designs for their projects changed as they developed understanding of circuits, coding, and the technologies they were using, but throughout the process aesthetics continued to play a role even if the final product looked significantly different from their original drawings. Further, in most studies of learning by design, there is a penchant for considering aesthetics as attractive and personally motivating without attention to how the integration of aesthetics into a project can become a tool for learning. While we too found that making an aesthetically pleasing e-textile project was motivating for students, aesthetics went beyond motivation to influence their learning of circuitry and programming. By looking at the step-by-step process of designing and re-designing e-textile projects, our analysis demonstrates that students aesthetic visions almost always promoted their learning in these areas. In particular, we noted how aesthetics played a role in fashioning more efficient and often more intricate circuitry designs as well as more challenging programmed lighting effects. In many cases, because of their aesthetic motivations for how their project would look and act, students sought out additional help, pressed through new challenges, and spent additional time outside of the workshop in order to learn more about circuitry and programming so that they could achieve a pleasing project. However, we do not mean to imply that simply making e-textiles project will always promote positive creative tensions between aesthetics and technological design. Rather, our experiences across workshops suggest that the local social context for making such a project must promote aesthetics and help students to create attractive projects that are also within their realm of reachable expertise. In our first workshop, when we knew less about how to help students design e-textile projects, some students created projects that were functional (i.e., the LEDs lit up) but not aesthetically pleasing to them, and they gave these projects back to us because they had no interest in keeping them. In other cases, some students were unwilling to alter their original elaborate blueprint designs according to the affordances of the LEDs and LilyPad and never even reached the point of constructing a project. In contrast, in our third workshop we successfully combined concrete, functional goals for each workshop (e.g. sew two lights, code two lights) with a focus on aesthetics. We paid special attention to prioritizing aesthetics in their projects and helping them translate their ideas into projects that were within or just beyond the bounds of what they could learn in the workshop. In this paper we tackled the issue of how aesthetics relate to learning in the context of e-textile projects, expanding our understanding of learning and creative imagination. In his discussion of the development of creativity in adolescence, Vygotsky (2004) argues that mastery of tools is as important as the development of creative imagination: First, for adolescents the mere exercise of creative imagination is not enough. A drawing done any old way fails to satisfy. To embody his creative imagination an adolescent needs to acquire special professional artistic skills and abilities We thus see the problem in all its complexity. It has two parts: on the one hand, we need to cultivate creative imagination; on the other hand, a special culture is needed for the process of embodying the images created by imagination. (p. 85) Thus according to Vygotsky, students need to develop mastery of technology (in our case, e-textiles) alongside their creative imagination. He suggests that doing so in a community that specializes in the particular techniques and uses of technology related to a specific kind of art can provide a zone of proximal development for the adolescent. Thus learning develops alongside aesthetic or creative imagination. Our analysis of students design processes demonstrates this, showing what they learned through the process of embodying their aesthetic ideas in their e-textile project. Because e-textile designs involve circuits and programming, this learning developed alongside the creation of their aesthetic projects. Notably, we did not explore the motivations behind students aesthetic ideas, but this is a ripe area for further analysis as their projects showed many of their personal interests or experiences. Students often used their projects to reflect their interests in popular culture using symbols from anime, comic strips, or movie series, geek culture especially in designs that showed off the technology, feminine culture with hearts or stars or indicating their family membership (e.g., gifts for mom, dad, brother, or sister). Future analysis is needed to understand how students e-textile projects bring together personal, peer, and pop culture values while also providing a context for learning academic skills such as circuitry and programming.
References
Barton, A., Tan, E. & Rivet A. (2008). Creating hybrid spaces for engaging school science among urban middle school girls. American Education Research Journal, 45, 68-103. Brickhouse, N. W., Schultz, K., & Lowery, P. (2000). What kind of a girl does science? The construction of school science identities. Journal of Research in Science Teaching, 37, 441- 458. Buechley, L., and Eisenberg, M. (2008). The LilyPad Arduino: Toward wearable engineering for everyone.
ISLS 202
ICLS2012
Wearable Computing Column in IEEE Pervasive, 7(2), 12-15. Buechley, L., Eisenberg, M. and Elumeze, N. (2007, June). Towards a Curriculum for Electronic Textiles in the High School Classroom. In Proceedings of the Conference on Innovation and Technology in Computer Science Education (ITiCSE), Dundee, Scotland. Charmaz, K. (2000). Grounded theory: Objectivist and constructivist methods. In N.K. Denzin & Y.S. Lincoln (Eds.), Handbook of qualitative research (pp. 509535). Thousand Oaks, CA: Sage Publications. Flannery, M. (1991). Science and aesthetics: A partnership for science education. Science Education, 75, 577593. Girod, M., Twyman, T., & Wojckiewicz, S. (2010). Teaching and learning science for transformative aesthetic experience. Journal of Science Teacher Education, 21(7), 801-824. Girod, M. (2009). Evaluating the science and literacy connection by comparing three curricular units in second grade classrooms. Journal of Elementary Science Education, 21(3), 13-32. Girod, M. (2007). A conceptual overview of the role of beauty and aesthetics in science and science education. Studies in Science Education, 43, 38-61 Hadzigeogiou, Y. P. (2011). Fostering a sense of wonder in the science classroom. Research in Science Education. DOI 10.1007/s11165-011-9225-6. Jacobson, B., & Wickman, P.-O. (2008). He roles of aesthetic experience in elementary school science. Research in Science Education, 38, 45-65. Jackson, P. (1998). John Dewey and the Lessons of Art. New Haven, CT: Yale University Press. Kafai, Y. B., Peppler, K. A., & Buechley, L. (2009). Computational textiles as materials for creativity. Proposal [funded] to the National Science Foundation, Arlington, VA. Keller, E. F. (1983). A Feeling for the Organism: The Life and Work of Barbara McClintock. New Haven, CT: Yale University Press. Lemke, J. (2010, July). Affect, identity, and representation. Paper presented at the 9th International Conference of the Learning Sciences (ICLS), Chicago, Illinois. Retrieved on March 9th at http://www.jaylemke.com/storage/Affect-identity-representation-2010%2009-54-29.pdf Post, E. R., Orth, M., Russo, P. R., & Gershenfeld, N. (2000). E-broidery: design and fabrication of textile-based computing. IBM Systems Journal, 39(3-4), 840-860. Pugh, K., & Bergin, D. (2005). The effect of schooling on students out-of-school experience. Educational Researchers, 34(9), 15-23. Pugh, K., & Girod, M. (2006). Science, Art, and Experience: Constructing a Science Pedagogy from Deweys Aesthetics. Journal of Science Teacher Education, 18, 9-17. Resnick, M., Berg, R., & Eisenberg, M. (2000). Beyond black boxes: Bringing transparency and aesthetics back to scientific investigation. Journal of the Learning Sciences, 9(1), 7-30. Seymour, S. (2010). Functional Aesthetics: Visions in Fashionable Technology. New York, NY: Springer Verlag. Traweek, S. (1995). Bodies of evidence: Law and order, sexy machines, and the erotics of fieldwork among physicists. In S. Foster (Ed.), Choreographing history (pp. ). Bloomington, IN: Indiana University Press Wickman P.-O. (2006). Aesthetic experience in science education. Mawhaw, NJ: Lawrence Erlbaum.
Acknowledgments
This work was supported by a collaborative grant (0855868) from the National Science Foundation to Yasmin Kafai, Leah Buechley and Kylie Peppler. Any opinions, findings, and conclusions or recommendations expressed in this chapter are those of the authors and do not necessarily reflect the views of the National Science Foundation or the University of Pennsylvania. We thank Joyce Wang for comments on earlier versions of the paper.
ISLS
203
ICLS2012
Learning Progressions, Learning Trajectories, and Equity

Cesar Delgado, Karisma Morton, University of Texas at Austin, 1 University Station D5700, Austin, TX 78712 cesar_delgado@austin.utexas.edu, kmorton@mail.utexas.edu Abstract: Learning progressions (LPs) and learning trajectories (LTs) are an approach to research that can guide the development of coherent, integrated curriculum, instruction, and assessment. They trace the development of student ideas as they grow in sophistication, across levels that are reasonably coherent networks of ideas and practices. This paper examines whether and how published LPs and LTs address issues of equity. It presents a case study where a curriculum guided by an LP closed or reduced gaps between mainstream and minority groups. We argue that implicit in most LP/LT work is a definition of equity as equal treatment for all students, and propose an alternate definition involving responsive instruction and materials that contemplate individual differences. From the analytic literature review and case study, we abstract guidelines for the development of LPs and LTs, and curriculum materials based on these, to make them more responsive to students and thus more equitable.
Introduction
Learning progressions (LPs) and learning trajectories (LTs) have become influential ways of conducting and synthesizing research to guide the development of science and mathematics curriculum, instruction, and assessment (Duschl, Maeng, & Sezen, 2011). From the outset, three design challenges were identified in the development of LPs: describing students knowledge and practice at different points in an LP, showing how students can realistically and in connected fashion progress across levels, and describing the variety of possibilities for meaningful learning for students with different personal and cultural resources or different instructional histories (National Research Council, 2007, p. 221). Of these three challenges, the issue of equity and diversity was at that time the challenge we [were] farthest from responding to effectively with the current research base. (NRC, 2007, p. 222). In this paper, we examine the current state of equity in LP/LT research through a comprehensive analytical literature review. We also present a case study of a teaching experiment that closed or reduced gaps between mainstream and minority groups, that used a curriculum based on an LP. From the analytic literature review and case study, we abstract guidelines for the development of LPs and LTs, and curriculum materials based on these, to make them more responsive to students and thus more equitable.
We adopt a cognitive constructivist perspective, in which individuals construct their understanding through interaction with the physical and social environment, by building upon their prior knowledge (e.g., Steffe & Gale, 1995). This perspective owes much to Piagets work (e.g., Piaget, 1983) although it revalues the role of context and does not adopt Piagetian stage theory. This perspective was explicitly adopted in the original formulation of the LT (Simon, 1995) and has been retained in most published LT papers. Cognitive constructivism is implicit in, and compatible with, the LP literature as well. The first policy report discussing LPs, Taking Science to School (NRC, 2007) builds on the seminal NRC report How People Learn, which synthesizes research bases including cognitive constructivism, sociocultural theories of learning, expert-novice studies, and cognitive psychology (NRC, 1999). In addition, the cognitive constructivist perspective is compatible with our definition of equity in education outlined below.
Learning Progressions and Learning Trajectories

LPs are descriptions of the successively more sophisticated ways of thinking about a topic that can follow one another as children learn about and investigate a [science] topic over a broad span of time (e.g., 6 to 8 years) (NRC, 2007, p. 214). The LP approach addresses the gap between research and practice in education by synthesizing fragmented, small-scale research studies into a form that is meant to be informative to educators and curriculum designers (Duschl et al., 2011). Being organized around core disciplinary ideas and/or practices, LPs provide an answer to the mile-wide, inch-thick curriculum (Schmidt, McKnight, & Raizen, 1997) that results from trying to meet standards that are primarily extensive lists of factual knowledge with little prioritization (NRC, 2007). LPs also address the fragmentation of knowledge that results from the use of modules or kits that are meant to be independent and can be sequenced in a variety of ways (Duschl et al., 2011). The first LPs were commissioned papers constructed by groups of science education experts, and dealt with evolution and the nature of matter as well as assessment (respectively, Catley, Lehrer, & Reiser, 2005; Smith, Wiser, Anderson, & Krajcik, 2006; Wilson & Bertenthal, 2005). LTs originated in the mathematics education community roughly a decade prior to LPs. Simon (1995) defined three components of hypothetical LTs (HLTs): the learning goal, learning activities or tasks, and a model of students thinking and understanding. Published LTs have tended to use this same definition or a
ISLS
204
ICLS2012
similar one, and the time frame for LTs is usually much shorter than that proposed for LPs a few classes or a unit (with exceptions, e.g., Confrey, Maloney, Nguyen, Mojica, & Myers, 2009). HLTs are iterative and malleable, the actual learning trajectories traversed by students vary by individual, and the teacher is involved in their creation and revision. Recently, LTs and LPs have begun to converge, with LTs seen as finer-grained components of LPs (Stevens, Delgado, & Krajcik, 2010; Plummer & Krajcik, 2010; Battista, 2011).
Equity
Arguments for equity in education include both instrumental and intrinsic rationales (Klasen, 2002). Greater equity in education is an intrinsic goal it is good in and of itself, on ethical grounds. It is also an instrumental goal: it serves the purpose of maximizing human capital and thus improving the economic competitiveness of a nation (Klasen, 2002). Over the years, there have been many definitions of equity in education (see Kahle, 1996). The first view, prevalent during the Sputnik scare, is purely instrumental, and involves providing the most resources to those students who are most likely to achieve. This view is consistent with tracking and other strategies that seek to maximize the learning of the most capable. Other views stress the intrinsic argument, but propose different strategies. A second definition, dating from the civil rights era, views equity as consisting of equal treatment for all students (Kahle, 1996). This view is consistent with the view that a rising tide raises all ships, e.g., improved curriculum will adequately address equity concerns by improving education for all. A third, postmodern view of equity also stresses the intrinsic argument but acknowledges existing inequities in society. Thus, it proposes responsive, individualized attention to students in order to compensate for past lack of opportunities and to promote social justice. Applying similar treatment to all students is not likely to result in equality of outcomes for equally capable students, given an uneven playing field. For example, African American and Hispanic students in the USA are three times as likely to be economically disadvantaged (i.e., qualifying for free/reduced lunch programs) and their average achievement on the National Assessment of Educational Progress is around 25 points lower (on a scale of 0-500) than their non-Hispanic White and Asian American peers in math and reading achievement (Hemphill & Vanneman, 2010; Vanneman, Hamilton, Baldwin Anderson, & Rahman, 2009). US females are outnumbered by their male counterparts in STEM fields and express less interest in pursuing STEM careers in college (Hill, Corbett, & St. Rose, 2010). Addressing these issues cannot be done simply by applying the same treatment to all students. Thus, we subscribe to the third definition of equity above. We feel that a definition of equity that involves being responsive to students individual needs is congruent with a cognitive constructivist learning theory, which proposes building on each individuals prior knowledge. We organize our study around the following research questions: 1. 2. To what degree do extant LPs and LTs explicitly consider issues of equity? What characteristics of an LP/LT and of the curriculum developed based on the LP/LT might be important in closing achievement gaps between ethnic/racial groups?
Methods
Analytical Literature Review
By means of the analytical literature review we address our first research question. The papers analyzed here propose LPs and LTs for disciplinary content and/or processes. They came primarily from journal special issues and conferences: the journal Mathematical Thinking and Learning (vol. 6 number 2, 2004), Journal of Research in Science Teaching (volume 6 number 6, 2009), and the 2009 conference Learning Progressions in Science Conference, Iowa City, IA, June 2009. We also searched leading journals in science and mathematics education for additional papers on LPs and LTs. LT papers examined included Battista, 2004, 2011; Confrey et al., 2009; Gravemeijer, Bowers, & Stephan, 2003; McGatha, Cobb, & McClain, 2002; Rousham, 2003; Sarama & Clements, 2009; and Steffe, 2004. LP papers included Stevens et al., 2010; Talanquer, 2009; Adadan, Trundle, & Irving, 2010; Smith et al., 2006; Claesgens, Scalise, Wilson, & Stacy, 2009; Merritt, 2010; Lee & Liu, 2010; Alonzo & Steedle, 2009; Duncan, Rogat, & Yarden, 2009; Duncan & Tseng, 2011; Roseman, Caldwell, Gogos, & Kuth, 2006; Songer, Kelcey, & Gotwals, 2009; Catley et al., 2005; Mohan & Anderson, 2009; Mohan, Chen, & Anderson, 2009; Plummer, 2009; and Plummer & Krajcik, 2010. We qualitatively analyzed each LP/LT paper along three dimensions. Influenced by Rodriguezs (2005) critique of color-blind language in the US National Science Education Standards (National Research Council, 1996), we decided to examine the demographics of the participants in each LP/LT paper. We also looked for specific mention of equity issues (e.g., differences in the LP/LT by group), and whether all students ideas were addressed. Additionally, based on a critique by Salinas (2009), we looked for evidence that the LP/LT considered funds of knowledge (Moll, Amanti, Neff, & Gonzalez, 1992). The term funds of knowledge refers to the knowledge that students bring to the classroom through their home and community experiences. Students gain this knowledge voluntarily, through their interests and questions. From a cognitive
ISLS
205
ICLS2012
constructivist perspective, considering students funds of knowledge is essential. Thus, in investigating our first research question, we examine the composition of the students involved in the research for each LP/LT, specific mentions of equity issues and all students ideas, and the use of students funds of knowledge.
Case Study
Through the case study, we address our second research question. The case study involves a teaching experiment conducted in a free, two-week summer nanoscience camp for 31 middle school students from a diverse, low SES public school district (described in Delgado, 2009a, 2009b; Delgado, Short, & Krajcik, 2009). One of the curriculum strands focused on size and scale, and was developed based on a learning progression (Delgado, 2009a). We analyzed and compared the pre- and post-camp achievement of two groups of students who agreed to participate, were attending the camp for the first time, and attended all sessions: seven mainstream or high-performing minority participants (one Asian-American and six non-Hispanic White students), and 17 historically lower-performing minorities (one Hispanic and 16 African-American students; two did not take the pre-test). While different minority groups have different cultural backgrounds, we placed the students into two groups because of the low number of participants from some racial/ethnic backgrounds. Two dimensions of knowledge were examined: consistency of knowledge across various aspects or ways of thinking about size (ordering, grouping, relative scale, and absolute size), which is a measure of conceptual understanding; and factual knowledge of the size of key objects such as cells, atoms, molecules, the Earth, humans, etc. (using the same four aspects). The means of the two groups were compared to each other, before and after camp. Our reanalysis for this paper used t-tests for normally distributed variables (pre-camp consistency and pre-camp factual knowledge) and Mann-Whitney non-parametric tests for the non-normally distributed variables (post-camp consistency and factual knowledge). Normality of distribution was tested using the Shapiro-Wilk test. We found that there was a statistically significant difference pre-camp between minority and mainstream groups on consistency (3.0 for mainstream, 1.78 for minority, on a scale of 0-5). Mainstream students also outperformed minority students on factual knowledge pre-camp (10.4 vs. 8.5, on a scale of 0-21), but the difference did not reach statistical significance at the 0.05 level. Thus, there was a gap favoring mainstream students pre-camp. After experiencing the curriculum developed based on an LP, however, the scores for consistency were practically identical for both groups (around 3), and factual knowledge increased for both groups but more for the minority students. Thus, this teaching experiment was effective in increasing achievement and closing gaps. Therefore, it is a good case to analyze in seeking to determine why an LP-guided curriculum was effective. We analyzed the movement of students from pre- to post-camp for both consistency and factual knowledge in order to detect patterns that might provide insights into what aspects of the curriculum might have been most effective in closing gaps across the racial/ethnic groups. We also used the analysis of the learning activities in terms of the levels of the LP each activity addressed (Delgado, 2009a).
Findings
Research Question 1
We examined the extant literature on LP/LTs for the composition of the student participants. The use of a nationally representative group would enable the LP or LT to reflect the ideas of differing groups of students. In most of the studies the students were not very diverse. Several of the studies used predominantly White, rural or suburban, middle class students (e.g., Adadan et al., 2010; Alonzo & Steedle, 2009; Mohan & Anderson, 2009; Mohan et al., 2009; Plummer & Krajcik, 2010). Many of the LPs or LTs omitted this information altogether. Three LPs (Lee & Liu, 2010; Songer et al., 2009; Stevens et al., 2010) did use student participants that were representative across different groups (e.g., achievement level and race/ethnicity) or groups that were traditionally underrepresented (e.g., low SES ethnic/racial minorities). We found that the current LP literature does not focus on all students ideas. Instead, the focus is only on those ideas that are shared by most students. Like many LPs, Mohan and Anderson (2009) include only one pathway to the upper anchor, based on the existing curriculum. In a subsequent study (i.e., Mohan, Chen and Anderson, 2009), they propose one alternative pathway, with little attention to how different groups of students may vary in traversing these pathways. While Talanquer (2009) laudably identifies a series of commonly held student ideas, he used student ideas that were held by a majority of students: We focused our attention on those ideas held by a large proportion of students or those that seem to persist at different learning stages (pg. 2127), in phase 1; in the next phase, he incorporated general beliefs identified in prior literature. Notably, Adadan, Trundle and Irving (2010) made sure to explore all of the alternative ideas of students in their LP development. It could be argued that the large grain-size and temporal scope of LPs precludes detailed descriptions of student ideas, but we argue that ignoring or neglecting ideas that are not in line with the mainstream way of thinking, discriminates against those students. Most LP authors raise few or no issues of equity in their publications. This is justifiable from a definition of equity as equal treatment, but not from our point of view.
ISLS
206
ICLS2012
LTs stem from an explicitly cognitive constructivist paradigm (Simon, 1995). Furthermore, a model of student thinking is a component of LTs. Nevertheless, the LT literature does not always pay close attention to diverse student ideas. For example, McGatha, Cobb, and McClain (2002) state that they excluded the ideas of some students that differed from the expected and dominant ideas. Steffe (2004) constructed an LT for commensurate fractions for two students, Jason and Laura, who displayed different abilities in partitioning objects. Jason has an equi-partitioning scheme that allows him to advance further than Laura, who relies on Jasons explanations to solve tasks. The author maintains that Lauras partitioning and iterating were not part of the same mental structure, as they were for Jason. However, Lauras estimates of a fraction of a stick were on repeated occasions uncannily accurate (p. 133). The proposal of one path (Jasons) to an understanding of commensurate fractions, and the way in which Lauras accurate estimates are not seen as suggestive of an alternative, productive pathway, illustrate less responsiveness to student ideas than would seem desirable. (See also Baroody, 2004). In our view, the case of Laura merits closer attention, as a potential alternative pathway to understanding of commensurate fractions. Closer attention to student ideas would be desirable in any LP or LT. The vast majority of the LPs and LTs did not consider the type of knowledge that students bring to the classroom from their community or family experiences. One exception was Plummer and Krajcik (2010), who specifically talked about including students observations of the world, and cultural interactions" when developing the lower anchor of their LP. Songer, Kelcey & Gotwals (2009) mentioned inclusion of student knowledge. In the case of Laura discussed previously (Steffe, 2004), a possible explanation for the uncannily accurate estimates of fractions could be experiences at home. For instance, if her grandmother happened to bake and sell cakes to neighbors, Laura may have had substantial experience slicing cakes accurately into the number of slices requested by a client. Alternatively, if her older sibling had a hobby or business constructing birdhouses, then Laura might have observed or participated in the sawing of long boards into shorter pieces of even lengths. Such concrete wisdom might not be as abstract or generalizable as the coordination of partitioning and iteration schemes, but it might be the kind of prior knowledge that an LT for fractions could be built on, possibly leading to an entirely different pathway to the same goal.
Research Question 2
We represented the observed changes in consistency (i.e., conceptual knowledge) from the teaching experiment from Delgado (2009a) in a new graphic form to better visualize these changes. See Figure 1. Each arrow represents the initial and final state of conceptual knowledge of a single participant. The start of the arrow represents the initial level, and the head of the arrow the final level. Vertical arrows represent students who had no change in their consistency of knowledge, arrows pointing to the right represent students whose consistency increased and arrows pointing to the left represent students whose consistency decreased. We found that the LPguided curriculum was most effective at helping students under level three, even though most activities were designed to help students achieve levels four and five, involving the coordination of relative scale and absolute size (e.g., realizing that if object A is X times longer than object B, and the absolute size of one object is known, then the absolute size of the other object can be calculated). One instructional activity involve using the optical microscope to successively visualize (at different magnifications): a hair (~0.1 mm in thickness) atop a thin plastic ruler with millimeter markings (1 mm between marks); cheek (~30 m) and skin cells (~15 m) next to the hair; and Staphylococcus aureus bacteria (~1 m) next to the cells. The strategy of visualizing successively smaller objects obeyed general pedagogical strategies, such as going from the concrete and familiar to the abstract and unfamiliar, and to gradually build skills using the microscope. But it concomitantly resulted in an ordered series of objects with successively smaller absolute sizes (supporting Level 3 knowledge, involving the coordination of absolute size and ordering) and larger relative scale factors compared to the reference object of the hair (supporting Level 2 knowledge, involving consistency across ordering and relative scale).
Figure 1. Changes in student consistency (conceptual knowledge), in the teaching experiment. Each arrow represents the initial and final level for one student. The dotted arrows represent non-Hispanic White/Asian American students. Vertical arrows indicate no change.
ISLS
207
ICLS2012
A subsequent instructional activity used a custom-built computer simulation that uses images of the same objects (hair, cells, bacteria) but extends down to atoms (see Figure 2). Along with a handout to scaffold students calculation of the absolute size of objects, this activity too targeted Level 4 and 5 understanding (the connection between relative scale and absolute size). There is a sidebar listing all of the objects by size, grouped into those that can be seen at a given magnification, those that are too small, and those that are too large to see completely. This sidebar was primarily meant to allow students to quickly identify objects, as scrolling over their names resulted in highlighting the image of the corresponding object. However, the sidebar itself scaffolds Level 1 understanding of the connection between ordering and grouping. The sidebar in conjunction with the simulation scaffolds understanding of the link between ordering and relative scale (Level 2), and along with the handout to calculate the size of the objects, scaffolds the comprehension of the relationship between ordering and absolute size (Level 3). By including a macroscopic object, the hair, both the optical and virtual microscope activities help Level 0 students who believe that no objects can exist that are too small to see.
Figure 2. Screen shot of virtual microscope simulation used in the teaching experiment. The sidebar provides ordering and grouping of objects; the main field calculates relative scale and a handout involves absolute size. An examination of changes in factual knowledge showed that most students, regardless of race/ethnicity or initial level of knowledge, increased from pre- to post-camp. See Figure 3.
Figure 3. Changes from pre- to post-camp factual knowledge of the size of objects. The diagonal line indicates no change, data points above the line represent an increase. Triangle markers represent Hispanic and African American students, circle markers non-Hispanic White and Asian American students.
ISLS
208
ICLS2012
Discussion and Recommendations

Based on the analytical literature review and case study, we issue the following recommendations to researchers developing LPs or LTs: 1. Use cross-cultural research to detect a diversity of student ideas. This can be achieved by conducting research in schools with very diverse populations, or in a variety of schools each serving more homogenous groups. The oversampling of traditionally underserved groups would be desirable, in order to ensure that LPs and LTs are responsive to those who face more difficult border crossings between home and school cultures (Jegede & Aikenhead, 1999). Make a greater effort to include all students ideas, rather than just the most common ones. Student ideas closer to the normative scientific or mathematical ideas may be easier to understand, and can be seen more clearly as productive intermediate steps on an LP or LT, but the premise behind LP and LT research is to build on student knowledge. Field analyses may be necessary to get a full understanding of the types of prior knowledge that students bring to the classroom (e.g., Moll et al., 1992). LP and LT developers may have to go into the communities where students reside and get first-hand exposure to the types of informal science and math experiences that students have, so that these can be leveraged in the LP or LT. When designing curriculum units and instructional activities based on an LP or LT, follow a broadspectrum approach that targets a specific level but simultaneously provides scaffolding for students at lower levels so they may build or reinforce foundational understandings. Project-based and inquiryoriented instructional approaches already seek to build basic knowledge and skills in the context of student-centered investigations that simultaneously build higher-level cognitive and metacognitive abilities, and may be useful in designing broad-spectrum lessons and units.
2.
3.
4.
Conclusions
These recommendations may seem daunting and time consuming, however, the collaboration between researchers, teachers, specialists, and communities can only serve to produce positive results. If we are serious about promoting equity and serving ALL students, we must be willing to do what it takes to make that happen. Research groups developing LPs and LTs should ideally include advocates for certain groups of students, for example, an expert on special education and team members that are deeply knowledgeable about the culture of minority students. Developing learning progressions and learning trajectories that do not address inequity in educational opportunities in math and science for students will only exacerbate the current problem. As the learning sciences, science education, and mathematics education fields continue to negotiate and define the nature of LPs and LTs, an expansion to include equity concerns at the forefront can greatly benefit groups that have been traditionally underserved.
References
Adadan, E., Trundle, K. C., & Irving, K. E. (2010). Exploring Grade 11 students conceptual pathways of the particulate nature of matter in the context of multirepresentational instruction. Journal of Research in Science Teaching, 47(8), 10041035. Alonzo, A. C., & Steedle, J. T. (2009). Developing and assessing a force and motion learning progression. Science Education, 93(3), 389421. Baroody, A. J., Cibulskis, M., Lai, M., & Li, X. (2004). Comments on the use of learning trajectories in curriculum development and research. Mathematical Thinking & Learning, 6(2), 227260. Battista, M. T. (2004). Applying cognition-based assessment to elementary school students development of understanding of area and volume measurement. Mathematical Thinking & Learning, 6(2), 185204. Battista, M. T. (2011). Conceptualizations and issues related to learning progressions, learning trajectories, and levels of sophistication. The Mathematics Enthusiast, 8(3), 507570. Catley, K. M., Lehrer, R., & Reiser, B. J. (2005). Tracing a prospective learning progression for developing understanding of evolution. Paper commissioned by the National Academies Committee on Test Design for K-12 Science Achievement. Claesgens, J., Scalise, K., Wilson, M., & Stacy, A. (2009). Mapping student understanding in chemistry: The perspectives of chemists. Science Education, 93(1), 5685. Confrey, J., Maloney, A., Nguyen, K., Mojica, G., & Myers, M. (2009). Equipartitioning/splitting as a foundation of rational number reasoning using learning trajectories. PME Conference Proceedings (Vol. 2, pp. 345352). Presented at the Psychology of Mathematics Education, Thessaloniki, Greece.
ISLS
209
ICLS2012
Delgado, C. (2009a). Development of a research-based learning progression for middle school through undergraduate students conceptual understanding of size and scale. Unpublished doctoral dissertation, University of Michigan, Ann Arbor. Delgado, C. (2009b). Learning progressions as a tool for equity (Ethics and Equity Scholars Symposium). Presented at the International Conference of the National Association for Research in Science Teaching (NARST), Garden Cove, CA. Delgado, C., Short, H., & Krajcik, J. S. (2009). Design, implementation, and evaluation of the effectiveness of a 12-hour middle school instructional unit for size and scale. Presented at NARST, Garden Cove, CA. Duncan, R. G., Rogat, A. D., & Yarden, A. (2009). A learning progression for deepening students understandings of modern genetics across the 5th-10th grades. Journal of Research in Science Teaching, 46(6), 655674. Duncan, R. G., & Tseng, K. A. (2011). Designing project-based instruction to foster generative and mechanistic understandings in genetics. Science Education, 95(1), 2156. Duschl, R. A., Maeng, S., & Sezen, A. (2011). Learning progressions and teaching sequences: A review and analysis. Studies in Science Education, 47(2), 123182. Gravemeijer, K., Bowers, J., & Stephan, M. (2003). Chapter 4: A hypothetical learning trajectory on measurement and flexible arithmetic. Journal for Research in Mathematics Education. Monograph, 12, 5166. Hemphill, F. C., & Vanneman, A. (2010). Achievement gaps: How Hispanic and White students in public schools perform in mathematics and reading on the National Assessment of Educational Progress. Washington, DC: National Center for Education Statistics. Hill, C., Corbett, C., & St. Rose, A. (2010). Why so few? Women in science, technology, engineering, and mathematics. Washington, DC: AAUW. Jegede, O. J., & Aikenhead, G. S. (1999). Transcending cultural borders: Implications for science teaching. Journal for Science & Technology Education, 14, 4566. Kahle, J. B. (1996). Thinking about equity in a different way. Washington, DC: American Association for the Advancement of Science. Klasen, S. (2002). Low schooling for girls, slower growth for all? Crosscountry evidence on the effect of gender inequality in education on economic development. The World Bank Economic Review, 16(3), 345 373. Lee, H., & Liu, O. L. (2010). Assessing learning progression of energy concepts across middle school grades: The knowledge integration perspective. Science Education, 94(4), 665688. McGatha, M., Cobb, P., & McClain, K. (2002). An analysis of students initial statistical understandings: developing a conjectured learning trajectory. Journal of Mathematical Behavior, 21(3), 339355. Merritt, J. D. (2010). Tracking students understanding of the particle nature of matter. Unpublished doctoral dissertation, University of Michigan. Retrieved from http://deepblue.lib.umich.edu/handle/2027.42/78789. Mohan, L., & Anderson, C. W. (2009). Teaching experiments and the carbon cycle learning progression. Presented at the Learning progressions in science conference, Iowa City, IA: Learning Progressions in Science. Mohan, L., Chen, J., & Anderson, C. W. (2009). Developing a multiyear learning progression for carbon cycling in socioecological systems. Journal of Research in Science Teaching, 46(6), 675698. Moll, L. C., Amanti, C., Neff, D., & Gonzalez, N. (1992). Funds of knowledge for teaching: Using a qualitative approach to connect homes and classrooms. Theory Into Practice, 31(2), 132141. National Research Council. (1996). National Science Education Standards: Observe, interact, change, learn. Washington, DC: National Academy Press. National Research Council. (1999). How people learn: Brain, mind, experience, and school. (J. D. Bransford, A. L. Brown, & R. R. Cocking, Eds.) (1st ed.). National Academies Press. National Research Council. (2007). Taking science to school: Learning and teaching science in grades K-8. (R. A. Duschl, H. A. Schweingruber, & A. W. Shouse, Eds.). Washington, D.C.: National Academies Press. Piaget, J. (1983). Piagets theory. Handbook of child psychology (4th ed., Vol. 1). New York: Wiley. Plummer, J. D. (2009). Early elementary students development of astronomy concepts in the planetarium. Journal of Research in Science Teaching, 46(2), 192209. Plummer, J. D., & Krajcik, J. S. (2010). Building a learning progression for celestial motion: Elementary levels from an earthbased perspective. Journal of Research in Science Teaching, 47(7), 768787. Rodriguez, A. J. (2005). Science for All and invisible ethnicities: How the discourse of power and good intentions undermine the National Science Education Standards. Multicultural science education: Theory, practice, and promise, Studies in the postmodern theory of education (pp. 2136). New York: Peter Lang.
ISLS
210
ICLS2012
Roseman, J. E., Caldwell, A., Gogos, A., & Kuth, L. (2006). Mapping a coherent learning progression for the molecular basis of heredity. Presented at the National Association for Research in Science Teaching Annual Meeting, San Francisco, CA: Project 2061. Rousham, L. (2003). The empty number line: A model in search of a learning trajectory. Enhancing primary mathematics teaching (pp. 29-40). Maidenhead: Open University Press. Salinas, I. (2009). Learning progressions in science education: Two approaches for development. Presented at the Learning Progressions in Science Conference, Iowa City, IA. Sarama, J., & Clements, D. H. (2009). Early childhood mathematics education research: Learning trajectories for young children. Taylor & Francis. Schmidt, W. H., McKnight, C. C., & Raizen, S. A. (1997). A splintered vision: An investigation of U.S. science and mathematics education. Boston: Kluwer Academic Publishers. Simon, M. A. (1995). Reconstructing mathematics pedagogy from a constructivist perspective. Journal for Research in Mathematics Education, 26(2), 114145. Smith, C. L., Wiser, M., Anderson, C. W., & Krajcik, J. S. (2006). Implications of research on childrens learning for standards and assessment: A proposed learning progression for matter and the atomic molecular theory. Focus Article. Measurement: Interdisciplinary Research and Perspectives, 14, 198. Songer, N. B., Kelcey, B., & Gotwals, A. W. (2009). How and when does complex reasoning occur? Empirically driven development of a learning progression focused on complex reasoning about biodiversity. Journal of Research in Science Teaching, 46(6), 610631. Steffe, L. P. (2004). On the construction of learning trajectories of children: The case of commensurate fractions. Mathematical Thinking & Learning, 6(2), 129164. Steffe, L. P., & Gale, J. E. (Eds.). (1995). Constructivism in education. Hillsdale, N.J: Lawrence Erlbaum. Stevens, S. Y., Delgado, C., & Krajcik, J. S. (2010). Developing a hypothetical multi-dimensional learning progression for the nature of matter. Journal of Research in Science Teaching, 47(6), 687715. Talanquer, V. (2009). On cognitive constraints and learning progressions: The case of structure of matter. International Journal of Science Education, 31(15), 21232136. Vanneman, A., Hamilton, L., Baldwin Anderson, J., & Rahman, T. (2009). Achievement gaps: How black and white students in public schools perform in mathematics and reading on the National Assessment of Educational Progress. Statistical analysis report. Washington, DC: National Center for Education Statistics. Wilson, M., & Bertenthal, M. W. (Eds.). (2005). Systems for state science assessment. Washington, DC: National Academies Press.
Acknowledgments
We wish to thank the following persons, who developed summaries of the LP/LT papers: Tara Craig, Ivy Girao, Sandra Jaramillo, Lance Kinney, Pat Ko, Wan Sin Lim, Tina Vega, and Hye Sun You. We would also like to thank Margaret Lucero and the 3 anonymous reviewers for their valuable feedback. This research was supported by the National Science Foundation through the National Center for Learning & Teaching in Nanoscale Science and Engineering (NCLT) grant no. 0426328. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors.
ISLS
211
ICLS2012
A Teachers Journey in Knowledge Building Pedagogy

Nancy Law, Johnny Yuen, Hidy Tse, The University of Hong Kong, Pokfulam, Hong Kong Email: nlaw@hku.hk, johnyuen@hku.hk, hidytse@hku.hk Abstract: A productive and engaging discourse is central to knowledge building (KB). How to guide students in identifying good questions and to engage in deepening discourse using the functions available in Knowledge Forum (KF) in accordance with the 12 KB principles is a major challenge to teachers interested in adopting KB pedagogy in their teaching. This paper reports on a study of a teachers pedagogical journey over three years, from the time when she was a novice teacher making the first attempt in introducing KB in her classroom to becoming fluent and confident in designing and executing curriculum units that will be successful in advancing students understanding through their engagement in synchronous discourse on KF. The findings reveal a shift in the teachers design focus, followed by a refinement in facilitation skills. The paper also reports on the changes in students discourse behavior and quality as the teachers pedagogical advancement progresses.
Introduction
Knowledge building (KB) is advocated by Scardamalia and Bereiter (1999) as a pedagogical approach that engages learners in a process of inquiry to advance the collective knowledge of the learners about a meaningful problem or issue as a community (in much the same way as researchers work to advance the knowledge of the academic community they belong to). Discourse plays a central role in this approach, but KB would not happen naturally just by putting a group of people into discourse, be they face-to-face or online. An important educational goal of KB pedagogy is to foster students socio-metacognitive capacity to build knowledge through intentional collaborative inquiry (Bereiter and Scardamalia, 1989). Scardamalia (2002) further identified 12 socio-cognitive determinants (or KB principles) that underpin the functional design of KF as a KB technology. The teacher faces a lot of challenges in their efforts to implement KB in their classroom (Lakkala, Lallimo and Hakkarainen, 2005), including not only the need to understand the theoretical underpinning of KB, but also how to apply the theories in practice. The latter involves task design, organization of the collaboration, and the role given to the web-based collaborative learning environment. Bielaczyc (2006) goes further to argue for the need to design an appropriate, four dimensional social infrastructure (cultural beliefs, practice, sociotechno-spatial relations and interaction with the outside world) in order to realize the potential of technology tools to support learning that involve social interactions. Research on teacher professional development to promote KB adoption points to the importance of creating a knowledge building community among teachers (T-KBC) (Chai and Merry, 2006; Chan and van Aalst, 2006) as a key success factor. In recent years, sustained network communities of KB teachers and researchers connected locally and internationally have demonstrated success in fostering an expanding community of teachers who make progressive improvements in their KB pedagogical practices (Laferrire et. al. 2010). These successes are encouraging evidence that deep changes in pedagogy are possible even though the larger educational context in terms of curriculum standards and public examinations remain largely traditional. On the other hand, our observation as participant co-learners since 2001 in the Learning Community Projects for Knowledge Building in Schools (http://lcp.cite.hku.hk, to be referred to as LCP in short) of the more than 100 teachers who have partiicpated in the LCP projects at some stage, only a small number have made sustained and progressive improvements in their pedagogical implementation of KB. Apparently, the teachers follow different learning trajectories, with different learning outcomes in terms of beliefs, knowledge, skills and practices. While there are research findings about teacher learning for KB implementation, there is not much in the literature that describes the pathways of change that a teacher goes through from being a novice KB teacher, to becoming an expert. Teacher learning and progress in teacher practice are connected, emerging processes. How does a teachers understanding evolve alongside his/her practice? Is there a progressive set of phases in the implementation path that a teacher would need to go through as Bielaczyc (2006) anticipates, or are there identifiably different pathways, which may be context dependent? If a teacher progresses in KB pedagogy, then arguably his/her students engagement in and outcomes from their KB activities should also demonstrate greater advancement. Following this line of reasoning, there have been preliminary attempts to study teachers trajectories of learning through examining changes in students discourse behavior (Law & Wong, 2003; Law et al, 2011). However, these studies do not include examinations of changes in teachers beliefs or practices. This paper reports on a study of a teachers journey over a period of three years in her efforts to introduce KB in her classrooms. This is an in-depth case study to understand how a teacher managed to traverse the deep changes needed in terms of knowledge, skills and beliefs to become a competent KB teacher. We also examine changes in students discourse behavior over the three years to find out if the changes in KB pedagogy brought about progressive advances in students learning using indicators of KB engagement and outcomes from the literature.
ISLS 212
ICLS2012
Research Context, Data Collection and Research Design

LCP was a University-School Partnership project organized as a design research (Barab, 2004) with teachers and teacher educators working together as co-researchers and co-learners in a professional Network to implement KB as a scalable pedagogical innovation in Hong Kong schools. Teachers within the project held scheduled meetings to co-plan KB curriculum units and share teaching plans, which are collected and archived in the LCP project database. All online discourse posted by students and teachers on Knowledge Forum since the beginning of the project were also archived and accessible for research and professional development purposes. Teachers in the project were encouraged to write reflection notes on their practice and invited for interviews from time to time. This rich archive of LCP data constitutes a core data source for the current study. In selecting a teacher for this study, we first identified a number of teachers known to have made significant advances in their understanding of KB as well as in their teaching practices over the years. We finally selected TH as the focus for our case study as she taught the same subject at the same class level over a period of three years. TH had five years of teaching experience in schools when she joined LCP in September 2005. She was attracted to the use of a discussion forum, which she considered to be an additional channel for students to learn some important scientific concepts in a more interesting way when she attended a teacher workshop on KB in 2004. She joined LCP in 2005-06 when she moved to a new school whose principal encouraged all teachers to adopt more student-centered inquiry-oriented approaches to teaching. She tried to implement KB in her grade 7 science classes during each of the three years she worked in that school. In addition to working collaboratively with teachers in the local network, TH participated in the Knowledge Building International Program (KBIP, Laferrierre and Law, 2010) during the 06-07 and 07-08 school years. In this paper, we will first report on the changes in pedagogical design and execution observed through an analysis of the teaching plans and the teacher interviews (interviews additional to those in the archive were conducted for the purpose of this study). We also examine the quality of the students discourse through (i) their participation statistics, (ii) their social interaction patterns, and (iii) the knowledge advances made by the students. We then discuss the nature of the observed changes in teacher practice and students outcomes, and the insight we gain on teacher learning for KB.
Changes in Pedagogical Design

05-06 Energy transformation and energy conservation
During the first year of her efforts to introduce KB in the classroom, TH focused on using forum discussions to support student learning. She chose the unit on energy in the Grade 7 Science curriculum as the context for her KB implementation. The conceptual focus for the unit was on transformation of different forms of energy and the principle of energy conservation. The plan basically followed the topic sequence and activities in the textbook. Some end-of-chapter questions were selected for the students to discuss on KF. Table 1 presents an outline of her teaching plan for the first three topics in the unit and the questions she asked the students to discuss at different points during the teaching process. It should be noted that TH did not post any question on KF. She only gave instructions to students on possible questions to discuss during lessons. Table 1 An excerpt of the teaching plan outline of TH for her first KB implementation in 05-06. Core content Teaching activities Introducing the different Conduct textbook expt. and forms of energy: heat, light, identify different forms of energy sound, kinetic, potential, Build simulated roller-coaster etc. using online tools Search web for info. on energy change in roller coaster rides Energy Energy conversion: Conduct activities and expt. in changes controlled and uncontrolled, textbook intermediate forms of Watch video on gas explosion, energy during conversion discuss energy changes involved Fuels Common fuels, safety in Follow textbook activities using fuels Find out the major forms of fuel used in Hong Kong Topic Forms of energy Questions for forum discussion How do different types of energy affect motion of a roller coaster? What is potential energy? At what point in a roller coaster ride does potential energy become kinetic energy? What are the intermediate forms of energy in roller coaster and horror tower rides?
What makes a good fuel? What are safety issues in the storage and transporting of fuels?
She used theme park rides in Disneyland and Ocean Park as an anchor for the unit to make the targeted scientific concepts more interesting and accessible, as she explained in the interview: I spent a lot more time [than normal] to prepare [for this unit], and found a lot of resources on the web. I also gave students a model roller coaster to investigate this topic. I didnt really do this before, The students found these to be interesting, ... There was not much discussion in the classroom.
ISLS 213
ICLS2012
06-07 If energy is conserved, why is there still an energy crisis?

In 06-07, TH also used the same curriculum unit on energy for her KB work. This time, she was collaborating with two other LCP teachers who were similarly in their second or third year of KB implementation in their classrooms, and were preparing,their students to present their learning outcomes in a KBIP video-conference event (Laferrierre and Law, 2010) to students in Canada and Spain. There was no curriculum co-planning, but they shared teaching plans and coordinated the videoconference schedule for student presentation. The actual teaching plan for this year (see Box 1) which she shared with other KBIP teachers was very different from the one used in the previous year. It no longer contained any reference to specific curriculum topics or the activities mentioned in the textbook. Much thought was given to making the discussion more engaging, particularly through drawing the students attention to a paradoxif energy is conserved, why is there still an energy crisis? The following is an excerpt from a section in her plan, titled rationale: The idea "conservation of energy" seems counter-intuitive to secondary school students, some of them may interpret the idea that "energy is not created or destroyed" to mean that energy is stored up in the system and can even be released again in its original form. . One particular interest is to investigate students' conceptual change on "conservation of energy". In order to discuss with others, they have to provide evidence to justify their viewpoints, e.g. experimental evidence or . If they use other scientists' opinions , then what support these scientists' opinion? Box 1. Summary of THs teaching plan for her KB work in 06-07. Phase 1 Aim: To elicit students' concepts about energy and energy source/fuel, relationship between fuel & energy. Seed questions (questions posted by the teacher to stimulate student discussion): Initial seed questions in December: 1. What is energy? 2. What is energy crisis? In January, introduce new seed: 3. If energy is conserved, why is there still energy crisis? (N.B. The first two seed questions focus on the "what" in energy crisis, followed by a "why" question.) Notes on facilitating students work on KF: It is the first time for students to use KF, good starting point to get them to use scaffolds. This is a stage of exploration and sharing of ideas, students should demonstrate a few KB principles, community knowledge, collective responsibility, democratizing knowledge. Besides the open exploration and sharing of ideas, students need to know that ideas can be improved, and to demonstrate the constructive use of authoritative sources. Students are also expected to write their learning diary on KF. Towards the end of phase 1, rise above notes will be written under teacher's guidance. Phase 2 Three learning tasks are designed for the students to conduct in parallel. Task 1: Class level activity - Tackling problems of understanding through discussions. Seed questions: 1. What are the differences, if any, between these 3 words: energy, fuel and power? 2. What is the difference between renewable and non-renewable energy? Are both forms of energy conserved during energy transformations? Student engagement: 1. Read notes on KF at least twice a week, post at least two notes by mid-Feb. 2. Write a reflection note on the most impressive/important ideas learnt from the discussion in the learning diary view by the end of Feb. Task 2: Group level activity - Solving the energy crisis by designing or developing something. Each group of students may decide to work in one of the following ways: Design artifacts (e.g. posters, songs, movie clips, animations, etc.) that would contribute to public education on energy crisis and energy conservation. Design technology (an operable product or a design plan) to help solve the energy crisis. Develop energy conservation policies/plans/proposals, with rationales, for the government/schools/families/individuals in Hong Kong to implement. Task 3: Community level activity Deepening discussion with the broader international community. Identify thoughtful notes from students phase 1 work and post on the KBIP collaboration view on KF to stimulate further discussions by all students from all schools. The plan in Box 1 reveals that THs planning was focused on designing and guiding the discussion, including the incorporation of the KB principles and use of the KF functions. The excerpt from her written plan below indicates that she considered the schedule to be tentative and dependent on students progress: The duration for this stage of discussion may vary. It depends on the discourse quality. After a week, sooner or later, I expect they can start to write their learning diary, and then, we will have a rise above view on top of their discussion.
ISLS
214
ICLS2012
07-08 Wildspacesustainability and water

We observed a further change in planning during the third year of THs ventures into KB pedagogy. In summer 2007, she participated in the IKIT Knowledge Building Summer Institute (SI) at OISE, University of Toronto (http://ikit.org/SummerInstitute2007/Highlights/index.php). The SI gave her and other participating teachers an opportunity to plan more integrated KBIP programs. Sustainability was selected as a common theme for the KBIP program for 07-08, under a fancy nameWildspace. TH and JQ (a science teacher in an Anglophone middle school in Quebec) agreed to engage their students in collaborative KB on water ecology. They conducted their planning through email and their plan as of December 07 is summarized in Box 2. Box 2. A summary of the Wildspace KB curriculum plan by TH and JQ. Phase 1: Field studies <Dec 2007-Feb 2008> (tbc* - to be confirmed) 1. Students' field work in January 2008 to (i) explore -what lives in the areas (understanding of life cycle and growth of specific plants and animals) - the relationship among plants, animals, humans and the environment. (ii) Prepare short description of the site with photographs ready to be shared. 2. Initial exchange between teachers by Skype of their chosen local areas, through photos and text. Phase 2: Discussion on KF < Feb 2008?> (tbc) Pupils work on perceived threats to their sites using KF Phase 3: international collaboration < Mar April 2008?> (tbc) Exchange of ideas and rise above notes between schools on shared view in KF. Phase 4: Video-conference <25-26 April, 2008> (tbc) Video-conference among all concerned (possibly camping at school for HK students because of timezone) Key questions to be investigated 1. What are the threats to the area? / How human activities impact the area? Investigate through experiments, field trip, KB talk, KF discourse on the following: a. What lives in the area? (Understanding of life cycle and growth of specific plants and animals) i. Construct a Food web of the organisms in the area incorporating observations of organisms in water samples collected in field trip under the microscope. ii. How do organisms (animals and plants) adapt to the natural habitat? Choose an animal in the area and study its special features/characteristics for adaptation in terms of food and habitat. iii. How do the habitats and living organisms in one site differ from the other? (Let students in the two schools read each others notes on this question.) iv. Why are oceans filled with salt water? Is the salt in the sea the same as table salt? What is salinity? Can animals from the sea live in fresh water? (Conduct related experiments.) b. How are the organisms affected by their environmental conditions? Conditions to include: i. Water quality: salinity, water temperature, visibility/turbidity, dissolved oxygen, pH, ii. Air Temperature, wind direction, wind speed. iii. What relationships exist among the living organisms in the area and the above conditions? c. What are the threats to the area? How do human activities impact the area (e.g. urbanization, pollutions, garbage, oil pill, climate change, chemicals, acid rain, etc.) 2. How do the threats to the area affect the organisms in the area? To investigate this problem, each student will choose an animal living in the area and: a. Research what each animal needs in terms of food and habitat, b. Research how the threats affect the habitat and food supply for the selected animal. There will be KB talks throughout, and a microscope available for examining microscopic life in water. TH and JQs joint KB curriculum plan was organized around water problems. The plan spanned over a period of five months and they coordinated the teaching schedule to include joint discussions on KF. The teaching in both sites were kicked off with a field trip: marine national park for Hong Kong students and fresh water pond for Quebec students. There were four phases of student activities planned, but no details were decided to allow for maximum flexibility. Much of the planning document was concerned with identifying the authentic problems for the students to explore. TH had a clear idea of the key concepts in the two grade 7 science curriculum units, water and living things, that can be learnt through the planned inquiry. However, unlike her plan in the previous year which contained the specific curriculum goals in the energy unit that she want to accomplish,, this plan did not refer to any curriculum content goals. The most important learning goal, as she explained in an interview held at the end of May 2008, is learning to learnfostering students ability to engage in inquiry-based learning. There were frequent interactions between TH and JQ. Their close liaison and collegial collaboration made it possible for the implementation to be so emergent and successful.
ISLS
215
ICLS2012
From introducing online discussions to orchestrating knowledge building

As is evident from the above descriptions, THs planning behavior changed significantly over the course of the three years. She mentioned in her interview that her focus in the first year was on achieving the prescribed curriculum objectives and activities in the textbook. Online discussion was included as an add-on activity to elicit students misconceptions so that she could more effectively teach them the correct concepts. In her second year, she began to understand that KB is not just getting students to do online discussions. While achieving the set curriculum content goals was still of high priority, her planning was very much guided by her understanding of the students major conceptual difficulties, such as confusing energy with fuel. The focus of the planning was on students discourse, and the seed questions she placed in KF were carefully thought through. Each of the three levels of discussion tasks in phase 2 was accompanied by carefully crafted learning activities, and there was continuity between the phase 1 and phase 2 discussion tasks. The KBIP activity during this year only involve a joint video-conference for sharing, but did not involve joint inquiry among students. In the third year, TH became even more KB-focused in her planning. Her attention was no longer focused on the curriculum content goals or the learning activities to be conducted, but how to foster students ability to engage in productive knowledge building. While she was still organizing activities, these served to provide the context for introducing and exploring real-life, authentic problems that do not have a ready answer. The questions she listed in the plan were no longer posted on KF as requiring responses from her as the teacher, but served as a reminder to herself about the kind of questions that could be raised with the students during class discussions. She did not post any seed question for conceptual clarification, but gave to students the cognitive responsibility to identify their own problems of understanding during their explorations of the authentic problem context. There was deep genuine collaboration between TH and JQ as they co-constructed a model of KBIP involving collaborative KB among the teachers as well as students. The changes in the curriculum plans over the three years reveal deep changes in THs understanding of KB and KB pedagogy. As she mentioned in her interviews, her participation in LCP as a seconded teacher from September 2006 gave her unprecedented opportunity to learn about KB and to reflect on her own teaching. (Seconded teachers in the LCP project are relieved from half of their teaching load to participate in weekly project team meetings and visit the classrooms of other teachers in the project to contribution to the scaling up of KB as a pedagogical innovation in Hong Kong schools.) Her understanding of KB changed from an online discussion serving as an extension of classroom learning to a way of deepening students understanding of difficult concepts through discussions around well selected questions, to genuine engagement in inquiry on authentic problems. THs deepening understanding of KB was accompanied by a widening repertoire of KB pedagogical strategies and skills. One important element in KB pedagogy is in helping students to understand what is inquiry and to be able to ask questions that contribute to deepening understanding (Hakkerainen, 2003). In the first year, she was not aware of the difference between textbook questions and questions that identify problems of understanding, which serve as the starting point in KB, and simply asked students to use end-ofchapter questions for their online discussion. In the second year, she posed paradoxical questions that help students clarify common misconceptions, but only attracted the participation of a small group of students. In her third year, she really opened up the space for exploration to the students. She used the field trip and various laboratory-based experiments and online simulations as stimulus and resources for the students to raise their own questions of inquiry and to pursue them. She also guided the students to identify which of the questions they raised were good questions that would lead to further inquiry and improvement of ideas, and the criteria for questions to be good inquiry questions. The prescribed curriculum no longer restricted the scope of her students inquiry. She was confident that if the students were able to identify inquiry questions they were genuinely interested in, they should be able to master the content knowledge associated with the activities from the curriculum that she provided to the students as opportunities for them to learn the knowledge needed to answer the students own authentic questions about the sustainability of the water ecology they were studying.
Changes in Students Discourse Behavior and Quality

Is there any evidence that the changes in THs KB pedagogical designs over the three years actually brought about differences in the students online behavior in KF? Can we see different extents of KB advancement in the students over those three years? In this section, we report on our investigation of the students online discourse behavior over the same period, using both quantitative and qualitative indicators commonly used in the KB literature, in order to validate our claim that TH has made significant advances in KB pedagogy. About 480 notes were found in THs students 05-06 discourse database, which is not too small a number. However, the quality of the notes was of a very low quality in terms of KB. While TH used roller coaster rides in Disneyland as the context to stimulate student discussions on energy transformation, nearly all of the students used KF as a platform for casual chats about the theme park. The following are typical excerpts: I like Disneyland in Japan but I don't like HK Disneyland. You think HK Disney is big? I think Disney land bigger than Ocean Park. Do you like Mickeymouse?
ISLS 216
ICLS2012
There were a handful of notes that showed more serious efforts to construct a learning discourse: New Idea The fireworks are very beautiful. I need to understand Is it will make the air pollute? Why? I need to understand If there is some accident [on the roller coaster], how can we evacuate fast? Unfortunately, none of these postings were followed up by any of the students for further discussion. Since there was no evidence that THs students were able to engage in a meaningful KB discourse on KF, our investigation of the students online discourse was confined to the later two school years, 06-07 and 07-08.
Indicators of student participation and interaction characteristics

Discourse participation statistics have conventionally been used as rudimentary indicators of students KB engagement (Burtis, 2002). The descriptive statistical in Table 2 indicates that students in the 07-08 cohort wrote on average, had a lot fewer singletons (notes that were not followed up) and had longer thread depths, all of which indicates a deeper level of KB engagement compared to the 06-07 cohort. Table 2. The total number of notes written per student and distribution of notes per thread lengths. # of notes per student Total # of Class Notes 06-07 542 Min 1 Max 56 Mean 6.7 # of notes in thread depth = T.Depth=3 88 T.Depth>=4 188
Median Singleton T.Depth=2 4 162 104
07-08 471 1 62 9.2 4 68 65 57 281 Social network analysis (SNA, Faust & Wasserman, 1994) is often used by researchers to reveal the social interaction pattern of participants in a forum discourse. More homogenous and dense connections among authors are interpreted as signs of more distributed, equitable and intense interactions among the authors, and hence more likely to bring about productive KB interactions that align with the KB principles. The SNA display in Figure 1 shows relatively little interaction between THs students (red dots) and another KBIP class (blue dots) on KF in 06-07 (left-hand side figure), while there was much denser interaction among the two classes of students in 07-08. The interactions among THs and JQs students through the KBIP program was so intensive that the two classes of students cannot be distinguished from the single cluster in the right-hand side figure.
Figure 1. Social Interaction Patterns of students build-on relationships in 06-07 (left) and 07-08 (right)
Students Best Knowledge Advancement as Perceived by the Teacher

While the non-semantic indicators do show reasonable student engagement in the online discourse for each of the two school years investigated, the semantic content of the notes needs to be explored to identify whether there was evidence of knowledge advancement in the process. We are not able to provide here a full semantic analysis of the students discourse, but will examine the students good notes selected by TH to gain some insight into their content and quality. For each of the two school years, TH selected what she considered to be the most thoughtful notes that indicated advances in understanding from the students online discourse. In 06-07, at the end of phase 1, TH selected four of the students notes to post in the thoughtful notes view to stimulate further discussion. Box 3 contains their note titles and content excerpts. These four notes already exhibited some rather nuanced understanding of energy, fuel, energy crisis and the principle of energy conservation. They were written by four different students, and remained as singletons. It is not clear how these students came to such sophisticated understanding of the scientific concepts at such an early phase of the discussion. TH kicked off the phase 2 discussions by writing a seed note to highlight the paradox she introduced in phase 1 by making reference to some students notes (other than the thoughtful notes). A few students responded, trying to make sense of the apparent contradiction. The key concepts and ideas discussed in phase 2 did not go beyond those listed in Box 3, and many of the notes were not really on task but social in nature.
ISLS
217
ICLS2012
Box 3. The titles and selected content of the four thoughtful notes selected by TH at the end of phase 1 Note 1: There is the principle of energy conservation in Physics. So why is there now a shortage of energy in the world? [2007, Jan 09] First, please do NOT be cheated by the term energy shortage... Energy shortage just means shortage of petroleum and coal. It does NOT mean the TOTAL energy in ALL forms on earth is decreasing. . However, human being rely too much on non-renewable energy like petroleum and coal. Once they are burnt into other form of energy, we can NOT obtain them back in reversed way. ... Note 2: Energy crisis [2007, Jan 09] Observation: I notice: energy crisis just means shortage of petroleum and coal. human being rely too much on non-renewable energy like petroleum and coal. Once they are burnt into other form of energy, we can NOT obtain them back in reversed way. they are non-renewable, Then eventually, ALL of them will be consumed in the future... This is known as energy crisis... . To conclude, the term energy crisis is just for our convenience. It has nothing to do with the physical law conservation of energy... Note 3: What is the difference between fuel and Energy? [2007, Jan 21] . Fuel is a kind of substance that when it is burnt it releases energy while energy has many different forms like kinetic, potential, and it can be transferred from one form to another form. Note 4: Fuel [2007, Jan 23] Fuel releases its energy either through chemical means, such as burning, or nuclear means, such as nuclear fission or nuclear fusion. An important property of a useful fuel is that its energy can be stored to be released only when needed, and that the release is controlled that the energy can be harnessed to produce work. In 07-08, TH did not ask the students to start the online discussion until after the field trip to the marine national park and the students had an opportunity to discuss their observations and questions in class. This time, TH did not showcase specific notes in a special view on the KF platform, but identified 33 notes from the students discourse on KF as good notes at the end of school year. the The selected notes can be categorized into three groups. The first group was from notes contributed by students during the four weeks after the field trip, which raised and explored significant questions such as: What can we do [to] stop polluting the Earth? How can we rescue them [animals] from extinction? Why there is a hole in [the ozone layer of the] atmosphere? How the carbon dioxide cause animal extinct? How planktons give birth to their babies? These discussions sparked off in a wide variety of discussion topics including: corals, global warming, water pollution, animal extinction, air pollution, energy conservation, and environmental protection. Some of the discussions revealed notable clarifications of misconceptions by the students, as illustrated by the following excerpts from a thread that debated about whether carbon dioxide is toxic and harmful: Student A: Carbon Dioxide is a kind of toxic gas. It is created by the breath of animals (plants), and will be distilled by photosynthesis by plants. Student B: I don't think carbon dioxide is a poisonous gas because the trees will intake the carbon dioxide during the photosynthesis. Student C: human being cannot breath pure oxygen, and air contains, oxygen, nitrogen, carbon dioxide... The third group of selected notes was from the period when THs and JQs students shared ideas and discussed online. They raised questions of understanding after reading the summary notes from students in the other country. For example, when the Hong Kong students read about the problem of sedimentation in the fresh water pond in Quebec, they asked for the meaning of sedimentation, and discussed whether sedimentation is also a problem in marine sites. The Quebec students asked Hong Kong students to explain how fish can eat planktons if these are too small to be seen without a microscope, and what do planktons eat. Unlike in the previous years, students initiated and sustained discussion topics that were personally meaningful and focused around the theme of sustainability of water and the biological world. They identified and tackled problems of understanding on concepts of scientific importance such as whether carbon dioxide is toxic or if animals can eat things that they cannot seemisconceptions that are not found in textbooks and often escape the attention of teachers.
Discussion
Whether there is KB happening in a classroom cannot be easily determined by the activities that takes place. Scardamalia (2002) explains that a KB classroom should be idea-centered rather than activity-centered, and the teacher should turn the cognitive responsibility of determining what and how to learn back to learners. She further argues that the change from an activity-centered view of education to an idea-centered one is like the Copernican Revolution in that you either get it or you dont. However, for a teacher to change from being a
ISLS 218
ICLS2012
novice to KB to the stage when s/he can induct and guide students to knowledge build is to cross a great canyon. How do teachers make that change? Are there certain implementation pathways (Bielaczyc, 2006) that teachers would need to go through to make that transition? Would there be different pathways for different individual and contextual backgrounds? The above questions cannot be answered by this study as we have only studied one teacher. However, it was a deeply exhilarating learning experience for us as researchers to witness the strides made by TH. Most teachers in THs position would have given up after the first year, as the students were apparently not able to engage in meaningful, on-task discussions. In an interview in October 2007, TH said she was disappointed with the students discussions on KF. She found KB to be really difficult. Why did she still believe that students could knowledge build? She confessed that she would not have persisted if she had not seen other teachers achieve it, and her participation as a seconded teacher was instrumental to her professional development. We have only scratched the surface of one teachers learning trajectory. Obviously, there had to be deep changes in terms of pedagogical belief and pedagogical conceptualization for TH to have made the jump. These need to be investigated. We also need to further investigate what were the critical factors/events that contributed to the changes observed. Teacher learning has to go beyond concepts and beliefs as teaching involves knowledge and skills in execution and performance. In her third year, TH did a lot of preparation in terms of organizing the field trips and classroom discussions. On the other hand, her way of handling the classroom was remarkably fluid. There was artful improvisation (Sawyer, 2011) in how she facilitated the classroom KB talks and adjusted planned activities in response to students interests. How did she learn to change from dutifully following textbook activities to becoming a creative improviser? This study opens up for us a goldmine of questions. We hope that it will also stimulate similar studies that would contribute to a better understanding of teacher learning for KB and for technology-supported pedagogical innovations more generally.
References
Barab, S. A. (Ed.). (2004). Design-based research: Clarifying the terms [Special issue]. Journal of the Learning Sciences, 13, 1128. Bereiter, C. & Scardamalia, M. (1989). Intentional learning as a goal of Instruction. In Resnick, L.B. (Ed.), Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 361-392). Hillsdale, NJ: Erlbaum. Bielaczyc, K. (2006). Designing social infrastructure: Critical issues in creating learning environments with technology. Journal of the Learning Sciences, 15(3), 301-329. Burtis, J. (2002). Analytic Toolkit for Knowledge Forum. Toronto, Ontario, Canada: Institute for Knowledge Innovation and Technology, University of Toronto. Chai, C. S., & Merry, R. (2006). Teachers perceptions of teaching and learning in a knowledge-building community: an exploratory case study. Learning, Media and Technology, 31(2), 133-148. Chan, C., & van Aalst, J. (2006). Teacher development through computer-supported knowledge building: Experience from Hong Kong and Canadian teachers. Teaching Education, 17(1), 7-26. Faust, K., & Wasserman, S. (1994). Social network analysis: Methods and applications. New York: Cambridge University Press. Hakkarainen, K. (2003). Progressive inquiry in a computer-supported biology class. Journal of Research in Science Teaching, 40(10), 1072-1088. Laferriere, T., & Law, N. (2010). Knowledge building international project (KBIP): a nested network of learning and knowledge creation. Proceedings of the 9th International Conference of the Learning Sciences (pp. 103-104). Laferrire, T. et al. (2010). Partnerships for Knowledge Building: An Emerging Model. Canadian Journal of Learning and Technology/La revue canadienne de lapprentissage et de la technologie, 36(1). Lakkala, M., Lallimo, J., & Hakkarainen, K. (2005). Teachers' pedagogical designs for technology-supported collective inquiry: A national case study. Computers & Education, 45(3), 337-356. Law, N., & Wong, E. (2003). Developmental Trajectory in Knowledge Building: An Investigation. In B. Wasson, S. Ludvigsen & Hoppe, U. (Ed.), Designing for Change in Networked Learning Environments (pp. 57-66). Dordrecht: Kluwer Academic Publishers Law, N., Yuen, J., Wong, W., & Leng, J. (2011). Understanding learners knowledge building trajectory through visualizations of multiple automated analyses. In S. Puntambekar, G. Erkens, & C. E. HmeloSilver (Eds.), Analyzing interactions in CSCL: Methods, issues and approaches. NY: Springer. Sawyer, R.K. (Ed.) (2011). Structure and Improvisation in Creative Teaching. NY: Cambridge University Press. Scardamalia, M. (2002). Collective Cognitive Responsibility for the Advancement of Knowledge. In B. Smith (Ed.), Liberal Education in a Knowledge Society (pp. 67-98). Chicago: Open Court. Scardamalia, M., & Bereiter, C. (1999). Schools as Knowledge Building Organizations. In D. H. Keating, C. (Ed.), Today's Children, Tomorrow's Society: the developmental health and wealth of nations (pp. 274289). New York: Guilford.
ISLS
219
ICLS2012
Is Computer Support More Significant than Collaboration in Promoting Self-Efficacy and Transfer?
Andreas Gegenfurtner, TUM School of Education, TU M nchen, Germany, andreas.gegenfurtner@tum.de Marja Vauras, Centre for Learning Research, University of Turku, Finland, vauras@utu.fi Koen Veermans, Centre for Learning Research, University of Turku, Finland, koevee@utu.fi Abstract: In the learning sciences, we assume that deep learning is more likely to occur in complex social and technological environments. Deep learning, in turn, is related to higher degrees of transfer and self-efficacy. It follows that population correlation estimates of the relationship between self-efficacy and training transfer should be higher in those conditions that afford computer support and collaboration. This meta-analysis (29 studies, k = 33, N = 4,158) tested this assumption. Based on social cognitive theory, results suggested positive population correlation estimates between self-efficacy and transfer ( = 0.39). Results also showed that effect sizes were higher in trainings with rather than without computer support, and higher in trainings without rather than with collaboration. These findings are discussed in terms of their implications for theories of complex social and computer-mediated learning environments and their practical significance for scaffolding technology-enhanced learning and interaction.
Introduction
In the learning sciences, we assume that deep learning is more likely to occur in complex social and technological environments (Sawyer, 2006). Deep learning, in turn, is related to higher degrees of transfer (Pugh & Bergin, 2006) and self-efficacy (Bandura, 1997). It follows that estimates of the relationship between selfefficacy and transfer of training should be higher in those conditions that afford computer supported collaborative learning (CSCL). To test the predictive validity of this assumption, the present meta-analysis sets out to investigate whether higher population correlation estimates between self-efficacy and transfer are found in training conditions that afford computer support and collaboration compared with other training conditions.
Self-Efficacy and Transfer of Training

Self-efficacy refers to beliefs in ones capabilities to organize and execute the courses of action required to produce given attainments (Bandura, 1997, p. 3). Efficacy beliefs are among the most widely documented predictors of achievement, which has been documented in domains including sports (Trost, Own, Bauman, Sallis, & Brown, 2002), work (Stajkovich & Luthans, 1998), and education (Nelson & Ketelhut, 2008; Van Dinther, Dochy, & Segers, 2011). According to social cognitive theory (Bandura, 1997), people with high selfefficacy set high and demanding goals; these goals create negative performance discrepancies to be mastered (Bandura, 2001). Expectations about the perceived efficacy of ones capability to master those discrepancies regulate whether effort is initiated, how much continuous effort is expended, and whether effort is maintained or even increased in face of difficulties during goal attainment. Because the power of self-efficacy to predict task achievement has been so widely documented (Bandura, 1997; Bandura, 2001; Efklides, 2011; Gegenfurtner, 2011; Gegenfurtner, Festner, Gallenberger, Lehtinen, & Gruber, 2009; Gegenfurtner & Vauras, 2012; Gegenfurtner, Vauras, Gruber, & Festner, 2010; Gegenfurtner, Veermans, Festner, & Gruber, 2009; Junttila, Vauras, & Laakkonen, 2007; Sadri & Robertson, 1993; Trost et al., 2002; Van Dinther et al., 2011; Zimmerman, 2000), it seems reasonable to assume that self-efficacy also predicts the initiation, expenditure, and maintenance of efforts toward transfer of training. Transfer of training can be defined as the productive use of newly acquired knowledge and skills (De Corte, 2003). If it is true that efficacy beliefs predict sufficient execution of effort to achieve successful outcomes, it follows that efficacy beliefs should also predict successful transfer of training. However, the literature shows mixed evidence. For example, some investigations showed high correlation estimates between self-efficacy and training transfer (Yi & Davis, 2003), while other investigations suggested that the magnitude of this relationship is negligible (Brown & Warren, 2009). One possible explanation for the mixed evidence is the influence of sampling error and error of measurement (Hunter & Schmidt, 2004) that may have induced biases on the true score population correlation. Therefore, one aim of the present study was to use meta-analytic methods to inquire whether performance self-efficacy, after controlling for sampling error and error of measurement, exhibits a stable influence on transfer and whether this relationship would be higher after training than before training. Another possible explanation for the mixed evidence is that population correlation estimates have been moderated by different study conditions. As Kanfer (2005) argued, identification of boundary conditions has important implications for testing the predictive validity of social cognitive theory. Therefore, a second aim of the study was to identify and estimate the boundary conditions under which self-
ISLS
220
ICLS2012
efficacy and training transfer correlate. Inquiring into these characteristics as boundary conditions is significant, because it enables accounting for artifactual variance in the total variance of a correlation, which, in turn, may explain some of the disagreements in the existing literature. Two boundary conditions were analyzed: computer support and collaboration.
Computer-Supported Collaborative Learning

The rationale for choosing CSCL as boundary conditions was derived from a belief in the learning sciences that deep learning is more likely in complex social and technological environments (Sawyer, 2006, p. 13). Deep learning, in turn, is related to higher degrees of transfer (Pugh & Bergin, 2006) and self-efficacy (Bandura, 2001). If these assumptions hold, it follows that population correlation estimates of the relationship between self-efficacy and training transfer should be higher in those conditions that afford CSCL. Conditions for CSCL can be examined as (1) computer support and (2) collaboration. For the purpose of this article, computer support was defined as technological material in learning environments intended to promote understanding (Lehtinen, Hakkarainen, Lipponen, Rahikainen, & Muukkonen, 1999); collaboration was broadly defined as the workingtogether of two or more individuals to attain the shared training goals and task at hand (Dillenbourg, 1999). We acknowledge substantial variation in how the term collaboration is defined in the literature (for an overview, see Dillenbourg, 1999; Volet, Vauras, & Salonen, 2009). We use the term collaboration here for the sake of simplicity and do acknowledge gradual nuances in how key conditions of the nature of joint working (e.g., shared goals, co-construction of knowledge, co-regulation, etc.) are reflected in prior literature to capture different sociopsychological processes of interpersonal coordination and their relation to actualizing motivation in training situations (Vauras, Salonen, & Kinnunen, 2008). An example of training including both computer support and collaboration is reported by Day and colleagues (2007), who trained participants with a collaborative computer game. An example of a training including computer support but no collaboration is Yi and Daviss (2003) study, in which they trained participants to use computer software individually and without interaction among trainees. An example of a training including no computer support but collaboration is Gibsons (2001) description of nursing team training, which included group discussions, brainstorming, and peer assessment. Finally, an example of a training program including neither computer support nor collaboration was a study by Karl, OLeary-Kelly, and Martocchio (1993), in which participants were trained in a speedreading skill individually with paper handouts. If it is true that complex social and technological environments promote self-efficacy and transfer (Bandura, 1997; Pugh & Bergin, 2006; Sawyer, 2006), then it follows that population correlation estimates should be higher in conditions with computer support rather than in conditions without and in conditions with collaboration rather than without. Importantly, population correlation estimates should be highest in conditions affording both computer support and collaboration. Figure 1 illustrates these four conditions. The top-left quadrant represents training conditions that neither afford computer support nor collaboration; it is thus assumed to have no particular positive effects. The bottom-left quadrant represents training conditions that afford computer support but no collaboration. The top-right quadrant represents training conditions that afford collaboration, but no computer support. Finally, the bottom-right quadrant represents training conditions that afford both computer support and collaboration. Conditions of CSCL here are understood from a systemic perspective (Arnseth & Ludvigsen, 2006). Examination of computer support and collaboration as boundary conditions can contribute to our understanding of how reshaping traditional training by technological media is related to trainee motivation and transfer on the job (Tynj & H l kkinen, 2005).
Does the training design afford collaboration among trainees?
No Yes
Does the training design afford computer support?
No
Yes
++
Figure 1: Hypothesized effects of study conditions on the relationship between self-efficacy and transfer.
ISLS
221
ICLS2012
The Present StudyHypotheses

In summary, the focus of the present study was the relationship between performance self-efficacy and transfer of training. The first aim was to cumulate previous research in order to correct the size of true score population correlations. A second aim of the study was to estimate the moderating effects of computer support and collaboration. Two hypotheses were formulated. Based on social cognitive theory (Bandura, 1997), we assumed that transfer of training would be positively related with performance self-efficacy (Hypothesis 1). Based on the assumption that deep learning is more likely to occur in complex social and technological environments (Sawyer, 2006), we hypothesized that the relationship between self-efficacy and transfer would be more positive in training conditions affording computer support and collaboration (Hypothesis 2).
Method
Literature Searches and Criteria for Inclusion
To test these hypotheses, we used meta-analytic methods (Hunter & Schmidt, 2004). Studies that reported correlations between performance self-efficacy and transfer of training were located. To be included in the database, a study had to report an effect size r or other effect sizes that could be converted to r ( coefficient; Cohens d; F, t, or Z statistics). Formulae reported in Peterson and Brown (2005; formula c) and Rosenthal and DiMatteo (2001) were used to convert effect sizes. Because the focus of inquiry was on self-efficacy as an individual capacity (Bandura, 1997), the database included studies that reported data on individuals. Studies reporting data on group efficacy were omitted. Studies on children as well as animal studies were also excluded, because they represent different premises on training and work performance. Using these inclusion criteria, the literature was searched in three ways. First, the PsycINFO, ERIC, and Web of Science databases were searched using the keywords self-efficacy, behavior change, training application, training use, and transfer of training. In addition, a manual search of journal issues covering a 25-year period (from January 1986 through December 2010) was conducted. Finally, studies that cited Noes (1986) seminal paper, which was the first to model the influence of motivation on transfer in professional training, were searched with the Web of Science and the Google Scholar databases. A total of 29 articles, book chapters, conference papers, and dissertations that contributed at least one effect size to the meta-analysis were included in the database. A full list of all included studies is available from the first author. The 29 studies offered a total of k = 33 independent data sources. Total sample size was N = 4,158 participants.
Recorded Variables
To answer the study hypotheses, different characteristics were tabulated from the selected research literature. Specifically, each study was coded for effect size estimates, computer support, and collaboration. Effect size estimates included Pearson product-moment correlation r of the self-efficacytransfer relationship, Cronbachs reliability estimate of the independent variables (self-efficacy), and Cronbachs reliability estimate of the dependent variable (transfer). We also coded the first author, publication year, the number of participants, their age (in years), and gender (percentage of females). Computer support for learning afforded during training was coded as 1 = computer support and 0 = no computer support. Collaboration among participants afforded during training was coded as 1 = collaboration and 0 = no collaboration. Two independent raters first coded fifteen of the studies. Because intercoder reliability was generally high (Cohens = .91), one rater continued to code the remaining studies. If a study reported more than one effect size, a single composite variable was created to comply with the assumption of independence. As an exception to this rule, linear composites were not created for the theoretically predicted moderator variables, as composite correlations would have obscured moderator effects and prohibited further analysis.
Meta-Analytic Methods Used

Analysis occurred in two stages. A primary meta-analysis aimed to estimate the true score population correlation of the pre- and post-training relationship between performance self-efficacy and transfer of training. A meta-analytic moderator estimation then aimed to identify moderating effects in those relationships. The primary meta-analysis was done using the methods of artifact distribution meta-analysis of correlations (Hunter & Schmidt, 2004). These methods provide an improvement from earlier statistical formulae when information such as reliability estimates is only sporadically reported in the original studies. First, study information was compiled on three distributions: the distribution of the observed Pearsons r of the transfer self-efficacy relationship, the distribution of Cronbachs of the independent variable, and the distribution of Cronbachs of the dependent variable. Next, the distribution of Pearsons r was corrected for sampling error. Note that the correction was conducted using a weighted average, not Fishers z transformation, since the latter was shown to produce upwardly biased correlation estimates (Hall & Brannick, 2002). The distribution corrected for sampling error was then further corrected for error of measurement using the compiled Cronbachs reliability estimates. This last step provided the final estimate of the true score population correlations
ISLS 222
ICLS2012
between self-efficacy and transfer. Finally, standard deviations of the corrected observed correlation rc and of the population correlation were calculated; these were used to derive the percentage of variance attributable to attenuating effects, the 95% confidence interval around rc, and the 80% credibility interval around . The meta-analytic moderator estimation followed the primary meta-analysis. Theory-driven nested sub-group analyses were used to estimate the moderating effects of computer support and collaboration. Nested sub-group analysis assumes that the moderator variables are independent and additive in their effects. A criticism of the use of sub-groups is that it reduces the number of data sources per analysis, resulting in secondorder sampling error. Although the present study contained a large number of data sources and participants, the possibility of second-order sampling error cannot be completely ruled out. This is therefore indicated when warranted for interpreting the results.
Results
Primary Meta-Analysis
Table 1 summarizes the number of studies, participants, and participant characteristics by condition. The mean estimates in Table 1 are age in years and percentage of females. Across all conditions, the uncorrected correlation coefficient r between performance self-efficacy and transfer of training is 0.34 (k = 33, N = 4,158). The population correlation estimate corrected for sampling error and error of measurement is = 0.39 (SD = 0.23; 80% CV = .10; .68). This estimate is in the positive direction, thus supporting Hypothesis 1. The difference between r and represents a depression of the true score population correlation through sampling error and error of measurement by 14.7%. Table 1: Number of studies, participants, and participant characteristics by condition. Age M 27.28 26.42 31.51 20.70 Gender M 39.60 61.43 50.88 21.87
Conditions Computer support and collaboration Computer support, but no collaboration No computer support, but collaboration No computer support and no collaboration
k 7 7 17 2
N 730 1,044 2,172 257
SD 4.55 9.90 10.11 0.99
SD 27.48 17.90 15.38 29.89
Meta-Analytic Moderator Estimation

The specific hypothesis was that the relationship between self-efficacy and transfer is moderated by computer support and collaboration. Four conditions were evaluated: trainings with computer support and collaboration (condition 1), trainings with computer support but no collaboration (condition 2), trainings with collaboration but no computer support (condition 3), and trainings with neither computer support nor collaboration (condition 4). There were no systematic age [2 (3,25) = 4.17, ns] or gender [2 (3,21) = 2.77, ns] differences between conditions (computer support, no computer support, collaboration, no collaboration). A nested sub-group analysis of computer support and collaboration as confounding moderator variables signaled two trends. First, computer support and collaboration were highly correlated, with Spearmans = .44 (95% CI = .43; .46). Second, effect sizes were highest when the training was computer-supported. Third, effect sizes were twice as high in computer-support trainings without collaboration (condition 2) compared to computer-supported trainings with collaboration (condition 1). Table 2 summarizes the results. However, unequal sample sizes and small cell sizes for some of the training conditions warrant caution when interpreting these results. Table 2: Nested moderator effects of computer support and collaboration. Conditions Computer support and collaboration Computer support, but no collaboration No computer support, but collaboration No computer support and no collaboration
0.31 0.62 0.30 0.25
SD 0.03 0.07 0.04 0.01
80% CV 0.27; 0.35 0.53; 0.71 0.25; 0.35 0.24; 0.26
Discussion
One aim of this meta-analysis was to cumulate the research of the past 25 years to correct the relationship between performance self-efficacy and transfer of training for sampling error and error of measurement. A second aim was to estimate the moderating effects of computer support and collaboration. The heterogeneity and disagreement in the training literature ultimately led this study to seek a better understanding of whether, to what extent, and under which conditions efficacy beliefs influenced transfer.
ISLS 223
ICLS2012
The results of the primary meta-analyses suggested positive relationships between performance selfefficacy and training transfer. These estimates provide support for Hypothesis 1 and are in line with previous meta-analytic examinations (Colquitt, LePine, & Noe, 2000; Gegenfurtner, 2011). These findings empirically support the theoretical assumption that efficacy beliefs influence transfer (Bandura, 1997; Pugh & Bergin, 2006), and are consistent with earlier conceptual frameworks in the training literature, such as the integrative theory of training effectiveness (Colquitt et al., 2000) or the integrative model of motivation to transfer training (Gegenfurtner, Veermans, Festner, & Gruber, 2009). The results of the meta-analytic moderator estimation suggested systemic effects of computer support and collaboration (Arnseth & Ludvigsen, 2006). Specifically, computer-supported collaborative learning does not per se promote the relationship between self-efficacy and transfer. The results showed that computer support played a more significant role than collaboration among trainees. Trainings affording CSCL were not generally more effective in promoting efficacy beliefs and transfer than trainings not affording CSCL (see estimates in Table 2). One possible explanation for this unexpected finding may be the form of collaboration (Dillenbourg, 1999; Lehtinen et al., 1999) in the individual study reports. We had no information on how social interaction emerged in the training situations, as the primary studies reported correlation estimates only but did not engage in analyzing interaction with methods currently available (Puntambekar, Erkens, & Hmelo-Silver, 2011). Nor had we information on the degree that the collaborative learning situations were scaffolded or scripted in the original studies. Without sufficient guidance and scaffolding of collaboration activities among training participants, efforts toward collaboration may result in unequal or heterogeneous participation (Barab & Duffy, 2000; Weinberger & Fischer, 2006), non-reciprocal interpretations of the learning situation (J rvel 1995; , Veermans, 2003), and/or lack of co-regulation (Vauras et al., 2008; Volet et al., 2009). Future research may take this meta-analytic evidence to test designs for scaffolding collaboration in technology-rich environments intended to promote self-efficacy and transfer. In summary, analysis of the moderating effects of computer support and collaboration illustrate boundary conditions for self-efficacy and transfer in professional training. Results of this study may have some practical value for the scaffolding of collaborative learning. Specifically, the low confounded moderator effect of computer support and collaboration tends to highlight the danger of ignoring adequate guidance and scaffolding of participatory interactions among trainees in the learning environment. This finding is reported in the literature elsewhere (e.g., J rvel 1995; Weinberger & , Fischer, 2006) and is now reiterated with a special emphasis of how important scaffolded collaboration is for promoting self-efficacy and transfer (Latham, 2007; Pugh & Bergin, 2006). It can be speculated that the provision of networks can scaffold trainees during training and that post-intervention enhancement of contacts among trainees could facilitate transfer after training. Methodologically, it would be interesting to follow these educational interventions with different methods, including social network analysis, to trace how they influence processes of sharing and co-construction during collaborative team learning and how they promote transfer to typically practice-bound situations at work (Gegenfurtner, 2011; S , 2010; Tynj & H lj l kkinen, 2005). This study has some limitations that should be noted. One limitation is that the population correlation estimates were corrected for sampling error and error of measurement. This decision was based on the frequent reporting and availability of sample size and reliability information. However, the original research reports may be affected by additional biases, such as extraneous factors introduced by study procedure (Hunter & Schmidt, 2004). Although the estimation of moderators sought to lessen this bias, the true population estimates may be somewhat greater than those reported here. An additional limitation is that some of the relationships in the nested subgroups were based on small sample sizes. However, some authors have noted that correcting for bias at a small scale mitigates sampling error compared to uncorrected estimates in individual studies (Hunter & Schmidt 2004; Rosenthal & DiMatteo, 2001). Still, although most of the cells contained sample sizes in the thousands, some did contain fewer, which indicates underestimation of sampling error in those few cases and that, therefore, computer-supported collaborative training may show more positive correlation estimates. Finally, the study reports two moderator variables. Although an analysis of the relationship between performance selfefficacy and transfer of training under different boundary conditions clearly goes beyond previous meta-analytic attempts, selection of the boundary conditions was, of course, eclectic and exclusively driven by an interest to better understand technological and social affordances in CSCL environments for motivation and transfer (Lehtinen et al., 1999; S , 2010; Sawyer, 2006). More conditions exist that would warrant inclusion in the lj meta-analysis and in turn raise concerns of the generalizability of the moderating effects. However, this limitation can be addressed only by additional original research reports that systematically vary different study conditions. As noted at the outset, self-efficacy and transfer were assumed to be more positive in in complex social and technological environments (Sawyer, 2006). The present meta-analytic study sought to test the predictive validity in an examination of the population correlation estimates between self-efficacy and transfer in computer-supported and collaborative training conditions. This examination was done by using meta-analysis to summarize 25 years of research on pre- and post-training self-efficacy, by cumulating 33 independent data sources from 4,158 participants, and by examining two confounded moderator variables (computer support and
ISLS
224
ICLS2012
collaboration) on the self-efficacytraining transfer relationship. The findings seem to imply that computer support is more significant for promoting self-efficacy and transfer than is collaboration. Future research is encouraged to extend these first steps reported here to the examination of social and technological conditions moderating self-efficacy and transfer in other educational and learning settings.
References
Arnseth, H. C., & Ludvigsen, S. (2006). Approaching institutional contexts: systemic versus dialogic research in CSCL. International Journal of Computer-Supported Collaborative Learning, 1, 167-185. Bandura A. (1997). Self-efficacy: The exercise of control. New York: Freeman. Bandura, A. (2001). Social cognitive theory: An agentic perspective. Annual Review of Psychology, 52, 1-26. Barab, S. A., & Duffy, T. M. (2000). From practice fields to communities of practice. In D. H. Jonassen & S. M. Land (Eds.), Theoretical foundations of learning environments (pp. 25-55). Mahwah, NJ: Erlbaum. Brosnan, M. J. (1998). The impact of computer anxiety and self-efficacy upon performance. Journal of Computer Assisted Learning, 14, 223-234. Brown, T. C., & Warren, A. M. (2009). Distal goal and proximal goal transfer of training interventions in an executive education program. Human Resource Development Quarterly, 20, 265-284. Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of training motivation: A metaanalytic path analysis of 20 years of research. Journal of Applied Psychology, 85, 678-707. Day, E. A., Boatman, P. R., Kowollik, V., Espejo, J., McEntire, L. E., & Sherwin, R. E. (2007). Collaborative training with a more experienced partner: Remediating low pretraining self-efficacy in complex skill acquisition. Human Factors, 49, 1132-1148. De Corte, E. (2003). Designing learning environments that foster the productive use of acquired knowledge and skills. In E. De Corte, L. Verschaffel, N. Entwistle, & J. van Merri nboer (Eds.), Powerful learning environments: Unraveling basic components and dimensions (pp. 21-33). Oxford: Elsevier. Dillenbourg, P. (1999). What do you mean by collaborative learning? In P. Dillenbourg (Ed.), Collaborative learning: Cognitive and computational approaches (pp. 1-19). Oxford: Elsevier. Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. Educational Psychologist, 46, 6-25. Gegenfurtner, A. (2011). Motivation and transfer in professional training: A meta-analysis of the moderating effects of knowledge type, instruction, and assessment conditions. Educational Research Review, 6, 153-168. Gegenfurtner, A., Festner, D., Gallenberger, W., Lehtinen, E., & Gruber, H. (2009). Predicting autonomous and controlled motivation to transfer training. International Journal of Training and Development, 13, 124138. Gegenfurtner, A., & Vauras, M. (2012). Age-related differences in the relation between motivation to learn and transfer of training in adult continuing education. Contemporary Educational Psychology, 37, 33-46. Gegenfurtner, A., Vauras, M., Gruber, H., & Festner, D. (2010). Motivation to transfer revisited. In K. Gomez, L. Lyons, & J. Radinsky (Eds.), Learning in the disciplines: ICLS2010 proceedings (Vol. 1, pp. 452459). Chicago, IL: International Society of the Learning Sciences. Gegenfurtner, A., Veermans, K., Festner, D., & Gruber, H. (2009). Motivation to transfer training: An integrative literature review. Human Resource Development Review, 8, 403-423. Gibson, C. B. (2001). Me and us: differential relationships among goal-setting training, efficacy and effectiveness at the individual and team level. Journal of Organizational Behavior, 22, 789-808. Hall, S. M., & Brannick, M. T. (2002). Comparison of two random effects methods of meta-analysis. Journal of Applied Psychology, 87, 377-389. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks: Sage. J rvel S. (1995). The cognitive apprenticeship model in a technologically rich learning environment: , Interpreting the learning interaction. Learning and Instruction, 5, 237-259. Junttila, N., Vauras, M., & Laakkonen, E. (2007). The role of parenting self-efficacy in children's social and academic behavior. European Journal of Psychology of Education, 22, 41-61. Kanfer, R. (2005). Self-regulation research in work and I7O psychology. Applied Psychology: An International Review, 54, 186-191. Karl, K. A., OLeary-Kelly, A. M., & Martocchio, J. J. (1993). The impact of feedback and self-efficacy on performance in training. Journal of Organizational Behavior, 14, 379-394. Latham, G. P. (2007). Work motivation. History, theory, research, and practice. Thousand Oaks, CA: Sage. Lehtinen, E., Hakkarainen, K., Lipponen, L., Rahikainen, M. & Muukkonen, H. (1999). Computer supported collaborative learning. In H. van der Meijden, R. J. Simons, & F. de Jong (Eds.), Computer supported collaborative learning networks in primary and secondary education. Project 2017. Final Report. University of Nijmegen.
ISLS
225
ICLS2012
Nelson, C. B., & Ketelhut, D. J. (2008). Exploring embedded guidance and self-efficacy in educational multiuser virtual environments. International Journal of Computer-Supported Collaborative Learning, 3, 413-427. Peterson, R. A., & Brown, S. P. (2005). On the use of beta coefficients in meta-analysis. Journal of Applied Psychology, 90, 175- 181. Pugh, K. J., & Bergin, D. A. (2006). Motivational influences on transfer. Educational Psychologist, 41, 147-160. Puntambekar, S., Erkens, G., & Hmelo-Silver, C. (2011). (Eds.) Analyzing interactions in CSCL: Methods, approaches, and issues. Berlin: Springer. Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59-82. Sadri, G., & Robertson, I. T. (1993). Self-efficacy and work-related behavior: A review and meta-analysis. Applied Psychology: An International Review, 42, 139-152. S , R. (2010). Digital tools and challenges to institutional traditions of learning: technologies, social memory lj and the performative nature of learning. Journal of Computer Assisted Learning, 26, 43-64. Sawyer, R. K. (2006). Introduction: The new science of learning. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 1-16). Cambridge: Cambridge University Press. Stajkovich, A. D., & Luthans, F. (1998). Self-efficacy and work-related performance: A meta-analysis. Psychological Bulletin, 124, 240-261. Trost, S. G., Owen, N., Bauman, A. E., Sallis, J. F., & Brown, W. (2002). Correlates of adults' participation in physical activity: review and update. Medicine & Science in Sports & Exercise, 34, 1996-2001. Tynj , P. & H l kkinen. P. 2005. E-learning at work: Theoretical underpinnings and pedagogical challenges. Journal of Workplace Learning 17, 318-336. Van Dinther, M., Dochy, F., & Segers, M. (2011). Factors affecting students self-efficacy in higher education. Educational Research Review, 6, 95-108. Vauras, M., Salonen, P. & Kinnunen, R. (2008). Influences of group processes and interpersonal regulation on motivation, affect and achievement. In M. Maehr, S. Karabenick, & T. Urdan (Eds.) Social psychological perspectives. Advances in motivation and achievement (Vol. 15, pp. 275-314). New York: Emerald Group. Veermans, M. (2004). Individual differences in computer-supported inquiry learning motivational analyses. Turku: Painosalama. Volet, S., Vauras, M. & Salonen, P. (2009). Psychological and social nature of self- and co-regulation in learning contexts: An integrative perspective. Educational Psychologist, 44, 1-12. Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction in computer-supported collaborative learning. Computers & Education, 46, 71-95. Yi, M. Y., & Davis, F. D. (2003). Developing and validating an observational learning model of computer software training and skill acquisition. Information Science Research, 14, 126-169. Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13-35). San Diego, CA: Academic Press.
ISLS
226
ICLS2012
Teacher Paradigm Shifts for 21 st Practice Skills: The Role of Scaffolded Reflection Within A Peer Community
Cheryl Ann Madeira, James D. Slotta Ontario Institute for Studies in Education, University of Toronto cheryl.madeira@utoronto.ca, jslotta@gmail.com Abstract: This paper presents a longitudinal study of teacher professional development that supports teachers through two interventions, namely scaffolded reflection and peer-exchange, and enables teachers to strengthen their learning of complex Web 2.0 practices within a community of learners. Teachers develop such knowledge only through authentic classroom practices. Nine secondary science teachers (N=9) designed, enacted and revised a technology-enhanced inquiry-based lesson. Teachers ideas, lesson plans and reflections were recorded, and their classroom enactments were captured on video in order to connect to their shifts in practice.
In science education, a broad community of scholars have focused on the need for 21st century knowledge skills including inquiry, collaboration, data-sharing, and critical thinking, as well as a wide spectrum of digital literacies (Slotta & Linn, 2009; Songer, 2006; Scardamalia, & Bereiter, 2006). These researchers are developing powerful new forms of inquiry that leverage Web 2.0 technologies, engage students in evidencebased reflection and critique, and promote an understanding of contemporary science. As academic research progresses toward new standards and expectations for K-12 science (e.g., AAAS project 2061; NRC, 2011; Partnership for 21st Century Skills, 2009) schools and teachers must find a way to transform their own conceptualizations and treatments of science. However, science teachers often hold traditional views about the nature of and learning of science, and they lack effective models of professional development that will enable them integrate new inquiry and technology practices (Fishman et al., 2003). Thus, if teachers are to reconceptualize their approaches to teaching science, they must be supported in their efforts to design and enact curricula that will engage students in authentic forms of 21st century science practices. In order to support such a shift in science education, teachers require in-service professional development that will direct them to learn about, apply, and reflect on the success of new pedagogical applications of inquiry and technology. This paper presents a longitudinal study of teacher professional development that supports teachers through two interventions, namely scaffolded reflection and peer-exchange, and enables teachers to strengthen their learning of complex Web 2.0 practices within a community of learners. Traditional learning opportunities for teachers (e.g., workshops, seminars) are far removed from the authentic context of classroom practice, and have limited peer-exchange. Hence, these are poorly suited for the development of teacher learning and instruction. In the current Web 2.0 era, social exchanges play a critical role in learning, and technology shapes the way students make sense of the world (Collins & Halverson, 2010). Research literature indicates that while technology-enhanced inquiry materials support students as they participate in scientific experimentation (McDonald & Songer, 2008; Metcalf & Tinker, 2004) and student learning of targeted concepts (Chang et al., 2010), teachers who have limited experience with implementing technology-designed curriculum are at a disadvantage with respect to the enactment of such curricula (i.e., implementation) (Fishman et al., 2003). Authentic learning opportunities for teacher learning, (e.g., lesson planning) mediated by Web 2.0 tools that scaffold reflection and promote peer-exchange offer science teachers new opportunities for the sharing ideas, strategies and artifacts about their practice. Such experiences for teachers could give them ideas in the how to implement Web 2.0 tools into their science curriculum. This design-based research examines two important mechanisms for teacher knowledge development: reflection and peer exchange. Nine science teachers participated in online and face-to-face scaffolded reflections and exchanges that were designed to connect directly to their relevant professional practices (i.e., lesson planning and lesson enactment) and to promote collaboration. The impact of these scaffolded reflections was examined during three stages: (1) teacher lesson planning, (2) enactment, and (3) revision of lesson. A coding schema was designed specifically for lesson planning and enactment utilizing a particular formalism of Activity Theory, drawn from Cultural Historical Activity Theory (Eng strom, 1987). This paper provides a rich description of how individual teachers developed understandings within the context of lesson planning and contributes to our understanding not only of situated collaborative learning, but also offers insight to models of teacher professional development. This paper will argue that teachers developed knowledge as direct result of these experiences.
Introduction
Theoretical Considerations
ISLS 227
ICLS2012
There is a substantial body of literature on teacher knowledge (De Jong, 2003; Gess-Newsome, 1999) and teaching professional development (Gerrard et al., 2011; Lawless and Pellegrino, 2007). Scholars have begun to clarify different kinds of knowledge held by teachers, including an understanding of Content Knowledge (CK), general instructional strategies (Pedagogical Knowledge, PK), specific student misconceptions and student learning difficulties (Gess-Newsome, 1999; De Jong, 2003), and Technology Pedagogical Content Knowledge (TPCK) (Mishra & Koehler, 2005). The difficulty in understanding about teacher knowledge is its implicit nature especially as it relates to their instructional practice. Special artifacts such as wiki lesson plans, and teacher wiki scaffolded reflections can help capture the tacit nature of teacher knowledge development and suggests the possibility of a shared knowledge among teachers. Reflection is a cognitive process that helps learners to construct their own coherent understandings (Slotta & Linn, 2009). Journaling can deepen teachers understanding of their learning, the quality of their learning, and strengthen their understanding of the experience (Moon, 1999). However, while most teachers would acknowledge that reflection is an important professional practice, they dont make an effort to incorporate explicit reflective activities in their own learning processes (Hargreaves, 1992). In addition, peer exchange of ideas and experiences about student learning and enactments of lessons, assists teachers in gaining insights they may not otherwise have recognized (Davis, Smit hey, & Petish, 2004). The notion of collaborative learning implies a relationship between individual and group, and reflects the position that social interaction provides not only background but also the context for the learning itself (Dillenbourg, 1999). Hoadley and Kilner (2005) reported that many teachers value collaborative practices and ultimately seek to initiate a community of practice that can provide opportunities for learning, peer exchange, and sharing of ideas. Such community-based exchanges can help individuals to better understand their own practice and can facilitate the construction of new knowledge (Scardamalia & Beireter, 2003). Hoadley and Kilner argued that knowledge artefacts that are constructed, shared, and reconstructed by individuals within the community benefit all community members through shared context and reflection. Teachers utilize prior knowledge in their teaching practice, but with respect to technology and 21 st Century skills, they lack in personal, pre-service and professional development experiences (Davis, 2003). This study investigates how scaffolded wiki reflections provide contextually relevant opportunities for teachers to build understandings that connect directly to their instructional practices, patterns of interaction, and achievement within their classroom. It builds on the aspects of collaborative learning of technology-enhanced inquiry-based lessons. This study reasons that through the mechanisms of reflection and peer-exchange mediated by Web 2.0 tools, teachers can incorporate similar mechanisms within their inquiry-based lessons, and develop adaptive expertise (Schwartz et al., 2005) about 21st Century science practices within their classes.
Methodology
This iterative, design-based study examined science teachers as they co-designed, enacted, and revised a technology-enhanced, inquiry-based lesson. After an initial interview about their existing ideas concerning project-based learning and technology, teachers were shown a generic set of characteristics for Project-Based Learning (Laffey et al., 1998; Blumenfeld et al., 1991) and reminded of various technologies including productivity software (e.g., Microsoft Office), visualization tools (e.g., Inspiration) social technologies (e.g., wikis or blogs) and interactive learning environments (e.g., WISE: The Web-based Inquiry Science Environment). Next, they worked closely with a mentor to design a project-based lesson, and integrated technology in a meaningful way. Reflections were collected at the time of lesson design, during classroom enactment, during assessment activities, and as the lesson was being redesigned. The teachers met at various at several points during the iterations to discuss and share their experiences with technology and inquiry-based implementation.
Participants
There were nine science teachers (N=9) who volunteered to participate in this study, with a range of experience and disciplinary expertise (i.e., physics, biology, chemistry, or general science). The teachers were from 5 different schools located in a large urban city in North America and had a wide variety of technology supports provided by their school base. They were randomly selected because of their interest in technology and initial understanding about project-based lessons. For confidentiality, all participants were given pseudonyms (see Table 1, p 4). All lessons were co-designed by the teacher working closely with the mentor, a doctoral student with 17 years of teaching experience.
Figure 1. Teacher Participants TechnologyEnhanced Co-Design Meeting
ISLS
228
ICLS2012
Materials and Data Sources

Pre-survey and Interview: In order to establish a measure of teachers background and pedagogical content knowledge, a pre-survey was administered to all teachers, followed by an interview for purposes of clarification and to orient the teacher to the study. The following questions were among those given to teachers before starting the research project: (1) What are some of your best learning experiences and why do you think they were important?; (2) What are some of your visions within your science classroom?; (3) What are some of your previous project-based lessons that you have conducted? Lesson Plan Template: A wiki site was designed with specific scaffolding categories for the teacher to design their new technology-enhanced project-based lesson. Some sample categories were (1) Determining Topic, (2) Challenges for students in science topic?, and (3) How can technology help?. Teacher Lesson Designs: Teachers co-designed a technology-enhanced project-based lesson using the scaffolded wiki-template with the research-mentor and their peers in the community. These lessons targeted a project-based learning pedagogy and included a variety of technology tools and materials. The lessons themselves were a focus for analysis. The table below provides an overview of the basic science topics and technology-enhanced project-based lesson themes selected by each teacher.
Figure 2. Teacher Lesson Designs on Wiki
Figure 3. Scaffolded Wiki Reflection
Reflection Questions: A wiki site and specific scaffolded reflection prompts were designed to capture teachers understandings about pedagogical content. Reflection questions were asked at three different points in each teachers lesson planning and enactment process (see Figure 3): Prior to the lesson planning: (1) What are the goals of your lesson? (2) What are your thoughts about the student ideas? (3) What are some of the key elements in your project-based design? After the lesson planning: (1) Where do you think the students will be challenged? (2) What will you try to do with your time, during the lesson? (3) What are some of the follow up concepts that require set up during lesson? After the lesson enactment: (1) Did you find the students had more or less difficulty than you expected? (2) What is one change or addition you would like to put into place for next time? (3) What was one advantage in using the technology within the project-based activity? Community: The peer community within this project includes both offline and online components. The teachers had periodic community meetings where they exchanged ideas and shared their stories about the project-based enactment, which served to establish a personal relationship between community members. The online component of the teacher community consisted of a website and a wiki site, developed to collect personal statements from teachers about their background and philosophy, as well as to collect details of lesson plans, and all reflections. Table 1. Technology-Enhanced Project-Based Lessons
Teacher Participant Bill Merle Charlie Grade Level 10 12 11 Science Concept Chemistry Indicators Physics interdisciplinary; several concepts Physics Sound, Electromagnetism, Technology-enhanced PB Lessons Photo Journal/Wiki/Poster of Experiment Wiki/Podcast Walking Tour of Science Exhibits Wiki/Video/Experiment Segment of
ISLS
229
ICLS2012 Newtons Laws Physics Newtons Laws, Renewable Energy Ecology Wolves management program Earth and Space Earths Crust project Earth and Space Earths Crust project Chemistry interdisciplinary; several concepts Biology Reproduction; Physics Reusable Energy
Volume 1: Full Papers concept Wiki/Video/Experiment Segment of concept Wiki/WISE/Video Wolves Ecology Commercial Wiki//Video/Powerpoint Presentation Wiki//Video/ Powerpoint Presentation Wiki/Poster/Debate-Presentations Wiki/ Powerpoint Presentation
Maddie Frank Placido Alex Olga Daryl
11 7/8 7/8 7/8 12 9
The online community supported peer exchanges, reviewed lesson plans and fostered discussions about the enactments. Upon completing the lesson plan an update of the lessons learned and the things I hope to add to the lesson next time was added to the wiki lesson page. Teachers in a community were asked to connect to their peers by asking questions and commenting to this additional wiki page. The wiki site for lesson design and reflection played a significant role in supporting socially constructed knowledge. It enabled teachers and the mentor to make their knowledge visible for themselves and all members of the community. These on-line artefacts became an asset for reference by all members of the community. Figure 4 and 5 displays the Website interface of the on-line community (left) and the face-to-face teacher community environment (right).
Figure 4. Community Website
Figure 5. Community Face-to-Face
Procedure
This paper reports on nine science teachers development of PCK and their experiences during 3 design phases of curriculum design and enactment. There were three main teacher activities that occurred within each iteration: (1) Lesson design, (2) Classroom enactment, and (3) Revision of lesson design. Data sources include teacher surveys, interview questions, wiki lesson plans, scaffolded reflections (captured in a wiki), videotaped classroom enactments, field notes, student artifacts and wiki responses, teacher peer-exchanges (on a wiki and in group meetings). The first design phase of the study included four teachers (n = 4) who worked individually with the researcher-mentor to co-design, enact and reflect on a science lesson. The second design phase added five more science teachers (n=5), increasing the community to a total of nine teachers and one researcher. Consistent with the design-based research paradigm, improvements were made on the reflective prompts (i.e., connecting reflection prompts to lesson planning and enactment) and community elements (face-to-face and online), which marked the beginning of the second design phase. The third phase of the design study continued to refine the scaffolded reflection and community exchange connecting teacher activities of lesson planning and enactment deeply with student prior knowledge, project-based learning and technology implementation.
Analysis and Findings

Analysis
ISLS
230
ICLS2012
The data collected were grouped into elements drawn from Grossmans (1990) taxonomy: pedagogical content knowledge (PCK), content knowledge (CK), pedagogical knowledge (PK), technology pedagogical content knowledge (TPCK), contextual knowledge (CXK). The coded elements became an empirical source to address an Activity Theory (AT) based coding of two activity systems: (1) lesson planning and (2) lesson enactment (adapted from Mzanwa, 2002). For each node of the two respective AT, a qualitative score of 1-3 was assigned and related to identifying elements (e.g., PCK, or Project-Based Learning). For example, the Subject node of each triangle was taken to represent a teachers PCK.
Teacher Knowledge Improvement

The first three elements of the coding scheme for the lesson planning comprised of a category named Subject and together the score reflected the teacher and their lesson planning knowledge. These three elements are as follows: (a) rationale for content choice, (b) connecting strategy to the subject-domain concept, and (c) understanding of technology and use of it to improve project design. Taken together, these three subject codes provide a measure of pedagogical content knowledge (PCK), including the subconstruct of technology pedagogical knowledge (TPCK). Although PCK and TPCK are seen as abstract terms in the literature, they are valid forms of teacher knowledge and can be measured within the context of authentic teaching practices. From the analysis, the Subject score reflecting some measure of PCK improved from Iteration 1 to Iteration 2 for the participants of this study. Thus, teachers demonstrated improved pedagogical content knowledge in their second planning cycle, as measured by the average score of the three elements. Figure 6, recognizes that at the end of the second design cycle not all the teachers completed two iterations. Three teachersMerle, Olga, and Maddiebegan their first iteration with a community meeting (i.e., they joined the study in Design Phase II), and it is evident from their design and reflections that the peer exchange had a strong influence on their lesson planning. It is suspected these teachers gained some pedagogical knowledge related to project-based technology-enhanced lessons from their initial participation in the peer community.
Figure 6. Measure of PCK-TPCK Iteration 1 Iteration 2 Isolating the element of technology pedagogical content knowledge (TPCK) from the other elements in the coding schema (i.e., restricting the analysis to the third element only) indicates a similar increase in score from Iteration 1 to Iteration 2 (see Figure 7). These patterns identify the changes in teacher-participants understanding about the way in which technology can enhance their lesson. Critical to their growth was reflection about their lesson planning activity, and, indeed, the reflection note they completed during planning was the primary source of coding for this element. Participants started off at varying levels of understanding about how technology could be embedded within the lesson activity. Technology was an important feature in the design of the project lesson, and as teachers became more aware of its potential, they began to independently incorporate technology as a way for students to share ideas and to scaffold student collaboration. For example, Merle started to realize the studentpeers were a valuable resource for each other. As students critiqued ideas of their peers during Merles project activity, they began to see that their voice was important, and because of this, Merle began to use technologybased student collaborations in other classroom activities such as note-taking. Bill decreased in teacher knowledge score and had no change in his TPCK score within the lesson design coding schema. Again, this could be linked to his poor wiki reflections (in written form) concerning his lesson design and the student learning processes that he hoped to target with technology.
ISLS
231
ICLS2012
Figure 7. Improvements in TPCK (or TPACK): Iteration 1 to Iteration 2
Technology and Effective Adapted Practices

Teachers who learned also adopted technology-enhanced project-based practices more effectively. As they participated in the community activities that included wiki reflections and on-line technology sharing, the teachers began to use the techniques of scaffolded peer-feedback on their on class wiki site. The adoption of these ideas offers a plan for teacher learning that includes the use of inquiry mediated by technology. Table 2 (below) summarizes the teacher-participants in this study with respect to project-based teaching and technology experience, with teachers scores based on the lesson planning and enactment schemas. The range of values from -3 to +3 allow teachers to be placed within a two dimensional matrix, which offers some measure of their baseline experience and understanding of project-based learning and technology, respectively. Table 2. Summary Initial Teacher Understandings of Project-Based and Technology-Enhanced Lessons TeacherParticipant Project-based Learning High Charlie Low -1 TechnologyEnhanced Lesson High Low -1 Some understanding of technology but design of project-based lessondidnt use affordances at start (limited community but grew) Only used PowerPointnot effective understanding of wiki (limited community) Used WISE and customized it (limited community) Limited use of technology, some understanding of projectbased learning (limited community) Great project-basedbut nervous at beginning about technology (was beneficial and benefited from community) Weak in both areas (although was able to benefit from the studys community) Had some understanding in both areas (although was really able to benefit from the studys community) Enthusiastic, but limited use of technology (benefited from community) Really used the same form of project as Alexhe had limited use for project-based learning (resistant to change) (benefited from community) Comments on Participant Background
Daryl Frank Bill Merle Maddie Olga Alex Placido 2
-3 -1 -2 1
-3
-3 -2
-1 -2 -2 -3
-2 -2 -2 -3
ISLS
232
ICLS2012
Figure 8 provides an overall sense of teachers initial experiences at the outset of their participation within the study. When the scores from Table 2 are plotted on a two-dimensional grid, only one teacher (Merle) was experienced with project-based instruction, and only one (Frank) had a high level of technology experience. Clearly, all teachers in the study stood to gain from their participation in a reflective process of lesson planning and enactment. The coding schemes allowed a mapping of several distinct kinds of data, resulting in scores or weights for the various nodes of the activity triangle. While such applications of the activity triangle are not generally favoured by activity theorists, this approach does serve to capture the complex temporal and Figure 8 Initial Teacher Understanding of cultural aspects of an activity like lesson planning (i.e., Project-Based Learning and Technology as opposed to, say, just scoring the teachers lesson plans). This coding schema made variables consistent (with common vocabulary) across iterations and across participants. Examining the various activity system nodes allowed an examination and discussion of the individual changes or shifts in practice, as influenced by the teacher reflection. One way to interpret knowledge development is by examining the teacher knowledge quadrants. While teachers were placed in their initial quadrants via a separate coding rubric (technology and project-based learning), it is possible to examine their movement within the quadrants by utilizing their Subject scores, which correspond to PCK and TPCK. The initial quadrants positioned teachers on their understanding of project-based learning and technology. PCK is knowledge that encompasses understanding of how to implement pedagogy (e.g., project-based learning) and TPCK is knowledge that specifically addresses technology enhancements to the pedagogy. Both PCK and TPCK are seen to be similar to the initial measures of projectbased learning and technology. Figures 9 and 10 (below) illustrate teachers in the knowledge quadrant diagram after each iteration of planning and enactment. The initial teacher knowledge grid scoring assesses project-based and technology understanding. This original grid is more representative of teachers prior thoughts seen as a more generic or static view of their understanding of project-based learning and technology implementation. However, as the teacher participated in the lesson planning and enactment activity, their understandings were identified more specifically through the rubric scoring as PCK and TPCK. The scores for the project-based dimension were obtained by combining the Subject scores (i.e., PCK) for planning and enactment, resulting in a single measure (displayed on the figure), which was then scaled to fit onto the quadrant scale of 1 (weak knowledge) to 6 (strong knowledge). The score for the technology dimension was obtained in a similar way, using only the technology element from the Subject nodes.
Figure 9. Teacher knowledge quadrants postIteration 1
Figure 10. Teacher knowledge quadrants post Iteration 2
Conclusion
While it is important to realize that teacher reflection and peer-exchange doesnt require Web 2.0 knowledge, the teachers in this study demonstrated adaptive effective instructional patterns when incorporating technologyenhanced science activities in their class. Teachers did not have a strong technology background but as the research progressed teachers had a paradigm shift about the uses of Web 2.0 and how it would enhance student learning.
ISLS
233
ICLS2012
The three teacher knowledge quadrants offer a visual representation of the growth in teacher knowledge over the course of this project. They do not say anything about the teachers overall level of knowledge. But they can make a specific qualitative claims, that planning, enacting, and then revising an inquiry-based science lesson while reflecting in peer-community can help teachers learn. It also suggests that interventions that model techniques and skills (such as 21st Century practice skills) can help teachers shift in their approaches and develop pedagogy appropriate to teaching 21st Century science.
References
Blumenfeld, P.C., Soloway, E., Marx, R.W., Krajcik, J.S., Guzdial, M., & Palincsar, A. (1991). Motivating ProjectBased Learning. Sustaining The Doing, Supporting the Learning. Educational Psychologist, 26 (3 & 4). Chang, H., Quintana, C., & Krajcik, J. S. (2010). The impact of designing and evaluating molecular animations on how well middle school students understand the particulate nature of matter. Science Education, 94, 7394. Collins, A. & Halverson, R. (2010). Rethinking Education in the Age of Technology: The Digital Revolution and Schooling in America. New York: Teacher College Press. Davis, E. A. (2003). Knowledge integration in science teaching: Analyzing teachers knowledge development. Research in Science Education, 34, 2153. De Jong, O. (2003). Exploring science teachers pedagogical content knowledge. In D. Psillos, P, Kariotoglou, V. Tselfes, E. Hatzikraniotis, G. Fassoulopoulos, M. Kallery (Eds.), Science Education Research in the Knowledge-Based Society (pp. 373-381) Dordrecht: Kluwer Academic Publishers. Engestr Y. (1987). Learning by expanding: An activity-theoretical approach to developmental research. Helsinki, m, Finland: Orienta-Konsultit. Fishman, B., Marx, R. W., Best, S., & Tal, R. (2003). Linking teacher and student learning to improve professional development in systemic reform. Teaching and Teacher Education, 19, 643658. Penuel, W. R., & Gallagher, L. P. (2009). Preparing teachers to design instruction for deep understanding in middle school earth science. Journal of the Learning Sciences, 18, 461508. Gerard, L.F., Varma, K., Corliss, S.B. & Linn, M.C. (2011). Professional Development for Technology Enhanced Inquiry Science. Review of Educational Research, 81(3), 408-448. Grossman, P.L. (1990). The making of a teacher: Teacher knowledge and Teacher Education. New York: Teachers College Press. Hargreaves, A. (1992). Cultures of Teaching: a focus for change. In A. Hargreaves & M. Fullan (Eds), Understanding Teacher Development (pp. 216-240). London: Cassell. Hoadley, C. M., & Kilner, P. G. (2005). Using technology to transform communities of practice into knowledgebuilding communities. ACM SIGGGROUP Bulletin, 25(1), 3140. Koehler, M. J., & Mishra, P. (2005). What happens when teachers design educational technology? The development of Technological Pedagogical Content Knowledge. Journal of Educational Computing Research, 32(2), 131152. Laffey, J., Tupper, T., Musser, D., & Wedman, J. (1998). A computer-mediated support system for project-based learning. Educational Technology Research and Development, 46 (1). Lawless, K. A., & Pellegrino, J. W. (2007). Professional development in integrating technology into teaching and learning: Knowns, unknowns, and ways to pursue better questions and answers. Review of Educational Research, 77, 575614. Leontev, A.N. (1978). Activity, Consciousness and Personality. Englewood Cliffs, NJ: Prentice-Hall. McDonald, S., & Songer, N. B. (2008). Enacting classroom inquiry: Theorizing teachers conceptions of science teaching. Science Education, 92, 973993. Mwanza, D. (2002). Towards an activity-oriented design method for HCI research and practice. Unpublished doctoral dissertation, The Open University, Milton Keynes, United Kingdom. Retrieved April 15, 2006, from http://iet.open.ac.uk/pp/d.mwanza/Phd.htm. Metcalf, S. J., & Tinker, R. (2004). Probeware and handhelds in elementary and middle school science. Journal of Science Education and Technology, 13, 4349. Moon, J. (1999). Reflecting in learning and professional development. London: Kogan Page. Roschelle, J., Penuel, W.R., Shechtman, N. (2006). Co-design of innovations with teachers: Definition and dynamics. Paper presented at the International Conference of the Learning Sciences, Bloomington. Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 97118). New York, NY: Cambridge University Press. Songer, N. B. (2006). BioKIDS: An animated conversation on the development of curricular activity structures for inquiry science. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 355369). New York, NY: Cambridge University Press. Schwartz, D., Bransford, J., & Sears, D. (2005). Efficiency and innovation in transfer. Transfer of learning from a modern multidisciplinary perspective, 151. Slotta & Linn, M. C. (2009). WISE science: Web-based inquiry in the classroom Teachers College Press.
ISLS
234
ICLS2012
The Impacts of Flexible Grouping in a Mobile-Assisted Game-based Chinese Character Learning Approach
Lung-Hsiang Wong, Learning Sciences Lab., National Institute of Education, 1, Nanyang Walk, Singapore 637616, lunghsiang.wong@nie.edu.sg Ching-Kun Hsu, National University of Tainan, 33, Sec 2, Shu-Lin St., Tainan 700, Taiwan, cyvs@cyvs.cy.edu.tw Jizhen Sun, Chinese Culture University, 231 Section 2, Jian-Guo S. Road, Taipei, Taiwan, ccsun@sce.pccu.edu.tw Ivica Boticki, Faculty of Electrical Engineering and Computing, University of Zagreb, Unska 3, Zagreb, Croatia, ivica.boticki@fer.hr Abstract: This paper reports on how two different flexible student grouping strategies in a mobile-assisted Chinese character learning game affect the players collaborative patterns. The game application assigns each student a Chinese character component on their smartphones. The students are required to form groups to assemble eligible Chinese characters using the components that the group members possess. Fifteen Primary 3 (3rd grade) students were involved in the study. The game process data and the transcriptions of focus group interviews were qualitatively analyzed in order to investigate the dynamics of student collaboration and competition during the games. The study aims to explore the patterns of social interactions during the activities, especially the varied impacts of the two different grouping rules (allowing versus not allowing each student to join more than one group at one time) on the students game behaviors and their learning gains.
Introduction
Chinese has long been regarded as a tough language to learn. One major challenge to the learners is the logographic nature of Chinese characters (Shen, 2005; Wong, Chai, & Gao, 2011). Most Chinese characters are composite made up of multiple components that fit into the square space. The spatial configurations of these as an example, there are compositions usually follow about 15 different patterns (Zhang, 1987). Taking many Chinese characters that fit the pattern, such as , , and so on. Psycholinguists stressed the importance of the awareness of part-whole relations and component knowledge in the processing of language (Taft & Chung, 1999). Therefore, certain studies favored the instructions on the structure and form of characters. Students are encouraged to figure out the associations using their creative thinking (W. M. Li, 1989). In turn, we developed a game-based learning approach on collaborative Chinese character formation, namely, Chinese-PP (). PP refers to or Pn y Pn in Chinese, which roughly means trial assembling. In playing the game, the students are assigned smartphones in 1:1 (one-device-per-student) basis. In each game round, a set of Chinese character components is randomly assigned by the system server via 3G connections to individual students. The students are required to recognize and compose eligible characters by grouping with their peers who have different components. This paper focuses on analyzing the social interactions, collaborative patterns and learning strategies that were emerged during several game playing sessions. In particular, we will discuss the different impacts of the two grouping rules (allowing or not allowing the players to join more than one group at a time) on the students emergent interactional patterns and learning strategies, and what are their implications to Chinese character learning.
Related Literature
Chinese character learning
Many studies emphasized the importance of Chinese character recognition (Allen, 2008). Research found that learners ability to recognize different components and identify their semantic, phonetic, and structural functions in a character is predictive of their reading and language proficiency (Fang, 1996). Li (1992) also found that the structure of a character is essential for the young learners to identify specific characters and to discriminate among different ones. Both studies pointed out that the cognitive aspect of recognizing a character lies on its contour and structure rather than its strokes, in a top-down order. In a related note, how strokes are composed into a component and how components are composed into a character follow certain orthographic rules. Indeed, teaching the structures of characters and simultaneously addressing the relationship between parts and wholes generate positive outcomes on character learning (Anderson, et al., 2002). When learners start to cultivate this orthographic awareness, which is a metalinguistic awareness, they also start the process of transforming their knowledge of characters from explicit to implicit (Jiang, 2006), from performance towards the competence.
ISLS
235
ICLS2012
Cooperative Learning and mCSCL

Collaborative learning (or cooperative learning both are different but overlapping learning models) is much more than competition and individualistic learning although does encompass some of these. The main difference is that in collaborative/cooperative learning students must learn how to sink or swim together (D. Johnson & Johnson, 1987). Johnson and Johnson (1994) identified eight main principles for cooperative learning. Here, we will elaborate only four of the principles which are arguably more tightly associated with our study: maximum peer interaction, equal opportunity to participate, individual accountability and positive interdependence. Positive interdependence exists when group members are linked with other group members in a way that they cannot succeed without each other. There are several ways of structuring it: devising a clear group goal (positive goal interdependence), rewarding the group if all members achieve a set criterion (positive reward), by combining distributed resources (positive resource interdependence) and by assigning complementary and interconnected group roles (positive role interdependence). Maximum peer interaction refers to the transformation of the normal classroom interaction pattern which is mainly sequential. Typical interaction pattern consists of the teacher talking, stopping in order to allow for student responses and evaluating students responses. Cooperative learning disrupts that pattern by transferring agency to the students who form groups working in parallel. Therefore, peer interaction is maximized with more students talking to each other at a time. Each student should have equal opportunity to participate in the activity. The activity design should not allow any student to dominate the activity and impede the participation of others. By receiving their portion of the task and being aware of their role(s), students should be able to fully demonstrate their abilities. In order to prevent social loafing (some group members hitchhiking the work of others), learning activities must come with sufficient individual accountability. That is, each student is held responsible by group mates for contributing his or her fair share to the groups success. Latest developments in the field of mobile Computer-supported Collaborative Learning (mCSCL) extend the idea of handheld technology mediated learning with the collaborative scaffolding in order to include both social and epistemic collaboration scripts encouraging small group participation (Nussbaum, et al., 2009). The design of collaborative scaffolding should encourage social interactions, facilitate joint problem solving, lead to richer knowledge construction and in the same time take into account different and emerging roles, joint group goals and actions and facilitate verbal explanations.
Method
Procedure
We adopted the design research methodology (Brown, 1992) to carry out the Chinese-PP study. Fifteen Primary 3 (3rd grade, 9-year-old) students who were learning Chinese as second language in a primary school in Singapore participated in the empirical study. The intervention was comprised of six learning sessions conducted every fortnight, unless pre-empted by school exams or holidays. Specifically, the intervention aimed to progressively establish and enhance the students orthographic awareness. Each learning session consisted of three sections, namely, warm up (15 minutes), game playing (30 minutes), and recalling (15 minutes). (1) Pre-task: warm up Each of the warm-up section began with the teacher going through a quick revision of what was covered in the previous Chinese-PP sections. She then made a brief Powerpoint presentation to deliver new knowledge of Chinese character structure (orthographic knowledge). One example is pictophonetic character, which refers to a character that is comprised of a component indicating the pronunciation and another representing the semantics, e.g., (means sunny, pronounced as qng), with (sun) hinting the semantic meaning or picture of the character, while (similarly pronounced as qng) indicating the pronunciation. The aim was to equip students with prerequisite knowledge for the subsequent game playing section. Since this paper focuses on the analysis of students game playing patterns and strategies, we will not present the details on the domain-specific curriculum design. (2) Main-task: Chinese-PP game playing Albeit taking place in a general classroom, we rearranged the furniture to set aside an empty space. We encouraged the students to walk around, form and re-form ad-hoc physical clusters to ardently discuss with different peers so as to explore alternative possibilities of characters. The students played two 15-minute game rounds in different modes, namely, single-group mode (SGM) and multi-group mode (MGM). The teacher facilitated the game to ensure tighter linkage between activity and content (i.e., the knowledge of Chinese character structure covered in the warm up section). Typically, the teachers facilitation tasks included controlling the game pace, hinting the students on-the-fly concerning possible groupings, verifying students groupings (the correctness of the formed characters), and determining when to terminate a round. (3) Post-task: Recall and reinforce all the characters composed during the game
ISLS
236
ICLS2012
After the game, the teacher facilitated a recalling activity where students were asked to relate the characters that they had composed during the game with the character structure knowledge that they learned in the present and past warm up sections (e.g., relating to pictophonetic character).
Design of game-based learning activities

The Chinese-PP game approach can be characterized as a spontaneous, flexible grouping model as no fixed student group is pre-determined. Each student is equipped with a smartphone in which they can see what components they have and what the other classmates are assigned. The students will identify their partners in order to collaboratively compose the components into an eligible Chinese character. When one game advances to the next round, the existing groups are all disbanded and a new set of components are assigned to them. The setting and the devices used in the study consists of a projector, the 3G wireless connection, a laptop with the Chinese-PP teacher console being installed, and 15 smartphones installed with the client application of Chinese character recognition game. The facilitator prepared several sets of Chinese components that are equal to the number of participating students in advance. When a game round starts, the smartphone application displays all the components for a student to select and configure (spatially) in order to form a Chinese character (see the left of Figure 1). Upon submission of her composed character to the server, the other students who own the components that she has selected will receive the character on their My Groups windows as a proposal for grouping (see the right of Figure 1). However, the proposed student cannot take for granted that her peers will join her group as the other students might have also formed their own characters or receive other proposals. This is the point where she will need to negotiate with the peers to join her group. The students are free to move around in the classroom to discuss with different classmates.
Figure 1. My Character interface (left), and My Group interface (right). A scoring scheme is applied in the game. Students earn and accumulate scores by forming eligible groups 10 points for a 2-component character, 20 points for a 3-component character, 30 points for a 4-component character, and so on (same score to be awarded to each member of the group). This is to encourage the students to form bigger eligible groups for identifying more complex characters. During the MGM, a student who joins more than one group will earn accumulated scores from all the groups carrying eligible characters. The system allows the teacher to tweak the game rule of whether to allow an individual student to form or join more than one group at a time (known as SGM and MGM respectively). Consider this scenario: a student, Sam, who is assigned the component (sun) submits a character (but) to Rita who owns the component (person) and Lisa who owns the component (one). At the same time, John who owns the component (moon) proposes (bright) and invites Sam to join him. If such a scenario takes place in SGM, Sam will have to choose between the two options; e.g., if he joins the group and rejects the group, he will win 20 points; or 10 points if he chooses otherwise. If it takes place in MGM instead, Sam may join both. In turn, he will earn 20 points for composing plus 10 points for composing respectively (see Figure 1). During our empirical study, the teacher alternated between the two modes across different game rounds in order to experiment their impacts on the students collaborative patterns and game strategies. Figure 2 depicts an instance in the middle of the game where the teacher made use of the projected teacher console UI to explain and give just-in-time feedback to the characters (which may consist of correct or
ISLS
237
ICLS2012
incorrect ones) formed by the students. Moreover, the students can refer to the characters composed by other students and the scores they win (see Figure 2) in the teacher console.
Data collection
In order to relate the participating students game behaviors and their academic performances in the formal Chinese class in our data analysis, we ranked the 15 students based on their academic results of the subject and divided them into three bands the top-5 students are known as high achievement (HA) students, the next 5 are medium achievement (MA) students, and the last 5 are low achievement (LA) students.
Figure 2. The interface of teacher console projected i the large screen. Throughout the interventions, we carried out video and audio recording, and took field notes of all the games to document the students game behaviors and collaborative patterns. The software logs of the students interactions and the automated screen captures of the teacher console during the phone games were also used for triangulation with the recorded material. By the end of the sixth session, we conducted a focus group interview with six students (including two randomly chosen HA students, two MA students and two LA students) to triangulate our game process analysis and probe them on the reasons behind their game habits. In this paper, we concentrate on analyzing the student-student interactions during the last three Chinese-PP sessions. The reason is that whereas the participating students had been more reliant on the teacher in the first three sessions, the teachers persistent strategy of advising the students to discuss with their peers had instead been progressively transforming the students help seeking behaviors. They became more independent and were willing to help each other, thus achieving effective peer interactions towards the last three sessions while the teacher-student interactions were fading out. Due to the space limit, we decide to focus on making sense of how the more frequent peer interactions had led to improved learning outcomes. We followed a coding procedure similar to that of grounded theory (Strauss & Corbin, 1990) on the collected data. This allows us to see new light from the collected data rather than conforming to existing theories. Due to the space limit, we will not present the detailed analysis but only provide a synoptic view of our findings in the next section. All the student names reported hereafter are pseudonyms to protect their identity.
Findings
The students exhibited high energy level during the last three Chinese-PP sessions. They walked around and carried out quick discussions on what would be viable in forming eligible characters. The dynamic grouping strategy enabled by mobile information exchange and face-to-face interactions had motivated them in game playing. In this section, we will present our findings in four themes, identified through our qualitative analysis, namely, overall patterns of individual game rounds, initial character forming and invitation for grouping, impacts of SGM and MGM on students game playing patterns, and competition versus collaboration. Hereafter, we refer to students who figured out Chinese characters and sent group invitations as student-leaders, while those who were invited to join a group as invitees. Due to the flexible grouping nature of the game, an individual student might rapidly switch her role and even assume both roles at the same time. For example, during a game round, a student might send out an invitation and then realizes that she is also invited by two other groups it is up to her to decide whether she wants to stick to her own proposed group or join another group (in SGM), or join more than one group (in MGM).
Overall Patterns of Individual Game Rounds

We observed a consistent pattern in the last three game sessions. In the beginning of each round, instead of going straight to peer interactions, they took their time to assemble characters individually on their phones, as though they were playing a one-player game. The trial-composing stage usually lasted about two minutes where
ISLS
238
ICLS2012
they stood or sat still at their original positions. Subsequently, as invitations were sent out one after another, they began to move around with their phone and look for their potential group-mates (who had the components that they needed) to discuss about the proposed characters, and perhaps switching between different clusters of peers to explore other possibilities. Occasionally, they approached HA peers to ask for quick advices even though they did not intend to form a group together. They remained active till the end of the game round. We cross-examined the invitation data in the server log and the inviters academic achievements and yielded the descriptive statistics in the last two Chinese-PP sessions, where the accumulated frequencies of HA, MA, and LA students (n = 5, respectively) in proposing eligible characters were 18, 21 and 11 respectively. The figures show that the HA students did not dominate the games. The MA/LA students had also made their share of contributions. We traced back to the data in the first three sessions and found that although HA students tended to be more proactive in these earlier sessions, the MA/LA students were not left alone. The HA students often invited them to join their groups as the MA/LA students were assigned the components that the HA students needed. In inviting the MA/LA students, the HA students needed to explain to them on the formed characters and the orthographic knowledge. Such emergent peer coaching seemed to have elevated the MA/LA students competencies and self-efficacy. Thus, the gaps between the HA and MA/LA students were gradually narrowed.
Initiating Character Forming and Invitation for Grouping
Impacts of SGM and MGM on Students Game-playing Patterns

A finer-grained analysis of the intra- and inter-group interactions has shed light on the varied game playing patterns of the students in playing SGM and MGM games. During the MGM games, the students were more motivated to trial composing alternative characters. They did not mind composing either a simple or complex character to start with; and the time they spent in working out and submitting their first characters tended to be shorter. Conversely, during the SGM, they were more inclined to deliberately composing the most complex Chinese character possible in one shot, and then giving up exploring other possibilities in composing alternative characters with their own components. In turn, they often took more time to form their first characters. Even though the game mechanism did allow them to disband their submitted group if they change their mind, they were in general not keen to do so but rather stick to their first (and only) formed characters throughout the game round. It was observed that there were more discussions but fewer characters were formed and submitted. Indeed, we discovered several emergent game patterns or strategies that were specific to either game mode. For example, during the SGM games, the students often delayed their decision or commitment in joining one group that they were invited to they wanted to spend more time in exploring other possibilities of forming more complex characters and earn higher scores. Instead, during the MGM games, members of a particular confirmed group might proceed to explore more character forming opportunities with one or a combination of the strategies as shown in Table 1. We did not teach them the strategies but they figured out these by themselves, perhaps as part of their socio-constructivist process, in internalizing the Chinese character knowledge that they learned from their teacher. We characterize the first three strategies as basic ones. The last two are derived from the basic strategies, which often involve more sophisticated student-student interactions. Table 1: Student strategies for figuring out alternative characters during the Chinese-PP games
Strategy Expansion Reduction Replacement Splitting Description To add component(s) owned by non-group member(s) in order to form a more complex character To remove one or more components in order to form a simpler character To replace one or more components of the group character in order to form another group with another similar character To split a character into multiple components or component sets, each forming new characters with other component(s) respectively To take one or more components from at least two characters and form a new character Example Splitting into components and , and form and respectively (i.e., double replacements) Taking from , from , and from ; and form
Reassembling
Thus, we observed a sequence of component manipulations in each MGM game round. During the first three Chinese-PP sessions, the students tended to be content in figuring out simple two- or three-component characters, and occasionally moving on to the possibility of replacing or adding one component to form another character. However, as they developed the skill of composing the most complex character possible in one shot during the SGM game rounds, they often transferred such a strategy to the MGM game rounds in the last three Chinese-PP sessions and extended it into a top-down, character reduction approach.
ISLS
239
ICLS2012
We take two instances of character compositions in session 4 as an example. During the SGM game, four students formed a group and submitted the character (fortune). They earned 30 points each and just stopped there. However, when they proceeded to play in the MGM game, two of the four students formed the five-component Chinese character (caution) with three other peers. They then took one component out each time and subsequently formed (respect), (thoughtless) and (sentence) (see Figure 3). Should they construct the character during the SGM mode instead, they might not bother to carry on with further reductions of the character as that would not increase their scores.
Figure 3. An example of students in multi-group mode composing a character in a top-down way. Apart from sticking to the reduction strategy, the students were getting more adept in applying multiple strategies to construct more alternative characters towards the last two Chinese-PP sessions. Such character altering sequences could be re-presented in the form of tree structure. Figure 4 depicts an example of a sequence occurred during Chinese-PP session 5. It started with two students grouping together to form the character (pile; with the components and ) and earn 10 points each. They then split up and grouped with two different peers to form (push) and (rubbish) respectively (splitting). Subsequently, three other students who formed the character (borrow) deployed the splitting strategy again to compose (wrong) and (location; to the students who formed , transforming to was actually the application of the replacement strategy instead). They then simplified and yielded (in the past) (reduction). In a nutshell, during the SGM games, as the invited students tended to spend more time to consider whether to join a proposed group, the student-leader of the proposed group often approached individual invitees or brought the potential group members together for a persuasion. During the MGM games, however, students often committed to a group faster. Members of the confirmed group usually proceeded to discuss within the group about various ways of altering the formed character. They did so by applying the strategies stated in Table 1, which would lead to forming new groups with different groups of members.
Competition versus Collaboration

During the Chinese-PP sessions, we observed both the competitive and collaborative behaviors displayed by the students. The scoring mechanism worked well in motivating the students to strive for composing greater amount of characters (usually in MGM games), or more complex characters (usually in SGM games). The teacher console offered the affordances of automated, live updates of student scores and their rankings. Most of the students were checking their scores and rankings from time to time during the game rounds. When they found out that they were lagging behind other peers, they tried harder to improve the situation. In a related note, some of the students initially found multiple invitations at a time irritating, as they felt that those had given them a hard time in decision making. Nonetheless, they became excited when they realized that their scores might be elevated as a result they would either be able to choose the most complex character during a SGM game, or join multiple groups during a MGM game. Figure 4. A character Much to our surprise, such a competitive mindset had neither hindered altering sequence student collaborations nor distracted them from assisting their peers who needed during session 5 helps. Most of the students, when being requested for advices to confirm or alter specific characters by their peers who belonged to different groups, they offered their help without reservation. Two HA students, Wendy and Nathan, often took the initiative to assist other groups in their game playing after they had formed their own groups respectively. In particular, Wendy paid special attentions to her peers who had yet to identify any character or being invited to join any group. Subsequently, their peers became more inclined to seek for their assistance. We learned from the focus group interview that the reason behind this was their self-esteem in bringing the best out of themselves in the process of helping others. The social relationships among the students did not play a prominent part in their groupings. For example, they used to exhibit gender-shy behaviors during the first two Chinese-PP sessions. However, towards the last few sessions, they seemed to ignore the social factors (gender or personal affiliation) and instead focused on optimizing their groupings during the games. Notwithstanding in a few rare occasions during the SGM games, some students opted to join a group with a simpler character being proposed. They did so for the reason of their personal affiliation with certain potential group member (especially if the latter was a left-out student, i.e., no other grouping option yet), despite having the chance to join another group to form a more complex character.
ISLS 240
ICLS2012
Discussion
The flexible, rapidly altered grouping model of Chinese-PP is a novel approach in mobile gaming. A similar mobile learning approach is participatory simulation; though it usually does not reinforce explicit student grouping. Even the more general CSCL studies have been focusing on fixed, often pre-determined student groupings (such as the mCSCL approach reported by Zurita & Nussbaum (2004)), perhaps for easier classroom/learning management by the facilitators or more robust executions of collaboration scripts. In designing Chinese-PP, we intended to leverage emergent peer scaffolding to keep the learning activities going (related to the principle of maximum peer interactions in Johnson & Johnsons (1994) model of cooperative learning). Each student possesses a piece of resource (a character component) and assumes full control on it (corresponding to the principles of individual accountability and positive resource interdependence). Nevertheless, in order to achieve the game goal (the principle of positive goal interdependence) of forming characters with the rest of the available resources (the 14 components possessed by other students), she will not only need to draw upon her own Chinese character knowledge but also her social skills to negotiate with her peers on group forming (reinforcing the principles of equal opportunity to participate and maximum peer interactions). In the process, she might be persuaded to join another group instead, and in turn learn from the latter in case the proposed character is unfamiliar to her. One important evidence of effective collaborations in the Chinese-PP games is that towards the last three sessions, there was no significant difference in the accumulated frequencies of student-leadership (i.e., proposals of eligible characters) among the HA, MA and LA students. It was no longer important that students in which band had earned more scores or how many groups had been successfully formed. It was the process of collaborations and peer coaching, which had led to learning gains and improvement in collaborative spirits and skills, that counted. Indeed, we observed a healthy balance of competition and collaboration among the students throughout the last three Chinese-PP sessions. When they were engaged in the games, they had the clear goal of forming complex characters in order to improve their scores and rankings. However, they usually would not hesitate to help their peers who did not belong to their groups even though that would bring the latter some competitive advantage. They did so for the sake of self-esteem and game-induced enjoyment in figuring out more characters. During the Chinese-PP games, we set aside some space in the classroom and allowed the students to move around with their phones to discuss with different individuals or clusters of peers. This is the power of mobility for both the devices and the students that resulted in a greater diversity in the students explorations of alternative solutions (the submitted characters). In particular, we facilitated two student-student interactional modes in the game, namely, technology mediated interactions (sending invitations and the visualization of the proposed characters) and face-to-face interactions (negotiation of group forming, peer coaching and discussions on the alterations of the characters). We observed that both interactional modes had been complementing each other so well that the students found no trouble in rapidly and smoothly switching between them. Perhaps the most inspiring discovery is the variation of the game-playing patterns/strategies emerged during the SGM and MGM game rounds. The students adapted themselves well in playing Chinese-PP games in both modes by figuring out different sets of strategies to maximize their winning chances. In the SGM games, they spent more time in composing the most complex characters possible in one shot; in the MGM games, they worked faster and more rapidly in composing and altering characters. Again, the two modes seemed to complement each other as students who engaged in both modes were able to develop both types of skills. Occasionally, they transferred the skill picked up during the SGM games to MGM games, and vice-versa. From Chinese character learnings point of view, the students had also demonstrated their orthographic awareness (Jiang, 2006) through their character alteration activities. Take Figure 4 as an example, we found that all the strategies they collaboratively used follow the orthographic rules of composing characters from different types of components. For instance, when they split into and , and reassembled them with and to form new characters and , not only did they need to know that the semantic components and can only be placed on the left side, they also needed to know that the phonetic components and cannot be placed on the right side, according to the orthographic rule for pictophonetic characters. The group that formed , a three-component character, needed to have the knowledge that component 2 and component 3 must be structured this way, with placed on the bottom, not at the top. Finally, it was not by chance that they reduced to , not to , as they must know that can only be place on the left side. This Chinese-PP game applies the object teaching via graphics, embedding the fundamental orthographic theory and rules in hands-on practice through collaborative learning, therefore enabling learners to implicitly grasp the concepts and rules of character structure through trial and practice (Sun, 2006; Wong, Boticki, Sun, & Looi, 2011). The flexible grouping model further encourages learners to contrast, associate, and discriminate different components in order to generate more eligible characters. Paired with a sound pedagogical design, this type of learning approach can effectively raise learners awareness of the principles of character structure and use this logical comprehension to learn characters in a more active and autonomous fashion.
ISLS
241
ICLS2012
Conclusions
We have unpacked the emergent game playing patterns and strategies exhibited by the students during the Chinese-PP activities in our study. We discovered significant patterns in individual game rounds, spontaneous student grouping, the balancing of competition and collaboration, as well as the varied impacts between SGM and MGM to the students socio-cognitive patterns. All these patterns are pointing to the achievement of the desired learning outcomes of the earning approach to establish the students orthographic awareness in Chinese characters. The novel flexible grouping feature, the mobility of both the smartphones and the students in the game, the joyfulness of game playing, and individual students self-esteem in both winning the game and assisting their peers are some of the critical success factors for the students commendable performances in the games. In the future, we plan to analyze the full set of qualitative data (i.e., the game playing process data in session 1-6) in lower level details to trace and uncover the transformations of the students in both socio-cognitive (the establishment of the orthographic awareness) and socio-constructivist (the improvement of game playing through emergent strategies) aspects. We will perform the analysis in both the lenses of CSCL and second language acquisition theories. In doing so, we hope to achieve better understanding in the general nature of such a flexible grouping model and how it may be applied to the learning of other subject domains. Furthermore, we will extract more conducive interactional patterns and game strategies, and incorporate them into our refined pedagogy in order to make future Chinese-PP players to become better learners and gamers.
Reference
Allen, J. R. (2008). Why learning to write Chinese is a waste of time: A modest proposal. Foreign Language Annals, 41(2), 237-251. Anderson, R. C., Gaffney, J. S., Wu, X., Wang, C. C., Li, W., Shu, H., et al. (2002). Shared-book reading in China. In W. Li, J. Gaffney & J. Packard (Eds.), Chinese Language Acquisition: Theoretical and Pedagogical Issues (pp. 131-155). Amsterdam: Kluwer Academic Publishers. Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Learning Sciences, 2(2), 141-178. Fang, J. Y. (1996). Study on the relationship between character recognition and the lexical knowledge of primary school students. Journal of Primary Education, 9, 211-259. Jiang, X. (2006). Study on the orthographic awareness of Chinese characters by American beginners. In D.-J. Sun (Ed.), Research on character learning as a second language (pp. 470-481). Beijing: Commercial Press. Johnson, D., & Johnson, R. (1987). Classroom instruction and cooperative learning. In H. C. Ean & H. J. Walberg (Eds.), Effective Teaching: Current Research (pp. 277-293). Berkeley, CA: McCutchen. Johnson, R., & Johnson, D. (1994). An overview of cooperative learning. In R. V. A. N. J. Thousand (Ed.), Creativity and Collaborative Learning: A Practical Guide to Empowering Students and Teachers (pp. 31-43). Baltimore, MD: Paul H. Brookes. Li, Q. W. (1992). Chinese childrens development of the strategies in character learning. Master Dissertation, Fu Jen Catholic University, Hsin-Chuang, Taiwan. Li, W. M. (1989). The investigation and implementation of creative thinking learning Chinese characters. Beijing: Peoples Education Publisher. Nussbaum, M., Alvarez, C., McFarlane, A., Gomez, F., Claro, S., & Radovic, D. (2009). Technology as small group face-to-face Collaborative Scaffolding. Computers & Education, 52(1), 147-153. Shen, H. H. (2005). An investigation of Chinese-character learning strategies among non-native speakers of Chinese. System, 33(1), 49-68. Strauss, A., & Corbin, J. (1990). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Newbury Park, CA: Sage. Sun, D.-J. (Ed.). (2006). Research on character learning as a second language. Beijing: Commercial Press. Taft, M., & Chung, K. (1999). Using radicals in teaching Chinese characters to second language learners. Psychologia, 42, 243-251. Wong, L.-H., Boticki, I., Sun, J., & Looi, C.-K. (2011). Improving the scaffolds of a mobile-assisted Chinese character forming game via a design-based research cycle. Computers in Human Behavior, 27(5), 1783-1793. Wong, L.-H., Chai, C.-S., & Gao, P. (2011). The Chinese input challenges for Chinese as second language learners in computer mediated writing: An exploratory study. The Turkish Online Journal of Educational Technology, 10(3), 233-248. Zhang, Z. G. (1987). Chinese characters and reading. Reading News, 8, 7-8. Zurita, G., & Nussbaum, M. (2004). A constructivist mobile learning environment supported by a wireless handheld network. Journal of Computer Assisted Learning, 20, 235-243.
ISLS
242
ICLS2012
Exploratory Study on the Physical Tool-based Conceptions of Learning of Young Students in a Technology-Rich Primary School
Lung-Hsiang Wong, Ming-Fong Jan, Yancy Toh, & Ching-Sing Chai, Learning Sciences Laboratory, National Institute of Education, 1, Nanyang Walk, Singapore 637616. [lunghsiang.wong, mingfong.jan, yancy.toh, chingsing.chai]@nie.edu.sg Abstract: This paper explores young students tool-based conceptions of learning in a technology-rich primary school in Singapore. By examining how young children represent the images of learning through their drawings, we have distilled the types of learning tools that are being accentuated from students emic perspective. We contend that the prevalence, absence or peripheral representation of tools (in particular technology) in students drawings will help us make sense of the collective cultural roles tools play in their world. Interviews are also conducted with students and teachers to further tease out the underpinnings of childrens conception of learning. The content analysis of 183 drawings revealed that technological tools were more prominently featured in mix-achievement classes than high-achievement classes. The reasons for such discrepancy emanate from the distinct learning priorities of and strategies used by these two groups of students. We also extend our discussion to explore the pedagogical implications of our findings with regard to 21st century learning dispositions.
Introduction
The mediating roles of tools in shaping human experience have long been recognized by educators (Bernhard, 2009; Jonassen, 2000; M 1940; Wong, Chen, & Jan, in-press). While the choices of learning tools shape the ller, learners learning experiences, individual learners conceptions of learning may also determine the tools that they choose or prefer in mediating their learning. The proliferation of technology-mediated learning in the past two decades may give rise to the evolution of dialectical relationships between learners, their learning experiences and their learning conceptions. However, research on students conception of learning has not been focusing much attention on the roles of technological tools for learning. Questions like this are important for educators to examine the gaps between theories and practice of technology-based learning. In this paper, we report on our recent exploratory study on young learners conceptions of learning in a technology-rich primary school in Singapore. Our aim is to find out how children in an ICT-enriched school represent the images of learning. In relation, what kinds of tools are being accentuated in the childrens drawings and how do children project themselves as learners when these tools are depicted? We surveyed 183 Primary 3 (3rd grade; 9-year-old) students in their general conceptions of learning by inviting them to draw a picture or write anything on What is learning to me? on a piece of paper respectively. We adopted such a picture questionnaire method due to the target students limited verbal capability for expressing their conceptions of learning. In addition, 38 students were invited for one-to-one interviews as a means to triangulate our interpretation. Preliminary coding and analysis led us to the focus of one specific element in the pictures the physical learning tools (i.e., physically existing entities), the most frequently portrayed or mentioned elements in the pictures. Furthermore, the drawings exhibit certain discrepancies when the pictures submitted by high-achievement (HA) and mixed-achievement (MA) students were compared. As such, we see the potential in making use of the referred tools in achieving deeper understanding in the students conceptions of learning. In this paper, we will delineate our coding and data analysis process in making sense of the picture data. We will also present preliminary findings arisen from our data analysis and relate them to the specific schooling and social context of the student participants. We hope that our report can serve as a foundation for more sophisticated analysis of the relationships between the tools and other elements portrayed in the pictures, as well as shed light on methods to investigate young childrens conceptions of learning.
Literature Review
Learning Tools
The Longman Dictionary refers to tool as something that you hold in your hand and used to do a particular job. This conventional definition implies physical nature of tools, with the purpose of enhancing human physical power, strength and capabilities. Conversely, Vygotsky (1978) argued that learning required two mediational means tangible tools (technical tools) and intangible tools or signs (semiotic tools). At Vygotskys time of writing, technical tools were general physical entities, while semiotic tools were (and still are) the means by which cognitive functions are manifested, such as language, numbers and diagrams. With the advent of ICT, the distinction between tools and signs in modern times, has blurred that is, computers have the ability to mediate cognitive processes (Duffy & Cunningham, 1996).
ISLS
243
ICLS2012
Conceptions of Learning
Conceptions of learning can be broadly defined as beliefs and understanding held by learners about learning (Lai & Chan, 2005, p. 3). One of the most prominent taxonomies on conceptions of learning is proposed by Marton, DallAlba and Beaty (1993) who built on Saljos (1979) work to define 6 hierarchically-related levels of learning: 1) Learning as increasing ones knowledge; 2) Learning as memorizing and reproducing; 3) Learning as applying; 4) Learning as understanding and abstraction of the meaning; 5) Learning as an interpretive process of seeing something in a different way and 6) Learning as the concept of changing as a person. Biggs (1994) re-conceptualized the 6 levels and interpreted the first 3 levels as quantitative outlook, where learners viewed acquisition and accumulation of content as indicators of proficiency; and the remaining 3 levels as qualitative outlook where the focus of learning is coalesced around understanding, associating and meaning-making. Van Rossum & Schenk (1984) visualized the 6 levels in terms of reproduction and construction of knowledge, and expounded that learners who emphasized on reproducing tend to adopt surface learning approaches whereas those with constructive orientation tend to adopt deep learning approaches. Apart from deep and surface learners, there is also another group of strategic learners who are motivated to achieve the highest possible grades. These learners regulate their learning to create a learning environment, maximize their use of resources as well as develop understanding of assessment orientation (Entwistle & Ramsden, 1983). Many studies have explored the inter-relationship between conceptions of learning and various variables such as approaches to learning (Lai & Chan, 2005), self-concepts (Burnett, Pillay, & Dart, 2003), learning outcomes (Martin & Ramsden, 1987), learner autonomy (Chuk, 2004), learners experience of learning (Prosser & Trigwell, 1997), and learning context (Van Rossum & Schenk, 1984). The literature supports the view that students conception of learning can influence their learning strategies or processes, and when these mediating variables are viewed together, they can be a good predictor of learning outcomes. Although there is much fruitful work done in exploring the conceptions of learning and learneroriented influences, there is tardiness in investigating the relationship between students conceptions of learning and their learning tools. Clarebout and Elen (2006) conducted a comprehensive literature review on the tools, defined as non-embedded support devices that were used in computer-based learning environment. After combing the vast body of literature that spanned from 1982-2002, they only surfaced 17 studies that addressed factors influencing tools used. They concluded there is some but not compelling evidence that the tools used by learners are shaped by learner, tool and task characteristics. Of relevance to our study is that high achievement students appear to benefit more from having more control over the tools used and that they use tools more frequently than low achievement students. Clarebout and Elens study is very useful in helping us gain perspectives about the macro trend of tool usage. However, the analysis of the meta-study is not situated within the framework of learners conceptions of learning, thus eschewing on learners emic sense-making perspectives. In view of this gap, we hope our study, guided by the theme of students tools-based conception of learning, will be able to shed light in this under-studied area.
The Research Methodology of Picture Questionnaire

In addition to abstracting the research themes that are related to the conceptions of learning, this section also reviews the methodologies used in prior studies. Most abovementioned studies are conducted using questionnaires or inventories and supplemented by interviews, if any. Our methodology departs from the above in that we adopt childrens drawings as our primary source of data and deploy semi-structured interviews as supplementary evidence. MacPhail and Kinchins (2004) methodological review showed that childrens drawings is an established methodology which had been used extensively in clinical studies as indicators of a childs intellectual maturity and psychological development; examination of childrens understanding of the classroom environment; family relationships and value-laden issues of what is good and bad. The use of childrens drawings is a child-centered procedure and evaluation tool where students have the autonomy to decide what to include without being influenced by researchers frame of reference. More importantly, the method is potentially powerful in conveying students salient range of experiences in association to different teaching and learning environments (MacPhail & Kinchin, 2004; Xu, Read, Sim, & McManus, 2009). Borthwick (2011) contended that childrens drawing can provide multi-perceptivities and encourage teachers to be more reflective in their teaching practices. Lodge (2007) asserted that childrens drawings can be viewed as visual discourse (p.147) which can be read in a similar way to textual discourse. Citing the views of Weber and Mitchell (1996), childrens learning can express that which is not easily put into words: the ineffable, the elusive, the not-yet-thought-through, the subconscious (p.304). Much like using discourse as a major source for research, there are limitations to using childrens drawing as the source of data. Reasons include: 1) childrens drawing are not the royal road of access to internal representations (Krampen, 1992, p. 41) as influences of conventions may affect the depiction of childrens internal model. Too many intervening factors create confounding intricacies, rendering it difficult to equate graphic representations with how internal representation is stored (Jolley, 2010); 2) there may be developmental lag in students drawing abilities and thus affect students production of representations (Jolley,
ISLS
244
ICLS2012
2010); 3) there is a tendency to fall into the trap of conceptual fallacy due to over-interpreting the drawings (Cotterall, 1995). Lange-Kuttner and Thomas (1995) elucidated that there is no consensus amongst experts on how pictures should be interpreted. However, most theorists would not dispute the fact that although childrens perceptions are influenced by convention and experience, some form of linkages do exist between picture perception and perception of real world. This argument stems from the premise that even very young children seem to be able to identify the subject-matter of many pictures without training (p.3). As a corollary to the potential drawbacks and benefits of using drawings as a data collection method, MacPhail and KinChin (2004) recommended the synthesis of both childrens drawings and direct questioning methods to gain insights about students conception. We agree with the authors and have deployed this methodological plurality in our study.
Study Description
Sample
The sample of this study consisted of 183 Primary 3 students (3 rd graders) from 6 classes within a primary school in Singapore. The first two classes admitted the top (HA) students of the level (n = 74), while the rest of the students were randomly assigned to the other four classes, which is known as MA students (n = 109). The school is a neighborhood school that in general admits average students. Nevertheless, it is one of the technology-rich schools in Singapore, with the managements long-term commitment in transforming itself into a 1:1, 24/7 (one-mobile-device-per-student, with 24 hours a day, 7 days a week access) learning environment (Looi, et al., 2010; Wong, Chai, Chin, Hsieh, & Liu, 2011; Wong, Chin, Tan, & Liu, 2010; Wong & Looi, 2010; Zhang, et al., 2010). We conducted the study during April, 2011. By then, the target students had already been accessing to their schools several e-learning portals through their home computers for close to 1 year. They had been using the portals for submitting their homework in digital formats, studying learning materials and/or attempting curriculum-related quizzes. Special arrangement was made for a small number of students who did not have computers at home to access computers at the computer labs after school hours. However, the target students had only episodic experience of 1:1 learning prior to our study. The only 1:1 experience that most of them had was the use of smartphones during a one-off teacher-facilitated educational fieldtrip at the zoo.
Data Collection
We arranged with the school to utilize one hour of class lesson time at each class to conduct the study. Each student was given a piece of A4-size blank paper. We asked them to draw a picture to describe What is learning to me? They were encouraged to write additional text to explain the picture, while writing pure text was also allowed. As such an activity instruction might still be too abstract to most of them, we asked them to close their eyes and imagine that they were learning, and draw or write to describe the picture that came across their mind. We did not give them any concrete example about what they could depict so as not to cause a leading effect in their picture contents. We advised them that this was not an arts assignment and would not be graded for the aesthetic quality. In addition, there was no standard answer; instead, they should depict whatever they personally thought relevant to learning. We also walked around the student desks and probed selected students on what they were drawing if we found their work-in-progress pictures having ambiguous contents. A researcher took field notes throughout the sessions, where such conversations were recorded.
Procedure of Data Analysis

We followed a coding procedure similar to that of grounded theory (Strauss & Corbin, 1990) in order to minimize the influence of preconceived theories. This allows us the possibility to see new light from the collected data rather than conforming to existing theories. Besides, we quantify the in-vivo codes (e.g., exam papers pictured by students) not only to bring out the frequency but also to demonstrate the intensity of related concepts and their meanings to participants. In addition to pictorial representations, we also coded textual descriptions, speech balloons between the different actors depicted in the pictures and reflection clouds of children. The field notes taken during the survey sessions and students interviews were used for triangulation. In addition, we selected 12 HA students and 26 MA students from various classes whose picture contents were either highly representative of many peers (i.e., depicting many common elements) or vastly unique for one-to-one semi-structured interviews. We probed the interviewees about ambiguous depictions and rationales of incorporating the elements in their pictures. We also interviewed 4 teachers from 4 different classes to find out more about their views on their own teaching methods and their students ways of learning. That helped us in better interpreting the pictures. Nevertheless, due to the complexity of the data set, we decided to focus on the category with the highest frequency the tools as our preliminary step in our longer term investigation of this topic. This is because as we are looking at the context of a technology-rich school, our primary focus is on how students will depict their learning tools, in particular ICT tools. The prevalence, absence or peripheral representation of ICT in their drawings can help us make sense of the roles which ICT play in their world. We can then infer
ISLS
245
ICLS2012
pedagogical implications based on the congruency/incongruency between students depicted ICT usage and the schools espoused usage. In order to ensure that such a decision is statistically justifiable, we ran a chi-square test on the number of pictures that depict at least one tool (n = 129) versus those without (n = 35), and yielded 2 = 53.88 (p < .05), which indicates that the amount of the former type of pictures is significantly higher. In particular, we concentrated on coding physical tools but ignored semiotic tools. This is because semiotic tools were scarcely depicted in the data set. The young students could easily relate to the physical tools and articulate how the tools were used for learning; but they in general would not consider semiotic tools as mediating tools, but as part of the contents. Furthermore, we are not looking at the micro analysis of individual students drawing, but common patterns that emerged from the 164 pictures, i.e., what we are investigating are the group data or groups cultural patterns (MacPhail & Kinchin, 2004, p. 89). Semiotic tools are imbued with subjective and heavily contextualized interpretations, thus rendering the generalization process across the 164 pictures ineffectual. We removed all other codes but focused on re-examining the tool-related codes, and worked out the coding scheme as shown in Table 1. The right-most column of Table 1 lists examples of the roles or functionalities of the corresponding tools as depicted or described in the pictures. Table 1: The coding scheme for physical tool-based conception of learning.
First-level category No tool Non-ICT tools Second-level category Personal stationery Code Pen / pencil / eraser / stationery case Paper notebook Ruler Folder Whiteboard / blackboard Pointing stick Desk / chair Homework / worksheets Past year exam papers Book shelf Any book Textbook Storybook Paper dictionary Basketball & net Soccer & net Badminton & net Running mill Skipping rope Equipment for science experiments Plants / seeds / watering container Specimen Piano Violin Assorted music instruments Color pencils / crayons iPhone / other smart-phone iPad / other tablet Calculator e-dictionary MacBook / Notebook computer Desktop computer Television Overhead projector / visualizer Functions Mentioned by Students (Examples) For doing worksheets For note taking at outdoor learning For drawing straight lines in the worksheets For filing the worksheets Teacher use in classroom lessons Teacher use in classroom lessons For sitting still and studying with concentration (either in the classroom or at home) For drill and practice To practice for exams For keeping books For all reading and studying purposes For studying the formal syllabus For easy reading or improving language Assisting language learning or general-purpose writing Basketball playing / training Soccer playing / training Badminton playing / training For keeping fit Playing or training rope skipping For science experiments in the lab For planting activities related to biology classes Exhibits in the science lab Used in piano lessons Used in violin lessons Used in music lessons Used in art lessons or personal drawing For supporting outdoor learning, Internet access or creating simple student artifacts For accessing to Internet or school e-learning portal To support math learning Assisting language learning or general-purpose writing For accessing to Internet or school e-learning portal; doing assignments For accessing to Internet or school e-learning portal; doing assignments; downloading past-year exam papers Learning by watching TV shows Teacher use in classroom lesson
Classroombased tools Furniture Assessmentrelated tools Books & references
Tools for sport activities / physical education Tools for science learning
Tools for music & arts
ICT tools
Handheld devices
Non-handheld devices
Classroombased tools
After re-coding the pictures with the above physical tool-based coding scheme (which were very minor revisions), we split the valid data set into two groups the HA group (72 pictures) and the MA group (92 pictures), based on which classes individual students came from. We ported the coding Excel sheet to the SPSS software and performed chi-square tests to compare the code frequencies of both groups in various categories
ISLS
246
ICLS2012
and sub-categories. In turn, we yielded some statistical findings that may assist us in making better sense of the target students tool-based conceptions of learning, which we will explicate below.
Findings and Discussion

As stated before, we performed chi-square tests on the data set in an exploratory manner after re-coding. A summary of the tests is shown in Table 2. Note that most of the students had depicted more than one physical tool in their respective pictures. The counts listed in Table 2 refer to the number of students who depicted the particular types of tools, not the accumulated frequencies of individual tool types across all the students. Table 2: Summary of chi-square tests on the coded data.
Topic Achievement: HA (n = 72) Count % within HA 16 22.2 48 66.7 3 4.2 5 6.9 53 73.6 7 9.7 Achievement: MA (n = 92) count % within MA 19 20.7 49 53.3 12 13.0 12 13.0 62 66.3 24 26.1 2
No tool Non-ICT tool(s) only ICT tool(s) only Non-ICT + ICT tools At least 1 non-ICT tool At least 1 ICT tool
0.06 3.00 3.83 * 1.62 1.02 7.06 *
* p < .05 In addition, we present two examples of drawings by a HA and a MA students respectively (see Figure 1). Both drawings are supplemented by textual annotations. The text in the HA students drawing at the left is originally written in Chinese (as we borrowed a Chinese lesson to conduct the survey). To benefit international readers, we overlay the English translation on the Chinese text. In this picture, we see (and coded) a whiteboard and a textbook in the top-left frame (classroom scenario), a textbook and an assessment book in the bottom frame (study room scenario), and desks/chairs in all frames. In the MA students drawing at the right, we see both non-ICT tools (pencil, eraser, ruler and assessment book) and ICT tools (computer, accessing to the schools e-learning portal, AskNLearn, which is an assessment-oriented portal) being depicted by the student.
Figure 1. Examples of drawings by a HA student (left) and a MA student (right). From Table 2, we notice that there is no significant difference between the number of HA and MA students who foregrounded non-ICT tools in their drawings (see the columns of non-ICT tool(s) only and at least 1 non-ICT tool). Conversely, general MA students had been far more inclined to highlight ICT tools than their HA peers. In any case, how do the students perceive the affordances of such tools that may be conducive for their learning effort? Indeed, the right-most column in Table 1 may delineate the most prevailing learning strategies in their mind. For example, the personal stationery, the books and references, and especially the
ISLS 247
ICLS2012
assessment-related tools, are mainly employed for assisting in the study of prescribed knowledge or drill-andpractice activities. The classroom-based tools are those used by the teachers in their classroom teaching; in such pictures, students usually portrayed themselves as attentive and obedient students in uniforms, sitting still on their respective chairs next to their desks. Such behaviorist- and transmissionist-inclined depictions (or, in Van Rossum & Schenks (1984) view, the reproduction of knowledge) of the functionalities of the non-ICT tools are also seen in their ICT counterparts. Examples are the use of phones, tablets, notebook or desktop computers to access to the school e-learning portals (containing essentially digital learning materials and online quizzes), doing assignments or even searching and downloading past year exam papers. What makes the use of ICT tools seemingly more open-ended is that they are also depicted by fewer students as the tools for searching for online learning resources, supporting outdoor learning and creating simple artifacts. There are also certain domain-specific tools such as those used for sports activities (or physical education), music, arts and science learning, which are however depicted by fewer students as compared to ICT tools and general non-ICT tools. The findings presented here suggest two salient themes we will discuss more in more depth: the discrepancy in the use of ICT tools between HA and MA students and the missing of social media. Though the two seem to be irrelevant phenomena, they are interconnected at a much deeper level.
The Discrepancy in the Use of ICT Tools between HA and MA Students

Based on Table 2, HA students, students who are academically superior to MA students, tend to associate nonICT tools, especially tools related to contents and standardized tests, with learning. MA students (average students), conversely, think of learning tools more broadly when it comes to the choices of physical tools for learning. Initially we were befuddled by the finding as we assume that high-achieving students should have been savvier than average students on using ICT tools for learning. Assumingly, HA students will use ICT tools as often as MA students, if not more. However, their drawings reveal a different storyICT tools are far less relevant to learning for HA students even when they are situated in an ICT rich school. Our follow-up interviews with teachers from the school shed more light to the interpretation of pictorial data. HA students are much more focused on success in exams and the most frequently depicted learning strategy is to work hard. Therefore, ICT tools, besides their limited affordances in providing access to school portals, exam questions and occasional on-line research, are often seen as distraction by many HA students as they do not serve the direct purpose of helping students score well in high-stake assessments. Oftentimes, the use of ICT tools in these families is limited, if not banned as they are perceived as irrelevant activities. Teachers also shared that HA students, with schedules packed for enrichment activities, have very limited informal learning spaces for other exploratory activities. MA students, conversely, have more informal learning spaces to dabble with technologies and are often savvier than HA students when it comes to computer and information literacies. On example cited by the teachers is that MA students appear to be more attuned to the nuances of technological affordances and are able to figure out how location-based technologies can be harnessed meaningfully on their own within a very short period of time during fieldtrips, as compared to HA students who encountered more problems trying to understand the functionalities of the technology.
The Missing of Social Media

Drawing ICT tools or not, students across HA and MA groups depict a consistent epistemology for how they will use tools for learning in their drawingsas carrying contents (such as books and blackboard), providing access to contents (such as desktop computers and iPad), and performing assessments (such as worksheet, and notebook with the assess to the schools e-learning portals). The finding is somehow expected as research from Kuhn, Cheney, and Weinstock (2000) reveal that students of this age group tend to hold an absolutist epistemology. As absolutists, students tend to identify knowledge as facts discovered by great people, and can be accessed via external sources such as books and the Internet. The unexpected to us is the rarity they depict artifacts that represent social media and participatory culture (Jenkins, Purushotoma, Clinton, Weigel, & Robison, 2009) in their drawings. Rarely do we find students depict how they hang out, mess around or geek out (Ito, 2010) in the world of new media and participatory culture (Jenkins, et al., 2009). Yes, some students do draw iPhone, iPad and computers. But they ways they associate these ICT tools with learning in the classrooms do not reflect the spirit of participatory culture. This finding raises an important question for educators who attempt to bring participatory culture to schools. Students participating in this research are all digital natives born around 2001 and 2002. In other words, their life experience should be parallel with the growth of Web 2.0 technologies. In fact, in the follow-up interviews, students reveal that most of them are using Facebook, Twitter and other social media. But why do students predominantly picture ICT tools as extension of classroom activities? How might the missing of social media and the discrepancy in the use of ICT tools between HA and MA students be connected? Both findings are pertinent to how students perceive the functions of technologies for learning. The missing of social media, together with the consistency in using ICT and non-ICT tools for carrying contents, providing access to contents, and performing assessments, is an indicator about the perceived
ISLS
248
ICLS2012
affordances (Norman, 1993) of technologies, which is pertinent to students learning epistemology (Jan, 2011) and desire for learning (Pea, 1993). The discrepancy in the use of ICT tools between HA and MA students is another revealing case about the perceived affordances of ICT tools based on the students concept of learning. When students perceive learning as equating to performing well in schools, ICT and non-ICT tools are often used as tools for achieving better performance in exams. That is, ICT tools, regardless of their affordances, are used by as content delivery tools like books. This argument supports Entwistle & Ramsdens (1983) notion of how strategic learners will maximize their resources primarily for the purpose of achieving high grades. When tools are used as content delivery tools, efficiency becomes the main concern when it comes to usage. Though ICT tools provide convenience to contents, they are also tools of distractions for parents and many students with just one click away from the targeted contents.
Conclusions
In this exploratory study, we investigate Primary 3 students concept of learning via analyzing their drawings. Specifically, we foreground the analysis of physical tools and the affordances of these tools perceived by students. The purpose of our analysis is not to depict how individual students picture the conceptions of learning, and provides a thick description. Rather, we aim at uncovering collective thinking patterns about students conceptions of learning. We found that these students, who grew up with the omnipresence of ICT, mostly identify non-ICT tools as their preferred tools for learning in the drawings. When computational media are depicted, they are often used to support learning in the same fashion non-ICT tools. Students learning epistemology, manifested via how the functions of physical tools are identified, suggest that these students predominantly view learning as familiarizing themselves with formal subject contents. Perhaps the most revealing finding is that the HA students, as defined by schools, are more inclined to identify non-ICT tools as preferred learning tools than average students. We believe that such an understanding is pivotal if schools wish to transform learning for the 21st century learning skillsa transformation from teaching standardized skills to fostering creativity and innovation (Shaffer & Gee, 2005). With the hefty investment in providing students new technologies, do we also equip students with frameworks for using technologies properly? If students view an iPad, for example, mainly as a tool for easy access to exam questions and contents, it is less likely for them to use iPad as a tool that situates their life experience in a participatory culture. We believe that such a framework for using technologies must encourage students to use technologies for sharing, collaboration, critical knowledge consumption and creative production. In other words, the framework for using technology fosters prosumer (Tapscott & Williams, 2008) identity via transforming epistemology and enculturing students in the practices of participatory culture.
References
Bernhard, J. (2009). Learning through artifacts in engineering education: Some perspectives from the philosophy of technology and engineering science. Proceedings of the 6th European Forum on Continuing Engineering Education, Quality Developmentin Lifelong Learning - in Theory and Use, Rotterdam, Netherlands. Biggs, J. (1994). Student Learning Research and Theory: Where Do We Currently Stand? In G.Gibbs (Ed.), Improving Student Learning: Using Research to Improve Student Learning (pp. 119). Oxford: Oxford Centre for Staff Development. Borthwick, A. (2011). Childrens perceptions of, and attitudes towards, their mathematics lessons. Proceedings of the British Society for Research into Learning Mathematics (pp. 37-42), Bristol, UK. Burnett, P. C., Pillay, H., & Dart, B. C. (2003). The Influences of Conceptions of Learning and Learner SelfConcept on High School Students Approaches to Learning. School Psychology International, 24(1), 54-66. Chuk, J. Y. P. (2004). Promoting learner autonomy in the EFL classroom: the Exploratory Practice way. Proceedings of the Supporting independent English language learning in the 21st Century: Independent Learning Association Conference (pp. 57-74), Melbourne. Clarebout, G., & Elen, J. (2006). Tool use in computer-based learning environments: towards a research framework. Computers in Human Behavior, 22, 389-411. Cotterall. (1995). Readiness for autonomy: Investigating learner beliefs. System, 23(2), 195-205. Duffy, T. M., & Cunningham, D. J. (1996). Constructivism: Implications for the design and delivery of instruction. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 170-198). New York: Simon & Schuster Macmillan. Entwistle, N., & Ramsden, P. (1983). Understanding Student Learning. London: Croom Helm. Ito, M. (2010). Hanging Out, Messing Around and Geeking Out: Kids Living and Learning with New Media. Cambridge: MIT Press. Jan, M. (2011). Fostering learning paradigm shift with game-based learning. Journal of Advanced Technology and Management, 1(1), 47-60.
ISLS 249
ICLS2012
Jenkins, H., Purushotoma, R., Clinton, K. A., Weigel, M., & Robison, A. J. (2009). Confronting the challenges of participatory Culture: Media education for the 21st century. Cambridge, MA: The MIT Press. Jolley, R. P. (2010). Children and Pictures: drawing and understanding: Wiley-Blackwell. Jonassen, D. (2000). Computers as Mindtools for Schools (2nd ed.). Upper Saddle River, NJ: Prentice-Hall. Krampen, M. (1992). Children's Drawings: Iconic Coding of the Envrionment. New York: Plenum Press. Lai, P.-Y., & Chan, K.-W. (2005). A structural model of conceptions of learning, achievement motivation and learning strategies of Hong Kong teacher education students. Paper presented at the AARE Conference, University of Western Sydney. Lange-Kuttner, C., & Thomas, G. V. (1995). Introduction. In C. Lange-Kuttner & G. V. Thomas (Eds.), Drawing and Looking: Theoretical approaches to pictorial representation in children. New York: Harvester Wheatsheaf. Lodge, C. (2007). Regarding learning: Childrens drawings of learning in the classroom. Learning Environ Res, 10, 145156. Looi, C.-K., Seow, P., Zhang, B. H., So, H.-J., Chen, W., & Wong, L.-H. (2010). Leveraging mobile technology for sustainable seamless learning: A research agenda. British Journal of Educational Technology, 42(1), 154-169. M ller, R. H. (1940). American apparatus, instruments, and instrumentation. Industrial and Engineering Chemistry: Analytical Edition, 12(10), 571-630. MacPhail, A., & Kinchin, G. (2004). The use of drawings as an evaluative tool: students' experiences of sport education. Physical Education & Sport Pedagogy, 9(1), 87-108. Martin, E., & Ramsden, P. (1987). Learning Skills or Skill in Learning. In J. T. E. Richardson, M. W. Eysenk & D. W. Piper (Eds.), Student Learning (pp. 155167). Milton Keynes, England: Open University Press. Marton, F., Dall'Alba, G., & Beaty, E. (1993). Conceptions of learning. International Journal of Educational Research, 19(3), 277-300. Norman, D. A. (1993). Things that Make Us Smart: Defending Human Attributes in the Age of the Machine. Reading, MA: Addison-Wesley. Pea, R. (1993). Practices of distributed intelligenceand designs for education. In G. Salomon (Ed.), Distributed Cognitions (pp. 47-87). Cambridge, U.K.: Cambridge University Press. Prosser, M., & Trigwell, K. (1997). Understanding Teaching and Learning: The Experience in Higher Education. Buckingham: Open University Press. Saljo, R. (1979). Learning from the Learners Perspective. I: Some Common Sense Conceptions (Vol. Rep. No. 76): Goteborg, Sweden: University of Goteborg, Institute of Education. Shaffer, D. W., & Gee, J. P. (2005). Before every child is left behind: How epistemic games can solvethe coming crisis in education WCER Working Paper No. 2005-7. Madison, WI: University of WisconsinMadison, Wisconsin Center for Education Research. Strauss, A., & Corbin, J. (1990). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Newbury Park, CA: Sage. Tapscott, D., & Williams, A. D. (2008). Wikinomics: How Mass Collaboration Changes Everything. New York: Portfolio Hardcover. Van Rossum, E., & Schenk, S. (1984). The Relationship between Learning Concepts, Study Strategy and Learning Outcome. British Journal of Educational Psychology, 54, 7383. Vygotsky, L. (1978). Mind in society. Cambridge, MA: Harvard University Press. Weber, S., & Mitchell, C. (1996). Drawing ourselves into teaching: Studying the images that shape and distort teacher education. Teaching and Teacher Education, 12, 303-313. Wong, L.-H., Chai, C.-S., Chin, C.-K., Hsieh, Y.-F., & Liu, M. (2011). Leveraging Ubiquitous Technology for Seamless Language Learning: From Move, Idioms! to MyCLOUD. Proceedings of the mLearn 2011 (pp. 231-239), Beijing, China. Wong, L.-H., Chen, W., & Jan, M. (in-press). How artefacts mediate small group co-creation activities in a mobile-assisted language learning environment? Accepted by: Journal of Computer Assisted Learning. Wong, L.-H., Chin, C.-K., Tan, C.-L., & Liu, M. (2010). Students' personal and social meaning making in a Chinese idiom mobile learning environment. Educational Technology & Society, 13(4), 15-26. Wong, L.-H., & Looi, C.-K. (2010). Vocabulary learning by mobile-assisted authentic content creation and social meaning-making: Two case studies. Journal of Computer Assisted Learning, 26(5), 421-433. Xu, D., Read, J., Sim, G., & McManus, B. (2009). Experience It, Draw It, Rate It Capture Childrens Experiences with Their Drawings. Paper presented at the IDC 2009, Como, Italy. Zhang, B. H., Looi, C.-K., Seow, P., Chia, G., Wong, L.-H., Chen, W., et al. (2010). Deconstructing and reconstructing: Transforming primary science learning via a mobilized curriculum. Computers & Education, 55(4), 1504-1523.
ISLS
250
ICLS2012
Alternate Reality Games: Platforms for Collaborative Learning

Elizabeth Bonsignore, Kari Kraus, June Ahn, Amanda Visconti, Ann Fraistat, Allison Druin, University of Maryland, College of Information Studies, 4105 Hornbake Bldg, South Wing, College Park, MD 20742 Derek Hansen, 265N CTB, Brigham Young University, Provo, UT 84602 ebonsign@umd.edu, dlhansen@byu.edu, kkraus@umd.edu, juneahn@umd.edu, visconti@umd.edu, ann.fraistat@gmail.com, allisond@umiacs.umd.edu Abstract: We review the first iteration of an Alternate Reality Game (ARG) designed to engage middle school students in the interpretive process behind history and the mathematical roots of cryptography and cartography. Building from the core characteristic of ARGs as collaborative sense-making systems, we sought to integrate elements of effective cooperative learning environments, such as positive interdependence, into our game design. We detail how cooperative learning constructs were designed into the narrative and game mechanics, and how evidence of these features emerged during play. We found that the jigsaw puzzle design of game challenges supported collaboration between groups; daily, whole-class discussions strengthened group processing; and real-time, chat-based dialogue among students themselves and students with an in-game character enhanced promotive interaction (e.g., providing positive reinforcement). However, we did note weaknesses in positive interdependence within groups. We conclude with design implications for future iterations of the ARG.
Introduction
Well-designed games often provide situated learning contexts that support deep learning: they include rich, compelling narratives; require players to engage in individual and collaborative problem-solving activities; and offer opportunities for players to assume authentic, community-valued roles as investigators, engineers, archeologists, artists, etc. (Gee, 2008; Shaffer et al., 2005). As socially situated, interactive media experiences, Alternate Reality Games (ARGs) incorporate several of the features that promote deep learning (Gee, 2008): they provide compelling narrative contexts and require collaboration to complete. An ARG is an immersive story-game hybrid whose core mechanics are collaborative problem-solving and participatory storytelling. The narrative context of an ARG is not bound by any one communications platform or media type: its story fragments can be scattered and hidden in websites, phone calls, text messages, or books. Because of the ways in which an ARGs narrative elements are hidden, players must collaboratively hunt for clues, solve puzzles, and synthesize disparate information to assemble and advance its ever-evolving storyline. An ARG rendition of a Sherlock Holmes mystery, for example, might involve Watson sending players a text message with GPS coordinates to the next clue, which the players must find, decode and email to Holmes before the story can continue. Just as the ARGs narrative is distributed across real-world platforms, players use real-world technologies such as blogs, chat, and online community forums to collaborate as they make sense of the unfolding story. A well-designed ARG engages learners in 21st century literacy practices, such as evaluating and sharing information across multiple media, analyzing complex problems, and using new media tools to re-interpret existing content or create new expressions (Bonsignore et al., in-press; Jenkins et al., 2006). Although the inherent community-based, collective intelligence characteristic of ARGs holds potential to promote socially situated learning (Jenkins et al., 2006), player participation is often lopsided. Active players, those who find clues, solve puzzles, and synthesize information to advance the story, represent a fraction of the total players involved in an ARG (Gurzick et al., 2010; Kim et al., 2008). The majority are bystanders or lurkers who do not participate in the story except at a minimal level, such as passively experiencing it through the latest story updates from a player community website. To mitigate this uneven active participation ratio and to realize more fully the potential of ARGs in education-based contexts, we are investigating ways to incorporate cooperative learning constructs (e.g., Johnson et al., 1994) into their design. In this paper, we examine the effects of cooperative learning designs on middle school students who participated in the first iteration of an ARG that engaged them in the interpretive process behind history and the mathematical roots of cryptography and cartography. Building from the core characteristic of ARGs as collaborative sense-making systems, we sought to integrate elements of effective cooperative learning environments, such as positive interdependence (Table 1), into our game design. Using a design narrative approach (Hoadley, 2002), we detail 1) how these cooperative learning constructs were designed into the narrative and game mechanics, and 2) how evidence of their effects emerged during game play.
Research Landscape and Related Work

Game-based learning studies over the past decade have investigated the knowledge-building activities of players in single and multiplayer videogames (Squire, 2006; Steinkuehler & Duncan, 2008); the authentic scientific
ISLS
251
ICLS2012
inquiry practices of players in multi-user virtual environments (MUVEs) (Barab et al., 2007; Dede et al., 2004); and the ways in which role-playing activities (playing an environmental scientist) enable players to adopt epistemic frames, or ways of knowing and working within specific in-game contexts that can transfer to realworld contexts (Shaffer et al., 2005). Socially situated opportunities for learning are embedded in the narrative frameworks for all of these games (e.g., players help a civilization similar to ours to solve a critical environmental issue). However, design-based techniques that directly engineer collaboration into these games have been largely a secondary, not central design concern. Further, most of this research has been focused on games that employ 3D gaming or virtual reality environments. Research by Klopfer, Perry, Squire and Jan (2005) on augmented reality game design represents one of the few studies that 1) focused explicitly on ways to embed cooperative learning elements into play and 2) moved the game-based environment beyond the desktop. Both augmented and alternate reality games remove traditional videogame requirements to remain tethered to a desktop or to invest in specialized tools necessary in virtual environments. Although augmented reality technologies can be used within an ARG to enhance the immersive experience of game-play, ARGs do not rely on augmented reality effects in their design or play. As real-world collaborative sense-making systems, ARGs can take advantage of relatively low-cost social media technologies and tools, such as online community wikis, forums, and blogs. We seek to extend Klopfer et al.s (2005) collaborative learning focus for augmented reality games by investigating the effects of integrating cooperative learning constructs into ARGs.
Alternate Reality Games (ARGs): collaborative, counterfactual learning experiences

Like their videogame counterparts, ARGs were initially developed for entertainment; however, they are garnering increasing attention as a potentially transformative vehicle for education, specifically in terms of the ways in which they promote 1) critical thinking and information literacy skills, and 2) collaborative problemsolving and sense-making (e.g., Johnson et al., 2010; Whitton, 2008). A small number of ARGs have already been developed with educational goals in mind, such as World Without Oil, which asked players to imagine their world in the midst of a global oil crisis, and Black Cloud, which engaged at-risk high school students in scientific investigations of environmental issues in their local neighborhoods (Niemeyer et al., 2009). ARGs invite players to imagine and inhabit a past, present, or future alternate world, requiring that they look critically at the information they find, constantly asking what if questions. Counterfactual thinking involves imagining what might have been, or considering what-if alternatives to specific events (Byrne, 2007). It can be a tool that fosters investigative reasoning across multiple disciplines, including science, history, and business (Gaglio, 2004; Hawkins & Pea, 1987; Owens, 2010). By embedding game play and story seamlessly into existing, everyday technologies, ARGs neither acknowledge nor promote the fact that they are games. The lines between whats real and whats not are unclear, fostering what-if interrogation. Known as the This is Not a Game (TINAG) principle by ARG designers, it can be the primary apparatus for prompting critical, counterfactual thinking and information literacy practices, because players are responsible for distinguishing truth from fiction. Mandatory collaboration is another primary design goal for ARG designers, who strive to create puzzles and challenges that no single person could solve on their own (McGonigal, 2008, p. 203). ARG designers immerse their players in hands-on experiences. Players have a central role in assembling the story world, by collecting and connecting pieces of the games narrative, then sharing their interpretations with each other. Through role-playing, videogames have offered similar collaborative means for players to participate as contributing members of virtual communities, such as SWAT teams or environmental research teams (Gee, 2008; Klopfer et al., 2005). In contrast, players in ARGs do not take on roles as digitally rendered avatars or specific character types. Instead, players inhabit the game space as themselves. As ARGs present the evidence of the story, and let the players tell it (McGonigal, 2008, p. 202), players often begin to view themselves as detectives, storytellers, and problem-solvers. Still, each individual player must be afforded opportunities to feel as though her existing and emergent individual skills can contribute to the communitys efforts. Else, she will remain one of the passive bystanders who watches the story unfold, but makes little effort to participate. Cooperative learning research offers techniques to help maximize the potential for ARGs to support problemsolving and critical thinking at individual as well as community levels.
Essential elements of Cooperative Learning

Cooperative learning, the strategic use of small groups working together to accomplish shared learning goals while maximizing individual learning, has been shown to be a highly effective instructional practice for over 40 years (Johnson & Johnson, 2009). Although diversity in cooperative learning methods exists between 8 and 10 variations have been applied and studied since the 1970s all of the methods have been found to produce higher achievement than competitive or individualistic learning approaches (Johnson & Johnson, 2009; Slavin, 1991). Improved academic performance is not the only indicator of the effectiveness of cooperative learning, however: improved psychological health, self esteem, and task-oriented and personal social support have been consistently confirmed as well (e.g., Johnson & Johnson, 2009; Slavin, 1991). As a repertoire of successful
ISLS
252
ICLS2012
learning techniques, cooperative learning offers an additional set of conditions that can translate into design principles for good games (Gee, 2008, p. 37). Cooperative learning approaches require group members to take individual responsibility for specific aspects of interdependent tasks (positive interdependence), and to promote each others successes as they work toward the groups shared goals (promotive interaction). These techniques offer ARGs a means to ensure interdependency among the games interactive puzzles while allowing players to contribute as individuals, based on a combination of their personal interests and their ARG communitys needs. This is the balance that we sought to achieve through both narrative and game mechanics in our ARG, the Arcane Gallery of Gadgetry.
The Arcane Gallery of Gadgetry (AGOG)

The Arcane Gallery of Gadgetrys mythology is grounded in the history of the U. S. Patent Office. From 1836 to 1932, the Patent Office building, then located in Washington, DC, was dubbed a Temple of Invention. Thousands of miniature models of patent submissions were put on display, and thousands of visitors came to view first-hand the ingenuity of their fellow citizens (Robertson, 2006). A number of historically significant figures were associated with the Patent Office, including Abraham Lincoln, Walt Whitman, and Clara Barton. In 1877, a fire damaged over 100,000 patent models housed there (Robertson, 2006). The fire, whose cause has never been completely resolved, provided the AGOG designers with the means to traverse fiction and reality: we could use this gap in historical knowledge to create counterfactual paths, filled with documents, inventions, and personalities that players would have to investigate and resolve.
Figure 1: From left to right: 1a) Working notes and artifacts that were posted along an "evidence" wall. 1b) Sample chat messages. 1c) Sample encoded document the students decoded. 1d) A student works on Morse-encoding the final message.
Goals
Throughout the design process of AGOG, we applied the notion of creating designed experiences, rather than content-delivery vehicles (Squire, 2006). Our goal was to have students view historical events and artifacts as possibility spaces that can be actively interpreted and reconstructed, not inevitabilities that could simply be memorized and retold. One design goal for building the historically grounded mythology of AGOG was to enable students to discover fault lines that exist in extant historical records, such as the 1877 Patent Office fire. We used these gaps to focus students attention on unanswered questions, or questions open to interpretation (e.g., What might have caused the Patent Office fire? How do you track and restore lost records? How do ideas get patented?). In this way, we sought to subtly foster student engagement with the National Center for Historys Historical Thinking Skills (Lesh, 2011), such as differentiating between historical facts and historical interpretations, and challenging arguments of historical inevitability (p. 17). Another primary design goal was to find ways that would not only encourage the collaboration typically experienced in ARGs, but also provoke interdependencies among players so that they would distribute team tasks more equally among themselves. To ensure that no one player could collect and reconstruct all the information required in the game timeframe, we distributed story content across multiple media types and tools, and embedded interdependent puzzle components within these story fragments. Historic maps contained over a hundred locations and landmarks into which we embedded several clues; several cryptographic key phrases were split across historic documents; a simple telegraph had to be reconstructed and tested; and archival data had to be evaluated to solve a logic puzzle about historical figures that were part of the games narrative.
Participants and Methods

AGOG is part of a larger, design-based research initiative that explores the potential of ARGs to create authentic contexts for collaborative learning and participatory design. This first iteration, AGOG Season 1, took place over a 2-week period in a public school in the U.S. Sixty 8th graders started the ARG, with about 55 students (13-15 years old) participating consistently. The students played during their American History class sessions (50 minutes each). To accommodate computer lab scheduling and to facilitate classroom management, the students remained in their respective class sessions (27 students in one class, 33 in the other), with only one instance of the game being played across both groups. Fewer boys participated than girls: 44% boys (26), 56%
ISLS 253
ICLS2012
girls (34); half the students were eligible for the schools free and reduced meal program; and six percent had limited English proficiency (LEP). Regarding ethnicity, 43% (26) of the students identified themselves as African American, 37% (22) as Caucasian, 13% (8) as Hispanic, one as Asian American, and four as mixed race. A teacher and school librarian worked with our research team throughout the game. We followed a multi-method case study framework (Yin, 2003). The main case involves the design and effects of AGOG in a formal education setting; the embedded unit of analysis for this paper focuses on student participation and collaboration. Our investigation covered several levels of student participation in terms of data collection: 1) observation and field notes on all in-class sessions, 2) brief daily summary meetings among the researchers and teacher before and after classes; 3) online, print, and physical artifacts created by students and in-game characters as part of AGOG; and 4) a post-game survey and discussion with students. We databased all of the students online interactions (e.g., chat interactions, wiki entries), as well as their post-game survey responses (e.g., some demographic data, free-form written responses to open-ended questions). The goals of our data analysis were to craft a design-based account of the students interactions with AGOG, while at the same time using our findings 1) to inform the integration of cooperative learning techniques into future iterations of the game, and 2) to develop design-based theories about their integration into immersive learning experiences like ARGs (cf. Barab et al., 2007; Hoadley, 2002). The essential elements of cooperative learning (Table 1, Johnson et al., 1994) formed our initial coding frame. For example, if one student encouraged another via chat, it was coded as an instance of promotive interaction. We followed a constant comparative analytic approach (Boeije, 2002). Each data source (e.g., blog posts) was first analyzed as a single set. Then, coding comparisons were made across sources (e.g., survey responses, blog posts, chat interactions) as a means of data triangulation.
Implementation
The game began when the students were recruited by university researchers to help inspect a set of historical artifacts that had allegedly been discovered by staff members from the Smithsonian Museum. After receiving background on the Junto, an actual but secret philanthropic society established by one of Americas founding fathers (Benjamin Franklin), the students accepted an invitation to be part of a modern version of the Junto, called JENIUS (Junto of Enlightened Naturalists and Inventors for a United Society). They discovered that JENIUS is an underground extension of the original Junto, and that its purpose is to curate a special subset of covert Patent Office designs known as the Arcane Gallery of Gadgetry. Most of the games narrative content and missions were presented and stored across two websites: a rabbit hole public site and a passwordprotected private community site. All students could see each others work, both online and on a working wall of evidence on which students and researchers posted ongoing questions, ideas, and findings (Figure 1a). JENIUS membership is distributed across four Orders, or community sub-groups. Each Order provides a different, but interrelated function. Archivists are responsible for determining the authenticity of documents and artifacts, and maintaining accurate records. Cryptographers are responsible for ensuring the security of JENIUS artifacts, through their knowledge of codes and ciphers. Inventors are responsible for maintaining existing JENIUS artifacts, as well as creating new designs. Surveyors are responsible for maintaining the societys maps and charts, along with other location-based data, such as the geographic coordinates of JENIUS artifacts that must remain distributed to ensure they do not fall into unsuspecting, non-JENIUS hands. JENIUS Orders provided the game-based means for students to band into interdependent groups. Students self-selected into Orders, based on an online Orders Quiz developed to help them decide which specialty was the best fit. The game was staged in two phases: a training phase and the final mission. During training, students worked to acquire various Order skills, such as using geographic tools like gazetteers and Google Maps (surveyors), searching patent and biographical databases (archivists), making an electric circuit (inventors), and learning about cryptographic keys, Morse code, and encryption using mono- and poly-alphabetic substitution (cryptographers). Students earned badges for every mission they completed, and a final certification badge when they completed all three missions for a specific Order. Badges for completed missions were posted on their profile pages. The training phase was intended to promote collaboration within groups and emphasize individual accountability (public posting of badges, encouraging and helping fellow Order members). The JENIUS students final mission required the skills of members from each Order, and entailed decoding, decrypting, and reassembling a message that had been scattered across documents they uncovered through their research. Surveyors pinpointed the locations of clues, cryptographers decrypted the clues, archivists validated and reassembled the contents of the decrypted message, and inventors repaired a notional telegraph and created a Morse-encoded recording of the message. The students would complete the game successfully only by collaborating across Orders as they applied the individual skills they had acquired. The game narrative was shared with students via two in-game characters: the JENIUS Ambassador and April Gravure. The fictional JENIUS Ambassador, a mysterious, omniscient leader in the society, provided information to players at key, pre-scripted stages in the game narrative (e.g., Introduction; Final Mission Orders; end-game congratulations). April was a fictional 21-year old college student who needed the students help to send the message that would successfully end the game. April was the games protagonist by proxy, a phrase
ISLS
254
ICLS2012
used to describe a character who seems to discover the story in tandem with the players (Anderson, 2008). The protagonist by proxy is a staple interactive dynamic in ARGs. During game-play, the player is presented with the same artifacts and information such as URLs, copies of documents, email as the in-game protagonist. Players have the chance to unearth secrets that this character may not have, but needs, and s/he has the means to communicate new information to the players. Moreover, the protagonist often models productive informationseeking and problem-solving behaviors that the players can emulate. Players often see the in-game protagonists as people just like them, and are motivated to regard them as both mentors and investigative partners. These characters were not avatars: members of the AGOG design team played the JENIUS Ambassador and April. We used the following media channels to deliver story fragments over the course of the game (see http://www.arcanegalleryofgadgetry.org/time_files/slide0003.htm for a visualization): Videos and text-/image-based blog posts were used by April G. to reveal secrets about the JENIUS Academy and clues to the final mission, as the students advanced through the game. Podcasts were the only communications used by the JENIUS Ambassador. As a stereotypical authority figure, his communications were one-way only; he did not engage in dialogue with the students. Facebook-like Status Wall updates were used as an informal, live chat mechanism. These updates were visible to all students, and posted in a prominent place on the JENIUS community website. Incomplete Wiki entries were seeded on the JENIUS website, providing hints and clues. The community wiki also included copies of JENIUS documents (e.g., historic maps, community ledgers) and images of artifacts (e.g., the Kairograph, an AGOG invention fashioned after the telegraph). April created some of the wiki entries, modeling ways to compose informative archival documents and accurate geographic data. The students could use any of these media during game-play. For example, in their first mission, all students created and posted videos of themselves taking the same oath to scientific and moral inquiry that Benjamin Franklins original Junto members used. Throughout the game, they also used the Status Wall updates to chat with April and each other, created blog posts, and edited wiki entries to complete missions and share information with their classmates. The research team, participating as fellow JENIUS members with the students, also used these media to help point players toward clues, and to model information literacy practices. At the beginning of class sessions each day, we would gather as a community to summarize events and activities of the day before, and introduce new plot points (via videos/blog posts by April G. or podcasts from the Ambassador). Students could ask for help or share information during these group processing sessions (Table 1), which lasted 10-15 minutes. Just before the final mission, and also in the post-game debrief, we allotted extra group processing time for the students to share burning questions or concerns that they had (~30 minutes). Throughout the game, students also posted any questions, concerns, or evidence they felt they should share with the community on big paper that became our wall of evidence (Figure 1a). Table 1: Elements of Cooperative Learning mapped to AGOG design features. Essential elements of Cooperative Learning How each was designed into AGOG Season 1 Individual Accountability Each student was required to complete a series of Individual performance is assessed and shared with increasingly complex problem-solving missions to both the individual and group. Individuals are held become certified in a specific Order. Badges were responsible for contributing to the group success. awarded and posted on individual member profiles for each mission completed. Positive Interdependence Members from each JENIUS Order had to solve Group members believe they are linked with each individual clues that, taken together, form the endother so that the success of one depends on the game solution. success of all (and vice versa). Promotive Interaction Students were encouraged to share information via Individuals encourage each others efforts to achieve the community chat feature, blogs and wiki, as well the group goals by sharing resources and providing as on the big-paper posted on a community wall positive, constructive feedback to each other. of evidence in the lab (Figure 1a-1b). Group Processing Daily whole group circle time was built into the (and associated constructive controversies) beginning of class to discuss individual mission Groups periodically reflect on how well they are status and to review plot updates. During class, functioning and how they may improve their researchers worked with small groups. An extensive processes. These sessions often reveal intellectual session was held midway through the game, just controversies, which, when managed constructively, before the final community mission was revealed to can promote active searches for more information, players. and a reconceptualization of knowledge/reasoning.
Findings
Table 1 summarizes the cooperative learning elements that we sought to design into AGOG. Key examples of the ways in which we saw these elements emerge during game-play are detailed in this section. We provide
ISLS 255
ICLS2012
evidence of the target behaviors that cooperative learning structures promote, such as providing mutual support when difficulties are encountered, awareness of ones interdependence with the larger community, sharing resources, and celebrating individual and joint successes. All student names are pseudonyms. Individual Accountability. We found that the training mission format worked well to support individual accountability. The training missions allowed us to embed content-based and information literacy lessons into the game in an authentic way, and enabled individual assessment and feedback. All students were very proud of their badge accomplishments, as evidenced by their Status Wall updates to each other and April (Figure 1b). In the post-game survey, almost half of the students noted that the missions and badges were the most fun, challenging, or surprising aspects of AGOG. Of these, 20% mentioned that the missions were both challenging and fun. One student let his colleagues know how he was doing in training via this short blog post: Anton (Day 4): Wheww!!! Just finished two of the missions of cryptographer. Very happy {kinda gettin into this game}. Another student shared both her frustration and success with April via chat as she decrypted an excerpt from the 14th Amendment, on due process (an American History content requirement): Myra (at the start of a class session, Day 3): i Got A 67% i Dont Get It. April G >> Myra: Hey, that's not bad, thoughyou're almost there. Ask the JENIUS people for help on that last question. Myra >> April G (toward the end of the class session): i Got 100% (: April G >> April: 100% --That's cool! You get a new badge!!! :) Myra >> April G: I Know, Right Positive Interdependence was most visible across groups during the final mission, because the interdependence of the Order tasks was strong (e.g., cryptographers decrypt a clue that surveyors pinpoint as the next clues location). One student captured her analysis of AGOGs increasingly interlocking parts as follows: There were 3 [missions] you played and then a master level that you have to use the skills you learned from the first 3. The last level is a group level where you put all of your skills together. Our wall of evidence proved particularly useful for students during daily group discussions, as they made visual cross-Order connections from individual Order discoveries (e.g., the same Ouroboros symbol imprinted on both surveyors' maps and cryptographers' clues). Cross-Order collaboration also occurred online, as students created blog posts to share information, or posted status updates asking for help from members outside their own Orders (samples below). A few students also logged in from home, or otherwise outside of their regular class sessions. Ben05 (Day 5, decoding April's notebook): Who is a cryptoghrapher? I NEED UR HELP! Ginny >> Claire746 (over the weekend): hey i have to tell you some clues i have found out Claire746 (over the weekend, after she had decoded a critical clue and wanted to solve more before class resumed Monday/Day 6): hahaim going solo! im trying to figure other things that i dont even know can someone help? Surveyors!!!! helpp!!!!! Individual accountability and positive interdependence within groups was more problematic due to the Order self-selection process coupled with the individual-only (versus group) badge awards. We allowed students to choose their JENIUS Orders because 1) in most ARGs, players self-organize into teams based on skills they feel they can contribute, and 2) we wanted the students to identify personally with their Order. To encourage the formation of informal, interrelated sub-groups within Orders, participating members of the design team encouraged students who had completed missions quickly to work with members of their Order who were struggling. This worked well in one class, because the distribution of students across Orders was fairly even (7 archivists, 7 cryptographers, 6 inventors, 7 surveyors). In the other class, an overwhelming number chose to be cryptographers (17), and only one was a surveyor. Consequently, training could be difficult to manage since students worked at different paces, and several students did not feel accountable to earn badges for their Order. Promotive interaction, in which students share resources and highlight each others success, appeared strongest during the final mission. During the training phase, in-game characters modeled promotive interaction for students. For example, April posted encouraging status updates (Figure 1b) and shared clues via videos, the Ambassador congratulated students on their progress in his podcasts, and participating members of the research team encouraged students. By the final mission, students were engaging in their own promotive interaction efforts, as shown in the Status updates (above) and in the blog posts shown below. In the post-game survey, over half the students commented that they had to work together and share information to complete the game. The stories of Ben05 and Claire746 offer evidence for both positive interdependence and promotive interaction. Early in the game, the teacher relayed to us that Claire746 typically did not participate much in class, yet her blog posts, status updates, badge certifications and efforts in class to find and decrypt clues tell a different story. In her post-game survey, she responded that she enjoyed how real [the game] felt and that the most fun and challenging part to her was when we felt like a detective figuring out things. During the final mission, she posted several blogs to share information she had found with her classmates. She also relayed her knowledge about navigating and working with the media on the site and her desire to collaborate via chat:
ISLS
256
ICLS2012
Claire746 (Day 7): If you go under collaborate-->wiki-->unsolved questions-->message to charles mason (CM) to see the message or to -->cabinet of curiousities document to find the hidden key phase: "YOU DELAY BUT TIME WILL NOT. i dont know what A PLACE FOR EVERYTHNG means can anybody help? Ben05, another quiet student, initially shared his frustration at the difficulty of his first mission and his elation at finally completing it (Figure 1b). By the final mission, he was one of the games most prolific bloggers: Ben05 (Day 6): As you know, last week we observed April G's notes on her phone recording. The morse code translated out to be: Messages Across Time. This is obviously linked to the Kairograph, which can send messages in the past, present, and future. SCAR is obviously on the hunt for this. We need April to write that message on the 15th!!!!! We need to work together for clues & find why SCAR wants April. You can talk to me about my blog at Ben05. Ben05 (Day 7): Me and Anton just got done decoding Walt Whitman message. It states: Time Lost is never found again. This is obviously leading to something, but what? We need more clues!!!!!!!!! Group processing. Questions about TINAG dominated interactions between the design team and the players during group processing discussions, highlighting the responsibility that educators have to balance the drama of uncovering new nuggets of history with the academic imperative to remain trustworthy models. Still, these TINAG talks also gave players an opportunity to debate and share insights with their peers. One student related AGOGs mechanics to the interlocking layers of fiction and reality in the film Inception, while another explained to her classmates that what they were doing was imaginary real. Yet another suggested that some of the information we were working with was like beliefs (i.e., opinions, and not always factual or grounded in evidence). We viewed our community TINAG discussions as opportunities for constructive controversy (Johnson et al., 1994), not as conflicts to be avoided or suppressed. As described by Johnson et al. (1994), constructively managed conflicts promote: an active search for more information, a reconceptualization of ones knowledge and conclusions, and, consequentlyhigher-level reasoning strategies (p. 29).
Design Implications and Conclusions

To help foster positive interdependence and accountability within groups (Orders) in future iterations of AGOG, we plan to design at least one training mission that requires two to three players to work together, with opportunities to earn Order-level badges (i.e., each member receives both an individual and team award for completing a training mission). To help students get a sense that their individual training efforts will contribute to the overall story and final community mission, we also plan to include more narrative information from ingame characters during the training phase. Finally, a long-term ARG design goal is to mitigate inherent replayability and scalability issues (Bonsignore et al., in-press). One option we are investigating involves the development of a repertoire of design patterns (Alexander, 1977) that can be used to create counterfactual, playable extensions from gaps in existing historical records or scientific knowledge (Bonsignore et al., 2012). Counterfactual design patterns could also be used as a participatory cultures design strategy (Jenkins et al., 2006) for developing in-game challenges that invite players to craft historically grounded extensions to the existing AGOG story world. Our data analysis reflected that we made strides toward more balanced participation from several students whose class interaction was typically underrepresented. We also believe that AGOG made an impact on the students perceptions of history, and their efforts to interpret information. We found that the points at which the games mythology blurred the lines between fact and fiction prompted them to move beyond rote memorization of content. Instead, they began to question, analyze, and make hypotheses about the data presented. In their post-game surveys, several players (~15%) mentioned that they felt like spies or detectives, helping to "save the world," and "saving the past." The ARGs dynamic, puzzle-quest nature engendered engagement as well as collaboration. Students noted that they felt AGOG made history interactive, and gave them opportunities to choose what to study: it let you learn more of the past and interacts with you and is fun; there were many jobs we could choose from. It was really fun; and we learned about codes, map reading, and old inventions. I invented things. Others observed that history was not just a facts-only or events-only endeavor, but required inter-disciplinary work: I was surprised we had to do a little math. A few students expressed frustration at being asked to move beyond regurgitating answers from textbooks they had previously been taught never to question. However, many expressed newfound awareness of historical thinking as an investigative process, involving detective work and interpretation. In particular, the porous boundary between the fictional game world and the real world (TINAG) provoked students to engage in constructive controversies with each other. As one student summarized in her post-game survey, It was cool. It was a strategizing game that keeps you thinking.
References
Alexander, C. (1977). A pattern language: towns, buildings, construction. New York: Oxford Univ. Press.
ISLS 257
ICLS2012
Anderson, M. (2008, Nov 12). An Interview with JC Hutchins: Personal Effects. ARGNet: Alternate Reality Gaming Network. http://www.argn.com/2008/11/an_interview_with_jc_hutchins_personal_effects/ Barab, S., Sadler, T., Heiselt, C., Hickey, D., & Zuiker, S. (2007). Relating Narrative, Inquiry, and Inscriptions: Supporting Consequential Play. Journal of Science Education & Technology, 16(1), 59-82. Boeije, H. (2002). A Purposeful Approach to the Constant Comparative Method in the Analysis of Qualitative Interviews. Quality and Quantity, 36(4), 391409. Bonsignore, E., Kraus, K., Visconti, A., Hansen, D., Fraistat, A., & Druin, A. (2012). Game Design for Promoting Counterfactual Thinking. In Proc. CHI 2012. ACM Press. Bonsignore, E., Hansen, D., Kraus, K. M., & Ruppel, M. (in-press). Alternate Reality Games as Platforms for Practicing 21st Century Literacies. International Journal of Learning and Media (IJLM). Byrne, R. (2007). The Rational Imagination: How People Create Alternatives to Reality. Cambridge, MA: MIT. Dede, C., Nelson, B., Ketelhut, D. J., Clarke, J., & Bowman, C. (2004). Design-based research strategies for studying situated learning in a multi-user virtual environment. In Proc. ICLS 2004, 158-165. Gaglio, C. M. (2004). The Role of Mental Simulations and Counterfactual Thinking in the Opportunity Identification Process. Entrepreneurship Theory and Practice, 28(6), 533-552. Gee, J. (2008). Learning and Games. In K. Salen (Ed.), The Ecology of Games: Connecting Youth, Games, and Learning, 21-40. Cambridge, MA: MIT Press. Gurzick, D., White, K. F., Lutters, W. G., Landry, B. M., Dombrowski, C., & Kim, J. Y. (2011). Designing the future of collaborative workplace systems: lessons learned from a comparison with alternate reality games. In Proc. iConference 2011, 174-180. Hawkins, J., & Pea, R. D. (1987). Tools for bridging the cultures of everyday and scientific thinking. Journal of Research in Science Teaching, 24(4), 291-307. Hoadley, C. P. (2002). Creating context: design-based research in creating and understanding CSCL. In Proc. CSCL 2002, 453-462. Jenkins, H., Clinton, K., Purushotma, R., Robinson, A. J., & Weigel, M. (2006). Confronting the Challenges of Participatory Culture: Media Education for the 21st Century. Chicago, IL: MacArthur Foundation. Johnson, D., Johnson, R. T., & Holubec, E. J. (1994). The New Circles of Learning: Cooperation in the Classroom and School. Alexandria, VA: Association for Supervision and Curriculum Development. Johnson, D. W., & Johnson, R. T. (2009). An Educational Psychology Success Story: Social Interdependence Theory and Cooperative Learning. Educational Researcher, 38, 365-379. Johnson, M., Buhler, A. G., & Hillman, C. (2010). The Library is Undead: Information Seeking During the Zombie Apocalypse. Journal of Library Innovation, 1(2), 29-43. Kim, J. Y., Allen, J. P., & Lee, E. (2008). Alternate reality gaming. Commun. ACM, 51(2), 36-42. Klopfer, E., Perry, J., Squire, K., & Jan, M-F. (2005). Collaborative learning through augmented reality role playing. In Proc. CSCL 2005, 311-315. Lesh, B. (2011). Why wont you just tell us the answer? Teaching historical thinking in grades 7-12. Portland, ME: Stenhouse Publishers. McGonigal, J. (2008). Why I Love Bees: A Case Study in Collective Intelligence Gaming. In K. Salen (Ed.), The Ecology of Games: Connecting Youth, Games, and Learning, 199-227. Cambridge, MA: MIT. Niemeyer, G., Garcia, A., & Naima, R. (2009). Black cloud: patterns towards da future. In Proc. Multimedia (MM 2009), 1073-1082. Owens, T. (2010). Modding the History of Science: Values at Play in Modder Discussions of Sid Meiers CIVILIZATION. Simulation & Gaming, 42, 481-495. Robertson, C. (2006). Temple of invention: history of a national landmark. London: Smithsonian American Art Museum National Portrait Gallery in association with Scala Publishers. Shaffer, D. W., Squire, K. R., Halverson, R., & Gee, J. P. (2005). Video Games and the Future of Learning. The Phi Delta Kappan, 87(2), 104-111. Slavin, R. E. (1991). Synthesis of Research of Cooperative Learning. Educational Leadership, 48(5), 71-82. Squire, K. (2006). From Content to Context: Videogames as Designed Experience. Educational Researcher, 35(8), 19-29. Steinkuehler, C., & Duncan, S. (2008). Scientific Habits of Mind in Virtual Worlds. Journal of Science Education and Technology, 17, 530-543. Whitton, N. (2008). Alternate reality games for developing student autonomy and peer learning. In Proc. Learners in the Co-creation of Knowledge (LICK) 2008, 32-40. Yin, R. K. (2003). Designing Case Studies. Case Study Research: Design and Methods (3rd ed., pp. 19-56). Thousand Oaks, CA: Sage Publications.
Acknowledgments
We thank the students, teachers, and librarians who participated in, and provided feedback on the first season of the Arcane Gallery of Gadgetry (AGOG). We also thank the National Science Foundation (NSF IIS-0952567).
ISLS
258
ICLS2012
Learning Innovation Diffusion as Complex Adaptive Systems through Model Building, Simulation, Game Play and Reflections
Junsong Huang, Manu Kapur National Institute of Education,1 Nanyang Walk, Singapore 643673 Email: junsong.huang@nie.edu.sg, manu.kapur.nie.edu.sg Abstract: To effectively foster innovation diffusion, school leaders need to learn innovation diffusion as Complex Adaptive Systems (CAS). In this study, two school leaders formed a dyad to learn both the knowledge about innovation diffusion and the knowledge in fostering innovation diffusion. Agent-based model building, model simulation, game play of a simulation game and reflections were designed as learning activities in this study. In the learning process, the learners developed the following understanding in innovation diffusion: teachers adoption decisions are based on limited rationality and local information; teachers have nonlinear influence on each other through social networks; teachers are heterogonous agents; and diffusion is a process of emergence. The learners also learnt to leverage on social networks to foster effective innovation diffusion. While agent-based model building faces challenges for learning CAS in the social science domain, this study shows that engaging learners in reflection activities helps to overcome the challenges.
Innovation Diffusion in Schools: Learning Challenges That School Leaders Face

The past decade saw many education innovations, but the diffusion (Rogers, 2003) of such innovations has yet to receive adequate attention in education research (Christensen, Horn, & Johnson, 2008; Surry & Ely, 2001). For school leaders to effectively diffuse innovations, it is essential that they (a) understand innovation diffusion as Complex Adaptive Systems (CAS) (Dooley, 1996; Holland, 1995), and (b) learn how to foster diffusion. Need to learn innovation diffusion as CAS: Rogers, Medina, Rivera and Wiley (2005) conceive innovation diffusion as CAS, a sub-set of Complex Systems (Bar-Yam, 2003). Different from Complex Systems, CAS has multiple semi-autonomous agents (such as human beings) (Holland, 1995) whose decision-making is based on local information and limited rationality (Gigerenzer & Selten, 2002). The agents are constantly evolving and adapting to what the other agents are doing by interacting with other agents (locally) so as to maximize their fitness in the environment (globally). The structures and patterns of a CAS tend to be emergent (Holland, 1995). Huang and Kapur (2007) regard innovation diffusion in school as a result of many teachers (who are seen as semi-autonomous agents in a CAS) interacting with each other in school social networks. As teachers adapt and change (similar to how agents self-organize in a CAS) over time, innovations are re-invented and adopted over time and space in a school. However, there is a strong top-down tradition in innovation diffusion (Niehaves, 2007), which maintains that imposing a top-down mandate in a school for a sufficiently long period will result in sustainable and pervasive adoption of innovations among teachers (Christensen, et al., 2008; Looi, Lim, Koh, & Hung, 2005). This perspective takes a mechanistic view rather than a CAS view on innovation diffusion. Need to learn how to foster innovation diffusion: There are differences between the knowledge about innovation diffusion (knowledge-about) and the knowledge in fostering innovation diffusion (knowledge-indoing). Using swimming as an example, the knowledge-about swimming includes the concept of buoyancy which can be used to describe and explain swimming whereas the knowledge-in-doing swimming refers to the personal knowledge that a swimmer has in the act of swimming. The characteristics of the two kinds of knowledge are summarized in Table 1. Table 1: Characteristics of Knowledge-about and Knowledge-in-doing. Knowledge-about Knowledge to describe and explain a phenomenon General and abstract Relatively static Explicit; can be described by the knower Evidenced by descriptions, artifacts and descriptionoriented actions Knowledge-in-doing Action dependent knowledge in changing a phenomenon Situational and embodied Temporal and dynamic Mostly tacit; cannot be totally reduced to mere words, but can be acted out in situations Evidenced by interpretations of actions and reflections towards actions
ISLS
259
ICLS2012
Innovation diffusion as CAS is a form of knowledge-about which is different from the knowledge in fostering innovation diffusion (knowledge-in-doing). There is a need for school leaders to learn both. The rest of the paper is organized into four sections. Section 1 syntheses the literature and concludes that the literature on learning Complex Systems and learning innovation diffusion rarely focus on the development of knowledge-in-doing, and the learning processes are largely uninvestigated. This section further draws design conjectures for learning innovation diffusion as CAS. Section 2 outlines the research methodology in this study: the interpretivist paradigm and the case study method, and the learning design framework which includes agent-based model building, model simulation, game play and reflections. Section 3 reports the four learning trajectories identified in the learning process: teachers adoption decisions based on local information and limited rationality; teachers nonlinear interactions through social networks, teachers heterogeneity, and diffusion as a process of emergence. The last section discusses the affordances and limitations of Agent-Based Modeling (ABM) and reflections for learning CAS in the social sciences domain.
1. Review of Literature
The review of literature is divided into three parts. The first part reviews the literature of learning Complex Systems: the literature focus primarily on learning the knowledge about Complex Systems in the science domain, where the processes of learning are rarely investigateded. The second part reviews the literature of learning innovation diffusion. The literature lacks of evidence of learning outcomes and understanding of learning processes. The third part reviews the design principles from the two bodies of literatures and draws four design conjectures: to activate learners prior knowledge, to provide opportunity for exploration and experimentation, to encourage collaboration and to provide opportunity for reflection.
Learning Complex Systems

Learning Complex Systems has proven to be difficult (Hmelo-Silver & Azevedo, 2006; Jacobson & Wilensky, 2006), and Kapur and Jacobson (2010) regard learning as an emergence. The research on learning Complex Systems is still at the emerging stage (Jacobson & Wilensky, 2006). Almost all the empirical studies (Azevedo, Guthrie, & Seibert, 2004; Levy & Wilensky, 2008; Son & Goldstone, 2009) measured students knowledge-about Complex Systems or CAS, in particular the concepts of emergence, nonlinearity (such as the butterfly effect) and self-organization (Jacobson & Wilensky, 2006). These concepts are difficult to learn because they are counterintuitive or conflict with learners existing beliefs (Wilensky & Resnick, 1995). Very rarely do studies focus on the learning in changing CAS (knowledge-indoing). There is also inadequate understanding on the process of learning Complex Systems. Many empirical studies (e.g., Azevedo, et al., 2004; Son & Goldstone, 2009; van Bilseb, Bekebrede & Mayer, 2010) adopt quantitative methods or mixed methods to analyze the students learning outcomes. Most of them are preliminary and only emphasize primarily on students learning outcomes. While many empirical studies (see (Wilensky & Reisman, 2006) engaged learners in learning Complex Systems in the science domain, such a learning design has limitations in learning CAS in the social science domain (e.g., in innovation diffusion) because human society is more complex. Currently, only one empirical study (van Bilseb, Bekebrede, & Mayer, 2010) had examined the learning in the social science domain. The researchers engaged port management professionals to learn port development as CAS and asked the learners to self-report their learning gain. The learning outcomes and the learning processes were largely not investigated. In summary, the existing studies generally engaged learners in learning the concepts of Complex Systems. Three main literature gaps are identified: (a) the need to have greater emphasis on learning the knowledge-in-doing in Complex Systems, (b) greater understanding of learners learning processes is desired, and (c) there is a lack in literature about learning CAS in the social science domain.
Learning Innovation Diffusion

At present, the learning of innovation diffusion is still at its infancy stage with few studies examining it. For example, Hallinger (2009) engaged part-time master students in learning innovation diffusion using a simulation game. Within the period of 2001-2005, a total of 50 classes and 1,221 students participated in the learning activities. However, Hallinger (2009) did not make any assessment on the students learning outcomes. The only paper proposing a measurement on students learning is a conference paper abstract by Ng and Hallinger (2000). Nevertheless, the paper did not report students learning outcomes. The review of the literature on learning Complex Systems and learning innovation diffusion revealed two main research gaps: (a) the need to develop both knowledge-about and knowledge-in-doing, and (b) the need to understand the process of learning. These literature gaps will be addressed in this paper.
Design for Learning

By synthesizing Jacobson and Wilenskys (2006) design principles for learning Complex System in general and the six empirical studies on learning CAS in specific, four design conjectures for learning CAS are identified: (a)
ISLS 260
ICLS2012
design learning activities that allow learners to activate their prior knowledge, (b) provide opportunities for learners to explore and experiment in CAS using tools that make the CAS emerging process explicit, (c) encourage collaboration, and (d) provide opportunity for reflection. Agent-Based Modeling (ABM) (Axelrod & Tesfatsion, 2006) and first-person role-play simulation game (Gee, 2004) are widely used as tools for learning CAS. Axelrod (2006) summarizes that ABM is particularly helpful in making the CAS emerging process explicit; supporting students reasoning capacity and aiding their intuition on CAS. In view of the challenges in learning CAS, all the six empirical studies engaged ABM to help students explore, experiment and experience CAS. This design conjecture is consistent with Jacobson and Wilensky (2006)s design principles for learning Complex Systems as well. Although learning and doing in the real world are not the same as learning and doing in a simulation game, Jarvis (2004) argues that a first-person role-play simulation game potentially has the power to engage learners in creating the impression of learning through primary experiences and to transpose their reallife experience and knowledge into a simulation game (White & Cornu, 2010). Consequently, the experience and knowledge developed in a simulation game can potentially be transposed to the real world. As such, firstperson role-play simulation games are often used to learn the knowledge-in-doing innovation diffusion. Reflection (Schon, 1987) challenges learners to critically examine the current situation, question the validity of the prior knowledge, and make sense of the experience. Reflection promotes metacognition (Kujawa, 1995). Metacognition has three basic elements: developing a plan of action, maintaining/monitoring the plan and evaluating the plan (Kujawa, 1995). Accordingly, reflection can also be categorized in three directions: reflection-for-action (Fitzgerald, 1994), reflection-in-action (Beck & Kosnik, 2001) and reflection-on-action (Fitzgerald, 1994). Reflection-for-action has a central role in achieving self-regulation (Zimmerman, 2000). Jacobson and Wilensky (2006) also regard self-regulation as a design principle for learning Complex Systems. The literature comparison reveals that none of the empirical studies reviewed follows all the four design conjectures. Two main design gaps are identified: (a) the researchers either engaged learners in reflection-on-action or did not engage learners in reflection at all. They rarely engage learners in reflection-inaction and reflection-for-action. One exception is the study by Klopfer, Yoon and Um (2005). The researchers engaged students in reflection-in-action during the task-based interview and reflection-for-action to plan for the next sessions; (b) sufficient opportunity to activate learners prior knowledge was not given in some studies.
2. Research Methodology
Under a broader framework of an interpretivist paradigm (Burrell & Morgan, 1976; Cohen, Manion, & Morrison, 2000), a case study method (Sorin-Peters, 2004) is adopted in this study. The purpose is to provide a descriptive interpretation of the process in which school leaders learn innovation diffusion as CAS. Learners: School leaders with innovation diffusion experience voluntarily took part in this study. There were a total of five dyad groups and only one dyad group is presented in this paper. This dyad group consists of a school principal (P1) and a Senior Teacher and ICT Mentor (P2) from a neighborhood secondary school. Prior to his appointment as Principal, P1 worked as a teacher, an ICT Department Head and a VicePrincipal in other secondary schools. He also worked as an Education Technology Officer in the Ministry of Education (MOE) before. P2 is a Senior Teacher in the English Department. He is the only ICT Mentor in the school appointed by the MOE. P1 and P2 had joint experience in diffusing iPad for learning in their school.
Figure 1. Learning Design Framework Learning Design Framework: The learning design framework adopted in this study is articulated in Figure 1. Three learning activities are engaged in sequence: (a) model-building building an agent-based diffusion model; (b) simulation of researchers model simulating an agent-based diffusion model provided by the researcher; and (c) game play playing a first-person role-play innovation diffusion simulation game which is also agent-based. In each learning activity, reflections and dialogs between the two learners are facilitated by the researcher. The three activities are selected in order to maximize the learners opportunities to activate prior
ISLS 261
ICLS2012
knowledge and to explore innovation diffusion as CAS. The sequence is proposed to given opportunities to activate the learners prior knowledge. The learners make explicit their assumptions in model building and they further activated their prior knowledge when justifying the model they built. Taking model building as the first activity would prevent the researchers model in limiting or directing the learners model-building towards some preconceived directions (Kapur & Bielaczyc, 2012). Asking the learners to simulate the researchers model after their model building activity allows the researcher to build a model that is relevant to the learners context and diffusion needs. The researchers model includes social network effects which are missed in the learners model. It better represents the CAS process in innovation diffusion. By comparing and contrasting between the simulation of their own model and the researchers model and by reflecting on their real world experiences, the learners further activate their prior knowledge. Game play was scheduled as the last activity as it not only enables learning but also allows observing learning transfer. Research Procedure and Instrument: A total of nine sessions were conducted. Session 1 and 2 were prelearning interviews with P1 and P2 respectively. Session 3 involved a pre-learning collaborative interview. In session 4, the researcher shared the ABM concept and engaged the learners in building the model. Session 5 was the simulation of the learners model and Session 6 was the simulation of the researchers model. Session 7 involved the game play. Finally, two post-learning interviews were conducted in sessions 8 and 9 respectively. Session 1 to 8 were conducted over 3 months and Session 9 was conducted 3 months after Session 8 to verify whether the learning developed had an impact on the learners diffusion strategies in their school context. Anylogic (XJ Technologies, 2008) was selected as the ABM tool in this study due to its robustness. It allows for defining agents local network connections and enables more complex diffusion strategies to be simulated. The learners were required to dialog and reflect before, during and after each modeling action in order to enhance and promote reflection-for-action, reflection-in-action and reflection-on-action. To address the time limitations in providing opportunities to activate learners prior knowledge, sufficient time was given to the learners for model building. The learners were also cautioned not to alter the model design to achieve certain desired simulation outcome without considering the sensibility and reality of the alteration in the real world. The learners model was then further enhanced by the researcher for a better model representation of innovation diffusion as CAS. This allowed the learners to see the differences from their own model and create learning opportunities. Dialogs and reflections were carried out during the simulation process. Frick, Ludwig, Kim and Huang (2007)s Diffusion Simulation Game (DSG) was used in the game play activity. DSG uses an agent-based approach to simulate a school context. The design of DSG includes heterogeneity of social connections (Rogers, 2003), heterogeneity of adopters (Rogers, 2003) and ConcernBased Adoption Model (CBAM) (Hall, Hord, Huling-Austin, & Rutherford, 1987) to represent agents adoption decision-making. The game allows players to select both bottom-up diffusion strategies (e.g. seeking help from people who are socially connected) and top-down diffusion strategies (e.g. organizing site-visits and information-sharing meetings) to achieve the desired diffusion goal. The game is widely played with nearly 9,000 game-plays between late 2006 and 2009 (Lara, Myers, Frick, Karabacak, & Michaelidou, 2009). Although CBAM is only one approach to model teachers adoption decision-making, the game is calibrated using real school data and the players self-reported that it is a realistic representation (Frick, et al., 2007).
3. Key Findings
The process of learning was analyzed using open coding content analysis method (Charmaz, 2006; Strauss & Corbin, 1998) which was also triangulated by the learning artifacts generated. Four key trajectories were identified: teachers adoption decisions based on local information and limited rationality; teachers nonlinear interactions through social networks, teachers heterogeneity, and diffusion as a process of emergence that is complex and adaptive. The four learning trajectories were related to the four characteristics of CAS, namely: semi-autonomous agents limited rationality, heterogeneity of agents, network and nonlinearity in interactions, and emergence. The four learning trajectories were not developed in isolation but were interrelated and coconstructed. An interview conducted three months after the learning activities revealed that the learners made corresponding changes when diffusing innovations in their schools. This section provides a descriptive interpretation of the four learning trajectories. As the agents in the agent-based models represent teachers in schools, the agents are referred to as teachers in this section.
Understanding the Limited Rationality in Teachers Adoption Decision-Making

During the pre-learning interviews and the agent-based model building sessions, the learners acknowledged teachers autonomy in decision-making but assumed that teachers make adoption decisions with full rationality (initial knowledge-about). This is manifested in the model they built where teachers needed to see value in innovation. The learners elaborated that teachers must see the rationale of the innovation. Under this assumption, teachers adoption decision-making was regarded primarily as an information processing process and the diffusion strategy (initial knowledge-in-doing) was to (a) make correct information available through school and departmental-level sharing; and (b) train teachers to increase their information processing ability.
ISLS
262
ICLS2012
Development in model building: The simulation of the learners model showed that innovation that works will be adopted fast among agents. The researcher then asked why the superior DVORAK keyboard is not able to replace the inferior QWERTY keyboard over the past 40 years (to promote reflection-in-action). P2 responded confidently that if the DVORAK keyboard is mandated, especially (if it is) mandated among the new users, it will overtake the inferior QWERTY keyboard. P1 concurred with P2. While this response revealed some slight shifts from a rationality model of decision-making (initial knowledge-about), the learners knowledge-in-doing was not well aligned with regards to innovation diffusion as CAS. Development in simulating the researchers model: The researcher made two modifications on the learners model: firstly, the social networks (e.g. friends in the school and colleagues in the same department) which the teachers were connected with and secondly, the spread of positive and negative opinions through the social networks. During the simulation, the learners encountered a simulation scenario whereby all teachers connected in a social network tend to be resistant to innovation diffusion. Even when the learners made attempts to convert some teachers in that social network, it was more difficult to convert them as compared to converting teachers who were not connected to any social networks. During reflection-in-action, the learners acknowledged that adoption decision-making is not just based on information, but also largely influenced by peers (developed knowledge-about). As such, the learners started to intentionally leverage on social networks to influence teachers adoption decision-making (developed knowledge-in-doing). Development in game play: In the game play session, the learners were fully aware of the social network effects and were actively engaging social networks to persuade adoption among teachers.
Understanding Social Networks that Connect Teachers

During the pre-learning collaborative reflection session, the learners were directly asked if the social networks had any effects on diffusion. P1 responded that he (did) not think they have a large impact on diffusion. P2 added that frankly we did not consider social networks in diffusion Even it is impactful, how can we control them? The reflection suggested that the learners initially regarded social networks as irrelevant to innovation diffusion (initial knowledge-about) and they had no intention to tap on social networks for effective diffusion (initial knowledge-in-doing). Development in model building: During the reflection on the model built by the learners (reflection-onaction), the researcher asked the learners to compare and contrast between innovation diffusion and the spreading of rumors. The reflection activated the learners prior knowledge on virus infection and through their experience on virus infection, the learners prior knowledge on rumor spreading and innovation diffusion were connected. However, the learners still did not think that they can leverage on social networks for diffusion. Development in simulating the researchers model: When simulating the researchers model, the learners noticed that the teachers from the same social networks tended to hold the same adoption status. They then made attempts to influence the teachers inside the social network so as to level up the people in the social networks (shifted knowledge-in-doing during reflection-in-action). They also experienced the difficulty in converting many teachers who were connected in social networks. This suggested their realization of the importance of social networks for innovation diffusion. As P2 mentioned during reflection-on-action, right now what we're doing is very structured. But I think we've overlooked the informal social interactions, so this is something we can really think about (on) how to use them. Despite this, the learners had yet to understand the influence of the social network and how to leverage on it. Development in game play: In the game play session, the learners identified opinion leaders to be their targets for converting as early adopters. This was supported by their developed knowledge-about during reflection-for-action: (in the game, when we choose teachers to convert), we talk about influence and credibility (that the teacher has in spreading the innovation). When reflecting for innovation diffusion in their own school, the learners showed some developed knowledge-in-doing: when we formed the core-group team for diffusion, we need to identify who the teachers (in the core-group) are and identify the opinion leaders in them (who) can help us spread the innovation to others.
Understanding the Heterogeneity of Teachers

The learners had an initial concept of who the innovators were. They regarded innovators as teachers who are new to the school environment or new in education service and they recognized that the innovators have creative juice in them, and always have new ideas to propose. The learners also had some understanding of the laggards who are more experienced, stubborn, resistant and more set in their ways. Development in model building: The learners were not aware of the other groups of adopters, such as early adopters, early majority and late majority (Rogers, 2003) (initial knowledge-about). Hence, they did not use differentiated diffusion strategy to target these teachers (initial knowledge-in-doing). Development in simulating the researchers model: The learners observed the social networks in this session. When the learners failed to convert teachers who were connected in social networks, special attention was paid to teachers who had a higher degree of connectedness in the social network. The learners also realized that opinions leaders are teachers who have centrality in social networks. They influence other teachers and are
ISLS 263
ICLS2012
concurrently influenced by these teachers (developed knowledge-about). The learners tried to convert the opinion leaders directly, which did not yield a productive diffusion outcome (developed knowledge-in-doing). Development in game play: The learners further developed their knowledge-in-doing by targeting different groups of teachers in the different stages of diffusion. The learners first identified innovators from the teachers profile (in the game) and worked on these teachers. The learners then selected the pro-innovation opinion leaders as their target group for early adopters. Subsequently, the learners worked on the early adopters both directly and indirectly by leveraging on the social networks to influence targeted opinion leaders. When opinion leaders adopted the innovation, the learners acted knowledge-in-doing by engaged them to conduct sharing sessions to influence other teachers (early majority and late majority). The learners also noticed during reflection-in-action that for those opinion leaders who have already adopted, if we continue talk to them, they will continue indirectly influencing others in their social circle.
Understanding Diffusion as an Emergence

In the first three learning trajectories, the learners identified a gap and subsequently constructed knowledge consciously over the learning sessions to fill the gap. The fourth learning trajectory did not start with any obvious gap, nor did the learners intentionally develop knowledge in this learning trajectory to address the gap. It is a rise-above during reflections. This learning trajectory is identified when conducting the data analysis. In the simulation session, P1 observed that a single strategy did not work and that the sequence of the strategies matters (reflection-in-action): actually the learning point is that, there is no one single strategy (that always works). It is really a multi-form of strategies and like P2 mentioned previously, the sequencing is also important. And sometimes we may need to repeat the strategies after a while. So it's quite hard to say I do one, let's say, ICT training throughout. It doesn't work. During the game play session, P1 further learnt that the sequence of diffusion strategies had a high impact on the diffusion discourse and the same sequence may not always produce the same outcome. He expressed his frustration (reflection-in-action) in dealing with innovation diffusion as CAS: I don't know because sometimes it affects (and) sometimes it doesn't. Finally, towards the end of the game play session, the learners acknowledged that innovation diffusion is unpredictable or controllable. They can only work on the feet, indicating that the learners developed an intuitive understanding of emergence. To summarize, for the case presented in this paper, learning innovation diffusion through agent-based model building, model simulation, game play and reflection is effective. The preliminary analysis of the case data clearly showed that the learners developed some intuitive understanding of innovation diffusion as CAS (knowledge-about), such as the limited rationality in teachers adoption decision-making, the socially connected teachers and their interactions, the heterogeneity of teachers, and diffusion as an emergence. The learners also learnt to be more effective in fostering innovation diffusion (knowledge-in-doing), such as differentiating their strategies for heterogeneous teachers and leveraging on social network effects for influencing and persuading teachers.
4. Discussion
Compared to previous studies (e.g. Slotta & Chi, 2006; Wilensky & Reisman, 2006) which engaged learning Complex Systems in the science domain, this study showed that agent-based model building alone is not effective in helping learners learn CAS in the social science domain. This is probably due to the complexity of the CAS in the social science domain (Eidelson, 1997) and the arbitrariness in setting numeric parameters when building ABM models. Manifested in the learning process, the learners were sometimes confused about how much they can trust the model they built for reasoning and learning. The learners reflection in the post-learning interview revealed that when an unfamiliar phenomenon emerged in the model simulation, they were not sure whether it is due to their limited or biased observations in their real world diffusion work, the fallacy of their assumption and reasoning; the over-simplification of model; or the inappropriate value assignment of parameters. The challenge is not due to the limitation of ABM as a tool, but the complexity nature of CAS. Even with this challenge, ABM is still very helpful in learning CAS. The case data in this study showed that ABM allowed the learners to activate and articulate their prior knowledge. It also allowed the researcher to better understand innovation diffusion in the learners context so as to build a diffusion model that is meaningful for the learners to simulate. These are important for learning CAS. Nevertheless, the challenges were addressed largely through reflections, which were crucial in this study. The reflections were facilitated in two areas: (a) the fallacies of the learners initial knowledge identified through the pre-learning interviews and the agent-based model building session; and (b) other CAS phenomena the learners may be familiar with, such as the spreading of rumors, the iPhone bandwagon effect, the failure in promoting the DVORAK keyboard, etc. The three forms of reflections, especially reflection-in-action, had proven to be effective and how they enabled learning will be unpacked further in future data analysis. With reflection as a critical design for learning CAS in the social science domain, the role of the researcher becomes controversial. The learning is no longer a function of learners and the learning activities; the
ISLS 264
ICLS2012
researcher forms part of the learning environment. As the researcher learns in the learners process of learning, he constantly becomes a changing agent in the learning environment. As such, the researchers ability to learn in the learners learning process, the researchers knowledge on innovation diffusion and CAS, and the researchers facilitation skill will affect the learners learning process and learning outcome. Hence, in this dynamic learning process, can the researcher be independent and is this study valid? To answer this question, Guba & Lincoln (1989) propose to use trustworthiness rather than validity to assess the quality of a qualitative study. Flyvbjerg (2004) maintains that trustworthiness needs to be coped within research design and research process, and not by following an inflexible set of standards and procedures. One strategy adopted by this study to improve the trustworthiness is by engaging the researcher in a self-reflexive process so that the researcher is positioned as a self-conscious, critical and participatory coparticipants (Fine, 1992). As such, the role of the researcher will be dealt with by engaging the researcher in the self-reflexive process and make his assumptions and intervention decisions overt to the readers (Morrow, 2005). Findings from this study are preliminarily as only one of the five cases are analyzed and discussed. A framework for understanding the learning of knowledge-about and knowledge-in-doing will be developed to describe, interpret and compare the learning trajectories within and across cases. The role of researcher in the learning environment as well as in the data analysis process will be dealt with in a self-reflexive process and made known to the readers in future reports.
References
Axelrod, R. (2006). Advancing the art of simulation in the social sciences. In J. P. Rennard (Ed.), Handbook of Research on Nature-inspired Computing for Economics and Management. Hersey, PA: Idea Group. Axelrod, R., & Tesfatsion, L. (2006). A Guide for Newcomers to Agent-based Modeling in the Social Sciences. In K. L. Judd & L. Tesfatsion (Eds.), Handbook of Computational Economics (Vol. 2): North Holland. Azevedo, R., Guthrie, J. T., & Seibert, D. (2004). The Role of Self-Regulated Learning in Fostering Students' Conceptual Understanding of Complex Systems with Hypermedia. Journal of Educational Computing Research, 30(1), 87-111. Bar-Yam, Y. (2003). Dynamics of Complex Systems: Westview Press. Beck, C., & Kosnik, C. (2001). Reflection-in-action: In defence of throughtful teaching. Curriculum Inquiry, 31(2), 217-227. Burrell, G., & Morgan, G. (1976). Sociological paradigms and organizational analysis. London: Heinemann. Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. London: Sage. Christensen, C. M., Horn, M. B., & Johnson, C. W. (2008). Disrupting Class: How disruptive innovation will change the way the world learns: McGraw Hill. Cohen, L., Manion, L., & Morrison, K. (2000). Research Method in Education (5th ed.). New York: Routledge Falmer. Dooley, K. (1996). Complex Adaptive Systems: A Nominal Definition. The Chaos Network, 8(1), 2-3. Eidelson, R. J. (1997). Complex adaptive systems in the behavioral and social sciences. Review of General Psychology, 1(1), 42-71. Fine, M. (1992). Disruptive voices: the possibilities of feminist research. Ann Arbor: University of Michigan Press. Fitzgerald, M. (1994). Theories of reflection for learning. In A. Palmer & S. Burns (Eds.), Reflective practice in nursing. Oxford: Blackwell Scientific. Flyvbjerg, B. (2004). Five misunderstandings about case-study research. In C. Seale, G. Gobo, J. F. Bubrium & D. Silverman (Eds.), Qualitative research practice (pp. 420-434): Sage. Frick, T., Ludwig, B., Kim, K. J., & Huang, R. (2007). Diffusion Simulation Game. Bloomington, US: Indiana University. Gee, J. P. (2004). Learning by Design: Games as Learning Machines. Interactive Educational Multimedia, 8, 15-23. Gigerenzer, G., & Selten, R. (2002). Bounded rationality: The adaptive toolbox: the MIT Press. Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage Publications. Hall, G. E., Hord, S., Huling-Austin, L., & Rutherford, W. L. (1987). Taking Charge of Change: Alexandria, VA: Association for Supervision and Curriculum Development. Hallinger, P. (2009). Learning to lead organizational change in Thailand - Assessment of a problem-based approach. Education Review. Hmelo-Silver, C. E., & Azevedo, R. (2006). Understanding Complex Systems: Some Core Challenges. The Journal of the Learning Sciences, 15(1), 53-61. Holland, J. H. (1995). Hidden Order: How Adaption Builds Complexity: Perseus Books.
ISLS
265
ICLS2012
Huang, J. S., & Kapur, M. (2007). Diffusion of Pedagogical Innovations as a Complex Adaptive Process Agent -Based Modeling as Research Method. In T. Hirashima & S. S. C. Young (Eds.), Supporting Learning Flow through Interactive Technologies - Frontiers in Artificial Intelligence and Applications. Japan: IOS Press. Jacobson, M. J., & Wilensky, U. (2006). Complex System in Education: Scientific and Educational Importance and Implications for the Learning Sciences. Journal of the Learning Sciences, 15(1), 11-34. Jarvis, P. (2004). Adult & Continuing Education (3rd ed.). London: Routledge. Kapur, M., & Bielaczyc, K. (2012). Designing for Productive Failure. The Journal of the Learning Sciences, 21(1), 45-83. Kapur, M., & Jacobson, M. J. (2010). Learning as an emergent phenomenon: Methodological implications Paper presented at the International Conference on the Learning Sciences Klopfer, E., Yoon, S., & Um, T. (2005). Teaching complex dynamic systems to young students with StarLogo. Journal of Computers in Mathematics and Science Teaching, 24(2), 157-178. Kujawa, S. (1995). Strategic Teaching and Reading Project Guidebook (Updated ed.): NCREL. Lara, M., Myers, R., Frick, T., Karabacak, S., & Michaelidou, T. (2009). A design case: Creating an enhanced version of the Diffusion Simulation Game. Paper presented at the annual conference of the Association for Educational Communications & Technology. Levy, S. T., & Wilensky, U. (2008). Inventing a" Mid Level" to Make Ends Meet: Reasoning between the Levels of Complexity. Cognition and Instruction, 26(1), 47. Looi, C. K., Lim, W. Y., Koh, T. S., & Hung, D. (2005). Systemic Innovations and the Role of ChangeTechnology: Issues of Sustainability and Generalizability. Paper presented at the ICCE. Morrow, S. L. (2005). Quality and trustworthiness in qualitative research in counseling psychology. Journal of Counseling Psychology, 52(2), 250-260. Ng, F. S., & Hallinger, P. (2000). Effects of an intelligent simulation system on knowledge acquisition among school leaders. Paper presented at the Annual Hawaii International Conference on Education Niehaves, B. (2007). Public Sector Innovation and Diffusion Process - Preliminary Results of a Qualitative Study in Japan. Paper presented at the PACIS. Rogers, E. M. (2003). Diffusion of Innovations (5th ed.): Simon and Schuster. Rogers, E. M., Medina, U. E., Rivera, M. A., & Wiley, C. J. (2005). Complex adaptive systems and the diffusion of innovations. The Innovation Journal: The Public Sector Innovation Journal, 10(3), 1-26. Schon, D. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass. Son, J. Y., & Goldstone, R. L. (2009). Fostering General Transfer with Specific Simulations Pragmatics and Cognition, 17(1), 1-42. Sorin-Peters, R. (2004). The case for qualitative case study methodology in aphasia: an introduction. Aphasiology, 18(10), 937-949. Strauss, A. L., & Corbin, J. M. (1998). Basics of qualitative research: Grounded theory procedures and techniques (2nd ed.). Thousand Oaks, CA: Sage. Surry, D. W., & Ely, D. P. (2001). Adoption, diffusion, implementation, and institutionalization of educational innovations. Trends & Issues in Instructional Design and Technology. Prentice-Hall, 183-193. van Bilseb, A., Bekebrede, G., & Mayer, I. (2010). Understanding Complex Adaptive Systems by Playing Games. Informatics in Education, 9(1), 1-18. White, D., & Cornu, A. L. (2010). Eventedness and disjuncture in virtual worlds. Educational Research, 52(2), 183-196. Wilensky, U., & Reisman, K. (2006). Thinking Like a Wolf, a Sheep, or a Firefly: Learning Biology through Constructing and Testing Computational Theories--An Embodied Modeling Approach. Cognition and Instruction, 24(2), 39. Wilensky, U., & Resnick, M. (1995). New thinking for new sciences: Constructionist approaches for exploring complexity. XJ Technologies. (2008). AnyLogic (Version 6). St. Petersburg, Russia: XJ Technologies. Zimmerman, B. J. (2000). Attaining Self-Regulation: a social cognitive perspective. In M. Boekaerts, P. Pintrich & M. Zeodmer (Eds.), Handbook of Self-Regulation: Academic Press.
ISLS
266
ICLS2012
Peer Collaboration and Mediation in Elementary Students Lamp Designing Process

Kaiju Kangas & Pirita Seitamaa-Hakkarainen, Department of Teacher Education, P. O. Box 8, 00014 University of Helsinki, Finland kaiju.kangas@helsinki.fi, pirita.seitamaa-hakkarainen@helsinki.fi Abstract: The present study examines peer collaboration during elementary students lamp designing process. By exploring how different aspects of discourse relate to each other as well as to the various tools being used, we aimed to unfold the nature of the collaborative design process. The video recorded design episodes of one student team constituted the data source of the study. Along with the analysis of the main design activities of the process, we created CORDTRA diagrams for analyzing the interaction and the use of tools. The results indicate that the lamp designing process was a successful learning process. Through repeated cycles of designing, and by simulating many professional design practices, the young students were able to collaboratively develop a design, where various constraints determining lamp and light designing were taken into account.
Introduction
The complex and multidisciplinary nature of designing calls for intensive collaboration across different domains. The activities of those in the design professions are often based on teamwork combining several fields of distributed expertise. Consequently, the use of collaborative settings in the area of Design and Technology (D&T) education has increased. Successful collaboration in D&T contexts appears to be based on authentic design tasks that allow students to confront the multidisciplinary characters of design practice (Murphy & Hennessy, 2001; Seitamaa-Hakkarainen, Viilo, & Hakkarainen, 2010). Professional design activities rely on the use of different tools and design representations, such as sketches and notes (Al-Doy & Evans, 2011; Goel, 1995). Particularly sketching has a crucial role in generating, developing, and communicating ideas; it is both a powerful form of thinking and the fundamental language of design (Seitamaa-Hakkarainen & Hakkarainen, 2000/2004; Welch et al., 2000). In the context of D&T education, the interaction with two- and three-dimensional models (sketches, prototypes) offers students direct possibilities to explore and evaluate a proposed solutions form and function. Involving students in modeling practices can help them build domain expertise, epistemological understanding, and skills to create and evaluate knowledge (Schwartz et al., 2009). In D&T settings material artifacts and tools have a central role in mediating the learning process; the design process involves parallel working through conceptual reflection and material implementation. However, research has shown that novice designers rarely use two-dimensional models, i.e., sketching, but tend to move immediately to three-dimensional modeling (Welch, 1998). Furthermore, when sketching or other forms of modeling occurs, they are primarily used for illustrative or communicative purposes, hence reducing the epistemic richness of the practice (Schwartz et al., 2009). The present study describes an elementary students collaborative lamp designing project, where the guidance was provided by a professional designer. We focus on design activities of one student team, and examine how their collaborative process unfolds. By opening up the process, the present study aims to shed more light on how different aspects of discourse relate to each other, and how the various tools and design representations being used mediate the process. We addressed the following specific research questions: (1) What was the nature of elementary students collaborative design process? (2) How did the tools and design representations created and used during the process mediate the design collaboration? As noted by Hmelo-Silver and her colleagues (2008; Hmelo-Silver, 2003), understanding collaboration and mediation requires going beyond coding individual speech acts. They proposed the use of ChronologicallyOriented Representations of Discourse and Tool-Related Activity (CORDTRA) diagrams as one way of achieving information about how social interaction and representations serve as tools for students collaborative thinking. The diagrams enable a researcher to combine the chronological picture of the coded discourse with other learning activities (Hmelo-Silver, Chernobilsky, & Jordan, 2008). In the present study, we produced a CORDTRA diagram from the student teams face-to-face peer collaboration, in order to visualize their interaction and the use of different tools.
ISLS
267
ICLS2012
Peer Collaboration and Mediation in Design Learning

We frame our study in sociocultural theory, since it provides an appropriate theoretical framework for understanding the complexity of collaborative design learning. Sociocultural and socio constructivist theories emphasize the socially situated nature of learning (Palinscar, 1998) and the critical role of tools that mediate learning (Cole, 1996). In the present study, we focus on peer collaboration and the role of tools and representations in design learning. Peer collaboration refers to interaction that occurs when students with incomplete but rather equal levels of competence share their ideas in order to jointly solve a challenging problem (Damon & Phelps, 1989; Goos, Galbraith, & Renshaw, 2002). The development and maintenance of the shared thought process is a central element in peer collaboration (Azmitia, 1988, Teasley & Roschelle, 1993); it is a reciprocal process of exploring each others reasoning and viewpoints in order to construct a shared understanding of the task (Goos, Galbraith, & Renshaw, 2002). Barrons study (2000) revealed three aspects defining good collaboration: mutuality, joint focus of attention, and shared task alignment. Mutuality refers to reciprocity with potential for all participants to meaningfully contribute, joint attention to the degree which attention is jointly focused, and shared task alignment the establishment of a collaborative orientation toward problem solving. In D&T contexts, successful peer collaboration is a process in which students actively work together in creating and sharing their design ideas, deliberately making joint decisions and producing shared design objects, constructing and modifying their design solutions, as well as evaluating their outcomes through discourse (Hennessy & Murphy, 1999; Rowell, 2002). The problem solving strategies and practices of designing can be used for learning something else than design (Davis et al., 1997), for example, mathematics (Jurow, 2005) and science (Fortus et al., 2004; Kolodner, 2002; Kolodner et al., 2003; Roth, 1998). In designing, the interaction with tools, concrete objects and materials is a central aspect and offers a potentially supportive environment for vital collaboration, i.e., for developing shared objects and understanding (Hennessey & Murphy, 1999; Johansson, 2006; Murphy & Hennessy, 2001; Rowell, 2002;). Through social interaction and visualization, design ideas, proposed solutions, and decisions are made verbally and visually explicit and visible, and joint decisions can be made. Various external representations (graphical and physical) in various phases of the design process provide different kinds of prompts to test the design ideas (Al-Doy & Evans, 2011). A review of the research examining the role of sketching for design professionals (Welch et al., 2000) shows that sketching has a crucial role in generating, developing, and communicating ideas; it is both a powerful form of thinking and the fundamental language of design. Consequently, sketching is seen as central to developing capability in D&T education. In the early stages of designing, sketching helps to define and clarify the task, and explicate the needs constraining the task. It also enables and encourages the designer to play with ideas, which is essential in creative problem solving. Furthermore, sketching facilitates the evaluation of ideas and elaboration of the design task. In addition, sketching can be used to communicate the design ideas with others; it also enables those others to contribute to the ideas (Al-Doy & Evans, 2011;Welch et al., 2000). In other words, various design representations allow students to interact with one another through the design object itself, as collaborating participants activities are mediated and made visible through them.
Method
Participants and the setting of the study
The lamp designing project was a part of larger study that was organized in an elementary school located in a middle-class suburb of Helsinki, Finland. 32 students (13 boys), aged 1011 years participated in the project; out of these, 7 students had linguistic or other educational problems. Ten computers were available for students working in the classroom; the teacher had her own computer and a data projector. The technical infrastructure of the project was provided by Knowledge Forum (KF, Scardamalia & Bereiter, 2006). During the lamp designing phase, the class collaboratively created the Lamp Designing view - a kind of on-screen workspace or bulletin board of present activity (with archives) - in the Knowledge Forum database. The lamp designing phase lasted 11 sessions (one session was 45135 minutes, depending on the class schedule) during a period of two months. The design process was carried through in 13 teams of 24 students, by sketching, drawing, and building prototypes or models. The students also regularly presented their designs to the whole class. The expert, a professional interior designer specialized in lamp and light designing, was present in the classroom; the interaction between him and the students varied from face-to-face discussions with the whole class, to small team conversations, and to the sharing of comments through the Knowledge Forum database.
Data Collection and Methods of Data Analysis

The lamp designing process was video recorded almost entirely, producing about 16 hours of video material from a single camera. During whole-class activities we recorded the designers activities, and during small-team
ISLS 268
ICLS2012
activities we followed three student teams. For the present study, we selected a team of three girls, whose design process we were able to follow from the very beginning to the end. We selected the peer collaboration episodes of the team and the episodes where they were interacting with the designer. The analysis was performed at three levels. Firstly, in order to form an overall view of the project, we divided all the selected episodes into five-minute-units (f=76) and defined the main focus of design activity in each unit. The development of the classification schema was a combination of theory-based and data-driven analysis. The schema was created on the basis of a) preliminary analysis of the video data, and b) reflection on the data in relation to the theoretical framework of our study. This assisted in identifying the relevant aspects of the phenomena in question (Seale, 2006), and produced the following eight categories: 1) ideation, 2) defining constraints, 3) elaborating ideas, 4) making drawings, 5) constructing prototype, 6) making poster, 7) process organizing, and 8) off-task activity. Secondly, on the basis of identifying the main foci of activities, we selected approximately 34 minute episode from sessions 2 and 3. This episode was from the beginning of the teams work, and included ideation, idea elaboration, and defining constraints. This episode was interesting, because the design task was still very open and required a great deal of idea generating and negotiating from the team. In order to analyze the nature of interaction within the team, we identified the discourse move of each statement (f=650). We defined whether the speaker was: 1) initiating, 2) accepting, 3) disagreeing, 4) rejecting, 5) elaborating own idea, 6) building on others idea, 7) asking clarification, or 8) discussing off-task issues. We used Barrons (2000) categorization of response types as a basis for analysis, and complemented the schema with our own data-driven categories. Thirdly, we examined how the participants used tools, design representations, and gestures to support their verbal communication. Each statement from the selected episode was coded according to its speakers use of 1) sketching tools, 2) shared view, 3) pointing (at sketches or other tools), or 4) hand gestures (to describe shape, size, etc.). In the further sections of the present paper, these are jointly referred to as tools. In order to visualize our data, we created CORDTRA diagrams from the analysis of the statements. Our CORDTRA diagram contains a timeline where the discourse moves and the tools being used are plotted in parallel, enabling their juxtaposing for understanding the nature of the collaborative design process. (cf. HmeloSilver, 2003; Hmelo-Silver, Chernobilsky, & Jordan, 2008).
Results
Overview of the Collaborative Design Process
The whole lamp designing process consisted of 11 design sessions during a period of two months. The sessions included the designers presentations (sessions 1, 5, 6, and 10) students presentations (2, 6, 7, and 11), and designing in small teams. The team selected for the present study worked on their lamp design from the second to the tenth session, altogether approximately 6,5 hours. The time they used per session varied from 15 to 90 minutes, depending on the class schedule and other activities (such as the presentations) during the sessions. The team started by creating and sketching several design ideas, and developing some of these ideas further (Figure 1). The final idea was chosen during session 3, and elaborated throughout the process. The team also made several drawings of their design, from hazy sketches to blueprints on scale 1:1. The most time consuming activity was the construction of the prototype (f=90 min, 23,7%), the team built their lamp model from paper, wire, masking tape, and copper pipe during sessions 6, 7, and 8. Making the poster representing their design process and the final product also required a great deal of the teams time (f=75 min, 19,7%). The distribution of the teams various design activities during the sessions are presented in Figure 1. The students were guided to consider the important constraints for lamp designing. The general constraints were presented by the designer and worked on during whole-class discussions; the more specific constraints were defined by each team with support from the designer. In Figure 1 the constraints are visible only in session 4, although the team selected for the present study considered the constraints on several occasions throughout the design process. However, these sequences were in most cases so short, that they were deluged with other design activities in our analysis of the main activity in each five-minute-unit of the design process. It is, in fact, typical, especially for novice designers, to consider the design constraints in short sequences (SeitamaaHakkarainen & Hakkarainen, 2001). The participants of the team, Emma, Leila, and Nina worked together well, and their design process proceeded rather smoothly. The distribution of statements was quite equal, Leila (f=229, 36,5%) and Nina (f=234, 37,3%) made almost same amount of statements, Emma (f=164, 26,2%) a little less. A closer look to the nature of their discourse revealed that the girls had different roles within the team (Figure 2), and this had an effect on their design process. Nina was the most dominant one, she made many initiatives and many of her statements elaborated her own ideas. Emma was the most silent of the girls, she made fewer initiatives and elaborated them less than the other two. Many of her ideas were disagreed with or completely rejected by Nina and Leila. However, many of
ISLS
269
ICLS2012
Emmas statements built on the other girls ideas, developing them further. Also Leila built on others ideas a lot, and she also made most of the initiatives as well as asked clarifying questions.
Focus of activ ity

15%
Ideation Defining constrai nts Elaborati ng i deas Constructing prototype Off -topi c acti vit y Process organizing Making drawi ngs Making poster
10%
5%
0% 2 3 4 5 6 7 8 9 10
Session number
Figure 1. Main focus of activities of the selected team during the design sessions
Off-topic talk
Participant
Emma Nina L ei la Bars show percents
Rejecti ng
Discourse move
Building on other's idea
Elaborating ow ideas n
Asking clarification
Disagreei ng
Accepting
Initiating 2% 4% 6% 8% 10%
Figure 2. The discourse moves of the selected team
Unfolding Collaboration and Mediation

As noted by Hmelo-Silver and her colleagues (2008), the frequencies provide one view of the collaborative design process, but they neither provide a sense of chronology nor inform the researchers about particular qualitative features of the discourse (p. 420). In order to examine how the different aspects of discourse relate to each other, and to the use of different design tools, we created a CORDTRA diagram from the first 34 minutes of the teams work (Figure 3). At the bottom of the diagram, there is a running count of the statements. The vertical axis shows the categories of participants (1-3 at the bottom), discourse moves (4-12 in the middle), and tools (13-16 at the top). The patterns of interaction, i.e., the speaker, the discourse move, and tools of each statement, are arranged on the horizontal axis in chronological order. During the first approximately 400 statements the team was generating design ideas, altogether 17 different ideas for a lamp. At first, their designing was mostly horizontal (Seitamaa-Hakkarainen & Hakkarainen, 2000/2004; see also Goel, 1995), they created one idea after another, without elaborating on them much further. Idea number seven, The Lamp with Hundred Strings (statements #44-114) was the first vertically developed idea, which indicated that the idea was provisionally accepted (see Al-Doy & Evans, 2011). This idea was initiated by Nina, and she also spontaneously started using sketching (#51) to support her verbal explanation.
ISLS
270
ICLS2012
20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 0 50 100 150 200 250 300 350 400 450 500 550 600 650
Representational gesture Pointing gesture Shared view Sketching tools Giving instructions Off-topic talk Rejecting Building on other's idea Elaborating own ideas Previous statement continues Asking clarification Disagreeing Accepting Initiating Other student Teacher Designer Leila Nina Emma
Figure 3. The selected teams first 650 statements of peer collaboration

ISLS 271
ICLS2012
Compared to the previous six design ideas altogether, this sequence contained almost a double amount of statements. Nina elaborated her idea verbally and visually, and both Leila and Emma built on her ideas. In addition, Leila asked Nina to clarify several features of the design. No off-task talk occurred during the sequence. Finally the idea was jointly rejected, because of difficulties in working out the technical solutions. After The Lamp with Hundred Strings sequence the team used sketching throughout the episode. From statement #135 onwards, and after the teacher had reminded them, they also used the shared view (i.e., the teachers computer screen shared through a data projector) to support their designing. On the screen was the designers note from the KF Lamp Designing view. He had listed questions to be considered while generating the design ideas, for example, the purpose of the lamp and users needs. When the team was discussing the second vertically developed idea, The Disco Lamp (idea number 10, statements #159-243), they first talked enthusiastically over the color of the light bulbs and the material of the shade. The designer's questions on the shared view helped them to focus their attention also beyond the superficial aspects (e.g., appearance) that novice designers tend to concentrate on. They considered where the lamp could be used, who would use it, and how well it would light up the area. In the end, the idea was jointly rejected, because the team agreed that there is no real need for their disco lamp. The final idea started to take shape around statement #378, when Nina presented an idea for a pendant lamp and Leila immediately showed her interest by asking clarifying questions and building on the idea:
Extract 1. The idea of a pendant lamp 378 Nina (showing her sketch): Ha ha, this kind, like this way around, like from Leilas direction, this is hanging from the ceiling. 379 Leila: Oh, so is it the wrong way around? 380 Nina: Yeah so this is hanging from the roof. 381 Leila: So how does the light then there, come from there? 382 Nina: It reflects off that. 383 Leila: So where is the lamp again? 384 Nina: Inside there. You could make a hole here, and then it comes from here. 385 Emma: Maybe not. 386 Nina: Yeah. 387 Leila (sketching): Ha-ha, hey I know, wait, were doing now, doing these chain things, like this. 388 Nina: A hole like that. 389 Leila: Look, now, this kind thats on both sides, because it comes from up there.
Nina was not so interested in her own idea and she moved on to the next idea, but Leila continued developing and sketching the idea of the pendant. She was eager to reach the decision about the final idea and kept pushing Nina and Emma to make the decision with her by asking repeatedly where and what for the lamp would be used. Finally Leila's attempts succeeded and the team made a decision that the lamp would be used over a dining table. While discussing, Nina was still sketching new ideas:
Extract 2. Reaching decision 437 Leila: Yeah, yeah, so, lets make it, um, whats the lamp, whats it needed for? For the living room or kitchen or where? 438 Emma: It could fit into the kitchen, but it doesnt really, because it doesnt produce much light and anyway you need quite a lot in the kitchen. 439 Leila: The dining room. 440 Emma: Dining room? 441 Leila: Yeah, on top of the table. 442 Emma: Maybe. 443 Nina (sketching): Well thats what I meant. 444 Leila: Oh but you said, you said the kitchen. 445 Emma: So that theres a table there, thats how it hangs off there tsiu. 446 Nina: Look I invented a new lamp. 447 Leila: Well try to decide which one we are using. 448 Nina: No but I invented a new lamp. 449 Emma: Why couldnt this one also have squares? 450. Leila: Yeah Im just drawing it. 451 Nina: It doesnt fit. Laura look, I made a good one, Laura. 452 Leila: Wait, what do you mean squares dont fit this one? 453 Nina: You dont know how to do it, its like this, oho, me neither. This doesnt fit here at all. 454 Leila: Yes it does. 455 Nina: Look, isnt it beautiful? 456 Leila: Yeah! Thats how it could be.
Through Nina's sketching, Leila's clarifying questions and repeated requests to consider the needs for the lamp, and Emma's idea to include squares on the shade (an idea presented previously) the final idea was finally jointly reached. After this the idea started to develop rapidly, the team elaborated the idea through active discussion and sketching. They made a decisions concerning, for example, the shape and coloring alternatives of the shade. They stayed mostly on the composition space, but time to time shifted also to the construction space (SeitamaaHakkarainen & Hakkarainen, 2001), and considered, for instance, how many light bulbs would be needed and how these should be positioned inside the shape.
ISLS 272
ICLS2012
Discussion
Designing puts emphasis on conceiving something new and clarifying as yet unknown details. Design is not description of what is, it is exploration of what might be (Edmonds et al., 1994, p. 43). That requires active knowledge creation and meaning making - aspects which make D&T settings potentially rich environments for successful learning. In addition, a central aspect of the design process is to conceptualize and visualize an idea of the emerging product (Goel, 1995; Lawson, 1997). Externalization and visualization helps intangible ideas to become concrete and allows them to be generated, modified and transmitted between people. Our previous studies (Kangas, Seitamaa-Hakkarainen, & Hakkarainen, 2011a; 2011b) provided evidence on the crucial role of the designers participation and guidance during the lamp designing process. The present study sheds more light on the importance of simulating professional practices in D&T education. Unfolding peer collaboration of one student design team revealed that the lamp designing process included many activities which are not common in D&T education at elementary level, but which are widely used by design experts. First of all, the design ideas were generated and developed collaboratively. Our analysis revealed that the participants had different roles within the team, and this had an effect on their design process. One of the students was slightly dominating, while one students ideas were sometimes disagreed with or completely rejected. However, she still continued to contribute by building on the other girls ideas. Thus, the collaborative design process in the team was impacted by the students willingness to listen to others ideas, to take risks and to share their ideas with others. It appears to us that in the beginning of the process the participants were still working out their roles within the team. During conversation, they were attending to each others understandings of the issues, while simultaneously settling social relationships (see also Rowell, 2002). Second, the process was long, much longer than most design processes that are carried through in schools. The lamp designing process lasted 11 sessions during a period of two months; from these sessions nine were mainly dedicated to the students idea generation and elaboration. Although construction of the prototype and making the poster required a great deal of time, also the ideation phase was long. The team selected for the present study produced 17 different design ideas before choosing the one to be developed further. This was possible, because they had enough time to both generate their ideas as well as develop the flow of togetherness. They were able to establish contact zones in which talk and tools were used to mediate the emerging design (Rowell, 2002). Our analysis indicated that the teams communication was centered on the design ideas and their visualizations, and also that the task was collaboratively shared and mutual. Third, both horizontal and vertical ideation occurred during the process, which is not typical for novice designers (Seitamaa-Hakkarainen & Hakkarainen, 2001). This might be the consequence of two conditions: the long ideation phase, and the requirement of producing numerous sketches before moving to three-dimensional modeling (cf. Welch, 1998, 2000). Sketches are valuable because they are rich in suggestions of what might be. A sketch is constructed as an arrangement of preconceived objects in a particular relationship, thus, it is a reflection of the meaning of these objects. Emergence results in the perception of new shapes that were not intended to be represented in the sketch (Edmonds et al., 1994). Finally, with the designers support, the young students were able to consider many important constraints related to lamp designing. They, for example, rejected ideas if there was no real need for the lamp. In their final idea they considered where and what for the lamp could be used, what kind of light would be needed, how the positioning of the lamp could be adjusted, and many other constraints that determined their lamp design. These are far beyond the superficial aspects of designing that novice designers tend to concentrate on. This indicates that the lamp designing process was also a successful, collaborative learning process. When analyzing design learning, it is important to evaluate the process and construction of the designs. Designing challenges students to think in new ways and take risks. The open-ended and complex design tasks require focusing on harnessing that engagement, and supporting students as they propose ideas, fail, and propose again. In the present study, the young students were able to develop a design, where various constraints determining lamp and light designing were taken into account. This was accomplished through repeated cycles of designing, and by simulating many professional design practices.
References
Al-Doy, N. & Evans, M. (2011). A review of digital industrial and product design methods in UK higher education. The Design Journal 14(3), 343368. Azmitia, M. (1988). Peer interaction and problem solving. When are two heads better than one? Child Development, 59, 8796. Barron, B. (2000). Achieving coordination in collaborative problem-solving groups. Journal of the Learning Sciences, 9(4), 403436. Cole, M. (1996). Cultural psychology: A once and future discipline. Cambridge, MA: Harvard. Damon, W. & Phelps, E. (1989). Critical distinctions among three approaches to peer education. International Journal of Educational Research 13, 919.
ISLS
273
ICLS2012
Davis, M., Hawley, P., McMullan, B., & Spilka, G. (1997). Design as a catalyst for learning. Alexandria, VA: Association for Supervision and Curriculum Development. Edmonds, E. A., Candy, L., Jones, R., & Soufi, B. (1994). Support for collaborative design: Agents and emergence. Communications of the ACM, 7(7), 4147. Fortus, D., Dershimer, R. C., Krajcik, J., Marx, R. W., Mamlok-Naaman, R. (2004). Design-based science and student learning. Journal of Research in Science Teaching, 41(10), 10811110. Goel, V. (1995) Sketches of thought. Cambridge, MA: MIT Press. Goos, M., Galbraith, P., & Renshaw, P. (2002). Socially mediated metacognition: Creating collaborative zones of proximal development in small group problem solving. Educational Studies in Mathematics, 49, 193223. Hennessy, S., & Murphy, P. (1999). The potential for collaborative problem solving in design and technology. International Journal of Technology and Design Education, 9(1), 136. Hmelo-Silver, C. E. (2003). Analyzing collaborative knowedge construction: Multiple methods for integrated understanding. Computers & Education, 41, 397420. Hmelo-Silver, C. E., Chernobilsky, E., & Jordan, R. (2008). Understanding collaborative learning processes in new learning environments. Instructional Science, 36, 409430. Johansson, M. (2006). The work in the classroom for sloyd. Journal of Research in Teacher Education, 2-3, 153171. Jurow, A. S. (2005). Shifting engagements in figured worlds: Middle school mathematics studentsparticipation in an architectural design project. The Journal of the Learning Sciences, 14(1), 3567. Kangas, K., Seitamaa-Hakkarainen, P. & Hakkarainen, K. (2011a). Design experts participation in elementary students collaborative design process. International Journal of Technology and Design Education, Online First. URL http://www.springerlink.com/content/11277067hk7647n0/ Kangas, K., Seitamaa-Hakkarainen, P. & Hakkarainen, K. (2011b). Figuring the world of designing: Expert participation in elementary classroom. International Journal of Technology and Design Education, Online First. URL http://www.springerlink.com/content/m6325g8m2715u91q/ Kolodner, J. (2002) Facilitating the learning of design practices: Lesson learned from an inquiry in science education. Journal of Industrial Teacher Education. 39(3), 131. Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., Puntambekar, S., & Ryan, M. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning by design into practice. Journal of the Learning Sciences, 12(4), 495547. Lawson, B. (1997). How designers think: The design process demystified (3rd ed.). Oxford: Architectural Press. Murphy, P. & Hennessy, S. (2001). Realising the potential and lost opportunities for peer collaboration in a D&T setting. International Journal of Technology and Design Education, 11, 203237. Palinscar, A. S. (1998). Social constructivist perspectives on teaching and learning. Annual Review of Psychology, 45, 345375. Roth, W-M. (1998). Designing communities. Boston: Kluwer Academic Publisher. Rowell, P. (2002) Peer interactions in shared technological activity: a study of participation. International Journal of Technology and Design Education, 13, 122. Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In K. Sawyer (Ed.) The Cambridge handbook of the learning sciences (pp. 97115). Cambridge, MA: Cambridge University Press. Schwarz, C. V., Reiser, B. J., Davis, E. A., Kenyon, L. Achr, A., Fortus, D., Shwartz, Y., Hug, B., & Krajcik, J. (2009). Developing a learning progression for scientific modeling: Making scientific modeling accessible and meaningful for learners. Journal of Research in Science Teaching, 46(6), 632654. Seitamaa-Hakkarainen, P. & Hakkarainen, K. (2001). Composition and construction in experts and novices weaving design. Design Studies, 22(1), 4766. Seitamaa-Hakkarainen, P. & Hakkarainen, K. (2000/2004). Visualization and sketching in design process. Design Journal, 3, 314. Seitamaa-Hakkarainen, P., Viilo, M., & Hakkarainen, K. (2010). Learning by collaborative designing: technology-enhanced knowledge practices. International Journal of Technology and Design Education, 20(2), 109136. Teasley, S. D. & Roschelle, J. (1993). Constructing a joint problem space: The computer as a tool for sharing knowledge. In S. P. Lajoie & S. J. Derry (Eds.), Computers as Cognitive Tools (pp. 229258). Hillsdale, NJ: Erlbaum. Welch, M. (1998). Students use of three-dimensional modeling while designing and making a solution to a technical problem. International Journal of Technology and Design Education, 8, 241260. Welch, M., Barlex, D., & Lim, H. S. (2000). Sketching: Friend or foe to the novice designer? International Journal of Technology and Design Education, 10, 125148.
ISLS
274
ICLS2012
Dynabook: Supporting Teacher Learning about Mathematical Thinking

Jeremy Roschelle, SRI International, Menlo Park CA, Jeremy.Roschelle@sri.com Charles Patton, SRI International, Menlo Park CA, Charles.Patton@sri.com Elizabeth Murray. CAST, Wakefield MA, bmurray@cast.org Abstract: To advance the conference theme of the future of learning, we report our investigations of how the change from paper textbooks to digital curriculum resources may change the Learning Sciences, and how, in response the Learning Sciences could grow to make significant advances to digital curriculum resources. We highlight the changing relationship of the reader and the resource in digital age. We believe something like a textbook will continue to be a central focus and source of structure in future learning environments, but that unreflectively carrying forward textbook conventions and assumptions about how reading relates to learning may not be productive. Building on universal design for learning (UDL) theories and our own Dynabook investigations, we sketch a framework for rethinking the nature of digital curriculum resources and highlight how the field of the learning sciences might grow to play a stronger role in the coming tidal wave of digital first learning resources.
Introduction
One of the largest, most powerful trends in education today is the transition from paper to digital textbooks (Education Week, 2011). Although some futurists imagine textbook-like resources disappearing, we suggest that there is a stable educational need for curricular resources that (a) structure the long-term learning progression in a course or topic (b) are carefully edited for coherence, etc. and (c) convey authority. Hence, something like the textbook will continue. But if something too much like the textbook continues, what a missed opportunity! As much as textbooks provide a central resources for organizing and structuring what teachers do, they have an insufficient theory of action with regard to teaching and learning (Ball & Cohen, 1996). Students buy or are given textbooks on the theory that student use of the textbook will advance learning. The most common use model is that a teacher reviews key definitions and solution methods from the text with students and then presents the students with selected problems from the book to practice on. However, this use model for a book does not lead to either effective teaching or meaningful student learning -- and thus supporting use in a digital medium may be an advance. An alternative to digitally-enhancing the existing textbook model is to start from learning sciences insights about how to prepare teachers, and to consider how those insights could be more systematically supported in a digital medium. Perhaps the single most valuable lesson from the learning sciences is that when teachers learn how to empathize with, make sense of, and work with student thinking, they often are able to enact better instructional strategies for their students. For example, this principle has very successfully been applied in Cognitively Guided Instruction (Carpenter & Fennema, 1996). Based on this principle, we have been exploring an alternative theory of action. We propose that a dynabook for use in teacher education can engage preservice teachers in considering how students think, examining their own mathematical thinking as a teacher, and framing more sophisticated pedagogical approaches towards helping their students. In this paper, we present some mid-term reflections on our progress in crafting the Proportionality Dynabook, testing it in teacher preparation courses in two universities, and reflecting on where we need to go next to fully realize the potential of the new digital medium for textbook-like resources. The main contributions of this paper are to (a) share some our early failures (b) describe a compelling purpose for a dynabook for teacher education (c) introduce a potential new metaphor for the endeavor and (d) elaborate some of the research questions that are still outstanding at this early stage of inquiry.
Background: The Dynabook Project and Team

The Dynabook Project has been a design research effort, aimed at crafting a new resource for use in teacher education based on the emerging potential for digital books. A quick sense of the project can be expressed by describing the partners. SRI International brought experience in creating effective digital materials for learning mathematics using dynamic graphical representations (Roschelle et al, 2010) as well as experience using a codesign methodology to build new things that reflect the input of multiple stakeholders (Penuel et al, 2007). A
ISLS 275
ICLS2012
nonprofit organization with a long history of work on reading and in special education, CAST, provided an engineering framework for creating interesting digital texts, based on its own work in Universal Design for Learning (Rose & Meyer, 2002; Rose, Meyer, & Hitchcock, 2005). Two campuses of the California State University system each brought innovative professors who were working to introduce prospective teachers to better ways to teach mathematics. The professors at San Francisco State University were more focused on special education teachers, whereas those at San Diego State were more focused on general mathematics education teachers. In addition to Universal Design for Learning (UDL), the TPACK framework has been influential in the project as a way to relate mathematical content, pedagogy and use of technology in learning (AACTE Committee on Innovation and Technology, 2008). Inverness Research, in the role of evaluator, has been acting as a critical friend to the project, seeking to engage the whole team in working on the most important and provocative questions in this area of work. As a whole, the project team has developed several iterations of a web-based dynabook, focused on the mathematics of proportionality, and tested the use of the dynabook in teaching experiments at both universities. Figure 1 shows a current version of the Proportionality Dynabook.
Figure 1. A challenging program in the current proportionality dynabook. One bit of terminology that has been useful throughout the project is to use the term candidates for students who use the dynabook, because these students are studying to be teachers, and thus both the word student and teacher are confusing. Teacher educators are the university professors who teach the candidates, and students are the middle school children who the candidates will eventually teach. We organize this paper by telling the story of the first and second versions of the dynabook we created and anticipating a possible third version, which has not yet been fully realized. Given the space limitations, we emphasize the design narrative over the presentation of a specific research study. In a nutshell, the research aspect of the project consisted of teaching experiments at the two participating universities, ranging from 6 hours to 6 weeks in duration. Teaching experiments were carried out during each design iteration. Data collected included observations, interviews, surveys, tests of mathematical knowledge, and the data collected within the tool automatically. A later report will focus on empirical findings in greater detail.
1. The Enriched Math Textbook Metaphor: A Dismal Failure

Our first attempt at a Dynabook was built by digitizing some existing textbooks and enriching them through a mix of dynamic representations and Universal Design for Learning features (see Figure 2). In particular, we started with a Singapore Math text to which we had access and a well-known reform oriented mathematics curriculum. UDL features included a notepad that allowed candidates to take notes on any page as they read,
ISLS
276
ICLS2012
colored highlighters, text-to-speech capabilities, a glossary, and a English-to-Spanish translation dictionary. A how do I say it? feature gave audible examples of how mathematics terminology could be spoken. Further, building on successful prior UDL work in reading, readers were periodically asked to stop and think about what they were reading.
Figure 2. UDL Principles Inspired the First Version of the Proportionality Dynabook. This first attempt was tested by our advisory board as well as in the context of the participating campuses. We present our primary research result with this version of a dynabook without much fanfare: everyone hated it. On one hand, advisors critiqued this dynabook was too texty, asking instead for a more elegant design with better use of graphics and interactive features. There was a grave worry that the metaphor of the book was holding the project back and that the rationale for including different existing textbooks was not strong. On the other hand, advisors wanted to see a revised dynabook that was built more around the unique affordances of interactivity and digital media. They urged a smaller scope of content for the overall effort, with more attention to making the content innovative. A particular important direction was to make the dynabook better suited to use by teacher educators and candidatesto give more attention to what the users could do with it. This feedback led the project team to do an extensive re-thinking of the project. As a consequence, the idea of starting from an existing text was dropped; we agreed to simplify the scope of content to be covered, and to focus our efforts where we could be more innovative through the creation of original content.
2. The Ways In Metaphor: Discovering Our Narrative Arc

Our new model for the Proportionality Dynabook was a matrix of 3 mathematics topics in proportionality by 4 different ways in to the material (see Figure 3). The reduction to 3 topics was intended to enable us to retain the idea of proportionality as a integrative middle school mathematics concept, while reducing the scope of material we would have to write. The four different ways in were based in part upon the strengths of the partners in writing material and in part upon their best guesses as to what would make for highly engaging digital materials for use in course. We imagined the resulting matrix of content as a collection of activities that could be flexibly realized by the teacher educators with their candidates, as they pursued the candidates learning about mathematical content, pedagogy and instruction technology. Doing these activities, we thought, could be a good way to develop candidates TPACK -- technological pedagogical content knowledge.
Figure 3. Our matrix of topics, each with four different ways in. At the same time, the failure of the first prototype also caused us to re-think the purpose of our book. Most generally, we realized that a book must be written for an implied reader (Weinberg & Weisner, 2010) -- that is, with a sense of the activities that a reader will undertake to make sense of our digital creation. However,
ISLS 277
ICLS2012
whereas what a reader does while reading fiction has been well-theorized, what a successful reader does while reading mathematics is not nearly so well understood. Without a better understanding of what success looks like for a reader of mathematics, it would be hard to create a more supportive resource. Further, as we thought about reading math it became clear that it is impossible to successfully learn from a mathematics text without also writing -- reading, doing, and expressing mathematics must be deeply intertwined for learning or even comprehension - to occur. Although mathematics and reading are not the same kind of activity, some lessons can potentially be learned from the reading comprehension literature. One consensus theory of reading comprehension (Duke, Pearson, Strachan, & Billman, 2011) outlines eight basic activities: 1. Setting purposes for reading 2. Previewing and predicting 3. Activating prior knowledge 4. Monitoring, clarifying, and fixing 5. Visualizing and creating visual representations 6. Drawing inferences 7. Self-questioning and thinking aloud 8. Summarizing and re-telling. Some of these have apparent analogies to mathematics, such as activating prior knowledge, visualizing and creating visual representations and drawing inferences. Others, such as previewing and predicting selfquestioning and thinking aloud dont feel quite right, and probably need either significant translation or replacement. One useful aspect of reading comprehension theory, however, is that it gives both a sense of the complexity of the enactment of reading and the sense of the arc of the process -- that it starts from a purpose and ends with a summarizing and retelling, and in the middle, there is a lot of active work with the text. What might be such an arc for mathematics? Fortunately, our design experiments in courses with teacher educators and candidates were much more successful on this round. From these successes, a narrative arc for the Proportionality Dynabook began to emerge. Like many engaging narrative arcs, the Dynabook narrative arc featured a tension in the beginning, some alternatively frustrating and productive muddling around through the middle, and the emergence of resolution towards the end. The most powerful beginning we found for a dynabook experience was for candidates to watch a video of a student who was struggling to solve (what should be) a routine mathematics problem. In particular, our design experiments found that one particular video was most powerful, a video of an actor pretending to be a student named Kayla. (Most Dynabook videos are unscripted recordings of current students; most users are unaware that some videos are of actors.) In the video, Kayla works to solve a ratio problem involving making salad dressing with a ratio of olive oil and vinegar. Kayla is not depicted as deficient in computational or procedural skill in fact shes quite competent. Nor is Kayla depicted as harboring some misconception or other about proportionality. Rather Kayla is portrayed as someone who is not cued by the problem - or even by subsequent queries - to engage in mathematical comprehension behaviors. Like a poor reader with a good vocabulary who believes reading and skimming are the same things, Kaylas deficit is a kind of illiteracy. Watching this video created a felt tension for candidates -- they wanted to help Kayla and could feel that it would be hard to do so, and thus they wanted to learn more about mathematics, pedagogy, and instructional technology as a means to creating a more satisfying conclusion to Kaylas story. The dynabook presented her video with a transcript, and candidates could use the UDL highlighting tool to mark up important teachable moments where they might intervene. However, our field tests with the dynabook suggested that listening to Kayla (not reading the transcript or watching her) was specifically important. From discourse processing research it is understood that people generally find it easier to detect inconsistencies through listening than through reading. And candidates viewing and listening to the video were clearly cued to this inconsistency detection aspect of comprehension. Indeed, they mention that Kaylas illogicallity and inconsistency was, for them, a highly salient take-away from their experience with the video. A powerful ending for the dynabook experience was for candidates to make scripts for how they would teach with Kayla so as to help her. For this we used an existing online tool, called Xtranormal. Xtranormal has the tag line If you can type, you can make movies. Xtranormal invites a user to choose a virtual setting, choose animate avatars, and craft the avatars dialog (with optional stage direction). The settings offered are both recognizable (office, home, park, etc.) and richly detailed. Similarly, the characters offered are iconic, each with
ISLS
278
ICLS2012
quirky - but photogenic - aspect, clothing style, and accoutrement. The dialog, however, is fully in the authors control and is read, verbatim, by a text-to-speech engine in the completed video file. Seeing and hearing their imagined dialogues come to life, so vividly, so simply, was a powerful draw for these candidates. In courses on both university campuses, we found that the activity of making a script for teaching Kayla was intensely engaging for candidates -- as they worked on this activity, opportunities to deepen their own teaching knowledge were abundant. Through field observations and video recordings, we noted that candidates discussed their theories of Kayla, attempted to relate their observations to a research-based theory of the stages in learning ratio and went to both dynabook and web-based resources to find more information about how to define and teach ratio. Importantly, the candidates found hearing the scripts they wrote enacted by avatars both motivating and clarifying -- in listening to the script performed, they could identify unreasonable conceptual leaps in the teacher-student interaction and ways to engage with a students comprehension challenges. In one of the courses we observed, the teacher educator went further by asking candidates to critique each others scripts from the point of view of standards for mathematical practices -- and to suggest improvements that would lead to greater mathematical depth. This seemed to be a powerful way for candidates to connect the ideas in the standards to more grounded conversational moves they might make with a student. One powerful midpoint activity, between the video case and Xtranormal scripting activities, was engaging candidates in solving a related challenging math problem on their own. This activity was supported by the challenging problems section of Dynabook. In this activity, a teacher educator asked the candidates to generate at least two ways to solve a particular challenging problem, involving ratio. In our observations, most candidates could not produce more than one solution. However, the Dynabook allowed the candidates to write and later share their solutions. When the classroom looked at the shared solutions, there was a great deal of astonishment among the candidates -- they were very surprised at the variety of solution methods they had collectively produced. Indeed, the solutions for a ratio problem varied from standard algorithms, to pictorial solutions, or more conceptual approaches. Investigating the properties of the different solutions turned out to be a very productive activity for the candidates to engage in; for example, it really opened their eyes to the potential conceptual power and relative simplicity of non-algorithmic solutions. This, in turn, influenced some of the candidates to propose teacher-student scripts for Kayla that featured problem solving approaches that were conceptually stronger and less algorithmic. Yet, although we observed productive learning experiences for candidates using this version of the Proportionality Dynabook, the tools still seemed less than perfectly tuned to the targeted activity of the implied readers. One major problem was that Xtranormal is oriented to a talk show format for the avatars, and does not provide any easy way for candidates to include mathematical representations in how they propose to teach Kayla. We found that without any concrete referents, the tutorial dialogues were far from optimal and hard to understand. In a quick experiment, we used the spoken dialog produced by Xtranormal but added graphics to the video stream, which we drew by hand. Viewers immediately noticed the graphic increase in how productive and comprehensible the tutorial dialogue seemed. Based on insights from our field tests, we have begun to yet further re-imagine what a dynabook should be.
3. A Dialogic Journal Metaphor: Mathematical Thinking for Teaching
A potential new metaphor for the Proportionality Dynabook is that is should be a journal that is written by teacher candidates as they reflect in their preservice teacher education courses (Yost, Sentner, & ForlenzaBailey, 2000). The use of the journal metaphor acknowledges that to read mathematics is to write and express mathematics too (Siegal, 1989). Journals are often also occasioned by trips; a journal can be the running notes about a journey. In this case the journey is through the Proportionality Dynabook, as they look for better ways to interact with a struggling student (depicted via a video). The practice of journaling often involves writing prompts, and a Dynabook can provide many opportunities for writing about mathematical thinking as candidates progress through our web-based digital resource. Conventional journals, however, are often completely private objects. Dynabook as a dialogic journal is designed to be not merely a private record but a source for interaction among preservice candidates as well as their professors. To accomplish this, we imagine building on existing features in the Proportionality Dynabook that give the candidate the opportunity to share otherwise private notes by publishing them to a shared work area. We imagine generalizing this feature somewhat so that not only notes, but also all forms of work in the Dynabook would be sharable and thus open to further dialog among participants in the teacher education course. In past work, we have found that giving participants easy control of private-to-public transactions is often a key feature in collaborative learning spaces (Vahey, Tatar & Roschelle, 2006).
ISLS
279
ICLS2012
What would candidates write about? Ball and Bass (2003) introduced the term mathematical knowledge for teaching to recognize that K-12 teachers need specialized knowledge of mathematics, relative to their teaching assignments. For example, relative to university mathematicians, teachers often need to be able to think about the unconventional things that K-12 students do when thinking about mathematics but which are completely subsumed as taken-for-granted conventions by the time students get to the university level. Based on our field test experience, however, knowledge seems like too formalized and static a term for the kind of work we expect to see candidates doing; mathematical thinking for teaching seems more apt. That is, we would like candidates to write in their journal in ways that demonstrate their growing ability to engage in extended mathematical thinking relevant to their specific teaching assignments and relative to the thinking their students may exhibit. While journaling typically involves writing, journaling in mathematics is more complex. As much of mathematics is understood through multiple representations, Dynabook should support creating multiple, connected representations through drawing, uploading images, audio recording, and text. Guiding questions can help candidates focus their thinking, and sentence starters or background images can provide hints of how to begin. A central posting wall can serve as a location in which candidates can share their thinking and build on each others work. As we have been planning a potentially new version of the Proportionality Dynabook that leaves behind the enriched textbook metaphor and focuses instead on dialogic journaling, two specific areas of the design have received the most attention. First, we want to more strongly feature the activity of responding to the Dynabook and encourage responses to take advantage of the flexibility of the digital medium. In particular, most digital books still have a fill in the textbox character when it comes to answering questions. In contrast, users of Facebook and other familiar social networking tools are becoming accustomed to being able to easily provide new information in a range of formats: text, pictures, videos, links, etc. The breadth of these options fits a UDL paradigm, however, the media types in a dynabook should more closely reflect the nature of mathematical activity. Hence, we want to feature easy ways to draw models and representations in a response area, and to call up mathematical visualizers for drawing a graph or exploring a geometric construction. We also want to allow spoken (audio) explanations of a mathematical idea as equally valid to textually written explanations. Our teaching experiments have found that candidates often want to write mathematics on paper and then simply take a picture of what they have done, to upload as their response. Most importantly, we want to encourage multimodal responses rather than responding in one format only, as the Multimedia Principle (Fletcher & Tobias, 2005) and related work focuses on the power of transcoding activities for developing and demonstrating comprehension. Hence, it should be possible for a response to be a demonstration of what I did with a virtual manipulative along with a voice over. Or more simply, for a response to be a picture of written mathematics with a short typed explanation. Allowing for responses to be dynamic and time-based, rather than inert and static also would seem to encourage more capture of mathematical thinking. Second, in a related effort, we want to create a tool for enacted scripts of teacher-student interactions that is better tuned to the tutorial context than Xtranormal is. Like the tool in the second dynabook version, candidates should be able to write proposed teacher-learner interactions by typing out a script. And because hearing the script read aloud is so powerful, both in terms of motivation but also in terms of detecting inconsistencies and ways to improve coherence, text-to-speech should be provided to enact the script. Further, the written script is useful when discussing and critiquing a particular approach to working with a student. However, instead of emphasizing a talk show format for the enactment of the script, we prefer to emphasize the models and drawings that teacher and student may be making, referring to, and modifying as they talk. In one simple modality, the candidate who is writing the script should be able to sketch what the teacher and student may be writing on a shared piece of paper. For example, A mathematically literate reader might naturally as a comprehension step sketch the diagram as seen in Figure 4 and then immediately replay the result, listening for inconsistencies, ambiguities and other forms of confusion while following along with the diagram.
ISLS
280
ICLS2012
Figure 4. Adding a sketch of a model and a transcript to a script for how to teach Kayla about ratio. Rather than suggesting a from-scratch generation of the entire alternative experience the new medium could suggest an intervention-as-editing approach. It would seem more natural for an instructor to ask candidates to consider whether Kayla could be understood to be solving some problem perfectly well, just not solving the problem that was posed. They might do so by asking candidates to edit the interviewers statement of the problem (only) so that in the resulting script, Kayla is actually correct in her analysis. Overall, we imagine the newest incarnation of the Proportionality Dynabook focusing on cuing candidates to reflect on their mathematical thinking and how they would deploy mathematical thinking in work with a student who is struggling to make sense of a particular mathematics problem. We imagine this dynabook as being particularly rich in ways to explore how different representations and encodings of mathematical thinking effect comprehension, which can be pursued both as an individual activity and through dialog with fellow teaching candidates. Finally, we imagine this Proportionality Dynabook as collecting examples of candidates growing ability to show how they would work with the thinking of students who are struggling with mathematical thinking, through the intervention-as-editing approach.
Discussion
In the course of three years of design experiments and teaching experiments conducted by a team of investigators at several organizations, our vision for a Proportional Dynabook has evolved considerably, moving quite a distance from the initial enriched textbook model. A key flaw of the initial model was that it resulted in too much text and not enough distinctively digital features. However, merely making a dynabook more multimedia failed to inspire us. Instead, this critique led us to think harder about the intended activities of the implied reader. Our interim model aggregated a collection of intriguing activities for teacher educators to do with their candidates, organized by a matrix of mathematical topics and ways in to the mathematics. Exploring this model in preservice teacher courses helped us to realize a purpose and narrative arc for a Dynabook. The purpose could be to create a unique resource to support intended activities of reading, doing, and expressing mathematics relative to the needs of struggling students. Our narrative arc started with a video of student struggling with apparently routine middle school mathematics and worked towards resolution in a script of how a candidate might engage in mathematical thinking with that same student. In the middle, the candidate might work to deepen her mathematical thinking for teaching, so as to be able to help that student. However, the ways in metaphor for a dynabook did not seem to fully capture the nature of the work of candidates in the dynabook that we intuitively wanted to support. This is presently leading us to a potential new metaphor of dialogic journaling -- the dynabook as a resource that elicits, supports, and enables mutual commentary on candidates writing about their evolving mathematical thinking for teaching. Relative to the 8-component reading comprehension model, we now have a better sense of the first and last component -- what the purpose for reading our proportionality dynabook might be and what the culminating summarizing and re-telling might involve. Videos of students struggling with mathematics, like that of Kayla, seem powerful for evoking a sense of purpose in candidates; they find it easy to envision themselves as working to teach students like Kayla and want to learn to do it better. Re-telling Kaylas story as a scripted teaching episode where Kayla learns more mathematics and summarizing what they learned about mathematical thinking through this exercise seem like a good way to address the last component.
ISLS
281
ICLS2012
Yet, we believe we still have much to learn about what middle components in a comprehension model for a digital resource of this nature might be, and how the digital medium could best support the processes in that model. How can a dynabook draw teachers into reflecting on the potential power and pitfalls of different modalities for comprehending, expressing, and reflecting on mathematics? How can it encourage them to monitor, clarify, and improve their mathematical knowledge for teaching? Can the tensions between journaling as a private activity and the value of dialog with other candidates about mathematical thinking be surmounted? The analogies based on the reading comprehension literature give inspiration but not firm answers; what we need, over time, is for a deeper understanding of how teachers enhance their comprehension of mathematical thinking in relationship to potential new affordances of the digital medium.
References
AACTE Committee on Innovation and Technology (Ed.) (2008), The handbook of technological pedagogical content knowledge (TPCK) for educators (pp. 3-29). Mahwah, NJ: Lawrence Erlbaum Associates. Ball, D. L., & Bass, H. (2003). Toward a practice-based theory of mathematical knowledge for teaching. In B. Davis & E. Simmt (Eds.), Proceedings of the 2002 Annual Meeting of the Canadian Mathematics Education Study Group, (pp. 3 - 4). Edmonton, AB: CMESG/GCEDM. Ball, D. L., & Cohen, D. K. (1996). Reform by the book: What is - or might be - the role of curriculum materials in teacher learning and instructional reform? Educational Researcher, 25(9), 6-8. Carpenter, T. P., & Fennema, E. (1996). Cognitively guided instruction: A knowledge base for reform in primary mathematics instruction. Elementary School Journal, 97(1), 3. Education Week (2011). S. Korea Outlines Ambitious Plans for All-Digital Scholastic Network. Digital Directions, 5 (1), p.11. Duke, N.D., Pearson, P.D., Strachan, S.L., & Billman, A.K. (2011). Essential elements of fostering and teaching reading comprehension. In S.J. Samuels, & A.E. Farstrup (Eds.), What research has to say about reading instruction, 4th edition (pp. 51-93). Newark, DE: International Reading Association. Fletcher, J. D., & Tobias, S. (2005) The multimedia principle. In R. E, Mayer (Ed.) The Cambridge Handbook of Multimedia Learning (pp 117-133). New York, NY: Cambridge University Press. Penuel, W.R., Roschelle, J. & Shechtman, N. (2007). Designing formative assessment software with teachers: An analysis of the co-design process. Research and Practice in Technology Enhanced Learning, 2(1), 51-74. Roschelle, J., Shechtman, N., Tatar, D., Hegedus, S., Hopkins, B., Empson, S., Knudsen, J. & Gallagher, L. (2010). Integration of technology, curriculum, and professional development for advancing middle school mathematics: Three large-scale studies. American Educational Research Journal, 47(4), 833878 Rose, D., & Meyer, A. (2002). Teaching every student in the digital age: Universal design for learning. Alexandria, VA: Association for Supervision and Curriculum Development. Rose, D., Meyer, A., & Hitchcock, C. (Eds.) (2005). The universally designed classroom: Accessible curriculum and digital technologies. Cambridge, MA: Harvard Education Press. Seigal, M. (1989). A critical review of reading in mathematics instruction: The need for a new synthesis (Report No. 301863). Retrieved from http://eric.ed.gov/ERICWebPortal/detail?accno=ED301863. Vahey, P., Tatar, D., and Roschelle, J. (2006). Using handheld technology to move between the private and public in the classroom. In M.A. van't Hooft and K. Swan (Eds.), Ubiquitous computing: Invisible technology, visible impact. Mahwah, NJ: Lawrence Erlbaum. 187-210. Weinberg, A. and E. Wiesner (2010). Understanding mathematics textbooks through reader-oriented theory. Educational Studies in Mathematics: 1-15.
Acknowledgements
This material is based on work supported by the National Science Foundation under Grant No. 0918339. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation..
ISLS
282
ICLS2012
Embodied Artifacts and Conceptual Performances

Dragan Trninic & Dor Abrahamson Embodied Design Research Laboratory, Graduate School of Education University of California at Berkeley, CA 94720-1670 USA trninic@berkeley.edu, dor@berkeley.edu Abstract: Learning scientists are only beginning to appreciate the potential of synergy between two concurrent developmentstheory of embodied cognition and technology of embodied interaction. We characterize and evaluate this prospective synergy from a sociocultural perspective. First we analyze learning in explicitly embodied cultural practices (e.g., surfing), then analogize to the implicitly embodied practice of mathematics. We next contextualize this analogy via interpreting data collected in a design-based research study, in which twenty-two 9-to-11-year-olds developed notions of proportionality through participating in guided problem-solving activities in an embodied-interaction space. In both surfing and mathematics, we argue, learners develop embodied artifacts, i.e. body-based and modular rehearsed actions. Embodied artifacts lend individuals entry into disciplinary competence via participation in action, refinement of operations, and integration into activity structures. Furthermore, embodied artifacts may become conceptual performances, wherein performance not merely augments, but stands for and constitutes understanding.
Introduction
Artifactscultural objects embedded in social practicecontinue to intrigue scholars of human cognition and development. Reasons are at least threefold: (1) philosophically, artifacts constitute essential cultural objects to think-and-act-with; (2) pedagogically, education involves learning to use the tools humanity has found indispensable; and (3) methodologically, artifacts render learning processes more visible for investigative scrutiny. As design-based researchers of educational technologies, the pedagogical artifacts we investigate are historically young. Nevertheless, we approach these novel artifacts from the same theoretical perspectives as we would a seemingly humdrum manual tool (say, a hammer), and ask, What learning gains can such an artifact foster? What can it teach us about human learning and the relation between performance and understanding? For the purposes of this report we are less interested in material artifacts per se, such as a piano or an abacus; neither are we presently concerned with symbolic artifacts, such as musical notes or numerals. Instead we focus our investigations on embodied artifactsrehearsed performances or trained routines, such as the dynamical physicality of playing Fr Elise or manipulating an abacus. As we shall argue, novel motion-sensitive cyber-technologies (e.g., Nintendo Wii, Xbox Kinect) are uniquely geared to both craft and leverage embodied artifacts as means of fostering learning and, for researchers, opening windows onto how learning occurs. We embark from embodied artifacts within explicitly embodied domainsdomains wherein the core practices are patently visible to interested observers. We do so for two interrelated reasons. Rhetorically, to assist the reader by providing transparent examples of embodied artifacts and, strategically, to bring attention to the pedagogical traditions of disciplines largely marginalized in the learning-sciences literature. Disciplines such as dance, martial arts, and various crafts are characterized by pedagogical practices apparently different than those of so-called abstract disciplines such as mathematics, a difference perhaps most clearly articulated by traditional perspectives on transition from performance to understanding. Namely, in abstract disciplines the transition from procedural to conceptual, often modeled by Piagetian encapsulation, is viewed as an epistemic necessity. In this process, a procedural performance is reorganized on a higher plane of thought and so comes to be understood (Beth & Piaget, 1966, p. 247). In contrast, pedagogical traditions of explicitly embodied disciplines seldom promote higher planes of thought as intrinsically necessary or even meaningful for the acquisition of competence (but see reflective practice by Schn, 1983). Instead, disciplinary knowledge is viewed matter-of-factly as emerging in, instantiated through, and assessed via performance per se. In the following section, we take the readers on a guided tour of a few decidedly low-tech instantiations of embodied artifacts. Following that, we frame our investigations of embodied artifacts against the backdrop of increasingly ubiquitous technologies leveraging embodied action. Finally, we introduce one of our recent design experiments as an example of what it may look like for learning-sciences researchers to apply in practice the notion that understanding emerges in and through performance.
Embodied Artifacts in Action: From Surfing and Kicking to Doing Mathematics

To begin, imagine a first surfing lesson in Honolulu, Hawaii. Despite the endless crowds at the sun-drenched Waikiki beach, a neophyte surfer is eager to get in the water. Doing so immediately, however, is likely to invite disappointment. His inability to distinguish the many types of waves, crowding by dozens of other nearby
ISLS
283
ICLS2012
surfers, neuromuscular fatigue from continuous paddling, an uneasy sense of unspoken social hierarchies among more experienced surfers, and a myriad other factors large and small all conspire to quickly dizzy and exhaust the novice. Yet the beachboys (surfing instructors) of Waikiki are famous for claiming they can make anyone ride a waveat least, that is, for a second or two. How? Before getting in the water, the beachboy asks the first-timer to lie down upon the surfboard on the sand. There, the beginner is taught the elementary Stand-Up sequence (SU; our nomenclature) on the surfboard, roughly thus from a prone position: (1) kneel; (2) raise one knee up; and (3) stand up. Only once the beachboy determines the neophyte is capable of executing this basic sequence with confidence does the surfer take to the water, where SUs utility quickly becomes apparent. In the water, the instructor waits for an appropriate wave a selection process beyond the novices capacitythen push his charge into said wave at the appropriate moment and velocity. At this point, all the neophyte must do is paddle hard into the wave and (attempt to) execute SU. A complex activity is thus partitioned into: (a) selecting a wave; (b) approaching a wave; and (c) SU. Hence the beachboys accomplish their claim of getting anyone to surf by performing (a) and (b) on behalf of their charge and sharing with the neophyte an embodied artifact, (c) the elementary SU sequence. Note that the function of an embodied artifact is modular, in the sense that it can be taught and learned as a standalone sequence of operations performed outside the ecological domains of deployment, yet later be contextualized via integration into a larger system. For example, the learner becomes more adept at timing, instigating, and performing SU in respect to his distance to the wave (contextualization via integrationrecall that the SU sequence was learned on sand); and learns a more optimal knee placement during the kneeling portion of SU (refinement via analysis). Learners therefore approach disciplinary competence by entering at the level of actions in the form of rehearsed performances. In other words, embodied artifacts serve as entry into disciplinary engagementas knowledge through practice (cf. Ericsson, 2002) and reflection (cf. Dewey, 1933; Schn, 1983). Importantly, the learner may rehearse elements of this modular action (SU) independently of any larger activity system (surfing; cf. Leont'ev, 1981). We revisit these critical ideas later in the report. Due to their modular nature, embodied artifacts tend to be adaptable in their application. Consider the Flying Sidekick, an aerial attack historically used to strike over ground fortifications (e.g., defensive spikes) and dismount enemy combatants off of warhorses and other beasts of war (see Figure 1b). In modern times, neither spike-barricades nor mounted warriors pose a serious concern, yet this artifact continues its existence as more than a quaint text-bound technique. Due to its modular nature, it survived the disappearance of its original context, mle warfare.
b. a. Figure 1. Embodied artifacts in the wild: (a) novice surfer prepares to use the Stand-Up sequence, as coach, seated, prepares to launch him into a wave; and (b) Soo Bahk Do expert demonstrates a Flying Sidekick. These two brief examples are meant to illustrate some of the variety of embodied artifacts. Whereas embodied artifacts may work in tandem with other artifacts (as in the case of surfing, operating an abacus, or playing the piano), they may merely require space and gravity (such as dance). The critical but common thread is that all embodied artifacts are rehearsed performances, ready-to-hand cultural equipment created by packaging procedures for skillfully encountering particular situations in the world. By mediating ones encounters with the world, embodied artifacts constitute an integral part of cultural and individual development. First, humans embody cultural procedures through participating in social activities. Through observation, demonstration, imitation, and training, these cultural procedures become our resources in the form of embodied artifacts. And, as we shall argue, embodied artifacts may become signified as conceptual performances, that is, embodied procedures that serve as concepts-in-action. Hence, through embodied artifacts we store cultural
ISLS
284
ICLS2012
knowledge in the body, using the body as both the material for and means of encountering the world (cf. Dourish, 2001; Dreyfus & Dreyfus, 1999). As researchers, our interest in embodied artifacts is twofold. First, current empirically supported theories of mind suggest that embodimenthaving and using a physical body in the worldis fundamentally linked to all reasoning, whether involving pure thought or getting ones hands dirty (literally or figuratively). Second, we hold that deliberate use of embodied artifacts in mathematics instruction may render hitherto undetectable learning processes open to both formative assessment in classrooms and empirical scrutiny in laboratories. The idea is simple: if students must perform physically in the service of, say, doing mathematics, then such doing becomes publicly observable rather than hidden away in their heads. Yet in addition to our research practice as learning scientists, we are designers of pedagogical artifacts. As designers, we are interested in availing of novel technologies to engineer learning environments wherein students craft embodied artifacts in pursuit of mathematical competence. We then observe students engaged with our design and, hopefully, we learn more about the process of learning. So doing, in turn, we also learn more about designing learning environments. The design investigations presented here begin from observations about the pedagogical potential of embodied artifacts in light of increasingly ubiquitous motion-sensor technologies. These observations, in turn, form the theme of the following section, where we situate our study in the broader context of research on the role of embodiment in human learning and knowing. From the perspective of educational design, we consider the following question: How, if at all, may novel motion-sensor technologies be pedagogically utilized, particularly in light of recent theoretical advances indicating the fundamental cognitive role of embodiment? In addressing this question, we present a proof-of-existence educational intervention that leverages novel technology, namely the Mathematical Imagery Trainer (hence, MIT, see Figure 2b). Working with the MIT for Proportion (MIT-P), students move their hands in an environment that changes its state in accord with the ratio of the hands respective heights. Via learning to maintain the target state even as they move about in the environmenti.e. to perform a dynamical conservationstudents effectively train an embodied artifact. They then reflect on, analyze mathematically, and ultimately contextualize this artifact as a case of proportionality. Before introducing this design, let us take a moment to briefly frame our work theoretically.
a. b. Figure 2. Embodied artifacts take many forms: (a) traditional Cham dancers, and (b) Mathematical Imagery Trainer for proportion (MIT-P) in use by two students (with instructor looking on).
Framing: The Body, Mathematics, and Technology

The Rise of Embodiment Theory and Its Application to Mathematics Education
Can conceptual understanding emerge from embodied interaction? One answer is that we are physical beings living in a physical world; hence, attempts to understand the development of conceptual thought need look to physical interaction. Perhaps due precisely to its apparent simplicity, this answer has been ignored by cognitive science throughout the last century. Instead, traditional cognitivist views partition mundane interaction into three mutually exclusive constituent facets: perception, thought, and action (e.g., Fodor, 1975). Thoughts and concepts, per those models, intervene between perception and action and are characterized as distinct from those real-time embodied processes by token of being symbolicpropositional. Yet in alternative views discussed below, cognition is neither secluded nor elevated from perception and action but is rather embedded in, distributed across, and inseparable from these corporeal processes. Embodiment studies rose fast in prominence towards the end of last century through the converging efforts of numerous pioneers across various fields like robotics, cognitive science, philosophy, psychology, and
ISLS 285
ICLS2012
computer science (Brooks, 1991; Varela, Thompson, & Rosch, 1991; Winograd & Flores, 1987). Though many of these perspectives initially emerged in opposition to then-prevalent symbolic-architecture models of mind, embodiment studies have, over the last two decades, burgeoned into a vast area of investigation in its own rightreplete with its own spectrum of proponents. Within this spectrum, one might roughly identify: conservatives, who cautiously posit that the body may have a role in cognition (e.g., Dove, 2009); moderates, who argue that physical action underpins or forms the substrate of cognition (e.g., Barsalou, 2010; SheetsJohnstone, 1990); and radicals, who hold that cognition itself is nothing but action (e.g., Melser, 2004). Indeed, the scope of embodiment studies has grown to the point where not a few scholars concern themselves with defining what, exactly, is meant by embodied (Kiverstein & Clark, 2009). In our current work we tend to hold with those who favor the middle ground, and we interpret available empirical evidence as indicating that physical action indeed undergirds thinking, including all so-called abstract thinking (e.g., solving for x). We are not concerned by the controversy over what precise role corporeality plays in thought: indeed, it gives us something to do. Namely, as interaction designers of mathematical learning we find ourselves in a unique position to contribute toward resolving this theoretical controversy. Finally, and particularly relevant to our work, embodiment has been presented as a useful framework for theorizing developmental processes inherent to abstract disciplinary mastery, including mathematics learning and reasoning (Abrahamson, 2009; Campbell, 2003; Namirovsky, 2003; Nez, Edwards, & Matos, 1999; Roth & Thom, 2009). One consequence of this view is that observations, measurements, and analyses of physiological activities associated with brain and body behavior can provide insights into lived subjective experiences pertaining to cognition and learning in general, and mathematical thinking in particular. In a strong form, we conjecture that physical action is neither epiphenomenal nor merely supportive to pure mental activity. Rather, conceptual understandingincluding reasoning about would-be abstract contents such as pure mathematicsemerges through and is embedded in actual and simulated perceptuomotor interactions in the world.
Technology for Using the Body

Even as cognitive scientists recognize this essential role of the body, industry has made dramatic advances in engineering technological affordances for embodied interaction. At the time of this writing, Nintendo Wii and Playstation Move players worldwide are waving hand-held wands so as to remote-control virtual tennis rackets; iPhone owners are tilting their devices to navigate a virtual ball through a maze; and Xbox Kinect users are controlling video-game avatars with their bare handsall increasingly commonplace activities hitherto confined to the realms of futuristic fantasy, like flying cars. Moreover, innovative designers tuned to this progress are constantly devising ways of adapting commercial motion-sensor technology in the service of researchers and practitioners (Antle, Corness, & Droumeva, 2009; Lee, 2008). As such, media that only recently appeared as esoteric equipment will imminently be at the fingertips of billions of potential learners. We are excited about the prospect of using these new media to create learning environments, wherein students discover, craft and rehearse embodied artifactsthen subsequently investigate them via mathematics, gaining embodied entry into this disciplinary domain. Specifically, our use of new media takes the form of Embodied-Interaction (EI) learning environments. In brief, EI is a form of technology-supported training activity, by which users develop or enhance cognitive resources that presumably undergird specialized forms of human practice, such as proportional reasoning. As is true of all simulation-based training, EI is particularly powerful when everyday authentic opportunities to develop the targeted schemes are too infrequent, complex, expensive, or risky. Emblematic of EI activities, and what distinguishes EI from hands on educational activities in general, whether involving concrete or virtual objects, is that EI users physical actions are intrinsic, not just logistically instrumental, to obtaining information. That is, the learner is to some degree physically immersed in the microworld, so that the embodied artifactinstantiated in finger, limb, torso, or even whole-body movementsemerges not only in the service of acting upon objects but rather the motions themselves become part of this learned culturalperceptuomotor structure. EI is hands in.
Pedagogical & Experimental Design: The Mathematical Imagery Trainer

Our overarching design conjecture is that some mathematical concepts are difficult to learn because mundane life does not occasion opportunities to embody and rehearse their spatialdynamical foundations. Specifically, we conjectured that students canonically incorrect solutions for rational-number problemsfixed difference solutions (see Lamon, 2007 for an overview)indicate students lack of appropriate dynamical imagery to ground proportion-related concepts (Pirie & Kieren, 1994). Accordingly, we engineered an EI computer-supported inquiry activity for students to discover, rehearse, and then objectify and signify the physical performance of a particular proportional transformation of our design. Hence, our solution to this general design problem is the Mathematical Imagery Trainer for Proportion (MIT-P, see Figure 3; see http://www.youtube.com/watch?v=n9xVC76PlWc).
ISLS
286
ICLS2012
Technologically, our instruction design leverages the high-resolution infrared camera available in the inexpensive Nintendo Wii remote to perform motion tracking of students hands. Details on technical, interface, and design properties can be found elsewhere (see Howison, Trninic, Reinholz, & Abrahamson, 2011).
b. a. Figure 3. MITP in action: (a) a students incorrect performance (raising both hands up at equal increments, thus maintaining fixed distance between the hands) turns the screen red; (b) correct performance (raising both hands up at proportionate increments, i.e. different distances) keeps the screen green. In practice, the MIT-P measures the heights of the users hands above the desk. When these heights match the unknown ratio set on the interviewers console, the screen is green (see Figure 4). Hence, the embodied artifact of the MIT activity is the dynamical physical articulation of all the pairs effecting a green screen. As such, the initial purpose of the MIT is to train a particular proportion-relevant embodied artifact.
a. b. c. d. Figure 4. The Mathematical Imagery Trainer (MIT) set at a 1:2 ratio, so that the right hand needs to be twice as high along the monitor than the left hand: (a) incorrect performance (red feedback); (b) almost correct performance (yellow feedback); (c) correct performance (green feedback); and (d) another correct performance. Participants included 22 students from a private K8 suburban school in the greater San Francisco (33% on financial aid; 10% minority students). Care was taken to include students of both genders from low-, middle-, and high-achieving groups as ranked by their teachers. Interviews took place in a quiet room within the school facility. Students participated either individually (see Figure 3) and/or paired with a classmate (see Figure 2b) in task-based, semi-structured clinical interviews (duration: mean 70 min.; SD 20 min.). In addition to the interviewer, at least one observer was present, whose duty included taking written notes in real-time, crewing the video camera, and assisting in operating the technological system. Study participants were initially tasked to move their hands so as to find a position that effects a green screen and, once this objective was achieved, to keep moving their hands while maintaining a continuously green screen. That is, the participants needed to discover a means of enacting a green-keeping performance. In a sense, the MIT offers students a pre-numerical Whats-My-Rule? mathematical game. The protocol included gradual layering of supplementary mathematical instruments onto this microworld, in the form of Cartesian grid without and then with numerals. Hence, once students have embodied the proportional-transformation dynamical artifact, semiotic resources (mathematical instruments) and discursive support (the tutor) are present for this embodied artifact to be mathematically signified, elaborated, and analyzed. The interview ended with an informal conversation, in which the interviewer explained the objectives of the study so as to help participants situate the activities within their school curriculum and everyday experiences. Finally, the interviewer answered any remaining questions participants may have had with the objective that participants achieve closure and depart with a sense of achievement in this challenging task. Our investigation of the empirical dataethnographic notes, daily debriefs, and video recordingswas conducted as collaborative, intensive microgenetic analyses (Schoenfeld, Smith, & Arcavi, 1991).
ISLS
287
ICLS2012
Findings
All students succeeded in devising and articulating strategies for making the screen green, and these initially amathematical strategies came to be aligned with the mathematical content of proportionality (with varying degrees of success). This particular finding serves as a proof-of-existence supporting the conjecture that embodied artifacts can create pedagogical opportunities to support student learning of targeted mathematical concepts. Naturally, there existed variations in individual participants initial interpretation of the task as well as consequent variation in their subsequent trajectory through the intervention protocol. But, importantly, the students progressed through similar problem-solving stages, with the more mathematically competent students generating more strategies and coordinating more among quantitative properties, relations, and patterns they noticed. We now elaborate on this learning trajectory. Each student began either by working with only one hand at a time, waving both hands up and down in opposite directions, or lifting both hands up at the same pace, occasionally in abrupt gestures. They realized quickly (< 1 min. on average) that the simultaneous actions of both hands are necessary to achieve green and, consequently, that the vertical distance between their hands was critical, although at first they viewed the distance between their hands as fixed. We found this default fixed distance approach of importance, as it arguably matches an enduring (mis)conception of students seeing 2/3 as the same as 4/5. Indeed, our hope was that by uncovering and addressing such conceptions physically, we could access and affect students prenumerical conceptual reasoning underlying their prospective arithmetic competence with this target concept of proportional relations. In brief, students were given the initial opportunity to amathematically develop embodied artifacts related to proportionally by enacting green-preserving action sequences, i.e., embodied artifacts. Gradually the protocol contextualized students actions within the broader world of mathematical proportions. This was achieved by providing mathematical tools as semiotic resources for re-articulating the embodied artifact (by way of analyzing and expressing it mathematically). As such, and via the tutors attentive guidance, the MIT-P introduced an embodied artifact via which students gradually instantiated and complexified the cultural practice of proportional reasoning. Similar to the Waikiki surfer who embodied, utilized, contextualized, and refined his surfing actions initially learned on sand, the students in our study integrated their initially amathematical moves into the broader world of proportional mathematics. While the pedagogical effectiveness of MIT cannot be determined through this pilot study, we find reason for cautious optimism.
A Brief Study Excerpt

As Liat, a 5th-grade female student, attempted to discover a means of making the screen green, she initially exhibited the fixed distance (i.e. same difference) approach described above. Liat: I think if I keep them [her hands] apart and keep going up [she keeps a fixed-distance between her hands], it stays the same... [i.e., remains green] Int.: If you keep them apart and you keep going up it stays the same? Liat: Its not becoming red, but... Int: So... how are you thinking about keeping them apart? Liat: [she attempts to effect green again] Oh maybe its more. If its farther up, then it has to bethey have to be more apart. Above we see Liat coming to verbally articulate her increasingly skillful performance, i.e., embodied artifact that enacts green. Later, upon the introduction of the Cartesian grid and other mathematical tools into the learning environment, she was asked to predict green locations without moving her hands. Referring to the green pair 1 (left cursor) and 2 (right cursor), Liat noted that the vertical interval in between the cursors is one row. Having objectified and quantified the interval, she said: Liat: And if you go 10 if you go up to 10, theres gonna be like 4 or 5 rows. [i.e., if the right hand is at 10, the interval down to the left hand should be 4-5 rows, so as to make green.] Thus, prior to the introduction of the grid, Liats articulation of the embodied artifact for green had been based on the qualitative relation of farther up and more apart, yet once the grid was introduced, she instrumentalized it so as to analyze her action, rendering the description quantitative. So doing, Liat reflexively improved her performance by introducing greater precision. However, the embodied artifact was yet to become articulated in an epistemic form supporting the ultimate emergence of the 1:2 ratio. To do so, Liat would need to calibrate her expression from approximate to specific, possibly via assimilating the grids numerical affordances into her enactment, so that some consistent quantitative principle will obtain across all green locations. Indeed, immediately after, Liat transitioned to a more powerful mode of reasoning that involved multiplicative relations. The transition was prompted by the interviewers request that she decide between 4 or 5 rows. Liat: No five! Int: Five? How did you do that so quickly? How did you know it was five? Liat: Half of ten is five.
ISLS
288
ICLS2012
In above excerpts and the ensuing discussion, Liat came to re-describe and refine her newly developed green finding embodied skill via mathematical operations. She does so by, first, noticing an empirical, functional commensurability between the qualitative rule further upmore apart and the quantitative halving/doubling property she had just discovered and, ultimately, by reconciling the manual performance as a case of proportion. So doing, she comes to see different differences as a legitimate mathematical concept, beginning to contextualize it more generally as a case of proportion. As another student chimed in at the end of the session, Now I know its OK to be different. Here, we posit, through reconciling rehearsed physical performance (moving hands at appropriate rates to make green) with disciplinary content (proportion), an embodied artifact became signified as a conceptual performance that stands for and constitutes understanding.
Discussion
We have been threading together two central themes. First is that movement matters. Physically interacting in a physical world is our mode of being and the roots of our thinking. This thread, then, dealt with the relation between performance and understanding: namely, we interpreted existing embodiment studies as suggesting that conceptual understandingincluding reasoning about would-be abstract contents such as pure mathematics emerges through and is embedded in actual and simulated perceptuomotor interactions in the world. We introduced the construct of an embodied artifact as a means of articulating how cultural practices are packaged and shared with novices, enabling their entry into the world of skillful action and disciplinary competence. Second theme is that recent decades have witnessed advances in remote-interaction cybertechnologies. We opined that critical questions have remained unanswered regarding the interaction of these two themes of theory and technology as this interaction pertains to design for learning. It is in embodied-interaction (EI) design that our two themes meet. We envision EI as a form of physically immersed instrumented activity geared to augment learning by crafting embodied artifacts targeted towards specific disciplinary practice. In pursuing design problems, our strategy has been to engage in conjecture-driven cycles of building, testing, and reflecting on these two themes and their potential synergy. Here we aim to share a conviction that EI offers unique affordances for mathematics learning via introducing and signifying an embodied artifact, such as a dynamical conservation discussed above, as a conceptual performance. Learning in EI occurs in contexts where participants embody practices in the forms of embodied artifacts and come to make meaning of those artifacts as conceptual performances (see Lave & Wenger, 1991 on learning as participation rather than acquisition; and Vygotsky, 1982 on the emergence of meaning) We anticipate that, when coupled with recent technological advances, EI stands to become a focus of design forand research onmathematics learning. As our work indicates, EI activities serve as highly useful empirical settings for research on the ontogenesis of mathematical concepts and, more generally, relations between performance and knowledge in mathematics education. In particular, these immersive activities create opportunities for design-based researchers to observe and resolve ostensible theoretical tension between: (a) unreflective orientation in multimodal instrumented spaces, such as riding a bicycle or playing ping-pong; and (b) reflective mastery over symbol-based re-description of this acquired competence, such as in mathematical form. We hope through this line of investigation to develop models of embodied mathematics instruction. Researchers could look to diverse culturalhistorical forms of physical performance, such as music, dance, and the martial arts, as ethnographic entries into traditional and indigenous pedagogical acumen accumulated over centuries or even millennia. The skills inherent to these cultural practices might, at first blush, be viewed as aconceptual and, as such, hardly bearing on mathematical reasoning and learning. Yet as recent theoretical and empirical work, including our own, suggests, our shared biology implies that even the most abstract of mathematical concepts may first be embodied, then verbally articulated, and finally reified in conventional semiotic forms (Nez, et al., 1999; Saxe, 1981). Such issues are more than academic. All too often proverbial lines are drawn in the sand regarding the importance of conceptual knowledge versus procedural performance (e.g., see Schoenfeld, 2004 on "math wars"). Yet physical actions performed in disciplinary activity constitute vital aspects of cognition and knowledge (cf. Ala & Hutchins, 2004; Kirsh, 2009, 2010). Embodied knowledge, we submit, is developed, elaborated, and expressed as situated conceptual performance. In future work, we will continue to investigate the embodiment of mathematical concepts through the reciprocal efforts of developing theories of embodied learning, and designing educational technologies.
References
Abrahamson, D. (2009). Embodied design: constructing means for constructing meaning. Educational Studies in Mathematics, 70(1), 27-47. Ala, M., & Hutchins, E. (2004). I see what you are saying: action as cognition in fMRI brain mapping practice. Journal of Cognition and Culture, 4(3), 629-661. Antle, A. N., Corness, G., & Droumeva, M. (2009). What the body knows: exploring the benefits of embodied metaphors in hybrid physical digital environment. Interacting with Computers, 21(1&2), 66-75.
ISLS 289
ICLS2012
Barsalou, L. W. (2010). Grounded cognition: past, present, and future. Topics in Cognitive Science, 2, 716-724. Beth, E. W., & Piaget, J. (1966). Mathematical epistemology and psychology. Dordrecht: D. Reidel. Brooks, R. A. (1991). Intelligence without representation. Artificial Intelligence, 47, 139-159. Campbell, S. R. (2003). Reconnecting mind and world: enacting a (new) way of life. In S. J. Lamon, W. A. Parker & S. K. Houston (Eds.), Mathematical modeling: a way of life (pp. 245-256). Chichester: Horwood Publishing. Dewey, J. (1933). How we think: a restatement of the relation of reflective thinking to the educative process. Boston: D.C. Heath. Dourish, P. (2001). Where the action is: the foundations of embodied interaction. Cambridge, MA: M.I.T. Press. Dove, G. (2009). Beyond perceptual symbols: a call for representational pluralism. Cognition, 110(3), 412-431. Dreyfus, H. L., & Dreyfus, S. E. (1999). The challenge of Merleau-Ponty's phenomenology of embodiment for cognitive science. In G. Weiss & H. F. Haber (Eds.), Perspectives on embodiment: The intersection of nature and culture. New York: Routledge. Ericsson, K. A. (2002). Attaining excellence through deliberate practice: insights from the study of expert performance. In M. Ferrari (Ed.), The pursuit of excellence in education (pp. 21-55). Hillsdale, NJ: Erlbaum. Fodor, J. A. (1975). The language of thought. Cambridge, MA: Harvard University Press. Howison, M., Trninic, D., Reinholz, D., & Abrahamson, D. (2011). The Mathematical Imagery Trainer: from embodied interaction to conceptual learning. In G. Fitzpatrick, C. Gutwin, B. Begole, W. A. Kellogg & D. Tan (Eds.), Proceedings of the annual meeting of CHI. Vancouver, May 7-12, 2011. Kirsh, D. (2009). Projection, problem space and anchors. In N. Taatgen, H. van Rijn & L. Schomaker (Eds.), Proceedings of the Cognitive Science Society 2009 (pp. 2310-2315). Mahwah, NJ: Lawrence Erlbaum. Kirsh, D. (2010). Thinking with the body. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Meeting of the Cognitive Science Society (pp. 2864-2869). Austin, TX: Cognitive Science Society. Kiverstein, J., & Clark, A. (2009). Mind, embodied, embedded, enacted: one church or many? Topoi, 28. Lamon, S. J. (2007). Rational numbers and proportional reasoning: toward a theoretical framework for research. In F. K. Lester (Ed.), Second handbook of research on mathematics teaching and learning (pp. 629667). Charlotte: Information Age Publishing. Lave, J., & Wenger, E. (1991). Situated learning: legitimate peripheral participation. New York: Cambridge University Press. Lee, J. C. (2008). Hacking the Nintendo Wii Remote. IEEE Pervasive Computing, 7(3), 39-45. Leont'ev, A. N. (1981). The problem of activity in psychology. In J. V. Wertsch (Ed.), The concept of activity in soviet psychology (pp. 37-71). Armonk, NY: M.E. Sharpe. Melser, D. (2004). The act of thinking. Cambridge: The MIT Press. Namirovsky, R. (2003). Three conjectures concerning the relationship between body activity and understanding mathematics. In N. A. Pateman, B. J. Dougherty & J. T. Zilliox (Eds.), Proceedings of PME 2003 (Vol. 1, pp. 105-109). Columbus, OH: Eric Claringhouse. Nez, R. E., Edwards, L. D., & Matos, J. F. (1999). Embodied cognition as grounding for situatedness and context in mathematics education. Educational Studies in Mathematics, 39, 45-65. Pirie, S., & Kieren, T. (1994). Growth in mathematical understanding: how can we characterize it and how can we represent it? Educational Studies in Mathematics, 26(2-3), 165-190. Roth, W.-M., & Thom, J. S. (2009). Bodily experience and mathematics conceptions: from classical views to phenomenological reconceptualization. In L. Radford, L. Edwards, & F. Arzarello (Eds.), Gestures and multimodality in the construction of mathematical meaning [Special issue]. Educational Studies in Mathematics, 70(2), 175-189. Saxe, G. B. (1981). Body parts as numerals: a developmental analysis of numeration among the Oksapmin in Papua New Guinea. Child Development, 52(1), 306-331. Schoenfeld, A. H. (2004). The math wars. Educational Policy, 18(1), 253-286. Schoenfeld, A. H., Smith, J. P., & Arcavi, A. (1991). Learning: the microgenetic analysis of one student's evolving understanding of a complex subject matter domain. In R. Glaser (Ed.), Advances in instructional psychology (pp. 55-175). Hillsdale: Erlbaum. Schn, D. (1983). The reflective practitioner: how professionals think in action. New York: Basic Books. Sheets-Johnstone, M. (1990). The roots of thinking. Philadelphia: Temple University Press. Varela, F. J., Thompson, E., & Rosch, E. (1991). The embodied mind: cognitive science and human experience. Cambridge: The MIT Press. Vygotsky, L. S. (1982). [Thought and word]. In V. Davydkv (Ed.), L.S. Vygotsky, collected works, vol. 2 (pp. 5-361). Moscow: Pedagogika. (In Russian). Winograd, T., & Flores, F. (1987). Understanding computers and cognition: a new foundation for design. Boston: Addison-Wesley Professional.
ISLS
290
ICLS2012
The Impact of Structural Characteristics of Concept Maps on Automatic Quality Measurement

H. Ulrich Hoppe, Jan Engler, Stefan Weinbrenner Collide Research Group, University of Duisburg-Essen, Duisburg, Germany Email: {hoppe, engler, weinbrenner}@collide.info Abstract: Concept maps are often used in early phases of learning scenarios to externalise and possibly develop the conceptual understanding of a certain domain. An assessment of such maps can help to detect misconceptions and can serve as a basis for learner feedback. If used as a basis for agent-based scaffolding, the assessment needs to be calculated automatically by the system at run-time. Previous approaches have used an expert concept map as a reference for calculating such quality measures. This paper describes an approach that works without an explicit expert concept map but only uses more general domain ontology. To determine and adjust an automated quality measure student concept maps were collected and analysed by experts. The expert judgment was compared to certain structural measures based on graph-theoretical concepts and to enriched measures that additionally included ontology relations. It turned out that the inherent characteristic of concept maps as scale-free networks rules out certain structural measures as positive quality indicators.
Introduction
In a variety of pedagogical scenarios, the creation of concept maps is used to externalise and represent the conceptual or qualitative understanding of a domain. A typical activity can be the semi-formal visual encoding of important statements from a given text on the part of the students. The creation of a concept map from a text requires the reader to identify relevant concepts of the text and find appropriate relations between them. In this sense, they can, e.g., be used to develop and reify conceptualisations of scientific knowledge at an early stage of an inquiry learning process. The context for the work presented in this paper is the European research project SCY (Science Created by YOU, http://www.scy-net.eu/). The SCY project aims at engaging and supporting students in inquiry learning activities using computer tools such as simulations and modelling software. In SCY-Lab (the SCY learning environment), students work on missions with specific challenges. To flexibly meet the challenges of these missions, the configuration of SCY-Lab is adaptive to learners capabilities and progress. In order to support the cyclic nature of inquiry learning, SCY-Lab provides tools and scaffolds, which provide justin time support. Among others, SCY features concept mapping for working on texts and ontologies to enable intelligent agents to support domain-aware scaffolding. A quality assessment of concept maps created by the students can provide valuable diagnostic information for teachers as well as for pedagogical agents and scaffolding mechanisms. Several suggestions have been made for automatically calculating such assessment measures, most of these using a reference concept map created by an expert on beforehand. In our approach, the student map is not compared to a predefined expert map but structural graph measures are used in combination with a projection of a given domain ontology. Such an ontology is part of every mission in SCY and provides semantic background knowledge for the pedagogical agents. The guiding question for our technical design and an ensuing empirical study was to check the possibilities of enabling the system to reliably interpret and analyse students maps using an ontology and structural graph measures and to explore the ensuing potential of scaffolding.
Related Work
As concept maps can be used for multiple purposes in education, much related research can be found in several disciplines. Especially the use of concept maps for assessment purposes has been examined in several research studies (e.g. Stoddart et al., 2000; Rye & Rubba, 2002; Ruiz-Primo, 2004). The Reasonable Fallible Analyser (RFA) by Conlon (2004) is similar to our approach in that it determines the quality of concept maps based on certain heuristic rules using structural and lexical properties. The score of a map is calculated by the comparison of the concepts of the learners map with the concepts of a map that was created by an expert. To be able to deal with synonyms, the RFA is connected to a web-based lexical database (WordNet) and a user dictionary that can be edited to define additional synonyms. Conlon claims (2004) that a quantitative score that represents an overall assessment of the concept map could help in giving the feedback to the learner. In contrast to our approach, the RFA utilizes (as many other concept map assessment approaches) a predefined expert concept map to compare it to the map produced by the learner.
ISLS
291
ICLS2012
Since, in the SCY project, a domain ontology has been created for each mission, we extract this expert map based on the ontology and the text that will be provided to the learners. Betty's Brain (Gupta et al., 2005) is an example of an educational environment based on the learning by teaching paradigm. The learner teaches an avatar called Betty by designing a concept map in different topics. By switching to the query mode of the system, the learner can ask questions about the previously inserted concepts. A reasoning mechanism tries to answer these questions based on the concepts in the map. The focus of this work is not on developing a quality measure of the concept maps but on gathering information about the learners knowledge in particular domains. The COMPASS project (Gouli, Gogoulou & Grigoriadou, 2003) also deals with the usage of concept mapping in the process of learning. The central component of COMPASS is the learning environment, which allows the learners to work out several concept map based activities. COMPASS is independent of a specific learning domain. COMPASS aims at combining the learning process (using concept maps) and the assessment of the results. An automatic assessment is provided by comparing the students solution with an expert map created by a teacher (Gouli et al., 2005) in a similar way as done by Conlon. Comparable to our approach, COMPASS makes use of ontologies to support the learner (and the teacher) during the learning process. Beside a domain ontology, which is used to generate semantic feedback, a student ontology and a content knowledge ontology are used to represent a learner model (level of domain knowledge, interaction data, etc.) and to store educational knowledge (Kilanioti et al., 2008).
Research Questions and Objectives

The initial leading goals for our research efforts were the creation and validation of a quality measure for concept maps and the testing of different scaffolds to be applied in the creation phase that might positively influence the map quality. To determine a quality measure, in addition to semantic matching also several graphbased measures were taken into consideration. As a reference criterion for our quality measure we asked four experts to assess the collected student maps. Our study was guided by the following research questions: 1. Which of the automatically determined variables correlate to the quality assessment of the experts and thus can contribute to a valid quality measure? 2. Do concept maps that were created with ontology-based scaffolding differ in any of the measures to the concept maps that were created without help? The quality assessment by the experts was based on guiding criteria inspired by Marra (2002). Among the four criteria suggested in this proposal, we decided to concentrate on the first two of which (crit1) covers the observed completeness or richness of the concept map as compared to the given text source, whereas (crit2) relates to the interconnectedness of the map. The other two original criteria focus on the descriptiveness and the adequacy of the links and were not used in this first order comparison. However, we tried to avoid this dimension of variation by restricting the link labels to a small given set of typical relations. This also ensured a better match with the ontology.
Implementation
Our implementation is based on the SCY system architecture, which provides a blackboard-based multi-agent framework with access to different user/system logs at the backend and the SCY-Lab frontend for user interaction. For our purposes, we have modified and extended the existing concept mapping tool SCYMapper.
SCYMapper
The SCYMapper is a simple and easy to use concept mapping tool, i.e. it allows for managing graph structures, creating labelled nodes and edges, both in different shapes and colours. Moreover, it is easily configurable for special needs, so that new shapes or colours can be inserted or the default ones can be deactivated. In our case, we configured the SCYMapper to have only one blue shape and one simple edge, because our backend does not impose any semantics on the shape of a node. One main advantage of the SCYMapper is that it is integrated in the SCY architecture. Technically, this means that all the actions that are done in SCYMapper are logged in a standardised format to the backend and that there are standardised message formats to easily send notifications to the SCYMapper.
Ontology
The knowledge that is used by the system to determine the progress and to calculate help proposals is encoded in an ontology that was created in the SCY project as a joint work of the educational and ontology experts. The core of this ontology was created in a traditional, manual way by using the Protg ontology editor. Additionally, a data mining process is used to semi-automatically add new keywords to the ontology. The advantage of such an ontology extension is that the system finds more concepts in a given text and, on the other hand, also recognises more concepts that students have created. This process uses the LDA algorithm
ISLS 292
ICLS2012
(Blei et al., 2003) to extract groups of keywords with a high probability of being related to ontology concepts from a given document repository. In most cases these resulting keyword clouds need manual renaming or regrouping, before they can be integrated with the ontology. In a next step, the ontology expert links these keyword clouds to the ontology in cooperation with the domain expert. This whole process is supported by a tool called Ontology Keyword Importer (OKI). Figure 1 shows the ontology on the left side, the keywords on the right side and the possibility to link keyword clouds to selected ontology nodes at the bottom. Links between ontology concepts and keyword clouds were created at such ontology nodes for which the domain expert saw a semantic connection.
Figure 1. The interface of OKI (Ontology Keyword Importer).
Architecture and Agent Support

The part of the SCY architecture that is most relevant to this work is the pedagogical agent framework (Weinbrenner & al., 2010). It is based on a blackboard architecture (Erman et al., 1980), i.e. the different agents only communicate over a shared platform (the blackboard) and not directly with each other. By monitoring and changing the contents of this platform, each agent contributes to solving a problem according to its specific competences. The SCY blackboard architecture is based on the TupleSpaces approach (Gelernter, 1985). In a TupleSpaces system, each participating system is a client of the central TupleSpaces server and is able to read, write and take tuples to and from the server. The concrete platform on which this is implemented is called SQLSpaces (Weinbrenner et al., 2007). The SQLSpaces server hosts several subspaces that are used to store different kinds of data: The action space contains all actions in the common action log format. There is another space for all inter-agent communication, which is called the command space. Finally, also the ontology is stored as a set of RDF tuples in the ontology space. Four agents are working on these three spaces: - The enricher is a so-called decision maker agent that only coordinates other agents. It recognises the need for help and triggers other agents. After having received the responses of the other agents, it writes a notification tuple in the command space. - The modeller constantly monitors the user actions in order to inform other agents about the current state of the users concept map. - The notifier waits for notifications to be written in the command space and routes these to the user.
ISLS 293
ICLS2012
- The proposer responds to the enricher to return the next hint to a learner. When the proposer is asked to generate concept or relation proposals, it first fetches the current user map from the modeller. Then, it determines the ontology projection by matching the given text against the concept in the domain ontology. This matching involves the following steps: First, all words in the text are stemmed and compared to the stemmed words in the ontology. Then all ontology terms that occur in the text are collected. Ensuingly, the same is done for the semi-automatically added keywords. Next, all the ontology concepts that have been found in that way are inter-linked by using the original ontology relations. This cut of the ontology is now compared to the users concept map and all the missing concepts and relations are determined. In order to propose the most important concepts, for each concept of the ontology projection a relevance value is calculated that depends on the graph centrality (degree) of the concept. Ontology terms that appear literally in the text are considered more relevant than keywords that have a common stemmed form with one of the words in the text. So all the terms that have been found are ranked according to their aggregated relevance value and the top ranked terms (in our case the top 3) are presented to the user. In spite of stemming and of the keyword cloud extension, still not all student terms can be matched to ontology terms even though there may be semantic equivalences. To solve this problem, students can explicitly declare a term used in a concept node of a map in SCYMapper as a synonym of a proposed concept (see Concept already exists! option in Figure 3). This will result in a message to the internal agents that will then establish an explicit synonym link and will handle this synonym accordingly. Figure 2 shows this matching process in a visual interface that has been used for inspecting and demonstrating the system. On the top the text, used by both the students and the agent, is displayed. The areas highlighted in yellow contain the terms that have been matched according to above mentioned matching strategy. Below this, on the right side the state of the current student concept map can be seen. On the left side the ontology projection that corresponds to the text is located. Here, the green labels mark those concepts of the students concept map that were found in the ontology, whereas the red labels identify the most relevant but missing concepts, which are to be proposed to the student as scaffolds.
Figure 2. Visualisation of the proposer agent
ISLS
294
ICLS2012
Study
Building Blocks for Quality Measurement
One main goal of our study was to find an adequate, automatically computable (possibly multi-factor) quality measure that would be maximally consistent with human expert judgement in terms of a high correlation to expert evaluation scores for crit1 and crit2 (see section 3). One type of factors or variables were based on the graph structure, such as the number of nodes (#N) and edges (#E), the relation/concept ratio (RC) and the density (D, calculated by (2#E / #N(#N-1) ). On the other hand, the ontology was used to calculate the number of nodes (#N) and edges (#E) in a map matched by the ontology. The matching process involves stemming of the terms, removing stopwords and ordering them alphabetically in order to cover different formulations of the same concept, for instance the concept increase of sea level would be processed to increas of sea level, then increas sea level and finally increas level sea. The ordering is particularly important in order to handle the concept increase of sea level in the same way as the concept sea level increase. Finally, also the weighted number of such concepts (#NW) was taken into consideration. The weighting calculated as the graph theoretical measure of degree centrality for each node of the ontology projection.
Study Design
Thirty-seven pupils of a secondary school in Duisburg, Germany, aged between 16-18 years participated in the study. Participants were introduced to concept mapping and the usage of the SCYMapper, including a demo, in a 45 minutes introduction. After the introduction, everyone received two pages of text about the global warming and a highlighter pen. Every pupil worked individually first highlighting the relevant keywords in the text, and afterwards on one computer. They were divided into two groups of twelve and one group of thirteen pupils. The three groups worked on different versions of SCYMapper offering different types of help. The on-demand group had two additional buttons, which were labelled "Request concept help" and "Request relation help". The second group received automatic help after an exploration phase of three minutes and a minimum count of five concepts. The last group did not receive any help. The help for the first two groups appeared in the right part of the SCYMapper screen (compare Figure 3).
Figure 3. Screenshot of SCYMapper with automatic help. The group without help worked in a different location to really prevent them from accessing help information. The other two groups worked together in one room. The task was to create a concept map from the given text using the SCYMapper tool. The three suggestions displayed in the example situation of Figure 3 were generated using the approach described above, i.e. considering not only their occurrence in the ontology
ISLS 295
ICLS2012
projection but also their degree centrality as a measure of relevance. The option Concept already exists! should be selected by the students to express that they had already covered this concept by another term (not in the ontology). In a next step, they would be asked to identify this term in their map. Then this term would be marked as a synonym of the suggested one. Pupils were assigned to groups based on a questionnaire that covered prior experience with concept mapping techniques as well as knowledge and interests in the domain of global warming and environmental issues. The results were mapped to a numerical scale, so that the pupils were assigned in a balanced way to the three groups. The text given to the pupils dealt with global warming. It consisted of 942 words and started with a definition of the phenomenon of global warming followed by a short explanation of the assumed causes. The main part of the text enumerated detailed consequences and effects on the ecosystem and the human living conditions. Finally, possible technological and geopolitical countermeasures were mentioned. Thirty-seven concept maps and the corresponding action log sequences were collected and analysed. Additionally, the pupils concept maps were assessed blindly by four experts on a scale from 0 to 4 based on the guiding criteria. These results were also evaluated according to the group division.
Results
For both criteria (completeness and interconnectedness) that were used to judge and to measure the quality of the students concept maps, correlations were calculated using Pearsons product correlation. As for research question 1, the correlations between the automatically calculated values and the experts judgements were calculated. The human assessed completeness correlates positively with the number of nodes #N (r=.792, p=.000), the number of edges #E (r=.741, p=.000) and the weighted nodes #NW (r=.387, p=.018), whereas the interconnectedness correlates with the number of edges #E (r=.625, p=.000) and the relation-concept ratio RC (r=.680, p=.000; all Pearson correlation). In sum, results showed that there is a medium to high correlation between the human judgement and the automatic assessment. Table 1: Correlations of structural measures with expert judgments Criterion 1 (completeness) r = 0.79, p = 0.000 r = 0.74, p = 0.000 r = -0.53, p = 0.001 Criterion 2 (interconnectedness) r = 0.63, p = 0.000 r = -0.26, p = 0.11
# Nodes (#N) # Edges (#E) Density
As for the structural measures, we found that the graph-theoretic density D was not a good quality predictor, but in contrast negatively correlated especially with the expert judgment on criterion 1. Very likely this is due to the nature of network/map evolution: The density of a map tends to decrease with growing number of nodes, whereas the RC coefficient tends to be stable. Thus a higher density indicates a smaller map and, accordingly, correlates negatively with crit1 (r= -.528, p=.001). This phenomenon corresponds to the evolution of so-called scale-free networks (Barabsi & Bonabeau, 2003) based on a mechanism called preferential attachment, expressing the tendency to attach new nodes to those already strongly connected. As an unexpected side-effect, yet retrospectively not surprising, our study corroborates that concept maps evolve in a similar way. It is indeed of high practical relevance that, as a consequence, graph-theoretic density of concept maps is inappropriate as a quality predictor. Instead, the ratio between number of links and number of concepts (#E/#N) can be used as an indicator of interconnectedness (r= .2, p=0.2 for crit1, r=.68, p=.000 for crit2). Due to unrecognised semantic equivalences between the terms used by students and ontology concepts, the values for #N and #E were not very high as compared to #N and #E. The main reason for that was the rare usage of the explicit synonym declaration (Concept already exists!). Therefore, the corresponding ontologybased variables also did not show a significant correlation to the expert judgments. However, we still decided to integrate them as a semantic correction into our formula for qm for two reasons: First, the structural values (#N, #E, etc.) lack a connection to the content so that a concept map with only dummy concepts would always score higher the bigger it gets, irrespective of node content. The second advantage is that the semantically weighted measure provides an estimation of expected values like size or relation-concept ratio. Without this a concept map for a long (concept-rich) text would not be handled differently to a concept map to a short text. To model the overall judgement, the different automatically assessed variables were composed additively after division by the corresponding value from the ontology projection (#N, #E, etc.). The resulting formula for the quality measurement qm is the following:
1 # 5 # 1 1 # + + + 3 # 6 # 2 3 # ,
ISLS
296
ICLS2012
The coefficients of the four quotients originate from the fact that #N, #E and #NW contribute to one third to the completeness and #E and RC to one half to the interconnectedness aspect. This composite quality measure strongly correlates to the overall expert assessment (crit1 + crit2) with r=.776, p=.000. Concerning research question 2, we wanted to investigate whether continuous help leads to a higher degree of interconnectedness. There is a positive indication for this from the RC value (1.17) as well as the crit2 assessment value (2.54), which is higher than the average (1.04, respectively 2.07). Probably due to the low sample size (N=37), this difference does not reach statistical significance. However, in informal discussions after the study the pupils reported a perceived benefit from the continuous help. Therefore, a continuation of the study in a larger scale is planned in order to confirm this.
Discussion
This study was originally conceived as a test of different versions of automated map assessment and different types of scaffolding support. Neither the benefit of enriching structural measures with ontology features nor the effects of automatic help and scaffolding could be clearly demonstrated. Yet, we have a found a clear indication of the scale-free nature of the student-created concepts maps. Scale-free networks as described by Barabsi & Bonabeau (2003) have certain characteristics that influence the adequacy of graph measures for quality assessment. A scale-free network has a small number of highly connected nodes (hubs) that, according to preferential attachment, are more likely to receive links from newly created nodes than others. Yet, the average degree in a growing scale-free network tends to be stable, which in turn implies that the density of the network will decrease as it grows (since density = avg.degree / (N-1) with N being the number of nodes). Figure 4 illustrates another typical feature of scale-free networks in our data: the inverse power law dependency of the number of nodes with a certain degree (degree distribution), for which a linear relation in the log-log graph can serve as a litmus test. 300 250 200 150 100 50 0 1 2 3 4 5 6 7 8 9 10 11 3 log N (no of nodes) 2,5 2 1,5 1 0,5 0 0 0,5 1 1,5 log d (degree values)
Figure 4. Aggregated degree distribution (from 37 maps) left: number of nodes per degree value, right: log-log graph In a panel contribution at ICLS 2010, Jacobson and Kapur (2010) have discussed the consequences of viewing ontologies as scale-free networks on modelling and understanding conceptual change. Their argument was a theoretical one, based on the assumption of ontologies (and also the personal conceptual models of learners) being of scale-free nature. Scale-free networks, although quantitatively dominated by nodes with lowest degrees, show a higher-than-random frequency of so-called hubs, i.e. very highly connected nodes. These hubs play a central role in the evolution of the model since they are the preferred points to which new nodes connect by preferential attachment. We can further hypothesize that preferential attachment will dominate the normal development of a conceptual model. Newly arising hubs will indicate hot spots of conceptual change. If ever we should see hubs collapsing (i.e. dramatically loosing connectivity) this would indicate an abnormal situation, i.e., a kind of revolutionary change or paradigm shift in conceptual understanding. Concept maps created by students can be seen as indicators for such processes. Ifenthaler, Masduki and Seel (2011) have used and compared a number of general graph theoretic measures to identify changes in cognitive structures, also relying on concept maps as external representations. Concept maps were collected at five different stages of a learning process. One of their findings was that the average degree of nodes tended to be constant over time (at a value close to 2). Yet, they did not trace this fact back to the scale-free nature of concept maps, but rather took it as an indicator of a de-centralised evolution of maps (p. 53). The model of hubs and preferential attachment, however, gives rise to more specific hypotheses about network growth.
ISLS
297
ICLS2012
General graph-theoretical measures provide a well-developed, yet still to be fully exploited inventory to analyse and model processes of conceptual development. Notwithstanding the lack of significance in the reported experiment, we still believe that a combination of such structural measures and semantic relations derived from an ontology is a key to a better understanding of learners conceptual models. As an example of a structural measure, we had used the centrality of a concept in the ontology as criterion to prefer this concept in scaffolding hints (in case this had not yet been used by the student). Also, high values of betweenness centrality could, e.g., be used as indicators for bridge concepts that inter-link different areas or sub-domains of a conceptual model. Missing bridge concepts could thus indicate a need for better knowledge integration. Overall, we are convinced that there is still much to gain from applying general methods of network analysis to conceptual models externalised by learners in the form of concept maps.
References
Barabsi, A.-L., & Bonabeau, E. (2003). Scale-Free Networks. Scientific American 288 (May 2003), 60-9. Blei, D.M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, vol. 3, 993-1022. Conlon, T. (2004). 'Please Argue, I Could Be Wrong': A reasonable fallible analyser for student concept maps. In: Proc. of Ed-Media 2004, Lugano, Switzerland. Erman, L.D., Hayes-Roth, F., Lesser, V.R., & Reddy, D.R. (1980). The Hearsay-II speech understanding system: Integrating knowledge to resolve uncertainty. ACM Comp. Surveys 12 (2), 213-253. Gelernter, D. (1985). Generative communication in Linda. ACM Transactions on Programming Languages and Systems, vol. 7 (1), 80-112. Gouli, E., Gogoulou, A., & Grigoriadou, M. (2003). A coherent and integrated framework using concept maps for various educational assessment functions. J. of Information Technology Education, 2, 215-240. Gouli, E., Gogoulou, A., Papanikolaou, K., & Griggoriadou, M. (2005). How to qualitatively + quantitatively assess concepts maps: the case of COMPASS. In Proc. of AIED 2005, Amsterdam, The Netherlands. Gupta, R., Wu, Y., & Biswas, G. (2005). Teaching about dynamic processes: a teachable agents approach. In Proc. of AIED 2005, Amsterdam, The Netherlands. Ifenthaler, D., Masduki, I., Seel, N.M. (2011). The mystery of cognitive structure and how we can detect it: tracking the development of cognitive structures over time. Instructional Science, vol. 39, 41-61. Jacobson, M. J., & Kapur, M. (2010). Ontologies as scale free networks: Implications for theories of conceptual change. In K. Gomez, L, Lyons, & J. Radinsky Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010), 193-194. Kilanioti, I., Gouli E., Gogoulou, A., & Grigoriadou, M. (2008). Representing knowledge of a concept map assessment and learning environment: an ontological approach. In Proc. of ICL 2008, Villach, Austria. Marra, R.M. (2002). The ideal online learning environment for supporting epistemic development: Putting the puzzle together. Quarterly Review of Distance Education, 3 (1), 15-31. Ruiz-Primo, M. (2004). Examining concept maps as an assessment tool. In A. J. Caas, J. D. Novak, & F. M. Gonzlez. (Eds), Concept Maps: Theory, Methodology, Technology. Proc. of CMC 2004. Pamplona, Spain, 555-562. Rye, J., & Rubba, P. (2002). Scoring concept maps: An expert map-based scheme weighted for relationships. School Science and Mathematics, vol. 102 (1), 33-44. Stoddart, T., Abrams, R., Gasper, E., & Canaday, D. (2000). Concept maps as assessment in science inquiry learning A report of methodology. The International Journal of Science Education, vol. 22 (12), 1221-1246. Weinbrenner, S., Engler, J., Wichmann, A., & Hoppe, H.U. (2010). Monitoring and analysing students systematic behaviour - the SCY pedagogical agent framework. In Proc. of ECTEL 2010, Barcelona, Spain. Weinbrenner, S., Giemza, A., & Hoppe, H.U. (2007). Engineering heterogeneous distributed learning environments using TupleSpaces as an architectural platform. In Proc. of ICALT 2007, Los Alamitos, USA (IEEE Press).
Acknowledgements
This study was partially supported by the SCY project funded by the EC under the ICT theme of FP7 (Grant agreement 212814). This text does not represent the opinion of the EC, and the EC is not responsible for any use that might be made of its content. We would like to thank the St. Hildegardis grammar school in Duisburg, Germany for cooperation and especially Maria Frank for her help in conducting the study as a teacher. We would like to acknowledge the contributions of the large list of people working on SCY to the work presented here.
ISLS
298
ICLS2012
Toward a Cognitive Framework of Interdisciplinary Understanding

Shannon Sung, Ji Shen, and Dongmei Zhang, University of Georgia, Math and Science Education, 212 Aderhold Email: shasung@uga.edu, jishen@uga.edu, dongmei@uga.edu Abstract: Students need to think and work across disciplinary boundaries in the 21st century. However, it is unclear what interdisciplinary thinking means and how to assess students interdisciplinary understanding. In this paper, drawing from multiple perspectives in the learning sciences, we aim to apply and refine a theoretical framework that helps define interdisciplinary learning in science. Specifically, we examine four cognitive dimensions of interdisciplinary understanding: integration, translation, transfer, and transformation (IT3 framework). We apply the framework in analyzing the conversations among university faculty members from different science disciplines who strive to improve college level science education. We report our results and further discuss the implications of using our framework in conceptualizing interdisciplinary learning.
Introduction
Collaborative research involving multiple disciplines is pervasive in many fields including Science, Technology, Mathematics, and Engineering (STEM) (Rhoten & Parker, 2004). This trend mandates college education to prepare students to think across disciplines in the 21st century (Engle, 2006; National Research Council, 2000). However, it is unclear what interdisciplinary understanding means, and little research has been conducted on establishing a cognitive model of interdisciplinary understanding (Boix Mansilla & Duraising, 2007). Furthermore, there are many barriers preventing students from becoming successful interdisciplinary thinkers and doers. For instance, typical assessment practices in a college science course focus primarily on specific disciplinary topics. As a result, college students not only develop fragmented understanding in science, but they are also reluctant to think beyond disciplinary constraints (Linn, 2006; diSessa, 1993). This paper aims to elaborate on a theoretical framework that can potentially answer the question What comprises interdisciplinary understanding? We developed the framework (Shen, Sung & Rogers, 2012) in the context of working on a larger project that aims to improve college students interdisciplinary science understanding. This framework can be used to guide the development of curricular materials, instructional approaches, and assessment items that target students interdisciplinary understanding. In the following, we first review relevant literature that informed our perspective. We then present our theoretical framework and elaborate on each key component of the framework. In the empirical section, we explain how we apply the framework in analyzing the discourse of a dynamic group whose goal is to develop interdisciplinary science assessment items for college students. We report the results of our analysis and discuss the educational implications.
Relevant Literature and The IT3 Framework

The learning sciences literature has provided many useful perspectives for examining the issue of interdisciplinary understanding. Here, we highlight the ones that influenced our framework. Boix Mansilla and Duraising (2007) defined interdisciplinary understanding as the capacity to integrate knowledge and modes of thinking in two or more disciplines or established areas of expertise to produce a cognitive advancementin ways that would have been impossible or unlikely through single disciplinary means (p.219). Recognizing that students develop fragmented understanding in science topics, Linn and colleagues developed the framework of knowledge integration (KI) that emphasizes students abilities in establishing connections among scientific ideas (Linn & Eylon, 2011; Linn, 2006). The framework promotes coherent understanding by encouraging students to add new ideas, distinguish new and existing ideas, develop scientific criteria to reconcile ideas, and build coherent connections between a science phenomenon with their prior knowledge or experiences across different dimensions of knowledge (Liu, Lee, & Linn, 2011). In this paper, we further explore the role of disciplinarity in scientific knowledge integration. Students constantly face the challenge of transferring scientific knowledge learned in one context to another, which includes transferring knowledge from one discipline to another (Bransford & Schwartz, 1999; Chin & Brown, 2000; Haskell, 2001). Research has shown that many factors contribute to or hamper student knowledge transfer, including prior knowledge and experience, opportunities to develop deep understanding, language, and context of learning (Klahr & Carver, 1988; Lave, 1987). It is critical to communicate the knowledge and outcome of interdisciplinary work with audiences from different disciplines. In an interview from her study, Boix Mansilla and Duraising (2007) highlighted the importance for learners to communicate their disciplinary knowledge to people who do not speak the same language (p.224). Learners who are able to acquire sufficient language to converse on a similar topic or
ISLS
299
ICLS2012
distinguish terminologies from another discipline are considered more competent in thinking across disciplines than those who require interpretation. Therefore, acquiring different terminologies used across disciplines is highly desired for students to achieve interdisciplinary understanding.
The IT3 Framework

Building upon related literature, we argue that deep interdisciplinary understanding has four interconnected dimensions: integration, translation, transfer, and transformation (henceforth, the IT 3 framework) (Shen, Sung & Rogers, 2012). We elaborate on each dimension in the following using examples from osmosis, an interdisciplinary science topic.
Integration
Considering the importance of linking distinctive ideas from different resources, the first dimension of our framework emphasizes knowledge integration across disciplinary boundaries. For instance, we may ask students to explain why eating a large amount of hyperosmotic food such as cake or chocolate without drinking water would cause an accumulation of water in the lumen of the digestive tract. To fully explain this phenomenon, students need to integrate knowledge in chemistry (e.g., solvation), biology (e.g., selectively permeable membrane of a cell), and physiology (e.g., structure and function of organs). Students with a higher level of interdisciplinary integration should identify the connections between disciplinary ideas in this context.
Translation
The comprehension of different terminologies used across disciplines is highly desirable for developing interdisciplinary understanding, and the translation component constitutes the second dimension of the IT3 framework. Since many disciplines develop their own terminologies to explain similar phenomena, students who develop interdisciplinary understanding need to be able to translate scientific terms in order to communicate effectively with people from different disciplinary backgrounds. For example, in plant biology, turgor pressure is the pressure of the cell contents enclosing the membrane (protoplast) against the cell wall due to osmosis, whereas in animal or medical physiology, intracranium pressure is the pressure exerted on the skull due to high fluid retention. These two discipline-bounded terms are similar as they present two concrete examples of osmotic pressure. A student who has developed interdisciplinary understanding of osmosis should be able to translate between these terms. Translation between terms does not have to occur en masse: in theory, one can just translate one term at a time.
Transfer
Interdisciplinary transfer occurs when students apply explanatory models and concepts learned from one discipline to another disciplinary context. One criterion is the ability to recognize the core structure of the system under studymatching the parallel elements or parts and their connections within the two systems. This falls into the category of deep transfer (e.g., Chin & Brown, 2000). Learners who use rote memorization will be less likely to succeed in an interdisciplinary task without applying knowledge they acquired in one field to a new context. In other words, a competent learner with cross-disciplinary understanding can relate what he or she has learned in one discipline to another discipline in order to recognize and identify the common model or shared ideas. Consider the following example: A student has learned the knowledge needed to explain the typical Utube scenario demonstrating osmosis in a chemistry class. Two solutions with different solute concentrations in two sides of a U-shaped tube separated by a selectively permeable membrane at the bottom which only allows certain ions or molecules, usually water, to pass through. The system reaches equilibrium when a certain amount of water from the side with lower solute concentration moves to the side with higher solute concentration. When the student is asked to explain the function and process of osmosis in a plant cell, he or she may be able to transfer his/her knowledge learned from the U-tube situation. For instance, to recognize the similar system component, i.e., the two solutions with different solute concentrations, the two solutions are separated by a selectively permeable membrane, and the movement of water reaches equilibrium when osmotic pressure is balanced by another external pressure. Transfer is different from translation in that it focuses on the understanding of the basic system or explanatory model that is being transferred as opposed to linking the conventional terminologies used in different disciplines.
Transformation
The fourth dimension of interdisciplinary understanding is transformation. Students need to be able to apply explanatory models and concepts learned from one discipline to physically or conceptually transform a system typically considered in a different discipline into another novel system. An example in this category is reverse osmosis, a process that is frequently used in food engineering. Reverse osmosis is achieved by applying
ISLS 300
ICLS2012
additional pressure to the higher solute-concentrated side. Because of the selectively permeable membrane, this process results in retaining large molecules and ions on the pressurized side of the membrane, forcing smaller molecules or ions to pass to the other side. Reverse osmosis is typically used to purify water. Since students have already acquired knowledge of regular osmosis process (existing system), they have to identify an additional system (pressure) in order to understand reverse osmosis (new system). The production of a new physical or conceptual system is a key feature of transformation, which is different from transfer. It is important to note that these four dimensions, as characterized above, are intertwined and nonexclusive to each other. For instance, as the translation process establishes links between parallel terms from two different disciplines, it also integrates these terms in a sense. When the models and concepts of the first discipline are transferred to a new discipline, they may also need to be translated. An interdisciplinary transformation process typically requires both a transfer and a translation process, as the learner has to acknowledge the target system and compare it to a referent system in order to change it.
Applying the Framework

Background and Data
In this study, we applied the IT3 framework to analyze the discourse of an interdisciplinary faculty group meeting. The group of faculty members came from different science disciplines and worked together to create interdisciplinary assessment items to be used in introductory science courses in physics, plant physiology, animal physiology, and chemistry. The faculty met roughly once every two weeks. The meetings were audiotaped and transcribed. We chose one particular meeting (38 minutes) out of 13 meetings to demonstrate our analysis. We chose this segment to start with because in this meeting, the faculty members encountered different and sometimes conflicting disciplinary perspectives while they were discussing a concept map on osmosis that they co-constructed. The conflicting views made these content experts eager to learn another disciplinary perspective and develop their interdisciplinary understanding.
Coding
The unit of our analysis is a coherent statement, defined as one or more sentences that deliver a stand-alone meaning. The segment of the meeting we analyzed consists of 298 individual statements in total. There are two layers of codes we apply to each statement (see Table 1 for sample statements). The first layer of codes emerged from the coding processit concerns the topic of the statement. That is, we first decided if a statement falls into one of the following three categories: A concept-specific statement involves specific scientific concepts and terminologies such as water movement or osmotic pressure. A metacognitive statement talks about scientific understanding at a general, abstract, or representational level without involving specific scientific concepts. An instructional statement touches upon issues related to teaching and learning. If a statement does not belong to any of the three categories, we categorized it as non-interdisciplinary other. If a statement falls into one of the three categories above, we then apply the second layer of codes, i.e., the four dimensions of interdisciplinary understanding. If a statement falls into one of the three categories but cannot be coded as one of the four dimensions in the IT3 framework, it is coded as interdisciplinary other. This process results in 16 different codes that may be applied to a statement. If a statement involves multiple codes, we assign equal weights to each code involved. For instance, if a statement involves both translation and transfer, we then assigned for each. We coded each statement in the context of the utterance. On many occasions we needed to infer the references of a pronoun as well as components omitted by the speakers. For example, in the stand-alone statement I think they are the same, the pronoun they referred to osmotic potential and water potential in a prior statement of another speaker. We employed an iterative coding process that took several cycles. The second author initiated the coding framework and trained two coders, one from a biology background (the first author) and the other from a physics background (the third author). The two coders independently coded all the statements. In each cycle, the coders coded a number of statements (30-50) and then compared their codes. The whole team then examined the inconsistent codes and discussed questions that arose in the coding until we reached agreement as a group. This process repeated for several cycles during which the interrater reliability (joint-probability of agreement) increased from 0.69 to 0.87. All inconsistent codes were resolved through discussion. Table 1: Examples of the two layers of coding.* Content-Specific: Metacognive: Instructional:
ISLS
301
ICLS2012
Integration:
So that the water potential is being negated by the increase of osmotic pressure? (S157)
So now we have two sorts of approaches, one is the core and connect to different context, and somehow we can put some that are related to biology and physiology or related to physics. (S51)
Chemical potential? The and maybe thats not what a change in internal energy of a biologist would say. In system when you add another which case, theres a, are one of those particle into the difference of language. system. Ok, thats the most (S182) general definition. (S97) Transfer: I think its actually maybe If you understand the the problem I have, with the central core of osmosis, and solute and pressure potential, you apply it in physics because when people come to situation or biology me and tell me well, that cell situation, isnt there a on its own in isolation has central core that enables you that pressure potential of that to see the links between number and that solute them? (S2) potential. (S117) Transformation: N/A N/A N/A *We did not include the other category in the table because statements falling into this category are usually situated in between the dialogue. Translation:
Well, yeah, part of what Kathrin might be saying is that a biology student or biology instructor you know is also a physicist or whatever approaching this from a different perspective. (S3) So I think at least for that part we could actually translate that into biological term so its more intuitive for people who are teaching that. (S184) And I found good students, that they learned things with a core, integrated knowledge, and then they extrapolate to physics, and biology. (S43)
Results
Table 2 lists the frequencies for each code for this data set. In this conversation, in terms of topics, the faculty members were more engaged in concept-specific discussion (overall, 63% in this category, compared to 12% metacognitive and 3% instructional). In terms of the interdisciplinary dimensions, they were mainly engaged in translation for each other (overall, 60% translation, 11% integration, and 8% transfer). Table 2: Frequencies of coding results. Integration 4 23 5.5 26 Translation 165.5 10.5 1 Transfer 19.5 2.5 1.5 Transformation 0 0 0 ID Other 11 24 4
Concept Metacognition Instructional Non-ID Other
It is noted that when the faculty members talked about interdisciplinary understanding at a metacognitive level, they emphasized integration over the other dimensions (within the metacognitive category: integration, 38%; translation 18%; transfer, 4%). In their concept-specific discussion, however, translation is the most common theme (within the concept-specific category: translation, 83%; transfer, 10%, integration, 2%). The first part of the discussion (~20% of the total time; not shown in the table) focused on the metacognitive aspect, while the rest of the discussion was much more concept-specific. The metacognitive comments mainly involved the structure of the concept maps that the faculty members created. For instance, the following conversations focused on how people from different disciplines would construct or perceive a concept map on osmosis (pseudonyms are used). Clark: .. Its like a physicist is focusing on one area of the concept map, almost to the exclusion of the other which is from one region and works his or her way out from there. Biologist might be starting from a different region, and eventually mixing those connections. But whats important in that concept map for the biologist is a lot of other stuff. Sam: So does it make sense what I heard from you is we could somehow incorporate this region (on the concept map) is more physics related and this region is sort of biological context or physiological context.
ISLS
302
ICLS2012
Kate: I think one of a big benefit of that project is that you give people the translation, because ultimately youre talking about the same thing. But in physics (people are) looking from angles as biologists do, but not because we impose physicists to teach biological way but you need to give something that you can approach this from a background of physics. Tim: sounds like a Venn diagram to me. you know biology and physics, and you got overlap in the middle, and you got that core, and the perspectives that give you insight to that, am I right? Kate: I personally would never draw a diagram like that, but thats again, thats personal. Im a more hierarchical person, and I would start with what Im really interested from the top, and then I would be more detailed of the big picture when I go down to the bottom. Later in the meeting, when the faculty brought up different terminologies on the concept map, they started a heated discussion on the specific concepts. That is why concept-specific discussion dominated the rest of the meeting. Tim: Solute potential, Ive never heard (of it) Sam: ok, so I just cross that out? Larry: What does that mean? Sam: I dont know. Its on the map. Kate: Oh, well, wait wait wait, that should be over here for plant cell. Sam: Plant cell? Kate: Yeah, in plant cells when you talk about water potential, you have two components, one is solute potential, Larry: Sorry, you said in plant cell what? Kate: .In plants the water potential is made up of two parts, what triggers osmosis has two parts, its the solute potential which is usually equivalent with the solute concentration except its backwards, if you look at the numbers. And the other part its the pressure potential, which in essence represents by the cell wall where the pressure starts to building up, results in turgor pressure . Similar discourse exchanges recurred in the rest of the discussion when animal physiologists used the term osmotic pressure to refer to the external pressure needed to stop water movement due to solute concentration gradient across a selectively-permeable membrane, while plant physiologists use the term solute potential to refer to the factor of adding solute in a solution that drives water movement in osmosis. Plant biologists see no osmotic pressure in a typical U-tube scenario (or an animal cell) because it is an open system, whereas plant cells have restrictions due to the rigid cell wall, related to turgor pressure. The group did not reach agreement on how to define osmotic pressure and reconcile the different terms at the end of this meeting.
Discussion
There were several interesting themes that emerged from examining the data. In this section, we discuss what we learned from the analysis in light of the IT3 framework.
Integration
Numerous studies had introduced much about knowledge integration in general (e.g., Linn, 2006). Here, we focus on interdisciplinary knowledge integration. From the meeting discussion, we see that two kinds of interdisciplinary knowledge integration were brought up. The first type, differential integration, is organizing concepts from different disciplines into a connected whole. When the faculty members discussed the concept map, they noticed that there were concepts bounded by different disciplines on the map (see the first segment of quotes on the previous page). Being aware that certain concepts are rooted in specific disciplines is a strong indicator of deep disciplinary knowledge, which we argue, along with other researchers (Boix Mansilla & Duraising, 2007), to be a prerequisite to true interdisciplinary integration. The second type of interdisciplinary knowledge integration, commonality integration, emphasizes the shared common set of knowledge. On several occasions, people in the meeting talked about a shared core when thinking of an interdisciplinary topic such as osmosis. For instance, at the very beginning, Tim pointed out that If we got a central connection about osmosis and then we relate those to biology and physics is there a central core how we relate it to? In these references, the central core is an integrated core set of concepts or big ideas that have been fused together from different disciplinary descriptions. This common core set of concepts can be used to describe the underlying processes applicable to different disciplines. The common core that emerged from the discussion at the meeting may be represented as the shared region in a Venn diagram or the top-level concepts in a hierarchical map. The second sense of interdisciplinary knowledge integration leads to transfer (or vice versa): as long as one develops the integrated common core, one can transfer it to different disciplinary contexts.
ISLS
303
ICLS2012
Translation
We noticed the beginning one-fifth of conversation was spent on the metacognitive aspect and the rest mostly touched upon concept-specific issues, which was cued by the terms used in different disciplines to describe osmosis. This indicates not only a shift of topics but also the groups engagement in reaching consensus regarding concepts and terminologies used to describe the same phenomenon. This process highlights the importance of the translation dimension of interdisciplinary understanding. The most common translation strategy a person used in this meeting was to elaborate on a term from his or her own disciplinary perspective to make it intelligible and plausible. This was an indication of a disciplinary-oriented system of thinking that may prevent successful interdisciplinary communication. For instance, when Kate was explaining the term water potential, she drew on her disciplinary knowledge and elaborated on the two typical components of water potential (see the second segment of quotes on the previous page). Another translation strategy witnessed in this data set is the common reference to a common scenario, in this case, referring to the U-tube case. This makes sense because it is typically introduced in all the disciplines when teaching / learning osmosis. In these translation processes, one would also expect transfer of knowledge to internalize the newly translated terms.
Differentiating Translation and Transfer

Although our coders interrater reliability improved significantly and reached a satisfactory level, we still encountered difficulties in determining whether a statement belonged to transfer or translation. Most of our disagreement resulted from the confusion between translation and transfer. This indicates that the two processes are probably more intertwined than the other processes and that there needs to be more clarification of the transfer and translation constructs. The following are some insights we gained through the data analysis process. First, a transfer process may include intradisciplinary and interdisciplinary transfer. The IT3 framework aims to address interdisciplinary understanding. Therefore, transfer in our framework only refers to interdisciplinary transfer. Second, at the surface level, a translation process may only involve the terminology level. But a deep translation may also occur. For instance, consider the scenario of the interdisciplinary meeting that we analyzed. When a person is translating some disciplinary-specific term for an audience from a different discipline, he or she is basically explaining the concept to make it meaningful to those who are not familiar with its disciplinary connections. This may involve several levels. She may simply introduce the terms (e.g., Kate introduced the terms such as water potential, solute potential, and pressure potential). We called this terminological translation. Furthermore, she may extend the translation by adding relations of the terms (e.g., Kate explained that water potential is the sum of solute potential and pressure potential). We called this relational translation. Finally, she may provide concrete examples to which one can apply the terms. These examples are typically within the disciplinary boundary and/or drawn from common experience. We called this concrete translation. The latter two levels are considered deep translation, compared to the first level. Confusion may arise when concrete translation overlaps with intradisciplinary transfer, the process of applying concepts to concrete examples within a discipline in order to translate for others. Specifically, in the context of interdisciplinary conversation, if one elaborates on any term or principle by explaining how it is applied within ones own discipline in order to solve intradisciplinary problems by providing specific examples from ones disciplinary understanding, this statement was coded as translationdeep translation, instead of interdisciplinary transfer. For example, when a plant physiologist attempted to clarify pressure flow to faculty from other disciplines by saying, That added sugar attracts water, and it comes out of the xylem, thats right next to the phloem and so you get all these water rushing in which builds up pressure, one might be tempted to code this statement as transfer; however, the speaker only provided this phenomenon from her discipline, so this statement was categorized as deep translation. Figure 1 represents the refined IT3 framework based on the discussion above.
ISLS
304
ICLS2012
Figure 1. The refined ITTT framework.
Limitations
In this study, we only analyzed one meeting. This limits the insights we can gather from this empirical data set. For instance, there was no statement that qualified as transformation. This may be because within the time limit, the participants had yet to absorb the inconsistent views and develop a better understanding to reach the transformation stage. We will analyze more faculty meetings as the next step. When we speak of interdisciplinarity, our views and data analysis are drawn from the perspectives within sciences. Applying this framework to include non-science domains is conceivable.
Conclusions and Implications

In this paper, we elaborated a framework on interdisciplinary understanding that has four interrelated dimensions: integration, translation, transfer, and transformation. Our framework provides a theoretical foundation to understand the construct of interdisciplinary understanding. We then developed corresponding codes using this framework to analyze a segment of an interdisciplinary faculty meeting, focusing on improving college-level integrated science education. We found that in this meeting, the faculty members spent most of the time discussing concept-specific issues and engaged in the process of interdisciplinary translation. The faculty members who sought to resolve the inconsistent usages of terminologies across disciplines demonstrated and confirmed the importance of nurturing the ability to translate in order to communicate with people who speak another language (Boix Mansilla & Duraising, 2007). The most difficult terms causing extensive clarification, objection, and discussion among the faculty members in this study were potential, osmotic potential, and osmotic pressure; this has significant educational implications when re-thinking teaching the topic of osmosis. We currently are carrying out a content analysis study to investigate how these terms are interpreted from different textbooks in different disciplines. The current validation for the IT3 framework seems a tedious and labor-intensive process; however, with the increasing demand to provide interdisciplinary education to the students with the expectation of fostering their higher order thinking, efforts to define and evaluate interdisciplinary understanding are necessary and valuable. Promoting interdisciplinary education does not mean discarding disciplinary courses. In fact, our framework involves advancement of disciplinary knowledge in order to communicate across disciplines. Students should be able to root their science knowledge in rigorous disciplinary training, and consciously elicit and apply their disciplinary perspectives to bring isolated concepts or inconsistent terminologies together in order to effectively communicate with others.
References:
Boix Mansilla, V., & Duraising, E.D. (2007). Targeted assessment of students interdisciplinary work: An empirically grounded framework proposed. The Journal of Higher Education, 78(2), 215-237.
ISLS
305
ICLS2012
National Research Council, (2000). How people learn: Brain, mind, experience, and school (Expanded Edition). Washington, DC: National Academy Press. Bransford, J. D. & Schwartz, D. L. (1999). Rethinking transfer: A simple proposal with multiple implications. In A. Iran-Nejad and P. D. Pearson (Eds.), Review of Research in Education, (pp. 61-100). Washington, D.C.: American Educational Research Association. Chin, C., & Brown, D.E. (2000). Learning in science: A comparison of deep and surface approaches. Journal of Research in Science Teaching, 37(2), 109-138. diSessa, A.A. (1993). Toward an epistemology of physics. Cognition and Instruction, 10(2&3), 105-225. Engle, R. A. (2006). Framing interactions to foster generative learning: A situative explanation of transfer in a community of learners classroom. Journal of the Learning Sciences, 15(4), 451-498. Haskell, J. P. (2001). The latitudinal gradient of diversity through the Holocene as recorded by fossil pollen in Europe. Evolutionary Ecology Research, 3, 345360. Hunt, E., & Minstrell, J. (1994). A cognitive approach to the teaching of physics. In K. McGilly (Ed.), Classroom lessons, 51-74. Cambridge: MIT Press. Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362404. Lave, J. (1987). Cognition in practice. New York: Cambridge University Press. Liu, O.L., Lee, H., & Linn, M.C. (2011) Measuring knowledge integration: Validation of four-year assessments. Journal of Research in Science Teaching, 48(9), 1079-1107. Linn, M. C. (2006). The knowledge integration perspective on learning and instruction. In R. K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (pp. 243264). New York: Cambridge University Press. Linn, M. C., & Eylon, B.-S. (2011). Science learning and instruction: Taking advantage of technology to promote knowledge integration. New York: Routledge. Mezirow, J. (2000). Learning to think like an adultCore concepts of transformation theory. In J. Mezirow (Ed.), Learning as transformation: critical perspectives on a theory in progress (3-33). San Francisco: Jossey-Bass Inc. Rhoten, D., & Parker, A. (2004). Risks and rewards of an interdisciplinary research path. Science, 306(5704), 2046-2046. Shen, J., Sung, S., & Rogers, W. (2012). Assessing interdisciplinary understanding in scienceThe IT3 framework. Poster presented at the annual conference of the National Association for Research in Science Teaching (NARST). Indianapolis, IN.
Acknowledgments
This material is based upon work supported by the National Science Foundation under award number DRL1043040. Any opinions, findings, and conclusions expressed in this work are those of the authors and do not necessarily reflect the views of the National Science Foundation.
ISLS
306
ICLS2012
Using Innovation with Contrasting Cases to Scaffold Collaborative Learning and Transfer
Seungyon Ha, David A Sears, College of Education, Purdue University, West Lafayette, IN, USA 47907 sha@purdue.edu, dsears@purdue.edu Abstract: The current study examined dyadic interaction and transfer resulting from a math lesson presented in traditional lecture-then-practice fashion versus in innovation then efficiency fashion (Schwartz, Bransford, & Sears, 2005). While dyads in the Traditional condition were able to solve the practice problems, only one out of 18 individuals successfully solved a difficult transfer problem on the posttest. By contrast, a clear pattern of steps in the problem solving process of the Innovation dyads showed that most students could perceive and verbalize the correct solution to the practice problems (7 of 9 dyads) but had trouble translating their insights into a mathematical solution. Only three Innovation dyads successfully translated their perceptions into correct mathematical formulationssuggesting an area for further scaffolding in future work. However, five of these six individuals correctly solved the difficult posttest transfer problem. The use of contrasting cases during innovation appeared instrumental to participants perception and formulation of correct solutions. There is a general consensus that collaborative learning leads to better academic outcomes than individual learning, at least under certain conditions (e.g., Cohen, 1994; Johnson, Johnson, & Stanne, 2000; ODonnell, 2006; Slavin, 1996). A primary benefit of learning in groups is that students can learn from other students by exchanging ideas, recognizing and reconciling different perspectives, and by co-constructing and practicing problem-solutions (Barron, 2003; Stacey, 1999; Webb, Troper, & Fall, 1995). However, merely putting students in groups does not guarantee effective collaborative learning. For example, Barron (2003), in a conversation analysis of high achieving 6th grader triads problem solving, found that some groups showed such disjointed interactions that the individuals in those groups would have likely performed better working alone. The less successful groups tended to ignore or reject correct proposals and failed to connect correct proposals to preceding conversational topics. Many studies of learning in small groups suggest that advantages can be obtained when certain scaffolds are used. Slavin (1983, 1996) has argued for group goals (or group rewards) and individual accountability as a critical motivating factor. Imposing scripts or specific roles is also a way to increase successful group learning outcomes (ODonnell & Dansereau, 1992). Training students for cooperation is also beneficial because cooperative learning requires different forms of behavior that students might not experience in typical classrooms (Cohen, 1994). From a nine-month study and a two-year follow-up study of junior high school students, Gillies (2002, 2003) concluded that students who worked in small groups in a structured way consistently benefitted more in terms of the quality of group work, enjoyableness, and social cohesion than students who were in unstructured groups that were not trained to collaborate. Generally, scaffolds such as reward-systems, scripts, and group training are intended to promote behaviors that support learning, such as generating hypotheses and explanations (Chi, Bassok, Lewis, Reimann, & Glaser, 1989; Koenig & Griggs, 2004; Webb et al., 1995). Interaction factors within a group are important for successful group outcomes (Dillenbourg, 1999; Dillenbourg, J rvel & Fischer, 2009). Barron (2000) , identified three interaction features for successful collaborative learning sharing task alignment, joint attention, and mutuality. Others have noted the importance of frequent task-related talk, wrestling with a task together, and explicit correspondence to a partners work (Cohen, 1994; Vedder, 1985). In the two-year qualitative study of Asian college students experiences in collaborative problem-based-learning classrooms, Remedios, Clarke and Hawthorne (2008) reported that although the Asian students tended not to speak frequently, probably due to language barriers and cultural differences, they tended to listen more, and when prompted by tutors or members, they were successful in task completion and sharing learning with peers. In other words, the quality of the interaction appears to be at least as important as the quantity for productive learning outcomes. Although the literature has constantly proposed scaffolding for better outcomes of collaborative learning, it is uncertain whether research-tested scaffolds are implemented effectively, if at all, in many classrooms. Antil, Jenkins, Wayne, and Vadasys (1998) survey of elementary school teachers revealed that although many teachers used collaborative learning in their classrooms, most of them used it without specific forms or were missing certain scaffolds. This suggests a need for research that identifies what types of tasks may be productive for collaboration without requiring extensive scaffolds. This approach could be an efficient way to foster the benefits of collaborative learning because task features may be relatively easy for teachers or curriculum designers to prepare and manipulate. Students interaction and willingness to collaborate are influenced by the nature of the task (Bennet & Dune, 1991; Cohen, 1994; ODonnell & Dansereau, 1992).
ISLS
307
ICLS2012
Simple tasks do not tend to benefit from unstructured collaborative learning (Cohen, 1994; Phelps & Damon, 1989). Cohen (1994) argued that collaborative tasks needed to be complex enough to prevent a single member from being able to solve the task without contributions from teammates. Recent evidence points to the potential of complex and innovation-oriented tasks to be productive for collaborative groups (Kapur, 2010; Kapur & Kinzer, 2009; Laughlin, Hatch, Silver, & Boh, 2006). Presumably, given complex problems, students find that they are not able to solve them alone, so they are naturally motivated to work together. Kapur and Kinzer (2009) and Kapur (2010) found benefits of complex and illstructured tasks for collaborative groups learning mathematical and science concepts. Although groups in the complex and ill-structured conditions struggled and even failed to solve the problems, they significantly outperformed the lecture-and-practice groups on a posttest that included near and far transfer measures. Kapur and Kinzer (2009) argued that there might be a hidden efficacy about learner-generated processes (p. 38) during the collaborative learning context. Kapur (2010) suggested that failure experiences during learnergenerated processes can be productive for deeper understanding (i.e., productive failure). The Innovation and Efficiency framework of Schwartz, Bransford, and Sears (2005) describes two complementary approaches to instruction that highlight potential benefits of productive failure for preparing students to learn from a subsequent lesson. Innovation emphasizes generation of solutions by students. Efficiency emphasizes traditional lesson-then-practice processes. Innovation has similarities with the concept of productive failure because when students engage in Innovation activities they are not expected to invent an expert formulation; instead, the experience is meant to help them notice and differentiate key features that a subsequent Efficiency activity can further organize, thereby better preparing students for future transfer. One possible mechanism by which Innovation tasks may help students prepare for future learning is by presenting them with opportunities to wrestle with contrasting cases that emphasize the central concepts of interest (Marton, 2008; Schwartz & Bransford, 1998; Schwartz, Sears, & Chang, 2007). While previous work has suggested ways in which innovation-oriented productive failure experiences may benefit students learning collaboratively, further study is needed to determine whether these instructional practices function as hypothesizedhelping students notice key factors and thereby preparing them to understand expert solutions more deeply. The current study builds upon a previous quantitative investigation of the Innovation and Efficiency framework for its ability to characterize naturally productive collaborative tasks (Sears, 2006). It examines dyadic interactions in an Innovation condition versus a Traditional lesson-thenpractice condition in an effort to explain how Innovation activity better prepared participants for subsequent individual transfer.
Methods
Participants
University students (N = 76) with little or no background in statistics those who took no more than A/P or introductory courses - were recruited through email lists and campus flyers. Forty participants worked individually and 36 worked collaboratively in same-sex dyads. They were randomly assigned to the Innovation and Efficiency condition or the Traditional lesson-then-practice condition. This study focuses on the dyads because they permit an analysis of group interactions. Nine dyads (6 female, 3 male) participated in each condition.
Materials
A learning packet, posttest, and transcription of videotaped interactions comprised the materials for the study (for more details on the materials see Sears, 2006). The learning packet consisted of nine pages of lessons and practice problems on the chi-square formula. The only difference in the materials used in each condition was the order in which they were completed. The Traditional condition read a lesson, then solved practice problems, then worked through a final example (with solution provided). The Innovation condition attempted to solve the practice problems first, then read the lesson, and received the final example. The practice problems used contrasting casesa feature that was expected to benefit participants in the Innovation condition by helping them notice key variables and scaffold their mathematical thinking. For example, the first set of practice problems required participants to differentiate three dice according to fairness. The results of a number of rolls for each of the three dice were provided and students had to construct a formula that would differentiate each die from least fair to most fair. The expert solution to this problem involves calculating the chi-square statistic: summing the squared differences between observed and expected values over the expected value. Because Traditional participants would have already seen this formula, they were expected to attend to the computational accuracy rather than comparing across the dice for solution insights. The Innovation participants were not expected to invent the chi-square formula, but they were expected to notice that one case was obviously least fair (because one side of the die appeared on 40% of the rolls). The other cases were harder to differentiate because each side of the dice appeared nearly equally often
ISLS
308
ICLS2012
(never off by more than one); however, one of these dice was rolled twice as much as the other (indicating it was more fair per roll). The posttest consisted of calculation and comprehension problems and one target transfer problem designed in preparation for future learning (PFL) fashion (Bransford & Schwartz, 1999). The target transfer problem required participants to adapt their understanding of the chi-square formula to invent a measure of inter-rater reliability similar to Cohens kappa. To help participants do this, a resource problem was presented earlier in the posttest that showed an inter-rater reliability matrix and gave suggestions for what to calculate. To succeed on the target transfer question, participants needed to complete a double-transfer: adapting the chisquare formula they learned during the lessons to work with the inter-rater reliability problem learned in the resource problem. The correct solution involved performing a chi-square calculation on just the agreement cells of the reliability matrix (rather than all of the cells, which would yield a nonsensical answer). This doubletransfer was expected to be very difficult for participants, but it provided an indicator of which condition better prepared students to learn from further instruction (i.e., the resource problem) and transfer their knowledge to new and challenging problems.
Procedures
Participants spent 35 to 65 minutes going through the learning packet. Those in the Innovation condition attempted to invent solutions to the practice problems before receiving the lesson and final example while those in the Traditional condition received the lesson, then the practice problems, and then the final example. Following a 5-minute break, participants completed the posttest (approx. 25 minutes).
Results
Regarding quantitative results previously reported by Sears (2006), the Innovation condition did not take longer than the Traditional condition during the learning or test phases, and they performed similarly on the calculation and comprehension problems. The key outcome of interest here is the analysis of dyadic interaction that may explain the advantage shown by the Innovation dyads on the difficult target transfer problem. Only 1 of 18 participants in the Traditional dyad condition successfully completed the double-transfer problem; 5 of 18 participants in the Innovation dyad condition did so (more than all other conditions combined). Sample interactions from dyads in each condition are presented followed by a description of the coding scheme and results. As shown in the following interactions, dyads in the Innovation condition tended to compare across the contrasting cases to estimate which of the three dice should be ranked least, most, or somewhat fair. Next they attempted to formalize their estimated rankings by developing a mathematical formula (as required by the problem). Most dyads showed accurate perceptions of the problem and many made progress on the mathematical formulation. However, only a third of the dyads invented a near-expert formula that successfully differentiated the three cases. Dyad 1(Innovation Condition): Noticing the conspicuous (least-fair) case and conjecturing the expected value as a way to compare the remaining two cases. They eventually succeeded in differentiating the cases verbally and mathematically. A: Yeah this (Case 2) one seems like the least fair. B: Least fair, exactly. A: And these two (Case 1 and Case 3) seem harder to tell. B: They seem the same to me. Are we supposed to circle these? So, if theyre equal, do we just rank them most fair together? A: Alright, lets just start with this one (Case 2) being the least fair. B: You seem to know whats going on. Let me see this. (laughing) A: So, for each number we should get 5 for this (Case 1 and Case 2) one and ten for this (Case 3) one. And each of them (each side of the die in Case 1) for this one should get five. B: Each of them, right? Dyad 1 noticed that case 2 looked the least fair (one of the numbers appeared on 40% of the rolls). Then, they tried to differentiate the similar two cases mathematically. First, they found the number of rolls that should appear for each side if the dice were fair (i.e., the expected value). This is a notable point because the expected value was not provided in the problem. We suspect that by inventing that variable, these students were more prepared to understand how it worked in the chi-square formula, the canonical solution that was taught in the lesson. Both participants in this dyad successfully solved the target transfer problem on the posttest (taken individually).
ISLS
309
ICLS2012
Dyad 16 (Innovation Condition): Noticing that the two challenging cases show a similar difference between the observed and expected values but one case has twice as many rolls as the other. They succeeded in differentiating the cases verbally and mathematically. C: Lets see. What if we say like the number these (Number of rolls in Case 1) should be is 5, the difference of each of these 5-5, 6-5 ah 4-5, 6-5 and then 4-5 equals '-1. And then, how do we do this one? D: So C: Ok. And then So these numbers (Number of rolls in Case 3) distribute the same way right? They are all have 2 on them, 2 -1s, but this (Case 3) is probably more fair than this (Case 2). D: Because its (Case 3) rolled a higher number of times? C: Right, right. Its like a better study right? . C: We dont know how to do that in terms of numbers. He he he he! Experimenter: Just try to invent one. It doesnt have to be perfect. D: I mean I guess you could average them, right? C: If you average them, these (Case 1 and Case 3) are both going to come out the same, right? So then because adding them up could get 0, so if you was it like squaring it? D: To make it positive? C: Yeah if you square this (Each of E-O). D: Thats still the same number. C: And then divide it by D: The number of times rolled? C: Yeah. Will that work? Cause if we squared it (each cell of Case 3) 1, 4. If we square this its 4 but if we divide it by 50 D: 50 then by 25. C: By 25. Then if we did it with this we have to do the same thing right? . D: Yeah, this one is .08. (Case 3) C: 4/50? D: Uhum. 1.28 (Case 2), 0.16 (Case 1) C: Ok, so the smaller the number, the more fair. He he he he. D: Yeah, might as well be. Group 16 shows a good example of how the students co-constructed a solution that differentiated all three cases, even the two most similar cases. After getting the expected value, this group subtracted the expected value from the observed values and summed them to see how much each case differed from expected. They found that case 1 and 3 yielded the same (O-E) value. The contrasting cases were designed to be tied by this approach in order to block this incomplete solution and push for considering another operation (i.e., dividing by N or E). Noticing the tie, the participants thought that case 3 should be more fair than case 1 because it was rolled more times. After struggling for a moment, they eventually thought of squaring the (O-E) to avoid negative values. Seeing that this still produced a tie, the partners co-constructed the idea of dividing by the number of rolls (N). This was considered an ideal solution to the difficult contrasting cases. One of the participants in this dyad correctly solved the target transfer posttest problem. Almost all groups in the Traditional condition went through the numerical solutions smoothly. Their activities showed considerable focus on calculations and error checking. They tended to plug in numbers into the formula they saw in the Lesson page right away. Examples from Dyads 2 and 5 are shown below. Dyad 2 (Traditional Condition): Getting the numbers for each component of the formula (E, E-O, E). E: Ok so um... for each one (cell) E should be 1/5th, right? F: Yeah. E: Then 1/5th of the number times, so 5 for the first 2 cases and 10 for the third case. So do you want to just write it off? F: Yeah, I just completely forgot the chi square formula. E: Ok so chi square = e-o squared over e. So for case 1 and 2 e = 5, e=5 then e=10. Then we just add up the 5-5 squared. E: 5-6 squared over 5, 5-4 squared over 5. Like that so, um so this (Number 5 in the cell) ones gonna be 0 because 5-5 is 0 but um So all these are going to be 1/5.
ISLS 310
ICLS2012
F: 1/5 1/5 1/5 so chi square for this is going to be 4/5ths. Right? E: Right. Dyad 5 (Traditional Condition): Plugging in numbers and checking calculations. G: Ok why dont you plug them in? H: Ok so it was sum of expected - observed squared over expected. So for the first one we had..um it was rolled 25 times so like this is the first side so um G: For this one its also 5 sided we rolled it. Does it matter that this (Case 1 and Case 2) is rolled 25 and this (Case 3) is rolled 50? H: I think its just gonna give us a different fraction. G: Alright. Ok. So then, this (Case 1) one we expect each one of these (Number of rolls) to be 5, um so again this (Case 2) is 5 sided. So this is 1/5 and 1/5 ok, this one is 2 squared is 4/5 thats 4/5ths. Did I do that right? H: Uhum. G: And this is also what oh no sorry that wasnt 2 that was 5 (Case 2) ok yeah so this is um so yeah, this is 10-5 no 5-10. Um yah so thats 5 squared 25/5 so 5. Ok and then for this one this is 2. H: 4/5 and then 1/5 again? G: Ok 2/5 3/5 7/5 and 5 7/5 is H: 6 and 2/5. G: 6 and 2/5. H: So this (Case 3) most fair ok. This is least fair (Case 2) this is somewhat fair (Case 1). Although the Traditional condition appeared to push for a focus on computational accuracy, it did not prevent participants from asking important questions, such as, does it matter that this is rolled 25 and this is rolled 50 [times]? (Dyad 5). However, as suggested by the partners response, it also did not necessitate developing answers to those questions that would yield insight into why the chi-square formula operated as it did. The subsequent results show the results of the coded transcripts across all 18 dyads. Table 1 shows the coding scheme. The codes are numbered to suggest a sequence by which Innovation dyads might proceed through the problemnoticing key features first and then formalizing their understanding mathematically in increasingly complex (and accurate) ways. Table 1: Coding scheme for dyadic interactions. Coding Goals 1st goal Perceptual Ranking Rank the three cases correctly without numerical solution Code 1. Case 2 is least fair. 2. Case 1 and Case 3 are hard to differentiate (e.g., they show the same range; tied according to [(E-O)]). 3. Case 3 has more rolls (50 vs. 25) and make a judgment. 4. Find E (the expected value) 5. (E-O) and/or (E-O)2 6. OR to differentiate cases N E Examples of quotes that earned credits 1 This one (case 2) is definitely least fair dyad #9 1 I was thinking like this (Case 3) deviates from 10 by 1 and this by 1 this by 1 total of 4 deviations . this one (Case1) has 4 too dyad #11 1 This ones 25 (Case 1) and this ones 50 (Case 3) dyad #1 1 1/5th of the number times [total number of rolls], so 5 for the first 2 cases and 10 for the third case dyad #2 1 it's going to be 5 minus 5 is zero squared over 5 is zero. dyad #17 1 so (5-6) squared over 5 so that 1/5 dyad #18
2nd goal Numerical Solution Provide a single value for each case that can uniquely rank them
Table 2 shows the results of the coding scheme for each condition. It also shows performance on the target transfer problem. Five of the 18 individuals in the Innovation dyad condition succeeded on the target transfer posttest problem versus only one of 18 individuals in Traditional dyads. Importantly, all five of those Innovation individuals worked in dyads that received credit on every code. Put another way, only one individual in the Innovation dyads that succeeded on the practice problems failed to successfully complete the double-transfer problem on the posttest. Thus, the learning activity, when done in Innovation fashion, was
ISLS 311
ICLS2012
diagnostic of participants level of understanding. Only those students who both perceived the key features of the problem and translated them into a mathematical operation successfully answered the target transfer posttest question. In addition, it is apparent from the decrease of success between code 4 and code 5 from seven out of the nine dyads to only four that many Innovation dyads struggled to translate their perceptions into mathematical operations. This finding could be valuable to educators because it suggests moments at which teacher scaffolding could be most effective, not just for initial success but potentially for subsequent individual transfer as well. In contrast to the Innovation condition, the results for the Traditional condition appeared neither successful nor diagnostic. Only one of 18 participants in the Traditional dyad condition succeeded on the target transfer posttest problem. However, nearly every dyad calculated the chi-square statistic on the practice problems. Thus, the practice problems not only failed to differentiate students with deep understanding from those needing additional support, they failed to prepare students to transfer their knowledge. This is not to say that lesson-then-practice instructional methods cannot facilitate deep understanding and transfer. We suspect, instead, that unless a problem requires students to compare across cases to make sense of how a formula works, they may fail to do so. Table 2: Number of dyads that got credit for each coding element, and number of those individuals who succeeded on the target transfer problem. Aggregated Data
1. Case 2 is least fair 2. Notice that Case 1 and Case 3 are hard to differentiate. 3. Notice that Case 3 has more rolls and make a judgment 4. Find E 5. (E-O) and/or (E-O)2 6. OR to differentiate cases N E
Innovation (n=9 dyads) Code Target Transfer 7 5 of the 14 dyads individuals 7 5 of 14 6 5 of 12 7 5 of 14 4 5 of 8 3 5 of 6
Traditional (n=9 dyads) Code Target Transfer 3 1 of the 6 dyads individuals 0 0 2 0 of 4 8 1 of 16 7 0 of 14 7 0 of 14
Discussion
In this study, there were clear differences between Innovation dyads and Traditional dyads in their approach to solving statistics problems. The Innovation condition successfully guided participants attention to relevant variables, and in some cases further guided them to develop near-expert mathematical solutions. In contrast, the Traditional dyads received the canonical formula and tended to focus on computational accuracy, giving less attention to understanding why or how the chi-square formula works. These behavioral differences appeared to have important implications for learning. First, many more participants in the Innovation condition subsequently succeeded on a challenging transfer problem (taken individually). Only one of the 18 Traditional participants solved the double-transfer problem successfully versus five for Innovation. Second, the Innovation practice problems were diagnostic. Innovation dyads tended to have difficulty in developing a general formula that differentiated all three contrasting cases (dice). Those who struggled at translating their perceptions or verbal understanding into a mathematical formula struggled on the subsequent transfer problem. However, members of Innovation dyads who succeeded at differentiating all three cases on the practice problems succeeded on the transfer problem (except one). This suggests a moment where teachers could offer targeted and effective scaffolding. Such teachable moments were apparent in the transcripts of the Innovation dyads but not of the Traditional dyads because they tended to calculate the correct answers during the learning phase. While these results support others findings that success during problem solving does not necessarily indicate deeper understanding, they also raise questions about what productive failure during innovationoriented learning activities consists of (Kapur, 2010). On average, the failure of the Innovation participants to invent the chi-square formula during the learning phase did not preclude them from future success on transfer (and greater success than the Traditional condition). In this sense, the notion of productive failure during innovation activity applies to this study. However, only those Innovation participants who invented a nearexpert solution during the learning phase succeeded on the target transfer posttest question. This suggests that meeting key indicators of progress during productive failure innovation activities may be necessary for deep understanding from subsequent lessons. The contrasting cases in the current study seemed particularly important to ensuring such progress. It was clear from the transcripts that many of the successful Innovation dyads may have settled for a more simplistic mathematical solution had the contrasting cases not blocked those
ISLS
312
ICLS2012
initial attempts. We suspect that inventing experiences that specifically push students to notice and relate key variables in ways that yield perceptible progress during the activity (e.g., allow students to differentiate more cases) are likely to facilitate their understanding and offer teachers more opportunities for effective scaffolding and teaching.
References
Antil, L. R., Jenkins, J. R., Wayne, S. K., & Vadasy, P. F. (1998). Cooperative learning: Prevalence, conceptualizations, and the relation between research and practice. American Educational Research Journal, 35(3), 419-454. Barron, B. (2000). Achieving coordination in collaborative problem solving groups. The Journal of the Learning Sciences, 9, 403-436. Barron, B. (2003). When smart groups fail. The Journal of the Learning Sciences, 12, 307-359. Bennett, N., & Dunne, E. (1991). The nature and quality of talk in co-operative classroom groups. Learning and Instruction, 1, 103118. Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups. Review of Educational Research, 64, 1-35. Chi, M., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145-182. Dillenbourg P. (1999) What do you mean by collaborative leraning?. In P. Dillenbourg (Ed) Collaborativelearning: Cognitive and Computational Approaches. (pp.1-19). Oxford: Elsevier. Dillenbourg, P., J rvel S., & Fischer, F. (2009). The evolution of research on computer-supported , collaborative learning. In Editor: N. Balacheff, S. Ludvigsen, T. Jong, A. Lazonder, & S. Barnes (Eds.) Technology-Enhanced Learning (pp. 3-19). Netherlands: Springer. Gillies, R. M. (2002). The residual effects of cooperative-learning experiences: A two-year follow-up. The Journal of Educational Research, 96(1), 15-20. Gillies, R. M. (2003). The behaviors, interactions, and perceptions of junior high school students during smallgroup learning. Journal of Educational Psychology, 95(1), 137-147. Johnson, D. W., Johnson, R. T., & Stanne, M. B. (2000). Cooperative learning methods: A meta-analysis. Minneapolis, MN: University of Minnesota. Kapur, M. , & Kinzer, C. K. (2009). Productive failure in CSCL groups. Computer-Supported Collaborative Learning, 4, 2146. Kapur, M. (2010). Productive failure in mathematical problem solving. Instructional Science, 38 (6), 523-550. Koenig, C. S., & Griggs, R. A. (2004). Analogical transfer in the THOG task. The Quarterly Journal of Experimental Psychology, 57A, 557-570. Laughlin, P. R., Hatch, E. C., Silver, J. S., & Boh, L. (2006). Groups perform better than the best individuals on letters-to-numbers problems: Effects of group size. Journal of Personality and Social Psychology, 90, 644-651. Marton, F. (2006). Sameness and difference in transfer. The Journal of the Learning Sciences, 15, 499-535. ODonnell, A. M. (1996). Effects of explicit incentives on scripted and unscripted Cooperation. Journal of Educational Psychology, 88(1), 74-86. ODonnell, A. M. (1999). Structuring dyadic interaction through scripted cooperation. In A. ODonnell and A. King (Eds.), Cognitive perspectives on peer learning (pp.179-196). Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc. ODonnell, A. M. (2006). The role of peers and group learning. In P.A. Alexander & P.H. Winne (Eds.), Handbook of educational psychology (2nd Ed.) (pp. 781-802). Mahwah, NJ: Lawrence Erlbaum Associates. ODonnell, A. M & Dansereau, D. F. (1992). Scripted cooperation in student dyads: A method for analyzing and enhancing academic learning and performance. In R. Hertz-Lazarowitz & N. Miller (Eds.), Interaction in cooperative groups: The theoretical anatomy of group learning (pp. 120-141). New York: Cambridge University Press. Phelps, E., & Damon, W. (1989). Problem solving with equals: Peer collaboration as a context for learning mathematics and spatial concepts. Journal of Educational Psychology, 81(4), 639-646. Remedios, L., Clarke, D.J., & Hawthorne, L. (2008). Framing collaborative behaviours: Listening and speaking on problem-based learning. The Interdisciplinary Journal of Problem-Based Learning, 2(1), 1-20. Schwartz, D. L. (1998). The productive agency that drives collaborative learning. In P. Dillenbourg (Ed.). Collaborative learning: Cognitive and computation approaches (pp. 197-218). New York: Elsevier Science/Permagon. Schwartz, D. L & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16, 475-522. Schwartz, D. L., Bransford, J. D. & Sears, D. A. (2005). Efficiency and innovation in transfer. In J. Mestre (Ed.), Transfer of learning from a modern multidisciplinary perspective. CT: Information Age Publishing.
ISLS
313
ICLS2012
Schwartz, D., Sears, D., & Chang, J. (2007). Reconsidering prior knowledge. In M. Lovett and P. Shah (Eds.), Thinking with data (pp. 319-344). Mahwah, NJ: Erlbaum. Sears, D. A. (2006). Effects of innovation versus efficiency tasks on collaboration and learning (Doctoral dissertation, Stanford University, 2006). Dissertation Abstracts International, 67, 1652A. Slavin, R. E (1983). When does cooperative learning increase student achievement? Psychological Bulletin, 94, 429-445. Slavin, R. E. (1996). Research on cooperative learning and achievement: What we know, what we need to know. Contemporary educational psychology, 21, 43-69. Souvignier, E. & Kronenberger, J. (2007). Cooperative learning in third graders' Jigsaw groups for Mathematics and Science with and without questioning training. British Journal of Educational Psychology, 77, 755771. Stacey, E. (1999). Collaborative learning in an online environment. Journal of Distance Education, 14(2), 14-33. Vedder, P. (1985). Cooperative learning: A study on processes and effects of cooperation between primary school children. Groningen, The Netherlands: University of Groningen. Webb, N. M., Troper, J. D., & Fall, R. (1995). Constructive activity and learning in collaborative small groups. Journal of Educational Psychology, 87(3), 406-423.
Acknowledgments
We thank Arnold Frias and Megan Wagner for their assistance in transcribing participants interactions.
ISLS
314
ICLS2012
Young Childrens Everyday Inquiry: A Field Study of a Young Girls Play Across Contexts
Danielle Keifert, Northwestern University, 2120 Campus Drive Evanston IL 60208, keifert@u.northwestern.edu Abstract: This paper documents the naturally occurring ways in which very young children encounter opportunities for inquiry in their everyday lives. Understanding these early childhood practices is a necessary first step in drawing on these practices as resources for science inquiry learning. Using approximately 35 hours of interactional video data of a twoyear-old girl in home and preschool settings, I describe the inquiry practices in which she engages at home, particularly how she orchestrates adult support for inquiry, and how she draws on aspects of her home inquiry practice in school. Based on her everyday experiences with inquiry and the common interactional arrangements of home and school environments, I suggest ways for conceiving of each setting as having affordances for the support of science inquiry among very young children.
Introduction
Researchers are seeking to identify resources for science classroom learning in the rich variety of practices found within childrens everyday lives. These resources include ways of thinking and talking about everyday experiences which are embedded within cultural communities and their practices (Bricker & Bell, 2007; Hudicourt-Barnes, 2003; Warren, Ogonowski, & Pothier, 2005). The literature documents everyday practices as rich, varied, and most importantly productive for science learning. However, few have spent time articulating the details of the interactional arrangements in which these practices are embedded (Stevens, Satwicz, & McCarthy, 2008), or examined how differences in the kinds of interactions across contexts impact participation in these practices as resources for learning. It is within interactions that individuals do the work of trying to understand everyday experiences. Individuals work together in interactions to mutually co-construct meaning, contexts, and the activity itself (Erickson & Schultz, 1997). If everyday activity provides resources for science inquiry, we must understand how people mutually orient to and do inquiry in their everyday lives. I begin this paper with a discussion of current conceptualizations of everyday inquiry. I then follow one child across two settings to understand the ways she works to draw in adults to support her questioning, the responses of adults in various settings, and the resulting co-constructed activity. I also explore the affordances of the interactional arrangements in which these naturally occurring inquiry experiences occur across the multiple contexts of this childs life, and how practices found in one context may be drawn upon in another.
What We Know About Everyday Inquiry

Current research shows that people build their knowledge based on everyday experiences. As toddlers, we build explanatory accounts to help us make predictions about what might happen in the future (Hawkins & Pea, 1987), and much of these explanatory accounts are influenced by the ways we learn to notice, ask questions, and provide explanations (Crowley, Callanan, Jipson, et al., 2001; Goodwin, 2007). From the very first moments of our lives, the communities in which we live shapes the ways we observe, think, and talk about the world (Cole, 2007). Despite the importance of these everyday experiences for very young children, we have an inadequate understanding of them (Callanan & Jipson, 2001). While existing research has examined young childrens causal questions in everyday settings (Callanan & Oakes, 1992; Hood & Bloom, 1979), causal questions are only one element of inquiry, and not all good inquiry takes the form of causal questions. There are also many ways of participating in inquiry beyond questioning. Thus, the existing literature provides only a start to understanding childrens early inquiry experiences. A more detailed examination of childrens encounters with inquiry within everyday contexts is needed. Many researchers argue there are important similarities between everyday thinking and how scientists think about inquiry: both attempt to provide explanations of our world, and both use many of the same intuitive practices to understand complex phenomena (Hawkins & Pea, 1987; Ochs, Gonzales, & Jacoby, 1996). Dewey (1981) argues inquiry in all settings is the process of noticing a problematic situation and working to better understand that situation in a way that allows for a satisfactory resolution to the problem. This may include questioning, relating the problem to other problems, or representing and discussing the problem. Deweys definition of inquiry is applicable to both an informal everyday sense of inquiry as well as a formal scientific one, although the rules and norms for a satisfactory resolution in everyday inquiry are different than those for formal science (Hawkins & Pea, 1987). While everyday inquiry is not the same as classroom or professional science inquiry, it shares important patterns and can help build a foundation for later inquiry learning. Rather than see the everyday and scientific as dichotomies, some (Dewey, 1981; Ochs & Taylor, 1992; Stevens & Hall, 1998; Warren, Ballenger, Ogonowski, Rosebery, & Hudicourt-Barnes, 2001 among others) argue for a
ISLS
315
ICLS2012
continuum view that allows us to avoid the prejudicial notion that there is some gap in kind (as distinct from degree) between the childs experience and the various forms of subject-matter that make up the course of study (Dewey, 1981, p. 472). Thus the challenge becomes how to characterize the inquiry activity of everyday practice in such a way that we can better understand how to draw upon these practices as resources for discipline learning. This includes understanding the practices themselves, but also considering how these practices do or do not move across context boundaries. Only a handful of researchers examine science and inquiry learning in both everyday and classroom environments, conceptualizing the everyday lives of children as places where they gain important resources for learning (Bricker & Bell, 2007; Hudicourt-Barnes, 2003; Mehus, Stevens, & Grigholm, 2010; Warren et al., 2001; Warren, et al., 2005; Zimmerman & Bell, 2007). However, only one set of these researchers has examined home and school contexts for preschool aged children (Mehus et al., 2010). This paper furthers prior work by examining everyday inquiry practices as they are embedded in naturally occurring activity. I focus on examining extensive video interactional data of one two-year old child, Marie, across both home and preschool settings over the course of several months. Through observations of Maries everyday interactions with her parents, brother, preschool peers, and teachers, I identify and analyze opportunities for inquiry within naturally occurring interactions. In this paper I (1) identify the characteristics of Maries inquiry, particularly with regard to the ways in which she orchestrates resources to understand phenomena and support her inquiry, and (2) examine how different interactional arrangements (across home and school settings) affect the inquiry practice. I conclude this paper with (3) a discussion of some of the ways we might begin to conceive of the affordances of each particular context as we consider the design of future science inquiry learning, and pose questions raised by this cross context analysis.
The data in this paper come from the LIFE Centers Early Learning Across Contexts (ELAC) project led by Reed Stevens. ELAC is designed to explore the everyday lives of young children across multiple settings. The data consist of over 500 hours of video data from both home and school settings for seven focal participants (2 to 5 years-old) across five classrooms in two different preschools in a large urban Western city. Observations were made over period of three to six months in 2009. By observing children in multiple settings and among different interactional arrangements (e.g., with parents, peers, siblings, teachers) ELAC seeks to capture the diversity and complexity of the social environments within which and through which young children learn. For the purposes of this paper I focus my analysis on one child, Marie. At the beginning of observation Marie is two years old but she turned three during the study. She has an older brother, Evan, who is six at the beginning of observations and turned seven during data collection. Maries mother (a doctor) and father (a marine biologist) share responsibility for child-care within their home. We have four months of observations for Marie at home (20 hours over 11 days), and three months of video of Marie at school (14.5 hours over 8 days). My process of analysis followed guidelines set forth by Erickson and Schultz (1997) and Pomerantz and Fehr (1997). After initial capture of video data, content logs were created describing major activity and interactions (Jordan & Henderson, 1995). I then viewed the video again, following interactional analysis methods to look for patterns in the interaction. Interactional analysis examines everyday social interactions and the underlying social organization of those interactions as demonstrated through the activity of the participants in their mutual orientation to each other and the context (Goodwin & Heritage, 1990). A primary assumption of this approach is that the conduct of everyday life is sensible and meaningful for those involved in the interaction (Pomerantz & Fehr, 1997). When observing children, Ochs (1979) articulates the need to pay special attention to the multiple ways in which children may participate in interactions, particularly as children may follow different norms than adults. Ochs highlights childrens gesture and gaze as particularly important for very young childrens communication. As such, my full transcripts (not included here for purposes of brevity) include separate columns for action, gaze, and body orientations temporally aligned with talk for each interaction. I focused my initial analysis of Marie at home on one particular sequence of activity. This activity occurs during one afternoon of pool play when Marie and Evan are home with Dad. I chose this sequence because it illustrates a broader pattern of an inquiry practice within the full corpus of Maries family data. During the 35 minutes of play in their backyard pool several series of questions, conversations, and explorations occur. In this article I focus on a sequence of questions and explorations about attempting to read a thermometer and to understand how water changes with temperature. Using this interactional pattern as a base of comparison, I discuss similar patterns within all of the data of Marie at home. My analysis of Marie at school examines whether she initiates or engages in this inquiry practice when at school. One particular school interaction will be examined in depth to determine to what extent Marie may be engaging in a similar pattern of activity, drawing on features of interaction similar to those within the home inquiry practice. Finally, I consider the interactional features that are present across contexts and raise questions about the extent to which interactional preferences and norms within contexts enhance or inhibit Maries inquiry practice.
Research Methods
ISLS
316
ICLS2012
Maries Inquiry at Home

In this section I provide a series of verbal snapshots from an unfolding interaction initiated by Marie and focused on a thermometer. I will examine this interaction with the thermometer, consider the broader activity of the afternoon of pool play in which it occurs, bring to bear relevant examples found throughout the home data, and compare these interactions with relevant literature. I begin this analysis by presenting the unfolding interaction in terms of Maries initiation and Dads response. I then consider the role of questions and interactional preferences, before concluding with a discussion of the co-construction of these interactions and argue that within these interactions Marie and Dad together orient to an established family practice.
Marie as Initiator
Marie frequently initiates inquiry interactions, often times with repeated requests. Marie repeatedly asks What is this for? while Dad is interacting with Evan, and is eventually successful in gaining Dads attention. These tactics for gaining Dads attention might not be considered appropriate within adult social norms, but Ochs (1979) asserts that we must recognize children operate under different norms for both verbal and non-verbal communication. An adult repeating a question eight times would likely be heard as inappropriate, but in this case, Maries repeated questioning is successful at initiating an interaction with Dad without admonishment from him, suggesting the repetitions are considered appropriate for this child-parent interaction. This pattern of Marie initiating inquiry interactions is also visible across multiple instances throughout the home data. In the afternoon of pool play Marie offers ten total invitations (e.g. Look Dada! Look Dada while pointing to a bug, or Look it! Look at down there! while jumping and pointing to a spot on the pool floor). Eight of these initiations result in interaction, and four interactions are extended consisting of more than one back-and-forth exchange. During other observations, Marie initiated extended inquiry interactions with Mom when at the zoo regarding a crows feather she found in a field, with Mom when observing their recently hatched butterflies, and with Dad when asking why he was moving a plant in their garden. These examples show a pattern of Marie orchestrating support for inquiry from both Mom and Dad. Through these initiations Marie demonstrates an ability to recognize phenomena and objects that she has questions about and that she sees as interesting or problematic. This evolving habit of noticing suggests Maries developing epistemology in a way similar to the developing habits of noticing described by Stevens and Hall (1998) in their description of disciplined perception. Marie also demonstrates her interest in exploring these phenomena and objects with Dad as well as her ability to successfully draw Dad into interaction to explore these problems.
Dad Responds To and Shapes Unfolding Inquiry

Dads response to Maries question is critical for supporting the unfolding inquiry. During pool play, both Marie and Evan make bids for Dads attention. When Dad responds to Maries initiations (as he does 8 out of her 10 initiations, although sometimes only after Maries repeated requests) he does so by shifting his focus of talk and gaze, as well as shifting his body orientation and location in the pool to engage with Marie. In the case of the thermometer, Dad begins by physically facing Marie while responding to her question: Dad: That's for telling the temperature of something. Telling whether something, it tells you whether it's hot or cold. Dad: Look if. See the red in there. The red line? If the red is, if the red line goes up in here in this red area its hot. And if it goes in the blue area its cold. Dad moves closer to Marie and thermometer. Dad points to various parts of thermometer in Maries hand as he describes how the thermometer works.
Through this interaction Dad provides a further explanation of how the thermometer works, describing hot and cold with regard to the colors and lines printed on the thermometer. After Dads initial response to Maries question, he prompts her to further explore the thermometer by encouraging Marie to try to read the thermometer, saying, So is it hot or cold right now? At this point we also Dad has shifted his gaze from the thermometer to looking at Marie as he asks the question. Both his question and his gaze indicate to Marie that an answer is expected of her. After Maries response that it is cold, Dad provides a correction of her thermometer reading by saying, Its sort of in the middle, isnt it? He continues to provide additional information to Marie about reading the thermometer by explaining that the temperature is starting to get warm and reading about twenty-five degrees. This interaction involves back and forth exchanges between Marie and Dad in which Dad supports an exploration of what the thermometer is and how to use it. Through these exchanges, Marie has an opportunity to explore the object about which she first asked. Together, Marie and Dad co-construct an opportunity for inquiry in response to Maries initial question. Although much literature tends to focus on the ways in which parent explanations shape childrens understanding and sometimes the ways in which parents shape the actual types of interactions with a phenomena that children have (Crowley, Callanan, Jipson, et al., 2001; Crowley, Callanan, Tenenbaum, & Allen, 2001; Crowley & Galco, 2001; Crowley & Jacobs, 2002; Fender & Crowley, 2007), most of these studies
ISLS
317
ICLS2012
do so within the context of a highly designed setting such as a science museum exhibit. However parents actions serve to shape childrens experience and interpretation of the world well before they are old enough to benefit from these kind of designed settings (Cole, 2007). In this section I presented some of the ways that, in response to a childs initiation, parents impact childrens experiences in daily encounters with everyday phenomena by shaping interactions and providing material resources to support inquiry.
The Role of Questions

Questioning, exploring, and observing are interrelated through this practice, as they are within the practices of science inquiry more broadly. Questions help to drive much of the activity throughout this and other inquiry practices (Penney & Stevens, 2011). While Maries first question (What is this for?) is an information query, other forms of questions play an important role in the interactions. Another question form includes those of a more imaginative nature: hypothetical What if? questions. A question of this form occurs in a later portion of the thermometer interaction when Dad asks Marie what will happen if the pool is one hundred Celsius. Dad: And, Marie, you know what happens if it goes up up up, way up to here? Marie: What? Dad: If it gets up to a hundred do you know what will happen? Marie and Evan: What? Dad: Water will start to boil and turn into steam. Dad: Would you like to be in there if it was a hundred degrees Celsius? Marie drops thermometer. Dad moves toward Marie, picks up thermometer.
Dad holds thermometer, looks towards Evan. Dad steps away. Marie watches thermometer. Evan picks up thermometer.
Dads last question asks Marie and Evan to imagine what it would be like to be near (or in) boiling water. This sort of hypothetical question asks the children to imagine being in the pool with boiling water, and what boiling water is like to experience. This form of imaginative embodiment is an important way that children can engage in inquiry and reasoning and a practice which is used by scientists (Ochs, et al., 1996). Hypothetical questions may also be causal questions. Later in the afternoon of pool play, we see Marie initiating an interaction focused on a white spot on the otherwise blue pool floor. Marie has been pointing out the spot with excited talk (Look it! Look it down there!) and gesture (pointing and jumping on the spot) for 40 seconds while Dad and Evan continue the thermometer interaction. Dad finally shifts to a shared focus on the spot with Marie, affirming that Marie has noticed something interesting through both his shift in gaze towards the spot and his comment of Oh! That is an interesting-. This affirmation reinforces for Marie that this sort of thing is of value to perceive, and his continuing response, Why do you think its white there?, encourages exploration, further reinforcing her evolving disciplined perception. This causal question about why the pool floor is white only in that one spot asks Marie to consider possible explanations for why something is the way it is. This is an important question form within scientific inquiry practices (Hawkins & Pea, 1987).
Interactional Preferences
Although at times we see Marie interacting with both a parent and her brother, there is interactional evidence to suggest that within the inquiry practice she has a preference for interaction that involves only herself and a parent. In the pool interactions we see this in a few ways. First, when Marie does invite a participant by name it is only Dad. Second, although there are a few occasions when Marie shares a joint-focus with Evan and Dad, in both cases she quickly leaves the interaction and shortly begins to attempt to draw Dads attention to a new phenomenon. For example, when Evan asks a question Marie terminates her interaction with the thermometer by walking to another place in the pool, and then attempts to draw Dads attention to a spot on the pool floor. Shortly thereafter, once Evan joins Dad and Marie near the spot on the pool floor, Marie walks away and returns to the thermometer. Through these actions Marie is working to initiate and maintain interactions in which only she and Dad share the joint focus. The interactional evidence to support a preference within this family for interaction with one child at a time is furthered by examining interactions from Dads standpoint. When Dad invites either child to engage in an activity by name, he only does so with one child at a time. We see this when he names Marie in his hypothetical question about the thermometer. Dad does occasionally split his attention between Marie and Evan, as seen in examining his gaze (after both Evan and Marie chime in with What? in response to Dads question about being in a pool that was one hundred degrees); Dad looks at Evan, not Marie, as he answers the question he posed. However, on the whole, Dad goes to great lengths to maintain an interaction with one child at a time.
Co-Constructors of a Family Practice

Marie and her parents play different roles within the co-constructed inquiry. Marie often serves as the identifier of the phenomenon of interest through her observations of phenomena or artifacts, and her requests for Mom
ISLS
318
ICLS2012
and Dad to pay attention to those foci. When these initiations are successful, Dad or Mom will then share a joint-focus with Marie. This joint-attention can be seen through both the talkwhen a shared talk focus is establishedas well as gaze and body observationshow gaze and bodies orient towards the focus and other participants within the interaction. Once a shared focus has been established, Mom or Dad tend to shape the nature of the inquiry that is occurs. For example, Dad does this, through the description of the thermometer and his attempts to help Marie learn how to read the thermometer. Adults are often conceptualized as providing support for children as they encounter phenomena, but we do not as often characterize children as agents in gaining that support. Maries act of drawing attention to the phenomena she finds problematic and interesting (as shown through her questioning) leads to sustained inquiry only when Dad or Mom engage with Marie, but also only as long as Marie demonstrates interest. Thus this is a co-constructed practice in which both the actions of Marie and parent are critical components of developing inquiry. However, although the practice is co-constructed, Marie often plays the pivotal role of bringing attention to the focus of inquiry. She has developed a sense of how to initiate the inquiry practice with her observations and questions. She is also gaining experience with hearing and asking many forms of questions (information seeking, imaginative hypothetical, causal) that are a part of the inquiry practice. The pattern of interaction demonstrated in this paper, along with reoccurring evidence of this pattern throughout the home data, support the argument that this inquiry practice is well established within this family. This claim is supported by the frequency with which the practice occurs in the short pool playtime (12 short, 7 extended interactions in 36 minutes, 30% of total pool playtime), suggesting the participants familiarity with the practice. Related to this is the participants repeated shared orientation to the practice, and postural movements to maintain the practice (McDermott, Gospondinoff, & Aron, 1978). Finally, the existence of the practice within other interactions between Marie and her parents on many of the observed days supports that this inquiry practice is a well-established routine through which Marie engages with both Mom and Dad.
It is an open question whether and how practices such as this familys inquiry travel across contexts (Stevens, Wineburg, Herrenkohl, & Bell, 2005). Does Marie carry her inquiry practice beyond the home context? This question is worth exploring in detail, because if want to leverage practices that are embedded within particular contexts for discipline learning, we need to understand the interactional arrangements in multiple settings that can support the practice. As shown in the above analysis, Marie competently engages her parents in inquiry of phenomena that interest her at home. In this section of analysis, I consider an interaction between Marie and a teacher to determine if Marie brings any characteristics of her home inquiry practice into her interactions within the preschool setting. In particular, I look to see if Marie has similar success in recruiting adults in preschool to joint inquiry as she does frequently at home, and examine how she attempts to draw others into interaction. As we consider the ways Marie may bring aspects of her home inquiry practice to the preschool setting, we can examine the ways in which Marie may draw attention to interesting phenomena or attempt to draw adults into interaction. However, the ways in which other participants within the preschool setting interact with Marie is a critical component of whether an interaction similar to the home inquiry practice occurs. Thus, as we examine the following episode that took place during snack at Maries preschool (approximately one month before the afternoon of pool play), we will look both for the ways in which Marie attempts to initiate interactions as well as the ways in which adults and other children respond to Maries initiations. During the episode that follows, several children are gathered around a table eating their morning snack. All the children in this particular classroom are ages two to three years. Jane, Scott, and Marie are seated on one side of the table, while Anne and Rose are on the other side. Stacey, a teacher, stands nearby as she takes care of the class cricket terrarium. After a question from Marie and a request from Scott to see the crickets, Stacey places the terrarium on the snack table in front of the children and then sits down between Anne and Rose. Shortly after Stacey moves the terrarium to the snack table, Marie begins to ask questions about the crickets. Marie begins by asking an imaginative hypothetical question: Whats if the crickets got out? This begins a pattern of back and forth questions and responses between Marie and Stacey. Stacey responds to Maries question that she doesnt think the crickets can escape from the cricket house. She says this is because they (the classroom) have a good cricket house. Maries question is of a similar type to a hypothetical question Dad might ask as part of the home inquiry practice (Would you like to be in here if it was a hundred degrees Celsius?). Staceys response emphasizes, and supports with evidence (they purchased a good terrarium), why the cricket escape wont happen. In the continuing interaction, Marie keeps asking imaginative hypothetical questions about a cricket escape (What if they got out and they crawled over your head and onto your eyes?, What if they got out and chase them?, What if crickets get out and got on your head for a whole week?) After each variation of the question, Stacey provides an explanation that communicates in one way or another that the crickets cannot escape and should not escape for safety reasons. There is a playful pattern to the back and forth in which Marie
Finding the Home Inquiry Practice at School
ISLS
319
ICLS2012
finds new and ever more dramatic ways of describing a cricket escape while Stacey explains how its not safe for them and she would return them to their home. Within this playful exchange Marie is once again demonstrating her ability to identify a phenomenon of interest and to initiate an interaction to explore that phenomenon. Marie uses imaginative hypothetical questions similar to those used by her parents during home inquiry interactions to start and maintain this interaction. Thus Marie is drawing on home inquiry practices (hypothetical questions) while at school. However, Maries repetition of her question may indicate she is not getting the kind of response from Stacey that she wants (and is therefore not exploring the idea of a cricket escape in the way she desires). Additionally, it is important to note that this is the only episode within the 14.5 hours of school data in which Marie asks this kind of question, though Marie does on two occasions propose imaginative hypothetical situations to teachers. The sparseness of inquiry practice or related components in school may be the result of Maries peers and teachers not orienting with Marie to shared inquiry in response to her initiations, as is the case in the example provided. This suggests a challenge for Marie if she continues to want to explore phenomena of interest to her in this school setting.
Home & School as Settings for Maries Inquiry Practice

This section addresses the issue of the relationship between the interactional arrangements and Maries inquiry practice. This is a comparative analysis of the home and school contexts. This includes a consideration of the similarities between home and school setting with regard to the interactional affordances of both contexts that support Maries inquiry. It also includes a discussion of some of the differences between the affordances that could allow for different kinds of inquiry interactions. The interactional arrangements both at home and school provide opportunities for children to ask questions. The afternoon of pool play and observation of crickets share several common interactional features. First, both arrangements had children and adults constrained within one area, allowing for the occasioning of mutual orientation to a shared focus. At the same time, neither of the arrangements has pre-defined topic of conversation (although of course every context has some limits). Because the children were physically constrained within an area, but not otherwise expected to interact in any particular way, they were free to discuss and explore anything within their immediate area. For Marie in the pool available material resources include the thermometer, while at school, once Stacey moves the terrarium to the snack table, available resources include the crickets. The provision of material resources of interest to Marie (e.g., thermometer, cricket terrarium) serves as a critical component of these arrangements. I would argue that these features of the arrangements supported the extended interactions, and in the case of the thermometer reoccurring interactions, around a shared focus. There are also important differences between the interactional arrangements found within the home and school settings. Most obviously, the number of children present in the school setting creates a different set of affordances for interactions. While at home it is possible and preferred to engage in single child-parent interactions, this is much more complicated to accomplish at school. Although the teacher to student ratio at Maries school allows for many single child-adult interactions, in the observations of Marie engaging in such interactions these tend to be short and interspersed with adult engagement with other children or adults. Additionally, although a few of these interactions include imaginative type exploration proposed by Marie to an adult (Marie imagines what might hatch out of an egg and discusses it with a teacher, Marie creates a clay snake and talks about its features with a teacher), the cricket episode is the only such case of extended imaginative questioning in which Marie is involved at school. There is some interactional evidence (not presented here) that suggest one-on-one is not a preferred form of interaction in the school setting. If Marie only engages in inquiry at home in one-on-one interactions with adults, then she may not initiate inquiry interactions with either her peers or groups of teachers and peers at school where these groups interactions are the preferred arrangements. These differing preferences, or norms, for interaction within home and school may have significant impacts on whether practices familiar within one context will be drawn upon in another context.
Conclusions
Dewey, among others (e.g., Nasir, Rosebery, Warren, & Lee, 2006; Stevens & Toro-Martell, 2003), argues for connecting inquiry experiences across settings in order to guide childrens development. If Dewey is right, we must consider the unique affordances of home, school, and other settings in which children have opportunities for everyday inquiry with regard to how educators (e.g. parents, teachers, museum designers) can best connect the types of everyday experiences described in the above analysis with future experiences. In particular we should consider how children orchestrate support for inquiry, and the role of adults in shaping naturally occurring everyday inquiry, the available material resources, and purposefully designed learning opportunities. We see Marie drawing adults (Mom, Dad, Stacey) into interactions about real-world phenomena that are problematic and interesting to her. Adults then impact the development of Maries observations of scientific phenomena as well as her epistemological theory of how to go about asking questions by shaping the way in
ISLS 320
ICLS2012
which those interactions unfold (Cole, 2007; Goodwin, 2007). In some ways Mom and Dad have already met Deweys insistence of providing future opportunities for the development of inquiry through repeated engagement in their shared inquiry practice. It is possible such a sustained series of experiences with inquiry, particularly if it is on one topic, would support the development of particularly rich science knowledge domains as Crowley and Jacobs have observed (2002). This possibility is supported by Crowley and Jacobs assertion that knowledge is collaboratively constructed within joint-engagement in examining phenomena of interest in a close child-parent relationship. If this is the case, then the analysis of Maries experiences at home may be the first step in connecting everyday inquiry and knowledge with the more highly developed inquiry and knowledge found in Crowley and Jacobs islands of expertise. This may happen over time if Maries questioning of everyday phenomenon is continually supported by Mom and Dad through sustained interactions, provision and exploration of resources (e.g., books, film, internet), or other activities described by Crowley and Jacobs. This sort of sustained and joint-attention activity is one that is particularly well suited to the affordances of an environment in which one-on-one interactions are preferred, as is the case in Maries home. Within the classroom, attention to connecting the cricket questioning to other opportunities may end in a different form of activity. Because of the affordances of the classroomthe large number of students in particularpeer discussion and exploration are possible (and may be preferred) forms of interaction. For example, Stacey could encourage all the students to think about why it would be dangerous for crickets to be loose in the classroom. Further interaction could lead to examining sources of information including their own observations of the classroom, experiences with bugs at home, or resources like books and the internet. Regardless of the direction in which the particular conversation flows, given the childrens interest in the crickets and their eagerness to discuss the crickets throughout snack time, by connecting this experience to others designed to foster the childrens inquiry, a community of science could be fostered. In so doing, Marie and the other childrens interest in the crickets could be the base for meeting the explicit goals of science inquiry education in providing opportunities for observing, questioning, seeking out resources, and explaining within a community of science learners (NRC, 1996). Within this paper I identified the way in which Marie draws adults into interactions about phenomena and objects of interest to her. Ive also discussed a particular practice of inquiry in which Marie often engages with her parents. Through examining how Marie draws on similar forms of noticing and questioning in the school context, it seems that Marie is carrying some aspects of this home practice across context boundaries. However, the cricket episode is one of few within the data of Marie at school during which Marie utilizes imaginative hypothetical exploration or otherwise seems to draw on features of her home inquiry practice. Thus, it may be that the differing affordances of common home and school interactional arrangements (particularly the preferences for one-on-one versus many-on-one arrangements) inhibit Marie from drawing on her home inquiry practice in school. If resources like Maries inquiry practice are to be utilized in classrooms for science inquiry learning, further studies examining interactional affordances of settings may shed greater light on the ways in which interactional arrangements can be designed to support childrens use of resources from other settings.
References
Bricker, L. , & Bell, P. (2007). "Um...since I argue for fun, I don't remember what I argue about": Using children's argumentation across social context to inform science instruction. Paper presented at the National Association for Research in Science Teaching. Callanan, M., & Jipson, J. (2001). Explanatory conversations and young children's developing scientific literacy In K. Crowley, C. D. Schunn & T. Okada (Eds.), Designing for science: Implications from everyday, classroom, and professional settings. Mahwah, NJ: Lawrence Erlbaum Associations. Callanan, M. A., & Oakes, L. M. (1992). Preschoolers' questions and parents' explanations: Causal thinking in everyday activity. Cognitive Development, 7, 213-233. Cole, M. (2007). Phylogeny and cultural history in ontogeny. Journal of Physiology - Paris, 101, 236-246. Crowley, K., Callanan, M., Jipson, J., Galco, J., Topping, K., & Shrager, J. (2001). Shared scientific thinking in everyday parent-child activity. Science Education, 85, 712-732. Crowley, K., Callanan, M., Tenenbaum, H., & Allen, E. (2001). Parents explain more often to boys than to girls during shared scientific thinking. Psychological Science, 12(3), 258-261. Crowley, K., & Galco, J. (2001). Everyday activity and the development of scientific thinking. In K. Crowley, C. Schunn & T. Okada (Eds.), Designing for science: Implications from everyday, classroom, and professional settings. Mahwah, NJ: Lawrence Erlbaum. Crowley, K., & Jacobs, M. (2002). Building islands of expertise in everyday family activity. In G. Leinhardt, K. Crowley & K. Knutson (Eds.), Learning conversations in museums. Mahwah, NJ: Lawrence Erlbaum. Dewey, J. (1981). The philosophy of John Dewey (Vol. 1 and 2). Chicago, IL: University of Chicago Press. Erickson, F., & Schultz, J. (1997). When is a context? Some issues and methods in the analysis of social competence. In M. Cole, Y. Engestr & O. Vasquez (Eds.), Mind, culture, and activity. New York, m NY: Cambridge University Press.
ISLS
321
ICLS2012
Fender, J., & Crowley, K. (2007). How parent explanation changes what children learn from everyday scientific thinking. Journal of Applied Developmental Psychology, 28, 189-210. Goodwin, C., & Heritage, J. (1990). Conversation analysis. Annual Review of Anthropology, 19, 283-307. Goodwin, M. (2007). Occasioned knowledge exploration in family interaction. Discourse Society, 18(1), 93-110. Hawkins, J., & Pea, R. (1987). Tools for bridging the cultures of everyday and scientific thinking. Journal of Research in Science Teaching, 24(4), 291-307. Hood, L., & Bloom, L. (1979). What, when, and how about why: A longitudinal study of early expressions of causality. Monographs of the Society for Research in Child Development, 44(6), 1-47. Hudicourt-Barnes, J. (2003). The use of argumentation in Haitian Creole science classrooms. Harvard Educational Review, 73(1), 73-93. Jordan, B., & Henderson, A. (1995). Interaction analysis: Foundations and practice. The Journal of the Learning Sciences, 4(1), 39-103. McDermott, R., Gospondinoff, K., & Aron, J. (1978). Criteria for an ethnographically adequate description of concerted activities and their contexts. Semiotica, 24, 245-275. Mehus, S., Stevens, R., & Grigholm, L. (2010). Interactional arrangements for learning about science in early childhood: A case study across preschool and home contexts. Paper presented at the 9th International Conference of the Learning Sciences, Chicago, IL Nasir, N., Rosebery, A., Warren, B., & Lee, C. (2006). Learning as cultural process: Achieving equity through diversity. In K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences. New York: Cambridge University Press. National Research Council (1996). National science education standards. Washington DC: National Academy Press. Ochs, E. (1979). Transcription as theory. In E. Ochs & B. Shieffelin (Eds.), Developmental Psychology. New York, NY: Academic. Ochs, E., Gonzales, P., & Jacoby, S. (1996). "When I come down I'm in the domain state": Grammar and graphic representation in the interpretive activity of physicists. In E. Ochs, E. Schegloff & S. Thompson (Eds.), Interaction and grammar. Cambridge: Cambridge University Press. Ochs, E., & Taylor, C. (1992). Science at dinner. In C. Kramsch & S. McConnell-Ginet (Eds.), Text and context: Cross-disciplinary perspectives on language study. Lexington, MA: CD Heath. Penney, L., & Stevens, R. (2011). Lines of question-based inquiry among young children. Paper presented at the Jean Piaget Society. Pomerantz, A., & Fehr, B. (1997). Conversation analysis: An approach to the study of social action as sense making practices. In T. van Dijk (Ed.), Discourse as social interaction (Vol. 2). Thousand Oaks, CA: Sage. Stevens, R., & Hall, R. (1998). Disciplined perception: Learning to see technoscience. In M. Lampert & M. Blunk (Eds.), Talking mathematics in school: Studies of teaching and learning. Cambridge: Cambridge University Press. Stevens, R., Satwicz, T., & McCarthy, L. (2008). In-game, in-room, in-world: Reconnecting video game play to the rest of kids' lives. In K. Salen (Ed.), The ecology of games: Connecting youth, games, and learning. Cambridge, MA: MIT Press. Stevens, R., & Toro-Martell, S. (2003). Leaving a trace: Supporting museum visitor interaction and interpretation with digital media annotation systems. The Journal of Museum Education, 28(2), 25-31. Stevens, R., Wineburg, S., Herrenkohl, L., & Bell, P. (2005). Comparative understanding of school subjects: Past, present, and future. Review of Educational Research, 75(2), 125-157. Warren, B., Ballenger, C., Ogonowski, M., Rosebery, A., & Hudicourt-Barnes, J. (2001). Rethinking diversity in learning science: The logic of everyday sense-making. Journal of Research in Science Teaching, 38(5), 529-552. Warren, B., Ogonowski, M., & Pothier, S. (2005). "Everyday" and "scientific": Rethinking dichotomies in modes of thinking in science learning. In R. Nemirovsky, A. Rosebery, J. Solomon & B. Warren (Eds.), Everyday matters in science and mathematics. Mahwah, NJ: Erlbaum. Zimmerman, H., & Bell, P. (2007). Seeing, doing, and describing everyday science: Mapping images of science across school, community, and home boundaries. Paper presented at the National Association for Research in Science Teaching.
Acknowledgments
I would like to acknowledge the efforts of Reed Stevens, Siri Mehus, Lauren Penney, Linda Grigholm and others involved in the initial collection of the Early Learning Across Contexts data. I would also thank to thank Reed Stevens and Lauren Penny for their input on the multitude of revisions of this paper.
ISLS
322
ICLS2012
Using the Idea Manager to Promote Coherent Understanding of Inquiry Investigations

Kevin W. McElhaney, Camillia F. Matuk, David I. Miller, Marcia C. Linn Graduate School of Education, University of California, Berkeley, Berkeley, CA kevin777@berkeley.edu, cmatuk@berkeley.edu, david.miller@berkeley.edu, mclinn@berkeley.edu Abstract: This study examines the ideas students generated during an online inquiry investigation and how students used these ideas to construct scientific explanations about everyday phenomena. We designed and implemented a four-day, technology-enhanced inquiry unit on the chemistry of recycling with 164 high school students. The unit incorporates the Idea Manager, a new inquiry scaffolding technology that helps students record their ideas and construct coherent explanations. We scored the conceptual content of the ideas students recorded, how students organized their ideas, and the coherence of students explanations. Regression models show that students ability to distinguish ideas predicted explanation coherence more strongly than the number of relevant ideas they recorded. Three cases illustrate difficulties some students had using molecular concepts to support their explanations. The findings demonstrate how this new technology can permit closer examination of students learning progress and inform curricular refinements.
Introduction
How do students integrate the range of ideas they hold into coherent scientific understanding while engaged in inquiry investigations? We investigate this question by studying how students collect, manage, and organize their ideas before they generate scientific explanations. We study a four-day inquiry unit titled How Can We Recycle Old Tires? (henceforth Tires) designed using the Web-based Inquiry Science Environment (WISE, Linn, Davis, & Bell, 2004). Tires features the Idea Manager, a powerful, new computer-based scaffolding technology that supports students as they manage ideas they generate from diverse sources and construct scientific explanations or arguments based on those ideas. Log files from the Idea Manager document how students assemble their ideas into coherent understanding as they progress through Tires. Tires addresses difficulties students have with integrating observable and sub-microscopic phenomena. The design of Tires is based on the Knowledge Integration framework and guided by design principles that have emerged from empirical studies (Kali, 2006; Linn & Eylon, 2011). Tires has been iteratively refined through multiple classroom trials. Tires scaffolds scientific inquiry by eliciting students everyday ideas and experiences with observable phenomena, guiding students interactions with molecular visualizations, providing opportunities for students to illustrate and animate their ideas, and prompting students to generate scientific explanations and reflect on their understanding. Tires takes advantage of the Idea Manager to help students bring diverse ideas together into coherent accounts of science. This study examines the nature and trajectories of students ideas (as illustrated by the Idea Manager) and connects the evolution of students ideas to the quality of their scientific explanations. We address two main research questions: (1) How do students integrate chemistry ideas about observable and molecular phenomena to achieve coherent understanding during inquiry investigations? and (2) How can technology-based inquiry scaffolding help students use chemistry concepts at the molecular level to explain observable phenomena?
Rationale and Idea Manager Design

Chemistry students struggle to integrate concepts at the observable and sub-microscopic levels (Johnstone, 1993; Kozma, 2003). Textbooks and typical classroom chemistry instruction add many ideas at the molecular level to students repertoire of ideas but provide students with few opportunities to link their understanding of these molecular concepts to their everyday understanding of the world. As a result, students may gain proficiency at recalling isolated facts at the sub-microscopic scale but fail to make meaning of them. Technology-enhanced inquiry investigations that help students bridge the gap between their everyday observations and ideas in the discipline show promise for helping students link observable and sub-microscopic concepts in chemistry (e.g. Schank & Kozma, 2002; Wu, Krajcik, & Soloway, 2001). Inquiry investigations compel students to integrate ideas having diverse origins such as everyday experiences, classroom instruction, scientific visualizations, experiments, and peer discussions. Research on inquiry instruction seeks effective scaffolds that help students achieve coherent understanding using diverse inquiry activities such as predictions, experiments, explanations, debate, and reflection (e.g. Linn & Eylon, 2011; Quintana et al., 2004). Furthermore, while studies examining inquiry investigations often demonstrate powerful learning outcomes as measured by pretests and posttests, pinpointing precisely how individual inquiry activities contribute to students understanding has proven difficult. While comparison studies can shed light on the value
ISLS 323
ICLS2012
of individual design features (e.g. Davis, 2003), researchers also need subtle embedded assessment methods to detect the contribution of each activity to students progress. We have designed the Idea Manager tools (Figure 1) to support students learning from complex inquiry activities and capture the individual and cumulative contributions of these activities to students learning. The Idea Manager consists of two interconnected tools, the Idea Basket and the Explanation Builder. The Idea Basket is a persistent repository for students ideas during WISE investigations. The Basket reduces demands on students memories and allows them to direct their efforts toward critically examining their ideas. Students add ideas (of up to 150 characters) to the Basket and can view, revise, and sort the Basket contents at any time. Embedded instructions can also prompt students to submit ideas to the Basket at specific points during the investigation. Students specify attributes of each idea entry, such as its source, tags, or flags (e.g. Important or Not Sure). WISE logs the attributes and time stamps of students Basket interactions to reveal detailed information about the evolution of students ideas. The Explanation Builder addresses difficulties students have using evidence to support explanations (McNeill & Krajcik, 2008; Sandoval & Millwood, 2005) by scaffolding the explanation process into several manageable steps. Rather than asking students to generate complex explanations by having to recall diverse pieces of evidence from memory, The Explanation Builder presents students with the Basket contents and prompts students to organize their ideas in response to an inquiry question. Students drag and drop their ideas from the Basket to an appropriate part of the organizing space. Students can then construct an explanation based on their organized evidence. This approach thus scaffolds the process of constructing complex explanations that require diverse pieces of evidence collected over an extended time. WISE logs the identity and coordinates of each idea that students drag to the organizing space. The Idea Manager tools build on the design of other scaffolds that support the management of inquiry evidence and the construction of evidence-based explanations and arguments, such as SenseMaker (Bell & Linn, 2000), ExplanationConstructor (Sandoval & Reiser, 2004), and IdeaKeeper (Quintana & Zhang, 2004, April). The Idea Manager differs in that it is designed to support the Knowledge Integration pattern (Linn & Eylon, 2011), which helps students integrate ideas across multiple scientific contexts. The Idea Basket encourages students to track both their current and new, normative scientific ideas. Meanwhile, the Explanation Builder helps students organize, distinguish, refine, and reflect on their Basket entries. Ultimately, these activities can help students construct more complete and coherent arguments and explanations. Because the Idea Manager captures students ideas at many points during an inquiry unit, it provides a window into how the students understanding evolves over the course of an investigation. In this paper, we analyze how students generated scientific ideas and used these ideas as evidence in scientific explanations in Tires. Using logs of students interactions with the Idea Manager, we examine how students explanations incorporate the conceptual content of the ideas they accumulated during the investigation. The Idea Manager captures sustained reasoning as students assemble their diverse ideas into accounts of chemical processes during several days of classroom instruction. We evaluate students explanations for coherence and completeness.
Figure 1. Screenshots of students views of the Idea Basket (left) and Explanation Builder (right).
Curriculum Design: How Can We Recycle Old Tires?

Tires helps students distinguish (1) the nature of covalent bonds from intermolecular attractions (e.g. van der Waals forces) and (2) the properties of tire rubber from recyclable plastics. Tire rubber is a polymer (natural rubber) strengthened by the addition of cross-linking covalent bonds. These bonds make tires sufficiently strong to be used on cars but also make tire rubber difficult to recycle. Plastics lack these additional covalent bonds; the weakness of the intermolecular forces holding plastic molecules together makes most plastics easy to recycle. By contrasting the atomic structure, bonding arrangements, properties, and recycling methods of tires and plastics, Tires guides students to make sense of chemistry concepts at the atomic and molecular level and to link this understanding to a familiar everyday context. In the four-hour Tires unit, students engage in a wide variety of activities that elicit their everyday ideas, add new normative ideas about chemical bonding and physical properties, and help them distinguish among these ideas, and apply and reflect upon them. Activity 1 introduces students to the environmental strain
ISLS 324
ICLS2012
associated with accumulating piles of unrecyclable used tires. Activity 2 illustrates the distinction between covalent bonds and intermolecular forces using the surface tension of water as an example. In Activities 3 and 4 students explore the properties, bonding, and atomic arrangements of tire rubber and plastics respectively. Students observe the properties of these substances using physical samples and videos, read about the bonding and atomic arrangements of these polymer substances, interact with molecular visualizations developed using Molecular Workbench (Pallant & Tinker, 2004), depicting the responses of these substances to changes in applied force and heat. Students also use a drawing tool to animate how electrons move within the chemical bonds, and use the evidence they gather to construct written explanations that articulate if and how tires can be recycled. In this study, students who completed Activities 1 through 4 were given the opportunity to study an optional activity on recycling ceramic materials (e.g. ionic solids). Students use the Idea Basket throughout Tires to keep track of ideas they think are relevant to recycling tires. At the end of Activities 3 and 4, students use the Explanation Builder to select and sort their ideas in order to respond to the inquiry questions Are tires recyclable? and Can we recycle tires like plastic? The Explanation Builder first guides students to organize their ideas according to whether they support a YES or NO answer to the question (Tables 2-4). Students then construct an explanation in response to the questions based on their organized evidence. These inquiry questions compel students to integrate evidence at both the observable level (physical properties of substances or recycling methods) and the molecular level (atomic arrangements and chemical bonding). Sophisticated explanations distinguish the observable characteristics of tire rubber and plastic based on their molecular characteristics. Tires thus goes beyond typical classroom exercises that require only textbook definitions by engaging students in making functional and evidence-based distinctions between molecular concepts such as covalent bonds and intermolecular attractions.
Methods
Participants
Chemistry students (N=164) in 10 classes with two teachers studied Tires at a public high school in the western United States. Both teachers had at least five years of teaching experience and had previously used WISE in their classrooms. The school has a diverse student population with 66% underrepresented ethnic minorities and 62% eligible for a reduced-price lunch. Students worked in dyads on the Tires unit. A total of 82 workgroups studied the unit for 4 days.
Data sources and scoring

Idea Basket contents. We analyzed the contents of students Idea Baskets at the end of Activities 3 and 4, the points where students used the Explanation Builder to organize ideas in response to the inquiry prompts. We scored the number of ideas (entries) in the Basket and recorded the type of step where each idea was generated (e.g. video, text, prompt, etc.). To link the conceptual content of students ideas to their explanations, we identified eight inquiry concepts about tire rubber and plastics that are critical to answering the inquiry questions about recycling tires. We coded each idea for the presence of these concepts. Four of these concepts represent observable phenomena (e.g. physical properties or recycling methods): (1) tire rubber is strong or resistant to melting, (2) plastics are weak or easily melted, (3) tires can be reused or repurposed rather than melted down, and (4) plastics are recycled by melting. The other four concepts represent molecular phenomena (e.g. atomic arrangements or chemical bonding): (5) tire rubber has cross-linking bonds that connect polymer chains, (6) plastics lack cross-linking bonds between polymer chains, (7) covalent bonds (such as between tire rubber molecules) are strong, and (8) intermolecular attractions (such as between plastic molecules) are weak. Explanation Builder organization. We evaluated how students organized their ideas using the Explanation Builder in Activities 3 and 4. We scored the placement of ideas in the organizing space in two ways. First, to capture whether students found their ideas relevant to the inquiry question, we computed the number of the eight inquiry concepts students placed in the organizing space. Second, we computed the number of these inquiry concepts that were validly organized (e.g. in a way that would support a valid explanation). We judged whether inquiry concepts were validly organized according to whether they supported one of two scientific views. The first view emphasizes the difficulty of recycling tires by melting because of the presence of covalent cross-links. In this case, we judged concepts appearing in the NO region of the organizing space to be validly organized. The second view emphasizes the possibility for tires to be recycled either by reuse or by selectively breaking covalent cross-links (a currently researched process known as devulcanization). In this case, we judged concepts 2, 4, 5, 6, 7, and 8 to be validly organized if they appeared in the YES region of the organizing space and if students also mentioned the possibility of breaking cross-links. We judged concept 3 to be validly organized on either side of the organizing space in Activity 3 because of the distinction some students made between recycling and reuse. Occasionally students placed an idea on the line dividing the two sides of the organizing space. We coded these as if they had been placed on both sides.
ISLS
325
ICLS2012
Embedded inquiry explanations. We scored students inquiry explanations in Activities 3 and 4 in two ways. First, to determine completeness we determined whether students used the organized inquiry concepts in their explanations by coding each explanation for the presence of each inquiry concept. Second, we used a Knowledge Integration rubric to score the coherence of each explanation from 1 to 5. Previous research shows Knowledge Integration rubrics, which reward valid conceptual connections between scientific concepts, to successfully distinguish levels of conceptual sophistication in scientific explanations (Liu, Lee, Hofstetter, & Linn, 2008). The rubric rewards connections between valid statements about physical properties, the strength of bonds or intermolecular attractions, and recycling methods. The number of inquiry concepts present in each explanation and the coherence score were highly consistent (r = 0.76, p < .001). Pretests and posttests. Though our analysis focuses on students use of the Idea Manager and their explanations, we also captured students broad understanding of atomic structure using two pretest/posttest transfer items. The items asked students to sketch the arrangement of atoms in a polymer and an ionic substance. They were transfer items in that students did not sketch atomic arrangements during the Tires unit, nor did most students study the optional activity on ceramics. We scored these sketches from 1 to 5 based on whether they represented the multitude of the atoms and their bonding arrangements. Students in one class took an alternate assessment as part of a different research study instead of the Tires transfer items. Some students did not take the pretest or posttest because of absence.
Analysis
To illustrate the paths students ideas took during inquiry, we traced the inquiry concepts from students Baskets to the Explanation Builder to their eventual articulation in an explanation. In our first analysis we merged the data from Activities 3 and 4 and divided students explanations into three tiers of coherence. Scores of 4 or 5 comprised the high coherence group (N = 24), scores of 3 comprised the moderate coherence group (N = 54), and scores of 1 or 2 comprised the low coherence group (N = 43). We examined differences in students ideas and in their use of the Explanation Builder preceding explanations of varying degrees of coherence. To clarify these findings, we used multiple linear regression to model the quality of students explanations for each activity. We used characteristics of the Basket ideas and organization choices in the Explanation Builder to model the coherence of students explanations. We used as predictors for the regression model (1) the total number of Basket ideas students had generated to that point in the unit, (2) the number of validly organized inquiry concepts, (3) the number of invalidly organized inquiry concepts, and (4) the number of unorganized inquiry concepts. We included the total number of Basket ideas in the model to control for the effort students made toward populating their Baskets. We omitted students who did not complete an explanation from the model.
Results
Fidelity of enactment and knowledge transfer
A researcher was present in the classroom throughout the enactment of Tires. Students engaged with the inquiry activities as intended, and there were no technical problems that would have interfered with data collection or students inquiry progress. Students made significant gains from pretest to posttest on the drawing transfer items even though they did not draw molecular structures during the unit [t(109) = 2.50, p = .014)]. Gains were significant on the polymer item (p = .018), but not on the ionic item (p = .17), consistent with the focus of the unit on the distinctions between two polymer materials, tire rubber and plastic. This gain is unlikely to have resulted from taking the pretest alone (i.e. drawing ideas prior to instruction). This result suggests students were able to transfer their understanding of the multitude and arrangement of atoms in polymer materials from interacting with the molecular visualizations to generating static representations with the drawing tool.
Basket characteristics
Students Basket ideas varied greatly in their content and complexity. In total, students generated 454 unique ideas. Of these, 214 contained at least one of the eight coded inquiry concepts, and 78 contained at least two inquiry concepts. Of these ideas, 119 contained inquiry concepts coded as observable, while 107 contained inquiry concepts coded as molecular (a few contained both observable and molecular concepts). By the end of Activity 4, students Baskets contained an average of 5.99 ideas (SD = 2.99). Although students could submit ideas to the Basket at any point in the unit, students added most of their ideas during three types of steps. Students submitted 24% of their ideas while viewing text-based evidence pages, 13% of their ideas while using the Explanation Builder, and 56% of their ideas after being prompted to submit ideas. The remaining 7% of ideas were added during other types of steps such as reflection prompts, drawing steps, videos of observable properties, or steps containing molecular visualizations. To examine the extent to which students Basket ideas were self-generated (rather than copied as text directly from the source), we coded ideas as verbatim if the entire text of the idea was taken directly from a text-
ISLS
326
ICLS2012
based evidence page. Only 65 of the 454 ideas (submitted by 14 of the 82 student groups) were coded as verbatim, illustrating that the vast majority of students generated ideas themselves.
Tracking inquiry concepts from Baskets to explanations

We illustrate the paths inquiry concepts took from the Idea Basket to the Explanation Builder and ultimately to their explanations. We compared the frequency with which the eight inquiry concepts were present in students Baskets, selected for organization in the Explanation Builder, organized validly in the explanation space, and included in inquiry explanations for high, moderate, and low levels of explanation coherence. Figure 1 presents this comparison separately for observable and molecular inquiry concepts.
Figure 2. Frequency of observable and molecular inquiry concepts present in Idea Baskets, selected for organization, organized validly in the explanation space, and incorporated in inquiry explanations. Highly coherent explainers added a relatively large number of molecular inquiry concepts to the Basket, organized them validly, and incorporated them coherently into explanations. Moderately coherent explainers generated nearly as many total inquiry concepts as highly coherent explainers (t = 0.71, p = .48), but validly organized only half as many molecular concepts as the highly coherent explainers (t = 2.43, p = .02) and incorporated less than one fourth as many molecular concepts into explanations (t = 5.94, p < .001). Overall, moderately coherent explainers relied primarily on observable concepts to support explanations and failed to incorporate molecular concepts into explanations, even when they recognized these concepts as relevant to the inquiry question. Incoherent explainers added nearly as many total inquiry concepts to the basket as highly coherent explainers (t = 1.48, p = .14) but validly organized concepts at a rate about equal to chance. Moreover, they seldom used any inquiry concepts (observable or molecular) to support explanations. Our regression models revealed that the number of validly organized inquiry concepts was a strong and significantly positive predictor of explanation coherence (Table 1). The number of inquiry concepts left unorganized was a strong and significantly negative predictor for the Activity 4 explanations. The remaining variables did not significantly predict explanation coherence. A linear combination of the inquiry concept predictors from both activities (weighted using the number of observations) showed that the total number of Basket inquiry concepts also did not significantly predict explanation coherence (p = .80). Table 1. Multiple linear regression models predicting explanation coherence Learning outcome Activity 3: Are tires recyclable? (N = 74) Activity 4: Can we recycle tires like plastic? (N = 47) * p < .05; *** p < .001 The multiple regression analysis highlights the importance of distinguishing and sorting ideas generated during the inquiry investigation. Neither the total number of Basket ideas nor the total number of
ISLS 327
Predictor Total Basket ideas Validly organized inquiry concepts Invalidly organized inquiry concepts Unorganized inquiry concepts Total Basket ideas Validly organized inquiry concepts Invalidly organized inquiry concepts Unorganized inquiry concepts
B .01 .38 -.31 -.06 .07 .50 .25 -.35
SE B .05 .19 .31 .22 .05 .15 .16 .16
.03 .41* -.13 -.06 .26 .81*** .29 -.60*
ICLS2012
inquiry concepts predicted explanation coherence. Accumulating ideas, even conceptually relevant ideas, is not sufficient for generating coherent explanations. Only when students used the Explanation Builder to validly distinguish among the key inquiry concepts did they proceed to construct coherent explanations. The higher regression coefficients in the Activity 4 model compared to the Activity 3 model likely reflect the additional opportunities students had to refine their understanding by distinguishing tire rubber from plastics in Activity 4.
Students inquiry progress toward explanations: Three cases

To illustrate how students used the Idea Basket and the Explanation Builder leading up to their Activity 4 inquiry explanation, we describe three cases. We contrast progress toward a highly coherent explanation (Table 2), a moderately coherent explanation (Table 3), and an incoherent explanation (Table 4). Highly coherent case. These students submitted two general ideas during the introductory activity, before Tires introduces specific chemistry concepts. Starting with Activity 3, they added sophisticated ideas that directly address recycling methods (Ideas 3 and 4), chemical bonding (Idea 5), and distinguishing the properties of tire rubber and plastic (Idea 6). In the Explanation Builder, they selected only the four specific ideas as relevant to the question and organized them appropriately. The placement of Idea 4 in the center suggests an awareness of devulcanization (though the students might simply have run out of room on the right side). The explanation that follows clearly distinguishes tire rubber from plastic on a molecular basis and, other than mistaking melting for burning, connects the nature of the chemical bonds to observable properties. Table 2. Idea Basket contents, organization space, and a highly coherent Activity 4 inquiry explanation. #
1 2 3 4 5
Idea text
unused tires collect standing water, creating a breeding gound for mosquitos and rodents, which can cause disease. very hard to recycle tires Cut the rubber tires into smaller pieces and make good use of them. chemically break down the molecular structures & reuse them. Tires cannot be recycled like plastics because tires are made up of covalent bonds.Covalent bonds are much stronger,while van der waals bonds are weak Plastic is capable of burning , tires aren't burned so easily
Organization
Explanation
No . Tires cannot be recycled like plastic because they contain a stronger bond, which is covalent. Plastic contains weak van der waals forces and therefore it is much easier to breakdown, it can easily be burned and made into other materials.
Moderately coherent case. Like the previous student group, these students added a general idea early in the project, then added several specific ideas at the sub-microscopic level during Activities 3 and 4. Ideas 3 and 5 reflect an incomplete understanding of the role of electrons in chemical bonding, while Ideas 2 and 4 are sophisticated ideas that are highly relevant to the inquiry question. Rather than selecting the most relevant ideas for organization, these students chose to organize all their ideas in an apparently arbitrary manner, as there is no clear connection between the content of the ideas and how they are organized. The explanation that follows refers only vaguely to the complexity of the bonds and omits the specific connections between bonding concepts and recycling that are present in Ideas 2 and 4. The haphazard organization and vague explanation suggest that these students had only a superficial understanding of the bonding concepts they submitted to their Basket. Table 3. Idea Basket contents, organization space, and a moderately coherent Activity 4 inquiry explanation. #
1 2 3
Idea text
Tires could be burned and the heat can be captured for energy. It is possible to recycle a tire but it is very difficult because of its complexity. A tire has very strong bonds that are not easy to break. What makes recycling tires so difficult is that it's not easy to take the electrons away from the bonds which makes the tire more resistant. It it easier to recycle plastics because there polymers are not cross linked like tire rubber polymers. Tire rubber would be easier to recycle if the electrons surrounding the bonds weren't so attracted to the protons in their nucleus.
Organization
Explanation
We cannot recycle tire rubber like we recycle plastic because the structure of bonds and polymers in tire rubber is much more complex than plastic.
4 5
Incoherent case. Like the previous two student groups, these students began by adding general ideas to the Basket. In Activity 3, these students added seven specific ideas (Ideas 3-9) about the atomic arrangement and bonding of tire rubber. These ideas are, for the most part, copied directly from pages in the unit. In Activity 4, these students neglected to add corresponding molecular ideas about plastic, instead adding only vague or
ISLS
328
ICLS2012
non-normative observations in Ideas 10 and 11. Given the opportunity to sort their ideas using the Explanation Builder, these students ignored their molecular ideas and organized only their vague observations. Thus, these students added many relevant ideas but failed to support their explanation with this evidence about chemical bonding or observable properties. Overall, they neglected to distinguish between tires and plastics at any level. Table 4. Idea Basket contents, organization space, and an incoherent Activity 4 inquiry explanation. #
1 2 7 8 9 10 11
Idea text
unused tires collect standing water, creating a breeding gound for mosquitos and rodents, which can cause disease. very hard to recycle tires Polymer chains in natural rubber are held together loosely by VAN DER WAALS forces. Natural rubber is strengthened using a process called vulcanization. Vulcanization uses sulfur atoms to cross-link polymer chains side-toside with more COVALENT BONDS. tires are different from plastic because when you hit a tire with ahammer it does nthing but when a plastic is hit by a hammer it bends and stays bent they are the same because when you melt them they both melt.
Organization
Explanation
We think that tires cannot be recycled like plastics because they are made out of nonrenewable resources and they are very hard and expensive substances. It requires high level of chemical breakdowns.
Discussion
The Idea Manager helps students collect, organize, and distinguish their ideas and allows researchers to observe how chemistry students grapple with molecular concepts to explain everyday science. Our findings show that most students add some relevant ideas while studying Tires. Students use of the Explanation Builder revealed that having relevant ideas did not ensure that they would generate coherent explanations. Thus, adding relevant ideas (often the main goal of typical science instruction) is not sufficient to ensure that students can use the ideas productivelyinstruction must provide students with opportunities to distinguish and sort their ideas. Indeed, many students generated incoherent explanations despite having identified a plethora of ideas relevant to the investigation. This study thus provides evidence to support the Knowledge Integration perspective, which describes learning as the process of connecting, distinguishing, sorting, and refining ideas rather than merely the accumulation of normative ideas. Most students in our study submitted Basket ideas that would have been appropriate answers to typical science assessments, but the total number of ideas was not a significant predictor of explanation coherence. Our analysis indicates that students need guidance to integrate their ideas into generative explanations. These findings resonate with our previous studies of a physics inquiry unit where students conducted virtual experiments to test their ideas about car collisions (McElhaney & Linn, 2011). In those studies, the validity of students experimental designs more strongly predicted students learning outcomes than the number of experimental trials they conducted. Just as valid experimental designs illustrate meaningful distinctions between relevant concepts, the valid organizing of chemistry ideas in the Explanation Builder illustrates the conceptual distinctions that are necessary to construct coherent explanations. In both studies, the quantity of evidence students produced was not sufficientthey also needed guidance to organize and distinguish ideas. Our findings also suggest that, consistent with previous research (e.g. Ben-Zvi, Eylon, & Silberstein, 1986; Kozma, 2003), students had particular difficulty using molecular concepts to support their explanations. The detailed accounts of learning provided by the Idea Manager suggest curriculum revisions to Tires that could help students better incorporate molecular views into their understanding. We designed the molecular visualizations to illustrate the relationship between physical properties and the molecular structure of substances. Our data logs indicate that less than one percent of students Basket ideas were generated while interacting with a molecular visualization, compared to 24% that were submitted while reading text-based descriptions of molecular concepts. Students molecular ideas might have been more productive had they been generated as observations of the visualizations rather than as summaries of text-based evidence. In a recent study on a WISE unit about the seasons, students submitted about 25% of their Basket ideas while interacting with dynamic visualizations (Matuk et al., 2012). The Seasons unit contains relatively little text-based evidence and encourages students to generate ideas while exploring the visualizations. The success of this unit suggests that a similar design approach with Tires might help students generate productive, molecular ideas that emerge from their understanding of the physical properties of substances.
Conclusions, Implications, and Next Steps

This paper illustrates how new technologies can guide students inquiry learning and help researchers closely examine the progress of students learning during inquiry investigations. Logs of students use of the Idea
ISLS 329
ICLS2012
Manager reveal the varied inquiry trajectories students follow. Tires helped some students achieve sophisticated insights about the chemistry of recycling. Other students accumulated productive ideas but were unable to integrate these ideas to reach deeper insights. These rich sources of data will be crucial to understanding the sequence of learning events that lead to students meaningful understanding of complex science. Furthermore, these data can inform curricular revisions that strengthen connections between disciplinary concepts and everyday science. Our subsequent studies will explore the causal relationships between students use of the Idea Manager and their learning outcomes. We will also investigate how inquiry scaffolding can encourage students to add more productive ideas and fewer inert ideas to their Baskets, as well as guide students to distinguish ideas in ways that promote more coherent scientific understanding.
Acknowledgements
We thank three anonymous reviewers for their comments on this manuscript and the classroom teachers who took part in this study. We acknowledge the support of National Science Foundation Grants DRL-0918743, DRL-1119670, and DRL-0822388 in conducting this research.
References
Bell, P., & Linn, M. (2000). Scientific arguments as learning artifacts: designing for learning from the web with KIE. International Journal of Science Education, 22(8), 797-817. Ben-Zvi, R., Eylon, B., & Silberstein, J. (1986). Is an atom of copper malleable? Journal of Chemical Education, 63, 64-66. Davis, E. (2003). Prompting middle school science students for productive reflection: Generic and directed prompts. Journal of the Learning Sciences, 12(1), 91-142. Johnstone, A. H. (1993). The development of chemistry teaching. Journal of Chemical Education, 70, 701-705. Kali, Y. (2006). Collaborative knowledge building using the Design Principles Database. International Journal of Computer-Supported Collaborative Learning, 1(2), 187-201. Kozma, R. (2003). The material features of multiple representations and their cognitive and social affordances for science understanding. Learning and Instruction, 13(2), 205-226. Linn, M., Davis, E., & Bell, P. (2004). Internet Environments for Science Education. Mahwah, NJ: Lawrence Erlbaum Associates. Linn, M., & Eylon, B. (2011). Science Learning and Instruction: Taking Advantage of Technology to Promote Knowledge Integration. New York: Routledge. Liu, O., Lee, H., Hofstetter, C., & Linn, M. (2008). Assessing knowledge integration in science: Construct, measures, and evidence. Educational Assessment, 13(1), 33-55. Matuk, C.F., McElhaney, K.M., King Chen, J., Miller, D.I., Lim-Breitbart, J., & Linn, M.C. (2012). The Idea Manager: A tool to scaffold students documenting, sorting, and distinguishing ideas in science inquiry. In The Future of Learning: Proceedings of the 10th International Conference of the Learning Sciences. International Society of the Learning Sciences: Sydney, Australia. McElhaney, K.W. & Linn, M.C. (2011). Investigations of a Complex, Realistic Task: Intentional, Unsystematic, and Exhaustive Experimenters. Journal of Research in Science Teaching, 48(7), 745-770. McNeill, K., & Krajcik, J. (2008). Scientific Explanations: Characterizing and Evaluating the Effects of Teachers Instructional Practices on Student Learning. Journal of Research in Science Teaching, 45(1), 53-78. Pallant, A., & Tinker, R. (2004). Reasoning with atomic-scale molecular dynamic models. Journal of Science Education and Technology, 13(1), 51-66. Quintana, C., Reiser, B., Davis, E., Krajcik, J., Fretz, E., Duncan, R., et al. (2004). A scaffolding design framework for software to support science inquiry. Scaffolding: A Special Issue of the Journal of the Learning Sciences, 13(3), 337-386. Quintana, C., & Zhang, M. (2004, April). The Digital IdeaKeeper: Extending Digital Library Services to Scaffold Online Inquiry. Paper presented at the American Education Research Association Annual Meeting, San Diego, CA. Sandoval, W., & Millwood, K. (2005). The Quality of Students Use of Evidence in Written Scientific Explanations. Cognition and Instruction, 23(1), 23-55. Sandoval, W., & Reiser, B. (2004). Explanation-driven inquiry: Integrating conceptual and epistemic scaffolds for scientific inquiry. Science Education, 88(3), 345-372. Schank, P., & Kozma, R. (2002). Learning chemistry through the use of a representation-based knowledge building environment. Journal of Computers in Mathematics and Science Teaching, 21(3), 253-280. Wu, H., Krajcik, J., & Soloway, E. (2001). Promoting conceptual understanding of chemical representations: Students use of a visualization tool in the classroom. Journal of Research in Science Teaching, 38(7), 821-842.
ISLS
330
ICLS2012
Using Heuristic Worked Examples and Collaboration Scripts to Help Learners Acquire Mathematical Argumentation Skills
Ingo Kollar1, Stefan Ufer1, Elisabeth Lorenz2, Freydis Vogel1, Kristina Reiss2, Frank Fischer1 1 Ludwig-Maximilians University of Munich, Leopoldstr. 13, D-80802 Munich, Germany 2 Technical University of Munich, Schellingstr. 33, D-80799 Munich, Germany ingo.kollar@psy.lmu.de, ufer@math.lmu.de, e.lorenz@tum.de, freydis.vogel@psy.lmu.de kristina.reiss@tum.de, frank.fischer@psy.lmu.de Abstract: Mathematical argumentation skills include an individual-cognitive component, which refers to the individual ability to generate and evaluate arguments on a mathematical conjecture, and a social-discursive component describing the skills necessary to develop and defend arguments on how to solve mathematical problems in collaborative situations. In an experimental 2x2 design (N = 119), we investigated whether both components can be fostered through heuristic worked examples and collaboration scripts which were implemented in a CSCL environment. Results indicate that the individual-cognitive component was facilitated by the provision of heuristic worked-out examples, whereas the social-discursive component was fostered through the use of collaboration scripts and of heuristic worked examples. Both scaffolds were especially helpful for learners with higher general ability. It appears that scaffolds on a content level as well as scaffolds on a social level are necessary to help learners attain high levels of mathematical argumentation skills.
Introduction
The ability to construct mathematical arguments for and against claims and to generate or inquire mathematical conjectures has shifted into the focus of mathematics curricula worldwide during the last decade (e.g., NCTM, 2000). Thus, corresponding skills of inquiring, evaluating, and adapting mathematical conjectures posed for example by students make up a relevant part of mathematics teachers professional skills (Krauss, Baumert, & Blum, 2008). Students, teacher students, and also teachers problems in generating mathematical arguments have been documented in several studies (e.g., Heinze, Reiss, & Rudolph, 2005). This contribution conceptualizes mathematical argumentation skills as the ability to inquire mathematical conjectures individually or in discursive resp. collaborative social contexts, finally arriving at a proof or refutation for the conjecture (e.g. Lin, 2005). Thus, we differentiate between an individual-cognitive and a social-discursive component of mathematical argumentation skill. The individual-cognitive component refers to the individual ability to generate adequate arguments for or against a mathematical conjecture, and finally combine these arguments to a mathematical proof. The social-discursive component refers to the ability to participate in and contribute to a constructive collaborative argumentation process in a small group. While the first component can be regarded as mostly genuine to mathematics, the second component is less specific to mathematical argumentation and might root in more general argumentation skills (Kollar, Fischer, & Slotta, 2007). In this paper, we investigate whether the two components can be facilitated by heuristic worked-out examples (as a content-related scaffold; Reiss & Renkl, 2002) and collaboration scripts (as a social scaffold; Kollar, Fischer & Hesse, 2006). We are specifically interested in whether a combination of the two scaffolds leads to more positive results than any of the two alone.
Collaboration scripts and the social-discursive component

The social-discursive component of mathematical argumentation skills is necessary to communicate adequate procedures and solutions to mathematical problems to other learners or to reach joint solutions that are based on collaboration between learning partners. Yet, research has produced a wealth of evidence demonstrating that collaborative learning is often less effective than individual learning as long as it is not structured appropriately (e.g., Mullins, Rummel & Spada, 2011). If such an external structure is lacking, collaborators tend to show lowlevel argumentation (e.g., Kollar et al., 2007), seldom relate their contributions to the contributions of their learning partners (e.g., Weinberger, Ertl, Fischer & Mandl, 2005) and often have problems achieving common ground concerning the problem at hand (e.g., Hansen & Spada, 2010). One way to structure collaborative learning is to provide learners with collaboration scripts that specify and distribute learning activities and collaboration roles among the learners of a small group and bring them into a certain sequence (Kollar et al., 2006). These scripts can be tailored to induce very specific collaboration processes that stand in a positive relation with individual learning outcomes. The classical MURDER script (ODonnell & Dansereau, 1992), for example, distributes learning activities and roles among the partners in a dyad that is supposed to learn text content and acquire text learning strategies. Numerous empirical studies have demonstrated the effectiveness of the MURDER script both compared to individual learning and to unstructured collaborative learning with
ISLS
331
ICLS2012
respect to different learning outcomes such as retention of the text, the acquisition of procedural knowledge, and strategy transfer to a new individual learning situation (for an overview, see ODonnell, 1999). Lately, the collaboration script approach has been transferred to computer-supported collaborative learning (CSCL; e.g., De Wever, van Keer, Schellens, & Valcke, 2010; Pozzi, 2011), and a number of CSCL scripts have been designed to facilitate collaborative argumentation. In one such study, Stegmann, Weinberger and Fischer (2007) used two collaboration scripts aiming at an improvement of collaborative argumentation in a case-based, asynchronous discussion board environment in which triads of learners were supposed to apply a psychological theory to solve authentic problem cases. The first script, the script for the construction of single arguments prompted learners to use data, grounds and warrants (Toulmin, 1958) while building arguments, whereas the second script, the script for the construction of argument sequences distributed among the learners the task to produce arguments, counterarguments and integrations (Leitao, 2000). The results revealed that the script for the construction of single arguments had positive effects on the frequency of complete arguments that student triads produced during collaboration, while the script for the construction of argumentation sequences had positive effects on the frequency of counterarguments and integrations in the collaborative process. In addition, both scripts fostered the acquisition of domain-general knowledge on argumentation, and their effects on this measure added up. However, none of the scripts was effective concerning the acquisition of domain-specific knowledge. Similar results were found by Kollar et al. (2007). However, a few studies have also found positive effects of collaboration scripts on domain-specific learning outcomes (e.g., knowledge about the content of discussions; e.g., Wecker, Kollar, Fischer & Prechtl, 2010; Weinberger et al., 2005). Yet, positive effects of collaboration scripts on domain-specific learning are much rarer than positive effects on domain-general knowledge. This finding motivates to search for further instructional interventions that may be used in combination with collaboration scripts to account for their weaknesses. Since collaboration scripts are interventions that are specifically designed to help with the interactive processes occurring between learners of a small group, but do not provide structure with respect to the domain-specific aspects of the task, a combination with more content-related scaffolds may be promising.
Heuristic worked examples and the individual-cognitive component

Mastering the domain-specific, individual-cognitive demands of mathematical argumentation requires coordinating knowledge on mathematical concepts, knowledge on the acceptable mathematical arguments, and problem solving processes to generate a line of argument (Ufer, Heinze, & Reiss, 2008). These problem solving processes may rely on several heuristic strategies like considering single examples and a systematization of examples, using informal arguments based for example on drawings, or more general strategies like means-endanalysis (e.g. Chinnappan & Lawson, 1996). The information provided by considering examples, informal proof ideas, general mathematical rules and concepts, possible arguments as well as acceptance criteria for mathematical arguments must be integrated within this problem solving process. Thus, mathematical argumentation tasks can be considered as complex problems. Worked examples are considered effective means of fostering skills to solve this kind of complex problems, since they free the learner from the working memory load necessary to solve the learning task. This is supposed to enhance working memory capacity available for learning processes. The effectiveness of studying worked examples as compared to solving the corresponding problems as a learning activity has been shown in numerous studies (e.g. Atkinson, Derry, Renkl, Wortham, 2001; van Gog & Rummel, 2010). Most research on worked examples has focused on well-structured domains and task types that can be solved by one specific strategy (Rourke & Sweller, 2009). Worked examples for more complex task types, which require the selection of appropriate strategies and principles in a heuristic manner, have been developed and studied recently (e.g., van Gog, Paas & van Merrienboer, 2008). Schworm and Renkl (2007) proposed that worked examples for such complex tasks address three content levels: the learning domain level (information on what should be learned), the exemplifying domain level (information on the context in which learning tasks are embedded), and the strategy level (information on how problems can be approached, e.g. through an expert). Based on this suggestion, Reiss and Renkl (2002) introduced heuristic worked examples, i.e. examples that include principled information on all three levels. Following Schworm and Renkl (2007), learning should be focused on one of the three levels, these examples focus learning on the strategy level. Heuristic worked examples targeting mathematical argumentation and proof skills that were based on an experts process model of mathematics proof ((1) generate a conjecture, (2) formulate a statement, (3) exploration of the statement, (4) proof idea, (5) proof draft, (6) proof formulation; adapted from Boero, 1999) have been studied by Reiss, Heinze, Kessler, RudolphAlbert, and Renkl (2007) and were found to be superior to ordinary instruction in an authentic school setting. One gap in research on (heuristic) worked examples, which is particularly important in view of skills that share a social-discursive component, is their frequent restriction to single-learner settings (Kirschner, Paas, Kirschner, & Janssen, 2011). It is plausible that social-discursive components of complex, interactive skills can hardly be obtained in single-learner settings. Kirschner, et al. (2011) compared worked examples and problem solving in individual and collaborative settings. In the problem solving condition, group learners showed better
ISLS
332
ICLS2012
post-test performance than individual learners, but there was no difference in the worked example condition. The authors argue that in collaborative learning, memory resources of both learners are joined to process relevant information on the problem solving process, making it unnecessary to reduce cognitive load by using worked examples. This raises the question, if unused resources could be activated by the provision of an additional scaffold, either to enhance learning from the worked example or to provide additional learning opportunities with respect to social-discursive components of mathematical argumentation skills.
The present study

Our main research question was: What are the effects of collaboration scripts and learning mode (problem solving vs. heuristic worked examples) on mathematical argumentation skills resp. their individual-cognitive and social-discursive components in a CSCL environment? In the CSCL environment utilized for this study, student dyads worked on mathematical proof and argumentation problems. Dyads were either supported by a set of heuristic worked examples, a collaboration script, both, or none of the two. Our expectations were: 1. When compared to unstructured collaboration, providing learners with collaboration scripts will foster the social-discursive component of learners mathematical argumentation skills. 2. Since heuristic worked examples may trigger high-level argumentation about the contents of the examples, we also expected a positive effect of the heuristic worked examples on the social-discursive component of mathematical argumentation skills. 3. When compared to problem solving, providing learners with a set of heuristic worked examples will foster the individual-cognitive component of learners mathematical argumentation skills. 4. Since collaboration scripts may stimulate a more thorough elaboration of learning material (see Wecker et al., 2011; Weinberger et al., 2005), we also expected a positive effect of collaboration scripts on the individual-cognitive component of mathematical argumentation skills. 5. For both (a) the social-discursive and the (b) individual-cognitive component of mathematical argumentation skills, learners equipped with both the collaboration script and the heuristic worked examples were expected to outperform learners from all other conditions.
Method
Participants and design
119 pre-service mathematics teacher students (66 female, 53 male; MAge = 20.00, SD = 2.25) from two different German universities participated as a part of a voluntary two-week preparatory course for university mathematics programs. After a pre-test, participants were divided into two groups using the median-split of their final school qualification grades. The best qualification grade reported by the participants was 1.00, the worst was 3.50 (M = 2.12, SD = 0.56). Qualification grade was used as an indicator of general ability in this study. Participants in each ability group were then randomized to the four learning conditions. The two factors were (1) learning mode (problem solving vs. heuristic worked example) and (2) collaboration script (without vs. with). Table 1: Overview over the experimental design. Collaboration script Without With N = 29 N = 29 N = 29 N = 32
Learning mode
Problem solving Heuristic worked example
All dyads either consisted of two learners with low general ability, or of two learners with high ability (no mixed dyads). New learning dyads were created for each session. Students who missed more than one session were excluded from our analysis. Thus, the number of participants in the cells can be odd.
Instructional setting and procedure

Prior to the experiment, students participated in courses about elementary number theory and other math content that comprised the first three days of the preparatory course. On the fourth day, the pre-tests took place. The intervention started on the fifth day. Students first received a short video explaining the use of the CSCL environment. After that, the first of three 45-minute treatment sessions started. The other two treatment sessions took place on day 6 and 7. For each treatment session, the students were assigned to a new learning dyad. On the eighth and ninth day of the course, the students completed the post-tests. The final, tenth day of the course consisted of lectures and seminars that were not part of the experiment. Lectures and seminars also proceeded during the days on which the experiment was conducted, but they dealt with content that was distal to that of the
ISLS
333
ICLS2012
intervention (e.g. mathematical mappings, limits, continuity of functions). The lecturer of the course and the teaching assistants were asked to give no instruction on heuristics of proof and collaborative learning strategies.
Materials and learning environment

Collaboration took place face-to-face, but was supported by a CSCL environment (Fig. 1). Each student received a laptop equipped with a mouse and a graphic tablet that enabled writing and drawing on the screen.
Fig. 1: Screenshot of the computer program (left side of the screen: private work space including the problem to be solved resp. the heuristic worked example; right side: shared work space displaying script prompts or not). The students of each dyad were placed vis--vis each other, so that each student could only see his or her own screen. The left side of the screen displayed either a problem to be solved by the students or a heuristic worked example that explained how a fictitious peer solved the problem. At the bottom left, students had access to a calculator and the lecture notes of the course. The left side of the screen was a private space, i.e. any changes or actions students undertook were only visible to them. The right part of the screen was a shared work space, i.e. when students typed text or drew or wrote on their graphic tablets, this was immediately visible on their partners screen. Depending on experimental condition, the right side of the screen included a range of script prompts or not (see below). All students, independently from the experimental condition, received instructions that implemented a common basic structure of the learning process for all four groups, which was visible in the top right section of the screen. According to this structure, students were first advised to think about the problem respectively the heuristic worked example individually and press the Finished button once they were done. When the learning partner also had completed this step, one of them was requested to explain his ideas about the problem resp. the worked example. Afterwards, the learners discussed their ideas on the solution of the given problem resp. on the approach that was described in the two heuristic worked examples.
Independent variables
Two independent variables were realized: Collaboration scripts (with vs. without) and learning mode (problem solving vs. heuristic worked examples). In the collaboration script condition (see Fig. 1), the right side of the screen (the shared workspace) was pre-structured in a way that the students of each dyad received pre-specified prompts that guided them through a three-step sequence of argumentation, based on Leitao (2000). First, one learner was asked to provide an argument for his or her suggestion how to solve the problem resp. a proargument for the way the character in the heuristic worked example solved the problem. Then, learner B was asked to produce a counterargument, and finally, both partners were asked to reach a consensus. To further support the argumentation process, students could click on a button labeled argumentation to find guidelines on how to produce arguments, which told them to produce arguments that include at least a claim, a warrant, and a qualification (Toulmin, 1958). This button was present in all conditions. Beyond that, participants learning without the collaboration script did not receive any prompts during the treatment session. In the learning environment, the students worked on tasks that posed open mathematical situations (e.g. Take five consecutive numbers and add them up. Repeat this and try to find regularities. Formulate a conjecture and prove it!). Depending on the variable learning mode, the learning materials implemented on the left side of the computer screen induced different activities concerning these tasks (see Fig. 1). The students either discussed a heuristic worked example, or they read the corresponding problem formulation and were requested to find a solution with their learning partner. Three different problems from elementary number theory
ISLS
334
ICLS2012
were presented. The heuristic worked examples included a fictitious peer, who found and proved a conjecture according to the six phases of the adapted process model by Boero (1999). A set of self-explanation prompts asked the students in each step of the heuristic worked examples, why the protagonist of the heuristic worked example had chosen this approach. Students were prompted in every second step of the worked example to discuss their thoughts with their learning partner. To trigger discussion, the two learning partners worked on two partly different examples, although the problem was the same. In the problem solving condition, students only received the problem formulation without any hints and were asked to solve the problem in their dyads.
Dependent variables and instruments

The social-discursive component of mathematical argumentation skills was measured by a post-test that asked participants to describe activities they would typically show in a discussion (e.g., make sure to reach consensus once opposing view points have been expressed) on a physics problem (i.e., if water should be put hot or cold into the fridge to freeze faster). Participants were asked the same question in the pre-test. Answers were analyzed regarding the naming of elements that were part of the script relating to the sequencing of a discussion and the structure of a good argument (pro-argumentation, counter-argumentation, consensus building, response to arguments, claiming, grounding, use of rebuttals). Two trained, independent raters coded ca. 12 % of all coded answers from pre- and post-test. Inter-rater reliability (Cohens Kappa) for each single code was acceptable (on average = .65). To calculate a value for the social-discursive component of mathematical argumentation skill, the frequency of the relevant seven elements was summed up and divided by the number of phases the student had described. Thus, the value for the social-discursive component indicates how many different relevant elements the learners named per each described phase on average. Students individual cognitive component of mathematical argumentation skills was measured with seventeen items in pre- and post-test, with different items for pre- and post-test. Five items focused on argumentation with elementary rules from number-theory (e.g., Show that for natural numbers, a and b, the following statement is true: If 7 divides (a+3b) then 7 divides (2a+13b).). Six items surveyed students proving skills in elementary number theory (e.g., Prove the following statement: The sum of five consecutive numbers is divisible by five.). Another six items required students to solve open-ended argumentation problems (e.g., Prove or refute the following statement for natural numbers a and b: If you multiply the sum of a and b with the difference of a and b, you will always obtain an even number.). A three-level coding (see also Heinze, Reiss & Rudolph, 2005), was applied to score students answers. One score was given for partially correct solutions; an extra score was given if the item was solved correctly. Additionally, the posttest assessed knowledge of heuristic strategies for the mathematical argumentation process (If you have to explore and prove a mathematical statement, how would you proceed?). Answers were scored based on the strategies that were implemented in the heuristic worked examples. Each correct strategy was awarded one point. Two independent raters coded all the items. Inter rater reliability values were good (Mean of ICCunjust =.83). The reliability of the instrument was found to be good for each point of measurement (Cronbachs alpha in the pre-test: = .82; post-test: = .78).
Statistical analyses
Univariate and multivariate analyses of covariance as well as regression analyses were used to test the hypotheses. The significance level was set to = .05. As an effect size measure, we used partial .
Results
Social-discursive component of mathematical argumentation skills
To test hypotheses 1 and 2, an ANCOVA with collaboration script and learning mode as independent factors, post-test social discursive argumentation knowledge as dependent variable and general ability as well as pretest argumentation knowledge as covariates was calculated. As single regressions of the covariate general ability on the learning gain (indicated by the residua of the regression of post-test measures on pre-test measures) of social-discursive argumentation knowledge showed differences in the regression slopes between experimental groups, the interaction effects between general ability and the independent factors were included in the model. The ANCOVA revealed a positive effect of the collaboration script on the post-test for social discursive argumentation knowledge, F(1,94) = 8.95, p < .01, part. = .09. Also, the learning mode had a positive effect, with learners who learned with heuristic worked examples reaching higher values in the post-test than learners who learned with problem solving, F(1,94) = 4.38, p < .05, part. = .05 (see table 2). The interaction effect between collaboration script and learning mode was not significant, F(1,94) = 0.59, n.s.. The interaction between general ability and the collaboration script was significant, F(1,94) = 7.59, p < .01, part. = .08. A comparison of the regression slopes for the two conditions with and without collaboration script revealed that in the scripted condition, general ability had a significantly positive correlation with the learning gain (stand. = -0.52, p < .001), while in the unscripted group, general ability did
ISLS
335
ICLS2012
not correlate with learning gain (stand. = 0.03, n.s.). I.e. while high-ability learners benefited from the script, this was not true for low-ability learners (note that high values represent low ability and vice versa). Regarding hypothesis 5a, a planned contrast comparing the experimental condition supported by the collaboration script and the heuristic worked example with the remaining three conditions revealed that students who learned in the combined condition significantly outperformed students who learned in any of the other three conditions, with respect to the social-discursive component, F(1,95) = 5.94, p < .05, part. = .02. Table 2: Mean pre- and post-test scores (standard deviations indicated in brackets) for the social-discursive component of mathematical argumentation skills in the four experimental conditions. Problem solving without collab. script with collab. script 0.61 (0.54) 0.64 (0.51) 0.68 (0.47) 0.72 (0.43) Heuristic worked example without collab. script with collab. script 0.69 (0.42) 0.58 (0.40) 0.81 (0.48) 0.94 (0.44)
Pre-test Post-test
Individual-cognitive component of mathematical argumentation skills

To test hypotheses 3 and 4, a 2 x 2 ANCOVA with collaboration script and learning mode as independent variables, post test performance in the test for the individual cognitive component of mathematical argumentation skills as dependent variable and the analogical scores in the pretest as well as general ability as covariates was computed. Because of the same reason mentioned above, the interaction effects between general ability and the independent factors were included in the model. Results revealed a significant main effect of the learning mode on the performance in the post-test, F(1,111) = 10.00, p < .01, part. = .08, favoring students from the heuristic worked examples condition over the students who engaged in problem solving (see table 3 for raw data). Neither the main effect for collaboration scripts nor the interaction effect between collaboration scripts and heuristic worked examples reached statistical significance, F(1,111) < 1, n.s.. Table 3: Mean pre- and post-test scores (standard deviations in brackets) for the individual-cognitive component of mathematical argumentation skills in the four experimental conditions. Problem solving without collab. script with collab. script 0.46 (0.25) 0.41 (0.21) 0.49 (0.17) 0.47 (0.18) Heuristic worked example without collab. script with collab. script 0.39 (0.17) 0.44 (0.19) 0.48 (0.19) 0.50 (0.15)
Pre-test Post-test
There was a significant interaction effect between learning mode and general ability, F(1,111) = 8.98, p < .01, = .08. Regression analyses contrasting the conditions with both kinds of learning mode showed that general ability had differential impact in the two conditions problem solving and heuristic worked example. Students with higher general ability learned better with heuristic worked examples (stand. = -0.45, p < .001) while in the problem solving condition, general ability showed no effect (stand. = 0.12, n.s.). Regarding hypothesis 5b, a planned contrast revealed that, counter to our expectations, students from the combined condition did not outperform students from the other three conditions, F(1, 113) = 0.10, n.s..
Discussion
This paper investigated whether combining collaboration scripts and heuristic worked examples is an effective way to facilitate students acquisition of mathematical argumentation skills in a CSCL environment. Mathematical argumentation skills were argued to include both a social-discursive component, which refers to the ability to participate in and contribute to a constructive collaborative argumentation process in a small group, and an individual-cognitive component necessary to generate arguments for or against a mathematical conjecture, to evaluate these arguments according to mathematical criteria, and to select and combine them for a mathematical proof or refutation. We expected the collaboration script and the heuristic worked examples to exert positive effects on both components. Moreover, we expected that a combination of collaboration scripts and heuristic worked example would lead to most positive effects with respect to both dependent measures. Our results largely confirm our expectations. First, the collaboration script indeed facilitated students acquisition of social-discursive aspects of mathematical argumentation skills. Thus, also for the mathematical domain, collaboration scripts can be designed that help students acquire argumentation skills, which adds to previous research from other domains (e.g., Kollar et al, 2007; Stegmann et al., 2008). Obviously, having students practice a strategy that is represented in a collaboration script leads to an at least partial internalization of this strategy, represented in higher learning gains by students who worked with vs. without the script. Second, the heuristic worked examples indeed supported the students acquisition of individual-cognitive aspects. In
ISLS
336
ICLS2012
contrast to the results of Kirschner et al. (2011), who found no positive effect of collaborative study of worked examples over collaborative problem solving, we found a supporting effect of heuristic worked examples in collaborative settings. This might be due to the more complex nature of the problems studied, which require the coordination of a rich repertoire of strategies. Third, as expected, heuristic worked examples had an additional positive effect on the social-discursive component, i.e. when compared to problem solving, they helped learners acquire knowledge about more argumentation processes. A possible interpretation of this result is that the heuristic worked examples gave learners more opportunities to argue about different approaches to solve the mathematical problem or to reflect on their argumentative discourse. When students try to solve such problems on their own, it may happen that the two learners converge on one specific solution path quickly, which may make further argumentation subjectively unnecessary. In contrast, having learners discuss two different heuristic worked examples that describe different solutions may trigger more profound argumentation processes. However, two of our hypotheses did not receive full empirical support. First, collaboration scripts did not exert a positive effect on the acquisition of individual-cognitive aspects of mathematical argumentation skills. This finding is surprising since it is often argued that high-level argumentation is related to learning about the content of argumentation (e.g., Andriessen, Baker & Suthers, 2003). Nevertheless, previous studies (e.g., Kollar et al., 2007) have shown that it is far from easy to learn a new collaboration strategy and apply it right away to elaborate content information more deeply. Possibly, students need more time to internalize the strategy proposed by a collaboration script to use it effectively for a more thorough elaboration of knowledge. Second, while we did find that the combination of collaboration script and heuristic worked examples was most effective with respect to the social-discursive component, no such effect could be found with respect to the individualcognitive component. Thus, at least with respect to the individual-cognitive component, there was no synergistic scaffolding effect (Tabak, 2004), i.e. the two scaffolds did not amplify each others effects. Possibly there was no real fit between the two scaffolds, which means that students in the combined condition had to separate their attention to two rather distinct tasks, namely to (a) follow interaction-related prompts provided by the script and (b) thoroughly process content-related information presented in the heuristic worked examples. Special consideration has to be given to the result that both the collaboration script and the heuristic worked example worked specifically for students with higher general ability. Obviously, both interventions were rather demanding, and to play out their potential, they required a lot of the students cognitive resources. Future research needs to investigate whether this pattern of results can also be found in other domains. Also, finding ways to design collaboration scripts and heuristic worked examples that also help learners with less positive learning prerequisites clearly is a didactical challenge for future studies as well as for educational practice. Of course, our study is not without limitations. First, the way the social-discursive component of mathematical argumentation competence was measured can be criticized, since we had students describe how they would act in an argumentative situation, without actually having them observing them in argumentation. Surely, more performance-based measures would be helpful. Second, our results may have been influenced by the specific way the two scaffolds were realized. Possibly, a script that only distributes roles without prompting them would lead to less pronounced effects than the script we used. Likewise, less detailed heuristic worked examples could be imagined that might yield different effects. Thus, future research should try to identify the crucial components of collaboration scripts and heuristic worked examples that are responsible for the effects. Despite these limitations, our study shows that, if mathematical argumentation skills are considered a compound of social-discursive and individual-cognitive components, learners require scaffolds on both a content-related and a social-discursive level. Heuristic worked examples and collaboration scripts seem to be a valuable combination of such scaffolds.
References
Andriessen, J., Baker, M. & Suthers, D. (2003). Argumentation, computer support, and the educational context of confronting cognitions. In J. Andriessen, M. Baker & D. Suthers (Eds.), Arguing to learn: Confronting cognitions in Computer-Supported Collaborative Learning environments (pp. 1-25). Dordrecht: Kluwer. Atkinson, R.K., Derry, S.J., Renkl, A. & Wortham, D. (2000). Learning from Examples: Instructional Principles from the Worked Examples Research. Review of Educational Research, 70(2), 181-214. Boero, P. (1999). Argumentation and mathematical proof: A complex, productive, unavoidable relationship in mathematics and mathematics education. International Newsletter on the Teaching and Learning of Mathematical Proof, July/August 1999. http://www.lettredelapreuve.it/. Chinnappan, M. & Lawson, M. J. (1996). The effects of training in the use of executive strategies in geometry problem solving. Learning and Instruction 6(1), 1-17. De Wever, B., van Keer, H., Schellens, T. & Valcke, M. (2010). Roles as a structuring tool in online discussion groups: The differential impact of different roles on social knowledge construction. Computers in Human Behavior, 26(4), 516-523. Hansen, M. & Spada, H. (2010). Supporting remote collaborative problem-solving. Applied Cognitive Psychology, 24(9), 1297-1323.
ISLS
337
ICLS2012
Heinze, A., Reiss, K., & Rudolph, F. (2005). Mathematics achievement and interest from a differential perspective. Zentralblatt fr Didaktik der Mathematik, 37(3), 212-220. Kirschner, F., Paas, F., Kirschner, P., & Janssen, J. (2011). Differential effects of problem-solving demands on individual and collaborative learning outcomes. Learning and Instruction, 21, 587-599. Kollar, I., Fischer, F. & Hesse, F. W. (2006). Collaboration scripts - a conceptual analysis. Educational Psychology Review, 18(2), 159-185. Kollar, I., Fischer, F. & Slotta, J. D. (2007). Internal and external scripts in computer-supported collaborative inquiry learning. Learning & Instruction, 17(6), 708-721. Krauss, S., Baumert, J., & Blum, W. (2008). Secondary mathematics teachers pedagogical content knowledge and content knowledge: validation of the COACTIV constructs. ZDM The international Journal of Mathematics Education, 40, 873-892. Leito, S. (2000). The potential of argument in knowledge building. Human Development, 43, 332-360. Lin, F.-L. (2005). Modeling students learning on mathematical proof and refutation. In H. L. Chick,. & J. L. Vincent (Eds.), Proceedings of the 29th Conference of the International Group for the Psychology of Mathematics Education, 1, (pp. 3-18). Melbourne, PME. Mullins, D., Rummel, N. & Spada, H. (2011). Are two heads better than one? Differential effects of collaboration on students computer-supported learning in mathematics. International Journal of Computer-Supported Collaborative Learning, 6(3), 421-443. National Council of Teachers of Mathematics (Ed.) (2000). Principles and Standards for School Mathematics. Reston, VA: NCTM. ODonnell, A. M. (1999). Structuring dyadic interaction through scripted cooperation. In A. M. ODonnell & A. King (Eds.), Cognitive perspectives on peer learning (pp. 179-196). Mahwah, NJ: Erlbaum. ODonnell, A. M., & Dansereau, D. F. (1992). Scripted Cooperation in student dyads: A method for analyzing and enhancing academic learning and performance. In R. Hertz-Lazarowitz, & N. Miller (Eds.), Interaction in cooperative groups: The theoretical anatomy of group learning (pp. 120-141). New York: Cambridge University Press. Pozzi, F. (2011). The impact of scripted roles on online collaborative learning processes. International Journal of Computer-Supported Collaborative Learning, 6(3), 471-484. Reiss, K. & Renkl, A. (2002). Learning to prove: The idea of heuristic examples. Zentralblatt fr Didaktik der Mathematik, 34(1), 29-35. Reiss, K., Heinze, A., Kessler, S., Rudolph-Albert, F., & Renkl, A. (2007). Forstering argumentation and proof competencies in the mathematics classroom. In M. Prenzel (Ed.): Studies on the educational quality of schools: the final report on the DFG priority programme (pp. 251-264). Mnster: Waxmann. Rourke, A. & Sweller, J. (2009). The worked-example effect using ill-defined problems: Learning to recognise designers styles. Learning and Instruction 19, 185-199. Schworm, S., & Renkl, A. (2007). Learning Argumentation Skills Through the Use of Prompts for SelfExplaining Examples. Journal of Educational Psychology, 99(2), 285-296. Stegmann, K., Weinberger, A., & Fischer, F. (2007). Facilitating argumentative knowledge construction with computer-supported collaboration scripts. International Journal of Computer-Supported Collaborative Learning, 2(4), 421-447. Tabak, I. (2004). Synergy: A complement to emerging patterns of distributed scaffolding. The Journal of the Learning Sciences, 13(3), 305-335. Toulmin, S. (1958). The uses of argument. Cambridge, UK: Cambridge University Press. Ufer, S., Heinze, A. & Reiss, K. (2008). Individual predictors of geometrical proof competence. In O. Figueras & S. Sepulveda (Eds.), Proceedings of the Joint Meeting of the 32nd Conference of the International Group for the Psychology of Mathematics Education, and the XXX. North American Chapter, - Volume 4 (pp. 361-368). Morelia, Michoacan, Mexico: PME. van Gog, T., Paas, F., & van Merrienboer, J. (2008). Effects of studying sequences of process-oriented and product-oriented worked examples on troubleshooting transfer efficiency. Learning and Instruction 18, 211-222. van Gog, T., & Rummel, N. (2010). Example-Based Learning: Integrating Cognitive and Social-Cognitive Research Perspectives. Educational Psychology Review, 22, 155-174. Wecker, C., Kollar, I., Fischer, F. & Prechtl, H. (2010). Fostering online search competence and domainspecific knowledge in inquiry classrooms: effects of continuous and fading collaboration scripts. In K. Gomez, L. Lyons, & J. Radinsky (Eds.) Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010) - Volume 1, Full Papers (pp. 810-817). International Society of the Learning Sciences: Chicago IL. Weinberger, A., Ertl, B., Fischer, F. & Mandl, H. (2005). Epistemic and social scripts in computer-supported collaborative learning. Instructional Science, 33, 1-30.
ISLS
338
ICLS2012
Technology Supports in CSCL

Heisawn Jeong, Hallym University, Chuncheon, Korea (South), heis@hallym.ac.kr Cindy E. Hmelo-Silver, Rutgers University, New Brunswick, NJ USA, cindy.hmelo-silver@gse.rutgers.edu Abstract: This study examined how technology supports collaborative learning in ComputerSupported Collaborative Learning (CSCL). Empirical CSCL studies published between 20052008 were selected from seven leading journals of the field and analyzed in terms of their technology supports. Analyses showed that communication technologies were most popular with asynchronous collaboration being the dominant mode of interaction. Although CSCL was predominantly supported by communication tools, diverse types of additional technologies (e.g., representational tool, simulations) were used to support both face-to-face and distributed collaboration. Technology supports varied depending on the educational levels, domains, and pedagogical approaches of the learning environments. Based on these analyses, we discuss five distinct ways that technology can support collaborative learning.
Introduction
Collaboration stimulates constructive processing, elaboration of knowledge and collaborative and social knowledge construction (Hmelo-Silver, 2009; Jeong & Chi, 2007). During collaboration, learners can learn from others, pool their resources toward a common goal, and come up with a solution that might not have been possible alone. For these reasons, collaborative learning, although it is not a panacea, is often considered to be one of the most effective forms of learning (Chi, 2009; Cohen, 1994; Scardamalia, 2002; Webb & Palincsar, 1996). Although computers and other technologies were initially used to support individual learning (Lajoie, 2000; Reiser, 2001), the development of networked technologies prompted the emergence of ComputerSupported Collaborative Learning (CSCL). CSCL aims to provide technology supports for collaborative learning and has been considered to be one of the most promising applications of modern information and communication technology toward the improvement of teaching and learning (Lehtinen, Hakkarainen, Lipponen, Rahikainen, & Muukkonen, 1999; Stahl, Koschmann, & Suthers, 2006). Although computers and digital technologies are actively used to support collaborative learning in CSCL, we do not understand exactly what kinds of technologies are used for this goal and how they support collaboration. In order to characterize the nature of supports computers provide for collaborative learning, researchers have used terms such as collaboration around versus through computers (Lehtinen, et al., 1999) or technology as a medium versus constraint (Suthers, 2006). These notions refer to the different ways that technology support collaborative learning. Metaphors such as through or medium refer to the role of technology as an enabler of communication, whereas the notion of collaboration around computers refers to their role as an object (i.e., contents) of collaboration (e.g., students discussing the contents of simulation). Technology as a constraint refers to its role to impose certain constraints on the interactive learning process (e.g., collaboration scripts). Given the proliferation of technologies and diverse ways that they are used to support learning and other human activities, there is a need to examine more systematically exactly how technologies support collaborative learning. Such an examination will not only help us to develop more accurate understanding about the role of technology support in CSCL but also to guide the efforts of the designers and instructors when they design and implement CSCL applications. The current study is a part of a larger project that attempts a comprehensive examination of CSCL research based on a content meta-analysis of recent CSCL investigations (Jeong & Hmelo-Silver, 2010a, 2010b, & 2011). In this paper, we focus on the nature of the support technologies provide for collaborative learning in CSCL and how it varied across educational levels, learning domains, and pedagogical approaches embodied in the learning environments.
Method
Journal Selection
Seven journals were selected for this study: (1) International Journal of Computer Supported Collaborative Learning (ijCSCL) (2) Journal of the Learning Sciences, (3) Learning and Instruction, (4) Computers and Education, (5) Journal of Computer Assisted Learning, (6) International Journal of Artificial Intelligence in Education, and (7) Computers in Human Behavior. The journals were selected based on a survey of 16 CSCL community leaders (e.g., CSCL committee of ICLS and the editorial board members of ijCSCL). These are all peer-reviewed journals published by well-known publishers with international author- and readership. Articles published during 2005-2008 period in these journals, that is, four years of publication and three years of publication in the case of ijCSCL were examined in the study.
ISLS
339
ICLS2012
Paper Selection
Excluding non-research articles (e.g., editorials), 1,422 articles were published in the seven journals during the 2005-2008 period. We screened them to identify empirical CSCL research papers. CSCL research meant that learners learned collaboratively using technological tools. Learning needed to be collaborative, but as long as parts of the learning process involved interaction (e.g., collaborative discussion after individual study), it was considered collaborative. Collaboration was broadly defined, so that interaction with computerized agents, intelligent system, or other types of indirect interaction such as interaction via persistent artifacts (Suthers, Dwyer, Medina, & Vatrapu, 2010) was included. The focus was on peer collaboration. Student-teacher interactions were excluded unless peer collaboration was included. The technologies applied did not necessarily have to be so-called collaboration technology such as e-mails or discussion boards, but needed to be specific. Studies that examined social and technical issues were included if they were studied in relation to CSCL. Analyses were restricted to empirical papers where primary data collection and analysis were carried out so as to focus on CSCL technologies that are actually implemented and researched. The selection process proceeded in two stages. First, initial selection of empirical CSCL papers was conducted based on title and abstract of the paper. At this stage, we were as inclusive as possible so as not to miss any potential CSCL papers. This initial screening was verified at the coding stage so that final eligibility judgment was made based on a more comprehensive examination of the papers. The selection and coding process is not yet complete. This paper reports on preliminary results based on 284 papers currently in the coding pool.
Content Analysis
A combination of inductive and deductive approaches was used to develop coding categories. Codes were initially developed based on a combination of several top-down schemes (e.g., categories drawn from the submission descriptors of the 2005 CSCL conference) and then later refined inductively.
Technology
Technology refers to the technologies used to assist learning and consisted of the following 15 categories: (1) Emails, (2) Discussion boards (including Knowledge Forum), (3) Chat, (4) Video-conferencing, (5) Mobile (e.g., phones, laptops, or other wireless devices), (6) Multi- or hyper-media (e.g., streaming videos), (7) Internet (e.g., websites, blogs), (8) Wiki, (9) Virtual reality (VR), (10), Multi-user virtual reality environments (MUVE), (11) Games, (12) Simulations, (13) Representational tools (e.g., concept maps, diagrams, or other visualization tools), (14) Intelligent System (e.g., simulated agents, Intelligent Tutoring Systems), (15) Others (e.g., hardware, generic software, or other miscellaneous technologies that did not fit into the preceding categories). There was considerable variance in how researchers described their technology. The same technology was described with different names (e.g., discussion boards versus conferencing system) and/or at different granularity (e.g., generic description versus detailed description of hardware specifications). Because the focus of the current analysis was to identify technologies at a level where learners interact with them, technical aspects of the implementation (e.g., servers, system architectures, SCORM standards, etc.) were not coded unless directly used to support learning or collaboration. As for the Internet/web, it was coded as a separate technology when its contents (e.g., information on specific websites) were used for learning, but not when used as a medium of communication (e.g., e-mails) or as a delivery system (e.g., online course management software). Integrated system or learning environments such as e-learning platforms (e.g., WebCT, Moodle) were coded for their individual technologies, but complementary (e.g., tutorials are offered to students to help them use the tool better) or irrelevant technologies (e.g., technologies that may be part of the environment but irrelevant to the study) were not coded separately. When a technology could be categorized in more than one ways (e.g., a MUVE game can be coded either in the MUVE or game category), the dominant aspect of the technology was selected based on the research contexts and study questions.
Collaboration
Collaboration refers to the types of interaction among learners in CSCL environment and consisted of the following three types: (1) Mediated face-to-face, (2) Synchronous distributed, and (3) Asynchronous distributed. Mediated face-to-face collaboration refers to co-located and synchronous interaction supported by technology (e.g., collaboration around a computer simulation), which is different from more traditional unmediated face-toface collaboration. Distributed collaboration refers to collaboration among distributed learners and can be either synchronous or asynchronous.
Educational Levels
Educational levels refer to the level of student learning examined in the study and consisted of the following three levels: (1) K-12 for primary and secondary education, (2) Higher education (i.e., undergraduate and graduate education), and (3) Other (e.g., professional training, life-long learning, learning networks outside of
ISLS
340
ICLS2012
schools, etc.). When data were collected from teachers while they served students at certain levels (e.g., teachers adoption and/or evaluation of tools for elementary students), we coded the level of the students that they served. When teachers were learners themselves, we coded it according to the study contexts. If the teachers enrolled in regular graduate courses, it was coded as higher education. If they were participating in a refresher course or learning networks, it was coded as other level.
Learning Domains
The domains of the study refer to academic disciplines, that is, domains whose knowledge and skills learners were trying to acquire. Learning domains were coded into five categories: (1) Science and math (e.g., physics, statistics), (2) Literacy and history, (3) Social sciences (e.g., psychology, instructional science), (4) Professional (e.g., nursing, engineering, teacher education, etc.), and (5) Other (e.g., workplace training, learning of social skills). These categories typically refer to academic disciplines in higher education, but they were also used to code related subject areas taught in secondary schools and other contexts. This coding was also applied to laboratory tasks if domains can be identified (e.g., science or educational problem solving in laboratory). When domain-independent knowledge or skills were learned in specific content domains (e.g., argumentation/writing about genetically modified food), we coded the content domain (e.g., science). When students learned about cross-disciplinary subjects (e.g., science communications, information technology in education), we coded them into one of the existing domains that best characterize them, but if they did not fit into existing domains, we coded them as other. Other domains also included hard-to-categorize domains, domain-independent learning (e.g., study skills), and unclear cases (e.g., university orientation).
Pedagogical Approaches
Pedagogical approaches refer to the instructional strategies used by teachers and reflect their beliefs about how learning occurs and how to promote it. Pedagogical approaches were coded into the following categories: (1) Traditional instructions (e.g., lectures), (2) Distance learning (e.g., learning in online courses, blended courses, learning networks, mobile learning), (3) Case-based instruction, (4) Problem solving, (5) Project/Problem-based learning (PBL), (6) Design-based learning, (7) Hands-on/Active learning, (8) Inquiry (e.g., modeling, Progressive Inquiry), (8), Game, (10) Collaboration, and (11) Other (e.g., informal learning, life-long learning, etc.). We distinguish problem- or project-based learning from problem solving as a pedagogical approach because in the former, problems are used as a context where students figure out what they need to know as they solve the problems (e.g., learning human physiology in the context of diagnosing disease; Nicol & MacLeod, 2005); in the latter, problems are more clear-cut, where learners apply and practice what they have learned previously (e.g., mathematical word problems; White, 2006). Although all papers used some kinds of collaboration in this study, it was not always the main feature of the pedagogy. Collaboration was coded as pedagogy when it was used as one of the main instructional features and can include (a) discussion, (b) argumentation, (c) role/scripted, (d) other miscellaneous collaboration (e.g., jigsaw, networked learning communities, wiki, or other unspecified group activities). Although pedagogy is typically used to describe the teaching activities orchestrated in classroom settings, laboratory tasks can also embody certain pedagogical principles (e.g., study of argumentation in the laboratory). We thus coded pedagogical approaches of the CSCL investigations regardless of the study settings, but if underlying pedagogical approaches cannot be identified, it was assigned to other category. Different pedagogical approaches were often incorporated in one CSCL application or environment. For example, distance learning courses were often implemented with regular online conferences or topic discussions (Hewitt & Brett, 2007). Problem-based learning was also implemented with script/role assignments (White, 2006). Instead of coding one overarching pedagogical approach, we coded all the salient instructional strategies used in the study. In all coding categories, multiple codes were allowed. For example, when a study compared effects of different collaboration conditions (e.g., shared wireless laptop vs. shared workspace as in Nicol & MacLeod, 2005) or when several different technologies were implemented together in an environment (e.g., chat and representational tool as in Lund, Molinari, Sejourne, & Baker, 2007), the study received multiple codes for each corresponding dimensions. Cohens Kappa for the 20% of the coded papers was all above .75 in all coding categories.
Results
Technology
The mean number of technologies used in a study was 1.68 as studies often used multiple technologies. Most (61%) of them used a single technology, but the rest examined multiple, up to seven, technologies in their research. The relative frequencies of CSCL technologies are shown in Figure 1.The most dominant CSCL technology was communication technology that was used in more than half of the studies (59%). Of these,
ISLS
341
ICLS2012
discussion boards were most frequent, being used in more than a third of the studies (39%), followed by chat (23%), e-mail (10%), and video-conferencing (6%). After the communication technology, the next frequent was other technology (28%), followed by the Internet/Web (14%), representational tools (13%), simulations (8%), multi/hyper-media and intelligent systems (each 6%), wiki (5%), mobiles (4%) and VR (4%), games (2%), and MUVE (1%).
Figure 1. Technologies used in CSCL investigations. Unexpectedly, a lot of the technologies fell in the other technology category (28%). A closer examination of the other category showed that it included diverse technologies such as peer assessment systems (Cho & Schunn, 2007), shared workspaces (Hwang, Chen, & Hsu, 2006), application/file/data sharing, and tools to support specific aspects of collaborative learning such as meta-cognitive and/or regulation tools (Manlove, Lazonder, & de Jong, 2006). It also included commercial software as students were asked to use word processors or text editors to compose messages for posting or as a reflection tool (Wang, 2005) and hardware and/or physical environments as physical layout of the classrooms or specific hardware (e.g., shared display) was part of an innovative learning environment (e.g., Dori & Belcher, 2005; Liu & Kao, 2007). The large proportion of other technology category suggests that our understanding (and coding categories) of the major CSCL technologies may have been biased toward the technologies that received more attention in the research literature, and yet researchers were not only developing and using new tools (e.g., group meta-cognitive tools) but also using common tools in unconventional ways. Such diversity and ingenuity in tool used was also evident in other categories. The Internet/web technology was used in a number of different ways in CSCL. Students used web-based learning platform (e.g., Bartholome, Stahl, Pieschl, & Bromme, 2006), used it as information resources (Lee, 2005), or constructed web pages or built communities as part of their learning activities (Turvey, 2006). Representational tools were often tools that allow graphical representations such as SenseMaker (Enyedy & Hoadley, 2006), but some of them were specifically designed to support certain collaborative activities (e.g., argumentation; Mirza, Tartas, Perret-Clemont, & de Pietro, 2007).
Collaboration
Learners in CSCL environments engaged in distributed asynchronous collaboration most frequently (56% of studies), followed by synchronous (36%) and mediated face-to-face collaboration (29%). Although the majority of the studies (80%) supported one type of collaboration, the rest (20%) supported more than one type of collaboration (e.g., Ellis, Goodyear, Prosser, & O'Hara, 2006; Veermans & Cesareni, 2005). How did technologies support collaborative learning? Because studies often supported multiple types of collaboration, we focused on the studies that supported one type of collaboration only (N=206) for this analysis and examined how each collaboration type was supported by technologies. As for the synchronous collaboration, not surprisingly, it was mostly supported by chat (58%) and video-conferencing (16%) but also by representational tools (21%) and other technologies (26%) such as virtual agents, (Holmes, 2007) or application sharing (Ertl, Kopp, & Mandl, 2008). As for the asynchronous collaboration, it was largely supported by communication technologies such as discussion boards (69%) and e-mail (8%), as well as by the Internet/Web (17%) and other technology (20%). Examples of other technology included peer review/evaluation system (Cho & Schunn, 2007), database (Kali, 2006), or other miscellaneous technologies. Mediated face-toface collaboration was supported mostly by non-communication technologies such as other technology (30%), simulations (19%), and hyper/multi-media (18%). They involved collaboration around computer (e.g., discussing a simulation while sharing the computer screen), about computers (e.g., collaborative programming in groups), or with computers (e.g., PDAs supplementing face-to-face interaction). In an example of mediated face-to-face with computers, students engaged in face-to-face collaboration each with their
ISLS
342
ICLS2012
own handheld device (White, 2006). The task was divided so that each student carried out a different portion of the task and the changes that individual students made with his or her handhelds were propagated to other members handhelds. In this case, the handhelds allowed learners to work on different parts of the problem simultaneously and yet be in synch with other members progress.
Educational Levels
The majority of the CSCL learners reported were in higher education (60%), followed by K-12 (34%) and other (8%). CSCL technology supports differed across educational levels. Communication technology, for example, was used 65% in the higher education, but only 50% in K-12 and 46% in other levels. There was also a difference in the kinds of communication tools used (see Figure 2a). Asynchronous communication tools (e.g., e-mails and discussion board) were more heavily used in higher and other education levels. For example, the discussion board was used in 24% of the K-12 studies, but 47% and 38% in higher education and other levels, respectively. On the other hand, synchronous tools were used equally frequently in K-12, so that chat tools were used 23% of the time in K-12 studies and 24% and 13% in higher education and other levels. Another difference between levels was the use of multiple communication technologies: the average number of communication technologies was 1.15 in K-12, 1.36 in higher education, and 1.55 in other level. Figure 2b also shows that K-12 also tends to rely more on non-communication technologies such as simulations and representational tools. It is unclear at this point whether this is related to the maturity of the K-12 learners or merely due to the fact that learners at K-12 level tend to spend more time together in schools, but it appears that the more advanced the learners were, the bigger the tendency to rely on communication technologies, especially asynchronous types.
Figure 2. CSCL technological supports across educational levels.
Learning Domains
CSCL technologies were used in a range of domains such as science/math (34%), professional (44%), literacy/history (13%), social science (9%) and other (9%). CSCL technology supports differed across domains. Table 1 presents the top five technologies used in different learning domains. This table shows that, although the use of communication and other technologies was dominant in all domains, it was less so in science/math. Technologies such as simulations and representational tools played more important roles in science/math. Table 1. Top five technologies used in different learning domains. Ranks Science/Math Literacy/History Social Science 1 DB (29%) DB (41%) DB (40%) 2 Other (24%) Chat (22%) Other (40%)* 3 Rep. tool (20%) Other (22%)* Chat (12%) 4 Chat (18%) Rep. tool (19%) Video C (12%)* 5 Simulation (16%) Internet (19%)* Internet (12%)* * indicates a tie with the preceding technology.
Professional DB (53%) Other (31%) Chat (26%) E-mail (12%) Internet (12%)
Other DB(36%) Other (32%) Chat (30%) Internet (23%) E-mail (9%)
Pedagogical Approaches
Not surprisingly, collaboration pedagogy was implemented most frequently (43%), followed by distance learning (19%), inquiry (10%), PBL (9%), problem solving (8%), traditional instruction (8%), design-based learning (5%), other (5%), case-based instruction (4%), and active/hands-on learning (4%). Within collaboration pedagogy, generic discussion was most frequent (18%), followed by misc. collaboration (9%), scaffolding (6%), argumentation (6%), and roles/scripted collaboration (5%). Although the majority (84%) of the studies employed single pedagogical approach, some used more than one approach. It appears that most CSCL
ISLS
343
ICLS2012
investigations were embedded in some kinds of innovative pedagogies. Traditional pedagogy was not frequent. In addition, distance learning which often rely on traditional lecture and can thus be quite passive was complemented with other pedagogical approaches half of the time in the studies analyzed in this paper. Still, it should be noted that inclusion of discussions or other activities may not necessarily ensure innovate teaching and learning, and cares need to be taken to ensure that appropriate learning activities occur. We also examine whether different technologies are associated with different pedagogical approaches. Due to space limitations, however, we restricted our analysis to six major technologiesthe two most frequent communication technologies (i.e., discussion board and chat), other technology, Internet/web, representational tool, and simulationand identified three key pedagogies associated with each technology (see Table 2). Analyses showed that, for example, although the discussion board was the dominant technology associated with almost all pedagogical approaches, it was associated more strongly with distance learning and collaboration pedagogy. On the other hand, the representational tools and simulations were strongly associated with pedagogical approaches such as inquiry, problem solving, active/hands-on approaches. These results suggest that different pedagogical approaches are likely to have different technological needs. Researchers and educators need to align technological affordances with the pedagogical needs of the curriculum. In addition, although other technology was not of the same kinds, but its strong association with some of the pedagogical approaches (e.g., active/hands-on, other, and case-based) suggests that these pedagogical approaches might have technological needs unarticulated and unmet by more established and well-known technologies. Table 2. Key pedagogical approaches associated with six major technologies. TechnologiesRanks 1 2 Discussion B Distance L (69%) Collaboration (48%) Chat Distance L (35%) Problem-solving (32%) Other Active/Hands-on (60%) Other (40%) Internet/Web Other (33%) Design-based (20%) Rep. Tool Inquiry (28%) Problem-solving (23%) Simulation Active/Hands-on (40%) Inquiry (24%) * indicates a tie with the preceding technology.
3 Inquiry (40%) Collaboration (25%) Case-Based, Traditional (36%) Distance L (20%)* Design-based, Active (20%) Problem-solving (14%)
Discussion
This study investigated technology supports in CSCL. A content meta-analysis of CSCL research showed that communication technologies were most common with the dominant mode of collaboration being asynchronous interaction. Although CSCL was largely supported by communication technologies, diverse other technologies (e.g., representational tool, simulations) were used to support both face-to-face and distributed collaboration. Technology supports varied depending on the educational levels, learning domains, and pedagogical approaches embedded in the learning environments so that the role of communication technology was lessened in K-12 and science/math domains. Different pedagogical approaches also appeared to have different technological needs. Although there are many different types of learning technologies, not all have adequate affordances for learning. The argument for learning technology is not about having students learn faster, but rather about helping them learn better. Likewise, although there are many ways that technologies support collaborative learning, they need to do more than making interaction possible. They need to enhance and facilitate processes of collaborative learning (Suthers, 2006). What are the processes of collaborative learning that current CSCL technology applications support? Integrating the results of the current analyses and earlier conceptualizations (Jeong & Hmelo-Silver, 2010a; Lehtinen, et al., 1999; Suthers, 2006), we identify five distinct types of technology supports for collaborative learning in CSCL applications. First, technologies support collaborative learning by providing means of communication. Technologies such as discussion boards and chat tools make distributed collaboration possible as partners in remote locations can interact synchronously with the help from these tools (Janssen, Erkens, Kanselaar, & Jaspers, 2007; Vizcaino, 2005). Asynchronous technologies also allow learners to interact whenever and wherever they want (Kitsantas & Chow, 2007; Yang & Liu, 2007). These types of technologies play more important roles in certain contexts such as in distance learning, but are important in other contexts as well. Second, technologies facilitate collaboration by providing a focal point of interaction around which learners discuss, argue, and explain (Lehtinen, et al., 1999). This type of facilitation typically occurred in face-to-face collaboration, but can also happen in distributed interaction (Susser & Ariga, 2006). Third, learners do not always engage in productive interaction during collaborative learning. Technologies can be used to guide learners toward task-specific interactions by imposing structures and constraints on the interaction process (Dillenbourg & Hong, 2008; Suthers, 2006; Weinberger, Ertl, Fischer, & Mandl, 2005). Fourth, technologies provide tools and resources to manage and control collaborative processes. Representational tools such as whiteboards and graphic tools complement communication processes by providing deictic cues and also make the interactive process explicit (Hwang, Chen, & Hsu, 2006; Lai & Wu, 2006; Yang & Liu, 2007). They become the basis of negotiating shared meaning and common grounds during
ISLS 344
ICLS2012
interaction. The resulting artifacts, be they argumentation maps or Wikipedia pages, serve as a basis of shared knowledge and collective memory. Technology also makes it possible for members to share resources more easily (Nicol & MacLeod, 2005). Although such supports are still possible in traditional modes of collaboration with pen and paper, technologies make these functions much more accessible to learners. Fifth, technologies support collaborative learning by making new forms of interaction possible. Traditionally, interaction meant direct person-to-person interaction. Communication technologies make it possible to interact through e-mails or chat, but they still mean direct person-to-person interaction. New forms of interaction are possible with technologies so that learners can now interact indirectly through Wikipedia pages, social navigational support (Lee, 2005; Nicol & MacLeod, 2005; Recker, Walker, & Lawless, 2003), or peer feedback system (Cho & Schunn, 2007). Interaction with non-human agents can also be used to monitor and support human learning (Vizcaino, 2005). These five types of technology supports are not equally realized in CSCL research with some types of supports being more mature and prevalent than other type of supports. We also do not understand yet how different types of supports interact with learner characteristics (e.g., educational levels) and various aspects of learning environments (e.g., learning domains and pedagogical approaches). Much CSCL research has been driven by the technologies rather than learners needs and underlying learning mechanisms. With technological advances, additional ways to support collaborative learning will emerge. Whatever the future technologies might be, we should bear in mind that technological advances will benefit CSCL to the extent that they support core mechanisms of collaborative learning. Increased awareness and attention to the underlying learning mechanisms in addition to the technology itself will help the designers and instructors to design and implement more effective CSCL applications.
References
Bartholeme, T., Stahl, E., Pieschl, S., & Bromme, R. (2006). What matters in help-seeking? A study of help effectiveness and learner-related factors. Computers in Human Behavior, 22, 113-129. Chi, M. T. H. (2009). Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in Cognitive Science, 1, 73-105. Cho, K., & Schunn, C. (2007). Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system. Computers and Education, 48, 409-426. Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups. Review of Educational Research, 64(1), 1-35. Dillenbourg, P., & Hong, F. (2008). The mechanics of CSCL macro scripts. Computer-Supported Collaborative Learning, 3, 5-23. Dori, Y. J., & Belcher, J. (2005). How does technology-enabled active learning affect undergraduate students' understanding of electromagnetism concepts? The Journal of the Learning Sciences, 14(2), 243-279. Ellis, R. A., Goodyear, P., Prosser, M., & O'Hara, A. (2006). How and what university students learn through online and face-to-face discussion: conceptions, intentions, and approaches. Journal of Computer Assisted Learning, 22, 244-256. Enyedy, N., & Hoadley, C. M. (2006). From dialogue to monologue and back: Middle spaces in computermediated learning. International Journal of Computer-Supported Collaborative Learning, 1, 413-439. Ertl, B., Kopp, B., & Mandl, H. (2008). Supporting learning using external representations. Computers & Education, 51, 1599-1608. Hmelo-Silver, C. E. (2009, August). Facilitating social knowledge construction in communities of learners: Are we there yet (and how would we know)?Invited keynote presentation. European Association for Research on Learning and Instruction. Amsterdam, Netherlands Holmes, J. (2007). Designing agents to support learning by explaining. Computers and Education, 48, 523-547. Hwang, W.-Y., Chen, N.-S., & Hsu, R.-L. (2006). Development and evaluation of multimedia whiteboard system for improving mathematical problem solving. Computers and Education, 46, 105-121. Janssen, J., Erkens, G., Kanselaar, G., & Jaspers, J. (2007). Visualization of participation: does it contribute to successful computer-supported collaborative learning? Computers and Education, 49, 1037-1065. Jeong, H., & Chi, M. H. (2007). Knowledge convergence and collaborative learning. Instructional Science, 35, 287-315. Jeong, H., & Hmelo-Silver, C. (2011). A portrait of CSCL methodologies. Paper presented at the ComputerSupported Collaborative Learning, Hong Kong. Jeong, H., & Hmelo-Silver, C. (2010). An overview of CSCL methodologies. Paper presented at the International Conference of the Learning Sciences, Chicago. Jeong, H., & Hmelo-Silver, C. E. (2010). Technology use in CSCL: A content meta-analysis. Paper presented at the Hawaiian International Conference on System Science. Kali, Y. (2006). Collaborative knowledge building using the design principles database. International Journal of Computer-Supported Collaborative Learning, 1, 187-201.
ISLS
345
ICLS2012
Kitsantas, A., & Chow, A. (2007). College students' perceived threat and preference for seeking help in traditional, distributed, and distance learning environments. Computers and Education, 48, 383-395. Lai, C.-Y., & Wu, C.-C. (2006). Using handhelds in a jigsaw cooperative learning environment. Journal of Computer Assisted Learning, 22, 284-297. Lajoie, S. P. (Ed.). (2000). Computers as cognitive tools: No more walls. Mahwah, NJ: Earlbaum. Lee, Y.-J. (2005). VisSearch: A collaborative Web searching environment. Computers and Education, 44, 423439. Lehtinen, E., Hakkarainen, K., Lipponen, L., Rahikainen, M., & Muukkonen, H. (1999). Computer-supported collaborative learning: A review. Available: http://www.kas.utu./clnet/clnetreport.html [Accessed 29 June 2007]. Liu, C.-C., & Kao, L.-C. (2007). Do handheld devices facilitate face-to-face collaboration? Handheld devices with large shared display groupware to facilitate group interactions. Journal of Computer Assisted Learning, 23, 285-299. Lund, K., Molinari, G., Sejourne, A., & Baker, M. (2007). How do argumentation diagrams compare when student pairs use them as a means for debate or as a tool for representing debate? International Journal of Computer-Supported Collaborative Learning, 2, 273-295. Manlove, S., Lazonder, A. W., & de Jong, T. (2006). Regulative support for collaborative scientific inquiry learning. Journal of Computer Assisted Learning, 22, 87-98. Mirza, N. M., Tartas, V., Perret-Clemont, A., & de Pietro, J.-F. (2007). Using graphical tools in a phased activity for enhancing dialogical skills: An example with Digalo. International Journal of ComputerSupported Collaborative Learning, 2, 247-272. Nicol, D. J., & MacLeod, I. A. (2005). Using a shared workplace and wireless laptops to improve collaborative project learning in an engineering design class. Computers and Education, 44, 459-475. Recker, M. M., Walker, A., & Lawless, K. (2003). What do you recommend? Implementation and analyses of collaborative information filtering of web resources for education. Instructional Science, 31, 299-316. Reiser, R. A. (2001). A history of instructional design and technology: Part II: A history of instructional design. Educational Technology Research and Development, 49(2), 57-67. Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith (Ed.), Liberal education in a knowledge society (pp. 67-98). Chicago: Open Court. Stahl, G., Koschmann, T., & Suthers, D. D. (2006). Computer-supported collaborative learning: A historical perspective. In R. K. Sawyer (Ed.), Cambridge handbook oh the learning sciences. New York: Cambridge University Press. Susser, B., & Ariga, T. (2006). Teaching e-commerce Web page evaluation and design: a pilot study using tourism destination sites. Computers and Education, 47, 399-413. Suthers, D. D. (2006). Technology affordances for intersubjective meaning making: A research agenda for CSCL. Computer-Supported Collaborative Learning, 1, 315-337. Suthers, D. D., Dwyer, N., Medina, R., & Vatrapu, R. (2010). A framework for conceptualizing, representing, and analyzing distributed interaction. International Journal of Computer Supported Collaborative Learning, 5(1), 5-42. Turvey, K. (2006). Toward deeper learning through creativity within online communities in primary education. Computers and Education, 46, 309-321. Veermans, V., & Cesareni, D. (2005). The nature of the discourse in web-based collaborative learning environments: Case studies from four different countries. Computers and Education, 45, 316-336. Vizcaino, A. (2005). A simulated student can improve collaborative learning. International Journal of Artificial Intelligence in Education, 15, 3-40. Wang, C.-H. (2005). Questioning skills facilitate online synchronous discussion. Journal of Computer Assisted Learning, 21, 303-313. Webb, N. M., & Palincsar, A. S. (1996). Group processes in the classroom. In D. C. Berliner & R. C. Calfee (Eds.), Handbook of Educational Psychology. New York: Simon & Schuster Macmillan. Weinberger, A., Ertl, B., Fischer, F., & Mandl, H. (2005). Epistemic and social scripts in computer-supported collaborative learning. Instructional Science, 33, 1-30. White, T. (2006). Code talk: Student discourse and participation with networked handhelds. International Journal of Computer-Supported Collaborative Learning, 1, 359-382. Yang, Z., & Liu, Q. (2007). Research and development of web-based virtual online classroom. Computers and Education, 48, 171-184.
Acknowledgments
This research was in part funded by the National Research Foundation of Korea (NRF) (Grant No. 20090068919) awarded to the first author and a Rutgers University Faculty Research Council grant to the second author.
ISLS
346
ICLS2012
Situating epistemological development

William A. Sandoval, University of California, Los Angeles, 2327 Moore Hall, Los Angeles, CA, USA, sandoval@gseis.ucla.edu Abstract: Personal epistemology as a field has been around for forty years and generated a wide assortment of models to explain epistemological development. Dominant frameworks are inspired either by Piagetian or information processing perspectives on cognition, and all have failed to explain a range of empirical results that question the stability and coherence predicted by these frameworks. Recent models from a more situated perspective seem much more capable of explaining the variability seen empirically. This paper adds to this situated perspective by outlining a research approach for "locating" epistemic beliefs within analyses that triangulate between people's participation in knowledge construction and evaluation, the artifacts such participation may produce, and individual's reflection upon both participation and artifacts. Examples of such work are used to demonstrate the fruitfulness of this approach.
Epistemic cognition and epistemic belief

What do people think knowledge is, or how we get it, or why we believe it, or what it means to know something? How do such ideas develop throughout people's lives? How is that development shaped by the experiences people have, in school, on the playground, in their homes, at their jobs, or anywhere else? How do these ideas influence how people think about their world, make judgments, solve problems, or learn new things? These questions are at the heart of the study of personal epistemology, people's beliefs about knowledge and knowing. The field of personal epistemology has been going strong for forty years now, generating a variety of models to account for the development of epistemological beliefs. This growth has occurred alongside, and perhaps even prompted, a concurrent growth in the educational concern that learning in the disciplines should include understanding of disciplinary epistemologies. Yet, none of the theoretical models put forth in the last forty years adequately explains the array of empirical findings within personal epistemology or related studies of epistemic cognition. Also over the last forty years, there has been a cultural turn in psychology, inspired by the theories developed by Vygotsky (1978) and his contemporaries, and further developed by many others (Cole, 1996; Engestrm, 1987; Lave & Wenger, 1991; Rogoff, 1990; Wenger, 1998; Wertsch, 1985). This cultural turn is largely absent in personal epistemology research. Indeed, it can be hard to discern in cognitive developmental research generally. Two features of sociocultural perspectives have particular importance to studies of epistemic cognition. One is the conceptualization of cognition as situated within activity, and the second is that all such activity is inherently cultural (and historical). The ways in which epistemic cognition may be situated has only recently begun to be theorized (Chinn, Buckland, & Samarapungavan, 2011; Hammer & Elby, 2002). My aim here is to build on these recent efforts in two ways. The first is to emphasize the ways in which epistemic cognition is culturally situated. That is, in order to properly understand the epistemic beliefs that individuals may develop it is necessary to understand how such beliefs derive from cultural activity, both everyday activity and activity situated within disciplinary cultures (e.g., of science). A second aim is to propose a particular approach to conducting situated research on epistemic cognition that can productively coordinate the range of research methods required to understand the complex terrain that is epistemic cognition. The argument proceeds in two parts. First, the major theoretical perspectives that have been proposed within the field of personal epistemology since Perry's (1970) seminal work are summarized. One might think such a summary hardly necessary given the quantity of recent reviews of personal epistemology research (Hofer & Bendixen, in press; Hofer & Pintrich, 2002; Khine, 2008; Muis, Bendixen, & Haerle, 2006). Such reviews are typically neutral in their evaluations of competing theories, however, whereas a critical examination emphasizes how existing models fail to explain available data. The second part of the argument then describes a situative perspective on epistemic cognition, what it looks like, what it can explain, and how it can be developed. Before getting into the main line of argument, it may be useful to clarify some terminology. The study of people's ideas about knowledge and knowing, and how these develop, has occurred primarily in two distinct strands of research. One of these is developmental research on the young child's theory of mind and the emergence of "folk epistemology" (Astington, Pelletier, & Homer, 2002; Montgomery, 1992). A related line of developmental research is that concerned with causal inference and causal reasoning (Moshman, 2004). This developmental work, targeted primarily but not exclusively at younger children, tends to examine particular forms of cognition: causal inference, reasoning about false belief, and so on. In contrast, personal epistemology research has largely focused on adolescents and adults and framed its focus primarily around the construct of "epistemological belief" (Hofer & Bendixen, in press; Hofer & Pintrich, 1997). R. Kitchener (2002) usefully joins the developmental and personal epistemology strands of research together around a mutual concern for what he favors as "folk" epistemology, meaning the commonsense ideas
ISLS 347
ICLS2012
that people develop about knowledge and knowing. Kitchener maintains that psychologists studying folk epistemology tend to introduce confusion between the epistemic and the epistemological. From a philosophical perspective, any particular epistemology, i.e., theory of knowledge, is comprised of epistemic beliefs beliefs about the nature of knowledge and knowing. Epistemological beliefs, from this perspective, would be beliefs about epistemology, i.e., a meta-epistemology. This distinction has unevenly penetrated into personal epistemology research, with some researchers adhering to Kitchener's distinction and others not. Here, I will try to abide by it, qualifying cognitive elements like belief as epistemic to connote that they refer to individual's ideas about knowledge and knowing, not about epistemology per se. Some personal epistemology scholars have begun to shy away from talk of beliefs, for reasons discussed below, and instead talk of epistemic cognition (Chinn, et al., 2011; Greene, Torney-Purta, & Azevedo, 2010). The focus on epistemic cognition rather than epistemic belief signals a focus on process over cognitive structure, on the kinds of reasoning people may be doing rather than the set of beliefs that may drive that reasoning. Of course, the relation between cognitive structures and the reasoning they enable is precisely the focus of developmental research, and much of what we have learned about epistemic cognition and/or belief has come from efforts to understand how cognitive structures enable forms of reasoning. Yet, from a sociocultural perspective on development, the typical cognitive view that knowledge structures in the head are retrieved and applied as needed fails to account for the ways in which cognition is influenced by the material and social aspects inherent to human activity (Packer, 2001). Development from this perspective involves qualitative changes in cognition mediated by cultural activity and the tools involved in such activity. The argument I make here is that personal epistemology researchers have largely ignored the culturally situated nature of knowledge and knowing, and thus the situated character of epistemic cognition and epistemological development. This is particularly apparent in how issues of "domain-specificity" are treated, which is almost entirely without reference to the practices of knowledge construction and evaluation of specific domains, such as history, math, or science. More fundamentally, the survey research favored by personal epistemology researchers suffers from two basic problems. One is that taking a survey, or any other form of belief measurement, is itself a particular, and peculiar, social interaction fraught with troubles of misperception (Suchman & Jordan, 1990). The second is that such measurement efforts are far removed from people's actual efforts to construct or evaluate knowledge, and unable to explain how material and social resources are recruited into those efforts. That is, they are inherently limited in their capacity to trace epistemological development.
Models of epistemological development

Perry's (1970) groundbreaking study of (male) Harvard students' trajectories of intellectual development during the college years created the field of personal epistemology. As Hofer and Bendixen (in press) discuss, it has been widely taken up by researchers in higher education interested in student development. Perry's model of epistemological development was heavily influenced by Piaget, and subsequent models have arisen both in response to Perry's work and in relation to the changing status of Piaget's theory and more recent information processing theories of cognitive development. I discern four kinds of models populating the personal epistemology landscape: classic Piagetian stage theories, neo-Piagetian developmental theories, belief systems models, and recent situative models.
Classical developmental stage theories

What I mean by a "classic" stage theory is one with three characteristics, derived from Piaget's theory of cognitive development. First, such theories posit an invariant sequence of stages of development, each one qualitatively different from the others. Stage theories propose that all individuals proceed through the same sequence of stages, though not necessarily at the same rate. Most importantly, in classical stage theories stages cannot be skipped. Second, each stage is presumed to entail broad qualitative changes in cognitive function, quite general in their scope. Movement to a new stage is presumed to lead to fundamental changes in cognitive structure and hence how one reasons across domains. Third, classic stage theories all have an endpoint, a final stage that reflects the telos of development. This final stage reflects the ultimate reach of human development. Within personal epistemology research, there are two classic stage theories. The first is Perry's model of epistemological development, and the second is the reflective judgment model developed by King and Kitchener (1994, 2004). Both theories propose seven stages of epistemological development, advancing in roughly the same way through three major phases. The first is an absolutist perspective in which knowledge is seen as objectively in the world, knowable with certainty, and derived from authoritative sources. The second phase is an unmoored relativism in which knowledge is uncertain, authoritative sources are untrustworthy, and all are free to believe what they wish. The resolution to this relativism is an evaluative stance that concedes that knowledge is constructed, and not knowable with absolute certainty, but asserts that knowledge claims can be evaluated according to standards of reason and evidence. That is, it is possible to say that some knowledge is more justifiable than others. These two theories differ in important respects. Notably, we owe the original conception of epistemic cognition to Kitchener (1983) and her argument that epistemic cognition is a kind of
ISLS
348
ICLS2012
processing reserved for ill-structured problem solving where gauging one's progress requires some idea of the form of an answer and appropriate processes for generating such an answer. In this sense, King and Kitchener's theory is much more clearly and cleanly epistemic than Perry's. There are a number of problems with these theories. One is that the stages at the lower and upper ends of the continuum are hypothesized but rarely, if ever, seen empirically. Perry and King and Kitchener propose that epistemological development may not even start until adolescence, making it quite difficult to reconcile their accounts with related developmental research in theory of mind and causal inference, both of which are inherently epistemic domains. While Piaget's theory has obviously been enormously influential, and beneficial, for the study of human development, central aspects of the original theory have not fared well within developmental psychology. Specifically, the basic notion of stages was largely abandoned by developmentalists decades ago, with Susan Carey (1986) going so far as to call Piaget's stage theory "fundamentally misleading." Theoretically, then, it seems unlikely that epistemic cognition would be the only area of cognitive development that proceeds in a stage-like manner. Empirically, these stage theories simply fail to explain available data. Research in science education on students' ideas about the epistemology of science reveals largely that these ideas are inconsistent and unstable (Sandoval, 2005), in contradiction to stage theory predictions. King and Kitchener themselves found in their own studies that subjects can rarely be placed at a single stage (King & Kitchener, 2004). Stage theories predict a global coherence to epistemic beliefs and epistemic cognition that is simply not found in empirical studies.
Neo-Piagetian developmental theories

The neo-Piagetian perspective characterizes development as a trajectory of increasing capacity or sophistication, but one that relies not on changes in underlying cognitive structure in the sense meant by Piaget, but on changes in knowledge. Contrary to Piaget, contemporary developmental psychologists argue that children in fact do think like adults, they simply have less experience and knowledge to bring to bear. Thus, changes in how children think are the result of changes in what they know and how they organize that knowledge, rather than changes in basic cognitive structures or mechanisms. Several personal epistemology models fit this approach, abandoning strong stances on general stagelike changes in favor of developmental trajectories (Baxter Magolda, 1992; Belenky, Clinchy, Goldberger, & Tarule, 1986; Hallett, Chandler, & Krettenauer, 2002; Hofer, 2000; Kuhn, Cheney, & Weinstock, 2000). These models differ in their generality, with some reflecting domain-specific aspects of models of expertise developed in cognitive science. These models share the progressive telos of the classic stage theories above, and the progression from absolutism through relativism to evaluativism. Both Kuhn and Hofer advance different theories about domain-specificity, but agree that within a domain epistemic development advances in the same general direction. A big improvement of these models over the classic stage theories is they directly concern themselves with young children and the lifespan of development. Yet, these neo-Piagetian theories suffer from the same inability to explain empirical discrepancies. Specifically, these theories predict that within a single domain, people can be attributed to a single, coherent epistemological stance (i.e., an absolutist), but this does not seem to be the case (Bromme, Kienhues, & Stahl, 2008; Rosenberg, Hammer, & Phelan, 2006; Sandoval, 2005). Chandler has argued that the developmental trajectory may be recursive, that children may cycle through these levels twice (Chandler, Hallett, & Sokol, 2002), but evidence for this claim seems lacking. The abundant evidence of a general developmental trend from absolutism toward evaluativism has been characterized as an effort to coordinate objective and subjective components of knowing (Kuhn, 2001). The discrepant evidence of clear developmental trajectories within or across specific domains presents a dilemma, and the apparent developmental trend from absolutist to relativist/multiplist, to evaluativist may be an artifact of the instruments researchers use to assess epistemological stances, as has been the case in studies of epistemological views of science (Lederman, Wade, & Bell, 1998). A situated approach to the study of epistemic cognition provides a more grounded way to explicate possible developmental trajectories and link such development to particular kinds of experiences.
Belief Systems Models

In contrast to frameworks proposing developmental trajectories, there are now several personal epistemology models that can be characterized as "belief systems" models. These frameworks are heavily influenced by information processing views on cognition, specifying a variety of inputs into the system, modules of the system and their interactions, and proposed outputs from the system. Interestingly, the belief systems models in personal epistemology rely almost exclusively on survey instruments to measure proposed components of the epistemological belief system. These models share an emphasis on two major components of epistemology (Hofer & Pintrich, 1997): the nature of knowledge and the nature of knowing. The knowledge component is conceptualized in terms of two dimensions, the certainty of knowledge and the complexity of knowledge. The knowing component is also defined in two dimensions: the source of knowledge and the means of justification.
ISLS
349
ICLS2012
This multi-dimensional systems approach was initiated by Schommer (Schommer, 1990; SchommerAikins, 2004), who posited five (later four) separate dimensions that, she argued, can develop independently. Schommer's approach to detecting and measuring these dimensions involved the development of a survey, the Epistemological Beliefs Questionnaire, that has been widely used throughout the world. Schommer's instrument and its many variants, however, consistently fail to produce the dimensions this model predicts (Clarebout, Elen, Luyten, & Bamps, 2001; DeBacker, Crowson, Beesley, Thoma, & Hestevold, 2008; Qian & Alvermann, 1995). These and many other studies (including Schommer's own as discussed by both Clarebout et al. and DeBacker et al.) cast a great deal of doubt on this dimensional model as an adequate explanation of epistemic beliefs and their development. It is surprising, in fact, that this survey remains popular given its well-documented problems. One response to these issues has been to propose systems models that attempt to account more explicitly for a multitude of other factors that might influence epistemic cognition. Schommer, now SchommerAikins, proposed expanding her model such that explicitly epistemic beliefs are just one component in a complex network of belief systems that might influence learning (Schommer-Aikins, 2004). Bendixen and her colleagues have proposed a variety of process models to attempt to characterize epistemic belief change and development (Bendixen & Rule, 2004; Muis, et al., 2006). So far, the complexity of these models appears to have resisted empirical test, nor have these models been applied to the explanation of extant data. Moreover, it is far from clear that the survey methods favored by researchers in this school are capable of explicating processes of change. Altogether, then, the prospects of systems models to explain epistemological development are cloudy, at best. Taken altogether, the variety of theoretical models proposed to explain epistemological development generally fail to explain the full range of relevant data, either generated within personal epistemology studies or related findings from science education or other disciplines. They all struggle to explain the widely documented variability of epistemic cognition within and across domains (Rosenberg, et al., 2006).
Locating epistemic belief within situated activity

A situative perspective on cognition (Greeno & The Middle School Mathematics Through Application Project Group, 1998; Lave & Wenger, 1991) provides a way to explain available data on epistemological development and account for conflicting evidence on the stability and coherence of epistemic cognition. This perspective explains cognition as tightly bound to social and material aspects of particular situations, thus variability across situations, rather than across broad domains, would be expected. The situated view argues not only that individual thinking and reasoning depend upon and are inherently mediated by social and material resources, but that these resources are themselves historical, cultural artifacts. On this view, epistemic beliefs emerge from and are linked to particular forms of activity. It may also be the case that such beliefs are not really beliefs in the sense that they are stable cognitive entities. Instead, what people express as beliefs about epistemic matters may be ad-hoc constructions built from networks of finer-grained, i.e., more situated, cognitive resources (Redish & Smith, 2008). The most important consequence of this characterization is that it is unlikely that potential epistemic beliefs can be straightforwardly "read" from people's responses to particular questions or performances on particular tasks. Rather, what such performances provide is a glimpse on some candidate resources that people bring to bear in the particular situation. This implies that research intended to characterize epistemic cognition must fundamentally be involved in triangulation across a variety of situations and performances, and should be skeptical that a person's response to a single task a survey, an interview, or some problem directly indicates a stable belief. Consequently, efforts to understand epistemic cognition and its development should be grounded in studies of how people go about constructing knowledge for themselves, and their efforts to evaluate knowledge claims made by others (including textbooks, newspapers, TV talking heads, teachers, their parents, or anyone else). Building knowledge for oneself and evaluating knowledge claims from the world both entail engaging in particular cultural practices tied to particular situations (at particular points of time). How people participate in these practices, and how they may think about their participation and its purposes, provides a window into the potential epistemic beliefs they bring to bear on particular kinds of situations. Consequently, epistemic beliefs can be located, so to speak, by triangulating among three facets of such participation, as shown in Figure 1. One important way to understand the epistemic ideas that people bring to bear is to examine their participation in practices of knowledge evaluation and construction. Changes in the form of participation are indicators of changes in the meaning that individuals make of the activity in which they are engaged. For example, changes in children's forms of participation in mathematical activity indicate changes in their ideas about what is mathematics (Cobb, Stephan, McClain, & Gravemeijer, 2001). Children's changing arguments about science topics over time suggests the internalization of certain norms for argumentation, and thus changed ideas about what it means to have a good scientific explanation for something (Rosebery, Warren, & Conant, 1992; Warren & Rosebery, 1996). Change in participation can indicate a shift in epistemic perspective, but it is the shift itself that suggests what particular epistemic ideas are brought to bear in the first place.
ISLS
350
ICLS2012
Studies of participation raise the issue of the unit of analysis, the unit to which epistemologies may be attributed. Rosenberg et al. (2006), for example, present a compelling case study demonstrating how a group of students shift their epistemic stance over the course of a single 15 minute episode. Their analysis highlights how changes in the framing of an activity shift participation, and the consequent epistemic resources that may be deployed in the situation. Their analysis draws on the epistemological resources theory's attempts to articulate epistemic modes, or frames constructs that attempt to characterize the interactional contexts in which a person may or may not deploy particular epistemic resources (Hammer, Elby, Scherr, & Redish, 2005). It should be stressed, though, that this is an analysis of social epistemology (Fuller, 1988) the epistemology of the groupin-activity. Individuals within the activity can be seen to contribute potential resources to the activity, but their uptake is a group effect. Attributing particular epistemic beliefs or orientations to individuals during single episodes of activity is fraught, as demonstrated by the conflicting interpretations of the same episode by (Hammer, Russ, Mikeska, & Scherr, 2008) and (Sandoval, 2008). Assessing the stability or coherence of epistemic cognition of individuals seems to require analyses of their participation across a range of similar and varied contexts over time.
Figure 1. Facets of epistemic activity within which epistemic beliefs can be located. Participation in knowledge construction and evaluation often produce artifacts that reflect epistemic commitments of the practice and participants. In particular, variations in the form of similar kinds of artifacts can indicate differences in the epistemic criteria used to create them. Within science education, for example, there has been a focus over the last decade on student construction of models and arguments. Looking at the models and arguments that children make can tell us something about what they think makes a good model or a good argument. Analyses of secondary students' written explanations, for example, suggest that explicit justification to link evidence to claims is not commonly recognized as an epistemic criterion for scientific explanations (McNeill, Lizotte, Krajcik, & Marx, 2006; Sandoval & Millwood, 2005). Studies of participation and artifacts can each shed some light on the epistemic beliefs people apply to efforts to make knowledge. Yet, people's participation in constructing knowledge artifacts is highly telegraphic. A crucial way that Chinn et al. (2011) have expanded the conceptualization of epistemic cognition is to include notions of epistemic aims and values. Epistemic aims are the goals people pursue through knowledge practices within a particular context, and epistemic values refer to the value that achieving such aims has for people. Analyses of participation or artifacts are insufficient to uncover the epistemic aims people may pursue. A student may fail to justify how some piece of data supports a claim because they do not know it is an important aspect of satisfying the epistemic aim of building a justified explanation, or they may hold a different epistemic aim. Chinn and colleagues also discuss epistemic virtues, which they describe as stable dispositions toward knowledge (e.g., skepticism or open-mindedness). These too are unlikely to be uncovered simply by looking at how people participate in knowledge construction or evaluation. Consequently, it is necessary to devise ways to have people reflect on their practices of knowledge construction and evaluation in order to produce a more complete account of epistemic belief. Such reflection is itself a particular cultural activity, of course, and has to be interpreted as such; i.e., it is a particular form of participation. Reflection here refers to reflection on one's participation and the artifacts produced by it. It seems critical that any attempt to ask people what they think about knowledge and knowing has to be grounded in situations close to their own experience. As an example, Metz (2004, 2011) has engaged quite young children, 5-7 years old, in sustained sequences of guided scientific inquiry, such that children take on increasing responsibilities for various roles and ultimately design and conduct their own investigation into some phenomenon (e.g., cricket behavior). They represent their work in research posters that then become the props for reflective interviews that focus primarily
ISLS
351
ICLS2012
on the children's judgments of the certainty of their results and conclusions. Metz has shown that these children are quite sensitive to sources of potential error in their own work, and consequently regard their conclusions as tentative. It may be the case that these children also believe they could achieve certainty if they corrected these errors, Metz' methods do not address this directly, but it remains that these very young children are sensitive to the idea that the epistemic status of particular claims is tied to the means of generating the claim. Clearly, to characterize these children's ideas about knowledge to be either "certain" or "simple," as most instruments of personal epistemology would do, or to characterize them as "absolutists," misses the point. Such characterizations are gross simplifications of the epistemic resources children bring to bear in this situation, and provide little explanatory power to account for children's reasoning. For another example, Sandoval and am (2011) asked 8 and 9 year olds to choose which story character had the "best reason" for deciding whether or not to believe a causal claim. When given the choice between data, even inconclusive data, and an authority figure, children were overwhelmingly more likely to choose the character that had data as having the better reason. At the same time, their reason was commonly based on the credibility that data gave to the story character. This suggests that children ascribe epistemic authority at least partially on the basis of evidence and not simply to an actor's status as an authority. These studies and others (Lehrer, Schauble, & Lucas, 2008; Rosenberg, et al., 2006; Ryu & Sandoval, in press) suggest children have developed a variety of productive epistemic resources even by the time they start school that can be triggered and deployed appropriately in contexts framed to make their use meaningful for achieving epistemic aims.
Conclusions
Theories, and consequent research, in personal epistemology must adopt a more situated perspective on epistemic cognition to productively confront and explain empirical findings from a variety of fronts. Piagetian and neo-Piagetian developmental theories predict consistency and coherence that is not seen in empirical studies. Belief systems models inspired by cognitive information processing perspectives fail their own efforts at empirical validation, or appear too unwieldy to enable empirical work. The epistemological resources framework of Hammer and Elby provides one starting point for a situated theory of epistemological development, and the recent re-conceptualization of epistemic cognition offered by Chinn and colleagues expands the purview of personal epistemology by fruitfully including epistemic aims as part of the context of epistemic cognition. The contribution offered here is a methodological approach for locating epistemic beliefs, conceived as the cognitive consequences of epistemic cognition, within situated activity by triangulating between multiple forms of analysis. I have presented some examples of such work, even if the researchers doing it may not identify themselves as personal epistemology researchers. At heart, the argument here is a plea for researchers of epistemological development to seriously attend to the demands of understanding how people actually go about trying to produce knowledge for themselves, and to adopt methods that can meet those demands. From the situated perspective outlined here, surveys or interviews of epistemological belief must be seen as particular, and highly unusual, cultural activities where the potential for participants to talk past each other is quite high. They hide the richness and nuance of the epistemic resources children can muster, and which provide starting points for instruction and development.
References
Astington, J. W., Pelletier, J., & Homer, B. (2002). Theory of mind and epistemological development: the relation between chidlren's second-order false-belief understanding and their ability to reason about evidence. New Ideas in Psychology, 20, 131-144. Baxter Magolda, M. B. (1992). Knowing and reasoning in college. San Francisco: Jossey-Bass. Belenky, M. F., Clinchy, B. M., Goldberger, N. R., & Tarule, J. M. (1986). Women's ways of knowing: the development of self, voice, and mind. New York: Basic Books. Bendixen, L. D., & Rule, D. C. (2004). An integrative approach to personal epistemology: A guiding model. Educational Psychologist, 39(1), 69-80. Bromme, R., Kienhues, D., & Stahl, E. (2008). Knowledge and epistemological beliefs: an intimate but complicated relationship. In M. S. Khine (Ed.), Knowing, knowledge, and beliefs: Epistemological studies across diverse cultures (pp. 417-435): Springer. Carey, S. (1986). Cognitive science and science education. American Psychologist, 41(10), 1123-1130. Chandler, M. J., Hallett, D., & Sokol, B. W. (2002). Competing claims about competing knowledge claims. In B. K. Hofer & P. R. Pintrich (Eds.), Personal Epistemology: The Psychology of Beliefs about Knowledge and Knowing (pp. 145-168). Mahwah, NJ: Erlbaum. Chinn, C. A., Buckland, L. A., & Samarapungavan, A. (2011). Expanding the dimensions of epistemic cognition: Arguments from philosophy and psychology. Educational Psychologist, 46(3), 141-167. Clarebout, G., Elen, J., Luyten, L., & Bamps, H. (2001). Assessing epistemological beliefs: Schommer's quesionnaire revisited. Educational Research and Evaluation, 7(1), 53-77.
ISLS 352
ICLS2012
Cobb, P., Stephan, M., McClain, K., & Gravemeijer, K. (2001). Participating in classroom mathematical practices. Journal of the Learning Sciences, 10(1&2), 113-163. Cole, M. (1996). Cultural psychology: a once and future discipline. Cambridge, MA: Belknap Press. DeBacker, T. K., Crowson, H. M., Beesley, A. D., Thoma, S. J., & Hestevold, N. L. (2008). The challenge of measuring epistemic beliefs: An analysis of three self-report instruments. Journal of Experimental Education, 76(3), 281-312. Engestrm, Y. (1987). Learning by expanding: an activity-theoretical approach to developmental research. Helsinki: Orienta-Konsultit. Fuller, S. (1988). Social epistemology. Bloomington, IN: Indiana University Press. Greene, J. A., Torney-Purta, J., & Azevedo, R. (2010). Empriical evidence regarding relations among a model of epistemic and ontological cognition, academic performance, and educational level. Journal of Educational Psychology, 102(1), 234-255. Greeno, J. G., & The Middle School Mathematics Through Application Project Group. (1998). The situativity of knowing, learning, and research. American Psychologist, 53(1), 5-26. Hallett, D., Chandler, M. J., & Krettenauer, T. (2002). Disentangling the course of epistemic development: parsing knowledge by epistemic content. New Ideas in Psychology, 20, 285-307. Hammer, D., & Elby, A. (2002). On the form of a personal epistemology. In B. K. Hofer & P. R. Pintrich (Eds.), Personal Epistemology: The Psychology of Beliefs about Knowledge and Knowing (pp. 169190). Mahwah, NJ: Erlbaum. Hammer, D., Elby, A., Scherr, R., & Redish, E. F. (2005). Resources, framing, and transfer. In J. Mestre (Ed.), Transfer of learning from a modern multidisciplinary perspective (pp. 89-120). Greenwich, CT: Information Age Publishing. Hammer, D., Russ, R., Mikeska, J., & Scherr, R. (2008). Identifying inquiry and conceptualizing students' abilities. In R. A. Duschl & R. E. Grandy (Eds.), Teaching scientific inquiry: Recommendations for research and application (pp. 138-156). Rotterdam, Netherlands: Sense. Hofer, B. K. (2000). Dimensionality and disciplinary differences in personal epistemology. Contemporary Educational Psychology, 25(4), 378-405. Hofer, B. K., & Bendixen, L. D. (in press). Personal epistemology: Theory, research, and future directions APA Handbook of Educational Psychology. Washington, DC: APA. Hofer, B. K., & Pintrich, P. R. (1997). The development of epistemological theories: beliefs about knowledge and knowing and their relation to learning. Review of Educational Research, 67(1), 88-140. Hofer, B. K., & Pintrich, P. R. (Eds.). (2002). Personal Epistemology: The Psychology of Beliefs about Knowledge and Knowing. Mahwah, NJ: Erlbaum. Khine, M. S. (Ed.). (2008). Knowing, knowledge, and beliefs: Epistemological studies across diverse cultures: Springer. King, P. M., & Kitchener, K. S. (1994). Developing reflective judgment. San Francisco: Jossey-Bass. King, P. M., & Kitchener, K. S. (2004). Reflective judgment: Theory and research on the development of epistemic assumptions through adulthood. Educational Psychologist, 39(1), 5-18. Kitchener, K. S. (1983). Cognition, metacognition, and epistemic cognition. Human Development, 26(4), 222232. Kitchener, R. F. (2002). Folk epistemology: an introduction. New Ideas in Psychology, 20(2/3), 89-105. Kuhn, D. (2001). How do people know. Psychological Science, 12(1), 1-8. Kuhn, D., Cheney, R., & Weinstock, M. (2000). The development of epistemological understanding. Cognitive Development, 15, 309-328. Lave, J., & Wenger, E. (1991). Situated learning: legitimate peripheral participation. Cambridge, UK: Cambridge University Press. Lederman, N. G., Wade, P. D., & Bell, R. L. (1998). Assessing the nature of science: what is the nature of our assessments? Science & Education, 7, 595-615. Lehrer, R., Schauble, L., & Lucas, D. (2008). Supporting development of the epistemology of inquiry. Cognitive Development, 23(4), 512-529. McNeill, K. L., Lizotte, D., Krajcik, J. S., & Marx, R. W. (2006). Fading scaffolds for argumentation and explanation. Journal of the Learning Sciences, 15(2), 153-191. Metz, K. E. (2004). Children's understanding of scientific inquiry: their conceptualization of uncertainty in investigations of their own design. Cognition and Instruction, 22(2), 219-290. Metz, K. E. (2011). Disentangling robust developmental constraints from the instructionally mutable: Young children's epistemic reasoning about a study of their own design. Journal of the Learning Sciences, 20(1), 50-110. Montgomery, D. E. (1992). Young children's theory of knowing: The development of a folk epistemology. Developmental Review, 12, 410-430.
ISLS
353
ICLS2012
Moshman, D. (2004). From inference to reasoning: the construction of rationality. Thinking & Reasoning, 10(2), 221-239. Muis, K. R., Bendixen, L. D., & Haerle, F. C. (2006). Domain-generality and domain-specificity in personal epistemology research: Philosophical and empirical reflections in the development of a theoretical framework. Educational Psychology Review, 18, 3-54. Packer, M. J. (2001). The problem of transfer, and the sociocultural critique of schooling. Journal of the Learning Sciences, 10(4), 493-514. Perry, W. G., Jr. (1970). Forms of intellectual and ethical development in the college years. New York: Holt, Rinehart & Winston. Qian, G., & Alvermann, D. E. (1995). Role of epistemological beliefs and learned helplessness in secondary school students' learning science concepts from text. Journal of Educational Psychology, 87(2), 282292. Redish, E. F., & Smith, K. A. (2008). Looking beyond content: Skill development for engineers. Journal of Engineering Education, 295-307. Rogoff, B. (1990). Apprenticeship in thinking: cognitive development in social context. New York: Oxford University Press. Rosebery, A. S., Warren, B., & Conant, F. R. (1992). Appropriating scientific discourse: findings from language minority classrooms. Journal of the Learning Sciences, 2(1), 61-94. Rosenberg, S., Hammer, D., & Phelan, J. (2006). Multiple epistemological coherences in an eighth-grade discussion of the rock cycle. Journal of the Learning Sciences, 15(2), 261-292. Ryu, S., & Sandoval, W. A. (in press). Improvements to elementary children's epistemic understanding from sustained argumentation. Science Education. Sandoval, W. A. (2005). Understanding students practical epistemologies and their influence on learning through inquiry. Science Education, 89, 634-656. Sandoval, W. A. (2008). Commentary: Exploring children's understanding of the purpose and value of inquiry. In R. A. Duschl & R. E. Grandy (Eds.), Teaching scientific inquiry: Recommendations for research and application (pp. 157-163). Rotterdam, Netherlands: Sense. Sandoval, W. A., & am, A. (2011). Elementary children's judgments of the epistemic status of sources of justification. Science Education, 95(3), 383-408. Sandoval, W. A., & Millwood, K. A. (2005). The quality of students use of evidence in written scientific explanations. Cognition and Instruction, 23(1), 23-55. Schommer, M. (1990). The effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82, 498-504. Schommer-Aikins, M. (2004). Explaining the epistemological belief system: Introducing the embedded systemic model and coordinated research approach. Educational Psychologist, 39(1), 19-29. Suchman, L., & Jordan, B. (1990). Interactional troubles in face-to-face survey interviews. Journal of the American Statistical Association, 85, 232-241. Vygotsky, L. S. (1978). Mind in society: the development of higher psychological processes. Cambridge, MA: Harvard University Press. Warren, B., & Rosebery, A. S. (1996). "This question is just too, too easy!" Students' perspectives on accountability in science. In L. Schauble & R. Glaser (Eds.), Innovations in learning: new environments for education (pp. 97-125). Mahwah, NJ: Erlbaum. Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge: Cambridge U. Press. Wertsch, J. V. (1985). Vygotsky and the social formation of mind. Cambridge, MA: Harvard U. Press.
ISLS
354
ICLS2012
Improving Revision in Wiki-based Writing: Coordination Pays off

Astrid Wichmann, Marina Becker, Nikol Rummel, Ruhr University Bochum, Universit tsstra 150, 44801 Bochum, Germany e Email: astrid.wichmann@rub.de, marina.becker@rub.de, nikol.rummel@rub.de Abstract: Despite the potential of wiki-based writing for learning, students struggle with collaborative writing especially when it is necessary to revise text-sections that are written by others. The study investigated whether collaboration scripts help improving students' revision activities and the quality of wiki-texts. We compared scripted (script+) with unscripted (script-) collaboration in a wiki-based writing setting that was adapted for educational purposes. Students from two university courses participated in a one week collaborative writing activity. Results showed that students in the scripted condition outperformed students in the unscripted condition with respect to revision behavior and text-quality. Results from analyzing students discussions during the writing activity revealed increased coordination concerning task division and increased communication frequency for students in the scripted condition. Results indicate that collaboration scripts may trigger coordination. Our findings suggest that wikis are suitable to foster writing skills in university settings if wikis are tailored to educational contexts.
Introduction
The following study explores ways to improve wiki-based writing by scripting students during main writing activities supporting coordination. Collaborative writing in online learning platforms has become a common activity in higher education and it is now a major ingredient of many Master Program requirements (e.g. Utrecht University, Uolu University, University of Helsinki). Based on the process model of writing developed by Flower and Hayes (1981), individual as well as collaborative activities take place during the main writing phases of planning, drafting and reviewing. Various technologies (Google Docs, MediaWiki, MoodleWiki) emerged that enable collaborative writing and support students in co-editing shared documents in collaboration with peers. Wiki-based writing settings provide excellent possibilities for learning, because students can asynchronously edit a wiki-text, monitor changes on the wiki-text and communicate with peers using discussion facilities. Often however, students are hesitant when it comes to revision activities. Especially revising text sections of others is a challenge for inexperienced writers. As a result, the quality of the joint text decreases and students miss potential learning opportunities. Students revision problems might have to do with increased coordination demands in collaborative writing. How can these higher coordination demands be met? Discussion facilities in wikis can facilitate the coordination of writing activities but often, the mere availability of discussion facilities is not sufficient. Instructional support is necessary for students to benefit from those facilities (Blakeslee, 1992). For instance, collaboration scripts provide support by structuring students interaction (Kollar, Fischer & Hesse, 2006; King, 2007). Collaboration scripts prescribe an optimal sequence of activities that specifies how and when a group of learners should work together. (ODonnell & Dansereau, 1992). For wiki-based writing settings, collaboration scripts seem particularly suitable, because they can help students structure their writing process by suggesting a task division resulting in an optimal sequence of individual and collaborative working phases (Hermann, Rummel, & Spada, 2001). Individual working phases are a central aspect during the main writing activities (planning, drafting and reviewing). A central goal of collaborative working phases is to communicate with peers, which is necessary for coordinating the main writing activities. We assume that following a collaboration script, affects not only the way students coordinate writing activities but also yields positive effects with regard to the quality of the wiki-text.
Collaborative Writing in Wikis

With the introduction of new collaborative writing technology into educational settings, collaborative writing has shifted towards a process with the focus on creating a joint product. In collaborative writing settings that are wiki-based, learners can access a shared document, which changes the way collaborative writing takes place. Earlier, the task of collaboratively producing a document included heavy exchange of email and a sequence of writing and reviewing until the document was finalized. Today, email exchange can be avoided using asynchronous collaborative writing technology such as wikis. In wiki-based writing, several users can collaborate as authors by remotely accessing and editing wiki-text. The wiki-text becomes a product with joint ownership, because authors do not only draft and edit sections created by them but also revise sections created by peers. Additionally, users have the possibility to discuss aspects related to the wiki-text, by adding comments in a simple discussion page. The original purpose of wikis was to offer software developers a tool that allows for communicating ideas concerning software design (Leuf & Cunningham, 2001). Recently, wikis have been investigated for educational purposes in the area of computer-supported collaborative learning research (Trentin,
ISLS 355
ICLS2012
2009). Wikis have been used for writing assignments, particularly in blended or distant learning settings in higher education (Minocha & Thomas, 2007). With wikis, learners collaborate by making changes to the wikitext, which can be observed by the learner itself as well as by co-learners (Weng & Gennari, 2004). Despite the agreed value of wikis in higher education, wikis place new demands on the learner. Therefore, wikis need to be adapted towards educational use taking into account those core processes.
Revisions in Wiki-Based Writing

As a starting point for characterizing wiki-based writing activities, we use the simple single-writing model by Hayes and Flower (1980): planning-drafting-reviewing. In wiki-based writing settings, these activities take part during individual or collaborative phases. Planning activities include reading relevant literature, generating ideas, outlining criteria for the text and planning the writing process. Drafting activities involve making notes, writing down ideas but they also include the production of first individual chunks of text. Reviewing activities consist of evaluating text and subsequently revising the text (for an extensive overview of the activities in collaborative writing see Neuwirth et al., 2000). The task of revising text needs to receive special attention, because students generally seem to have problems with revision activities (Proske, Narciss & McNamara, 2010). In collaborative writing, this problem is increased, because students are challenged to revise text parts that were not written by themselves but by co-authors. Existing research on wiki-based writing has shown that students tend to avoid editing or revising text of others (Judd, Kennedy, & Cropper, 2010). After contributing to the wiki-text individually, students are supposed to edit the text parts including revising text produced by peers. But students hesitate to restructure the shared text or to adapt existing paragraphs written by others. Often, contributions result in individually adding paragraphs into the existing wiki-text without revising the text written by peers. As a result, the overall text quality is suffering. According to writing expertise research, little text revision affects the text quality negatively. Lacking revision leads to less coherent text and more errors with respect to grammar and orthographical errors (Proske et al., 2010). Revising paragraphs of others in order to construct a coherent text is not only important with respect to text quality. Also from a learning perspective, text revision is of great importance, even though challenging. Students need to integrate multiple views to achieve a coherent text structure. This integration process often leads to cognitive conflicts. According to Kimmerle and colleagues (Kimmerle, Moskaliuk, & Cress, 2009), those cognitive conflicts resulting from the incongruity between the individual knowledge and the information in the shared document can lead to learning gains. However, they only can lead to learning gains if those conflicts can be reconciled (Chi, Siler, Jeoug, Yamauchi, & Hausmann, 2001). In wiki-based writing, reconciliation takes place through revision activities. Therefore, revision activities are very important for learning. One central question that guided the design of our study was: How can we adapt wiki-based writing settings to engage students in revision activities and to increase text quality?
Coordination
For collaborative activities to be successful, coordination is required of both, content (what are we agreeing to do) and process (how are we doing it) (Clark, 1996). Coordination strategies need to be applied to ensure a joint focus among co-authors, resulting in meaningful task division (who does what) during individual and collaborative working phases (Herrmann et al., 2001). Coordination in collaborative writing settings is required during all activities of the writing process. One way how coordination occurs is through communication using available communication channels such as discussion boards. In wiki-based writing settings, learners should discuss and agree on a writing process before starting the writing activities. A discussion board is helpful to reassign tasks and attune own activities to the activities of others. In addition, taking advantage of a discussion board is crucial, because it helps to resolve cognitive conflicts. This is important, because from a sociocognitive point of view, conflicting views are only beneficial for learning if conflicts can be resolved (Renkl & Mandl, 1995). Communicating while collaboratively writing a text might enable learners to overcome making superficial revision and to re-organize and rewrite paragraphs of peers for the purpose of integrating ideas and thus resolving cognitive conflicts. The extent of coordination that needs to take place is depending on several aspects. One aspect is the group size: In large groups including large numbers of authors, spending time with coordination is not helpful because negotiations become ineffective and authors start relying on decisions that were made in the first place not willing to change those decisions. Findings from Wikipedia research indicate that collaboration in large Wikipedia communities were less successful when heavy coordination was involved (Kittur & Kraut, 2008). However, coordination was more effective in small Wikipedia communities. Another aspect that affects the need for coordination is the duration of the activity. Findings from Wikipedia research indicated that authors tended to rely on coordination more heavily in the beginning of a writing process. As Wikipedia-authors developed a shared model over time, coordination was not necessary anymore (Kittur & Kraut, 2008). In the area of higher education, collaborative writing usually takes place in smaller groups and the time frame is smaller than in Wikipedia communities. Therefore, we assume that in higher education, coordination becomes
ISLS
356
ICLS2012
crucial for the process of writing and its outcome, the shared text. Another aspect affecting the amount of coordination is related to the way knowledge is distributed among co-writers. In a co-authorship where students share a joint/overlapping knowledge base that is relevant to the writing task at hand, little coordination might suffice. However, if students prior knowledge is disjoined but complementary with respect to the task, a higher amount of coordination is necessary. Task characteristics are yet another aspect influencing the amount of coordination needed. Depending on the theme and structure of a collaborative writing task, the writing requires different levels of prior knowledge integration. If a writing task demands the integration of multiple views, the collaborative writing process requires not only task division but also communication about how the perspectives can be described so that a coherent picture is presented. As a result the amount of coordination is increased. In sum, besides group-size and task duration, knowledge distribution and task characteristics determine how much it is necessary to coordinate the writing process.
Collaboration Scripts
Collaboration scripts could be a promising way to support writing in wiki-based settings, because they help structuring a learning process by dividing tasks resulting in an optimal sequence of individual and collaborative working phases (Hermann et al, 2001). Typically, collaboration scripts aim at supporting coordination among group members, by prompting particular types of cognitive, metacognitive and social activities (King, 2007). Research studies in the area of computer supported collaborative learning indicate that collaboration scripts can promote learners problem solving activities and knowledge acquisition (Kollar et al., 2006, ODonnell, 1999). For instance, Weinberger and colleagues (Weinberger, Stegmann, & Fischer, 2010) successfully scripted students to engage in a discussion. The collaboration script divided the task of argumentation into subtasks of first formulating an argument, then constructing a counterargument and so on. This structure was used cyclically to engage in a successful discussion. Rummel and Spada (2005) applied a collaboration script in a videoconference setting to support students of Medicine and Psychology collaborating on complicated patient cases. Scripted collaboration resulted in an improved diagnosis and treatment suggestion when compared to an unsupported collaboration control group. The collaboration script prescribed the separation of individual and collaborative work phases encouraging students to exchange information effectively. Yet, results on scripting effectiveness are not homogeneous. Some findings in the area of problem solving suggest that collaboration scripts are not always effective (Haake & Pfister, 2010; Kollar, Fischer, & Slotta, 2007; Stegmann, Weinberger, & Fischer, 2007). Under what conditions are scripts ineffective? Deiglmayr and Spada (2010b) found that in a collaborative problem solving activity that required collaborative drawing of inferences, a collaboration script did lead to improved performance. The authors concluded that for the purpose of collaborative inference drawing, a separation of activities was inappropriate. Concluding from the studies described above, it seems as though collaboration scripts are only effective under certain task conditions. First, collaboration scripts should be employed for tasks that can be divided into sub-activities. Second, collaboration scripts might be beneficial for tasks being carried out in a simple repeatable sequence (Deiglmayr & Spada, 2010a). Therefore, it is our assumption that collaboration scripts can support learners in wiki-based writing settings effectively, because in wiki-based writing, the task can be divided in activities of planning, drafting and reviewing (which includes revising) and these activities can be cyclic depending on the duration and complexity of the writing task. Because wiki-writing activities are divisible and can be carried out cyclically, we think that collaboration scripts are suitable to support students revision activities and to improve overall quality of produced wiki-texts.
Aims
Based on the idea that coordination is an important precondition for university students to engage in revisions of peer authored text in a wiki setting, we hypothesized that providing coordination support in the form of a collaboration script would help students to engage in effective revision behavior. We further hypothesized that the improved revision behavior would have a positive impact on the overall wiki-text quality. More generally, we were interested in the students perception of the wiki-based writing setting.
Method
Participants
Participants were freshmen university students enrolled in BA education courses. All students were recruited from two introductory academic writing courses (class sizes: 35 and 38 students, respectively) at the institute of educational research. The courses were taught by different instructors.
Design
Students were randomly assigned across courses to one of two conditions, script+ (n=36) and script- (n=37) condition. Students participated in a wiki-based writing task, which included the activities planning, drafting
ISLS
357
ICLS2012
and reviewing. Altogether, 23 groups worked in triads and 1 group (script-) consisted of four students. 12 groups worked with a collaboration script (script+) and 12 groups worked without a collaboration script (script-). Distributed prior knowledge was invoked by instruction (Figure 1). Every student in a triad read a different article (except the group with four students in which 2 students received the same article for reading). The topic of the writing assignment was chosen so that the knowledge of all three students was relevant, hence knowledge was complementary. The task was to write a summary on the topic of schools contribution to student development. Each wiki-text included a pre-structure that required students to not only list but integrate their prior knowledge. The students were asked to write not more than 1000 words for the whole assignment. Article A Article B Article C Student I Student II Student III Shared document Threaded discussion board Wiki Script Worked example Versioning
Figure 1. Article assignment in every group
Procedure
Prior to the planning activity, the pre-questionnaire was administered. With start of the planning activity, every student was assigned to one article via email including the topic of the writing assignment. Students had one week to read the article they were assigned to. After the planning activity, the wiki access code for each student was sent personally via email. Access codes were anonymised by providing login names consisting of random letters and numbers (e.g. AB3). Anonymisation was done to ensure that students could not communicate with each-other outside the wiki setting. Every student only had access to one wiki-text that they shared with their peers. Students worked for one week on the wiki-text in their groups. Halfway through the week, an email was sent to everyone, to encourage the writing and reminding that the deadline was two days later without extension. After the deadline, the wikis were not accessible anymore for the students. During the following week, the postquestionnaire was administered and afterwards the writing experience using the wiki was discussed in each course separately.
Wiki-based Writing Setting

We used MediaWiki (Barrett, 2009) for the wiki-based writing setting. MediaWiki software, best known from Wikipedia, includes three functionalities that are crucial for collaborative writing: article, versioning and talk pages. A MediaWiki may include several articles. An article can be modified using formatting attributes such as titles or hyperlinks. Multiple authors can login to the wiki-system and view as well as edit an article. Furthermore, authors can use the versioning functionality of MediaWiki to observe and compare revisions of the article. The versioning functionality tracks every revision made. Considering that author I edited an article and saved it, author II can view the changes by comparing the previous version of the article with the current one. The third functionality, the talk pages, is designed to support rudimentary communication. MediaWiki is not designed for educational purposes; especially the talk pages have their limitations when using it in an educational setting. Talk pages allow the user communicate with other users by adding comments (Cunningham, 2006). For educational purposes, talk pages are insufficient, because they provide little means to structure a discussion, to reply to others and to lead topic-related discussions. Hence, we drastically re-designed the standard MediaWiki for short-term writing tasks in university settings. Each wiki that was accessed by a group, included 6 tabs: 1. Text display, 2. Editing 3. Discussion board, 4. Worked example, 5. Versioning, 6. Wiki-help. 1. Text display: The text display showed the wiki-text document. This included a pre-structure consisting of 6 Titles and references were listed at the end of the page. We included a pre-structure because the pre-structure required the learners to integrate their prior knowledge (instead of just listing different views). 2. Editing: The editing of the wiki-text document was restricted to linear text only. Hyperlinks were excluded. 3. Discussion board: Instead of the talk pages being used in Wikipedia, a threaded discussion board was included. The threaded discussion board is an extension of MediaWiki called LiquidThreads (Garrett, 2009). Users can create threads, which can be nested and sorted by date. Below every entry, the user can press a reply button, which conveniently creates a new entry that is associated with the thread replied to. Every entry in a threaded discussion is automatically signed and dated. New threads can be started depending on the topic or subject of discussion. 4. Worked example: This tab was dedicated to display four rules and examples for writing. For every rule, one positive and one negative example were included. The rules referred to writing problems that reduce text quality, specifically lack of logic (rule A), nested sentences (rule B), erroneous expressions (rule C) and grammar errors (rule D). 5. Versioning: The versioning functionality was the same as in Wikipedia.
ISLS
358
ICLS2012
6. Wiki-help: A help page was added to facilitate the technical handling of the wiki.
The Collaboration Script

In the script+ condition, the collaboration script was displayed in the wiki adjacent to the discussion board. The collaboration script included instructions for interaction, specifically a description of an optimal sequence with respect to task division resulting in individual and joint working phases. The main sequence consisted of the following activities: text production, text review and text revision. The collaboration script suggested a sequence in which these activities would be carried out best by each group within the time frame. For every day of the week, one activity was suggested: For day one, the students were instructed to agree with their co-authors on a schedule for the week. It was suggested to spend the first day producing their individual text parts. For day two, the students were asked to finalize their individual text part in the wiki and to start reading the sections of their co-authors. Day three and four focused on revising the overall text. The following sequence was suggested for the revisions: One person selects a sentence that needs to be revised and pastes that sentence into the discussion board. The next person edits the sentence. The next person takes the sentence and integrates it into the wiki-text. For day three, the script suggested to focus on writing rules of the worked example A and B. For day four, rules C and D. Day five instructed the students to reread the wiki-text and to discuss and resolve open issues and to revise the wiki-text again. Students had to distribute the activities by themselves. The reviewing activities (review & revision) were suggested to take place in two cycles during the available time. In addition, the students were encouraged to discuss problems and read the worked example that was included in both conditions, script+ and script-.
Questionnaires
The pre-questionnaire included two sections. Section A included items regarding computer attitudes (Likert scale in which 1= Dont agree at all and 5= I absolutely agree) and section B assessed experiences with the computer (Likert scale in which 1= never and 5= very frequently). Reliability was .851 and .685, respectively. The post-questionnaire included several sections including one open-response item and several Likert-type items (Likert scale in which 1= Dont agree at all and 5= I absolutely agree). The first section asked students to let us know whether they communicated with their co-authors outside the wiki. Perceived use was assessed for the main functionalities of the wiki respectively, those are: text editing, discussion board, worked example and versioning. Cronbachs Alpha of the perceived use scale was .875.
Content Analysis
A content analysis was performed on both, the wiki-texts and discussion contributions. All content analyses were based on frequencies. The wiki-text was analyzed by looking at contributions per student and by looking at the wiki-text of each triad. Per student analysis included frequency measure with respect to number of characters written and number of peer edits per student for all 73 students. Number of characters was measured using a customized MediaWiki user contribution page, which includes revision history and difference between revisions. This page offers menus to select and filter contributions and edits by every student separately. We investigated students revision behavior by counting the number of times a student editing text-sections. Edits included the operations move, delete and add. Every operation was counted equally as an edit. Peer edits per student included only edits done on sections that were written by peers, to tease apart the difference between edits on own sections and peer-edits. Two raters used MediaWiki user contribution page to analyze the wikitexts (Krippendorfs = .98). Content analysis was also done for all 24 wiki-texts. The unit of analysis for the wiki-text was the document itself as only frequencies were calculated. Text quantity was calculated based on the number of characters included in a wiki document. Text quality was evaluated on the dimensions grammar and orthographical errors as well as coherence. We used MS WORD spell checker functionality as a technique for checking grammar and orthographical errors. As an indicator for coherence, we looked for evidence to integrate perspectives induced by the articles that the students read. Indicators were for instance In contrast to author II, Similar to, Different than author I, author II views. These indicator words establish coherence by comparing and contrasting perspectives (Proske et al., 2010). Calculating interrater reliability between two raters resulted in an Alpha of .96 (Krippendorf). For manipulation check purposes, we looked at whether students took into account the collaboration script during discussions, particularly discussions about task division. We counted indicators for task division agreements with respect to time and task (Krippendorfs = .92). In addition, discussion contributions were analyzed in terms of number of contributions by counting the discussion contributions in all 24 wikis.
Results
From 73 course participants 8 students logged in to the wiki system, but did not participate at all in the wikibased writing activity. Hence, they were excluded from further analysis resulting in n = 65. Not all participants
ISLS
359
ICLS2012
took the pre- and post-questionnaire and some filled in the questionnaire only half-way. Therefore, the number of participants varies for every analysis. Results from the pre-questionnaire indicate that there are no significant differences between students in script+ and students in script- condition with respect to computer attitudes (F(1; 54) = .219, p = .64; partial 2 = .004) and computer experiences (F(1; 54) = .002, p = .97; partial 2 = .000). Wiki-texts in the script+ condition were on average a bit longer than wiki-texts in the script- condition but this difference did not reach statistical significance (Table 1). All students reported that they solely relied on the wiki for communication purposes. Hence, no other communication channels (e.g. chat, phone) were used. Results from the content analysis: Groups in both conditions wrote on average 10344 characters per wiki (SD = 2486). Since the writing instruction included a word limit, there were no differences between conditions concerning number of characters per wiki (M = 10901, SD = 2343 for Script+; M = 9787, SD = 2600 for Script-). Results (Table 1) from a MANOVA on the individual dataset showed differences between students behavior. Students in the script+ condition wrote significantly more text than students in the script-condition (F(1; 65) = 4.61, p = .04; partial 2 = .068).With respect to text revision, students in script+ condition engaged in more editing of peer-authored wiki-text sections (F(1; 65) = 4.60, p = .04; partial 2 = .068). A second MANOVA was calculated on the group data-set for differences between wiki-texts. With respect to text quality, texts in the script+ condition showed significantly more efforts of integrating perspectives (F(1; 24) = 8.92, p = .01; partial 2 = .288). Wiki-texts in the script+ condition included less grammar and orthographical errors but difference did not reach statistical significance. From analyzing discussion-board contributions, we have evidence that implementing the collaboration script was successful. Overall, there was a significantly higher occurrence of discussion contributions in the script+ condition (F(1; 24) = 6.37, p = .02; partial 2 = .224). From analyzing the content of the discussions, we found that task division mentions were more frequent in the script+ condition than in the script- condition (F(1; 24) = 5.20, p = .03; partial 2 = .191). Table 1. Mean frequencies Script+ vs. ScriptScript+ Unit Individual Wiki-text level Coding category Nr of characters Nr of peer edits (revision) Nr of perspective integration (coherence) Nr of grammar & orthographical errors Nr of contributions Discussion contributions Nr of task division mentions M SD M ScriptSD
4157.45 1660.78 3383.65* 1230.91 2.58 10.50 5.58 45.75 8.58 2.56 5.73 5.62 31.02 4.46 1.35* 4.75* 7.83 19.67* 4.58* 2.04 3.41 5.75 17.89 4.12
Wiki-text Grouplevel
* Significant difference, p<0.05 Results from post-questionnaire: We were interested in students perceptions concerning the wiki-based writing setting. For the purpose of using a wiki for educational purposes, we drastically adapted the MediaWiki functionality towards students needs. The post-questionnaire looked at differences of perceived use with respect to the adapted wiki functionalities (Text editing, discussion board, worked example and versioning). On average, students perceived working with the editing functionality (M=3.45, SD=.79) and working with the discussion board (M=3.28, SD=.91) as useful. For instance, a high percentage of students reported that editing functionality (70.9 %) and discussion board (75%) were helpful. The worked example (M=2.84, SD=.70) and versioning (M=2.91, SD=.61) were perceived as less useful. Only 25% agreed that the worked example was helpful and only 27.8 % agreed that versioning was helpful.
Discussion
Based on our assumption that coordination is a prerequisite for students to revise peer authored texts effectively, we investigated whether the provision of a collaboration script improves revision behavior of students. Furthermore, we measured whether providing a collaboration script also affects the overall text quality. Our findings indicate that students benefit from the collaboration script: We found that following the script resulted in increased revision behavior. Hence, the collaboration script seems to have counteracted students hesitation to rewrite texts of others. Based on literature in academic writing performance (Proske et al., 2010), revision
ISLS
360
ICLS2012
behavior is an important factor that affects the quality of texts positively. Indeed we were able to show that not only revision behavior, but also the quality of wiki-texts increased for students who had a collaboration script available. Also from a learning perspective, supporting revision behavior is important, because it provides opportunities to resolve cognitive conflicts. Another assumption was that discussion boards alone are not sufficient for wiki-based writing in educational settings. We assumed that the sole availability of a discussion board would not be sufficient to meet students coordination demands. An analysis of students discourses revealed that the collaboration script in combination with the discussion board resulted in increased coordination activities: Students not only contributed more to the discussion board, they also used the discussion board for coordinating task division, which was the intended purpose of the collaboration script. Our overall goal was to tailor wiki-based writing settings to educational contexts. Besides the experimental variation, we wanted to know if our adaptations of the MediaWiki were perceived as useful. Findings show that students perceived the main components, the wiki-text and wiki-editing as useful. Based on questionnaire results, the LiquidThread implementation (Garrett, 2009) as an alternative to wiki talk pages was perceived as useful. In contrast, the worked example was perceived as less useful. This result is confirmed by the contents that were focus of discussion in the discussion board. Only in two wikis, the presence of a worked example was mentioned. One reason might have to do with the fact that the worked example was not adjacent to the wiki-text but situated in another tab. Hence, students always had to switch between tabs and did not have the wiki-text and the worked example side by side. Situating the worked example adjacent to the wiki-text might have resulted in a higher usefulness of the worked example. The versioning tab, which was implemented as known in Wikipedia, was also not perceived as useful and not mentioned at all in the discussions. Future efforts should think of better ways to include the versioning.
Limitations and Future Questions

We were able to show that our collaboration script is suitable for the wiki-based writing setting described in the paper: Wikis provide the possibility to collaborate asynchronously, that is, only one author at a time can make changes in the wiki-text. Asynchronicity leads to a very procedural writing process in which every author has to wait until a change by a co-author has been made. Therefore, a fixed collaboration script was appropriate for this writing task. For collaborative writing settings that allow synchronous access such as Google Docs, the drafting and revision process becomes much more intertwined. A separation of activities is difficult and might be ineffective (Deiglmayr & Spada, 2010b). Hence, in synchronous collaborative writing settings, a fixed collaboration script might not be sufficient. Support that is adaptive to the authors writing activity might be more appropriate. The study was conducted in a field setting controlling conditions as much as possible. Some conditions of the study design were unrealistic in favor of controlling the setting. For instance, under common university course conditions, students might be in contact with their co-authors from the beginning and therefore start negotiating criteria for the wiki-text already during the planning activities. In our study, participants were not able to communicate with their co-authors until the planning activity was finished. Hence, the planning activity was determined to be individual work. Finally, the study just took a snapshot of a writing process. It included just two cycles of reviewing. Common writing assignments include longer and more frequent cycles of reviewing and editing. Future research on wiki-based writing should look at changes of coordination during longer periods of time.
References
Barrett, D. J. (2009). MediaWiki. Beijing: O'Reilly. Blakeslee, A. (1992). Inventing scientific discourse: dimensions of rhetorical knowledge in physics. Carnegie Mellon University, Pittsburgh. Chi, M. T. H., Siler, S., Jeoug, H., Yamauchi, T., & Hausmann, R. (2001). Learning from human tutoring. Cognitive Science, 25, 471-533. Clark, H. H. (1996). Using language. Cambridge: Cambridge University Press. Cunningham, W. (2006). Design principles of Wiki: How can so little do so much? Retrieved from http://c2.com/doc/wikisym/WikiSym2006.pdf Deiglmayr, A., & Spada, H. (2010a). Developing adaptive collaboration support: the example of an effective training for collaborative inferences. Educational Psychology Review, 22(1), 103-113. Deiglmayr, A., & Spada, H., (2010b). Collaborative problem-solving with distributed information: the role of inferences from interdependent information. Group Processes and Intergroup Relations, 13(3), 361-378. Garrett, A. (2009): LiquidThreads (LQT 2.0). Retrieved from http://www.mediawiki.org/wiki/Extension:LiquidThreads Flower, L., & Hayes, J. R. (1981). A cognitive process of theory of writing. College Composition and Communication, 32(4), 365-387.
ISLS
361
ICLS2012
Haake, J. M., & Pfister, H. R. (2010). Scripting a distance-learning university course: Do students benefit from net-based scripted collaboration? International Journal of Computer-Supported Collaborative Learning, 5(2), 191-210. Hayes, J. R., & Flower, L. (1980). Identifying the organization of writing processes. In L. Gregg & E. Steinberg (Eds.), Cognitive processes in writing: An interdisciplinary approach (pp. 3-30). Hillsdale, NJ: Lawrence Erlbaum. Hermann, F., Rummel, N., & Spada, H. (2001). Solving the case together: The challenge of netbased interdisciplinary collaboration. In P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.), Proceedings of the first European conference on computer-supported collaborative learning (pp. 293300). Maastricht: McLuhan Institute. Judd, T., Kennedy, G., & Cropper, S. (2010). Using Wikis for Collaborative Learning: Assessing Collaboration through Contribution. Australasian Journal of Educational Technology, 26 (3), 341-354. Kimmerle, J., Moskaliuk, J., & Cress, U. (2009). Individual learning and collaborative knowledge building with shared digital artifacts. International Journal of Behavioral, Cognitive, Educational and Psychological Sciences, 1, 25-32. King, A. (2007). Scripting collaborative learning processes: A cognitive perspective. In F. Fischer, I. Kollar, H. Mandl, & J. M. Haake (Eds.), Scripting computer-supported collaborative learning: Cognitive, computational, and educational perspectives (pp. 13-37). New York: Springer. Kittur, A., & Kraut, R. E. (2008). Harnessing the wisdom of crowds in wikipedia: Quality through coordination. In CSCW'08: Proceedings of the ACM conference on computer-supported cooperative work. New York: ACM Press. Kollar, I., Fischer, F., & Hesse, F. (2006). Collaboration scripts - A conceptual analysis. Educational Psychology Review, 18, 159185. Kollar, I., Fischer, F., & Slotta, J.D. (2007). Internal and external scripts in computer-supported collaborative inquiry learning. Learning and Instruction, 17, 708-721. Leuf, B., & Cunningham, W. (2001). The wiki way: Quick collaboration on the Web. Upper Saddle River, NJ: Addison-Wesley. Minocha, S., & Thomas, P. (2007). Collaborative learning in a wiki environment: Experiences from a software engineering course. New Review of Hypermedia and Multimedia, 13(2), 187-209. Neuwirth, C. M., Chandhok, R., Kaufer, D. S., Morris, J. H., Erion, P., & Miller, D. (2000). Computer supported collaborative writing: A coordination science perspective. In G. M. Olson, T. W. Malone & J. B. Smith (Eds.), Coordination Theory and Collaboration Technology (pp. 535-557). Mahwah, NJ: Lawrence Erlbaum Associates. ODonnell, A.M. (1999). Structuring dyadic interaction through scripted cooperation. In A. M. ODonnell & A. King (Eds.), Cognitive perspectives on peer learning (pp.179-196). Mahwah, NJ: Erlbaum. ODonnell, A. M., & Dansereau, D. F. (1992). Scripted Cooperation in student dyads: A method for analyzing and enhancing academic learning and performance. In R. Hertz-Lazarowitz & N. Miller (Eds.), Interaction in cooperative groups: The theoretical anatomy of group learning (pp. 120-141). New York: Cambridge University Press. Proske, A., Narciss, S. & McNamara, D. (2010). Computer-based scaffolding to facilitate students development of expertise in academic writing. Journal of Research in Reading 33(1), 1-17. Renkl, A., & Mandl, H. (1995). Kooperatives Lernen: Die Frage nach dem Notwendigen und dem Ersetzbaren. Unterrichtswissenschaft, 23, 292-300. Rummel, N., & Spada, H. (2005). Learning to collaborate: an instructional approach to promoting collaborative problem solving in computer-mediated settings. The Journal of the Learning Sciences, 14(2), 201241. Stegmann, K., Weinberger, A., & Fischer, F. (2007). Facilitating argumentative knowledge construction with computer-supported collaboration scripts. International Journal of Computer-Supported Collaborative Learning, 2(4), 421-447. Trentin, G. (2009). Using a wiki to evaluate individual contribution to a collaborative learning project. Journal of Computer Assisted Learning, 25, 43-55. Weinberger, A., Stegmann, K., & Fischer, F. (2010). Learning to argue online: Scripted groups surpass individuals (unscripted groups do not). Computers in Human Behavior, 26, 506-515. Weng, C., & Gennari, J. J. (2004). Asynchronous collaborative writing through annotations. In Proceedings of the 2004 ACM conference on computer supported cooperative work (pp. 578-581). New York, NY: Association for Computing Machinery.
ISLS
362
ICLS2012
Challenging Assumptions: Using Sliding Window Visualizations to Reveal Time-Based Irregularities in CSCL Processes
Gregory Dyke, Carnegie Mellon University, Pittsburgh, PA, USA, gregdyke@gmail.com Rohit Kumar, Raytheon BBN Technologies, Cambridge, MA, USA, rohit.kumar@gmail.com Hua Ai, Georgia Tech, Atlanta, GA, USA, hua.ai@cc.gatech.edu Carolyn P. Ros Carnegie Mellon University, Pittsburgh, PA, USA, cprose@cs.cmu.edu , Abstract: Temporality and the unfolding of collaborative processes over time are an important aspect in understanding the underlying mechanisms of collaborative learning. In this paper, we describe how interactive sliding window visualizations can integrate with existing methodologies to provide new insight into how interactions develop over time. We illustrate the value of these visualizations by performing a secondary analysis of a corpus analysed in a previous study to show how new insight can be gained and how results at a scale of a whole interactive session can be challenged when looking for time-based irregularities. We conclude by discussing the kinds of insights that can be gained from sliding window visualizations, how they overcome certain methodological limitations of existing techniques for dealing with temporality in CSCL and, in turn, the methodological limitations which they introduce.
Introduction
In recent years, many authors and collaborations have addressed the problem of temporality in Computer Supported Collaboration (Reimann, 2009; Kapur, 2011; Wise & Chiu, 2011; Suthers, Teplovs, De Laat, Oshima, Zeini, 2011). The variety of beliefs (Suthers, 2006) as to the mechanisms that render collaboration beneficial for learning require that these collaborative processes be studied in detail. The irregularities of these processes are of fundamental importance in CSCL for two reasons. First, learning is primarily about change: acquiring or creating new knowledge and skills. Second, the introduction of technological artifacts to mediate collaboration allows us to study the ways in which collaborative meaning-making is affected by the environment, in an appropriation process which is expected to change over time. Chiu & Khoo (2005), Jeong (2008), and Reimann, Frerejean, & Thompson (2009) have described and successfully employed a variety of statistical techniques which apply to sequences of events and allow the identification and characterization of recurrent patterns in interaction. While these techniques have revealed fascinating results, allowing time to be taken into account rather than factoring it out as an intractable element, they have some weaknesses. They are essentially concerned with identifying regularities at all points in time and must be creatively adapted to show change over time. Furthermore, the timescale of these regularities (if and when they exist) is at that of the event, or of the adjacency pair, and such granularities fail to account for the prior history of the collaborative and learning processes. Last, many social processes related to learning and collaboration, are not expected to happen at the smaller timescales (McGrath and Tschan, 2004), but occur concurrently at multiple timescales. Borrowing techniques from time-series analysis and as suggested by several authors in CSCL (e.g. Muukkonen et al., 2007; Reimann, 2009) interactive time-series visualizations, calculated from sliding window analyses can make a methodological contribution to understanding the irregularities of the unfolding of collaborative processes over time. A sliding window analysis takes an indicator which can be calculated from a set of events, and rather than producing a summary value for a whole period of interaction, computes this indicator for a window of a given size (e.g. 5mn) which is then slid over the period of interaction by small chunks (e.g. 10s), producing the value of the indicator for each window, and showing how it changes over time. In the remainder of this paper, we examine in greater detail the existing techniques for taking time into account and present a methodology based on interactive sliding window visualizations. We illustrate this methodology by performing a secondary analysis on a corpus analysed in a previous publication (Ai, Kumar, Nguyen, Nagasunder, & Ros 2010 ) and highlight the ways in which such visualizations can lead to a better , understanding of the data, give a more subtle account of existing results, and allow new hypotheses and variables to be suggested and confirmed by triangulation using other methodologies. We conclude by discussing the theoretical and practical limitations of such visualizations, the affordances that make their interpretation easier, and outline their relevance for real-time automated analysis and support for computer-mediated collaboration.
Identifying temporal (ir)regularity

The (stereo)typical coding and counting approach of Content Analysis (De Wever, Schellens, Valcke, & Van Keer, 2006) can be criticized for its adaptation from fields where temporality is less central, reducing
ISLS
363
ICLS2012
interactions to summary indicators which no longer have a temporal component. Such indicators or summary statistics, have nevertheless proven useful, particularly in hypothesis testing scenarios where a set of interactions are designed to vary only in the variables isolated for study. Indeed, time is not entirely lost as the act of manually coding assumes the situatedness of each turn or event being coded. However, removing the temporal aspect can lead to misattributions being made when statistical associations are found between counts over whole sessions and outcome measures, because the distribution in behaviors over time is not taken into account. In some of our recent work, for example, a certain form of scripting was found to be ineffective in producing a positive learning outcome (Dyke, Adamson, Howley, & Ros 2012). A closer examination of the temporality , revealed that it produced the desired interactions (which might theoretically have produced a positive learning outcome), but these interactions disrupted the natural flow of interaction to such a degree that the benefits were balanced out. Methods adapted from Conversational Analysis (e.g. Suthers, Dwyer, Medina, & Vatrapu, 2010) have had the opposite problem, with their costliness in analytic time making it hard to identify generalizations, and have lead to the refinement of methods of describing collaborative processes in ways that focus as much on the relationships (or contingencies) between events as on the events themselves. Reimann (2009) argues that eventdriven approaches that seek to model sequential process can provide a methodological link between the thick descriptions produced by fine-grained analyses and the robust generalizations of quantitative studies. He also provides a detailed overview of the kinds of techniques which could be used, along with their methodological implications. Jeong (2005), Reimann et al. (2009), Thompson, Kennedy-Clark, Markauskaite, & Southavilay (2011) use Markov models to model collaborative processes and describe distinctive features which characterise certain conditions or certain populations of learners. Chiu & Khoo (2005) and Kapur (2011) describe statistical techniques which introduce lag-variables to examine the effect that the preceding turn (at a lag of -1) or older turns (at lags of -2, -3, etc) have on the current turn. While these approaches can be generalized to the non-linear structure of artefact-mediated interactions, they nevertheless tend to only apply to small time scales and to the search for regularities. In a study by Jadallah et al. (2011), lag variables ceased to have a discernable effect after seven turns. Even short-term effects, such as the introduction of technology for a one hour session would be expected to produce a gradual effect over the course of several hundred turns. As Reimann (2009) highlights, learning processes happen at multiple timescales, making it challenging to find new coherent temporal units over which larger-scale variables can be meaningfully computed. Chiu & Khoo (2005) have used breakpoint analysis to identify focal points in interaction in which the nature of a variable shifts, marking, for example, a transition from a period of low to a period of high microcreativity. Unfortunately, such statistical techniques are computationally expensive, both requiring datasets with many events, and limiting the number of breakpoints that the model can attempt to find. Mayfield, Rudnicky, & Ros(2012) have used dynamic time warping (a technique for aligning temporal processes which are not of the same duration or do not happen at the same speed) to characterize differences between populations performing a given collaborative task. Jain, McDonough, Gweon, Raj, & Rose (2011) have used a dynamic bayesian network models to show how linguistic stylistic accommodation happens over the duration of an interaction. These techniques, along with techniques developed for mining collaboration patterns in Educational Data Mining (e.g. Maldonado, Yacef, Kay, Kharrufa, & Al-Qaraghuli, 2011) tend to posit a determinism which may not be present or which may be influenced by too many variables which can neither be measured nor controlled. In our work, we take an Exploratory Sequential Data Analysis (Sanderson & Fisher, 1994) approach, providing the means to examine an overview of trends over time (and over multiple time periods) in order to quickly gain an appreciation for how a given interaction unfolds, what might be characteristic of a particular class of interactions and what might distinguish different classes of interactions (e.g. in different conditions, or with differing degrees of success).
Sliding Window Visualizations

Sliding window visualizations extend the ideas of indicators computed as a summary of an interaction by reintroducing the notion of time. If we take a very simple indicator (e.g. distribution of turns among participants), this initially produces a summary of the distribution of turns over a whole interaction (e.g. telling us that all participants participated equally), but giving us no idea of change. By taking the smallest possible time-period (1 turn), the values of this distribution become very noisy (during a single turn, a single participant has 100% of the participation and the identity of this participant changes at each turn), telling us, that, at this micro level, interactions are very irregular. During a small portion of an interaction, however, two of the participants might be monopolizing the discussion, producing a 45/45% distribution for them, with the remaining 10% of turns contributed by the remaining participants. Sliding window visualizations show how the distribution changes over time throughout the interaction. As this window slides over times, it gradually leaves certain events out of the calculation of the indicator, and introduces new ones. At each iteration, the indicator is calculated for that window and plotted. The size of the window produces a certain amount of
ISLS
364
ICLS2012
smoothing or blurring, with one extreme being the window over the whole interaction (equivalent to the usual summary statistics) and the other being the jumpiness of the value per event. Our approach differs from other visualization approaches (e.g. Jermann, Mullins, Nuessli, & Dillenbourg, 2011; Hmelo-Silver, Chernobilsky, & Nagarajan, 2009; Suthers et al., 2010) in that we automatically make the transition in granularity from the turns or events which are present in the raw data. In order to compensate for this (as there is no reason that there should be any comparability in terms of interactive and learning processes between two windows just because they contain the same number of events or the same amount of time), we provide the means to change the amount of smoothing (the size of the window) on the fly. Furthermore, we implement these visualizations within the Tatiana analysis framework (Dyke, Lund, & Girardot, 2009; Dyke, Lund, & Girardot, 2010), which provides synchronization between visualizations, allowing a human analyst to transition from the visualization to the raw data and back. In this way, any salient irregularities can be examined, verified to not be an artifact of smoothing at a particular granularity, and be used as a means of diving into the raw data. The interactive nature of our visualizations distinguish them from existing work in CSCL which has examined change over time (e.g. Wang & Ros 2007). , In accordance with the Tatiana data model, each event has a certain number of properties, including, when applicable, a contents, which is the message, utterance or text for that event. In order to introduce a certain amount of genericity in these visualizations, any ad-hoc indicator can be computed, with default indicators showing number of turns and number of words, as distributed, not only among participants (one of the key properties of an event), but also among other properties, such as a set of codes (e.g. ontask and offtask). We furthermore provide the means to view each visualization either in absolute numbers (A had 5 turns, B had 10) or in proportions (A had 1/3, B had 2/3).
Social Conversational Agents for Scaffolding CSCL

To illustrate the ways in which such visualizations can be exploited, we will perform a secondary analysis on the dataset analyzed by Ai et al. (2010). This research was conducted to investigate how conversational agents (chatbots) can be used, not only to script the content of a collaborative discussion task, but also to provide social support to groups (such as showing solidarity, precipitating tension-release, and agreeing). It also investigates the effect of bias and alignment with student goals, in a situation where students are given competing goals. The study builds on an activity for which it had already been shown that providing social support produced a significant effect on learning (Kumar, Ai, Beuth, & Ros 2010). In this activity, students must design a Rankine , cycle (a type of power plant), working in pairs to understand the trade-offs between power output and environmental friendliness. The study was conducted in a 3x3 factorial design across 53 pairs of students in which two aspects of the conversational agents were manipulated. The first was the amount of social support: None, Low (at most 15% of agent turns are social), and High (at most 30%). The second was the goal-alignment: Green, Power, and Neutral. In all conditions, the agents guide the students through a carefully designed script going through a series of 4 KCDs (Knowledge Construction Dialogues), each of which discusses a parameter of the cycle and its implications: Tmax (maximum temperature), Pmin, Pmax (minimum, maximum pressure) and fuel. The wording of these dialogues is slightly different in the Green and Power conditions to highlight the influence of these parameters on environmental friendliness and power output respectively. For example, where the Green agent might say What is bad about increasing the heat input to the cycle is that it increases the heat rejected to the environment. The neutral tutor would simply say Increasing heat input to the cycle increases the heat rejected to the environment. After each KCD, the tutor prompts the students to discuss which value they would like to choose for that parameter and gives them a few minutes to conduct this discussion. At the end, the tutor concludes by prompting the students to review their choices for each parameter and come up with a final design. Students worked in pairs, with each pair having a student whose goal was to maximize power output and a student whose goal was to maximize environmental friendliness. Thus, each student can be considered to either be in the Match, Mismatch or Neutral condition, depending on their goal and that of the agent. In the original analysis, the learning gains between pre- and post-tests were evaluated across each of the 9 conditions, as was the amount of offtask behaviour, the amount of agent abuse (the agent sometimes makes remarks at inappropriate times, leading the students to comment on how much they disliked it), and the amount of social behaviour. The main results were: - Learning was significantly better in the low social condition than in the other two (particularly in the Match condition), with an effect size of .83 standard deviations. - Social behaviour was significantly higher in both of the social conditions than in the None condition (effect sizes of 1.8 and 1.2) - There was significantly more offtask behaviour in the None social condition than either of the other two (effect sizes of .35 in both cases) - There was significantly more tutor abuse in the High social condition than in the Low or None condition (effect sizes of 1.1 and 1.28 respectively)
ISLS
365
ICLS2012
A Secondary Analysis Looking for Irregularities

We now show how creating various forms of sliding window visualizations on this corpus allows us to reexamine the results presented above and to identify new insights and irregularities to produce new statistically significant results. More specifically, we would like to identify whether the low social condition affected participation over time in some way which can explain the increase in learning. Furthermore, we would like to examine the extent to which the social, offtask and tutor abuse behaviors were uniform throughout the conversation or whether they occurred at certain specific times.
Single group All groups

While the visualization of a single group gives a good idea of what happened in that group, it is very hard to get a feel for what is normal across 53 different groups. For this reason, and because the timing of the agents scripting was reasonably consistent (to within about 10s) across all groups, we combined all the sessions into a single time-aligned transcript and visualized it to get an idea for what is normal across all groups. Figure 1 (top) shows the general trends of how the conversations flowed over time (horizontal axis, each session being around 40mn in length), in words per minute (vertical axis), with tutor participation (pastels) separated according to the KCD or the type of social turn to which it belongs showing clearly the flow from one KCD to the next, with minima in tutor participation in between. The student participation (pink, yellow and orange) shows noticeable peaks during the times when the tutor is not in a KCD. As a result of this observation, we coded each student utterance according to where it fit in the discussion (offtask, social, tutor abuse, on task in the Tmax, Pmax, Pmin, Fuel KCDs, or on task outside of one of the KCDs), producing Figure 1 (bottom), which hides all tutor turns and amplifies the scale of the student participation (the pink above is the sum of all the colors below), showing clearly 4 equal bouts of interaction with the tutor within KCD, and showing high participation in the interval after the first KCD (A), slightly lower participation after the second KCD (B) and lowest participation after the third KCD (C), before rising after the last KCD and during the final stages of the decision (D). These regularities (irregularities over time, regularities across groups), while not particularly surprising (students do a lot of reading during KCDs and participate less there is also less to discuss as the power plants parameters become increasingly fixed), were not known to us previously. We also examined various student participation ratios: green vs. power; most learning vs. least learning; match vs. mismatch, but all these revealed near equal participation, throughout the interaction. Concerned that this might be a smoothing effect due to cumulating all groups over time, we visualized a more precise indicator which calculates within group ratios and again obtained the same results. Although the match condition produced a slight effect on learning, it apparently produced no effect on participation over time: a consistent 50-50 split in student participation at all window sizes greater than 1mn.
Figure 1. Sliding window visualization for windows of 1mn40s across all groups, showing distribution of features over time (horizontal axis). Units are in words per minute per group. Tutor (pastel) vs student (pink) is shown at top. Student turns are decomposed (bottom) into their component KCD and non-KCD sections.
ISLS
366
ICLS2012
Partitioning groups
Armed with the knowledge of what a normal group (or subset of groups) would look like, we partitioned the groups according to the experimental conditions, contrasting the graphs for each. In Figure 2, for example, we see that the Power condition produced less talk in the (circled) discussion immediately after the Fuel KCD than the Neutral or Green conditions. Examining the transcripts in more detail, this appears to be because the introduction of the discussion related to Fuel, particularly when combined with the environmental bias (which discourages the use of fuels which produce a lot of waste heat and increase power) generated a lot of rediscussing of values that might not otherwise have been reconsidered.
Figure 2. Sliding window visualizations for windows of 1mn40s partitioned by Goal condition (Top: green; Middle: power; Bottom: neutral). The curves are strikingly similar across conditions, apart from the discussion (green, circled) immediately after the fuel KCD (orange), which is noticeably lower in the power condition.
Figure 3. Sliding window visualization (window of 4mn43) for offtask, social and tutor abuse, portioned by Social condition (Top: high; Middle: low; Bottom: none). We see high social in the beginning for the social conditions, and high offtask at the end for the none condition. We were particularly able to refine our understanding of the interaction between social conditions and offtask, social talk and tutor abuse. First, as is apparent in Figure 3, the majority of social talk happened during the beginning of the session in the High and Low social conditions and was almost completely absent in the None social condition. Upon examination of the transcripts the reason was immediately apparent: in the social
ISLS 367
ICLS2012
conditions, the tutor suggested that participants introduce themselves which they then did, almost consistently in all groups. Tutor abuse can be seen to be steadily rising in the high condition, while being much more varied in the others. This shows the growing frustration that the students in the high social condition experienced with the tutor, which was performing too many social turns to remain a credible participant in the discussion. Last, the offtask towards the end is particularly marked in the end of the None condition, possibly as a reaction to the tutor leaving, presumably leading the students to believe that the task was over.
Identifying new variables for statistical analysis

The differences between the Green and Power conditions, along with other, less noticeable ones when we partitioned both by learning outcome (post-test, adjusted to account for pretest values) and by condition lead us to believe that separating participation into individual discussion portions (the various KCD and non-KCD phases) would produce productive variables for examining correlation with learning. It turns out that in the Green condition, the discussion about Tmax was significantly greater (in total number of words) for high learners than for low learners, F(2,43) = 3.98, p < .05, effect size 1.47 standard deviations, whereas there was no difference within the Power and Neutral conditions. In the Power condition, however, it was the discussion about Pmin which was greater for the high learners than for the low learners, F(2,43) = 3.65, p < .05, effect size .43 standard deviations. Interestingly, in both cases, these particular subjects were the more cognitively demanding with their respective biases (Tmax should go as high as possible to get greater power, but needs compromise to get environmental friendliness; Pmin affects cycle efficiency and so is the only one of the four parameters for which there isnt a tradeoff between power and environmental friendliness). Examining the direct relationship between the various social conditions and the learning outcome did not reveal any immediate answers. When combining this with information about offtask, however, a clearer picture can be drawn. We examined the amount of offtask during each portion of the discussion and its correlation with learning outcomes. Low learners had significantly more offtask during the KCD portion of the discussion than high learners, F(1,36) = 10.3, p < .005, effect size 1.62 standard deviations. On the other hand, with respect to the Power and Green conditions there was no difference between higher and lower performing students regarding how much tutor abuse they uttered, but surprisingly, within the Neutral condition, higher performing students did more tutor abuse than lower performing students, F(2,43) = 3.87, p < .05, effect size 1 standard deviation. This dual effect between social condition and on/off task behavior and between on/off task behavior and learning outcome illustrates how intricate such interactions between variables can be, particularly when time is factored back into the analysis. To summarize, our new analysis shows that there is a distinct fluctuating pattern that can be expected of all the conversations scripted by this conversational agent. It shows that the effect of tutor social support on student social interaction is mostly a product of the introductory phase of the interaction (although it does extend to some extent over the whole interaction). It suggests an interaction between social condition, amount of offtask, and learning outcome, indicating that certain points of the interaction are more important (during the KCDs and during the final discussion) than others and that appropriate agent scaffolding might attempt to divert offtask behavior from happening at these points, or that offtask at this behavior might be diagnostic of groups with low learning performance. Finally, we identified a previously unsuspected interaction between participation during specific KCDs, respectively in the Green and Power conditions, and learning outcomes. Again this relationship can currently only be suggested as a predictor of learning, and not a cause of learning. Further studies might attempt to scaffold the interaction to favor patterns which have, in the past, been indicative of high learning and attempt to discern a causal relationship.
Discussion
In this example, we show how interactive sliding windows can be integrated in different ways into both descriptive and quantitative methodologies, particularly when focusing on a search for irregularities. This example has certain properties which render it particularly suited to the methodology we are developing: it is strongly scripted over time, making it easy and feasible to cumulate groups together and to partition data into distinct sets of groups. Furthermore, there is a strong homogeneity between groups (each has two students, with similar goals) and an experimental design which provides a number of randomly assigned existing conditions. The data in this example was only coded in a limited way to show offtask and ontask and all visualizations were calculated using turn word count as a proxy for participation. More complex coding might have revealed more pertinent results, but it would then have been harder to attribute new findings to the introduction of sliding window visualizations as opposed to the new codings. Interactive sliding windows can intervene at different times within existing methodologies. For an initial data exploration phase, they provide an interesting lens for understanding how a process develops over time, drawing a continuous link between session-wide summary statistics and properties of individual turns, and allowing the analyst to delve down into the raw data at any time. They also provide a way to examine what regularities and distinctions that exist in different conditions or sub-populations. This helped us show both than
ISLS
368
ICLS2012
some of the results of the initial analysis were less interesting than they appeared to be (the effect of social condition on amount of social participation was most marked in the introductions) and that others were more interesting (the interaction between both goal and social conditions and offtask and tutor abusing participation is particularly complex). On their own however, these visualizations are not suitable as a quantitative tool. As in our example, they are useful for identifying new variables that might potentially be used to show significant quantitative results. We did not report any abortive analytic paths, but the visualizations frequently helped us to cheaply explore different avenues in which quantification might have yielded results, but for which it was immediately apparent that there would be nothing significant to report. For example, we also used subpopulation partitioning on post-hoc values such as the learning outcome. This showed an apparent linear relation between participation in the final discussion and learning, with its value being smallest in the low learners, largest in the high learners and in between in medium learners. This relationship turned out to be artifacts of the data portioning, rather than a significant result. Partitioning by learning outcome and condition, however produced the significant result reported above. Finally, the dynamic nature of these visualizations, helps overcome a major limitation of time-scale changes that there is no reason that an arbitrarily sized window (either in time or number of turns) should have a constant relation with complex group processes. The visualizations proved particularly challenging to read, frequently suggesting patterns which turned out (on closer examination of the data) to be artifacts caused by certain groups or by looking at a specific window size. One of the issues is that our indicators only report averages and not distributions informing us of variance, etc. This effect can be slightly mitigated by synchronizing across multiple visualizations, so that their timescale is constant, the windowsize currently under examination is identical and the units are coherent across window sizes. The units must also be coherent across subpopulations of different sizes. In order to meaningfully compare visualizations, we made each indicator be computed per group and per minute. Not only did synchronization from Tatiana help us transition between visualizations and raw data, it also helped understand the non-intuitive relationship between an indicator computed over a given window and the events making up that window. It is useful to consider each event has having an area of influence comprising multiple windows, and to consider each window as being the first, chronologically, to include certain events and to exclude others.
Conclusion
In this paper, we presented interactive sliding window visualizations and showed the role they can play in integrating with existing methodologies to explore data, produce more refined results and new insights into the irregular temporal processes which occur during collaborative learning. These visualizations address some of the granularity issues found in previous work attempting to quantify processes over time. They also draw upon work in visualization and exploratory sequential data analysis providing a method which fits nicely in the gap between descriptive and quantitative methodologies, allowing transitions between event properties and global indicators, and providing the means both to dive into the raw data and suggest new variables for quantification, fundamentally looking for regularity across populations and irregularities between populations and over time. We showed how these visualizations provided new insights on a previously analyzed corpus, showing that certain results were more interesting than they seemed, some less, and providing additional entirely new insights. We then discussed the limitations of these visualizations, highlighting that they are not intended to stand on their own, but to be used as a means of identifying potential irregularities which can then be explained qualitatively or confirmed quantitatively as related to learning and developmental processes. These visualizations also prove particularly appropriate when the data is scripted, providing a link between the intended script and the actual path through the script. Future work should investigate how these visualizations can be used for larger timeframes and in natural, rather than quasi-experimental, and non-educational settings. We chose to focus on very superficial features of the conversations without using complex coding. This was in part in order to show the benefits over an existing analysis using the same features, but also because we see this as fitting into two wider strains of research. The first is the problem of data analysis where researchers attempt to find anchor points which allow them to explore a corpus which has not yet been analyzed or coded sliding window visualizations assist in the identification of salient features which may be worthy of further interest. The second is the problem of providing dynamic support for collaborative learning. In our research into conversational agents, we have been hampered by the fact that agents do not have a deep understanding of how conversation unfolds over time. In a first step, we would like to identify temporal features which allow a conversational agent to gain insight into how well or poorly a group is performing. Once we have identified such features (and that they correlate with positive outcomes) we will be able to address the more complex issue of whether the agent is able to influence the interaction in a positive way, or in other words to examine whether we can also posit a causal effect between some kinds of features the agent could influence and a positive outcome.
Acknowledgments
This work was funded by NSF SBE-0836012.
ISLS 369
ICLS2012
References
Ai, H., Kumar, K., Nguyen, D., Nagasunder, A., and Ros C. 2010. Exploring the Effectiveness of Social , Capabilities and Goal Alignment in Computer Supported Collaborative Learning. Proceedings of 10th International Conferences on Intelligent Tutoring Systems (ITS). Chiu, M. M., & Khoo, L. (2005). A new method for analyzing sequential processes: Dynamic multi-level analysis. Small Group Research, 36, 600-631. De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyse transcripts of online asynchronous discussion groups: A review. Computers & Education, 46(1) 6-28. Dyke, G., Adamson, D., Howley, I., Ros C. (2012) Towards Academically Productive Talk Supported by , Conversational Agents. ITS 2012. Dyke, G., Lund, K., & Girardot, J.-J. (2010). Tatiana : un environnement daide lanalyse de traces dinteractions humaines. Technique et Science Informatiques, 29(10), pp.1179-1205. Dyke, G., Lund, K.,& Girardot, J.-J. (2009). Tatiana: an environment to support the CSCL analysis process. CSCL 2009, Rhodes, Greece, 58-67. Hmelo-Silver, C. E., Chernobilsky, E., & Nagarajan, A. (2009). Two sides of the coin: Multiple perspectives on collaborative knowledge construction in online problem-based learning. In K. Kumpulainen, C. E., Hmelo-Silver, & M. Cesar (Eds.). Investigating Classroom interaction: Methodologies in action. Sense Publishers. Jadallah, M., Anderson, R.C., Nguyen Jahiel, K., Miller, M., Kim, I. Kuo, L. Wu, X., & Dong, T. H., J., (2011). Influence of a teacher's scaffolding moves during child small led group discussions. American Educational Research Journal, 48(1), 194 230. Jain, M., McDonough, J., Gweon, G., Raj, B., & Rose, C (2012). An Unsupervised Dynamic Bayesian Network Approach to Measuring Speech Style Accommodation. EACL 2012. Jeong, A. (2008). Sequentially analyzing and mapping the interactional processes of knowledge construction in online learning. AERA 2008, New York. Jermann, P., Mullins, D., Nuessli, M.-A., Dillenbourg, P. (2001). Collaborative Gaze Footprints: Correlates of Interaction Quality. In Proceedings of CSCL 2011. Hong Kong. Kapur, M. (2011) Temporality matters: Advancing a method for analyzing problem-solving processes in a computer-supported collaborative environment. ijcscl 6 (1), pp. 39-56. Kumar, R., Ai, H., Beuth, J., Ros C. P. (2010) Socially-capable Conversational Tutors can be Effective in , Collaborative-Learning situations. Intelligent Tutoring Systems (ITS) 2010, Pittsburgh, PA Maldonado, R. M., Yacef, K., Kay, J., Kharrufa, A, & Al-Qaraghuli, A. (2011). Analysing frequent sequential patterns of collaborative learning activity around an interactive tabletop. Proceedings of EDM 2011. Mayfield, E., Rudnicky, A., and Rose, C. (2012). Computational Representation of Discourse Practices Across Populations in Task-based Dialogue. ICIC 2012. McGrath, J. E., & Tschan, F. (2004). Temporal matters in social psychology: Examining the role of time in the lives of groups and individuals. Washington: American Psychological Association. Muukkonen, H., Hakkarainen, K., Konsonen, K., Jalonen, S., Heikkil, A., Lonka, K., et al. (2007). Process-and context-sensitive research on academic knowledge practices: Developing CASS-tools and methods. In Proceedings of CSCL 2007 (pp. 541543). New Brunswick: ISLS. Reimann, P. (2009) Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. ijcscl 4 (3), pp. 239-257. Reimann, P., Frerejean, J., & Thompson, K. (2009). Using process mining to identify decision making processes in virtual teams. Paper presented at CSCL2009. Rhodes/Greece. Sanderson, P. M., & Fisher, C. (1994). Exploratory sequential data analysis: Foundations. Human-Computer Interaction, 9(3/4), 251317. Suthers, D. D. (2006) Technology affordances for intersubjective meaning making: A research agenda for CSCL. ijcscl 1 (3). Suthers, D. D., Dwyer, N., Medina, R., & Vatrapu, R. (2010). A framework for conceptualizing, representing, and analyzing distributed interaction. ijcscl 5(1), 5-42. Suthers, D. D., Teplovs, C. De Laat, M., Oshima, J., Zeini S. (2011). Connecting Levels of Learning in Networked Communities. Workshop at CSCL 2011. Thompson, K., Kennedy-Clark, S., Markauskaite, L., & Southavilay, V. (2011). Capturing and analysing the processes and patterns of learning in collaborative learning environments. CSCL 2011. Hong-Kong. Wang, H., & Ros C. P. (2007). A Process Analysis of Idea Generation and Failure. Proceedings of the Annual , Meeting of the Cognitive Science Society. Wise, A. F., & Chiu, M. M. (2011). Analyzing temporal patterns of knowledge construction in a role-based online discussion. ijcscl 6 (3), pp. 445-470.
ISLS
370
ICLS2012
Distributing Practice: Challenges and Opportunities for Inquiry Learning

Vanessa Svihla, University of New Mexico, Hokona Hall, Albuquerque, NM 87131, vsvihla@unm.edu Marcia Linn, University of California at Berkeley, Tolman Hall, Berkeley CA 94720, mclinn@berkeley.edu Abstract: This study investigates the impact of differing lag times on distributed practice compared to the traditional clustered approach. High school students studied the Web-based Inquiry Science Environment unit, Global Climate Change in one of three conditions. We compare student learning outcomes. We find that traditional spacing resulted in larger gains, in apparent contrast to the findings of many studies of distributing practice. We further investigate this finding to understand how spacing might have impacted revisiting and restudying of earlier work. Specifically, we use log files to determine how students revisited earlier work, finding that condition moderated the effect of revisiting on outcomes. Students in the distributed condition tended to revisit proximal steps, whereas those in the clustered condition tended to revisit steps while writing explanations at the end of the unit. We discuss implications and directions for further study.
Introduction
Helping students develop coherent understanding of science concepts is challenging, particularly given the fragmented curriculum in place in many schools. Research focusing on how to help students integrate their ideas has employed the Knowledge Integration (KI) framework to investigate shorter units focused on specific concepts (Linn, 2006), and recently has shifted to integrating core ideas across science disciplines, such as energy transfer and transformation (Svihla et al., 2010). One approach to helping students develop such integrated understanding is to distribute practice. Experimental laboratory studies have demonstrated benefits of distributed practice for learning a new concept or category. For instance, in one study, participants who studied paintings in a distributed manner, with other artists paintings interleaved, performed better than those who studied in a massed manner, viewing the paintings consecutively. Interestingly, the participants in the latter condition rated it as a better learning condition, even though their performance was lower (Kornell & Bjork, 2008). This finding held for repetition tests as well (Kornell, Castel, Eich, & Bjork, 2010). In an experiment simulating classroom practice with eight hours of video-delivered statistics lectures, a significant advantage was found for distributed practice over massed practice on a retention test given five days after the training period (Smith & Rothkopf, 1984). Smith and Rothkopf explain this finding as demonstrating that distributed practice supports learning, whereas massed practice primarily supports retrieval, making a distributed approach the more efficient approach to instruction (Smith & Rothkopf, 1984). In a series of studies ranging from the laboratory to the classroom, Seabrook and colleagues found an advantage for distributed practice (Seabrook, Brown, & Solity, 2005). In their laboratory experiments, they found that spacing was beneficial across age groups, and introduced a clustered condition, intermediate to massed and distributed conditions and better reflecting teaching, finding that clustering was statistically similar to massed practice, and both conditions performed significantly lower compared to the distributed condition. In their classroom-based study, which involved young children learning phonics, children in the distributed condition also outperformed those in the clustered condition. Though spacing has been recommended for the classroom (Pashler et al., 2007), few studies have sought to examine the spacing effect in real world settings. Sobel and colleagues (Sobel, Cepeda, & Kapler, 2010) conducted a study in a fifth grade classroom, in which students were taught the definitions of unfamiliar words. Students studied some words in a distributed manner, with two 15-min learning sessions separated by seven days, and studied other words in a massed manner, with two 15-min consecutive learning sessions. A vocabulary test given five weeks after the second learning session revealed a significant advantage for the words learned in the distributed condition, though in both conditions, retention was poor (Sobel et al., 2010). The classroom and classroom-like studies have not investigated the effect of distributing practice on helping students integrate their understanding of complex science content. This comparison study, set in intact classrooms, investigates the impact of different spacingtraditional or longer lag between sessions on how well students integrated their understanding of a core science concept.
Methodological Approach Materials

Students studied a previously-tested Web-based Inquiry Science Environment (WISE) (Slotta & Linn, 2009) unit called Global Climate Change (Svihla & Linn, 2011). This unit teaches students about the greenhouse
ISLS
371
ICLS2012
effect and global warming. Students investigate NetLogo visualizations (Wilensky & Reisman, 2006) representing the earth and atmosphere, and variables involved in climate change. The Global Climate Change project includes a focus on energy transfer and energy transformation in global climate change processes. WISE units scaffold students using an inquiry map (Figure 1) to supports them in developing coherent understanding of science ideas (Linn, 2006). We sought to examine the impact of different lag times on both what students learned, and how they studied the CLEAR unit on Global Climate Change.
Figure 1. Inquiry map for the WISE 4 environment.
Participants
The participants of this study were students in intact sections of Chemistry and Earth Science at a suburban high school serving a diverse community. Students in the chemistry class were primarily in their junior year whereas those in Earth Science were primarily in their first year of high school. Across classes, the teacher hosted guest speakers each Wednesday, and these ranged in topic. On the following class meeting (which due to block scheduling may not have been the following day) the teacher began with a starter activity in which students first completed a worksheet with review questions about the speaker, followed by whole class discussion in which he reviewed the questions and asked students what they learned. Several of the invited speakers discussed climate change. Thus, all conditions had the opportunity to learn about climate change outside of the WISE unit during the semester. Students studied the WISE unit in dyads, but completed assessments as individuals.
Conditions and Study Design

To examine how distributed practice might be used in classroom settings with complex instruction, we compared three conditions that varied in terms of lag (N=291). In contrast to other classroom and laboratory research focusing on restudy of identical items specifically, we considered the distribution of a WISE unit in one of the following ways: Clustered conditions. The WISE unit was used for the majority of instruction time, across consecutive class meeting times. In this context, the lag period between each class session with instructional time is two or three days, either because of intervening weekend days or block scheduling. Within this condition, two timing conditions were also designed: an early condition (n=31) and a late condition (n=91). The teacher found it challenging to schedule the early condition separately from his other courses; thus, as implemented, the early condition ended up being a mid-point between the clustered and distributed conditions, with students completing activities in the unit at their own pace, but with variable lag time between days spent on the unit (Figure 2). Distributed condition. The WISE unit was used for a portion of instruction time each week, with sufficient time for students (n=169) to complete a given activity within the unit (Figure 2). More class periods were assigned to this condition because of numerous findings of a benefit of distributing practice. Three individually-completed tests were given to track students progress. The unit began with a pretest given in week one, and ended with a post test as soon as the dyads completed the activities in the unit. A delayed post test was given at the end of the semester; thus the students in the clustered, late condition completed the delayed post four weeks after finishing the unit, the distributed condition did so five weeks after, and the clustered early did so six weeks after. Pre-, post- and delayed post items, as well as dyad responses to embedded assessments were scored using previously developed KI rubrics (Svihla & Linn, 2011). Log files of dyad progress were examined for total time spent, number of unique visits to steps within the unit, and unique visits to steps across days.
ISLS
372
ICLS2012
Figure 2. Implementation of conditions, with grey showing times the unit was studied We consider two competing hypotheses in this study. Implicit in both of these hypotheses is the notion that restudying is valuable to learning. We operationalize restudying as revisiting earlier steps in the unit, but recognize the limitations this presents. There are many reasons a student might decide to revisit a step apart from deliberate restudy; however, we view any revisiting as affording an opportunity to restudy, and partially address the limitations by considering how and when students revisited steps. Our hypotheses are: The distributed condition should outperform other conditions because they have more opportunities to restudy. The clustered conditions should outperform the distributed condition because-- though the distributed condition provides apparent opportunities for students to restudy-- the traditional classroom context tends to fragment learning activity, such that students in the distributed condition are less likely to actually restudy.
Analysis and Results

Because this study took place in intact classrooms, we first sought to determine whether the natural clustering of class periods or the conditions explained variance in our measures. We used regression analysis to model the total scores for each time point using contrast codes; these allowed us to model variance in scores as a function of course type (chemistry or earth science), class period, and condition. When controlling for pretest, course type did not explain significant variance, t=0.99, p=.33.By comparing the models, we find that condition accounts for significant variance in most cases (Figure 3). In this figure, which summarizes a sequence of regression models, we use grey lines to depict significant differences by condition, and red lines to show significant differences by class period.
Figure 3. Scores on pre-, post- and delayed post-tests by class period, grouped within condition. Grey lines represent differences by condition; red lines by class period. In general, condition accounted for variance. Through this modeling approach, we found that significant variance could be explained between the clustered and late condition and all other conditions for the total scores at each time point. Additionally, at the post-test, significant variance is explained by including a contrast code for clustered early and distributed conditions. No further variance is explained using class period contrast codes. For the Energy Story, contrast codes for clustered early and distributed conditions explain significant variance in KI level at the pretest and delayed post test, and contrast codes for clustered late compared to all other conditions explain significant variance in KI level at the post and delayed post time points. The only additional significant variance explained
ISLS 373
ICLS2012
by class period within a condition was found for the delayed post test Energy Story score, between Period 3 and all other periods within the distributed condition. We therefore primarily focus our analyses on differences by condition, rather than by class period. At all time points, condition explained significant variance in scores. We therefore continued using regression modeling to seek best-fit models using contrast codes for condition and pre- and posttest scores as we sought to explain variance in delayed post scores. First, we modeled variance in the total KI scores from the delayed post test as a linear combination of scores from the pre-test and post-test, as well as contrast codes comparing the clustered early condition to the distributed condition, and comparing the clustered late condition to all other conditions. This explained a significant amount of variation in delayed post-test scores, F(4, 99) = 9.26, p < .01. A more parsimonious model, including only the post-test scores and the contrast codes for clustered late condition compared to all other conditions was sought while still explaining a significant amount of variation, F(2, 101) = 17.88, p < .01 (Table 1). This latter model may be interpreted as follows: In general, higher scores on the post test predict higher scores on the delayed post-test. For the mean post test score of 11.93, the delayed post test score would be predicted to be 12.61 for the clustered late condition, and 11.74 for the other conditions. Table 1: Regression modeling of delayed post test scores Unstandardized Coefficients B Intercept Pre test scores Post test scores Contrast: M+E v. D Contrast: M+L v. others Intercept Pre test scores Post test scores Contrast: M+L v. others Intercept Post test scores 5.88 0.13 0.41 -0.01 0.26 5.88 0.13 0.41 0.26 7.02 0.42 Std. Error 1.46 0.11 0.09 0.26 0.14 1.46 0.11 0.09 0.13 1.11 0.09 0.41 0.11 0.40 0.18 0.11 0.40 0.00 0.18 Standardized Coefficients
t 4.01** 1.21 4.33** -0.05 1.88 4.03** 1.21 4.39** 1.95 6.32** 4.59**
Model 1: All contrast condition codes and prior scores
Model 2: clustered early versus distributed omitted
Model 3: Parsimonious model
Contrast: M+L v. others 0.29 0.13 0.19 2.14** Model 1 r2 = .24, r2 change**; Model 2 r2 = .25, r2 change NS; Model 3 r2 = .25, r2 change NS. * Significant at p<.05; ** Significant at p<.01 We also modeled variance in the KI levels for Energy Stories from the delayed post test as a linear combination of Energy Story KI scores from the pre-test and post-test, as well as contrast codes comparing the clustered early condition to the distributed condition, and comparing the clustered late condition to all other conditions. This explained a significant amount of variation in delayed post-test Energy Story scores, F(4, 79) = 3.34, p < .05. A more parsimonious model, including only the pre-test scores and the contrast codes was sought while still explaining a significant amount of variation, F(3, 80) = 4.03, p < .01 (Table 2). This latter model may be interpreted as follows: In general, higher scores on the pre test Energy Story predict higher scores on the delayed post-test. For the mean pre test score of 3.12, the delayed post test Energy Story score would be predicted to be 3.91 for the clustered late condition, 3.27 for the clustered early condition, and 3.68 for the distributed condition. Overall, we find that the clustered late condition achieved the highest scores and the clustered early condition the lowest scores (Tables 3 & 4). The KI scores are significantly different across times, F(2, 204) = 34.21, p <.01. For the Energy story, students generally began with a score of a three, meaning that they gave answers such as The energy comes from the sun and goes to Earth where it warms the planet. Only the clustered late condition made gains by the post test, shifting to answers that additionally included details about how the
ISLS
374
ICLS2012
energy was transferred to Earth (e.g., The energy from the sun travels through space as electromagnetic waves and reaches Earth). These findings partially support our second hypothesis, that the clustered conditions should outperform the distributed condition because though the distributed condition provides apparent opportunities for students to restudy, the classroom context tends to fragment learning activity, such that students in the distributed condition are less likely to actually restudy. Table 2: Models of delayed post scores for Energy Stories Unstandardized Coefficients B Intercept Pre test scores Post test scores Contrast: CE v. D Contrast: CL v. others Intercept Pre test scores Contrast: CE v. D 2.69 0.21 0.09 -0.24 0.11 2.94 0.22 -0.23 Std. Error 0.41 0.11 0.08 0.12 0.07 0.34 0.11 0.12 0.23 -0.22 0.22 0.12 -0.22 0.19 Standardized Coefficients
t 6.61** 2.00* 1.11 -2.02* 1.69 8.80** 2.13* -1.99*
Model 1: All contrast condition codes and prior scores
Model 2: Parsimonious model with test scores omitted
Contrast: CL v. others 0.13 0.06 0.23 2.10* Model 1 r2 = .10, r2 change*; Model 2 r2 = .10, r2 change NS. * Significant at p<.05; ** Significant at p<.01 Table 3: Test scores for each condition Pretest Mean SD clustered early distributed clustered late 9.86 10.43 10.88 1.66 1.84 1.60 Post test Mean SD 10.36 11.51 12.65 2.31 1.85 2.24 Delayed Post test Mean SD 10.96 11.58 12.79 2.06 2.12 2.11
Table 4. KI scores for Energy Stories Pretest Mean SD 2.97 0.76 3.28 0.79 2.89 0.97 Post test Mean SD 3.04 0.96 3.20 1.31 3.78 1.42 Delayed Post test Mean SD 3.26 0.81 3.77 0.82 3.93 0.81
clustered early distributed clustered late
We therefore next how explore students progressed through the unit. We assumed an advantage for revisiting prior steps in the unit, and predicted students in the distributed condition would revisit steps more frequently. We also wanted to know whether students spent the same amount of time on the unit, across conditions. For these investigations, we examined log files of dyad interactions with the unit. We computed the average number of visits to steps as the average number of unique visits to a step, provide the visit lasted five seconds or longer. This eliminated situations in which students either clicked the submit button repeatedly (each time an answer is submitted, it is recorded as a separate visit in the log file) or when the student clicked the Next or Back button in rapid succession. The latter behavior was observed during classroom observations, as when students would click Back repeatedly to reach a desired step, rather than navigating to it through the interface, or when playing and simply clicking back and forth through the
ISLS
375
ICLS2012
steps. We also computed scores for the number of revisits to steps across days, predicting an advantage when students revisited a step they had studied on a previous day. Using the same criteria as described for the visits, we further calculated the number of days on which unique visits occurred, then summed the total number of steps revisited by each dyad on more than one day. We computed the total amount of time each dyad spent logged into the unit. We used regression analysis to model these three variables of interest: the average of the total number of unique visits a dyad made to steps; the total amount of time spent by each dyad, and the number of visits to steps across days. Using the contrast codes we modeled variance in each as a function of class period, and as a function of condition. By comparing the models, we find that condition accounts for significant variance in most cases (Figure 4). In this figure, which summarizes a sequence of regression models, we use grey lines to depict significant differences by condition, and red lines to show significant differences by class period. Both clustered conditions spent more time on the unit, and revisited steps more frequently, both across the unit in general and across days specifically. This also supports our second hypothesis, that the clustered conditions should outperform the distributed condition because though the distributed condition provides apparent opportunities for students to restudy, the classroom context tends to discourage making connections between activities over time, such that students in the distributed condition are less likely to actually restudy.
Figure 4. Amount of time and revisiting behavior of dyads during the unit. Grey lines represent differences by condition; red lines by class period. Variability in number of visits to steps is not well accounted for by condition. To further explore revisiting of steps in the unit, we combined the steps containing visualizations. These steps also contained the majority of content, and therefore are of interest in terms of how and when students revisited them. By condition, there was no significant difference in the overall number of times students visited steps containing visualizations. However, we wished to better understand the relationship between revisiting and integrating ideas. We modeled variance in the delayed post test scores as a linear combination of pre-test and post-test scores and included revisiting across days. We focused on revisits to steps that included visualizations, because these were the steps that tend to be most difficult for students to initially understand. We further wanted to test the idea that condition moderated the revisiting activity across days. We created contrast codes and applied them to the clearest cases, the distributed condition (-1) and the clustered late (1) condition (note the semi-distributed profile of the clustered early condition, Figure 1). We then generated an interaction variable using these codes and the revisiting variable. The full model explained a significant amount of variation in delayed post-test scores, F(3, 81) = 10.77, p < .01 (Table 4). This model may be interpreted as follows: In general, higher scores on the pre test and post test predict higher scores on the delayed post-test, with the post explaining significant variance in delayed post scores. For the distributed condition, (coded -1), less frequent revisiting of visualizations predicts higher scores; whereas for the clustered late condition (coded 1), more frequent revisiting of visualizations predicts
ISLS 376
ICLS2012
higher scores, but this was not statistically significant. A parsimonious model omitted pre-test scores, F(3, 81) = 10.77, p < .01., and revisiting steps across days approached significance (p<.10). Delayed post test = 6.305 + 0.49 (post test score) + 0.48 (contrast code)(revisiting score) where the distributed condition is coded as -1 and the clustered late condition is coded as 1. Table 4: Model of delayed post scores, using revisiting across days and time Unstandardized Coefficients B Intercept Pre test scores Post test scores Revisit x condition Intercept Post test scores
2
Std. Error 1.669 0.12 0.11 0.18 1.301 0.11
Standardized Coefficients
t 3.306**
Model 1: Delayed post scores with interaction between revisiting across days and condition 5.519 0.09 0.48 0.27 6.305 0.49 0.07 0.45 0.15 0.76 4.51** 0.15 4.846** 0.46 4.66**
Model 2: Parsimonious model
Revisit x condition 0.30 0.18 0.16 1.67 2 2 2 Model 1 r = .28, r change*; Model 2 r = .28, r change NS. * Significant at p<.05; ** Significant at p<.01 This model suggests that though there was no significant difference between conditions in the frequency of revisiting steps containing visualizations across days, there is likely a difference in why or how they revisited steps. This would explain why revisiting may have been beneficial for one condition. It is possible that in the clustered condition, when students began a new session, they revisited steps more deliberately, because they remembered questions or confusions from the previous session; whereas in the distributed condition, they may have revisited less deliberately, with a goal to remind themselves of the unit, but not in a way that encouraged knowledge integration. While we cannot determine this with certainty, we can determine whether there are differences in the patterns of revisiting by condition. For instance, we might hypothesize students in the clustered condition would tend to revisit steps upon reaching a point of confusion, whereas students in the distributed condition would primarily revisit in the first minutes of a new session. Preliminary results suggest that revisits across days tended to be proximal jumps within one-two steps for the distributed condition, and to occur in the beginning of a new session (Figure 5).
Figure 5. Patterns of revisiting steps containing visualizations across days, by condition. Students in the clustered condition revisited visualizations towards the end of the unit, but students in the distributed condition revisited proximal steps.
ISLS
377
ICLS2012
Students in the clustered condition tended to return to earlier steps while composing their Energy Stories or other later steps, which would have provided clearer opportunities for restudy, better supporting students to integrate their understanding.

In WISE, clustered practice supported the development of an integrated understanding of the role of energy in climate processes, whereas distributed practice reinforced the idea that each activity is self-contained. We interpret the further advantage for the clustered late condition as evidence that the students were also able to integrate ideas learned during the course, but outside of the unit. Ultimately, this study is not focused merely on clustered or distributed practice, but rather how they occur when there is not an emphasis on cumulative learning in classrooms. In the distributed condition, more revisiting was associated with lower scores on the later assessments, but in the clustered condition, more revisiting was associated with higher scores on later assessments. This is explained through the analysis of log files, showing that revisiting patterns differed by condition, affording restudying that would be expected to lead to different opportunities for knowledge integration. In the distributed condition, students revisited proximal, related steps, meaning that they had the opportunity to develop a better understanding of particular ideas. In the clustered late condition, students revisited various earlier steps while writing explanations late in the unit, affording opportunities to integrate their ideas across the unit. Further study is indicated, as this work was conducted in the classrooms of a single school and using a particular curricular unit. Based on these findings, further research is warranted on possible scaffolds to support the types of revisiting observed in the clustered condition, should distributed practice be used.
References
Kornell, N., & Bjork, R. A. (2008). Learning Concepts and Categories. Psychological Science, 19(6), 585. Kornell, N., Castel, A., Eich, T., & Bjork, R. A. (2010). Spacing as the Friend of Both Memory and Induction in Young and Older Adults. Psychology and Aging, 25(2), 498. Linn, M. C. (2006). The knowledge integration perspective on learning and instruction. In K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 243-264). New York: Cambridge University Press. Pashler, H., Bain, P. M., Bottge, B. A., Graesser, A., Koedinger, K., McDaniel, M., & Metcalfe, J. (2007). Organizing Instruction and Study to Improve Student Learning. IES Practice Guide. NCER 2007-2004. National Center for Education Research, 63. Seabrook, R., Brown, G., & Solity, J. (2005). Distributed and massed practice: From laboratory to classroom. Applied Cognitive Psychology, 19(1), 107-122. Slotta, J., & Linn, M. C. (2009). WISE Science: Web-Based Inquiry in the Classroom. New York: Teachers College Press. Smith, S., & Rothkopf, E. (1984). Contextual enrichment and distribution of practice in the classroom. Cognition and Instruction, 1(3), 341-358. Sobel, H. S., Cepeda, N. J., & Kapler, I. V. (2010). Spacing effects in real world classroom vocabulary learning. Applied Cognitive Psychology. Svihla, V., Gerard, L., Ryoo, K., Sato, E., Visintainer, T., Swanson, H., . . . Dorsey, C. (2010, June 29 - July 2). Energy across the Curriculum: Cumulative Learning Using Embedded Assessment Results. Paper presented at the International Conference of the Learning Sciences, Chicago, IL. Svihla, V., & Linn, M. C. (2011). A Design-based Approach to Fostering Understanding of Global Climate Change. International Journal of Science Education, 1-26. doi: 10.1080/09500693.2011.597453 Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theoriesan embodied modeling approach. Cognition and Instruction, 24(2), 171-209. Acknowledgments Though funded by the NSF (Grant #0822388), the views expressed here are not necessarily those of the NSF. We would like to acknowledge the teachers and students who participated in this research, and express thanks to other members of the research group for their feedback.
ISLS
378
ICLS2012
Technology for Learning: Moving from the Cognitive to the Anthropological Stance
Michael Eisenberg Dept. of Computer Science and Institute of Cognitive Science University of Colorado, Boulder USA duck@cs.colorado.edu Abstract: Most current research in educational technology proceeds from a philosophical stance based upon cognitive science. Typical questions implied by this stance include: How do we help students develop more accurate or fruitful knowledge structures? How do we overcome misconceptions in science or mathematics? How do we train skills more efficiently, employing what we know of human memory and learning? How do we encourage the growth of metacognitive skills? While such questions are interesting, this paper argues that they are ultimately peripheral for the purposes of designing effective educational technology. Indeed, the "cognitive science stance" tends to ignore or take for granted issues that are, in fact, much more central and productive in guiding design. These are questions rooted in anthropology, not cognitive science. They include: How and why do children make friends? What features of physical settings encourage (or discourage) the development of children's interest? Why, and in what situations, do children develop intense intellectual passions and obsessions? This paper discusses the sorts of questions that designers of educational technology need to ask in order to make progress beyond the confines of cognitive science research.
Introduction
Educational technology researchlike any other sort of researchtends to begin with a set of (often unstated) philosophical assumptions; these assumptions in turn guide the nature of questions that researchers feel encouraged or permitted to ask. In the current intellectual and political climate of educational technology, it is fair to say that the dominant philosophical assumptions are derived from the (by now) half-century-old field of cognitive science. Indeed, for many researchers, there is a natural tendency to blend (if not equate) the notions of "applied cognitive science" and "educational research". This paper will, in fact, challenge the merits of cognitive science as a disciplinary foundation for educational design. Nonetheless, before proceeding further in this discussion, it would be worthwhile to illustrate the central concerns and findings of the "applied cognitive science" approach. Probably the best exemplar of this approach can be found in the widely cited National Research Council report How People Learn [2000], which has justifiably been regarded as a landmark publication in the learning sciences. Several "key findings" reported in that book are summarized in a smaller companion volume [NRC 1999, ch. 2], and can be presented here as an illustration, in distilled form, of the applied cognitive science approach: Key Finding #1: Students come to the classroom with preconceptions about how the world works. If their initial understanding is not engaged, they may fail to grasp the new concepts and information that are taught, or they may learn them for purposes of a test but revert to their preconceptions outside the classroom. Key Finding #2: To develop competence in an area of inquiry, students must: (a) have a deep foundation of factual knowledge, (b) understand facts and ideas in the context of a conceptual framework, and (c) organize knowledge in ways that facilitate retrieval and application. Key Finding #3: A metacognitive approach to instruction can help students learn to take control of their own learning by defining learning goals and monitoring their progress in achieving them. It is worth reflecting upon these key findings, to see what they imply for the pursuit of design in educational technology. A designer responding, for example, to the first finding might attempt to create a computer application that identifies children's misconceptions about (e.g.) physics, and that then presents children with examples whose purpose is to challenge or contradict those misconceptions. A designer responding to the second finding would plausibly focus on creating intelligent educational (or tutoring) applications that (a) represent knowledge structures characteristic of expert thinking, (b) interpret or diagnose children's knowledge structures, and (c) compare the two in order to see what material or ideas to convey next. A designer responding to the third finding might well focus on the pace of presentation of material: perhaps we might design an application that, from time to time, encourages children to reflect upon their own progress, or to identify their own learning styles.
ISLS
379
ICLS2012
These suggestions do not, of course, exhaust the realm of "applied cognitive science" design for education, but they are typical of that approach, and directly responsive to the key findings reported in How People Learn. More broadly, these types of design approaches are now generally regarded as "scientifically based" (or, in another popular phrase, "evidence-based"), in that they are responsive to well-designed, reliably reported experiments investigating human learning and cognition. Identifying such approaches as "scientifically based" tends to imply, by contrast, that alternative approaches to educational technology design are unsound, vague, sloppy, irrational, or counterproductive. The remainder of this paper is devoted to the argument that this "cognitive science based" approach to educational technology design is, at best, radically incomplete. To preview the argument at this early stage: the point is not that the research behind (e.g.) How People Learn is in any way flawed or unreliable. The point is rather that this research is ultimately peripheral to the real challenges of education, and to the most interesting and important dimensions of educational design. The key findings quoted above are not so much wrong, as simply not all that important. To design productive, meaningful new technologies and systems for learning, we will have to ask questions that are presently invisible to the cognitive scientists. What follows, then, is an expansion of this argument. The goal here is to suggest an alternative theoretical perspectiveone informed by cognitive science, and consistent with its findings, but based largely in children's anthropology. The anthropological perspective tends to highlight different issues in children's intellectual development. It inquires about friendships, role models, family dynamics, peer pressure, physical settings, neighborhoods, rituals, intellectual passions and obsessions, and patterns of day-to-day activities (among others). It seeks to ground technological design in response to these factors, and thus tends to view "technology" as an interwoven system of material and computational artifacts (rather than, e.g., a single particular computer application). It tends to de-emphasize classroom experience, if only because that experience is no longer identified as the primary or default source of children's learning. It is less dependent on laboratory findings and computational models of learning, and thus more alert to the crucial elements of intellectual biography that are of necessity ignored or suppressed in the laboratory. The following (second) section of this paper begins by taking a closer look at the origins, occasional strengths, and numerous limitations of the cognitive science approach to design in educational technology. The goal here is affirmatively not to critique any particular research project or finding, but rather to question the reflexive identification between "cognitive science" and "scientific educational research". The third section presents a variety of contrasting anthropological themes, based in literature from that field, that highlight important questions for the design of novel educational technology. The fourth and final section suggests a sampler of potential design projects and approaches emerging from anthropological questions. Along the way, throughout the paper, we will make use of occasional illustrative examples from our own work, and that of others.
Cognitive Science as an Incomplete Foundation for Educational Design

In order to re-examine the role of cognitive science in education, we would do well to view the discipline as itself a type of intellectual cultureone that (as all intellectual cultures do) highlights certain questions, or types of questions, while suppressing others. For the purposes of our argument here, we can group our observations along three axes: namely, the cognitive portrait of intellectual growth and development, the cognitive stance toward educational infrastructure and institutions, and the cognitive view of intellectual motivation. All of these various dimensions are in fact tightly interwoventhere is a certain (probably laudable) internal consistency within the cognitive science culture. Still, for the sake of clarity, we can disentangle elements of each dimension for individual discussion. Intellectual Growth and Development. In general-and at the risk of only a bit of caricaturethe cognitive portrait of intellectual growth and development is one derived from a natural mapping between intellectual performance and prominent characteristics of traditional (von Neumann) computational systems. Thus, a cognitive account of education will stress such activities as problem solving; skill acquisition; retrieval, recall, and reconstruction of factual material; mental models of complex structures or dynamical phenomena; and metacognition. This is of course only a partial list, but it is representative of the traditional core of the field. The computational roots of the cognitive tradition are worth noting here. A view of problem solving rooted in algorithmic research on search strategies will tend to ask questions such as: what kind of search strategy is the student using to solve a problem (e.g., in solving an integral equation)? How flexible are the student's strategies? What parameters (e.g., depth of search, pruning of alternatives) characterize effective searching? Likewise, a view of factual retrieval rooted in computational models of memory will focus on questions such as precision (relevance) and recall (coverage) of the memory process; training and testing schedules that lead to more effective retrieval; elaborative structures (such as analogies) that aid retrieval; and so forth. More recently, the landscape of cognitive science has expanded, with encouraging prospects: there has been an increased attention to hitherto underexplored topics such as the role of mental imagery in intellectual performance [Mathewson 1999], gestural and "whole-body" aspects of learning and cognition [Nemirovsky et al.
ISLS
380
ICLS2012
1998; Goldin-Meadow 2003; Howison et al., 2011], and the supportive role of cognitive neuroscience in understanding and assessing learning (see [Johnson, 2011] for a good introductory text). These are fascinating and important developmentsthough again, at the risk of painting with something a broad brush, they have not greatly altered the traditional intellectual emphasis of the cognitive approach to learning. That is, one might (e.g.) study the role of mental imagery in employing search strategies without altering the disciplinary focus on search as a central element of learning. Likewise, one might focus on the neurological changes observable during skill acquisition without altering one's identification of skill acquisition as the basis of educational performance. Structural assumptions: Educational Institutions and Infrastructure. The intellectual foundations of educational cognitive science, as described in the preceding paragraphs, are accompanied by a number of underlying structural assumptions about the nature and settings of learning and development. For the most part, cognitive science research in education devotes its attention to direct instruction as a motor for human learning. This is not to say that the proposed low-level computational structures (search strategies, rule bases, semantic networks, and so forth) are themselves the targets of direct instruction, though that is sometimes the case; rather the style of research focuses on settings in which a teacher (human or computer) is placed in the role of moving the student from one intellectual state to another. As a corollary, a disproportionate level of attention is paid to classroom settings in the cognitive science literature on education. In principle, there is no necessary connection between the intellectual focus described earlier (problemsolving, skill acquisition, and so forth) and the structural settings of schools and classrooms. One might study (e.g.) skill acquisition in vocational situations [cf. Rose 2004], or problem solving as experienced in informal conversations, or metacognition in the context of free exploratory play. Still, classroom research represents a path of least resistance: direct instruction is, after all, relatively easy to implement, alter, observe, and measure. Studying a topic such as informal conversationswhere there are no control groups to be foundnecessitates an approach closer to ethnography or anthropology (a point to which we will return). A structural emphasis on classroom instruction carries with it a variety of related (often unstated) structural assumptions about how and where learning occurs, and how it is to be studied. For instance, the disciplinary domains of cognitive educational research rarely stray beyond the boundaries of school material. It would be unlikely to study skill acquisition for idiosyncratic topics, even when those topics may be of some historical relevance (how do children learn to tend gardens in family settings?), personal meaning (how do children acquire skill in caring for pets?), or cultural resonance (how do children develop and judge performance in skateboarding?). Likewise, by assumption, most research focuses on learning that takes place at designated times and locations, rather than (say) intermittently over the course of months or years; and most research focuses either on individual learning (e.g., as measured by tests) or learning in the company of a restricted cohort of age-mates. All of these implicit assumptions may appear natural when the focus is on classrooms; but they all place strong and to some degree arbitrary restrictions on what is permitted under the aegis of "educational research". In this context, there is still one more structural assumption to be mentioned: because of the cultural emphasis on "scientific evidence" that has come to dominate the cognitive approach to education, there are strict constraints placed on the design process for those who would create novel educational artifacts. In particular, the constraints of research strongly point designers toward creating, and studying, a single individual system or artifact that permits of direct experimental assessment. Often this takes the form of a specific computational system or curricular intervention; and the standard form of research is to compare the experimental student (using the system) with the control student (who doesn't use the system). It is rarely acknowledged how arbitrary and impoverished this single-intervention stance can appear to those who study the history of technological design. It is much more likely for technology to be experienced as a widespread ecosystem of innovationsa collection of tools, devices, and techniques that collectively alter personal experience. To take an example, one might ask what historical effect "the printing press" had on the children of the sixteenth century, but such a question would have to take into account the growth of printingrelated professions (such as bookbinding), the growth of a paper industry, and the advent of new literacy-based professions such as financial accounting, to name just a few factors. In a similar vein, a present-day educational designer might wish to study (say) the impact of fabrication technologies, or 3D display devices, on education; but the single-intervention approach would limit the broad scope of this study to something far less meaningful, such as comparing a program presented on a 3D screen to a similar program on a traditional screen. In short, then, the constraints of "evidence-based" research place counterproductive restraints on educational design, limiting our thinking to specific, artificially isolated technological changes and not to larger systemic or cultural innovation. Intellectual Motivation. As noted in the previous paragraphs, the disciplinary center of gravity of the cognitive approach tends toward those subjects that are representative of classroom instruction. Thus, one will tend not to see educational research that focuses on (e.g.) hobbies, personal interests, or subject matter that the school system might view as exotic or unorthodox. Likewise, there is little encouragement for technological design that might impact learning in a preferentially non-school context.
ISLS
381
ICLS2012
In fact, the disciplinary limitations of the current research culture are even somewhat more severe. It's fair to say thateven within the bounds of the classroomthere is an emphasis on subject matter perceived as "rational", or at least utilitarian. Broadly speaking, the most prominent candidates for study are reading comprehension, mathematics, and the natural sciences. These subjects are also the foci of technological innovations for educational research. It is less typical to see technological design or research aimed at (e.g.) music appreciation, color sense, creative writing, or performance. It is interesting to step back a bit and speculate on why, precisely, the disciplinary focus should be as relatively narrow as it is. In part, one might argue that the underrepresented disciplines are precisely those in which affective, aesthetic elements are most prominentand these elements are least familiar within the cognitive science tradition. While some cognitive scientists have indeed advanced fascinating ideas about the psychology of aesthetics and the arts [Solso 1994, Ramachandran 2004 ch. 3] these ideas have been far less straightforward to implement in computational form than their counterparts in the realm of (e.g.) mathematical problem solving or vocabulary retrieval. At the same time, it is worth noting that, arguably, there really is a strong affective and aesthetic component to even the "allowed" subjects of mathematics, engineering, and the natural sciences. Often, biographies of scientists and mathematicians place these elements into strong relief. Still, these are aspects of the subjects that tend to be underexplored in learning sciences research: it is far more typical to ask (say) why a student might misunderstand Newtonian mechanics than to ask why a student might (or might not) be drawn to study physics. We will return to these themes (of affect and aesthetics) shortly; for the present, however, we might focus on yet another aspect of the cognitive approach that impacts choice of subject matter. This is the almost relentlessly utilitarian portrait of education and learning that appears to motivate much of the research community in its decisions on what to study. That is, there is a general (if unstated) syllogism at work that (a) the purpose of education is to acquire a job, (b) there are relatively more jobs to be had in technical fields, and therefore (c) the most important subjects for research and study are technical subjects. By the same reasoning, subjects that offer few prospects for employment (or monetary reward), such as classics or philosophy, are rarely the subject of technological design or cognitive research. It is an open question whether the utilitarian view of education implicit in the research choices of the learning sciences community is in fact shared by youngsters. There are reports in the popular press of continuing problems with academic retention among students in technical fields [NY Times, 2011]; at the same time, children and teens spend large amounts of time in activities (e.g., sports) for which there is little prospect of gainful employment. Children persist in doing what they like; while the stock images promoted by the educational community that purport to motivate an interest in scientific study (e.g., the need to "acquire skills for the twenty-first century", or to "compete with China") seem to have little or no resonance with the students. Overall, as noted in a recent National Science Board report [2010], "[f]or at least the past two decades, about one-third of all freshmen planned to study science and engineering"; or in other words, the per capita level of college student interest in technical fields has remained approximately unchanged. Thus, measured in indicators of student interest, the result of a decade of work since How People Learn has been a disappointment. As a result, it has become something of a commonplace to say that there is a need for more attention to "motivation" within the cognitive science community, particularly with regard to education. Indeed, the authors of How People Learn anticipated as much: Although cognitive psychologists have long posited a relationship between learning and motivation, they have paid little attention to the latter, despite its vital interest to teachers. Research has been done on motivation, but there is no commonly accepted unifying theory, nor a systematic application of what is known to educational practice. [NRC, 2000, p. 280] In point of fact, there is much more missing in the cognitive approach than an attention to motivation alone; "motivation" is rather a thin term for the myopia that is at work here. More generally, there is a lack of attention to the element of biography, of personal narrative, that underlies young people's choice of activities and commitments. Because the cognitive approach emphasizes a purely rational or utilitarian view of education, it exhibits a concomitant de-emphasis on the individual or idiosyncratic nature of intellectual development. Even the favored research scenario of the learning sciences communityimplementing some innovation within a classroom (or better yet, a school district) and studying the average, large-scale impact of that innovation ignores, by its very nature, the sort of innovation that would have an intense but highly personal impact on the occasional student's thinking. The idea that a student might have a passionate, obsessive, aesthetically based, or highly individual pattern of interests is not addressed. To sum up the argument to this point: the research culture of the learning sciences community, based as it is in the disciplinary tradition of cognitive science, exhibits important blind spots. Some of these blind spots are disciplinarysubjects (such as the arts) that are relatively underrepresented both in the research literature and in
ISLS
382
ICLS2012
technological design. Some of these blind spots are structuralan overemphasis on classroom settings, formal instruction, and short-term learning. Some of these blind spots are philosophicala view of education as exclusively utilitarian, a lack of attention to personal or idiosyncratic factors in intellectual growth, and an overreliance on computational models. Collectively, these blind spots have serious consequences for the designer interested in creating new technologies for education. Sometimes, it is the subject matter that is problematic: designing a computational system to help students learn (e.g.) choreography, or pottery, or costuming would be seen as an offbeat (and perhaps frivolous) choice of disciplinary focus, even if creating such a system could raise fascinating issues for the designer and have tremendous value for the eventual student. Sometimes the structural bias toward the classroom is problematic: designing novel artifacts to help youngsters tend to their backyard, or to enjoy birdwatching in their neighborhood, or to add programmable computational elements to their own clothing and home furnishings, would be seen as somehow less than fully "educational". Sometimes it is the philosophy that is problematic: designing (say) a novel musical instrument, or science kit, or mathematical puzzle that would capture the imagination of a small number of students would be seen as "low-impact", and hence a waste of time. The limitations of the current research culture have, in this fashion, collectively hobbled the imaginations of the designers of educational technology. In the following section, we discuss a variety of more productive educational themes, based in anthropology, that could reinvigorate technological design for children.
Children's Anthropology and Technological Design

The previous section has sketched a problematiceven depressingstate of affairs in educational technology research. The current research culture is limited in subject matter, in its vision of the setting and timeframe of intellectual development, in its inattention to personal narrative, and in its utilitarian view of education in general. Where, then, can educational designers turn for an alternative theoretical foundation for research? A source of some hope for the field can, we argue, be found in the burgeoning field of children's anthropology. Sources such as [Opie & Opie, 1959; Csikszentmihalyi et al. 1993; Lancy, 2008; Lancy et al., 2010] are rich in thoughtful description of the ways in which children grow, develop (or fail to develop) intellectual interests, and arrive (for better or worse) at an understanding of their own biography. The themes and issues raised in this field of research offer a fresh lens through which to approach technological design for children. What follows in this section is an outline of five prominent themes in children's anthropology, and suggestions for how these themes could inform novel design projects. In the final section, we will expand on several of the brief suggestions introduced here, and sketch several potential directions for design informed by anthropology. Friendship and Peer Culture. A recurring theme in the anthropological literature on children is the role of friends (and peers more generally) on development [Corsaro, 2003; Milner, 2004; Harris, 1998]. Children's tastes and language areas has been well-documentedstrongly influenced by their peers. Perhaps more important for our purposes, there seems to be a relation between the role of peer culture and the question of "intellectual motivation" discussed earlier: for instance, one of the memorable findings of Coleman's [1961] famous mid-century study of high school students was that relatively few students actually admired academic success among their peers. There is thus a strong need to design educational technology that not only instructs students, but that in some sense is answerable to the question of creating a supportive peer culture. There might be stronger attempts, for instance, to build online social networks among students based not on a loose definition of "friendship", but supporting shared intellectual interests or construction projects. Likewise, one might design educational technology whose effects include not only intellectual advancement but increased popularity: for example, one might argue that computational kits for designing programmable clothing, or for creating innovative public artwork, would have much greater motivational force for students than (e.g.) programs to help raise test scores. Economies. It should come as no surprise to parents and teachers that an important element of children's culture is the presence of various informal "economies" based on (among other objects) baseball or character cards, model cars, doll clothing and accessories, "virtual" (game-related) goods, and the like. Children collect, trade, and display these objects in numerous forms, andapparently since the late medieval period if not earlier [Orme 2001, p. 177]these objects play a recurring role in knitting together the friendship and peer cultures mentioned earlier. From the standpoint of educational technology design, an attention to children's economies would likely lead to (e.g.) the design of "collectibles" that attempt to push the envelope of challenging content. One might imagine (for example) "mathematical collectibles", or perhaps small computationally-enriched pieces that, when combined together, can produce larger constructions. (For a survey of still-early ideas along these lines, see [Schweikardt & Gross, 2007].)
ISLS
383
ICLS2012
Holidays and rituals. As noted in [Opie & Opie, 1959], the yearly cycle of holidays and rituals plays an important role in children's lives; many holidays (Halloween, April Fool's Day, Christmas) have especially strong associations with children's activities, and birthday parties are likewise important events. A reasonable prospect for educational technology might be to exploit the importance of these events through the design of personalized or creative activities: children might (e.g.) design high-tech costumes for Halloween, magical illusions for April Fool's, or aesthetically appealing packages and wrapping for Christmas presents. Another possibility for educational design would be to create technologically-enriched construction activities, puzzles, or games specifically geared toward events such as birthday parties. Role models. In biographies of scientists and mathematicians, the accounts of early interest often (perhaps surprisingly often) include the presence of an inspirational role modela teacher, relative, or historical figure. Just to take one example (many more could be mentioned): the classic book Microbe Hunters [De Kruif 1926], with its capsule biographies of intellectually adventurous scientists, appears to have strongly influenced a generation of twentieth-century biologists. (See, for instance, [Hargittai 2011, p. 53].) Technological design for education could attempt to play upon the power of role models in a variety of ways: one might (e.g.) design laboratory spaces intended to reproduce settings, or re-enact events, associated with famous historical figures; or one might create a series of ongoing public scientific projects or experiments whose explicit aim is to highlight the narrative, biographical dimension of science education, showing the scientist at work over time. Neighborhoods and Hangouts. The settings in which young people live and congregate constitute yet another important theme in children's anthropology. Historically, it is not uncommon for older children and teenagers to find places ("hangouts") in which to meet; and these places in turn take on the character of the local youth culture (for good or ill). More recently, as Mintz [2004, p. 347] has documented, children spend more time within their homes (and within their rooms), perhaps attenuating the role of the neighborhood hangout; though in other cultures, teenagers still find their way to the local mall. An educational designer might, in response to these factors, attempt to create settings that have perhaps more challenge or possibility than a shopping mall (maybe incorporating some of the features of interactive science museums); or we might design interesting activities that can be incorporated within the settings where young people tend to congregate. There are still numerous other anthropological themes that could be included in a list of the sort given here. We might, for example, inquire about the relationships that children have with special objects in their lives (souvenirs, hand-constructed items, sentimental gifts), and use that as a basis for designing novel types of educational artifacts. Or we might note that for many children (and older students), a great deal of productive time is conducted on their own, without the company of friends or peers [Csikszentmihalyi et al. 1993, p. 90; Arum & Roksa 2011, p. 100]; so it might be useful to design technologies that can, in some fashion, alleviate the burdens of loneliness while still permitting children to work in solitude. Or we might study the design and decoration of children's rooms, with an eye toward creating display artifacts (perhaps computationally-enhanced) that offer intellectual challenge. Or we might study the patterns of children's intellectual passions and obsessions, and attempt to design educational technology to fit or exploit those patterns. The following section will outline a couple of potential projects based on themes of this sort. Before moving on, however, it is worth returning to the opening paragraphs of this paper; and it is worth comparing the sorts of design projects encouraged by the "key findings" of How People Learn with the sorts of design projects sketched here. There is nothing intrinsically wrong with (e.g.) identifying students' misconceptions, building knowledge structures, and promoting metacognitive skills; but these are peripheral issues when the surrounding culture of children's lives (counterproductive friendships or peer pressure, anti-intellectual hangouts, absence of role models) mediate against intellectual growth. Cognitive computational models are impoverished representations of children's minds and lives, and impoverished bases for technological innovation. As designers, the themes of anthropology collectively tell us much more about how and where to direct our energies in technological research and creation.
Moving Toward the Anthropological Stance: a Sampler of Potential Projects

In this final section, we describe several themes for larger research projects, based upon the anthropological ideas introduced in the previous pages. The suggestions of this section are only a very small sampling of the multitude of projects that could be undertaken; the reason for presenting them here is not that these are the best possible project ideas (a wildly optimistic claim), but rather that they are directly responsive to major issues raised in the literature of children's anthropology.
ISLS
384
ICLS2012
Project theme 1. Incorporating computational elements into children's performance. There are numerous opportunities within children's culture for various types of public performance. Students take part in sporting events; they take the stage in talent shows, magic shows, plays, glee clubs, and public performances; they participate in events that (at times) occur outside the boundary of the school, such as slam poetry competitions. In all these events, there are elements of costume, choreography, props, sets, and so forth. One potential project, then, would be to incorporate novel types of technologies and design activities into these performance settings. One might employ computationally-enhanced textiles [Cf. Buechley & Eisenberg, 2008] to create costumes, sports uniforms, and theatrical backdrops; one might use computer-generated lighting effects in stage performances; one might employ novel musical instruments or sound effects for glee clubs or poetry competitions; one might design novel optical illusions or trick apparatus for magic performances. In summary, then, there are ways of blending ideas from programming, the physical and natural sciences, engineering, and computer-generated arts into the types of events that matter to many students. It should perhaps be pointed out that these types of activities are associated with creative peer communities among students; they are the places where (often) close and lifelong friendships are formed. In this sense, incorporating novel technology into these activities is a potentially powerful way of placing creative learning opportunities within contexts where peer pressure will not run counter to academic achievement. Project theme 2. Public displays and activities. The previous project idea focused primarily on settings for established traditions of youngster's performances (sports, glee clubs, etc.). There are still other modes of public performance that might serve as excellent venues for novel designs in educational technology. For example, it might be possible to create large outdoor surfaces suitable for projected displays, and to permit youngsters to employ various types of projectors (ranging from large-scale to nanoprojectors) to create massive animated or graphical effects in public spaces. Still other types of group public displaysranging from flash mobs to political protests to Christmas carolingmight allow for interesting technological enhancement. In some cases, public displays are associated with the individual home: one might imagine creative child-accessible technological enhancements for holiday decorations such as Halloween or Christmas lights, or one might create novel techniques for presenting interesting scientific ideas (rather like a "home scientific museum") in a local front yard. The larger idea here is to rethink the possibility of public settings for educational purposes. Project theme 3. Children's rooms as objects of design. Thus far, we have focused our discussion on public eventsperformances and outdoor settings. As mentioned earlier [cf. Mintz 2004] present-day children spend much of their time alone at home, in their rooms. One potential response to this demographic shift would be to create technological activities that are geared toward beautifying, personalizing, or otherwise increasing the intellectual interest and affective poignancy of children's rooms. Electronic textiles might create interesting effects on curtains or wallpaper; long-term computational simulations could be programmed to run over a period of weeks or months, projected on the walls or ceiling of the room; computationally-controlled mobiles, wall hangings, and kinetic artwork could be designed with novel fabrication and design tools; children might be able to personalize or interact in interesting ways with nightlights, a home terrarium, or displays of collectibles. Here, the overarching theme is to design technology that speaks in personalized, perhaps private, ways to children's intellectual passions and obsessions, and that allows them to recreate their physical environment to reflect their own interests and personality back to them. These types of projects are suggestive of the directionsanthropological directionsin which the design of educational technology needs to turn. Cognitive science, on its own and unaided by the anthropological perspective, has taken us in the wrong direction for too long. The goal of education should be to fashion a living biography, not an internal mechanism.
References
Arum, R. and Roksa, J. [2011] Academically Adrift. Chicago: University of Chicago Press. Buechley, L. and Eisenberg, M. [2008] The LilyPad Arduino: Toward Wearable Engineering for Everyone. IEEE Pervasive Computing, 7:2, pp. 12-15. Coleman, J. [1961] The Adolescent Society. New York: Free Press. Corsaro, W. [2003] We're Friends, Right?: Inside Kids' Culture. Washington, DC: Joseph Henry Press. Csikszentmihalyi, M.; Rathunde, K. and Whalen, S. [1993] Talented Teenagers. New York: Cambridge U. Press De Kruif, P. [1926] Microbe Hunters. Orlando, FL: Harcourt. (1996 edition) Goldin-Meadow, S. [2003] Hearing Gesture. Cambridge, MA: Harvard University Press. Hargittai, I. [2011] Drive and Curiosity. New York: Prometheus Books. Harris, Judith Rich [1998] The Nurture Assumption. New York: Free Press. Howison, M. et al. [2011] The mathematical imagery trainer: from embodied interaction to conceptual learning. Proceedings of SIGCHI 2011, pp. 1989-1998.
ISLS
385
ICLS2012
Johnson, M. [2011] Developmental Cognitive Neuroscience (3rd edition). Malden, MA: Wiley. Lancy, D.; Bock, J.; and Gaskins, S., eds. [2010] The Anthropology of Learning in Childhood. Lanham, MD: Altamira Press. Lancy, D. [2008] The Anthropology of Childhood. New York: Cambridge University Press. Mathewson, J. H. [1999] Visual-spatial thinking: an aspect of science overlooked by educators. Science Education, 83: 33-54. Milner, M. [2004] Freaks, Geeks, and Cool Kids. New York: Routledge. Mintz, S. [2004] Huck's Raft: a History of American Childhood. Cambridge: Harvard University Press. National Science Board [2010] Science and Engineering Indicators 2010; www.nsf.gov/statistics/seind10/c2/c2s2.htm National Research Council [1999] How People Learn: Bridging Research and Practice. Washington, DC: National Academy Press. National Research Council [2000] How People Learn. Washington, DC: National Academy Press. Nemirovsky, R.; Tierney, C.; and Wright, T. [1998] Body motion and graphing. Cognition and Instruction, 16:2, pp. 119-172. New York Times (C. Drew) [2011] "Why Science Majors Change Their Mind (It's Just So Darn Hard)", Nov. 4, 2011 (Retrieved from www.nytimes.com website.) Opie, I. and Opie, P. [1959] The Lore and Language of Schoolchildren. New York: New York Review of Books (2001 edition). Orme, N. [2001] Medieval Children. New Haven: Yale University Press. Ramachandran, V. S. [2004] A Brief Tour of Human Consciousness. New York: Pearson Rose, M. [2004] The Mind at Work. New York: Viking. Schweikardt, E. and M. D. Gross [2007] "A Brief Survey of Distributed Computational Toys" DIGITEL 2007: The First IEEE International Workshop on Digital Game and Intelligent Toy Enhanced Learning Solso, R. [1994] Cognition and the Visual Arts. Cambridge: MIT Press.
Acknowledgments
Thanks to Mike Petrich, Clayton Lewis, Gerhard Fischer, Mitchel Resnick, Leah Buechley, Mark Gross, and Yasmin Kafai for many helpful and provocative ideas and conversations.
ISLS
386
ICLS2012
Re-presenting Complex Scientific Phenomena Using Agent-Based Modeling in Engineering Education

Paulo Blikstein, Stanford University, 520 Galvez Mall, CERAS 232, CA, USA, paulob@stanford.edu Abstract: A chronic problem in engineering education is that students knowledge is superficial, fragmented, and ungenerative. I embark from the conjecture that these shortcomings result from cognitive discontinuity between intuitive forms of sense-making that deal with objects and actions and algebraic formulas traditionally used in this discipline. I propose and that a new form of representation, agent-based models, may offer greater cognitive continuity from sense-making to disciplinary knowledge. Study 1, an analysis of textbook content and lecturer practice, confirmed the cognitive-discontinuity conjecture by implicating the epistemic challenges of converting sense-making into formulas. Study 2 was a longitudinal evaluation of an agent-based, complementary curricular unit for materials science undergraduates. A micro-ethnographic case study of one students progress through this unit documents the challenges and achievements of appropriating the agent based perspective as means of empowering sense-making.
Introduction
Representations are not benign curators of content. Rather, the medium in which knowledge is encoded may profoundly affect its message. It took a vast body of twentieth century scholarship in semiotics, philosophy, anthropology, and cognitive sciences to arrive at this counterintuitive thesis by which representations are far more than impartial containers for transferring abstract ideas (Brock & Price, 1980). Wilensky and Papert (2006) illustrate this relation between knowledge encoding and generativity by evoking paradigm shifts in the history of numeral systems. Consider the task of using Roman numeral to do multiplication. This task is so challenging that, in ancient Roman times, only a very specialized caste of workers, professional multipliers with years of training, could perform it. That multiplication would one day become a normative goal for all elementary-school children would have appeared ludicrous to the Romans, because their notion of what multiplication is was vested in its cumbersome semiotic system. Yet what multiplication is was irrevocably transformed with the advent of HinduArabic numerals and the place-value system and, with them, the prospects of democratizing multiplication (see also diSessa, 2000; Kaput, Noss, & Hoyles, 2002). Restructurations (Wilensky & Papert, 2006) of such epic epistemic magnitude occur in science, too. In particular, computational representations of phenomena are deeply transforming scientific practice (Oden et al., 2006). The redescription of natural phenomena via new ontologies is challenging knowledge encoding schemes previously taken for granted and consequently reorganizing discipline-based theory and practice (Kaput et al., 2002; Wolfram, 2002). Implicated in this current paradigm shift is a new domain of research, complexity science, which is changed what is being researched as well as the nature and knowledge this research is generating. The essence of the epistemic revolution is in focusing on micro-behaviors, emergent properties and stochasticity of natural and social phenomena, thus complementing or replacing the traditional emphasis on equational, differential, and deterministic modeling (Ioannidou, Repenning, Lewis, Cherry, & Rader, 2003; Wilensky & Resnick, 1995). Learning sciences researchers have examined the cognitive and pedagogical affordances of complexity science methodology in STEM education (e.g., Chi, 2005; Hmelo-Silver & Pfeffer, 2004; Jacobson, 2001; Jacobson & Wilensky, 2006), and whereas a number of studies have evaluated the prospects of introducing complexity as a new content area at the elementary (Danish, Peppler, & Phelps, 2011), secondary (Levy & Wilensky, 2009), and university levels, few studies have evaluated the utility of restructurating traditional engineering content via principles and methodologies of complexity. In this paper, I will focus on materials science as a content area, where agent-based modeling is widely used by experts. As I will demonstrate via interpreted ethnographic observations, although expert sense-making of material science phenomena is essentially agent-based, their inscriptions and documents make this sense-making process opaque to their students. During lectures, material scientists agent-based sense-making is manifest in their semiotic practices, most emphatically in narratives, sketches, and gestures they employ in order to communicate to students aspects of the phenomenon itself. Yet immediately the lecturers leap to a parallel system of practice involving the production and manipulation of symbolic inscriptionsequations describing the same phenomenon and they are regarded as the official representational form, and what students are supposed to really learn. As such, inherent yet undisclosed in expert practice is the recognition that agent-based narratives of materials science processes better enable learners to grasp the essence of phenomena, yet that these ultimately are second-class or na epistemic forms. ve
ISLS
387
ICLS2012
The current study is part of a larger research program that seeks to fill this gap in the literature by exploring the potential of restructurating existing college-level engineering content via epistemic practices of complexity science, including a re-taxonomization of the curriculum highlighting common micro-mechanisms across several different phenomena. In particular, this paper explores two research questions: (1) What are experts standard forms of content representation apparent in their multimodal discourse (utterance, gesture, inscription) during course lectures, and to what extend they use micro-mechanisms and principles of complexity science in their representations?; (2) Do students who design and build computational models in alignment with experts forms of content representation develop new forms of reasoning that are more productive in approaching familiar and unfamiliar phenomena?
Methods
The project was conducted in the design-based research approach (DBR). In particular, iterative cycles of design, implementation, and reflection generated empirical data whose analysis informed the development and refinement of theoretical models as well as improvement to the design (Collins, 1992; Confrey, 2005). DBR was selected as a suitable approach since I wished to investigate the experimental intervention in the actual context for which it was designed, an ongoing materials science college course, so that my data would authentically reflect the complexity of instructional practice. This study was part of a larger project in which 18 volunteer students from one sophomore materials science classroom participated in model-building activities, pre/post semi-clinical interviews, and questionnaires (Blikstein, 2009). The data are divided into two main parts. Part A: classroom observations and materials collected during a ten-week period of a materials science course, Microstructural Dynamics. I recorded, transcribed, and coded classroom episodes, collected handouts, homework, and textbooks, and kept a timestamped log of all equations, plots, drawings, and other materials instructors used during the lectures. Part B: materials collected as I worked with students in the course. I facilitated a three-week classroom assignment in full coordination with the lecturer. The outcome of the assignment was a computer model, a written report with a description of the algorithm, and an account of its validation with published data. I met each participant multiple times every week in videotaped sessions of 30-45 minutes. Sessions were individual, in the form of semi-structured, task-based tutorial sessions, during which students would simultaneously advance their modeling project and learn about the modeling language. In Year 2, I conducted 20-minute follow-up interviews with the participants, asking content-related questions and engaging in simple modeling tasks. The classroom observations and materials (part A) were analyzed categorizing different explanatory episodes accordingly to the prevalent representational infrastructure (agent-based or equation-based.) For the work with students (part B), in order to measure for any conceptual change resulting from the intervention, I collected field notes, conducted pre- and post-interviews and administered questionnaires. In particular, these instruments were designed to evaluate (a) knowledge about common content topics, (b) the identification of common micro-mechanisms across phenomena, and (c) their usage when articulating explanations about known and unknown topics. To elicit narrative about micro-mechanisms and their articulation, I created the paper modeling instrument, which consisted of computer modeling tasks done on paper. For example, I would ask students to draw on paper the user interface of their computer models, the elements of the simulation, and to subsequently animate the elements on paper using verbal explanations, arrows, multiple colors, or frame-by-frame representations. The corpus of data, including videotapes, student artifacts, and computer-interaction logs, informed a qualitative microethnographic analysis of participants multimodal behavior (Nemirovsky, 2011; Siegler & Crowley, 1991). In line with the study objectives, I especially sought to identify relations between, on one hand, the construction and mobilization of dynamical virtual representations and, on the other hand, evidence of conceptual change engendered via these activities. In this paper, I focus on one student for an in-depth case study, selected as paradigmatic exemplar of prototypical cognitive shifts observed throughout the study. The goal of this extended case study is to generate thick, rich descriptions (Geertz, 1973) of participants experience, not necessarily to determine the particular causal mechanism, or try to establish a basis for deterministic predictions in other subjects or circumstances (Nemirovsky, 2011, p. 316), but to offer evocative accounts of how students experienced the modeling activity.

This paper investigates an alleged misfit between intuitive understanding and formal representation in the domain of materials science. This misfit would be substantiated if we could witness broken chains of signification from simple language concerning a phenomenon itself to its algebraic re-modeling. For example, we would witness discontinuity in discursive referencewhat a text is talking aboutas the text shifts from intuitive to formal register. If indeed these broken chains exist, they could be embodied in a variety of texts, such as a lecturers oral presentation, course textbooks, and student argumentation. In the first part of this section, we will demonstrate these broken chains in lectures and textbooks. The second part contains data from
ISLS 388
ICLS2012
an extended case study about one student, Mike, who participated in an alternative curricular unit in which he authored and ran agent-based models of materials science phenomena.
The Invention of the Average Atom

The classroom data revealed that current instruction heavily relies on equation-based representations. On average, during a 30-minute class period, students were exposed to as many as 12 unique equations (approximately 2.5 minutes for each equation; all this, not counting intermediate steps in a derivation). Instruction consisted in a dense, linear narrative, in which sequences of equations and analytical expressions were the main content pieces, with occasional use of aids such as plots, photos of atomic structures, diagrams. In related work, I have shown that this linear narrative, based on the unpacking of these myriad equations, resulted in students superficial, fragmented, and ungenerative understanding (Blikstein, 2009; Blikstein & Wilensky, 2009, 2010). In particular, this equation overload impeded learner engagement in deep sensemaking as evidenced in their subsequent failure to reconstruct with understanding these sequences of equations. However, embedded within this formal, explicit portrayal of materials science as an equation-based practice, I detected an informal, clandestine grounding of the discipline in agent-based sense-making. In particular, instructors initial utterance and gesture, as they described a phenomenon to students for the first time, connoted that sense was made at the atomic level, yet they inscribed it algebraically in formulas. The instructors appeared to appreciate an epistemic discontinuity between agent-based and algebraic structurations, and offered heuristics for converting between these two semiotic registers. Yet students were confused and frustrated by these heuristics, which not only required highly sophisticated mathematical machinery but black-boxed the very intuitions that had made the agent-based register more generative in the first place. The same pattern of epistemic discontinuity, which I observed in the lectures, can be readily implicated in the textbooks accompanying the course. In what follows, I substantiate the above characterization of current instructional practices with excerpts from the official textbook and transcribed lectures. We begin with two selections from Phase Transformations in Metals and Alloys (Porter & Easterling, 1992). The first selection is from a chapter on crystal growth (note that grain is one of the technical term for crystal, and used in many of the transcriptions). The following text offers an agent-based description of one crystal as it grows through the migration of atoms from neighboring crystals.
101 102 103 104 105 106 107 108 [] the atoms in the shrinking grain detach themselves from the lattice on the high pressure side of the boundary and relocate themselves on a lattice site on the growing grain. Normally an atom in a crystal oscillates about a given site and is surrounded by neighboring atoms on similar sites. Normally the movement of an atom is limited by its neighbors and the atom cannot move to another site. However, if an adjacent site is vacant it can happen that a particularly violent oscillation results in the atom jumping over on to the vacancy.
The second selection is from a chapter on diffusion, describing the random walk of atoms within the material.
These excerpts demonstrate that the authors use micro-behaviors so as to introduce crystal growth, diffusion, and solidification. In line 101, the atoms detach themselves, and in 104-107, an atom oscillates and jumps over, if an adjacent site is vacant. Lines 105-108 essentially describe the canonical agent-based model of solid diffusion. These textbook introductions, which relied heavily on agent-based language, were mirrored in the instructors discourse during lectures about these same phenomena, in which instructors would first present agent-based narratives (and also gesture individual atoms moving around, colliding, and jumping from one crystal to another) and then rapidly plunge into differential equations asking students to forget the agentified atom and consider the average atom, embedded in a mathematical variable in which the microbehaviors are backgrounded. This brutal conversion from image to symbolthe very crux of students learning challenge in this unithappened in a mere few seconds, leaving students with few resources to reconstruct the conversion, thus departing from their intuitions about the behaviors of the atoms, moving from sense-making to formula-making.
Previous conceptions and model building

Materials science experts, it appears, make sense of phenomena at the atomic level (level of mechanism) but inscribe at the macroscopic level (level of formalization). Plausibly, then, materials science novices, too, should make sense at the level of mechanism and inscribe at the level of formalization. This section presents a microgenetic account of the data suggesting problematic consequences of this epistemic model of building expertise for student reasoning in the disciplines. Specifically, I will present a case-study analysis of one student who reasoned about a particular materials science phenomenon from both agent-based and equational perspectives. The data analysis will be guided by six dimensions, which with I interpret students transcription: macro vs. micro, static vs. dynamic, morphological vs. structural, grounded vs. ungrounded, analogy identification, and mechanism estimation. The macro vs. micro dimension refers to the idea of levels for example, atomic (the atoms are moving faster) versus aggregate (the liquid is heating up.). The dynamic
ISLS
389
ICLS2012
versus static dimension refers to students describing the phenomena as a snapshot (the atoms are organized into a crystal structure) or as an unfolding process (the atoms in the structure are vibrating and moving). The morphological versus structural dimension denotes students mentioning the form (the crystals are round and small) rather than the deeper physical structure (the crystals are just regular arrangements of atoms). A grounded explanation (note that the term grounded can have many different uses in the literature) is one that refers to a general trend for example, materials want to go to their lower energy state, or atoms are trying to minimize their energy. Mechanism estimation builds on work on mechanistic reasoning (Russ, Scherr, Hammer, & Mikeska, 2008) and points to students ability to induce the mechanism behind a particular phenomenon with partial information. The data in this subsection are divided into three main episodes: (a) a summary of the Mikes interview, prior to the modeling assignment; (b) an interaction with the teaching assistant during the modeling assignment; and (c) a follow up interview, one year later.
First clinical interview

In the first round of clinical interviews I conducted with students, I quizzed each on materials science content they had studied one week prior, including grain growth and recrystallization. I found that even with all the study materials at hand, these students struggled on conceptual items (Blikstein, 2009; Blikstein & Wilensky, 2009). Specifically, whereas the students possessed at hand all the pieces they had studiedmicro behaviors, macro behaviors, engineering laws, practical heuristics, and analogiesthey appeared to lack the sense-making glue that would compile this list into a coherent cognitive structure in the form of a relationally coherent narrative (diSessa, Gillespie, & Esterly, 2004). Conversely, students would merely recite back equations. In general, I attribute these difficulties to two strategies: (a) recalling ungenerative heuristics employed at the macroscopic level of the phenomena and not at the level of the mechanism, which is microscopic, and (b) inferring similarities at the macroscopic level (equations, shapes, appearance, morphology) and not at the level of mechanism and deep structures (Clement, 1993). In summary, these data suggests that students try to draw similarities between phenomena and infer mechanisms of new phenomena not by examining possibly correspondences of micro-behaviors, but by looking at aggregate aspects that oftentimes were just coincidental. For example, in his pre-interview, Mike considered crystal growth and recrystallization as equivalent phenomena because he identified circular shapes (a similarity trigger) growing in both, ignoring that the microbehaviors were considerably different. Therefore, when tasked with guessing the mechanism (mechanism estimation task) driving a new phenomenon, he would depart from the macro-morphology instead of structure. Present instructional methods, based on equational, aggregate descriptions, do not address these issues.
Mikes Model Building

A productive approach to the analysis of science learning has been to articulate sense-making as linking specific elements of a given display, such as a diagram or algebraic expression, to specific aspects of the phenomenon it represents (see, for example, Sherin, 2001). A remaining question, however, about the nature of the early explorative work that subsequently yields the more sophisticated linking. The following excerpts from Mikes model building process can be particularly illuminating. The analysis will focus on Mikes guided cognitive process, by which he built macro-to-micro connections via assigning new meanings to elements in his model. A week after the initial tutorial, Mike chose recrystallization as the phenomenon he would model for his assignment, and we had the first programming session. During the session, Mike developed a simple model that began by instantiating multiple circular particles on the screen and then made them all grow according to the known recrystallization formula. When Mike ran his simulation, he witnessed that it generated the behaviors he had intended: the circles grew on the screen, but he realized a fatal flaw. As the circular particles grew, they began to overlap, so that a simple summation of circular areas would clearly overshoot the calculation of the amount of material recrystallized, which was the outcome of the model. We studied the geometry of the problem for a long time to develop an equation that would account for the overlapping area, but it turned out to be a nontrivial problem. But then Mike had an insight: Well, if you had a grid, couldnt it count everything that has one thing over it, it could count as one, everything that has two things over it, it could count as zero? Mike deduced that if he were to model the screen as a grid and inspect the state of each patch within that grid, he would be able to circumvent his calculation challenge: every patch could look up and s ee how many particles were over it. This insight was a fundamental departure from his previous algorithmit was the first time he considered dividing the system into smaller elements, albeit still just as a counting tool. Importantly for my thesis, Mike began shifting toward a new model of the phenomenon under inquiry via a mundane fix to pragmatic problem: his micro breakthrough resulted from macro breakdown as mediated by a particular representational system. More intriguingly, we will now witness how this technical solution evolved into a cognitive model, as iconic elements took on the semiotic burden of phenomenal elements. I explained to Mike the idea of patches in NetLogo and showed him examples. But then he proposed an even more radical change: Would it be easier to just, instead of just creating all these recrystallized areas, with circles, would it be easier to just create them with the patches? I explained the right commands, and he continued: Transcription 1
ISLS
390
ICLS2012
201 202 203 204 205 206 207
Mike: I mean, just set them to different colors, like, ask patches [types the code] I don't even know if this will work. Hum. If you ask patch at random xcor..., ask them to set patch color blue. So, if you want to increase the radius, you are going to set more patches adjacent to them to be larger Are the patches aware of the radius around them? Hum, I'm asking the patches that are already blue, initially, I'm going to ask them, oh, that's going to be a problem
Mikes algorithm was simple and effective. He would start with a user-defined number of random seeds, and then make them grow while there still was unrecrystallized material (white patches, see Figure 1) around them. The code initially locates a non-white patch and asks it to do the following: look around yourself; if there is a white patch around you, expand yourself into that patch (i.e. set that patchs color to be your own color).
Figure 1. Mikes new patch-based recrystallization model In just some minutes, Mike transitioned from traditional representations in engineering classrooms (equations, heuristics, macroscopic descriptions) to an agent-based representation. As we track his transition, there is also a clear shift in epistemological resources employed. Before the interaction with the agent-based model (see First clinical interview) knowledge as propagated stuff, and authority-based explanations (Hammer & Elby, 2003) are more common. In the end, those resources are replaced almost entirely by knowledge as fabricated stuff, free creation, or direct perceptionall being resources more related to the practices of inquiry and discovery. As Mike switched to this new representation, he also mobilized his own connections to the domain. Before the ABM model, when he was examining the phenomena from a macroscopic/morphological perspective, he could not inspect the formalization (equations, heuristics) in a way that would give him new insights. After the ABM, conversely, he was generating new ideas about behaviors just based on the available information, and even finding bugs in his own reasoning. After this first tutorial session, he successfully completed his ABM model and report on recrystallization. But for the purposes of this paper, I will now skip one year ahead to Mikes return interview. In this interview, I was interested in evaluating out whether or not Mike retainedand could possibly still leveragehis insights and representational shifts, which I just described in this section.
One Year Later

One year following the modeling session described above, I conducted a set of interviews with the study participants as well as with a comparison group (see Blikstein, 2009 for an analysis of the comparison), and Mike was one of the self-selected individuals who volunteered to return for the interview. The goal of this follow-up interview was to evaluate the study participants recall of the agent-based heuristics. For the following analyses, I will introduce the idea of a representational element and of plausible interaction schemes. Representational elements are any graphical symbols used in paper modeling tasksfor example, a circle representing a particle or a rectangle illustrating a container. Plausible interaction schemes are physically-viable modes of interaction between these elements. At first, Mike was presented with the task of planning and thinking-aloud a computer model of solidification, initially on paper. This models required components were primarily a liquid metal inside a container. Solidification was understood as the process by which the liquid would transform into a solid due to a drop in temperature along the container walls. Mike begins his explanation by drawing an empty square (the container) and assuming it was homogeneous. This cursory assumption expresses a macroscopic, equation-based encoding. The macroscopic homogeneity and symmetry are typical starting points of differential models, due to the relative ease in building and solving for symmetrical/homogeneous situations. He then draws four arrows, each attached to one of the walls (see Figure 2, left), and states that the solidification comes from the walls. He adds that by knowing the temperature inside and outside the container, one could determine the solidification rate. After this initial description, he is left with three initial representational elements: (1) an empty rectangle, (2) the outside of the rectangle, (3) the inside of the rectangle. Seeing the phenomenon through these three representational elements, the combinations of plausible interaction schemes would be: (a) a flux (of energy, or matter) from the inside of the rectangle to the outside, or (b) vice versa, and (c) both regions would be at equilibrium. Then he attempted to build an equational relationship of the speed of the process: by the difference between temperature outside and temperature inside, [] you will be able to determine the cooling rate. Finally, his model ends when the system is at equilibrium, once the rectangle was filled with solid and the system stops. Mikes solidification model was correct, but bears little insights into the actual mechanism of solidification. Actually, his explanation followed a much-repeated method in engineering classrooms: (1) represent a phenomenon in terms of symmetrical, differentiable terms, (2) define initial and final states, and (3)
ISLS
391
ICLS2012
hypothesize a prototypical path linear, exponential, quadratic, etc. However, as I showed earlier, there is an earlier phase in experts sense-making process: examining micro-behaviors and make approximations to transform them into differentiable mathematical elements, which Mike skipped, and was apparent in the next step of the task. So far, I showed how the macroscopic approach determined the way Mike represented his solution, and how that representation left him with plausible interaction schemes that did not afford insight into mechanism. This is further confirmed by my next question, when I asked him about modeling the opposite phenomenon solid turning into liquid. His answer was that the system would simply go the other way, gesturing to show that there would be liquid coming from the walls until the whole container was filled with it. In other words, the process was also symmetric in terms of running it forwards and backwards. Thinking about the reverse phenomena did not elicit any new insights into the process. I will return to this issue after the next episode to compare what happened when Mike was embarking in agent-based sense-making. Episode 2. An interesting turn of affairs then took place, when I next prompted Mike to recall the NetLogo model he had built a whole year prior. Mikes second explanation, just seconds after the first, was significantly different from it. Once Mike recalled his agent-based model, he set out with a different representational stance: a rectangle divided into a grid with particles in it. Onto the grid, he added a layer of solid by shading one row of patches thus rendered immobile (Figure 2). His next step was to create the liquids, and assign them a fairly random velocity and direction. This time, solid and liquid were not represented as continuous invisible entities but as two categories of micro-entities belonging to the grid. Having created three types of representational elements (two types of particles and a wall), Mikes plausible interaction schemes became considerably different than in the previous case: (a) particles would move; (b) particles could collide; (c) particles could hit the walls of the container. These potential interaction schemes, in turn, gave rise to verbal articulation, for example, For the liquid atoms, when they hit each other they could [] have a term for sticking to each other [], so when it is stuck, it would just be there, and join the solid. As we will now discuss, microscopic views bear further affordances.
Figure 2. Mikes drawings of his solidification/melting models; equational (left) and ABM (right). Episode 3. This third and last episode will shed more light on the process of mechanism estimation and show how reasoning at the level of agents can give students powerful tools to understand yet unknown phenomena. Let us examine what happened when I asked Mike, again, about the opposite phenomenon: liquefying the solid metal. Table 1: Mikes Paper Model for the melting phenomenon.
[a] so for these ones on the edge, [b] and then give them a set I guess I would give them a energy that they need to break random kind of vibrational off of this, energy based on the temperature next to them,
[c] so as their temperature [d] As it goes eventually they all increases, they will have a go into the solution. greater chance of breaking off and heading off to the solution
Recall that Mikes equational explanation for melting had been a straightforward reversal of his equational explanation for solidification. From an agent-based perspective, however, his new algorithm was not a reversal of the old one. The mechanism for the liquids to stick to the solids (collision) was fundamentally different than the solids melting (vibrate and detach). This provides further evidence of the generativity of the second approach, which led Mike to generate a new mechanism by just rearranging the elements of the previous model. Note that then his estimation of mechanism was quite different from before. In the macroscopic, morphological paper model, his explanation had been based on a prototypical function that, given a set of parameters, simply returned the result. Albeit obviously useful for engineering practice, the process of inquiry
ISLS 392
ICLS2012
into this function did not elicit in Mike new insight into mechanism or any interaction schemes that would lead him to dive into what was under the hood of the phenomenon. This is not to say that this engineering heuristic is not powerful, but in materials science, where the fundamental mechanisms of interest take place at the microscopic level, structure matters more than morphology, dynamicity is more important than static processes, the atom outplays the aggregate. Those three dimensions are what enable students to identify meaningful similarity triggers so as to conduct coherent mechanism estimation.
Conclusion
As technologies and instrumentation evolve, so do representational systems. How we encode knowledge is at best a portrait of the tools available to scientists. At the same time, as Mikes comments illustrate, how we encode knowledge may also hold us captive in historical ways of seeing and thinking. This paper suggests misfit or discontinuity between intuitive understanding and formal representation in materials science as well as the potential of agent-based modeling as a means of ameliorating this misfit. Central to the first goal was the identification and analysis of broken chains of signification from a phenomenon, via simple language, and through to its algebraic reformulation. Namely, by analyzing a lecturers oral presentations, course textbooks, and student argumentation, we have witnessed extant discontinuity in discursive reference, as materials science texts shift from intuitive to formal registers. My analyses of the empirical data also revealed how that discontinuity happens, and its implications. Yet, I suggest that the informalformal dichotomy, which traditional curricula have handed us down as an axiom of engineering pedagogy, in fact need not map onto an intuitive unintuitive dichotomy: formal semiotic systems in science and engineering can and should be intuitive. To evaluate the potential specifically of agent-based modeling as a more intuitive representational, I compared for differences in how students make sense of the same phenomenon with or without the agent-based epistemic lenses. I examined an alternative curricular approach by which I eschewed the algebraic formalization entirely and, instead, explored the potential of formalizing phenomena in computer-based propositions that express agent-based observations/intuitions in the form of procedures. As we witnessed through our case study, agentbased language attempts to re-evoke rather than revoke the subjective experience. However, the project here is by no means an agent-based call to abolish algebra. Algebraic expression enables practitioners to calculate, predict, and generalizeactions that are all central to the practice of science and engineering. The concern is, rather, that students sustain a sense of groundingthat students know what formal expressions are about, what they mean with respect to the objects and processes they are studying, how they refer to the elements and mechanisms whose cumulative interactivity gives rise to the macro-phenomena. Indeed, our case study as well as the entire corpus of data point to the cognitive and epistemic advantages of agent-based modeling for this target content. Using ABM, students stand a better chance to make sense of and recall the phenomena they study. Moreover, they develop and later apply a new set of heuristics for reasoning about other phenomena that traditionally belong in different curricular units yet, it turns out, share in structure and function. That is, we have witnessed students linking productively among several phenomena that according to traditional taxonomy are ontologically disparate. As such, this study concurs with Jacobson and Wilensky (2006) to question the epistemic consequences of extant curricular taxonomy. Perhaps showing these affordances is the inferential power that agent-based perspectives bear as compared to macro, equation-based models. To examine this difference, I created two analytical constructs. The first, which I called similarity trigger, refer to aspects of a system that trigger the identification of similarity shape, color, mechanism, speed. Those triggers can be microscopic or macroscopic, structural or morphological. The second is mechanism estimation, which refers to the ability of infer mechanistic explanations with incomplete information. For example, when students identify similarity triggers at the level of mechanism, they can more easily use that perception of similarity to inform the discovery of a new phenomenon, since the atomistic behaviors are normally invariant across phenomena in materials science. Unequipped with an agentbased view students are liable to establish similarities that are misleading, taking into consideration shape, color, geometry, or other macroscopic or morphological aspects. From the agent-based perspective, a student can grasp the core mechanism of each phenomenon and connect them across content topics (see Markman & Gentner, 1993 on determining structural alignment as a prerequisite for successful judgments of similarity). Wolfram (2002) has suggested that the advent of agent-based modeling should revolutionize humanitys obsession with algebraic formalisms, which have constrained the scope of science to a small part of the vast spectrum of natural phenomena. Agent-based modeling is a viable candidate for empowering intuition; for working with, rather than against, humans natural vision and reasoning.
References
Blikstein, P. (2009). An Atom is Known by the Company it Keeps: Content, Representation and Pedagogy Within the Epistemic Revolution of the Complexity Sciences. PhD. dissertation, Northwestern University, Evanston, IL.
ISLS
393
ICLS2012
Blikstein, P., & Wilensky, U. (2009). An Atom is Known by the Company it Keeps: A Constructionist Learning Environment for Materials Science Using Agent-Based Modeling. International Journal of Computers for Mathematical Learning, 14(2), 81-119. Blikstein, P., & Wilensky, U. (2010). MaterialSim: A Constructionist Agent-Based Modeling Approach to Engineering Education. In M. J. Jacobson & P. Reimann (Eds.), Designs for Learning Environments of the Future: International Perspectives from the Learning Sciences (pp. 17-60). New York: Springer. Brock, W. H., & Price, M. H. (1980). Squared paper in the nineteenth century: Instrument of science and engineering, and symbol of reform in mathematical education. Educational Studies in Mathematics, 11(4), 365-381. Chi, M. T. H. (2005). Commonsense conceptions of emergent processes: Why some misconceptions are robust. The Journal of the Learning Sciences, 161-199. Clement, J. J. (1993). Using bridging analogies and anchoring intuitions to deal with students' preconceptions in physics. Journal of Research in Science Teaching, 30(10), 1241-1257. Collins, A. (1992). Toward a design science of education. In E. Scanlon & T. OShea (Eds.), New directions in educational technology. Berlin: Springer-Verlag, 1992. Confrey, J. (2005). The evolution of design studies as methodology. The Cambridge handbook of the learning sciences, 135-151. Danish, J. A., Peppler, K., & Phelps, D. (2011). BeeSign: Designing to Support Mediated Group Inquiry of Complex Science by Early Elementary Students. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA. diSessa, A. A. (2000). Changing minds: computers, learning, and literacy. Cambridge, MA: MIT Press. diSessa, A. A., Gillespie, N., & Esterly, J. (2004). Coherence vs. fragmentation in the development of the concept of force. Cognitive Science, 28(6), 843-900. Geertz, C. (1973). Interpretation of Cultures. New York, NY: Basic Books. Hammer, D., & Elby, A. (2003). Tapping epistemological resources for learning physics. The Journal of the Learning Sciences(12), 53-91. Hmelo-Silver, C. E., & Pfeffer, M. G. (2004). Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions. Cognitive Science, 28(1), 127-138. Ioannidou, A., Repenning, A., Lewis, C., Cherry, G., & Rader, C. (2003). Making Constructionism Work in the Classroom. International Journal of Computers for Mathematical Learning, 8(1), 63-108.. Jacobson, M. J. (2001). Problem Solving, Cognition, and Complex Systems: Differences Between Experts and Novices. Complexity, 6(3), 41-49. Jacobson, M. J., & Wilensky, U. (2006). Complex systems in education: Scientific and educational importance and implications for the learning sciences. Journal of the Learning Sciences, 15(1), 11-34. Kaput, J. J., Noss, R., & Hoyles, C. (2002). Developing new notations for a learnable mathematics in the computational era. In L. D. English (Ed.), Handbook of international research in mathematics education. Mahwah, NJ: Lawrence Erlabum Associates. Levy, S. T., & Wilensky, U. (2009). Crossing Levels and Representations: The Connected Chemistry (CC1) Curriculum. Journal of Science Education and Technology, 18(3), 224-242. Markman, A. B., & Gentner, D. (1993). Structural alignment during similarity comparisons. Cognitive Psychology, 25, 431-431. Nemirovsky, R. (2011). Episodic Feelings and Transfer of Learning. The Journal of the Learning Sciences, 20(2), 308-337. Oden, J. T., Belytschko, T., Fish, J., Hughes, T. J. R., Johnson, C., Keyes, D., Yip, S. (2006). Simulation-Based Engineering Science: Revolutionizing Engineering Science through Simulation: NSF. Porter, D. A., & Easterling, K. E. (1992). Phase transformations in metals and alloys (2nd ed.). London ; New York: Chapman & Hall. Russ, R. S., Scherr, R. E., Hammer, D., & Mikeska, J. (2008). Recognizing mechanistic reasoning in student scientific inquiry: A framework for discourse analysis developed from philosophy of science. Science Education, 92(3), 499-525. Sherin, B. L. (2001). A Comparison of Programming Languages and Algebraic Notation as Expressive Languages for Physics. International Journal of Computers for Mathematical Learning, 6(1), 1-61. Siegler, R. S., & Crowley, K. (1991). The microgenetic method: A direct means for studying cognitive development. American Psychologist, 46(6), 606-620. Wilensky, U., & Papert, S. (2006). Restructurations: Reformulations of knowledge disciplines through new representational forms. Unpublished Manuscript. Wilensky, U., & Resnick, M. (1995). New Thinking for New Sciences: Constructionist Approaches for Exploring Complexity. Paper presented at the Presented at the annual conference of the American Educational Research Association, San Francisco, CA. Wolfram, S. (2002). A new kind of science. Champaign, IL: Wolfram Media.
ISLS
394
ICLS2012
Metadiscourse to Foster Student Collective Responsibility for Deepening Inquiry

Jianwei Zhang, Jiyeon Lee, Jane Wilde State University of New York at Albany Email: jzhang1@albany.edu, jlee26@albany.edu, wildejk@gmail.com Abstract: This study examined two complementary designs of metadiscourse to foster collective responsibility for sustained, progressive inquiry in two comparable Grade 5/6 classrooms that investigated astronomy. Class As metadiscourse focused on reviewing student questions to formulate deepening goals. Class Bs focused on co-monitoring disciplinary key concepts in readings that suggested possible areas for deeper discourse. Analyses of classroom videos and online discourse suggest the positive impact of such metadiscourse on sustained knowledge building, along with specific conditions required.
Introduction
To prepare students for creative careers in a knowledge-based society, schools need to cultivate collaborative, inquiry-based practices by which knowledge-creating communities (e.g., R&D networks) expand our societys knowledge. Key to the productivity of these communities is a self-sustained, progressive trajectory of inquiry by which ideas are generated, refined, and further built upon by peers to formulate more advanced ideas and problems that continually inform further initiatives (Bereiter, 2002; Dunbar, 1995; Sawyer, 2007). Inquiry-based learning in schools needs to similarly foster such a self-sustained, progressive trajectory among students in order to develop their creative capacity (Hakkarainen & Sintonen, 2002). This sustained course of inquiry is not predefined by the teacher but co-developed by students through an interactional, unfolding process. Students need to take on high-level collective responsibility for continually advancing their communitys knowledge (Scardamalia, 2002). Our recent studies shed light on conditions and designs to foster collective cognitive responsibility among young students (Zhang et al., 2007, 2009). Among other conditions, metadiscourse collaborative, metacognitive conversations to review progress of understanding and formulate deeper goals and actions as a communitywas identified as a critical means for students to enact collective cognitive responsibility. The present study intends to test designs to facilitate metadiscourse, focusing on co-monitoring of progressive questions and key concepts in knowledge building discourse. Elaborating the vision of self-sustained inquiry, we (Zhang et al., 2007, 2009; Zhang, 2012) recently analyzed a set of design experiments conducted in elementary classrooms that implemented knowledge building pedagogy using Knowledge Forum, a collaborative online environment (Scardamalia & Bereiter, 2006). Instead of relying on teacher-specified steps, scripts, and resources of inquiry, students elaborated what they needed to know, set forth theories, searched for resources, and designed experiments to test and improve their ideas. Students in a whole class collaborated opportunistically to investigate a focal area (e.g. optics) and progressively identify important issues to be understood (e.g., how light travels, how people see colors, how lenses work). Small groups formed and reformed based on evolving needs. Knowledge Forum provided the online space in which student collective works were recorded, in views (workspaces) corresponding to the progressive goals identified. Ideas contributed became shared objects of continual online discourse accessible to all students. Essential to this emergent, opportunistic approach to inquiry, students engaged in metacognitive practices to reflect on what they had learned and what they needed to better understand on an ongoing basis. Leveraging such personal efforts, whole class metacognitive conversationsmetadiscoursewere carried out to review progress, highlight important insights, and identify deeper issues and weak areas as the focus of further inquiry. To highlight important knowledge goals and structure their work accordingly, students discussed what views should be created in Knowledge Forum and how the views should be structured and linked, with contributions (i.e. notes) in each view organized into theme-based clusters (Zhang, 2012; Zhang & Messina, 2010). Despite the importance of metadiscourse, this discourse pattern is rarely seen in classrooms even in inquiry-based settings (van Aalst, 2009). We as a field have little understanding about how metadiscourse works to sustain knowledge building and how it can be facilitated in classrooms. The present study aims to test and elaborate two complementary strategies to structure metadiscourse in knowledge building communities. (a) Metadiscourse focusing on collaborative, progressive questioning. This strategy supports student collaborative efforts to monitor what is known and what is missinggaps and problems of understandingso as to identify knowledge goals based on their deepening wonderment (Zhang et al., 2007). Although inquiry is entering more and more classrooms, in practice students often address pre-specified problems or tasks; rarely do they spontaneously generate curiosity-driven questions (Chinn & Malhotra, 2002; Rop, 2003). Part of the problem is the difficulty elementary and middle school students are found to have in formulating questions that guide inquiry in productive directions (Krajcik et al., 1998). This problem is, of course, heightened by concerns
ISLS
395
ICLS2012
with meeting curriculum standards and assessments. It is not essential that students initial questions target core concepts or issues, however, provided there is a process through which questions are progressively deepened as student understanding is advanced. Such a process has been formulated by Bereiter and Scardamalia (1993) as progressive problem solving and by Hakkarainen and Sintonen (2002) as an interrogative model of inquiry. This study engages students in metadiscourse to examine questions contributed by individual members so as to co-develop shared, promising research goals. (b) Metadiscourse focusing on co-monitoring disciplinary key concepts used in knowledge building discourse. This strategy engages students to monitor what is out there in the larger world in order to better reflect on their own work and discourse in a domain area. Students deepen their inquiry into the intellectual heart of a discipline (Gardner, 1999) by making use of authoritative sources that bring core concepts of the domain into their focus. A recent study provided preliminary evidence showing that productive use of key concepts from reading materials stimulated new lines of inquiry (Zhang et al., 2007). This study further tests designs to support student collaborative metacognitive efforts to identify key concepts from reference materials, review their own discourse to identify progress made related to the key concepts, and speculate over what needs to be further investigated. In short, this study aims to test the above two ways to structure metadiscourse in knowledge building communities. The purpose is not to find out which design is superior, but to examine and elaborate each in depth so we can refine and integrate them to foster student collective responsibility for sustained inquiry. Our research question asks: How did the metadiscourse using the two strategies take place in the classrooms to help students develop a progressive course of inquiry, as reflected in their knowledge building discourse?
Method
Classroom Contexts
This study was part of a larger initiative to develop pedagogical designs and visualization tools to foster a selfsustained, progressive trajectory of inquiry among elementary students using a design-based research methodology (Collins, Joseph, & Bielaczyc, 2004). It was conducted in two comparable Grade 5/6 classrooms at the Dr. Eric Jackman Institute of Child Study in Toronto, which has been implementing Knowledge Building pedagogy and Knowledge Forum (see Scardamalia & Bereiter, 2006 for details) for more than a decade. The two classeseach involving 22 studentsinvestigated astronomy over approximately eight weeks, taught by two teachers who had equivalent experience (2-3 years) with using Knowledge Forum to support inquiry. The astronomic study integrated face-to-face activities (e.g. whole class conversations, small-group experiments, reading) and online discourse in Knowledge Forum that extended and enriched each other. A pretest revealed no significant difference between the two classes in student prior knowledge (p > .10).
Focal Designs
Each classroom tested a focal design of metadiscourse: collaborative progressive questioning in Class A and comonitoring of key disciplinary concepts in knowledge building discourse in Class B. Each classroom could also use the strategy of the other, although not as explicitly. About once every two weeks, students in Class A held a whole class, metacognitive meeting (20-30 minutes) to reflect on progress and identify problems of understanding as the focus of their further inquiry. In preparation for each meeting, students were given time to read the online entries of their peers to identify knowledge advances and problems. Using the discourse scaffold I need to understand as a sentence starter, students contributed new and deeper questions in Knowledge Forum. Questions identified were then listed on chart paper and reviewed in the whole class meetings. In such meetings, the teacher worked with her students to reflect on progress made and identify promising questionsquestions that seemed important and might stimulate deep inquiry and understanding. These questions were highlighted to focus further inquiry and revisited in the subsequent metacognitive meetings to review progress. Classroom B also held whole class metacognitive meetings, about once every two weeks, to reflect on progress and identify deeper goals of inquiry. In preparation for each meeting, students were given time to read Knowledge Forum entries of their peers and use curriculum guidelines and other materials to identify key concepts that seemed important but yet to be used in their inquiry. Using a discourse scaffold, Key concept, as a sentence starter, students wrote notes to introduce new key concepts and propose issues to be explored. During their metacognitive meetings, students reviewed the key concepts identified, with their teacher recording the key concepts on a digital whiteboard. A concept map was thereby co-constructed to highlight the concepts and their connections. The concept map was used as a reflective planning tool to help students review progress, identify weak areas, and plan deeper inquiries. In the subsequent meetings, the concept map was revisited and updated in reflection of new concepts and deeper goals identified.
ISLS
396
ICLS2012
Data Analysis
Following design-based research (Collins et al., 2004), we integrated qualitative (e.g. videos, online discourse patterns) and quantitative analyses (e.g. counting domain words in discourse) to elaborate and evaluate the interventions. Primary data analysis was conducted after each metacognitive meeting and further discussed with the teachers to refine the classroom designs. To understand how students deepened inquiry, we conducted inquiry thread analysis (Zhang et al., 2007) of their online knowledge building discourse in relation to the metacognitive conversations recorded in videos. Unlike physically linked threads of online discourse (e.g. build-on trees), an inquiry thread represents a line of inquiry composed of a series of conceptually related discourse entrieswhich may involve multiple build-on treesthat address a shared principal problem over an extended period of time. Visualizing online discourse using inquiry threads helps to understand how diverse themes of inquiry emerge, evolve, and deepen in a community space and trace idea improvement and student participation within and across the threads. Two coders read and re-read all the Knowledge Forum notes and identified 19 major themes (e.g. planets, moons, atmosphere) addressed by the two communities. Using these themes as the tracers, the primary coder clustered the notes that addressed the same theme into one inquiry thread. The second coder then reviewed the notes in each thread and discussed with the primary coder about any disagreements. To gauge reliability, two coders independently coded 27 notes with an inter-rater consistency of 95%. To examine deepening moves in each inquiry thread, we further conducted content analysis (Chi, 1997) of questions in student notes. Using a previously tested coding scheme (Zhang et al., 2007), each question was coded as (a) factual, to be answered with factual information (e.g. where, when, how many), or (b) explanatory, to be satisfactorily answered with an explanation (why, how, what-if). Although both types of questions are valuable, explanatory questions represent deeper epistemic efforts to elaborate on reasons and mechanisms behind scientific phenomena. Additionally, we conducted lexical analysis of student online discourse to trace the emergent use of domain-specific terms as an indicator of the depth and scope of their inquiry (Sun et al., 2010). A word list containing 59 terms was developed including 39 from the Ontario Curriculum and 20 additional terms used in student online discourse. A software tool, Simple Concordance Program (SCP), was used to trace the occurrences of these termsincluding their grammatical variationsin the online discourse. Following a narrative approach to video analysis (Derry et al., 2010), we further analyzed the metacognitive meetings recorded in videos to understand the complex processes by which students codeveloped and deepened inquiry goals through metadiscourse. We first browsed the videos and transcriptions to develop an overall sense of the knowledge building processes, aided by the inquiry threads identified from the online discourse. We then identified digestible chunks in the videosmajor episodes in which students proposed questions or key concepts, reviewed connections and progress, and formulated new and deeper goals. These chunks were contextualized and linked based on thematic and conceptual connections (e.g. questions and concepts deepening the same theme) to develop a storyline for each classroom, showing how the two focal designs worked out to foster progressive inquiry.
Results
Inquiry Thread Analysis
Students wrote a total of 121 discussion notes (3966 words) in Class A and 190 (5370 words) in Class B (excluding individual summary notes). Inquiry thread analysis identified a total of 19 themes addressed by the two classes in the online discourse. Discourse entries focusing on each theme constituted an inquiry thread that extended from the first to the last note contributed (see Figure 1). The inquiry thread map of each class visualizes the emergence of the various themes over time and contributions and contributors involved. The themes of inquiry were not pre-specified by the teachers, but emerged through student interactive discourse that brought important issues to their focus. Each class investigated multiple themes in parallel through extended and interconnected discourse input, with a large number of notes addressing multiple themes simultaneously, serving as interconnectors between different lines of work. Several core themes in astronomy (e.g. planets, moons, gravity) engaged the most intensive contributions in both classes. Class B covered more themes (n=18) then Class A (n=16). Through student use of key concepts to monitor and focus their discourse, issues related to nebulas, constellations, and orbit were brought to the focus of Class B. These themes were absent from the discourse in Class A. Consistently, Class B conducted more extended discourse on themes that are more distant from student personal experience, such as the origin of the universe, black holes, stars, and space technology. Class A carried out more intensive inquiry in more personally relevant areas, such as the Earth, planets, moons, gravity, and atmosphere. Interestingly, Class B did not explicitly discuss issues about the Earth at all. To further examine deepening moves of discourse in the inquiry threads, we analyzed student questions as well as domain-specific terms (e.g. planet, moon, orbit) used in the discourse (Table 1). Consistent with the focal designs of the two classrooms, Class A had a much higher percentage of notes raising questions, including fact-seeking (e.g. Are some planets smaller than some moons?) and explanation-seeking questions (e.g. Why are
ISLS 397
ICLS2012
Jupiters moons so big? How did planets get their moons?). Among the 59 domain-specific terms identified, Class B used more unique terms, and at a higher frequency, in the online discussion. Overall, the online discourse in Class A was more driven by students interest, curiosity, ideas, and experiences that were presented using more personal and informal language, with more deepening moves (e.g. problematizing); while that in Class B addressed more astronomic topics using sophisticated scientific language, including topics relatively far away from student personal experience.
Figure 1. Inquiry threads of Class A (left) and B (right). The numbers following the title of each theme show the number of notes and authors involved, respectively. Vertical lines linking notes in different threads denote notes shared by the threads addressing multiple related themes. Table 1: Questions and domain-specific terms incorporated in the online discourse. Class A B % of notes raising questions 41.32% 23.16% % of notes raising fact-seeking questions 14.88% 8.42% % of notes raising explanatory questions 26.45% 14.74% Unique domain terms 34 out of 59 45 out of 59 Total occurrences of domain terms 425 603
Progressive questioning in Class A helped students to investigate deeper issues beyond their current understanding, leading to progressively deepening goals and focuses. Questions that were more interesting to students and of a greater intellectual value in the domain tended to engage more active and extended conversations, resulting in more sophisticated understanding that further illuminated deeper issues to be understood. For example, in the inquiry thread about moons (A2 in Figure 1), students first contributed information to address basic, factual questions such as what the moon is made of (rock, not cheese) and which planets have moons. Deeper questions were then asked: Why dont Mercury and Venus have moons? Why are Jupiters moons so big? How did a planet get its moons? Such deeper, explanatory questions stimulated continual inquiries (e.g. reading, modeling, discussion) leading to advances of understanding, as shown in the following note: I think that Mercury and Venus (the two closest planets to the Sun) don't have moons because whenever they get moons the Sun's gravitational pull pulls the moons away. Jupiter has so many moons because it has such a strong pull, that asteroids and comets get pulled in. The advances of understanding further helped students to problematize their knowledge at a deeper level. Even simple facts, such as how many moons a planet has, were examined for deeper inquiry: Determined Moons by nw [2008, Oct 29] I need to understand: how can you tell how many moons a planet has? My Theory I think that maybe astronomers use telescopes to see, but how can they see each side of all the different planets? I need to understand: How can people determine that? What facts can lead up to it?...
ISLS
398
ICLS2012
Through identifying and monitoring disciplinary key concepts, students in Class B engaged in active online knowledge building discourse, with the key concepts introduced serving to inform new and deeper focuses of inquiry. More extended online discourse were conducted about stars (B5), life of stars (B8), nebulae (B7), and the origin of the universe (B10)all these threads were initiated through introducing key concepts from reference materials. For example, the inquiry thread on nebulae began with a student note: Key concept is NEBULASNebulas are the [remains] of stars they are created when stars explode. Nebulas also create new stars. The discourse was deepened when her peers identified deeper things to be further understood: do nebulas stay where they are or do they just disappear? why there[re] more stars than nebulas, because nebulas are dead starsthere[re] so many stars in the sky that there should be a nebula for each star that dies. Theories were further developed to address these issues, such as the following: This is why!!!!!!!!!!!!!!!!!!!!!!! by dc [2008, Oct 23] Stars live for a few billion years but nebulas turn into stars in about a million so over time. [T]here will only more stars. But even though you would expect to see some nebulas but they are really hard to see, so you never see them in the city.
Metadiscourse Focusing on Collaborative, Progressive Questioning

We analyzed the metacognitive meetings recorded in videos in relation to the online inquiry threads. Students in Class A began the space study in late September by contributing initial ideas and information about the Sun and the planets and sharing questions in Knowledge Forum. The first metacognitive meeting was then held on October 3. Students gathered around a piece of chart paper that recorded a rich set of initial questions. The questions were read aloud, including: How long ago did Earth begin? How was the Sun formed? What are the dwarf planets, and why are they not normal planets? What is beyond Pluto? Is there anything beyond the Milky Way? Is there other life in space? Interactive discussions occurred focusing on some of the questions (e.g. planets and moons, galaxies). The remainder of the meeting was then focused on reviewing the questions and identifying good, promising ones as the communitys focus. The teacher encouraged student reflection by asking: Which questionthat seems like really meaty? So, thats a question that shows a lot of promise. We could probably do lots with that...A question we could generate lots of discussions. Lots of people can have input into it. Contrasting with the lively discussions to generate specific questions and ideas in the first part of the meeting, reflecting on the questions to identify promising ones was challenging for the students. To further scaffold such reflection, the teacher then worked with the students to examine a few examples: Teacher A: How big is the Sun? Would that be a question that would generate lots of different conversation and people could input lots of different things? Student 1: No. Teacher A: No, and why not? (Students murmur. Student 2s voice gains their attention) Student 2: Its pretty Its like a simple question How does it heat people?... that would be a bigger, deeper questionLike, How does the Sun heat the Earth? or something. Teacher A: Right. So There are lots of questions about the Sun, that would be deeper, richer kinds of questions, that would generate lots of discussion, and those are the questions were really looking for. In the analysis of the example questions, the teacher provided metacognitive prompts such as: Why do you think thats a solid, great question? As you find out more and more about , where can you go from there? Will this question take [us] in a productive path? Although the first meeting did not end up with any explicit conclusion about which questions should be set as their common focus, questions related to several themes (e.g. the difference between the moons and planets, different types of galaxies) were highlighted to invite student input. Such themes became the major focuses of the online discourse in early October, with planets as the most intensive and long-lasting theme (see thread A1 in Figure 1). Another metacognitive meeting was held two weeks later, on October 14. Students worked together to review progress made to address questions identified earlier, generate deeper and new questions, and identify promising and productive ones as their focus. First, a student shared her finding about what is beyond Plutoa question identified in the first meeting: There are more galaxies, some of which scientists have not observed yet, so they just call them Dark Matter, or negative space. Questions were raised by her peers pertaining to what Dark Matter is and how it is identified. The teacher then reminded the students to record such questions in Knowledge Forum if these were promising and could help further their understanding. However, the topic of Dark Matter did not attract any contribution in the subsequent online discourse, probably because of the difficulty to advance understanding in this area. As the meeting proceeded, atmosphere was identified as a new theme based on information contributed online earlier: What is atmosphere made of? Are there atmospheres for other planets, other stars? Students further commented on the importance of this topic because atmosphere
ISLS
399
ICLS2012
supports life. In contrast to the dismissal of Dark Matter as a possible discourse focus, rich and extended discourse was conducted focusing on atmosphere following this meeting (see thread A4 in Figure 1). Thus, in monitoring the promisingness (Bereiter & Scardamalia, 1993) of questions, the students were able to consider the importance of such questions as well as the challenges and difficulties involved. Another metacognitive meeting was recorded on October 24. While understanding was deepened to address existing questions, students reformulated the problems in deeper and more productive ways. Deepening the question of what the atmosphere is made of, students asked how it is held in place and affects life on the Earth. Deepening questions about which planets have moons, students asked why only some planets have moons, how they got the moons and keep the moons in orbit. In the end of this meeting, three areas were explicitly identified calling for deeper work: moons, sun (in relation to planets), and atmosphere. Issues related to moons became a new major theme in the subsequent online discourse (see A2 in Figure 1) while discourse about atmosphere (A4) and planets (A1) continued to attract deeper input in relation to gravity (A9). Overall, through generating and deepening questions and collaboratively reflecting on the value and importance of the questions for knowledge advancement, students in Class A were able to focus their efforts on core issues in the domain that are intellectually engaging and rich (e.g. planets, moons, atmosphere, gravity). However, several topics expected by the curriculum (e.g. constellations) were not explicitly addressed because of the lack of student questions in these directions.
Metadiscourse Focusing on Co-Monitoring of Key Concepts

Students in Class B also began their inquiry in late September by contributing initial thoughts and information. In the online discourse, they introduced key concepts to their community and suggested related issues to be understood. Whole class metacognitive meetings were held to review the key concepts, reflect on progress, and identify focal areas for further inquiry. These key concepts were recorded on a chart paper by the teacher to ground and focus the first metacognitive meeting on October 10, during which students reviewed the existing key concepts and related questions and ideas and identified major themes (e.g. galaxies, planets and life, orbit). As the class reviewed the concepts and ideas related to each theme, the teacher recorded the themes and related concepts on a digital whiteboard. For example, focusing on the theme of galaxies, concepts were identified including stars, Solar System, planets, and size (big), etc., with connections between related concepts highlighted (e.g. between the Solar System and planets). A concept map was thereby created and further expanded as students identified key concepts related to other themes. Figure 2 shows the full concept map cocreated through this meeting. This map was used by students to discuss what areas had been explored and what needed to be better understood. It was later uploaded to their Knowledge Forum view as a background picture, which served to focus students collaborative discourse on key themes and additionally provide a source of domain-specific vocabulary to be used in their discourse.
Figure 2. Concept map generated through the first metacognitive meeting in Class B. Circled terms represent major themes identified. Numbers show the sequence by which the terms and links were added. Another metacognitive meeting was held on October 21. With the concept map (Figure 2) displayed on the digital whiteboard, students reviewed progress made in areas indicated by the existing concepts and identified new key concepts that had not been explored much yet. In reflection of students deepened understanding, a new concept, satellites, was added to the map and connected to orbit as well as moons. Issues related to orbit (B6 in Figure 1) and satellites (B14) became a focus in the online discourse and were addressed at a much deeper level than in Class A. On October 30, students held their third metacognitive meeting. Several new concepts were added to the map to elaborate the theme of gravity. A new theme was added about space exploration. To facilitate
ISLS 400
ICLS2012
reflective conversations to review progress and plan further exploration, the teacher asked: What of the things up there are the things that you know a lot about and what do we want to know more of? Atmosphere was identified as a topic that had been overlooked. It was added to the map and linked to gravity. Altogether, eight topics were identified as the focus of their further inquiry, including atmosphere, Asteroid Belt, galaxies, black holes, jovian/terrestrial, the moon and tides, white dwarves, spaceships. Students then formed into temporary small groups to study these topics and contribute findings in Knowledge Forum, as reflected in the discourse in threads B4, B19, B11, B12, B1, B2, B5, and B14. Overall, through co-monitoring key concepts used in readings and in their own knowledge building discourse, students monitored and shaped the conceptual landscape of their collective knowledge, identified weak areas, and focused their personal and collaborative efforts accordingly. Key concepts introduced were not simply something for the students to remember, but suggested main directions of search for new information and deeper thoughts. Responding to the key concepts introduced (e.g. constellations), students developed personal thoughts and asked questions, such as cant you just see any cluster of stars and say that is a constellation? Such deepening moves are critical for students to make productive use of key concepts to deepen inquiry. However, in several inquiry threads, such deepening moves were scarce and emerged late, making the online discourse factual and dry.
Discussion
Metadiscourse provides a social, epistemic structure for a knowledge building community to execute high-level collective responsibility for sustained knowledge advancement. The results shed light on two complementary strategies to structure metadiscourse, each with partial success and facing specific challenges. Metadiscourse focusing on progressive questions represents an inside out process to deepen inquiry natively among students, with their personal wonderment and curiosity serving as the driving force. It was implemented in Class A productively, resulting in fruitful questions and ideas that deepened one another over time in the interactive, lively knowledge building discourse. With student questions setting directions for their inquiry, the online discourse engaged student thinking more deeply from the onset in each inquiry thread. The metacognitive meetings further provided the opportunity for students to collaboratively examine their personal questions and envision promising directions. Questions addressing core disciplinary issues were highlighted to the attention of all members, and initial questions were later reformulated and deepened, increasing their potential for knowledge advancement. As a result, the inquiry driven by student questions in Class A addressed almost all the expected curriculum themes, as reflected in their inquiry thread map, even though some of the specific terms noted in the curriculum guidelines were missed from the students discourse. The analyses suggest a few conditions of productive metadiscourse focusing on progressive questioning. First, students need to work as a community to co-reflect on the individually identified questions to examine their importance and potential for knowledge advancement. Second, the community needs to be allowed and supported to go through a continual, progressive process by which questions are revisited and refined drawing upon new understanding achieved. Working with the progressive process is a challenge for both the teacher and students. For example, one of the questions initiating the inquiry thread on the Earth presented a wild wonderment: What would happen if scientists turned the Earth inside out? This question attracted many responses but with little knowledge advancement for a long time; although a few educationally productive issues did eventually surface, such as the structure of the Earth and how gravity works in the center of the Earth. Helpful moves were made when students connected their wild discourse to concepts from readings, with fruitful disciplinary topics coming to their attention. Metadiscourse focusing on co-monitoring of key concepts used by a community in reference to those in authoritative sources represents an outside in process for students to monitor what is out there in the larger world and selectively adopt ideas from the field to help focus and grow their own inquiry. Key concepts in a field serve as conceptual landmarks using which students can understand the landscape of the discipline so to better navigate in it. Bringing such key concepts into their own discourse to help review their ideas and initiatives helps to focus, connect, and deepen their discourse in productive directions. Class B conducted such metadiscourse at multiple points of the inquiry with positive impacts. Important topics and issues, including those that were relatively far away from student experience, were brought to student focus. Concepts introduced were used to develop ideas and identify gaps and problems, leading to deeper inquiry. The concept maps created through the metadiscourse served to make the communitys knowledge visible, so important advances were shared and weak areas identified and further addressed. Rich cross-theme connections were built surrounding core concepts in the domain, such as orbit, which was not explicitly discussed in Class A. Several conditions seem essential to productive metadiscourse aided by disciplinary key concepts. First, students need to engage in personal and collaborative reflection on the potential value of concepts, so key concepts and ideas can be differentiated from simple facts and information (e.g. the size of the sun). Second, a constructive epistemological stance needs to be fostered in the community so students do not see key concepts introduced as the answer but resources for deeper questioning and thinking. This stance is critical for the new
ISLS
401
ICLS2012
concepts to take root in the community to grow into productive lines of inquiry. As the analyses suggest, when such constructive moves are missing, student discourse tends to be dry and focus on digesting the concepts and adding factual information. In conclusion, each of the two designs of metadiscourse supports students to enact collective responsibility to sustain and deepen inquiry under favorable conditions. An integration of both will likely lead to more productive and balanced metadiscourse to foster deep and sustained inquiry. Our future research will test integrated designs of metadiscourse supported by the development of new visualization tools (e.g. inquiry threads) that make collective progress and problems visible to support ongoing reflection and co-planning.
References
Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah, NJ: Lawrence Erlbaum Associates. Bereiter, C., & Scardamalia, M. (1993). Surpassing ourselves: An inquiry into the nature and implications of expertise. Chicago: Open Court. Chi, M. T. H. (1997). Quantifying qualitative analysis of verbal data: A practical guide. Journal of the Learning Sciences, 6, 271-315. Chinn, C., & Malhotra, B. A. (2002). Epistemologically authentic inquiry in schools: A theoretical framework for evaluating inquiry tasks. Science Education, 86, 175-218. Collins, A., Joseph, D. & Bielaczyc, K. (2004) Design research: Theoretical and methodological issues. Journal of the Learning Sciences, 13(1), 1542. Derry, S. J., Pea, R. D., Barron, B., Engle, R.A., Erickson, F. Goldman, R. et al. (2010). Conducting video research in the learning sciences: Guidance on selection, analysis, technology, and ethics. Journal of the Learning Sciences, 19, 353. Dunbar, K. (1995). How scientists really reason: scientific reasoning in real-world laboratories, in: R. J. Sternberg and J. E. Davidson (Eds.), The nature of insight (pp. 365-395). Cambridge, MA: MIT Press Gardner, H. (1999). The disciplined mind. New York: Simon & Schuster. Hakkarainen, K., & Sintonen, M. (2002). Interrogative model of inquiry and computer-supported collaborative learning. Science & Education, 11(1), 2543. Krajcik, J., Blumenfeld, P. C., Marx, R. W., Bass, K. M., & Fredricks, J. (1998). Inquiry in project-based science classrooms. Journal of the Learning Sciences, 7(3/4), 313-350. Rop, C. J. (2003). Spontaneous inquiry questions in high school chemistry classrooms. International Journal of Science Education, 25(1) 1333. Sawyer, R. K. (2007). Group genius: The creative power of collaboration. New York: Basic Books. Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith (Ed.), Liberal Education in a Knowledge Society (pp. 6798). Chicago, IL: Open Court. Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 97-115). New York: Cambridge University Press. Sun, Y., Zhang, J., & Scardamalia, M. (2010). Knowledge building and vocabulary growth over two years, Grades 3 and 4. Instructional Science, 38(2), 247-271. van Aalst, J. (2009) Distinguishing knowledge-sharing, knowledge-construction, and knowledge-creation discourses. International Journal of Computer-Supported Collaborative Learning, 4 (3), 259-287. Zhang, J., Scardamalia, M., Lamon, M., Messina, R., & Reeve, R. (2007). Socio-cognitive dynamics of knowledge building in the work of nine- and ten-year-olds. Educational Technology Research and Development, 55(2), 117145. Zhang, J., Scardamalia, M., Reeve, R., & Messina, R. (2009). Designs for collective cognitive responsibility in knowledge building communities. Journal of the Learning Sciences, 18(1), 744. Zhang, J., & Messina, R. (2010). Collaborative productivity as self-sustaining processes in a Grade 4 knowledge building community. In K. Gomez, J. Radinsky, & L. Lyons (Eds.), Proceedings of the 9th International Conference of the Learning Sciences (pp. 49-56). Chicago, IL: International Society of the Learning Sciences. Zhang, J. (2012). Designing adaptive collaboration structures for advancing the communitys knowledge. In: D. Y. Dai (Ed.), Design research on learning and thinking in educational settings: Enhancing intellectual growth and functioning (pp. 201-224). Philadelphia, PA: Taylor & Francis.
Acknowledgements
The work presented here was supported by the U.S. National Science Foundation under grant 1122573 and the Faculty Research Award Program of the University at Albany. The authors would like to thank Ben Peebles, Rheanne Stevens and their students at the Dr. Eric Jackman Institute of Child Study for their insights and work enabling this research and Kevin Goodman for his assistance with data analysis.
ISLS
402
ICLS2012
Consequential Feedback as a Means of Supporting Student Engagement and Understanding

Melissa Sommerfeld Gresalfi, Jacqueline Barnes Indiana University, 1900 E. 10th St., Eigenmann 543, Bloomington, IN 47406 mgresalf@indiana.edu, jacqbarn@indiana.edu Abstract: This paper presents two cycles of a design-based research project that considers whether and how consequential feedback supports students engagement with mathematics. Consequential feedback describes a form of feedback that is embedded in the context with which a student is engaging, and allows the student to see how their solution to a problem plays out in that context. Comparing two classes across two years, we considered how the timing of feedback impacted student engagement. Findings suggest that providing consequential feedback in the form of a narrative outcome supported students to offer more mathematical justification, consequential justification, and to engage critically with the mathematical content within their written recommendations. The impact of consequential feedback appears to be potentially heightened by total time in class discussion, presence of more frequent discussions, and increased immersion in the narrative.
Introduction and Framing

In this paper, we present findings from the first two rounds of a design-based experiment that sought to consider how particular forms of feedback might effectively support productive forms of mathematical engagement. The conjecture underlying this research is that a particular form of feedback, called consequential feedback, can serve to both help students to interrogate their own mathematical work and provide information for students to use to improve their performance. Consequential feedback describes a form of feedback that is embedded in the context (mathematical or narrative) with which a student is engaging, and allows the student to see how their solution to a problem plays out in the context. In this way, consequential feedback provides students with information about their reasoning. This paper considers how the timing of consequential feedback impacts the nature of students engagement with content. In general, the literature on feedback has been inconsistent. Feedback can both support (Baron, 1998; Mory, 2004; Sweller, Van Merrinboer, & Paas, 1998) and thwart (Bangert-Drowns, Kulik, Kulik, & Morgan, 1991; Kluger & DeNisi, 1996; Schimmel 1983) learning. Even when it successfully supports learning, feedback is used in different ways and for different purposes. Much of the research on feedback focuses on general characteristics of feedback, such as timing, specificity, length, and complexity. Feedback is treated as a factor in students work, and not as an integral part of their activity. Specifically, the relationship among the content on which students are working, the nature of the activity, and the role of feedback is typically not discussed. Indeed, in her recent review of formative feedback, Shute (2008) notes, Despite the plethora of research on the topic, the specific mechanisms relating feedback to learning are still mostly murky, with very few (if any) general conclusions (p. 156, emphasis added). Additionally, research on feedback typically focuses on outcomes such as performance and learning (measured by pre-post change). Although to be sure, there is reason to care about the content that students learn, this is a distal measure that doesnt yield insight into how feedback is functioning. Understanding how feedback functions therefore requires a closer examination of the nature of feedback as it exists as part of larger activity. The work presented in this paper builds directly on findings that suggest that what students learn cannot be separated from how they learn it (Barab & Plucker, 2002; Boaler, 2000; Brown, Collins, & Duguid, 1989; Cobb, Stephan, McClain, & Gravemeijer, 2001; Greeno, 1991) and that knowing is an interaction among components of complex systems. This shift in the conceptualization of learning highlights the importance of attending to the practices associated with engaging content as part of the learning process, rather than focusing on learning outcomes as a means of determining effectiveness (Greeno & Gresalfi, 2008). Building on this framework, the research presented in this paper speaks directly to this gap in our understanding of the design of feedback by considering how a particular form of feedback impacts the nature of the activity with which students engage. This work also speaks to the need for different methodological approaches for studying feedback by leveraging a design-based research methodology that looks iteratively at the nature of activity as it is impacted by design. In what follows, we define and justify the intervention studied in this paper, consequential feedback. We then share findings from two rounds of implementation of design-based research, and discuss the implications of those findings for future research.
ISLS
403
ICLS2012
Consequential Feedback
Consequential feedback is a form of feedback that is embedded in the context (mathematical or narrative) with which a student is engaging; the way the students solution to the problem plays out gives the student information about their reasoning (Gresalfi, 2011). Consequential feedback is elaborated and task level in that it occurs in the context of the problem that is being completed and focuses on process (as opposed to outcome). However, rather than informing students about the mathematical accuracy of their work, consequential feedback offers agency for students to consider whether their mathematical reasoning resulted in the outcome they had envisioned (positioning students as being responsible determining what might have gone wrong). In this way, consequential feedback is more facilitative than directive (Black & William, 1998). Any mathematical task could afford consequential engagement as long as the feedback takes place in terms of the context (rather than in terms of the calculations). For example, if the task is to figure out how long it would take for two trains to pass each other given particular starting points and rates of speed, consequential feedback would allow students to see the actual scenario based on their mathematical calculations, through some sort of simulation or engagement in a virtual world (Nathan, Kintsch, & Young, 1992). Creating opportunities for students to engage consequentially often involves embedding a mathematics problem in a real world situation. The context of these problems serves a purpose beyond simply providing relevance for disciplinary work. Instead, when well designed, contexts can push back on students understanding by forcing the learner to consider the usefulness, impact, or significance of particular tools on outcomes (Cobb, McClain, & Gravemeijer, 2003). For engagement to be truly consequential, it must therefore involve engaging procedurally with rules and formalisms, and conceptually by considering why particular solution paths are sensible (Gresalfi & Barab, 2011; Gresalfi, Barab, Siyahhan, & Christensen, 2009). Thus, consequential feedback has the potential to support opportunistic use and meaningful application (not procedural replication) of content (Carr & Claxton, 2002). This way of engaging mathematics supports meaningful disciplinary understanding, as actions cannot have power unless they can be legitimately shown (through proof or defense) to impact particular situations.
Methods
The data reported for this paper come from two iterations of a design-based research project, which sought to better understand how consequential feedback could be integrated into a project-based mathematics unit situated in a virtual educational game called Quest Atlantis. As a design-based research project, our goal was to design an intervention (consequential feedback), which, we conjectured, would support particular forms of mathematical engagement, which we call consequential and critical. The two rounds of data collection presented here, however, are best conceptualized as a quasi-experiment, in that ultimately we compared two iterations of the same curriculum across very similar groups of students. However, these iterations were not randomly assigned; version 1 of the curriculum was used in year 1, and version 2 of the curriculum was used in year 2. These versions were not used in both years because the second version of the curriculum was developed in response to findings from year 1.
Intervention
This study examined consequential feedback through a mission within the online immersive game Quest Atlantis (www.questatlantis.org). The narrative of the mission is based on a well-known project-based mathematics activity from the Adventures of Jasper Woodbury, called Adventure at Boones Meadow (CTGV, 1997). When students enter Quest Atlantis, they are given an online persona (an avatar) who navigates the virtual world, explores the environment, interacts with non-player-characters, and makes choices to resolve complex real world dilemmas. Thus, the world itself is a part of the information they can explore, and that world can change in response to particular decisions students make. The mission used here informs students that an endangered eagle has been shot in Boones Meadow, which cannot be reached by car and takes 10 hours to hike by foot. Luckily, they meet a a non-player character who has an ultralight flying machine. Students must decide which route to take, the length and time of the journey, how much gasoline will be required (and where to stop to get it), who will pilot the plane, and whether any additional cargo is necessary (or possible) given the weight limit of the small aircraft. Consequential feedback is built into this narrative by showing students the consequence or outcome of their choices--in particular, whether the flight was completed successfully (without crashing) and whether the eagle survived. The outcome is shown through an image (of the crashed ultralight) or through dialogue with a non-player character. For example, students who make a mathematical error (such as not calculating the amount of fuel needed accurately) see their ultralight crashing, and then learn from the veterinarian that the eagle has died because no one was able to reach her in time. Students who offer a possible but non-optimal solution (i.e. a solution that will take too long) successfully complete their rescue mission to discover that the birds wing has been amputated. And finally, students who select both an accurate and efficient route are able to save the eagle. Examples of the kinds of feedback that students experience can be seen in Figure 1.
ISLS 404
ICLS2012
Figure 1. Examples of feedback students receive if they proposed a correct solution (top) or incorrect solutions (bottom).
Study Design
Among other questions, the project considered how the timing of consequential feedback impacted students engagement with content. For this paper, we present data two rounds of implementation across two years. Sixty-five students participated in this study. Year one included a 6th grade class of seventeen students, taught by Ms. Bell, and a 7th grade math class of sixteen students, taught by Ms. Kent. Year two included Ms. Bells eighteen 6th grade students and Ms. Kents fourteen 7th grade students. Both teachers have over five years of teaching experience and have both used Quest Atlantis for several years. Both teachers taught in schools in a suburban part of the Midwestern United States. In year 1, students were given consequential feedback on their strategy after they had invested time in their mathematical calculations (i.e. after they turned in their solution, but before the final reflection). In year 2, students were given consequential feedback based on their initial guesses, before they had engaged deeply with the mathematics. In every other way the curricular experiences were identical: they had the same immersive experience, were given the same information, saw the same possible consequential feedback, and made a recommendation in response to the same prompts. The only curricular difference was whether students received consequential feedback based on their initial solution before making a formal recommendation, or after. It is important to note that these implementations were not controlled, in that the same teacher was using the same curriculum in year 1 and year 2. Thus there could be some kind of effect of experience on the findings from year 2. We suspect this is not the case as both teachers needed a refresher about what the activities were about and the mathematics that was covered in the activities. Additionally, the students differed between year 1 and year 2, and although there were no systematic differences, it is certainly the case that individual students can dramatically change the dynamic of a classroom.
Data Sources
Data for this paper draw from several resources: videotapes of whole-class discussion, videotapes of small group discussion, and submitted work that outlined the final recommendation. Videotapes were collected from each day of the implementation (approximately five hours of classroom time from each classroom in both years, a total of approximately 20 hours of videotape). During whole-class time, cameras were set up to capture the entire class discussion while focusing on the written work presented in the public space. During small group time, several cameras were placed around the room in order to capture a range of discussions. Groups were chosen randomly based primarily on where students were located in the classroom. Students submitted work was in the format of a written recommendation in response to targeted questions about how to best serve the bird (this was submitted before getting feedback in Year one, and after getting feedback in Year two. The recommendation was identical across both years, and included the following information: Your task is to give Larry a plan for how to get the eagle back as quickly and safely as possible (in case something like this happens again). Larry will need to know: What's the quickest way to transport an eagle from Boone's Meadow to Cumberland City? (include the route, what you need to bring, and who should fly)
ISLS
405
ICLS2012
How long will it take? TIP: Decide what the outcome and travel time would be for multiple plans, and then decide which is best. There isn't one right answer, but some plans may be better than others. It's up to you to decide which plan is best.
Analysis
The first round of analysis focused on the submitted recommendations as primary indicators of students reasoning about the scenario and their understanding of the mathematical content. The recommendations were coded using both a priori and emergent codes. A priori codes included indicators of the accuracy of students submitted work (instances where mathematics that was unreasonable were noted), and their use of mathematical or consequential justification. Additionally, because a key interest in the study was whether students were engaging critically with content, codes that had been developed to capture these forms of engagement were used (Gresalfi & Barab, 2011; Gresalfi, Barab, Siyahhan, & Christensen, 2009). Additionally, researchers perceived emergent patterns within the students submissions, and sought to quantify those differences through the other codes included below in Table 1. Two researchers discussed the codes and further developed more explicit definitions and coding rules. The two researchers separately coded a sample of the Quests (approximately 23% of the total) until 88% of their codes were in common. After confirming 88% reliability, the two researchers independently coded the remaining Quests. Codes were applied in terms of instance. For example, each unreasonable mathematical claim made, or each distinct justification using consequences, was coded as an instance. The second round of analysis included looking in depth at both whole-class and small-group discussion. These videotapes were examined primarily to document the types of activities that occurred in the classrooms, measuring the duration of whole-class and small group discussions. In addition, the nature of the conversations during whole-class discussions were coded in terms of the amount of time where either mathematical content or narrative content was discussed. Table 1. Coding scheme Codes Mathematically unreasonable Narrative immersion Uncertainty Certainty Critical engagement Mathematical justification Consequential justification Description The numbers mentioned in the quest show a significant misunderstanding of the mathematics involved. Something that is mathematically inaccurate but reasonable is not included in this category. Expression that student feels like a character in the game, or displays immersion in the narrative of the game. This is aimed to capture when students are imagining what might or should happen, rather than being sure of the consequences of their plan. There is a sense of guessing, a lack of certainty. Noticeable expression of certainty in the consequences of their plan, such as I KNOW this will work (rather than I THINK). This shouldnt be used to show lack of tentative language. Shows acknowledgement of other options. The student expresses that they had a choice, and chose one thing over another. Using mathematics to justify their decision, for instance, justifying route (miles, distance), justifying cargo (weight), justifying pilot (weight), justifying time (time), describing one static mathematical claim rather than a causal statement or connection. Using consequences of a plan to justify it (in addition to OR opposed to mathematics), such as Justifying route, justifying cargo, justifying pilot, justifying time WITH an event, NOT A NUMBER, using causal statements.
Results
We found that offering consequential feedback in the course of problem solving as in year two led to significantly higher levels of consequential justification (i.e. considering the relationship between mathematics and the story), with students including justification that related to the consequences of their decisions at a rate of five times higher than in year one. Likewise, students in year two included many more instances of mathematical justification (Gresalfi, 2011; see Figure 2) in year two, with the rate of mathematical justification more than doubling. Interestingly, the incidence of mathematical unreasonableness was quite low across both years, indicating that although students might have had some calculational errors, they were primarily engaging with the mathematics in a way that indicated their understanding of the content. Thus, it is not the case that mathematical justification increased because students understanding of the mathematical procedures increased.
ISLS
406
ICLS2012
Likewise, students narrative immersion was quite high in year one and in year two, suggesting that the context was equally vivid for students regardless of whether they had received feedback before or after they made their recommendation. This suggests that it is not students immersion in the narrative (how real or important it seemed to be) that supported increased consequential engagement. Finally, students critical engagement increased significantly, indicating that students realized that they had some kind of choice about how they might solve the problem. This difference in critical engagement might give insight into the differences in justification, as discussed below.
Total count of codes across classes
Mathema0cal Jus0ca0on
Matheam0cal Unreasonableness
Consequen0al Jus0ca0on
Figure 2. Coded Final Recommendations Collapsed Across Classes For Year 1 and Year 2 The trends captured by collapsing the classes reflect what happened in the individual classes, although looking between the classes an interesting difference can be seen. In comparing Ms. Bells class from Year 1 and Year 2, differences are apparent in the codes of critical engagement, consequential justification, mathematical justification (see Figure 3). Submitted recommendations from Year 2 displayed over six times more critical engagement than year one, nearly five times more consequential justification, and about three times more mathematical justification. In comparing Ms. Kents class from Year 1 and Year 2, similar differences are apparent in the codes of critical engagement and consequential justification; students submitted recommendations in Year 2 showed about four times more critical engagement, and six times more consequential engagement than students submitted recommendations in Year 1. Interestingly, the trend did not apply for the code mathematical justification, which showed no change.
Narra0ve Immersion
Mathema0cal Jus0ca0on
Figure 3. Coded Final Recommendations For Ms. Bell and Ms. Kent for Years One and Two.
ISLS
Cri0cal Engagement
Matheam0cal Unreasonableness
Consequen0al Jus0ca0on
50 45 40 35 30 25 20 15 10 5 0
Total count of codes
Cri0cal Engagement
Narra0ve Immersion
70 60 50 40 30 20 10 0
Year 1 Year 2
Bell Year 1 Bell Year 2 Kent Year 1 Kent Year 2
407
ICLS2012
The coding of the final recommendations suggest that embedding feedback in advance of submitting final recommendations supports students to include more justification in their responses. This is important as students use of justification gives teachers greater insight into their students thinking (important as an aspect of formative assessment). In addition, justification creates opportunities for students to think more deeply about their own mathematical ideas, and thus more and better justification likely supports deeper understanding of mathematical content (Niemi 1996). However, these quantitative results give no indication of why the change in timing of consequential feedback led to differences in justification, nor why changes in mathematical justification occurred in one classroom and not the other. In seeking to better understand students activity, we examined classroom discussions in more detail, characterizing both the nature of the activity and the content of the conversations that occurred in the two classrooms over the two years.
Further Investigation
As discussed above, the ratios of change for each category were nearly the same, with the exception of mathematical justification. In one class (Kent), the codes for mathematical justification did not change at all, while in the second class (Bell) increased by over four times its original number. The change is particularly interesting when considering that all classes except for Bell Year 2 had mathematical justification totals of 14 or 15. The comparisons between the classrooms across two years allow us to eliminate some potential hypotheses. First, differences in the teaching styles of Ms. Kent and Ms. Bell would not (alone) have been likely produce the increase in mathematical justification for Ms. Bell in the second year. Any teaching differences would have also have been likely to impact the students submitted recommendations for year one as well. One contributing factor might be the time the frequency of whole class discussionsperhaps Ms. Bell simply talked more with her students, thus giving them more opportunities to understand justification and its purposes. As shown in Table 2, whole class discussion time varied among the four environments observed. Discussion in Ms. Kents class remained nearly the same while discussion in Ms. Bells class increased dramatically in Year 2, considering both amount of time and frequency of discussion. This presence of more frequent discussions, and a very large overall time in discussion, occurred only in Ms. Bells Year 2 class, and therefore, could account for some difference present within that class. Table 2. Amount and percent t of time spent in whole class discussion Class Bell Y1 Bell Y2 Kent Y1 Kent Y2 Number of Discussions 2 6 2 3 Average Time In Each Discussion 19.45 minutes 10.6 minutes 15.9 minutes 9 minutes Time in Discussion Total 47.5 minutes 69.5 minutes 31.7 minutes 27 minutes Percent of total class time in discussion 26% 52% 13% 8%
One interesting note is that although the total amount of discussion time increased in Ms. Bells class in year two, the actual time discussing mathematics was almost identical. Indeed, the actual percent of the discussion that focused on mathematics decreased significantly for year two. Thus it is likely not the case that mathematical justification increased for students in Ms. Bells class in year two simply because they had received more instruction about mathematics. Table 3. Amount of time spent in mathematical and consequential discussion Class Bell Year 1 Bell Year 2 Time in mathematical discussion 24 minutes 32 seconds 24 minutes 50 seconds Percent of total discussion time 52% 36% Time in consequential discussion 103 seconds 225 seconds Percent of total discussion time 4% 5%
Considering the evidence that the discussions in Ms. Bells class did not change drastically in their content, it is still plausible that having more time in discussion, and having more frequent discussions may have contributed to the sensitivity of students to the consequentiality factor. The mechanism of the contribution may not be as simple as having more mathematical or consequential discussion, but perhaps the timing of the discussion, or the ways in which discussion was structured. In addition to instructional factors that may have impacted the effects of consequentiality within the game, an additional conjecture might be that there is a difference between the two classes with respect to students sensitivity to consequential feedback. The two classes were very different environments and had quite different student populations. In addition to individual differences of students, the Kent classes were in 7th grade
ISLS 408
ICLS2012
within a project based school, while the Bell classes included 5th grade students enrolled in 6th grade math in an elementary school. It seems possible that the younger students were more likely to play in the narrative of the storyline, thus actually engaging the complexity of the situation more deeply. Indeed, in examining small group interactions, there were several conversations amongst students in Ms. Bells class wherein they seemed to be deeply immersed in the narrative, seen in the ways they played out scenarios beyond what had actually happened (using their imagination to fill in the gaps that the videogame had left). For example, in a small group interaction wherein students were reviewing the feedback they had received, one student summarized his experience as follows: The first time I didnt really think about it. I just said, this time will be Im just going to play around with it. So, the first time I tried it, I took Route Two. It was straight up to Boones Meadow and straight back. Um, I took the extra tank of gas, and I didnt take the cargo box because I thought it would overdo the payload. And, I thought, um, basically (laughs) because when I got there, the eagle had to obviously ride somewhere. And, where was it going to fit? It was a one-person seat, so, um (pause) the eagle fell out (laughs), and I ran out of gas. I found the eagle and took it back to the vet, um, but it died. Several pieces of this students explanation were not included in the actual experience: the eagle cant fall out of the ultralight, and if the student ran out of gas, they experienced the ultralight crashing and then immediately arrived at the door of the vet (they did not have an opportunity to find an eagle after the crash). Thus, in recounting his experience to his two classmates, this student was introducing new details that made his experience narratively consistent, thus immersing himself deeply into the context. Likewise, students in this class sometimes spontaneously offered factors that might impact the birds survival beyond the information given in the scenario. For example, the following solution was offered as an alternative suggestion (note the consequential justification): K: I was going to say. The reason I chose going to here is that it might be a little easier on the eagle because a) there would be no predators T: Because they couldnt get in the plane? K: No outside bacteria or germs or it couldnt get shot again, which is kind of like still (a possibility). And one more thing. You could immediately give first aid to the eagle if you pick it up. In both of these instances, students were immersing themselves in the real world of saving the eagle based on their experience of playing out the mathematical solutions they had proposed. This kind of imaginative conversation, wherein students were generating their own information and examples, was only seen in a single utterance in Ms. Kents classroom.
Discussion
This paper has presented the results of two implementations of a curricular activity that considered how the timing of consequential feedback might impact students engagement with mathematics. Findings suggest that when students have an opportunity to experience consequential feedback before making formal recommendations, their solutions include much higher rates of justification, both mathematical and consequential. These findings are particularly promising given that they bore out across implementations in two very different classrooms. It is not clear why including feedback in advance of careful mathematical calculation, as opposed to after completing these calculations (when feedback typically occurs) is so significant in terms of impacting the ways students engage mathematically. One hypothesis is that locating consequential feedback in the midst of students problem solving is truly formative in that the feedback helps students gain insight into the impact of their mathematical calculations. Consequential feedback can become another data point against which students can contrast their mathematical decisions. For example, in a recent implementation, two students discussed two different outcomesone solution that led to an eagle having his wing amputated, and another that allowed the eagle to live unscathed. These contrasting outcomes prompted the students to re-calculate the time it would take to complete a targeted route, which involved engaging in the mathematics of ratio. Thus, one hypothesis of this study is that consequential feedback actually prompts students to engage more deeply with mathematical content because it prompts them not just to correct, but also to think about how the mathematical works in terms of leading to solutions that arent desired. Indeed, the awareness of possible different outcomes might also explain why instances of justification increased; students realized that there was something that actually had to be justified. However, specific differences between the two classrooms, in particular with respect to mathematical justification, led to deeper investigation into the nature of the implementations across these two classrooms. As detailed above, we had several conjectures about what might have contributed to these differences, some of
ISLS
409
ICLS2012
which were not borne out. Specifically, it wasnt the case that the classroom that had higher rates of mathematical justification simply talked more about math; nor what it the case that the teacher in that classroom gave different instructions that emphasized mathematical justification. Differences in the total amount of classroom conversation might have contributed to these differences. Alternatively, something about the nature of students engagement with the narrativeand the extent to which that helped them become aware of the myriad possible solutionsmight also have contributed to this difference. The differences between the nature of students engagement with the narrative suggests a possible avenue for future research: considering to what extent narrative immersion becomes a resource for students meaningful conceptual engagement.
References
Bangert-Drowns, R.L., Kulick, J.A. & Morgan, M.T. (1991) The instructional effect of feedback in test-like events. Review of Educational Research, 61,213-238. Barab, S. A. & Plucker, J. A. (2002). Smart people or smart contexts? Cognition, ability, and talent development in an age of situated approaches to knowing and learning. Educational Psychologist 37, 165-182. Barab, S. A. & Roth, W.M. (2006). Curriculum-based ecosystems: Supporting knowing from an ecological perspective. Educational Researcher 35, 3-13. Barron, B. J., Schwartz, D.L., et al. (1998). Doing With Understanding: Lessons From Research on Problemand Project-Based Learning. Journal of the Learning Sciences 7, 271. Black, P. & William, D. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan 80, 139-148. Boaler, J. (2000). Exploring situated insights into research and learning. Journal for Research in Mathematics Education, 31, 113-119. Brown, J. S., Collins, A., et al. (1989). Situated cognition and the culture of learning. Educational Researcher 18, 32-42. Carr, M. & Claxton, G. (2002). Tracking the development of learning dispositions. Assessment in Education 9, 9-37. Cobb, P., Stephen, M., McClain, K., & Gravemeijer, K. (2001). Participating in classroom mathematical practices. The Journal of the Learning Sciences, 10, 113163. Cognition and Technology Group at Vanderbilt. (1997) The Jasper Project: Lessons in Curriculum, Instruction, Assessment, and Professional Development. Mahwah, NJ: Lawrence Erlbaum Greeno, J. G. (1991). Number sense as situated knowing in a conceptual domain. Journal for Research in Mathematics Education, 22, 170-218. Greeno, J. G. (1994). Gibson's affordances. Psychological Review, 101, 336-342. Greeno, J. G., & Gresalfi, M. S. (2008). Opportunities to learn in practice and identity. In P. A. Moss, D. C. Pullin, J. P. Gee, E. H. Haertel, & L. J. Young (Eds.), Assessment, Equity, and Opportunity to Learn. New York: Cambridge University Press. Gresalfi, M.S., & Barab. S. (2011) Learning for a Reason: Supporting Forms of Engagement by Designing Tasks and Orchestrating Environments. Theory Into Practice. 50, 300-310. Gresalfi, M. S., Barab, S.A. Siyahhan, S., & Christensen, T. (2009). Virtual worlds, conceptual understanding, and me: Designing for Critical engagement. On the Horizon, 17: 21-34. Kluger, A.N. & Denisi, A. (1996) The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory, Psychological Bulletin, 119, 254-284. Mory, E. H. (2004) Feedback Research Revisited. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology. New York: Simon & Schuster Macmillan. Nathan, M. J., Kintsch, W. et al. (1992). A Theory of Algebra-Word-Problem Comprehension and Its Implications for the Design of Learning Environments. Cognition and Instruction 9, 329-389. Niemi, D. (1996). Assessing conceptual understanding in mathematics: Representations, problem solutions, justifications, and explanations. Journal of Educational Research, 89, 351-363. Reed, E. (1996). Encountering the world; Toward an ecological psychology. Oxford: Oxford University Press. Schimmel, B. J. (1983). A meta-analysis of feedback to learners in computerized and programmed instruction. Paper presented at the annual meeting of the American Educational Research Association, Montreal. Schute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78, 153-189. Sweller, J., van Merrinboer, J.J.G., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251296.
ISLS
410
ICLS2012
Multiple Trajectories for Understanding Ecosystems

Catherine Eberbach, Cindy Hmelo-Silver, Rebecca Jordan, Suparna Sinha Rutgers University, 10 Seminary Place, New Brunswick, NJ 08901 catherine.eberbach@gse.rutgers.edu, cindy.hmelo-silver@gse.rutgers.edu, jordan@aesop.rutgers.edu, suparana.sinha@gse.rutgers.edu Ashok Goel, Georgia Institute of Technology, Atlanta, GA 30332, goel@gatech.edu Abstract: This is an exploratory study about how middle school students develop an increasingly coherent understanding of ecosystems. As part of a broader design research program, we coded and analyzed 12 middle school students drawings of aquatic environments collected before, during, and after a technology-rich instructional intervention. Coding considered several dimensions: relations (macro/micro, biotic/abiotic, structurebehavior-function) as well as coherence, and extraneous structures. Findings suggest that students may follow multiple non-linear trajectories towards an increasingly coherent understanding of ecosystems. Even so, the ability to observe phenomena at multiple macro/micro and biotic/abiotic levels may be an underlying constraint to also observing integrated structure-behavior-function relations and the development of a more coherent understanding of ecosystems.
Systems thinking is at the forefront of science education standards and is important for being scientifically literate citizens (Committee on Conceptual Framework for the New K-12 Science Education Standards, 2011; Jordan et al, 2009). Yet systems thinking poses many challenges for learners (Hmelo-Silver & Azevedo, 2006; Jacobson, 2001). Some of these challenges include making connections across system levels, including a systems structures, underlying mechanisms (or behaviors), and functions (Goel et al, 1996; Hmelo-Silver, Marathe, & Liu, 2007), connecting macro level phenomena with micro level phenomena (Penner, 2001), as well as connecting a systems biotic aspects with abiotic aspects such as sunlight (Eilam, in press; Shepardson, Wee, Priddy, & Harbor, 2007). Although learning research has documented these challenges as well as identifying what students learn after participating in instruction about systems (e.g., Assaraf & Orion, 2005; Eberbach & Hmelo-Silver, 2010; Yoon, 2008), little research has documented the learning trajectories that students take to understanding ecosystems. Our focus here on learning trajectories is consistent with the recent emphasis on learning progressions (Duncan & Hmelo-Silver, 2009; Duschl, Schweingruber, & Shouse, 2007). In particular, Mohan, Chen, & Anderson (2009) developed a learning progression for carbon cycling. Through a cross-sectional analysis, they identified four levels that represented increasingly sophisticated reasoning. Across these levels, students ideas became more coherent as they progressed from informal accounts dominated by agents to more scientific accounts in which processes (i.e., matter transformation) occur at multiple levels and scales. At the lowest level, students largely generated macroscopic narratives focused on objects and events but did not observe connections between biotic and abiotic aspects. At the next level, students composed narratives of causal sequences of events, which included unseen mechanisms but did not account for constraining principles such as conservation of matter. The authors described the third level as school science narrative about processes (Mohan et al., p. 688), where students explained some processes in terms of chemical interactions. At the highest levelwhere national standards are setstudents gave detailed qualitative process accounts of systems. For example, students should be able to explain organismal changes in terms of chemical processes and consistently identify materials being exchanged in living systems. It is not surprising that few students reached this highest level. In fact, most middle school students functioned at levels one and two and high school students at levels two and three. Although Mohan et al (2009) characterize student thinking and achievement, they do not describe learning trajectories over multiple time points for individual learners, nor do they account for how different aspects of these levels develop. Often, student understanding of curricula is assessed using pre- and posttests that may not reveal detailed learning trajectories in designed settings, such as the technology-rich learning environment that is the focus of our research. This kind of testing assumes that growth is linear over time, which is not necessarily a justifiable assumption when trying to understand individual change. Consider that multiple paths may lead to the same outcome and that outcomes at a group level may be different from those that occur at an individual level (Sloane & Kelly, 2008). In a discussion about the measurement of learning progressions, Wilson (2009) notes that different dimensions of learning may be staggered, meaning that there may be different trajectories for the different dimensions. One way to look at this kind of complex change in learning is the microgenetic method (Siegler & Crowley, 1991), which gathers detailed data from learners over an extended period to gain deep understanding
ISLS 411
ICLS2012
of changes in understanding. Initially developed for laboratory conditions, Chinn (2006) argued that a microgenetic approach can be adapted for classrooms to understand the rate and timing of learning, the intermediate steps along the way, and how different changes might be clustered. To adapt this method to the classroom, the grain size must be necessarily larger and measurement occasions less dense. One approach could be through video analysis of intensive learning episodes, which is not particularly practical for capturing learning on large numbers of students. Alternatively, student performance on related tasks could be examined over multiple time points. This is the approach we adopted to explore and identify possible learning trajectories of middle school students as they learn about aquatic ecosystems.
Methods
This study is part of a larger program of design research that develops and explores computer-based instructional interventions that facilitate student understanding of aquatic ecosystems (Hmelo-Silver et al, 2011). To promote systems thinking in a way that enables students to think about multiple interacting components, we organized instruction around the structure, behavior, and function (SBF) conceptual representation (Goel, et al., 1996; Hmelo-Silver, et al., 2007). Doing so explicitly portrays a systems structures (i.e., configuration of components and connections), functions (i.e., output), and behaviors (i.e. causal mechanisms that enables the components functions). Thus, the SBF conceptual representation functions as a lens that enable students to observe, model, and understand the relationships among form and function, as well as the causal behaviors and mechanisms of complex systems. The context for this analysis was a five to six week enactment of a new curriculum unit on aquatic ecosystems in two middle school classrooms. The unit was organized around the problem of fish mysteriously and suddenly dying in a local pond. Students engaged in inquiry through evaluation of assorted forms of scientific evidence, the use of NetLogo simulations (Wilensky & Reisman, 2006), and the creation and revision of SBF models using the Ecosystem Modeling Toolkit (EMT) (Vattam, et al., 2011). As students used these simulations, they had many opportunities to explore factors that affect the dynamic balance in a pond and an aquarium. Students also read supporting information in the form of hypermedia text that was organized around function-oriented questions and content (Liu & Hmelo-Silver, 2009). The simulations and instruction emphasized photosynthesis, cellular respiration, and decomposition, all of which are processes critical to explaining how and why the fish died suddenly in the pond. Students learned about pond ecosystems and the aquarium as a model aquatic system in both classrooms. However, due to challenges with initially sequencing lessons in Classroom 1, we redesigned some sequences for Classroom 2. In classroom 1, students were initially presented with a problem about fish dying in a local pond followed by designing aquariums as model systems. In classroom 2, students initially designed aquariums as model systems and then were introduced to the problem about the pond.
Participants and Data Sources

Study participants included 12 students from two public middle school science classrooms in two schools in northeastern United States. Students were selected for this analysis because they showed improvement in articulating more complex SBF and Macro/Micro relationships at posttest. Six students were selected from both classrooms. Data sources included student drawings from two sets of drawing assessment tasks. In all, we analyzed four drawings from each student (totaling 47 because of student absence). The first set, which was administered before and after classroom instruction, asked students to draw what happens in an aquatic ecosystem. The second set, described here as the Aquarium Assessment, was administered twice during classroom instruction. Based upon what they learned about how the fish died in the pond, students were asked how they would set up an aquarium to keep fish healthy. Both sets of drawing tasks also instructed students to draw lines to indicate relationships between parts of the drawing and to label the lines to explain why they made these connections. Because the curriculum was modified after its enactment in Classroom 1, students completed the Aquarium Assessment at slightly different points in time. Specifically, in Classroom 1, Aquarium Assessment 1 (AA1) was implemented after introductory modeling and evidence assessment activities and EMT model revisions. Before Aquarium Assessment 2 (AA2), students revised EMT models and interacted with simulations focused on behaviors and functions of macro and micro level phenomena and nitrification simulations. In Classroom 2, AA1 was implemented after introductory modeling activities, the construction of a live aquarium model, and interactions with a simulation on carrying capacity. Before AA2, students revised EMT models and interacted with simulations focused on behaviors and functions of macro and micro level phenomena.
Coding
All drawings were coded along multiple dimensions to better understand how interacting components and processes may affect increasingly complex systems thinking. We developed coding along the following dimensions: Structure-Behavior-Function (SBF) relations, Macro/Micro relations, Biotic/Abiotic relations,
ISLS 412
ICLS2012
Extraneous Structures, and Coherence. Scoring criteria are summarized in Table 1. The first two authors discussed each student drawing and reached agreement through discussion. Table 1. Scoring Criteria for Student Drawings Level 1 Identifies only macro structures or processes* Level 2 Identifies both macro and micro structures or processes Level 3 Identifies relationship(s) between macro and micro structures or processes
Macro/Micro (e.g., fish, plants/ bacteria, oxygen) Biotic/Abiotic (e.g., fish, plants/ ammonia, sun) SBF
Identifies only biotic structures* **
Identifies both biotic and abiotic structures
Identifies relationship(s) between biotic and abiotic structures Identifies structures in relation to behaviors and functions Includes at least 1 extraneous structure
Single cluster covering Multiple clusters with everything or multiple connections among clusters disconnected clusters *Based on our observations, students only began with macro or biotic structures. **If only one abiotic structure appeared in a largely biotic scene, we coded the drawing at Level 1. A higher score represents a more desirable outcome, except for extraneous structures. To illustrate how the coding was applied to student drawings, we examine the pre and post-test drawings of Student 1 (See examples in Figure 1). We applied the Macro/Micro code as Level 1 in the pre-test example because all structures (e.g., fish, coral, seaweed) are macroscopic, whereas the posttest example is coded as Level 3 because the student identifies relations between macro and micro levels (e.g., fish and ammonia, algae and oxygen). We applied the Biotic/Abiotic code as Level 1 in the pre-test example because the student drew a largely biotic scene and included only one abiotic structure (ocean floor). In the posttest example, we coded this as Level 3 because the student included examples of biotic and abiotic structure relations (e.g., algae and sunlight; bacteria and nitrate). In both drawings, no structures were deemed irrelevant so Extraneous Structures was coded as Level 1 for each. For SBF, the pre-test example was coded as Level 2 because the student related structure and behavior relations (e.g., starfish eats the clams; fish lives in the coral). In the posttest example, Student 1 reached Level 3 of the SBF code (e.g., sunlight causes algae to grow links to algae makes oxygen for fish). For coherence, we coded the pre-test example as Level 2 because it could be interpreted as a single cluster covering everything, whereas the posttest example was coded as Level 3 because the student included multiple connecting clusters (see SBF posttest example).
Extraneous (e.g., castles, divers) Coherence
Identifies structures without connecting to behaviors or functions Includes no extraneous structures Phenomena is isolated
Identifies structures in relation to behaviors or functions N/A
Results
Exploring Learning Trajectories in Two Classrooms
To support our analysis, we constructed two graphs (Figure 2) that depict the group means for each systems dimension in each classroom. In looking at the graph of Classroom 1, students appear to have made steady gains towards higher levels of macro, biotic, SBF, and Coherence dimensions, and essentially converging at posttest. From the start, students represented structures in terms of either their behaviors or functions and made connections between phenomena (Coherence). However, these structures were mostly macroscopic and biotic. Even so, they could articulate relationships between structures either by AA2 (Biotic/Abiotic) or by posttest (Macro/Macro). Their understanding of SBF appeared to be flat at AA1, but otherwise showed steep growth so that by posttest all students could represent phenomena at the highest level of SBF. On average, Coherence trailed all other dimensions and approached Level 3 by posttest. Finally, the inclusion of extraneous structures spiked at AA2 and then declined until no drawings included extraneous structures by posttest.
ISLS
413
ICLS2012
Figure 1. Student 1s drawing at pretest (left) and posttest (right) with students explanatory labels.
In Classroom 2, students appear to have made quick gains in the macro and biotic dimensions, but to more gradually shift to more complex representations of SBF and Coherence. Between pre-test and AA1 there is a steep change in Biotic/Abiotic and Macro/Micro structures. On average, students quickly shifted from largely macroscopic and biotic structural representations to ones that depicted relations between both Biotic/Abiotic and Macro/Micro structures. This change remains stable through posttest for the macro/micro dimension, but declined slightly for the biotic/abiotic dimension at posttest. Students started with some understanding that structures have behaviors or functions within a system. However, it was only at AA2 that modest growth in SBF was apparent, but even at posttest not all students reached the highest SBF level. Coherence also showed a relatively gentle slope that approached a mean Level 3 at posttest. Few extraneous structures were included at pre-test and no longer appeared after AA1. Recall that each classroom experienced slightly different activity sequences. Despite these differences, certain patterns emerged across classrooms. On average, students in both classrooms started with understanding that structures have some behaviors or functions in a system. Students made more complex connections between macro and micro structures as well as between biotic and abiotic structures before reaching the highest levels of SBF or Coherence. Thus, from these graphs, we might expect that before students observe more complex relations between structure, behavior, and function, they need to master connections among other system levels. Finally, Coherence did not perfectly track any one dimension, suggesting that Coherence detects something different from SBF.
Individual Student Learning Trajectories

Until now we have only considered the mean classroom learning trajectories, but this level of analysis is only part of the story. What of individual student trajectories? Visually inspecting the trajectories of Student 1 and Student 2 (Figure 3) reveals that there can be considerable variability in the paths that individual students take to achieve the same endpoint. In the case of Student 1, it is evident that while starting with a mid-level understanding of SBF, these are limited to macroscopic and biotic structures, possibly with no particular connections. By AA2, the student has reached the highest levels on all dimensions, but still included some extraneous structures. In comparison, Student 2 has also started with a mid-level understanding of SBF and has also focused largely on macroscopic and biotic structures, but could identify mid-level coherent relationships between the structures. By the AA2, this student also reached the highest levels of Biotic/Abiotic, Macro/Micro structure relations, but it was not until posttest that the student could also represent SBF and Coherence at the highest levels as well. Unlike Student 1, this student never included extraneous structures in any of the assessments.
ISLS
414
ICLS2012
Figure 2. Mean level of system dimensions in each classroom, Classroom 1 (top) and Classroom 2 (bottom)
Looking more closely at the assessment drawings of Student 1, we see that the student begins by working at a simple macroscopic level and is focused on biotic structures only (Figure 1, left). As the student moves from AA1 (Figure 4, left), his/her drawing reflects connections among Macro/Micro and Biotic/Abiotic levels. Structures remain connected to overall behavior or function, but elaborated mechanisms do not yet appear. Additionally, decorative elements (i.e., castle) have entered the picture. In AA2 (Figure 4, Right), which was completed after the students engaged in modeling activities and simulation-supported inquiry, connections between SBF appear in the drawing. Although the castle is still present, we note that it is now in the background. There is also an increased level of connectedness within the drawing that persists to the posttest (Figure 1, Right). In addition, the posttest includes key ecosystem processes such as photosynthesis. Although space does not permit detailed qualitative analyses, we found that assigning thematic narratives to student drawings to be a useful lens for characterizing shifts in student conceptualizations of ecosystems. Student 1s drawings are typical. At pre-test, the dominant narrative habitat depicts a static set of structures with fixed relations. In comparison, the posttest narrative dynamic processes depicts ecosystems as being driven by processes, such as photosynthesis and cellular respiration that both affect and are affected by changing conditions across multiple levels.
ISLS
415
ICLS2012
Figure 3. Learning trajectory for Student 1 (top) and Student 2 (bottom) both from Classroom 1
Discussion
We conducted an exploratory study of how middle school students develop coherent understanding of a natural system, using aquatic ecosystems as an example. Students drawings were generally the most coherent when they made connections across all levels. Even so, both classes showed a dip in understanding biotic and abiotic relations at posttest after making gains through AA2. This is a good reminder that learning does not always happen in linear patterns and that different dimensions of learning may have different paths (e.g., Wilson, 2009). Likewise, new transitional knowledge can sometimes be fragile (e.g., Siegler & Crowley, 1991). There also seems to be a lot that can be learned by studying the variability among individual learning trajectories. At the same time, the findings suggest that there are potentially foundational dimensionsMacro/Micro and Biotic/Abiotic relationsthat are necessary for learning about aquatic ecosystems. Ongoing analyses of a more heterogeneous sample also suggest that students who stalled at mid-levels of Coherence included macro and micro as well as biotic and abiotic components, but seldom in relation to one another. Moreover, those students who start at moderate levels of Coherence or SBF typically include macro and biotic components only. This may be the start of a learning progression for ecosystems that could be empirically validated in the future. Additional investigations might examine whether students include relationships between levels of Macro/Micro and Biotic/Abiotic structures because these are relatively easier to achieve or because they are prerequisites for
ISLS
416
ICLS2012
observing dynamic processes that require integrating across levels and developing higher levels of SBF thinking and Coherence.
Figure 4. Drawings by Student 1 for Aquarium Assessment 1 (top) and Aquarium Assessment 2 (bottom).
These findings are encouraging, but much more work is needed before drawing firm conclusions. Future analyses will include a larger and more diverse sample. However, we now have preliminary evidence that these dimensions may provide a plausible account of student learning by adapting microgenetic techniques to the classroom. This illuminates how changes in systems thinking may occur in a technology-rich inquiry environment.
ISLS
417
ICLS2012
References
Ben-Zvi Assaraf, O. & Orion, N. (2005). Development of system thinking skills in the context of earth system education. Journal of Research in Science Teaching, 42(5), 518-560. Chinn, C. A. (2006). The microgenetic method: Current work and extensions to classroom research. In J. L. Green, G. Camilli & P. Elmore (Eds.), Complementary methods for research in education (3rd ed., pp. 439-456). Washington, D. C.: Washington, DC: American Educational Research Association. Committee on Conceptual Framework for the New K-12 Science Education Standards. (2011). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press. Duncan, R. G. & Hmelo-Silver, C. E. (2009). Learning progressions: Aligning curriculum, instruction, and assessment. Journal of Research in Science Teaching, 46, 606-609. Duschl, R. A., Schweingruber, H. A., & Shouse, A. W. (Eds.). (2007). Taking science to school: Learning and teaching science in grades K-8. Washington, D.C.: The National Academies Press. Eberbach, C. & Hmelo-Silver, C. E. (2010). Observing the seen and unseen: Computer and social mediation of a complex biological system. In Z. C. Zacharia, C. P. Constantinou & M. Papaevripidou (Eds.), Computer Based Learning in Science (pp. 213-224). Warsaw: OEliZK. Eilam, B. (in press). System thinking and feeding relations: Learning with a live ecosystem model. Instructional Science. Goel, A. K., Gomez de Silva Garza, A., Gru, N., Murdock, J. W., Recker, M. M., & Govindaraj, T. (1996). Towards designing learning environments -I: Exploring how devices work. In C. Fraisson, G. Gauthier & A. Lesgold (Eds.), Intelligent tutoring systems: Lecture notes in computer science. NY: Springer. Hmelo-Silver, C. E., Jordan, R., Honwad, S., Eberbach, C., Sinha, S., Goel, A., Rugaber, S., Joyner, D. (2011). Foregrounding behaviors and functions to promote ecosystem understanding. In Proceedings of Hawaii International Conference on Education (pp. 2005-2013). Honolulu HI: HICE. Hmelo-Silver, C. E. & Azevedo, R. (2006). Understanding complex systems: Some core challenges. Journal of the Learning Sciences, 15, 53-61. Hmelo-Silver, C. E., Marathe, S., & Liu, L. (2007). Fish swim, rocks sit, and lungs breathe: Expert-novice understanding of complex systems. Journal of the Learning Sciences, 16, 307-331. Jacobson, M. J. (2001). Problem solving, cognition, and complex systems: Differences between experts and novices. Complexity, 6(2), 1-9. Jordan, R.C., F. Singer, J. Vaughan, and A. Berkowitz. 2009. What should every citizen know about ecology? Frontiers in Ecology and the Environment, 7, 495-500. Liu, L. & Hmelo-Silver, C. E. (2009). Promoting complex systems learning through the use of conceptual representation in hypermedia. Journal of Research in Science Teaching, 46(9), 1023-1040. Mohan, L., Chen, J., & Anderson, C. W. (2009). Developing a multi-year learning progression for carbon cycling in socio-ecological systems. Journal of Research in Science Teaching, 46, 675-698. Penner, D. E. (2001). Complexity, emergence, and synthetic models in science education. In K. Crowley, C. Schunn & T. Okada (Eds.), Designing for science (pp. 177-208). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Shepardson, D. P., Wee, B., Priddy, M., & Harbor, J. (2007). Students' mental models of the environment. Journal of Research in Science Teaching, 44, 327-348. Siegler, R. & Crowley, K. (1991). The microgenetic method: A direct means for studying cognitive development. American Psychologist, 46, 606-620. Sloane, F. C. & Kelly, A. E. (2008). Design research and the study of change: Conceptualizing individual growth in designed settings. In A. E. Kelly, R. A. Lesh & J. Y. Baek (Eds.), Handbook of Design-based Research Methodology (Vol. 441-448, pp. 299-319). New York: Routledge. Vattam, S., Goel, A. K., Rugaber, S., Hmelo-Silver, C. E., Jordan, R., Gray, S., & Sinha, S. (2011). Understanding complex natural systems by articulating structure-behavior-function models. Educational Technology & Society 14(1): 66-81. Wilensky, U. & Reisman, K. (2006). Thinking like a wolf, a sheep, or firefly: Learning biology through constructing and testing conceptual theories - an embodied modeling approach. Cognition and Instruction, 24, 171-209. Wilson, M. (2009). Measuring learning progressions: Assessment structures underlying a learning progression. Journal of Research in Science Teaching, 46, 716-730. Yoon, S. (2008). An evolutionary approach to harnessing complex systems thinking in the science and technology classroom. International Journal of Science Education, 30(1), 1-32.
Acknowledgements
This research was funded by IES grant #R305A090210. Conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of IES.
ISLS 418
ICLS2012
Supporting Learners to Conceptualize the Vast Range of Imperceptible Smallness with Temporal-aural-visual Representations
Minyoung Song, Chris Quintana University of Michigan School of Education 610 E. University Ave. Ann Arbor, Michigan 48109 USA mysong@umich.edu, quintana@umich.edu Abstract: Teaching and learning about the sizes of the objects that are too small to see with human eyes (called imperceptible sizes) have been a difficult issue in science education. Because representations are the only external input that learners use to make sense of imperceptible phenomena, the design of the representations becomes critical. However, studies imply that commonly used conventional visual representations tend to direct learners to overestimate imperceptible sizes and underestimate the difference between them. Hence, the need for a novel representation that can provide the way learners an alternative experience with the imperceptible sizes emerges. To address this issue, we designed a multimodal representation that incorporates temporal sense to transmit imperceptible sizes. In this paper, we introduce our recent research that explored the different effects of the compressed and extended form of temporal experience on student learning of the range of imperceptible sizes.
Introduction
When students try to conceptualize imperceptible phenomena such as the size and behavior of subatomic particles, it requires them to think about things that humans cannot see with naked eyes. To support learners to construct such knowledge that are beyond human senses, teachers provide various forms of representations to learners. Representations allow him/her to understand a concept which could have been beyond his or her cognitive capability (Hutchins, 1995; Pea, 1993; Salomon, Perkins, & Globerson, 1991; Zhang & Norman, 1994) and shape or promote a learners mental model of the knowledge. In this way representations have made critical contributions to the advancement and education of science. The role of representations becomes particularly critical in teaching and learning about imperceptible phenomena because what learners perceive and conceptualize is mediated only by the representations they exploit. Particularly, a number of learning technologies have been developed to support learners to understand imperceptible sizes by adopting various forms of visual representations such as logarithmic scale, static graphics (e.g., aligning enlarged images of a strand of hair and a red blood cell for size comparison), video (e.g., Powers of Ten) or interactive graphics (e.g., Scale Ladder). Such learning technologies convey imperceptible sizes usually by providing alternative visual experience of imperceptible objects (e.g., atoms, molecules, bacteria, virus, and cells). For example, the visual representations of imperceptible objects are enlarged to a visible scale or they gradually grow as a student interacts with them. In many cases such visual representations are incorporated with the mathematical descriptions (e.g., The diameter of a Hydrogen atom is about 10,000,000 smaller than one millimeter) that require students to mentally visualize the sizes of the objects through proportional reasoning. Sometimes the visual representations are accompanied even with absolute sizes such as, The diameter a Hydrogen atom is about 0.1 nanometers. However, research indicates that learners are likely to face cognitive challenges in interpreting and comprehending such visual representations of imperceptible sizes. Frequent exposure to the macroscopic visual depictions of imperceptible objects seems to mislead students to overestimate the sizes, even those who are aware of the fact that such representations are controversial in terms of size. For example, in Tretter et al.s study(Tretter, Jones, Andre, Negishi, & Minogue, 2006a), their participants answered, when I picture [microscopic objects] I see the drawing [in textbooks]. In one of our interviews with middle school students we noticed that some 7th grade students even thought that bacteria and viruses (germs, in their term) are as big as dust or a particle of fine-grained salt, but they are only invisible because they are transparent, inferring from what they have observed in educational video. Some other students thought that the germs would be visible under a simple magnifier according to what they have seen in media such as a disinfectant spray TV commercial. As a result, students construct na beliefs that all imperceptible objects are roughly the same size, ve even similar with small macroscopic objects such as a grain of rice or an ant (Tretter, et al., 2006a) although there exist a vast range of different imperceptible sizes. Students such misconceptions seem to be developed under the significant influence of the bigenough-to-see images of imperceptible objects. Visual representations are very effective for illustrating the
ISLS 419
ICLS2012
shape, features, movement or physical relationship between certain objects, as the way they are frequently and effectively used to illustrate many scientific concepts, however, they tell learners the opposite of too small to see. When representations stand between the goal knowledge and learners, they act as an intermediary, requiring them to be proficient in both interpreting the representations and making sense of the concept. Inasmuch as cognitive processes required of the intermediary are extra cognitive burdens and foreign to learners, it can add to the overall cognitive complexity for learners. The more premises and premise-based inconsistencies, the more alternative mental models are constructed, making the cognitive task more difficult (Rapp, 2005). There are too many cognitive steps that learners have to deal with to conceptualize an imperceptible size with its macroscopic depiction. Learners have to accept the macroscopic images but at the same time have to cognitively resist them to grasp the concept that the objects are not in fact seeable to human eyes, then they have to try to imagine how actually small the objects would be through mental calculation that they are not capable of. Therefore, there rises a need for a novel form of representation that tells learners, This is what too small to see means, and this is how small they are via non-visual modality and with less cognitive burdens. Inappropriately designed representation can bring disadvantages in learning. Without proper designs, students are less likely to know what to look and what is being conveyed by the representation because learners tend to focus on the surface features of a representation (Kozma, Chin, Russell, & Marx, 2000). Preexisting naive beliefs are resistant to modify, but this resistance can be lessened if a learner is provided with an explanation that show why the flawed beliefs are incorrect. It is known that a new type of representation that guides learners to explore the concepts in a different way may help them realize and revise their misconceptions by revealing the inconsistencies between their exiting mental model and the new representation (Ainsworth, 1999; Chi, 2005). To refute the na conceptions in learners and to provide an alternative way to think about ve the range of imperceptible sizes, we designed a temporal-aural-visual representation (TAVR).
TAVR: Temporal-aural-visual Representation

TAVR does not employ a visual modality as a main vehicle for conveying imperceptible sizes; it takes the temporal sense as a main modality. The sense of time here refers to the duration of a sequential accumulation across the head of a pin (1 millimeter in diameter). This sequential accumulation of an imperceptible object is continued until the object fully spans across the pinhead. Hence the duration of the sequential accumulation inferentially implies the size of the placed object via the inverse relationship between the accumulation time and size; the longer the accumulation, the smaller the object and vice versa. The temporal representation is incorporated with two other modalities aural and visual, which are adopted to augment learners temporal experience. They were added based on the dual coding theory (Paivio, 1986), which explains that information is processed through two separate but parallel channels - visual and aural. When one object is placed, an audio clip that sounds like a single click of a mouse button (aural representation) is played once. The accumulation within imperceptible scale is indicated via the click, and visual representation (the red line) is added only after the length of the accumulation enters macroscopic scale. The elapsed time is shown as illustrated in Figure 1. Figure 2 is the screen captures of the accumulation of Hydrogen atom at three different phases: (a) before a learner presses the Play button, (b) the progress of the accumulation during about 11 hours (notice the short red line that indicates the current progress of the accumulation), (c) the accumulation is completed. The completion is notified by the bright cyan-colored square around the icon and the label on the button (says Replay). Students can jump ahead the accumulation by clicking >>FF button (meaning Fast Forward) below the Play button.
Figure 1. An illustration of the accumulation process in TAVR. (a) The first object is placed on the pinhead, and one click is played. (b) The second object is placed on the pinhead next to the first one, and one click is played. (c) The process continues until the object spans the pinhead.
(a)
(b)
(c)
Figure 2. A screen capture of the TAVR of Hydrogen atom.
ISLS
420
ICLS2012
We believe that a temporal modality may be able to provide an alternative perceptual experience of sizes because the studies on spatial cognition imply that the concept of time and the concept of space are interwoven in peoples mental model (Droit-Volet, 2001; McCormack & Hoerl, 2008; Tretter, Jones, & Minogue, 2006b; Vallesi, Binns, & Shallice, 2008). Our mental representations of things we can never see or touch may be built, in part, out of representations of physical experiences in perception and action that happens over a certain period of time (Casasanto & Boroditsky, 2008). For example, the students participated in Tretter et al.s (2006a) study compared the time that took for driving across their town and the state of North Carolina not only to judge which distance is longer and but also to reason how distinctively different they are with each other. Furthremore, the units of time (e.g., second, minute, hour, day, week, month, and year) may work as the conceptual metaphors for the vast range of imperceptible sizes. Conceptual metaphor is created by analogically extending the conceptual structure from richer, experience-based domains (the units of time in this study) to structure learners understanding of relatively more abstract domains (the sizes of imperceptible objects) (Boroditsky, 1997; Boroditsky & Ramscar, 2002; Lakoff & Johnson, 1980). Literature on size cognition (see Kosslyn, 1980) indicates that conceptual size categories (e.g., small, large) are critical components of the mental model of sizes, as categorization is one of the main ways that people make sense of experience and repair misconceptions with (Chi & Roscoe, 2002; Lakoff, 1987). From the daily experiences most of learners have a very firm idea about the units of time and their orders in terms of the length. It may be easier for them to interpret the difference between the sizes represented in TAVR time (e.g., the size of Rhinovirus is about 1 hour of TAVR time) than in the relative sizes (e.g., the diameter of a Rhinovirus is about 40,000 times smaller than one millimeter) or in the measurement units that are used to discuss absolute sizes (e.g., the diameter of a Rhinovirus is about 25 nanometer) 1.
Application: Wow, it is small! (WIIS)

We designed Wow, It Is Small! (WIIS), a Flash-based learning environment where students can interact with TAVRs for selected imperceptible objects following the scaffolding work in Quintana et al. (2004) (see a screen capture in Figure 4 in the next page). In WIIS, the largest units of the accumulation time of the selected sample imperceptible objects match with the scale category they belong to (see Figure 3). For example, it takes several seconds for microscopic objects, hours for nanoscopic objects, and days for sub-nanoscopic objects. A simplified demo of TAVRs and WIIS are available at http://www.wow-it-is-small.info/index.html.
Red blood cell ~6 micrometer
E coli bacterium ~2 micrometer
Rhinovirus ~24 nanometer
DNA helix ~2 nanometer
Water molecule ~0.3 nanometer
Hydrogen atom ~0.1 nanometer
Figure 3. The sizes and the duration of accumulation for the imperceptible objects used in this study.
1
1 nanometer equals 10-9 meter, and 1 micrometer equals 10-6 meter

421
ISLS
ICLS2012
Students interact with TAVRs
Students exhibit their hypotheses
Figure 4. A screen capture of WIIS. Students can exhibit their prior knowledge and hypothesis in the lower part of the screen (white background) and interact with TAVRs in the upper part of the screen (light blue background). For more information and to try the full version of WIIS, visit http://www.wow-it-issmall.info/index.html.
Prior Findings
In a pilot study we explored whether middle school students could understand temporal-aural-visual representation (Song & Quintana, 2009) and how they make use of their experience with TAVR to revise their mental model of the range of imperceptible sizes. The students performance on learning tasks and interviews showed that students could understand that: (1) one click corresponds to one object placed on the head of a pin, (2) the smaller the object, the longer the accumulation (the inverse relationship between the accumulation time and size), and (3) sound is the only indicator for the accumulation until the accumulation becomes macroscopic. The inverse relationship between the accumulation time and the size of the object was not difficult to understand for middle school students. We also observed that TAVR was particularly more effective for supporting students to realize extremely small sizes (i.e., the size of an atom and molecule), which consequently lead them to construct refined mental models of the range of imperceptible sizes. It was apparent that time is a meaningful concept with which learners can construct a comprehension of abstract spatial knowledge.
Compressed vs. Extended Temporal Experience of Imperceptible Sizes

The goal of our recent study is to investigate the different learning effects Table 1: Comparison between two different accumulation velocities with the compressed and extended and the corresponding total accumulation time in TAVR. temporal experience of imperceptible. The Accumulation rapidity first version of TAVR places 10 objects in Object 1 object / sec 10 objects / sec one second and was designed for the (extended) (compressed) practical in-class use. If the simulation took Hydrogen atom 115 d 17 h 46 min 40 s 11 d 13 h 46 m 40 s too long teachers would not be able to use Water molecule 23 d 3 h 30 min 35 s 2 d 7 h 33 m 35 s WIIS in their classrooms that allows DNA helix 5 d 18h 53 min 20 s 13 h 53 m 20 s limited amount of instruction time although Rhino virus 12 h 37 min 30 s 1 h 15 m 45 s students could fast forward the E coli bacterium 8 min 20 s 50 s accumulation process with the jump Red blood cell 2 min 40 s 16 s ahead button. However, the initial prototype was designed to place one object per second. We thought in this way it might be easier for learners to approximately grasp the sense of the number of the object that was placed on the pin during the elapsed time.
ISLS 422
ICLS2012
Additionally, as Table 1 shows, the total durations of accumulation become dramatically longer when the rapidity of accumulation is one object in one second, and hence the relative difference between the accumulation durations of different objects become greater. Because in a prior study we noticed that learners tend to make meaningful interpretations of the differences between the accumulation durations of each imperceptible object, we expected that the radical differences between the total durations of accumulation may direct them to form even more distinguished conceptual categories of imperceptible sizes. Hence, we hypothesized that the students who interact with the TAVRs with the slower accumulation rapidity (extended, 1 object/sec) would experience more discrete and distinguished imperceptible sizes and therefore construct more refined imperceptible size categories (i.e., more number of groups and elaborated labels) than the students with the TAVRs with faster accumulation rapidity (compressed, 10 objects/sec). The result of the study would let us discuss the potential design suggestions for a temporal representation for an adjusted concept such as abstract spatial information that requires learners to construct fine grained conceptual categories of it.
Methods
Fifty seven 7th grade students from a public middle school in a Table 2: The composition of the mid-western state in the United States were recruited for this study participants in this study. through their science teacher. Students were divided into two groups, Groups Male Female Total by their class hour. See Table 2 for the composition of the participants 1/sec in each group. One of the researchers went into each groups science 16 13 29 (Extended) class and instructed them for one class hour (50 minutes) and collected 10/sec the data with the help of their science teacher. Every student was 15 13 28 (Compressed) assigned to an individual computer in a computer lab. Students were not allowed to go back to previous phases of the learning activity to revise their answers. They were told to move onto the next page of the learning activity all together at the same time with an instruction. Before beginning to work Phase 3, the instructor made sure that the students understood the inverse relationship between the accumulation time and size. The students were provided with the objects in Figure 3 for this study. They were allowed to discuss about what the tasks are about and how TAVRs (and WIIS) work with peers, but they were asked to create hypotheses on their own. Each student was provided with a real pin for easier visualization of the size of the head of a pin. During the instruction students were continuously reminded of the fact that the accumulation is happening on the tiny little pin head in front of them. The goal of this study is not to compare the numeric increase in student performance to judge which group scored higher; rather, it is to see how students change their hypotheses through the learning activity with TAVRs that provide different temporal experiences (extended vs. compressed). To meet this goal we had the students set up hypotheses for the range of the imperceptible sizes and later compare it with the result. Then they were asked to reflect on the difference between their hypotheses (their preexisting knowledge) and the result. The data that show the changes in the students mental models of the range of imperceptible sizes would let us investigate what kinds of different learning impact the different accumulation conditions have on students. Student learning activity for was composed of five phases. First, students represented their preexisting knowledge in card sorting task. The card sorting task has three sub-tasks; (1) ordering the given imperceptible objects from the smallest to the biggest, (2) grouping the objects by similar sizes (The number of groups could vary from 1 to 6), and (3) labeling the groups. For the labeling task the students were provided with prompts (a set of adjective - small, tiny, mini, teeny, and a set of adverbs - extremely, very, strikingly, surprisingly). Students could also use any additional vocabulary they can come up with. In the second phase students were introduced to mini learning activities that showed how TAVR works and what it represents. They solved a quiz (on the computer screen) that aimed to examine whether they understood the mechanism of TAVR correctly. In the third phase, the students represented their existing idea of the sizes of the imperceptible objects in the durations of accumulation time. For this task, a TAVR of the thickness of human hair was shown to the students to provide them with an idea of how long it takes for a small macroscopic object to accumulate (hair takes 1 second for the compressed condition, 10 seconds for the extended condition). In fourth phase, the students finally played and interacted with the TAVRs for all objects. During this time they were told to revise their hypotheses, if they wanted to, observing the TAVRs. At the final phase, students reflected on the differences between the actual accumulation time and their hypotheses on seven point bipolar Likert scale that ranged between extremely smaller than I thought and extremely bigger than I thought centered by similar with what I thought after the learning activity. The card sorting task was adopted as a main student task because the conceptual size categories play a critical role in making sense of what learners experience (Chi & Roscoe, 2002; Lakoff, 1987) and constructing the mental model of sizes (Kosslyn, 1980; Tretter, et al., 2006a). Having more conceptual sizes categories means more refined mental model of the range of sizes and being able to make shifts in conceptual categories (which is the process of repairing misconceptions). Hence, the increase in the number of groups and the
ISLS
423
ICLS2012
descriptive words in the labels would imply the positive effects of TAVR (in either compressed or extended temporal condition) on the students mental model. Data were collected at phase 1, 3, 4, and 5. The student activities in their computers were automatically transferred and stored in a database. Observation notes were also recorded. The student performance on the card ordering task was analyzed by a coding scheme that scored between 0 and 5. For the grouping task, we counted the number of groups that the students created in phase 4, then we compared it with what the students did in phase 1 to see how much they increased. To analyze the labeling task, we counted the number of adverbs and adjectives in the labels in phase 4 and also compared with the data from phase 1. Data were quantitatively analyzed with one-way analysis of variance (ANOVA) to test for the statistical significance between two groups.
Results and Discussions

In phase 1 and 3, where the students represent their preexisting knowledge, the students in both groups performed poorly and did not show any significant group difference (p>0.05). All students overestimated the sizes of the imperceptible objects and underestimated the range of imperceptible sizes (i.e., number of groups less than 2, minimal use of descriptive words in labels). In phase 4 and 5, the students in both groups had increases in their card sorting task. This is a natural result since students were allowed to refer to the TAVRs in their computer screens to sort the cards. The students in both groups scored perfectly at the ordering task, however, statistically significant differences were found for the grouping task and labeling task. As we predicted, the students in the extended group outperformed the compressed group.
Ordering by size
In Phase 4, all participating students in both groups correctly ordered (scored 5 points) the objects by size. The students actively referred to the total duration of accumulation shown in TAVRs in both compressed and extended conditions when they were asked to think about the order of the sizes of the imperceptible objects. The accumulation rapidity did not make difference in the ways the students perceived and conceptualized the order of different imperceptible sizes because they could complete this task only by looking at the total accumulation durations once they understood the inverse relationship between the duration and the represented size.
Grouping by similar size

In this task the students exhibited different patterns in their Table 3: The average number of the size answers according to the treatment group they belonged to. As groups that the students created in phase 4. Table 3 shows, the students in the extended group have created Group Mean SD min max a more number of groups (M=5.1, SD=0.85); (F(1, 55) = 1 / sec 29.73, p < 0.001) than the students in the compressed group 5.1 0.85 4 6 (Extended) (M=3.9, SD=0.81) as we hypothesized. This implies that the 10 / sec difference between the accumulation times of the objects 3.9 0.81 2 5 (Compressed) seemed to be relatively bigger to the students in the extended group than to the students in the compressed group. We also examined the difference between the phase 1 and 4 to examine if the increases in the number of size groups were also significantly different between two groups. The increase in the number of size groups was calculated by subtracting the number of the student-generated size groups in Table 4: The average increases in the number Phase 1 from that in Phase 4 (see Table 4 for the brief of the the size groups. summary of the result). We noticed that none of the students in Group Mean SD min max both groups had a decreased number of the size groups in 1 / sec phase 4 although few of them had zero increase. The students 2.9 0.95 1 4 (Extended) in the extended groups had more increases that were 10 / sec statistically significant (M=2.9, SD=0.95); (F(1, 55) = 24.08, p 1.6 0.97 0 4 (Compressed) < 0.001) than the students in the compressed group (M=1.6, SD=0.97). Taken together, after the learning activity with TAVRs, the students in the extended group had created more refined classifications of the sizes than the students in the compressed group.
Labeling the groups
Our hypothesis for the labeling task was that the extended Table 5: The average number of the descriptive group would outperform the compressed group in Phase 4. words used for the smallest size group. Accordingly, the students in the extended group generated Group Mean SD min max significantly richer descriptions for each object group they 1 / sec created than the compressed group (see Table 5), and the 6.6 1.66 5 11 (Extended) descriptions for the smallest size group were even richer; all of 10 / sec the students in the extended group used more adjectives, 5.6 1.33 4 9 (Compressed) adverbs to name the smallest size group in phase 4
ISLS
424
ICLS2012
(M=6.6, SD=1.66); (F(1, 55) = 6.424, p = 0.014) than the compressed group (M=5.6, SD=1.33). Additionally, out of our expectations, some students even voluntarily used a number of exclamation marks or capital letters to emphasize the smallness of the size of the smallest group. For example, they labeled the smallest group such as, This is the smallest of the smallest x 1000000000!!!!!!, or the really surprisingly teeny-weeny smallest!!! while the students in the compressed group were likely to use a less number of adjectives and exclamation marks in the labels compared to the extended group. Although Table 6: The average increases in the number the students in the compressed group did actively use the of the the descriptive words for the smallest descriptive words to label their groups, the students in the size group. extended group used significantly more descriptive words than Group Mean SD min max the compressed group. As shown in Table 6, the increases in 1 / sec 4.43 1.81 2 8 the number of the descriptive words for the smallest size group (Extended) from Phase 1 to Phase 4 also was also significantly higher for 10 / sec 3.51 1.22 1 6 the extended group (M=4.43, SD=1.81); (F(1, 55) = 5.027, p = (Compressed) 0.029) than the compressed group (M=3.51, SD=1.22). This result implies that the extended form of temporal experience not only supports learners to construct a vaster range of imperceptible sizes but also guides them to form more refined and distinctively different conceptual categories for imperceptible sizes than the compressed temporal experience.
Reflections
The students hypotheses on the accumulation time of each object were similar between both groups. The students predictions of the accumulation time for the smallest object (according to their hypotheses) were below ten minutes. Interestingly, even the students in the extended group, who used the TAVRs with slower accumulation rapidity, made predictions that were under ten minutes. We think this is natural because mentally calculating the accumulation time without the knowledge of the sizes of each imperceptible object is beyond their capability, and ten minutes is the long enough duration for students to sit in front of a computer and watch it. In general the students in both groups made more extreme response for the object that their prediction had bigger difference with the actual accumulation time; the bigger the difference, the more extreme their response. For example, a student who predicted that it would take less than one hour (it takes about 11 days and 13 hours for Hydrogen atom in the compressed condition) answered that the size of an atom was extremely smaller than I thought in the bipolar Likert scale. The same student responded that the size of a red blood cell was similar with what he thought because his prediction was 10 seconds (it takes about 16 seconds for a red blood cell). However, we did not find a statistically significant difference between the groups for all objects. Both groups showed a similar pattern of reflections (i.e., most of the students in both groups marked on extremely smaller than I thought for Hydrogen atom and somewhat smaller than I thought for red blood cell). Considering the obviously different student performances in the grouping and labeling tasks, we believe this result is due to the limits in the test method that provided only seven points in the bipolar Likert scale, which only allows three degrees for smaller than I thought extremely, much, and somewhat. In fact many students in the extended condition group explicitly asked for more options that would allow them to mark on a smaller scale than extremely smaller than I thought.
Implications and Concluding Remarks

In this paper we have presented a multimodal representation that incorporates a temporal modality to support learners to explore the range of imperceptible sizes in two different temporal conditions; extended and compressed. The result showed that the extended temporal experience was consistently more effective for supporting learners to perform the object grouping and labeling tasks than the compressed temporal experience. It was not because the compressed temporal experience was not effective, but because the extended temporal experience provided more refined analogical size categories for the learners with wider variety of time units. It also seems that the students with slower accumulation rapidity tend to give more meanings to their temporal experience than those with faster accumulation rapidity. The students knew that 23 days is close to one month and 115 days is almost four months. They actively converted the days into a more convenient unit of time and attached meanings to them. For example, one student said, For 115 days, I height will grow 2 inches. Another student said, Wow, think about how many homework I should do for 23 days. In addition to interpreting the temporal experience within the context of their daily lives, the students in the extended group were more critically influenced by the labor-intensive FF button clicking. To see the accumulation completed, for an example, for a Hydrogen atom, students had to vigorously click the FF button for at least 10 minutes. This observation reminds us of Baddeley (1986)s discussion on human working memory model that the tactile and kinesthetic senses take in information and help learners better understand the information. Overall, the results imply that the extended form of temporal experience can guide learners to form a mental model of vaster range of imperceptible sizes than the compressed temporal experience of imperceptible sizes. The temporal experience
ISLS
425
ICLS2012
can be used to generate the fine grained categorizations of an abstract spatial concept by using different rapidity of the sequential accumulation.
References
Ainsworth, S. (1999). The functions of multiple representations. [Article]. Computers & Education, 33(2-3), 131-152. Baddeley, A. (1986). Working memory. New York, NY US: Clarendon Press/Oxford University Press. Boroditsky, L. (1997, Aug 07-10). Evidence for metaphoric representation: Perspective in space and time. Paper presented at the 19th Annual Conference of the Cognitive-Science-Society, Stanford, CA. Boroditsky, L., & Ramscar, M. (2002). The roles of body and mind in abstract thought. [Article]. Psychological Science, 13(2), 185-189. Casasanto, D., & Boroditsky, L. (Writers). (2008). Time in the mind: Using space to think about time [doi: DOI: 10.1016/j.cognition.2007.03.004]. Chi, M. T. H. (2005). Commonsense Conceptions of Emergent Processes: Why Some Misconceptions Are Robust. Journal of the Learning Sciences, 14(2), 161-199. doi: 10.1207/s15327809jls1402_1 Chi, M. T. H., & Roscoe, R. D. (2002). The Processes and Challenges of Conceptual Change In M. L. a. L. Mason (Ed.), Reconsidering Conceptual Change: Issues in Theory and Practice (pp. 3-27). Dordrecht, Netherlands: Springer Netherlands. Droit-Volet, S. (2001). Temporal Bisection in Children. Journal of experimental child psychology, 80(2), 142159. Hutchins, E. (1995). Cognition in the wild. Cambridge, Mass.: MIT Press. Kosslyn, S. M. (1980). Image and mind. Cambridge, Mass.: Harvard University Press. Kozma, R., Chin, E., Russell, J., & Marx, N. (2000). The Roles of Representations and Tools in the Chemistry Laboratory and Their Implications for Chemistry Learning. [Article]. Journal of the Learning Sciences, 9(2), 105-143. Lakoff, G. (1987). Women, fire, and dangerous things : what categories reveal about the mind. Chicago: University of Chicago Press. Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago: University of Chicago Press. McCormack, T., & Hoerl, C. (2008). Temporal Decentering and the Development of Temporal Concepts. Language Learning, 58(s1), 89-113. Paivio, A. (1986). Mental representations : a dual coding approach. Oxford psychology series no. 9 Pea, R. D. (1993). Practices of distributed intelligence and designs for education. In G. Salomon (Ed.), Distributed cognitions (pp. 47-87). New York: Cambridge University Press. Quintana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G., et al. (2004). A Scaffolding Design Framework for Software to Support Science Inquiry. Journal of the Learning Sciences, 13(3), 337 - 386. Rapp, D. N. (2005). Mental Models: Theoretical Issues for Visualizations in Science Education. In J. K. Gilbert (Ed.), Visualization in Science Education (Vol. 1). Dordrecht: Springer. Salomon, G., Perkins, D. N., & Globerson, T. (1991). Partners in Cognition: Extending Human Intelligence with Intelligent Technologies. Educational Researcher, 20(3), 2-9. doi: 10.3102/0013189x020003002 Song, M., & Quintana, C. (2009). WIIS: multimodal simulation for exploring the world beyond visual sense. Paper presented at the Proceedings of the 27th international conference extended abstracts on Human factors in computing systems, Boston, MA, USA. Tretter, T. R., Jones, M. G., Andre, T., Negishi, A., & Minogue, J. (2006a). Conceptual boundaries and distances: Students' and experts' concepts of the scale of scientific phenomena. [Article]. Journal of Research in Science Teaching, 43(3), 282-319. doi: 10.1002/tea.20123 Tretter, T. R., Jones, M. G., & Minogue, J. (2006b). Accuracy of Scale Conceptions in Science: Mental Maneuverings across Many Orders of Spatial Magnitude. Journal of Research in Science Teaching, 43(10), 1061-1085. Vallesi, A., Binns, M. A., & Shallice, T. (2008). An effect of spatial-temporal association of response codes: Understanding the cognitive representations of time. [doi: DOI: 10.1016/j.cognition.2007.10.011]. Cognition, 107(2), 501-527. Zhang, J., & Norman, D. (1994). The Representation of Relational Information. Proceedings of the Sixtheenth Annual Conference of the Cognitive Science Society, 952-957.
Acknowledgement
This material is based on work supported by the National Science Foundation under Grant No. ESI-0426328. Any opinions and findings expressed in this material are those of the authors and do not necessarily reflect those of the National Science Foundation.
ISLS
426
ICLS2012
Inter-Identity Technologies for Learning

Robb Lindgren, University of Central Florida, 12461 Research Parkway Suite 500, Orlando, FL 32826-3241, USA, robb.lindgren@ucf.edu Roy Pea, Stanford University, 485 Lasuen Mall, Stanford, CA 94305-3096, USA, roypea@stanford.edu Abstract: We propose a new framework for conceptualizing media technologies that blur the distinction between the self and the other. We refer to these as inter-identity technologies (IITs) and we describe both theoretical and empirical support for these technologies as having special properties for generating learning. Several examples of IITs are offered such as video games where a first-person perspective is augmented with data visualizations, or digital characters to which a learners attributes can be imparted and their activity subsequently observed. We review several studies that have been conducted on digital environments we classify as IITs and discuss their impact on various types of learning. In an effort to forge a coherent research agenda, we propose three lines of inquiry pertaining to inter-identity and the design of digital media.
Introduction
In keeping with the ICLS 2012 theme of The Future of Learning, we introduce a new and generative framework for conceptualizing developments in new media technologies for learning and education. While we lay out our argument below for why we have chosen to articulate these opportunities in terms of the phrase inter-identity technologies for learning, we foreshadow our reasoning with two key points. The first point is the deepening recognition of the profoundly social nature of learning, and the integral relationship of learning to the development of personal identity (Gee, 2000; Hull & Greeno, 2006; Lave & Wenger, 1991), because selfconceptions of who one is, who one may become, and who and with what domains and tasks one affiliates may open up or close down learning opportunities (Nasir, 2011; Wortham, 2006). The second is that information and communication technologies do not simply serve as amplifiers of cognition and social relations, but restructure them in new functional and activity systems (Cole & Griffin, 1980; Engestrom, 1999; Pea, 1985, 2004). If we take seriously the social construction of identity as integral to learning, and the integral nature of technologies in the functional systems of intelligence in which people act, we should work to deeply mine the new opportunities for advancing the future of socially-mediated learning. We begin that quest in this paper by turning our attention to the pedagogical prospects wrought by bringing together the affordances of new media technologies and the inter-relating of the identities of the learner and others, real and virtual.
Inter-Identity: New Possibilities of an Increasingly Digital Life

Sherry Turkle has highlighted how "we come to see ourselves differently as we catch sight of our images in the mirror of the machine" (Turkle, 1999, p. 643; Turkle, 1995). Her work has highlighted how individuals negotiate what she calls the virtual and the real when people represent themselves on the computer screens available to others through the Internet. Through such identity presentations, people can experience how others respond to them in the identity or identities that they are formulating and expressing. Her ethnographic studies of the performative nature of identity work online lead Turkle to place a primary emphasis on the multiplicity and flexibility of a person's identity, more fluid and multi-faceted than many identity theorists from the social sciences have previously advocated. Over the past several decades, a voluminous literature has been developed on how interacting by means of textual and avatar representations of the self in virtual worlds or in multi-player games provides fertile occasions for identity play, and for the social construction of identity, that are uniquely contributed by means of technology mediation rather than in face to face interaction (Calvert, 2002; Huffaker & Calvert, 2006; Ito, Baumer, Bittanti, Boyd, Cody, Herr-Stphenson, et al., 2009; Kendall, 1998; Thomas, 2007; Turkle, 1995; Turkle, 1999; Valkenburg, Schouten, & Jochen, 2005). Among the many aspects of identity experimentation that have been emphasized in these works on identity online are gender, race, class, ethnicity, culture, age, religion, profession, power, and orientation. While such sociological, psychological, anthropological, and cultural studies are important and defining in significant ways for our era, our concerns are more focused on the powerful potentials of the representational and interactive affordances of information and communication technologies for developing new frameworks for pedagogy and for learning - not only about the self and others in social relations, but about the multiplicity of aspects of expertise, in terms of conceptual, spatial and procedural knowledge, involved in performing complex actions in the communities of practice represented in the activities of school, work, play, and society more broadly. Before we define the new framework opportunities, consider the face-to-face communicative infrastructure for identity development in a pre-technological era. Either observed or reported
ISLS
427
ICLS2012
human action and associated characteristics of personal identity served as the available cultural resource for identity development. Children had occasions to observe different models in their accessible social groups and to emulate aspects of the behaviors, beliefs, orientations of those individuals. Enactments of aspects of ones developing personal identity could then be subject to social feedback and regulation, with extreme behaviors subject to stigma, isolation or exclusion from social groups. The powerful human technology of narrative storytelling opened up the possibilities of identity models from by-gone eras, even before texts made their permanent inscription in a fixed form possible. In a pre-technological era, identity expressions among adults and identity enactments among youth and feedback from others were transient in experience, unrepresented in an inspectable medium. Today, beyond the stories and myths wrought in tale and text, video, film and computer games all serve as representational media of possible identities and subject of attention for youth and adults who are always in process of developing and elaborating their personal identities. These technologies have become more powerful and intimate as they have become interactive technologies among people, not only for watching, as in television and film. What special opportunities does this representational ecology offer for advancing the learning of complex competencies? While we cannot in this short paper provide an account of landmarks in the history of film, television, and computing that brought about innovations in the experiential possibilities of media as resources for identity development, salient examples from popular media of the power of the technical affordances of video and computer representations include: (1) Point-of-view films, such as the Blair Witch Project, in which first-person camera techniques are used to create the sensation for moviegoers that they are present in the scene that is captured on film, with greater reality-quotient to the events, dangers, risks, uncertainties that are present in the narrative line of the movie. This genre of film-making is often described as found footage, and is most typically used for horror movies. (2) First-person gaming perspective. Video games where the user can select a first person rather than a third person perspective to experience the gameplay in a more immersive manner. First-person shooters are perhaps best known for uses of this graphical perspective, but flight simulators and adventure games are well-known exemplars as well. (3) Virtual worlds co-populated with graphical avatars representing humans and agents representing computer programs. In a variety of multi-user virtual environmentsfrom the text-based MUD (multi-user dungeon), LambdaMoo to todays Second Life and other virtual worldsand massively multiplayer role playing games, human players encounter not only other players but embodied agents in the game that are running computer programs that generate their behaviors and communications to the user community. In developing expertise in game play, interactions with both avatars and agents contribute to learning, but the usual framing of agents as contributors to players learning is by analogy to the role of a face-to-face human teacher or a guide who offers advice or impacts knowledge, wisdom, or clues. New media technologies like immersive online games, interactive video and virtual worlds enable players and game designers to shift the point of view from which people play or watch, blurring the participants perspective with that of another. We refer to technologies that blur the boundaries between self and other as inter-identity technologies (IITs). We argue in this paper that the learning-relevant properties of these new media infrastructures for IITs have been underutilized for fostering learning and expertise development. We see enormous opportunity in the design, development, and study of IITs for learning and teaching. Inter-identity refers to the co-mingling of a learners identity with a virtual character/avatar that can produce psychological phenomena such as heightened arousal, attention to key features, engagement, memory, learning, performance and a sense of responsibility. The inter-identity technological system might, for example: (1) enable the learner to see through the avatars point of view things that the learner on his or her own might not notice, leading to more expert-like attention and behavior, or (2) enable the learner to teach a computer agent to do things which the learner then takes responsibility for in the performance of the agent, or (3) occasionally take over control of the learners avatar in a virtual environment so that it acts more expertly an intervention we allude to as scaffolding on steroids.
The Power of Blending: Theoretical Support for Using Inter-Identity Technologies for Learning
Emerging media technologies make it possible to create immersive experiences where the resulting activity is a blending of the knowledge and behavior of multiple individuals. The notion that this blending can be leveraged for powerful effect in learning environments has support from theoretical work in educational psychology, neuroscience, and human-computer interaction.
ISLS 428
ICLS2012
The Social Foundations of Learning

IITs present a unique mode of social interaction in that there is typically an absence of face-to-face or direct conversational exchange with others. There is, however, an implicit awareness of others presence and the influence they bring on the activity at hand. We argue that this awareness provokes the mechanisms of learning that stem from our makeup as social creaturesworking together to solve problems and using each other as models and guides towards our own inquiry and understanding. Recent research on learning has shown the extent to which our social nature pervades our thinking and learning processes. From an early age children use subtle cues from parents and others that shape not only the content of their learning but the focus of their learning pursuits (Meltzoff, Kuhl, Movellan, & Sejnowski, 2010). Studies have demonstrated the critical importance of these cues originating from human actors (or at least the belief as such) compared to inanimate or robotic sources (Meltzoff, 2007; Okita, Bailenson, & Schwartz, 2007). Unlike other forms of computermediated interaction, IITs actively evoke social cues, albeit in sometimes covert and subtle ways that integrate themselves with the individuals own perceptual system. In addition to providing a raw social stimulus to activate basic mechanisms of learning, IITs can also effectively situate an individual within the authentic and complex social contexts where learning typically occurs. Theories of situated and distributed learning maintain that the activity structures that emerge from authentic social practices and the tools available within these environments should themselves be viewed as cognitive processes that support learning (Brown, Collins, & Duguid, 1988; Lave & Wenger, 1991; Pea, 1993; Scardamalia & Bereiter, 1994). Fueled by this perspective many of the new designs and reforms in education involve making traditionally independent learning activities more social and collaborative, and frequently this has produced positive results. For example, a meta-analysis of computer-based activities in schools showed that students tended to learn more when these activities were performed in groups (Lou, Abrami, & dApollonia, 2001). Of course, the quality of the interaction in a collaborative activity has much to say about what is ultimately learned (Barron, 2003). With IITs it is possible for the characteristics of a productive social interaction to be designed into experience (e.g., constraints that ensure that the correct ideas and interpretations remain the focus). A good example of digital media technologies situating an individual within a complex social environment are video games. Gee (2003) describes the unique ability of these gameswhether their focus is combat, adventure, solving puzzles, etc.to embed someone within an authentic environment and equip them with the tools and knowledge to perform optimally. These games make it possible for players to assume (partially) the identity of a competent other, creating the conditions for new learning.
Overlapping Neurological Systems for Self and Other

The effectiveness of technology designs that blend the experiences of one person with those of another relies on our ability to effectively integrate our own perceptions with information we receive from other sources. Recent research in perceptual psychology suggests that people instinctively incorporate visual perspectives of others when making visual judgments, and that we are capable of doing so quite rapidly (Samson, Apperly, Braithwaite, Andrews, & Bodley Scott, 2010). Amongst neuroscientists it has been proposed that humans (and animals) have a system of mirror neurons in the motor cortex that are activated both when performing an action and observing those actions as performed by others (Rizzolatti & Craighero, 2004). The implication is that there are strong perception-action connections built into our brains that can facilitate their integration into a coherent experience (e.g., I see actions being performed as my own even if they are partially controlled by others). A particularly relevant study in neuroscience showed that while there was activation in the motor cortex by participants when observing an actors video recorded movement (hand and foot motion), this activation was greatest when viewing these actions from the actors viewpoint (i.e., as if the actions were their own) (Jackson, Meltzoff, & Decety, 2006). Findings such as these suggest that people are capable of naturally assimilating sensory experience transmitted via digital media.
Opportunities for Perspective-Taking

There has been a great deal of research in psychology demonstrating broad cognitive benefits for perspectivetaking interventions on learning (e.g., Anderson, Pichert, & Shirey, 1983; Siegler, 1995), and in education the positive effects of giving students new perspectives on content have been shown in history (Davis, Yeager & Foster, 2001), science (Linn, Bell, & Hsi, 1998), and mathematics (Resnick & Wilensky, 1998). In most cases perspective-taking interventions involve the mental simulation of another persons point of view, but with IITs it is possible to give a person novel perspectives in a much more literal way, by co-opting their perceptual system and allowing them to experience aspects of the world the way someone else experiences them. Perspective-taking with digital media can conceivably be accomplished in numerous ways, but studies using a diverse set of approaches and technologies have shown learning benefits akin to those shown for non-computermediated perspective-taking. For example, Grant and Spivey (2003) improved participant performance using eye tracking technology to digitally highlight features of a problem solving task that had proven to be critical to the success of previous participants. Likewise, it was shown that being given multiple perspectives in a virtual
ISLS
429
ICLS2012
environment (both egocentric and exocentric) improved learning related to a design task (Salzman, Dede, & Loftin, 1999). We are particularly excited about the possibility of using IITs to convey expert perspectives simulations of how experts in some domain interact with artifacts and representations (e.g., what features do they look at to make critical decisions). The hope is to create a process analogous to the way experts have been shown to convey field knowledge to novices in domains such as archeology (e.g., highlighting important perceptual contrasts: Goodwin, 1994).
Demonstrations of Inter-Identity Technologies Impacting Learning

As discussed above, several digital media designs exist that could be considered IITseverything from firstperson films to certain kinds of video gamesbut there have been relatively few studies focused on the features of these technologies that facilitate identity blending and the impact that they have on learning. We review a few examples here that we consider to be good examples of designs that attempt technology-mediated identity development targeted at enhancing learning, but by no means do we consider this list exhaustive, and we invite others to offer up their ideas and designs for what constitutes inter-identity.
First-Person Expert Perspectives in Digital Video

Giving viewers of video a first-person perspective (e.g., handheld camera recording) of events has been used for good effect in generating engagement and a feelings of gritty realism in films, but can these same conditions of video be used to promote learning? And if so, does it matter who is holding the camera? These were the questions asked in a set of studies that examined peoples ability to learn from an instructional video about how a toaster works. An initial study compared learning when the video was shot from the first-person perspective of the expert (head-mounted camera) compared to a third-person (tripod-mounted camera) recording of the same event (Figure 1). Results showed that participants in the first-person condition actually learned more based on their responses to a set of inference questions administered after watching the video (Lindgren, Pea, & Lewis, 2008). A follow-up study attempted to isolate the features of the video that produced the learning effect (e.g., motion, perspective), and results indicated that learning and engagement was highest when the motion in the video was consistent with the actions and utterances of the expert (Lewis, Lindgren, & Pea, under review). The suggestion is that learning from digital video can be enhanced when viewers are given the opportunity to view events as they are seen by an expert, and perhaps even feeling what it is like to be an expert in that domain. These studies highlight that a media technology as accessible and as ubiquitous as digital video can be leveraged as an IIT when the appropriate design considerations are applied.
Figure 1. Instructional video shot as an IIT showing events from the experts perspective (left) and the same event shot from the perspective of a passive observer (no camera movement) (right).
Imparting Knowledge to Computer Agents

A second paradigm of IIT involves the transfer of personal characteristics to a computer agent and then monitoring the behavior of that agent as an outside observer. This could involve, for example, creating a character in a video game that looks like you and watching to see how people respond to that character. More pertinent to the topic of IITs for learning, however, is the notion of a computer character to whom one can impart their knowledge or their reasoning about some topic and then observe how that character fairs when confronted with challenges in that domain. It is a process that can be seen as analogous to teaching a student and then watching that student perform on a test or execute some other demonstration of what theyve learned, and it is the basis of the Teachable Agents paradigm of computer software being built at Vanderbilt and Stanford University (Biswas, Schwartz, Leelawong, Vye, & TAG-V, 2005). Recent research on Teachable Agents in the classroom have shown that the process of instructing computer agents can have lasting benefits for future learning experiences without the agents (Chin, Dohmen, Cheng, Oppezzo, Chase, & Schwartz, 2010) and that
ISLS 430
ICLS2012
students will actually work harder to learn for their agents than they will to learn for themselves (Chase, Chin, Oppezzo, & Schwartz, 2009). Unlike other IITs where the technology allows the user to bring characteristics of others identities into their own, Teachable Agents provide an opportunity to transfer part of ones identity (i.e., their understanding of a domain) to an external entity that can be observed and evaluated.
Expert Perspectives in Virtual Environments

Just as the perspective of a knowledgeable other can be built into digital video, so too can expert perspectives be built into the interactions users have with screen-based virtual environments such as video games and social virtual worlds. Lindgren (in press) showed that participants who experienced an experts actions from the firstperson perspective in a virtual world training simulation performed better and learned more than participants who viewed the same actions from a fixed virtual camera in the simulation space. When allowed to perform the simulated actions independently, participants who were trained through the eyes of an expert committed fewer errors and exhibited less help-seeking behavior. Outside of the simulation participants with the expert view could recall more of the important simulation elements and drew more accurate diagrams of the virtual environment. These IIT effects within virtual environments can likely be integrated directly into the simulation activity itself. For example, Lindgren (2009) describes a version of the training simulation that is fully under the control of the user until they made a critical error, at which point the simulation takes over (Figure 2) and shows the user what it would be like to perform the action correctly, followed by an opportunity to perform the correct actions on their own. This is the type of intervention we refer to as scaffolding on steroids. Similar approaches for mixing in expert identities in real time have already been utilized in commercial video games. The game Batman Arkham Asylum, for example, has a detective mode where important objects and areas of interest are highlighted to encourage more productive inquiries into the virtual environment.
Figure 2. The eye icon in the upper left indicates that the simulation has temporarily taken control away from the user to highlight the correct actions in the first-person perspective.
Altered Identity and Behavior in Virtual Reality

As a final example of IIT we describe the capacity of fully immersive virtual environments such as gogglebased VR to convey realistic but altered depictions of experience in real time. Jeremy Bailenson has created and studied numerous of these experiences in a paradigm referred to as Transformed Social Interactions (TSIs) (Bailenson, Beall, Loomis, Blascovich, & Turk, 2004). TSIs involve altering the way that actions in a virtual environment are rendered, such as augmenting non-verbal behavior or gaze enhancement where a single actor can have direct eye contact with multiple people at the same time (i.e., all participants in the environment receive a rendering of the environment where the speaker is looking at them). Bailenson and colleagues have demonstrated that TSIs enacted in simulated learning environments such as classrooms can have positive consequences on learning and attention (Bailenson, Yee, Blascovich, Beall, Lundblad, & Jin, 2009). In this case, inter-identity is achieved by merging an individuals real time behavior with the strategic alterations of a system designed to optimize the outcome of that behavior. Modifying the way that we see others or ourselves in the course of activity can change what and how we learn.
Solidifying an IIT Research Agenda

Given strong theoretical backing and the few provocative exemplars described above, we believe that interidentity technologies presents itself as a fascinating and important research strand for developing the future of learning. In order to move beyond a few interesting studies, however, it will be necessary for future research to address head-on some important questions elicited by the notion of inter-identity. We outline what we see as the three central questions here.
ISLS
431
ICLS2012
(1) In what ways can identity be convincingly and practically blended? The examples above suggest a few ways in which the identities of learners, experts, designers, etc. can be comingled into a coherent interactive experience, but there are certainly other ways that this can be achieved and should be explored. One possible dimension that could be used to define these methods of integration is whether the characteristics of new identities are brought within, such as watching video recorded from a head camera, or whether they are imparted upon external entities, as is done with Teachable Agents. Perhaps more importantly, this line of research will require new instruments sensitive to whether learners feel that a transaction involving identity has occurred. Similar to instruments that have been developed to detect abstract constructs such as presence (e.g., Witmer & Singer, 1999), we believe that it would be useful to develop tools for gauging the degree of experienced interidentity. (2) What are characteristics of existing and future technologies required to support inter-identity interventions? The demonstrated utility of IITs described in this paper suggests that identity blending is a focus that should be taken up not just by psychologists and philosophers, but by computer scientists and developers. Creating an authentic and fluid experience that shares attributes of the self and otherpotentially in real timeis a non trivial task that can require expertise in graphics, AI, haptics, computer vision, and HCI. In fact, we believe that IITs constitute a new frontier in user experience design that could drive technology innovation. It is our hope that these advances are accompanied by rigorous study to determine the mechanisms by which inter-identity is achieved. (3) What are the specific learning affordances of the different types of identity blending? We readily acknowledge that all types of IITs are likely not optimal for all types of learning. For example, it is hard to imagine (though not impossible) how first-person digital video could be used to teach organic chemistry. Thus in order to aid in the design of IITs it is imperative that we begin to map their characteristics to the ability to support specific types of learning. Ideally the bulk of IIT research would reside here as these studies would further both our understanding of inter-identity effects on cognition as well as to develop practical guidelines for developing effective educational platforms.
Conclusion
We have introduced a new framework for understanding and designing learning technologies that we hope will be instrumental to shaping the future of learning. Inter-identity technologies are a spectrum of digital media designs that in some way blend the experience, knowledge, and beliefs of multiple individuals. We have presented both theoretical and empirical support for the efficacy of IITs for generating learning, and we have suggested new lines of inquiry to fuel a promising research agenda. Identity is a highly complex and abstract construct, but emerging media technologies are making it possible to explore identity in concrete and productive ways. The idea that simple manipulations using IITs could lead one to be more expert-like in their learning pursuits or to view themselves as being more capable of tackling difficult challenges is exciting, and we believe it is an area of research worthy of active investigation.
References
Anderson, R. C., Pichert, J. W., & Shirey, L. L. (1983). Effects of the readers schema at different points in time. Journal of Educational Psychology, 75(2), 271-279. Bailenson, J., Beall, A., Loomis, J., Blascovich, J., & Turk, M. (2004). Transformed social interaction: Decoupling representation from behavior and form in collaborative virtual environments. Presence, 13(4), 428-441. Bailenson, J. N., Yee, N., Blascovich, J., Beall, A. C., Lundblad, N., & Jin, M. (2009). The use of immersive virtual reality in the learning sciences: Digital transformations of teachers, students, and social context. The Journal of the Learning Sciences, 17, 102-141. Barron, B .J. (2003). When smart groups fail. Journal of the Learning Sciences, 12, 307-359. Biswas, Schwartz, Leelawong, Vye, & TAG-V (2005). Learning by teaching: A new agent paradigm for educational software. Applied Artificial Intelligence, 19, 363-392. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Education Researcher, 18(1), 32-42. Calvert, S. L. (2002). Identity construction on the internet. In S. L.Calvert, A. B. Jordan & R. R. Cocking (Eds.), Children in the Digital Age: Influences of Electronic Media on Development (pp. 5770). Westport , CT : Praeger. Chase, C. C., Chin, D. B., Oppezzo, M. A., & Schwartz, D. L. (2009). Teachable agents and the protg effect: Increasing the effort towards learning. Journal of Science Education and Technology, 18(4), 334-352.
ISLS
432
ICLS2012
Chin, D. B., Dohmen, I. D., Cheng, B. H., Oppezzo, M. A., Chase, C. C., Schwartz, D. L., (2010). Preparing students for future learning with teachable agents. Education Tech Research Dev., 58, 649669. Cole, M., & Griffin, P. (1980). Cultural amplifiers reconsidered. In D. R. Olson (Ed.), The social foundations of language and thought: Essays in honor of Jerome S. Bruner (pp. 343-364). New York: Norton. Davis, O. L., Yeager, E. A., & Foster, S. J. (2001). Historical empathy and perspective taking in the social studies. Lanham, MD: Rowman & Littlefield. Engestrom, Y. (1999). Activity theory and individual and social transformation. In Y. Engestrom, R. Miettinen & R-L Punamaki (Eds.), Perspectives on activity theory (pp. 19-38). New York: Cambridge University Press. Gee, J. P. (2000). Identity as an analytic lens for research in education. Review of Research in Education, 25, 99-125. Gee, J. P. (2003). What video games have to teach us about learning. New York: Palgrave. Goodwin, C. (1994). Professional vision. American Anthropologist. 96(3), 606633. Grant, E. R., & Spivey, M. J. (2003). Eye movements and problem solving. Guiding attention guides thought. Psychological Science, 14, 462466. Huffaker, D. A., & Calvert, S. L. (2006). Gender, identity, and language use in teenage blogs. Journal of Computer-Mediated Communication, 10(2). Hull, G., & Greeno, J. (2006). Identity and agency in non-school and school worlds. In Z. Bekerman, N. C. Burbules & D. Silberman-Keller (Eds.), Learning in places: The informal educational reader (pp. 7797). New York: Peter Lang. Ito, M., Baumer, S., Bittanti, M., Boyd, D., Cody, R., Herr-Stephenson, B., et al. (2009). Hanging out, messing around, and geeking out: Kids living and learning with new media. Cambridge, MA: MIT Press. Jackson, P.L., Meltzoff, A.N., Decety, J. (2006). Neural circuits involved in imitation and perspective-taking. NeuroImage, 31(1), 429-439. Kendall, L. (1998). Meaning and identity in cyberspace: The performance of gender, class, and race online. Symbolic Interaction, 21(2), 129-153. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press. Lewis, S., Lindgren, R., & Pea, R. (under review). Learning with media: Harnessing viewpoint and motion to generate fields of potential action. Lindgren, R. (2009). Learning in a virtual world simulation environment: A perspective-taking approach. Paper presented at the annual meeting of the American Education Research Association, San Diego, CA. Lindgren, R. (in press). Generating a learning stance through perspective-taking in a virtual environment. Computers in Human Behavior. Lindgren, R., Pea, R., & Lewis, S. (2008). The impact of perspective in digital video on conceptual learning outcomes. Paper presented at the annual meeting of the American Education Research Association, New York, NY. Linn, M., Bell, P., & Hsi, S. (1998). Using the Internet to enhance student understanding of science: The Knowledge Integration Environment. Interactive Learning Environments, 6, 438. Lou, Y., Abrami, P. C., & d'Apollonia, S. (2001). Small group and individual learning with technology: A metaanalysis. Review of Educational Research, 71(3), 449-521. Meltzoff, A. N. (2007). 'Like me': A foundation for social cognition. Developmental Science, 10, 126-134. Meltzoff, A. N., Kuhl, P. K., Movellan, J., & Sejnowski, T. J. (2009). Foundations for a new science of learning. Science, 325(5938), 284288. Nasir, N. S. (2011). Racialized identities: Race and achievement among African American youth. Stanford, CA: Stanford University Press. Okita, S. Y., Bailenson, J., & Schwartz, D. L. (2007). The mere belief of social interaction improves learning. Proceedings of the Twenty-ninth Meeting of the Cognitive Science Society. Nashville, TN. Pea, R. D. (1985). Beyond amplification: Using computers to reorganize human mental functioning. Educational Psychologist, 20, 167-182. Pea, R. D. (1993). Practices of distributed intelligence and designs for education. In G. Salomon (Ed.), Distributed cognitions (pp. 47-87). Cambridge, UK: Cambridge University Press. Pea, R. D. (2004). The social and technological dimensions of scaffolding and related theoretical concepts for learning, education and human activity. The Journal of the Learning Sciences, 13(3), 423-451. Resnick, M., & Wilensky, U. (1997). Diving into complexity: Developing probabilistic decentralized thinking through role-playing activities. Journal of the Learning Sciences 7(2), 153172. Rizzolatti, G., & Craighero, L. (2004). The mirror neuron system. Annual Review of Neuroscience, 27, 169-192. Salzman, M. C., Dede, C., & Loftin, R. B. (1999). VRs frames of reference: A visualization technique for mastering abstract information spaces. In Proceedings of CHI 99 (pp. 489495). Pittsburgh, PA: ACM Press.
ISLS
433
ICLS2012
Samson, D., Apperly, I. A., Braithwaite, J. J., Andrews, B. J., Bodley Scott, S. E. (2010). Seeing it their way: Evidence for rapid and involuntary computation of what other people see. Journal of Experimental Psychology: Human Perception and Performance, 36(5), 1255-1266. Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences, 3(3), 265-283. Siegler, R. S. (1995). How does change occur: A microgenetic study of number conservation. Cognitive Psychology, 28, 225273. Thomas, A. (2007). Youth online: Identity and literacy in the digital age. London: Peter Lang. Turkle, S. (1995). Life on the Screen: Identity in the age of the Internet. New York : Simon and Schuster. Turkle, S. (1999). Cyberspace and identity. Contemporary Sociology, 28(6), 643-648. Valkenburg, P. M., Schouten, A. P. & Jochen, P. (2005). Adolescents' identity experimentation on the internet. New Media and Society, 7(3), 383-402. Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoperators and Virtual Environments, 7, 225240. Wortham, S. E. F. (2006). Learning identity: The joint emergence of social identification and academic learning. New York: Cambridge University Press.
ISLS
434
ICLS2012
Predicting Idea Co-Construction in Speech Data using Insights from Sociolinguistics

Gahgene Gweon, Mahaveer Jain, John McDonough, Bhiksha Raj, Carolyn Penstein Rose, Carnegie Mellon University, 5000 Forbes avenue, Pitsburgh, PA 15213, Email: {ggweon, mahaveercse, johnmcd, bhiksha, cprose}@cs.cmu.edu Abstract: Automatic assessment of group processes in collaborative groups is one of the holy grails of the computer supported collaborative learning community. In this paper, we present work towards detecting one type of group process which provides an important window into the inner workings of a group, namely idea co-construction (ICC). What is unique about our approach in relation to other educational data mining techniques is that we adopt insights from sociolinguistic theories by modeling stylistic convergence of speech. We present an unsupervised machine learning technique that is able to generate a predictor of the prevalence of ICC in face-to-face debates with an R2 value of .13, p < .05.
Introduction
Automatic analysis of collaborative processes has value both as part of dynamic interventions in the midst of collaborative learning as well as for research towards understanding how collaboration works. Early work in collaborative learning process analysis focused on text based interactions (Ros et al., 2008; McLaren et al., 2009; Mu et al., 2011). This work has enabled a whole series of studies where interactive support for collaborative learning was triggered by real time analysis of collaborative processes (Chaudhuri et al., 2008; Ai et al., 2010; Kumar et al., 2011). As communication technologies such as cell phones and voice over IP become more ubiquitous and allow for communication and collaboration over multiple modalities including video, audio, and text to be accessible any time and any place, the line between online group learning and face-to-face group learning begins to blur. Furthermore, as more and more collaboration takes place over video and audio channels, the need grows for the CSCL community to think about how to extend collaboration support technologies from the text realm into audio and eventually video. Recently, interest in group learning supported by robots has also begun to emerge. These shifts towards face-to-face group interactions in the three dimensional world around us rather than online require a corresponding shift in analysis technology from textbased input to speech based input. To meet this challenge, early work towards analysis of collaborative processes from speech has begun to emerge as well (Gweon et al., 2011), although the early results showed predictive value that was just above random. Machine learning algorithms are designed with the goal of finding mappings between sets of input features and output categories. When it comes to applications of machine learning to speech, the algorithms are not applied to the speech data in its raw form. Instead, it must first be represented in terms of a list of attributevalue pairs referred to collectively as a vector space representation of the speech. Thus, first the researcher must decide what is the set of features that every segment of speech will be represented in terms of. And then for each segment, these features must be extracted so that each attribute is associated with a value that was extracted from the speech. Supervised machine learning algorithms find stable patterns within these feature vector representations by examining collections of hand-coded training examples for each output category, then using statistical techniques to find characteristics that exemplify each category and distinguish it from the other categories. The goal of such an algorithm is to learn general rules from these examples, which can then be applied effectively to new data. In order for this to work well, the set of input features must be sufficiently expressive, and the training examples must be representative. The contribution of this paper is the application of an unsupervised machine learning technique that is able to generate a predictor of the prevalence of ICC in face-to-face debates with an R2 value of .13. The advantage of an unsupervised technique is that it can be trained on data that has not been hand labeled. The area of automatic collaborative process analysis has focused on discussion processes associated with knowledge integration such as transactivity (Berkowitz & Gibbs 1983; Teasley 1997; Weinberger & Fischer 2006), inter subjective meaning making (Suthers 2006), and productive agency (Schwartz 1998). We refer to the discussion process that we are targeting as idea co-construction (ICC), which is similar in nature to these earlier constructs. More specifically, ICC is defined as the process of building on an idea expressed earlier in a conversation using a reasoning statement. Research has shown that such deep knowledge integration processes provide opportunities for cognitive conflict to be triggered within group interactions, which may eventually result in cognitive restructuring and learning (de Lisi & Golbeck 1999). While the value of this general class of processes in the learning sciences has largely been argued from a cognitive perspective, these processes undoubtedly have a social component. At the heart of the Piagetian roots of the concept of transactivity comes the idea that it occurs when there is a balance of assimilation and accommodation within an
ISLS
435
ICLS2012
interaction. We expect this balance to occur when discussion partners share a certain equality in footing. Thus, with respect to the goal of automatic analysis of ICC from speech data, we hypothesize that a feature representation that captures speech structure related to social processes that reflect effort to build this social balance into an interaction will be highly predictive of the prevalence of ICC in an interaction. We test our hypothesis on a corpus of face-to-face debate discussions collected as part of research on arousal and learning (Nokes et al., 2010). In the remainder of the paper, we first situate our work in the midst of current directions in speech processing and review the literature on speech style accommodation in order to motivate our hypothesis. Next, we present both our manual and automatic approach for measuring the prevalence of ICC in debate discussions. We present an evaluation of the predictive value of our model. We conclude with a discussion of future directions.
Motivation and Background

The research goal of the work presented in this paper is to develop a speech processing technique that is capable of predicting the prevalence of idea co-construction (ICC) contributions in speech, where an ICC contribution is one in which reasoning is made explicit, and that reasoning builds on a prior reasoning statement in the discussion. Automatic analysis of deep knowledge integration processes is not a new direction in the CSCL community in itself. For example, several researchers have applied machine learning using text, such as newsgroup style interactions (Ros et al., 2008), chat data (Joshi & Ros, 2007), and transcripts of whole group discussions (Ai et al., 2010). The unique contribution of the work presented here is that it is one of the first approaches to detection of ICC in speech, and the first that does so by leveraging insights from sociolinguistics. In state of the art approaches to applying machine learning technology to speech data, including the earlier work on detecting ICC from speech (Gweon et al., 2011), the speech signal is first processed using basic audio processing techniques in order to extract features from segments of speech, which are then used for classification using a machine learning model. Gweon and colleagues (2011) used the same acoustic and prosodic features that are typically used in the language technologies community for predicting emotion from speech. For example, Ranganath and colleagues (2009) used acoustic and prosodic features extracted from speech data to predict whether a speaker came across as flirting or not in a speed dating encounter. Similarly, Ang and colleagues (2002) and Kumar and colleagues (2006) applied a similar technique to the problem of detecting emotions such as boredom, confusion, or surprise, whereas Liscombe and collegues (2005) applied these techniques to the problem of detecting student uncertainty. All of this work makes use of signal processing tools that are able to extract basic acoustic and prosodic features such as variation and average levels of pitch, intensity of speech, amount of silence and duration of speech. What this general approach misses is the way the values of these features change over the course of an interaction and how that change itself is meaningful. It is the social interpretation of the change in speech style characteristics that is the crux of the contribution of this paper. Acoustic and prosodic features are frequently associated with intuitive interpretations that make them an attractive choice to play a role in baseline techniques for these stylistic classification tasks. For example, increased variation in pitch might indicate that the speaker wants to deliver his ideas more clearly. Likewise, volume and duration of speech may signal that the speaker is explaining his ideas in detail, and is presenting his point of view about the subject matter. What is different about our work is that we base our approach on insights about the way in which speech style specifically (Coupland, 2009; Eckert & Rickford, 2001; Jaffe, 2009) and language style more generally (Fina et al., 2006) reflect both intentional and subconscious aspects of the way in which a speaker positions him or herself within an interaction at multiple levels. These recent accounts build on decades of work beginning with Labovs work on speech characteristics that signal social stratification (Labov 1966) and Giles work developing Social Accommodation Theory (Giles 1984), which describes how speech characteristics shift within an interaction, and how these shifts are interpreted. While the essence of transactivity has been characterized in prior work in terms of content level distinctions, we argue that it also has a social interpretation. For example, Azmitia and Montgomery (1993) have demonstrated that friends exhibit higher levels of transactive conversational moves, which are operationalized in a way that is similar to ICC, than pairs that are not friends. Furthermore, it makes sense to consider that in order to build on a partners reasoning, one must be attending to the partners reasoning and deem it worth referring to in the articulation of ones own reasoning. What we argue here is that we expect a construct known as Speech Style Accommodation to reflect these social processes and thus predict prevalence of ICC. Defining Speech Style Accommodation: The concept of what we refer to as Speech Style Accommodation has its roots in the field of the Social Psychology of Language. In this field, the many ways in which social processes are reflected through language, and conversely, how language influences social processes, are the
ISLS
436
ICLS2012
target of investigation (Giles & Coupland, 1991). As a first step towards leveraging this broad range of language processes, we refer to one very specific topic, which has been referred to as entrainment, priming, accommodation, or adaptation in other computational work (e.g., Levitan, Gravano, & Hirschberg, 2011). Specifically we refer to the finding that conversational partners may shift their speaking style within the interaction, either becoming more similar or less similar to one another. Our usage of the term accommodation specifically refers to the process of speech style convergence within an interaction. Stylistic shifts may occur at a variety of levels of speech or language representation. For example, much of the early work on speech style accommodation focused on regional dialect variation, and specifically on aspects of pronunciation, such as the occurrence of post-vocalic r in New York City, that reflected differences in age, regional identification, and socioeconomic status (Labov, 2010). Distributions of backchannels and pauses have also been the target of prior work on accommodation (Levitan et al., 2011). These effects may be moderated by other social factors. For example, Bilous and Krauss (1988) found that females accommodated to their male partners in conversation in terms of average number of words uttered per turn. Additionally, Hecht, Boster, and LaMer (1989) reported that extroverts are more listener adaptive than introverts and hence extroverts converged more in their data. Social interpretation of Speech Style Accommodation: It has long been established that while some speech style shifts are subconscious, speakers may also choose to adapt their way of speaking in order to achieve strategic social effects within an interaction (Sanders, 1987). One of the main motives for accommodation is to decrease social distance. On a variety of levels, speech style accommodation has been found to affect the impression that speakers give within an interaction. For example, Welkowtiz and Fledstein (1970) found that when speakers become more similar to their partners, they are liked more by partners. Another study by Putman and Street (1984) demonstrated that interviewees who converge to the speaking rate and response latency of their interviewers are rated more favorably by the interviewers. Giles and colleagues (1987) found that more accommodating speakers were rated as more intelligent and supportive by their partners. Conversely, social factors in an interaction affect the extent to which speakers engage in, and some times chose not to engage in, accommodation. For example, Purcell (1984) found that Hawaiian children exhibit more convergence in interactions with peer groups that they like more. Bourhis and Giles (1976) found that Welsh speakers while answering to an English surveyor broadened their Welsh accent when their ethnic identity was challenged. Scotton (1985) found that few people hesitated to repeat lexical patterns of their partners to maintain integrity. Computational models of speech style accommodation: Prior research has attempted to quantify accommodation computationally by measuring similarity of speech and lexical features either over full conversations or by comparing the similarity in the first half and the second half of the conversation. For example, Edlund and colleagues (2009) measure accommodation in pause and gap length using measures such as synchrony and convergence. Levitan and Colleagues (2011) found that accommodation is also found in special social events of conversation such as backchannels. They show that speakers in conversation tend to use similar kinds of speech cues such as high pitch at the end of utterance to invite a back channel from their partner. In order to measure accommodation on these cues, they compute the correlation between the numerical values of these cues used by partners. When stylistic shifts are focused on specific linguistic features, then measuring the extent of the stylistic accommodation is simple since a speakers style may be represented on a one or two dimensional space, and movement can then be measured precisely within this space using simple linear functions. However, the rich sociolinguistic literature on speech style accommodation highlights a much greater variety of speech style characteristics that may be associated with social status within an interaction and may thus be beneficial to monitor for stylistic shifts. Unfortunately, within any given context, the linguistic features that have these status associations, which we refer to as indexicality, are only a small subset of the linguistic features that are being used in some way. Furthermore, which features carry this indexicality are specific to a context. Thus, separating the socially meaningful variation from variation in linguistic features occurring for other reasons can be like searching for a needle in a haystack. To meet this challenge, we measure accommodation using Dynamic Bayesian Networks (DBNs) (Jain et al., 2012). More details can be found in the methods section (Step 3.2).
Method
Our experiment setup for predicting the idea co-construction process is outline in Figure 1. In the current section, we detail our approach for steps 1, 2, 3.1 and 3.2.
ISLS
437
ICLS2012
Figure 1. The Process overview.
1. Data collection using speech recorders

Our corpus was collected in a laboratory setting where pairs of participants were engaged in a debate in which they took opposing sides on a controversial topic. The specific task that the participants were asked to discuss was the cause of the decline of the Ottoman Empire, which has prompted controversy among historians. One perspective emphasizes factors internal to the Empire, while the other emphasizes external factors. Each of the participants were provided with background materials that supports the role of internal or external factors, and were asked to argue for the side that was assigned. Participants were all male undergraduate students between the ages of 18 and 25. In prior studies, it has been shown that accommodation varies based on gender, age and familiarity between partners. Because this corpus controls for most of these factors, it is appropriate for our experiment. Furthermore, because the participants did not know each other before the debate, we can assume that if accommodation occurred, it was only during the conversation. Each debate lasted 8 minutes. In order to collect clean speech with each student on a separate channel, each student wore a directional microphone. Nevertheless, although it is possible to clearly identify the main speaker from an audio file, crosstalk, which is the other participants voices, could still be heard in the background. A total of 20 sessions (40 participants) were collected and used for further analysis.
2. Transcribing and segmenting the recorded data

For each audio file, the eight-minute discussion sessions were transcribed and manually segmented for further analysis. A total of 40 meetings were collected, transcribed, and segmented according to the following three main rules. The resulting data contained a total of 1890 segments. Independent-clause rule: a segmentation boundary should be placed as soon as an independent clause is identified. Dependent-clause rule: a sentence that cannot stand alone should be unitized with either the preceding or following unit Analyze-from-Beginning rule: sentences should be analyzed from the beginning of the sentence towards the end, i.e. Match the subject in the sentence with the closest verb.
3.1. Computing the amount of the Idea co construction

When students are working on a given task or a project in a team, they receive a certain amount of information that would help them solve the problem, in the form of a task statement and training materials. In order to solve the given problem, students discuss the materials that are given to them and try to apply them to a potential solution. We are interested in capturing instances when students display reasoning during group discussions that goes beyond what is given and displays some understanding of a causal mechanisms behind the information. Typically some causal mechanism would be referenced in a discussion of how something works or why something is the way it is. In segmenting student talk and identifying which segments display reasoning we are able to quantify amount of reasoning displayed. However, it is important to note that since what we are coding is attempts at displayed reasoning, we need to allow for displays of incorrect, incomplete, and incoherent reasoning to count as reasoning. That will necessarily be quite subjective especially in the case of incoherent explanations. We begin by operationalizing the distinction between non-reasoning statements and reasoning statements, and then we focus on the distinction between reasoning statements that represent new directions within a conversation (i.e., externalizations) from those that build on prior contributions (i.e., ICC contributions). One important goal in detecting the knowledge integration process is to distinguish instances when students are making their own reasoning explicit from ones that just parrot what they have heard. In our formulation, we consider the task and training materials provided during the experiment to be given, and we look for contributions where students go beyond that.
ISLS
438
ICLS2012
Operationalization step 1: Reasoning process

Our formulation of what counts as a reasoning display comes from the Weinberger and Fischers (2006) notion of what counts as an epistemic unit, where what they look for is a connection between some detail from the given task (which in their case is the object of the case study analyses their students are producing in their studies) with a theoretical concept (which in their work comes from the attribution theory framework, which the students are applying to the case studies). When they have seen enough text that they can see in it mention of a case study detail, a theoretical concept, and a connection between the two, they place a segment boundary. Occasionally, a detail from a case study is described, but not in connection with a theoretical concept. Or, a theoretical concept may be mentioned, but not tied to a case study detail. In these cases, the units of text are considered degenerate, not quite counting as an epistemic unit. We have adapted the notion of an epistemic unit from Weinberger and Fischer (2006) because the topic of our conversations is very different in nature. The conversations that we analyzed come from a debate where two participants are asked to take opposing sides on the cause of the fall of the Ottoman Empire. As in Weinberger and Fishers (2006) notion of epistemic unit, we look for a connection between two or more concepts. We describe our operationalization in detail below. First, examine a segment of a conversation provided in Table 1. The fourth column indicates whether the given contribution contains reasoning (R) or no reasoning (N). Table 1: Sample contribution Line 14 Speaker B Contribution I think that the economic downfall of the Ottoman Empire was due to internal problems because of the first World War uh, and other civil wars going on uh, beforehand which took place over the hundreds and thousands of years that people have been in that area. Um, this lead to, these wars lead to population problems. Uh, people were either being killed or they couldn't farm, and if you can't farm, you can't feed people R/N R I/E E
15 16 17
R N R
I N I
The simple way of thinking about what constitutes a reasoning display is that it has to communicate an expression of some causal mechanism. Often that will come in the form of an explanation, such as X because Y. However, it can be more subtle than that, for example Russian invasion in 1914 led to a decrease in their population. The basic premise was that a reasoning statement should reflect the process of drawing an inference or conclusion through the use of reason. Note that in the example with the Russian invasion, although there is no because clause, one could rephrase this in the following way, which does contain a because clause: The population decreased because of the Russian invasion in 1914. More generally, we defined reasoning display as a relationship between two or more concepts. A concept could be prior knowledge, or one of the facts provided to the participants. The presence of multiple concepts in a statement by itself does not determine whether a statement contains reasoning. Rather, the relationship between multiple concepts is the determining factor. For example, a simple list of concepts (e.g., population decreased) is information sharing, and not reasoning. We identified two types of relationships that signal a reasoning process; 1. Compare & contrast, 2. Cause & effect. 1. Compare and contrast, tradeoff: When the speaker compares two concepts, the speaker is making a judgment, which involves thinking about how two concepts are related to another. The speaker compares two time periods (at the time & today): At the time if you look at the technology, it wasnt that advanced as we have today. When a speaker makes an analogy, he is making a link due to the similarity between two concepts. Outside powers were like the match lighting the fire. Cause and effect: When the speaker uses a cause-and-effect relationship, this process involves establishing the relationship between two concepts through a reasoning process. The general relation in this category is doing x helps you achieve y. Examples are illustrated below. A because of B: they forced the Empire to be economically dependent because they set up trading posts and banks A in order to achieve B: Great Britain came in and introduced capitulations to control schools and health systems.
2.
ISLS
439
ICLS2012
Operationalization step 2: Idea co-construction (ICC) vs. Externalization

Statements that display reasoning can be either Externalizations, which represent a new direction in the conversation, not building on prior contributions, or ICC contributions, which operate on or build on prior contributions. In our distinction between Externalizations and ICC contributions, we have attempted to take an intuitive approach by determining whether a contribution refers linguistically in some way to a prior statement, such as through the use of a pronoun or deictic expression. Take the sample conversation we used earlier to illustrate the reasoning contribution in table 1. The last column of table 1 is marked as either an externalization (E), idea co-construction (I) for the statements that are marked as (R). The first statement by B is an externalization since B starts a new topic, thus this contribution is not building on a prior contribution. Subsequent reasoning contributions in this discussion are coded as (I) because they each build on statements that directly precede them.
Reliability of Annotation
Two coders were initially trained using a manual that describes the above operationalization of reasoning displays and ICC in detail along with an extensive set of examples. After each coding session, the coders discussed disagreements and refined the manual as needed. Most of the disagreements were due to the interpretation of what the students meant rather than the definition of reasoning itself. Therefore, later efforts focused more on defining how much context of a statement could be brought to bear on the interpretation and how. In a final evaluation of reliability for reasoning process, we calculated kappa agreement of 0.72 between two coders over all the data. After calculation of the kappa, disagreements were settled by discussion between the two coders. For detecting instances of ICC and externalization, our coding yielded a kappa value of 0.7.
Amount of Idea co-construction (ICC)

Because the accommodation scores were computed for each 8 minute session, we computed a comparable score by adding up the number of idea co-construction for each session. This resulted in an average score of 36. The minimum score for a session was 22, and the maximum score was 60.
3.2. Computing the amount of accommodation

What we evaluate in our study is the ability of this measurement of speech style accommodation to predict prevalence of ICC in debate conversations. In order to measure the amount of accommodation, we used Dynamic Bayesian Networks (DBNs). What we are specifically interested in is the manner in which the influence of one partner on the other is modeled. What is novel in our approach is the introduction of the concept of an accommodation state. The accommodation state models structurally the insight that accommodation occurs over time as a reflection of a social process, and thus has some consistency in the nature of the accommodation within some span of time. The major advantage of modeling consistency of motion within the style shift over time is that it provides a sign post for identifying which style variation within the speech is salient with respect to social interpretation within a specific interaction. Therefore, the model may remain agnostic to which style features are shifting and may thus be applied to a variety of interactions that differ with respect to which stylistic features have strategic social value for the participants. The technical details of the inner workings of the model as well as a validation that it is able to measure speech style accommodation are described in a separate paper (Jain et al., 2012). However, the types of features used are briefly described in the following subsection.
Speech Features
Speech stylistic information is reflected in prosodic features such as pitch, energy, and speaking rate. In our work, we leverage on several of these prosodic speech features to quantify accommodation using Dynamic Bayesian Networks (DBNs). To extract features, speech from each session was segmented into a window of 50ms, with adjacent overlapping windows by 40ms. From each window, a total of 7 features were computed using OPENSmile toolkit (openSmile 2011). These features are voice probability, harmonic to noise ratio, voice quality, and three different measurement for pitch (F0, F0 raw , F0 env), and loudness. Frame size within each window was set to 50 milliseconds and frame step was set to 10 milliseconds. Next a 10 bin histogram of feature values were computed for each of these features, which was then normalized to sum to 1.0. The normalized histogram represents both the values and the fluctuation in the feature. For example, a histogram of loudness feature captures the variations in the loudness of the speaker within the turn.
Amount of Accommodation
Possible range of accommodation score ranged from 0 to 1, where 0 means there was no accommodation between the two partners. The average accommodation score was 0.43.
ISLS
440
ICLS2012
Results
Our hypothesis motivated by the social interpretation of ICC discussed above that we will see a significant positive correlation between the prevalence of ICC in a conversation based on our hand coding of the transcript and the amount of Accommodation detected in the speech using our Dynamic Bayesian Network model. Over the 20 discussion transcripts discussed above, we counted the number of contributions that were coded as transactive, and labeled this sum Prevalence of ICC. Similarly, we labeled the score computed by our network model as Accommodation score. When we computed a linear regression model between the Accommodation score and Prevalence of ICC, we found a statistically significant correlation, R2 = .13, p < .05.
Conclusions and Current Directions

In this paper, we presented our work towards automatic detection of prevalence of idea co-construction contributions in speech data. The goal of this paper was to develop technology to address such needs. To this end, we have demonstrated the feasibility of using data mining techniques by modelling stylistic convergence of speech. Our work shows promise in that our model generated a predictor of amount of idea co-construction with an R2 value of 0.13. In our future work, we plan to compute the amount of accommodation at the clause level, the unit of analysis used for idea co-construction coding. With this level of analysis, we would have enough data points to run machine learning in order to predict whether each clause is an instance of displaying idea co-construction rather than just predicting the prevalence of ICC over the whole interaction. In addition, more sophisticated adaptations of sociolinguistic work might suggest follow-up techniques. For example, we plan to investigate sequencing and timing rather than just quantity, in line with work conducted by Kapur and colleagues (2009). In terms of data, we are currently collecting and annotating audio data from additional meetings as well as other contexts to validate our result further as well as testing its generality across a wider variety of student groups.
References
Ai, H., Kumar, R., Nguyen, D., Nagasunder, A., & Ros, C. P. (2010). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, Proc. ITS. 134-143. Ang, et. al. (2002). Prosody based automatic detection of annoyance and frustration. In Proc. International conference spoken language processing, Denver, Colorado, USA, 2002, pp. 2037 2039. Azmitia, M., & Montgomery, R. (1993). Friendship, transactive dialogues, and the development of scientific reasoning. Social Development, 2(3): 202-221 Baker, R.S., Corbett, A.T., Koedinger, K.R. Detecting Student Misuse of Intelligent Tutoring Systems. Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 531-540 (2004) Berkowitz, M., & Gibbs, J. (1983). Measuring the developmental features of moral discussion. Merrill-Palmer Quarterly, 29, 399-410. Bilous, F. & Krauss, R. M. (1988). Dominance and accommodation in the conversational behaviour of same and mixed-gender dyads. Language and Communication, 8, 183-194. Bourhis, R. Y. & Giles, H. (1977). The language of intergroup distinctiveness. In H. Giles (Ed.), Language, Ethnicity and Intergroup Relations (pp. 119-135) Chen, M. (2003). Visualizing the pulse of a classroom. In Proc. MM, ACM Press, 555-561. CMUSphinx Wiki. (2010). http://cmusphinx.sourceforge.net/wiki/ Coupland, N. (2009). Style: Language Variation and Identity: Key Topics in Sociolinguistics, Cambridge University Press. De Cheveign, A., & Kawahara, H. (2002). "YIN, a fundamental frequency estimator for speech and music," The Journal of Acoustical Society of America, 111(4), 1917-1930. De Lisi, R., & Golbeck, S. (1999). Implications of Piagetian Theory for Peer Learning, A. ODonnell & Alison King (Eds.) Cognitive Perspectives on Peer Learning, Lawrence Erlbaum Associates, New Jersey. DiMicco, J., et. al. . (2004). Influencing group participation with a shared display. Proc. CSCW, 614-623. Eckert, P., & Rickford. J. (2001). Style and Sociolingusitic Variation, Cambridge University Press. Edlund, J., Heldner, M., and Hirschberg, J. (2009). Pause and gap length in face-to-face interaction. In Proceedings of Interspeech. Faidley, J., et. al. (2000). How are we doing? Methods of assessing group processing in a problem-based learning context. In Evensen, D. H., and Hmelo, C. E. (eds.), Problem-Based Learning: A Research Perspective on Learning Interactions, Erlbaum, NJ, 109-135. Feng, M., Beck, J., & Heffernan, N. Using Learning Decomposition and Bootstrapping with Randomization to Compare the Impact of Different Educational Interventions on Learning. In Barnes, Desmarais, Romero & Ventura (Eds) In Proc. 2nd Int. Conference on Educational Data Mining. pp. 51-60 (2009) Freund, Y., & Schapire, R. E. (1995). A decision-theoretic generalization of on-line learning and an application to boosting. European Conference on Computational Learning Theory. 23-37. Fina, A., Schiffrin, D., & Bamberg, M. (2006). Discourse and Identity, Cambridge University Press
ISLS 441
ICLS2012
Forbes-Riley, K., & Litman, D. (2009). Adapting to Student Uncertainty Improves Tutoring Dialogues. In Proc AIED 2009: 33-40. Giles, H. (1984). The Dynamics of Speech Accommodation, Amsterdam: Mouton. Gweon, G., Kumar, R. Ros, C. (2009). GRASP: The Group Assessment Platform, In Proc CSCL. Gweon, G., Agarwal, P., Udani, M., Raj., B., Ros, C. P.(2011). The Automatic Assessment of Knowledge Integration Processes in Project Teams, in Proceedings of Computer Supported Collaborative Learning, p.462-469. Hecht, M., Boster, F. and LaMer, S. (1989). The effect of extroversion and differentiation on listener-adapted communication. Communication Reports, Volume 2, pp 1-8. Jaffe, A. (2009). Stance: Sociolinguistic Perspectives, Oxford University Press. Jain, M., McDonough, J., Gweon, G., Raj, B., Ros, C. P. (2012). An Unsupervised Bayesian Network Approach to Measuring Speech Style Accommodation, in Proceedings of the European Chapter of the Association for Computational Linguistics. Joshi, M., & Ros, C. P. (2007). Using Transactivity in Conversation Summarization in Educational Dialog. Proceedings of the SLaTE Workshop on Speech and Language Technology in Education Jensen, F. V. (1996). An introduction to Bayesian networks. UCL Press. Kapur, M., & Kinzer, C. K., (2009). Productive failure in CSCL groups. In Proc, CSCL 2009. 4: 21-46. Kay, J.,et al. (2006). Wattle Tree: Whatll It Tell Us?, University of Sydney Technical Report 582, January 06. Kim, T., et. al. (2008). Meeting Mediator: Enhancing Group Collaboration with Sociometric Feedback, In Proc. of CSCW, San Diego, CA, 457-466. Kumar, R., Ros, C. P., & Litman, D. (2006). Identification of Confusion and Surprise in Spoken Dialogusing Prosodic Features , Proceedings of Interspeech 2006. Labov, W. (1966). The social stratification of English in New York City, Washington DC: Center for Applied Linguistics. Labov, W. (2010). Principles of Linguistic Change, Volume I, Internal Factors. Wiley-Blackwell. Levitan, R., Hirschberg, J. (2011). Measuring acoustic-prosodic entrainment with respect to multiple levels and dimensions. In In Proceedings of Interspeech. Levitan, R., Gravano, A., Hirschberg, J. (2011). Entrainment in speech preceding backchannels. In Proc. ACL/HLT Liscombe, J. Venditti, J, & Hirschberg, J. (2005). Detecting certainness in spoken tutorial dialogues. In Proc. Interspeech 2005. 1837-1840. Nokes, T., Levine, J. M., Belenky, D., & Gadgil, S. (2010). Investigating the impact of dialectical interaction on engagement, affect, and robust learning, Proc. International Conference of the Learning Sciences. Purcell, A. K. (1984). Code shifting in Hawaiian style: Childrens accommodation along a decreolizing continuum, International Journal of the Sociology of Language, volume 46, pp 71-86. Putman, W. and Street, R. (1984). The conception and perception of noncontent speech performance: Implications for speech accommodation theory. International journal of the sociology of language, Volume 46, pp 71-86. Pianesi, F., et al. (2008). Multimodal support to group dynamics. Personal and Ubiquitous Computing. 12(3), 181-195. Pon-Barry., et al. (2006). Responding to Student Uncertainty in Spoken Tutorial Dialogue Systems. IJAIED 16(2) Ranganath, R., Jurafsky, D., & McFarland, D. (2009). It's Not You, it's Me: Detecting Flirting and its Misperception in Speed-Dates. Proceedings of EMNLP 2009. 334-342. Ros, C. P., et al. (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning , IJCSCL. 237-271 Soller, A., & Lesgold, A. (2003). A computational approach to analyzing online knowledge sharing interaction. In Proc. AIED, 253-260. Teasley, S. D. (1997). Talking about reasoning: How important is the peer in peer collaboration? In L. B. Resnick (Eds.), Discourse, tools and reasoning: Essays on situated cognition, 361-384. Weinberger A., & Fischer F. (2006). A framework to analyze argumentative knowledge construction in computer supported collaborative learning. Computers & Education; 46, 71 95.
Acknowledgments
This research was supported in part by NSF grants 0648487 and 0935127.
ISLS
442
ICLS2012
Automatic extraction of interpretable topics from online discourse

Yonghe Zhang, Beijing Normal University, China, yonghe617@gmail.com Nancy Law, The University of Hong Kong, Pokfulam, Hong Kong, nlaw@hku.hk Yanyan Li, Beijing Normal University, China, liyy1114@gmail.com Ronghuai Huang * , Beijing Normal University, China, huangrh@bnu.edu.cn Abstract: Teachers adopting CSCL often face the challenge of handling massive textual information, and finding it difficult to have a clear grasp of the topics being addressed in the discourse. Topic modeling, an emerging field in machine learning, has the potential to solve this problem by automatically extracting from text collections formal representations of latent topics. However, the interpretation of latent topics is still a challenge, which hinders the use of this state-of-the-art technology from wider use in CSCL contexts. In a recent paper, we put forward a novel topic discovery method, the fLDA model, based on Minskys Frame theory. This method has the advantage of providing outputs that are potentially more easily interpretable for generating the topic of each thematic cluster. In this paper, we show how fLDA can be used in extracting and visualizing the topics of asynchronous online discourse from four classrooms.
Introduction
Online discussion forums provide students with the opportunity to explore different problems and topics through discussion in an unconstrained manner, allowing more student-centered interactions to take place. It also provides a record of the explorations that teachers can use to gain an understanding of the students concerns and on that basis to make facilitation decisions. However, making sense of the massive amounts of text in the posted messages is a daunting challenge for teachers adopting CSCL in their teaching repertoire. How can a teacher readily find out the key topics of discussion among hundreds of posted messages? There is a need for semantic tools that can identify key topics in online discussions to support pedagogical decision-making. Summarizing massive free style text has been a long-standing issue in computer science, esp. in machine learning. One of the most relevant research areas to tackle this problem is text clustering (also referred to as document clustering). Text clustering methods can be used to automatically group documents into a list of clusters. Each cluster is regarded as a collection of documents on a latent topic. By interpreting latent topics and examining their distribution, users can gain an overview of the content focus of a collection of texts. Recent research in text clustering shows that the interpretability of latent topics is still a challenge (Blei, in press). Earlier text clustering methods based on the Vector Space Model (Salton, Wong, and Yang, 1975) using vector similarity measures (e.g. cosine similarity) do not provide further semantic clues for the interpretation of topics. Latent Semantic Analysis (Deerwester, Dumais, Landauer, Furnas, Harshman, 1990) uses a set of latent variables to represent topics, but still fails to provide intuitive interpretation for each topic. Topic models such as LDA (Blei, Ng and Jordan, 2003) use a set of weighted words to represent each topic. These have been used successfully to identify research topics among tens of thousands of scientific articles (Griffiths and Steyvers, 2004). Other work related to the interpretability of topic models include coherence measures of topics manually (Chang, Boyd-Graber, Gerris, Wang, and Blei, 2009), automatically (Newman, Han-Lau, Grieser and Baldwin, 2010; Musat, Velcin, Trausan-Matu, and Rizoiu, 2011), interpretability improvement by semi-supervised method (Zheng, 2008) and connecting topics, e.g. hLDA (Blei, Griffiths, Jordan and Tenenbaum, 2003), PAM(Wei and Andrew, 2006), hPAM (David,Wei and Andrew, 2007). This paper introduces our recent work on a novel topic discovery method, the fLDA model (Zhang, Li and Huang, under review), and presents our preliminary exploration on using this to identify and interpret the online discourse data collected from four classrooms that used Knowledge Forum to support students learning.
Topic Models and the Interpretability Challenge

Probabilistic topic models are a suite of algorithms whose aim is to discover the hidden thematic structure in large archives of documents (Blei, in press). The most famous topic model is Latent Dirichlet Allocation (LDA) (Blei et al, 2003). The intuition behind LDA is that documents exhibit multiple topics. A topic is formally defined as a distribution over a fixed vocabulary. For example, a document on the topic of education would have a high probability of containing words about pedagogy and learning while one on the topic of computer
*
Corresponding author is Ronghuai Huang, huangrh@bnu.edu.cn. This paper is supported by the Chinese National Natural Science Fund (61075048) and the Information Technology Strategic Research Theme of the University of Hong Kong.
ISLS
443
ICLS2012
science would likely contains words about hardware and software. Technically, LDA assumes that topics are specified before any document is generated. Each document in a collection is regarded as generated in a twostage process (document generative process): 1. Randomly choose a distribution over topics. 2. For each word in the document (a) Randomly choose a topic from the distribution over topics in step #1. (b) Randomly choose a word from the corresponding distribution over the vocabulary. Based on this assumption, the goal of topic modeling is to automatically discover the topics from a collection of documents. The documents themselves are treated as observed data, while the topic structurethe topics, per-document topic distribution, and the per-document per-word topic assignmentis the hidden structure. The central computational problem for topic modeling is to use the observed documents to infer the hidden topic structure, and hence is a reverse generative process. Topic modeling algorithms generally fall into to two categoriessampling-based algorithms (Steyvers and Griffiths, 2006) and variational algorithms (Wainwright and Jordan, 2008). For a good discussion of the merits and drawbacks of both, see Asuncion, Welling, Smyth and Teh (2009). Topic modeling is an emerging field in machine learning. Among the many challenges in this field, the most basic one may be the evaluation and model checking problemhow to evaluate a topic model by the meaningfulness of the topics identified (Blei, in press). This problem has led to many efforts on evaluation and improvement of topic interpretability. Current topic interpretability research focuses on evaluation of topic interpretability and interpretability improvement. To evaluate topic interpretability, popular strategies include the use of coherence measure of topics using a qualitative approach (Chang et al, 2009) and a lexicon-based automatic approach (Newman et al, 2010; Musat et al, 2011). To improve topic interpretability, many efforts have focused on discovering topic hierarchy. Ning (2008) incorporates human intervention in the topic modeling process, such as shaping topics with human knowledge. Current topic models use a set of weighted words to represent a topic.
The fLDA Model

Our work focuses on improving topic interpretability by representing each topic in a more human-readable form. To improve user interpretability of topics, we need to understand how humans achieve understanding of situations represented by words. There are many theories in discourse psychology that try to explain text comprehension, e.g. Frame Theory (Minsky, 1975), Script Theory (Schank, 1975), Story Grammar (Rumelhart, 1975) and propositional representation of discourse (Kintsch and van Dijk, 1978). Among these theories, Minskys Frame Theory can be more easily implemented in artificial intelligence applications in terms of its data structure. According to Minskys (1975) Frame Theory (or Schema Theory), when one encounters a new situation or makes a substantial change in one's view of the situation, one selects from memory a structure, called a frame. A frame is a data-structure for representing a stereotypical situation, like being in some ordinary living spaces or performing certain activities. From a computational perspective, a frame is a set of slot-value pairs. Slots are stable for a frame while the value for each slot is adaptively assigned to represent each specific situation. For example, a story frame may have slots named time, place, thing and event. So a theft story may be: At 8:00 pm inside a jewelry store, a diamond watch was stolen when the shop-keeper was distracted by a mob shouting outside. Then a frame for this story can be written as ( time=8:00 pm, place=a jewelry store, things=a diamond watch, shop-keeper, a mob shouting, event=distracted, stolen). The level of detail in value assignment of frame slots is assumed to reflect the extent of ones understanding of the situation. For example, event=the shop-keeper was distracted by a mob, a diamond watch was stolen is more understandable than event=distracted, stolen. The Frame Theory has inspired much productive research in cognition (Solso, MacLin and MacLin, 2004). It is apparent that the externalization of topics in the form of frames helps one to understand the topics. Hence in this study we explore the use of a frame-based method to extract topics from document collections in order to find out whether the output is more readily interpretable as meaningful topics by human readers. In the remainder of this section, we will briefly describe the frame-based topic discovery model we developed based on the LDA model, named fLDA (Zhang et al, under review).
Definitions of key terms in the fLDA model

The following are the definitions of some basic terms used in the fLDA model. WordA word w is the basic unit in text, and is defined as a string in its original form. DocumentA document d is a sequence of words extracted from the text. It can be denoted as d = {w1,w2, . . . , wn}. The i-th element of d is referred to as the i-th word token or i-th token, which is conceptually different from the term word. CorpusA corpus c is a collection of documents. Term frequencyThe term frequency of a word w in corpus c is the number of occurrences of w in all documents contained in c.
ISLS
444
ICLS2012
Co-occurrence frequencyThe co-occurrence frequency of word w1 and w2 in corpus c is the number of occurrences of w1 in any document containing w2 in c. Topic frameA topic frame f is a quadruple of semantic slots: focus, features, events and related things. A focus is a single word representing an entity, and can be regarded as a central concept of a topic. A feature is a weighted word representing an entity property. Each frame contains a set of features. An event is a weighted word representing some action. A related thing is a weighted word representing entities. Each of the features, events and related things is weighted by its co-occurrence frequency with respect to the focus of the frame, for frames having non-zero weight. It should be noted that ontologically the focus words are also things. Features, events and related things can be regarded as foil concepts or foils to the central concept. A "foil" is used in text comprehension studies to refer to a concept associated with a central concept. In this study, a frame is make up of a focus and its foils, and a foil of a focus can be a feature, an event or a thing related to the focus. TopicA topic T of corpus c is a set of topic frames and their weights pair, and these topic frames reflect the main content of the topic-related documents. It can be denoted as T = ( (f1,pf1), (f2,pf2), . . . , (ft ,pft) ), where pfi is equal to the term frequency of the focus of fi.
Input and output of fLDA topic modeling algorithm

The topic modeling approach of fLDA is to input a corpus with POS (Part-Of-Speech) tags for each word, and output a set of topics defined by the previous sub-section. POS-tagging tools are available online in several languages, e.g. the Stanford POS tag tool for English and the ICTCLAS tool for Chinese. In the fLDA model, each word has a semantic class (one of thing, feature, event or other) and a topic. fLDA assumes a similar corpus generative process as LDA. Based on this assumption, we have developed a Gibbs sampling-based algorithm to assign a semantic class and a topic for each word. In Zhang, Li and Huang (2011), we use a pseudo dataset generated using the corpus generative process to test the recall rate and performance of this algorithm. Intuitively, every 10~30 new documents stay on one topic, and each word token in a document has 50% probability to be on the document topic and 50% to be of other topics randomly. Due to the lack of a fully trusted quality indicator for the topic-modeling algorithm, we present a visualization of the topic discovery process. This visualization is generated by the algorithm in JPG image format. Due to space limitations, only a part of the visualization is shown in figure 1. The numbers at the top of the images indicate the iteration number and each image represents the matrix for the document-topics. Each row of an image represents a document, and each column a topic. The darkness of a pixel indicates the topic word frequency of a document on a topic. Tiny horizontal lines are used to indicate the boundary of every 10 documents. At iteration zero, the color of blocks in the image is uniformly distributed, as it is assumed that every document has similar word frequency on topics. As the number of iteration increases, four vertical line segments become more and more clear. Each of these segments is made up by several adjoining blocks in the same column. This indicates that the corresponding documents have high word frequency in the corresponding topic of the relevant blocks. The documents with darkened blocks in the same column relates to the same topic. The image of iteration 400 shows a clear topic distribution, and this distribution is consistent with the parameters predefined in the data generation rule. column 2 : topic 2 column 3: topic 3 column 4 : topic 4 column 1: topic 1 row 1 : document 1 row 2 : document 2 row 3 : document 3 iteration darkness indicate the word frequency on a topic 3 in both document 2 and 3 is high
Figure 1. Visualization of the topic discovery process by the fLDA algorithm on a pseudo dataset. We chose one of the four Knowledge Forum discourse datasets used in this study to test the quality of tagging on the semantic classes, which is the basis of frame slots using this algorithm (the 6B dataset referred to in the
ISLS 445
ICLS2012
next section), and used Cohens Kappa (Jean, 1996) as the cross-validation indicator. Two graduate students achieved kappa=0.73 for their tagging, while the algorithm-human kappa=0.69. Although this is far from perfect, the result can be regarded as an acceptable one.
Interpretability of fLDA Generated Topic Outputs Using Authentic CSCL Data

To explore whether fLDA can help one to interpret the topic foci of online discourse in CSCL settings, we analyzed the discourse corpuses from four primary school Chinese Language classrooms participating in the Knowledge Building Teacher Network led by HKU-CITE (http://kbtn.cite.hku.hk). Some basic information about these corpuses is shown in Table 1. A note is a message posted by one or more students. The four classrooms belong to two schools: 6B, 6C belong to one school and 5A, 5B belong to another. The classrooms from the same school shared the same discussion topic based on the set school curriculum. The theme Hong Kong (HK) Kids was centered around negative media reporting about Hong Kong childrens lack of general life skills and inability to handle adversity. Ice duck is the title of a passage in the Chinese Language textbook, which is a story about the pursuit of ones ideal in life. Table 1: Basic information of the CSCL discourse corpuses selected for this study. Class 6B 6C 5A 5B Number of notes generated 274 76 167 199 Discussion duration (weeks) 10 8 18 15 Theme of discussion HK kids HK kids Ice duck Ice duck
Topic extraction
In order to be able to compare the topic frequencies in the discussions generated from different classrooms sharing the same theme, we combined the corpuses of 6B and 6C in one analysis, and the corpuses from 5A and 5B in another. fLDA is an exploratory method and the user needs to pre-set the number of topics to be extracted. In our preliminary exploration, analyses using pre-set topic numbers from 4 to 10 were conducted. We find that outputs from too few or too many pre-set topics are more difficult to make meaningful interpretations. In this study, the topic number used is 5. In all runs of the algorithm, the number of iterations was set to 1000. In this paper, we focus our reporting on the 6B_6C corpus. The theme of the discussion is Hong Kong Kids. The output generated by the fLDA algorithm is reprocessed to show the top 3 weighted frames of each topic and the three documents that are most representative for the topic. Table 2 presents the high co-occurrence words for the top three frames generated by fLDA for each of the five topics. The original text of the students discourse was in Chinese. The English translation of the words are provided in brackets. The meaning of each topic is not immediately evident based only on the frame words in Table 2. The following steps are taken to formulate an understanding of the topics: 1. For each focus word in a topic, identify the co-occurring words in the other frames and rank order them in decreasing frequency. 2. For the high co-occurrence pairs, locate the text segments containing these pairs to understand the relationships expressed in the documents on these pairs. 3. Synthesize the meanings of the co-occurring words in the top three frames of each topic to generate the meaning of each of the five identified topics. To illustrate how we can follow the above steps to arrive at the meaning of a topic, we use the first topic in Table 2 as an example. Table 2: High co-occurrence content words in frames of Topic 1 from a five-topic analysis of the 6B_6C corpus (English translation of words and frequency of occurrence in brackets). Topic Focus (things) Feature 1 (difficult, 2) (system, 22) (poverty, 2) (HK, 19) (difficult, 2) (education, 12) (maid, 13) Event Related things (learn,4), (education,4) (education, 13), (Hong (occur,3), (reduce, 3) Kong, 8),(elitism, 7) (learn, 4), (system, 8), (occur,3), (spread,3) (elitism, 5) (education, 4), (country, 3) (take care, 2) (learn, 4), (occur, 3), (system, 13), (elite, 6), (admission, 2) (Hong Kong, 4), (parent, 3), (government, 3) (encounter ,4), (training of maid, 4), (help ,4), (train ,3), (child, 2), (government, 2), (listen,3),(independent,2) (method, 2)
ISLS
446
ICLS2012
First, as we can see in Table 2, the foci of topic 1 are system, Hong Kong and education. According to our contextual knowledge and the high co-occurrence of these three words in the topic frames, we predict this topic is about the relation between Hong Kongs education system and the cause of the HK Kids phenomenon (which refers to the childrens lack of general life skills and inability to handle adversity). In the second step, we find that among the word pairs containing the foci words, the words with the highest co-occurrence frequency are elitism and elite. We locate the text segments containing these pairs as bellow. The English translation of some of these segments are listed below: Prof. Chen point out that Hong Kongs education system lead to elitism. Since the whole society is focused on fostering elites, parents in our education system wants to send their kids to prestigious schools. Otherwise, it will be difficult for these kids to survive in an elitist society. If the Hong Kong education system does not encourage elitism, will the Hong Kong Kids phenomenon gradually disappear? Hong Kong Kids may be caused by the elitism oriented education system in Hong Kong. It is clear from these text segments that there is a strong semantic overlap across the 4 text segments and it is not controversial to identify that topic 1 is about the Hong Kong elitist education system being the root cause of the HK Kids phenomenon. Table 3 presents the interpretations we arrived at on the five topics generated by fLDA. The topic words found in the frames are highlighted in colors according to their semantic classes (things in blue, features in green, events in red). Table 3: Interpretation of the topics identified from the_6B_6C discourse corpus. Topic 1 Interpretation of topic (The HK kids syndrome occur because of the elitist education system. Parents drill children for admission to famous schools. Parents pay too much attention to childrens school performance, ignoring the development of childrens integrity and independence.) (3 low means low self-care capacity, low EQ and low resilience to adversity. HK kids are helped by others in their daily life, not able to take care of themselves, with poor socialization capacity. ) , (Some HK children have mood disorders caused by overly protective parents who interfere with their childrens disputes with others, afraid of their kids being disadvantaged and always fighting on their kids behalf, causing their children to be unable to handle any adversity.) , (Parents should lead by example, and reflect on their own behavior and ideas, release their control as early as possible to let the children learn and experience failure so that they can build the capacity to solve problems.) (Parents hire maids to look after their children, making them rely on others for everything. Parents should be taught to train their childrens capacity to be independent. )
Towards the end of this unit, the teacher asked the students to summarize their discussions. The content of the students summaries can be used as a reference for validation of our interpretation of the topics. The following is a translation of the students discussion summaries: First, we think the HK Kids parents would not try to remedy their ways of disciplining their children because they think sending their children to prestigious schools and making them study all the time is correct, and do not pay attention to the real needs of their children. Second, we do not think that HK Kids are caused by the government enforcing the education system. Maybe parents are too busy with their work, and want to compensate for not giving enough love to their children by giving them more material comforts, and hiring maids to look after them. However, in this way, children cannot learn to become independent and they are used to rely on others to solve their problems. Some students consider the HK education system as fostering elitism which cause HK Kids Third, we think that in order to solve HK Kids problem, parents should pay attention to their children's habits and train them to become independent starting from childhood. The government should offer courses for children to improve their self-understanding, and to train them to manage their own emotions.
ISLS
447
ICLS2012
Fourth, the emergence of "Hong Kong Kids" is not a regional problem. Because of fast-paced lifestyle in Hong Kong, many parents do not have time to take care of their children. Another reason may be related to economic conditions. HK parents can employ maids to take care of their children leading to children's weak problem-solve skill. But in mainland China, children in poor families have to take care of themselves from childhood.) The first part of the above students summaries is similar to our interpretation of topic 1: parents overemphasizes childrens school education performance too much, and overlook the training of their childreds independence. However, in this part, students do not mention elitism. The second part of the summary is similar to our interpretation of topic 5: parents employ maids to take care of their children, resulting in their weak ability to be independent. Here, elitism is mentioned as a factor associated with the HK Kid syndrome, though not as central as picked up by the fLDA algorithm. The third part is similar to topic 4: parents should foster childrens independence. The childrens summary is written in more concrete terms than the interpretation presented in Table 3. The final part of the students summary is similar to topic 5: Due to the better economic conditions in HK, parents are able to employ maids to take care of their children. The content of this summary is also related to topic 1: in the frames of topic 1, the words poverty and countries turn out to be mentioned when comparing HK with other regions. Through this comparison, we find that our interpretation can cover most of the ideas students expressed in their own summary of discussions. On the other hand, we cannot find summaries with content similar to topics 2 and 3. So our approach may have potential in helping readers identify and interpret topics.
Topic trend analysis

One use of topic analyses is to compare the discussion topics across corpuses and over time. To do so, we first tag each note with relevant topics. A note is tagged with a specific topic if it contains at least 2 topic words of the topic. It is possible for a note to have all five or none of the identified topics tagged. Then we create for each corpus a visualization, called per-week topic trend graph, to show the number of notes containing each of the topics for each week. As shown in figure 2, this visualization of the 6B corpus reveals that the number of notes change significantly over time. In week 2, there are peaks for all topics. Afterwards, all topics have a low occurrence except for the none-of-the-topics category, which had another peak in week 6.
Note#oftopicsperweekforClass6B
60 50 Note# 40 30 20 10 0 1 2 3 4 5 6 Week 7 8 9 10 none-oftopics t2 t3 t4 t5 t1
Figure 2. Per-week topic trend graph of the 6B corpus. However, per-week topic trend graph of class 6B cannot reveal clear changes in topical focus when the level of discourse activity changes significantly over time. To provide a better representation of the change in topic focus in terms of the concerns expressed in notes, we make another visualization, per-N-notes topic trend graph, based on grouping every N=20 notes in order of created time. Per 20-notes topic trend graphs for all datasets are shown from figure 3 to 5. As shown in figure 3, the wave shape curves in the visualization reveal note numbers of topics have declined and increased for more than two times. By looking at the curves within first 60 notes, we find that topic 3 is more stable than other topics. This may indicate that the 6B students focus more on family problems than other causes of Hong Kong kids. Another interesting finding can be recognized at the point of 121-140 notes where the note number of none-of-the-topics category suddenly increases while other categories reduce a lot, indicating possibly some new topics may have emerged. Figure 4 shows the topic trend graph for the 6C corpus, which is quite different from that for the 6B corpus. From the point of 20-40 notes to the one of 40-60 notes, the note number of the none category goes down while the numbers of topical notes increase. This indicates different topical foci for the classes 6B and 6C in their discussions on the same theme.
ISLS
448
ICLS2012
Note#oftopicsper20notesforClass6B
16 14 12 Note# 10 8 6 4 2 0
Note:1- N:21-40 N:41-60 N:61-80 20 W:1 W:1-2 W:2 Week:1 N:81100 W:2 N:101120 W:2 N:121- N:141140 160 W:2 W:2 N:161- N:181180 200 W:2 W:2-3 N:201- N:221220 240 W:3-5 W:5-6 N:241- N:241274 274 W:6-7 W:7-10
t1 t2 t3 t4 t5 none
Figure 3. Per-week topic trend graph of 6B data set.

16 14 12 Note# 10 8 6 4 2 0 N:1-20 W:1 21-40 1 40-60 1-2 61-76 2-8 Note#oftopicsper20notesforClass6C t1 t2 t3 t4 t5 none
Figure 4. Per-week topic trend graph of the 6C corpus. We analyzed another set of corpus from a pair of classrooms from another school, 5A and 5B, and compare the topic trends between them based on their per-20-notes topic trend graphs, as shown in figure 5. It seems 5A had very few notes related topic 4 and 5. In contrast, 6B had very few notes on topics 1, 2 and 3. The ways the discourse developed in terms of the major topics are different as well. For 5A, topic 1 became a hot topic in the stage of 1-80 notes, then cooled down slowly afterwards, and finally disappeared. For 5B, topic 4 become a hot topic in the stage of 1-40 notes, became a rare one in the 41-80 stage, and then become a normal topic found in 25% of the notes in the later stage.
Note # of topics per 20-notes for Class 5A 20 18 16 Note# 14 12 10 8 6 4 2 0 20 40 60 80 100 120 140 160 180 t5 none t1 20 t2 Note# t3 t4 18 16 14 12 10 8 6 4 2 0 20 40 60 80 100 120 140 160 180 200 t2 t3 t4 t5 none Note # of topics per 20-notes for Class 5B t1
Figure 5. Per-week topic trend graph of the 5A (left) and 5B(right) corpuses.
Conclusion
We have demonstrated two promising applications of fLDA in analyzing CSCL discourse corpuses: topic extraction and topic trend analysis. Topic extraction starts with performing the fLDA algorithm to generate
ISLS
449
ICLS2012
frames for topics from a collection of notes. Notes can be tagged with different topics according to topic word frequency. Based on these frames, we can make meaningful interpretations for each topic through a few steps. Topic trend analysis can be done with per-N-notes topic trend visualization. Interpretation of this kind of visualization can potentially be used to help teachers to identify the focal concerns in students online discussions. More work involving the participation of teachers in evaluating the validity and usefulness of such analyses is necessary to determine the value of such a topic discovery and visualization tool.
References
Asuncion, A., Welling,M., Smyth, P. & Teh, Y. (2009). On smoothing and inference for topic models. Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, p.27-34, June 18-21, 2009, Montreal, Quebec, Canada. Blei, D. (in press). Introduction to probabilistic topic models. Communications of the ACM, Retrieved March 17, 2012, from http://www.cs.princeton.edu/~blei/papers/Blei2011.pdf. Blei, D., Ng, A., & Jordan,M. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993-1022. Blei, D., Griffiths, T., Jordan, M. & Tenenbaum, J. (2003). Hierarchical topic models and the nested Chinese restaurant process. Advances in Neural Information Processing Systems 16, Cambridge, Retrieved March 17, 2012, from http://books.nips.cc/papers/files/nips16/NIPS2003_AA03.pdf. Chang, J., Boyd-Graber,J., Gerris,S., Wang,C. & Blei, D. (2009). Reading tea leaves: How humans interpret topic models . Advances in Neural Information Processing Systems 22, Retrieved March 17, 2012, from http://books.nips.cc/papers/files/nips22/NIPS2009_0125.pdf . David, M., Wei, L. & Andrew M. (2007). Mixtures of hierarchical topics with Pachinko allocation, Proceedings of the 24th International Conference on Machine Learning, p.633-640, June 20-24, Corvalis, Oregon. Deerwester, S., Dumais, S., Landauer, T., Furnas, G. & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41(6): 391-407. Griffiths, T. L., & Steyvers, M. (2004). Finding scientific topics. Proceedings of the National Academy of Science, 101, 5228-5235. Jean, C. (1996). Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2), 249254. Kintsch, W. & van Dijk, T. A. (1978). Toward a model of text comprehension and production, Psychological Review, 85, 363-394. Minsky, M. (1975). A framework for representing knowledge. In P.Winston (Ed.), The Psychology of Computer Vision. New York: McGraw-Hill, pp. 211-277. Musat, C. C., Velcin, J., Trausan-Matu, S. & Rizoiu, M. A. (2011). Improving topic evaluation using conceptual knowledge. Proceedings of the 22nd International Joint Conference on Artificial Intelligence, Retrieved March 17, 2012, http://www.aaai.org/ocs/index.php/IJCAI/IJCAI11/paper/view/3010/3754. Newman,D., Han-Lau,J., Grieser, K. & Baldwin, T. (2010). Automatic evaluation of topic coherence. The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 100-108. June 2-4, 2010, Los Angeles, California, USA.. Rumelhart, D.E. (1975). Notes on a schema for stories. In: Bobrow D.G. & Collins, A.M. (Eds.) Representation and understanding: Studies in cognitive science, pp. 211-236. New York: Academic Press. Salton, G., Wong, A., & Yang, C. S. (1975). A vector space model for automatic indexing. Communications of the ACM, 18 (11), 613-620. Schank, R.C. (1975). Conceptual information processing. Amsterdam: North-Holland. Solso,R. L., MacLin, M. K., MacLin, O. H. (2004). Cognitive Psychology (7th Edition) . Allyn & Bacon. Steyvers, M. & Griffiths, T.(2006). Probabilistic topic models. In T. Landauer, D. McNamara, S. Dennis & W. Kintsch, (Eds.), Latent Semantic Analysis: A Road to Meaning. Lawrence Erlbaum. Wainwright, M. & Jordan, M. (2008). Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2), 1-305, December 2008. Wei, L. & Andrew, M. (2006). Pachinko allocation: DAG-structured mixture models of topic correlations. Proceedings of the 23rd International Conference on Machine Learning, pp.577-584, June 25-29, Pittsburgh, Penn. Zhang, Y., Li, Y. & Huang, R. (under review). fLDA: A Topic Model based on Frame Representation. Zheng, N. (2008). Discovering interpretable topics in free-style text: diagnostics, rare topics, and topic supervision. Doctoral dissertation, The Ohio State University, USA.
Acknowledgments
This study is supported by funding from the Information Technology Strategic Research Theme of the University of Hong Kong and the Chinese National Natural Science Fund.
ISLS
450
ICLS2012
Finding Voices and Emerging Agency in Classroom Learning

Tuck Leong Lee, Beaumie Kim, Mi Song Kim, Jason Wen Yau Lee Learning Sciences Lab, National Institute of Education, Nanyang Technological University 1 Nanyang Walk, Singapore 637617 tuckleong@gmail.com, beaumie@gmail.com, misong.kim@gmail.com, jasonleewy@gmail.com Abstract: This paper describes the development of a class as a learning community that transits from one established classroom practice that elevated monologic voices of authorities to another practice that encouraged dialogic participation. The project illustrated in this paper took a design-based approach in nurturing a learning community in a classroom, with which we re-designed the Secondary 1 Geography curriculum with teachers around six design principles. This paper proposes that the effort of a teacher, Nicole, helped learners find their inner persuasive voices and developed their conceptual agency in learning Geography. The transition of Nicoles Geography class is examined by comparing episodes from different periods of the project implementation, with the goal of finding out how students agency emerges from this transition. Nicole: 24 hours (referring to North Poles time without daylight). Ok, now looking at this, when do you think the date would be? Which month would this occurrence happen? Student 1: December Nicole: December? Why December here? Why, why not Student 2: Cause the video says so. Student 3: Because the book says so. (During a lesson on Day and Night, 21 Jan 2010) Nicole: We're going to hear Luke and John (students) explain to the class. Hopefully they are better teachers than Mrs. Toh (herself) who will not confuse you. Can? Student 4: What if they confuse us even further? (During a lesson on Atlas and Grids ,9 Feb 2010) The above excerpts are taken from the lessons conducted in two different days with a Geography teacher, whom we call Nicole (or Mrs. Toh) in this paper they provide a picture of the existing learning practices as we began the project. In the first excerpt, Nicole tried to open a dialogue with the students on why North Pole does not receive daylight for 24 hours, during particular time of the year. However, students closed the dialogue (up to this point) by answering that they already accepted this factual information from authoritative sources (i.e., video and textbook). In the second excerpt, Nicole invited two students to provide their explanations on how longitudes differ from latitudes. That had the potential to open up a critical discourse. However, one student expressed his reservation toward this practice as he thought that multiple explanations might be more confusing. These two excerpts demonstrate that students were comfortable with monological voices of authorities (i.e., teacher, textbook, and media) and were still dubious of a dialogic practice in their learning. Nonetheless, drawing from Bakhtinian thought, the language of the authoritative voice is half someone elses until it is incorporated with the hearers own intentions and accents (Bakhtin, 1992, p. 294) or, in Nicoles own words: until they are internalized and [not just] regurgitated [from] the textbook.
Preparing the grounds for dialogic competencies

Voyage to the Age of the Dinosaurs is a design-based research project that has grown over the last four years It was implemented around a Secondary One (for students aged 11-12) Geography curriculum, covering topics from the Earth Sciences. We started with an informant design approach (Kim, Tan, & Kim, 2011a; Scaife, Rogers, Aldrich, & Davies, 1997), with which we conducted a series of five workshops over the years 20072009 to develop an educational game. During our third year, we worked closely with teachers from one of the collaborating schools. Aims of the curriculum design were: (a) to help learners understand and advance their own resources (i.e. experiences and ideas) as epistemic communities and (b) to engage learners in meaning-making and experiential learning activities that integrate a computer game and other tools developed through the informant design workshops. With these aims, we set out principles for our design of curriculum to prepare a space where a dialogic discourse might naturally emerge. These principles are: (i) Engage students in group activities to develop epistemic communities; (ii) start with open-ended questions that could encourage the use of their preconceptions and their own approaches; (iii) engage them in activities that provide relevant experience using tools and hands-on activities; (iv) require them to produce group or individual artifacts, thus encouraging their ownership
ISLS 451
ICLS2012
of the process and product; (v) develop a sharing mechanism of the class, thus furthering their conversations; and (vi) engage them in consolidation and linking activities to understanding connections and gain confirmation about their ideas. In centering the interaction spaces as sites of learning, we hoped to prepare the grounds to allow an emergence of a dialogic space, signaled by indicators that characterize such a space. According to Kubilis (2005) application of Bakhtinian thoughts to an education context, a transformation towards a dialogic discourse is seen when the voice of the textbook gives way to shared experience, shared knowledge and common evaluation of the [learning] situation (pp. 513-514). In such a space at a micro-interaction level, words exchanged are polyvalent signs generated through attempts towards intersubjective sharing of meanings; and at a macro level of class participation, inquiry is carnivalesque, entered in a spirit of enterprise, with freer and more active exchange of ideas (pp. 529-530). Such an attempt in encouraging a dialogic discourse is consonant with some qualities of an effective education space, which Dhingra (2008), in his review paper on science education, argued should include activities for practice of skills and personal meaning making, validation of student funds of knowledge and authoring process, and exchange of students funds of knowledge (p. 141). A dialogic discourse also resonates with sound pedagogic principles. While monologic discourse is important for transmission of knowledge, it provides no guarantee against misinterpretation and other communicative failures; it is a dialogic participation that encourages a state of intersubjectivity, where shared perspectives can clarify misunderstanding (Wells & Arauz, 2006, p. 393-395). We believe that changing the classroom culture towards a dialogic practice would not happen with just few lessons. As such in our implementation, instead of using such fundamentally different design principles in few lessons, we decided to work alongside the teacher for the entire duration of the Secondary 1 geography curriculum, which covered half a year. Kress (2003) puts forward a theory of constant transformation of both resources and subjectivity, whereby learning is perceived as a change in resources resulting from active transformative engagement with an aspect of the world. The learner (the sign-maker) constantly transforms the set of resources and herself, at the same time building funds of knowledge (Moll, Amanti, Neff, & Gonzalez, 2001) in the classroom which bridge both school and out-of-school practices. Any preparation towards a dialogic discourse then needs to be patient with the sedimentation of actions, resource use, and shifting subjectivities which together constitute this transformation towards a dialogic space. The question for us to address in this paper then is how students develop conceptual agency within this emerging practice partly shaped by our design principles, thus initiating actions that recursively transform both resource use and learners roles and identities. Our construct of agency aligns with a body of work that views agency as emergence from activity systems (Dama, Kirschner, Andriessen, Erkens, & Sins, 2010; Gresalfi, Martin, Hand, & Greeno, 2007; Rainio, 2008). There is another significant body of literature (Adler, Rougle, Kaiser, & Caughlan, 2003; Greeno & van de Sande, 2007; Linn, 2006; Martin, 2004; Shanahan, 2009; Stefanou, Perencevich, DiCintio, & Turner, 2004) that locates agency primarily in a learners self-direction and autonomy, which is seen in a learners ownership of the learning process, using ones own range of cognitive skills, and taking actions towards collective goals. However, Rainio (2008) argues that that agency cannot be considered a stable property or an attitude of an individual (p. 116). Dama et al (2010) reasons that agency progressively emerges out of interactions between participants and their activities (p. 115). Gresalfi et al (2009) dismiss distinguishing between whether people have or lack agency, noting that both active and passive participations are equally agentive gestures (p. 53). What is important, for Gresalfi et al (2009), is not whether the students exercise agency, but how they do so in contexts that provide sets of affordances, with their implied values of what constitute competent participation. Nonetheless, both bodies of literature provide some insight as to the kinds of agency students might take on which serve as useful indicators for us to locate agentive actions, for us to examine how these actions constitutes the emergent classroom practice and is in turn supported by the latter. These indicators include acting with appropriate structures of dialogic argumentation (Andriessen, 2006); participating with constructive listening that allows conceptual changes within the learner (Greeno & van de Sande, 2007); using process-related activities that support, direct, and structure knowledge-related actions (Dama et al, 2010, p. 149-152), playing identity roles that position each learner in distinct ways in the learning environment (Anderson & Zuiker, 2010), and managing ones own disposition and attitude appropriate to a specific learning context (Rainio, 2008). This paper contributes to this conversation over agency by examining the relationships between some of these agentive actions and a dialogic space that is shaped by our design principles.
Understanding Agency in a Dialogic Community

The activities in all of Nicoles class were recorded with video and audio recording devices. The classroom interactions were transcribed vertabim. We also wrote field diaries and collected various class artifacts, such as drawings the students produced, photographs of diagrams on the board and students journals these allowed us to interrogate our observations. We selected two significant class lessons on the 21st of January and the 13th of
ISLS
452
ICLS2012
April, for discourse analysis (Gee, 2005) in this paper, based on how they represented a fair range of our design principles and how they instantiate the classroom practices at the beginning of our implementation and towards the end. In this section, we first describe what happened in each of the lessons, briefly account for changes between these two lessons, and then describe the emergent classroom discourse in the latter lesson. In the January lesson, Nicole first reviewed the previous lesson on seasons and diurnal changes. Then, Nicole led a discussion of two Thinking Questions, the open-ended questions that the researchers prepared for classroom discussion. The questions for that day were: (i) Why do people draw maps? and (ii) Why are there different types of maps?. Finally, the students were given a task of drawing, in groups, a map showing the direction from one location in school to another. The class ended with the students evaluating each others work. In the April lesson, Nicole reviewed the last lesson where they learned that atmospheric temperature decreases with increasing altitude. What they did not previously discuss was how that came to be that gap served as the Thinking Question for the first part of this class. The students speculated on some possible reasons. Nicole asked the students to draw schematic models to support their answers. One of the students presented his schematic drawing to the class, which in turn led to an intense discussion that, in the words of one student, was more like a debate than a question and answer session. With the discussions completed, Michele moved to the next topic on the hydrological cycle. Over a schematic diagram of the water cycle, the students explained the different elements in the cycle, filling in details of relations among the elements, which were absent in the schematic representation.
Learning, from in Unison to with Polyphonic voices

Between the lessons selected for our analysis, we see a transformation of a classroom practice. Table 1 below summarizes the changes that took place in the classroom practices between the two lessons. Column (A) identifies the design principles that provide our categories for comparison. These design principles had been earlier expanded upon in this paper, in exact order. Column (B) contains descriptions that qualify how the design principles were translated in a lived practice in the January class. Column (C) contains descriptions that qualify how the design principles translated into practice in the April class. Table 1: Characteristics of the classroom practice in January and in April No. (A) Design Principles (1) Working within Groups (2) Thinking Questions (3) Activities (4) Artifacts (5) Sharing Mechanism (6) Linking Ideas (B) Class on 21 Jan 2010 (1B) Cooperatively/ Alone (2B) Researcher-generated (3B) Researcher generated (4B) Researcher generated (5B) Group leader/ Vocal member (6B) Predetermined by curriculum (C) Class on 13 April 2010 (1C) Working with distributed responsibilities and conversing within/ beyond (2C) Student generated (3C) Emerged in the classroom (4C) Ideas and diagrams emerged (5C) More students volunteering (6C) Teacher-mediated/ Student mediated
Between the classes in January and April, we can identify a few changes. First, the nature of the students participation changed. Table 2 describes the changes in the students participation, taking into account only utterances from the students, substantial enough to describe a concept, offer an explanation, or extending an elaboration. Table 2: Comparison of nature of class interaction No (7) (8) (9) Types of Participation Total number of substantial contributions from students Total number of students who spoke substantially of their own accord Number of utterances from most frequent speaker 21 January 14 6 7 13 April 22 16 3
The pattern of how students participated in class changed in the following ways: (i) more ideas were shared within the class discussions (7), more students volunteered to contribute (5 and 8) and vocal members had somewhat restrained their active participation, allowing other members to contribute (5 and 9). Furthermore, the patterns of interactions within smaller groups also changed. For example, in the January activity in which students were tasked to draw a map, a student named James took most of the initiatives for the group work, analyzing the task, giving instructions, evaluating the process even though he was not the nominated group leader. The rest of the members were simply cooperative with James (1B), who also happened to be the dominant speaker at the whole-class interactions (9). On the other hand, in April, students were asked
ISLS
453
ICLS2012
to choose a representative to speak for each group. James did not volunteer while one student in the group encouraged another member to represent them. Group members were shifting the procedural responsibilities among themselves (1C). Second, Nicole exercised greater flexibility over how she used our lesson designs this had the effect on allowing her to attend to the conceptual turns in the class discussions, and enacting a platform for the students to present their views. For example, in the January class, Nicole followed the lesson plans very closely down to the exact procedures of a group activity (3B) of drawing a map (4B). She also used the exact Thinking Questions (2B) we provided. On the other hand, in the April lesson, Nicole began with her own inquiry thread that began with a gap from a previous discussion, asking how temperature is affected by altitude. Noting that the students needed first to consolidate the ideas on their own, she asked the students to draw schematic diagrams to illustrate their explanations (3C and 4C). A great amount of discussion ensued in which she allowed a broad range of answers. Thereafter, she helped the students to consolidate their own Thinking Questions, queries that arose from the class discussion (2C). Third, the movement from using prescribed Thinking Questions towards using open-ended questions that arose from the situated discussions encouraged students ownership over the dialogues, and that in turn, generated a huge amount of discussions. Consider the earlier discussion over one Thinking Question in January: Table 3: Extract of a class discussion over the Thinking Question Why do people draw maps? (10) Nicole Can I have a volunteer to share with us, what you have discussed with your (.) Er, group members. Why do people draw maps? (11) James It's to prevent them from getting lost. ((The students start talking within the group without volunteering)) (12) Nicole Bryan. Hello, I asked for a volunteer to give the answer, why are you talking among your groups. That means you finish your discussion already. (.)Thank you. Now, we're all ears for you, Bryan. (13) Bryan People draw map so that they can navigate easier around (.) the places that they are not very sure of. (14) Nicole Thank you very much. Anyone wants to add to Bryan? He mention so that people can navigate ((The class is interrupted by an announcement)). Alright, now. Bryan mentioned something very interesting, it's about using maps for navigation. Can somebody explain what navigation means? (15) James Moving around? (16) Daniel Moving around. (17) Richard To find their way. (18) James To move around. (19) Alfie Find their way. (20) Nicole To find their way. Find their way where? The contributions from the different students (11, 13, 15-19) converged around a simple idea of a use of a map for navigational purposes. The discussion over that Thinking Question soon closed after the excerpt, with many students, answering in chorus to their destination. Nicole commented that was interesting, and moved to the next Thinking Question. Here, the students were acting on the learning process with supportive and constructive initiatives (Rainio, 2008, p. 123), responding to Nicoles questions and probes; however, from a dialogical perspective such an agency was counteractive towards broadening the dialogic space. On the other hand, in the April lesson, the students contributions both diverged and converged in cycles, thus opening up the dialogic space. Table 4 is a simplified interaction map of the key ideas exchanged publicly in the April class. In that table, Nicole facilitated a discussion where cumulative contributions are shown by increasing indentation in the subsequent rows (e.g. 22-24) and divergent contributions are shown by a change in indentation direction (e.g. 41-44). Descriptions in italics either contextualize the events, or describe actions that were not part of the flow of the class dialogue. Table 4: Simple Interaction Map of a discussion over a Thinking Question on 13 th April Event Class in dialogue (21) Nicole poses the Thinking Question: How is the temperature affected by altitude? Projection of an isotherm image, illustrating that the temperature of the atmosphere decreases (22) with height (23) Oliver: As you go higher, it gets colder.
ISLS
454
ICLS2012
(24) (25) (26) (27) (28) (29) (30) (31) (32) (33) (34) (35) (36) (37) (38) (39) (40) (41) (42) (43) (44) (45) (46) (47)
Lijian: The air gets thinner Unidentified: There is less carbon dioxide. Nicole: It's not so much the composition of the air but more because of the air molecules... Closer to the ground the air is denser. Nicole asks the students to draw representations of their understanding. Nicole then asks Weiyang to share his diagram on the board. Weiyang shows a class a diagram he draws, illustrating densely packed molecules at lower ground and loose molecules at higher ground. He labels the diagram: "Air is denser as altitude increases" Nicole corrects Weiyangs label to read: "Air is denser as molecules are closer" At a small group discussion, Erickson asks Nicole: Dense air sinks not rise. How can it be dense air at the top?" Group discussion and individual seatwork ends here. Nicole continues with the class lesson. Nicole: Erickson has an interesting question. Erickson: When air is denser, it traps more heat. Where air is less dense, it will trap less heat and hence it is colder. Nicole: When hot air is closer to the ground, it gets heated. Won't the molecules be further apart? Unidentified: Air will expand when it is heated, Nicole: What do you understand about hot air when it is heated up? Unidentified: When it rises, there is lesser air molecules. On the 31st March, the class discussed the movement of wind: How air flows from a high pressure to a low pressure. Richard: In a previous lesson on air pressure, we learned that hot air rises. So why is it hotter at the base of the mountain? Nicole: If hot air rises, why is it colder up there? Minghan: When hot air rises... the air molecules are more spaced out so it loses all heat. That's why it becomes cold air... cold air sinks... Carlo: At higher ground, there is snow. At lower, there is rain. At higher ground, snow does not melt, that's why it is cold. Sony: Is Carlo saying that there is hot air in the higher atmosphere to melt the snow? Carlo: No. When snow passes through the atmosphere, it melts. Sony: Why does it snow? Thinking Question 1: Why does air become less dense at higher ground? [Ryan] Thinking Question 2: Shouldn't the temperature be higher at higher ground since we are closer to the sun? [James] Thinking Question 3: If hot air rises, why is it colder at higher altitude? [James] Utterances are paraphrased for clarity
The interaction map illustrates how the process of linking ideas were no longer constrained by an adherence to the research design (6B), but mediated by Nicole (6C), as she arbitrated the exchanges of ideas. When the students took ownership of their discussion, we find them not only activating cognitive tools of engaged reasoning such as making claims and challenging them (24, 25, 30, 32), looking out for implications and inferences (36, 42); but we also find them doing more than that they drew on knowledge from an earlier lesson (37-38) and even from another discipline (34, the student learned about the expansion of air in General Sciences). Furthermore, they even attempted to look at air convection from a larger system of weather phenomena (42, 44). One very important emergence from such an ownership of the knowledge collaboration process is that some students took on Nicoles role as a knowledge mediator (6C). At event (38), we find Richard assembling together two seemingly conflicting ideas (i) that hot air rises, from an earlier class; and (ii) it is hotter at the base of the mountain, from the present discussion. At a next lesson, Richard also noted down two conflicting claims offered in the class discussion: (i) that air near the ground is denser because of gravity and (ii) that air molecules are loose near the ground because of the expansion due to heat. Gresalfi et al (2009) explained that participation structure and tasks partly affords learners' agency in a particular discourse (p. 52). Nicoles modeling of a mediating role, and the divergent contributions from class members might have afforded Richard the resources to imitate this mediatory role. This role has an important place in collaborative learning because it allowed a temporary synthesis which, while never fully the answer, was a genuine gesture towards a quest for deep learning.
ISLS
455
ICLS2012
Polyphonic Voices, Symphonic Spaces, Authentic Sounds

A question then remains: How did the emergent practice take on the quality of a carnival-like exchange of ideas (Kubli, 2005), and afforded students like Richard to act with such agency towards dialogic discourse? We propose three important reasons that allowed students to exercise cognitive autonomy (Stefanou et al., 2004): (i) the role of the students own inner persuasive voices; (ii) Nicoles positioning of students as competent participants in a dialogic space; (iii) ideas surfacing from students learning from each others inner persuasive voices; and (iv) the emergence of parallel dialogic spaces that ran alongside the whole-class discussions. First, we will consider the interplay between the Nicoles authoritative voice and the students 'inner voices of their own knowledge artifacts. In the following extract from the April class, Nicole led a discussion on how the rainforest is an important source of water in the hydrological cycle. She shared a perspective that is constructed out of textbook facts: Nicole: The other way is that it goes into the sea or into the river. This leads to something very important. You did the chapter on tropical rainforest right? Did you, did you find out along the way that tropical rainforest is an important resource (.) of water? Student: Yes. Alfie: When you are running through a rainforest, you find that it is very humid. Nicole: Exactly. When you were running into the rainforest, you find that it is very humid right? Alfies contribution came from his own experience, a voice he has found persuasive in [his] previous thinking (Alvermann, 1998 p. 292). The embodied experience of finding the rainforest humid was used as a knowledge tool to understand Nicoles authoritative claim, thus internalizing her voice with his own intention and accents (Bakhtin, 1992, p. 294). And when Nicole repeated Alfies contribution, her voice is enriched with not just the traces of Alfies experience, but also the isomorphic traces of the class members shared experience of how humidity felt to the individual students. Second, Nicole used her authoritative voice to emphasize every contributions count, thus positioning the students as co-contributors. This led to an attitude change in the students that supported dialogic exchanges. In the April class, she commented that an idea Erickson made in his small group was interesting (33) and encouraged him to make his views public to the class, effectively using a gesture of positioning (Wagner, 2008., p. 145) to mark Erickson as a competent member of a discourse community. Even though she exercised her teachers monologic authority to constrain the direction of the discussion (31, 33, 35) and corrected misconceptions at times (29), these gestures are retrospectively heteroglossic, in that they open up more spaces for dialogues (Wagner, 2008. p. 155). In turn, Nicoles positioning enacted a new space that favoured knowledge experimenting. Consider how Minghan supported Ericksons contribution by saying to him even when its wrong, its interesting. The emerging dialogic practice changed an older discourse of what counted as competent participation (Gresalfi et al., 2009) to one that is inclusive of all sorts of interesting ideas. In this new dialogic space, Minghan could draw on his existing social role of a friend and sidekick to Erickson as a resource (Anderson & Zuiker, 2010) to initiate his own process-related action (Dama et al., 2010) to direct Erickson towards a goal of enriching the dialogue from Ericksons own voice. Third, students in voicing their inner persuasions that is articulated as interesting contributions regardless of whether they are right or wrong allowed ideas to surface not from authority but from robust debates that enabled students to learn from sets of others internally persuasive voices. This is illustrated in the next extract in Table 5, which took place at event (32). Small group exchanges are italicized. Table 5: A small group conversation around Ericksons contribution (48) Erickson When the air is denser it attracts more heat so at the sea level it will be hotter but then when but then when the air is less dense it attracts less heat so it will be cooler. (49) Weiyang Cos its denser it don't trap heat. (50) Ching (51) Nicole (52) Ching Why its denser? I think Daniel don't understand. Anyone understand what Erickson is saying? Because the sun will reach the mountain first what.
(53) Weiyang Yea reach first but its // (54) Ching I don't really get you// (55) Weiyang When the air molecules are spaced out so it doesnt trap heat Ericksons perspective (48) of air that is less dense [attracting] less heat clashed with Weiyangs perspective (49) that it is less dense air that trap[s] heat. Nonetheless, they both share an implicit conceptual
ISLS 456
ICLS2012
schema (Greeno & van de Sande, 2007, p. 12) of air as container of heat except Erickson conceived of the less dense air molecules as having less holding power on heat, as an object, between them; and Weiyang conceived of less dense air molecules with spaces between them to hold more heat. Eventually, Weiyang accepted (55) Ericksons perspective. Even though none of them were scientifically correct in their explanations, the dialogue Erickson shared with his friends allowed him to activate the schema of air as a container in language, and used the affordances of such a conceptual schema as a knowledge tool, to enact his own conceptual growth (Greeno & van de Sande, 2007, p. 12) and arrive at a tentative conclusion on why less dense air is cooler. Finally, in the emergent practice the April class, there were smaller dialogic spaces that ran alongside the public exchanges, while students waited for the next student to speak, or even while the teacher was teaching. This allowed room for local exchanges and students could still continue to converse within their group, or even across their own group (1C). Prior to the next extract, the class were discussing why, given that hot air rises, the higher altitudes are colder. Richard suggested looking at snow but never had a chance to complete that idea. He turned to across to Erickson in another group (1C) Richard (to Erickson): You dont understand. What Im saying that I have a block of ice. I - I heat it up, become water right? Lets say for example. A big chunk of snow among the clouds because of the higher atmosphere thats why the hot air from the bottom rises up so it melts the snow and becomes rain.

We have described Nicoles class as a learning community that transited from one established classroom practice that elevated monologic voices of authority to another practice that encouraged dialogic participation. In conclusion, we want to first frame our research findings within a larger context of how the lived dialogic space emerged together with transformed participations, both from the Nicole and her students. In our earlier paper (Wang, Kim, Lee, & Kim, 2011), we examined how Nicoles role in the emerging epistemic community developed from that of a (i) teacher, to (ii) a facilitator, and then finally to (iii) a co-learner. Dhingra (2008) identified the teacher-roles of a listener, interpreter, cultural broker, facilitator, and science knowledge resource person as highly suitable to allow students to emerge as co-participants in an inclusive and respectful community of practice (p. 125). We would like to add that it is a shift from a teacher to a co-learner that allowed Nicole an access to a wider range of roles, accumulated in the transition, to support student dialogue. At the same time, the students participation is characterized by an expanded use of the funds of knowledge (Moll, Amanti, Neff, & Gonzalez, 2001) that is counted legitimate in the emergent practice. Some of these funds of knowledge came explicitly from other lessons, other academic disciplines, and from their own experiences; others came as implicit conceptual schemas (Greeno & van de Sande, 2007) that can be used as knowledge tools. We find it useful to draw on Linns (2006) identification of four learning trajectories: (i) strategization of learning that maximizes learning efficiency by accepting definite authoritative voices in the discipline; (ii) conceptualization of a broad range of ideas about a phenomenon, (iii) contextualization of ideas within a specific learning context, without seeking other connections out of that context; and (iv) experimentation of new ideas and testing them out across different learning context. While Linn (2006) did not intend these trajectories to be sequenced developmentally, we find that from the point of view of developing a dialogic space that is signaled by a profusion of authentic voices acting in collaboration sequencing them in this order characterizes the development of the students participation, as students increasingly become more confident in drawing from a wider range of knowledge sources into the learning of Geography. The change in the students participation together with Nicoles shifting position in the class worked in tandem to construct a different system of competence, which in turn reifies a dialogic participation (Gresalfi et al, 2009). However, there is one very important aspect that this research into agency within a dialogic participatory space has not addressed. While we have focused on primarily on the spoken interactional discourse, we have not examined the roles of other components of the activity system (Gresalfi et al, 2009) of this dialogic site components that are important in our design principles. Some of these components include the affordances of specific shared activities, including the manipulation of knowledge representations and the making of artifacts (Kim, Wang, Tan, Kim, Lee, & Pang, 2009); the actions that emerge from the development of sharing mechanisms, such as information uptakes and the types of evaluation offered that extend discussions (Wells & Arauz, 2006); and the high cognitive demands of ideas presented in linking and consolidating activities, which require a learner to bring together different inputs and performing further analysis or synthesis (Wells & Arauz, 2006). Further work needs to address this gap and is the future direction that we will undertake.
ISLS
457
ICLS2012
References
Adler, M., Rougle, E., Kaiser, E., & Caughlan, S. (2003). Closing the gap between concept and practice: Toward more dialogic discussion in the language arts classroom. Journal of Adolescent & Adult Literacy, 47(4), 312-322. Anderson, K. T., & Zuiker, S. J. (2010). Performative identity as a resource for classroom participation: Scientific shane vs. Jimmy neutron. Journal of Language, Identity & Education, 9(5), 291-309. Andriessen, J. (2006). Arguing to learn. In R. K. Sawyer (Ed.), The Cambridge handbook of the Learning Sciences. (pp. 443-60). Cambridge University Press. Alvermann, D. E. (1998). Reconceptualizing the literacies in adolescents' lives. Mahwah, N.J.: L. Erlbaum Associates. Bakhtin, M. M. (1992). On dialogism and heteroglossia (the other(s)' word). In M. Holquist (Ed.), The dialogic imagination : Four essays. Austin: University of Texas Press. Dama, C. I., Kirschner, P. A., Andriessen, J. E. B., Erkens, G., & Sins, P. H. M. (2010). Shared epistemic agency: An empirical study of an emergent construct. Journal of the Learning Sciences, 19(2), 143-186. Dhingra, K. (2008). Towards science educational spaces as dynamic and coauthored communities of practice. Cultural Studies of Science Education, 3(1), 123-144. doi:10.1007/s11422-007-9077-6 Gee, J. P. (2005). An Introduction to Discourse Analysis: Theory and Method. London: Routledge. Greeno, J. G., & van de Sande, C. (2007). Perspectival understanding of conceptions and conceptual growth in interaction. Educational Psychologist, 42(1), 9-23. Gresalfi, M., Martin, T., Hand, V., & Greeno, J. (2009). Constructing competence: An analysis of student participation in the activity systems of mathematics classrooms. Educational Studies in Mathematics, 70(1), 49-70. doi:10.1007/s10649-008-9141-5 Kim, B., Tan, L, & Kim, M. S. (2011). Why we should design educational games with learners: The affordances of informant design. Proceedings of the 2011 International Conference on Computers in Education. Chiang Mai, Thailand: Asia-Pacific Society for Computers in Education. Kim, B., Wang, X., Tan, L., Kim, M., Lee, J.W.Y., & Pang, A.L.H. (2009). Designing with Stakeholders for Learning Innovations: Voyage to the Age of Dinosaurs. Paper presented at the Annual Meeting of American Educational Research Association, San Diego. Kress, G. R. (2003). Literacy in the new media age. New York: Routledge. Kubli, F. (2005). Science teaching as a dialogue Bakhtin, Vygotsky and some applications in the classroom. Science and Education, 14, 501-534. doi: 10.1007/s11191-004-8046-7 Linn, M. C. (2006). The knowledge integration perspective on learning and instruction. In R. K. Sawyer (Ed.), The Cambridge handbook of the Learning Sciences. (pp. 243-64). Cambridge University Press. Martin, J. (2004). Self-Regulated learning, social cognitive theory, and agency. Educational Psychologist, 39(2), 135-145. Moll, L. C., Amanti, C., Neff, D., & Gonzalez, N. (2001). Funds of knowledge for teaching: Using a qualitative approach to connect homes and classrooms. Theory Into Practice, 31(2), 132-141. Rainio, A. P. (2008). From resistance to involvement: Examining agency and control in a playworld activity. Mind, Culture, and Activity, 15(2), 115-140. Scaife, M., Rogers, Y., Aldrich, F., & Davies, M. (1997). Designing for or designing with? Informant design for interactive learning environments. In Proceedings of the SIGCHI conference on human factors in computing systems. Shanahan, M. -C. (2009). Identity in science learning: Exploring the attention given to agency and structure in studies of identity. Studies in Science Education, 45(1), 43-64. Stefanou, C. R., Perencevich, K. C., DiCintio, M., & Turner, J. C. (2004). Supporting autonomy in the classroom: Ways teachers encourage student decision making and ownership. Educational Psychologist, 39(2), 97-110. Wagner, D. (2008). "Just don't": The suppression and invitation of dialogue in the mathematics classroom. , 143-157. Wang, X., Kim, B., Lee, J.W.Y. & Kim, M.S. (2011). Developing an epistemic community in the classroom as teacher development. In S. Barton et al. (Eds.), Proceedings of Global Learn Asia Pacific 2011 (pp. 1272-1281). Wells, A. H., & Arauz, R. M. (2006). Dialogue in the classroom. The Journal of the Learning Sciences, 15(3), 387-428. doi:10.1080/0023970660855708
ISLS
458
ICLS2012
Processes of decision-making with adaptive combinations of wiki and chat tools

Kate Thompson and Nick Kelly, CoCo Research Centre, The University of Sydney, Australia Email: kate.thompson@sydney.edu.au, nick.kelly.mail@gmail.com Abstract: Debate in the field of event-based analysis of collaborative process data has focused on how best to account for the temporal nature of processes. We reanalyzed data from an online learning environment that included multiple tool use (chat and wiki) in synchronous and asynchronous collaborative settings. In this paper we modeled decision-making, and used time to identify four patterns of tool use to make decisions: single chat, multiple chat, chat/s and follow-up wiki, and integrated. Each pattern of tool use was associated with a unique process model, which indicated the affordances of each tool for that particular activity. The examination of the integrated use of tools, and the integrated decision-making process in a collaborative online setting has implications for designers, instructors and researchers.
Introduction
The field of event-based analysis of processes of collaboration has opened up since Reimanns seminal paper (2009). New methods of analysis (Kapur, 2011; Reimann, 2009; Wise & Chiu, 2011a, 2011b) have been suggested, and critiqued (Goggins, Laffey, & Amelung, 2011). In this paper, we suggest that prior to making a decision about the method of analysis, the role that time plays in the composition of the dataset is important to consider. We build on Reimann, Frerejean & Thompsons (2009b) research by taking their dataset (chat data for a month-long collaborative exercise) and incorporating data from the wiki that was used by the groups. Rather than assuming that linear, or even cyclic processes, that apply to the entire dataset, we identified a number of threads, related to combinations of tool use, and examined the processes involved in decision making for each of these. Unlike the original study, we use first-order Markov models to derive the process models. This study has implications for the design and management of collaborative learning environments, as well as making a contribution to the development of methods in this emerging field of research.
Background
With the ability to collect more complex datasets (for example, log file data, time sequenced video and conversations) has come methodological advances. Research into the processes of learning have included perspectives such as argumentation (Weinberger & Fischer, 2006), knowledge building (Aalst, 2009), decision making (Reimann, 2009) or knowledge construction (Wise & Chiu, 2011a). Research has addressed both synchronous (Ding, 2009) and asynchronous (Weinberger & Fischer, 2006) collaboration, but few address both (Thompson & Kelly, accepted). The units of analysis differ across many of the studies, depending on the research question. As yet, there is not a clear framework in which to place further research. While both synchronous and asynchronous online learning spaces have been researched, few have studied the combination of tools, and even fewer in association with the processes of collaboration. Zenios and Holmes (2010) focused on the social affordances of tools such as chats and wikis. They described a process of collaboration in which students used the online chat tool to reflect on, discuss and modify what they had written in the wiki. They discussed the importance of dialogue before and after the development of collaborative wikis and suggested that chat was needed in order to produce the ideas, whereas the wiki acted as the organisational memory for the group. They concluded that the combination of the chat and the wiki were necessary for epistemic tasks to take place, and group cognition and new knowledge to be developed. Making use of the large amounts of data produced during online activities, such as that produced with chat and wiki, is something that has received significant attention in recent years (Reimann, Frerejean, & Thompson, 2009a; van der Aalst & Weijters, 2005). Given a set of states (for example, learning phases, decision stages, steps interacting with a computer model) that changes over time for a group of students, what can be mined from this data? Techniques that have been applied successfully for identifying patterns within state transitions include heuristics mining for dependency relationships (Reimann, et al., 2009a; Weijters, Van der Aalst, & Medeiros, 2006), first-order Markov models for state transitions where symbols in the dataset and states of the target are isomorphic (Eddy, 1998; M. S. Poole & Roth, 1989a), and hidden Markov models for determining hidden states from a set of symbols (Jeong, Biswas, Johnson, & Howard, 2010; Shih, Koedinger, & Scheines, 2010; Southavilay, Yacef, & Calvo, 2010).
Methods and analysis

The data collection method and the sample have been described in other papers (Reimann, et al., 2009a). Briefly, data were collected using a tool called Snooker (Ullman, Peters, & Reimann, 2005) from a group of
ISLS
459
ICLS2012
graduate students who worked on a design task of adding instructional design features to an existing system dynamics model, without meeting face to face (Reimann, Thompson, & Weinel, 2007a). Students were expected to coordinate their own work for this task, which required frequent decision-making about the task and managing the group work. We focus on just one of the groups in this paper, composed of three female students and one male student. The learning environment combined synchronous and asynchronous communication components. Students had access to both a chat tool and a wiki. This paper builds on previously reported studies (Reimann, et al., 2007a; Reimann, Thompson, & Weinel, 2007b). In particular, the analysis of the processes of decision-making were reported on in Reimann et al. (2009a). Process modelling and mining were used to produce a generalised model group of decision-making in a chat. Process was viewed as a sequence of events, which placed it somewhere between an atomistic and holistic granularity of process, with an event-oriented unit of analysis, rather than a variable-oriented view. The DFCS has seven main categories: 1) problem definition; 2) orientation; 3) solution development; 4) non-task; 5) simple agreement; 6) simple disagreement; and 7) implementation. Category Solution Development (3) has five subcategories: 3a solution analysis, 3b solution suggestions, 3c solution elaboration, 3d solution evaluation, 3e solution confirmation. Poole & Roths (1989a, 1989b) model was used, stating that groups work on multiple threads, or decisions at the same time, and that the decisions are mixed together in observable behaviour. The DFCS was selected as it includes problem definition, orientation and solution development. In other work (Kennedy-Clark, Thompson, & Richards, 2011) a modified version of the scheme was adopted, based on the availability of data that showed implementation of decisions, and we have included this additional code in our analysis. The 2009 study concluded that the decision process was unstructured, complex, and cyclic, but that within that, differences could still be identified in the two groups of students. In this paper we focus on just one of the groups, which had a more pronounced cyclic pattern to their decision-making. We have included the wiki data in our analysis in order to examine the way in which learners combined their use of the tools.
(a) Heuristics Miner (Reimann et al., 2009) (b) First order Markov model Figure 1: Process models based on original data Part (a) of Figure 1 shows the original model presented in 2009, based on an analysis using a heuristics miner. As described in Reimann et al. (2009), this showed that the group began with a period of problem definition, followed either by discussion of solution alternatives, group orientation, and discussion of solution criteria. The dependencies scores indicated that problem definition statements were most often followed by extensive periods of group orientation. The group then discussed either solution analysis, or solution alternatives, generally
ISLS 460
ICLS2012
followed by decision confirmation, although sometimes it was followed by solution elaboration and evaluation. If there was disagreement after deciding on criteria, the group often started from the top again. The confirmation of solutions was followed by positive evaluation or agreement. In some cases the positive evaluation statements triggered discussion of the criteria again. The second model (b), is the process model derived from the application of a first-order Markov model. The dataset can be considered as a sequence of states (the eleven listed earlier) ordered in time. The set of states {S1, ..., Sk} occur as the series {x1, , xn} where k = 11, the number of states, and n = 1828, the number of coded entries. A first-order Markov chain, is created by calculating a table of probabilities for each state transition, expressed by Equation 1. In Figure 1(b) all state transitions with a probability greater than or equal to 0.25 have been represented. For example, the line from solution confirmation to agreement is labeled 11 (to indicate the number of state transitions) and 0.275, the probability from within this dataset that agreement will follow solution confirmation. (1) It is useful to compare this Markov chain to the heuristic miner representation. The heuristic miner representation conveys dependency relationships, which are heavily influenced by the directional relationships between states if there is an equal number of transitions from state Si->Sj as from Sj->Si then this will not show up as a strong relationship using the heuristic miner. The Markov chain simply shows the probability of each state transition and is appropriate for this work looking at a single group of students and this is adopted for the remainder of the paper. Data from wiki entries and chats were assigned to the decisions that were made as part of the collaboration (Reimann, et al., 2009a), and coded using a modified version of the Decision Function Coding Scheme. Of the data that was coded by the authors, 2517 lines of chat data were assigned, and 69% agreement was achieved. Further discussion resulted in 100% agreement. Similar coding of wiki data resulted in initial agreement of 98%. In total, 28 decisions were made during four weeks, and were assigned to cases according to the area about which they were deciding (choosing a model to alter, adding to the model, implementing the changes, and general planning). The decisions related to general planning were disregarded not analysed for this paper. 1649 lines of chat data were used, and 179 wiki revisions. A selection of the 28 cases, the distribution of the decisions, and the tools that were used in this process, are displayed in Figure 2.
cccase2
cpcase2

Cases
aicase1
aicase2
aicase3
aicase6
arcase1

iicase1 ircase1
Figure 2. Distribution of a selection of cases over time, by tool use
Figure 2 shows the temporal distribution of the types of decisions that were being made, as well as the tool use that was involved. Those decisions concerned with choosing the model were made at the beginning of the collaborative process. Some crossover (time-wise) occurred with the decisions that focused on additions to the model. This was repeated for the transition between deciding on the additions to the model and making decisions regarding the implementation of the additions. The squares and circles represent chat and wiki tool use
ISLS
461
ICLS2012
respectively. Patterns of tool use were identified: single chat (for example cccase2), multiple chat (for example ircase1), chat/s with follow-up wiki (for example cpcase2), and integrated (for example aicase1). This data was also examined at the level of the DFCS (see Figure 3 below).
cccase2
cpcase2

Cases
aicase1
aicase2
aicase3
aicase6
arcase1
iicase1 ircase1
Figure 3. DFCS for selected cases over time, by tool use
Figure 3 shows the different patterns of tool use (circles for chats and squares for wikis) in the states of decision-making (e.g. 0.2 is cccase2, code 2 orientation). It can be seen that most elements of the decisionmaking process were observed during chats (e.g. aicase1, in the first chat, codes 1-5 were used: problem definition, orientation, solution development, off-task, and simple agreement), and wikis were used for solution development (e.g. cpcase2) and non-task related postings (e.g. cpcase2, mostly formatting). In the cases of integrated tool use, the wiki was also used for orientation (e.g. aicase6). We used the patterns identified above and reallocated the cases according to the role that the tools played (not the topic of the decision). The new groups, as outlined above, were single chat (6 cases), multiple chat (4 cases), multiple wikis (1 case), chat/s and follow-up wiki (5 cases), and integrated (4 cases). The multiple wiki group was excluded from any further analysis. This re-classification of the type of case (e.g. implementation) into the role of tool produced four different threads through the data. Each thread was then mined for processes to compare the state transitions in decision making with a high probability of occurrence.
ISLS
462
ICLS2012
Figure 4. State transitions for decision making in single chats In the single chats, students began with problem definition, and then moved on to either solution analysis or orientation. Once engaged in orientation there was a high probability that they would continue there, and either agree, or move on to solution analysis. During these single chats, students would move through from solution analysis to solution suggestions, or agreement. Once at solution suggestions, they had a high probability of remaining here (brainstorming) and then either returning to solution analysis or to solution elaboration, at which point students would most probably continue to do this. Links to all other elements of the decision-making process from solution elaboration were fairly evenly distributed. Also worth noting is the high probability that solution confirmation would be followed by agreement.
Figure 5. State transitions for decision making in multiple chats Students who used multiple chats began with problem definition and either continued defining the problem, moved to orientation, or straight to solution suggestions. At orientation, students were most likely to continue with orientation, or begin non-task discussion, however once on non-task they were most likely to return to orientation. Other than these, students either returned to define the problem, moved to solution suggestion, or agreement. Solution analysis was not connected to any other elements, with the probability that students move from here to other aspects of solution development relatively equal. Once students began solution suggestions, they were most likely to return to orientation, or to continue in a brainstorming-like activity. In some instances they returned to problem definition or began non-task discussions. When students engaged in solution elaboration, they either returned to orientation or to solution suggestions. Solution evaluations and confirmations were not connected to other elements, with low numbers of instances and options spread between all other elements of solution development and orientation.
ISLS
463
ICLS2012
Figure 6. State transitions for decision making in chat/s with follow-up wiki Students using chats with follow up wiki began with problem definition and were most likely to follow with solution suggestions. If they began their solution analysis, they were most likely to follow with solution evaluation, although this was not common. When they began solution suggestions, they were most likely to follow up with solution elaboration or orientation, they did not continue with solution suggestions. Students engaged in orientation, mostly stayed there, returned to problem definition, agreed, or returned to solution suggestions. When they went to solution elaboration, they most like followed this with non-task (which in almost all cases occurred on the wiki, and is indicative of formatting changes to the text), where they tended to stay. Solution evaluation was likely to be followed by orientation, and evaluation was either followed with nontask, orientation or further confirmation.
Figure 7. State transitions for decision making in integrated tool use When students used an integrated approach to their tool use, they began with problem definition, followed by orientation. Once engaged in orientation, they were most likely to stay there. Small probabilities were recorded that flowed back to problem definition, solution suggestion, agreement, solution elaboration, non-task, solution evaluation, and confirmation. When engaged in solution analysis, students either returned to orientation, or proceeded to solution suggestion. At solution suggestion, students would engage in brainstorming, elaboration, or return to orientation. Solution elaboration was most probably followed by further elaboration, orientation, non-task discussion, or a return to solution suggestions. Once students engaged in evaluating solutions, they either moved to non-task discussions (almost all of which were associated with wiki use, and so represented minor formatting changes applied to the text) or returned to solution suggestions. When students engaged in solution confirmation, they tended to remain there, or return to orientation. Evaluation statements followed the two instances of implementation.
Discussion
Processes of decision-making in four patterns of tool use were identified using first-order Markov models. The patterns identified were single chat, multiple chat, chat/s with wiki follow-up and integrated. Relatively efficient decision-making processes characterized decisions made using a single chat. Alternatives were reconsidered early in the process, and once elaborated, were most likely confirmed and agreed upon. Specific decisions that students used this strategy to address were deciding on the subject area on which to focus (cccase2), and adding instructions to the software (aicase3). A different pattern was observed in those decisions that were made over multiple chats. This decision-making process was characterized by a consistent return to defining the problem and identifying features important to making the decision. Topics such as the resources to add to the model (arcase1), and what to do about ongoing software problems (ircase1) were addressed over multiple chats. They needed ongoing discussion, but were relatively straightforward. Nevertheless, the model indicates a careful process of considering options, over several sessions. The next two patterns of tool use were more complex, as the wiki was incorporated into the decisionmaking process. From examination of the chat/s with follow-up wiki process, along with the figures presented earlier identifying the temporal distribution of states of the decision-making process, it appears that the chat was used to identify initial solutions and agree on these, and the wiki for further elaboration, asynchronously. Students addressed the process of choosing a model (cpcase2) and the instructional approach adopted (aicase2)
ISLS 464
ICLS2012
in this way. For these decisions, the group did not return to the chat to evaluate the elaborated solutions, or to confirm them. Further consideration needs to be given as to whether these instances could be classified as implementation instead. Finally, the integrated approach to tool use in the decision-making process was examined. Questions such as what to add (aicase1) and the issues around the addition (aicase6) and implementation (iicase1) of the storytelling function in the software were addressed in this way. This type of decision-making was clearly separated into stages of understanding the problem, identifying solutions, using the wiki to elaborate on these (often during the chat session, but not always), and then evaluation and confirmation took place in the chat, both of which did, at times, result in further solution development. This is a complex, cyclical process, with truly integrated tool use. This group took advantage of the affordances of the tools identified in other studies. By dividing the cases by patterns of tool use, we were able to use first order Markov chains to differentiate patterns of decision-making, that fit the different strategies devised by students. Further analysis of the types of decisions included needs to occur, however it would seem at first glance, that time was important in identifying combinations of tool use and then patterns of decision-making within that. The Markov chains were useful in the analysis for showing with clarity the transitions between states that had a high probability of occurring. They lend themselves to a straightforward graphical comparison of different groups. The limitations of this technique of analysis are that the representations only show sequential state transitions and do not reveal hidden states they assume that all states are known and are present within the dataset. Higher-order Markov models need to be used in cases where non-sequential state transitions are relevant, and hidden Markov models in cases where hidden states may be revealed.
Conclusions
Time is important in investigating processes of learning. In this paper we used time to identify four patterns of tool use and then within these, examined the processes of learning. First-order Markov models were a valuable addition to our analysis. We examined a complex system of collaboration, students made decisions over weeks, and within any one chat session discussed multiple topics. They, seamlessly it seems, used the wiki as both an asynchronous and synchronous collaboration tool. Systematic investigation of the processes involved provides valuable information for designers, manager/instructors and researchers. Generally, we examine patterns of collaboration in order to aid our understanding of the processes of collaboration, to identify indicators of good collaboration in order to aid in classroom management, or to aid in the design of collaborative learning environments. In this study we were able to visualize the temporal nature of the decision-making process, and use this to identify patterns of tool use. The repetition of complexity in both the decision-making process and tool use, suggests that tool use may be a useful indicator of complexity in other elements of the collaborative process (see (Thompson & Kelly, accepted) for more details on this). This has implications for automated feedback to managers of collaborative learning environments, which will be followed up in future research. In addition, the observation of the adaptive nature of tool use in this group would indicate that collaborative learning environments should allow students to have this flexibility to adapt their strategies to meet their needs. Finally, this study contributes to the discussion around the analysis of processes in collaborative learning environments, and recommends consideration of more than time, but of as many elements of the complex process as are possible.
References
Aalst, J. (2009). Distinguishing knowledge-sharing, knowledge-construction, and knowledge-creation discourses. International Journal of Computer-Supported Collaborative Learning, 4, 259-287. Ding, N. (2009). Visualizing the sequential process of knowledge elaboration in computer-supported collaborative problem solving. Computers & Education, 52, 509-519. Eddy, S. R. (1998). Profile hidden Markov models. Bioinformatics, 14(9), 755-763. Goggins, S. P., Laffey, J., & Amelung, C. (2011). Context aware CSCL: Moving toward contextualised analysis. Paper presented at the Connecting computer-supported collaborative learning to policy and practice. Jeong, H., Biswas, G., Johnson, J., & Howard, L. (2010). Analysis of productive learning behaviors in a structured inquiry cycle using hidden markov models. Paper presented at the Proceedings of the 3rd International Conference on Educational Data Mining. Kapur, M. (2011). Temporality matters: Advancing a method for analyzing problem-solving processes in a computer-supported collaborative environment. International Journal of Computer Supported Collaborative Learning, 6(1), 39-56. Kennedy-Clark, S., Thompson, K., & Richards, D. (2011). Collaborative Problem Solving Processes in a Scenario-Based Multi-User Environment. Paper presented at the CSCL2011: Connecting computer supported collaborative learning to policy and practice.
ISLS
465
ICLS2012
Poole, M. S., & Holmes, M. E. (1995). Decision Development in Computer/Assisted Group Decision Making. Human Communication Research, 22(1), 90-127. Poole, M. S., & Roth, J. (1989a). Decision Development in Small Groups IV: A Typology of Group Decision Paths. Human Communication Research, 15(3), 323356. Poole, M. S., & Roth, J. (1989b). Decision Development in Small Groups V: Test of a Contingency Model. Human Communication Research, 15(4), 549589. Reimann, P. (2009). Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning, 4, 239-257. Reimann, P., Frerejean, J., & Thompson, K. (2009a). Using process mining to identify models of group decision making in chat data. 98-107. Reimann, P., Frerejean, J., & Thompson, K. (2009b). Using process mining to identify models of group decision making in chat data. Paper presented at the Computer Supported Collaborative Learning Practices, CSCL 2009. Reimann, P., Thompson, K., & Weinel, M. (2007a). Collaborative learning by modelling: Observations in an on-line setting. Paper presented at the ICT: Providing choices for learners and learning., ASCILITE, Singapore. Reimann, P., Thompson, K., & Weinel, M. (2007b). Decision making patterns in virtual teams. Paper presented at the Supporting learning flow through integrative technologies, Amsterdam. Shih, B., Koedinger, K. R., & Scheines, R. (2010). Discovery of Learning Tactics using Hidden Markov Model Clustering. Paper presented at the Proceedings of the 3rd International Conference on Educational Data Mining, 2010. Southavilay, V., Yacef, K., & Calvo, R. A. (2010). Analysis of Collaborative Writing Processes Using Hidden Markov Models and Semantic Heuristics. Paper presented at the Proceedings of the Third International Workshop on Semantic Aspects in Data Mining 2010, in conjunction with the 10th IEEE International Conference on Data Mining (ICDM 2010). Thompson, K., & Kelly, N. (accepted). Combining collaboration spaces: Identifying patterns of tool use for decision-making in a networked learning environment. Paper presented at the Eighth International Conference on Networked Learning. Ullman, A., Peters, D., & Reimann, P. (2005). Developing a research supportive web-based learning System. Paper presented at the ODLAA Conference 2005, Adelaide. van der Aalst, W. M. P., & Weijters, A. J. M. M. (2005). Process Mining Process-Aware Information Systems: John Wiley & Sons, Inc. Weijters, A. J. M. M., Van der Aalst, W. M. P., & Medeiros, A. K. A. d. (2006). Process Mining with Heuristics Miner Algorithm, BETA Working Paper Series. Eindhoven University of Technology, Eindhoven. Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction in computer-supported collaborative learning. Wise, A., & Chiu, M. M. (2011a). Analyzing temporal patterns of knowledge construction in a role-based online discussion. International Journal of Computer-Supported Collaborative Learning, 6(3), 445-470. Wise, A., & Chiu, M. M. (2011b). Knowledge construction patterns in online conversation: A statistical discourse analysis of a role-based discussion forum. Paper presented at the Connecting computersupported collaborative learning to policy and practice. Zenios, M., & Holmes, B. (2010). Knowledge creation in networked learning: combined tools and affordances. Paper presented at the 7th International Conference on Networked Learning 2010.
Acknowledgements
This work was funded by the Australian Research Council grant FL100100203.
ISLS
466
ICLS2012
Examining the Adequacy of Students Priors and Teachers Role in Attention to Critical Features in Designing for Productive Failure
Pee Li Leslie Toh & Manu Kapur National Institute of Education, Nanyang Technological University, 1 Nanyang Walk, Singapore 637616 leslie.toh@nie.edu.sg, manu.kapur@nie.edu.sg Abstract: This paper describes two classroom-based studies that examine the productive failure learning design from the following perspectives: (i) the adequacy of students prior knowledge, and (ii) the necessity of teacher-led discernability of critical conceptual features. The entire investigation, set in the domain of monohybrid inheritance, involved 138 ninth and tenth grade science students. In the first study, students were either provided or not provided with monohybrid inheritance pre-requisites for their complex problem-solving activity. The second study compared a teacher-led with a student-led comparison and contrast during the consolidation phase. Findings of the first study suggested that students may not have adequate priors to generate representations and solution methods that explore multi-level conceptions. Findings of the second study suggested that teacher involvement in the compare-and-contrast may be beneficial for better knowledge assembly.
Introduction
There is now a growing body of research that suggests an efficacy of engaging students in exploratory or invention activities, prior to receiving an appropriate form of instructional intervention (e.g. Schwartz & Bransford, 1998; Schwartz & Martin, 2004). Kapur (2008) proposed the notion of Productive Failure (PF) to examine this efficacy, and more recently developed it into a learning design (Kapur & Bielaczyc, 2012). The PF learning design embodies four core interdependent mechanisms: a) activation and differentiation of prior knowledge in relation to the targeted concepts, b) attention to critical conceptual features of the targeted concepts, c) explanation and elaboration of these features, and d) organization and assembly of the critical conceptual features into the targeted concepts. These mechanisms are embodied in a two phase design: a generation and exploration phase (Phase 1) followed by a consolidation and knowledge assembly phase (Phase 2). Phase 1 affords opportunities for students to generate and explore the affordances and constraints of multiple representations and solution methods (RSMs). Phase 2 affords opportunities for organizing and assembling the relevant student-generated RSMs into canonical RSMs. The designs of both phases were guided by the following core design principles that embody the abovementioned mechanisms: (1) create problem-solving contexts that involve working on complex problems that challenge but do not frustrate, rely on prior mathematical resources, and admit multiple RSMs (mechanisms a and b); (2) provide opportunities for explanation and elaboration (mechanisms b and c); (3) provide opportunities to compare and contrast the affordances and constraints of failed or sub-optimal RSMs and the assembly of canonical RSMs (mechanisms b d). Over a series of studies (e.g., Kapur, 2010, 2011, 2012), Kapur and colleagues have demonstrated the efficacy of the PF learning design over a direct instruction design in terms of significantly better conceptual and transfer gains without compromising procedural fluency. More importantly, findings suggest that the more RSMs students are able to generate, the more they learn from PF (Kapur, 2012; Kapur & Bielaczyc, 2012). Kapur and Bielaczyc (2012) argue that a greater RSM diversity implies greater prior knowledge activation and differentiation, which in turn helps students to better attend to and understand the critical features of the targeted concepts from subsequent instruction (Schwartz & Bransford, 1998; Schwartz & Martin, 2004). The purpose of this paper is to more closely examine an assumption and a key mechanism of the PF learning design through two classroom-based studies in Singapore biology classrooms. In the PF learning design, there is an assumption that even though students may not have learnt the target concepts, they may still be able to leverage their informal and intuitive priors along with their formal priors to generate RSMs to complex, novel problems. Our first aim in this paper is to show that this assumption does not hold true more generally, especially when the targeted concepts are not entirely experientially grounded. Our second concern deals with one of the key mechanisms in learning design, which is, getting students to attend to the critical features of the targeted concept by comparing and contrasting student-generated RSMs and the canonical RSMs. To achieve this, PF emphasizes the role of the teacher in leading such a comparison and contrast (C&C) activity. However, the need for the teacher (as opposed to the student) doing the C&C has not been tested. Testing this mechanism forms our second aim.
Assumption of Students Priors for Generating RSMs
ISLS
467
ICLS2012
Given the importance of student-generated RSMs and their correlation with learning from PF, it is important to examine more closely the assumption that students have the necessary priors to generate RSMs especially without having had any formal instruction on the targeted concepts. The complex problems of past PF studies (e.g., Kapur, 2008; Kapur & Bielaczyc, 2012) had contexts that were grounded in students own experiences, for instance, a speeding offence (involved concepts of speed, friction, forces, acceleration, etc.), finding average speeds in a biking scenario (involved concepts of speeds, distance, time, etc.), and finding consistency of a soccer player given his goal-scoring record (involved concepts of central tendencies, graphing, deviation, etc.). Invariant across all these problem-solving contexts is that although students had not formally learned the targeted concepts, it is reasonable to suggest that they already had informal and experientially-grounded conceptions of these concepts (Palmer, 1999). If so, this only begs the question: What about problem-solving contexts based on phenomena that are not entirely experientially-grounded? An example of such a phenomenon is monohybrid inheritance, which is a multi-level complex phenomenon. This is a specific area in biology, which comprises concepts from inheritance (macro-level) and genetics (micro-level). Essentially, inheritance is concerned with the gain of characteristics from ones parents, while genetics entails inheritance patterns, transmission of genes through generations and genetic expression (Kibuka-Sebitosi, 2007). Whereas the macro-level phenomenon of inheritance can be seen and observed, the underlying micro-level causal mechanisms cannot. Consequently, if students are placed in problem-solving contexts that that target monohybrid inheritance, they may well be able to generate RSMs at the macro-level, but may find it difficult to generate RSMs at both the macro and micro levels. We designed Study 1 to test this conjecture.
Teachers Role in Drawing Attention to Critical Conceptual Features

A critical mechanism in the PF learning design is to draw students attention to critical features of the targeted concepts. This is done in the consolidation phase by the teacher leading a C&C between student-generated RSMs and the canonical RSMs (Kapur & Bielaczyc, 2012). Thus, the PF learning design conjectures that the efficacy of RSM diversity is contingent upon a teacher-led C&C activity. This is in contrast with past research where students themselves were able to compare and contrast and attend to the critical features so long as the generation and exploration phase prepares the students to see the significance of expert solutions and potential resources for learning from the later formal consolidations (Schwartz & Martin, 2004, p. 168). Even earlier studies of PF suggested the same (Kapur, 2008; Kapur & Kinzer, 2009), as do others (e.g., Snchez, GarcaRodicio & Acua, 2009). If so, why do more recent studies of PF (e.g., Kapur, 2010, 2011, 2012) emphasize the teachers involvement in the consolidation phase? Is there a marginal gain of teachers leading the C&C as opposed to students doing it by themselves? Expert-novice research does suggest that experts tend to attend to more critical features of a problem situation than novices, and are better at discerning features and significant informational patterns (Bransford, Brown, Cocking, Donovan & Pellegrino, 2000; Chi, Feltovich, & Glaser, 1981). If so, we can conjecture that having teachers (as relative experts in subject matter) may be better in drawing attention to the critical features during the C&C than if students were do it themselves. This in turn, may also result in better organizing and assembling these features into the targeted conceptions. However, this remains a conjecture, and Study 2 was designed to test this conjecture. Sections 2 and 3 present the two studies. We conclude by discussing the findings from the two studies, and deriving implications for theory and future research.
Study 1: Testing the Assumption of Students Priors for Generating RSMs

Participants and Design
Participants (N = 80; 53 female, 27 male) were ninth grade students from three intact science classes at a mainstream, co-educational secondary school in Singapore. They had not been instructed on monohybrid inheritance and its pre-requisites specifically, nuclear division (included chromosomes and meiosis) and molecular genetics (included DNA and genes). The 80 students were randomly divided into two conditions: a) High Micro-level Conception (HMiC) condition (N = 40, 28 female, 12 male) was provided with three one-hour lessons on nuclear division and molecular genetics by the first author with six years of experience teaching eleventh and twelfth grade biology, and b) Low Micro-level Conception (LMiC) condition (N = 40, 25 female, 15 male) was provided with three one-hour lessons about the eye, a topic unrelated to monohybrid inheritance or its pre-requisites. While both conditions were assumed to have similar levels of the macro-level conceptions, the experimental manipulation was designed to make them different in terms of the micro-level conceptions. Students in each condition were then randomly divided into dyads/triads, and were instructed to generate RSMs for a complex problem (see following section) on their own, except for the occasional affective support provided by the first author to
ISLS
468
ICLS2012
ensure that groups remained on task. The students were given blank sheets of A4 size paper for their group work.
Design of the Complex Problem

The complex problem targeted concepts of monohybrid inheritance. It was adapted from Gregor Mendels classical pea plants experimental data as well as the outcomes from the various genetic crosses (see below for an example). Parental Generation Genetic cross between pea plants with purple flowers & pea plants with white flowers 1st Generation Offspring Purple flowers: All; White flowers: 0 Further Genetic Cross Genetic cross between offspring with purple flowers 2nd Generation Offspring Purple flowers: 700; White flowers: 230
The complex problem involved effects-to-cause (Stewart, 1988, p. 244) reasoning, and required students to generate multi-level (macro and micro) explanations for the data provided. More specifically, students needed to generate an explanation at the macro-level about the predominance of the purple flower characteristic, and concomitantly, generate micro-level causal explanations to account for how the predominance effect is possibly genetically controlled.
Data Source and Analysis

The diversity of student-generated RSMs formed the outcome measure. In previous PF studies, RSM diversity was defined as the number of different RSMs generated by a group (Kapur & Bielaczyc, 2012). In this study, RSM diversity was defined in terms of the macro and micro levels, that is, whether the RSMs generated by a group constituted explanations at the macro-level only or at both the macro and micro levels. Based on this operationalization of RSM diversity, a rubric (see Table 1) was developed to categorize the generated RSMs into either macro-level conceptions or macro- and micro-level conceptions based RSMs. RSMs in each category were further categorized according to the following critical conceptual features they targeted: Dominant-Recessive, Gametes Formation and Union and Genetic Crosses. Two experienced biology teachers independently scored the RSMs with an inter-rater reliability of 0.67. Table 1. Rubric to analyze and categorize student-generated RSMs Type of RSM Macro-level conceptions based RSM (See Figure 1a) Macro- and micro-level conceptions based RSM (See Figure 1b) Conception (C) C1: Dominant-Recessive C2: Gametes Formation and Union C3: Genetic Crosses Descriptions of Conception Genetic materials interacted to generate the characteristics for each character Separation and union of genetic materials Rules for genetic crosses and the matching of characteristics of each character to their respective genetic makeup
Table 2 presents the descriptive statistics. There was a significant association between the students levels of taught pre-requisites and RSM Diversity ( 2 = 14.15, df = 1, p < 0.001). For example, the students from the HMiC condition represented genes in the form of DNA or chromosomal structuresan attempt at using the taught micro-level concepts for their RSM (See Figure 1b). However, students from the LMiC condition, M = 1.00, SD = 0.00, relied only on the macro-level conceptions. For example, they developed conceptions of dominant with macro-level conceptions, for example, stronger (See Figure 1a). Table 2. Frequency counts of student groups from the LMiC and HMiC conditions Number of Student Groups Macro-level RSMs Macro- & Micro-level RSMs LMiC (20 groups) HMiC (19 groups) 20 9 0 10
Results
ISLS
469
ICLS2012
Figure 1a: C1: Developed conceptions of dominant with macrolevel conceptions, e.g., stronger or healthier
Figure 1b:
C2
C1 C3 C1 C2
C3
C1: Developed dominant conceptions with varying thickness (macro-level conception) and also with chromosomes (micro-level conception)
Figure 1a. LMiC: Student-generated RSM with macro-level conceptions; Figure 1b. HMiC: Similar studentgenerated RSM but with conceptions at both the macro and micro levels.
Study 2: Teachers Role in Drawing Attention to Critical Features

Study 2 was designed to address the following question: How does having a teacher-led C&C versus a studentled C&C influence how much students learn from PF?
Participants
Participants (N = 58; 47 female, 11 male) were tenth grade students from two intact science classes at the same mainstream, co-educational secondary school in Singapore, and taught by the same biology teacher. Tenth grade students were chosen because at the time of the study, these students had not been taught monohybrid inheritance in the curriculum but had received instruction on its pre-requisites. Choosing students with prerequisite knowledge was necessary as the findings of the first study indicated that students do not have adequate priors for the generation of RSMs that explored conceptions beyond the macro-level.
Design
Table 3 presents the design of the study. First, all students took a 50-minute single item pretest to ascertain their prior knowledge of monohybrid inheritance. This item pertained to a particular biological character (that is, humped versus smooth shoulders) of a mouse. It required students to first consider the first generation offspring outcomes that were obtained from a cross between two parental mice. Thereafter, they had to determine the genotypes (that is, the genetic makeup) and phenotypes (that is, the inherited characteristics) of the second generation offspring. Each class had four 60-minute periods over three weeks. In the first period, all students in dyads/triads generated RSMs to the complex problem (the same as in Study 1) on their own, except for the occasional affective support by the teacher. Students were given blank A4-sized papers for their work. In the second period, the teacher carried out a lecture of the targeted concepts of monohybrid inheritance and explained the canonical solution.
ISLS 470
ICLS2012
The experimental manipulation happened in the third period after students had learnt the canonical concepts in the second period. In one class, the teacher led a whole-class discussion, where he brought students attention to the critical conceptual features of monohybrid inheritance, that is, the conceptions of dominantrecessive, gametes formation and union and genetic crosses. He then explained and elaborated on these features. To highlight the affordances and constraints of the student-generated RSMs, he further compared and contrasted these RSMs with the critical features. Thereafter, he showed how the various critical conceptual features of monohybrid inheritance were organized and assembled into the canonical solution of the complex problem. We call this the Generation and Teacher-led (GTL) C&C condition. In the other class, students worked individually to compare and contrast on their own their generated RSMs with the canonical concepts taught in the previous period. They also had to suggest improvements for their generated RSMs. The students were given blank sheets of A4 size paper for their individual work. We call this the Generation and Student-led (GSL) C&C condition. The students also rated their level of engagement for this period with a nine items Likert scale survey (modified from Kapur, 2011) that had an alpha reliability of 0.94. The engagement rating was designed to capture differences, if any, between the GTL and GSL students on their self-reported engagement during the key manipulation, that is, the teacher- or student-led C&C activity. The fourth period was designed for problem-solving practice, where students in both the conditions individually solved four questions on a worksheet, followed by a teacher-led discussion of the answers. After the implementation, all students took a 50-minute, two-part, posttest, with the first part testing for procedural knowledge (one item), and the second part (four items) testing for conceptual knowledge. The procedural knowledge item was isomorphic with the pre-test item and the four worksheet questions. The conceptual knowledge items targeted the critical features such as: (i) What is the relationship between DNA, chromosomes, genes and alleles? (ii) What is monohybrid inheritance? (iii) What is the relationship of meiosis to monohybrid inheritance? (iv) What is the relationship between dominance and recessive? Ideally, one would want several items for each of the four conceptual questions for greater scale reliability. However, curriculum time constrained what we could test for, and we decided to look for greater coverage of the conceptual features rather than having more items for one particular feature. Consequently, however, the overall scale reliability of the four conceptual questions was low at 0.19. After the completion of each posttest item, the students rated the confidence of their answers using a 5-point Likert scale from 0 (no confidence) to 4 (100 % confidence). Table 3. The overall design of Study 2 Sequence of Events in Study 2 Phases of Productive Failure Learning Design Generation Direct Comparison & & Instruction Contrast (C&C) Exploration (2nd Period) (3rd Period) (1st Period) Students individually compared and contrasted own Student group generated groups Teacher RSMs with generated lectured on canonical concepts RSMs for targeted complex concepts Teacher led wholeproblem class C&C of generated RSMs with canonical concepts
Condition
Pretest
ProblemSolving Practice (4th Period)
Posttest
GSL For all conditions
GTL
Students solved four problems; teacher discussed solutions with the whole class
For all conditions
Data Sources and Analyses

For process measures, we analyzed the student-generated RSMs with the same approach as in the first study. Two independent coders coded the student-generated RSMs with inter-rater reliability of 1.00. Other process measures included students self-reported lesson engagement ratings during the key experimental manipulation period, where the C&C activity was either led by the teacher or the students themselves. The posttest scores on the procedural and conceptual knowledge and the item confidence ratings formed the outcomes measures. Two independent raters scored the posttest items with an inter-rater reliability of 0.92. All the instruments were validated by two experienced biology teachers from the participating school. The instruments were piloted on students who were not included in the main study. The teachers feedback and the trial results were used to refine all instruments.
ISLS 471
ICLS2012
Results
Table 4 presents the descriptive as well as the inferential statistics, based on which the studys findings can be summarized as following: 1. Pretest. As expected, students in both the conditions demonstrated no knowledge of the targeted concepts, and there was statistically no significant difference between the two conditions. 2. RSM Diversity. As expected, there was variation within each condition in terms of student-generated RSMs. In each condition, some dyads or triads generated RSMs at the macro-level only, whereas others generated at both the macro and micro levels. However, a non-parametric chi-squared test (not shown in Table 4) showed there was statistically no significant difference between the two conditions on the generation of macro-level or a combined, macro- and micro-level RSMs, 2 (1) = 2.23, p = 0.136. In other words, there was statistically no difference between the two conditions in terms of what students were able to generate. 3. Self-Reported Engagement during the C&C Activity. There was generally high level of self-reported engagement in both the conditions, but there was statistically no significant difference between the two. 4. Posttest. a. Procedural Knowledge. Controlling for the effect of prior knowledge (as determined by the pretest score), an ANCOVA showed that there was statistically no significant difference between the conditions on procedural knowledge. Students from both the conditions scored close to the maximum score. This was expected given the isomorphic problem-solving practice in the final lesson for both the conditions. b. Conceptual Knowledge. Controlling for the effect of prior knowledge, an ANCOVA showed that students from the GTL condition performed significantly better than GSL condition on conceptual knowledge. This effect had a high effect size of .14. c. Confidence. As expected, students reported greater confidence for the procedural knowledge item than conceptual knowledge items. However, controlling for the effect of prior knowledge, ANCOVAs showed that there was statistically no significant difference between the conditions on their self-reported confidence for the procedural knowledge item. For conceptual knowledge items, GSL students reported slightly greater confidence than GTL students, but this effect was only marginally significant with a low-medium effect size of .06. Table 4. Summary of descriptive and inferential statistics for Study 2 GTL: Teacher-Led C&C M SD 0.13 0.34 3.78 0.67 GSL: Student-Led C&C M SD 0.04 0.19 3.72 0.65
Pretest Self-Reported Engagement (during C&C activity) Posttest Procedural Conceptual Confidence (Procedural) Confidence (Conceptual)
Max 5 5
F F(1,56) = 1.54 F(1,56) = 0.12
p .220 .726
p2 .03 <.01
2 13 4 4
2.00 8.39 3.26 1.34
0.00 2.01 0.82 0.71
1.93 6.81 3.52 1.69
0.27 2.02 0.58 0.74
F(1,56) = 2.20 F(1,56) = 8.98 F(1,56) = 1.91 F(1,56) = 3.30
.144 .004* .172 .075
.04 .14 .03 .06
In the light of previous PF findings that RSM diversity is a significant predictor of conceptual understanding and transfer (Kapur, 2012; Kapur & Bielaczyc, 2012), we further examined the relationship between RSM diversity and posttest performance on conceptual knowledge. Within each condition, we compared the conceptual knowledge scores of students who had generated RSMs at macro-level with those who had generated at both macro and micro levels. Table 5 presents the descriptive and inferential statistics, based on which we can surmise that the descriptive trend in both conditions was that dyads or triads who generated macro- and micro-level RSMs (that is, had greater RSM diversity) performed better on average on conceptual knowledge than those who only produced macro-level RSMs. For GSL students, this trend did not reach significance and had a low effect size of .03. For GTL students, however, this trend reached marginal significance (perhaps due to low n) with a moderate-high effect size of .10. The moderate-high effect size for the GTL condition but not for the GSL condition tentatively confirms previous findings that RSM diversity plays a significant role in how much students learn from PF. More interestingly, it also shows that the effect of RSM diversity is mediated by whether the teacher or a student carries out the compare and contrast between the student-generated RSMs and the canonical RSMs.
ISLS
472
ICLS2012
Table 5. RSM diversity and conceptual knowledge scores by condition Posttest Conceptual Knowledge Score of Dyads/Triads who Produced: Macro-Level RSMs Macro- & Microonly Level RSMs M SD M SD 6.40 2.32 7.00 1.90 7.95 2.04 9.30 1.70
Experimental Condition GSL GTL
F F(1,24) = 0.61 F(1,29) = 3.09
p .445 .090
p2 .03 .10
Discussion
We had two aims. First, we wanted to show that the assumption of students priors for generating RSMs to complex problems does not hold true more generally, especially when the problem targets concepts that are not entirely experientially grounded. Our findings from Study 1 provided evidence for this. Second, we wanted to test the marginal gain of having teacher-led C&C activity over a student-led one. Findings from Study 2 suggested that a teacher-led C&C activity is significantly better than a student-led one in terms of conceptual gains. This was consistent with our hypothesis grounded in expert-novice literature that teachers as subject matter experts are perhaps better in knowing what and how to draw attention to the critical features, whereas students may not have been able to do this as well by themselves. In Study 1, students were either provided (HMiC condition) or not provided (LMiC condition) with pre-requisites on the micro-level conceptions of monohybrid inheritance. Findings suggested that the HMiC condition students generated RSMs at both the micro and macro levels. However, the fact that these students had not received any formal instruction on monohybrid inheritance made it difficult for them to generate canonical conceptual connections on their own. Therefore, the overall interactions between the micro and macro levels gave rise to failed or suboptimal RSMs. However, having failed in generating canonical explanations of the phenomenon, such attempts at generating diverse RSMs are precisely the locus of powerful learning from PF (Kapur & Bielaczyc, 2012). This was not the case for the LMiC condition students who relied only on macro-level conceptions. These results suggest that one cannot always assume that students have the necessary prior resources to generate RSMs, especially for phenomenon with multi-level conceptions that are not entirely experientially-grounded. Some form of formal prior resources maybe necessary prior to getting students to generate RSMs. In Study 2, after receiving instruction on the targeted concepts, students experienced either a teacherled comparison and contrast (C&C) activity or a self-led C&C. Findings revealed that both the GSL and GTL conditions were effective in equipping the students with procedural fluency. However, students from the GTL condition had significantly better conceptual gains than their peers in the GSL condition. The better conceptual gains in the GTL condition may be attributed directly to the experimental manipulation of having a teacher rather than a student carrying out the C&C. For the GTL condition, the teacher, being a subject matter expert and better at discerning critical conceptual features (Bransford et al., 2000), might not only discuss the affordances and constraints of the student-generated RSMs, but also reinforce the critical conceptual features (Kapur & Bielaczyc, 2012). With his rich content knowledge and in-depth understanding of monohybrid inheritance (Bransford et al., 2000), he might also explain and elaborate on these various conceptual features, and further help in organizing and assembling them into the targeted conceptions (Kapur & Bielaczyc, 2012). GSL condition students, on the other hand, were probably aware of the critical conceptual features, especially after the complex problem solving activity. However, these students, being novices, might not have been able to effectively explain and elaborate on these features, and fail to organize and assemble them further into the targeted conceptions. It is rather premature to generalize the above findings beyond the conditions and settings of these two studies. The low reliability of the posttest further constrains the reliability of our inference. Hence, larger samples and more reliable instruments are needed. Nonetheless, our findings can be seen as initial warrants that can drive further work to examine the boundary conditions of PF, and unpack its design elements more systematically. For example, our work going forward will be to examine how the provision of micro-level conceptions influences how much students learn from a subsequent consolidation phase. Additionally, we will also seek to understand if the collaborative participation structure of the PF learning design is a critical and necessary design decision to support student generation and exploration of RSMs (Kapur & Bielaczyc, 2012). This is essential because other studies (e.g. Snchez, Garca-Rodicio & Acua, 2009; Schwartz & Bransford, 1998) demonstrated learning gains even when they allowed students to explore or organize their RSMs individually. We believe that such efforts that unpack the design elements further stand to contribute substantively to on-going debates on how best to support students in their initial learning (Collins, 2012; Kapur & Rummel, 2009, 2012)
ISLS 473
ICLS2012
References
Bransford, J.D., Brown, A.L., Cocking, R.R., Donovan, M.S. & Pellegrino, J.W. (2000). How People Learn. Brain, Mind, Experience, and School (p. 31). Expanded Edition. Washington: National Academy Press. Chi, M.T.H., Feltovich, P.J. & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5 (2), 121 152. Collins, A. (2012). What is the most effective way to teach problem solving? A commentary on Productive Failure as a method of teaching. Instructional Science. Howitt, D. & Cramer, D. (2008). Introduction to SPSS in Psychology. For Version 16 and Earlier. Fourth Edition. England: Pearson Education Limited. Kapur, M. & Bielaczyc, K. (2012). Designing for productive failure. Journal of the Learning Sciences, 21 (1), 45 83. Kapur, M. & Kinzer, C.K. (2009). Productive failure in CSCL groups. International Journal of ComputerSupported Collaborative Learning, 4 (1), 21 46. Kapur, M. & Rummel, N. (2009). The assistance dilemma in CSCL. In A. Dimitracopoulou, C. OMalley, D. Suthers, & P. Reimann (Eds.), Computer Supported Collaborative Learning Practices - CSCL2009 Community Events Proceedings, Volume 2 (pp. 37 42). International Society of the Learning Sciences. Kapur, M. & Rummel, N. (2012). Productive failure in learning from generation and invention activities. Instructional Science. DOI: 10.1007/s11251-012-9235-4 Kapur, M. (2008). Productive failure. Cognition and Instruction, 26 (3), 379 424. Kapur, M. (2010). Productive failure in mathematical problem solving. Instructional Science, 38 (6), 523 550. Kapur, M. (2011). A further study of productive failure in mathematical problem solving: Unpacking the design components. Instructional Science, 39 (4), 561 579. Kapur, M. (2012). Productive failure in learning the concept of variance. Instructional Science, DOI: 10.1007/s11251-012-9209-6 Kibuka-Sebitosi, E. (2007). Understanding genetics and inheritance in rural schools. Journal of Biological Education, 41 (2), 56 61. Palmer, D.H. (1999). Exploring the link between students' scientific and nonscientific conceptions. Science Education, 83 (6), 639 653. Snchez, E., Garca-Rodicio, H. & Acua, S.R. (2009). Are instructional explanations more effective in the context of an impasse? Instructional Science, 37 (6), 537 563. Schwartz, D.L. & Bransford, J.D. (1998). A time for telling. Cognition and Instruction, 16 (4), 475 522. Schwartz, D.L. & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22 (2), 129 184. Stewart, J. (1988). Potential learning outcomes from solving genetics problems: A typology of problems. Science Education, 72 (2), 237 254.
Acknowledgements
The study was funded by a Ministry of Education grant to the second author through the Office of Education Research of the National Institute of Education of Singapore. The authors would like to thank the principal, teachers and students of the participating school for their support for this research study.
ISLS
474
ICLS2012
Volume 1: Author Index
VOLUME 1: AUTHOR INDEX A Abrahamson, Dor, 283 Ahn, June, 251 Ai, Hua, 363 Aleven, Vincent, 64 Avramides, Katerina, 151 B Baker, Ryan, 167 Barnes, Jacqueline, 403 Becker, Marina, 355 Bennett, Sue, 88 Bereiter, Carl, 111 Bientzle, Martina, 1 Blikstein, Paulo, 387 Bonsignore, Elizabeth, 103, 251 Boticki, Ivica, 235 C Chai, Ching-Sing, 243 Chen, Bodong, 111 Chen, Ying-Chih, 159 Chuy, Maria, 111 Clark, Jim, 40 Clegg, Tamara, 103 Craft, Brock, 151 Cress ,Ulrike, 1 D Dalgarno, Barney, 88
ISLS
475
ICLS2012
Delgado, Cesar, 204 Dasgupta , Chandan, 9 DeVane, Ben, 143 Druin, Allison, 103, 251 Dyke, Gregory, 363 E Eberbach, Catherine, 411 Eisenberg, Michael, 379 Elinich, Karen, 180 Engler, Jan, 291 Engle, Randi A., 40 F Fields, Deborah A., 188, 196 Fischer, Frank, 331 Forest, Dominique, 17 Fraistat, Ann, 251 Frmmel, Cornelius, 48 G Gegenfurtner, Andreas, 220 Gelderblom, Helene, 103 Gobert, Janice, 167 Goel, Ashok, 411 Goh, Sao-Ee, 72 Goldwasser, Lloyd, 40 Gresalfi, Melissa, 403 Guha, Mona Leigh, 103 Gweon, Gahgene, 435 H Ha, Seungyon, 307
ISLS
476
ICLS2012
Hand, Brian, 159 Hershkovitz, Arnon, 167 Hmelo-Silver , Cindy, 339, 411 Holbert, Nathan R., 119 Hoppe, H. Ulrich, 291 Hsiao, Ying-Ting, 56 Hsu, Ching-Kun, 235 Huang, Junsong, 259 Huang, Ronghuai, 443 J Jacobson, Michael J., 25 Jain, Mahaveer, 435 Jan, Ming-Fong, 243 Jeong, Heisawn,339 Jordan, Rebecca, 411 K Kafai, Yasmin B., 188, 196 Kai, Hakkarainen, 127 Kangas, Kaiju, 267 Kapur, Manu, 259, 467 Keifert, Danielle, 315 Kelly, Nick, 25, 459 Kennedy, Gregor, 88 Kim, Beaumie, 451 Kim, Mi Song, 451 Kimmerle, Joachim, 1 Kirschner, Paul, 96 Klopfer, Eric, 72 Kollar, Ingo, 331
ISLS
477
ICLS2012
Kraus, Kari, 251 Kumar, Rohit, 363 L Lachner, Andreas, 48 Lam, Diane P., 40 Law, Nancy, 212, 443 Lee, Jason Wen Yau, 451 Lee, Jiyeon, 395 Lee, Tuck Leong, 451 Lewittes, Becky, 103 Li, Yanyan, 443 Lin, Emily S., 135 Linn, Marcia, 323, 371 Lindgren, Robb, 427 Lorenz, Elisabeth, 331 Luckin, Rosemary, 151 Lyons, Leilah, 9 M McDonough, John, 435 McElhaney, Kevin, 323 Madeira, Cheryl Ann, 227 Marbouti, Farshid, 56 Markauskaite, Lina, 25 Matuk, Camillia, 323 McKenney, Susan, 96 Meyer, Xenia S., 40 Miller, David, 323 Minor, Emil, 9
ISLS
478
ICLS2012
Morton, Karisma, 204 Moskaliuk, Johannes, 1 Murray, Elizabeth, 275 N Nakama, Adam, 167 Naves, Erica, 40 Newnham, Denise Shelley, 127 Nleya, Paul, 127 Nckles, Matthias, 48 O Orrill, Chandra Hawley, 175 P Pacilio, Laura, 64 Park, Soonhye, 159 Patton, Charles, 275 Pea, Roy, 427 Perera, Nishan, 56 Perez, Sarah, 40 Q Quintana, Chris, 419 R Raj, Bhiksha, 435 Rau, Martina A., 64 Read, Janet C, 151 Reiss, Kristina, 331 Resendes, Monica, 111 Rey, Sabine, 48 Engestrm, Ritva, 127
ISLS
479
ICLS2012
Rosas, Hernan, 40 Rose, Carolyn, 363, 435 Roschelle, Jeremy, 275 Rummel, Nikol, 64, 355 S Sandoval, William, 347 Sawyer, R. Keith, 33 Scardamalia, Marlene, 111 Schmidt, Kristin, 48 van Schooneveld, Jacqueline, 180 Searle, Kristin A., 188, 196 Sears, David A., 307 Seitamaa-Hakkarainen, Pirita, 267 Sensevy, Grard, 17 Shaffer, David Williamson, 175 Shelley, Tia, 9 Shen, Ji, 299 Sherman, Milan, 80 Sinha, Suparna, 411 Slattery, Brian, 9 Slotta, James D., 227 Song, Minyoung, 419 Southavilay, Vilaythong, 25 Speer, Jennifer, 56 Stucke, Bjrn, 48 Sun, Jizhen, 235 Sung, Shannon, 299 Svihla, Vanessa, 371
ISLS
480
ICLS2012
T Tan, Danny, 40 Taylor, Holly A., 135 Thiel, Ansgar, 1 Thompson, Kate, 459 Toh , Pee Li Leslie, 467 Toh, Yancy, 243 Trninic, Dragan, 283 Tse, Hidy, 212 Tsepo, Batane, 127 Tunc-Pekkan, Zelha, 64 U Ufer, Stefan, 331 V Vauras, Marja, 220 Veermans, Koen, 220 Virkkunen, Jaakko, 127 Visconti, Amanda, 251 Vogel, Freydis, 331 Voogt, Joke, 96 W Walkington, Candace, 80 Wang , Joyce, 72, 180 Weinbrenner, Stefan, 291 Wichmann, Astrid, 355 Wilde, Jane, 395 Wilensky, Uri, 119 Wong, Lung-Hsiang, 235, 243
ISLS
481
ICLS2012
Wise, Alyssa Friend, 56 Y Yang, Zhitong, 72 Yip, Jason C., 103 Yoon, Susan A., 72, 180 Yuen, Johnny, 212 Z Zellner, Moira, 9 Zhang, Dongmei, 299 Zhang, Jianwei, 395 Zhang, Yonghe, 443 Zheng, Kathleen, 40
ISLS
482

ICLS2012 Proceedings Vol 1 2012

Enviado por

Dados do documento

Descrição original:

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

ICLS2012 Proceedings Vol 1 2012

Enviado por

Direitos autorais:

Formatos disponíveis

10th International Conference of the Learning Sciences

The Future of Learning

Centre for Research on Computer Supported Learning & Cognition (CoCo)

ICLS 2012 Conference Proceedings Volume 1 Full Papers

THE FUTURE OF LEARNING

ICLS2012 would like to thank our sponsors:

Volume 1: Full Papers

ORGANISERS AND COMMITTEES CONFERENCE CHAIRS

Twente, the Netherlands

Sydney, Australia EARLY CAREER WORKSHOP

University, Taiwan PRE-CONFERENCE WORKSHOPS AND SPECIAL SESSIONS

Germany PAPER REVIEW COORDINATION

Kong, Hong Kong

Kong, Hong Kong DOCTORAL CONSORTIUM

Volume 1: Full Papers

Volume 1: Full Papers

Volume 1: Full Papers

Volume 1: Full Papers

Volume 1: Full Papers

Volume 1: Full Papers

ABOUT "COCO" AT THE UNIVERSITY OF SYDNEY

Volume 1: Full Papers

Volume 1: Full Papers

Volume 1: Full Papers

Understanding How Learners Grapple with Wicked Problems in Environmental Science

Semiosis Process in Joint Didactic Action

Learning How to Create: Toward A Learning Sciences of Art and Design

How Metacognitive Awareness Caused A Domino Effect in Learning

Investigating the Relative Difficulty of Various Complex Systems Ideas in Biology

The Impact of Students Exploration Strategies in Discovery-Based Instructional Software

Design Research in Early Literacy within the Zone of Proximal Implementation

Kitchen Chemistry: Supporting Learners Decisions in Science

Volume 1: Full Papers

Beyond the Screen: Game-Based Learning As A Nexus of Identification

Exploring Connectedness: Applying ENA to Teacher Knowledge

Investigating the Effects of Varying Labels as Scaffolds for Visitor Learning

Functional Aesthetics for Learning: Creative Tensions in Youth e-Textile Designs

196 204 212

Learning Progressions, Learning Trajectories, and Equity

A Teacher's Journey in Knowledge Building Pedagogy

Alternate Reality Games: Platforms for Collaborative Learning

Volume 1: Full Papers

Peer Collaboration and Mediation in Elementary Students Lamp Designing Process

267 275 283

Dynabook: Supporting Teacher Learning about Mathematical Thinking

Embodied Artifacts and Conceptual Performances

The Impact of Structural Characteristics of Concept Maps on Automatic Quality Measurement

Toward a Cognitive Framework of Interdisciplinary Understanding

Using the Idea Manager to Promote Coherent Understanding of Inquiry Investigations

Technology Supports in CSCL

339 347 355

Situating epistemological development

Improving Revision in Wiki-based Writing: Coordination Pays off

Distributing Practice: Challenges and Opportunities for Inquiry Learning

Metadiscourse to Foster Student Collective Responsibility for Deepening Inquiry

Consequential Feedback as a Means of Supporting Student Engagement and Understanding 403

Multiple Trajectories for Understanding Ecosystems

Volume 1: Full Papers

Inter-Identity Technologies for Learning

427 435 443 451 459

Predicting Idea Co-Construction in Speech Data using Insights from Sociolinguistics

Automatically extract interpretable topics from online discussion

Finding Voices and Emerging Agency in Classroom Learning