Most influential research done by our lab members, including best award papers at conferences!
Examining player estimates of prior video gaming experience: comparing single-item and multiple-item measures.
Abstract:
The construct of game experience encompasses two distinct dimensions: experience within a video game and experience with a video game. For the purposes of this study, the authors focused on prior experience with video games. The construct of prior video gaming experience has been typically defined in terms of time spent playing and operationalized in two distinctly different ways: 1) an estimate of overall time spent playing video games (e.g., single-item measure); or 2) a cumulative estimate of time spent playing split by video game genre (e.g., multiple-item measure). Despite these extant measurement perspectives, a search of the literature did not reveal a rationale for the disparate granularity with which the prior video game experience construct is captured. However, these variations in approach to measuring prior video game experience may impact the findings of game-based research. This study was conceptualized to begin a methodological conversation centering on how construct operationalization could influence study outcomes. A common scale adaptation process was applied to examine whether prior gaming experience measured by a single-item compared to a multiple-item instrument impacted the predicted likelihood of game-based outcomes. Results demonstrated that while both overall and genre-based measures of prior video game experience measures are tenable in game-based research, genre-based measures appear to yield more clarity for both task-specific and outcome-based research questions.
McCreery, M. P., Head, D. L., Leif, S. A., Fiorentini, J., Krach, S. K., & Leung, Z. K. (2024). Examining player estimates of prior video gaming experience: comparing single-item and multiple-item measures. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-023-17919-w
Games as Stealth Assessments A volume in the Advances in Educational Technologies and Instructional Design (AETID) Book Series
Abstract:
In the world of assessment, traditional methods often fall short, providing limited insight into individuals’ skills and abilities while being susceptible to response biases. Recognizing these shortcomings, researchers have delved into the realm of stealth assessments, a novel approach that embeds traditional measurement techniques within a game-based environment. By seamlessly integrating assessment into gameplay, stealth assessments offer a contextually rich and unobtrusive method of data collection, allowing for a comprehensive understanding of the constructs being assessed. Games as Stealth Assessments unveils the promising field of stealth assessment, exploring its design considerations, research methods, and practical applications. Drawing upon a foundation of psychometrically-sound assessment practices, this book delves into the intersection of thoughtful game design and empirical support for the use of stealth assessments. It justifies the adoption of stealth assessments in academic disciplines such as mathematics, science, and literacy, as well as in the assessment of psychological constructs like aggression, social skills, and self-regulation. Catering to instructors, researchers, and practitioners in the fields of Curriculum and Instruction, Media Psychology, Applied Psychology, and Educational Psychology, this comprehensive guide offers a framework for designing, developing, and conducting research in the emerging field of stealth assessment. Covering essential topics such as the history of stealth assessments, evidence models, technical architecture, psychometrics, data mining, and game design principles, this book equips readers with the knowledge and tools needed to harness the power of stealth assessments effectively. Through a systematic review of stealth assessments, readers gain insights into successful strategies for assessing specific constructs and capturing their manifestation over time and context. The book also provides practical guidelines, highlighting the do’s and don’ts of stealth assessments, ensuring that practitioners can navigate this innovative assessment approach with confidence.
McCreery, M. P., & Krach, S. K. (Eds.). (2023). Games as Stealth Assessments: IGI Global. https://doi.org/10.4018/979-8-3693-0568-3
Best Paper Award: A Hardcore Look at the Casual Measurement of Gamer Experience
Abstract:
Gamer experience is measured in terms of time spent playing video games. Two main perspectives exist regarding the operationalization of gamer experience: a single-item construct grounded in frequency and a composite construct grounded in game type or genre. Currently, despite these two dominant measurement perspectives, a search of the literature did not reveal a rationale for the disparate granularity with which the gamer experience construct is captured. However, the noted variations in the approach to measuring the gamer experience construct may have substantive impacts on the findings and conclusions in game-based research. This led to the development of the overarching question associated with the current study: how might a singular construct versus a composite construct of gamer experience influence research findings? Single-item and composite measures of gamer experience (N=158) were regressed on outcomes associated with two playthroughs of a video game (i.e., The Deed). Results demonstrated clear advantages in employing a composite construct over a single-item construct. Implications for these findings are discussed.
McCreery, M., Head, D., & Leif, S. (2022). A Hardcore Look at the Casual Measurement of Gamer Experience. In International Organization of Social Sciences and Behavioral Research Fall Conference. Las Vegas, Nevada.
Examining the Influence of Cognitive Ability on Situating to a Video Game: Expanded Discussion
Abstract:
The current study examined the relationship between an individual player’s cognitive ability and their in-game experience. The focus of this work was examining how players think within, and acclimate to, video games as complex and contextualized systems. Specifically, researchers examined the relationship between a player’s cognitive ability (i.e., score on the Shipley-2) and video gameplay outcomes (i.e., summative data from The Deed) in a sample of college-age students (N = 144). Gameplay outcomes were evaluated and contrasted after two discrete playthroughs, each lasting approximately 30 min. Logistic regression indicated that cognitive ability predicted individual outcomes for both the first and second playthrough, but did not predict growth between playthroughs (i.e., outcome change). The differences between playthrough performance appear to be related to acclimatization. Each playthrough experience with the game highlighted separate cognitive abilities needed to acclimate (crystallized intelligence) and progress (fluid intelligence) within the video game. Implications of these findings for video game-based research design are discussed.
McCreery, M. P., Head, D. L., Leif, S. A., Fiorentini, J. P., Bacos, C. A., Laferriere, J. R., Kathleen Krach, S., & Cole, L. Q. L. (2022). Correction to: Examining the Influence of Cognitive Ability on Situating to a Video Game: Expanded Discussion. In D. Ifenthaler, P. Isaías, & D. G. Sampson (Eds.), Orchestration of Learning Environments in the Digital World (pp. C1–C1). Springer International Publishing. https://doi.org/10.1007/978-3-030-90944-4_17
Can Video Games Be Used as a Stealth Assessment of Aggression?: A Criterion-Related Validity Study
Abstract:
The current pilot study examined how well a reflective moral-choice video game predicted the rating scale scores of aggression types. To begin, the authors used a coding system to examine in-game proactive and reactive behaviors. This analysis resulted in a tallied score for each construct. These game-based scores were then included in regression models, examining how well within-game behaviors predict scores on a pre-existing rating scale of both proactive and reactive aggression. Findings indicated that game-based proactive scores were not predictive of proactive aggression ratings; however, reactive game-based scores were predictive of reactive aggression ratings. Implications for these findings are discussed.
McCreery, M. P., Krach, S. K., Bacos, C. A., Laferriere, J. R., & Head, D. L. (2022). Can Video Games Be Used as a Stealth Assessment of Aggression?: A Criterion-Related Validity Study. In I. R. Management Association (Ed.), Research Anthology on Game Design, Development, Usage, and Social Impact (pp. 774–784). IGI Global. https://doi.org/10.4018/978-1-6684-7589-8.ch038
Best Paper Award: Time to Acclimate: Examining the Influence of Cognitive Ability on Situating to a Video Game
Abstract:
The current study was designed to examine the relationship between cognitive ability and player experience in shaping how players think within, and acclimate to, video games as complex systems. Specifically, researchers examined the relationship between player’s cognitive ability and gameplay outcomes within a video game (i.e., The Deed). Outcomes were evaluated and contrasted after two discrete playthroughs, each lasting approximately 30 minutes. Logistic regression indicated that cognitive ability predicted individual outcomes for both the first and second playthrough, but did not predict growth between playthroughs (i.e., outcome change). Findings are then discussed in terms of an acclimation stage.
Time To Acclimate: Examining The Influence Of Cognitive Ability On Situating to a Video Game. (2020). Proceedings of the 17th International Conference on Cognition and Exploratory Learning in the Digital Age (CELDA 2020). 17th International Conference on Cognition and Exploratory Learning in the Digital Age. https://doi.org/10.33965/celda2020_202014L020
Machine Learning and Education in the Human Age: A Review of Emerging Technologies
Abstract:
Today’s emerging technologies are moving in the direction of the human age. As technologies emerge in the educational setting, humans can evolve in the way they learn because of technology; and machines can evolve in the way they learn because of humans. Technology can be used to effectively observe and assesses human behaviors to better understand and respond to them. Whether the behaviors need to be adjusted or the behaviors are worth modeling, technology can provide support such as tools to track and collect data, assess performance, and provide meaningful feedback to the learner. In the human age of machine learning, the focus is less on technology and more on being human. To adapt to changing educational contexts, more effective applications of emerging technologies are needed. This paper explores the following novel applications of human-centered approaches to using technology in education: the quantified self, affective computing, emotional design, and pedagogical agents.
Bacos, C. A. (2020). Machine Learning and Education in the Human Age: A Review of Emerging Technologies. In K. Arai & S. Kapoor (Eds.), Advances in Computer Vision (Vol. 944, pp. 536–543). Springer International Publishing. https://doi.org/10.1007/978-3-030-17798-0_43
Transformative Social and Emotional Learning: Examining Learning Management Systems Through the Pairing of Digital Learning Environments
Abstract:
COVID-19 required teachers and administrators to swiftly transition traditional education into learning management systems (LMS). However, LMSs are designed as delivery trucks, providing content to students and serving as digital filing cabinets that organize and deliver information in an appropriate manner. This inherent limitation allows for the disenfranchisement of underprivileged groups, which can be addressed via the transformative social and emotional learning framework (TSEL). TSEL is a method of alleviating inequitable learning experiences by accounting for racial oppressions that marginalized groups encounter within education. Here, the authors discuss how to leverage additional digital learning environments to enhance and support identity, belongingness, intersectionality, and agency during online learning. These digital learning environments are detailed from a research perspective as well as a practical one, allowing readers to immediately implement concurrent extensions to their LMS.
Fiorentini, J. P., McCreery, M. P., Cole, L. Q. L., Leif, S. A., Monk, M. M., Bagneris, J. R., & Head, D. L. (2021). Transformative Social and Emotional Learning: Examining Learning Management Systems Through the Pairing of Digital Learning Environments. In C. M. Crawford & M. A. Simons (Eds.), Advances in Educational Technologies and Instructional Design (pp. 180–202). IGI Global. https://doi.org/10.4018/978-1-7998-6956-6.ch009
Examining Designed Experiences: A Walkthrough for Understanding Video Games as a Performance Assessments
Abstract:
Although there is an extensive body of video game research, very few resources exist that outline the potential for games to provide process-oriented data related to human behavior and learning. The current chapter offers guidance for researchers to extract dynamic, emergent, and complex data from video game contexts, and thus unlock the potential for games to function as performance assessments. This chapter describes how to leverage behavioral observation as a means to examine player interactions that are associated with knowledge acquisition, formative (within-game) activities, and summative (end-game) outcomes. Specifically, through employing a moral-choice video game as an example, we walk through deconstructing the game, defining game elements, and the construction of a behavioral observation protocol. Although some games aren’t well-suited for this approach, we provide a framework for how in-game player interactions can be captured, coded, and analyzed. Each step is used to illustrate how games can function as performance assessments.
McCreery, M. P., Schrader, P. G., Krach, S. K., Laferriere, J. R., Bacos, C. A., & Fiorentini, J. P. (2019). Examining Designed Experiences: A Walkthrough for Understanding Video Games as Performance Assessments. In D. Ifenthaler & Y. J. Kim (Eds.), Game-Based Assessment Revisited (pp. 105–119). Springer International Publishing. https://doi.org/10.1007/978-3-030-15569-8_6
Interactive Narratives, Counterfactual Thinking and Personality in Video Games
Abstract:
Interactive narratives in video games allow players to experience a variety of storyline pathways, inviting players to think about alternative choices that might lead to different outcomes. These branching story structures can induce counterfactual thinking, the process of forming mental representations of past events to imagine alternative outcomes. Video games can be used to track and analyze behaviors during gameplay that are indicative of cognitive processes. As a gameplay analytics approach to assessing such behaviors, video games afford the measurement of factors that may influ-ence those behaviors. Little is known about how personality influences the use of counterfactuals. As such, the focus of the present study was to inves-tigate the effect of the Big Five personality traits (i.e., agreeableness, conscientiousness, emotional stability, extraversion, and intellect) on behavioral choices indicative of counterfactual thinking (CFT). Participants (N=132) played an interactive, narrative-based video game twice. In-game behaviors indicative of CFT (i.e., changes in answer choices across gameplays) were coded and analyzed to determine whether they were dependent on partici-pants’ in-game experiences (i.e., outcomes and valence of answer choices) and individual differences (i.e., personality dimensions). The outcome of failure and the valence of answer choices in the first gameplay had significant effects on CFT. The results also indicated a significant interaction between the outcome of the first gameplay and conscientiousness in their ef-fect on CFT. Implications for these findings are discussed.
Bacos, C. A., McCreery, M. P., & Laferriere, J. R. (2018). Interactive Narratives, Counterfactual Thinking and Personality in Video Games. In C. Stephanidis (Ed.), HCI International 2018 – Posters’ Extended Abstracts (Vol. 850, pp. 340–347). Springer International Publishing. https://doi.org/10.1007/978-3-319-92270-6_49
