A transition evaluation model with probability-based effectiveness indicators—a new measurement model for problem-solving process data

doi:10.3758/s13428-026-02994-2

A transition evaluation model with probability-based effectiveness indicators—a new measurement model for problem-solving process data

2026-04-16

0

PRE

AI

DOI：10.3758/s13428-026-02994-2

Original

Original request for help

Save

Abstract

En 中文

Computer-based interactive tasks generate rich process data that capture respondents’ problem-solving behaviors, particularly sequences of actions that trigger transitions between problem states. In recent years, the process-based measurement models analyzing transition sequences have emerged as a promising approach for estimating latent problem-solving ability. A fundamental step in developing these models is the predefinition of the transition effectiveness. However, existing effectiveness indicators are often limited to restricted value ranges (e.g., dichotomous or polytomous scales) and theoretical perspective of expert evaluation, thereby constraining the flexibility of process-based models. To address these limitations, this study introduces two probability-based indicators: state effectiveness $${p}_{s}$$ and transition effectiveness $${\Delta p}_{s\to {s}{\prime}}$$ . When validated using empirical data from the PISA 2012 problem-solving assessment, the probability-based effectiveness indicators exhibited a broader range of numerical values, enabling finer-grained discrimination among states and transitions. Subsequently, we developed the transition evaluation model (TEM), a process-based model that incorporates the transition effectiveness $${\Delta p}_{s\to {s}{\prime}}$$ to better differentiate transition characteristics. Simulation study demonstrated TEM’s robust parameter estimation, satisfactory model-data fit, and high estimation accuracy across diverse conditions. In an empirical study, TEM outperformed three models, including the Sequential Response Model (SRM), the State Response Measurement Model (SRMM), and SRM with Polytomous Effectiveness Indicators (SRM-PEI) in terms of model-data fit, and yields more nuanced transition characteristic curves and interpretable ability estimates. These findings underscore the value of probability-based effectiveness indicators and TEM as advanced tools for analyzing complex problem-solving assessments.

Keywords:

Process data

Problem-solving assessment

State and transition effectiveness

Probability-based effectiveness indicators

Transition evaluation model

A transition evaluation model with probability-based effectiveness indicators—a new measurement model for problem-solving process data

Abstract

Journal

Organization