.png){kind=icon}
.png){kind=icon}
Return
Promoting open science in test-driven software experiments
.png){kind=icon}
OA
AI
DOI：10.1016/j.jss.2024.111971.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
A core principle of open science is the clear, concise and accessible publication of empirical data, including rawobservational data as well as processed results. However, in empirical software engineering there are no established standards (de jure or de facto) for representing and openingobservations collected in testdriven software experiments - that is, experiments involving the execution of software subjects in controlled scenarios. Execution data is therefore usually represented in ad hoc ways, often making it abstruse and difficult to access without significant manual effort. In this paper we present new data structures designed to address this problem by clearly defining, correlating and representing the stimuli and responses used to execute software subjects in test-driven experiments. To demonstrate their utility, we show how they can be used to promote the repetition, replication and reproduction of experimental evaluations of AI -based code completion tools. We also show how the proposed data structures facilitate the incremental expansion of execution data sets, and thus promote their repurposing for new experiments addressing new research questions.
Keywords:
Software
Engineering
Empirical
Experimentation
Observation
Behavior
Reproducibility
Replication
Data structures
Open science
Large language models
Machine learning
Generative artificial intelligence
Benchmark
Language-to-code
HumanEval
Automation
Measurement
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
4.1
Papers:
5.4K
Citations:
8.4K