A multiple comparative study of test-with development product changes and their effects on some attributes of team speed and product quality

Thesis

While studying Test-First Development (TFD), researchers have typically compared it to Test-Last Development (TLD), across groups or projects, and for relatively short durations. But, we have observed that teams testing fine-grained changes do not necessarily develop tests for those changes first. And, comparing across groups or projects is difficult, due to the "contextual variables" specific to each. Finally, many "real-life" projects have long durations.

Because of TFD’s potential to increase team speed or product quality, and because of these previous limitations, we were motivated to design and execute our own related study. Our work on this study has resulted in three original contributions to knowledge: (1) our software change model; (2) our measurement method; and (3) our study results.

First, as part of our software model, we defined Test-With Development (TWD) as more general than TFD, and as a step that can be practised to a degree. Then, we defined team speed measures for a developer's product change: initial net size, discards over time, and net size over time. As well, we adopted product quality measures for a developer's product change: change in the number of potential bugs and change in the average method complexity.

Second, we defined a measurement method which allowed us to perform two studies, across six open-source projects and thousands of developer changes. With it, we performed an auxiliary study to calibrate our discards over time measure. Then, we used it to perform our main study to compare the effects of TWD product changes on some attributes of team speed and product quality.

Finally, the results from our main study indicate that there were significant differences between the effects of TWD product changes. We interpret these results to indicate that teams that practice TWD can increase their efficiency, by reducing their product’s complexity faster.

Authors: Bannerman, S

• DOI: 10.5287/ora-r5kgjn17m
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford