[Context and motivation] Trace matrices are lynch pins for the development of mission- and safety-critical software systems and are useful for all software systems, yet automated methods for recovering trace links are far from perfect. This limitation makes the job of human analysts who must vet recovered trace links more difficult. [Question/Problem] Earlier studies suggested that certain analyst behaviors when performing trace recovery tasks lead to decreased accuracy of recovered trace relationships. We propose a three-step experimental study to: (a) determine if there really are behaviors that lead to errors of judgment for analysts, (b) enhance the requirements tracing software to curtail such behaviors, and (c) determine if curtailing such behaviors results in increased accuracy. [Principal ideas/results] We report on a preliminary study we undertook in which we modified the user interface of RETRO.NET to curtail two behaviors indicated by the earlier work. We report on observed results. [Contributions] We describe and discuss a major study of potentially unwanted analyst behaviors and present results of a preliminary study toward determining if curbing these behaviors with enhancements to tracing software leads to fewer human errors.
Automated tracing, generating or recovering the relationship between artifacts of the software development process, has been well researched over the past 15 years [4], but this automation doesn't come without inherent costs. One such cost is the need for human analysts to interact with the results of the automated methods. What we currently know about such interactions is that they tend to end disappointingly [1,2,6]. As long as we are using automated tracing methods for safety-and mission-critical systems, we must have humans vet the links. Therefore, we need to figure out how to make humans more accurate as they work with the results of automated methods. In prior studies we noticed some unwanted behaviors [1,2,6]. Can we curb them? Will curbing them yield fewer human errors?
A trace matrix is a collection of trace links, defined as “a specified association between a pair of artifacts, one comprising the source artifact and one comprising the target artifact.” by the Center of Excellence for Software and System Traceability (COEST) [3]. A plethora of researchers have designed techniques for automatically or semi-automatically generating trace matrices, many discussed in a comprehensive survey by Borg [4]. Most of the focus in that work was on improving the quality of the candidate trace matrix, the matrix generated by a software method. While that work continues, recent work has segued into study of the analyst who works with the candidate matrix to generate the final trace matrix -the one that is used in application.
A typical trace tool, such as RETRO.NET used in this work [5], displays the candidate trace matrix and shows the list of source (high level) elements, and the list of candidate target (low level) elements that were automatically mapped to the source element. The texts of all elements can also be viewed. The key function of a tracing tool is to allow the analyst to vet individual candidate links.
Cuddeback et al. [1] and Dekhtyar et al. [2] studied the work of analysts with candidate trace matrices produced by automated software. The analysts were presented a candidate trace matrix and were asked to evaluate the individual links and correct any errors of omission or commission. The accuracy of candidate trace matrices varied from analyst to analyst -from high-accuracy matrices that contained few omitted links and few false positives to low-accuracy ones which contained many errors of both types. The studies found that analysts working with high accuracy candidate traces tended to decrease the accuracy -i.e., introduce false links into the matrix and remove true links, whereas the analysts who had low accuracy matrices tended to improve the accuracy significantly 1 . A follow-up study collected logs of analyst activity during the tracing process, and looked at behaviors that correlated with improved or decreased accuracy [6]. While that study did not have enough data points to allow for statistical significance of the results, the authors observed a number of analyst behaviors that tended to lead to errors of judgement. Specifically, two behaviors briefly described below were observed.
Long time to decide. When analysts took unusually long (for their pace) time to decide whether a candidate link needed to be kept in the trace, they tended to make an incorrect decision [6].
When analysts revisited a link on which they already entered a decision and reversed that decision, they tended to err [6].
Our motivation for the continuing study of analyst behavior in tracing tasks comes from the key observations from the prior work [4,1,2,6]. On one hand, the lack of traceability as a byproduct of development in large software projects demonstrates a clear need for accurate automatic tracing methods [4]. At the same time, human analysts, when asked to curate automatically obtained traceability relations, make mistakes and decrease the overall accuracy of the trace [1,2]. We observe that one possible way to resolve this, and to improve the accuracy of curated trace relations is, potentially, to curb analyst behaviors that result in errors. In fact, psychologists studying human decision-making have observed that humans tend to operate in one of two decision-making systems -System 1 (S1) (or fast, instinctive thinking) or System 2 (S2) (slow, deliberate, logical thinking) [8]. The observed behaviors leading to decrease in accuracy belong to System 2. This motivates an additional research question expressed below.
The latter observation serves as the inspiration for our next step in the study of the behavior of human analysts. In this section we discuss the overall plan for the study, as well as the preliminary work we conducted.
The study we are planning to undertake consists of three key research questions.
We hypothesize that such behaviors can be observed as statistically significant. We additionally conjecture that such behaviors would correspond to the decision-making System 2 [8].
We hypoth
This content is AI-processed based on open access ArXiv data.
