Methodological Variation in Studying Staff and Student Perceptions of AI

In this paper, we compare methodological approaches for comparing student and staff perceptions, and ask: how much do these measures vary across different approaches? We focus on the case of AI perceptions, which are generally assessed via a single quantitative or qualitative measure, or with a mixed methods approach that compares two distinct data sources - e.g. a quantitative questionnaire with qualitative comments. To compare different approaches, we collect two forms of qualitative data: standalone comments and structured focus groups. We conduct two analyses for each data source: with a sentiment and stance analysis, we measure overall negativity/positivity of the comments and focus group conversations, respectively. Meanwhile, word clouds from the comments and a thematic analysis of the focus groups provide further detail on the content of this qualitative data - particularly the thematic analysis, which includes both similarities and differences between students and staff. We show that different analyses can produce different results - for a single data source. This variation stems from the construct being evaluated - an overall measure of positivity/negativity can produce a different picture from more detailed content-based analyses. We discuss the implications of this variation for institutional contexts, and for the comparisons from previous studies.

💡 Research Summary

The paper investigates how methodological choices shape the findings on AI perceptions among university students and staff. Recognising that generative AI has rapidly entered higher education, the authors note that existing studies often rely on a single method—typically a survey or interview—leading to potentially contradictory conclusions about stakeholder attitudes. To address this gap, the authors collected three qualitative data sources from the same participant pool: (1) free‑form written comments embedded in a large‑scale survey, (2) structured digital comments posted on a Padlet board, and (3) semi‑structured focus‑group discussions. Each source contains responses from both students and staff, allowing direct comparison across groups and methods.

Four analytical lenses were applied to each data set: (i) sentiment analysis, using a hybrid of lexicon‑based (VADER) and BERT‑based classifiers to produce positive, negative, and neutral polarity scores; (ii) stance analysis, which categorises each utterance as supportive, oppositional, or neutral toward AI integration; (iii) word‑cloud generation based on TF‑IDF weighting to visualise the most frequent lexical items; and (iv) thematic analysis following Braun & Clarke’s six‑step framework to identify overarching themes and sub‑themes.

The results reveal substantial divergence depending on the analytical approach. Sentiment analysis shows a relatively optimistic picture for students (≈62 % positive) and a more mixed picture for staff (≈48 % positive, 38 % negative). However, stance analysis uncovers that roughly 45 % of both groups express reservations about AI, indicating that a simple polarity metric masks nuanced concerns. Word‑clouds highlight distinct lexical emphases: students frequently mention “efficiency,” “help,” and “time‑saving,” whereas staff foreground “ethics,” “privacy,” and “assessment.” Thematic analysis extracts four shared macro‑themes—efficiency, academic integrity, ethics/privacy, and educational quality—and several group‑specific sub‑themes (e.g., students focus on task‑specific benefits, staff on pedagogical quality and job security).

Crucially, the study demonstrates that even when the raw data are identical, the choice of analytic lens can produce different narratives about the same phenomenon. This methodological variability stems from (a) differences in granularity (sentence‑level polarity versus discourse‑level stance) and (b) the contextual nature of the data source (written comments tend toward socially desirable positivity, while live focus‑group dialogue elicits more spontaneous, affect‑laden expressions).

The authors argue that policy makers and institutional leaders should not rely on a single survey‑based sentiment score when shaping AI governance. Instead, a triangulated approach that combines sentiment, stance, lexical profiling, and thematic coding provides a richer, more reliable picture of stakeholder concerns. They also call for further work on improving the accuracy of automated stance detection and for meta‑analyses that account for cultural and linguistic variation in qualitative AI perception studies.

In sum, the paper makes three key contributions: (1) empirical evidence that methodological choices materially affect AI perception findings; (2) a demonstration of the added value of multi‑method triangulation for capturing both surface‑level attitudes and deeper, context‑specific worries; and (3) practical guidance for designing robust perception studies and for developing AI policies that are responsive to the distinct yet overlapping concerns of students and staff.