Evaluating the Effectiveness and Appeal of Universitas XYZ’s SIAKAD Using the User Experience Questionnaire and the UX-GAP Index Approach

Abstract

The evaluation of academic information systems (SIAKAD) in Indonesia consistently reveals a gap between high pragmatic-cluster scores and low hedonic-cluster scores on the User Experience Questionnaire (UEQ). Prior studies typically stopped at reporting per-dimension scores without compressing this gap into a single diagnostic index or testing it through hypotheses. This study operationalized the UX-GAP Index as a scalar metric (G = the directed difference between the pragmatic and hedonic cluster means) and tested three hypotheses on a university SIAKAD together with its consistency against verified literature. Data were collected from 124 active students using the 26-item UEQ; the analysis comprised a paired t-test, Pearson inter-dimension correlation, and a directional synthesis of UEQ studies on Indonesian higher-education SIAKAD/SIA selected through an explicit procedure and verified against primary sources. The pragmatic cluster significantly exceeded the hedonic cluster (t = 4.37; p < 0.001) with a UX-GAP Index of G = 0.22 (Mild Imbalance), and Perspicuity correlated strongly with Efficiency (r = 0.79; p < 0.001). Four verified comparator studies confirmed the pragmatic > hedonic direction with G ranging 0.32–0.57, although one system was an exception. The pragmatic-hedonic gap is therefore a general tendency rather than a universal law. The proposed UX-GAP Index is operational, with heuristic category thresholds that still require further validation.