The Effect of Gemini AI-enhanced E-portfolios on Students’ Speaking Performance: A Quantitative Analysis in ESP Context
Abstract
This study investigates the quantitative impact of Gemini AI-enhanced e-portfolio assessment on English speaking performance among ESP students in a vocational education context. Despite the growing integration of artificial intelligence in language education, empirical evidence examining the effectiveness of multimodal AI systems on speaking skill development remains limited, particularly within ESP contexts where students require targeted professional language competencies. Using a one-group pretest-posttest design, 30 second-semester International Business Management students at Politeknik Negeri Bali participated in a 14-week intervention utilizing Gemini AI-integrated e-portfolio assessment. Speaking performance was measured through comprehensive assessments based on the Complexity, Accuracy, and Fluency (CAF) framework, incorporating formal business presentations and structured role-play negotiation scenarios representative of authentic professional discourse demands. Results from paired samples t-test analysis revealed statistically significant improvements following the intervention (t (29) = 15.847, p < 0.001, Cohen's d = 2.89), with mean speaking scores increasing substantially from 62.45 (SD = 8.23) to 78.67 (SD = 6.45). Component-level analysis demonstrated differential yet consistently strong effects across all CAF dimensions: fluency (d = 2.52), complexity (d = 2.27), and accuracy (d = 1.87). The very large effect sizes indicate substantial practical significance, exceeding effect sizes reported in previous technology-enhanced speaking interventions by factors of four to eight. These findings contribute critical empirical evidence supporting AI integration in vocational language education and suggest that Gemini AI-enhanced e-portfolio assessment represents a paradigm shift in ESP pedagogy, with important implications for curriculum redesign and institutional technology adoption decisions in professional preparation programs.
Full text article
References
Ali, A. H., Alajanbi, M., Yaseen, M. G., & Abed, S. A. (2023). Chatgpt4, DALL· E, Bard, Claude, BERT: Open possibilities. Babylonian Journal of Machine Learning, 2023, 17–18. https://doi.org/10.58496/BJML/2023/003.
Alrasheedi, S. (2020). Investigation of factors influencing speaking performance of Saudi EFL learners. Arab World English Journal (AWEJ) Volume, 11(4). https://ssrn.com/abstract=3764220.
Badshah, A., Rehman, G. U., Farman, H., Ghani, A., Sultan, S., Zubair, M., & Nasralla, M. M. (2023). Transforming educational institutions: Harnessing the power of internet of things, cloud, and fog computing. Future Internet, 15(11), 367. https://doi.org/10.3390/fi15110367.
Bolliger, D. U., & Shepherd, C. E. (2010). Student perceptions of ePortfolio integration in online courses. Distance Education, 31(3), 295-314.
Carlà, M. M., Gambini, G., Baldascino, A., Giannuzzi, F., Boselli, F., Crincoli, E., D’Onofrio, N. C., & Rizzo, S. (2024).
Exploring AI-chatbots’ capability to suggest surgical planning in ophthalmology: ChatGPT versus Google Gemini analysis of retinal detachment cases. British Journal of Ophthalmology. https://doi.org/10.1136/bjo-2023-325143.
Chaisiri, P. (2023). Promoting Thai EFL learners’ willingness to communicate in the virtual classroom through technology-mediated oral tasks. LEARN Journal, 16(1), 97-120.
Chantaraphat, Y., & Jaturapitakkul, N. (2023). Use of peer tutoring in improving the English speaking ability of Thai undergraduate students. rEFLections, 30(3), 826-849.
Diyyab, E. A. E. (2023). Using contemporary talk shows to develop faculty of education English major’s EFL creative speaking skills and speaking self-efficacy. BSU Journal of Pedagogy and Curriculum, 2(3), 43-79. https://doi.org/10.21608/bsujpc.2023.270980.
El-Gawad, A. M. A. S. A. (2022). The effect of using a technological authentic task-based approach on enhancing EFL secondary stage students’ language use. Online Submission. ERIC.
Gozali, I., Wijaya, A. R. T., Lie, A., Cahyono, B. Y., & Suryati, N. (2024). Leveraging the potential of ChatGPT as an automated writing evaluation (AWE) tool: Students’ feedback literacy development and AWE tools integration framework. JALT CALL Journal, 20(1), 1-22. https://doi.org/10.29140/jaltcall.v20n1.1200.
Hill, J., Berlin, K., Choate, J., Cravens-Brown, L., McKendrick-Calder, L., & Smith, S. (2021). Exploring the emotional responses of undergraduate students to assessment feedback: Implications for instructors. Teaching and Learning Inquiry, 9(1), 294-316.
Housen, A., Kuiken, F., & Vedder, I. (2012). Complexity, accuracy and fluency. Dimensions of L2 Performance and Proficiency, 32, 1-20.
Ibrahim, K. A. A., & Basim, N. (2024). Applying task supported language learning via Instagram: The effects on EFL learners’ self-esteem, academic buoyancy, academic engagement, and language achievement. Asian-Pacific Journal of Second and Foreign Language Education, 9(1), 2. https://doi.org/10.1186/s40862-023-00223-3.
Imran, M., & Almusharraf, N. (2024). Google Gemini as a next generation AI educational tool: A review of emerging educational technology. Smart Learning Environments, 11(1), 22. https://doi.org/10.1186/s40561-024-00310-z.
Jabbari, N., & Peterson, M. (2023). Complexity, accuracy, and fluency improvements through massively multiplayer online gaming: A longitudinal mixed-methods case study. The Language Learning Journal, 51(4), 416-450. https://doi.org/10.1080/09571736.2023.2219713.
Klenowski, V., Askew, S., & Carnell, E. (2006). Portfolios for learning, assessment and professional development in higher education. Assessment & Evaluation in Higher Education, 31(3), 267-286.
Kotmungkun, S., Chompurach, W., & Thaksanan, P. (2024). OpenAI ChatGPT vs Google Gemini: A study of AI chatbots’ writing quality evaluation and plagiarism checking. English Language Teaching and Education Journal, 7(2), 90-108. https://doi.org/10.12928/eltej.v7i2.11572.
Kusuma, I. P. I., & Waluyo, B. (2023). Enacting e-portfolios in online English-speaking courses: Speaking performance and self-efficacy. Iranian Journal of Language Teaching Research, 11(1), 75-95.
Kusuma, I. P. I., Pratiwi, N. P. A., Paramartha, A. G. Y., & Utami, I. A. M. I. (2022). Can e-portfolios reduce foreign language anxiety in online speaking courses during the COVID-19 pandemic? Computer-Assisted Language Learning Electronic Journal, 23(2), 38–55.
Kusuma, I. P. I., Roni, M., Dewi, K. S., & Mahendrayana, G. (2024). Revealing the potential of ChatGPT for English language teaching: EFL preservice teachers’ teaching practicum experience. Studies in English Language and Education, 11(2), 650–670. https://doi.org/10.24815/siele.v11i2.34748
Lee, G.-G., Shi, L., Latif, E., Gao, Y., Bewersdorff, A., Nyaaba, M., Guo, S., Wu, Z., Liu, Z., & Wang, H. (2023). Multimodality of AI for education: Towards artificial general intelligence. arXiv preprint arXiv:2312.06037. https://doi.org/10.48550/arXiv.2312.06037.
Milne-Ives, M., Selby, E., Inkster, B., Lam, C., & Meinert, E. (2022). Artificial intelligence and machine learning in mobile apps for mental health: A scoping review. PLOS Digital Health, 1(8), e0000079.
Nyaaba, M. (2024). Comparing human and AI’s (GPT-4 and Gemini) understanding of the nature of science. Society for Information Technology & Teacher Education International Conference (p. 10). Association for the Advancement of Computing in Education.
Ogawa, C. (2022). CAF indices and human ratings of oral performances in an opinion-based monologue task. Language Testing in Asia, 12(1), 4. https://doi.org/10.1186/s40468-022-00154-9.
Rane, N., Choudhary, S., & Rane, J. (2024). Gemini versus ChatGPT: Applications, performance, architecture, capabilities, and implementation. Journal of Applied Artificial Intelligence, 5(1), 69-93. https://doi.org/10.48185/jaai.v5i1.1052.
Saeidnia, H. R. (2023). Welcome to the Gemini era: Google DeepMind and the information industry. Library Hi Tech News, ahead-of-print. https://doi.org/10.1108/LHTN-12-2023-0214.
Santosa, M. H., Putra, I. B. Y. B. S., & Pratiwi, N. P. A. (2024). EFL Students’ Online Learning Readiness: Performing Out the Basic Functions of Technology. Pedagogy: Journal of English Language Teaching, 12(1), 59–80. https://doi.org/10.32332/joelt.v12i1.3590.
Shahzad, M. F., Xu, S., Lim, W. M., Yang, X., & Khan, Q. R. (2024). Artificial intelligence and social media on academic performance and mental well-being: Student perceptions of positive impact in the age of smart learning. Heliyon, 10(8), e29523.
Suardewa, P. M. S., Kusuma, I. P. I., & Dewi, K. S. (2024). The The Impacts of Preservice English Teachers’ Self-efficacy of Using AI Towards Their Intentions of Teaching Writing Skills Using AI. Jurnal Pendidikan Bahasa Inggris Undiksha, 12(1), 110–119. https://doi.org/10.23887/jpbi.v12i1.80827.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., & Millican, K. (2023). Gemini: a family of highly capable multimodal models. ArXiv Preprint ArXiv:2312.11805. https://doi.org/10.48550/arXiv.2312.11805.
Topping, K. J. (2021). Face-to-face peer assessment in teacher education/training: A review. Educational Review USA.https://discovery.dundee.ac.uk/en/publications/face-to-face-peer-assessment-in-teacher-educationtraining-a-review.
Yahya, M. Y. (2019). Improving speaking performance and L2 motivation through task-based language teaching on Malaysian undergraduate students. University of Reading. https://doi.org/10.48683/1926.00089016.
Yu, S., & Liu, C. (2021). Improving student feedback literacy in academic writing: An evidence-based framework. Assessing Writing, 48, 100525. https://doi.org/10.1016/j.asw.2021.100525.
Yuliantini, N., Ratminingsih, M., Putra, N. A. J., & Adnyani, L. P. S. (2024). The Effect of Job-Related English Learning Materials for Mobile Learning Implementation on Students’ English Mastery. Theory and Practice in Language Studies, 14(12), 3778–3784.
Zrekat, Y., & Al-Sohbani, Y. (2022). Arab EFL university learners’ perceptions of the factors hindering them to speak English fluently. Journal of Language and Linguistic Studies, 18(1), 775-790. https://doi.org/10.52462/jlls.219.
Authors
Copyright (c) 2026 I Putu Yoga Laksana, Ni Made Ratminingsih, Made Hery Santosa, I Putu Indra Kusuma, Thijs Hemmer
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.