Abstract
As cyber threats continue to expand across information technology (IT) and operational technology (OT) domains, cybersecurity education increasingly relies on practice-oriented methods beyond traditional theoretical instruction. Capture the Flag (CTF) competitions have emerged as a widely adopted form of experiential and gamified learning that aims to develop both technical and transversal cybersecurity competences. This systematic literature review searched for empirical studies published between 2015 and 2025 and synthesizes evidence from 18 eligible studies published between 2017 and 2025 examining the use of CTF methodologies in cybersecurity education. The review maps CTF formats (Jeopardy-style, Attack–Defense, hybrid, and immersive), educational contexts, target populations, and reported learning outcomes. It also evaluates the methodological quality and certainty of the available evidence using the GRADE framework. Across the reviewed studies, CTF participation was frequently associated with increased learner engagement and self-reported confidence. Some studies also reported short-term improvements in technical skills and cybersecurity awareness; however, these findings were generally based on self-reported measures, indirect performance indicators, small samples, or uncontrolled study designs. Several recurring design features were identified across the included studies, including scaffolded difficulty, teamwork, and progressive challenges; however, their independent effects on learning were rarely evaluated. Nevertheless, the current evidence base is limited by heterogeneous methodologies, inconsistent outcome measures, and the lack of longitudinal evaluation. Overall, CTF activities were frequently associated with learner engagement and perceived confidence, although the certainty of evidence for these outcomes was Low. Evidence for demonstrated technical competence, learning retention, and transfer to practice was Very Low. More rigorous comparative and longitudinal research is therefore needed to determine the educational effectiveness of CTF-based learning and its potential contribution to sustained cybersecurity competence.
- Ahmed, A., Watterson, C., Alhashmi, S., & Gaber, T. (2024). How universities teach cybersecurity courses online: A systematic literature review. Frontiers in Computer Science, 6, Article 1499490. https://doi.org/10.3389/fcomp.2024.1499490
- Alnajim, A. M., Habib, S., Islam, M., AlRawashdeh, H. S., & Wasim, M. (2023). Exploring cybersecurity education and training techniques: A comprehensive review of traditional, virtual reality, and augmented reality approaches. Symmetry, 15(12), Article 2175. https://doi.org/10.3390/sym15122175
- Antonioli, D., Ghaeini, H. R., Adepu, S., Ochoa, M., & Tippenhauer, N. O. (2017). Gamifying education and research on ICS security: Design, implementation and results of S3. In Proceedings of the 2017 Workshop on Cyber-Physical Systems Security and PrivaCy (pp. 93–102). Association for Computing Machinery. https://doi.org/10.1145/3140241.3140253
- Bandura, A. (1997). Self-efficacy: The exercise of control. W. H. Freeman.
- Barker, T. H., Habibi, N., Aromataris, E., Stone, J. C., Leonardi-Bee, J., Sears, K., Hasanoff, S., Klugar, M., Tufanaru, C., Moola, S., & Munn, Z. (2024). The revised JBI critical appraisal tool for the assessment of risk of bias for quasi-experimental studies. JBI Evidence Synthesis, 22(3), 378–388. https://doi.org/10.11124/JBIES-23-00268
- Collins, J., & Ford, V. (2023). Teaching by practice: Shaping secure coding mentalities through cybersecurity CTFs. Journal of Cybersecurity Education, Research and Practice, 2022(2). https://doi.org/10.32727/8.2023.8
- Costa, G., De Francisci, S., Renieri, M., & Valiani, S. (2025). Tackling the gender gap in cybersecurity education. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 (pp. 234–240). Association for Computing Machinery. https://doi.org/10.1145/3641554.3701807
- Costa, G., Lualdi, M., Ribaudo, M., & Valenza, A. (2020). A nerd dogma: Introducing CTF to non-expert audience. In Proceedings of the 21st Annual Conference on Information Technology Education (pp. 413–418). Association for Computing Machinery. https://doi.org/10.1145/3368308.3415405
- Demetrio, L., Lagorio, G., Ribaudo, M., Russo, E., & Valenza, A. (2019). ZenHackAdemy: Ethical hacking @ DIBRIS. In Proceedings of the 11th International Conference on Computer Supported Education (Vol. 1, pp. 405–413). SciTePress.
- European Cyber Security Organisation. (2020). Understanding cyber ranges: From hype to reality. https://www.evanssion.com/wp-content/uploads/2025/02/ECSO_Paper_on_cyber_ranges_understanding-cyber-ranges-from-hype-to-reality_2.pdf
- European Union Agency for Cybersecurity. (2018). Cybersecurity culture guidelines: Behavioural aspects of cybersecurity.
- Ford, I., Soneji, A., Kokulu, F. B., Vadayath, J., Basque, Z. L., Vipat, G., & Shoshitaishvili, Y. (2024). "Watching over the shoulder of a professional": Why hackers make mistakes and how they fix them. In 2024 IEEE Symposium on Security and Privacy (SP) (pp. 350–368). IEEE. https://doi.org/10.1109/SP54263.2024.00202
- Ford, V. S. (2017). Capture the flag unplugged: An offline cyber competition. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (pp. 225–230). Association for Computing Machinery. https://doi.org/10.1145/3017680.3017783
- Gasiba, T., Lechner, U., Pinto-Albuquerque, M., & Zouitni, A. (2020). Design of secure coding challenges for cybersecurity education in the industry. In International Conference on the Quality of Information and Communications Technology (pp. 223–237). Springer. https://doi.org/10.1007/978-3-030-58793-2_18
- Gervais, J. (2016). The operational definition of competency-based education. The Journal of Competency-Based Education, 1(2), 98–106. https://doi.org/10.1002/cbe2.1011
- GRADE Working Group. (2025). GRADE. https://www.gradeworkinggroup.org/
- Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O'Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P. (2018). The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Education for Information, 34(4), 285–291. https://doi.org/10.3233/EFI-180221
- İpek, Z. H., Gözüm, A. İ. C., Papadakis, S., & Kalogiannakis, M. (2023). Educational applications of the ChatGPT AI system: A systematic review research. Educational Process: International Journal, 12(3), 26–55. https://doi.org/10.22521/edupij.2023.123.2
- Junior, A. O., Funchal, G., Queiroz, J., Loureiro, J., Pedrosa, T., Parra, J., & Leitão, P. (2022). Learning cybersecurity in IoT-based applications through a capture the flag competition. In 2022 IEEE 20th International Conference on Industrial Informatics (INDIN) (pp. 560–565). IEEE. https://doi.org/10.1109/INDIN51773.2022.9976079
- Kalogiannakis, M., Papadakis, S., & Zourmpakis, A.-I. (2021). Gamification in science education: A systematic review of the literature. Education Sciences, 11(1), Article 22. https://doi.org/10.3390/educsci11010022
- Karampidis, K., Panagiotakis, S., Vasilakis, M., Lamari, A. T., Markakis, E., & Papadourakis, G. (2023). Digital training for cybersecurity in industrial fields via virtual labs and capture-the-flag challenges. In 2023 32nd Annual Conference of the European Association for Education in Electrical and Information Engineering (EAEEIE) (pp. 1–6). IEEE. https://doi.org/10.23919/EAEEIE55804.2023.10181644
- Khoo, L. J., Yatim, M. H. M., & Wong, Y. S. (2025). Research on capture the flag exercises for cybersecurity skill training among Malaysian undergraduates. Journal of Human Centered Technology, 4(1), 1–9. https://doi.org/10.11113/humentech.v4n1.87
- Korkiakoski, M., Antila, A., Annamaa, J., Sheikhi, S., Alavesa, P., & Kostakos, P. (2023). Hack the room: Exploring the potential of an augmented reality game for teaching cyber security. In Proceedings of the Augmented Humans International Conference 2023 (pp. 349–353). Association for Computing Machinery. https://doi.org/10.1145/3582700.3583955
- Ksiezopolski, B., Mazur, K., Miskiewicz, M., & Rusinek, D. (2022). Teaching a hands-on CTF-based web application security course. Electronics, 11(21), Article 3517. https://doi.org/10.3390/electronics11213517
- Larroca, F., Belcredi, G., García, R., González, G. G., Inglés, L., Mariño, C., & Randall, M. (2024). Building a radio community across students, industry and enthusiasts with capture the flag. In 2024 XVI Congreso de Tecnología, Aprendizaje y Enseñanza de la Electrónica (TAEE) (pp. 1–6). IEEE. https://doi.org/10.1109/TAEE59541.2024.10604950
- Lazarov, W., Schafeitel-Tähtinen, T., Squillace, J., Martinasek, Z., Coufalikova, A., Helenius, M., & Fujdiak, R. (2025). Lessons learned from using cyber range to teach cybersecurity at different levels of education. Technology, Knowledge and Learning. Advance online publication. https://doi.org/10.1007/s10758-025-09840-y
- Leune, K., & Petrilli, S. J., Jr. (2017). Using capture-the-flag to enhance the effectiveness of cybersecurity education. In Proceedings of the 18th Annual Conference on Information Technology Education (pp. 47–52). Association for Computing Machinery.https://doi.org/10.1145/3125659.3125686
- Malone, M., Wang, Y., James, K., Anderegg, M., Werner, J., & Monrose, F. (2021). To gamify or not? On leaderboard effects, student engagement and learning outcomes in a cybersecurity intervention. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (pp. 1135–1141). Association for Computing Machinery. https://doi.org/10.1145/3408877.3432544
- Mulder, M. (2014). Conceptions of professional competence. In S. Billett, C. Harteis, & H. Gruber (Eds.), International handbook of research in professional and practice-based learning (pp. 107–137). Springer.
- National Institute of Standards and Technology. (2025). Information Technology Laboratory: Computer Security Resource Center. https://csrc.nist.gov/
- Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., . . . Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, Article n71. https://doi.org/10.1136/bmj.n71
- Rege, A., & Bleiman, R. (2021). Collegiate social engineering capture the flag competition. In 2021 APWG Symposium on Electronic Crime Research (eCrime) (pp. 1–11). IEEE.
- Sahin, M., Ünlü, T., Hébert, C., Shepherd, L. A., Coull, N., & Mc Lean, C. (2022). Measuring developers' web security awareness from attack and defense perspectives. In 2022 IEEE Security and Privacy Workshops (SPW) (pp. 31–43). IEEE. https://doi.org/10.1109/SPW54247.2022.9833858
- Schreuders, Z. C., Shaw, T., Mac Muireadhaigh, A., & Staniforth, P. (2018). Hackerbot: Attacker chatbots for randomised and interactive security labs, using SecGen and oVirt. In 2018 USENIX Workshop on Advances in Security Education (ASE 18). USENIX Association. https://www.usenix.org/conference/ase18/presentation/schreuders
- Stouffer, K., Pease, M., Tang, C., Zimmerman, T., Pillitteri, V., Lightman, S., Hahn, A., Saravia, S., Sherule, A., & Thompson, M. (2023). Guide to operational technology (OT) security (NIST Special Publication 800-82r3). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-82r3
- Švábenský, V., Čeleda, P., Vykopal, J., & Brišáková, S. (2021). Cybersecurity knowledge and skills taught in capture the flag challenges. Computers & Security, 102, Article 102154. https://doi.org/10.1016/j.cose.2020.102154
- Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019). Cognitive architecture and instructional design: 20 years later. Educational Psychology Review, 31(2), 261–292. https://doi.org/10.1007/s10648-019-09465-5
- Vigl, W., & Abramova, S. (2024). Design and use of privacy capture-the-flag challenges in an introductory class on information privacy and security. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 (pp. 618–624). Association for Computing Machinery. https://doi.org/10.1145/3649217.3653572
- Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
- Vykopal, J., Švábenský, V., & Chang, E.-C. (2020). Benefits and pitfalls of using capture the flag games in university courses. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (pp. 752–758). Association for Computing Machinery. https://doi.org/10.1145/3328778.3366893
- Zouahi, H. (2023). Gamifying cybersecurity education: A CTF-based approach to engaging students in software security laboratories. Proceedings of the Canadian Engineering Education Association (CEEA).https://doi.org/10.24908/pceea.2023.17071
APA 7th edition
In-text citation: (Muulmann et al., 2026)
Reference: Muulmann, T., Lugo, R. G., & Sütterlin, S. (2026). A systematic literature review of capture the flag (CTF) methodologies in developing cybersecurity competence.
European Journal of STEM Education.
https://doi.org/10.20897/ejsteme/19494
AMA 10th edition
In-text citation: (1), (2), (3), etc.
Reference: Muulmann T, Lugo RG, Sütterlin S. A systematic literature review of capture the flag (CTF) methodologies in developing cybersecurity competence.
European Journal of STEM Education. 2026.
https://doi.org/10.20897/ejsteme/19494
Chicago
In-text citation: (Muulmann et al., 2026)
Reference: Muulmann, Triin, Ricardo Gregorio Lugo, and Stefan Sütterlin. "A systematic literature review of capture the flag (CTF) methodologies in developing cybersecurity competence".
European Journal of STEM Education (2026).
https://doi.org/10.20897/ejsteme/19494
Harvard
In-text citation: (Muulmann et al., 2026)
Reference: Muulmann, T., Lugo, R. G., and Sütterlin, S. (2026). A systematic literature review of capture the flag (CTF) methodologies in developing cybersecurity competence.
European Journal of STEM Education.
https://doi.org/10.20897/ejsteme/19494
MLA
In-text citation: (Muulmann et al., 2026)
Reference: Muulmann, Triin et al. "A systematic literature review of capture the flag (CTF) methodologies in developing cybersecurity competence".
European Journal of STEM Education, 2026.
https://doi.org/10.20897/ejsteme/19494
Vancouver
In-text citation: (1), (2), (3), etc.
Reference: Muulmann T, Lugo RG, Sütterlin S. A systematic literature review of capture the flag (CTF) methodologies in developing cybersecurity competence. European Journal of STEM Education. 2026.
https://doi.org/10.20897/ejsteme/19494