The Development and Evaluation of a Tool to Determine the Characteristics of STEM Activities

STEM activities integrate various disciplines and are organized around domain-specific concepts and skills. What is achieved in terms of learning is related to how well the important STEM education characteristics are reflected in the implemented and integrated activities. This study aims to develop and evaluate a measurement tool for determining the characteristics of STEM activities by using a case study method. The STEM Activity Evaluation Form was developed based upon a literature review and experts’ opinions. The observation-based form was applied to 21 different STEM activities, grouped in four categories: curiosity arousing activities, encouraging and upskilling activities, research and design projects, and one science fair. Students in science education partook in the activities in an eight-month period. The evaluation form has four categories: STEM learning environment, activitation of students, STEM content and practices, and connecting STEM. Application of the form revealed that the tool is useful to determine the appropriateness of activities with respect to the nature of STEM education.


INTRODUCTION
STEM (Science, Technology, Engineering, and Mathematics) education plays a significant role in the economic development for both developed and emerging countries (Kennedy and Odell, 2014). It not only supports the development of a STEM-capable workforce (Honey, Pearson, and Schweingruber, 2014) but also has the potential to increase inventiveness, scientific discovery, and efficiency in STEM disciplines (Modi, Schoenberg, and Salmond, 2012).
Many instructional approaches and practices like inquiry-based learning, project-based learning, argumentation and reasoning, digital learning, computer programming, and cooperative learning are used in STEM education as well as in the STEM disciplines themselves (Bender, 2016;McDonald, 2016). Scientific inquiry is an essential element of those practices. It reflects how scientists come to understand the natural world and how students learn science (National Science Teachers Association [NSTA], 2018). In science education, scientific inquiry is defined as a way of thinking that includes content knowledge and procedural processes. Over time, the definition has expanded and evolved a structure that includes knowledge and hands-on/minds-on approaches (National Research Council [NRC], 2000).
The integration of disciplines is also critical for STEM education. As the highest level of integration, the transdisciplinary approach aims to shape learning experience and combines knowledge and skills from at least two STEM disciplines to apply to real-world problems and projects (English, 2016). Engineering has a unique role

Development of the STEM Activity Evaluation Form
In order to develop the STEM Activity Evaluation Form, the first step was examining studies in the literature that provide answers to the following questions: (1) What are the basic or constituting elements of STEM education? (2) How is the overall quality of STEM education measured? (3) How are STEM/science education programs elaborated into activities, such as projects? (4) How are STEM/science activities evaluated? and (5) Which classroom observation protocols are used to evaluate teaching and learning? The first two questions meant to determine the characteristics of STEM education and evaluation of STEM education on the level of the curriculum or program. The third and fourth questions comprised the studies relevant for the development and evaluation of STEM/science education activities within specific programs or courses. The last question is even more detailed and focuses on concrete observation.
According to the literature, discipline integration plays an essential role in STEM education (Bybee, 2010). STEM education should include engineering design challenges of relevant technologies to develop students' problem-solving abilities and higher-order thinking skills (Moore, Johnson, Peters-Burton, and Guzey, 2016). Scientific inquiry promotes learning and success in STEM education (McDonald, 2016). STEM education studies focus often on learning outcomes such as attitudes (Wendell and Rogers, 2013), knowledge (Cotabish, Dailey, Robinson, and Hughes, 2013), and STEM interest (Mohr-Schroeder et al., 2014).
Pri-Sci-Net (2014), Fibonacci (2013), and The Partnerships in Education and Resilience (PEAR) Institute (2014) projects were prominent long-term and large-scale research projects focusing on the evaluation of STEM or science activities. The project outcomes contained many teaching materials and assessment tools relevant for this study. Pri-Sci-Net was a European Union (EU) funded project aiming to promote inquiry-based learning on a large scale in Europe and to develop inquiry-based science activities (Pri-Sci-Net, 2014). Fibonacci, a three-year project, aimed at dissemination of inquiry-based science and mathematics education throughout the EU. The project provided observation-based formative assessment tools for teaching practices (Harlen and Lena, 2013;Fibonacci, 2013). PEAR (2014) designed the Dimensions of Success (DoS) observation tool to evaluate the quality of out-of-school STEM programs. These dimensions are features of the learning environment, activity engagement, STEM knowledge and practices, and youth development in STEM. The observation-based protocols can be used to evaluate domain-specific teaching and learning in STEM/science activities (Schultz and Pecheone, 2015;Shah et al., 2018).
In the second step, the basic elements of science activities and STEM education were categorized into themes and sub-themes that were mentioned in the literature. Two researchers of science education independently classified the categorizes and sub-themes and worked out a consensus framework (see Table 1). Table 1 represents the four categories and eleven sub-themes that resulted from the literature analyses and constitute the framework for the STEM Activity Evaluation Form. The first category or theme is 'STEM learning environment', which contains and organizes the space and the materials to be used. Hence, two sub-themes were

STEM learning environment
The space and the materials should be available and organized to achieve learning objectives (Peters and Stout, 2006). The required time for the activity is an essential factor (Ainley, Pratt, and Hansen, 2006). Besides, a proper flow of activity steps prevents gaps, which negatively affect learning (Martin, Sexton, Franklin, and Gerlovich, 2014). Suitable materials for learning objectives should be accessible and complete for all participants (Harris, Miske, and Attig, 2004).

Activation of students
What students do in an activity is determined by the learning objectives. Students should be enabled to participate (Ainley et al., 2006). Each participant should be actively involved for effective STEM education (National Research Council [NRC], 2011). The activity should ensure that participants are involved both minds-on and hands-on (Fibonacci, 2013;PEAR, 2014;Pri-Sci-Net, 2014). The activity should be accompanied by formative assessment and evaluation techniques (Fibonacci, 2013).

STEM content and practices
Discipline integration plays an essential role in STEM education (Bybee, 2010). The engineering design process is essential in STEM activities (Basham and Marino, 2013). Mistakes are not negative. They are a vital part of learning (Hodson, 1988). Reflection on what the participants have learned provide evidence on the effectiveness of the activity. Thus, students should be encouraged to reflect (Harlen and Lena, 2013;PEAR, 2014). If the activity includes integrated engineering, artifact design is expected (Basham and Marino, 2013;Bybee, 2010). Scientific inquiry is one of the most common elements of STEM education. Activities can engage students in posing scientific questions, evidence-based thinking, and developing skills to identify existing ideas and analyzing and interpreting data. In this process, participants are expected to be aware that there is no ultimate solution to research problems. The participants should share their ideas and the evidence with peers and draw evidence-based conclusions (Harlen, 2014;Harlen and Lena, 2013;NRC, 2000).
Connecting STEM STEM activities are connected to real-life situations and student are enabled to experience the relations between the activity, daily life problems and STEM solutions (Tseng, Chang, Lou, and Chen, 2013). Hence, the activity influences students' interest in STEM (Honey et al., 2014;Maltese and Tai, 2010). STEM activities can promote 21st century skills that are essential to the STEM workforce (Trilling and Fadel, 2009). defined: organization and materials. The second category is 'activitation of students' and describes how actions are related to learning. Hence, the sub-themes are learning objectives, participation, engagement, and evaluation. The third category is on 'STEM content and practices.' STEM teaching has a topic with multidisciplinary aspects; it evokes practices such as scientific inquiry, and students need to reflect in order to improve their learning. Hence, the sub-themes are STEM disciplines, STEM practices and scientific inquiry, and reflection. The fourth category is on 'connecting STEM.' STEM activities connect real-life problems to STEM approaches and solutions, which may develop more generic, or 21 st century skills and attitudes of students. Hence, the sub-themes are connection to real-life and 21 st century skills and attitudes. The descriptions of the categories are shown in Table 2. Table 2 represents essential elements and characteristics of STEM activities that evolve from an analysis of the literature.
In the next step, these descriptions were transformed into 29 items to be used in the evaluation tool. The items were modified into a 4-point (0 to 3) observation scoring scale following the rubric specified by PEAR (2014). The criteria are presented in Table 3. Table 3. Scoring of the STEM Activity Evaluation Form

Score
Scoring criteria 0 This quality has never been observed in the activity. 1 There is weak evidence that this quality is supported in the activity.

2
There is clear evidence that this quality is supported in the activity. However, the quality is not fully reflected in the activity, or it is not adequate to get 3 points. 3 This quality is observed in the activity based on compelling/consistent evidence.
After this step, the activity evaluation form was prepared for the pilot study. In this pilot, the form was applied by two researchers to two STEM activities. The Inter-Rater Reliability (IRR) value was calculated for the pilot study data. IRR refers to the degree of consistency between the scoring of two or more raters based on their observations of the same situation (Aiken, 2000). IRR values were 89% and 93% for the two activities, respectively. We concluded from these values that the STEM Activity Evaluation Form was reliable (Miles and Huberman, 1994). Thus, the activity evaluation form took its final version (see Appendix A). STEM education researchers and implementers/teachers are the target audience of the STEM Activity Evaluation Form. Users can evaluate STEM activities by observations. In this way, the limits and strengths of the activities can be revealed. Users of course need to have certain competencies with respect to STEM education to utilize this form validly. For example, two criteria in the STEM Activity Evaluation Form are "the activity includes STEM integration." and "the activity includes the engineering design process." In order to observe and score these two criteria validly, users should have sufficient understanding of STEM disciplines and the engineering design processes.

Participants
The data were obtained from STEM activities that occurred within the scope of an astronomy education course, conducted with 67 third grade science teacher candidates during eight months at a university in the Western Anatolia Region. The participants were divided into two groups as A (N=34) and B (N=33). All STEM activities were carried out in two groups. The research data were obtained by observing each activity in groups A and B. The purpose of performing the observations repeatedly for each activity was to improve the consistency of the research data.

Intervention
STEM education activities selected for investigating the STEM Activity Evaluation Form were chosen according to two criteria. First, the activity should be relevant to the learning objectives of astronomy according to the science curriculum as prescribed by the Turkish Ministry of National Education. Second, another STEM discipline, next to astronomy, is needed in the activity. In total, 21 activities were selected and categorized in four modules: curiosity arousing activities (1), encouraging and upskilling activities (2), research and design projects (3), and one science fair (4).
The module of curiosity arousing activities focused on astronomy, the known universe, and the connections between astronomy and STEM disciplines. It introduced students to naked eye and telescope observations and included trips to research laboratories and replica museum activities. These activities aimed to contribute to an understanding of the importance of scientific methods in astronomy, the use of advanced observation tools such as telescopes and other techniques and devices, and the relevance of social sciences for communication with STEM experts.
The encouraging and upskilling activities aimed to teach students about the design of artifacts such as basic observation tools, rockets, and holograms that are used in astronomy. The primary purpose was to foster STEM skills of the participants using the engineering design process as proposed by Tayal (2013). A teaching format, this engineering design process is appropriate for a single one hour-long activity as well as for projects that last for several class periods (see Figure 2).
The engineering design process can be summarized as follows: In the first step ('ask'), the participants define the engineering problem, determine the requirements and constraints, and gather scientific knowledge essential for the solution. In the second step ('imagine'), the participants' ideas for the solution of the problem are expressed and discussed. The solution that best meets the requirements and constraints is selected. The third ('plan') step includes sketching the design and determining the material required for making a prototype. A prototype is constructed and tested in the fourth ('create') step. In the last ('improve') step, the prototype is redesigned and reconstructed according to the test results. This ends when the prototype meets the requirements. In this module, activities are organized individually or in group workshops using simple and inexpensive materials. The aim here is that the participants develop their STEM skills and also gain self-confidence by designing functional artifacts. As an example, the Telescope activity is presented in Appendix B.
The research and design projects focused on solving an engineering problem which was given by the teacher, or identified by participants in previous activities. In these activities, it is essential that participants use the necessary STEM skills gained in previous modules. For these medium and long-term projects, the engineering design approach recommended by Kampe and Oppliger (2012) was used. Participants were divided into teams and encouraged to use this approach in their projects (see Figure 3).
The steps in this design cycle can be summarized as follows. At the start, the problem is defined. Information about the problem is collected from various sources such as books, websites, or previous designs. The criteria, limitations, and restrictions are determined. Thus, the essential features of the design are determined. A flow plan,  including steps and strategies to be followed, is formed. Alternative solutions are developed and discussed. The solution that is most rational according to the criteria is chosen. A prototype is constructed. The prototype is analyzed to confirm compatibility with the design and is tested. After the test, the prototype is evaluated whether it meets all requirements and solves the engineering problem. If the criteria are met, the prototype has reached the final version and is ready for presentation. Otherwise, the prototype is revised, analyzed, tested, and evaluated, or the students fall back to one of the previous steps of the design process.
The research and design projects included student workshops and design challenges. The engineering problems of the projects were on how science works in space, how time can be measured in alternative ways, how we can reach space, how holograms work, and how we observe space. These problems do not have a single solution and encourage participants to use scientific research and design processes. It was essential to use STEM integration and collaborative teamwork to deal with these challenging problems. Moreover, the participants had to define the research and design stages and organize the design plan by themselves. The teams were given eight weeks for research and design projects. At least one discussion meeting was held with each team per week.
The science fair module included only one activity. The main aim here was to gain self-confidence through sharing designed artifacts in previous modules through interactive presentations and workshops.

Data Collection Process
A researcher collected data during the activities by taking the role of an external observer (Lodico, Spaulding, and Voegtle, 2010). The focus of the observations was the activity itself, not the activity plan or the competence of the teacher. The observer was a Ph.D. student experienced with classroom observation protocols in science teaching and had been in the same learning environment with the participants for three weeks.

Data Analysis
Activities were scored on the 11 items (the sub-themes as described above) on a four-point Likert scale (0, 1, 2, and 3). The arithmetic means of the scores were calculated and transformed into charts. Additional qualitative information was gathered by note taking during the observations. An example of the scores on two items for two activities is presented in Table 4.

Results for the Curiosity Arousing Activities
The results obtained for the five activities in the Curiosity Arousing module are presented in Figure 4. What draws the attention in Figure 4 are the relatively low scores for STEM content, scientific inquiry, and engagement. All other sub-themes were scored at a high level. We provide some details and qualitative observations to illustrate the scores for one of the activities, called Discovering the Universe.
The learning environment was appropriately arranged. The classroom layout had a U-shaped desk arrangement that encourages discussions, and the participants could easily watch the astronomy videos about the known universe (Organization). The required materials were available, as well as technical support. Various videos and software were prepared. Worksheets and other materials were ready for participants (Materials). The activity enhanced various goals mentioned in the curriculum, such as understanding the size, distance, and position in the universe, calculating in terms of powers of tens, and the connections between STEM and astronomy. Students wrote sticky notes with their questions about space and astronomy. These notes were posted visibly in the classroom (Learning objectives). Students were enabled to participate equally in all sub-activities. Questions were The activity includes the engineering design process.

3
In the activity, participants were given the task that they should carry a cargo (duster) 8 m away, using the thrust of the air within the shortest time. Participants performed all the steps of the engineering design process, such as defining the problem, revealing ideas for solutions, choosing a solution, drawing sketches, constructing a prototype, testing and evaluating the prototype, improving, redesigning, retesting, and re-evaluation the prototype. For example, after the implementer confirmed the sketches of the design, it was possible to create the prototype. Designing and sketching steps were the most time-consuming parts of the activity for participants. Naked Eye and Telescope Observations The activity focuses on artifact construction.

0
The activity focused on discovering the sky using a telescope. There was no artifact-oriented strategy in the activity.
posed to all participants and all answers received attention (Participation). The activity did not involve hands-on activities. The emphasis was on answering questions and making inferences. Hence, the activity only engaged students' minds (Engagement). The activity includes process-oriented measurement and evaluation forms. Worksheets were used by participants to evaluate their own performance and learning (Evaluation). The activity did not involve an engineering design sub-activity nor did the participants have the opportunity to conduct their own research, so it received rather low scores for these items. This is not particularly worrying, since the activities in this module aim to foster curiosity, more than knowledge and skills (STEM disciplines). The participants did not have the opportunity to conduct their own research (STEM practices and scientific inquiry). The participants filled out worksheets and reflected on what they learned through writing and discussing. This was reinforced by questions posed by the teacher (Reflection). The participants were observed discussing how many days will it take to get to Mars, and whether life is possible outside Earth. A simulation of the power of ten used in this activity was directly related to daily life observations of the participants with respect to the limits of science (Connection to real-life). The activity highlighted the relation and importance of several 21 st century skills to STEM, such as communication, reasoning, and critical thinking (21 st century skills and attitudes).

Results for the Encouraging and Upskilling Activities
The results obtained for the ten activities in the Encouraging and Upskilling module are presented in Figure 5.
As can be seen in Figure 5, all activities except the Earth, Sun and Moon activity received the highest scores and thus encouraged students and helped them develop STEM skills. We illustrate this with observations from one of the activities in this module, on designing your own water rocket.
We observed that the duration (of four hours) was quite sufficient for defining the problem, sketching a design, creating a prototype, conducting experiments, collecting data, evaluating, and revising the prototype   (Organization). The materials, also additional materials required by the students such as chalk and nylon rope were available or provided in a short time (Materials). The activity aimed at conceptual and skill-oriented achievements pertaining to energy transformations, thrust, air resistance, measurement, and the engineering design process (Learning objectives). Students were assigned to different design tasks, and they all participated in the sub-activities (Participation). The activity involved hands-on and minds-on activities. The participants expressed their ideas to complete the given task. They evaluated different ideas according to the evidence and realized their designs using their hands-on skills (Engagement). The worksheets used in the activity provided the teams to evaluate themselves and shape their learning (Evaluation). The participants had to convert a given bottle into a durable rocket using water and pressurized air. The rocket had to have a nose cone, fines, and a body. After launch, the rocket should move straight and reach the farthest distance. The participants performed all the steps of the engineering design process, such as defining the problem, revealing ideas for solutions, and choosing a solution. They spent considerable time to meet the durability criteria (STEM disciplines). They collected and recorded data in tables when testing the prototype. The teams analyzed and evaluated the results by comparing them with peers in the evaluation part (STEM practices and scientific inquiry). The teams verbally reflected on their understanding of relevant concepts, such as air resistance, pressure, and force. Questions on the worksheet on the primary variables for a rocket to reach space enabled participants to reflect on what they learned (Reflection). The activity connected phenomena that participants' experience and observe in real-life to scientific concepts such as kinetic energy, potential energy, air resistance, and the engineering design process (Connection to real-life). The participants constructed their artifacts using the engineering design process. Thus, the activity was particularly beneficial to support engineering interest (21 st century skills and attitudes). The Earth, Sun, and Moon activity did not receive high scores on all items. Analysis of qualitative observations confirmed this. The activity focused on astronomy simulations and animations. Hands-on activities were limited (Engagement), and there was no evidence of implementation of the engineering design process (STEM disciplines).

Results for the Research and Design Projects and the Science Fair
The results obtained for the five Research and Design Projects and the Science Fair are presented in Figure 6. As can be seen, all research and design projects scored high on all sub-themes. We illustrate the scores for the research and design projects through the observations for the activity 'How can we reach space?' This activity was carried out in approximately two months. During this period, weekly workshops were organized for the participants. Each team designated their work plan, and had sufficient time to conduct the research and design tasks effectively (Organization). Often-used materials were ready at hand. When different or unique materials were required, these materials were provided to the participants (Materials). The objective was learning to solve an engineering problem. All sub-activities were related to the solution of the determined problem (Learning objectives). Three teams of four participants participated in the activity. All team members actively shared their scientific knowledge about the solution of the design problem. They gathered information from various sources and decided on a prototype through group discussions. The team distributed the tasks, such as obtaining materials and performing experiments and focused on the solution to the design problem (Participation). Initially, minds-on activities were prioritized by the teams because they had to decide via group discussions on solutions and constitute a design according to scientific evidence. Later, the teams predominantly were hands-on, constructing a prototype.  In the last steps, they were both hands-on and minds-on, engaged in testing, evaluating test results, and solving encountered design problems. For example, one of the teams designed a model rocket. In the test, the model failed because of flight instability. After evaluating data, the team found the problem: the center-of-gravity was not in front of the center-of-pressure. They improved the design and re-constructed the prototype (Engagement). The teams created a portfolio electronically or in written format. The portfolio included detailed sections on scientific concepts, principles, formulas, and sketching relevant for rocket science; the data obtained in the tests; and the features of the designs arranged according to results. The portfolio allowed participants to evaluate their progress (Evaluation). The activity was transdisciplinary. The participants applied a variety of knowledge and skills, especially from physics, chemistry, engineering, technology, and mathematics. For example, one of the team used computer simulations and mathematical calculations to determine the maximum altitude that the model rocket could reach (STEM disciplines). All teams executed the steps from the research and design process by themselves. For example, one of the team designed two different types of nose shapes for a model rocket. They tested the nose shapes and decided to use or give type nose shape (STEM practices and scientific inquiry). The teams used the portfolio to share and discuss what they had learned with respect to identifying and solving the engineering problem (Reflection). The challenge to reach space is a common theme and dream in daily life; participants see how STEM skills and knowledge are relevant for meeting such challenges (Connection to real-life). The activity involved solving a challenging problem through sustained teamwork. Thus, collaboration, effective communication, critical thinking, and creativity were supported (21 st century skills and attitudes). The STEM-based Science Fair also scored high on most characteristics, but it did not contain evaluation procedures, nor did it activate inquiry or design practices at the highest level. The learning environment of the fair was the showroom of a regional shopping mall. Here, workshops, exhibitions, and rocket tests were organized. The premises were large enough to perform the activities effectively (Organization). In the workshops, the teams showed rocket and hologram designs to the science fair participants. The fair used the materials and the artifacts designed by participants in previous modules, which were of course available (Materials). The fair aimed to raise self-confidence of the participants concerning STEM to show that they could reach large audiences via interactive presentations and workshops (Learning objectives). Choosing a shopping mall as an environment increased participation in the science fair. In this context, the participants met with children and shared their artifacts. While the participants were learners in the previous modules, they took the role of an implementer in this module. This encouraged them to feel and behave like a teacher. (Participation). The fair supported hands-on and minds-on activities. Design activities and demonstrations contributed to learning by doing for both participants and science fair participants (Engagement). However, no process-oriented measurement and evaluation techniques were applied (Evaluation). The engineering design process was limited in the fair. For example, only the design and testing steps were included in the water rocket designs (STEM disciplines). There was no evidence for inquiry concerning data collection, analysis, and interpretation (STEM practices and scientific inquiry). When sharing their knowledge, skills, and experiences with the public in workshops and interactive exhibitions, participants reflected on what they had learned in the previous modules (Reflection). By sharing their designs and their experiences with children, the participants, as prospective teachers, could connect to the real-life experiences of children (Connection to real-life). The activity concerned sharing STEM-focused artifacts with the public. Thus, participants' collaboration and communication skills were supported by the activity. Besides, the STEM-based science fair was beneficial for the participants to gain self-confidence (21 st century skills and attitudes).

DISCUSSION AND CONCLUSION
This study aimed to develop and validate an activity evaluation form that is compatible with the nature of STEM education. The STEM Activity Evaluation Form we developed consisted of four categories: STEM learning environment, activation of students, STEM content and practices, and connecting STEM. It was tested and validated by evaluating all 21 STEM activities that occurred in an astronomy course for prospective science teachers. We found that we could use the form without inconsistencies. The majority of the activities scored high on all four themes of the form. Lower scores could be explained through analysis of the aims of the activity and of the observation data, confirming that the form applies well to a variety of activities with a bigger or lesser number of STEM characteristics. For example, activities meant to arouse curiosity such as excursions were not meant to develop specific engineering design practices, and as a consequence these activities scored lower on 'STEM disciplines. ' Only a limited number of studies in the literature investigated the evaluation of STEM activities. Hug and Eyerman (2018) concluded that the moving car activity in their case studies research significantly met the qualities of STEM education. Shah et al. (2018) developed an observation tool evaluating STEM program quality in out-ofschool environments. They applied their Dimensions of Success tool to 340 activities. Their focus was limited to the engagement of students with STEM content and their development in STEM disciplines, which are comparable to the learning environment and activation themes in our form.
We applied our instrument to a course with four modules and 21 activities, with a sequential and complementary structure. This structure consists of raising curiosity and awakening interest, developing skills and gaining selfconfidence in STEM disciplines, designing an artifact from the beginning to the end, and sharing the artifact with an audience through interactive presentations. Such a sequential and complementary pattern, we think, effectively reflects the nature of STEM education.
According to the results of the research, the following points for further discussion and research can be raised. In STEM education, artifact construction should not be restricted to cookbook-type workshop assignments but should include inquiry and engineering design practices. In this manner, students acquire STEM content knowledge and skills through activities that are both motivating and valid.
STEM education activities should be evaluated from a holistic and goal-oriented perspective and foster interest, attitude, knowledge, skills, and understanding of STEM practices and their relations to real-life problems. Whether activities use fancy or costly equipment or materials such as robotics is less important. The use of simple and inexpensive materials that meet the essential elements of STEM education can equally well lead to high-quality STEM activities.
Activity-oriented observation tools are important to establish a correlation or causal relation between teaching and learning outcomes (Shavelson and Towne, 2002). These tools support teachers' content knowledge, teaching practices, and the learning of their students (Schultz and Pecheone, 2015). The STEM Activity Evaluation Form developed in this study is a useful measurement tool that can be used by (prospective) teachers to evaluate and improve STEM education practices.
The STEM Activity Evaluation Form developed in this study can be used to evaluate STEM activities based on various pedagogical approaches such as project-based learning, collaborative learning, problem-based learning, inquiry-based learning, and design-based learning. It may be less suitable to evaluate purely direct instruction and 'cook book' approaches.
In today's education, the diversification of challenges and information sources makes the teacher or the textbook not the only role model and authority to the student. Nowadays, the teacher and the student are in a position to continually renew themselves. To monitor the performance of a generation that can think creatively and critically, shows commitment and patience, is not afraid to make mistakes and can learn from mistakes, and collaborates with others, we need process-oriented evaluation tools and not just standardized assessment and evaluation tools.
There are some limitations to the current study. The observation-based evaluation form was validated on a limited number of STEM activities from only one, astronomy-based STEM course, which was taken by one specific group of students, namely prospective science teachers. There may be more STEM characteristics to STEM activities, to be discovered from applying the STEM Activity Evaluation Form to a larger variety of courses and students.

APPENDIX A The STEM Activity Evaluation Form Theme
Sub-theme Item Score Observation, evidence, or example 0 1 2 3

STEM Learning environment Organization
A proper learning environment for the activity is organized.
The duration of the activity is sufficient.
Transitions between the sub-activities occur systematically.

Material
Activity materials are proper for learning objectives.
Required materials are available for the activity.

Learning objectives
The activity is relevant to learning objectives.
Sufficient time is spent on achieving for learning objectives.

Participation
The activity enables students to participate equally in the sub-activities.
Students are encouraged to participate in the subactivities.

Engagement
The activity involves hands-on and minds-on activities.

Evaluation
The activity includes process-oriented measurement and evaluation techniques.

STEM disciplines
The activity emphasizes various STEM disciplines.
The activity includes STEM integration.
The activity includes the engineering design process.
The mistakes are essential parts of learning.
The activity allows students to provide evidence that they accurately understand scientific concepts.
The activity focuses on artifact construction.

STEM practices and scientific inquiry
The activity allows students to engage in scientific questions. The activity supports students' evidence-based thinking skills.
The activity allows students to reveal their current ideas.
The activity allows students to do their research.
The activity enables students to use data collection, analysis, and interpretation. The activity allows students to suggest different solutions for a problem. The activity allows students to share their ideas or findings with their peers.

Reflection
The students are encouraged to reflect on what they have learned.
The activity allows students to reflect on what they have learned.

Connection to real-life
The activity makes a connection between problems or subjects related to real-life and students' experiences.

st century skills and attitudes
The activity supports students' interest in STEM disciplines.
The activity supports the development of students' 21 st century skills.