-
Teusner, R., Matthies, C., Staubitz, T.: What Stays in Mind? - Retention Rates in Programming MOOCs. 2018 IEEE Frontiers in Education Conference (FIE) (2018).
This work presents insights about the long-term effects and retention rates of knowledge acquired within MOOCs. In 2015 and 2017, we conducted two introductory MOOCs on object-oriented programming in Java with each over 10,000 registered participants. In this paper, we analyze course scores, quiz results and self-stated skill levels of our participants. The aim of our analysis is to uncover factors influencing the retention of acquired knowledge, such as time passed or knowledge application, in order to improve long-term success. While we know that some participants learned the programming basics within our course, we lack information on whether this knowledge was applied and fortified after the course's end. To fill this knowledge gap, we conducted a survey in 2018 among all participants of our 2015 and 2017 programming MOOCs. The first part of the survey elicits responses on whether and how MOOC knowledge was applied and gives participants opportunity to voice individual feedback. The second part of the survey contains several questions of increasing difficulty and complexity regarding course content in order to learn about the consolidation of the acquired knowledge. We distinguish three programming knowledge areas in the survey: First, understanding of concepts, such as loops and boolean algebra. Second, syntax knowledge, such as specific keywords. Third, practical skills including debugging and coding. We further analyzed the long-term effects separately per participant skill group. While answer rates were low, the collected data shows a decrease of knowledge over time, relatively unaffected by skill level. Application of the acquired knowledge improves the memory retention rates of MOOC participants across all skill levels.
-
Matthies, C., Teusner, R., Hesse, G.: Beyond Surveys: Analyzing Software Development Artifacts to Assess Teaching Efforts. 2018 IEEE Frontiers in Education Conference (FIE) (2018).
(accepted)
This Innovative Practice Full Paper presents an approach of using software development artifacts to gauge student behavior and the effectiveness of changes to curriculum design. There is an ongoing need to adapt university courses to changing requirements and shifts in industry. As an educator it is therefore vital to have access to methods, with which to ascertain the effects of curriculum design changes. In this paper, we present our approach of analyzing software repositories in order to gauge student behavior during project work. We evaluate this approach in a case study of a university undergraduate software development course teaching agile development methodologies. Surveys revealed positive attitudes towards the course and the change of employed development methodology from Scrum to Kanban. However, surveys were not usable to ascertain the degree to which students had adapted their workflows and whether they had done so in accordance with course goals. Therefore, we analyzed students' software repository data, which represents information that can be collected by educators to reveal insights into learning successes and detailed student behavior. We analyze the software repositories created during the last five courses, and evaluate differences in workflows between Kanban and Scrum usage.
-
Matthies, C.: Scrum2Kanban: Integrating Kanban and Scrum in a University Software Engineering Capstone Course. Proceedings of the 2nd International Workshop on Software Engineering Education for Millennials. pp. 48-55. ACM, New York, NY, USA (2018).
Using university capstone courses to teach agile software development methodologies has become commonplace, as agile methods have gained support in professional software development. This usually means students are introduced to and work with the currently most popular agile methodology: Scrum. However, as the agile methods employed in the industry change and are adapted to different contexts, university courses must follow suit. A prime example of this is the Kanban method, which has recently gathered attention in the industry. In this paper, we describe a capstone course design, which adds the hands-on learning of the lean principles advocated by Kanban into a capstone project run with Scrum. This both ensures that students are aware of recent process frameworks and ideas as well as gain a more thorough overview of how agile methods can be employed in practice. We describe the details of the course and analyze the participating students' perceptions as well as our observations. We analyze the development artifacts, created by students during the course in respect to the two different development methodologies. We further present a summary of the lessons learned as well as recommendations for future similar courses. The survey conducted at the end of the course revealed an overwhelmingly positive attitude of students towards the integration of Kanban into the course.
-
Hesse, G., Reissaus, B., Matthies, C., Lorenz, M., Kraus, M., Uflacker, M.: Senska - Towards an Enterprise Streaming Benchmark. In: Raghunath, N. and Meikel, P. (eds.) Performance Evaluation and Benchmarking for the Analytics Era. pp. 25-40. Springer International Publishing, Cham (2018).
In the light of growing data volumes and continuing digitization in fields such as Industry 4.0 or Internet of Things, data stream processing have gained popularity and importance. Especially enterprises can benefit from this development by augmenting their vital, core business data with up-to-date streaming information. Enriching this transactional data with detailed information from high-frequency data streams allows answering new analytical questions as well as improving current analyses, e.g., regarding predictive maintenance. Comparing such data stream processing architectures for use in an enterprise context, i.e., when combining streaming and business data, is currently a challenging task as there is no suitable benchmark.
-
Matthies, C., Treffer, A., Uflacker, M.: Prof. CI: Employing Continuous Integration Services and GitHub Workflows to Teach Test-Driven Development. 2017 IEEE Frontiers in Education Conference (FIE). pp. 1-8 (2017).
Teaching programming using Massive Open Online Courses (MOOCs) is gaining popularity due to their scalability and efficiency of knowledge distribution. However, participating in these courses usually means fully committing to the supplied programming environment in the browser. While this allows a consistent and controllable setup, learners do not gain experience with actual development tools, such as local code editors, testing frameworks, issue trackers or continuous integration (CI) services, which is critical for subsequent real-world projects. Furthermore, the tests for the functionality that is to be developed are oftentimes already available in MOOCs and simply need to be executed, leading to less involvement with developing appropriate tests. In order to tackle these issues while maintaining a high degree of automation and scalability, we developed Prof. CI, a novel approach to conducting online exercises. Prof. CI leverages the existing automation infrastructure that developers use daily, i.e. CI services and Github workflows, to teach test-driven development (TDD) practices. Participants work on their own repositories in Github and receive feedback and new challenges from the CI server when they push their code. We have successfully applied this approach in a pilot project with 30 undergraduate students learning the Ruby on Rails web development framework. Our evaluation shows that the exercise effectively increased students' motivation to write tests for their code. We also present the results of participant surveys, students' experiences and teachers' observations.
-
Teusner, R., Matthies, C., Giese, P.: Should I Bug You? Identifying Domain Experts in Software Projects Using Code Complexity Metrics. 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS). pp. 418-425 (2017).
In any sufficiently complex software system there are experts, having a deeper understanding of parts of the system than others. However, it is not always clear who these experts are and which particular parts of the system they can provide help with. We propose a framework to elicit the expertise of developers and recommend experts by analyzing complexity measures over time. Furthermore, teams can detect those parts of the software for which currently no, or only few experts exist and take preventive actions to keep the collective code knowledge and ownership high. We employed the developed approach at a medium-sized company. The results were evaluated with a survey, comparing the perceived and the computed expertise of developers. We show that aggregated code metrics can be used to identify experts for different software components. The identified experts were rated as acceptable candidates by developers in over 90% of all cases.
-
Hesse, G., Matthies, C., Reissaus, B., Uflacker, M.: A New Application Benchmark for Data Stream Processing Architectures in an Enterprise Context: Doctoral Symposium. Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems. pp. 359-362. ACM, New York, NY, USA (2017).
Against the backdrop of ever-growing data volumes and trends like the Internet of Things (IoT) or Industry 4.0, Data Stream Processing Systems (DSPSs) or data stream processing architectures in general receive a greater interest. Continuously analyzing streams of data allows immediate responses to environmental changes. A challenging task in that context is assessing and comparing data stream processing architectures in order to identify the most suitable one for certain settings. The present paper provides an overview about performance benchmarks that can be used for analyzing data stream processing applications. By describing shortcomings of these benchmarks, the need for a new application benchmark in this area, especially for a benchmark covering enterprise architectures, is highlighted. A key role in such an enterprise context is the combination of streaming data and business data, which is barely covered in current data stream processing benchmarks. Furthermore, first ideas towards the development of a solution, i.e., a new application benchmark that is able to fill the existing gap, are depicted.
-
Matthies, C., Kowark, T., Uflacker, M., Plattner, H.: Agile Metrics for a University Software Engineering Course. 2016 IEEE Frontiers in Education Conference (FIE). pp. 1-5. IEEE (2016).
Teaching agile software development by pairing lectures with hands-on projects has become the norm. This approach poses the problem of grading and evaluating practical project work as well as process conformance during development. Yet, few best practices exist for measuring the success of students in implementing agile practices. Most university courses rely on observations during the course or final oral exams. In this paper, we propose a set of metrics which give insights into the adherence to agile practices in teams. The metrics identify instances in development data, e.g. commits or user stories, where agile processes were not followed. The identified violations can serve as starting points for further investigation and team discussions. With contextual knowledge of the violation, the executed process or the metric itself can be refined. The metrics reflect our experiences with running a software engineering course over the last five years. They measure aspects which students frequently have issues with and that diminish process adoption and student engagement. We present the proposed metrics, which were tested in the latest course installment, alongside tutoring, lectures, and oral exams.
-
Kowark, T., Matthies, C., Uflacker, M., Plattner, H.: Lightweight Collection and Storage of Software Repository Data with DataRover. Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. pp. 810-815. ACM, New York, NY, USA (2016).
The ease of setting up collaboration infrastructures for software engineering projects creates a challenge for researchers that aim to analyze the resulting data. As teams can choose from various available software-as-a-service solutions and can configure them with a few clicks, researchers have to create and maintain multiple implementations for collecting and aggregating the collaboration data in order to perform their analyses across different setups. The DataRover system presented in this paper simplifies this task by only requiring custom source code for API authentication and querying. Data transformation and linkage is performed based on mappings, which users can define based on sample responses through a graphical front end. This allows storing the same input data in formats and databases most suitable for the intended analysis without requiring additional coding. Furthermore, API responses are continuously monitored to detect changes and allow users to update their mappings and data collectors accordingly. A screencast of the described use cases is available at https: //youtu.be/mt4ztff4SfU.
-
Matthies, C., Kowark, T., Richly, K., Uflacker, M., Plattner, H.: How Surveys, Tutors, and Software Help to Assess Scrum Adoption in a Classroom Software Engineering Project. Proceedings of the 38th International Conference on Software Engineering Companion. pp. 313-322. ACM, New York, NY, USA (2016).
Agile methods are best taught in a hands-on fashion in realistic projects. The main challenge in doing so is to assess whether students apply the methods correctly without requiring complete supervision throughout the entire project. This paper presents experiences from a classroom project where 38 students developed a single system using a scaled version of Scrum. Surveys helped us to identify which elements of Scrum correlated most with student satisfaction or posed the biggest challenges. These insights were augmented by a team of tutors, which accompanied main meetings throughout the project to provide feedback to the teams, and captured impressions of method application in practice. Finally, we performed a post-hoc, tool-supported analysis of collaboration artifacts to detect concrete indicators for anti-patterns in Scrum adoption. Through the combination of these techniques we were able to understand how students implemented Scrum in this course and which elements require further lecturing and tutoring in future iterations. Automated analysis of collaboration artifacts proved to be a promising addition to the development process that could potentially reduce manual efforts in future courses and allow for more concrete and targeted feedback, as well as more objective assessment.
-
Matthies, C., Kowark, T., Richly, K., Uflacker, M., Plattner, H.: ScrumLint: Identifying Violations of Agile Practices Using Development Artifacts. Proceedings of the 9th International Workshop on Cooperative and Human Aspects of Software Engineering. pp. 40-43. ACM, New York, NY, USA (2016).
Linting tools automatically identify source code fragments that do not follow a set of predefined standards. Such feedback tools are equally desirable for "linting" agile development processes. However, providing concrete feedback on process conformance is a challenging task, due to the intentional lack of formal agile process models. In this paper, we present ScrumLint, a tool that tackles this issue by analyzing development artifacts. On the basis of experiences with an undergraduate agile software engineering course, we defined a collection of process metrics. These contain the core ideas of agile methods and report deviations. Using this approach, development teams receive immediate feedback on their executed development practices. They can use this knowledge to improve their workflows, or can adapt the metrics to better reflect their project reality.
-
Matthies, C., Kowark, T., Uflacker, M.: Teaching Agile the Agile Way — Employing Self-Organizing Teams in a University Software Engineering Course. 2016 American Society for Engineering Education (ASEE) International Forum (2016).
Teaching students the basics of agile software development, by pairing lectures with a practical project, has been a part of computer science curricula for the last decades. However, the structure of these projects is often not agile and adaptable itself, in the way how teams are formed, workload is assigned, and meetings are organised. This restricts the areas that students can gather firsthand experiences in. In this paper we describe a university software engineering course which applies the values of the Agile Manifesto to the course design itself. Students work in self-organizing teams and are given significant authority and responsibility for their work, including planning, scheduling, and workload assignment. During the project they are guided by tutors and supported by a minimal amount of traditional lectures. We regularly assess learning progress and satisfaction via surveys and compare the results to those of more controlled course setups. We have observed positive student attitudes towards the self-organizing approach, with no adverse effects of the novel course design in terms of participant satisfaction or learning success. This confirms that the benefits of agile values, reported in the literature, are also applicable in the domain of education on the subject and can help increase student engagement and provide a better learning experience.
