Software using machine learning: how to test and verify? (Sommersemester 2020)

Lecturer: Dr. Leen Lambers (Systemanalyse und Modellierung)

General Information

• Weekly Hours: 2
• Credits: 3
• Graded: yes
• Enrolment Deadline: 06.04.2020 - 22.04.2020
• Teaching Form: Seminar
• Enrolment Type: Compulsory Elective Module
• Course Language: English
• Maximum number of participants: 10

Programs, Module Groups & Modules

Description

During the last decade, we can observe a rise in software systems using machine learning (ML).  In particular when it comes to handling uncertainty, ML techniques promise to add a higher level of flexibility and automation.  They observe usually large amounts of data relevant for the system w.r.t. tackling this uncertainty, and encode their observations into a ML model.  Software incorporating ML components bases part of its automated decision making (in particular also for unseen data) on such ML models.

In this seminar, we focus on the challenges arising for the successful adoption of quality assurance techniques to software incorporating ML components. Tackling these challenges  becomes increasingly important, given the increased integration of ML components in particular also in the field of e.g. safety-critical systems. We first study selected approaches specifically developed to test and verify ML components, and in particular explore their inherent limitations. From there we explore which additional challenges therefore arise when it comes to assuring quality of the overall software system, and we evaluate first approaches with the aim of tackling these challenges.

Requirements

Basic knowledge in machine learning techniques and/or classical software testing and verification techniques is required.

Literature

• Rob Ashmore, Radu Calinescu, Colin Paterson:
  Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges. CoRR abs/1905.04223 (2019)
• Marsha Chechik, Rick Salay, Torin Viger, Sahar Kokaly, Mona Rahimi:
  Software Assurance in an Uncertain World. FASE 2019: 3-21
• Marta Z. Kwiatkowska: Safety Verification for Deep Neural Networks with Provable Guarantees (Invited Paper). CONCUR 2019: 1:1-1:5
• Jie M. Zhang, Mark Harman, Lei Ma, Yang Liu: Machine Learning Testing: Survey, Landscapes and Horizons. CoRR abs/1906.10742 (2019)

Further literature will be announced in the course.

Learning

The course is expected to take place entirely online. Detailed information on the technical implementation (based on a combined use of Moodle and an online conference system) will follow. Please follow this link to reach the course in Moodle. You can self-enroll to the course using the keyword qaml-20.

There will be two introductory appointments. At the first appointment there is an introduction to the topic and a description of possible presentation topics. The goals of the second appointment are to select a presentation topic (some of the topics can be worked on by a group of two students) and to arrange an appointment for the block seminar.

There is an individual preparation period for the block seminar between the introductory lectures and the block seminar appointment. This period is accompanied by two individual preparation appointments with the lecturer and an online discussion forum.

In the block seminar, individual topics are illuminated by student presentations and reflected in an open discussion after the presentation. This reflection will be continued individually after the block seminar and summarized into a reflection document.

Examination

Active participation (in the form of e.g. asking questions, giving feedback, and participating in group discussions) during individual as well as group appointments is required. The grade for this seminar is composed from the presentation (80%) of the selected topic during the block seminar and from a post-processing (20%) of this presentation, reflecting on questions and comments in the open discussion after the presentation, and summarized into a reflection document. Both parts must be passed.

Dates

Two introductory appointments: April 28, 11:00 and May 5, 11:00.

For the individual preparation phase, two individual appointments are arranged with the lecturer: The first appointment should take place within the first two weeks after the topic has been assigned. The second appointment should take place at the latest two weeks before the block seminar appointment.

The rest of the seminar will take place as a block seminar (probably also online) in the second half of the semester. We agree on a suitable appointment during the introductory course.

The deadline for the presentation slides is three working days before the block seminar appointment. The deadline for the reflection document is one week after the block seminar appointment.

Zurück