Automated Testing Summit 2019: Full Schedule

07:30 CET

Registration

Thursday October 31, 2019 07:30 - 16:00 CET
Terreaux Reception Hall

Breaks & Meals

09:00 CET

Keynote: Welcome & Opening Remarks - Tim Bird, Sr. Staff Software Engineer, Sony

Speakers

Tim Bird

Principal Software Engineer, Sony Electronics

Tim Bird is a Principal Software Engineer for Sony Corporation, where he helps Sony improve the Linux kernel for use in Sony's products. Tim is also a member of the Board of Directors of the Linux Foundation. Tim is active in technical projects related to embedded Linux testing and... Read More →

Thursday October 31, 2019 09:00 - 09:10 CET
Salon Pasteur

Keynote Sessions

Experience Level Any

09:10 CET

Keynote: Report on Recent Testing Meetups - Kevin Hilman, Co-founder & Sr. Engineer, BayLibre

Kevin will report on results and discussions from recent meetings where Linux and Open Source testers got together. Specifically, Kevin will report on the Linux Plumbers testing and fuzzing microconference, and the CKI hackfest, both of which were held in Lisbon, Portugal in September. He will give an overview of the key discussions and point out areas where there was agreement, and where work still needs to be done to come to consensus.

Speakers

Kevin Hilman

CTO, BayLibre, Inc.

Kevin has been a Linux user since 1994, and a kernel hacker since 1999when he started writing drivers and working on kernel ports to new embedded platforms. He has been a driver/kernel developer for Equator Technologies, MontaVista, Texas Instruments, Linaro and currently a co-founder... Read More →

Thursday October 31, 2019 09:10 - 09:30 CET
Salon Pasteur

Keynote Sessions

Experience Level Intermediate

09:30 CET

LKFT Status Update - Milosz Wasilewski, Linaro

Speakers

Milosz Wasilewski

Engineering manager, Linaro Ltd.

Milosz is an Engineering Manager at Linaro. He spent most of his career in jobs related to software testing and validation. He is currently involved in Linux Kernel Functional Testing project. Past experience includes testing software for Nokia mobile phones and telecom OSS software... Read More →

Thursday October 31, 2019 09:30 - 09:40 CET
Salon Pasteur

Lightning Talks

09:40 CET

Fuego Status Update - Tim Bird, Sony

Speakers

Tim Bird

Principal Software Engineer, Sony Electronics

Thursday October 31, 2019 09:40 - 09:50 CET
Salon Pasteur

Lightning Talks

09:50 CET

Kernelci Status Update- Kevin Hilman, BayLibre

Speakers

Kevin Hilman

CTO, BayLibre, Inc.

Thursday October 31, 2019 09:50 - 10:00 CET
Salon Pasteur

Lightning Talks

10:00 CET

CKI Status Update - Veronika Kabatova, Red Hat

Speakers

Veronika Kabatova

Principal Software Engineer, Red Hat

CKI tech lead

cki overview ats pdf

Thursday October 31, 2019 10:00 - 10:10 CET
Salon Pasteur

Lightning Talks

Session Slides Included Yes

10:10 CET

SLAV Status Update - Pawel Wieczorek, Samsung

Speakers

Pawel Wieczorek

Software Development Engineer, Samsung R&D Institute Poland

Paweł Wieczorek works at Samsung R&D Institute Poland since 2014. Starting as an access control developer, Paweł contributed to the security framework of Tizen operating system. At that time, he introduced testing automation practices to Tizen and still actively develops automated... Read More →

SLAV Status Update pdf

Thursday October 31, 2019 10:10 - 10:20 CET
Salon Pasteur

Lightning Talks

Session Slides Included Yes

10:20 CET

Open Testing Philosophy - Kevin Hilman, BayLibre

Modularity and interoperability of different CIs: standardized API points to "share" different parts of CI systems, resource usage optimization, bisections, ... see the modularity document at https://docs.google.com/document/d/15F42HdHTO6NbSL53_iLl77lfe1XQKdWaHAf7XCNkKD8
for more details.

Speakers

Kevin Hilman

CTO, BayLibre, Inc.

Thursday October 31, 2019 10:20 - 10:40 CET
Salon Pasteur

Sessions - Track 1

10:40 CET

Coffee Break

Thursday October 31, 2019 10:40 - 11:00 CET
Place Haute

Breaks & Meals

11:00 CET

Automatic Bootloader Testing - Rémi Duraffort

CI is becoming a standard for many projects with tests running on cloud resources. Thanks to LAVA, running test on real ARM hardware (like rpi, juno, ...) is now way easier and allow to build large testing systems like KernelCI.

However, theses projects are testing either the kernel or the userspace but never the lower software levels like bootloaders.

In this talk we will present some recent improvements in LAVA that allow to build a bootloader CI system on real hardware.

Speakers

Remi Duraffort

Principal Tech Lead, Linaro

I'm a principal tech lead, working for Linaro. I've been contributed to OSS since 2007 when I started working on VLC Media player at university.I have been core developer and maintainer of LAVA , a widely adopted framework to test software (bootloader, kernel, user space) on real... Read More →

Bootloader testing in LAVA pdf

Thursday October 31, 2019 11:00 - 11:40 CET
Salon Pasteur

Sessions - Track 1

Experience Level Intermediate
Session Slides Included Yes

11:00 CET

Labgrid: Real-world Examples - Jan Lübbe & Rouven Czerwinski, Pengutronix e.K.

Labgrid provides low-overhead access to a board farm for testing with CI, custom automation or interactive use. It supports controlling boards connected to multiple hosts and is implemented as a Python library with some CLI tools.

This talk will give a brief introduction before diving into some real use-cases from our lab.
Depending on interest and time, we'd show:
- regression tests via pytest
- interactive use with automated software provisioning
- integrating existing hardware control APIs

Speakers

Jan Lübbe

CTO, Pengutronix e.K.

After building Linux smartphones with OpenMoko and deploying open source GSM networks to cruise ships, Jan Lübbe joined Pengutronix in 2012 as a kernel hacker. Since then he helps customers understand Linux and how it can solve their problems. While not hacking Linux, Jan builds... Read More →

Rouven Czerwinski

Embedded Software Developer, Pengutronix e.K.

After working with embedded testing in 2016, Rouven worked on the security side of things by contributing to OP-TEE and shipping products with it. Nowadays he has an interest in media pipelines and the corresponding kernel drivers to provide a flawless recording and viewing exper... Read More →

ATS2019 jlu labgrid pdf

Thursday October 31, 2019 11:00 - 11:40 CET
Rhone 3AB

Sessions - Track 2

Experience Level Intermediate
Session Slides Included Yes

11:50 CET

Test Metadata Extraction - Cyril Hrubis, SUSE

Test metadata extraction is an attempt to propagate test metadata, i.e.
requirements, constraints, and documentation, from testcases to the test
execution framework.

The reason behind all this work is to make the execution framework aware
of this information in order to make better decisions. There are many
improvements that could be implemented based on this work starting with
simple things such as skipping unsupported tests without need to execute
the test binary or automatically annotating test failures with kernel
commit ids. More advanced uses would be building browseable and
indexable test documentation, selecting right system to run the test on
based on test requirements, or running tests in parallel.

Speakers

Cyril Hrubis

Kernel QA, SUSE

Cyril Hrubis was born in a small village in Czech Republic, played with electronics, until finally got his first PC. After that he quickly realized that proprietary systems are no fun, became a Linux user, and never looked back. All this lead employment as a kernel QA for SUSE, which... Read More →

Thursday October 31, 2019 11:50 - 12:30 CET
Salon Pasteur

Sessions - Track 1

11:50 CET

New Ways Out of the Struggle of Testing Embedded Devices - Chris Fiege, Pengutronix e.K.

Continuous Testing (CT) of Software for Embedded Devices is gaining more and more attention. Pengutronix is currently running 60+ boards for interactive development and CT of Embedded Software. This talk begins with an overview of the equipment currently used in the Pengutronix board farm and takes a look at the (dis-)advantages of the current setup.

Afterward this talk gives suggestions on how the current issues can be solved using custom devices for multiplexing and testing of electronic buses as well as the use of non-USB-based bus systems for controlling the test equipment. This talk concludes with a Q&A where the audience is invited to give a brief summary of their solutions and share their experience.

Speakers

Chis Fiege

Electron Distribution Manager, Semiconductor and PCB Division, Pengutronix

Chris has several years of experience as an electronics designer. Before starting his employment at Pengutronix he developed electronics for special measurement devices for the German automotive industry. At Pengutronix, Chris designs and maintains hardware for the embedded device... Read More →

ATS2019 cfi embedded testing handout pdf

Thursday October 31, 2019 11:50 - 12:30 CET
Rhone 3AB

Sessions - Track 2

Experience Level Intermediate
Session Slides Included Yes

12:30 CET

Lunch (Provided for Automated Testing Summit Attendees)

Thursday October 31, 2019 12:30 - 14:00 CET
Place Haute

Breaks & Meals

14:00 CET

Beaker Project: Automated Testing at Red Hat - Tomas Klohna, Red Hat

Beaker is open-source software for managing and automating labs of test computers

With Beaker, users and administrators can:
- Manage systems of different architectures across multiple labs
- Maintain an automated inventory of system hardware details
- Provision task execution environments on systems
- Schedule tasks to run on one or more systems
- Store and view task results

Beaker is a mission-critical tool for testing at Red Hat, has been adopted by several partner companies and recently leveraged by the CKI project to run its tests. This talk will give a short product introduction before diving into some areas of Beaker, specifically the provisioning, managing and scheduling part.

Speakers

Tomas Klohna

Red Hat

Tomas joined Red Hat at the start of 2019 and is responsible for Beaker's product vision, its growth and the well-being of Beaker community. Before Red Hat, Tomas worked as a Software Engineer for NXP Semiconductors working on the MCUXpresso project.

Thursday October 31, 2019 14:00 - 14:20 CET
Rhone 3AB

Sessions - Track 2

14:00 CET

Test Plan Templating for LAVA - Milosz Wasilewski, Linaro Ltd.

Repeatable testing with LAVA might become a challenge when dealing with a lot of test cases and a growing number of hardware targets. This talk is about proposed solution to this issue. Since LAVA test job definitions usually convolute the test and device part it’s hard to create a templating system that allows for easy test job re-use. Using Jinja2 might be a solution if designed carefully. Presented solution is shared between 3 teams and has a potential of being generic enough for others to use.

Speakers

Milosz Wasilewski

Engineering manager, Linaro Ltd.

Testjob templating for LAVA pdf

Thursday October 31, 2019 14:00 - 14:40 CET
Salon Pasteur

Sessions - Track 1

Experience Level Any
Session Slides Included Yes

14:20 CET

SLAV: Test Stack Abstraction Layers - Pawel Wieczorek, Samsung R&D Institute Poland

This session will cover the benefits and consequences of having custom hardware in a testing laboratory as well as using generic, off-the-shelf parts. Paweł will discuss API design decisions for testing laboratories focusing on SLAV test lab stack case. The presentation will also include lessons learned from shifting to a user-centric perspective and possible outcome of such a process.

Speakers

Pawel Wieczorek

Software Development Engineer, Samsung R&D Institute Poland

Test stack abstraction layers pdf

Thursday October 31, 2019 14:20 - 14:40 CET
Rhone 3AB

Sessions - Track 2

Session Slides Included Yes

14:50 CET

How AGL Tests Its Distro and What Challenges We Face - Jan-Simon Möller, The Linux Foundation

The Automotive Grade Linux project (AGL) has setup a extensive test infrastructure and does use it not only for its releases but also during the daily development. This talk wil show what components are used and discusses the challenges that exist and what future development might be needed.

Speakers

Jan-Simon Moeller

AGL Release Manager, The Linux Foundation

Jan-Simon Möller is Release Manager of the Automotive Grade Linux Project (AGL). He’s an active contributor to open source projects for over a decade. His dedication is to advance open source in general and Projects like AGL in particular. He serves on the Yocto Project board representing... Read More →

Thursday October 31, 2019 14:50 - 15:10 CET
Rhone 3AB

Sessions - Track 2

Experience Level Beginner

14:50 CET

A Survey of Open Source Test Definitions - Tim Bird, Sony

Different test frameworks have commonalities and differences in the elements of their test definitions. In this presentation, Tim will present the results of his research into different systems' test definitions, with the goal of coming up with those canonical fields which could be used for test interchange between systems. Last year, Tim conducted a survey of different test systems, and recorded the results on the elinux wiki. Combined with research this year, Tim will show the different categories of elements, describe the intersection of elements between systems, and describe difficulties in harmonizing the elements for a single unified test definition.

Speakers

Tim Bird

Principal Software Engineer, Sony Electronics

Thursday October 31, 2019 14:50 - 15:30 CET
Salon Pasteur

Sessions - Track 1

Experience Level Intermediate

15:10 CET

A Guide to CIP Testing - Chris Paterson, Renesas Electronics Europe & Michael Adler, Siemens AG

The core aim of the Civil Infrastructure Platform (CIP) project is to provide support for a selection of Linux Kernels (SLTS) and a reference filesystem (CIP Core) for 10+ years. In order to ensure quality CIP must have a vigorous test setup that is not only be flexible, but maintainable for years to come. In order to achieve this CIP intends to use (and contribute to) existing projects, rather than reinvent the wheel.

This talk seeks to increase discussion and collaboration by providing details on CIP's test infrastructure, both hardware and software, including our gitlab-cloud-ci tool.

Speakers

Michael Adler

Linux Consultant, Siemens AG

Working on Linux container solutions and firmware updates for embedded devices. Interested in functional programming and exotic languages. Passionate NixOS user.

Chris Paterson

Project Leader, Renesas Electronics Europe

Project Leader in the Linux team at Renesas Electronics Europe. Working with the Civil Infrastructure Platform (CIP) and KernelCI projects.

2019 10 31 ATS Lyon CIP Testing Overview v3 pdf

Thursday October 31, 2019 15:10 - 15:30 CET
Rhone 3AB

Sessions - Track 2

Experience Level Beginner
Session Slides Included Yes

15:30 CET

Coffee Break

Thursday October 31, 2019 15:30 - 15:50 CET
Place Haute

Breaks & Meals

15:50 CET

Working Together to Build a Modular CI Ecosystem - Discussion Session - Tim Bird, Sony

This session will be a working session to discuss how developers and companies can work together to create an automated test ecosystem. There are technical issues to be investigated, and resources to be shared. How do we go from monolithic test systems, to an test ecosystem of modular pieces?

To some degree this is already happening - different test systems are often composed of different major open source projects (e.g. LKFT/Squad/Lava, KernelCI/LAVA, Fuego/Jenkins/ttc or CKI/Beaker), but there are various hurdles to overcome in order for test users to be able to plug-and-play between different modules.

Come join the discussion about where the module boundaries are, and how to deploy our different pieces so they can be reused by other systems and frameworks.

Speakers

Tim Bird

Principal Software Engineer, Sony Electronics

Thursday October 31, 2019 15:50 - 16:30 CET
Salon Pasteur

Sessions - Track 1

Experience Level Intermediate

15:50 CET

Defining a Standard Board Management API - Jan Lübbe, Pengutronix e.K. & Pawel Wieczorek, Samsung R&D Institute Poland

In this session the group will discuss different approaches to the board management or DUT control layer, and how to unity the industry around a modular architecture. The desired outcome is a description of the API between the Test framework (specifically the test scheduler) and the DUT controller, and a set of guidelines for
projects that implement board management layers, so that they can easily be incorporated into multiple CI frameworks.

Speakers

Jan Lübbe

CTO, Pengutronix e.K.

Pawel Wieczorek

Software Development Engineer, Samsung R&D Institute Poland

Thursday October 31, 2019 15:50 - 16:30 CET
Rhone 3AB

Sessions - Track 2

16:40 CET

Summit Wrap-up - Tim Bird, Sony

In this session, summit attendees will compare notes from the 2 tracks, review and finalize action items taken from the event, and discuss next steps for continuing the progress being made to collaborate on open source automated testing.

Speakers

Tim Bird

Principal Software Engineer, Sony Electronics

Thursday October 31, 2019 16:40 - 17:15 CET
Salon Pasteur

Keynote Sessions