Introducing resinOS: An Operating System Tailored for Containers and Built for the Embedded World

Editor's Notes

• #2: Ποιοι είμαστε Γιατί κάνουμε το resin Πως δουλεύει το resin
• #4: We will upload the slides The links next to the github logo link to the repo for what is mentioned in the slide
• #5: We believe containers give some unique capabilities when applied to IoT Developers use very familiar tools They don’t have to reboot for every iteration If their container completely crashes the OS is still there to recover But also have some unique challenges A container in the cloud interacts with its environment by requiring computation, memory, networking, storage All of these are very well abstracted and have no different if they are virtual or not You can’t have a virtual GPIO Containers in IoT have to interface with hardware sometimes Devices can be connected with to very poor networks or via 3G, bandwidth matters We’re also aiming at making the OS developer friendly and providing the tools that allow you to build great applications We’ve made a lot of progress during the past years and now we’re making an official release as a standalone project
• #6: We were managing a fleet of 200 devices across London Most of the tooling felt primitive. It was a hard problem We wanted to bring more modern tools to the embedded world Bridge yocto world with docker/cloud world Started with `git push` workflow, only supported nodejs projects
• #7: Most distributions are not optimised for remote management and embedded devices We had a very early prototype which sometimes it would start being slow Turned out filesystem indexing was kicking off in the background We had to manually build tooling around creating images, installing software Most distributions support a limited set of architectures
• #8: resinOS is the result of our efforts to have a reliable system in production Has been running on thousands of devices We moved the development to the open a bit more than a year ago We’re now making it a proper open source project
• #10: Yocto had drawbacks, namely steep learning curve, resource-hungry builds Benefits outweight them by far We can disable compile-time features that we don’t use Very big community in the embedded space Provides the right tools for distribution maintainer
• #11: For each board we support there is a GH repo with a well-defined structure We manage yocto dependencies by using git submodules We can move each board independently through versions without blocking the development of meta-resin Normally each such repo has no code, but sometimes some glue code is put there in a non-submoduled layer
• #12: This is where the bulk of the work is happening We need AUFS for docker - we’ll talk about it later on why we chose this backend Meta-resin auto-detects if the kernel provided by the bsp has aufs support (like beaglebone) and automatically applies patches if not Meta-resin automatically extends the kernel config provided by the BSP to enable the right features for container to run Prepopulating docker images means easier mass-production/provisioning
• #13: We use a docker container for the build env yocto runs in Well defined versions of host dependencies One of the artifacts of the build process is a docker image containing the userspace and boot files We are actually pushing this image in dockerhub for every release This can be used as a base for a container or for testing It will also become relevant in the talk when we talk about OS updates
• #14: Meta-resin creates this layout by default Data partition is where all containers and container state is kept State partition is where partition state is kept We generally try to keep the boot partition as a FAT partition This allows our development tools to read and write to it cross-platform
• #15: We chose to have a read-only root partition to make failures much less likely During board boot we bind-mount a lot of relevant directories from the state partition to the appropriate locations Another approach, used by OpenWRT, would be having a read-only root (squashfs in their case) and using overlayfs on top. We didn’t want to be dependent on a union filesystem for the rootfs Some POSIX semantics are lost You can still get random mutations of the rootfs that will bite you later when you update
• #18: The goal of the OS is to be fully updatable but also flexible. If all the application code was living in the host then a complex host OS update would be required for every change. This is where containers come in. Our userspace is designed to do the absolute necessary things to bring up the board and connect to the internet Using this distinction we can have very fast reboot-less updates for containers But we can still update the host with traditional methods but much more infrequently
• #20: The initial versions of our OS were based on sysvinit But we quickly found ourselves having a hard time managing dependencies Sometimes restarting a service wouldn’t kill all its processes Each program was managing its own logs We plan to leverage it a lot more in the future
• #21: Networking outside the data center is a wild place Glibc only supports max 3 nameservers In some networks DHCP dns is broken In some networks google DNS is broken In some networks DNS server take forever to reply In some networks DNS servers reply with the wrong data D-Bus can be easily exposed to containers Have also tried connman We had to patch it to convince it to not do NTP Documentation buried into repo and examples in test suite We didn’t have a good way of persisting a config file describing 3G connections ModemManager gets us that In general much more developer friendly
• #22: Overlay needs 3.18+ and uses a lot of inodes Overlay2 needs 4.0+ BTRFS is not stable enough, especially for low storage devices Devicemapper doesn’t work on NAND Zfs… yeah Docker’s default logging driver will keep appending your stdout into a very big JSON file on disk We configure docker to send logs to journald Meta-resin also enables seccomp in the kernel and docker uses a default profile for all containers
• #23: We don’t allow anything to write its own logs One can easily send the stream of journald logs to a remote server If configured, ResinOS can store logs in the
• #25: Yocto by default produces the host OS image but sometimes this needs to be written to internal storage Dependening on the board this might involve a few steps ResinOS can create a “flasher” type image that copies itself into internal storage and shuts down the board
• #26: There are so many options for OS updates. I think there are 4 talks on this subject in this ELCE Resinhup is how are addressing our current production needs It’s still experimental and tested in a very carefully selected set of customers
• #27: Host OS updates is a very hot topic and is definitely
• #28: One important feature we added recently is integration with docker Pulling down an fs image using a tarball felt wrong when we had docker Currently we’re using docker only for pulling an image and then copy it to the rootfs Docker has a lot of nice features that can be leveraged. Signed images for example
• #29: Automate all possible ways of booting a board ( SD card / DFU / Network booting? ).* Emulate the peripherals that could be connected to the SBC ( GPIO/SPI/I2C/Bluetooth).* All external hardware used to run the tests should be on USB to enable for portability of the entire rig.* Simulate and test different network conditions under which a SBC operates.* Accompanying software components should have an easy DSL/way of writing and extending tests, so that peripheral abstractions can be re-used.* Accompanying software component should enable Continuous Integration with Jenkins* All custom hardware designs should be in KiCad to allow for easy contribution as the project is open source.
• #30: Automate all possible ways of booting a board ( SD card / DFU / Network booting? ).* Emulate the peripherals that could be connected to the SBC ( GPIO/SPI/I2C/Bluetooth).* All external hardware used to run the tests should be on USB to enable for portability of the entire rig.* Simulate and test different network conditions under which a SBC operates.* Accompanying software components should have an easy DSL/way of writing and extending tests, so that peripheral abstractions can be re-used.* Accompanying software component should enable Continuous Integration with Jenkins* All custom hardware designs should be in KiCad to allow for easy contribution as the project is open source.
• #31: 20 boards
• #34: Normally you have to connect with ethernet/serial cable Configuration is tricky Flashing
• #42: All of our code is open source and our goal is to build a community around this project We’ll be more than happy to chat with you on our gitter channel
• #43: Meetup info