We've added support for the following new Erlang/OTP releases:

We've added support for the following new Erlang/OTP releases:

In additional to those new versions, now it's possible to read Erlang/OTP documentation in Erlang shell. The docs .chunk files get generated during build time and one could easily install it by adding the package erlang-modules-dev.

Intro​

From atomvm.net website, atomvm is:

AtomVM is a lightweight implementation of the the Bogdan Erlang Abstract Machine (aka, the BEAM), a virtual machine that can execute byte-code instructions compiled from Erlang or Elixir source code. AtomVM supports a limited but functional subset of the BEAM opcodes, and also includes a small subset of the Erlang/OTP standard libraries, all optimized to run on tiny micro-controllers. With AtomVM, you can write your IoT applications in a functional programming language, using a modern actor-based concurrency model, making them vastly easier to write and understand!

One of the atomvm goals is the possibility to run BEAM code on really small systems like MCUs.

For a while, I was wondering what could be the benefits of integrating atomvm into meta-erlang recipes. It didn't look to make any sense for me. Then, I started playing with that just to see if I was able to run an atomvm program in Qemu emulation.

Suddenly, I ended up with all pieces to run atomvm programs integrated with meta-erlang. So this post is about how it is possible to use atomvm in Linux images based on YP/OE.

atomvm recipe​

The atomvm.bb recipe is very simple because the atomvm project is based on CMake and YP has support to it.

However, I had to make a specific recipe to isolate the tool packbeam. The packbeam tool lives inside atomvm source code I've created a recipe called packbeam.bb which compiles packbeam as native (that is runs on build host machine). That was necessary because the atomvm recipe crosscompile the atomvm and packbeam is used by CMake to pack all .beam files to create .avm files.

To make atomvm recipe works, I applied a patch (0001-Remove-PackBEAM-dependency-rules.patch) to disable the packbeam dependency internally. That allows me to use my packbeam tool from sysroot-native instead.

atomvm examples​

There is the atomvm_examples project that provides great source of ideas and examples. I wish to create some recipes to build and pack each example. So I started with the system_info.

The following code is the atomvm-examples-system-info_0.1.0.bb recipe. And should be enough for any Erlang project build which uses atomvm_rebar3_plugin.

SUMMARY = "Collects and displays various information about AtomVM and the environment in which it is running."
SECTION = "examples"
LICENSE = "Apache-2.0"
LIC_FILES_CHKSUM = "file://LICENSE;md5=745e8b23501916820b8a509f8e3ba3d4"

ATOMVM_EXAMPLE = "erlang/system_info"

S = "${WORKDIR}/system_info"

SRCREV = "8e54aaf475a74b59a20f914e575202b1810a7954"
PV = "0.1.0+git${SRCPV}"
SRC_URI = "git://github.com/atomvm/atomvm_examples;branch=master;subpath=${ATOMVM_EXAMPLE};protocol=https"

inherit atomvm

The inherit atomvm inherits the atomvm.bbclass which implements rebar3 commands to compile and create avm files.

Build and run session​

To build the atomvm-examples-system-info recipe, we call bitbake like that:

bitbake atomvm-examples-system-info

Now, we want to run system_info application inside Qemu. The first step is to install the atomvm-examples-system-info in the final image.

In conf/local.conf file add the recipe name to the IMAGE_INSTALL variable:

IMAGE_INSTALL:append:pn-core-image-minimal = " atomvm-examples-system-info"

When building the core-image-minimal image and running it with Qemu:

runqemu core-image-minimal slirp nographic serialstdio

Inside the Qemu, let's run our first atomvm program:

root@qemux86-64:~# uname -a
Linux qemux86-64 6.5.7-yocto-standard #1 SMP PREEMPT_DYNAMIC Thu Oct 19 14:51:09 UTC 2023 x86_64 GNU/Linux
root@qemux86-64:~# atomvm /usr/share/atomvm-examples-system-info/system_info.avm
Unsupported line_ref tag: 0
SystemInfo:
===========
atom_count: 162
port_count: 0
process_count: 1
system_architecture: <<"Linux--x86_64">>
word_size: 8

PlatformInfo:
=============

ProcessInfo:
============
Pid: <0.1.0>
heap_size: 51
memory: 848
message_queue_len: 0
stack_size: 9

Return value: ok

That works as expected.

Integration outcomes and questions​

The recipes created so far covers running atomvm for generic unix (Linux in my case). Well, if you can afford running Erlang/Elixir on Linux, then there is no selling point to use atomvm. That is true if we look at the current state of atomvm project for generic unix platform.

Maybe in the future we can see extensions to generic unix allowing it to talk with i2c, gpio, spi on Linux for example, then atomvm and meta-erlang starts to cover a lot of possibilities. Projects like Eclipse Mraa could be integrated with atomvm providing all the low level access for low speed IO.

There is another idea for meta-erlang and atomvm which is the heterogeneous system with a combination of:

• Linux based images running on the "application processing unit" (CPU)
• and "real-time processing unit" (MCU) running an atomvm program.

meta-erlang could build atomvm images for MCU as well for CPU. The seeds for this integration was described in this talk One Build to Rule Them All: Building FreeRTOS & Linux Using Yocto - Alejandro Hernandez (the pdf is here). In that talk Alejandro shows how YP/OE projects can build MCU target images. Using the same principles also works for meta-erlang.

Intro​

According to Wikipedia X32 ABI page:

The x32 ABI is an application binary interface (ABI) and one of the interfaces of the Linux kernel. The x32 ABI provides 32-bit integers, long and pointers (ILP32) on Intel and AMD 64-bit hardware. The ABI allows programs to take advantage of the benefits of x86-64 instruction set (larger number of CPU registers, better floating-point performance, faster position-independent code, shared libraries, function parameters passed via registers, faster syscall instruction) while using 32-bit pointers and thus avoiding the overhead of 64-bit pointers.

So, I'm wondering if it would be possible to enable x32 support in Erlang/OTP build. That way, I could make a Yocto image for x32 that runs on x86-64 machines.

Here is some references about the subject:

• Yocto, Using x32 psABI
• x32-abi
• X32 – A Native 32bit ABI For X86-64
• Debian x32 port
• System V Application Binary Interface AMD64 Architecture Processor Supplement (With LP64 and ILP32 Programming Models

In fact, x32 seems to be around since 2011/2012 and has been integrated in many platforms. Like Ubuntu, Debian, Gentoo.

Building Erlang/OTP using x32 toolchain​

In order to follow this experiment, you can download a specific toolchain with x32 enabled here: poky-glibc-x86_64-core-image-minimal-x86_64_x32-qemux86-64-toolchain-4.2.sh.

It will be necessary to install it in a temporary folder like the steps below:

chmod +x poky-glibc-x86_64-core-image-minimal-x86_64_x32-qemux86-64-toolchain-4.2.sh
poky-glibc-x86_64-core-image-minimal-x86_64_x32-qemux86-64-toolchain-4.2.sh -y -d /tmp/poky/4.2

Following the Erlang/OTP INSTALL-CROSS.md document, we first need to build a Bootstrap System:

cd $ERL_TOP
./configure --enable-bootstrap-only
make

Next, we have to source the toolchain environment configurations:

. /tmp/poky/4.2/environment-setup-x86_64_x32-poky-linux-gnux32

After sourcering the environment variable, the shell gets configured with some extra variables using during the build:

x86_64-poky-linux-gnux32-gcc -mx32 \
 -fstack-protector-strong \
 -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security \
 --sysroot=/tmp/poky/4.2/sysroots/x86_64_x32-poky-linux-gnux32

It's important to not the GCC flag -mx32:

The -mx32 option sets int, long, and pointer types to 32 bits, and generates code for the x86-64 architecture.

Finally, start the second part of Erlang/OTP build, which is the Cross Build:

./configure  $CONFIGURE_FLAGS  --disable-silent-rules --disable-dependency-tracking \
    --with-ssl-rpath=no --disable-static  --without-javac --without-dynamic-trace --without-observer --without-odbc

Installing the build output and inspecting the erlexec binary to see what it looks like:

make install DESTDIR=/tmp/e

file /tmp/e/usr/local/lib/erlang/erts-14.0.2/bin/erlexec
erlexec: ELF 32-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /libx32/ld-linux-x32.so.2, BuildID[sha1]=013e32ef8c57686a59a812ca452f09d677ff8e37, for GNU/Linux 5.15.0, with debug_info, not stripped

Well, the build is correct. But I couldn't test this build in my machine.

Enabling Erlang for Yocto​

In the previous section we just build Erlang/OTP using a toolchain with x32 support. Now, it's time to build Erlang/OTP inside the Yocto project and test the results using qemu instance.

Enabling it for Yocto is simple, just adding the follow snippet in your local.conf file:

MACHINE = "qemux86-64"
DEFAULTTUNE = "x86-64-x32"
baselib = "${@d.getVar('BASE_LIB:tune-' + (d.getVar('DEFAULTTUNE') or 'INVALID')) or 'lib'}"

Then, building erlang:

bitbake erlang

The build failed in one point related to ASM code in erts/lib_src/pthread/ethread.c. Looks like an ASM incompatibility issue. In order to address it here is a patch that just adds one more compiler check to pick up the correct ifdef branch.

Afer that, the build runs as expected. And testing it using QEMU shows exactly what I had in mind:

runqemu core-image-minimal-qemux86-64.ext4 slirp nographic serialstdio

• check the current kernel

root@qemux86-64:~# uname -a
Linux qemux86-64 6.1.32-yocto-standard #1 SMP PREEMPT_DYNAMIC Mon Jun  5 13:43:33 UTC 2023 x86_64 GNU/Linux

• check /proc/cpuinfo to see the 'lm' (long mode)

root@qemux86-64:~# grep -o -w 'lm' /proc/cpuinfo
lm

• check Erlang shell

Erlang/OTP 26 [erts-14.0.2] [source] [32-bit] [smp:4:4] [ds:4:4:10] [async-threads:1]

Eshell V14.0.2 (press Ctrl+G to abort, type help(). for help)
1> application:ensure_all_started(crypto).
{ok,[crypto]}

Well, looks like we are running Erlang/OTP 32-bits in a x86_64 machine. Also, it was able to correct load the crypto (with ssl libraries compiled for x32 too). By the way, there is a second patch need to proper compile the crypto application.

Some raised questions for further investigations:

• What tests are necessary to prove that the x32 Erlang build is safe ?
• Are there any other code change in order to fit the x32 build ?
• Would BeamAsm be available for x32 ?

We've added support for the following new Erlang/OTP releases:

In additional to those new versions, a new beamtools is available here.

This new version was made with Yocto mickledore release and brings Erlang/OTP 26.0.2 and Elixir 1.15.2.

Yocto Project has released a new version 4.2 (Mickledore). As consequence meta-erlang brings a new branch mickledore updating most of the recipes. mickledore release is not a LTS release and community support will be available until November 2023.

We've also upgraded to the latest Erlang/OTP and Elixir releases supporting the following new Erlang/OTP releases:

• Erlang/OTP 26.0.1
• Erlang/OTP 25.3.2.2
• Elixir 1.14.4

It's worth mentioning the meta-erlang master has also been updated.

We've added support for the following new Erlang/OTP releases:

Removed recipes:

• For langdale the recipes ejabberd and emqx have been removed due the lack of supporting cross-compilation build in a stable way.

Updated recipes:

• For langdale the following recipes have been updated: lux, vernemq, yaws, tsung, riak, couchdb. It is important to mention about langdale EOL in May 2023. So, this branch will not receive updates.

It has been decided:

• Dropped support to Erlang/OTP 24 in langdale. Due the lack of deterministic build flags in 24. Elixir 1.12 has also been dropped (because it dependes on Erlang/OTP 24).

Intro​

The meta-erlang on cloud II, build your own application post has shown the basic idea for how would be possible (and in fact it is possible) to build a custom Linux distribution and run it on Graviton AWS instances.

Now, it's time to refactor out the previous result. Removing what are not necessary in order to create a slim meta-erlang cloud ready image.

So, this post is more like a tutorial to reach two main targets:

• Explore a demo Yocto layer which shows the basic pieces to run a distro on Graviton
• Add livebook to that image and try it on cloud. Why not trying some GPU processing :)

meta-axon cloud aware​

meta-axon is my playground layer focused on demonstrate how Erlang and Elixir applications could be integrated with Yocto ecosystem. Until now, meta-axon has been tailored to run on embedded devices (like raspberry pi). But, I'm also added, along side with the former axon-slim, a new distro called axon. The axon distro is focused on cloud, for the purpose of this tutorial AWS cloud.

I also introduced a new image called axon-image-cloud with all the pieces needed to run it on Graviton instances. This image relies on systemd plus cloud-init scripts.

It is important to mention that to run images on Graviton it is necessary to add two layer dependencies:

• meta-arm, implementing a generic ARM 64 bits machine
• meta-virtualization, bringing cloud-init recipe

The rest of the needed components are all provided by standard Yocto layers (like: poky and meta-openembedded).

Build instructions​

We want fast feedback and test the build results quickly. So here is the procedure that will build an distro linux image with Erlang, Elixir and livebook:

1. Clone the meta-axon layer:

   git clone https://github.com/meta-erlang/meta-axon.git
   cd meta-axon
   

2. Install the kas tool to setup a bitbake based project.

   git clone https://github.com/siemens/kas
   cd kas
   pip3 install .
   

3. Call kas tool to build everything:

   kas build kas/machines/meta-erlang-graviton2-livebook.yaml
   
   note

   The final image has been created in the deploy folder: tmp/deploy/images/generic-arm64.

4. Run the script create-ami.sh to convert and upload a image made with Yocto into AWS AMI. Use your s3 bucket and specific how much extra space you want:

   ../scripts/create-ami.sh <add your s3 bucket here> 8
   

Launching EC2 instances​

After uploading the image, it's time to lunch an EC2 instance like the following:

And, finally here is it running:

Accessing the instance via ssh and login with the user axon should work as expected. After all it is a normal Linux distro.

Results​

The following two screenshots show the livebook instance up and running on Graviton EC2 instance. It is pretty cool.

Just showing the top command to check all the running process:

And here is the livebook system dashboard to inspect some usage statistics, and system information:

As you can see, it is running on ARM64, with JIT enable.

Conclusion​

Now we are converging to create the basic building blocks for creating custom Linux distribution based on Yocto Project that runs on Cloud.

That could be the perfect case for projects that need to take control on every detail of the software stack. From linx kernel configuration, building flags, software configuration flags, anything. It's really flexible.

And for projects that relies on Erlang and Elixir, it is also amazing bringing it to Cloud ecosystem.

The first blog part meta-erlang on cloud I: EWAOL based has introduced what is possible to build. This blog post is more related to explore the introduced idea into something feasible and aligned with meta-erlang purposes.

Intro​

The Yocto Project has been made to create specific embedded Linux distributions. And still does a great job on this field making it possible to run softwares in a range of target architectures.

But, the term embedded Linux distribution should be expanded a bit more; it's not only about creating tiny filesystem within cross-compiled software, scripts and configurations targeting platforms with small footprint, CPU and memory constraints.

We could do more, actually there are people using the not so newly but still new ARM processors that runs on cloud environment. For sure the most famous one is the Graviton EC2 instances, on AWS cloud.

Looking into these Graviton processors and Yocto Project, what would it be possible ? Maybe running Yocto images on Graviton instances could be feasible enough to run some workload. And what about adding Erlang/OTP and Elixir to that mix ? Or running some advanced database or broker made with Erlang ?

That is exactly what we will try to experiment. In this first post about running meta-erlang on cloud, we are going to look how it would be possible.

Actually there are folks whose made possible to run Yocto images on Graviton instances. Technically is pretty simple and all bits and bytes have been described in the blog post Building an Automotive Embedded Linux Image for Edge and Cloud Using Arm-based Graviton Instances, Yocto Project, and SOAFEE. The companion repository also worth the reading The meta-aws-ewaol repository.

The idea that [1] brings is part of another idea related to Software-Defined Vehicles through Cloud [2] which is very excited. These references also cites a framework to build Edge Workloads [3] and in fact the EWAOL (Edge Workload Abstraction and Orchestration Layer) was used as a base to produce the meta-aws-ewaol layer.

Putting everything together, first experiment​

So, based on all what has been described so far and as a first experience, I've tested meta-aws-ewaol setup and added meta-erlang into that. The aim was to run erl and iex console on Graviton EC2 instances.

I had to make a few fixes on meta-aws-ewaol and pushed the results here: meta-aws-ewaol with meta-erlang. The rest of this session is a small guide that I made in order to run our first experiment.

With a ready Yocto environment, following the Building EWAOL session is enough to get the system built. It's just Yocto, nothing more.

The next step is to create and setup all the AWS infrastructure needed in order to push the Yocto image as AWS AMI and start the Graviton Instance. Lucky, there is a Cloud Formation template which performs the initial setup. The session Pre-requisites helps to clarify it.

Then, having the Yocto image and the basic AWS account in place. It's time to use a special script to convert Yocto image to an AMI image. Most of the job is done by awscli commands. There is nothing related to Yocto here. The session Creating AMI from image file tells what is necessary to do.

Finally, the last part. After pushing the AMI image to the S3 bucket. The fun part is to instantiate a Graviton instance (please refer to the session Launch the EC2 Image as usual using your newly created AMI) an run the EWAOL based image and test how Erlang and Elixir looks like.

I've added two screenshots below just to have a taste of the final results:

Conclusion​

This first experiment has finished and I can conclude two main bullets:

• It is possible to build a Yocto image and run it on Graviton instances
• meta-aws-ewaol is a great start point. And the EWAOL framework looks solid for what it has been planned for.

However, what I am looking for is to run a tiny image with just the necessary to instantiate an ARM cloud instance running Erlang/Elixir. That is what the next blog post will do: meta-erlang on cloud II, build your own application

References​

• [1]: Building an Automotive Embedded Linux Image for Edge and Cloud Using Arm-based Graviton Instances, Yocto Project, and SOAFEE
• [2]: Accelerating Software-Defined Vehicles through Cloud-To-Vehicle Edge Environmental Parity
• [3]: Edge Workload Abstraction and Orchestration Layer (EWAOL) Documentation

We've added support for the following new Erlang/OTP releases:

Fixed bugs:

• QA Issue buildpaths (reproducible builds) when building with langdale, a specific patch has been integrated in order to avoid building flags into the final beam.smp binary.

To be decided:

• Yocto Langdale release has enabled reproducible build checks. Erlang/OTP from 25 and later implements special flags to fix reproducible issues. But Erlang/OTP 24 does not have these special flags as a consequence langdale Erlang/OTP 24 builds warns about: packages contains reference to TMPDIR [buildpaths]. There are two paths that we could get. One is to drop Erlang/OTP 24 support OR disable QA buildpaths checking.

This new website is based on Docusaurus documentation framework. The previous site was using docsify, which is also great. But Docusaurus address others concerns when writing documentation for open source project.

So, the site has a new frontend page and a blog session where we want to add latest meta-erlang release highlights. The documentation session is pretty much the same plus some improvements on Introduction chapter.