2 posts tagged with "cloud"

Intro​

The meta-erlang on cloud II, build your own application post has shown the basic idea for how would be possible (and in fact it is possible) to build a custom Linux distribution and run it on Graviton AWS instances.

Now, it's time to refactor out the previous result. Removing what are not necessary in order to create a slim meta-erlang cloud ready image.

So, this post is more like a tutorial to reach two main targets:

• Explore a demo Yocto layer which shows the basic pieces to run a distro on Graviton
• Add livebook to that image and try it on cloud. Why not trying some GPU processing :)

meta-axon cloud aware​

meta-axon is my playground layer focused on demonstrate how Erlang and Elixir applications could be integrated with Yocto ecosystem. Until now, meta-axon has been tailored to run on embedded devices (like raspberry pi). But, I'm also added, along side with the former axon-slim, a new distro called axon. The axon distro is focused on cloud, for the purpose of this tutorial AWS cloud.

I also introduced a new image called axon-image-cloud with all the pieces needed to run it on Graviton instances. This image relies on systemd plus cloud-init scripts.

It is important to mention that to run images on Graviton it is necessary to add two layer dependencies:

• meta-arm, implementing a generic ARM 64 bits machine
• meta-virtualization, bringing cloud-init recipe

The rest of the needed components are all provided by standard Yocto layers (like: poky and meta-openembedded).

Build instructions​

We want fast feedback and test the build results quickly. So here is the procedure that will build an distro linux image with Erlang, Elixir and livebook:

1. Clone the meta-axon layer:

   git clone https://github.com/meta-erlang/meta-axon.git
   cd meta-axon
   

2. Install the kas tool to setup a bitbake based project.

   git clone https://github.com/siemens/kas
   cd kas
   pip3 install .
   

3. Call kas tool to build everything:

   kas build kas/machines/meta-erlang-graviton2-livebook.yaml
   
   note

   The final image has been created in the deploy folder: tmp/deploy/images/generic-arm64.

4. Run the script create-ami.sh to convert and upload a image made with Yocto into AWS AMI. Use your s3 bucket and specific how much extra space you want:

   ../scripts/create-ami.sh <add your s3 bucket here> 8
   

Launching EC2 instances​

After uploading the image, it's time to lunch an EC2 instance like the following:

And, finally here is it running:

Accessing the instance via ssh and login with the user axon should work as expected. After all it is a normal Linux distro.

Results​

The following two screenshots show the livebook instance up and running on Graviton EC2 instance. It is pretty cool.

Just showing the top command to check all the running process:

And here is the livebook system dashboard to inspect some usage statistics, and system information:

As you can see, it is running on ARM64, with JIT enable.

Conclusion​

Now we are converging to create the basic building blocks for creating custom Linux distribution based on Yocto Project that runs on Cloud.

That could be the perfect case for projects that need to take control on every detail of the software stack. From linx kernel configuration, building flags, software configuration flags, anything. It's really flexible.

And for projects that relies on Erlang and Elixir, it is also amazing bringing it to Cloud ecosystem.

The first blog part meta-erlang on cloud I: EWAOL based has introduced what is possible to build. This blog post is more related to explore the introduced idea into something feasible and aligned with meta-erlang purposes.

Intro​

The Yocto Project has been made to create specific embedded Linux distributions. And still does a great job on this field making it possible to run softwares in a range of target architectures.

But, the term embedded Linux distribution should be expanded a bit more; it's not only about creating tiny filesystem within cross-compiled software, scripts and configurations targeting platforms with small footprint, CPU and memory constraints.

We could do more, actually there are people using the not so newly but still new ARM processors that runs on cloud environment. For sure the most famous one is the Graviton EC2 instances, on AWS cloud.

Looking into these Graviton processors and Yocto Project, what would it be possible ? Maybe running Yocto images on Graviton instances could be feasible enough to run some workload. And what about adding Erlang/OTP and Elixir to that mix ? Or running some advanced database or broker made with Erlang ?

That is exactly what we will try to experiment. In this first post about running meta-erlang on cloud, we are going to look how it would be possible.

Actually there are folks whose made possible to run Yocto images on Graviton instances. Technically is pretty simple and all bits and bytes have been described in the blog post Building an Automotive Embedded Linux Image for Edge and Cloud Using Arm-based Graviton Instances, Yocto Project, and SOAFEE. The companion repository also worth the reading The meta-aws-ewaol repository.

The idea that [1] brings is part of another idea related to Software-Defined Vehicles through Cloud [2] which is very excited. These references also cites a framework to build Edge Workloads [3] and in fact the EWAOL (Edge Workload Abstraction and Orchestration Layer) was used as a base to produce the meta-aws-ewaol layer.

Putting everything together, first experiment​

So, based on all what has been described so far and as a first experience, I've tested meta-aws-ewaol setup and added meta-erlang into that. The aim was to run erl and iex console on Graviton EC2 instances.

I had to make a few fixes on meta-aws-ewaol and pushed the results here: meta-aws-ewaol with meta-erlang. The rest of this session is a small guide that I made in order to run our first experiment.

With a ready Yocto environment, following the Building EWAOL session is enough to get the system built. It's just Yocto, nothing more.

The next step is to create and setup all the AWS infrastructure needed in order to push the Yocto image as AWS AMI and start the Graviton Instance. Lucky, there is a Cloud Formation template which performs the initial setup. The session Pre-requisites helps to clarify it.

Then, having the Yocto image and the basic AWS account in place. It's time to use a special script to convert Yocto image to an AMI image. Most of the job is done by awscli commands. There is nothing related to Yocto here. The session Creating AMI from image file tells what is necessary to do.

Finally, the last part. After pushing the AMI image to the S3 bucket. The fun part is to instantiate a Graviton instance (please refer to the session Launch the EC2 Image as usual using your newly created AMI) an run the EWAOL based image and test how Erlang and Elixir looks like.

I've added two screenshots below just to have a taste of the final results:

Conclusion​

This first experiment has finished and I can conclude two main bullets:

• It is possible to build a Yocto image and run it on Graviton instances
• meta-aws-ewaol is a great start point. And the EWAOL framework looks solid for what it has been planned for.

However, what I am looking for is to run a tiny image with just the necessary to instantiate an ARM cloud instance running Erlang/Elixir. That is what the next blog post will do: meta-erlang on cloud II, build your own application

References​

• [1]: Building an Automotive Embedded Linux Image for Edge and Cloud Using Arm-based Graviton Instances, Yocto Project, and SOAFEE
• [2]: Accelerating Software-Defined Vehicles through Cloud-To-Vehicle Edge Environmental Parity
• [3]: Edge Workload Abstraction and Orchestration Layer (EWAOL) Documentation