Yocto: Unravel the Way Files Take When Packaging Software With BitBake

Introduction

Ever wondered where files go when you package software with bitbake?

In my last post I discussed where files go when you build software with bitbake. I did that in an example project and used openssl as an example software package. If you haven’t read that post, I encourage you to do so before reading this one. I know that the last post is kind of long. If you don’t want to read all of it, go at least through the Introduction to see how the project is set up in order to follow along with the current post. I use the exact same set up here.

In the last post, it would have been possible to run bitbake openssl to run through the whole process from downloading the source code all the way to a finished openssl package. I didn’t do that. Instead, I split the process up and ran individual tasks manually. This allowed me to explain everything in more detail. It also allowed me to stop after the compile task. Missing was what happens afterwards. And that is what this post is all about. But instead of keep using openssl as an example, I decided to use tzdata for this post. Why? Because with tzdata it is a little easier to explain bitbake’s FILES variable.

I will start by building tzdata the same way I build openssl in the last post (without going into details about that process) and then I will explain the steps (tasks) necessary to wrap everything up into packages.

Build tzdata with Bitbake

OK, let’s build the tzdata software the same way I built openssl in my last post. The very first step is to cd into the project directory and source the oe-init-build-env script:

cd <path/to/example/project>
source poky/oe-init-build-env build/

After that, the current working directory is <path/to/example/project>/build. All the following command have to be run in this build directory. Also all given paths are relative to the build directory.

By the way, the recipe for tzdata can be found under ../poky/meta/recipes-extended/timezone/ and is called tzdata.bb. To build tzdata, we need to run the following commands:

bitbake -c fetch tzdata
bitbake -c unpack tzdata
bitbake -c patch tzdata
bitbake -c configure tzdata
bitbake -c compile tzdata

Note: We could have just invoked bitbake -c compile tzdata and bitbake would have taken care of running the tasks that the compile task depends on. But I wanted to show the five steps I ran in the last post once more.

Note: Just like in the last post, it’s a good idea to create an environment file for tzdata to analyze some bitbake variables. This can be done by running: bitbake -e tzdata > environment.txt. I will refer to that file as the “environment file”.

bitbake’s WORKDIR variable is a different for tzdata compared to openssl. I won’t analyze how it is set, as I have done that in detail in the last post. Just this much, tzdata sets the PACKAGE_ARCH variable explicitly to all. To do so, tzdata.bb inherits allarch. allarch.bbclass can be found under ../poky/meta/classes/ and sets the PACKAGE_ARCH variable. The whole WORKDIR variable for tzdata is set to the tmp/work/all-poky-linux/tzdata/2023c-r0 directory within the build directory.

After the compile task you end up with a WORKDIR that looks like this:

$ tree -L 1 tmp/work/all-poky-linux/tzdata/2023c-r0/
tmp/work/all-poky-linux/tzdata/2023c-r0/
├── build
├── deploy-source-date-epoch
├── recipe-sysroot
├── recipe-sysroot-native
├── source-date-epoch
├── sstate-install-deploy_source_date_epoch
├── temp
├── tz
└── configure.sstate

Except for configure.sstate all other entries are directories with more or less sub-directories and files in them.

The compiled files ended up in the build directory inside the WORKDIR. That is because the recipe for tzdata sets the B variable to ${WORKDIR}/build and the recipe defines the whole compile task and used the B variable within it.

Note: tzdata is not a “normal” software where the output of the compilation task is one or multiple executables. Instead the compilation yields files in the timezone information format (TZif). This does not matter for the purpose of this article.

Package tzdata with Bitbake

Now let’s look into the steps necessary to package tzdata and where files end up during that process.

do_install

The first step is to run the install task:

bitbake -c install tzdata

This task places the compiled files into a “holding” area that bitbake’s D variable points to. D is set to ${WORKDIR}/image. The install task is entirely defined in tzdata.bb and it actually uses the D variable 😀.

That means that after the install task finishes you can find a new directory called image in the WORKDIR:

$ tree -L 4 tmp/work/all-poky-linux/tzdata/2023c-r0/image/
tmp/work/all-poky-linux/tzdata/2023c-r0/image/
├── etc
│   ├── localtime -> /usr/share/zoneinfo/Universal
│   └── timezone
└── usr
    └── share
        └── zoneinfo
            ├── Africa
            ├── (...)    // output truncated

The image directory contains two sub-directories (etc and usr). These are the sub-directories that would contain files from the tzdata package on a Linux machine.

do_package

Once the holding area is filled. It’s time for the next step which is the package task:

bitbake -c package tzdata

The package task creates three new directories in the WORKDIR that are of interest here:

• package (pointed to by bitbake’s PKGD variable)

  It contains the whole tzdata package before it is split up into individual packages. In this case it’s basically a copy of the image directory.

• packages-split (pointed to by bitbake’s PKGDEST variable)

  We will look into this directory in just a bit.

• pkgdata (pointed to by bitbake’s PKGDESTWORKDIR variable)

  This is a temporary work directory used by the package task to keep some metadata about the individual packages.

If you look into the packages-split directory, you will find the following:

$ tree -L 1 tmp/work/all-poky-linux/tzdata/2023c-r0/packages-split/
tmp/work/all-poky-linux/tzdata/2023c-r0/packages-split/
├── tzdata
├── tzdata-africa
├── tzdata-americas
├── tzdata-antarctica
├── tzdata-arctic
├── tzdata-asia
├── tzdata-atlantic
├── tzdata-australia
├── tzdata-core
├── tzdata-europe
├── tzdata-misc
├── tzdata-pacific
├── tzdata-posix
├── tzdata-right
└── tzdata-src

Where do these directories come from?

If you look into the environment file and search for the PACKAGES variable, you will find that this variable is a list of packages with the same names as the directories. These PACKAGES are defined in the recipe file tzdata.bb. Also in tzdata.bb, you can find multiple lines that look like these examples:

FILES:tzdata-antarctica += "${datadir}/zoneinfo/Antarctica"

FILES:tzdata-arctic += "${datadir}/zoneinfo/Arctic"

These definitions define which files from tzdata go into which of the individual packages and therefore different directories in ${WORKDIR}/packages-split.

do_packagedata

This step creates metadata for the subsequent step of generating the actual packages. The data can be found in ${WORKDIR}/pkgdata-pdata-input.

do_package_write_rpm

Now it is finally time to generate the packages that can then be installed into a distribution image. In this case we need to run the package_write_rpm task.

Why to we create rpm packages?

If you have a look into the set up of the example project that we work in all the time, you find the PACKAGE_CLASSES variable set to package_rpm in the conf/local.conf file (see the Introduction of my last post). This variable determines which type of packages will be build. An alternative to rpm packages would, for example, be deb packages.

All right let’s run the package_write_rpm task:

bitbake -c package_write_rpm tzdata

This will create all the individual packages in a sub-directory of the ${DEPLOY_DIR_RPM} directory. The name of this sub-directory depends on the PACKAGE_ARCH variable. Usually it is just exactly the value of that variable, but if PACKAGE_ARCH is set to all (see above), the sub-directory is called noarch. The reason for why that is the case can be found in the documentation file ../poky/documentation/migration-guides/migration-2.3.rst.

All right, looking into tmp/deploy/rpm/noarch (${DEPLOY_DIR_RPM}/noarch) reveals the following:

$ tree -L 1 tmp/deploy/rpm/noarch/
tmp/deploy/rpm/noarch/
├── tzdata-2023c-r0.noarch.rpm
├── tzdata-africa-2023c-r0.noarch.rpm
├── tzdata-americas-2023c-r0.noarch.rpm
├── tzdata-antarctica-2023c-r0.noarch.rpm
├── tzdata-arctic-2023c-r0.noarch.rpm
├── tzdata-asia-2023c-r0.noarch.rpm
├── tzdata-atlantic-2023c-r0.noarch.rpm
├── tzdata-australia-2023c-r0.noarch.rpm
├── tzdata-core-2023c-r0.noarch.rpm
├── tzdata-europe-2023c-r0.noarch.rpm
├── tzdata-misc-2023c-r0.noarch.rpm
├── tzdata-pacific-2023c-r0.noarch.rpm
├── tzdata-posix-2023c-r0.noarch.rpm
└── tzdata-right-2023c-r0.noarch.rpm

There you have it, readily build tzdata packages that can be installed into a distribution image, but this would be a topic for another post.

Take care,
Andreas

---

References