Andreas Schuster

Andreas Schuster

Yocto: Project Directory Structure

Updated: 9 minute read

Introduction

When I started out using Yocto, I was never sure about the best way to set up a directory structure for a project that uses Yocto. By now I do have a way to set up project directories that works for me. I thought, I’d share it. On one hand, as inspiration for others, and on the other hand, to possibly receive feedback from experts who can tell me if this makes sense or suggest improvements.

There are certain things that I like the project directory structure to support. They are:

  • Everything that is not built/generated in one way or the other, should be under version control (in a Git repository). I don’t like losing work.
  • Everything that is built, generated, or downloaded should be in a very limited number of places. I want to be able to get rid of it and get back to a clean build environment quickly. Additionally this requirement ensures that ignoring such data by means of entries in .gitignore is fairly easy. Ideally there are only a few directories that have to be added to .gitignore and not a long list of direcoties and files that are scattered all over the place.
  • It should be possible to generate a new clean build environment and trigger a build process quickly and more or less automatically.

One thing I did to figure out how to set up a project directory structure was reading the Yocto Project Reference Manual [1]. Well, at least parts of it. In that manual you can find a chapter about the Source Directory Structure [2]. This chapter describes the internal directory structure of the poky directory that you can obtain by cloning the poky Git repository:

git clone git://git.yoctoproject.org/poky

But this directory structure is not what this article is about. The poky directory that the previous command generates is just one part of the project directory. I just mention the Source Directory Structure [2] chapter of the Yocto Project Reference Manual [1], because it talks about the build directory within the poky directory. Although I’m a big fan of Yocto’s documentation (it is really good), I don’t like the idea of building anything inside source directories. This is a general thing for me and not only related to Yocto. Luckily the reference manual also mentions that you can run “out of tree” builds which I much prefer.

All right, enough of the introduction. Here comes my current take on how to set up a project using Yocto. This might change over time and I would love to hear your thoughts about it. So don’t hesitate to contact me about it. Whether you do or don’t like my approach, any feedback is welcome.

Project Directory Structure

Assume your current working directory contains a directory for your (embedded) Linux project that uses Yocto as a build system and is called <project_dir>. I like the project directory to be set up in such a way that running tree on it, reveals the following:

$ tree -a -L 2 <project_dir>/
<project_dir>/
├── bitbake               // ignored by Git
│   ├── downloads
│   └── sstate-cache
├── build                 // ignored by Git
│   └── conf
├── .git
│   └── (...)             // output truncated
├── layers
│   ├── meta-bootstrap
│   ├── <meta-layer-1>    // possibly added as git submodule
│   ├── <meta-layer-2>    // possibly added as git submodule
│   ├── (...)
│   └── <meta-layer-n>    // possibly added as git submodule
├── poky                  // added as git submodule
│   └── (...)             // output truncated
├── .gitignore
├── .gitmodules
└── README.md

(...)

The following sections will describe each of the directories within the <project_dir> in more detail.

<project_dir>/

The <project_dir> is the top level directory of the project and should be named meaningfully. It is also the top-level directory for version control (the .git directory resides in it). Besides some sub-directories which will be discussed next, the project_dir contains three files:

  • .gitignore

    The .gitignore file is used to ignore the bitbake and the build directory. It might look like this:

    # .gitignore
bitbake
build

  • .gitmodules

    This file is generated by Git, when adding submodules.

  • README.md

    Well, any decent project should have a README.md file giving an introduction and possible further useful information about the project.

project_dir/bitbake/

The bitbake directory is used for files downloaded or generated by bitbake. Sources that bitbake downloads during a build process will go into the downloads sub-directory. In order for that to work, bitbakes’s DL_DIR variable has to be set appropriately in the build/conf/local.conf file. The shared state cache files that bitbakes generates during a build process will go into the sstate-cache sub-directory. In order for that to work, bitbakes’s SSTATE_DIR variable has to be set appropriately in the build/conf/local.conf file.

The bitbake directory is just an additional hierarchy level to keep the <project_dir> clean and I feel that the two containing sub-directories kind of belong into the same context.

Since everything in this directory is either downloaded or generated, the bitbake directory is not under version control (ignored by git, see above: .gitignore).

There may be cases when I want the downloads or sstate-cache direcetory to be physically stored somewhere else, but not within the project itself. In such case, I still like to have them in the bitbake directory, but I just use symbolic links.

project_dir/build/

The build directory is were bitbakes builds software, packages that software, and generates images. Everything in this directory is either generated by the poke/os-init-build-env script (see section about the meta-bootstrap directory) or during bitbake build runs. Therefore this directory is not under version control (ignored by git, see above: .gitignore).

Note: Even though everything in the build directory is generated, be careful when deleting it. It should be possible to delete this directory and recreate it fairly easily, but the conf sub-directory is a little bit special. It is perfectly fine to edit the contents of the conf sub-directory, mainly the local.conf and the bblayers.conf files during development. But once you reach a state that you feel builds your project nicely, you should make sure the configuration of your build environment is saved in your project repository. This is what I like to use the meta-bootstrap layer for (see corresponding section below). Before wiping out the build directory, make sure you can automatically recreate the configuration files and you don’t lose any local configuration that you might still need later on.

project_dir/.git/

The directory that Git does its thing in. No need to worry about it too much. You just should not mess with it.

project_dir/layers/

I’m not sure if this is just me, but to me it seems as if most people place additional layers that are not part of the poky directory directly into the <project_dir>. I like to introduce an additional level (layers) into the directory hierarchy. To me this feels better: it keeps the top-level <project_dir> cleaner and all layers (except the ones that belong to poke) are within one directory.

The names <meta-layer-1> through <meta-layer-n> are just placeholders for the real layer names. These layers may come from different sources:

  • External sources like the OpenEmbedded Layer Index [3] for example. In this case, they should be added as git submodules.
  • They may be layers developed by you or your team. If such a layer is specific to the very project and it is not planned that it will be shared publicly or with other internal project, it might just live as a normal sub-directory within the project’s Git repository. In case you do want to share it, it makes sense to develop the layer within an additional external Git repository and to add it as a submodule.

The directory structure and the name of any layer within the layers directory should follow the recommendations from the Yocto project. You can read about how to create layers in the Yocto Project Development Tasks Manual [4], specifically in the chapter about Understanding and Creating Layers [5].

The meta-bootstrap directory within layers is kind of special and will be explained separately.

project_dir/layers/meta-bootstrap/

To set up a build environment or to start working with an existing build environment (within the build directory), you usually go into the <project_dir> and run:

source poky/oe-init-build-env build

If you don’t have a build environment yet and you have not set the TEMPLATECONF environment variable, this sets up a more or less empty build directory based on templates in the poky directory. All it contains are some configuration files, namely:

  • bblayers.conf
  • local.conf

These files are used by bitbake to build your project.

This is all fine, if you just start out with a project. But once you have set up a working build environment, you what to have some means for new developers or yourself to recreate the build environment automatically. This is where the meta-bootstrap directory comes into play. It allows you to integrate a template configuration into your project. There is a Yocto Project Development Tasks Manual [4] that explains how this is done. It is called Creating a Custom Template Configuration Directory [6].

project_dir/poky/

I don’t want to go into what’s inside the poky directory, as it is very well described here [2]. One way to get the poky directory is to download a release tar ball from the Yocto Project Website [7], but the What I wish I’d known about Yocto Project [8] article recommends using Git instead. Compared to the Introduction, I actually prefer adding it as a submodule to my projects.

Conclusion

The outlined project directory structure with its four “main” directories (bitbake, build, layers, poky) fulfills all my requirements:

  • Everything that is not generated by a build process is version controlled in Git. The only exception might be some configuration files in build/conf during the development. But everything that is needed to generate a new clean build environment should be merged into the meta-bootstrap layer anyways. Any configuration in build/conf/local.conf that is actually “local” to the current build host should not be part of the project repository. An example for such a “local” configuration might be the setting of Bitbake’s BB_NUMBER_THREADS [9] variable.
  • Everything that is generated during a build process is located in the directories bitbake or build. This allows me to ignore these directories easily in .gitignore and I can get rid of these directories without the fear of loosing work (with the mentioned exception of some configuration files in build/conf/.
  • By setting up the layers/meta-bootstrap layer, I am able to quickly set up a new build environment and trigger a build process within it.



Well, this is how I like to setup a project that uses Yocto. As mentioned before, I would love to hear your feedback about it (either by email or as a comment at the very bottom of this post).

Change Log

  • 2024-01-18:

    • Add a paragraph about storing the downloads and the sstate-cache directory outside the project directory.
    • Improve the very first paragraph a little bit.

  • 2024-01-19:

    • Change the order of some paragraphs (requirements moved up).
    • Improve wording in some places.
    • Add a few internal links to sections.

  • 2024-01-22:

    • Add the Conclusion section.
    • Improve the very first paragraph a little bit.
    • Improve the requirements a little bit.



Take care,
Andreas

References

  1. The Linux Foundation, “Yocto Project Reference Manual,” 11-Jan-2024. [Online]. Available at: https://docs.yoctoproject.org/ref-manual/index.html. [Accessed: 15-Jan-2024].
  2. The Linux Foundation, “Source Directory Structure,” 11-Jan-2024. [Online]. Available at: https://docs.yoctoproject.org/ref-manual/structure.html. [Accessed: 15-Jan-2024].
  3. Openembedded.org, “OpenEmbedded Layer Index.” [Online]. Available at: http://layers.openembedded.org/layerindex/branch/master/layers/. [Accessed: 15-Jan-2024].
  4. The Linux Foundation, “Yocto Project Development Tasks Manual,” 11-Jan-2024. [Online]. Available at: https://docs.yoctoproject.org/dev-manual/layers.html. [Accessed: 15-Jan-2024].
  5. The Linux Foundation, “Understanding and Creating Layers,” 11-Jan-2024. [Online]. Available at: https://docs.yoctoproject.org/dev-manual/layers.html. [Accessed: 15-Jan-2024].
  6. The Linux Foundation, “Creating a Custom Template Configuration Directory,” 11-Jan-2024. [Online]. Available at: https://docs.yoctoproject.org/dev-manual/custom-template-configuration-directory.html. [Accessed: 15-Jan-2024].
  7. The Linux Foundation, “The Yocto Project,” 2023. [Online]. Available at: https://www.yoctoproject.org/. [Accessed: 15-Jan-2024].
  8. The Linux Foundation, “What I wish I’d known about Yocto Project,” 11-Jan-2024. [Online]. Available at: https://docs.yoctoproject.org/what-i-wish-id-known.html. [Accessed: 15-Jan-2024].
  9. The Linux Foundation, “Variable Glossary – The Yocto Project,” 22-Jan-2024. [Online]. Available at: https://docs.yoctoproject.org/ref-manual/variables.html. [Accessed: 22-Jan-2024].

Updated:

Leave a comment