diff options
Diffstat (limited to 'docs/devel/build-system.rst')
| -rw-r--r-- | docs/devel/build-system.rst | 500 |
1 files changed, 500 insertions, 0 deletions
diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst new file mode 100644 index 0000000000..58bf392430 --- /dev/null +++ b/docs/devel/build-system.rst @@ -0,0 +1,500 @@ +================================== +The QEMU build system architecture +================================== + +This document aims to help developers understand the architecture of the +QEMU build system. As with projects using GNU autotools, the QEMU build +system has two stages, first the developer runs the "configure" script +to determine the local build environment characteristics, then they run +"make" to build the project. There is about where the similarities with +GNU autotools end, so try to forget what you know about them. + + +Stage 1: configure +================== + +The QEMU configure script is written directly in shell, and should be +compatible with any POSIX shell, hence it uses #!/bin/sh. An important +implication of this is that it is important to avoid using bash-isms on +development platforms where bash is the primary host. + +In contrast to autoconf scripts, QEMU's configure is expected to be +silent while it is checking for features. It will only display output +when an error occurs, or to show the final feature enablement summary +on completion. + +Because QEMU uses the Meson build system under the hood, only VPATH +builds are supported. There are two general ways to invoke configure & +perform a build: + + - VPATH, build artifacts outside of QEMU source tree entirely:: + + cd ../ + mkdir build + cd build + ../qemu/configure + make + + - VPATH, build artifacts in a subdir of QEMU source tree:: + + mkdir build + cd build + ../configure + make + +For now, checks on the compilation environment are found in configure +rather than meson.build, though this is expected to change. The command +line is parsed in the configure script and, whenever needed, converted +into the appropriate options to Meson. + +New checks should be added to Meson, which usually comprises the +following tasks: + + - Add a Meson build option to meson_options.txt. + + - Add support to the command line arg parser to handle any new + `--enable-XXX`/`--disable-XXX` flags required by the feature. + + - Add information to the help output message to report on the new + feature flag. + + - Add code to perform the actual feature check. + + - Add code to include the feature status in `config-host.h` + + - Add code to print out the feature status in the configure summary + upon completion. + + +Taking the probe for SDL as an example, we have the following pieces +in configure:: + + # Initial variable state + sdl=auto + + ..snip.. + + # Configure flag processing + --disable-gnutls) sdl=disabled + ;; + --enable-gnutls) sdl=enabled + ;; + + ..snip.. + + # Help output feature message + sdl SDL UI + + ..snip.. + + # Meson invocation + -Dsdl=$sdl + +In meson_options.txt:: + + option('sdl', type : 'feature', value : 'auto') + +In meson.build:: + + # Detect dependency + sdl = dependency('sdl2', + required: get_option('sdl'), + static: enable_static) + + # Create config-host.h + config_host_data.set('CONFIG_SDL', sdl.found()) + + # Summary + summary_info += {'SDL support': sdl.found()} + + + +Helper functions +---------------- + +The configure script provides a variety of helper functions to assist +developers in checking for system features: + +`do_cc $ARGS...` + Attempt to run the system C compiler passing it $ARGS... + +`do_cxx $ARGS...` + Attempt to run the system C++ compiler passing it $ARGS... + +`compile_object $CFLAGS` + Attempt to compile a test program with the system C compiler using + $CFLAGS. The test program must have been previously written to a file + called $TMPC. + +`compile_prog $CFLAGS $LDFLAGS` + Attempt to compile a test program with the system C compiler using + $CFLAGS and link it with the system linker using $LDFLAGS. The test + program must have been previously written to a file called $TMPC. + +`has $COMMAND` + Determine if $COMMAND exists in the current environment, either as a + shell builtin, or executable binary, returning 0 on success. + +`path_of $COMMAND` + Return the fully qualified path of $COMMAND, printing it to stdout, + and returning 0 on success. + +`check_define $NAME` + Determine if the macro $NAME is defined by the system C compiler + +`check_include $NAME` + Determine if the include $NAME file is available to the system C + compiler + +`write_c_skeleton` + Write a minimal C program main() function to the temporary file + indicated by $TMPC + +`feature_not_found $NAME $REMEDY` + Print a message to stderr that the feature $NAME was not available + on the system, suggesting the user try $REMEDY to address the + problem. + +`error_exit $MESSAGE $MORE...` + Print $MESSAGE to stderr, followed by $MORE... and then exit from the + configure script with non-zero status + +`query_pkg_config $ARGS...` + Run pkg-config passing it $ARGS. If QEMU is doing a static build, + then --static will be automatically added to $ARGS + + +Stage 2: Meson +============== + +The Meson build system is currently used to describe the build +process for: + +1) executables, which include: + + - Tools - qemu-img, qemu-nbd, qga (guest agent), etc + + - System emulators - qemu-system-$ARCH + + - Userspace emulators - qemu-$ARCH + + - Some (but not all) unit tests + +2) documentation + +3) ROMs, which can be either installed as binary blobs or compiled + +4) other data files, such as icons or desktop files + +The source code is highly modularized, split across many files to +facilitate building of all of these components with as little duplicated +compilation as possible. The Meson "sourceset" functionality is used +to list the files and their dependency on various configuration +symbols. + +Various subsystems that are common to both tools and emulators have +their own sourceset, for example `block_ss` for the block device subsystem, +`chardev_ss` for the character device subsystem, etc. These sourcesets +are then turned into static libraries as follows:: + + libchardev = static_library('chardev', chardev_ss.sources(), + name_suffix: 'fa', + build_by_default: false) + + chardev = declare_dependency(link_whole: libchardev) + +The special `.fa` suffix is needed as long as unit tests are built with +the older Makefile infrastructure, and will go away later. + +Files linked into emulator targets there can be split into two distinct groups +of files, those which are independent of the QEMU emulation target and +those which are dependent on the QEMU emulation target. + +In the target-independent set lives various general purpose helper code, +such as error handling infrastructure, standard data structures, +platform portability wrapper functions, etc. This code can be compiled +once only and the .o files linked into all output binaries. +Target-independent code lives in the `common_ss`, `softmmu_ss` and +`user_ss` sourcesets. `common_ss` is linked into all emulators, `softmmu_ss` +only in system emulators, `user_ss` only in user-mode emulators. + +In the target-dependent set lives CPU emulation, device emulation and +much glue code. This sometimes also has to be compiled multiple times, +once for each target being built. + +All binaries link with a static library `libqemuutil.a`, which is then +linked to all the binaries. `libqemuutil.a` is built from several +sourcesets; most of them however host generated code, and the only two +of general interest are `util_ss` and `stub_ss`. + +The separation between these two is purely for documentation purposes. +`util_ss` contains generic utility files. Even though this code is only +linked in some binaries, sometimes it requires hooks only in some of +these and depend on other functions that are not fully implemented by +all QEMU binaries. `stub_ss` links dummy stubs that will only be linked +into the binary if the real implementation is not present. In a way, +the stubs can be thought of as a portable implementation of the weak +symbols concept. + +The following files concur in the definition of which files are linked +into each emulator: + +`default-configs/*.mak` + The files under default-configs/ control what emulated hardware is built + into each QEMU system and userspace emulator targets. They merely contain + a list of config variable definitions like the machines that should be + included. For example, default-configs/aarch64-softmmu.mak has:: + + include arm-softmmu.mak + CONFIG_XLNX_ZYNQMP_ARM=y + CONFIG_XLNX_VERSAL=y + +`*/Kconfig` + These files are processed together with `default-configs/*.mak` and + describe the dependencies between various features, subsystems and + device models. They are described in kconfig.rst. + +These files rarely need changing unless new devices / hardware need to +be enabled for a particular system/userspace emulation target + + +Support scripts +--------------- + +Meson has a special convention for invoking Python scripts: if their +first line is `#! /usr/bin/env python3` and the file is *not* executable, +find_program() arranges to invoke the script under the same Python +interpreter that was used to invoke Meson. This is the most common +and preferred way to invoke support scripts from Meson build files, +because it automatically uses the value of configure's --python= option. + +In case the script is not written in Python, use a `#! /usr/bin/env ...` +line and make the script executable. + +Scripts written in Python, where it is desirable to make the script +executable (for example for test scripts that developers may want to +invoke from the command line, such as tests/qapi-schema/test-qapi.py), +should be invoked through the `python` variable in meson.build. For +example:: + + test('QAPI schema regression tests', python, + args: files('test-qapi.py'), + env: test_env, suite: ['qapi-schema', 'qapi-frontend']) + +This is needed to obey the --python= option passed to the configure +script, which may point to something other than the first python3 +binary on the path. + + +Stage 3: makefiles +================== + +The use of GNU make is required with the QEMU build system. + +The output of Meson is a build.ninja file, which is used with the Ninja +build system. QEMU uses a different approach, where Makefile rules are +synthesized from the build.ninja file. The main Makefile includes these +rules and wraps them so that e.g. submodules are built before QEMU. +The resulting build system is largely non-recursive in nature, in +contrast to common practices seen with automake. + +Tests are also ran by the Makefile with the traditional `make check` +phony target. Meson test suites such as `unit` can be ran with `make +check-unit` too. It is also possible to run tests defined in meson.build +with `meson test`. + +The following text is only relevant for unit tests which still have to +be converted to Meson. + +All binaries should link to `libqemuutil.a`, e.g.: + + qemu-img$(EXESUF): qemu-img.o ..snip.. libqemuutil.a + +On Windows, all binaries have the suffix `.exe`, so all Makefile rules +which create binaries must include the $(EXESUF) variable on the binary +name. e.g. + + qemu-img$(EXESUF): qemu-img.o ..snip.. + +This expands to `.exe` on Windows, or an empty string on other platforms. + +Variable naming +--------------- + +The QEMU convention is to define variables to list different groups of +object files. These are named with the convention $PREFIX-obj-y. The +Meson `chardev` variable in the previous example corresponds to a +variable 'chardev-obj-y'. + +Likewise, tests that are executed by `make check-unit` are grouped into +a variable check-unit-y, like this: + + check-unit-y += tests/test-visitor-serialization$(EXESUF) + check-unit-y += tests/test-iov$(EXESUF) + check-unit-y += tests/test-bitmap$(EXESUF) + +When a test or object file which needs to be conditionally built based +on some characteristic of the host system, the configure script will +define a variable for the conditional. For example, on Windows it will +define $(CONFIG_POSIX) with a value of 'n' and $(CONFIG_WIN32) with a +value of 'y'. It is now possible to use the config variables when +listing object files. For example, + + check-unit-$(CONFIG_POSIX) += tests/test-vmstate$(EXESUF) + +On Windows this expands to + + check-unit-n += tests/vmstate.exe + +Since the `check-unit` target only runs tests included in `$(check-unit-y)`, +POSIX specific tests listed in `$(util-obj-n)` are ignored on the Windows +platform builds. + + +CFLAGS / LDFLAGS / LIBS handling +-------------------------------- + +There are many different binaries being built with differing purposes, +and some of them might even be 3rd party libraries pulled in via git +submodules. As such the use of the global CFLAGS variable is generally +avoided in QEMU, since it would apply to too many build targets. + +Flags that are needed by any QEMU code (i.e. everything *except* GIT +submodule projects) are put in $(QEMU_CFLAGS) variable. For linker +flags the $(LIBS) variable is sometimes used, but a couple of more +targeted variables are preferred. + +In addition to these variables, it is possible to provide cflags and +libs against individual source code files, by defining variables of the +form $FILENAME-cflags and $FILENAME-libs. For example, the test +test-crypto-tlscredsx509 needs to link to the libtasn1 library, +so tests/Makefile.include defines some variables: + + tests/crypto-tls-x509-helpers.o-cflags := $(TASN1_CFLAGS) + tests/crypto-tls-x509-helpers.o-libs := $(TASN1_LIBS) + +The scope is a little different between the two variables. The libs get +used when linking any target binary that includes the curl.o object +file, while the cflags get used when compiling the curl.c file only. + + +Important files for the build system +==================================== + +Statically defined files +------------------------ + +The following key files are statically defined in the source tree, with +the rules needed to build QEMU. Their behaviour is influenced by a +number of dynamically created files listed later. + +`Makefile` + The main entry point used when invoking make to build all the components + of QEMU. The default 'all' target will naturally result in the build of + every component. Makefile takes care of recursively building submodules + directly via a non-recursive set of rules. + +`*/meson.build` + The meson.build file in the root directory is the main entry point for the + Meson build system, and it coordinates the configuration and build of all + executables. Build rules for various subdirectories are included in + other meson.build files spread throughout the QEMU source tree. + +`rules.mak` + This file provides the generic helper rules for invoking build tools, in + particular the compiler and linker. + +`tests/Makefile.include` + Rules for building the unit tests. This file is included directly by the + top level Makefile, so anything defined in this file will influence the + entire build system. Care needs to be taken when writing rules for tests + to ensure they only apply to the unit test execution / build. + +`tests/docker/Makefile.include` + Rules for Docker tests. Like tests/Makefile, this file is included + directly by the top level Makefile, anything defined in this file will + influence the entire build system. + +`tests/vm/Makefile.include` + Rules for VM-based tests. Like tests/Makefile, this file is included + directly by the top level Makefile, anything defined in this file will + influence the entire build system. + +Dynamically created files +------------------------- + +The following files are generated dynamically by configure in order to +control the behaviour of the statically defined makefiles. This avoids +the need for QEMU makefiles to go through any pre-processing as seen +with autotools, where Makefile.am generates Makefile.in which generates +Makefile. + +Built by configure: + +`config-host.mak` + When configure has determined the characteristics of the build host it + will write a long list of variables to config-host.mak file. This + provides the various install directories, compiler / linker flags and a + variety of `CONFIG_*` variables related to optionally enabled features. + This is imported by the top level Makefile and meson.build in order to + tailor the build output. + + config-host.mak is also used as a dependency checking mechanism. If make + sees that the modification timestamp on configure is newer than that on + config-host.mak, then configure will be re-run. + + The variables defined here are those which are applicable to all QEMU + build outputs. Variables which are potentially different for each + emulator target are defined by the next file... + +`$TARGET-NAME/config-target.mak` + TARGET-NAME is the name of a system or userspace emulator, for example, + x86_64-softmmu denotes the system emulator for the x86_64 architecture. + This file contains the variables which need to vary on a per-target + basis. For example, it will indicate whether KVM or Xen are enabled for + the target and any other potential custom libraries needed for linking + the target. + + +Built by Meson: + +`${TARGET-NAME}-config-devices.mak` + TARGET-NAME is again the name of a system or userspace emulator. The + config-devices.mak file is automatically generated by make using the + scripts/make_device_config.sh program, feeding it the + default-configs/$TARGET-NAME file as input. + +`config-host.h`, `$TARGET-NAME/config-target.h`, `$TARGET-NAME/config-devices.h` + These files are used by source code to determine what features + are enabled. They are generated from the contents of the corresponding + `*.h` files using the scripts/create_config program. This extracts + relevant variables and formats them as C preprocessor macros. + +`build.ninja` + The build rules. + + +Built by Makefile: + +`Makefile.ninja` + A Makefile conversion of the build rules in build.ninja. The conversion + is straightforward and, were it necessary to debug the rules produced + by Meson, it should be enough to look at build.ninja. The conversion + is performed by scripts/ninjatool.py. + +`Makefile.mtest` + The Makefile definitions that let "make check" run tests defined in + meson.build. The rules are produced from Meson's JSON description of + tests (obtained with "meson introspect --tests") through the script + scripts/mtest2make.py. + + +Useful make targets +------------------- + +`help` + Print a help message for the most common build targets. + +`print-VAR` + Print the value of the variable VAR. Useful for debugging the build + system. |