====== Compiling Individual Packages ======

You may know me from former titles such as "Batocera Linux Buildroot Modifications" and "How to work on Batocera (and not recompile everything)".

<WRAP center round info>
RPi3, the "per platform" example used here, was formerly known internally as ''rpi3'' in Batocera **v35** and lower. This is now ''bcm2837'' in **v36** and higher.
</WRAP>

<WRAP center round tip>
If you'd instead like to only test [[:notable_files|certain config files]] (like es_features.cfg), you can edit them directly in the live system and save them [[:modify_the_system_while_it_s_running|via overlays]].
</WRAP>

[[:compile_batocera.linux|Compiling the entirety of Batocera]] is optional, but recommended as it automatically sets up your enviroment for compiling individual packages. batocera.linux is based on Buildroot, [[https://buildroot.org/downloads/manual/manual.html|manual available here]].

===== Package compilation =====

For each package (ie. **kodi**, **emulationstation**), Buildroot does the following:
  - do prerequisites
  - download
  - extract
  - patch
  - configure
  - build
  - install staging
  - install target

More exactly:
  - do the prerequisite packages recursively if not yet done (ie **sdl2** if you want to build **emulationstation**)
  - download the package in the dl directory according to information from ''package/package-name/package-name.mk'' or ''package/batocera/package-name/package-name.mk''
  - extract the package in ''output/build/package-name-version''
  - patch the package from ''*.patch'' files (files next to the ''.mk'' file or ''board/batocera/patches/'' or ''board/batocera/bcm2837/patches/'')
  - configure the package according to the ''.mk'' file
  - build the package according to the ''.mk'' file
  - install in the ''output/bcm2837/host/'' staging directory, in case of libraries
  - install in the ''output/bcm2837/target/'' directory

For each step, Buildroot creates an empty flag file. For example: ''%%.stamp_downloaded%%'' in ''output/bcm2837/build/package-name-version''.

In case a step failed, just run ''make'' or ''make package-name'' again to restart/continue the build where you left.

To completely rebuild a package (for example an RPi 3 package):

  rm -rf output/bcm2837/build/package-name-version
  make package-name

For example, to build EmulationStation again from the latest ''master'' git version:

  rm -rf output/bcm2837/build/batocera-emulationstation-master
  make batocera-emulationstation
  
Individual packages can also be compiled using the container using:

<code bash>
make x86_64-pkg PKG=batocera-splash
</code>

replacing the components as necessary.

<WRAP center round important>
Although this will build package itself, it does not do so in the same way that the regular ''make x86_64-build'' does. Using a package build cannot be used for testing a package's successful build, always use ''-build'' or ''-cleanbuild'' to confirm before making a pull request.
</WRAP>

===== Main directories/files =====

When you clone the batocera.linux git repository, you mainly get the following directories:

  * ''.config'': the list of all packages, stating if you are going to build it or not.
  * ''dl'': the directory where all files are downloaded, for example, ''batocera-emulationstation-master.tar.gz''
  * ''output'': the directory where everything is built.

In other words, in order to reinitialize your build, you can type:

  rm -rf .config dl output

I personally use the following command to confirm I have no other remaining files:

  git status --ignored

My ''dl'' directory is a link on ''../DL'' in order to use the same download directory for all my builds, to avoid to download the source archives each time again and again. Note that you may have to remove some packages like ''batocera-emulationstation-master.tar.gz'' or ''batocera-configgen-master.tar.gz'' while the contains is the master branch of their respective git repositories. In other words, whenever the content is dynamic (''master'') and not a precise GitHub version. If you commit something on ''configgen'', while the file is in the ''dl'' directory, buildroot will not download it again as it is seen as the same filename.

===== Working on EmulationStation =====

EmulationStation is a [[https://github.com/batocera-linux/batocera-emulationstation|separate git repository]] from Batocera system. To work on EmulationStation, the simplest way is to directly get it on your computer and compile it without buildroot.

===== How to add an individual package ====

This is for generic packages that aren't necessarily emulators.

<WRAP center round tip>
The following is a (very) brief summarization of [[https://buildroot.org/downloads/manual/manual.html#adding-packages|chapter 19 of the Buildroot manual]]. It is recommended to read that in full in case of any questions.
</WRAP>

<WRAP center round todo>
Todo:
  * Explain the differences between each different kind of package container (generic, cmake, autotools, etc.).
  * Provide rough syntax for each one.
  * Cover all stages.
  * Define when certain steps aren't absolutely necessary.
</WRAP>

  - Add the package's folder to the appropriate location. For generic packages like this, it will most likely be ''package/batocera/utils/package-name/''. <WRAP center round tip>
It's worth checking if the package you want is already included in buildroot, just check in the ''buildroot/package/Config.in'' file. If so, skip steps 2 and 3.
</WRAP>
  - In that folder, create the ''Config.in'' file. This should contain the package's "metadata" for compilation purposes. Add into this file its:
    - Buildroot package header: ''%%config BR2_PACKAGE_<package>%%''
    - Its bool shortname used for compilation and other packages it may depend on: ''%%bool "<package shortname>"%%''
    - Specify other packages depends on: ''%%depends on BR2_PACKAGE_<other package>%%''
    - A help section describing the package in more detail. <WRAP center round info>
For more information on Buildroot's ''Config.in'' syntax, refer to [[https://buildroot.org/downloads/manual/manual.html#writing-rules-config-in|chapter 16.1 in Buildroot's manual]].
</WRAP>
  - Create the makefile. Its filename should be the same as the package's bool shortname, followed by the ''.mk'' extension: ''<package-shortname>.mk''
    - If pulling from a Github source, use the following syntax: <code bash><package>_VERSION = <commit hash>
<package>_SITE = $(call github,<author>,<repository>,$(<package>_VERSION))</code>
    - Each [[#package_compilation|step of package compilation]] needs to be put into its appropriate function.
      - If needing to set particular config options while building, place them in ''<package>_CONF += <options>''. For example: <code bash>GZDOOM_CONF_OPTS += -DCMAKE_BUILD_TYPE=Release -DDYN_GTK=OFF -DDYN_OPENAL=OFF
</code>
      - If needing to set particular environmental variables before building, place them in ''<package>_CONF_ENV += <environment variables>''. For example: <code bash>GZDOOM_CONF_ENV += ZMUSIC_LIBRARIES="/x86_64/target/usr/lib/" ZMUSIC_INCLUDE_DIR="/x86_64/target/usr/lib/cmake/ZMusic/"
</code>
      - If compiling from the package's downloaded build folder (such as when downloading a Github repo), put this in ''define <package>_BUILD_CMDS''. Inside of that function, be sure to use the appropriate flag to set the correct working directory: ''%%$(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D)%%'' The "make" compiler uses its own instance, which may not be sharing the same current working directory. A generic compilation script would be as follows: <code bash>define <package>_BUILD_CMDS
    $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D)
endef</code>
      - When compilation is finished, if not compiled as a part of a large over-arching package, copy over the resulting files/binaries to the appropriate locations in the install step (if the compilation process itself doesn't already do so). For example, when compiling an ordinary binary: <code bash>define <package>_INSTALL_TARGET_CMDS
    mkdir -p $(TARGET_DIR)/usr/bin
    cp $(@D)/output_executable $(TARGET_DIR)/usr/bin
endef</code>
    - When done defining actions to perform for each step, end the makefile with one file line defining which package container to use. For "generic" packages: ''%%$(eval $(generic-package))%%''.<WRAP center round info>
For more information on Buildroot's makefile syntax, refer to [[https://buildroot.org/downloads/manual/manual.html#writing-rules-mk|chapter 16.2 in Buildroot's manual]].
</WRAP>
  - Add the package to the list of compiled packages at ''package/batocera/core/batocera-system/Config.in''. This is where you would refer to its Buildroot package header, not its shortname.
  - Add the package to the list of sources in the appropriate section in ''Config.in''. That's right, the file in the root directory of the repo.

An example of a recent pull request that adds a simple package: https://github.com/batocera-linux/batocera.linux/pull/5812/files

===== How to add an emulator =====

  - Add the initial scripts for successful compilation. Ensure that there are no valid build errors with it. Refer to [[https://buildroot.org/downloads/manual/manual.html#adding-packages|buildroot's documentation]] for further info.
    - Create its initial configuration script: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/emulators/<new emulator>/Config.in
    - Call it from https://github.com/batocera-linux/batocera.linux/blob/master/Config.in
    - Create its makefile for package compilation: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/emulators/<new emulator>/<new emulator>.mk
    - If the emulator requires certain files in certain locations in userdata, copy them to ''/usr/share/batocera/data/''.
    - If the emulator cannot bind functions to the gamepad's buttons and needs to use a virtual keyboard, add the appropriate [[https://github.com/kempniu/evmapy|evmapy]] ''.keys'' file to ''<shortname>.<emulator>.keys''.
    - If building from source, add the git repository check to the update script: https://github.com/batocera-linux/batocera.linux/blob/master/scripts/linux/checkPackagesUpdates.sh
  - Add your BR2 package to the compilation process (and limit your emulator to particular archs (typically, complex standalone emulators are x86/x86_64 only)) in https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-system/Config.in.
    * If making a new category of systems, be sure to add it to the included ''%%config BR2_PACKAGE_BATOCERA_ALL_SYSTEMS%%'' under the ''#### systems ####'' section.
  - Add your system/emulator to the EmulationStation system configuration at https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/emulationstation/batocera-es-system/es_systems.yml (there needs to be at least one default core for the system, described later).
    * You can check a list of system shortnames that can be automatically scraped for here: https://github.com/batocera-linux/batocera-emulationstation/blob/master/es-app/src/PlatformId.cpp
  - Add your system/emulator to the features list (https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/emulationstation/batocera-es-system/es_features.yml) and any advanced system settings if necessary.
  - If BIOS files are required, set them in https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-scripts/scripts/batocera-systems
  - If the emulator has a desktop configuration application, add it to the Applications menu in the file manager:
    - Add the desktop entry (shortcut metadata) at [[https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-desktopapps/apps]].
    - Add its launch commands to a script in [[https://github.com/batocera-linux/batocera.linux/tree/master/package/batocera/core/batocera-desktopapps/scripts]].
    - Add its icon in [[https://github.com/batocera-linux/batocera.linux/tree/master/package/batocera/core/batocera-desktopapps/icons]].
    - Add it to the list of desktop entries to be compiled at [[https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-desktopapps/batocera-desktopapps.mk]].
  - Create and test its config generator at https://github.com/batocera-linux/batocera.linux/tree/master/package/batocera/core/batocera-configgen/configgen/configgen/generators.
    - Define the generator with its system shortname here: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configgen/configgen/GeneratorImporter.py (you can test this locally with ''/usr/lib/python#.#/site-packages/configgen/Generator Importer.py'')
      * Syntax: <code python>
if emulator == '<emulator name>'
    from generators.<emulator name>.<emulator name>Generator import <CamelCased emulator name>Generator'')
</code> <WRAP center round info>
In Batocera **v34** and earlier, this process is instead:
    * Define the generator (syntax: ''from generators.<shortname>.<shortname>Generator import <CamelCased shortname>Generator'', shortname is the [[:systems|system name]]) with its system shortname here: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configgen/configgen/emulatorlauncher.py (you can test this locally with ''/usr/lib/python#.#/site-packages/configgen/emulatorlauncher.py''). The ''commandArray'' is ultimately the line that is used to run the emulator from bash.
</WRAP>
    - Include it in the packages list (syntax: ''configgen.generators.<shortname>'') here: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configgen/setup.py.
    - Create the "main" configuration generator Python script here: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configgen/configgen/generators/ followed by <shortname>/<shortname>Generator.py.
    - If appropriate, split the file into multiple Python scripts called by <shortname>Generator.py in the ''generators/<shortname>/'' folder.
    - Call the files if required here: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configgen/configgen/batoceraFiles.py.
  - Configure the default emulator (if you've added a whole new system) with https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configs/configgen-defaults.yml (if you're merely adding an emulator to an already existing system and you don't want to make it the default, no need to touch this file).
  - Configure the default settings for particular architectures (such as if your emulator requires certain settings to function on a particular architecture) at https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configs/.
  - Add the system's shortname and a link to a page explaining the system (Wikipedia tends to be a good "default" but there are better sources out there) to the appropriate section in the [[:systems|systems page]] of this wiki.
  - (Optional) If you need to add an explanation for the emulator's exclusion from a particular platform for the [[https://batocera.org/compatibility.php|compatibility chart]], add it to https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/emulationstation/batocera-es-system/systems-explanations.yml

If you'd like an example of a recent pull request that adds a whole new emulator: https://github.com/batocera-linux/batocera.linux/pull/4742/files

===== Patching a package =====

Buildroot features the ability to patch the source code of a package before compiling it, read [[https://buildroot.org/downloads/manual/manual.html#patch-policy|section 19.2 in the buildroot user manual]] for more info. Here we will do a quick rundown.

Let's say you want to work on the package "retroarch" to make it evolve via a patch.

First, compile a complete version of Batocera for the architecture you will test. In this example, we will do so for RPi3.

<code bash>
cd batocera.linux
make bcm2837-build
</code>

Once this is done, you don't need to build the entirety of it again for your tests. You can just rename/remove the output's ''target'' folder for that architecture, like so:

<code bash>
mv output/bcm2837/target output/bcm2837/target_
</code>

Now for our patch. First change the working directory to that of the //build// folder for the respective package:

<code bash>
cd output/bcm2837/build/retroarch-version
</code>
  
Establish the package state using ''git'':

<code bash>
git init
</code>

Then add the file(s) you want to modify to the current commit:

<code bash>
git add <the file(s) you modified>
</code>

<WRAP center round tip>
Add all files with ''git add -A''.
</WRAP>

Make the modifications to the files you want now.

Then compare the differences and save them to a "patch" file:

<code bash>
git diff > ../../package/batocera/retroarch/001-mymodification.patch
</code>

Now clear out the package's respective ''build'' folder to make way for testing a new build with it:

<code bash>
cd ../../..
rm -rf ouput/bcm2837/build/batocera/retroarch-version
</code>

Then build the package (replacing ''make'' if using a different make tool):

<code bash>
make retroarch-patch # to confirm the patch is applied
make retroarch # to finish other steps
</code>

===== Testing a package without compiling the entirety of Batocera =====

It's possible to compile //just// the package you've modified and then transfer it over to your live Batocera installation using ''rsync''. First, compile the individual package [[#package_compilation|as explained above]] with ''make bcm2837-pkg PKG=<package name>'' and then run:

<code bash>
rsync -av output/bcm2837/target/ root@batocera.local:/
</code>

This will sync up the contents of the target directory on your build machine with the Batocera machine. Feel free to immediately test your changes. If the changes are bad and you need to restore, simply reboot and the changes will not persist. To have them persist instead, run [[modify_the_system_while_it_s_running|batocera-save-overlay]] from the machine itself.