Differences

This shows you the differences between two versions of the page.

--- batocera.linux_buildroot_modifications [2022/03/31 09:28] – [How to add an emulator] you actually do need to edit es_systems, even if all you do is add an additional emulator atari
+++ batocera.linux_buildroot_modifications [2022/09/28 07:14] (current) – specifics atari
@@ Line 2: / Line 2: @@
 You may know me from former titles such as "Batocera Linux Buildroot Modifications" and "How to work on Batocera (and not recompile everything)".
+<WRAP center round info>
+RPi3, the "per platform" example used here, was formerly known internally as ''rpi3'' in Batocera **v35** and lower. This is now ''bcm2837'' in **v36** and higher.
+</WRAP>
 <WRAP center round tip>
@@ Line 25: / Line 29: @@
   - download the package in the dl directory according to information from ''package/package-name/package-name.mk'' or ''package/batocera/package-name/package-name.mk''
   - extract the package in ''output/build/package-name-version''
-  - patch the package from ''*.patch'' files (files next to the ''.mk'' file or ''board/batocera/patches/'' or ''board/batocera/rpi/patches/'')
+  - patch the package from ''*.patch'' files (files next to the ''.mk'' file or ''board/batocera/patches/'' or ''board/batocera/bcm2837/patches/'')
   - configure the package according to the ''.mk'' file
   - build the package according to the ''.mk'' file
-  - install in the ''output/rpi3/host/'' staging directory, in case of libraries
+  - install in the ''output/bcm2837/host/'' staging directory, in case of libraries
-  - install in the ''output/rpi3/target/'' directory
+  - install in the ''output/bcm2837/target/'' directory
-For each step, Buildroot creates an empty flag file. For example: ''%%.stamp_downloaded%%'' in ''output/rpi3/build/package-name-version''.
+For each step, Buildroot creates an empty flag file. For example: ''%%.stamp_downloaded%%'' in ''output/bcm2837/build/package-name-version''.
 In case a step failed, just run ''make'' or ''make package-name'' again to restart/continue the build where you left.
-To completely rebuild a package:
+To completely rebuild a package (for example an RPi 3 package):
-  rm -rf output/rpi3/build/package-name-version
+  rm -rf output/bcm2837/build/package-name-version
   make package-name
 For example, to build EmulationStation again from the latest ''master'' git version:
-  rm -rf output/rpi3/build/batocera-emulationstation-master
+  rm -rf output/bcm2837/build/batocera-emulationstation-master
   make batocera-emulationstation
@@ Line 52: / Line 56: @@
 replacing the components as necessary.
+<WRAP center round important>
+Although this will build package itself, it does not do so in the same way that the regular ''make x86_64-build'' does. Using a package build cannot be used for testing a package's successful build, always use ''-build'' or ''-cleanbuild'' to confirm before making a pull request.
+</WRAP>
 ===== Main directories/files =====
@@ Line 70: / Line 78: @@
 My ''dl'' directory is a link on ''../DL'' in order to use the same download directory for all my builds, to avoid to download the source archives each time again and again. Note that you may have to remove some packages like ''batocera-emulationstation-master.tar.gz'' or ''batocera-configgen-master.tar.gz'' while the contains is the master branch of their respective git repositories. In other words, whenever the content is dynamic (''master'') and not a precise GitHub version. If you commit something on ''configgen'', while the file is in the ''dl'' directory, buildroot will not download it again as it is seen as the same filename.
-===== Working on packages =====
-Let's say you want to work on the package "retroarch" to make it evolve via a patch.
-First, compile a version of Batocera for the architecture you will test.
-Once this is done, you don't need to build it for your tests again. Just run the following commands:
-  mv output/rpi3/target output/rpi3/target_
-  cd output/rpi3/build/retroarch-version
-  git init
-  git add <the file(s) you want to modify>
-  # edit the file(s)
-  git diff > ../../package/batocera/retroarch/001-mymodification.patch
-  cd ../../..
-  rm -rf ouput/rpi3/build/batocera/retroarch-version
-  make retroarch-patch # to confirm the patch is applied
-  make retroarch # to finish other steps
-At this step, output/target contains only the retroarch package. You can install it on your target machine via the command:
-  rsync -av output/target/ root@batocera.local:/
-Remember, Batocera system modification are done on the Linux overlay, i.e. in RAM. Rebooting Batocera without saving your overlay acts as if you hadn't changed it -- in other words, it's a nice way to make tests while your system is not modified, but those changes are not persistent. If you want to permanently save the modification and keep them upon reboot, you need to use [[modify_the_system_while_it_s_running|batocera-save-overlay]] from the machine itself.
 ===== Working on EmulationStation =====
@@ Line 116: / Line 99: @@
 </WRAP>
-  - Add the package's folder to the appropriate location. For generic packages like this, it will most likely be ''package/batocera/utils/package-name/''.
+  - Add the package's folder to the appropriate location. For generic packages like this, it will most likely be ''package/batocera/utils/package-name/''. <WRAP center round tip>
+It's worth checking if the package you want is already included in buildroot, just check in the ''buildroot/package/Config.in'' file. If so, skip steps 2 and 3.
+</WRAP>
   - In that folder, create the ''Config.in'' file. This should contain the package's "metadata" for compilation purposes. Add into this file its:
     - Buildroot package header: ''%%config BR2_PACKAGE_<package>%%''
@@ Line 128: / Line 113: @@
 <package>_SITE = $(call github,<author>,<repository>,$(<package>_VERSION))</code>
     - Each [[#package_compilation|step of package compilation]] needs to be put into its appropriate function.
+      - If needing to set particular config options while building, place them in ''<package>_CONF += <options>''. For example: <code bash>GZDOOM_CONF_OPTS += -DCMAKE_BUILD_TYPE=Release -DDYN_GTK=OFF -DDYN_OPENAL=OFF
+</code>
+      - If needing to set particular environmental variables before building, place them in ''<package>_CONF_ENV += <environment variables>''. For example: <code bash>GZDOOM_CONF_ENV += ZMUSIC_LIBRARIES="/x86_64/target/usr/lib/" ZMUSIC_INCLUDE_DIR="/x86_64/target/usr/lib/cmake/ZMusic/"
+</code>
       - If compiling from the package's downloaded build folder (such as when downloading a Github repo), put this in ''define <package>_BUILD_CMDS''. Inside of that function, be sure to use the appropriate flag to set the correct working directory: ''%%$(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D)%%'' The "make" compiler uses its own instance, which may not be sharing the same current working directory. A generic compilation script would be as follows: <code bash>define <package>_BUILD_CMDS
     $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D)
 endef</code>
-      - When compilation is finished, copy over the resulting files/binaries to the appropriate locations in the install step (if the compilation process itself doesn't already do so). For example, when compiling an ordinary binary: <code bash>define <package>_INSTALL_TARGET_CMDS
+      - When compilation is finished, if not compiled as a part of a large over-arching package, copy over the resulting files/binaries to the appropriate locations in the install step (if the compilation process itself doesn't already do so). For example, when compiling an ordinary binary: <code bash>define <package>_INSTALL_TARGET_CMDS
     mkdir -p $(TARGET_DIR)/usr/bin
     cp $(@D)/output_executable $(TARGET_DIR)/usr/bin
@@ Line 150: / Line 139: @@
     - Create its makefile for package compilation: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/emulators/<new emulator>/<new emulator>.mk
     - If the emulator requires certain files in certain locations in userdata, copy them to ''/usr/share/batocera/data/''.
+    - If the emulator cannot bind functions to the gamepad's buttons and needs to use a virtual keyboard, add the appropriate [[https://github.com/kempniu/evmapy|evmapy]] ''.keys'' file to ''<shortname>.<emulator>.keys''.
     - If building from source, add the git repository check to the update script: https://github.com/batocera-linux/batocera.linux/blob/master/scripts/linux/checkPackagesUpdates.sh
-  - Add your system/emulator to the EmulationStation system configuration at https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/emulationstation/batocera-es-system/es_systems.yml (there needs to be at least one ''default="true"'' core, even if that's not necessarily applied here).
+  - Add your BR2 package to the compilation process (and limit your emulator to particular archs (typically, complex standalone emulators are x86/x86_64 only)) in https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-system/Config.in.
+    * If making a new category of systems, be sure to add it to the included ''%%config BR2_PACKAGE_BATOCERA_ALL_SYSTEMS%%'' under the ''#### systems ####'' section.
+  - Add your system/emulator to the EmulationStation system configuration at https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/emulationstation/batocera-es-system/es_systems.yml (there needs to be at least one default core for the system, described later).
     * You can check a list of system shortnames that can be automatically scraped for here: https://github.com/batocera-linux/batocera-emulationstation/blob/master/es-app/src/PlatformId.cpp
   - Add your system/emulator to the features list (https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/emulationstation/batocera-es-system/es_features.yml) and any advanced system settings if necessary.
+  - If BIOS files are required, set them in https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-scripts/scripts/batocera-systems
+  - If the emulator has a desktop configuration application, add it to the Applications menu in the file manager:
+    - Add the desktop entry (shortcut metadata) at [[https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-desktopapps/apps]].
+    - Add its launch commands to a script in [[https://github.com/batocera-linux/batocera.linux/tree/master/package/batocera/core/batocera-desktopapps/scripts]].
+    - Add its icon in [[https://github.com/batocera-linux/batocera.linux/tree/master/package/batocera/core/batocera-desktopapps/icons]].
+    - Add it to the list of desktop entries to be compiled at [[https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-desktopapps/batocera-desktopapps.mk]].
   - Create and test its config generator at https://github.com/batocera-linux/batocera.linux/tree/master/package/batocera/core/batocera-configgen/configgen/configgen/generators.
-    - Define the generator (syntax: ''from generators.<shortname>.<shortname>Generator import <CamelCased shortname>Generator'', shortname is the [[:systems|system name]]) with its system shortname here: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configgen/configgen/emulatorlauncher.py (you can test this locally with ''/usr/lib/python3.9/site-packages/configgen/emulatorlauncher.py''). The ''commandArray'' is ultimately the line that is used to run the emulator from bash.
+    - Define the generator with its system shortname here: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configgen/configgen/GeneratorImporter.py (you can test this locally with ''/usr/lib/python#.#/site-packages/configgen/Generator Importer.py'')
+      * Syntax: <code python>
+if emulator == '<emulator name>'
+    from generators.<emulator name>.<emulator name>Generator import <CamelCased emulator name>Generator'')
+</code> <WRAP center round info>
+In Batocera **v34** and earlier, this process is instead:
+    * Define the generator (syntax: ''from generators.<shortname>.<shortname>Generator import <CamelCased shortname>Generator'', shortname is the [[:systems|system name]]) with its system shortname here: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configgen/configgen/emulatorlauncher.py (you can test this locally with ''/usr/lib/python#.#/site-packages/configgen/emulatorlauncher.py''). The ''commandArray'' is ultimately the line that is used to run the emulator from bash.
+</WRAP>
     - Include it in the packages list (syntax: ''configgen.generators.<shortname>'') here: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configgen/setup.py.
     - Create the "main" configuration generator Python script here: https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configgen/configgen/generators/ followed by <shortname>/<shortname>Generator.py.
@@ Line 162: / Line 167: @@
   - Configure the default emulator (if you've added a whole new system) with https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configs/configgen-defaults.yml (if you're merely adding an emulator to an already existing system and you don't want to make it the default, no need to touch this file).
   - Configure the default settings for particular architectures (such as if your emulator requires certain settings to function on a particular architecture) at https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-configgen/configs/.
-  - Limit your emulator to particular platforms (typically, complex standalone emulators are x86/x86_64 only) with https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-system/Config.in.
+  - Add the system's shortname and a link to a page explaining the system (Wikipedia tends to be a good "default" but there are better sources out there) to the appropriate section in the [[:systems|systems page]] of this wiki.
-  - If BIOS files are required, set them in https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/core/batocera-scripts/scripts/batocera-systems
   - (Optional) If you need to add an explanation for the emulator's exclusion from a particular platform for the [[https://batocera.org/compatibility.php|compatibility chart]], add it to https://github.com/batocera-linux/batocera.linux/blob/master/package/batocera/emulationstation/batocera-es-system/systems-explanations.yml
 If you'd like an example of a recent pull request that adds a whole new emulator: https://github.com/batocera-linux/batocera.linux/pull/4742/files
+===== Patching a package =====
+Buildroot features the ability to patch the source code of a package before compiling it, read [[https://buildroot.org/downloads/manual/manual.html#patch-policy|section 19.2 in the buildroot user manual]] for more info. Here we will do a quick rundown.
+Let's say you want to work on the package "retroarch" to make it evolve via a patch.
+First, compile a complete version of Batocera for the architecture you will test. In this example, we will do so for RPi3.
+<code bash>
+cd batocera.linux
+make bcm2837-build
+</code>
+Once this is done, you don't need to build the entirety of it again for your tests. You can just rename/remove the output's ''target'' folder for that architecture, like so:
+<code bash>
+mv output/bcm2837/target output/bcm2837/target_
+</code>
+Now for our patch. First change the working directory to that of the //build// folder for the respective package:
+<code bash>
+cd output/bcm2837/build/retroarch-version
+</code>
+Establish the package state using ''git'':
+<code bash>
+git init
+</code>
+Then add the file(s) you want to modify to the current commit:
+<code bash>
+git add <the file(s) you modified>
+</code>
+<WRAP center round tip>
+Add all files with ''git add -A''.
+</WRAP>
+Make the modifications to the files you want now.
+Then compare the differences and save them to a "patch" file:
+<code bash>
+git diff > ../../package/batocera/retroarch/001-mymodification.patch
+</code>
+Now clear out the package's respective ''build'' folder to make way for testing a new build with it:
+<code bash>
+cd ../../..
+rm -rf ouput/bcm2837/build/batocera/retroarch-version
+</code>
+Then build the package (replacing ''make'' if using a different make tool):
+<code bash>
+make retroarch-patch # to confirm the patch is applied
+make retroarch # to finish other steps
+</code>
+===== Testing a package without compiling the entirety of Batocera =====
+It's possible to compile //just// the package you've modified and then transfer it over to your live Batocera installation using ''rsync''. First, compile the individual package [[#package_compilation|as explained above]] with ''make bcm2837-pkg PKG=<package name>'' and then run:
+<code bash>
+rsync -av output/bcm2837/target/ root@batocera.local:/
+</code>
+This will sync up the contents of the target directory on your build machine with the Batocera machine. Feel free to immediately test your changes. If the changes are bad and you need to restore, simply reboot and the changes will not persist. To have them persist instead, run [[modify_the_system_while_it_s_running|batocera-save-overlay]] from the machine itself.