316 lines
11 KiB
Markdown
316 lines
11 KiB
Markdown
`https://wiki.raptorcs.com/wiki/Compiling_Firmware`
|
|
|
|
# Compiling Firmware
|
|
|
|
## Purpose
|
|
|
|
The following steps can be used to compile and update the firmware on Raptor Computing Systems's OpenPOWER systems, such as the Talos II or Blackbird.
|
|
|
|
## Applicability
|
|
|
|
These specific instructions are for the Talos II, though the process for compiling firmware for other OpenPOWER systems like Blackbird should be very similar.
|
|
|
|
## Requirements
|
|
|
|
* At least 25GB of free hard drive space
|
|
* 16GB of free RAM
|
|
|
|
## Building on Debian
|
|
|
|
The build system (op-build) has been primarily tested using Debian Stretch. Ensure you install the following packages:
|
|
```
|
|
# Packages needed for OpenPOWER Firmware builds
|
|
$ sudo apt install cscope ctags libz-dev libexpat-dev python texinfo build-essential g++ git bison flex unzip libssl-dev libxml-simple-perl libxml-sax-perl libxml2-dev libxml2-utils xsltproc wget bc rsync
|
|
|
|
# Packages needed for OpenBMC builds
|
|
$ sudo apt install git build-essential libsdl1.2-dev texinfo gawk chrpath diffstat
|
|
```
|
|
|
|
## Building on other Linux distributions
|
|
|
|
If you are on a different distribution, such as Fedora 28, a Debian chroot is recommended:
|
|
```
|
|
$ sudo yum install debootstrap dpkg
|
|
$ sudo debootstrap stretch debian-chroot http://httpredir.debian.org/debian
|
|
$ sudo mount -t proc none debian-chroot/proc/
|
|
$ sudo mount -o bind /sys/ debian-chroot/sys/
|
|
$ sudo mount -o bind /dev/shm/ debian-chroot/dev/shm/
|
|
```
|
|
|
|
Enter the chroot and install the needed packages:
|
|
```
|
|
$ sudo chroot debian-chroot/
|
|
# apt install software-properties-common locales
|
|
|
|
# Packages needed for OpenPOWER Firmware builds
|
|
$ sudo apt install cscope ctags libz-dev libexpat-dev python texinfo build-essential g++ git bison flex unzip libssl-dev libxml-simple-perl libxml-sax-perl libxml2-dev libxml2-utils xsltproc wget bc rsync
|
|
|
|
# Packages needed for OpenBMC builds
|
|
$ sudo apt install git build-essential libsdl1.2-dev texinfo gawk chrpath diffstat
|
|
```
|
|
|
|
Also create a user inside the chroot to build under:
|
|
```
|
|
$ useradd -m build-user -s /bin/bash
|
|
$ su build-user
|
|
$ cd
|
|
```
|
|
|
|
You can now use the chroot to build the firmware.
|
|
|
|
To enter the chroot in the future, you can run the following from any terminal:
|
|
```
|
|
sudo chroot debian-chroot/
|
|
su build-user
|
|
cd
|
|
```
|
|
|
|
## Building the OpenPOWER Firmware
|
|
|
|
### Downloading the sources
|
|
|
|
Raptor Computing Systems maintains a public Git repository containing the complete source code for the firmware.
|
|
To download the source code for Talos systems:
|
|
```
|
|
git clone -b raptor-v2.00 --recursive https://scm.raptorcs.com/scm/git/talos-op-build
|
|
```
|
|
|
|
To download the source code for Blackbird systems:
|
|
```
|
|
git clone -b raptor-v2.00 --recursive https://scm.raptorcs.com/scm/git/blackbird-op-build
|
|
```
|
|
|
|
*Note:* The `master` branch is often in a non-functional state. The latest firmware branch (either `raptor-v2.00` or `raptor-v1.05` at the time of this update) should be used.
|
|
|
|
### Building the firmware
|
|
|
|
Before building the firmware, check the `README.md` file to ensure that all needed packages are installed.
|
|
|
|
The firmware can then be built using the following commands:
|
|
```
|
|
$ cd talos-op-build
|
|
$ git submodule update
|
|
$ . op-build-env
|
|
$ export KERNEL_BITS=64 # needed when building on ppc64, or libopenssl will try to build in 32 bit mode
|
|
$ op-build talos_defconfig
|
|
$ op-build
|
|
```
|
|
|
|
You can pass `-j<num-cores>` to perform a parallel build (`op-build` invokes `make`), though this may result in very high memory usage.
|
|
|
|
If the build completes successfully, the final firmware image is at `output/images/talos.pnor`.
|
|
|
|
### Rebuilding an individual package
|
|
|
|
To rebuild an individual package (such as Hostboot) and recreate the `talos.pnor` image, run:
|
|
```
|
|
$ op-build <pkgname>-rebuild openpower-pnor-rebuild
|
|
```
|
|
|
|
where `<pkgname>` is the name of the package to rebuild.
|
|
|
|
For example:
|
|
```
|
|
$ op-build hostboot-rebuild openpower-pnor-rebuild
|
|
```
|
|
|
|
Note when recompiling hostboot into a PNOR image with openpower-pnor-rebuild, it is usually recommended to force a machine XML rebuild as well:
|
|
```
|
|
$ rm -rf output/build/machine-xml-*
|
|
$ rm -rf output/build/hostboot-*
|
|
$ ./op-build openpower-pnor-rebuild
|
|
```
|
|
|
|
## Installing the OpenPOWER firmware
|
|
|
|
### Transfer image to BMC
|
|
|
|
Copy the firmware to the BMC:
|
|
```
|
|
$ scp ./output/images/talos.pnor root@$TALOS_BMC_ADDR:/tmp/
|
|
```
|
|
|
|
### Establish BMC sessions
|
|
|
|
At this point, you should connect two SSH sessions to OpenBMC.
|
|
In the first session, run the following to display the console during boot:
|
|
```
|
|
$ ssh -p 2200 root@$TALOS_BMC_ADDR
|
|
```
|
|
|
|
The console log will be useful in debugging any issues with the firmware that could occur.
|
|
|
|
In the second session, get a shell on the BMC via SSH:
|
|
```
|
|
$ ssh root@$TALOS_BMC_ADDR
|
|
root@talos:~#
|
|
```
|
|
|
|
*Ensure the system is off* before proceeding:
|
|
```
|
|
root@talos:~# obmcutil state
|
|
CurrentBMCState : xyz.openbmc_project.State.BMC.BMCState.Ready
|
|
CurrentPowerState : xyz.openbmc_project.State.Chassis.PowerState.Off
|
|
CurrentHostState : xyz.openbmc_project.State.Host.HostState.Off
|
|
```
|
|
|
|
The CurrentHostState must be Off before continuing with the procedure.
|
|
If the CurrentHostState is not Off, please turn off the machine:
|
|
```
|
|
root@talos:~# obmcutil chassisoff
|
|
```
|
|
|
|
### Running the firmware temporarily
|
|
|
|
You can test the firmware without installing it, though this requires rebuilding OpenBMC to use a modified `mboxd` binary. (clearnet url: https://gerrit.openbmc-project.xyz/#/c/openbmc/mboxbridge/+/14384/ )
|
|
|
|
First, stop `mboxd`:
|
|
```
|
|
root@talos:~# systemctl stop mboxd
|
|
```
|
|
|
|
Restart `mboxd` with the additional `-b` argument:
|
|
```
|
|
root@talos:~# mboxd -f 64M -w 1M -b file:/tmp/talos.pnor
|
|
```
|
|
|
|
You can now test the new firmware image by starting the machine:
|
|
```
|
|
root@talos:~# obmcutil poweron
|
|
```
|
|
|
|
When you have finished testing the image, stop the machine:
|
|
```
|
|
root@talos:~# obmcutil poweroff
|
|
```
|
|
|
|
*Note:* Ensure the machine is off before proceeding. Verify this by running `obmcutil state`.
|
|
|
|
Finally, terminate `mboxd` and restart the normal `mboxd`:
|
|
```
|
|
root@talos:~# systemctl start mboxd
|
|
```
|
|
|
|
### Flashing the firmware
|
|
|
|
*Warning: Some PNOR firmware updates may require a BMC update to function. Before flashing ensure that your installed BMC firmware is capable of booting the image. It is also possible to downgrade using these instructions in the event of a non-functioning firmware image.*
|
|
|
|
Ensure the system is off.
|
|
|
|
Perform the update:
|
|
```
|
|
root@talos:~# pflash -E -p /tmp/talos.pnor
|
|
```
|
|
|
|
Start the machine:
|
|
```
|
|
root@talos:~# obmcutil poweron
|
|
```
|
|
|
|
*Note:* The machine may reboot multiple times when first booted after a firmware update. This is normal; do not interrupt the process.
|
|
|
|
## Troubleshooting the OpenPOWER Firmware
|
|
|
|
### General advice
|
|
|
|
*Always upgrade PNOR and BMC together*
|
|
Many mismatched PNOR/BMC version combinations lead to weird failures.
|
|
|
|
*Try downgrading the PNOR+BMC firmware*
|
|
Firmware package 1.04 seems the most reliable at updating the [[Self-Boot_Engine|SBE SEEPROM]] inside the POWER9 chip package.
|
|
|
|
*Always use processor socket 0 for SBE updates*
|
|
The BMC firmware and/or FSI driver seem to either forget to update the SBE SEEPROM in the second CPU socket, leading to a boot with only CPU 0 active. When you get a brand new chip you need to install it in CPU socket 0 leaving socket 1 empty, wait for the double-reboot to update the SEEPROM, and then you can move that chip to socket 1 if you like.
|
|
|
|
*Try unplugging the HSF fan power during [[Self-Boot_Engine|SBE]] update*
|
|
Not kidding about this. The BMC is insanely complicated -- it's got an entire operating system in there for some reason. It even has systemd. The BMC's systemd often gets into a funky loop restarting `hwmon` over and over and over, interrupting the SBE SEEPROM reflash every time it does this. Unplugging the PROC0 HSF 4-pin connector gets it to fail hard (due to inability to read the tachometer) and stay failed so the SBE update can proceed. Ugly as this is, it's easier than trying to figure out what systemd thinks it's doing.
|
|
|
|
### SBE_MASTER_VERSION_DOWNLEVEL
|
|
If you see the following message reported in the console, then the Self-Boot_Engine update process did not work as expected:
|
|
```
|
|
16.74709|Error reported by sbe (0x2200) PLID 0x90000008
|
|
16.74823| SBE Image Version Miscompare with Master Target
|
|
16.74824| ModuleId 0x0d SBE_MASTER_VERSION_COMPARE
|
|
16.74825| ReasonCode 0x2215 SBE_MASTER_VERSION_DOWNLEVEL
|
|
16.74826| UserData1 Master Target HUID : 0x0000000000050000
|
|
16.74826| UserData2 Master Target Loop Index : 0x0000000000000000
|
|
```
|
|
|
|
The machine needs to be reset to finish the update procedure:
|
|
```
|
|
root@talos:~# obmcutil chassisoff
|
|
root@talos:~# systemctl stop xyz.openbmc_project.State.Host.service
|
|
root@talos:~# systemctl start xyz.openbmc_project.State.Host.service
|
|
root@talos:~# obmcutil poweron
|
|
```
|
|
|
|
The update should now complete as expected.
|
|
|
|
A bug report is open to track this issue. (clearnet url: https://github.com/open-power/sbe/issues/7 )
|
|
|
|
### internal compiler error: Killed
|
|
|
|
Building the Hostboot source code requires a large amount of RAM. If your machine runs out, you may see an error similar to the following:
|
|
```
|
|
powerpc64le-buildroot-linux-gnu-g++.br_real: internal compiler error: Killed (program cc1plus)
|
|
```
|
|
|
|
To continue you have a few options:
|
|
- Reduce the number of parallel jobs being run by appending -j<num> to you build command line
|
|
op-build -j4
|
|
- Increase the swap space (not recommended)
|
|
- Install additional RAM
|
|
|
|
### Bugs
|
|
|
|
Firmware issues should be reported preferably upstream.
|
|
|
|
If they are specific to Raptor Computing Systems products, please report them on their bug tracker. (clearnet url: https://bugs.raptorengineering.com/ )
|
|
|
|
## Building the OpenBMC firmware
|
|
|
|
### Downloading the sources
|
|
|
|
Raptor Computing Systems maintains a public Git repository containing the complete source code for the firmware.
|
|
|
|
To download the source code:
|
|
```
|
|
$ git clone -b raptor-v{{CURRENT_BMC_VERSION}} https://git.raptorcs.com/git/talos-openbmc
|
|
```
|
|
|
|
### Building the firmware
|
|
|
|
Ensure that all needed support packages are installed. See the `README.md` for information on needed packages.
|
|
|
|
The firmware can then be built using the following commands:
|
|
```
|
|
$ cd talos-openbmc
|
|
$ export TEMPLATECONF=meta-openbmc-machines/meta-openpower/meta-rcs/meta-talos/conf
|
|
$ . openbmc-env
|
|
$ bitbake obmc-phosphor-image
|
|
```
|
|
|
|
The resulting firmware image can then be found in the `tmp/deploy/images/talos/` directory.
|
|
|
|
*Note:* If `mboxd` fails to build, you may need to [ patch `mboxd.bb`]. (clearnet url: https://github.com/openbmc/openbmc/issues/2780 )
|
|
|
|
*Note:* If building newer versions of the firmware, TEMPLATECONF has changed to TEMPLATECONF=meta-rcs/meta-talos/conf. This should be set before running `. open-env`. If not, do a git clean and start over with the new TEMPLATECONF.
|
|
|
|
### Installing the firmware
|
|
*Warning: If you are attempting to upgrade a Talos system from firmware 1.06 or earlier to the 04-16-2019 beta branch, you must follow the directions at Talos_II/Firmware/Public_Beta. Failure to do so may result in a non-booting BMC. (clearnet url: https://wiki.raptorcs.com/wiki/Talos_II/Firmware/Public_Beta )*
|
|
|
|
Once firmware has been built, the resulting `image-kernel` and `image-rofs` binaries must be copied to `/run/initramfs/` on the BMC:
|
|
```
|
|
scp tmp/deploy/images/talos/image-rofs tmp/deploy/images/talos/image-kernel root@$TALOS_BMC_ADDR:/run/initramfs/
|
|
```
|
|
|
|
Once the images have been transferred, reboot the BMC. The new firmware files will be detected and automatically applied.
|
|
```
|
|
root@talos:~# reboot
|
|
```
|
|
|
|
The reboot may take some time. Once complete, you will be able to log back in via SSH.
|
|
|
|
### Recovering from failed firmware updates
|
|
See [Debricking the BMC](debricking-the-bmc.md).
|