add raptorcs documents relating to secureboot and firmware building.
This commit is contained in:
yafox
parent
502a345bdd
commit
fbc52a5e9b
|
|
@ -0,0 +1,315 @@
|
|||
`https://wiki.raptorcs.com/wiki/Compiling_Firmware`
|
||||
|
||||
# Compiling Firmware
|
||||
|
||||
## Purpose
|
||||
|
||||
The following steps can be used to compile and update the firmware on Raptor Computing Systems's OpenPOWER systems, such as the Talos II or Blackbird.
|
||||
|
||||
## Applicability
|
||||
|
||||
These specific instructions are for the Talos II, though the process for compiling firmware for other OpenPOWER systems like Blackbird should be very similar.
|
||||
|
||||
## Requirements
|
||||
|
||||
* At least 25GB of free hard drive space
|
||||
* 16GB of free RAM
|
||||
|
||||
## Building on Debian
|
||||
|
||||
The build system (op-build) has been primarily tested using Debian Stretch. Ensure you install the following packages:
|
||||
```
|
||||
# Packages needed for OpenPOWER Firmware builds
|
||||
$ sudo apt install cscope ctags libz-dev libexpat-dev python texinfo build-essential g++ git bison flex unzip libssl-dev libxml-simple-perl libxml-sax-perl libxml2-dev libxml2-utils xsltproc wget bc rsync
|
||||
|
||||
# Packages needed for OpenBMC builds
|
||||
$ sudo apt install git build-essential libsdl1.2-dev texinfo gawk chrpath diffstat
|
||||
```
|
||||
|
||||
## Building on other Linux distributions
|
||||
|
||||
If you are on a different distribution, such as Fedora 28, a Debian chroot is recommended:
|
||||
```
|
||||
$ sudo yum install debootstrap dpkg
|
||||
$ sudo debootstrap stretch debian-chroot http://httpredir.debian.org/debian
|
||||
$ sudo mount -t proc none debian-chroot/proc/
|
||||
$ sudo mount -o bind /sys/ debian-chroot/sys/
|
||||
$ sudo mount -o bind /dev/shm/ debian-chroot/dev/shm/
|
||||
```
|
||||
|
||||
Enter the chroot and install the needed packages:
|
||||
```
|
||||
$ sudo chroot debian-chroot/
|
||||
# apt install software-properties-common locales
|
||||
|
||||
# Packages needed for OpenPOWER Firmware builds
|
||||
$ sudo apt install cscope ctags libz-dev libexpat-dev python texinfo build-essential g++ git bison flex unzip libssl-dev libxml-simple-perl libxml-sax-perl libxml2-dev libxml2-utils xsltproc wget bc rsync
|
||||
|
||||
# Packages needed for OpenBMC builds
|
||||
$ sudo apt install git build-essential libsdl1.2-dev texinfo gawk chrpath diffstat
|
||||
```
|
||||
|
||||
Also create a user inside the chroot to build under:
|
||||
```
|
||||
$ useradd -m build-user -s /bin/bash
|
||||
$ su build-user
|
||||
$ cd
|
||||
```
|
||||
|
||||
You can now use the chroot to build the firmware.
|
||||
|
||||
To enter the chroot in the future, you can run the following from any terminal:
|
||||
```
|
||||
sudo chroot debian-chroot/
|
||||
su build-user
|
||||
cd
|
||||
```
|
||||
|
||||
## Building the OpenPOWER Firmware
|
||||
|
||||
### Downloading the sources
|
||||
|
||||
Raptor Computing Systems maintains a public Git repository containing the complete source code for the firmware.
|
||||
To download the source code for Talos systems:
|
||||
```
|
||||
git clone -b raptor-v2.00 --recursive https://scm.raptorcs.com/scm/git/talos-op-build
|
||||
```
|
||||
|
||||
To download the source code for Blackbird systems:
|
||||
```
|
||||
git clone -b raptor-v2.00 --recursive https://scm.raptorcs.com/scm/git/blackbird-op-build
|
||||
```
|
||||
|
||||
*Note:* The `master` branch is often in a non-functional state. The latest firmware branch (either `raptor-v2.00` or `raptor-v1.05` at the time of this update) should be used.
|
||||
|
||||
### Building the firmware
|
||||
|
||||
Before building the firmware, check the `README.md` file to ensure that all needed packages are installed.
|
||||
|
||||
The firmware can then be built using the following commands:
|
||||
```
|
||||
$ cd talos-op-build
|
||||
$ git submodule update
|
||||
$ . op-build-env
|
||||
$ export KERNEL_BITS=64 # needed when building on ppc64, or libopenssl will try to build in 32 bit mode
|
||||
$ op-build talos_defconfig
|
||||
$ op-build
|
||||
```
|
||||
|
||||
You can pass `-j<num-cores>` to perform a parallel build (`op-build` invokes `make`), though this may result in very high memory usage.
|
||||
|
||||
If the build completes successfully, the final firmware image is at `output/images/talos.pnor`.
|
||||
|
||||
### Rebuilding an individual package
|
||||
|
||||
To rebuild an individual package (such as Hostboot) and recreate the `talos.pnor` image, run:
|
||||
```
|
||||
$ op-build <pkgname>-rebuild openpower-pnor-rebuild
|
||||
```
|
||||
|
||||
where `<pkgname>` is the name of the package to rebuild.
|
||||
|
||||
For example:
|
||||
```
|
||||
$ op-build hostboot-rebuild openpower-pnor-rebuild
|
||||
```
|
||||
|
||||
Note when recompiling hostboot into a PNOR image with openpower-pnor-rebuild, it is usually recommended to force a machine XML rebuild as well:
|
||||
```
|
||||
$ rm -rf output/build/machine-xml-*
|
||||
$ rm -rf output/build/hostboot-*
|
||||
$ ./op-build openpower-pnor-rebuild
|
||||
```
|
||||
|
||||
## Installing the OpenPOWER firmware
|
||||
|
||||
### Transfer image to BMC
|
||||
|
||||
Copy the firmware to the BMC:
|
||||
```
|
||||
$ scp ./output/images/talos.pnor root@$TALOS_BMC_ADDR:/tmp/
|
||||
```
|
||||
|
||||
### Establish BMC sessions
|
||||
|
||||
At this point, you should connect two SSH sessions to OpenBMC.
|
||||
In the first session, run the following to display the console during boot:
|
||||
```
|
||||
$ ssh -p 2200 root@$TALOS_BMC_ADDR
|
||||
```
|
||||
|
||||
The console log will be useful in debugging any issues with the firmware that could occur.
|
||||
|
||||
In the second session, get a shell on the BMC via SSH:
|
||||
```
|
||||
$ ssh root@$TALOS_BMC_ADDR
|
||||
root@talos:~#
|
||||
```
|
||||
|
||||
*Ensure the system is off* before proceeding:
|
||||
```
|
||||
root@talos:~# obmcutil state
|
||||
CurrentBMCState : xyz.openbmc_project.State.BMC.BMCState.Ready
|
||||
CurrentPowerState : xyz.openbmc_project.State.Chassis.PowerState.Off
|
||||
CurrentHostState : xyz.openbmc_project.State.Host.HostState.Off
|
||||
```
|
||||
|
||||
The CurrentHostState must be Off before continuing with the procedure.
|
||||
If the CurrentHostState is not Off, please turn off the machine:
|
||||
```
|
||||
root@talos:~# obmcutil chassisoff
|
||||
```
|
||||
|
||||
### Running the firmware temporarily
|
||||
|
||||
You can test the firmware without installing it, though this requires rebuilding OpenBMC to use a modified `mboxd` binary. (clearnet url: https://gerrit.openbmc-project.xyz/#/c/openbmc/mboxbridge/+/14384/ )
|
||||
|
||||
First, stop `mboxd`:
|
||||
```
|
||||
root@talos:~# systemctl stop mboxd
|
||||
```
|
||||
|
||||
Restart `mboxd` with the additional `-b` argument:
|
||||
```
|
||||
root@talos:~# mboxd -f 64M -w 1M -b file:/tmp/talos.pnor
|
||||
```
|
||||
|
||||
You can now test the new firmware image by starting the machine:
|
||||
```
|
||||
root@talos:~# obmcutil poweron
|
||||
```
|
||||
|
||||
When you have finished testing the image, stop the machine:
|
||||
```
|
||||
root@talos:~# obmcutil poweroff
|
||||
```
|
||||
|
||||
*Note:* Ensure the machine is off before proceeding. Verify this by running `obmcutil state`.
|
||||
|
||||
Finally, terminate `mboxd` and restart the normal `mboxd`:
|
||||
```
|
||||
root@talos:~# systemctl start mboxd
|
||||
```
|
||||
|
||||
### Flashing the firmware
|
||||
|
||||
*Warning: Some PNOR firmware updates may require a BMC update to function. Before flashing ensure that your installed BMC firmware is capable of booting the image. It is also possible to downgrade using these instructions in the event of a non-functioning firmware image.*
|
||||
|
||||
Ensure the system is off.
|
||||
|
||||
Perform the update:
|
||||
```
|
||||
root@talos:~# pflash -E -p /tmp/talos.pnor
|
||||
```
|
||||
|
||||
Start the machine:
|
||||
```
|
||||
root@talos:~# obmcutil poweron
|
||||
```
|
||||
|
||||
*Note:* The machine may reboot multiple times when first booted after a firmware update. This is normal; do not interrupt the process.
|
||||
|
||||
## Troubleshooting the OpenPOWER Firmware
|
||||
|
||||
### General advice
|
||||
|
||||
*Always upgrade PNOR and BMC together*
|
||||
Many mismatched PNOR/BMC version combinations lead to weird failures.
|
||||
|
||||
*Try downgrading the PNOR+BMC firmware*
|
||||
Firmware package 1.04 seems the most reliable at updating the [[Self-Boot_Engine|SBE SEEPROM]] inside the POWER9 chip package.
|
||||
|
||||
*Always use processor socket 0 for SBE updates*
|
||||
The BMC firmware and/or FSI driver seem to either forget to update the SBE SEEPROM in the second CPU socket, leading to a boot with only CPU 0 active. When you get a brand new chip you need to install it in CPU socket 0 leaving socket 1 empty, wait for the double-reboot to update the SEEPROM, and then you can move that chip to socket 1 if you like.
|
||||
|
||||
*Try unplugging the HSF fan power during [[Self-Boot_Engine|SBE]] update*
|
||||
Not kidding about this. The BMC is insanely complicated -- it's got an entire operating system in there for some reason. It even has systemd. The BMC's systemd often gets into a funky loop restarting `hwmon` over and over and over, interrupting the SBE SEEPROM reflash every time it does this. Unplugging the PROC0 HSF 4-pin connector gets it to fail hard (due to inability to read the tachometer) and stay failed so the SBE update can proceed. Ugly as this is, it's easier than trying to figure out what systemd thinks it's doing.
|
||||
|
||||
### SBE_MASTER_VERSION_DOWNLEVEL
|
||||
If you see the following message reported in the console, then the Self-Boot_Engine update process did not work as expected:
|
||||
```
|
||||
16.74709|Error reported by sbe (0x2200) PLID 0x90000008
|
||||
16.74823| SBE Image Version Miscompare with Master Target
|
||||
16.74824| ModuleId 0x0d SBE_MASTER_VERSION_COMPARE
|
||||
16.74825| ReasonCode 0x2215 SBE_MASTER_VERSION_DOWNLEVEL
|
||||
16.74826| UserData1 Master Target HUID : 0x0000000000050000
|
||||
16.74826| UserData2 Master Target Loop Index : 0x0000000000000000
|
||||
```
|
||||
|
||||
The machine needs to be reset to finish the update procedure:
|
||||
```
|
||||
root@talos:~# obmcutil chassisoff
|
||||
root@talos:~# systemctl stop xyz.openbmc_project.State.Host.service
|
||||
root@talos:~# systemctl start xyz.openbmc_project.State.Host.service
|
||||
root@talos:~# obmcutil poweron
|
||||
```
|
||||
|
||||
The update should now complete as expected.
|
||||
|
||||
A bug report is open to track this issue. (clearnet url: https://github.com/open-power/sbe/issues/7 )
|
||||
|
||||
### internal compiler error: Killed
|
||||
|
||||
Building the Hostboot source code requires a large amount of RAM. If your machine runs out, you may see an error similar to the following:
|
||||
```
|
||||
powerpc64le-buildroot-linux-gnu-g++.br_real: internal compiler error: Killed (program cc1plus)
|
||||
```
|
||||
|
||||
To continue you have a few options:
|
||||
- Reduce the number of parallel jobs being run by appending -j<num> to you build command line
|
||||
op-build -j4
|
||||
- Increase the swap space (not recommended)
|
||||
- Install additional RAM
|
||||
|
||||
### Bugs
|
||||
|
||||
Firmware issues should be reported preferably upstream.
|
||||
|
||||
If they are specific to Raptor Computing Systems products, please report them on their bug tracker. (clearnet url: https://bugs.raptorengineering.com/ )
|
||||
|
||||
## Building the OpenBMC firmware
|
||||
|
||||
### Downloading the sources
|
||||
|
||||
Raptor Computing Systems maintains a public Git repository containing the complete source code for the firmware.
|
||||
|
||||
To download the source code:
|
||||
```
|
||||
$ git clone -b raptor-v{{CURRENT_BMC_VERSION}} https://git.raptorcs.com/git/talos-openbmc
|
||||
```
|
||||
|
||||
### Building the firmware
|
||||
|
||||
Ensure that all needed support packages are installed. See the `README.md` for information on needed packages.
|
||||
|
||||
The firmware can then be built using the following commands:
|
||||
```
|
||||
$ cd talos-openbmc
|
||||
$ export TEMPLATECONF=meta-openbmc-machines/meta-openpower/meta-rcs/meta-talos/conf
|
||||
$ . openbmc-env
|
||||
$ bitbake obmc-phosphor-image
|
||||
```
|
||||
|
||||
The resulting firmware image can then be found in the `tmp/deploy/images/talos/` directory.
|
||||
|
||||
*Note:* If `mboxd` fails to build, you may need to [ patch `mboxd.bb`]. (clearnet url: https://github.com/openbmc/openbmc/issues/2780 )
|
||||
|
||||
*Note:* If building newer versions of the firmware, TEMPLATECONF has changed to TEMPLATECONF=meta-rcs/meta-talos/conf. This should be set before running `. open-env`. If not, do a git clean and start over with the new TEMPLATECONF.
|
||||
|
||||
### Installing the firmware
|
||||
*Warning: If you are attempting to upgrade a Talos system from firmware 1.06 or earlier to the 04-16-2019 beta branch, you must follow the directions at Talos_II/Firmware/Public_Beta. Failure to do so may result in a non-booting BMC. (clearnet url: https://wiki.raptorcs.com/wiki/Talos_II/Firmware/Public_Beta )*
|
||||
|
||||
Once firmware has been built, the resulting `image-kernel` and `image-rofs` binaries must be copied to `/run/initramfs/` on the BMC:
|
||||
```
|
||||
scp tmp/deploy/images/talos/image-rofs tmp/deploy/images/talos/image-kernel root@$TALOS_BMC_ADDR:/run/initramfs/
|
||||
```
|
||||
|
||||
Once the images have been transferred, reboot the BMC. The new firmware files will be detected and automatically applied.
|
||||
```
|
||||
root@talos:~# reboot
|
||||
```
|
||||
|
||||
The reboot may take some time. Once complete, you will be able to log back in via SSH.
|
||||
|
||||
### Recovering from failed firmware updates
|
||||
See [Debricking the BMC](debricking-the-bmc.md).
|
||||
|
|
@ -0,0 +1,104 @@
|
|||
`https://wiki.raptorcs.com/wiki/Debricking_the_BMC/Watchdog`
|
||||
|
||||
# Debricking the BMC/Watchdog
|
||||
|
||||
Once standby power has been applied to the Talos II (i.e. you have plugged it in), the FPGA waits for the power to stabilize, and then signals the BMC to reset.
|
||||
|
||||
At this point, a watchdog counter in the FPGA begins running. This is done because of a hardware issue in the ast2500, where very occasionally the chip would get stuck when attempting to start. (clearnet url: https://git.raptorcs.com/git/talos-system-fpga/commit/main.v?id=e90ca898402a250e9d2f6e303e25ddaceb0cf8d6 ) If the BMC boot gets stuck in the U-Boot phase, the FPGA will force a reset after approximately 40 seconds. This interferes with BMC recovery, as that leaves you with a limited window to run commands.
|
||||
|
||||
U-Boot on the Talos II has been slightly modified to disable this watchdog counter immediately before transferring control to the OpenBMC kernel. (clearnet url: https://git.raptorcs.com/git/talos-obmc-uboot/tree/common/bootm.c?id=cfee45a3ef2a592d130573ce3b7d8bcfe056060b#n707 )
|
||||
|
||||
Since we will want to spend more than 40 seconds at the U-Boot shell during recovery, we need to disable this watchdog ourselves.
|
||||
|
||||
However, general GPIO support is not built into the Talos II U-Boot loader, so the hardware must be manipulated directly.
|
||||
|
||||
Using an ARM cross-compiler, we can build a tiny program to do the same thing as the bootm.c code.
|
||||
|
||||
## Payload creation
|
||||
|
||||
### For Talos II:
|
||||
|
||||
```
|
||||
/* watchdog.c - minimal code to disable the FPGA watchdog.
|
||||
* Derived from Raptor Engineering changes to U-Boot common/bootm.c.
|
||||
* SPDX-License-Identifier: GPL-2.0+
|
||||
*/
|
||||
#include <stdint.h>
|
||||
int main() {
|
||||
uint32_t* gpio_ctl_reg = 0x1e780084;
|
||||
uint32_t* gpio_data_reg = 0x1e780080;
|
||||
|
||||
*gpio_ctl_reg |= 0x00800000;
|
||||
*gpio_data_reg &= ~0x00800000;
|
||||
return 0;
|
||||
}
|
||||
```
|
||||
|
||||
### For Blackbird:
|
||||
|
||||
```
|
||||
/* watchdog.c - minimal code to disable the FPGA watchdog.
|
||||
* Derived from Raptor Engineering changes to U-Boot common/bootm.c.
|
||||
* SPDX-License-Identifier: GPL-2.0+
|
||||
*/
|
||||
#include <stdint.h>
|
||||
int main() {
|
||||
uint32_t* gpio_ctl_reg = 0x1e780024;
|
||||
uint32_t* gpio_data_reg = 0x1e780020;
|
||||
|
||||
*gpio_ctl_reg |= 0x00010000;
|
||||
*gpio_data_reg &= ~0x00010000;
|
||||
return 0;
|
||||
}
|
||||
```
|
||||
|
||||
Compiling this to an object and then converting it to an s-record will give you a file that can be directly loaded into U-Boot without using special tools.
|
||||
```
|
||||
$ arm-linux-gnueabihf-gcc -ffreestanding -march=armv6 -mfloat-abi=soft -marm -Os -c watchdog.c
|
||||
$ arm-linux-gnueabihf-objcopy -O srec watchdog.o watchdog.srec
|
||||
$ cat watchdog.srec
|
||||
```
|
||||
|
||||
### For Talos II:
|
||||
```
|
||||
S01000007761746368646F672E73726563C3
|
||||
S11300001C309FE50000A0E3842093E5022582E3F1
|
||||
S1130010842083E5802093E50225C2E3802083E5E4
|
||||
S10B00201EFF2FE10000781E11
|
||||
S9030000FC
|
||||
```
|
||||
|
||||
### For Blackbird:
|
||||
```
|
||||
S01000007761746368646F672E73726563C3
|
||||
S11300001C309FE50000A0E3242093E5012882E34F
|
||||
S1130010242083E5202093E50128C2E3202083E502
|
||||
S10B00201EFF2FE10000781E11
|
||||
S9030000FC<
|
||||
```
|
||||
|
||||
Due to compiler differences, your output may be slightly different than the provided example s-record code. The example was compiled with `arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0`.
|
||||
|
||||
- Copy the srec data to the clipboard, as we will need to send it to the BMC within a limited time window in a bit.
|
||||
|
||||
## Main Procedure
|
||||
|
||||
To load and execute this code, do the following at the `ast#` shell within the watchdog time window:
|
||||
|
||||
- `ast# loads 83000000`
|
||||
- Paste the contents of the srec file into the terminal when prompted. Some summary data will be shown on screen.
|
||||
|
||||
(todo: grab and stick the example output in here.)
|
||||
|
||||
Run the code using the `go` command.
|
||||
|
||||
- `ast# go 83000000`
|
||||
|
||||
(todo: stick the output in here)
|
||||
|
||||
The program will run and then return control to U-Boot. At this point, the watchdog has been disabled, and you can take your time with the rest of the recovery commands. The loaded program code is no longer needed and the memory range can be reused.
|
||||
|
||||
## Notes
|
||||
|
||||
- For reference, the GPIO pin associated with this watchdog (For Talos II / Lite) is GPIOS7 (GPIO 151, physical pin AA20 on the ast2500). The signal is labelled SEQ_CONT if you are looking at it on the schematics.
|
||||
- On Blackbird, this pin is instead GPIOG0 (GPIO ???, Physical pin A19 on the ast2500). The signal is labelled BMC_BOOT_PHASE if you are looking at it on the schematics.
|
||||
|
|
@ -0,0 +1,138 @@
|
|||
`https://wiki.raptorcs.com/wiki/Debricking_the_BMC`
|
||||
|
||||
# Debricking the BMC
|
||||
|
||||
## Purpose
|
||||
|
||||
This guide explains how to debrick the BMC when the BMC has been rendered inoperable, for example due to a defective firmware update.
|
||||
|
||||
## Applicability
|
||||
|
||||
All RCS OpenPOWER systems.
|
||||
|
||||
## Overview
|
||||
|
||||
There are three means of debricking the BMC:
|
||||
|
||||
- Remove the BMC SPI flash chip and reflash it with a flash programmer
|
||||
Note: flashrom versions earlier than 1.1 do not support the BMC flash chip (clearnet url: https://www.flashrom.org/Flashrom )
|
||||
- Flash new BMC firmware via U-Boot TFTP (requires that U-Boot is still intact on the flash)
|
||||
- Flash new BMC firmware via serial port (requires proprietary BMC chip vendor tool)
|
||||
|
||||
## Reset persistent storage
|
||||
This is applicable if somehow the persistent storage (SSH keys, passwords, IPMI error logs, etc.) has been corrupted, but the read only data (U-boot, kernel, initramfs) are all intact. This is also the easiest and least invasive recovery method if you have forgotten the BMC password.
|
||||
|
||||
From the U-boot prompt on the BMC serial console, run the following (must be run quickly, to avoid watchdog timeouts):
|
||||
|
||||
`printenv`
|
||||
|
||||
Look at the bootargs command, set the same environment variable but insert `overlay-filesystem-in-ram` before the `rw` keyword.
|
||||
|
||||
Example for Blackbird HW version 1.01:
|
||||
|
||||
`setenv bootargs console=ttyS4,115200n8 root=/dev/ram overlay-filesystem-in-ram rw`
|
||||
|
||||
Then run `boot` to continue the boot process.
|
||||
|
||||
This will start the BMC with default settings, but the existing persistent data has not yet been cleared. To clear it, log in as root, then run:
|
||||
|
||||
```
|
||||
flash_eraseall /dev/mtd/rwfs
|
||||
reboot
|
||||
```
|
||||
|
||||
## Flash new BMC firmware via U-Boot TFTP
|
||||
|
||||
-Note:* While these instructions have been successfully applied in practice, they are still preliminary. Ask questions in IRC if you are unclear on what to do!
|
||||
|
||||
```
|
||||
Hi fellow wiki people! Ask Bdragon in IRC if you have questions about this procedure.
|
||||
IRC user dragon_pilot was successfully able to recover a nonworking BMC from u-boot, these instructions are the result of that experiment.
|
||||
Further testing and refinement would be appreciated, preferably by someone who has easy access to an external flasher.
|
||||
```
|
||||
|
||||
In the event of a failure when updating the BMC, but with a functioning U-boot, you can still recover by using U-Boot to manually bootstrap the BMC by manually loading a boot image over the network or the BMC serial port.
|
||||
|
||||
If your BMC flash is corrupted to the extent that U-Boot does not load properly, these instructions will not work; you will need to remove and reflash the BMC flash chip externally, or flash new firmware via serial port.
|
||||
|
||||
- Prepare a TFTP server, and place `image-bmc`, `image-rofs`, and `image-kernel` in the root.
|
||||
- Connect a serial console to the BMC serial port (J7701, serial port bracket required). The serial port configuration is `115200,8n1`.
|
||||
- Disconnect and reconnect power to the machine to force a BMC restart. Press a key to interrupt auto-boot when prompted.
|
||||
- If you are having trouble with U-Boot resetting while you are trying to run these steps, have a slow network, or you are going to be loading over serial, you can [disable the FPGA watchdog](debricking-the-bmc-watchdog.md).
|
||||
- Run `dhcp x.x.x.x:image-bmc`, replacing the IP address of your TFTP server. This will load a copy of the stock boot image into RAM.
|
||||
- Run `bootm 83080000`. This will prepare and boot off of the loaded virtual image.
|
||||
- If your rofs partition is not functional, you will be dropped into the systemd emergency shell at this point. Try both the password you set as well as the default `0penBmc`, it may be one or the other depending on the state of the rwfs partition. If it boots up properly instead of dropping you into the emergency shell, the problem is probably in your kernel partition and you can retry flashing your `image-kernel` using the normal procedure. (The rest of these instructions are for the systemd emergency shell.)
|
||||
- `mount -t tmpfs none /tmp`
|
||||
- run `udhcpc` to get an IP address. (TODO: verify that this is the actual command that you run. Do you have to specify the network interface too?)
|
||||
- `cd /tmp`
|
||||
- `tftp -g -r image-rofs x.x.x.x`
|
||||
- `tftp -g -r image-kernel x.x.x.x`
|
||||
- IMPORTANT: Use `md5sum`, `sha1sum`, or `sha256sum` to verify successful transfer of image-rofs and image-kernel! tftp is a very barebones protocol and relies on transport layer checksumming, which is optional and not always available in UDP!
|
||||
- Verify that the output of `cat /sys/class/mtd/mtd3/name` is `kernel` and the output of `cat /sys/class/mtd/mtd4/name` is `rofs`. We will be flashing mtd partitions directly in the next step and this is the last chance to verify that they will be flashed to the correct partition.
|
||||
- `flashcp -v image-kernel /dev/mtd3`
|
||||
- `flashcp -v image-rofs /dev/mtd4`
|
||||
- (TODO: Describe how to reset rwfs in case it was damaged as well?) note: the kernel param for bypassing rwfs is "overlay-filesystem-in-ram". Append it to the existing boot-args before running the bootm command. This can also be used as part of a password reset procedure.
|
||||
- After the flash is complete, you can run restart the BMC and it should boot successfully.
|
||||
|
||||
- (TODO: Discussion of using Kermit to upload the image without network access) note: I (Bdragon) have successfully done a ram-only boot using cu's built in xmodem support (escape sequence ~X) to do an image transfer into RAM over the BMC serial interface.
|
||||
- (TODO: Discuss using u-boot's built in cmp tool to perform basic validation of the u-boot image against a second copy loaded into RAM.)
|
||||
- (TODO: Load recovery images over USB?) note: The onboard USB port is connected to the USB switch after all, so this might be problematic.
|
||||
- (TODO: Discussion of u-boot memory map) Short version is: flash lives at 0x20000000 and default base address for the memory loading tools is 0x83000000. So add 0x63000000 to any flash address to get the eqivilent address for an image-bmc file loaded into RAM. For example, the bootable image of a loaded image-bmc is at 0x83080000.
|
||||
|
||||
## Flash new BMC firmware via serial port (Open Source Method)
|
||||
|
||||
Tools required:
|
||||
|
||||
- BMC serial port
|
||||
- A secure computer with a serial port (usb to serial works fine) - preferably running linux (Linux on POWER is fine).
|
||||
|
||||
Software:
|
||||
|
||||
- flashrom with serial ASpeed flash support from [https://gitlab.raptorengineering.com/raptor-engineering-public/ast2050-flashrom]
|
||||
- BMC Firmware bundle: System Package 1.06 2a92dec044239591244b6ed69c3fac162a6b9ea4 (clearnet url: https://wiki.raptorcs.com/wiki/File:Talos_ii_openbmc_v1.07_bundle.tar.bz2 )
|
||||
|
||||
Procedure:
|
||||
|
||||
1. Build flashrom on your Linux or BSD PC.
|
||||
2. Extract the BMC firmware bundle.
|
||||
3. Set the FPGA RUN/RESET switch to RESET.
|
||||
- On a Blackbird, this switch is located roughly between the flash chips and the PCIe slots. If you have a GPU installed in the x16 slot, you may need to remove it.
|
||||
4. Apply standby power to the mainboard
|
||||
5. Run the following command '''./flashrom --verbose --programmer 'ast2400:serial=/dev/ttyUSB0,cpu=halt,spibus=0' -c MX25L25635F/MX25L25645E/MX25L25665E -w image-bmc'''
|
||||
- if your serial interface can handle the baudrate 921600 add the parameter: `high_speed_uart=true`
|
||||
- NOTE: If you are using updated firmware (Talos II/Lite 2.0 beta firmware or later) or are using a Blackbird, U-boot will shut down access to this interface after about 3 seconds of standby power, so you will need to run the command _immediately_ after plugging in the power supply to bypass this.
|
||||
6. Be Patient: this will take a _long_ time.
|
||||
7. Once the flash has been verified, set the FPGA RUN/RESET switch to RUN.
|
||||
|
||||
## Flash new BMC firmware via serial port (Proprietary Method)
|
||||
|
||||
_This method was discovered by Centurion Dan as an alternative to pulling and reflashing the BMC SPI chip after a failed update had corrupted/wiped U-Boot._
|
||||
|
||||
Tools required:
|
||||
|
||||
- BMC serial port
|
||||
- An x86 computer with a serial port (usb to serial works fine) - preferably running linux.
|
||||
|
||||
Software:
|
||||
|
||||
- Proprietary SOC Flash Utility from https://www.aspeedtech.com/support.php : at least version 1.18.00. (clearnet url: http://upload.aspeedtech.com/SOC/v11800.zip) Since version 1.20.x Aspeed requires being registered as developer to download this util:
|
||||
```
|
||||
ASPEED SOC Flash Utility --- The utility has been moved to Document Download Page for ASPEED registered developers to access.
|
||||
```
|
||||
- BMC Firmware bundle: System Package 1.06 2a92dec044239591244b6ed69c3fac162a6b9ea4 (clearnet url: https://wiki.raptorcs.com/wiki/File:Talos_ii_openbmc_v1.07_bundle.tar.bz2 )
|
||||
|
||||
Procedure:
|
||||
|
||||
1. Unzip the SOC FLASH Utility on your other computer, and unzip the appropriate SOC Flash Utility bundle for that computer.
|
||||
2. Extract the BMC firmware bundle.
|
||||
3. Run the following command `./socflash -s option=u comport="4" cs=0 if=image-u-boot gpio_b=S71 gpio_a=S70 option=f`
|
||||
- You can drop the `option=f` for a slower but verified write process
|
||||
- if your serial interface can handle the baudrate 921600 add the parameter: `baudrate=921600`
|
||||
- if you want to see what is going on, you can strace it by prepending: `strace -e trace=open,close,read,write` to the command above.
|
||||
- NOTE: If you are using updated firmware (Talos II/Lite 2.0 beta firmware or later) or are using a Blackbird, U-boot will shut down access to this interface after about 3 seconds of standby power, so you will need to run the command _immediately_ after plugging in the power supply to bypass this.
|
||||
4. Be Patient: it took me about 45 minutes to complete the flash process.
|
||||
|
||||
Notes:
|
||||
|
||||
- `gpio_b=S71` and `gpio_a=S70` are used to turn off the fpga watchdog timer before the flash process and then re-enables it after it's completed.
|
||||
- On a Blackbird, replace `gpio_b=S71` with `gpio_b=G01` and `gpio_a=S70` with `gpio_a=G00`. Due to the new HDMI interface, the BMC watchdog GPIO was moved to a different pin on the AST2500.
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
-----BEGIN PGP SIGNED MESSAGE-----
|
||||
Hash: SHA256
|
||||
|
||||
11654 compiling-firmware.md
|
||||
4282 debricking-the-bmc-watchdog.md
|
||||
9879 debricking-the-bmc.md
|
||||
6163 secure-boot-with-your-own-keys.md
|
||||
31978 total
|
||||
-----BEGIN PGP SIGNATURE-----
|
||||
|
||||
iQIzBAEBCAAdFiEE0ZVR9dPnC9rIvgC+tQHDCzf0gGwFAl++u3wACgkQtQHDCzf0
|
||||
gGyr8g//bpLm5zS89HWz0S5TxGzjVXCeQvKRuksxiamFkNi4b61M1ujmc5L5nY6e
|
||||
EME83ewaJ+Vn/nc8YCP58xa35A43yMgM/6jJZavATWpBzp1uQOFdAu4BpEm2NpEd
|
||||
Oed5qtwoDGF6JsCdOgyQjmj2nKrK/5z/nafFMgKE2WGQueXn0wxgsJyAz7ZrXaE2
|
||||
faGge3VoHb6BXl6+nD2RezIO7YZ4lUTshxA2qGplLMr7fc16OsJfMRJrVXoZl+mV
|
||||
kYS/JW8AYPjaOIPrP+ZVu0fOKGYf1nWo8Gw/cKm+RCjgkyrgu0t23NUJ7roNEqRq
|
||||
00sg0GtVk0bghpMyXPzRplm1+kYdF1hM25UVrWXc2+mYtuXFaEiZU4S9DzzmWP+s
|
||||
TfqcP9rJaK4oPARxOUo11QGixwTGZYIiVUJkXgybBbFjgrKFe3oaPBDnyburwtto
|
||||
aE6/Y5Lkmtpa6uEVjmP4N/I1gKI1XhtiTRlbXr54zz7tcF0LAXrHLnrxTI3dPEuN
|
||||
IberZaNdbeJkUJvJCYkWXof4IHLsdUNQC3FN323buoPN09VX2I7DVHXUv9ThNIA6
|
||||
NnB1M7DTLdjRhiS2fi7R7uV4Yg6FliMmGFJsLX9etVzxZMBYQwjgXqv+eSvpA8PI
|
||||
HGTQbapR9uYcqbuZqLzFi/Om9UKTEEhOtjXYPOC2oCpX0T4Olhw=
|
||||
=y1qJ
|
||||
-----END PGP SIGNATURE-----
|
||||
|
|
@ -0,0 +1,132 @@
|
|||
`https://wiki.raptorcs.com/wiki/Secure_Boot_with_your_own_keys`
|
||||
|
||||
# Secure Boot with your own keys
|
||||
|
||||
_Disclaimer: There may be a cleaner way to do this. There may also be a way to speed the process up using partial rebuilds. This guide was created by Fizzbuzz and successfully tested on a Blackbird system using the raptor-v2.00 branch of the blackbird-op-build firmware repository._
|
||||
|
||||
1. Set up a Debian Stretch build environment in accordance with the instructions on the [Compiling Firmware](compiling-firmware.md) page.
|
||||
2. Set environmental variables (for the purposes of this walkthrough and its helper functions) referring to the OpenPower build directory and the directory where you will be placing your firmware signing keys:
|
||||
```
|
||||
export OPBUILDDIR=~/blackbird-op-build
|
||||
export KEYDIR=~/new-keys
|
||||
```
|
||||
|
||||
3. Generate the keys you will be using to secure your hardware:
|
||||
```
|
||||
mkdir $KEYDIR
|
||||
cd $KEYDIR
|
||||
openssl ecparam -genkey -outform pem -noout -name secp521r1 -out hw_key_a.key
|
||||
openssl ecparam -genkey -outform pem -noout -name secp521r1 -out hw_key_b.key
|
||||
openssl ecparam -genkey -outform pem -noout -name secp521r1 -out hw_key_c.key
|
||||
openssl ecparam -genkey -outform pem -noout -name secp521r1 -out sw_key_a.key
|
||||
```
|
||||
|
||||
4. Edit `$OPBUILDDIR/openpower/configs/hostboot/blackbird.config` or `$OPBUILDDIR/openpower/configs/hostboot/talos.config` (depending on which system you have) and remove or comment out the `unset SECUREBOOT` line.
|
||||
5. Follow the [Compiling Firmware](compiling-firmware.md) instructions for building the OpenPower firmware. Run `op-build menuconfig` before the final `op-build` if you want to customize your build in some way.
|
||||
6. Replace the keys and key hashes in the pulled code with your own. I've provided some helper functions for this purpose and described the procedure below.
|
||||
```
|
||||
# Helper functions:
|
||||
|
||||
get_keyhash () {
|
||||
$OPBUILDDIR/output/host/bin/create-container -v -w0 \
|
||||
-a $1/hw_key_a.key -b $1/hw_key_b.key -c $1/hw_key_c.key \
|
||||
--payload /dev/zero --imagefile /dev/null | grep "HW keys hash";
|
||||
}
|
||||
|
||||
check_keys () {
|
||||
while read p;
|
||||
do echo `dirname $p`;
|
||||
get_keyhash `dirname $p`;
|
||||
echo '--';
|
||||
done <<< $(find -name hw_key_a.*)
|
||||
}
|
||||
|
||||
replace_keys () {
|
||||
NEWKEYHASH=$(get_keyhash $KEYDIR);
|
||||
while read p; do
|
||||
OLDKEYHASH=$(get_keyhash `dirname $p`);
|
||||
if [ "$OLDKEYHASH" != "$NEWKEYHASH" ]; then
|
||||
echo "Replacing keys in `dirname $p`"
|
||||
cp -a $KEYDIR/. $(dirname $p)
|
||||
fi
|
||||
done <<< $(find -name hw_key_a.*)
|
||||
}
|
||||
|
||||
check_imprints () {
|
||||
while read p; do
|
||||
echo "$p"
|
||||
cat $p | xxd -p
|
||||
done <<< $(find -name imprintHwKeyHash)
|
||||
}
|
||||
|
||||
replace_imprints () {
|
||||
NEWIMPRINT=$(get_keyhash $KEYDIR | cut -d' ' -f8);
|
||||
while read p; do
|
||||
OLDIMPRINT=$(cat $p | xxd -p);
|
||||
OLDIMPRINT="${OLDIMPRINT//[$'\t\r\n ']}"
|
||||
if [ "$OLDIMPRINT" != "$NEWIMPRINT" ]; then
|
||||
echo "Replacing imprint $p"
|
||||
echo "$NEWIMPRINT" | xxd -p -r > $p
|
||||
fi
|
||||
done <<< $(find -name imprintHwKeyHash)
|
||||
}
|
||||
|
||||
untar_code () {
|
||||
untar_pkg () {
|
||||
cd $OPBUILDDIR/dl/$1/
|
||||
tar xzf $1-*.tar.gz
|
||||
}
|
||||
untar_pkg hostboot
|
||||
untar_pkg libflash
|
||||
untar_pkg sb-signing-utils
|
||||
untar_pkg skiboot
|
||||
untar_pkg pnv-lpc
|
||||
cd $OPBUILDDIR
|
||||
}
|
||||
|
||||
retar_code () {
|
||||
retar_pkg () {
|
||||
cd $OPBUILDDIR/dl/$1/
|
||||
tar czf $1-*.tar.gz $1-*/
|
||||
rm -fr $1-/
|
||||
}
|
||||
retar_pkg hostboot
|
||||
retar_pkg libflash
|
||||
retar_pkg sb-signing-utils
|
||||
retar_pkg skiboot
|
||||
retar_pkg pnv-lpc
|
||||
cd $OPBUILDDIR
|
||||
}
|
||||
|
||||
# Procedure:
|
||||
|
||||
cd $OPBUILDDIR
|
||||
untar_code
|
||||
replace_keys
|
||||
replace_imprints
|
||||
retar_code
|
||||
```
|
||||
|
||||
You can confirm the process worked by running `get_keyhash $KEYDIR` and comparing the hash returned with the hashes returned by `check_keys` and `check_imprints`. They should all match.
|
||||
|
||||
7. Configure the build system to produce a key transition container in the PNOR image. Run `op-build menuconfig` and set `External options -> OpenPower -> OpenPower Packages -> OpenPower PNOR assembly options -> Secure Boot key transition type` to `Transition existing keys to development keys`. "Development keys" in the context of the build system means "keys existing locally on my hard drive." "Production keys" refer to keys on a signing server somewhere for which the build system will generate signing requests if any "Production key" options are set.
|
||||
8. Clear the old build output and re-build. Our aim in building the first time was only to force the build system to populate the `dl` directory with all the source code it would be compiling.
|
||||
```
|
||||
cd $OPBUILDDIR
|
||||
mv output/.config .
|
||||
rm -fr output/*
|
||||
mv .config output/
|
||||
op-build
|
||||
```
|
||||
|
||||
9. Save a copy of `output/images/` to an external drive. Rename the external drive's copy to `transition-images` or something similar.
|
||||
10. Run `op-build menuconfig` again and set `Secure Boot key transition type` back to `None`.
|
||||
11. Repeat the "clear the old build output and re-build" step.
|
||||
12. Save a copy of `output/images/` to an external drive.
|
||||
13. Shutdown the computer.
|
||||
14. Follow the instructions on the [Compiling Firmware](compiling-firmware.md) page to install the `transition-images` PNOR image on your external drive.
|
||||
15. Make sure the "secure mode disable" jumper on your mainboard is set. (See the user manual for your mainboard if you need help locating this.)
|
||||
16. Boot up the computer and then let it shut itself down. If you did not manually clear and set the ECC bits for your firmware during the installation process, the system may reboot itself first to set those bits. Once it is finished, the hash of your own keys should be "imprinted" on the processor's SEEPROM. To confirm this, you can run `cat /proc/device-tree/ibm,secureboot/hw-key-hash | xxd -p` in either the host OS or the Petitboot shell to see what value is currently set in the SEEPROM after replacing the key transition PNOR image with a non-transition one.
|
||||
17. Follow the instructions on the [Compiling Firmware](compiling-firmware.md) page to install the `images` PNOR image on your external drive.
|
||||
18. Make sure the "secure mode disable" jumper on your mainboard is unset.
|
||||
19. Boot up the computer. It should now be booting in secure mode using your signed firmware!
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
-----BEGIN PGP SIGNED MESSAGE-----
|
||||
Hash: SHA256
|
||||
|
||||
3710fe8deac54ac3a444e2146a830ea900ba0e66d6de9fed2f7ba1d97f18daa8287f03dc5c7248fba6ca20c3b157ace7c0dd3e88471d1c4cabb0477f720bf916 compiling-firmware.md
|
||||
b805cb96a7945c3ba46f2dff588f523a2d2fd18614dd15c58a6bcfead6059dd9d825c3bc6950626c7d8705e6d063f064cafd39164b498c74061faafa25355dc7 debricking-the-bmc-watchdog.md
|
||||
a1d10403aa9e02a8115388702c0ad7cbbb93f785ac6859db84595924dfa70a6615e8826e28c2ddd63bd90672a5bf3ddf311d689876768532c2132190ec0e7385 debricking-the-bmc.md
|
||||
3f89eb8b34e71beb6c5208440fee520a3c58cc69e8026d61d3a60f75eef9a32209170ab4e87e68a99de8fc257a32a3dd985e60df64f2a1a7abc9006615255dc0 secure-boot-with-your-own-keys.md
|
||||
-----BEGIN PGP SIGNATURE-----
|
||||
|
||||
iQIzBAEBCAAdFiEE0ZVR9dPnC9rIvgC+tQHDCzf0gGwFAl++u18ACgkQtQHDCzf0
|
||||
gGzmhg/8DBlvpoaViLJRF+55hZhwP4zxUw9maaBT/DkpqcbPioGxecpKpZgNJSR3
|
||||
kVx447zbOHpVhALa/zlFdNWeICx+hL6sLOswfNr4iU+VVYYttjaGt/8ilWUagu0s
|
||||
X0svEzF3NmGypW5SYvPYm2GbJUfwLMUp87PkJ1Wdo1ELdXknBw9guDWrDDoMvdbL
|
||||
17M4+9H15MiO83sCfdPXDtGLSga8AFgYo1I+g+MV6ra6R4xLg+y7qXq5Samcv09/
|
||||
A1r+uyI/e7qsPhkYQ9B6H606Fdslw/ye+7PNVcZCApscPdM67kjtoSHdxMiM5kpW
|
||||
p6wM9QkC8U+IuYmGC5wRwrZxjAICoNBgSycisXgd0wRQf7o98bk3YYX7h1LiTVSI
|
||||
kAn4Tjs3/km/QMs1MjK5xPyEhcdjkJKGwd2aYgjIMiVzaIC7M2LD4r/KwkXLfDb/
|
||||
TNID2RvVry+CPhtk6e7zqY493Q4XNFBrhUOsNSJbQAifORDtdgmyU3EzPHvXiNMW
|
||||
+MIQPsK5rrTtvnqn1oE6nymEkOvW/UFI5dzlbNWf8xkHejz5K5lfpQABuyGdcDG2
|
||||
q+CkUsoXQl9qoDHdrApdROoA7Tm/H665gjJVnwbL4KdCSAJOfMd9ms77NN4Jh8k0
|
||||
L3pCVWY9Q1KtG2Fjln6QT9qkPifySxX9V8cSQ7U72k7l3UBOzmU=
|
||||
=GfDC
|
||||
-----END PGP SIGNATURE-----
|
||||
Loading…
Reference in New Issue