A PCI/PCI-Express Primer Part 3: Let’s Play

A PCI/PCI-Express Primer Part 3: Let’s Play

 

Note: This is part three of a three-part blog series. Part one is available here and part two is available here.

Now that we understand the basics, it's time to see what the interface looks like in the real world.

In the old days, one would need to open the entire /dev/mem file and seek the defined offset in order to interact with the BAR.

The BARs informations are shown by the lspci tool with the verbose flag:

Figure 1: lspci BARS info

In this example, 4 BARs were declared: three in memory at offsets 0xdc000000, 0xb0000000, 0xc0000000, and one in I/O space at 0xe000 with the various attributes as discussed above.

Nowadays, modern Linux distribution make the BARs directly accessible in the /sys tree. You may prefer this method if the /dev/mem device is not accessible/present in your environment.

BAR0 => /sys/bus/pci/devices/[device ID dir]/resource0
BAR1 => /sys/bus/pci/devices/[device ID dir]/resource1
BAR1 => /sys/bus/pci/devices/[device ID dir]/resource1_wc
BAR3 => /sys/bus/pci/devices/[device ID dir]/resource3
BAR3 => /sys/bus/pci/devices/[device ID dir]/resource3_wc
BAR5 => /sys/bus/pci/devices/[device ID dir]/resource5

The Linux documentation does not really indicate what the resourceX_wc corresponds to, but an educated guess would be that this file maps to the prefetchable version of the BAR.

Once you have identified which file you need, mmap it into your process memory, keep the initial address pointer for the region, and operate on it directly via memory operations, within the limit of what is permitted by the BAR attributes (Type and Prefetchable).

Beyond this, the standard does not dictate what the BARs should contain. This is where the vendor documentation (or lack thereof) comes into play.

The BARs are very good targets for fuzzing, but be careful. If the BAR expose functionalities that controls things like card voltage or thermal cutoff, you might be in for a wild ride!

Fuzzing BARs may also trigger very subtle bugs in the kernel itself that are not easily detectable or that only disturb/crash the system much later on.

Something to keep in mind as well is that the entire BAR space is not necessarily functional (i.e., some registers area of the BAR are not linked to anything) and will return seemingly random value when read and/or have no effect when written. For example, NVidia devices will return a value of the form 0xBAD0XXXX if the address register is not used, and no amount of write will ever change this value.

Among the observed behavior during fuzzing BARs, kernel crashes are usually the most frequent but carry a low risk rating. This is because if you have access to the BARs, you are already root, and root has much simpler ways of crashing a system (memory or PID saturation, for example). Although they should not be discarded, crashing a system as root will not make many vendors panic.

This is another story if the system is shared between customers using a virtualized environment, which means that the PCI BUS is going to be shared as well. In this case, a hardware crash has a high probably of crashing the hypervisor and shutting down other customers’ VMs. That could be a direct hit to the core business bottom line, something our customers takes very seriously!

Once you have exhausted the BARs testing, it is worth noting that the PCI specification details a common set of functionalities that are frequently used across all extension cards. Instead of leaving their definitions to each vendors, they are explicitly described by the standard.

Capabilities and Extended Capabilities

Capabilities are basic functionalities that are explicitly defined in the PCI standard.

Extended Capabilities are similar to Capabilities, but are defined in the extended Configuration Space thus only relevant for PCI-e devices.

Both are stored within (relatively) simple linked-lists, as described by the standard. The Capabilities list starts at the Configuration Space offset value defined in one of the field of the PCI header (located at 0x34 in the Configuration Space), whereas the Extended Capabilities list simply starts right after the PCI header (offset 0x100 in the Configuration Space).

These offsets can be shown by the lspci command with the verbose flag:

Figure 2: lspci Capbilities info

The offsets are given between square brackets in hexadecimal. Each Capability is described in detail within the standard and, although many of them contain read-only registers, some may be able to modify the hardware configuration. These are usually designated as Control Registers (as opposed to State Registers).

Capabilities worth looking at include (but are not limited to):

• Power Management (sounds a bit dangerous)
• ACS Extended Capabilities (Access Control at the PCI bus level)
• Resizable BAR Capability (Dynamic resizing of the target BARs, may allow access to non-default memory regions from the extension card memory)
• Vendor Specific Extended Capability (as defined by the vendor, usually interesting)

Once again shutting down or damaging the hardware using the Power Management capability of your laptop has limited value, but the game changes in industrial environments where some extension cards costs thousands of dollars each, and making them vulnerable to local attacks may have serious consequences for the business.

This is once more amplified by the 'Cloud Principle', where hardware is shared among multiple customers.

Summary

We have looked over the basics of in-memory register access and write, and took a deep dive into how PCI/PCI-e cards expose their functionalities to user space.

In principle, this is not really difficult as long as you have access to the PCI specifications and comprehensive vendor documentation. However, given the numerous functionalities exposed by the PCI standard when coupled with vendor specifications, this can quickly turn into tedious work if you have to write the interfaces by yourself.

Additionally, we haven't talked about the I/O space BARs but they work similarly as the Memory Space, the main difference being that each I/O address is much closer to a simple cable, which adds some complications due to timing constraints. Having said that looking at the wording in various documentations, it looks like they are trying to make them obsolete.

If you are tasked with such tests, be sure to ask for both the PCI-Express specifications, if you don't have them already, and the vendor’s documentations describing exposed functionalities in detail. When looking at the latter, be sure to check the undocumented registers for their liveliness—this could lead to some interesting discoveries.