Last time we looked at using the SWD interface of an STM32 ARM chip. This isn’t the only way we can interact with this device though. It also contains a serial interface on the bootloader than allows us to read and write to the flash memory. It’s nowhere near as powerful as the SWD interface.

This example is rather contrived. On a device this small, with pins around the edge, it is unlikely that SWD will be harder to access than serial. Both SWD and the serial bootloader can be “locked” by using RDP (Read Protection). However, we’ve seen many chips where different debug/programming interfaces have different security levels applied. Equally, serial ports are often more obvious or physically accessible than JTAG/SWD.

This is simply to teach some of the principles around reading datasheets, reading schematics, putting a device into another bootmode, and using different tools.

What is a bootloader?

A bootloader is generally the first code to run on a processor. It has the task of initialising hardware and executing the firmware. On complex systems, there will be a significant ROM bootloader permanently stored in the device to call a second bootloader.

On very simple ARM microcontrollers (like the Cortex-M0 we are using) there is no need for anything this complex. The processor looks to the second entry in the “vector table” (the reset vector), and executes the code at this address.

However, STM32 have a ROM bootloader stored in something called “system memory” (STMicroelectronics call it that, not sure why). Using pins and flash options, you can choose different boot modes:

• Execute from flash (normal operation)
• Execute from system memory (which we will look at)
• Execute from SRAM (sometimes used for programming)

The system memory on the STM32F030R8 allows use to use a serial interface to communicate with the device. Some other STM32s allow I2C or USB.

Datasheets Everywhere

With the STM32 range, there are four sets of documents of most interest to us:

• Product Specification (often just “the datasheet”) – this largely concerns the physical and electrical interfaces. The pinouts will be here, and in STM32 ones, the memory map is also here.
• Reference Manual – this contains detailed descriptions of the peripherals and registers inside the device.
• Programmer’s Manual – this contains detailed information about the instruction set of the processor.
• Application Notes – smaller documents concerning the implementation of specific functionality in a device

For the STM32 range, most or all documents are publicly available.

The nomenclature used and the information contained in various documents is totally inconsistent across the semiconductor industry.

Boot Modes

In the product specification, we can see reference to the “Boot Modes”. There is not very much information.

In the reference manual, detail is given on how to go into various boot modes. We want to go into “system memory”. There are two inputs to this function: nBOOT1 bit and BOOT0 pin.

By default, the device is booting from flash memory. The BOOT0 pin must therefore be pulled low.

nBOOT1 bit is “x” which means “don’t care”. We need to check what this is, and if we need to change it. Searching the reference manual for nBOOT1 shows us it is single bit in the option byte.

The default value is 0x00FF55AA – we need to work out what bit 20 is. If you work much with hex, it’s obvious, but it’s useful to be able to use a calculator to do this.

We can type the value into Windows calculator and see the binary value immediately.

Now, we could painfully count out the bits, but by pressing the little “dots” button, the window shows a convenient display.

Bit 20 – nBOOT1 – is set to 1 by default. This is what we want!

(0x00FF55AA clearly has bit 20 set as the byte it is in is 0xFF – all 1s)

Now we need to work out how to set pin BOOT 0.

For this we want to look back to the product specification.

It’s a clearly labelled pin – pin 60. Just a bit of advice – never assume that images in datasheets have searchable text! You might have to scan page after page.

We could use a multimeter to find where this pin goes, but it’s a development board – we have the schematic.

There’s a few things worth noting about schematics.

The two yellow boxes – U5A and U5B – both refer to U5 – the microcontroller. It’s just split into different parts to make the document easier to understand.

The pin numbers are not organised sequentially. Again, they are put into whatever order makes the document easier to understand.

The yellow diamond boxes mean that a signal (or group of signals) are jumping to another page.

BOOT0 – pin 60 – can be seen on U5B. It then leads off to R33 – a 10K resistor. This is a “pull down” resistor – designed to hold the logic level at 0 unless the pin is connected directly to a high level.

Where does the BOOT0 signal go? It jumps to another page.

Pin 7 on connector CN7 – nice! All we need to do is connect this to a high logic level, and the device will enter the system memory. Conveniently, immediately above pin 7 – pin 5 is “VDD”, which is the supply voltage for the processor. We can use one of the spare jumpers from CN11 or CN12 (which are just holders for spare jumpers) to bridge the two.

This is easily tested – your board should be running the blinky LED code by default. Place the jumper on, and press the black reset button. The LED should no longer be blinking as the flash code won’t run.

Finding the UART

The product specification mentions that the bootloader can be accessed on pins PA14/PA15 or PA9/PA10. We want the most readily accessible ones.

Back to the schematic. PA14/15 are brought out on CN7 pins 15 and 17 – adjacent to each other.

Interacting with the bootloader

We have the device starting up in the bootloader, and we know which pins to connect to. Whilst the development board does have a built-in USB serial adapter, it is connected to USART2 on PA2/PA3. It’s not trivial to reconfigure the board, so we will use an external USB serial adapter.

Look back to the post on interacting with serial ports if you can’t remember how to use them. Remember that TX goes to RX and RX goes to TX.

The software we will be using is called stm32flash. Yes, it is SourceForge, and yes, SourceForge is somehow still alive. Download the latest release called “stm32flash-0.5.tar.gz“.

The following should be done to unpack, build, and install it:

tar -zxvf stm32flash-0.5.tar.gz 
cd stm32flash
make
sudo make install

With the STM32 development board powered up, the jumper pulling BOOT0 high, and our serial adapter connected, you should then be able to run:

sudo stm32flash /dev/ttyUSB0

This will query the device for information. If it doesn’t work, try flipping TX and RX.

And now, using other commands, we can read and write the flash.

Conclusion

We’ve used a very similar trick on the LPC5460x series of chips, which can have SWD/JTAG disabled, but you can set the device into a USB boot mode that allows you to read the firmware back over USB. Always check the datasheets for the processor you are working with to uncover “hidden” functionality like this.

STMicroelectronics STM32 processors are some of the most common ARM microcontrollers we see in IoT devices. We often want to read the firmware from these so we can examine it and find any vulnerabilities or hidden functionality. We will use the simple software utility stlink to do this.

Target Device

We will be looking at the Nucleo-F030R8. This is a small development board that is available for under £10 from many sources. The manual is here and the schematic is here.

Why a development board, instead of a real target? They are cheap, consistent and documented. Whilst you are learning how things work

The main microcontroller on it is an STM32F030R8. The datasheet is here. This is an ARM Cortex-M0, the smallest of the ARM cores. There are 64Kbytes of Flash and 8 Kbytes of RAM, and it can run at 48Mhz. As this is a microcontroller, they are inside the same package as the processor itself.

The top part of the PCB has another microcontroller, which is acting as an STLink SWD programmer for the main microcontroller. SWD (Serial Wire Debug) is a debug interface that provides very similar functionality to JTAG. We can read and write memory and interact with the processor.

This STlink be separated from the lower board entirely by snapping cutting across where the break is. Strangely, the physically smaller programmer microcontroller is more capable than the lower one.

We will use this STLink hardware with the stlink software to interact with the device and read and write the flash.

Interacting with STLink

As with the tutorial on using USB Ethernet, we want to pass the STMicroelectronics STM32 STLink through to our Ubuntu VM. VM->Removable Devices->STMicroelectronics STM32 STLink->Connect (Disconnect from Host).

You can then type:

dmesg

And you should see multiple USB devices being enumerated – this is not just an STLink!

Let’s examine what we have by running:

lsusb

You can see the STMicroelectronics device here. It’s on USB bus 003, and is device 003.

Running:

lsusb -t

Will show us a break down of the device. We can see it’s made up of 4 different interfaces:

• Vendor Specific  – the STLink part
• Mass Storage – this is for the built-in Mbed functionality
• Two cdc_acm – this is a USB serial port

We will just be working with the STLink today.

In an earlier post, we built and installed a tool called st-link. These are the Linux tools for interacting with the STLink programmer.

Run:

st-info --probe

And you should receive back information about the development board, including the size of flash and SRAM. Communications are working.

Reading the flash using stlink software

Now we want to read back the flash – this is the non-volatile storage where the firmware is stored. The flash this device is directly memory-mapped – we don’t need to initialise a flash controller before reading it.

The first thing we need to do is work out what address to read and how many bytes. For this, the memory map from the datasheet is of use.

The specific part of that diagram is in the middle. Flash memory is at 0x08000000.

We know this chip has 64KByte of flash – 0x10000. The command we want to write is:

st-flash read firmware.bin 0x08000000 0x10000

You can be lazy and input a large length – st-flash queries the chip and only reads back what is expected.

We now have firmware.bin, the contents of which can he checked with hexdump:

hexdump -C firmware.bin | less

If the data here is all 0xFF or 0x00, then something has likely gone wrong! The default state of erased flash is 0xFF.

Writing the flash using stlink software

There are times when we want to write to the flash memory in a device, say if we have patched the firmware.

The command here takes the flash address and infers the length from the file.

st-flash write firmware.bin 0x080000000

You will notice how much longer this takes than reading. Writing to flash is not trivial – it needs to be erased in blocks first, then specific commands issued to write it. To do this, a “flash loader” is injected by st-flash into the SRAM of the device. This then carries out the programming.

You can see the assembly for the STM32F0 flash loader here. If the flash is more complex – say external SPI flash that is not memory mapped – sometimes you need to do the same simply to read the flash.

Alternatives

OpenOCD can also be used to read and write flash in devices. It’s not a one-liner however, so for now we’ll use stlink.

There is also a utility provided by STMicroelectronics. Unfortunately it is Windows only and requires registration to download.

Although I love working with hardware, if I can avoid hardware attacks, I will. The network interfaces on a device are often all we need to compromise it.

There are three different ways we will need to connect to Ethernet devices:

1. Receiving an IP address from the device via DHCP
2. Setting a manual IP address to communicate with the device
3. Offering an IP address to the device via DHCP

DHCP is Dynamic Host Configuration Protocol. For the purposes of this post, it is simply the way that devices connecting to a network are assigned an IP address as they connect.

Basic Setup

We will be using a USB Ethernet adapter which will be passed through to our Ubuntu VM. This is preferable to using a bridged connection to our built-in Ethernet, as it ensures that the host operating system is not going to interfere. It also allows multiple network connections.

I am using an Amazon Basics USB 3.0 Gigabit Ethernet adapter and a switched Sabrent USB hub (both discussed in this post). Switched hubs allow us to power cycle connected devices without unplugging them. Unfortunately, it is very common to have to disconnect and reconnect USB devices for them to work correctly, especially in VMs.

The most common chipsets for USB Ethernet are ASIX AX88179 (USB 3.0 Gigabit), ASIX AX88772A (USB 2.0 100Mbit) and Realtek RTL8152B (USB 3.0 Gigabit). As far as I know, all are supported natively in Windows and Ubuntu without installing any drivers.

USB 3.0 (or 3.1) support needs to be enable for the VM itself. VM->Settings, then navigate to the USB section and ensure it is set correctly. If you leave this on 1.0, the device will partially enumerate but fail to work correctly – I was once losing random part of packets due to this!

The USB device needs to be passed through to the VM. In VMware Workstation, go to VM->Removable Devices->(name of device) -> Connect (Disconnect from Host). You should get some USB sounds to indicate this happening

Type:

dmesg

And you should see the most recent log entries show the device.

Type:

ifconfig

or

ip

And you should also see the adapter.

Notice the name of the adapter: enx0050b6fff820. The last part is the MAC address of the adapter. In some respects, this is annoying – when you need to type the name of the adapter in (remember – copy and paste!). In other respects, it means you can have multiple adatpers plugged in and not get confused.

If at this stage you are having issues, try power cyling the Ethernet adapter.

Lazy Method of Setting up Connections

I’m fundamentally quite lazy. I’ve found that Ubuntu’s built-in network manager handles the networking setups we need, and does it quickly and efficiently. I’m sure some people won’t be happy with this, but this method works and is easy to remember.

Run:

nm-connection-editor

You should be presented with a window showing two network connections – the VM NAT connection and the new USB Ethernet.

Rather frustratingly, they are not named using MAC address here. Generally, the higher numbered one is the most recently plugged in. Here I select “Wired connection 2” and press the settings (cog) button at the bottom.

Confirm that the “Device” is the same as the MAC address of the adapter.

Now click on the “IPv4 Settings” tab. This is where we can choose which of the three options we want.

Receiving an IP address from the device via DHCP

This is the situation if we are connecting to the LAN side of a router, or simply connecting to a network to test.

This is the default “Method” in Ubuntu – “Automatic (DHCP)”. Change or set it to this.

Ensure that there are no “Addresses” listed below – if there are, delete them.

Setting a manual static IP address to communicate with the device

This will often be used when the device under test already has a static IP address. This is common on DVRs and ICS equipment. It is also used when devices enter recovery mode – a lot of routers will take a manual IP of 192.168.x.1, and you need to manually set yourself to 192.168.x.2 or similar to communicate.

Change the “Method” to “Manual”.

Press “Add” next to the addresses. Add the address. The netmask should automatcially populate – generally a 24 subnet will work for IoT. Unless you want to try and route traffic out, leave the gateway blank.

Offering an IP address to the device via DHCP

Most consumer IoT devices will connect out, and assume that they will receive an IP address via DHCP. We want to act as a router for this traffic, allowing us to intercept and tamper with any communications from the device.

Ubuntu can natively do this. Set the “Method” to “Share to other computers”.

Ensure that there are no “Addresses” listed below – if there are, delete them.

That’s it. Your machine will assume the IP address of 10.42.0.1, offer IPs to connecting devices, and route traffic. Simple.

For all methods

Press “Save”. You don’t need to quit out of the list of connections if you don’t want to.

If you have changed from one connection to another, then you will need to drop the connection and bring it back up. There are two ways I do this.

You can simply unplug the network connection, wait 10 seconds, and plug it back in.

Or run:

nmcli device disconnect enx0050b6fff820
nmcli device connect enx0050b6fff820

Remember that not all IoT devices will request a new DHCP lease if the connection drops – some only do this at startup. We will look at working with Wireshark analyse traffic in a later post to diagnose issues.

In the last post, we setup an Ubuntu 18.04 system for hardware hacking, and used the built-in package manager to install some software we will use.

There are several tools that are better to be installed from source, for a few reasons:

• They will be as up-to-date as possible
• You can modify them and rebuild them if need be.

Organisation

I tend to keep all of my tools in a subdirectory called “tools”. Not everything is installed into the path, and this helps keep everything neat.

Binwalk

Binwalk is a tool used to examine embedded filesystems and extract them for analysis. The one installed using the package manager in Ubuntu and Kali is out-of-date and missing dependencies that are essential. Crucially, it will not unpack JFFS2 filesystems, which are incredibly common.

Installation is easy:

sudo apt install python
git clone https://github.com/ReFirmLabs/binwalk
cd binwalk
sudo ./deps.sh

The deps.sh script will install the packages required, downloads some repositories, build and install them. This is around 350MByte of downloads, so be prepared to wait a bit.

Answer “Yes” to Ubuntu 18 being detected.

Finally run:

sudo python3 setup.py install

Binwalk should now be installed in the path.

Flashrom

Flashrom is a tool used to interact with SPI flash chips. You can use many USB adapters like the CH431A and FT2232H, or a single-board computer (SBC) like the Raspberry Pi or Beaglebone Black.

Note that if you are going to use an ARM-based SBC, you should compile the tool on the device rather than try and cross-compile in on your Intel machine.

I often find that I need to tweak how Flashrom interacts with chips, especially with the proliferation of cloned devices that misreport JEDEC IDs.

Installation is easy:

sudo apt install build-essential libpci-dev libusb-dev libusb-1.0-0-dev libftdi-dev linux-headers-generic
git clone https://github.com/flashrom/flashrom
cd flashrom
make
sudo make install

That should be it!

OpenOCD

OpenOCD is used to interact with devices using JTAG and SWD. Packaged versions tend to be old and not support all the tools.

sudo apt install libusb-1.0-0-dev libhidapi-dev libhidapi-libusb0 libftdi-dev libtool automake pkg-config
git clone --recursive https://github.com/ntfreak/openocd
cd openocd
./bootstrap
./configure
make
sudo make install

ST-Link

Although we can do nearly everything we need to with STM32 processors using OpenOCD, some scripts and tutorials use ST-Link instead.

sudo apt install make cmake libusb-1.0-0-dev gcc build-essential
git clone https://github.com/stlink-org/stlink
cd stlink
cmake .
make
sudo make install
sudo ldconfig

That last step is just to reload the shared libraries, as the make install does not do it.

We need access to Linux tools to be able to analyser firmware and work with hardware. Some of these tools are already in Kali Linux, but many are absent or out-of-date. Because of this, I prefer using Ubuntu. Ideally, we would be using the latest version (20.04), but binwalk is not supported yet.

To VM or not VM

I generally prefer working inside a virtual machine. Being able to quickly reinstall the OS and take snapshots are huge advantages.

However, you’ll need to accept some issues. USB passthrough is not perfect. You’ll get very used to power cycling USB devices to get them to work correctly. I still can’t get full USB3.0 speed on a passthrough port, hence things like the Saleae Logic get used with my host OS.

Main machine or not?

When you are working with hardware, there is always the chance that you damage your machine. This could range from breaking one USB port to wiping out the entire machine. This said, for work on low-voltage devices, I have not had anything worse than a broken USB port.

If you are careless or your main machine is extremely high value, consider using a secondary machine.

USB isolators are available, but they are not affordable for USB3.0.

If we need genuine isolation (e.g. working on mains equipment), a Raspberry Pi and a network connection are a an effective way of keeping safe. Do not work on mains equipment without suitable training.

VM Software

I personally use VMWare Workstation. It mostly works, though you do need to pay.

I have had mixed experiences with VirtualBox, often finding USB performance abysmal. Ensure you install the VirtualBox Extension Pack to get USB 3.0 support.

I have not used Parallels on OSX enough to say much.

VM Settings

I would suggest:

• 4-8GB of RAM
• 40-80GB of disk space – this might seem excessive, but toolchains and firmware unpacking quickly eat it
• Enable USB 3.0 or 3.1 in the settings – both VMWare and VirtualBox simply fail to work with USB 3.0 devices otherwise
• Allow a means of Internet access, most likely NAT – this is for software updates

Getting up and running

You will likely get prompted to upgrade to 20.04 LTS. Decline this.

First, update and upgrade the system.

sudo apt update && sudo apt upgrade

Next, we want to install some tools that help us build software for the machine we are working on.

sudo apt install git build-essential cmake autogen m4

Ubuntu does not come with Python 2.7, and some tools still rely on it:

sudo apt install python

Followed by some common libraries that are used for building tools like flashrom and stlink.

sudo apt install libpci-dev libusb-dev libusb-1.0-0-dev libftdi-dev linux-headers-generic

Some tools that are best install from apt rather than built.

sudo apt install net-tools nmap wireshark hexdiff wget curl hydra minicom meld

And if you want to work with ARM targets:

sudo apt install gcc-arm-none-eabi gdb-multiarch

With these, you should have the packages required to get a lot of software built and a lot of tasks carried out.

In previous posts, we saw how we could identify a serial console on a DVR, connect and interact with it, and – if full shell access was enabled – recover the firmware using a USB stick. But what happens if we can’t get a full shell on the device? What happens if the kernel doesn’t have a serial console enabled?

We can use the U-boot console to dump the data out over serial, and rebuild it into a binary file!

Getting to the U-boot console

You want to connect up a serial adapter, start your terminal emulator (minicom) and watch the screen as the device boots. It’s highly likely you will see a message:

Hit any key to stop autoboot:

There will normally also be a countdown timer for a few seconds. This is U-boot prompting to see if you want to go into the U-boot console. Reboot, and press a key when prompted!

We are now at the U-boot console. The “hisilicon” prompt is because this is a Hisilicon SoC using their version of U-boot.

In a later post we will look at ways of getting into the U-boot even when there is no obvious key sequence, by glitching one of the SPI flash signals.

Supported Commands

Type help to see what commands are supported.

The number of commands supported varies from device to device, but most low-cost DVRs will have a fairly comprehensive list. Unfortunately (for us), U-boot is mostly concerned with copying data onto the SPI flash, whereas we want to copy data from the SPI flash. The USB commands are not of help.

Frequently a protocol called TFTP is supported – Trivial File Transfer Protocol. On most DVRs this allows data to be uploaded and downloaded. Other embedded devices vary; it’s common to find U-boot only allows data to be download to the device.

There is also a command md – memory display, which we can use to read the memory on the device out over serial.

Let’s look at both methods.

Dumping Memory over TFTP

TFTP is Trivial FTP, a very simple file transfer protocol that U-boot often integrates. We can use this to copy data off the device.

Setting up a TFTP server

In Ubuntu 18.04 and 20.04, the package tftpd-hpa sets up a TFTP server:

sudo apt install tftpd-hpa

That’s it. Files that are received by it will be stored by default in /srv/tftp.

Setting your IP address

U-boot stores settings in something called “environment variables”. If you run

printenv

These will be displayed.

You are looking for ipaddr (the DVR’s IP) and serverip (the IP of the TFTP server). These will generally have default values. You can either change the IP of the machine hosting the server, or you can modify the environment variables. To change both, run:

setenv ipaddr 10.42.0.2
setenv serverip 10.42.0.1

Copy the flash to RAM

This device has a 16MByte SPI flash chip. You can look up the size using the part number from the board, or just read it from the boot log.

This means we want to read back 0x1000000 bytes. SPI flash is not directly memory mapped on Hisilicon DVRs, which means we can’t directly access it. We need to copy it into RAM first.

First we need to initialise the flash:

sf probe 0

Now we need to copy the flash into RAM. A suitable address in RAM is nearly always 0x82000000. This command copies 0x1000000 bytes from address 0x0 of the flash into RAM.

sf read 0x82000000 0x0 0x1000000

Copy out with tftp

A quirk of tftp is that you can’t, by default, create files on the server. The easiest way to resolve this is to create an empty file on the server, and let it be overwritten.

cd /srv/tftp
sudo touch firmware.bin
sudo chmod 666 firmware.bin

Note the file has 0 size and is read/write by everyone.

Then, back in the uboot prompt, we run:

tftp 0x82000000 firmware.bin 0x1000000

And now, heading back to the server – we will have the full flash firmware. Simple!

Dumping Memory over Serial

We can use memory display to show us the contents of flash memory.

If you type the follwing command you with get 0x10 (16) bytes of memory from address 0x0.

md.b 0 10

How do we leverage this to extract the flash?

Capture to file

You want to store the output to a file. In minicom, you use:

Ctrl-A L

And choose a filename.

Now we need to dump the RAM:

md.b 0x82000000 0x1000000

This will take some time. Why? Each row is 80 characters long – 80 bytes – and only contains 16 bytes of data. That means for our 16Mbyte flash memory, we need to transfer 80Mbytes over a serial connection! At 115200bps (where each byte takes 10 bits including start and stop bits), that’s just over 2 hours!

This might seem like a long time, but given how safe serial connections are, and how ubiquitous U-boot is, it’s a highly effective mechanism for smaller SPI flash memories. It’s clearly not workable for the 32GByte+ eMMC found in some devices though – that would take months!

Once this is done, close the capture file using:

Ctrl-A L

Trimming the file

Use a text editor to trim anything before and after the memory dump output itself.

Post-processing the file

We’ll use a tool called uboot-mdb-dump to allow us to convert the verbose md output into a binary.

First, make sure git is installed.

sudo apt install git

Now clone the repository

git clone https://github.com/gmbnomis/uboot-mdb-dump

Go into the cloned directory:

cd uboot-mdb-dump

Finally, we can take our capture file (flash.cap in my case), and process it to obtain a binary file representing the flash.

python3 uboot_mdb_to_image.py < flash.cap > flash.bin

We should now have an image of the entire flash, which can be carved up into partitions (details in a later post) or extracted with binwalk.

Unfortunately, serial extraction like this is prone to errors which this tool might not be able to parse. You can generally edit the file manually, maybe using specific md.b commands to correct damaged areas.

Much more advanced work can be carried out with an impressive tool called Depth Charge , which we will cover in another post. The method here just shows the basic process.

Conclusion

That’s another two methods to get firmware from a device. The first is fast, the second slow. The more tools you have in your arsenal to do this, the more effective you will become.

You’ve managed to get a serial console or telnet shell onto a device, but this can be very limiting. You may not have tools like find, grep and strings. Watchdog timers could reset the device. There may only be a very limited amount of storage space. So how do we copy the firmware off a device like a DVR when we don’t have FTP or SSH available?

If you have a USB socket, the quickest solution is to use a USB memory stick.

Suitable USB memory sticks

I’ve found a lot of DVRs dislike USB3 memory sticks. They should fall back to USB2, but this doesn’t always work. Personally, I really like the Sandisk Cruzer Blade drives, which come in packs of 3 for less than £10.

Size is largely irrelevant. The flash is going to be 16Mbytes or so, and the unpacked filesystem less than 512Mbytes. 8GB or 16GB memory sticks are fine.

The drive needs to be formatted in something that the DVR will support. Generally, FAT is the lowest common denominator. Don’t assume EXT2/3/4 or similar – many DVRs have custom file systems for the hard drives.

Getting a shell

We already have a shell via serial onto the Zosi DVR, as root. Telnet, command injection, a dodgy limited reverse shell – whatever. We want to get the files and firmware off.

We could maybe come up with some xmodem type transfer over serial, but it’s going to be painfully slow.

Mounting the USB memory stick

Put the USB memory stick into the DVR. Next, on the device, try running “dmesg”. This should show you if the stick has been seen.

Sometimes, dmesg does not exist. If you have a serial console, these messages may just be dumped out anyway.

Otherwise, you can check /dev/sd* for devices:

Now you want to mount the drive. The following steps are needed:

1. Create an empty directory to mount the drive (/mnt/usb in this case)
2. Mount the /dev/sd*1 drive that showed up before
3. Go into the directory, and test that a file can be written (I used touch)

Copying Firmware

There are different levels at which we can copy the firmware off:

1. The files inside the filesystem
2. The raw flash memory

Both are useful, but the files tend to be lower effort.

File copy

There are some challenges here. The USB memory stick is FAT formatted and will not support all the Linux features like symbolic links. We also have limited tools on the device itself to help us here.

I tend to target individual directories. In this case, the /mnt/mtd/ directory is most interesting – it appears to contain the settings and application.

Most DVRs will support tar, but only the limited busybox version. The following command will put the entire /mnt/mtd into mnt_mtd.tgz in the current directory, which is the USB memory stick we mounted.

tar -zcvf mnt_mtd.tgz /mnt/mtd

There are more efficient ways to do this if you have full tar, but busybox tar does not support directory exclusion. You can create a list of files and provide that to tar.

Raw Flash Copy

There will be some aspects in the flash that you may not be able to see from the OS level, such as U-boot, the U-boot environment variables, and fallback/factory reset images. We can access these at the raw flash level.

First, determine if you can access the flash by running:

ls /dev/mtd*

MTD is Memory Technology Device. You can’t directly tell how big they are using ls.

You can just copy these directly onto the memory stick using:

cp /dev/mtd0 /mnt/usb/mtd0.bin

However, this one-liner is probably easier:

for f in /dev/mtd?; do cp $f /mnt/usb/${f##*/}.bin; done

umount /mnt/usb

Many embedded Linux devices will have a serial port on them, which can allow us access at either the bootloader or operating system level. This can be a powerful tool to the reverse engineer, allowing recovery of firmware, dynamic analysis, and access to other hardware components.

Interacting with the USB serial adapter

I am using Ubuntu 18.04 LTS as my operating system. Ideally, we should be using 20.04, but some tools that are heavily used in hardware hacking (like Binwalk) do not currently work under 20.04. You can work inside a virtual machine.

Windows is a world of pain; it’s probably tolerable on a Mac but I have not tried.

Connect the USB serial adapter to the machine. In a terminal, run the command “dmesg”. This should show you if the device has been recognised. They mostly show up as /dev/ttyUSB0.

You now want to install minicom. This is a terminal emulator that we will use to connect to the DVR. There are alternatives, but I find minicom is the easiest.

sudo apt install minicom

Next up, we want to run minicom. We need to tell it which device we are interacting with. By default, you need to run minicom as root to access serial devices.

sudo minicom -D /dev/ttyUSB0

You should now get minicom running.

On the USB serial adapter, use a jumper wire to connect RX back onto TX. Why? You always need to test your tools and instruments are working before using them in anger. This should loopback what you send, demonstrating that the adapter can both send and receive.

Trying typing. Nothing will happen – our tools are not working! This is because hardware flow control is on, and we need to turn it off.

Press the following keys:

Ctrl-A Z

(as in Ctrl with A, let go, Z)

This shows you the different commands you can carry out. We want to cOnfigure Minicom – so press O.

Scroll down with the cursor keys to “Serial Port Setup”.

Press enter, then press F to turn OFF hardware flow control. Press enter to leave the menu.

Now scroll down to “Save setting as dfl” (default) – this saves the change so we don’t need to make it again.

Finally, scroll down to “Exit”.

You should now be able to type and see the characters come back! Try disconnecting the wire to confirm that it is being carried that way.

By default, minicom is at 115200bps. Nearly all DVRs seem to use this, but if you do need to change it, press

Ctrl-A P

And then you can use A/B to shift up or down the common baud rates.

To exit minicom, do

Ctrl-A X

Finding the Physical Serial Port

For this particular example, I am looking at a Zosi CCTV NVR/DVR. This has a labelled and obvious serial port on the board. In a previous post, we looked at identifying security-relevant parts of a DVR, including serial ports. In a later post, we will look at using a logic analyser to work out if something is a serial port.

The pitch of this connector is not the typical 2.54mm that we’d find. It’s 2mm. We want to connect to it – so how do we do it?

• Get hold of 2mm pitch pin-headers. I recommend that people have a stock of common pitches of pin headers at hand.
• Shoe-horn 2.54mm pin-headers onto the board. This sounds awful, but if you chop the plastic separator apart, they fit very well.
• Solder wires on. This is the best option, but I’m very lazy and would rather avoid it.

In this particular instance, I am not even going to solder the connections on. Push-fit is good enough for my needs today.

Silence is Golden

Nearly all DVRs have a buzzer on them. It gets annoying very quickly. Plug it with some Blu Tack to reduce the volume, or desolder it.

Yes you could copy my fingerprint

Checking Ground

Even though we have a silkscreen showing one of the pins is ground, we should absolutely confirm this. Whilst it is unlikely to cause serious harm, connecting 12V on the DVR to ground on your laptop could have consequences.

To do this, we use a multimeter in continuity mode. This is normally indicated by the buzzer symbol and sometimes needs a button press from resistance (“SELECT” is common).

Never perform continuity tests on a powered board. The continuity test works by testing for resistance – the meter does this by applying a voltage to the board. It will get very confused if there is already voltage present.

Always confirm the meter is working by touching the probes together first.

Find a known ground. The following are good places to look.

USB connector shields – be warned you might need some pressure to break through oxides or coating/

The outer connection on the barrel jack power connector for most DVRs. This is normally on the side of the black plastic socket.

The middle two pins of a Molex power connector for the hard drives (not present on this DVR board).

The top (-) of the coin cell for the RTC backup battery.

The BNC video connection outers (not present on this DVR board as it uses WiFi cameras).

Screw mounts for the PCB to case often have a grounded ring around them.

Pushing needle probes through the solder mask into a ground pour.

Visual inspection can also show which pin is ground. As can be seen on this DVR, the pin marked UART_GND is connected to a large ground pour around it, whereas the UART_TXD and UART_RXD pins have thin signal traces.

Checking Voltage

Serial ports can work at different voltages. We need to make sure we are using an appropriate USB serial interface for the voltage. In order of descending popularity:

• 3.3V – this is typical for a DVR
• 5V – older DVRs may use this
• 12V – some DVRs use this for interacting with external devices
• 1.8V – some DVRs with newer chips may use this.
• RS232 – this is +/-25V – sticking this into a normal 3.3V USB serial adapter will not be good.

The best tool for checking this is an oscilloscope, but we don’t all have them. So let’s use a multimeter and acknowledge the limitations.

Power the DVR. Immediately measure the voltage between ground and TX and the voltage between ground and RX. Then wait a couple of minutes for it to boot, and measure again.

On this board, with this multimeter I see:

• Ground to TX -> 3.29V with some slight fluctuations
• Ground to RX – > 2.36V mostly constant

I can be fairly confident that this is 3.3V logic. Let’s talk through some of what we saw there:

The quiescent state for TX on nearly all serial ports is the high state. If we saw 0V most of the time, this assumption would be false.

The TX signal will have data on it as the device boots. It will fluctuate between 0V and 3.3V. A multimeter only shows an average value, and the mechanism by which it averages is fairly arbitrary. We measure the voltage right at the beginning (before serial output) and at the end (generally, the device will go quiet after a while) to avoid this issue.

We measure both TX and RX because we often don’t know which is which at this stage. They are also frequently labelled incorrectly – TX and RX flip depending on which perspective you are looking at it from!

Why is RX at this strange middle ground of 2.36V? It is highly likely that there is no pull-up or pull-down resistor on the board, leaving it “floating” when not connected.

An oscilloscope would let us observe the high and low voltage.

Connecting the USB serial adapter to the DVR

The device is working at 3.3V, so we can use one of the readily available 3.3V USB serial adapter. Any of these will do.

The USB serial adapter will have the pin-out labelled on it. We want GND, RX and TX.

With both the DVR and the USB serial adapter turned off, connect up the three wires. GND always goes to GND. RX on the adapter goes to TX on the DVR, and TX on the adapter goes to RX on the DVR.

I would recommend always using black for ground wires, just to keep things clear.

If you get RX and TX the wrong way round, nothing bad will happen. You just won’t get any output. If this happens, try swapping the wires around.

I used test clips for this particular situation as they were to hand.

Bringing it all together

Connect the USB serial adapter to your machine. Start minicom as we did earlier.

Turn the power onto the DVR. You should see text very quickly! You can capture this into a text file using:

Ctrl-A L

Choose a file name. Press the same key combination to close the file. We’ll use this to recover the firmware in another post.

The first part is the bootloader – U-boot. Considering the DVR was bought in 2020, U-boot from 2010 might be considered out of date.

You can see that the SPI flash chip has been identified as an MX25L128XX. It’s 128Mbits or 16Mbytes in size.

You might then see a prompt saying that you can press a key to stop autoboot. This is a prompt to get into the U-boot console. You might have to hammer a key repeatedly to get into the console. This isn’t the operating system – this is the bootloader before that stage. You can still read and write the flash, which we will use in another post to recover the firmware.

If you don’t go into this console, the device will carry on booting.

Many DVRs stop immediately after U-boot, as the serial console is disabled in the kernel. Not this time. We see normal system logs fly past, and then what looks like logging from the DVR processes:

Don’t make the assumption that you don’t have control when you see this. A process might be outputting to the console, but that doesn’t mean you can’t provide input.

Yep, we are straight in as root, no password. The annoying DVR logging sticks about though – I think it’s actually partially built into the kernel, making it impossible to suppress.

And, with a bit of exploration, we quickly find a file called config.dat in /mnt/mtd/config/ that contains the admin password as plaintext.

Not great for a new DVR!

Conclusion

Hopefully, you have seen how easy it can be to get a serial console on a DVR. In this case, we have full OS-level access.

In the next post, we’ll look at a quick way of recovering files and firmware from a DVR to analyse on our own machines.

DVRs (Digital Video Recorders) are interesting targets for hardware hacking. They are generally powerful Linux machines with multiple network interfaces and have historically been full of vulnerabilities. 

Identifying the security-relevant components on a DVR board make it far easier to quickly attack and gain access to the various interfaces: serial ports and SPI flash memory being the most important. We can also quickly isolate areas of the board that are not of interest; power supply, RAM, video ADCs, HDMI etc.

Disassembly and Powering

I would recommend removing the main PCB from the case to make access and inspection easier. Nearly all DVRs run on 12V, with a few running on 48V. Less than 120VDC and 50VAC is classified as SELV (Separated Extra Low Voltage) in the UK and is safe to work on in electrical shock terms. These low-voltages are still capable of causing heat, smoke and fire if shorted.

You do not need to have the hard drives connected when working on the DVRs. For 99% of DVRs, the drives only store footage, no operating system or configuration. Disconnect them, especially if you want to recover footage from them.

For nearly all 12V DVRs, a barrel jack power connector is used.

You can use the provided wall-wart style power supply but be warned that the quality and safety of many is in question. You can use a better quality power supply, but check that:

• The voltage matches. This should be a direct match – a 12V power supply for a 12V device.
• It can supply enough current. The replacement should be able to supply more current e.g. a 2A supply is suitable to replace a 1A supply. A DVR without drives will generally only consume a fraction of the power needed during normal operation.
• The polarity matches. Nearly all DVRs are centre positive. You can check this by looking for the pictogram on the original power supply.

The power supply below is 12V, maximum 2000mA (2A), DC (the straight line above dots) and centre positive.

Personally, I use a bench power supply when working on them, as this allows me to limit the amount of current provided. Simply chop the barrel jack off the end of an existing power supply and use this.

Many DVRs will have a heatsink on the main chip. This can make finding out the part number difficult, or finding out where pins go to. You can often remove it with careful levering. Generally, because we will be running the DVR without video feeds, the chip will not get hot enough to need a heatsink. As a rule of thumb, if you can’t touch the chip for more than a few seconds, it is too hot.

High-level Overview

Many of the low-cost DVRs are based around a Hisilicon chipset. These are large, powerful ARM SoCs (System-on-Chips) containing a wide array of peripherals.

SPI Flash

They tend to use SPI (Serial Peripheral Interface) NOR flash memory for storing the firmware and configuration. These range from 8Mbytes to 64Mbytes in size. This is a particularly convenient flash memory format for the reverse engineer.

Firstly, most of them are packaged in 8-pin SOIC (Small Outline Integrated Circuit). This is a comparatively large package, with the pins being accessible and 1.27mm apart. This makes physical access to the signals easy.

Secondly, the SPI interface is very simple. There are only 4 signal wires:

• DI (Data In) or MOSI (Master Out, Slave In)
• DO (Data Out) or MISO (Master In, Slave Out)
• CS (Chip Select)
• CLK (Clock)

As with all interfacing, we will need to find a common ground. Sometimes we also need to provide power as well. Regardless, this is a small number of signals, especially compared to parallel NAND flash which needs more than 14 connections on a finer pitch package.

Thirdly, although SPI only technically allows a single bus master, we can bend the rules and add another master to directly interact with the flash memory.

Fourthly, NOR flash does not suffer from bad blocks as NAND flash does. The data we read out from the flash will be the same as seen on the filesystem. There is no overhead to handle bad blocks.

Serial Ports

The majority of DVRs will have a serial port on them. These are sometimes called UARTS or USARTS (Universal Synchronous Asynchronous Receiver/Transmitter). This will be a conventional asynchronous serial connection with receive (RX), transmit (TX) and ground (GND). This serial console will often allow access to the bootloader, and sometimes access to the Linux console itself.

RAM

Nearly all DVRs copy data from flash into RAM before executing it. This is why the flash is small (~16Mbytes) and the RAM large (>256MBytes). It would be desirable for us to access the RAM to inspect the OS and see any data the device is processing.

Unfortunately, on these devices, the RAM is DDR3 running at 800MHz or above, using a parallel bus. There is simply no low-cost or accessible means to intercept these signals.

JTAG

It is likely that some DVRs have JTAG on them, but I have had very limited success in interacting with a DVR using it. It would allow flash access, RAM access, and the ability to debug.

Typical DVR

This is a Sansco DVR bought from Amazon in 2020. It is representative of a typical DVR.

In red is the large Hisilicon Hi3520 SoC. I have removed the heatsink, but not the compound holding it on. In blue is the DDR3 RAM. In green is the SPI flash.

Sansco DVR

The PCB will generally be 4 layers or above. This means that there are layers inside the PCB that you cannot easily see. It is possible for these to carry signals across the board. Most interesting signals will be carried on the surface, with the inner layers reserved for power and ground.

All PCBs will have a solder mask, which is the green layer. This is used to stop solder sticking to places it shouldn’t be. You can generally see traces through the solder mask, and you can push needle probes through this layer to reach traces below.

Most PCBs will have a silkscreen. These are the white letters showing names of components and connectors. The most common letters are:

• R – resistor
• C – capacitor
• L – inductor
• CN – connector
• U – semiconductor
• T – transistor
• X – crystal

Looking at the Hisilicon SoC in more detail, you will note it has pins all around the outside. This is an LQFP package. The signals being accessible around the edge can be extremely helpful when reverse engineering. It has a very fine pitch of 0.4mm – you will struggle to clip or solder to these.

Some DVRs have chips in BGA packages, with the connections under the chip. This can slow down reverse engineering.

To identify pin 1 on a chip such as this, you tend to look for a dot or dimple. You’ll see there are three on this chip – generally, it’s the smallest and deepest one. You can also see a “• 1” marked on the white silkscreen. The logo and text orientation to not have any relevance as to pin 1 – it’s arbitrary.

Next up is the RAM. This will nearly always be in BGA package, physically close to the SoC. There will be a lot of parallel traces going to it, some of which may be “squiggly”. These are called matched-length traces, and are often indicative of RAM. They have to be the same length to ensure that the extremely fast signals arrive at the chip at exactly the same time.

DDR3 RAM

Onto the SPI flash. This will nearly always be in 8 or 16-pin SOIC packages. Part numbers with “25” in them are common, as are binary multiples such as 16, 32, 64, 128, 256 (indicating the size). Manufacturers include Winbond, Spansion, Macronix, and Atmel. The pin-out of the chips is nearly always the same, and the protocol used to communicate with them is very similar. In this case, we have a device made by a smaller company I don’t recognise.

We actually determined that this was the SPI flash partially by elimination. The brand (XMD? XMC?) and part number did not yield good results on google.

The only other 8-pin chips were a small memory (setting and MAC address) and switch-mode power supply ICs. The small memory is an FM24C08D – a 2-wire (or I2C) serial EEPROM, only 8kbytes in size. Not big enough for the firmware!

Serial EEPROM

Nearly all DVRs will have an RTC on them. This is to keep accurate track of the time, even if the power is cycled. We can quickly identify if one is present by two components.

First, they will have a coin cell to provide battery power. Some use different forms, but this is by far the most common.

Coin cell

Second, they will have a 32.768kHz crystal. Why this odd frequency? If you divide it by 2^15, you get one pulse per second – ideal for timekeeping! These crystals are nearly always small cylindrical silver cans.

There will be other crystals to drive other chips – normally the SoC has one at 24MHz, in an oval-shaped can.

32.768kHz crystal at top and 24MHz at bottom.

Next up is the video ADC (Analog to Digital Converter) or RX (Receiver). There will be one or more of these chips on the board, taking analog video and digitising it. There tends to be one chip for every 4 channels. This is in a QFN (Quad Flat No-leads) package, which means you can just about access the signals around the edge. This part of the board is generally not of security interest though.

Video RX or ADC

Nearly all DVRs will have an Ethernet port. Ethernet is isolated, using a small set of transformers. Often there is a large package near to the port containing these transformers. It is not a chip and is not interesting in security terms.

Ethernet jack and magnetics

The area near to the power input will be the power supply. There will generally be a switch-mode power supply to drop the voltage down from 12V to 5V, then linear regulators for the 3.3V for the SoC. Again, this has little security relevance to us.

Power supply area

We can quickly isolate power supply areas by looking for certain components. 4-pin packages with one very large tab are normally linear regulators. The large tab allows heat to be conducted onto the PCB to keep it cool.

Regulator

Large cylindrical capacitors are called electrolytics, and are power reserves, used for smoothing.

Electrolytic capacitor

Finally, small coils of wire are inductors, very common in switch-mode power supplies. They will be marked with an “L”

Inductor

Finding serial ports is key. I’ve found that for most DVRs, this will be 3 pin – GND, RX and TX. There is also sometimes a voltage supply as well.

On this DVR it was a 3-pin connector. You can easily make out the ground – it’s the left, pin, connected to the large “ground pour” on the PCB. The other two pins have very fine traces, leading away – likely signals.

Find all the suitable candidates for serial ports when performing an overview of the board. We’ll look into working out if they are serial and how to connect in a later post.

Serial port

Conclusion

I hope that’s helped in understanding the major components on a DVR. This can be applied to most small embedded devices, but many routers and other IoT now use different forms of flash memory such as eMMC.

If you are starting out in hardware hacking, then you need a toolkit. For some aspects of this, it’s worth spending good money for a quality tool. For other parts, low-cost alternatives can be better.

This post details what I would consider the absolute starting point.

Multimeter

Your multimeter will be one of your most commonly used tools. Most of the time, you will be in continuity or voltage mode. Sometimes you will be measuring resistance. Rarely will you measure current. Frequency, capacitance, and duty cycle are virtually useless for embedded systems.

You want a responsive and loud continuity buzzer – ideally a “latching” one. This is virtually impossible to determine from specifications – YouTube reviews will often cover this.

From a usability perspective, it is desirable that the meter has a dedicated continuity mode or stays in continuity mode through power-off. Many meters require a button-press to go from resistance to continuity mode, which leads to using the meter in the wrong mode.

An auto-ranging meter generally makes work quicker, as you don’t need to change the range. Be warned though, some cheap meters have incredibly slow auto-ranging, which becomes frustrating quickly.

Separate current and voltage terminals are essential. Some meters share a single terminal for both, with the function being changed by a selector dial. This changes the meter from high-impedance (>1MOhm) to low-impedance (<1Ohm), which can lead to mistakenly shorting out power supplies.

In terms of resolution, accuracy, and precision, we generally aren’t bothered as long as the meter isn’t truly shocking. Anything with less than 4000 counts may not be adequate.

Multimeters have varying levels of safety. Unless you know the meter is a genuine one from a reputable manufacturer (Fluke, Brymen, Agilent/Keysight, Amprobe etc.), be extremely cautious using them on mains voltages or in high-current situations.

This isn’t the place to be discussing the difference between CAT II/III/IV meters, but if you are working on mains installations, I would strongly recommend following the route electricians have: two pole testers that simply cannot measure current.

My recommendation is the Brymen BM235. This is a reliable and accurate meter that does nearly everything you will need. If you think you will be in this game for the long run, I’d really recommend getting a decent meter.

Multimeter Probes

The probes that come with your meter will likely have large, relatively blunt tips. In my opinion, a major upgrade is switching to needle probes.

There are very fine probes which are as sharp as a needle. They make it easy to probe individual pins on fine-pitch semiconductor packages. Because they are so sharp, they allow you to break through oxides, conformal coating, and solder masks easily.

The image below shows a 0.5mm pencil, a needle probe, and the sharpest normal multimeter probe I have. There is a very clear difference.

I recommended Pomona 6275 for this. These ones come with stainless steel tips, which are very sharp and robust, but at the cost of slightly increased contact resistance.

Be warned, this can only carry 3A at 60V. They are not suitable outside electronics.

USB-to-serial adapters

Connecting to serial consoles is extremely common. I’m a big fan of cheaper USB-to-serial adapters based on the CP2102/4, FT232R, and PL2303. There’s very little to differentiate between them in functionality. Some support 5V and 3.3V operation, but nearly all of the time 3.3V will be what you need.

These are the kind of devices that you lose, get borrowed, or break. You can order packs of 5 from Amazon for around £10-12. However, you can also buy them for around $0.70 from Aliexpress, which works out to around 70p each with postage.

USB Ethernet adapter

You are likely going to end up working with Ethernet connected devices, and want to pass traffic through to a virtual machine. Rather than bridge to the physical adapter, I find it is far more versatile to use a USB Ethernet adapter.

I like the Amazon basics ones. They work well, the link and activity lights are exposed, and they have the MAC address printed on the back of them.

It is also worth sourcing a USB 2.0 Ethernet adapter – these have wider support on older OS and systems. You would be surprised how many devices offer up an unfiltered network interface when you plug one in! I have several of the Plugable ones from Amazon that have lasted well.

USB WiFi adapter

For WiFi connected devices, you will want to make a hotspot on your machine. For this, you need a USB WiFi adapter that can act as an access point.

I have a stock of TP-Link TL-WN722N, but only the V1 is suitable for this. The readily available Alfa devices are suitable as well.

USB Logic analyser

A logic analyser allows you to monitor multiple digital signals at the same time. There are a lot of options on the market here.

Generally, when reverse engineering, you are looking at signals over a long period of time. SPI, serial, I2C. You favour sample length, convenience and the ability to quickly export data to files for post-processing.

It’s for this reason that I prefer USB logic analysers. They facilitate reverse engineering. Dedicated instruments with screens are more useful for timing issues and lots of parallel channels.

Professionally, we all use Saleae Logic Pro 16. They are, however, expensive. The analog inputs, whilst helpful from time-to-time, are not strictly required.

At the opposite end of the market are the low-cost Hobby Components devices. 8 channels are adequate for a lot of work, but the 24MHz sample rate can be limiting when dealing with faster signals. They often struggle to get even 10MHz on many laptops. Adequate for serial ports, but not SPI.

A good middle-ground are the Kingst range of analysers. The LA1010 is available for around £70 from Amazon or $40 from Aliexpress.

USB Hub

With all of these USB devices, you are going to want a USB hub. Not only do these give you more ports, but they provide a degree of isolation between your laptop and the device you are working on.

The Sabrent switched hub from Amazon is great. This has individual power switches per-port, allowing you to quickly power cycle devices. If you have ever worked with USB in a Linux VM, you will quickly see why this is desirable!

Making Connections

The part missing from this post is making connections to devices. They don’t have pin headers fitted all of the time. That’s coming up.