The Wonderful World of MIPS
INTRO
MIPS is a Reduced Instruction Set Computer (RISC) that has been around since the early 1980's and it is commonly used in residential gateway routers and switches. You can see why malware authors would want to target MIPS based devices. Today we will demonstrate how to compile, run, analyze, and debug an example MIPS program written in C on an x86 system. While the code we are using is not malware, the techniques and methods in this article will allow you to analyze MIPS based binaries/malware.
CROSS COMPILING MIPS ELF ON x86
We will be using Kali Linux 2018 for our base VM. In order to compile our own MIPS ELF, we will need to install a few dependencies
apt-get install linux-libc-dev-mips-cross libc6-mips-cross libc6-dev-mips-cross binutils-mips-linux-gnu gcc-mips-linux-gnu g++-mips-linux-gnu
Today we will use some sample C code that prints out the current PID and PPID as well as the current iterations of a WHILE loop. It will also perform FORK and again print the PID and PPID of the current process. You will notice after the FORK there will be two outputs: one from the parent process and one from the child. It is important to practice debugging FORKs in MIPS because debuggers have a tricky time dealing with them (CLONEs as well.)
Here is the example code we will use for compilation. Be sure to save it with a name like mips-test.c or something similar.
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
int main(){
printf("PID: %d\n",getpid());
printf("PPID: %d\n",getppid());
int i = 0;
while(i < 4){
sleep(1);
printf("ITERATION #%d\n",i);
i = i + 1;
}
fork();
for (int w = 0; w < 10; w=w+1){
printf("%d: %d\t%d\n",getpid(),getppid());
sleep(1);
}
printf("EXITING: %d\n",getpid());
return 0;
}
We can now utilize the following command to cross compile our code to MIPS-Little Endian:
mipsel-linux-gnu-gcc -xc -static -o mips-test.elf mips-test.c
You should now have a file named mips-test.elf. You will need to CHMOD +X mips-test.elf in order to run it. However, if you try to run the file you will encounter and error stating that the architecture is not supported on your machine. In the next section we go through how to run the MIPS ELF on your x86 machine.
RUNNING MIPS ELF ON x86
Again we start by retrieving dependencies:
apt-get install qemu
apt-get install bridge-utils
QEMU is an emulator which allows you to run many different architectures within your host machine and bridge-utils will be needed to enable network functionality later on.
QEMU has two modes: User Mode and Full System Emulation.
QEMU USER MODE
User mode allows us to run a single binary natively through the command line (like the one we just compiled) using the following command:
qemu-mipsel [-strace] [-g 12345] mips-test.elf
To run the binary as-is disregard the -strace and -g 12345 options of the command line. Simply use qemu-mipsel mips-test.elf.
The option -strace outputs the binaries' system calls to the terminal. This can be helpful to get a quick understanding of what the binary is doing.
The -g 12345 option tells QEMU to open the ELF with a GDB Stub loader listening on port 12345. This pauses the ELF at the entry point and waits for a GDB session to be established. We will discuss how to attach to this GDB Stub loader later in the article.
QEMU SYSTEM EMULATION
Get ready for an epic adventure. To start, we will need to download a couple files from:
https://people.debian.org/~aurel32/qemu/
Specifically for this demonstration you will need the MIPS Little Endian flavor of Debian and its corresponding kernel image.
https://people.debian.org/~aurel32/qemu/mipsel/debian_wheezy_mipsel_standard.qcow2
https://people.debian.org/~aurel32/qemu/mipsel/vmlinux-3.2.0-4-4kc-malta
Once you have downloaded these two files you will need to create a shell script in the same folder. You can name it start.sh or something similar. The script will look like this (be sure to modify the path variables at the top to correspond to where you put your files):
#!/bin/bash
qemu=qemu-system-mipsel
path="$HOME/Documents/Debian-MipsEL"
hda="$path/debian_wheezy_mipsel_standard.qcow2"
kernel="$path/vmlinux-3.2.0-4-4kc-malta"
iface=tap0
echo "Stopping eth0, starting tap0"
/etc/qemu-ifup tap0 || quit 1 "Failed to start tap0"
echo "Starting Debian MIPS"
$qemu -net nic -net tap,ifname=$iface,script=no,downscript=no \
-M malta -kernel $kernel -hda $hda -append "root=/dev/sda1 console=tty0" -nographic
QEMU automatically creates the file /etc/qemu-ifup which is supposed to enable networking for the emulated MIPS system, however, I have had many issues with this, so we will be using a modified version for this demonstration. Feel free to backup the current qemu-ifup file because we will be replacing it's contents with the following code (WARNING: These commands work on Kali 2018.1. Be sure the variables at the top are being configured correctly for your distro. ):
#!/bin/bash
ETH0IPADDR=$(ifconfig eth0 | grep inet | cut -d: -f3 | awk '{print $2}')
GATEWAY=$(ip route | awk '/default/ { print $3 }')
BROADCAST=$(ifconfig eth0 | grep broadcast | cut -d: -f3 | awk '{print $6}')
USER=$(whoami)
# First take eth0 down, then bring it up with IP address 0.0.0.0
/sbin/ifconfig eth0 down
/sbin/ifconfig eth0 0.0.0.0 promisc up
# Bring up the tap device (name specified as first argument, by QEMU)
/usr/sbin/openvpn --mktun --dev $1 --user $USER
/sbin/ifconfig $1 0.0.0.0 promisc up
# Create the bridge between eth0 and the tap device
/sbin/brctl addbr br0
/sbin/brctl addif br0 eth0
/sbin/brctl addif br0 $1
# Only a single bridge so loops are not possible, turn off spanning tree protocol
/sbin/brctl stp br0 off
# Bring up the bridge with ETH0IPADDR and add the default route
/sbin/ifconfig br0 $ETH0IPADDR netmask 255.255.255.0 broadcast $BROADCAST
/sbin/route add default gw $GATEWAY
Next we will need a way to undo this bridged network, so we will edit the /etc/qemu-ifdown file and replace it's contents with the following code:
#! /bin/sh
# Bring down eth0 and br0
/sbin/ifconfig eth0 down
/sbin/ifconfig br0 down
# Delete the bridge
/sbin/brctl delbr br0
# Bring up eth0 in "normal" mode
/sbin/ifconfig eth0 -promisc
/sbin/ifconfig eth0 up
# Delete the tap device
/usr/sbin/openvpn --rmtun --dev $1
Now we are off to the races. To start your emulated MIPS system all you need to do is run your start.sh script (don't forget to CHMOD +X). Bootup takes a bit, but if everything went correctly you should be greeted with a login prompt for your MIPS Debian emulated system with user/pass being root/root:
Your emulated system should be fully functional and have networking capability. You can verify this with a quick ping to Google's DNS ping 8.8.8.8. Hopefully you receive some TTLs.
Now, to get files in/out of your newly created Debian MIPS instance, I found that python is the easiest method. On your Kali host VM (not in the Debian MIPS machine) navigate to the location of your compiled mips-test.elf file. Now type the following command in the terminal:
python -m SimpleHTTPServer
This will start an HTTP server in the folder where your mips-test.elf is. Now go back to your Debian MIPS machine and type the following command:
wget http://[YOUR KALI ETH0 IP HERE]:8000/mips-test.elf
This will download the mips-test.elf file to your Debian MIPS system. Perform and LS and you should see the file. CHMOD +x mips-test.elf and run it. Tada! Easy file transfer and now you are running a cross compiled MIPS Little Endian binary on an emulated Debian MIPS Little Endian system inside of a virtualized Kali Linux OS with a base machine of Windows/Linux/Mac. We heard you like virtualized emulation so we emulated your virtualized emulator's emulations. :)
To restore networking functionality to your Kali Machine run the following command:
/etc/qemu-ifdown tap0
DEBUGGING MIPS ELF ON x86
We finally made it to debugging. Our system should now be set up to compile our own MIPS binaries and run them singularly using QEMU User Mode or natively in an emulated Debian MIPS system using QEMU System Emulation.
Before we continue, lets grab a couple more tools called radare and gdb-multiarch. Gdb-Multiarch is a version of GDB that supports multiple architectures. Radare is a fantastic tool that allows disassembly and debugging of multi-architecture files. Use the following command to retrieve the latest version from GitHub (Kali already comes with Radare, but it is very old):
apt-get install gdb-multiarch
git clone https://github.com/radare/radare2
cd radare2/sys
chmod +x install.sh
./install.sh
Now that Radare2 and GDB-Multiarch are installed we have a few options to proceed with debugging. Each one of these methods has pros and cons and you will need to experiment with each to find the method that works best for your situation. I will demonstrate the commands needed to launch each instance and how to attach the debugger of choice.
1.) QEMU USER MODE, GDB STUB, AND GDB-MULTIARCH
From your Kali Terminal:
qemu-mipsel -g 12345 mips-test.elf
gdb-multiarch
target remote 127.0.0.1:12345
2.) QEMU USER MODE, GDB STUB, AND RADARE2 GDB MODE
From your Kali Terminal:
qemu-mipsel -g 12345 mips-test.elf
radare2 -a mips -b 32 -D gdb -d gdb://127.0.0.1:12345
e dbg.follow.child=true
e dbg.forks=true
3.) QEMU USER MODE AND RADARE2 NATIVE DEBUGGER
From your Kali Terminal:
radare2 -a mips -b 32 -d dbg:///mips-test.elf
e dbg.follow.child=true
e dbg.forks=true
aa
s sym.main
af
VV
4.) QEMU USER MODE AND IDAPRO REMOTE GDB ATTACH
From your Kali Terminal:
qemu-mipsel -g 12345 mips-test.elf
From your machine running IDAPRO:
select the remoteGDB debugger from the drop down debugger menu
specify the IP of your Kali machine and Port 12345
5.) QEMU SYSTEM EMULATION, GDB-MULTIARCH NATIVE COMPILED
From your Kali Terminal:
./start.sh
From your Debian MIPS Terminal:
apt-get install gdb-multiarch
gdb-multiarch mips-test.elf
6.) QEMU SYSTEM EMULATION, RADARE2 NATIVE COMPILED
From your Kali Terminal:
./start.sh
From your Debian MIPS Terminal:
wget https://radare.mikelloc.com/get/2.4.0/radare2_2.4.0_mipsel.deb
dpkg -i radare2_2.4.0_mipsel.deb
radare2 -a mips -b 32 -d dbg:///mips-test.elf
e dbg.follow.child=true
e dbg.forks=true
aa
s sym.main
af
VV
7.) QEMU SYSTEM EMULATION, GDBSERVER, REMOTE GDB-MULTIARCH
From your Kali Terminal:
./start.sh
From your Debian MIPS Terminal:
gdbserver 127.0.0.1:12345 mips-test.elf
From your Kali Terminal:
gdb-multiarch
target remote [Debian MIPS IP]:12345
8.) QEMU SYSTEM EMULATION, GDBSERVER, REMOTE RADARE2 GDB MODE
From your Kali Terminal:
./start.sh
From your Debian MIPS Terminal:
gdbserver 127.0.0.1:12345 mips-test.elf
From your Kali Terminal:
radare2 -a mips -b 32 -D gdb -d gdb:///[Debian MIPS IP]:12345
e dbg.follow.child=true
e dbg.forks=true
aa
s sym.main
af
VV
9.) QEMU SYSTEM EMULATION, GDBSERVER, IDAPRO REMOTE GDB ATTACH
From your Kali Terminal:
./start.sh
From your Debian MIPS Terminal:
gdbserver 127.0.0.1:12345 mips-test.elf
From your machine running IDAPRO:
select the remoteGDB debugger from the drop down debugger menu
specify the IP of your Debian MIPS machine and Port 12345
TIPS, TRICKS, AND PITFALLS
From the nine options above you should be able to find a suitable debugging style that suits your needs. Here are some observations and difficulties I've noticed when using various setups listed above.
- When using the QEMU User Mode method to run the MIPS binary you will notice that your actual process space is QEMU. If you attach directly to the PID you will be in QEMU process space which is recognized as x86 instruction set by the debugger and not MIPS. It is difficult to locate the MIPS entry points within the process memory space and pause on valid instructions as shown here:
- Radare2 has difficulty with the MIPS architecture when it comes to debugging FORKS, CLONES, and Single Stepping Instructions. If you issue the "e dbg.forks=true" command the debugger will pause correctly when a FORK is called. However, once you attach to the child process and SINGLE step you lose control of the program. It will run as if you told it to continue execution. I've found that you must manually place a breakpoint a couple instructions after the FORK in order to pause the program. Then you can use F7, ds, etc to step through the code.
- For best results with Radare2 on Debian MIPS system you should open the file without debugging "radare2 -a mips -b 32 mips-test.elf" and then issue the command "doo" once you are at the command line. This seems to load symbols/code analysis better than opening it straight into debugging mode.
- IDAPRO remote GDB Debugging has limitations in that you cannot specify GDB command line options such as "set follow-fork-mode child", therefore, you cannot debug child processes or threads.
- Using radare2 or gdb-multiarch natively within the emulated Debian MIPS system is generally the most effective method of debugging as you do not have to deal with the process being emulated inside of an x86 QEMU User Mode process. However, this can be a much slower method though due to the processor being emulated at ~200MHz and only being able to have one terminal session open (unless you setup SSH, etc.)
CONCLUSION
Congratulations on making it this far. While there is a fair amount of information out there regarding MIPS emulation, debugging, and analysis; we found most of it to be incomplete in one area or another. For instance an article might show how to compile MIPS but not how to run it. Or how to emulate it but not how to get networking to work.
The culmination of material in this article came from numerous sources as well as personal experience. We hope that you found this article informative and that the information presented here helps with your malware analysis in the future. Happy hunting.
Email
YouTube
Twitter
Reddit
