Things about System Performance Evaluation

Kernel Config, Build and Update Grub Memuentry

make config: starts a character based questions and answer session
make menuconfig: starts a terminal-oriented configuration tool (using ncurses)
- dependency: sudo apt-get install libcureses-ocaml-dev
make oldconfig: It reads the existing .config file and prompts the user for options in the current kernel source that are not found in the file. This is useful when taking an existing configuration and moving it to a new kernel.
make: comile the kernel
make modules_install: copy all the modules to a new directory, "/lib/modules/a.b.c" where a.b.c is the kernel version
make install: prepare bzImage and initrd, and update grub

Q: What if I want to disable iommu in UEFI boot?

A: Modify GRUB_CMDLINE_LINUX="intel_iommu=off" in /etc/default/grub, and # update-grub. It will search all the available kernel images in the system and prepare grub menuentry automatically.

Q: How to reboot into non-default entry temporarily?

A: In order for the grub-reboot command to work, several required configuration changes must be in place:

The default entry for grub must be set to saved. One possible location for this is the GRUB_DEFAULT= line in /etc/default/grub.
Use grub-set-default to set your default entry to the one you normally use.
Update your grub config (e.g. update-grub).

This should take care of the initial set-up. In the future, just do grub-reboot for a one-time boot of .

If you have a submenu in your grub.cfg, you should use two digits like this: grub-reboot '1>3'. The first is menu index, and the second is submenu index. For example, to boot last entry from config like this:

grep menu /boot/grub/grub.cfg

menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-1ffa16a9-fda2-43b5-91b9-c91a08ff190f' {
submenu 'Advanced options for Debian GNU/Linux' $menuentry_id_option 'gnulinux-advanced-1ffa16a9-fda2-43b5-91b9-c91a08ff190f' {
    menuentry 'Debian GNU/Linux, with Linux 4.6.0-0.bpo.1-amd64' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.6.0-0.bpo.1-amd64-advanced-1ffa16a9-fda2-43b5-91b9-c91a08ff190f' {
    menuentry 'Debian GNU/Linux, with Linux 4.6.0-0.bpo.1-amd64 (sysvinit)' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.6.0-0.bpo.1-amd64-init-sysvinit-1ffa16a9-fda2-43b5-91b9-c91a08ff190f' {
    menuentry 'Debian GNU/Linux, with Linux 4.6.0-0.bpo.1-amd64 (recovery mode)' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.6.0-0.bpo.1-amd64-recovery-1ffa16a9-fda2-43b5-91b9-c91a08ff190f' {
    menuentry 'Debian GNU/Linux, with Linux 3.16.0-4-amd64' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.16.0-4-amd64-advanced-1ffa16a9-fda2-43b5-91b9-c91a08ff190f' {
    menuentry 'Debian GNU/Linux, with Linux 3.16.0-4-amd64 (sysvinit)' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.16.0-4-amd64-init-sysvinit-1ffa16a9-fda2-43b5-91b9-c91a08ff190f' {
    menuentry 'Debian GNU/Linux, with Linux 3.16.0-4-amd64 (recovery mode)' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.16.0-4-amd64-recovery-1ffa16a9-fda2-43b5-91b9-c91a08ff190f' {

You should use grub-reboot '1>5' and reboot.

Reference: Jem's Guide: How to compile and install a new Linux kernel

How To Enable KVM and Qemu

check the output of egrep -c '(vmx|svm)' /proc/cpuinfo is non-zero.
sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils
check user in the libvirtd group by sudo adduser `id -un` libvirtd

Q: By default, the qemu-kvm installed is Qemu 2.5.0. What if I want to upgrade Qemu to 2.8.1?

A: See example below.

$ wget http://download.qemu.org/qemu-2.8.1.1.tar.xz
$ tar xvJf qemu-2.8.1.1.tar.xz
$ cd qemu-2.8.1.1
$ ./configure
$ make
$ make install
$ kvm -version
$ qemu-system-x86_64 -version

Or you can custom the config like below.

$ ./configure --prefix=/usr             \
                --enable-kvm                 \
                --disable-xen             \
                --enbale-debug-info     \
                --enable-debug            \
                --enable-sdl                \
                --enable-vhost-net        \
                --disable-debug-tcg        \
                --target=list=x86_64-softmmu
$ make && make install

How To Build and Boot Guest

$ qemu-img create ubuntu-64.img 15G
$ cat install_iso.sh
#!/bin/bash

qemu-system-x86_64 \
    -m 2048 -smp 2 -M pc -name my-guest -cpu host -enable-kvm -machine kernel_irqchip=on \
    -hda /path/to/ubuntu-64.img
    -cdrom /path/to/ubuntu-64.iso

# ./install_iso.sh

Optional,

# kpartx -a -v /path/to/ubuntu-64.img
add map loop0p1 (253:0): 0 29638656 linear /dev/loop0 2048
add map loop0n2 (253:1): 0 1075202 linear /dev/loop0 29642750
add map loop0n5 : 0 1075200 linear 253:1 2
# mount /dev/mapper/loop0n1 /mnt/
## dosomeing like file copy
# umount /mnt.
# kpartx -d -v /path/to/ubuntu-64.img

$ qemu-img creaet -b /path/to/ubuntu-64.img -f qcow2 /path/to/ubuntu-64.qcow2

And then restart with following script.

qemu-system-x86_64 \
    -m 2048 -smp 2 -M pc -name my-guest -cpu host -enable-kvm -machine kernel_irqchip=on \
    -hda /path/to/ubuntu-64.qcow2

Upgrade kernel to 4.10

This release of Ubuntu 16.04.2 uses kernel 4.4, we want to upgrade to the latest stable kernel of 4.x, i.e., 4.4.0-72-generic. This 4.4 kernel does not support io polling which is what the specification sheet for Optane requires because the default method to test specifications requires io polling mode. So follow these steps on Ubuntu with a network connection to the internet to download and install from kernel.ubuntu.com a 4.10-stable kernel release.

$ cd /tmp
$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10/linux-headers-4.10.0-041000_4.10.0-041000.201702191831_all.deb 
$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10/linux-headers-4.10.0-041000-generic_4.10.0-041000.201702191831_amd64.deb
wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10/linux-image-4.10.0-041000-generic_4.10.0-041000.201702191831_amd64.deb
### Now install the kernel packages:
$ sudo dpkg -i *.deb
$ sudo reboot

$ uname –r
4.10.0-041000-generic

Performance Testing and FIO Example

When conducting maximum performance testing, better to enable TURBO, enable hyper threading, set cpufreq governor to "performance", diasble ondemand service.

$ sudo apt-get install cpufrequtils
$ echo GOVERNOR=performace > /etc/default/cpufrequtils
$ sudo update-rc.d ondemand disable

# reboot

$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_govenor
performance

Fio example

[global]
direct=1
filename=/dev/nvme0n1
log_avg_msec=500
time_based
ioengine=libaio
## Test 2
#ioengine=sync
## Test 3
#ioengine=pvsync2
#hipri
percentile_list=99:99.999


[precondition]
runtime=30
bs=128K
iodepth=16
numjobs=1
rw=write
cpus_allowed=0
stonewall


[rand-read-4k-qd1]
runtime=90
bs=4K
iodepth=1
numjobs=1
rw=randread
cpus_allowed=0
stonewall


[rand-write-4k-qd1]
runtime=90
bs=4K
iodepth=1
numjobs=1
rw=randwrite
cpus_allowed=0
stonewall


[rand-read-4k-qd16]
runtime=90
bs=4K
iodepth=4
numjobs=4
group_reporting
rw=randread
cpus_allowed=0-3
stonewall


[rand-write-4k-qd16]
runtime=90
bs=4K
iodepth=4
numjobs=4
group_reporting
rw=randwrite
cpus_allowed=0-3
stonewall


[rand-rw-4k-qd16]
runtime=90
bs=4K
iodepth=4
numjobs=4
group_reporting
rw=randrw
rwmixread=70
rwmixwrite=30
cpus_allowed=0-3
stonewall

How to measure CPU usage

Add -monitor stdio to Qemu boot command. Use info cpus in Qemu monitor to check Qemu threads. And use cpustat -p $Qemu-pid -g to calculate the average CPU usage. In general, top or htop is sufficient.

Knowledge on Evaluation