Linux Kernel 1: How to Compile Linux Kernel

I. Introduction:

This post is the first post for linux kernel hacking. Before you can dig into linux kernel source you should first learn how to build a kernel yourself.

There are many distributions out there, and I prefer to use Slackware 14, gentoo or debian (not dibian based distro such as ubuntu, mint). The reason I recommend to use these two distro is because that most modern distributions have changed so many linux codes that may confused you while your are tracing linux code.

II. Environment:

• Linux Distro: Slackware 14 x86_64
• kernel version: 3.2.37
• gcc version: 4.7.1

III. Contents:

1. pre-requirement:

The first thing you should do is download the kernel source that you want to build. The kernel source file can be found in : http://www.kernel.org/pub/linux/kernel/.

In my case, "linux-3.2.37.tar.bz2". Just extract the file and put it into the directory that you want. I put the source code in "$HOME/kernel/".

2. config:

Before building the kernel, you have to config it first.  type "make help" and you can see a list of options. In this part, we just focus on the config sections.

The following  are some options that are commonly used.

---

config          - Update current config utilising a line-oriented program
menuconfig      - Update current config utilising a menu based program
xconfig         - Update current config utilising a QT based front-end
gconfig         - Update current config utilising a GTK based front-end
oldconfig       - Update current config utilising a provided .config as base
localmodconfig  - Update current config disabling modules not loaded
localyesconfig  - Update current config converting local mods to core

---

The description is very self-explained therefore I'm not going to explain it. Just a quick note, if you prefer the GUI interface, type "make xconfig/gconfig" which will give you gui interface to config the kernel. In my case I use "make localmodconfig" which will set the config file according to your system modules.

------------------------------------[update]---------------------------------------------------------
 you can also use "gcat /proc/config.gz > .config" if your previous kernel has enable the IKCONFIG  and IKCONFIG_PROC flag. This will copy the previous kernel configuration to your current kernel source. After you finished the above command you can still type "make oldconfig" if you are using a newer kernel. (The newer kernel may have some new features )
Thank you +Eric Garland. :)

3. build the kernel:

If this is the first time you compile this kernel Type "make all -j8" to compile the kernel. The "make help" output tell us that the make install will build the target marked * and the default one is "vmlinuz" and "modules".

However, if u have compiled the kernel before and there is no new featured that is added in the config file.You can type "make vmlinuz -j8" instead. It will only build the vmlinuz and will not build the modules.

If you have add a new modules or edit the modules source code you can type "make modules -j8" to only compile the linux modules.

P.S Compiling the linux kernel may take some time according to your hardware. (get a cup of coffee or watch a movie :P)

4. Install the modules:

If this is the first time you build the linux kernel: type "sudo make modules_install". If your haven't modify any kernel modules or add a new one in the config file, this step can be skiped.

5. Install the kernel:

Instead of typing "make install" I prefer using the following command.

---

sudo cp arch/x86_64/boot/bzimage /boot/vmlinuz-3.2.37

sudo mkinitrd -c -k 3.2.37 -m ext4 -f ext4 -r /dev/sdaN -o /boot/initrd-3.2.37.img

---

More detailed about mkinitrd command, type "man mkinitrd" or check out this link: http://mirrors.slackware.com/slackware/slackware-11.0/extra/linux-2.6.17.13/README.initrd

6. Update your bootloader:

Since I'm using slackware and the default bootloader of slackware is lilo, so I have to edit the /etc/lilo.conf. After modified the /etc/lilo.conf type "sudo lilo".

7. Command Walk through:

---

        // if this is the first time you compile the kernel

make mrproper;

// if this is the first time you compile the kernel or want to add some new features to your kernel.

make localmodconfig/menuconfig/xconfig/gconfig;

make all && make modules_install; // if this is the first time you compile the kernel or adding new stuff to the

make vmlinuz; // if you just change the kernel source

make modules && make modules_install; // if you have add a new module in config file or modify the module source code

cp arch/x86_64/boot/bzimage /boot/vmlinux-3.2.37

mkinitrd -c -k 3.2.37 -m ext4 -f ext4 -r /dev/sda2

-----update your bootloader----

echo "done";

---

IV Conclusion:

This is basically how to compile a linux kernel. I will talk about some more configuration and some tools to help you trace the linux code. Happy Hacking.

SimpleOS source code

I finally upload my OS source code to the github. The following is the link:
https://github.com/mike820324/SimpleOS

Recently I'm very busy because the school work. When my school work is finished, I will post some article about the source code, from the booting process to the protected mode in detailed. :P

Simple OS - note

While I was working on my simple operating system project. I found something interesting.
Almost all the x86 system boot up in 16-bit real mode. And the way to enable the protected mode is quite easy.
As osdev wiki suggest, using the following instruction can take us from real mode to protected mode.

....

mov  eax, cr0   ; switch to pmode by
or al,1         ; set pmode bit
mov  cr0, eax

.....

But there are one thing that bother me a lot, since our code is still in the real mode, 

how can we use the 32-bit register and instructions.

After google for a while I found a very helpful website that completely solved my question.

answer in stackoverflow 

answer in nasm forum 

The answer is that

When intel introduced 32-bit code - they used the same opcodes! 

When using 32bit register in 16 bit real mode, assembler will place a prefix in front of the instruction. (0x66 according to the nasm forum) This tell the cpu that I'm using 32-bit register

in 16-bits real mode.

I take some picture to verified the result.

My environment is ubuntu 10.10 and gcc 4.4.5

I' using qemu and gdb to verified the result.

As you can see, there are 0x66 prefix in front of the mov eax, 0 instruction.

Simple OS - bootloader part1

The past few days, I've finished writing the bootloader. I summarize some of them and will post them on the blog.

Introduction:
Before we started writing our code, there are some background knowlege.


Bootloader:
what is a bootloader?
A bootloader is a program that will load the kernel image into the memory, and jumps to it.

how does it works?
First when you press the power bottom, the bios will start first. And after the bios is loaded into the memory,
it will first check which device you want to boot and check the first sector(The MBR) of the device. If it is OK, the bios will put the MBR code into the memory address 0x7c00 and jump to it.

MBR:
what is MBR?
MBR is the abbreviation of master boot record. As the name suggest the code inside the MBR is the bootloader. In most cases, the MBR is in the first sector of your devices, such as the hard disk, floopy disk , compact disk and so on.

The size of the MBR is 512 bytes. There are many fields contains in a MBR.
a. code                 440bytes
b. Disk signature   4 bytes.
c. null                   2 bytes
d. Partition tables. 64bytes
e. MBR signature 2 bytes.

BIOS interrupt:


what is bios interrupt?
bios interrupt is a low level interrupt which is loaded before the bootloader. BIOS interrupt contains many useful functions which can communicate with the I/O without fully understand the architecture.

why using bios interrupt?
as I previous mentioned, bootloader is to load the kernel image into the memory, and therefore there is no os system call or drivers to help you communicate with the I/O. The best way and the most convenient way is to use the bios interrupt to handle the I/O.

Coding time:
After understand the information, it's time to write a simple hello world program in the boot loader.
Helloworld.S

.code16
.section .text
.global main
main:
#FAT12 file system format
#there is nothing to change in this part
        jmp start_prog
        .byte   0x90
        .ascii  "MicrMike"
        .word   512
        .byte   1
        .word   1
        .byte   2
        .word   224
        .word   2880
        .byte   0xf0
        .word   9
        .word   18
        .word   2
        .long   0
        .long   2880
        .byte   0
        .byte   0
        .byte   0x29
        .long   0x19900303
        .ascii  "HELLO-OS   "
        .ascii  "FAT12   "
        .fill   18, 1, 0
start_prog:
        movw $0, %ax
        movw %ax, %ss
        movw %ax, %ds
        movw %ax, %es
        movw $msg, %si
#using bios interrupt 10h
#parameter of bios interrupt 10h
#%ah: function number
#%al: the offset of the message
loop:
        movb    $0xe, %ah
        movb    (%si), %al
        cmpb    $0, %al
        je      fin
        int     $0x10
        addw    $1, %si
        jmp     loop
fin:
        jmp     fin
msg:
        .ascii  "Hello world!!."
        .byte 0
        .org    0x1fe, 0x00
        .word   0xaa55

P.S the above source code is using the at&t syntax. In this article I won't tell you how to write assembly, but you can google to find some great tutorial.

Comment:
There are many things that is worth notice in the source code.
1. The red highlight is the FAT12 file system format. Do not change this part.
2. Since we are in the real mode and the ld will assume that the code is in 0x0, I need to initial the whole base register, such as ds, ss and so on.
3. In order to print a message on the screen, I use the bios interrupt.
    bios interrupt 10h
    %ah stores the function number, in this case use the $0xe.
    %al stores the address of the message.
4. in the bottom of the code, don't forget to put the MBR signature, otherwise the bios will think the MBR is     useless.


Compile:
1. use gcc to compile the source code:
  gcc -c Helloworld.S
2. use ld to link the obj file into the binary:
  ld -Ttext=0x0 --oformat binary Helloworld.o -o Helloworld.bin
3. use mkdosfs to create a virtual floppy disk
  mkdosfs -C os.flp 1440
4. install the binary file into the virtual floppy disk by using dd 
  dd status=noxfer conv=notrunc if=$boot_bin of=os.flp
reference website:

Result:
using qemu to test the result.
qemu -fda os.flp
and you will see a hello world in the qemu.
http://duartes.org/gustavo/blog/post/how-computers-boot-up

nachos (cont.)

After building and install the nachos, the second project is to fix a bug in nachos.

Bug description:
When nachos is executing two different executable files, the result will be weird.
The following is picture is how the bug looks like.

Finding the Bug:
Now I know what the bug looks like, It's time to find out what cause the bug.
Since our TA have told us that the bug may be in the ./userprog/addrspace.h and ./userprog/addrspace.cc, my team mate and I start to understand what this two files are doing.

what we are interested are 
1. AddrSpace::Load(), which will load the image into the physical addrspace.
2. AddrSpace::AddrSpace, which is the constructor of this class, and will create the page table of this process.


In the source code of AddrSpace::Load()
these code is to load the .text and .data into the physical memory

if (noffH.code.size > 0) {        
        executable->ReadAt(&(kernel->machine->mainMemory[noffH.code.virtualAddr]),noffH.code.size, noffH.code.inFileAddr);
    }
if (noffH.initData.size > 0) {        
        executable->ReadAt(&(kernel->machine->mainMemory[noffH.initData.virtualAddr]),noffH.initData.size, noffH.initData.inFileAddr);
    }


The allocating process will allocate the virtual address space into the physical address space.
However as the red highlight suggest, the physical address of the process is exactly the same as the virtual address of the process.
If the second process is allocate into the memory, it will overwrite the previous process. Hence, when the previous process restore the state, it will execute the code of another process, not the one belongs to it.
Here, we may find our problem.


But not so fast, before we modified the code and fixed the bug, there is one thing that we need to be aware.Nachos is using paging to map the virtual address to physical address. Therefore, when we modified the allocating part, we also need to modified the paging (page tables).The mapping of the page table is in the constructor. As the code suggest, the page table is one to one linear mapping. Therefore, if another process is load into the memory, and the process 1 restore the state. The page table of process 1 will tell that the physical address space of process1 in in the old place where it has already been overwritten by process 2.


Fixed the bug:
Now we have the information we need. It's time to modified the code and fix the bug.
The method of our team is to add a static unsigned int variable which will record how many physical frame is being used, and when process 2 is load into the memory, it will not overwritten the address space of process 1.


1. first add a static unsigned int variable in the addrspace.h

addrspace.h

class AddrSpace {

  public:

   .....

  private:

    TranslationEntry *pageTable;

    unsigned int numPages;

    static unsigned int usedPhys;

    

    bool Load(char *fileName);

    void InitRegisters();

};

2. initialize the static variable in the addrspace.cc

    #include "addrspace.h"

  unsigned int AddrSpace::usedPhys=0;

3. modified the AddrSpace::Load()

    executable->ReadAt(

&(kernel->machine->mainMemory[noffH.code.virtualAddr+128*usedPhys]),noffH.code.size, noffH.code.inFileAddr);

P.S 

why usedPhys multiply 128 before adding to the virtualAddr?

The reason is simple, cuz the usedPhys is the page frame which is being used, not the physical memory.

Therefore, when we are adding the usedPhys we also need to multiply the page size of the page frame, which is 128 byte per page.

4.remapping the page table

   for (unsigned int i = 0; i < numPages; i++) {

pageTable[i].virtualPage = i;

pageTable[i].physicalPage = i+usedPhys; 

    }

P.S

   the numPages is the number of the page frame the current process is using.

   there is no need to map the whole page again, just those the process is using.

5. record the usedPhys

    usedPhys += numPages;

P.S

   after those steps, adding the numPages to the usedPhys. Therefore, when the process 2 is loading it will not overwritten the address space of process 1.

6. the last steps is to compile the source code again and test the result.

Result:

building nachos

My Operating system final project came out recently. The project is to build the nachos and modify some of the source code to let us understand more about operating system.

However, the GCC version to compile the nachos OS is pretty low(gcc 2.95). This is not good since my environment is ubuntu 10.10 and the gcc version under ubuntu is 4.45.(way too high for building nachos). Fortunately, the source code of the nachos  is not very long, so I modified some of the source code and Makefile to accomplished the building process.

Here is how I done it:
1. download the nachos source code.
    wget http://neuron.csie.ntust.edu.tw/homework/99/os/materials/nachos-4.0.tar.gz
2.download the mips cross compile tool
   wget http://neuron.csie.ntust.edu.tw/homework/99/os/materials/mips-decstation.linux-xgcc.gz
3.move the mips cross compile tool to the root directory
  sudo mv  mips-decstation.linux-xgcc.gz /
4.unzip the file
  sudo tar zxvf mips-decstation.linux-xgcc.gz /
5.unzip the nachos 
  tar zxvf nachos-4.0.tar.gz


It's time to build the source file.
  cd ${nachos directory}/code
  make
after doing this some problems pop up.
First
     /bin/sh: gmake: not found
solution:
    use any text editor to change the Makefile. 
    Change
    MAKE = gmake make
Second
   the second problem is the ../lib/sysdep.h
   the iostream.h not found
solution:
   simply change the <iostream.h> to <iostream> and add a new line using namespace std;
Third
   the reason of the third problem is the gcc version
solution:
   use text editor the modified the Makefile.common,
   modified the line called "CFLAGS =  -g -Wall -fwritable-strings ........"
Fourth
   the terminal will echo many errors.
solution:
   the solution is very simple, add this-> to the error variables or functions.

this is it, the nachos is finished, enjoyed.

Here is the video tutorial by Isaias

 
The original link:http://os-fime.blogspot.com/2011/08/how-to-compile-nachos.html

reference website:http://neuron.csie.ntust.edu.tw/homework/99/OS/homework/homework1/B9715017-hw1-1/#ubuntu10

building the android adt for eclipse

After building the android source code and the sdk. I tried to build the adt as well.
But remember when you make adt, android will delete the .sdk and the .img which generated from android source code. (I took a lot of time to make the image files T T )
Therefore, my advice is backup those file before building the android adt.

OS: ubuntu 10.10
java version: sun-jdk1.5

Ok, now this is how I built my android adt.
1. Download  eclipse from here. (I prefer Download manually. When I use apt-get install eclipse, the building process failed.)
P.S I prefer the RAP and RCP version (since the package is smaller).

2. After downloading, extract the file to any directory you want.
   for example: ~/eclipse

3. set the environment variable ECLIPSE_HOME.
    export ECLIPSE_HOME=${the directory you extract your eclipse}

4.the final step is to make the adt using the script contain in the source code.
    ${android home}/tools/eclipse/scripts/build_server.sh ${destination-directory}
    
P.S the build_server.sh script may be contains in different folder depend on which version of source code you download
           ${android home} is where your android source code is
           ${destination-directory} is where you want to put your adt zip file.


After all these steps, you can see a zip file in your destination folder.


reference website: Build android device tool for eclipse

building the android sdk

After compiling the android source, It's time to compile the sdk.
Remember before start compiling the sdk, you need to use the sun jdk 1.5.
use this command: update-alternatives --config java 
to switch to jdk5.
However, when I'm compiling the sdk when follow the above steps, there are still error.
My solution is:
1. use lunch sdk-eng before make sdk(if the error still occur)
2. remove all the java sdk in my ubuntu such as openjdk and sun-java1.6.jdk(except jdk5)

And after these two steps, I successfully finished compiling the sdk
you can find the jdk in ~/out/host/${your_machine}/sdk
here is the layout:
drwxr-x--- 3 user user      4096 2011-04-18 10:11 .
drwxr-xr-x 9 user user      4096 2011-04-18 10:10 ..
drwxrwx--- 7 user user      4096 2011-04-18 10:10 android-sdk_eng.user_linux-x86
-rw-r--r-- 1 user user 172889941 2011-04-18 10:11 android-sdk_eng.user_linux-x86.zip
-rw-r--r-- 1 user user    648232 2011-04-18 10:11 sdk_deps.mk


steps review:
                     sudo update-alternatives --config java(change to jdk5)
                     lunch sdk-eng
                     make sdk

Before building the android source

What I forgot to mention in the previous is that the environment setting.
Before we start downloading and compiling the android source code, there are
some important stuff we need to do first.
1.according to the website, we need to download the jdk6 for Gingerbread and newer. And jdk5 for froyo or older.

  The following is how to do this.
   add deb http://us.archive.ubuntu.com/ubuntu/ jaunty multiverse
          deb http://us.archive.ubuntu.com/ubuntu/ jaunty-updates multiverse
   in your /etc/apt/source.list
  
   and sudo apt-get update
          sudo apt-get install sun-java5-jdk(for jdk5)
  
after that the sun java jdk5 is installed.
you can use : update-alternatives --config java 
to change which version you want.


2. the rest of the package is just like the website says. follow those steps and you can start download and compile your android.

compiling android source code

Yesterday, I finally finished compiling the android source code. It took me the whole afternoon to accomplish this job.
The following is how it work.
My environment is: ubuntu 10.10 netbook edition. 

1.download the android source code.(Tooks me lots of time. 5GB source code)
    this step is referenced from the android official website.
2.compile the android source code
    the official website has it all. However, there are some detail that the website doesn't mentioned.
     when compiling the source code, I met two problems.
      a.the first problem happened when compiling the qemu,
         and the solution is install some extra packages.
         sudo apt-get install xllproto-core-dev (provides Xatom.h)
         sudo apt-get install libxll-dev (provides Xlib.h)
      referenced website: Issue 15
    
     b.another problem is when compiling the adb,
        Error message is like: /usr/bin/ld: cannot find -lncurses
        solution is install an extra package.
        sudo apt-get install libncurses5-dev
     reference website: problem compiling adb....

After these problems, everything works really fine.

And after compile is finished, u will see a directory called "out" in you android root directory.