Day 2 at LISA10

Linux Performance tuning

Slides: http://sites.google.com/site/linuxperftuning/
Speaker: http://en.wikipedia.org/wiki/Theodore_Ts'o

I was very excited to take the Linux Performance tuning.  I wasn't sure who the teacher was, but it was an engineer from Google.  A class in performance tuning by Google, I figured this would have to be good.  So did everyone else.  It was one of the few classes at LISA that was not only sold out, but they had to add extra seats to the room.

The class began by establishing what it is we are trying to accomplish.  Goals of performance tuning are to speed up single task,  or graceful degradation of application as load increases.  You don't want your web server to come crashing to a halt when load increases, you want it to gracefully degrade as it takes on more and more load.

In order to start tuning something, you have to know where it stands currently so you can measure whether or not you actual did something.  You must first establish a baseline.  Using the baseline as the starting point, you make a single change to the system, test, then measure the results against the baseline.  Then you just keep repeating the process, making only one change at a time.

Some basic tools to start with are 'free', 'top', and 'iostat'.  Using the free command, see if the swap space is being used.  If you are using swap, you should increase the memory of the system.   Run the top command, and check what's running.  Check the I/O with iostat and use the -k option to get kilobytes instead of blocks.  These are just some of the simple tips we started with.

We got into deep details of file system optimizations by the guy that maintains the code for ext4.  There were a lot in the slides that I still have to decipher which is why this post took so long to post.

One interesting thing I learned was something called short stroking.  Short stroking is where you partition the disk to use the outer rim (the beginning of the disk) where you get 100% performance vs the interior of the disk.  Short stroking is about 10% - 30% of the disk so you are going to lose some space, but gain a lot of speed.  Combine this with multiple disks and you can get near SSD speeds.

Other bits that jump out at me:

Mounting with noatime can help I/O because you no longer have to make a write for each file access.

Increasing the size of the journal can help performance a little bit.  'sudo dumpe2fs -h /dev/sda5`

ionice - “This  program  sets  or gets the io scheduling class and priority for a program.  If no arguments or just -p is given, ionice  will  query  the current io scheduling class and priority for that process.”

nttcp - (new test TCP program) - The nttcp program measures the transferrate (and other numbers) on a TCP, UDP or UDP multicast connection.

When tuning the network, you can tune for latency or for throughput.

NFS - no_subtree_check - If a subdirectory of a filesystem is exported, but the whole filesystem isn't then whenever a NFS request arrives, the server must check not only that the accessed file is in the appropriate filesystem (which is easy) but also that it is in the exported tree (which is harder). This check is called the subtree_check. This option disables subtree_check.

You can bump the number of NFS threads up from the default.  This is in /etc/sysconfig/nfs or /etc/defaults/nfs-kernel-server

“Use the hard option with any resource you mount read-write. Then, if a user is writing to a file when the server goes down, the write will continue when the server comes up again, and nothing will be lost.” (http://uw714doc.sco.com/en/NET_nfs/nfsT.mount_cmd.html)

Remove outdated fstab options.  Just use “rw,intr”

More NFS performance info: http://www.linuxselfhelp.com/HOWTO/NFS-HOWTO/performance.html

strace – system call tracer
ltrace – library tracing

Optimize the stuff that used often versus once

perf – kicks ass.  Learn it. (https://perf.wiki.kernel.org/index.php/Main_Page)

After the class, I went to a few talks.  There was Splunk, PostgreSQL, and some cloud computing thing.  Then I attended the Minecraft get-together.  I heard of Minecraft, but now I saw a demo on the large projection screen.  Now (a week later) I have four account for the family and we are playing on our server.  We're all hooked.  I can't recommend this thing enough.  (http://minecraft.net/)