A Simple Webserver Performance Comparison Test

Last week, I had a long weekend (3 days off). It made me very bored. During that time I read PC Magazine. In it there is an article regarding several web performance tools. One of them is ab2 that is included in the Apache webserver.

For a long time I want to measure the performance of several webservers. That article inspired me to test the performance of several webservers.

The webservers I want to measure are Apache 2.0.55, Yaws, lighttpd and WEBrick (included with Ruby, actually it is not a webserver per se but a low-level web framework). My testing method may not be scientific enough, but at least it gives me a clue about the performance of the webservers I mentioned above.

Here are the methods I will use :

• create a static webpage (it is a copy of http://tedi.heriyanto.net)
• Apache will listen on port 80
• Lighttpd will listen on port 80
• Yaws will listen on port 8000
• WEBrick will listen on port 8080
• using ab2 request for a static webpage for 200 times from each webserver
• using ab2 request for a static webpage for 2000 times from each webserver

Notes :
In this document, I will not give details about the installation of each webserver. You can consult with the appropriate documents for that.

Apache Setup

I setup the Apache so it can handle each user website from the URL : http://website/~user/

It's a default setting in my SUSE Prof 9.3 Apache (version 2.0.55).

The directory is /home/tedi/public_html
There I put file index.html to that directory

Lighttpd Setup

I just use the default Lighttpd configuration. The webpage is store in /srv/www/htdocs

Yaws Setup

I created a special directory for Yaws (test-yaws) and it contains static webpage (index.html). I configure Yaws (yaws.conf) with the following virtual server :




WEBrick Setup

For WEBrick, I had to write a simple webserver code like the following :

#!/usr/bin/env ruby
require 'webrick'
include WEBrick

def start_webrick(config={})
config.update(:Port => 8080)
server = HTTPServer.new(config)

['INT', 'TERM'].each {|signal|
trap(signal) {server.shutdown}
}

server.start
end

start_webrick(:DocumentRoot => '/home/tedi/public_html')


Starting Apache

To start apache just use the following command :

# rcapache start


Starting Lighttpd

To start lighttpd just type the following command :

# /etc/init.d/lighttpd start

Starting Yaws

To start Yaws, I use the following command :

$ bin/yaws -i
Erlang (BEAM) emulator version 5.4.12 [source] [hipe]

Eshell V5.4.12 (abort with ^G)
1>
=INFO REPORT==== 10-Apr-2006::13:30:21 ===
Yaws: Using config file /home/tedi/yaws.conf
yaws:Add path "/home/tedi/software/erlang/yaws/scripts/../examples/ebin"
yaws:Add path "/home/tedi/software/erlang/yaws/examples/ebin"
yaws:Running with id="default"
Running with debug checks turned on (slower server)
Logging to directory "/home/tedi/yaws_logs"

=INFO REPORT==== 10-Apr-2006::13:30:21 ===
Yaws: Listening to 0.0.0.0:8000 for servers
- http://suse:8000 under /home/tedi/software/erlang/yaws/scripts/../www
- http://localhost:8000 under /home/tedi/test-yaws


Starting WEBrick

To start WEBrick just type the following command :

$ ruby webserver.rb
[2006-04-10 13:26:52] INFO WEBrick 1.3.1
[2006-04-10 13:26:52] INFO ruby 1.8.2 (2004-12-25) [i686-linux]
[2006-04-10 13:26:52] WARN TCPServer Error: Address already in use - bind(2)
[2006-04-10 13:26:52] INFO WEBrick::HTTPServer#start: pid=6893 port=8080


Testing

Here are the commands I used to test Apache webserver performance :

/usr/sbin/ab2 -n 200 -c 10 http://localhost/~tedi/index.html
/usr/sbin/ab2 -n 2000 -c 10 http://localhost/~tedi/index.html

Here are the commands I used to test Lighttpd webserver performance :

/usr/sbin/ab2 -n 200 -c 10 http://localhost/index.html
/usr/sbin/ab2 -n 2000 -c 10 http://localhost/index.html

Here are the commands I used to test Yaws webserver performance :

/usr/sbin/ab2 -n 200 -c 10 http://localhost:8000/index.html
/usr/sbin/ab2 -n 2000 -c 10 http://localhost:8000/index.html

Here are the commands I used to test WEBrick webserver performance :

/usr/sbin/ab2 -n 200 -c 10 http://localhost:8080/index.html
/usr/sbin/ab2 -n 2000 -c 10 http://localhost:8080/index.html

Results

For 200 connections

Apache

Software: Apache/2.0.55
Document Length: 4782 bytes

Concurrency Level: 10
Time taken for tests: 0.89339 seconds
Complete requests: 200
Total transferred: 1013400 bytes
HTML transferred: 956400 bytes
Requests per second: 2238.66 [#/sec] (mean)
Time per request: 4.467 [ms] (mean)
Time per request: 0.447 [ms] (mean, across all concurrent requests)
Transfer rate: 11070.19 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.9 0 4
Processing: 2 2 1.4 3 6
Waiting: 1 1 0.8 1 4
Total: 3 3 1.0 3 6

Percentage of the requests served within a certain time (ms)
50% 3
66% 3
75% 3
80% 3
90% 6
95% 6
98% 6
99% 6
100% 6 (longest request)

Yaws

Server Software: Yaws/1.57
Document Length: 4782 bytes

Concurrency Level: 10
Time taken for tests: 0.171134 seconds
Complete requests: 200
Total transferred: 1026120 bytes
HTML transferred: 975528 bytes
Requests per second: 1168.67 [#/sec] (mean)
Time per request: 8.557 [ms] (mean)
Time per request: 0.856 [ms] (mean, across all concurrent requests)
Transfer rate: 5855.06 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.1 2 5
Processing: 1 2 1.0 3 6
Waiting: 0 1 1.0 1 4
Total: 4 4 0.8 4 7

Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 4
80% 4
90% 5
95% 7
98% 7
99% 7
100% 7 (longest request)

Lighttpd

Server Software: lighttpd/1.4.11
Document Length: 4782 bytes

Concurrency Level: 10
Time taken for tests: 0.53806 seconds
Complete requests: 200
Total transferred: 1003000 bytes
HTML transferred: 956400 bytes
Requests per second: 3717.06 [#/sec] (mean)
Time per request: 2.690 [ms] (mean)
Time per request: 0.269 [ms] (mean, across all concurrent requests)
Transfer rate: 18195.00 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.0 1 3
Processing: 1 1 0.8 1 4
Waiting: 0 0 0.7 0 3
Total: 2 2 0.6 2 4

Percentage of the requests served within a certain time (ms)
50% 2
66% 2
75% 2
80% 2
90% 4
95% 4
98% 4
99% 4
100% 4 (longest request)

WEBrick

Server Software: WEBrick/1.3.1
Document Length: 4782 bytes

Concurrency Level: 10
Time taken for tests: 1.435260 seconds
Complete requests: 200
Total transferred: 1009582 bytes
HTML transferred: 960496 bytes
Requests per second: 139.35 [#/sec] (mean)
Time per request: 71.763 [ms] (mean)
Time per request: 7.176 [ms] (mean, across all concurrent requests)
Transfer rate: 686.29 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 1
Processing: 35 55 23.6 54 277
Waiting: 20 38 23.4 37 258
Total: 35 55 23.7 54 278

Percentage of the requests served within a certain time (ms)
50% 54
66% 56
75% 58
80% 59
90% 66
95% 70
98% 77
99% 272
100% 278 (longest request)

For 2000 connections

Apache

Software: Apache/2.0.55
Document Length: 4782 bytes

Concurrency Level: 10
Time taken for tests: 0.658329 seconds
Complete requests: 2000
Total transferred: 10134000 bytes
HTML transferred: 9564000 bytes
Requests per second: 3037.99 [#/sec] (mean)
Time per request: 3.292 [ms] (mean)
Time per request: 0.329 [ms] (mean, across all concurrent requests)
Transfer rate: 15032.00 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.5 0 4
Processing: 1 2 0.8 2 7
Waiting: 1 1 0.5 1 5
Total: 2 2 1.0 2 7

Percentage of the requests served within a certain time (ms)
50% 2
66% 3
75% 3
80% 3
90% 3
95% 4
98% 6
99% 6
100% 7 (longest request)

Lighttpd

Server Software: lighttpd/1.4.11
Document Length: 4782 bytes

Concurrency Level: 10
Time taken for tests: 0.416057 seconds
Complete requests: 2000
Total transferred: 10060090 bytes
HTML transferred: 9592692 bytes
Requests per second: 4807.03 [#/sec] (mean)
Time per request: 2.080 [ms] (mean)
Time per request: 0.208 [ms] (mean, across all concurrent requests)
Transfer rate: 23612.15 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 2
Processing: 0 1 0.7 1 5
Waiting: 0 0 1.0 1 4
Total: 0 1 0.8 1 7

Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 2
80% 2
90% 2
95% 3
98% 3
99% 4
100% 7 (longest request)

Yaws

Server Software: Yaws/1.57
Document Length: 4782 bytes

Concurrency Level: 10
Time taken for tests: 1.32380 seconds
Complete requests: 2000
Total transferred: 10060000 bytes
HTML transferred: 9564000 bytes
Requests per second: 1937.27 [#/sec] (mean)
Time per request: 5.162 [ms] (mean)
Time per request: 0.516 [ms] (mean, across all concurrent requests)
Transfer rate: 9515.88 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 0 3 18.1 2 605
Waiting: 0 3 18.1 2 605
Total: 0 3 18.1 2 605

Percentage of the requests served within a certain time (ms)
50% 2
66% 2
75% 3
80% 3
90% 4
95% 4
98% 6
99% 7
100% 605 (longest request)

WEBrick

Server Software: WEBrick/1.3.1
Document Length: 4782 bytes

Concurrency Level: 10
Time taken for tests: 13.272744 seconds
Complete requests: 2000
Total transferred: 10054582 bytes
HTML transferred: 9568096 bytes
Requests per second: 150.68 [#/sec] (mean)
Time per request: 66.364 [ms] (mean)
Time per request: 6.636 [ms] (mean, across all concurrent requests)
Transfer rate: 739.71 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 4 95.0 0 2999
Processing: 4 61 15.2 66 251
Waiting: 0 43 13.4 48 236
Total: 27 65 95.4 66 3059

Percentage of the requests served within a certain time (ms)
50% 66
66% 67
75% 68
80% 69
90% 70
95% 71
98% 77
99% 87
100% 3059 (longest request)

Remarks

Here are the summary of the time taken to complete requests for each webserver :

200 connections

Yaws :

Time taken for tests: 0.171134 seconds
Requests per second: 1168.67 [#/sec] (mean)
Time per request: 8.557 [ms] (mean)
Time per request: 0.856 [ms] (mean, across all concurrent requests)
Transfer rate: 5855.06 [Kbytes/sec] received

Lighttpd :

Time taken for tests: 0.53806 seconds
Requests per second: 3717.06 [#/sec] (mean)
Time per request: 2.690 [ms] (mean)
Time per request: 0.269 [ms] (mean, across all concurrent requests)
Transfer rate: 18195.00 [Kbytes/sec] received

Apache :

Time taken for tests: 0.89339 seconds
Requests per second: 2238.66 [#/sec] (mean)
Time per request: 4.467 [ms] (mean)
Time per request: 0.447 [ms] (mean, across all concurrent requests)
Transfer rate: 11070.19 [Kbytes/sec] received

WEBrick :

Time taken for tests: 1.435260 seconds
Requests per second: 139.35 [#/sec] (mean)
Time per request: 71.763 [ms] (mean)
Time per request: 7.176 [ms] (mean, across all concurrent requests)
Transfer rate: 686.29 [Kbytes/sec] received


2000 connections

Lighttpd :

Time taken for tests: 0.416057 seconds
Requests per second: 4807.03 [#/sec] (mean)
Time per request: 2.080 [ms] (mean)
Time per request: 0.208 [ms] (mean, across all concurrent requests)
Transfer rate: 23612.15 [Kbytes/sec] received


Apache :

Time taken for tests: 0.658329 seconds
Requests per second: 3037.99 [#/sec] (mean)
Time per request: 3.292 [ms] (mean)
Time per request: 0.329 [ms] (mean, across all concurrent requests)
Transfer rate: 15032.00 [Kbytes/sec] received


Yaws :

Time taken for tests: 1.32380 seconds
Requests per second: 1937.27 [#/sec] (mean)
Time per request: 5.162 [ms] (mean)
Time per request: 0.516 [ms] (mean, across all concurrent requests)
Transfer rate: 9515.88 [Kbytes/sec] received


WEBrick :

Time taken for tests: 13.272744 seconds
Requests per second: 150.68 [#/sec] (mean)
Time per request: 66.364 [ms] (mean)
Time per request: 6.636 [ms] (mean, across all concurrent requests)
Transfer rate: 739.71 [Kbytes/sec] received

Based on the information above, I can infer that the followings are the fastest webserver in descending order :

• Lighttpd
• Apache
• Yaws
• WEBrick

Please note that the file used is a static webpage. The result may be different if we use a dynamic webpage.