Server tasting

Here’s a page about how to test Linode’s network performance at a given data center and also the disk read and write speeds of a Linux server.  I’ve been looking for an easy way to evaluate/compare how virtual servers will hold up under load.  Since Jacktrip is all about moving audio data around, I’ve landed on testing disk performance as a good way to test all the parts of the server that are going to be used by Jacktrip (CPU, i/o busses, etc) to do that.

Linode has introduced a very helpful network performance testing tool which is available here.  I would use this if a given player is having trouble as it is meant to test performance to a specific player location — results will vary widely depending on where the player is.

I titled this web page “Server Tasting” because that’s how I do it.  I build a gaggle of servers and conduct a little tasting session that compares the results of these two command-line tests (thanks to Linode-support for these).  As soon as I find a tasty one in a location that will work for the session, I stop building new ones and delete the rejects.  I can taste a lot of servers for not much money this way and the results have been Pretty Good.

More recently I’ve started using a series of support tickets in which I ask Linode to migrate a server to a less-crowded server and network-segment which these tests aren’t coming back the way I want them.  They’ve been very responsive and willing to do those migrations.

The Two Tests

I got these from Linode Support — who have always been there when I needed help.  Thanks folks!

This “dd” command tests how fast the disk can create and then read files some great big disk files.  This version creates a pretty big file set 500 4M (mByte) files or 4 gBytes which can tip over the disk of a 25 gig Nanode if it’s got other stuff already there.  Consider reducing those numbers if the disk isn’t empty.  It’s a good idea to hunt down and remove that test_file_DELETE file if the process aborts in the middle.

dd if=/dev/zero of=test_file_DELETE bs=4M count=500

My speedy Nanode (smallest/cheapest Linode) is returning 1.2 gBytes/second just now.  I would trigger a migration with Linode support if that dropped below 600 mBytes/second.  Here’s how the 1.2 gig result looks.

dd if=/dev/zero of=test_file_DELETE bs=4M count=500
4000+0 records in
4000+0 records out
16777216000 bytes (17 GB, 16 GiB) copied, 13.7779 s, 1.2 GB/s

This “fio” command line runs a similar test that produces more information and stresses the server a little more.  You may need to add it to the server – use “apt install fio” to do that.

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=500M --readwrite=randrw --rwmixread=75

Here are the results, on Speedy Nanode, showing lower throughput (reading 373 mbytes/sec and writing 124 mbytes/second).   I would trigger a migration with Linode support if either of these drops below 150 mBytes/sec.  I’m especially sensitive to the speed with which the server writes data, as a bottleneck here corrupts the audio of a session that’s being recorded with weird-sounding periodic “gremlin” buzzes every 3-5 seconds.

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=500M --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.25
Starting 1 process
test: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1)
test: (groupid=0, jobs=1): err= 0: pid=460525: Mon Sep 5 16:29:40 2022
read: IOPS=95.5k, BW=373MiB/s (391MB/s)(375MiB/1005msec)
bw ( KiB/s): min=362784, max=401816, per=100.00%, avg=382300.00, stdev=27599.79, samples=2
iops : min=90696, max=100454, avg=95575.00, stdev=6899.95, samples=2
write: IOPS=31.9k, BW=124MiB/s (130MB/s)(125MiB/1005msec); 0 zone resets
bw ( KiB/s): min=120952, max=134176, per=100.00%, avg=127564.00, stdev=9350.78, samples=2
iops : min=30238, max=33544, avg=31891.00, stdev=2337.70, samples=2
cpu : usr=6.47%, sys=45.92%, ctx=7617, majf=0, minf=8
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=95984,32016,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=373MiB/s (391MB/s), 373MiB/s-373MiB/s (391MB/s-391MB/s), io=375MiB (393MB), run=1005-1005msec
WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=125MiB (131MB), run=1005-1005msec

Disk stats (read/write):
sda: ios=74188/24613, merge=0/0, ticks=33804/9000, in_queue=42804, util=90.34%

Results from a recent tasting:

I was preparing a server to host a public 7-person performance that was also recorded full multi-track.

I built five servers of two types (virtual and dedicated CPUs) in two locations  (California and New Jersey).  I had a duplicate sneak in because I wasn’t paying attention to what I was doing.  The Nanode rounds out the bottom of the table.

The “dd” test results are the most startling in their difference, ranging over 10 times the speed.  The more-complex “fio” test doesn’t show as much difference.  But the “dd” and “fio” tests agree on which server is fastest.  I chose the highlighted East-location dedicated-CPU server and was happy with the performance.

But also note how well the tiny little last-row Speedy Nanode stacks up.  That rascal is a terrific server — and can run all month for five dollars.