WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] disk I/O problems under load? (Xen-3.4.1/x86_64)

To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-users] disk I/O problems under load? (Xen-3.4.1/x86_64)
From: Luca Lesinigo <luca@xxxxxxxxxxxxx>
Date: Thu, 1 Oct 2009 13:33:28 +0200
Delivery-date: Thu, 01 Oct 2009 04:35:42 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20091001071235.GK1434@xxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <7B8F8B0B-0A65-4042-873D-64422ED94775@xxxxxxxxxxxxx> <20091001071235.GK1434@xxxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
[resending to list as I erroneously sent directly to Pasi]

Il giorno 01/ott/09, alle ore 09:12, Pasi Kärkkäinen ha scritto:
You have only 2x 7200 rpm disks for 7 virtual machines and you're wondering why there's a lot of iowait? :)
Actually, no. I just reported the observed behaviour of the system.
I fully expect, as load increases, to hit the disk i/o bottleneck before having any cpu/ram/network problem.

Normally iostat in Domain-0 shows more or less high tps (200~300 under normal load, even higher if I play around with rsync to artificially trigger the problems) on the md device where all the DomU reside,
Here we're talking about the MD 'virtual' device, it's a linux kernel artifact...
and much less (usually just 10-20% of the previous value) on the two physical disks sda and sdb that compose the mirror.
...and here we're talking about the infamous 7200rpm disks, that usual under normal load are floating at 20-30 iops.

What I did _not_ expect was that all domains would literally freeze up for the whole duration of a single i/o intensive job (rsync example), or any qemu-dm / HVM domain to crash (but I'm not sure if this is related or not). We're not talking about a slow website, but about people not even connecting to HTTP (or POP, FTP, etc...) servers, without the latter ever hitting any application-level hard limit like number of worker threads/processes and the like.

I expected the CompletelyFairScheduler in linux to be fair, maybe not completely fair but a little fair at least ;), in distributing I/O load in the system so it would gracefully slow down as load increased. I do not have scientific tests, but my guess is that a single, non virtualized system, would keep up with that load, maybe slowing down during peaks but not freezing up anywhere.

While I'm still learning how to get the most out of Xen, I'm not 100% sure about my choice of kernel configurations (see my questions about I/O schedulers and tickless kernels) and hypervisor usage (see free ram, cpu pinning, etc). If my guess that the load should be 'light' on that system is correct, I'm probably just hitting some underoptimization issues in my setup. On the same line, I can add two other disks to that system (no networked storage for now, I have to rely on 4 local sata bays), I'll study up how to get the most iops out of the hardware I have.

Thanks,
--
Luca Lesinigo
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

<Prev in Thread] Current Thread [Next in Thread>