Re: [Xen-devel] Paravirtualised drivers for fully virtualised domains, rev9

[Steve Dobbelstein <steved@xxxxxxxxxx>]

Steven Smith <sos22@xxxxxxxxxxxxxxxx> wrote on 08/11/2006 05:17:04 AM:

> > Here is what I have found so far in trying to chase down the cause of
the
> > slowdown.
> > The qemu-dm process is running 99.9% of the CPU on dom0.
> That seems very wrong.  When I try this, the device model is almost
> completely idle.  Could you see what strace says, please, or if there
> are any strange messages in the /var/log/qemu-dm. file?

Looks like I jumped the gun in relating the 99.9% CPU usage for qemu-dm and
the network.  I start up the HVM domain and without running any tests
qemu-dm is chewing up 99.9% of the CPU in dom0.  So it appears that the
100% CPU qemu usage is a problem by itself.  Looks like the same problem
Harry Butterworth is seeing.

> > It appears that a lot of time is spent running timers and getting the
> > current time.  Not being familiar with the code, I am now crawling
through
> > it to see how timers are handled and how the xen-vnif PV driver uses
them.
> Timer handling isn't really changed by any of these patches.  Patch
> 02.ioemu_xen_evtchns.diff is in vaguely the same area, but I can't see
> how it could cause the problems you're seeing, assuming your
> hypervisor and libxc are up to date.
>
> What changeset of xen-unstable did you apply the patches to?

10968

> > P.S.  This just in from a test running while I typed the above.  I
noticed
> > that qemu will start a "gui_timer" when VNC is not used.  I normally
run
> > without graphics (nographic=1 in the domain config file).  I changed
the
> > config file to use VNC. The qemu-dm CPU utilization in dom0 dropped to
> > below 10%.   The network performance improved from 0.19 Mb/s to 9.75
Mb/s
> > (still less than the 23.07 Mb/s for a fully virtualized domain).
> When I try this, I see about 1600Mb/s between dom0 and a
> paravirtualised domU, about 30Mb/s between dom0 and an ioemu domU, and
> about 1200Mb/s between dom0 and an HVM domU running these drivers, all
> collected using netpipe-tcp.  That is a regression, but much smaller
> than you're seeing.
>
> There are a couple of obvious things to check:
>
> 1) Do the statistics reported by ifconfig show any errors?

No errors.

> 2) How often is the event channel interrupt firing according to
>    /proc/interrupts?  I see about 50k-150k/second.

I'm seeing ~ 500/s when netpipe-tcp reports decent throughput at smaller
buffer sizes and then ~50/s when the throughput drops at larger buffer
sizes.

> 3) Is there any packet loss when you ping a domain?  Start your test
>    and run a ping in parallel.

No packet loss.

> The other thing is that these drivers seem to be very sensitive to
> kernel debugging options in the domU.  If you've got anything enabled
> in the kernel hacking menu it might be worth trying again with that
> switched off.

Kernel debugging is on.  I also have Oprofile enabled.  I'll build a kernel
without those and see if it helps.

> > It appears there is some interaction between using the xen-vnif
> > driver and the qemu timer code.  I'm still exploring.
> I'd be happier if I could reproduce this problem here.  Are you
> running SMP?  PAE?  64 bit?  What kernel are you running in the domU?

UP kernels in both the domU and dom0 (although the scheduler likes to move
the 1 vcpu in dom0 around to different physical CPUs).  64-bit kernels on
both.
Steve D.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel