Alan Hargreaves' Blog

The ramblings of an Australian SaND TSC* Principal Field Technologist

Mutex Contention vs number of cpus

I’ve had a few cases recently that have brought this issue to the fore.

It’s amazing how many people think that the answer to all performance issues is to simply throw more cpu at the problem.

Let’s work through this thought (and this holds true for other queuing type locks too).

  1. On Solaris, if the mutex holder is on proc, the waiter spins instead of blocking and sleeping
  2. If we have sufficient threads wanting the same mutex, we can quickly utilize all cpus in a box
  3. The more threads we have in the queue for a mutex, logically the longer any given thread will take to progress through this queue to obtain it.

What does this tell you about what is likely to happen if you add more cpus into the mix?

It’s relatively obvious now, isn’t it.

  1. More cpus stuck spinning in kernel space
  2. Longer mutex queues
  3. Longer average time to obtain mutex for each thread that wants it.

The obvious consequence being that adding more cpus can actually have the effect of making the problem worse.

Sigh.

Advertisements

Written by Alan

November 19, 2007 at 4:23 pm

Posted in OpenSolaris

2 Responses

Subscribe to comments with RSS.

  1. Is there a way to improve the locking in the case you’ve hit ? For example, replace the plain mutex by a version which allows multiple readers concurrently (may drain the queue faster) … or somehow re-arrange the locking…
    … offtopic: Does Solaris allow memory barriers in the kernel ?

    Roland Mainz

    November 20, 2007 at 8:07 am

  2. Yes, you can use mem barriers in the kernel. Yes, the solution to this type of problem is to re-work the code in question to avoid contention, use a different locking approach, …

    Nico

    November 20, 2007 at 10:49 am


Comments are closed.

%d bloggers like this: