Nov 182010
 

By Default, without setting worker_cpu_affinity directive in nginx.conf, linux kernel will spread all nginx’s worker all over CPUs.

I have 4 logical CPUs on my server, which is CPU0 – CPU3

Cpu0  :  2.9%us,  0.9%sy,  0.0%ni, 88.9%id,  7.2%wa,  0.0%hi,  0.2%si,  0.0%st
Cpu1  :  1.8%us,  0.6%sy,  0.0%ni, 95.3%id,  2.2%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu2  :  2.4%us,  0.7%sy,  0.0%ni, 94.3%id,  2.5%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu3  :  1.9%us,  0.7%sy,  0.0%ni, 96.7%id,  0.6%wa,  0.0%hi,  0.0%si,  0.0%st

Using default setting, nginx’s worker always bind to those 4 logical CPUs. which is has “f” bitmask

# taskset -p 12348
pid 25748's current affinity mask: f
# taskset -p 12349
pid 25749's current affinity mask: f
# taskset -p 12351
pid 25751's current affinity mask: f
# taskset -p 12352
pid 25752's current affinity mask: f
# taskset -p 12353
pid 25753's current affinity mask: f

CPU affinity is represented as a bitmask (given in hexadecimal), with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU.
Examples:

CPU #0: 0x00000001
CPU #1: 0x00000002
CPU #2: 0x00000004
CPU #3: 0x00000008
CPU #0 and CPU #1: 0×00000003
CPU #2 and CPU #3: 0x0000000C
All CPUs: 0xFFFFFFFF

Now, i have worker_processes set to 5 in nginx.conf. I want to bind 4 worker proccess to CPU0 – CPU2 ( 1 CPU per worker proccess), 5th worker proccess to  CPU0 and CPU1. Here’s the config:

worker_processes  5;
worker_cpu_affinity 0001 0010 0100 1000 0011;

Explaination:
0001: mean, set first worker to CPU0 (bitmask 0x00000001)
0010: mean, set second worker to CPU1 (bitmask 0x00000002)
0100: mean, set third worker to CPU2 (bitmask 0x00000004)
1000: mean, set fourth worker to CPU3 (bitmask 0x00000008)
0011: mean, set sixth worker to CPU0 and CPU1 (bitmask 0×00000003)

Restart nginx, verify affinity setting using taskset

25747 ?        S<s    0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
25748 ?        S<     0:00  _ nginx: worker process
25749 ?        S<     0:00  _ nginx: worker process
25751 ?        S<     0:00  _ nginx: worker process
25752 ?        S<     0:00  _ nginx: worker process
25753 ?        S<     0:00  _ nginx: worker process

Here’s new affinity for each worker proccess

# taskset -p 25748
pid 25748's current affinity mask: 1
# taskset -p 25749
pid 25749's current affinity mask: 2
# taskset -p 25751
pid 25751's current affinity mask: 4
# taskset -p 25752
pid 25752's current affinity mask: 8
# taskset -p 25753
pid 25753's current affinity mask: 3

That’s it, i hope this help someone else..

  7 Responses to “Nginx worker_cpu_affinity”

Comments (7)
  1. AMD Phenom II 6x.

    Since it’s not any HT at all on those cpus it’s easy to setup the numbers.

    worker_cpu_affinity 000001 000010 000100 001000 010000 100000;

    Opteron 6168 with 12 cores, now there is motherboards for up to 4 cpu’s, 48 cores totaly. Bind one worker on 1 core/cpu and only use 12 workers ?

  2. i’ve should admitted that i don’t know how to deal with those 12 or 48 cores, yet.
    here, there’s a discussion about 48 cores you might be interest

    http://hardware.slashdot.org/story/10/09/30/1528229/Linux-May-Need-a-Rewrite-Beyond-48-Cores

  3. mmm, i get it now.
    32-bit machines, it’s represented by a 32-bit bitmask.

    11111111111111111111111111111111 = 0xFFFFFFFF
    

    it’s the default CPU affinity mask for all processes.

    on 64-bit machine (with operating system support that)
    the bitmask should be:

    1111111111111111111111111111111111111111111111111111111111111111 = 0xFFFFFFFFFFFFFFFF
    

    you should be able to set affinity for all 48 cores cpus

    cmiiw

  4. thanks for sharing this 🙂 how would the masks look for say cpu 4 through 7? I’ve tried 00010000 through 10000000 but that resulted in masks of 10,20,40,80 on a 64bit machine, so I had to revert to 0001 through 1000 but this sticks them to the first 4 cpus, instead of the last 4 cpus, I want them to live on….

  5. can i see your configuration?

  6. First of all I would like to say great blog! I had a quick question that I’d like to ask if you don’t
    mind. I was curious to find out how you center yourself and clear your thoughts before writing.
    I have had a tough time clearing my thoughts in getting my thoughts
    out. I do enjoy writing however it just seems like the first 10
    to 15 minutes are wasted just trying to figure
    out how to begin. Any recommendations or hints? Cheers!

  7. most of my writing based on real experiments i’ve done. not all my tutorial work out of the box if someone try to apply it. i guess writing is not a hobby for me, just sharing my knowledge and opinions here, so i don’t have any special recipe when writing, not mention i’m also not an english native speaker :))

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

(required)

(required)

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.