Nov 182010
 

By Default, without setting worker_cpu_affinity directive in nginx.conf, linux kernel will spread all nginx’s worker all over CPUs.

I have 4 logical CPUs on my server, which is CPU0 – CPU3

Cpu0  :  2.9%us,  0.9%sy,  0.0%ni, 88.9%id,  7.2%wa,  0.0%hi,  0.2%si,  0.0%st
Cpu1  :  1.8%us,  0.6%sy,  0.0%ni, 95.3%id,  2.2%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu2  :  2.4%us,  0.7%sy,  0.0%ni, 94.3%id,  2.5%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu3  :  1.9%us,  0.7%sy,  0.0%ni, 96.7%id,  0.6%wa,  0.0%hi,  0.0%si,  0.0%st

Using default setting, nginx’s worker always bind to those 4 logical CPUs. which is has “f” bitmask

# taskset -p 12348
pid 25748's current affinity mask: f
# taskset -p 12349
pid 25749's current affinity mask: f
# taskset -p 12351
pid 25751's current affinity mask: f
# taskset -p 12352
pid 25752's current affinity mask: f
# taskset -p 12353
pid 25753's current affinity mask: f

CPU affinity is represented as a bitmask (given in hexadecimal), with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU.
Examples:

CPU #0: 0x00000001
CPU #1: 0x00000002
CPU #2: 0x00000004
CPU #3: 0x00000008
CPU #0 and CPU #1: 0×00000003
CPU #2 and CPU #3: 0x0000000C
All CPUs: 0xFFFFFFFF

Now, i have worker_processes set to 5 in nginx.conf. I want to bind 4 worker proccess to CPU0 – CPU2 ( 1 CPU per worker proccess), 5th worker proccess to  CPU0 and CPU1. Here’s the config:

worker_processes  5;
worker_cpu_affinity 0001 0010 0100 1000 0011;

Explaination:
0001: mean, set first worker to CPU0 (bitmask 0x00000001)
0010: mean, set second worker to CPU1 (bitmask 0x00000002)
0100: mean, set third worker to CPU2 (bitmask 0x00000004)
1000: mean, set fourth worker to CPU3 (bitmask 0x00000008)
0011: mean, set sixth worker to CPU0 and CPU1 (bitmask 0×00000003)

Restart nginx, verify affinity setting using taskset

25747 ?        S<s    0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
25748 ?        S<     0:00  _ nginx: worker process
25749 ?        S<     0:00  _ nginx: worker process
25751 ?        S<     0:00  _ nginx: worker process
25752 ?        S<     0:00  _ nginx: worker process
25753 ?        S<     0:00  _ nginx: worker process

Here’s new affinity for each worker proccess

# taskset -p 25748
pid 25748's current affinity mask: 1
# taskset -p 25749
pid 25749's current affinity mask: 2
# taskset -p 25751
pid 25751's current affinity mask: 4
# taskset -p 25752
pid 25752's current affinity mask: 8
# taskset -p 25753
pid 25753's current affinity mask: 3

That’s it, i hope this help someone else..

  7 Responses to “Nginx worker_cpu_affinity”

Comments (7)
  1. First of all I would like to say great blog! I had a quick question that I’d like to ask if you don’t
    mind. I was curious to find out how you center yourself and clear your thoughts before writing.
    I have had a tough time clearing my thoughts in getting my thoughts
    out. I do enjoy writing however it just seems like the first 10
    to 15 minutes are wasted just trying to figure
    out how to begin. Any recommendations or hints? Cheers!

  2. most of my writing based on real experiments i’ve done. not all my tutorial work out of the box if someone try to apply it. i guess writing is not a hobby for me, just sharing my knowledge and opinions here, so i don’t have any special recipe when writing, not mention i’m also not an english native speaker :))

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

(required)

(required)

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.