By Default, without setting worker_cpu_affinity directive in nginx.conf, linux kernel will spread all nginx’s worker all over CPUs.
I have 4 logical CPUs on my server, which is CPU0 – CPU3
Cpu0 : 2.9%us, 0.9%sy, 0.0%ni, 88.9%id, 7.2%wa, 0.0%hi, 0.2%si, 0.0%st Cpu1 : 1.8%us, 0.6%sy, 0.0%ni, 95.3%id, 2.2%wa, 0.0%hi, 0.1%si, 0.0%st Cpu2 : 2.4%us, 0.7%sy, 0.0%ni, 94.3%id, 2.5%wa, 0.0%hi, 0.1%si, 0.0%st Cpu3 : 1.9%us, 0.7%sy, 0.0%ni, 96.7%id, 0.6%wa, 0.0%hi, 0.0%si, 0.0%st
Using default setting, nginx’s worker always bind to those 4 logical CPUs. which is has “f” bitmask
# taskset -p 12348 pid 25748's current affinity mask: f # taskset -p 12349 pid 25749's current affinity mask: f # taskset -p 12351 pid 25751's current affinity mask: f # taskset -p 12352 pid 25752's current affinity mask: f # taskset -p 12353 pid 25753's current affinity mask: f
CPU affinity is represented as a bitmask (given in hexadecimal), with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU.
Examples:
CPU #0: 0x00000001 CPU #1: 0x00000002 CPU #2: 0x00000004 CPU #3: 0x00000008 CPU #0 and CPU #1: 0×00000003 CPU #2 and CPU #3: 0x0000000C All CPUs: 0xFFFFFFFF
Now, i have worker_processes set to 5 in nginx.conf. I want to bind 4 worker proccess to CPU0 – CPU2 ( 1 CPU per worker proccess), 5th worker proccess to CPU0 and CPU1. Here’s the config:
worker_processes 5; worker_cpu_affinity 0001 0010 0100 1000 0011;
Explaination:
0001: mean, set first worker to CPU0 (bitmask 0x00000001)
0010: mean, set second worker to CPU1 (bitmask 0x00000002)
0100: mean, set third worker to CPU2 (bitmask 0x00000004)
1000: mean, set fourth worker to CPU3 (bitmask 0x00000008)
0011: mean, set sixth worker to CPU0 and CPU1 (bitmask 0×00000003)
Restart nginx, verify affinity setting using taskset
25747 ? S<s 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf 25748 ? S< 0:00 _ nginx: worker process 25749 ? S< 0:00 _ nginx: worker process 25751 ? S< 0:00 _ nginx: worker process 25752 ? S< 0:00 _ nginx: worker process 25753 ? S< 0:00 _ nginx: worker process
Here’s new affinity for each worker proccess
# taskset -p 25748 pid 25748's current affinity mask: 1 # taskset -p 25749 pid 25749's current affinity mask: 2 # taskset -p 25751 pid 25751's current affinity mask: 4 # taskset -p 25752 pid 25752's current affinity mask: 8 # taskset -p 25753 pid 25753's current affinity mask: 3
That’s it, i hope this help someone else..
AMD Phenom II 6x.
Since it’s not any HT at all on those cpus it’s easy to setup the numbers.
worker_cpu_affinity 000001 000010 000100 001000 010000 100000;
Opteron 6168 with 12 cores, now there is motherboards for up to 4 cpu’s, 48 cores totaly. Bind one worker on 1 core/cpu and only use 12 workers ?
i’ve should admitted that i don’t know how to deal with those 12 or 48 cores, yet.
here, there’s a discussion about 48 cores you might be interest
http://hardware.slashdot.org/story/10/09/30/1528229/Linux-May-Need-a-Rewrite-Beyond-48-Cores
mmm, i get it now.
32-bit machines, it’s represented by a 32-bit bitmask.
it’s the default CPU affinity mask for all processes.
on 64-bit machine (with operating system support that)
the bitmask should be:
you should be able to set affinity for all 48 cores cpus
cmiiw
thanks for sharing this 🙂 how would the masks look for say cpu 4 through 7? I’ve tried 00010000 through 10000000 but that resulted in masks of 10,20,40,80 on a 64bit machine, so I had to revert to 0001 through 1000 but this sticks them to the first 4 cpus, instead of the last 4 cpus, I want them to live on….
can i see your configuration?
First of all I would like to say great blog! I had a quick question that I’d like to ask if you don’t
mind. I was curious to find out how you center yourself and clear your thoughts before writing.
I have had a tough time clearing my thoughts in getting my thoughts
out. I do enjoy writing however it just seems like the first 10
to 15 minutes are wasted just trying to figure
out how to begin. Any recommendations or hints? Cheers!
most of my writing based on real experiments i’ve done. not all my tutorial work out of the box if someone try to apply it. i guess writing is not a hobby for me, just sharing my knowledge and opinions here, so i don’t have any special recipe when writing, not mention i’m also not an english native speaker :))