Hyper-threading and CPU time

From Wikistix
Jump to: navigation, search

When is a CPU second not a CPU second? When you are running with hyper-threading (aka HT, HTT, Symmetric Multi-Threading (SMT), etc) enabled. Here's a simple demonstration.

NetBSD 4.0 on a Pentium 4

The system here has a "Intel(R) Pentium(R) 4 CPU 2.80GHz", single core (one "physical" CPU) with hyper-threading enabled (giving two "logical" CPUs), running NetBSD 4.0 with an SMP kernel. We run a deterministic unit of work on an idle system:

ksh$ time gzip -9 < zz > /dev/null
   10.28s real    10.05s user     0.24s system
ksh$ time gzip -9 < zz > /dev/null
   10.26s real    10.05s user     0.20s system
ksh$ time gzip -9 < zz > /dev/null
   10.31s real    10.08s user     0.23s system

The times are fairly consistent, and, roughly, real = user + sys. Next we add an arbitrary load to the system. We assume the kernel will now schedule each thread on each logical CPU, and it is then up to the CPUs hyper-threading algorithm how the instructions are scheduled on the single core.

ksh$ perl -e 'while(1){}' &
[1] 9382
ksh$ time gzip -9 < zz > /dev/null
   15.36s real    14.96s user     0.36s system
ksh$ time gzip -9 < zz > /dev/null
   15.49s real    14.97s user     0.34s system
ksh$ time gzip -9 < zz > /dev/null
   15.41s real    14.95s user     0.37s system

OK, so what has happened here? The real time has increased by about 50%, but so has the user time. On the same system with hyper-threading disabled, you would expect the user time to remain about the same, and the real time to approximately double. Here, because both threads are really sharing the same core and its resources, they tend to compete and slow each other down. However, as the real time has not doubled, the overall throughput of the system has increased over the uni-processor case.

Also, adding more load only increases the real time, as only two threads can ever be executed in parallel.

ksh$ perl -e 'while(1){}' &
[2] 12480
ksh$ perl -e 'while(1){}' &        
[3] 29686
ksh$ perl -e 'while(1){}' &        
[4] 12019
ksh$ time gzip -9 < zz > /dev/null
   38.14s real    15.12s user     0.33s system
ksh$ time gzip -9 < zz > /dev/null
   34.45s real    15.11s user     0.25s system
ksh$ time gzip -9 < zz > /dev/null
   37.96s real    15.04s user     0.34s system

For reference, the CPU tested was:

cpu0: Intel Pentium 4 (686-class), 2798.79 MHz, id 0xf25
cpu0: features 0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features 0xbfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu0: features 0xbfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu0: features2 0x4400<CID,xTPR>
cpu0: "Intel(R) Pentium(R) 4 CPU 2.80GHz"
cpu0: I-cache 12K uOp cache 8-way, D-cache 8KB 64B/line 4-way
cpu0: L2 cache 512KB 64B/line 8-way
cpu0: ITLB 4K/4M: 64 entries
cpu0: DTLB 4K/4M: 64 entries
cpu0: Initial APIC ID 1
cpu0: Cluster/Package ID 0
cpu0: SMT ID 1
cpu0: family 0f model 02 extfamily 00 extmodel 00

Linux 2.6 on a Xeon X5650

Second test, on Linux 2.6.38 on a 6-physical core Xeon (Intel(R) Xeon(R) CPU X5650 @ 2.67GHz). We use taskset to select which cores we're going to run these processes on:

bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
11.27user 0.07system 0:11.34elapsed 99%CPU (0avgtext+0avgdata 2944maxresident)k
0inputs+0outputs (0major+229minor)pagefaults 0swaps
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
11.18user 0.01system 0:11.19elapsed 99%CPU (0avgtext+0avgdata 2944maxresident)k
0inputs+0outputs (0major+230minor)pagefaults 0swaps
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
11.21user 0.05system 0:11.26elapsed 99%CPU (0avgtext+0avgdata 2928maxresident)k
0inputs+0outputs (0major+228minor)pagefaults 0swaps

Start a CPU burning thread on the second thread on that core, and retest:

bash$ taskset -c 11 perl -e 'while(1){}' &
[1] 4391
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
16.90user 0.09system 0:17.00elapsed 99%CPU (0avgtext+0avgdata 2944maxresident)k
0inputs+0outputs (0major+229minor)pagefaults 0swaps
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
16.80user 0.03system 0:16.84elapsed 99%CPU (0avgtext+0avgdata 2944maxresident)k
0inputs+0outputs (0major+230minor)pagefaults 0swaps
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
16.71user 0.07system 0:16.79elapsed 99%CPU (0avgtext+0avgdata 2928maxresident)k
0inputs+0outputs (0major+229minor)pagefaults 0swaps

And just to complete our set of tests:

bash$ taskset -c 11 perl -e 'while(1){}' &
[2] 4730
bash$ taskset -c 11 perl -e 'while(1){}' &
[3] 4731
bash$ taskset -c 11 perl -e 'while(1){}' &
[4] 4734
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
16.66user 0.06system 0:16.73elapsed 99%CPU (0avgtext+0avgdata 2928maxresident)k
0inputs+0outputs (0major+228minor)pagefaults 0swaps
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
16.60user 0.07system 0:16.68elapsed 99%CPU (0avgtext+0avgdata 2944maxresident)k
0inputs+0outputs (0major+229minor)pagefaults 0swaps
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
16.71user 0.08system 0:16.80elapsed 99%CPU (0avgtext+0avgdata 2944maxresident)k
0inputs+0outputs (0major+229minor)pagefaults 0swaps

Whoa, what happened here? Since we're selecting each virtual core to run on explicitly, the second virtual core now has 4 threads (perl) running on it, while the first virtual core only gets the gzip. For a matching test to the NetBSD case, we could do:

bash$ taskset -c 5,11 perl -e 'while(1){}' &
[1] 4966
bash$ taskset -c 5,11 perl -e 'while(1){}' &
[2] 4969
bash$ taskset -c 5,11 perl -e 'while(1){}' &
[3] 4970
bash$ taskset -c 5,11 perl -e 'while(1){}' &
[4] 4972
bash$ taskset -c 5,11 time gzip -9 < /tmp/zz > /dev/null
16.63user 0.04system 0:42.45elapsed 39%CPU (0avgtext+0avgdata 2944maxresident)k
0inputs+0outputs (0major+229minor)pagefaults 0swaps
bash$ taskset -c 5,11 time gzip -9 < /tmp/zz > /dev/null
16.72user 0.11system 0:42.89elapsed 39%CPU (0avgtext+0avgdata 2944maxresident)k
0inputs+0outputs (0major+229minor)pagefaults 0swaps
bash$ taskset -c 5,11 time gzip -9 < /tmp/zz > /dev/null
16.83user 0.08system 0:43.64elapsed 38%CPU (0avgtext+0avgdata 2928maxresident)k
0inputs+0outputs (0major+228minor)pagefaults 0swaps

NetBSD 7.0 on Intel Core i7

And a more modern example on NetBSD, on a Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, first a baseline:

ksh$ sudo schedctl -A 3,7 time gzip -9 < /tmp/zz > /dev/null
       10.37 real        10.06 user         0.30 sys
ksh$ sudo schedctl -A 3,7 time gzip -9 < /tmp/zz > /dev/null
       10.37 real        10.17 user         0.18 sys
ksh$ sudo schedctl -A 3,7 time gzip -9 < /tmp/zz > /dev/null
       10.40 real        10.08 user         0.28 sys

With a single spinning process:

ksh$ sudo schedctl -A 3,7 perl -e 'while(1){}' &
[1] 20565
ksh$ sudo schedctl -A 3,7 time gzip -9 < /tmp/zz > /dev/null
       14.63 real        13.69 user         0.21 sys
ksh$ sudo schedctl -A 3,7 time gzip -9 < /tmp/zz > /dev/null
       14.46 real        14.24 user         0.22 sys
ksh$ sudo schedctl -A 3,7 time gzip -9 < /tmp/zz > /dev/null
       14.46 real        14.26 user         0.20 sys

And now with 3 more spinning processes:

ksh$ sudo schedctl -A 3,7 perl -e 'while(1){}' &            
[2] 19974
ksh$ sudo schedctl -A 3,7 perl -e 'while(1){}' &
[3] 25182
ksh$ sudo schedctl -A 3,7 perl -e 'while(1){}' &
[4] 27197
ksh$ sudo schedctl -A 3,7 time gzip -9 < /tmp/zz > /dev/null
       32.05 real        14.22 user         0.29 sys
ksh$ sudo schedctl -A 3,7 time gzip -9 < /tmp/zz > /dev/null
       28.45 real        14.22 user         0.27 sys
ksh$ sudo schedctl -A 3,7 time gzip -9 < /tmp/zz > /dev/null
       38.47 real        14.28 user         0.21 sys

All pretty much as expected. Single thread latency increases about 36%, for a multi-threaded instruction throughput increase of around 47%.

For reference, the CPU is:

ksh$ sudo cpuctl identify 3
cpu3: highest basic info 0000000d
cpu3: highest extended info 80000008
cpu3: "Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz"
cpu3: Intel Xeon E3-12xx, 2nd gen i7, i5, i3 2xxx (686-class), 3392.45 MHz
cpu3: family 0x6 model 0x2a stepping 0x7 (id 0x206a7)
cpu3: features 0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE>
cpu3: features 0xbfebfbff<MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2>
cpu3: features 0xbfebfbff<SS,HTT,TM,SBF>
cpu3: features1 0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MONITOR,DS-CPL,VMX,SMX,EST>
cpu3: features1 0x1fbae3ff<TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE41,SSE42,X2APIC>
cpu3: features1 0x1fbae3ff<POPCNT,DEADLINE,AES,XSAVE,OSXSAVE,AVX>
cpu3: features2 0x28100800<SYSCALL/SYSRET,XD,RDTSCP,EM64T>
cpu3: features3 0x1<LAHF>
cpu3: xsave features 0x7<x87,SSE,AVX>
cpu3: xsave instructions 0x1<XSAVEOPT>
cpu3: xsave area size: current 832, maximum 832, xgetbv enabled
cpu3: enabled xsave 0x7<x87,SSE,AVX>
cpu3: I-cache 32KB 64B/line 8-way, D-cache 32KB 64B/line 8-way
cpu3: L2 cache 256KB 64B/line 8-way
cpu3: L3 cache 8MB 64B/line 16-way
cpu3: 64B prefetching
cpu3: ITLB 64 4KB entries 4-way, 2M/4M: 8 entries
cpu3: DTLB 64 4KB entries 4-way, 2M/4M: 32 entries (L0)
cpu3: L2 STLB 512 4KB entries 4-way
cpu3: Initial APIC ID 6
cpu3: Cluster/Package ID 0
cpu3: Core ID 3
cpu3: SMT ID 0
cpu3: DSPM-eax 0x77<DTS,IDA,ARAT,PLN,ECMD,PTM>
cpu3: DSPM-ecx 0x9<HWF,EPB>
cpu3: SEF highest subleaf 00000000
cpu3: microcode version 0x23, platform ID 1

Linux 3.13 on Xeon E5-1650

Slightly more modern CPU:

bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
12.06user 0.08system 0:12.16elapsed 99%CPU (0avgtext+0avgdata 812maxresident)k
0inputs+0outputs (0major+253minor)pagefaults 0swaps
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
12.03user 0.06system 0:12.11elapsed 99%CPU (0avgtext+0avgdata 812maxresident)k
0inputs+0outputs (0major+253minor)pagefaults 0swaps
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
12.23user 0.06system 0:12.31elapsed 99%CPU (0avgtext+0avgdata 812maxresident)k
0inputs+0outputs (0major+253minor)pagefaults 0swaps

Busying the other hyper-thread core:

bash$ taskset -c 11 perl -e 'while(1){}' &
[1] 15995
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
17.02user 0.07system 0:17.12elapsed 99%CPU (0avgtext+0avgdata 812maxresident)k
0inputs+0outputs (0major+253minor)pagefaults 0swaps
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
16.92user 0.09system 0:17.04elapsed 99%CPU (0avgtext+0avgdata 808maxresident)k
0inputs+0outputs (0major+253minor)pagefaults 0swaps
bash$ taskset -c 5 time gzip -9 < /tmp/zz > /dev/null
16.82user 0.09system 0:16.94elapsed 99%CPU (0avgtext+0avgdata 808maxresident)k
0inputs+0outputs (0major+253minor)pagefaults 0swaps

So, in this very primitive test, about a 40% increase in CPU (equating to single-thread latency), which also means approx 43% increase in overall throughput [math]({2}/{1.4})[/math] by enabling hyper-threading (overall instruction throughput by multiple threads).

CPU for this test was:

Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz.

Additional

In truth, similar effects can be seen with other shared resources, just not as easily. Some examples include shared L2/L3 caches, and memory bandwidth. Both may increase the CPU time required for a given unit of work.

See Also


Misinformation found herein copyright Paul Ripke (aka “stix”) stixpjr@gmail.com.