计算机及Linux操作系统原理、CPU负载观察及调优方法

计算机及Linux操作系统原理、CPU负载观察及调优方法

2023年7月3日发(作者:)

计算机及Linux操作系统原理、CPU负载观察及调优⽅法Understanding the Linux operating systemOS, cell32bit, 64bit,2^32=4G*4GBCPU  运算器  控制器  寄存器 registerPAE:  Physical Address Extension 物理地址扩展  32bit, + 4bit = 64G程序,局部性置换策略空间局部性、时间局部性N路关联Write through: 通写write Back: 回写Video Card 显卡South Bridge 南桥固态硬盘IO Port65535⼀批连续端⼝poll中断控制器临界区DMA: 直接内存访问monitor  OS --> VMProcess: ⼀个运⾏单位  系统资源: CPU时间,存储空间OS: VM  CPU:    时间:切⽚      缓存:缓存当前程序数据    进程切换:保存线程、恢复现场  内存:线性地址 <-- 物理地址    空间:映射4k, page, page frame(页框)  I/O:    内核 --> 进程Readysleeping可打断不可打断Schematic interaction of different performance componentsThe process descriptor and task listList 链表进程描述符:  进程元数据  双向链表Context switchingLinux: 抢占  tick: 滴答    时间解析度  100Hz  1000Hz时钟中断A: 5ms, 1msC:进程类别:  交互式进程(I/O)  批处理进程(CPU)  实时进程(Real-time)桌⾯:服务器:CPU  CPU: 时间⽚长,优先级低  IO: 时间⽚短,优先级⾼Linux优先级:priority  实时优先级:1-99,数字越⼩,优先级越低  静态优先级: 100-139,数据越⼩,优先级越⾼    实时优先级⽐静态优先级⾼  nice值:调整静态优先级调度类别:  实时进程:    SCHED FIFO: First In First Out    SHCED_RR:Round Robin    SCHED_Other: ⽤来调度100-139之间的进程      100-139        10:100        30:115        2:130动态优先级:  dynamic priority = max (100, min ( static priority - bonus + 5, 139))    bonus: 0-10  110, 10  110⼿动调整优先级:  100-139: nice    nice N COMMAND 启动进程时候指定优先级    renice -n # PID 指定运⾏中的进程的优先级    chart -p [prio] COMMAND 正在100-139之间的优先级  1-99:    chrt -f -p [prio] PID 已经运⾏的进程调整优先级,-f FIFO类别    chrt -r -p [prio] PID 已经运⾏的进程调整优先级,-r RR类别    chrt -f -p [prio] COMMAND 启动⼀个进程指定优先级;    ps -e -o class,rtprio,pri,nice,cmd 查看优先级O(1)SCHED_OtherCFS: Complete Fair Scheduler 完全公平调度器Process address spaceCOW  Kernel --> init    init      fork(): 系统调⽤      task_struct        Memory --> Parent        COW: Copy On Write        prefork[root@Smoke ~]# top(查看cpu使⽤情况)top - 19:56:19 up 1 day, 22:22, 3 users, load average: 0.00, 0.00, 0.00Tasks: 119 total, 1 running, 118 sleeping, 0 stopped, 0 zombieCpu(s): 0.0%us, 3.8%sy, 0.0%ni, 95.8%id, 0.0%wa, 0.0%hi, 0.4%si, 0.0%stMem: 1938968k total, 1909364k used, 29604k free, 106984k buffersSwap: 2097144k total, 180k used, 2096964k free, 1521396k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28 root 20 0 0 0 0 S 15.6 0.0 0:01.08 kswapd0 4 root 20 0 0 0 0 S 0.3 0.0 0:01.78 ksoftirqd/0

369 root 20 0 0 0 0 S 0.3 0.0 0:44.95 kjournald

6382 root 20 0 2696 1124 876 R 0.3 0.1 0:00.42 top 1 root 20 0 2900 1388 1224 S 0.0 0.1 0:01.65 init

2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd

3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0

6 root RT 0 0 0 0 S 0.0 0.0 0:19.02 watchdog/0

7 root 20 0 0 0 0 S 0.0 0.0 0:13.24 events/0

8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cgroup

9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper

10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns

11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr

12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pm

[root@Smoke ~]# ps -e -o class,rtprio,pri,nice,cmd(查看优先级)CLS RTPRIO PRI NI CMDTS - 19 0 /sbin/initTS - 19 0 [kthreadd]FF 99 139 - [migration/0]TS - 19 0 [ksoftirqd/0]FF 99 139 - [migration/0]FF 99 139 - [watchdog/0]TS - 19 0 [events/0]TS - 19 0 [cgroup]TS - 19 0 [khelper]TS - 19 0 [netns]TS - 19 0 [async/mgr]TS - 19 0 [pm]TS - 19 0 [sync_supers]TS - 19 0 [bdi-default]TS - 19 0 [kintegrityd/0]TS - 19 0 [kblockd/0]TS - 19 0 [kacpid]TS - 19 0 [kacpi_notify]TS - 19 0 [kacpi_hotplug]TS - 19 0 [ata/0]TS - 19 0 [ata_aux]TS - 19 0 [ksuspend_usbd]TS - 19 0 [khubd]TS - 19 0 [kseriod]TS - 19 0 [md/0]TS - 19 0 [md_misc/0]TS - 19 0 [khungtaskd]TS - 19 0 [kswapd0]TS - 14 5 [ksmd]TS - 19 0 [aio/0]TS - 19 0 [crypto/0]TS - 19 0 [kthrotld/0]TS - 19 0 [pciehpd]TS - 19 0 [kpsmoused]TS - 19 0 [usbhid_resumer]TS - 19 0 [scsi_eh_0]TS - 19 0 [scsi_eh_1]TS - 19 0 [mpt_poll_0]TS - 19 0 [mpt/0]TS - 19 0 [scsi_eh_2]TS - 19 0 [scsi_eh_3]TS - 19 0 [scsi_eh_4]TS - 19 0 [scsi_eh_5]TS - 19 0 [scsi_eh_6]TS - 19 0 [scsi_eh_7]TS - 19 0 [scsi_eh_8]TS - 19 0 [scsi_eh_9]TS - 19 0 [scsi_eh_10]TS - 19 0 [scsi_eh_11]TS - 19 0 [scsi_eh_12]TS - 19 0 [scsi_eh_13]TS - 19 0 [scsi_eh_14]TS - 19 0 [scsi_eh_15]TS - 19 0 [scsi_eh_16]TS - 19 0 [scsi_eh_17]TS - 19 0 [scsi_eh_18]TS - 19 0 [scsi_eh_19]TS - 19 0 [scsi_eh_20]TS - 19 0 [scsi_eh_21]TS - 19 0 [scsi_eh_22]TS - 19 0 [scsi_eh_23]TS - 19 0 [scsi_eh_24]TS - 19 0 [scsi_eh_25]TS - 19 0 [scsi_eh_26]TS - 19 0 [scsi_eh_27]TS - 19 0 [scsi_eh_28]TS - 19 0 [scsi_eh_29]TS - 19 0 [scsi_eh_30]TS - 19 0 [scsi_eh_31]TS - 19 0 [scsi_eh_32]TS - 19 0 [kjournald]TS - 23 -4 /sbin/udevd -dTS - 19 0 [vmmemctl]TS - 19 0 [flush-8:0]TS - 19 0 [bluetooth]TS - 19 0 [kstriped]TS - 19 0 [kjournald]TS - 19 0 [kauditd]TS - 19 0 /usr/local/memcached/bin/memcached -d -p 11211 -u nobody -c 1024 -m 128TS - 23 -4 auditdTS - 19 0 /sbin/rsyslogd -i /var/run/ -c 5TS - 19 0 dbus-daemon --systemTS - 19 0 nginx: worker process

TS - 19 0 /usr/sbin/sshdTS - 19 0 /usr/libexec/postfix/masterTS - 19 0 qmgr -l -t fifo -uTS - 19 0 crondTS - 19 0 /usr/bin/rhsmcertdTS - 19 0 login -- root

TS - 19 0 /sbin/mingetty /dev/tty2TS - 19 0 /sbin/mingetty /dev/tty3TS - 19 0 /sbin/mingetty /dev/tty4TS - 21 -2 /sbin/udevd -dTS - 19 0 /sbin/mingetty /dev/tty5TS - 21 -2 /sbin/udevd -dTS - 19 0 /sbin/mingetty /dev/tty6TS - 19 0 /usr/sbin/console-kit-daemon --no-daemonTS - 19 0 -bashTS - 19 0 php-fpm: master process (/usr/local/php/etc/)

TS - 19 0 php-fpm: pool www

TS - 19 0 php-fpm: pool www

TS - 19 0 php-fpm: pool wwwTS - 19 0 php-fpm: pool wwwTS - 19 0 php-fpm: pool wwwTS - 19 0 php-fpm: pool www

TS - 19 0 php-fpm: pool wwwTS - 19 0 php-fpm: pool wwwTS - 19 0 sshd: root@pts/0

TS - 19 0 -bashTS - 19 0 sshd: root@notty

TS - 19 0 /usr/libexec/openssh/sftp-serverTS - 19 0 /usr/libexec/openssh/sftp-serverTS - 19 0 pickup -l -t fifo -uTS - 19 0 sshd: root@pts/1

TS - 19 0 -bashTS - 19 0 ps -e -o class,rtprio,pri,nice,cmdTS - 19 0 nginx: master process /usr/sbin/nginx -c /etc/nginx/ - 19 0 /bin/sh /usr/local/mysql/bin/mysqld_safe --datadir=/mydata/data --pid-file=/mydata/data/ - 19 0 /usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/mydata/data --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=/var/log/ --pid-fi

[root@Smoke ~]# man chrt(查看chrt的man帮主) chrt - manipulate real-time attributes of a process chrt [options] prio command [arg]... chrt [options] -p [prio](优先级) pid(进程) -b, --batch(批处理进程) set scheduling policy to SCHED_BATCH (Linux specific) -f, --fifo(调整fifo类别) set scheduling policy to SCHED_FIFO -i, --idle(空闲进程) set schedulng policy to SCHED_IDLE (Linux specific) -r, --rr(调整rr类别) set scheduling policy to SCHED_RR (the default)调度类别:  RT    SCHED_FIFO    SCHED_RR  100-139    SCHED_Other  SCHED_BATCH  SCHED_IDLE抢占  tick: 时钟终端    100Hz    1000HzRHEL 6.4  tick less ⽆滴答  interrupt-driven    硬中断    软中断    深度睡眠I1(⼀级指令缓存), D1(⼀级数据缓存)SMP对称多处理器NUMA ⾮对⼀致性内存访问Local and Remote Memory Access In NUMA Topolog

CPU affinity: CPU姻亲关系numastatnumactlnumad 只是在硬件级别将某个进程或某些进程跟我们cpu和node绑定的,⽽要实现cpu姻亲关系,要将进程跟cpu绑定;taskset: 绑定进程⾄某CPU上  mask:    0x0000 0001(底0颗CPU)0001: 0号cpu;0x0000 00030011:0号和1号cpu;1010x0000 0005: 0号和2号cpu;1110x0000 0007: 0-2号cpu;# taskset -p mask pid101, 3# CPU#taskset -p 0x0000 0004 101(把101进程绑定第三号cpu上)0100tasket -p -c 3 101(把101进程绑定在第3号cpu上)tasket -p -c 0,1 101(把101进程绑定在0号和1号cpu上)tasket -p -c 0-2 101(把101进程绑定在0-2号cpu上)tasket -p -c 0-2,7 101Tuning run queue length with tasketRestrict length of a CPU run queue  Isolate a CPU from automatic scheduling in /etc/ isolcpus=cpu number,...,cpu number(将cpu从操作系统隔离出来,内核不会让已经启动的进程使⽤这些cpu的,就将这些cpu预留出来了)  Pin tasks to that CPU with taskset(tasket将某个任务订在某颗cpus上)  Consider moving IRQs off the CPU(移除cpu上的中断)echo cpu_mask > /proc/irq//smp_affinity应该将中断绑定⾄那些⾮隔离的CPU上,从⽽避免那些隔离的CPU处理中断程序;echo CPU_MASK > /proc/irq//smp_affinty(CPU_MASK CPU号码,irq_num终端号码)Viewing CPU performance data(查看cpu性能数据命令)Load average: average length of run queues(查看cpu平均利⽤率)  Considers only task in TASK_RUNNABLE and TASK_UNNTERRUPTABLE  sar -q(查看cpu平均使⽤率)  top(查看系统⼈物)  w(显⽰登陆⽤户的任务)  uptime(显⽰系统运⾏时长)  vmstat 1 5(显⽰虚拟内存统计报告)CPU utilization  mpstat 1 2(显⽰对称多处理器每个cpu整个使⽤率)  sar -P ALL 1 2(查看cpu使⽤情况)  iostat -c 1 2  /proc/stat  dstat -csar -w  查看上下⽂切换的平均次数,以及进程创建的平均值Scheduler domains(调度器域)Group processors into cpusets(将cpu化好组)  Each cpuset represents a scheduler domian(将某个进程绑定在某个组内的cpu上)  Supports both multi-core and NUMA architectures  Simple management interface through the cpuset virtual file systemThe root cpuset contains all system resourcesChild cpusets  Each cpuset must contain at least one CPU and one memory zone  Child cpusets can be nested  Dynamically attach tasks to a cpusetConsequences  Control latency due to queue length,cache,and NUMA zones  Assign processes with different CPU characteristics to different cpusets  Scalable for complex performance scenariosConfiguring the root cpusetCreate a mount ponit at /cpusets(创建挂载点)Add an entry to /etc/fstab(添加fstab⽂件配置)cpuset(设备) /cpusets(挂载点) cpuset(类型) defaults(选项) 0 0Mounts the filesystem to automatically create the cpuset/cpusets/cpus/cpusets/mems/cpusets/tasks/cpusets/tasks  All CPUs and memory zones belong to the root cpuset  All existing PIDs are assigned to the root cpuset

测试:NUMA相关命令需要使⽤rhel的64位操作系统;[root@node1 ~]# numa(查看numa开头相关命令)numactl(显⽰命令) numad(服务器进程) numademo(事例命令) numastat(状态)

[root@node1 ~]# numastat(显⽰每个numa节点内存信息)

node0numa_hit 227475numa_miss 0numa_foreign 0interleave_hit 14980local_node 227475other_node 0[root@node1 ~]# man numastat(查看numastat的man帮助) numastat - Show per-NUMA-node memory statistics for processes and the operating system numa_hit is memory successfully allocated on this node as intended. numa_miss is memory allocated on this node despite the process preferring some different node. Each numa_miss has a numa_foreign on another node. numa_foreign is memory intended for this node, but actually allocated on some different node. Each .I numa_foreign has a .I numa_miss on another node. interleave_hit is interleaved memory successfully allocated on this node as intended. -p or (查看某个特定进程内存分配) Show per-node memory allocation information for the specified PID or pattern. If the -p argument is only digits, it is assumed to be a numerical PID. If the argument characters are not only digits, it is assumed to be a text fragment pattern to search for in process command lines. For example, numastat -p qemu will attempt to find and show information for processes with "qemu" in the command line. Any command line arguments remaining after numa- stat option flag processing is completed, are assumed to be additional or process specifiers. In this sense, the -p option flag is optional: numastat qemu is equivalent to numastat -p qemu -s[](查看某⼀个node) Sort the table data in descending order before displaying it, so the biggest memory consumers are listed first. With no specified , the table will be sorted by the total column. If the optional argument is sup- plied, the data will be sorted by the column. Note that must follow the -s immediately with no intermediate white space (e.g., numastat -s2).[root@node1 ~]# numastat -s(显⽰所有节点内存状态信息)Per-node numastat info (in MBs): Node 0 Total --------------- ---------------Numa_Hit 887.74 887.74Local_Node 887.74 887.74Interleave_Hit 58.52 58.52Numa_Foreign 0.00 0.00Numa_Miss 0.00 0.00Other_Node 0.00 0.00[root@node1 ~]# numastat -s node0(显⽰node0节点内存信息)Found no processes containing pattern: "node0"Per-node numastat info (in MBs): Node 0 Total --------------- ---------------Numa_Hit 886.74 886.74Local_Node 886.74 886.74Interleave_Hit 58.52 58.52Numa_Foreign 0.00 0.00Numa_Miss 0.00 0.00Other_Node 0.00 0.00[root@node1 ~]# man numactl(查看numactl的man帮助⽂件) numactl - Control NUMA policy for processes or shared memory(numa策略进程或共享内存控制) --cpunodebind=nodes, -N nodes(只将cpu运⾏在⾃⼰所属的Node上,将cpu跟node完成绑定,不让cpu去访问其他node) Only execute command on the CPUs of nodes. Note that nodes may consist of multiple CPUs. nodes may be specified as noted above.--physcpubind=cpus, -C cpus(将进程跟cpu绑定) Only execute process on cpus. This accepts cpu numbers as shown in the processor fields of /proc/cpuinfo, or

relative cpus as in relative to the current cpuset. You may specify "all", which means all cpus in the current cpuset. Physical cpus may be specified as N,N,N or N-N or N,N-N or N-N,N-N and so forth. Relative cpus may be specifed as +N,N,N or +N-N or +N,N-N and so forth. The + indicates that the cpu numbers are relative to the process’ set of allowed cpus in its current cpuset. A !N-N notation indicates the inverse of N-N, in other words all cpus except N-N. If used with + notation, specify !+N-N. --show, -s(显⽰当前进程运⾏的设定) Show NUMA policy settings of the current process.[root@node1 ~]# numactl --show(显⽰当前进程运⾏的设定)policy: defaultpreferred node: currentphyscpubind: 0 1

cpubind: 0

nodebind: 0

membind: 0

[root@node1 ~]# cat /proc/cpuinfo(查看cpu信息)

processor : 0vendor_id : GenuineIntelcpu family : 6model : 58model name : Intel(R) Core(TM) i5-3210M CPU @ 2.50GHzstepping : 9microcode : 18cpu MHz : 2494.389cache size : 3072 KBphysical id : 0siblings : 1core id : 0cpu cores : 1apicid : 0initial apicid : 0fpu : yesfpu_exception : yescpuid level : 13wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb pln pts dts fsgsbase smepbogomips : 4988.77clflush size : 64cache_alignment : 64address sizes : 42 bits physical, 48 bits virtualpower management:processor : 1vendor_id : GenuineIntelcpu family : 6model : 58model name : Intel(R) Core(TM) i5-3210M CPU @ 2.50GHzstepping : 9microcode : 18cpu MHz : 2494.389cache size : 3072 KBphysical id : 2siblings : 1core id : 0cpu cores : 1apicid : 2initial apicid : 2fpu : yesfpu_exception : yescpuid level : 13wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb pln pts dts fsgsbase smepbogomips : 4988.77clflush size : 64cache_alignment : 64address sizes : 42 bits physical, 48 bits virtualpower management:提⽰:numactl调整完成以后下次开机重启这些效果就没有了,因为这是命令强⾏将进程绑定;[root@node1 ~]# man numad(查看numad命令的man帮助) numad - A user-level daemon that provides placement advice and process management for efficient use of CPUs and memory on systems with NUMA topology.(⽤户空间级别的守护进程,能够提供策略⾃我启动,能够观察cpu上每⼀个运⾏状况来⾃动的将某进程给它关联到cpu上,将那个cpu关联到那个node上)[root@node1 ~]# numastat(显⽰每个numa节点内存信息)

node0numa_hit 230557numa_miss 0(查看miss量是不是很⾼)numa_foreign 0interleave_hit 14980local_node 230557other_node 0[root@node1 ~]# man taskset(查看taskset的man帮助) taskset - retrieve or set a process’s CPU affinity 0x00000001(cpu0) is processor #0 0x00000003(cpu0和cpu1) is processors #0 and #1 -c, --cpu-list(指定第⼏号cpu) specify a numerical list of processors instead of a bitmask. The list may contain multiple items, separated by comma, and ranges. For example, 0,5,7,9-11.[root@node1 ~]# cat /proc/irq/0/smp_affinity(查看第0号终端线smp_affinity⽂件内容)ffffffff,ffffffff,ffffffff,ffffffff(0号终端可以运⾏在所有cpu上)[root@node1 ~]# rpm -qf `which sar`(查看sar⽂件所属的程序包)6.x86_64[root@node1 ~]# rpm -ql sysstat(查看sysstat安装⽣成那些⽂件)/etc/cron.d/sysstat/etc/rc.d/init.d/sysstat/etc/sysconfig/sysstat/etc/sysconfig//usr/bin/cifsiostat/usr/bin/iostat/usr/bin/mpstat/usr/bin/pidstat/usr/bin/sadf/usr/bin/sar/usr/lib64/sa/usr/lib64/sa/sa1/usr/lib64/sa/sa2/usr/lib64/sa/sadc/usr/share/doc/sysstat-9.0.4/usr/share/doc/sysstat-9.0.4/CHANGES/usr/share/doc/sysstat-9.0.4/COPYING/usr/share/doc/sysstat-9.0.4/CREDITS/usr/share/doc/sysstat-9.0.4/FAQ/usr/share/doc/sysstat-9.0.4/README/usr/share/doc/sysstat-9.0.4/TODO/usr/share/locale/af/LC_MESSAGES//usr/share/locale/da/LC_MESSAGES//usr/share/locale/de/LC_MESSAGES//usr/share/locale/es/LC_MESSAGES//usr/share/locale/fi/LC_MESSAGES//usr/share/locale/fr/LC_MESSAGES//usr/share/locale/id/LC_MESSAGES//usr/share/locale/it/LC_MESSAGES//usr/share/locale/ja/LC_MESSAGES//usr/share/locale/ky/LC_MESSAGES//usr/share/locale/lv/LC_MESSAGES//usr/share/locale/mt/LC_MESSAGES//usr/share/locale/nb/LC_MESSAGES//usr/share/locale/nl/LC_MESSAGES//usr/share/locale/nn/LC_MESSAGES//usr/share/locale/pl/LC_MESSAGES//usr/share/locale/pt/LC_MESSAGES//usr/share/locale/pt_BR/LC_MESSAGES//usr/share/locale/ro/LC_MESSAGES//usr/share/locale/ru/LC_MESSAGES//usr/share/locale/sk/LC_MESSAGES//usr/share/locale/sv/LC_MESSAGES//usr/share/locale/vi/LC_MESSAGES//usr/share/locale/zh_CN/LC_MESSAGES//usr/share/locale/zh_TW/LC_MESSAGES//usr/share/man/man1//usr/share/man/man1//usr/share/man/man1//usr/share/man/man1//usr/share/man/man1//usr/share/man/man1//usr/share/man/man5//usr/share/man/man8//usr/share/man/man8//usr/share/man/man8//var/log/sa[root@node1 ~]# vmstat 1 7(查看当前cpu使⽤率)procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 320912 10260 114364 0 0 16 1 16 26 0 0 100 0 0

0 0 0 320872 10260 114384 0 0 0 0 33 50 0 0 100 0 0

0 0 0 320872 10260 114384 0 0 0 16 36 55 0 0 100 0 0

0 0 0 320872 10260 114384 0 0 0 0 20 42 0 0 100 0 0

0 0 0 320872 10260 114384 0 0 0 0 26 49 0 0 100 0 0

0 0 0 320872 10260 114384 0 0 0 0 21 48 0 0 100 0 0

0 0 0 320872 10268 114380 0 0 0 12 40 57 0 1 100 0 0

[root@node1 ~]# sar -q(查看当前cpu使⽤情况)Linux 6.x86_64 (iZ25j00d1grZ) 07/26/2016 _x86_64_ (1 CPU)12:00:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-1512:10:01 AM 0 179 0.00 0.00 0.0012:20:01 AM 0 179 0.00 0.00 0.0012:30:01 AM 0 179 0.00 0.00 0.0012:40:01 AM 0 179 0.00 0.00 0.0012:50:01 AM 0 179 0.00 0.00 0.0001:00:01 AM 2 183 0.00 0.00 0.0001:10:01 AM 0 179 0.00 0.00 0.0001:20:01 AM 0 178 0.00 0.00 0.0001:30:01 AM 0 178 0.00 0.00 0.0001:40:01 AM 0 178 0.00 0.00 0.0001:50:01 AM 0 178 0.00 0.00 0.0002:00:01 AM 2 182 0.00 0.00 0.0002:10:01 AM 0 178 0.00 0.00 0.0002:20:01 AM 0 178 0.00 0.00 0.0002:30:01 AM 0 178 0.00 0.00 0.0002:40:01 AM 0 179 0.00 0.00 0.0002:50:01 AM 0 178 0.00 0.00 0.0003:00:01 AM 2 182 0.00 0.00 0.0003:10:01 AM 0 179 0.00 0.00 0.0003:20:01 AM 0 179 0.00 0.00 0.0003:30:01 AM 0 179 0.00 0.01 0.0003:40:01 AM 0 179 0.00 0.00 0.0003:50:01 AM 0 179 0.00 0.00 0.0004:00:01 AM 2 182 0.00 0.00 0.0004:10:01 AM 0 178 0.00 0.00 0.0004:20:01 AM 0 178 0.00 0.00 0.0004:30:01 AM 0 178 0.00 0.00 0.0004:40:01 AM 0 178 0.00 0.00 0.0004:50:01 AM 0 178 0.00 0.00 0.0005:00:01 AM 2 182 0.00 0.00 0.00[root@node1 ~]# sar -q 1(查看当前cpu使⽤情况,实时采样)Linux 6.x86_64 () 2016年07⽉25⽇ _x86_64_ (2 CPU)23时56分40秒 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-1523时56分41秒 0 114 0.00 0.00 0.0023时56分42秒 0 114 0.00 0.00 0.0023时56分43秒 0 114 0.00 0.00 0.0023时56分44秒 0 114 0.00 0.00 0.0023时56分45秒 0 114 0.00 0.00 0.0023时56分46秒 0 114 0.00 0.00 0.0023时56分47秒 0 114 0.00 0.00 0.0023时56分48秒 0 114 0.00 0.00 0.0023时56分49秒 0 114 0.00 0.00 0.0023时56分50秒 0 114 0.00 0.00 0.00[root@node1 ~]# man sar(查看sar命令的man帮助) sar - Collect, report, or save system activity information. -b Report I/O and transfer rate statistics. The following values are displayed: -B Report paging statistics. Some of the metrics below are available only with post 2.5 kernels. The following values are displayed:(查看内存页⾯置换情况) -d Report activity for each block device (kernels 2.4 and newer only). When data is displayed, the device specifica- tion dev m-n is generally used ( DEV column). m is the major number of the device. With recent kernels (post 2.5), n is the minor number of the device, but is only a sequence number with pre 2.5 kernels. Device names may also be pretty-printed if option -p is used or persistent device names can be printed if option -j is used (see below). Val- ues for fields avgqu-sz, await, svctm and %util may be unavailable and displayed as 0.00 with some 2.4 kernels. Note that disk activity depends on sadc options "-S DISK" and "-S XDISK" to be collected. The following values are displayed:(查看每秒的事务数) -q Report queue length and load averages. The following values are displayed:(查看队列长度和负载平均值) runq-sz(运⾏队列的长度) Run queue length (number of tasks waiting for run time). plist-sz(task列表中进程的个数) Number of tasks in the task list. ldavg-1(过去1分钟的平均值) System load average for the last minute. The load average is calculated as the average number of runnable or running tasks (R state), and the number of tasks in uninterruptible sleep (D state) over the specified inter- val. ldavg-5(过去5分钟的平均值) System load average for the past 5 minutes. ldavg-15(过去15分钟的平均值) System load average for the past 15 minutes.[root@node1 ~]# sar -q 1(查看当前cpu使⽤情况,实时采样)Linux 6.x86_64 () 2016年07⽉26⽇ _x86_64_ (2 CPU)00时05分21秒 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-1500时05分22秒 0 120 0.00 0.00 0.0000时05分23秒 0 120 0.00 0.00 0.0000时05分24秒 0 120 0.00 0.00 0.0000时05分25秒 0 120 0.00 0.00 0.0000时05分26秒 0 120 0.00 0.00 0.0000时05分27秒 0 120 0.00 0.00 0.0000时05分28秒 0 120 0.00 0.00 0.0000时05分29秒 0 120 0.00 0.00 0.0000时05分30秒 0 120 0.00 0.00 0.0000时05分31秒 0 120 0.00 0.00 0.0000时05分32秒 0 120 0.00 0.00 0.0000时05分33秒 0 120 0.00 0.00 0.0000时05分34秒 0 120 0.00 0.00 0.0000时05分35秒 0 120 0.00 0.00 0.0000时05分36秒 0 120 0.00 0.00 0.0000时05分37秒 0 120 0.00 0.00 0.0000时05分38秒 0 120 0.00 0.00 0.00[root@node1 ~]# ab -n 10000 -c 300 127.0.0.1/(压⼒测试,-n执⾏指定请求数,-c每次多少请求数量,)This is ApacheBench, Version 2.3 <$Revision: 655654 $>Copyright 1996 Adam Twiss, Zeus Technology Ltd, /Licensed to The Apache Software Foundation, /Benchmarking 127.0.0.1 (be patient)Completed 1000 requestsCompleted 2000 requestsCompleted 3000 requestsCompleted 4000 requestsCompleted 5000 requestsCompleted 6000 requestsCompleted 7000 requestsCompleted 8000 requestsCompleted 9000 requestsCompleted 10000 requestsFinished 10000 requestsServer Software: Apache/2.2.15Server Hostname: 127.0.0.1Server Port: 80Document Path: /ument Length: 0 bytesConcurrency Level: 300Time taken for tests: 7.560 secondsComplete requests: 10000Failed requests: 0Write errors: 0Total transferred: 2661596 bytesHTML transferred: 0 bytesRequests per second: 1322.84 [#/sec] (mean)Time per request: 226.785 [ms] (mean)Time per request: 0.756 [ms] (mean, across all concurrent requests)Transfer rate: 343.83 [Kbytes/sec] receivedConnection Times (ms) min mean[+/-sd] median maxConnect: 0 2 37.4 1 3000Processing: 3 159 939.0 23 7539Waiting: 0 158 938.8 22 7537Total: 12 161 943.5 24 7549Percentage of the requests served within a certain time (ms) 50% 24 66% 26 75% 27 80% 27 90% 29 95% 34 98% 995 99% 7543 100% 7549 (longest request)[root@node1 ~]# sar -q 1(查看当前cpu使⽤情况,实时采样)Linux 6.x86_64 () 2016年07⽉26⽇ _x86_64_ (2 CPU)00时13分55秒 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-1500时13分56秒 1 363 0.33 0.08 0.0300时13分57秒 1 362 0.33 0.08 0.0300时13分58秒 11 361 0.33 0.08 0.0300时13分59秒 2 360 0.33 0.08 0.0300时14分00秒 1 360 0.31 0.08 0.0200时14分01秒 1 360 0.31 0.08 0.0200时14分02秒 1 359 0.31 0.08 0.0200时14分03秒 2 358 0.31 0.08 0.0200时14分04秒 1 357 0.31 0.08 0.0200时14分05秒 2 356 0.44 0.11 0.0400时14分06秒 2 355 0.44 0.11 0.0400时14分07秒 13 354 0.44 0.11 0.0400时14分08秒 2 353 0.44 0.11 0.0400时14分09秒 1 352 0.44 0.11 0.0400时14分10秒 6 351 0.49 0.12 0.0400时14分11秒 1 350 0.49 0.12 0.0400时14分12秒 3 349 0.49 0.12 0.0400时14分13秒 1 348 0.49 0.12 0.0400时14分14秒 2 347 0.49 0.12 0.0400时14分15秒 2 346 0.45 0.12 0.0400时14分16秒 3 345 0.45 0.12 0.0400时14分17秒 2 344 0.45 0.12 0.0400时14分18秒 2 343 0.45 0.12 0.04提⽰:runq-sz运⾏队列长度,[root@node1 ~]# man mpstat(查看mpstat命令的man帮助) mpstat - Report processors related statistics.(cpu相关使⽤状况) mpstat [ -A ] [ -I { SUM | CPU | ALL } ] [ -u ] [ -P { cpu [,...](查看那颗cpu) | ON | ALL } ] [ -V ] [ interval [ count ] ] -I { SUM | CPU | ALL }(cpu处理中断的次数) Report interrupts statistics.[root@node1 ~]# mpstat(查看cpu相关使⽤情况)

Linux 6.x86_64 () 2016年07⽉26⽇ _x86_64_ (2 CPU)00时31分46秒 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle00时31分46秒 all 0.63 0.00 1.13 0.18 0.00 0.73 0.00 0.00 97.33[root@node1 ~]# mpstat -P 0 1(查看第0号cpu使⽤情况,1表⽰每秒显⽰⼀次)Linux 6.x86_64 () 2016年07⽉26⽇ _x86_64_ (2 CPU)00时32分21秒 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle00时32分22秒 0 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.0100时32分23秒 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.0000时32分24秒 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00提⽰:%usr⽤户空间占⽤率,%sys内核空间的,%iowait I/O等待的,%irq⽤于处理中断的,%soft软中断的,%steal被虚拟机偷⾛的,%guest由虚拟机使⽤的,%idle空闲的;[root@node1 ~]# man mpstat(查看mpstat的man帮助) %irq(硬中断) Show the percentage of time spent by the CPU or CPUs to service hardware interrupts. %soft(软中断) Show the percentage of time spent by the CPU or CPUs to service software interrupts.[root@node1 ~]# mpstat -I CPU 1(查看cpu上对于中断处理)Linux 6.x86_64 () 2016年07⽉26⽇ _x86_64_ (2 CPU)02时02分26秒 CPU 0/s 1/s 8/s 9/s 12/s 14/s 15/s 16/s 17/s02时02分27秒 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.0018/s 19/s 24/s 25/s 26/s 27/s 28/s 29/s 30/s 31/s

0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0032/s 33/s 34/s 35/s 36/s 37/s 38/s 39/s 40/s 41/s 42/s0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0043/s 44/s 45/s 46/s 47/s 48/s 49/s 50/s 51/s 52/s

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0053/s 54/s 55/s NMI/s LOC/s SPU/s PMI/s IWI/s RES/s CAL/s TLB/s0.00 0.00 0.00 0TRM/s THR/s MCE/s MCP/s ERR/s MIS/s[root@node1 ~]# sar -P 0 1(查看第0号cpu每秒显⽰⼀次)Linux 6.x86_64 () 2016年07⽉26⽇ _x86_64_ (2 CPU)02时04分01秒 CPU %user %nice %system %iowait %steal %idle02时04分02秒 0 0.00 0.00 4.00 0.00 0.00 96.0002时04分03秒 0 0.00 0.00 3.00 0.00 0.00 97.0002时04分04秒 0 0.00 0.00 3.00 0.00 0.00 97.0002时04分05秒 0 0.00 0.00 3.00 0.00 0.00 97.00[root@node1 ~]# man iostat(查看iostat的man帮助) iostat - Report Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions and network filesystems (NFS).(报告cpu和i/o设备统计数据及分区和⽹络⽂件系统统计) -c Display the CPU utilization report.(显⽰cpu) -d Display the device utilization report.(显⽰设备)[root@node1 ~]# iostat -c(显⽰cpu使⽤状况)Linux 6.x86_64 () 2016年07⽉26⽇ _x86_64_ (2 CPU)avg-cpu: %user %nice %system %iowait %steal %idle 0.31 0.00 0.96 0.10 0.00 98.63[root@node1 ~]# iostat -c 1(显⽰cpu使⽤情况,每1秒刷新⼀次)Linux 6.x86_64 () 2016年07⽉26⽇ _x86_64_ (2 CPU)avg-cpu: %user %nice %system %iowait %steal %idle 0.31 0.00 0.96 0.10 0.00 98.63avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00[root@node1 ~]# iostat -c 1 6(显⽰cpu使⽤情况,每1秒刷新⼀次,总共刷6次)Linux 6.x86_64 () 2016年07⽉26⽇ _x86_64_ (2 CPU)avg-cpu: %user %nice %system %iowait %steal %idle 0.30 0.00 0.94 0.09 0.00 98.66avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.50 0.00 0.00 99.50提⽰:iowait数值过⾼,io设备可能称为瓶颈,%system在内核空间消耗的时间太长了,[root@node1 ~]# cat /proc/stat(查看cpu数据统计)

cpu 8159 0 15627 2735069 2535 18 9714 0 0cpu0 4185 0 7897 1367336 1257 18 4643 0 0cpu1 3974 0 7730 1367733 1277 0 5071 0 0intr 1391893 206 88 0 0 0 0 0 0 1 0 0 0 143 0 0 118 10860 7757 145 29467 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0ctxt 4647980btime 1469457843processes 3153procs_running 1procs_blocked 0softirq 9351126 0 799440 5552 6961625 7704 0 80 363031 989 1212705[root@node1 ~]# man dstat(查看dstat的man帮助) dstat - versatile tool for generating system resource statistics(系统资源统计数据) -c, --cpu enable cpu stats (system, user, idle, wait, hardware interrupt, software interrupt) -C 0,3,total include cpu0, cpu3 and total -d, --disk enable disk stats (read, write) -m, --mem enable memory stats (used, buffers, cache, free) -n, --net enable network stats (receive, send) -N eth1,total include eth1 and total -p, --proc enable process stats (runnable, uninterruptible, new) -s, --swap enable swap stats (used, free) -r, --io enable I/O request stats (read, write requests) --aio enable aio stats (asynchronous I/O) --fs enable filesystem stats (open files, inodes) --ipc enable ipc stats (message queue, semaphores, shared memory) --lock enable file lock stats (posix, flock, read, write) --socket enable socket stats (total, tcp, udp, raw, ip-fragments) --udp enable udp stats (listen, active) --unix enable unix stats (datagram, stream, listen, active) --vm enable vm stats (hard pagefaults, soft pagefaults, allocated, free) -v, --vmstat equals -pmgdsc -D total -a, --all equals -cdngy (default) --battery-remain(电池情况) battery remaining in hours, minutes (needs ACPI) --cpufreq(cpu频率) CPU frequency in percentage (needs ACPI) --disk-util(磁盘利⽤率) per disk utilization in percentage --gpfs GPFS read/write I/O (needs mmpmon) --innodb-buffer(innodb存储引擎的相关情况) show innodb buffer stats --lustre show lustre I/O throughput --memcache-hits(memcache命中率) show the number of hits and misses from memcache --mysql5-cmds show the MySQL5 command stats --top-cpu(谁最消耗cpu) show most expensive CPU process --top-cputime(最消耗cpu时间的) show process using the most CPU time (in ms) --top-cputime-avg(最消耗cpu时间⽚) show process with the highest average timeslice (in ms) --top-io show most expensive I/O process --top-latency(那个进程最⼤的延迟) show process with highest total latency (in ms) --top-mem(谁⽤了最多的内存) show process using the most memory[root@node1 ~]# dstat --top-cpu(显⽰那个进程最占⽤cpu)-most-expensive- cpu process

events/1 0.1

sshd: root@pt0.5events/0 0.5

mysqld 0.5[root@node1 ~]# dstat --top-mem(查看那个进程最占⽤内存)--most-expensive- memory process

mysqld 445Mmysqld 445Mmysqld 445Mmysqld 445M[root@node1 ~]# dstat --top-mem --top-cpu(查看最占⽤内存和cpu的进程)--most-expensive- -most-expensive- memory process | cpu process

mysqld 445M|events/1 0.1mysqld 445M|events/1 0.5[root@node1 ~]# dstat --top-mem --top-cpu --top-io(查看最占⽤内存和cpu的进程,谁最占i/o)--most-expensive- -most-expensive- ----most-expensive---- memory process | cpu process | i/o process

mysqld 445M|events/1 0.1|sshd 44k 6867Bmysqld 445M|events/1 0.5|crond 5595B 0

mysqld 445M| |sshd: root@ 155B 208Bmysqld 445M| |sshd: root@ 155B 208B[root@node1 ~]# dstat -c(显⽰cpu使⽤率)----total-cpu-usage----usr sys idl wai hiq siq 0 1 99 0 0 0 0 0 100 0 0 0 0 0 100 0 0 0 0 0 100 0 0 0[root@node1 ~]# vmstat 1(查看进程使⽤硬件情况)procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 2488 217176 8720 191188 0 0 5 4 47 148 0 1 99 0 0

0 0 2488 217168 8720 191188 0 0 0 0 44 57 0 1 100 0 0

0 0 2488 217168 8720 191188 0 0 0 0 34 59 0 0 100 0 0

0 0 2488 217168 8720 191188 0 0 0 0 29 51 0 0 100 0 0

提⽰:cs是上下⽂切换;[root@node1 ~]# man sar(查看sar命令的man帮助) -w Report task creation and system switching activity. proc/s Total number of tasks created per second. cswch/s Total number of context switches per second.(查看上下⽂件切换)[root@node1 ~]# sar -w 1(查看上下⽂件切换,每秒刷新⼀次)Linux 6.x86_64 () 07/26/2016 _x86_64_ (2 CPU)03:21:15 AM proc/s cswch/s03:21:16 AM 0.00 112.8703:21:17 AM 0.00 116.0003:21:18 AM 0.00 113.0003:21:19 AM 0.00 114.14提⽰:上下⽂切换次数过多,可能进程有点多;[root@node1 ~]# mkdir /cpusets(创建cpusets⽂件)[root@node1 ~]# vim /etc/fstab(编辑fstab⽂件)## /etc/fstab# Created by anaconda on Mon Jul 25 18:51:23 2016## Accessible filesystems, by reference, are maintained under '/dev/disk'# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info#UUID=cf43ec31-f59f-423f-aa0b-d091c18b2fa4 / ext4 defaults 1 1UUID=8d02aaa8-1714-4377-9925-a3d6e4cdca1e /boot ext4 defaults 1 2UUID=2273581c-907e-423b-95f3-cc95e035dfda swap swap defaults 0 0tmpfs /dev/shm tmpfs defaults 0 0devpts /dev/pts devpts gid=5,mode=620 0 0sysfs /sys sysfs defaults 0 0proc /proc proc defaults 0 0/cpuset /cpusets cpuset defaults 0 0[root@node1 ~]# mount -a(挂载/etc/fstab所有⽂件系统)[root@node1 ~]# mount(查看系统挂载的⽂件系统)/dev/sda2 on / type ext4 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")/dev/sda1 on /boot type ext4 (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)/cpuset on /cpusets type cpuset (rw)[root@node1 ~]# ls /cpusets/(查看/cpusets⽬录⽂件内容)_control cpus(根域) memory_migrate memory_spread_page notify_on_release sched_relax_domain_ mem_exclusive memory_pressure memory_spread_slab release_agent taskscpu_exclusive mem_hardwall memory_pressure_enabled mems(关联那些内存) sched_load_balance[root@node1 ~]# cat /cpusets/cpus(查看cpus⽂件内容)0-1[root@node1 ~]# cat /cpusets/mems(查看那⼀段内存属于这⾥)

0[root@node1 ~]# cat /cpusets/tasks(查看运⾏在这个域的进程有那些)

82936373946545556585967338764565676958234624462455247924982524254425692573257425782589232924292529262927292829292933293429352936293729382939294329442945294629472948294929532954295529562957295829592963296429652966296729682969297329742975297629772978298[root@node1 ~]# cd /cpusets/(切换到/cpusets⽬录)[root@node1 cpusets]# ls(查看当前⽬录⽂件及⼦⽬录)_control cpus memory_migrate memory_spread_page notify_on_release sched_relax_domain_ mem_exclusive memory_pressure memory_spread_slab release_agent taskscpu_exclusive mem_hardwall memory_pressure_enabled mems sched_load_balance[root@node1 cpusets]# mkdir domain1(创建domian1⽬录)[root@node1 cpusets]# ls(查看当前⽬录⽂件及⼦⽬录)_control cpus mem_hardwall memory_pressure_enabled mems sched_load_ domain1 memory_migrate memory_spread_page notify_on_release sched_relax_domain_levelcpu_exclusive mem_exclusive memory_pressure memory_spread_slab release_agent tasks[root@node1 cpusets]# cd domain1/(切换到domian1⽬录)[root@node1 domain1]# ls(查看当前⽬录⽂件及⼦⽬录)_control cpu_exclusive mem_exclusive memory_migrate memory_spread_page mems sched_load_balance cpus mem_hardwall memory_pressure memory_spread_slab notify_on_release sched_relax_domain_level提⽰:⾃动创建很多⽂件,设定这个域内cpu有那些,内存有那些,⽽后将进程绑定在这个域内;[root@node1 domain1]# cat cpus(查看cpus⽂件内容)

提⽰:刚创建⼦域⾥⾯没有绑定任何cpu;[root@node1 domain1]# cat mems(查看mems⽂件内容)

[root@node1 domain1]# cat tasks(查看tasks⽂件内容)[root@node1 domain1]# echo 0 > cpus(绑定0号cpu)[root@node1 domain1]# echo 0 > mems(绑定0段内存)提⽰:如果将某个进程绑定在这个域,那么这个进程就只能运⾏在这颗cpu和这个内存上;[root@node1 domain1]# ps axo pid,cmd(查看所有终端进程,只显⽰pid和cmd字段) PID CMD 1 /sbin/init 2 [kthreadd] 3 [migration/0] 4 [ksoftirqd/0] 5 [stopper/0] 6 [watchdog/0] 7 [migration/1] 8 [stopper/1] 9 [ksoftirqd/1] 10 [watchdog/1] 11 [events/0] 12 [events/1] 13 [cgroup] 14 [khelper] 15 [netns] 16 [async/mgr] 17 [pm] 18 [sync_supers] 19 [bdi-default] 20 [kintegrityd/0] 21 [kintegrityd/1] 22 [kblockd/0] 23 [kblockd/1] 24 [kacpid] 25 [kacpi_notify] 26 [kacpi_hotplug] 27 [ata_aux] 28 [ata_sff/0] 29 [ata_sff/1] 30 [ksuspend_usbd] 31 [khubd] 32 [kseriod] 33 [md/0] 34 [md/1] 35 [md_misc/0] 36 [md_misc/1] 37 [linkwatch] 39 [khungtaskd] 40 [kswapd0] 41 [ksmd] 42 [khugepaged] 43 [aio/0] 44 [aio/1] 45 [crypto/0] 46 [crypto/1] 54 [kthrotld/0] 55 [kthrotld/1] 56 [pciehpd] 58 [kpsmoused] 59 [usbhid_resumer] 60 [deferwq] 89 [kdmremove] 90 [kstriped] 165 [scsi_eh_0] 166 [scsi_eh_1] 170 [mpt_poll_0] 171 [mpt/0] 172 [scsi_eh_2] 302 [jbd2/sda2-8] 303 [ext4-dio-unwrit] 387 /sbin/udevd -d 645 [bluetooth] 656 [vmmemctl] 700 [jbd2/sda1-8] 701 [ext4-dio-unwrit] 740 [kauditd] 979 /sbin/dhclient -1 -q -lf /var/lib/dhclient/ -pf /var/run/ eth1 1024 auditd 1044 /sbin/rsyslogd -i /var/run/ -c 5 1091 dbus-daemon --system 1129 /usr/sbin/sshd 1146 /bin/sh /usr/local/mysql/bin/mysqld_safe --datadir=/mydata/data --pid-file=/mydata/data/ 1432 /usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/mydata/data --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql - 1481 sendmail: accepting connections 1491 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue 1502 crond 1515 login -- root

1517 /sbin/mingetty /dev/tty2 1519 /sbin/mingetty /dev/tty3 1521 /sbin/mingetty /dev/tty4 1523 /sbin/mingetty /dev/tty5 1525 /sbin/mingetty /dev/tty6 1533 /sbin/udevd -d 1534 /sbin/udevd -d 1981 /usr/sbin/httpd 2399 /usr/sbin/httpd 2426 /usr/sbin/httpd 2446 /usr/sbin/httpd 2455 /usr/sbin/httpd 2479 /usr/sbin/httpd 2498 /usr/sbin/httpd 2524 /usr/sbin/httpd 2544 /usr/sbin/httpd 2560 /usr/sbin/httpd 2561 /usr/sbin/httpd 2564 /usr/sbin/httpd 2569 /usr/sbin/httpd 2573 /usr/sbin/httpd 2574 /usr/sbin/httpd 2578 /usr/sbin/httpd 2581 /usr/sbin/httpd 2583 /usr/sbin/httpd 2590 /usr/sbin/httpd 2592 /usr/sbin/httpd 2615 /usr/sbin/httpd 2916 /usr/sbin/console-kit-daemon --no-daemon 2983 -bash 3001 sshd: root@pts/0

3010 -bash 12828 [flush-8:0] 12892 ps axo pid,cmd[root@node1 domain1]# echo 2615 > tasks(将2615进程绑定到该区域)[root@node1 domain1]# man ps(查看ps命令的man帮助)psr PSR processor that process is currently assigned to.(那个进程分配给那颗cpu)[root@node1 domain1]# ps -e -o psr,pid,cmd | grep httpd(查看所有进程,只显⽰psr,pid,cmd字段将结果送给管道只显⽰httpd相关) 1 1981 /usr/sbin/httpd 1 2399 /usr/sbin/httpd 1 2426 /usr/sbin/httpd 1 2446 /usr/sbin/httpd 1 2455 /usr/sbin/httpd 1 2479 /usr/sbin/httpd 1 2498 /usr/sbin/httpd 1 2524 /usr/sbin/httpd 1 2544 /usr/sbin/httpd 1 2560 /usr/sbin/httpd 1 2561 /usr/sbin/httpd 1 2564 /usr/sbin/httpd 1 2569 /usr/sbin/httpd 1 2573 /usr/sbin/httpd 1 2574 /usr/sbin/httpd 0 2578 /usr/sbin/httpd 1 2581 /usr/sbin/httpd 1 2583 /usr/sbin/httpd 1 2590 /usr/sbin/httpd 1 2592 /usr/sbin/httpd 1 2615 /usr/sbin/httpd 1 12921 grep httpd[root@node1 domain1]# cat tasks(查看tasks⽂件内容)

2615[root@node1 domain1]# watch -n 0.5 'ps -e -o psr,pid,cmd | grep httpd'(每0.5秒执⾏ps命令,查看所有进程,只显⽰psr,pid,cmd字段将结果送给管道只显⽰httpd相关)Every 0.5s: ps -e -o psr,pid,cmd | grep httpd Tue Jul 26 04:05:12 2016 0 1981 /usr/sbin/httpd 1 2399 /usr/sbin/httpd 1 2426 /usr/sbin/httpd 1 2446 /usr/sbin/httpd 1 2455 /usr/sbin/httpd 1 2479 /usr/sbin/httpd 1 2498 /usr/sbin/httpd 1 2524 /usr/sbin/httpd 1 2544 /usr/sbin/httpd 1 2560 /usr/sbin/httpd 1 2561 /usr/sbin/httpd 1 2564 /usr/sbin/httpd 1 2569 /usr/sbin/httpd 1 2573 /usr/sbin/httpd 1 2574 /usr/sbin/httpd 0 2578 /usr/sbin/httpd 1 2581 /usr/sbin/httpd[root@node1 ~]# ab -n 10000 -c 300 127.0.0.1/(http压⼒测试,-n只i的那个总共请求数量,-c指定⼀次多少请求)This is ApacheBench, Version 2.3 <$Revision: 655654 $>Copyright 1996 Adam Twiss, Zeus Technology Ltd, /Licensed to The Apache Software Foundation, /Benchmarking 127.0.0.1 (be patient)Completed 1000 requestsCompleted 2000 requestsCompleted 3000 requestsCompleted 4000 requestsCompleted 5000 requestsCompleted 6000 requestsCompleted 7000 requestsCompleted 8000 requestsCompleted 9000 requestsCompleted 10000 requestsFinished 10000 requestsServer Software: Apache/2.2.15Server Hostname: 127.0.0.1Server Port: 80Document Path: /ument Length: 0 bytesConcurrency Level: 300Time taken for tests: 3.077 secondsComplete requests: 10000Failed requests: 0Write errors: 0Total transferred: 2660000 bytesHTML transferred: 0 bytesRequests per second: 3250.03 [#/sec] (mean)Time per request: 92.307 [ms] (mean)Time per request: 0.308 [ms] (mean, across all concurrent requests)Transfer rate: 844.25 [Kbytes/sec] receivedConnection Times (ms) min mean[+/-sd] median maxConnect: 0 6 54.9 2 1028Processing: 0 63 295.7 27 3027Waiting: 0 62 295.8 26 3027Total: 20 69 302.7 30 3053Percentage of the requests served within a certain time (ms) 50% 30 66% 31 75% 33 80% 33 90% 35 95% 37 98% 561 99% 1468 100% 3053 (longest request)提⽰:2615进程不会切换,⼀直在0号cpu上;[root@node1 domain1]# cd ..(切换到上级⽬录)[root@node1 cpusets]# ls(查看当前⽬录⽂件及⼦⽬录)_control cpus mem_hardwall memory_pressure_enabled mems sched_load_ domain1 memory_migrate memory_spread_page notify_on_release sched_relax_domain_levelcpu_exclusive mem_exclusive memory_pressure memory_spread_slab release_agent tasks[root@node1 cpusets]# cat /etc/fstab(查看fstab⽂件内容)

## /etc/fstab# Created by anaconda on Mon Jul 25 18:51:23 2016## Accessible filesystems, by reference, are maintained under '/dev/disk'# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info#UUID=cf43ec31-f59f-423f-aa0b-d091c18b2fa4 / ext4 defaults 1 1UUID=8d02aaa8-1714-4377-9925-a3d6e4cdca1e /boot ext4 defaults 1 2UUID=2273581c-907e-423b-95f3-cc95e035dfda swap swap defaults 0 0tmpfs /dev/shm tmpfs defaults 0 0devpts /dev/pts devpts gid=5,mode=620 0 0sysfs /sys sysfs defaults 0 0proc /proc proc defaults 0 0/cpuset /cpusets cpuset defaults 0 0[root@node1 cpusets]# ps -e -o psr,pid,cmd | grep httpd(查看所有进程,只显⽰psr,pid,cmd字段将结果送给管道只显⽰httpd相关) 0 1981 /usr/sbin/httpd 1 2524 /usr/sbin/httpd 1 14408 /usr/sbin/httpd 1 14447 /usr/sbin/httpd 1 14477 /usr/sbin/httpd 1 14496 /usr/sbin/httpd 1 14505 /usr/sbin/httpd 1 14512 /usr/sbin/httpd 1 14529 /usr/sbin/httpd 1 14532 /usr/sbin/httpd 1 14533 /usr/sbin/httpd 1 14535 /usr/sbin/httpd 1 14538 /usr/sbin/httpd 1 14547 /usr/sbin/httpd 1 14555 /usr/sbin/httpd 1 14558 /usr/sbin/httpd 1 14561 /usr/sbin/httpd 1 14563 /usr/sbin/httpd 1 14574 /usr/sbin/httpd 0 14575 /usr/sbin/httpd 1 14821 /usr/sbin/httpd 1 14870 grep httpd[root@node1 cpusets]# taskset -p -c 0 14821(将14821进程绑定在0号cpu)pid 14821's current affinity list: 0,1pid 14821's new affinity list: 0[root@node1 cpusets]# taskset -p -c 1 14821(将14821进程绑定1号cpu)pid 14821's current affinity list: 0pid 14821's new affinity list: 1

发布者:admin,转转请注明出处:http://www.yc00.com/web/1688329579a121141.html

相关推荐

发表回复

评论列表(0条)

  • 暂无评论

联系我们

400-800-8888

在线咨询: QQ交谈

邮件:admin@example.com

工作时间:周一至周五,9:30-18:30,节假日休息

关注微信