一、前提

1 nginx編譯安裝

假設已經安裝好了nginx服務器，切能正常啟動訪問，個人版本為 `nginx version: nginx/1.14.0`，使用strace/pstack進行調試（或者gdb，沒調試過）。strace 和ltrace 可以查看一個應用程序在運行過程中所發起的系統調用

2 strace命令

strace - trace system calls and signals 跟蹤系統調用和信號，最重要的參數 -p，

strace -p pid

Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to many processes in addition to command (which is optional if at least one -p option is given). -p "`pidof PROG`" syntax is supported

詳細參數，自行查看 man strace（yum install strace）

3 nginx配置

nginx採用master-worker模式，且只有一個worker進程，方便調試

<code>worker_processes  1;
events {
    worker_connections  1024;
}/<code>

二、gstack 調試

<code>ps aux | grep nginx | grep -v grep
www      13366  0.0  0.1  21784  2164 ?        S    Mar23  0:00 nginx: worker process
root    15221  0.0  0.1  21784  2132 ?        ts  Mar13  0:00 nginx: master process /usr/local/nginx/sbin/nginx
/<code>

<code>
[@bx16-77-246 /usr/local/nginx]# gstack 13366
#0  0x00007fe98da830f3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x00000000004320b8 in ngx_epoll_process_events (cycle=0xd7e1b0, timer=18446744073709551615, flags=1) at src/event/modules/ngx_epoll_module.c:800
#2  0x0000000000429c48 in ngx_process_events_and_timers (cycle=cycle@entry=0xd7e1b0) at src/event/ngx_event.c:242
#3  0x0000000000430644 in ngx_worker_process_cycle (cycle=0xd7e1b0, data=<optimized>) at src/os/unix/ngx_process_cycle.c:750
#4  0x000000000042ed72 in ngx_spawn_process (cycle=cycle@entry=0xd7e1b0, proc=proc@entry=0x4305d3 , data=data@entry=0x0, name=name@entry=0x480ead "worker process", respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:199
#5  0x000000000042f90b in ngx_start_worker_processes (cycle=cycle@entry=0xd7e1b0, n=1, type=type@entry=-4) at src/os/unix/ngx_process_cycle.c:359
#6  0x0000000000431358 in ngx_master_process_cycle (cycle=cycle@entry=0xd7e1b0) at src/os/unix/ngx_process_cycle.c:244
#7  0x000000000040c84b in main (argc=<optimized>, argv=<optimized>) at src/core/nginx.c:382/<optimized>/<optimized>/<optimized>/<code>

從main()函數到epoll_wait()函數的調用關係一目瞭然，說明：（調用棧從下自上）

1 入口是nginx.c文件裡的main函數，解析命令行參數，初始化

2 根據ngx_process變量，進入分支ngx_master_process_cycle（另一個分支是ngx_single_process_cycle）

3 進入master後，就需要啟動worker進程，調用ngx_start_worker_processes

4 調用ngx_spawn_process啟動worker進行，並執行回調ngx_worker_process_cycle

5 進入worker主循環，其中最重要的邏輯是ngx_process_events_and_timers

6 調用ngx_worker_process_init，執行各個模塊的init鉤子，並設置action為ngx_epoll_process_events

7 epoll開始進入事件等待 __epoll_wait_nocancel

三、strace 調試

<code>#參數1：9是epoll_create創建文件描述符，等待事件觸發，1代表rdlist就緒隊列有事件要處理
epoll_wait(9, [{EPOLLIN, {u32=14344048, u64=14344048}}], 512, -1) = 1
#接收client請求，3是socket的文件描述符
accept4(17, {sa_family=AF_INET, sin_port=htons(57695), sin_addr=inet_addr("10.2.82.135")}, [16], SOCK_NONBLOCK) = 3
#將3 （accept4 產生）加入到epoll的事件監聽機制
epoll_ctl(9, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=14344720, u64=14344720}}) = 0
#等待就緒事件，1表示有事件要處理
epoll_wait(9, [{EPOLLIN, {u32=14344720, u64=14344720}}], 512, 60000) = 1
#從client接收數據，長度484字節
recvfrom(3, "GET /ae/test HTTP/1.1\\r\\nHost: tem"..., 16384, 0, NULL, NULL) = 484
#判斷_not_exists_是否存在，-1表示不存在
stat("/usr/local/nginx/html/_not_exists_", 0x7fff9e42c1c0) = -1 ENOENT (No such file or directory)
#更新事件註冊
epoll_ctl(9, EPOLL_CTL_MOD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=14344720, u64=14344720}}) = 0
#初始化新的socket 4
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 4
#對設備的I/O通道進行管理，0表示成功
ioctl(4, FIONBIO, [1])                  = 0
#將socket 4 加入epoll的事件監聽機制
epoll_ctl(9, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=14344944, u64=14344944}}) = 0
# socket 4 建立連接
connect(4, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
#rdlist就緒隊列裡有兩個事件待處理
epoll_wait(9, [{EPOLLOUT, {u32=14344720, u64=14344720}}, {EPOLLOUT, {u32=14344944, u64=14344944}}], 512, 60000) = 2
#獲取套接字的選項信息
# 第一個參數：socket套接字
# 第二個參數：套接字層次
# 第三個參數：要獲取的信息項 

getsockopt(4, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
#響應頭509字節內容寫入socket4緩衝區
writev(4, [{"GET /ae/test HTTP/1.0\\r\\nX-Forward"..., 509}], 1) = 509
#rdlist就緒隊列裡有一個事件待處理，即socket 3
epoll_wait(9, [{EPOLLIN|EPOLLOUT|EPOLLRDHUP, {u32=14344944, u64=14344944}}], 512, 60000) = 1
#從socket 4 讀取響應頭及內容
recvfrom(4, "HTTP/1.0 200 OK\\r\\nContent-Length:"..., 4096, 0, NULL, NULL) = 4096
readv(4, [{"m' => 'input',\\n  "..., 4096}], 1) = 4096
readv(4, [{"  'file_size' => 0,\\n "..., 4096}], 1) = 4096
readv(4, [{"put',\\r\\n      'rule' "..., 4096}], 1) = 4096
readv(4, [{"#39;2' => array(\\r\\n      "..., 4096}], 1) = 4096
readv(4, [{"  'file_size' => arr"..., 4096}], 1) = 4096
readv(4, [{"\\r\\n        'file_size' =&"..., 4096}], 1) = 4096
readv(4, [{"            'predownload'"..., 4096}], 1) = 4096
readv(4, [{"y (\\n      'label' => "..., 4096}], 1) = 4096
#向socket 3 寫入數據
writev(3, [{"HTTP/1.1 200 OK\\r\\nServer: nginx/1"..., 165}, {"\\n\\n  \\n"..., 3949}, {"m' => 'input',\\n  "..., 4096}], 3) = 8210
writev(3, [{"  'file_size' => 0,\\n "..., 4096}, {"put',\\r\\n      'rule' "..., 4096}], 2) = 8192
writev(3, [{"#39;2' => array(\\r\\n      "..., 4096}, {"  'file_size' => arr"..., 4096}], 2) = 8192
writev(3, [{"\\r\\n        'file_size' =&"..., 4096}, {"            'predownload'"..., 4096}], 2) = 8192
writev(3, [{"y (\\n      'label' => "..., 4096}], 1) = 4096
readv(4, [{"  array(\\n                    \u0003"..., 4096}, {"9;file_ext' => 'mp4,g"..., 4096}, {"\\r\\n    
9 array(\\r\\n    'gotourl"..., 4096}, {"tle\" id=\"exampleModalScrollableT"..., 4096}, {"", 4096}, {"", 4096}, {"", 4096}, {"", 4096}], 9) = 17810
readv(4, [{"", 2670}, {"", 4096}, {"", 4096}, {"", 4096}, {"", 4096}], 5) = 0
writev(3, [{"  array(\\n                    \u0003"..., 4096}, {"9;file_ext' => 'mp4,g"..., 4096}, {"\\r\\n    
9 array(\\r\\n    'gotourl"..., 4096}, {"tle\" id=\"exampleModalScrollableT"..., 1426}], 5) = 17810
#關閉socket 4
close(4)                                = 0
write(28, "10.2.82.135 - - [24/Mar/2020:12:"..., 220) = 220
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
#查看epoll的rdlist就緒隊列是否有事件，0表示無
epoll_wait(9, [], 512, 65000)          = 0
#關閉socket 3
close(3)                                = 0
#繼續等待事件觸發
epoll_wait(9,

整個過程，你應該清晰的知道epoll對象上的rdlist就緒隊列和rbtree的變化情況 

延伸閱讀
[nginx 源碼註釋項目](https://blog.csdn.net/cjqh_hao/article/details/104076693)
參考文檔
1 [epoll_create](http://www.man7.org/linux/man-pages/man2/epoll_create.2.html#NAME)
2 [strace工具](https://people.gnome.org/~newren/tutorials/developing-with-gnome/html/ch03s02.html)
3 [nginx 源碼註釋](https://github.com/HelloMrShu/nginx_comment_1.17.x)
"
/<code>

閱讀更多 martianboy 的文章

關鍵字: Nginx 深度 源碼