TCP Socket Internals: Kernel-Level Lifecycle, Buffers & Queues Explained
This article dissects the complete TCP socket lifecycle from a kernel perspective, covering socket creation, three-way handshake, connection queues, send/recv buffer behavior, sticky packet solutions, and graceful closure with code examples and troubleshooting commands.
From an application perspective, a socket is simply a file descriptor returned by socket(). Behind this integer, the kernel maintains a data structure holding peer IP, port, send/receive buffers, congestion window, sliding window, and the TCP state machine. Calling send() copies user data into the kernel send buffer; the protocol stack decides when to segment, transmit, and retransmit.
Three-Way Handshake: Kernel Actions
The handshake synchronizes initial sequence numbers (ISN) and verifies bidirectional reachability.
Client sends SYN with seq=x, enters SYN_SENT.
Server replies SYN+ACK with its own seq=y and ack=x+1, enters SYN_RCVD.
Client sends ACK with ack=y+1 and seq=x+1; both sides enter ESTABLISHED.
SYN/FIN each consume one sequence number; pure ACK does not. TIME_WAIT lasts 2MSL to ensure the final ACK reaches the peer and lets stale packets expire.
Why Not Two Handshakes?
Two handshakes cannot confirm the client's receive capability and cannot prevent delayed duplicate SYNs from creating spurious connections. The third ACK proves the client is alive.
Half-Connection vs Full-Connection Queues
Upon receiving SYN, the kernel places the connection in the half-connection queue (syns queue) while in SYN_RCVD. After the third ACK, it moves to the full-connection queue (accept queue) awaiting accept().
Half-connection queue : holds SYN_RCVD connections; overflow drops new SYNs (client retransmits); controlled by tcp_max_syn_backlog.
Full-connection queue : holds established connections not yet accepted; overflow behavior set by tcp_abort_on_overflow (drop ACK or send RST); sized by listen() backlog argument.
Use ss -lnt to inspect Recv-Q (current full-queue length) and Send-Q (queue limit). A persistently full Recv-Q indicates slow accept() calls.
send and recv Internals
(1) send Success ≠ Data Delivered
In blocking mode, send() returns once data is copied to the kernel send buffer. Actual transmission, ACK processing, and retransmission happen asynchronously. If the peer crashes after send() returns, the application learns of the failure only on a subsequent write() (yielding EPIPE) or read() (yielding ECONNRESET). Writing to a broken pipe also raises SIGPIPE by default.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <signal.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <errno.h>
int main(void) {
signal(SIGPIPE, SIG_IGN);
int lfd = socket(AF_INET, SOCK_STREAM, 0);
int opt = 1;
setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
struct sockaddr_in serv_addr;
serv_addr.sin_family = AF_INET;
serv_addr.sin_port = htons(9999);
serv_addr.sin_addr.s_addr = INADDR_ANY;
bind(lfd, (struct sockaddr *)&serv_addr, sizeof(serv_addr));
listen(lfd, 5);
printf("server listen on 9999
");
printf("1. telnet 127.0.0.1 9999
");
printf("2. Close telnet abruptly (sends RST)
");
printf("3. Observe two send returns
");
int cfd = accept(lfd, NULL, NULL);
printf("client connected, fd=%d
", cfd);
const char *msg = "test data";
ssize_t n;
n = send(cfd, msg, strlen(msg), 0);
printf("first send return: %zd, errno=%d(%s)
", n, errno, strerror(errno));
sleep(4);
n = send(cfd, msg, strlen(msg), 0);
printf("second send return: %zd, errno=%d(%s)
", n, errno, strerror(errno));
char buf[128];
n = recv(cfd, buf, sizeof(buf)-1, 0);
printf("recv return: %zd, errno=%d(%s)
", n, errno, strerror(errno));
close(cfd);
close(lfd);
return 0;
}EPIPEoccurs when writing to a closed connection (accompanied by SIGPIPE); ECONNRESET signals receipt of an RST; recv() returning 0 indicates graceful closure via FIN.
Interview follow-up: Can send() wait for ACK before returning? Standard send() has no such option; TCP ACKs are kernel-internal. Application-level acknowledgments are required for business-level delivery confirmation.
(2) TCP Is a Byte Stream — No Message Boundaries
TCP guarantees in-order byte delivery but does not preserve application write boundaries. Two 100-byte sends may be received as one 200-byte read or ten 20-byte reads — the classic "sticky packet" and "unpacking" problem. The protocol itself has no concept of packets; framing is an application-layer concern.
Framing Solutions
Fixed-length messages : Pad to a constant size (e.g., 128 bytes). Simple but wastes bandwidth and forbids variable-length payloads.
#define PKG_FIX_LEN 128
int send_fixed_pkg(int fd, const char *data, int len) {
char buf[PKG_FIX_LEN] = {0};
int copy_len = len < PKG_FIX_LEN ? len : PKG_FIX_LEN;
memcpy(buf, data, copy_len);
return send(fd, buf, PKG_FIX_LEN, 0);
}
int recv_fixed_pkg(int fd, char *out) {
int total = 0;
while (total < PKG_FIX_LEN) {
int n = recv(fd, out + total, PKG_FIX_LEN - total, 0);
if (n <= 0) return n;
total += n;
}
return total;
}Delimiter-based : Use a unique marker (e.g., HTTP's \r\n\r\n) to separate header and body. Requires escaping if payload contains the delimiter; parsing scans byte-by-byte.
int recv_delim_pkg(int fd, char *buf, int buf_sz, const char *delim) {
int total = 0;
while (1) {
int n = recv(fd, buf + total, buf_sz - total - 1, 0);
if (n <= 0) return n;
total += n;
buf[total] = '\0';
if (strstr(buf, delim)) return total;
}
}Correction: HTTP uses \r\n\r\n only to delimit headers; the body relies on Content-Length or chunked encoding — a common interview mistake.
Length-prefixed (industry standard) : Prepend a 4-byte big-endian length field. Receiver reads 4 bytes, converts to host order, then reads exactly that many payload bytes. Efficient, no escaping needed, but must handle partial reads and network byte order.
#include <arpa/inet.h>
#include <string.h>
#include <unistd.h>
int send_len_prefix_pkg(int fd, const char *body, int body_len) {
char head[4];
uint32_t net_len = htonl((uint32_t)body_len);
memcpy(head, &net_len, 4);
if (send(fd, head, 4, 0) != 4) return -1;
return send(fd, body, body_len, 0);
}
int recv_len_prefix_pkg(int fd, char *out, int max_body_sz) {
char head_buf[4];
uint32_t body_len;
int total = 0;
while (total < 4) {
int n = recv(fd, head_buf + total, 4 - total, 0);
if (n <= 0) return n;
total += n;
}
memcpy(&body_len, head_buf, 4);
body_len = ntohl(body_len);
if (body_len > max_body_sz) return -1;
total = 0;
while (total < (int)body_len) {
int n = recv(fd, out + total, body_len - total, 0);
if (n <= 0) return n;
total += n;
}
return (int)body_len;
}Minimal TCP Echo Server
The following blocking example illustrates each system call's kernel interaction.
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <string.h>
#include <stdio.h>
int main() {
int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
int opt = 1;
setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
sockaddr_in addr;
memset(&addr, 0, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_ANY);
addr.sin_port = htons(8080);
bind(listen_fd, (sockaddr*)&addr, sizeof(addr));
listen(listen_fd, 128);
while (true) {
sockaddr_in client;
socklen_t len = sizeof(client);
int conn_fd = accept(listen_fd, (sockaddr*)&client, &len);
char buf[1024];
ssize_t n;
while ((n = recv(conn_fd, buf, sizeof(buf), 0)) > 0) {
send(conn_fd, buf, n, 0);
}
close(conn_fd);
}
close(listen_fd);
return 0;
}Compile: g++ echo_server.cpp -o echo_server && ./echo_server. Test with nc 127.0.0.1 8080. socket() returns an unbound fd; bind() associates it with a port.
After listen(), the fd can only be used for accept(), not read/write. accept() returns a new conn_fd for data exchange. recv() returns 0 on graceful close, -1 on error — handle distinctly.
This example is blocking and single-threaded; production uses epoll + non-blocking I/O.
Operational Troubleshooting Commands
ss -lnt— view listening ports and full-connection queue usage. ss -ant state time-wait | wc -l — count TIME_WAIT sockets. netstat -s | grep -i listen — show full-queue overflows and SYN drops. tcpdump -i any port 8080 -nn — capture handshake/teardown sequences.
Conceptually, a socket is "a file descriptor + a kernel connection state machine + two buffers." Tracing the "establish → transfer → close" lifecycle equips you to answer most TCP socket interview follow-ups.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
