Secure Live HLS Streaming: Nginx, RTMP, and Lua Authentication Guide

HLS Overview

HTTP Live Streaming (HLS) is an Apple‑proposed protocol that delivers media over HTTP by splitting the stream into small TS segments and an extended M3U8 playlist. Clients download the playlist, then fetch segments sequentially, allowing adaptive bitrate selection based on network conditions. Because it uses plain HTTP, HLS works well through firewalls, proxies, CDNs, and supports load balancing, but it introduces higher latency for live streams.

Advantages and Disadvantages

Advantages : Simple client support (HTML5 video), good network compatibility, easy firewall traversal, straightforward load balancing, strong CDN support, built‑in multi‑bitrate adaptation.

Disadvantages : Higher latency unsuitable for interactive live scenarios; many TS fragments require dynamic generation and cleanup, demanding more complex storage logic for high performance.

Live Encryption Implementation

Playable HLS URLs

https://hls-auth.tinywan.com/hls/202403101/index.m3u8

https://hls-auth.tinywan.com/hls/202403102/index.m3u8

HTTPS Key Requirements

Install a trusted SSL certificate signed by a recognized authority on the HTTPS server.

The key file’s domain must match the domain of the first playlist; the simplest approach is to serve the variant playlist over HTTPS while other playlists can use HTTP.

You must provide an authentication dialog or store credentials on the client device, as HLS itself does not supply a user prompt. In custom iOS clients, credentials can be stored (cookie or HTTP digest) and supplied in the didReceiveAuthenticationChallenge callback.

Note: you must obtain a trusted SSL certificate to use HTTPS with HTTP Live Streaming.

RTMP Live Configuration

rtmp {
    notify_method get;
    server {
        listen 1935;
        chunk_size 4000;
        application live {
            live on;
            hls on;
            hls_path /tmp/hls;
            hls_fragment 6;
            hls_keys on;
            hls_key_path /tmp/keys;
            hls_key_url https://hls-auth.tinywan.com/keys/;
            hls_fragments_per_key 10;
            hls_playlist_length 36s;
            hls_sync 100ms;
            hls_fragment_naming system;
            hls_fragment_slicing aligned;
        }
    }
}

Push URL: rtmp://live.tinywan.com/live_auth/123

HLS Virtual Host Configuration

server {
    listen 443 ssl http2;
    server_name live.tinywan.com;
    ssl on;
    ssl_certificate /etc/letsencrypt/live/www.tinywan.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/www.tinywan.com/privkey.pem;
    server_tokens off;
    location /live {
        add_header Cache-Control no-cache;
        add_header 'Access-Control-Allow-Origin' '*'
            always;
        add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range';
        add_header 'Access-Control-Allow-Headers' 'Range';
        types {
            application/vnd.apple.mpegurl m3u8;
            video/mp2t ts;
        }
        root /tmp;
    }
}

Configuration notes:

Play stream URL: https://live.tinywan.com/live_auth/12345.m3u8 m3u8 file directory: /tmp/live_auth Matching principle: location uses prefix matching. Without a trailing slash, location /abc/def matches /abc/defghi and /abc/def/ghi. With a trailing slash, location /abc/def/ only matches paths that start with that exact prefix, e.g., /abc/def/anything.

Therefore, location /live matches the address /live_auth/12345.m3u8.

Final matched file path:

/tmp/live_auth/12345.m3u8

HLS‑KEY Virtual Host Configuration

server {
    listen 443 ssl http2;
    server_name hls-auth.tinywan.com;
    ssl on;
    ssl_certificate /etc/letsencrypt/live/www.tinywan.com-0002/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/www.tinywan.com-0002/privkey.pem;
    server_tokens off;
    # hls keys
    location /keys {
        add_header Cache-Control no-cache;
        add_header 'Access-Control-Allow-Origin' '*'
            always;
        add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range';
        add_header 'Access-Control-Allow-Headers' 'Range';
        access_by_lua_file /usr/local/openresty/nginx/conf/lua/filter_domain_ip_limit.lua;
        root /tmp;
    }
}

Lua Script filter_domain_ip_limit.lua

local redis = require("resty.redis")
local var = ngx.var
local log = ngx.log
local ERR = ngx.ERR
local EXIT = ngx.exit

-- redis config
local redis_host = "127.0.0.1"
local redis_port = 6379
local redis_auth = "tinywanredisamaistream"
local redis_timeout = 1000

local red = redis:new()
red:set_timeout(redis_timeout)
local ok, err = red:connect(redis_host, redis_port)
if not ok then
    log(ERR, "connect to redis error : ", err)
end

-- client info
local client_ip = ngx.req.get_headers()['X-Real-IP']
if client_ip == nil then
   client_ip = ngx.req.get_headers()['X-Forwarded-For']
end
if client_ip == nil then
   client_ip = ngx.var.remote_addr
end

local req_origin = ngx.req.get_headers()['origin']
if req_origin == nil then
   red:zadd('HTTP_REQUEST_UNAUTHORIZED', ngx.time(), client_ip)
   log(ERR, 'not http request origin is nil IP::'..client_ip)
   EXIT(ngx.HTTP_UNAUTHORIZED)
end

if tostring(req_origin) ~= "https://www.tinywan.com" then
      red:zadd('HTTP_REQUEST_FORBIDDEN', ngx.time(), req_origin)
      log(ERR, "error : request origin is error req_origin = "..req_origin)
      EXIT(ngx.HTTP_FORBIDDEN)
end

Note: through the above encrypted authorization, playback is only possible under the tinywan.com domain; other domains, including VLC, will be unable to play.