Designing an Enterprise-Scale Short URL Service for Hundreds of Millions of Requests

Background

Long URLs in SMS and template messages increase cost and reduce readability. A short‑URL service maps a short code to the original long URL and redirects when the short URL is accessed.

Design Overview

The service maintains a one‑to‑one mapping between short and long URLs. Two HTTP endpoints are exposed: one for generating a short URL and one for redirecting to the long URL. High‑concurrency reads are handled with load balancing and a distributed cache.

Application Example

SMS from Sephora contains the short link http://ew7.cn/?M2Fj which redirects to the long e‑commerce page https://m.sephora.cn/v2/html/rewardsBoutique/.

Compression Code Generation

Short codes use Base64. Six‑character codes provide 64⁶ ≈ 68 billion possibilities, sufficient for the expected traffic. Seven‑character codes provide 64⁷ ≈ 4 trillion possibilities.

Hash‑based generation : Compute MD5 or SHA‑256 of the long URL, Base64‑encode the hash, then truncate to 6 characters. Collisions are possible because the truncated Base64 segment may repeat; therefore the service must check for existing codes and regenerate if a conflict is found.

Auto‑increment ID : Use a database auto‑increment primary key or a distributed ID generator, then Base64‑encode the numeric ID. This guarantees uniqueness but produces predictable codes, making them vulnerable to enumeration attacks.

Pre‑generated codes : Randomly generate a pool of 6‑character Base64 strings, filter duplicates with a Bloom filter, and store the pool in Redis. Because the pool is prepared ahead of time, runtime generation avoids collision checks and improves performance.

Short‑URL Generation Flow

When a request to shorten a URL arrives, the service:

Validates the URL format.

Checks Redis for an existing short code for the same long URL (keyed by the URL's MD5). If found, returns the existing short URL.

Otherwise obtains a unique code from the pre‑generated pool (Redis set short_url_unique_code).

Persists the mapping (id, unique_code, short_url, long_url, long_url_md5) in the url_link table.

Writes two hash entries in Redis: SHORT_LONG_MAP[unique_code] → long_url and LONG_MD5_CODE_MAP[long_url_md5] → unique_code.

Implementation Details

Database Schema

-- Table structure for unique_code
DROP TABLE IF EXISTS `unique_code`;
CREATE TABLE `unique_code` (
  `id` bigint NOT NULL,
  `code` varchar(16) NOT NULL COMMENT '压缩码',
  `status` tinyint(4) NOT NULL DEFAULT '0' COMMENT '状态 :0:未使用 1:已使用 -1:失效',
  `type` tinyint(4) NOT NULL DEFAULT '0' COMMENT '生成方式：0：随机数 1：分布式id 2：hash',
  `deleted` tinyint(4) NOT NULL DEFAULT '0',
  `creator` bigint DEFAULT NULL,
  `updater` bigint DEFAULT NULL,
  `create_time` datetime DEFAULT NULL,
  `update_time` datetime DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='压缩码表';

-- Table structure for url_link
DROP TABLE IF EXISTS `url_link`;
CREATE TABLE `url_link` (
  `id` bigint NOT NULL,
  `unique_code` varchar(255) NOT NULL COMMENT '唯一压缩码',
  `short_url` varchar(255) NOT NULL COMMENT '短链接地址',
  `long_url` varchar(1000) NOT NULL COMMENT '长链接地址',
  `long_url_md5` varchar(255) NOT NULL COMMENT '长链接地址md5',
  `deleted` tinyint(4) NOT NULL DEFAULT '0',
  `create_time` datetime DEFAULT NULL,
  `update_time` datetime DEFAULT NULL,
  `creator` bigint DEFAULT NULL,
  `updater` bigint DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='链接映射表';

-- Table structure for visit_record
DROP TABLE IF EXISTS `visit_record`;
CREATE TABLE `visit_record` (
  `id` bigint NOT NULL COMMENT '主键',
  `url_link_id` bigint NOT NULL COMMENT 'url映射id',
  `unique_code` varchar(16) NOT NULL COMMENT '压缩码',
  `client_id` varchar(128) NOT NULL COMMENT '唯一身份标识,SHA-1(客户端IP-UA)',
  `client_ip` varchar(64) NOT NULL COMMENT '客户端IP',
  `visit_time` datetime NOT NULL COMMENT '访问时间',
  `user_agent` varchar(2048) DEFAULT NULL COMMENT 'UA',
  `country` varchar(32) DEFAULT NULL COMMENT '国家',
  `province` varchar(32) DEFAULT NULL COMMENT '省份',
  `city` varchar(32) DEFAULT NULL COMMENT '城市',
  `isp` varchar(32) DEFAULT NULL COMMENT '网络服务运营商',
  `browser_type` varchar(64) DEFAULT NULL COMMENT '浏览器类型',
  `browser_version` varchar(128) DEFAULT NULL COMMENT '浏览器版本号',
  `os_type` varchar(32) DEFAULT NULL COMMENT '操作系统型号',
  `device_type` varchar(32) DEFAULT NULL COMMENT '设备型号',
  `os_version` varchar(32) DEFAULT NULL COMMENT '操作系统版本号',
  `deleted` tinyint(4) DEFAULT '0' COMMENT '软删除标识',
  `creator` bigint DEFAULT '0' COMMENT '创建者',
  `updater` bigint DEFAULT '0' COMMENT '更新者',
  `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
  `update_time` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='访问记录';

Pre‑generation Service

@Service
@Slf4j
public class UniqueCodeServiceImpl extends ServiceImpl<UniqueCodeDAO, UniqueCode> implements UniqueCodeService {
    @Resource
    private UniqueCodeDAO uniqueCodeDAO;
    @Resource
    private StringRedisTemplate stringRedisTemplate;
    @Resource
    private IdGenerator idGenerator;
    @Resource
    private ExecutorService executorService;
    @Resource
    private ShortUrlBloomFilter shortUrlBloomFilter;

    @Value("${unique-code.max-size}")
    private Integer maxSize;
    @Value("${unique-code.min-size}")
    private Integer minSize;

    private static final String UNIQUE_CODE_KEY = "short_url_unique_code";

    @Override
    public String getUniqueCode() {
        String code = stringRedisTemplate.opsForSet().pop(UNIQUE_CODE_KEY);
        asyncGenerateUniqueCode();
        return code;
    }

    @Override
    public List<String> getUniqueCode(Integer size) {
        List<String> codes = stringRedisTemplate.opsForSet().pop(UNIQUE_CODE_KEY, size);
        asyncGenerateUniqueCode();
        return codes;
    }

    @Transactional(rollbackFor = Exception.class)
    public void generateUniqueCode() {
        log.info("==========开始生成压缩码===========" );
        long startTime = System.currentTimeMillis();
        Set<String> codes = new HashSet<>();
        Set<String> existCodes = new HashSet<>();
        for (int i = 0; i < maxSize; i++) {
            String code = RandomUtils.generateCode(6);
            Boolean exist = shortUrlBloomFilter.isExist(code);
            if (exist) {
                existCodes.add(code);
            } else {
                codes.add(code);
            }
        }
        if (!CollectionUtils.isEmpty(existCodes)) {
            log.info("=========以下压缩码已存在：{}", existCodes);
        }
        stringRedisTemplate.opsForSet().add(UNIQUE_CODE_KEY, codes.toArray(new String[0]));
        List<UniqueCode> uniqueCodeList = new ArrayList<>();
        codes.forEach(code -> {
            UniqueCode uniqueCode = new UniqueCode();
            uniqueCode.setId(idGenerator.nextId());
            uniqueCode.setCode(code);
            uniqueCodeList.add(uniqueCode);
        });
        saveBatch(uniqueCodeList);
        long costTime = System.currentTimeMillis() - startTime;
        log.info("===============结束生成压缩码, costTime:[{}],Count:[{}]==============", costTime, codes.size());
    }

    public void asyncGenerateUniqueCode() {
        Long size = stringRedisTemplate.opsForSet().size(UNIQUE_CODE_KEY);
        if (size < minSize) {
            executorService.execute(this::generateUniqueCode);
        }
    }
}

Short‑URL Generation

@Override
public String generateShortUrl(String longUrl) {
    // 1. Validate URL
    if (!isValidUrl(longUrl)) {
        throw new BizException("无效的url");
    }
    // 2. Return existing short URL if present
    Object value = redisTemplate.opsForHash().get(LONG_MD5_CODE_MAP, longUrl);
    if (Objects.nonNull(value)) {
        return domain + value.toString();
    }
    // 3. Create new short URL
    long id = idGenerator.nextId();
    String uniqueCode = uniqueCodeService.getUniqueCode();
    String longUrlMd5 = DigestUtils.md5DigestAsHex(longUrl.getBytes());
    String shortUrl = domain + uniqueCode;
    UrlLink urlLink = new UrlLink();
    urlLink.setId(id);
    urlLink.setUniqueCode(uniqueCode);
    urlLink.setShortUrl(shortUrl);
    urlLink.setLongUrl(longUrl);
    urlLink.setLongUrlMd5(longUrlMd5);
    urlLinkDAO.insert(urlLink);
    // short → long
    redisTemplate.opsForHash().put(SHORT_LONG_MAP, uniqueCode, longUrl);
    // long md5 → short code
    redisTemplate.opsForHash().put(LONG_MD5_CODE_MAP, longUrlMd5, uniqueCode);
    return shortUrl;
}

Redirect Logic

@Override
public void redirect(HttpServletRequest request, HttpServletResponse response, String uniqueCode) throws IOException {
    String longUrl = shortUrlService.getOriginUrl(uniqueCode);
    if (StringUtils.isBlank(longUrl)) {
        throw new BizException("短链接地址不存在");
    }
    executorService.execute(() -> visitRecordService.addVisitRecord(request, uniqueCode));
    response.sendRedirect(longUrl);
}

Visit Record Capture

@Override
@Transactional(rollbackFor = Exception.class)
public void addVisitRecord(HttpServletRequest request, String uniqueCode) {
    VisitRecord visitRecord = new VisitRecord();
    long id = idGenerator.nextId();
    visitRecord.setId(id);
    UrlLink urlLink = shortUrlService.getUrlLink(uniqueCode);
    visitRecord.setUrlLinkId(urlLink.getId());
    visitRecord.setUniqueCode(urlLink.getUniqueCode());
    visitRecord.setVisitTime(new Date());
    String agent = request.getHeader(USER_AGENT);
    String clientIp = IpUtils.getRemoteHost(request);
    IpRegion ipRegion = IpUtils.getIpRegion(clientIp);
    visitRecord.setUserAgent(agent);
    visitRecord.setClientIp(clientIp);
    String clientId = DigestUtil.sha1Hex(clientIp + "&" + agent);
    visitRecord.setClientId(clientId);
    visitRecord.setCountry(ipRegion.getCountry());
    visitRecord.setProvince(ipRegion.getProvince());
    visitRecord.setCity(ipRegion.getCity());
    visitRecord.setIsp(ipRegion.getIsp());
    if (StringUtils.isNotBlank(agent)) {
        try {
            UserAgent userAgent = UserAgent.parseUserAgentString(agent);
            OperatingSystem os = userAgent.getOperatingSystem();
            Optional.ofNullable(os).ifPresent(x -> {
                visitRecord.setOsType(x.getName());
                visitRecord.setOsVersion(x.getName());
                Optional.ofNullable(x.getDeviceType()).ifPresent(dt -> visitRecord.setDeviceType(dt.getName()));
            });
            Browser browser = userAgent.getBrowser();
            Optional.ofNullable(browser).ifPresent(x -> visitRecord.setBrowserType(x.getGroup().getName()));
            Version browserVersion = userAgent.getBrowserVersion();
            Optional.ofNullable(browserVersion).ifPresent(x -> visitRecord.setBrowserVersion(x.getVersion()));
        } catch (Exception e) {
            log.error("解析UserAgent异常,事件内容:", e);
        }
    }
    visitRecordDAO.insert(visitRecord);
}

Sample Data

Short‑URL generation record:

id          unique_code  short_url                                 long_url                                            long_url_md5                         deleted create_time          update_time          creator updater
3362951037714432 j51YO8   http://127.0.0.1:18800/j51YO8   https://gitee.com/plasticene3/plasticene-boot-starter-parent   c411c28da0302d3f0c9e34872c3b66d1   0      2022-08-18 17:00:21 2022-08-18 17:00:21 1       1

Visit‑record example:

id            url_link_id      unique_code  client_id                                 client_ip   visit_time           user_agent                                                                 country province city  isp   browser_type browser_version os_type device_type os_version deleted creator updater create_time          update_time
2632861324673024 2620608022052864 ava5R7      7fb0070a4cb212b7dc0efce2cd08b017477787ae 10.8.4.7   2022-08-16 16:39:14 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36 中国   浙江   杭州 电信 Chrome      104.0.0.0        Mac OS X Computer    Mac OS X      0       1       1       2022-08-16 16:39:14 2022-08-24 18:15:18

Repository

Project source code: https://github.com/plasticene/plasticene-infra/tree/main/short-url-service

Conclusion

Pre‑generating a pool of collision‑free short codes and storing them in Redis eliminates runtime validation of code uniqueness, reduces latency for the generate‑short‑URL endpoint, and prevents predictable code patterns while supporting high QPS for both generation and redirection.