FedMRL: Federated Meta Reinforcement Learning for Cold-Start Slice Resource Management

Background

In 6G networks, the high dynamism and uncertainty of services degrade QoS and raise management costs. Applying deep reinforcement learning (DRL) to this problem suffers from a cold‑start issue: a new service provider (SP) must train a model from scratch, which requires extensive data collection and long convergence time. Moreover, conventional DRL centralizes training data on a server, compromising SP data privacy.

Solution Overview

The proposed FedMRL approach introduces two learning loops.

Outer loop: Gradient information from existing tasks updates a meta‑policy on the central server, providing an initial model for subsequent tasks.

Inner loop: Each SP downloads the meta‑policy, performs a few local training steps to quickly converge to a task‑specific policy, and then uploads the resulting gradients to refine the meta‑policy.

FedMRL thus enables rapid learning for new slices while protecting SP data privacy.

FedMRL Method Details

FedMRL employs reinforcement learning, modeling the problem as a Markov Decision Process (MDP).

State space: Three elements represent (1) computational resources of physical nodes occupied by flows, (2) bandwidth of physical links used by flows, and (3) request rate of flow k in time slot τ.

Action space: VNF placement and flow scheduling are unified; an action selects a path for flow k from its candidate paths.

Reward function: Defined to minimize total cost.

The meta‑learning component uses the Model‑Agnostic Meta‑Learning (MAML) framework. MAML’s meta‑training consists of an inner loop that updates task‑specific policies using the maintained meta‑policy parameters, and an outer loop that aggregates task gradients to update the meta‑policy.

The article explains the rationale for centralizing gradient updates: by aggregating gradients from all tasks, the parameter server can discover a set of parameters that allow all tasks to converge quickly, achieving a globally beneficial initialization rather than a locally optimal one.

FedMRL Framework

Outer loop diagram: The parameter server collects gradients from all tasks and performs gradient descent to update the meta‑policy parameters θ, aiming for fast convergence and improved performance across tasks.

Inner loop diagram: For a new slice, the SP downloads the initial parameters θ, trains locally on its own data to obtain task‑specific parameters θ′, computes the loss and gradient g, and uploads g to the server for updating θ.

Experimental Results

The study compares FedMRL with three baselines—TUNE, TDSC, and IOSP—across four algorithms evaluating three cost metrics and end‑to‑end latency. Results (shown in the accompanying figures) demonstrate that regardless of network condition changes—such as path set variations, transmission rate fluctuations, or different topologies—FedMRL consistently achieves the best performance, effectively solving the cold‑start problem.

Author Information

Author: Duan‑Jie Duan, graduate student (2023) at NIRC, research focus on network slicing.