Advances and Challenges in Controllable Text Generation with Pretrained Language Models

In recent years, transformer‑based pretrained language models have shown strong generalization abilities in long‑text and controllable text generation, attracting increasing attention from both academia and industry.

Background of controllable text generation

Controllable text generation extends ordinary text generation by adding constraints (e.g., keywords, knowledge graphs) to guide the output, which often requires the model to produce much more content from relatively little input, demanding strong imagination and divergence capabilities.

Early text generation relied on template‑based methods; the rise of pretrained models such as GPT, BART, and T5—built on the Transformer architecture and trained with self‑supervised objectives—has dramatically improved generation quality and enabled multi‑task learning.

Current challenges

Despite progress, models still suffer from logical errors, incoherence, and repetitive outputs, especially in long‑text scenarios where memory and computational costs become prohibitive.

Research progress

Typical methods include:

Adjusting decoding strategies (e.g., adding restriction words to increase target token probabilities).

Modifying training objectives (e.g., sentiment‑aware objectives).

Prompt‑learning approaches that inject controllable factors at inference time.

GPT‑based controllable generation is popular, but fine‑grained control (e.g., generating alternating positive/negative sentences) remains difficult with a single prompt.

Recent work proposes encoding prompts separately and using a non‑residual attention decoder that jointly attends to prompt and text, enabling token‑level replacement of prompts for finer control.

Memory‑augmented models store historical generation information in a matrix‑shaped memory unit, compressing past tokens and allowing long‑document generation without revisiting every token.

Continual learning addresses catastrophic forgetting when adapting models to new tasks; typical strategies involve parameter sharing, task‑specific adapters, or selective insertion of historic parameters, followed by a lightweight fine‑tuning stage.

Contrastive learning at the token level introduces a margin‑based loss to push different tokens apart, improving diversity and coherence during decoding.

Fact‑correctness remains an issue; two main remedies are enhancing knowledge acquisition during pretraining (e.g., converting structured knowledge graphs into text) and injecting external knowledge at generation time via retrieval.

Application practice at LanZhou Technology

LanZhou built a lightweight “Mengzi” pretrained model and fine‑tuned it for various vertical domains, creating generation engines for literary assistance, smart copywriting, and template‑based content creation.

Key steps include large‑scale data construction, domain‑specific fine‑tuning, automatic evaluation, and post‑processing.

For literary assistance, the system offers entity rendering and custom templates with selectable writing styles; for smart copywriting, users select a template, input keywords, and generate copy in one click.

Summary and outlook

Pretrained language‑model based generation is rapidly moving from research to real‑world deployment across marketing, literature, and report writing. Future work should focus on long‑text generation, coherence, factuality, automatic evaluation, lightweight models, rapid domain adaptation, and few‑shot learning.

Q&A highlighted data acquisition for long texts, ensuring logical consistency via keywords and knowledge graphs, and concluded with a thank‑you note.