Prompt Engineering Essentials: Best Practices & Attack Defense for LLMs

1. What Is a Prompt?

In short, a prompt is the instruction that tells the model what task to perform. While this sounds simple, some prompts yield good results, others fail, and some even produce harmful output — a degree of "black magic" remains. Prompt engineering aims to write "good" prompts by deriving best practices from rigorous evaluation of model outputs, along with measures to defend against harmful prompts.

1.1 Prompts with Different Roles

In practice, model APIs assign different roles to prompts. For example, DeepSeek's official documentation shows two roles: system and user.

client = OpenAI(api_key="<your API key>", base_url="https://api.deepseek.com")

response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Hello"},
    ],
    max_tokens=1024,
    temperature=0.7,
    stream=False
)

The final input to the transformer is a token sequence, so the different role prompts must be concatenated in a specific format. Using AutoTokenizer we can observe how DeepSeek's tokenizer assembles them:

tokenizer = AutoTokenizer.from_pretrained("./deepseek-tokenizer")
messages = [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Hello"},
]
formatted_text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=False
)

print("Concatenated result:")
print(formatted_text)

The output is:

<｜begin▁of▁sentence｜>You are a helpful assistant<｜User｜>Hello

Inspecting the model's tokenizer_config.json reveals a chat_template field (usually in Jinja2 format) that defines the concatenation template. Notably, the system prompt appears first in the final prompt, and models are better at handling instructions that appear earlier . Additionally, during post-training models are adjusted to pay more attention to the system prompt.

2. Best Practices

Major model providers such as Anthropic and OpenAI publish detailed prompt engineering guides, and Chip Huyen's book AI Engineering provides an excellent summary. The key practices are:

Write clear, explicit instructions

Explain the task unambiguously.

Ask the model to adopt a persona (place in the system prompt).

Provide examples (few-shot).

Specify the output format.

Provide sufficient context

Retrieval-Augmented Generation (RAG) data retrieval.

Web search.

Decompose complex tasks into simpler subtasks

Give the model time to think

Chain-of-Thought (CoT) prompting.

Self-critique prompts.

Many high-quality examples are available online, such as the "Baoyu's Sharing" series, which includes a rigorous yet entertaining "Lei Jun Speech Generation Prompt".

3. Prompt Attacks

Where there is value, attackers follow. Prompt attacks fall into three categories:

Prompt extraction — extracting the application's prompt to replicate or exploit the application.

Jailbreak and prompt injection — tricking the model into performing malicious actions.

Information extraction — coaxing the model to leak training data or context information.

Defenses can be applied at multiple layers:

Model-level defense — model vendors guide the model during post-training to better follow system prompts, blocking many attacks.

Prompt-level defense — write more attack-resistant prompts, explicitly stating what the model must not do.

System-level defense — adopt isolation mechanisms in system design to protect user safety.

References

[1] GPT-3 paper: https://arxiv.org/abs/2005.14165

[2] Anthropic prompt engineering guide: https://docs.claude.com/en/docs/build-with-claude/prompt-engineering/claude-4-best-practices

[3] OpenAI best practices: https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-the-openai-api

[4] Chip Huyen, AI Engineering : https://www.oreilly.com/library/view/ai-engineering/9781098166298/

[5] Baoyu's Sharing: https://baoyu.io/