Key Considerations for Deploying Large Language Models in Cloud Services

After the 2024 NJSD Generative AI Application Development session, the author shares personal reflections on how large language models (LLMs) are being applied in Alibaba Cloud services, highlighting four business dimensions—service experience, efficiency, capability, and insight—and three primary product forms.

1. Business Scenarios and Product Forms

Intelligent Customer‑Facing Chatbot : Uses the Tongyi large model combined with domain knowledge to enable self‑service and improve customer experience.

Copilot + Agent for Service Staff : Provides end‑to‑end assisted workflows that integrate deeply with service processes, boosting staff efficiency.

AI Insights for Management : Applies global service‑experience analytics to uncover product and service improvement opportunities, raising overall service quality.

2. Three Core Questions for Foundation Models

The author proposes three essential questions when bringing generic foundation models to real‑world scenarios:

Do they possess sufficient domain capability?

Are the models themselves enough?

Must organizations change in the LLM era?

Question 1: Domain Capability

Using Claude (Sonnet 3.5) as an example, the author notes that current foundation models lack the depth required for highly specialized tasks. To raise domain capability, two dimensions are suggested: internal model optimisation ("inner skill") and external domain augmentation ("outer skill").

Prompt Engineering is identified as the highest‑ROI approach. Although many prompt‑engineering guides exist, effective prompts must capture deep business logic, role definition, task scope, interaction style, safety, output format, and example responses. The article shows an OpenAI prompt‑writing guideline side‑by‑side with a Claude‑generated chatbot prompt.

Retrieval‑Augmented Generation (RAG) offers the second‑highest ROI but introduces many practical pitfalls. The author references a "12 RAG Pain Points and Proposed Solutions" diagram, emphasizing that successful RAG requires careful handling of data quality, retrieval latency, and relevance.

Question 2: Model Sufficiency

The answer is a clear "no"—foundation models alone are insufficient. A full service‑technology stack is needed, including domain data layers, specialized small models, and robust LLMOps engineering. The author stresses that an LLM is not a finished product.

Question 3: Organizational Change

In the LLM era, traditional role boundaries blur. Data engineers, algorithm researchers, and platform engineers all see expanded responsibilities, such as using prompts to build end‑to‑end pipelines. The required team shape depends heavily on company culture and existing structures.

3. Practical Recommendations

The author presents a prioritized roadmap (high ROI to low ROI) for enhancing domain capability:

Establish a domain data advantage.

Build domain understanding ability.

Improve business‑process efficiency.

Expand into additional deployment scenarios.

Choosing the right scenario, form, and pace reflects both technical judgment and business acumen; not every problem should be "AI‑ified".

4. Post‑Event Takeaways

Discussions with peers revealed mixed adoption—some teams are already experimenting, others remain hesitant due to uncertainty. The author calls for "organizational sharpness" to explore these unknowns. While LLM technology continues to evolve rapidly, the author remains optimistic, quoting a recent cloud conference remark: "New technological revolutions grow amid doubt, and many miss out because they hesitate."

(End)