LangChain Video Analysis: Fixed Workflows First, Bounded Agents Only When Needed

Distinguishing Workflow from Agent

A workflow's path is predefined in code; an agent decides which tool to use next within allowed boundaries. Automatic execution, branching, or loops do not inherently imply an autonomous agent. LangChain and LangGraph differences do not map to an "autonomous vs. non-autonomous" dichotomy. Frameworks help organize execution but do not decide judgment rules or turn unstable inputs into reliable evidence.

01: Define Five Verifiable Nodes Before Choosing Tools

Each node owns a single verifiable artifact. Nodes pass structured state, not long conversation histories. The frame extraction and API call functions from earlier lessons can be wrapped as node capabilities without rewriting everything.

Input Validation Node: video reference → validated task input

Confirm authorization, file existence, media format, and duration. External callers submit a video_id; the server resolves it to an authorized media location. Never trust arbitrary paths or download URLs.

Frame Sampling Node: input + sampling strategy → frame manifest

Output frame_id, image reference, timestamp, window index, and sampling strategy version. On failure, return an explicit error — do not mask it with an empty list.

Judgment Node: image batch + prompt → candidate observations

Send actual image content or model-accessible authorized image URLs, not internal server filenames. Retain model name, prompt, latency, and raw response reference.

Validation Node: candidate observations + input manifest → valid per-frame results

Check structure, enums, frame numbers, and evidence requirements. Invalid responses go to failure or controlled retry; they must not flow into summarization.

Summarization Node: valid results + missing ranges → final report

Apply the rules from lesson two to produce person conclusions. Explicitly label execution status and coverage range. This step does not need another LLM call.

Shared State Contents

Store task_id, video_ref, sampling_version, frame_manifest_ref, batch_results, missing_batches, attempts, deadline, and budget counters. Prefer references over copying entire videos, Base64 images, or full message histories. Merge parallel batch results by batch_id or frame_id with a defined overwrite rule; avoid two nodes overwriting the same array concurrently. Start with serial execution; add limited concurrency only after each node's I/O can be independently inspected.

02: Express the Fixed Workflow in a Small Structured Code Block

The following shows only LangGraph wiring. State and node functions implement the contracts above; model configuration, persistence, and exception branches are omitted. It is a skeleton, not a runnable application.

from langgraph.graph import StateGraph, START, END

builder = StateGraph(State)
nodes = {
    "check": check_input, "sample": sample_frames,
    "analyze": analyze_batches, "validate": validate_results,
    "summarize": summarize_results,
}
for name, handler in nodes.items():
    builder.add_node(name, handler)
builder.add_edge(START, "check")
for a, b in zip(list(nodes), list(nodes)[1:]):
    builder.add_edge(a, b)
builder.add_edge("summarize", END)
workflow = builder.compile()
result = workflow.invoke(initial_state)

Where LangChain Fits

In the analyze node, use the unified chat model interface to organize prompts, multi-image messages, and response parsing. Wrap controlled frame sampling, validation, etc., as clear tool interfaces. The model adapter layer preserves platform parameter differences; do not assume all models support identical multi-image, structured output, or tool-calling capabilities just because the call form is unified. You can keep the lesson-one HTTP access functions, finish model integration testing, then gradually replace the adapter layer. LangChain's value is unified capability organization and future extensibility, not rewriting working functions.

"Resample on Uncertainty" Can Be Pure Code

Example rule: if uncertain appears and budget remains, resample neighboring frames around the corresponding frame once; if still uncertain, escalate for review. Such fixed conditional edges do not require model planning. Only when "where to look next" and "which analysis tool to call" vary by scenario and the extra choices demonstrably improve outcomes should an agent be introduced. Configuring persistence, retries, and tracing for the workflow is subsequent engineering work; the compile() call above does not enable those capabilities automatically.

03: When Dynamic Selection Is Needed, Give the Agent a Small Fence

Suppose frame F004 contains a blurry human silhouette. The agent may propose examining the small windows before and after it, rather than re-analyzing the entire video. It has choice rights but no arbitrary tool execution rights; every action passes through an executor check.

Tools can be limited to two:

sample_extra_frames : resample within an authorized video's specified window.

inspect_frames : run visual judgment on already registered frames. Tools return new frame numbers and observation records; if downstream models need to see images, the images must be passed as multimodal content, not just local paths.

from langchain.agents import create_agent

# model and two controlled tools are pre-configured by the application; only composition shown.
agent = create_agent(model=model,
    tools=[sample_extra_frames, inspect_frames],
    system_prompt="Only supplement evidence needed for judgment; allow ending and review when uncertain.")

The executor constrains video scope, tool whitelist, resampling time windows, cumulative frame count, model call count, and total time limit. Budget is reserved before calls and settled after returns; failures and retries also consume budget. Tool loop limits cannot replace time, permission, or cost limits. The tool list excludes arbitrary shell, arbitrary file reads, direct notifications, or device control. Final business actions remain in separate rule and authorization chains.

04: The Other Half of Automation Is Handling Failure

A 30-frame task split into three batches: first two succeed, third times out. Correct handling is not discarding the first two batches and re-sending the whole video, but saving valid results, locating incomplete batches, and deciding retry or termination within budget.

Graph state can be given to a persistent checkpointer; images and raw responses stay in controlled storage while state holds only references. An in-memory checkpointer cannot provide durable recovery after process restart and should not be treated as a production recovery solution.

Network timeout does not guarantee the platform hasn't processed the request. Check call records before retry; if completion cannot be confirmed, accept the boundary of possible duplicate calls and billing. Checkpoints do not automatically grant external interfaces "exactly-once" execution semantics.

Classify failures: transient rate limits or network errors may retry within budget; invalid model ID, missing permissions, format incompatibility should stop and fix configuration; insufficient evidence is an observation result, not a system fault that raw retry will necessarily fix.

Example budget: at most one resampling round, at most three model requests, and a task deadline. Numbers must be determined by actual latency, frame scale, and cost testing — they are not universal recommended thresholds.

05: Accept a Process, Not a Single Demo

Traceability from Submission to Report

Input task has a unique identifier; each stage has start, end, and error logs; every observation has image and time index; report states planned_frames, validated_frames, and missing batches; model, prompt, sampling, and aggregation strategies all have versions. A natural language summary cannot replace these records.

At Least Five Acceptance Paths

Happy path: Authorized video enters, frames sampled per strategy, batch validation succeeds, fixed rules summarize, every evidence image is traceable.

Partial failure: One batch unavailable, valid results retained, report explicitly marks partial and missing ranges; when no valid person evidence found, do not write incomplete task as not_observed.

Model output anomaly: JSON truncation, enum error, duplicate numbering — validation node blocks downstream, no silent field deletion or bracket patching to fake success.

Agent out-of-bounds: Requests another video, oversized time window, or over-budget calls — executor refuses; task has readable termination reason, no infinite loop.

Interruption recovery: After restart, same task, version, and evidence are found; confirmed artifacts reused; no duplicate business notifications from recovery.

When to Keep Using Fixed Workflows

If judgment goals, input scope, and resampling rules are already clear, fixed workflows are usually easier to explain and accept. If tasks are open-ended and evidence paths truly vary, add a constrained agent node within the stable main pipeline. Whether production can use an agent depends on its necessity, verifiability, and execution boundaries — not the framework name.

References and Scope

LangChain Overview and Agent Composition Interfaces —

https://python.langchain.com/docs/concepts/#langchain-overview

LangGraph: Workflows and Agents —

https://langchain-ai.github.io/langgraph/concepts/why-langgraph/

Graph API and State Merging — https://langchain-ai.github.io/langgraph/concepts/graph_api/ Persistence and Checkpointer —

https://langchain-ai.github.io/langgraph/concepts/persistence/

Reference verification date: 2026-09-03. This article is a solution design; sample inputs, parameters, and results illustrate rules only — no real model effectiveness testing or production integration was performed.