Same AI, Different Roles: Why Usability Depends on Context, Not Accuracy

Many teams have experienced the same large language model being praised as "finally usable" in one role but dismissed as "still not safe to use" in another. This gap is often blamed on poor prompts or incomplete knowledge bases, but the root cause lies deeper: the same output carries different consequences depending on who acts on it.

Consider a common scenario: the system prioritizes a batch of materials with a "suggested priority" ranking. For an information organizer, this may only save reading time; for a frontline operator, it changes the day's work order; for a reviewer, it may affect resource allocation and form conclusions that require explanation. The output looks identical, but the users stand in completely different positions.

This is why many AI projects experience a usability gap after launch: teams often only ask "how accurate is the model?" but fail to ask "who will act on this, can they see the basis, and who catches errors?"

NIST's AI Risk Management Framework places use environment, operator proficiency, human-AI roles, and oversight mechanisms on the same map; this is not to complicate governance but to remind us that AI usability is inherently contextual. The AI RMF Core explicitly includes human-AI configuration, user types, human oversight, and system environment in the continuous risk management process.

Model Unchanged, What Changes Is "What Happens Next"

Evaluating AI with a single overall score easily yields a false average: it seems "acceptable" for everyone, but not stable enough for any critical task.

A more realistic assessment breaks usability into three questions:

Does the judgment match the task? Does this role need clues, suggestions, comparisons, or a conclusion that can enter a formal process?

Does the evidence match the responsibility? Can the user see sources, scope, confidence boundaries, and uncovered gaps?

Does the handoff match the consequences? When uncertainty, conflict, or exceptions arise, where does the task stop and who takes over?

These three seemingly simple questions explain the vast temperature difference of the same model across roles.

Role Position: Information Organization & Auxiliary Analysis

AI Output Role: Clues, summaries, drafts.

Capabilities Needed: Traceable sources, quick revision allowed.

Role Position: Business Operations & Collaborative Disposal

AI Output Role: Suggestions, prioritization, to-dos.

Capabilities Needed: Exceptions visible, impact scope assessable.

Role Position: Review, Decision & Responsibility Bearing

AI Output Role: Reference opinions, not replacement conclusions.

Capabilities Needed: Evidence verifiable, human confirmation recorded.

The most overlooked column in the table is the last one. Many systems invest heavily in "making output look like an answer" but fail to equip each role with its needed evidence, exceptions, and confirmation mechanisms. Users then rely on experience to add manual judgment; AI appears to have entered the process but actually remains outside it.

Why the Same Answer Saves Effort for Some but Adds Work for Others

A counterintuitive phenomenon: the closer a role is to the responsibility endpoint, the more AI sometimes increases workload.

If the system only gives a fluent conclusion without explaining which materials were used, which conditions were unmet, and which steps still need human judgment, the recipient must re-verify after the fact. The model saves generation time, but the role bears the explanation cost.

At this point, "human-AI collaboration" easily becomes another division of labor: the machine gives an answer first, then the human searches for why it holds. It appears to retain human oversight but actually pushes the hardest verification to the end.

True smooth collaboration is the opposite: let people see necessary evidence before acting, receive clear handoff packages at boundaries, rather than retroactively tracing the entire chain after consequences appear.

A More Useful Perspective: Not Assigning Functions to Roles, but "Judgment Interfaces"

Many product designs divide by function: Q&A, retrieval, summarization, form filling, reminders. This is necessary but insufficient to explain why different roles need different AI forms.

A more useful unit is the "judgment interface" — what the user can see, confirm, and reject before taking the next action.

The same underlying capability can present at least three different judgment interfaces:

For assistant roles, focus on speed : sources, highlights, missing info must be easy to read and modify quickly.

For operational roles, focus on stability : priority, impact scope, anomaly signals must support continuous collaboration.

For responsibility roles, focus on clarity : applicable scope, key evidence, human confirmation points must withstand post-hoc review.

This does not mean each role needs a separate model. Often the model, knowledge, and tools can be shared; what should be separated are output presentation, human checkpoints, and recording methods. Merging them into a single universal page often yields a "universal assistant" that everyone finds cumbersome.

Observe Whether AI Truly Enters Work: Look at Three "Handovers"

Compared to tracking call volume or satisfaction, how a task passes between AI and humans often better indicates whether the system has integrated into real work.

From materials to suggestions. Does the system explain which information supports the suggestion, and which information is missing or conflicting?

From suggestions to actions. Can the user understand what actions the suggestion will trigger, who is affected, and whether there are irreversible consequences?

From actions to feedback. Can results return to the relevant role as calibration for future judgments, rather than disappearing after a click?

These three handovers do not require every step to slow down. Their goal is to place uncertainty where it can still be handled, not let it explode at the process end.

Generative AI risk management literature also notes that different application scenarios may require different levels of human oversight and different human-AI configurations; the diversity of generative outputs precisely requires organizations to re-examine review, recording, and management arrangements. NIST AI 600-1 provides cross-industry risk profiles for this.

Conclusion

That the same AI shows completely different usability across roles is not surprising. It shows AI is not a process-detached "capability component" but a new relationship embedded among tasks, evidence, and responsibility.

When teams shift attention from "how strong is the model?" to "what judgment interface does this role need?", previously contradictory feedback becomes understandable: some need a faster starting point, some need a more stable process, some need clearer responsibility boundaries.

A truly mature AI application does not show the same answer to all roles, but lets each person at their responsibility position get just enough information to make the next judgment.

Sources & References

NIST AI Risk Management Framework 1.0: Framework basis for human-AI collaboration, role positions, use context, and oversight mechanisms.

NIST AI RMF Core: Four continuous risk management functions — GOVERN, MAP, MEASURE, MANAGE — and related practices.

NIST AI 600-1: Generative AI Risk Management Profiles: Discussion of generative AI risks under different human-AI configurations and oversight levels.

Interim Measures for the Management of Generative AI Services: Public rule background for generative AI services provided to the domestic public; this article does not draw compliance conclusions for specific internal applications based on this.