06 Prompt Strategy Diagnosis: Influence, Failure Modes, and What Lives Outside the Prompt

Expected classification:

• Examples axis: zero-shot — no example review is demonstrated.
• Additive techniques: role priming + rubric-based + structured-output.

Why:

• Examples axis = zero-shot because no demonstration examples of good reviews are provided.
• It is role-primed because the model is asked to act as an expert reviewer.
• It is rubric-based because it lists explicit evaluation criteria to judge against.
• It uses structured-output prompting because it specifies named answer sections.
• It is not chain-of-thought because it does not ask for a visible reasoning trace.

Primary lever: the rubric. The main risk in a broad expert review is not output format but coverage: without an explicit list of dimensions the model focuses on surface-level feedback. The rubric is what makes the review systematic and comparable across future lecture drafts.

Discussion point:

This card is a clean warm-up because the two axes separate cleanly: it is plainly zero-shot, and the three additive techniques are each visible in distinct phrases. Notice that “no examples → zero-shot” is now its own question, so students never have to wonder whether “zero-shot” belongs in the same list as “structured-output.”

Classification of each version:

Primary lever: for all three versions the dominant lever is the bare instruction itself — the precise, enumerated behaviour spec (“strip whitespace, lowercase, raise ValueError unless exactly one @”). That is what actually pins down a correct implementation. The worked examples (1b) sharpen the edge cases; the output constraint (1c) tidies the shape; neither would save a vague instruction. Specify the behaviour first; add examples and shape second.

Why this strategy may be appropriate:

The task is small, familiar, and low-risk. The behaviour is explicitly specified, so even bare 1a is probably enough to get a plausible implementation.

Likely failure modes:

• The model may implement a simplistic email check and overinterpret the validation requirement.
• It may include extra explanation despite “Only output.”
• It may not handle non-string inputs unless specified.
• It may pass obvious examples but fail edge cases.

What belongs outside the prompt:

• Unit tests for whitespace, uppercase, missing @, multiple @, empty string.
• Static analysis or type checking if the project uses type hints.
• None of the three versions includes tests; the examples in 1b are demonstrations for the model, not an executable test suite.

Possible improvement:

Write a Python function called `normalize_email(email: str) -> str`.

Requirements:
- Strip leading and trailing whitespace.
- Convert the result to lowercase.
- Raise `ValueError("invalid email")` if the normalized string contains anything other than exactly one `@`.
- Do not perform full RFC email validation.
- Return the normalized email address.

Only output the function body with the function definition. No explanation.

Teaching point:

Zero-shot is not bad. For small, well-specified, low-risk tasks, it is often appropriate. The important question is not “Did we use an advanced prompting technique?” but “Does the task require one?” The triple also shows the two axes in isolation: 1b changes only the examples axis, 1c changes only an additive technique, neither touches the other.

Examples axis: few-shot. Two worked demonstrations (get_order_by_id, get_invoice_by_id) built on sibling artifacts — different functions from the one being written. A sibling-artifact example still counts as a demonstration.

Additive techniques: structured-output + context grounding.

• Structured-output: “Only output the completed function” constrains the output to a code completion.
• Context grounding: the example functions embed the project’s naming, return-type, and SQLAlchemy conventions the model conditions on while writing a different function.
• Not role priming: “You are helping in our Python backend project” is project membership, not an “Act as a <role>” expertise persona.

Double-count disclosure: the same Example 1 / Example 2 block legitimately fires both the few-shot axis and the context-grounding box. As worked demonstrations they set the examples axis; as embedded project conventions the model conditions on, they ground the output. The phrase “our Python backend project” alone is only membership — it is the example code that does the grounding.

Primary lever: the worked examples (few-shot). The challenge is not general Python knowledge but matching local conventions, and the examples carry naming, SQLAlchemy usage, return style, and type hints all at once. Remove them and the model falls back to generic ORM code; remove the “only output” constraint and you lose only a little tidiness.

Why this strategy may be appropriate:

The main challenge is not general Python knowledge but matching local project conventions. Few-shot examples are useful because they show naming, SQLAlchemy usage, return style, and type hints.

Likely failure modes:

• The model may assume imports exist.
• It may hallucinate the User model or table name.
• It may not know whether User.id is the correct field.
• It may reproduce a pattern that is locally consistent but architecturally wrong.

What belongs outside the prompt:

• Type checking.
• Unit or integration test using a test database.
• Code review to confirm the query belongs in this layer.

Possible improvement:

Add the relevant imports or surrounding repository context if the model is expected to produce a full patch.

Assume `Session`, `UUID`, `select`, and `User` are already imported. Do not add imports.

Teaching point:

Few-shot is especially useful when the desired behavior is “do it like these examples,” not when the task requires knowing external facts.

Examples axis: zero-shot. No example review is demonstrated; the rubric and field list are instructions and a schema, and the diff is the artifact under review — none is a worked demonstration.

Additive techniques: role priming + rubric-based + structured-output + context grounding.

• “Act as a senior backend engineer and security-conscious code reviewer” is role priming.
• The numbered list of evaluation categories is a review rubric (criteria to judge against).
• The per-finding fields (severity / file / issue / why / change) are an explicitly named output schema.
• The [DIFF] is embedded background the review conditions on (context grounding).
• Not chain-of-thought: nothing asks the model to think step by step.

Primary lever: the rubric. Code review is a coverage problem: without an explicit list of dimensions the model flags only what catches its eye. The rubric forces systematic coverage; role priming sets perspective and the schema makes findings actionable, but remove the rubric and the review becomes shallow and uneven.

Why this strategy is appropriate:

Code review is a judgment task. A role helps set the perspective, while a rubric improves coverage. Structured output makes the feedback actionable.

Likely failure modes:

• The model may over-report minor issues.
• It may miss subtle security issues.
• It may misunderstand diff context if surrounding code is missing.
• It may suggest changes inconsistent with team conventions.
• It may sound authoritative even when uncertain.

What belongs outside the prompt:

• Static analysis.
• Security scanners.
• Tests.
• Human review for architecture and business logic.

Possible improvement:

Separate findings into:
- Must fix before merge
- Should consider
- Questions for the author

If a finding depends on missing context, mark it as "needs context" instead of stating it as fact.

Teaching point:

Rubric-based review prompts are strong for systematic critique, but the review remains advisory. A model is not a replacement for CI or human approval.

Examples axis: zero-shot. Nothing demonstrates a desired input→output behaviour; the buggy function and the discounted_price() method are artifacts to reason about, not worked example calls.

Additive techniques: context grounding.

• Context grounding: the Item.discounted_price() method from pricing/discounts.py is genuine background — distinct from the total_price function under diagnosis — that the model conditions on to see that None is an expected return. The buggy function is the task’s direct object; the pricing method is the surrounding background.
• Not structured-output: “Explain the likely cause and propose a fix” is two prose imperatives joined by “and” — a task description, not an output-format spec.
• Not chain-of-thought: “Explain” alone does not invoke step-by-step reasoning.

Primary lever: context grounding. The supplied discounted_price() definition is what lets the model see that None is an expected return rather than guessing — it turns a blind diagnosis into an evidence-based one. Remove that background and the model has to assume what discounted_price() does, which is exactly where it collapses uncertainty too early.

Why this may be appropriate:

The error message is unambiguous and the supplied code is small, so with the discounted_price() definition in hand the model can confirm — not just guess — that None is an expected return. The model can infer that discounted_price() sometimes returns None.

Likely failure modes:

• The model may assume too much about discounted_price().
• It may suggest hiding the bug by treating None as zero.
• It may fail to ask whether None is valid domain behavior.
• It may propose a fix without tests.

What belongs outside the prompt:

• Reproduction test.
• Unit test for items with missing discounted prices.
• Domain decision: should None be allowed?

Possible improved prompt:

Diagnose this bug, but separate evidence from assumptions.

For the fix, provide two alternatives:
1. A defensive fix if `None` is expected domain behavior.
2. A stricter fix if `None` indicates invalid data.

Also suggest one regression test.

Error:
[STACK TRACE]

Relevant code:
[CODE]

Teaching point:

Debugging prompts should prevent the model from collapsing uncertainty too early. Ask it to separate evidence, assumptions, alternatives, and validation.

Examples axis: zero-shot. No example test is demonstrated; the function shown is the unit under test, not a demonstration.

Additive techniques: None of these. No role, no rubric, no output-format spec, no reasoning instruction, no tool. The function source is the unit under test, not embedded background, so it does not trigger context grounding.

Evidence of what is missing:

• No role.
• No examples.
• No test framework specified.
• No coverage criteria specified.
• No output format specified.

Primary lever: there is no additive technique, so the only lever is the bare instruction “Write tests for this function” — and it is far too thin to carry the task. The real fix is not to nominate a present technique but to add one: a coverage specification (a rubric-plus-structured-output upgrade). The lever this task needs is missing.

Why this is weak:

The task is underspecified. The model does not know whether to use pytest, unittest, property-based testing, table-driven tests, or project-specific conventions.

Likely failure modes:

• Tests may use the wrong framework.
• Boundary cases may be incomplete.
• Negative ages may be missed.
• The model may only test happy paths.
• The model may duplicate implementation logic rather than test behavior.

What belongs outside the prompt:

• Running the tests.
• Coverage measurement.
• Mutation testing if desired.

Possible improved prompt:

Write `pytest` tests for `is_valid_age`.

Use table-driven tests and cover:
- below minimum: -1, 0, 17
- lower boundary: 18
- typical valid value: 35
- upper boundary: 120
- above maximum: 121

Only output the test code.

Teaching point:

For test generation, the best prompt is often a coverage-specification prompt, not merely “write tests.”

Examples axis: zero-shot. No example acceptance-criteria document is demonstrated; the bulleted output format is a schema, and the stakeholder notes are the input to transform.

Additive techniques: role priming + structured-output.

• “Act as a product-minded requirements engineer” is an expertise persona (role priming).
• The output sections (User story / Given-When-Then criteria / Open questions / Assumptions) are an explicitly named section schema (structured-output).
• “Do not silently invent requirements” directly targets hallucination; “Open questions” and “Assumptions” force the model to separate what was given from what it inferred.
• Not rubric-based: the four sections organise the output; they are not evaluation criteria to judge the notes against.
• Not context grounding: the notes are the input to transform, not background.

Primary lever: the structured output — specifically the “Open questions” and “Assumptions” sections plus “do not silently invent requirements.” This is what forces the model to separate what was given from what it inferred, the dominant risk for ambiguity-heavy requirements work. Role priming helps set tone, but remove the uncertainty-surfacing structure and the model will quietly invent precision.

Why this strategy is appropriate:

Requirements work is ambiguity-heavy. The prompt is designed to prevent the model from converting uncertainty into invented precision.

Likely failure modes:

• The model may still over-specify compliance requirements.
• It may invent export formats or performance thresholds.
• It may miss security/privacy implications.
• It may produce criteria that are not testable.

What belongs outside the prompt:

• Stakeholder validation.
• Legal/compliance review.
• Product prioritization.

Possible improvement:

Add instruction:

Mark each acceptance criterion as either directly supported by the notes or inferred.

Teaching point:

For requirements tasks, good prompting often means making uncertainty visible rather than asking the model to complete the specification confidently.

Examples axis: zero-shot. No example call, documentation excerpt, or demonstration of the endpoint usage; the prompt assumes the model already knows the current API surface.

Additive techniques: structured-output.

• Structured-output: “return a dictionary mapping YYYY-MM to question count” is a concrete output-shape spec (a named mapping with a specified key format).
• Not tool-augmented: the prompt names no tool or documentation to invoke.
• Not context grounding: no documentation or background is embedded.

The structured-output box is correctly checked — but the shape of the dict is not where this prompt fails.

Primary lever: the present technique (the dict spec) is weak — it shapes the container, not the contents. The quality of this output hinges on a technique that is absent: grounding the model in the real API (documentation or a tool call). So the task’s true lever — context grounding / tool use — is missing, and that gap is exactly what makes this an anti-pattern.

Why this is risky:

This task depends on an external API surface. The model may hallucinate an endpoint, parameter, pagination behavior, rate limit, or response schema.

Likely failure modes:

• Factual hallucination: non-existent endpoint.
• Structural hallucination: plausible but wrong response fields.
• Missing pagination or date handling.
• Code that looks reasonable but fails at runtime.

What belongs outside the prompt:

• Reading official API documentation.
• Running the code against the API.
• Handling HTTP errors and rate limits.
• Tests with mocked API responses.

Possible improved prompt:

Use the official Stack Exchange API documentation provided below. Do not invent endpoints or fields.

First identify which documented endpoint can answer the task. If the API does not provide monthly aggregation directly, say so and compute monthly counts from available data.

Return:
1. A short explanation of the endpoint used.
2. Python code.
3. Assumptions and limitations.
4. Suggested mock tests.

Documentation excerpt:
[PASTE RELEVANT DOCS]

Teaching point:

External API tasks require grounding. Zero-shot prompting is often the wrong tool when the central risk is hallucinating facts about a changing interface.

Examples axis: zero-shot. No example extraction is demonstrated; the field list and enum are an output schema, and the bug report is the input to transform.

Additive techniques: structured-output.

• An explicit JSON request, named fields with types, and an enum of allowed severity values — a named serialization with an enumerated schema.
• Not rubric-based: the fields are a data schema to fill, not evaluation criteria.
• Not context grounding: the bug report is the input to transform, not background.

Primary lever: the structured-output schema. The named JSON fields, types, and severity enum are what make the output pipeline-consumable. Remove the schema and you get prose no downstream system can ingest. Note the schema biases toward valid JSON but does not enforce it — that is the enforcement layer’s job.

Why this strategy is appropriate:

The output is intended to be machine-readable. A structured format helps integrate the model output into a downstream workflow.

Likely failure modes:

• Invalid JSON.
• Missing required fields.
• Severity may be guessed without enough evidence.
• The model may put inferred information where null is more appropriate.
• The model may violate enum constraints.

What belongs outside the prompt:

• JSON schema validation.
• Retry/repair loop if invalid.
• Human review for severity.
• Possibly constrained decoding or structured output API (§15.1).

Possible improved prompt:

Return only valid JSON matching this schema. Use `null` when the report does not provide the information. Do not infer severity beyond the evidence; if uncertain, use `medium` and include the uncertainty in `missing_information`.

[JSON SCHEMA]

Teaching point:

Prompting for JSON is not the same as enforcing JSON. Production use requires schema validation or constrained decoding.

Examples axis: zero-shot. No example comparison is demonstrated; the options, rubric, and output sections are instructions and schema.

Additive techniques: role priming + rubric-based + structured-output.

• “Act as a pragmatic software architect” is an expertise persona (role priming).
• The eight-criterion list (latency, complexity, cost, security, …) is a set of evaluation dimensions to judge each option against (rubric).
• The four numbered output sections (table, recommendation, conditions, assumptions) are an explicitly named section schema (structured-output).
• Not context grounding: the three options are part of the task, not embedded background about the model’s own application.

Primary lever: the rubric. Architecture decisions are trade-off decisions, and the eight-criterion rubric is what prevents a one-dimensional answer (“just use a worker”). Role priming sets tone and the output schema makes the comparison legible, but remove the rubric and the model evaluates on whatever axis it happens to favour.

Why this strategy is appropriate:

Architecture decisions are trade-off decisions. A rubric and structured comparison help prevent a one-dimensional answer.

Likely failure modes:

• Recommendation may depend heavily on unstated context.
• The model may understate operational complexity.
• Security/privacy risks may be generic.
• Cost assumptions may be wrong.

What belongs outside the prompt:

• Real traffic estimates.
• Cost modeling.
• Security review.
• Team capability assessment.
• Architecture decision record reviewed by humans.

Possible improvement:

Before recommending, list the three most important missing pieces of context that could change the decision.

Teaching point:

For architecture, the model is useful as a decision-support tool, not as a decision-maker.

Examples axis: zero-shot. No example middleware is demonstrated; the prompt is a bare behavioural instruction.

Additive techniques: None of these. No role, no rubric, no output-format spec, no reasoning instruction, no embedded background, no tool. “Attach the user ID to req.userId” describes behaviour, not an output schema.

Primary lever: there is no additive technique, so the only lever is the bare instruction — and for a security-critical task it is dangerously thin. It specifies the happy path and nothing about the security policy (algorithms, issuer/audience, expiry, error behaviour). What the task actually needs is layered technique (a security-requirements rubric) plus enforcement and review — the present lever is not enough.

Why this is insufficient:

Authentication is security-critical. A generic zero-shot prompt may omit algorithm restrictions, issuer/audience checks, expiration handling, key rotation, error behavior, logging rules, and dependency choices.

Likely failure modes:

• Accepting unsigned tokens or wrong algorithms.
• Not checking expiration.
• Not checking issuer/audience.
• Leaking token details in errors.
• Attaching unvalidated claims to the request.
• Using outdated or unsafe libraries.

What belongs outside the prompt:

• Security review.
• Dependency review.
• Tests with valid, expired, malformed, wrong-audience tokens.
• Static analysis and secret handling checks.
• Runtime configuration management.

Possible improved prompt:

Act as a security-conscious Node.js backend engineer.

Write Express.js middleware for JWT authentication using `[SPECIFIC LIBRARY]`.

Security requirements:
- Verify signature.
- Accept only algorithm `[ALGORITHM]`.
- Validate issuer `[ISSUER]`.
- Validate audience `[AUDIENCE]`.
- Reject expired tokens.
- Do not log token contents.
- Return a generic 401 on failure.
- Attach only the validated subject claim to `req.userId`.

Also provide pytest/jest-style test cases for:
- valid token
- expired token
- wrong issuer
- wrong audience
- malformed token
- missing token

If any required configuration is missing, state it explicitly instead of inventing it.

Teaching point:

Security-sensitive tasks should not rely on prompt influence alone. They require enforcement, testing, and human review.

Examples axis: zero-shot. No demonstration of any kind — two terse imperatives, nothing to imitate.

Additive techniques: None of these. No role, rubric, output schema, reasoning instruction, or embedded context. Crucially, not tool-augmented: “commit them” implies tool use but names no tool and scopes nothing — per the tool-augmented decision rule, an implied action is not a named tool.

Evidence of what is missing:

• Very little context.
• No acceptance criteria.
• No constraints.
• No testing requirements.
• No permission boundaries.
• Allows commit without review.

Primary lever: there is no additive technique, so the only lever is the bare instruction — far too thin for an autonomous agent with write access. The task’s true levers are absent and external: scoped tool use, task boundaries, a stop-before-commit gate, branch protection, and review. The classification correctly shows “None,” and that emptiness is precisely the danger.

Why this is risky:

Agentic coding tasks can modify many files. Without boundaries and validation requirements, the agent may make broad, low-quality, or incorrect changes.

Likely failure modes:

• Misunderstanding the issue.
• Over-modifying unrelated files.
• Not running tests.
• Creating a commit that passes superficially but breaks behavior.
• Ignoring project conventions.
• Failing silently on edge cases.

What belongs outside the prompt:

• Branch protection.
• CI.
• Human review before merge.
• Test suite.
• Commit hooks.

Possible improved prompt:

Work on issue #214 in a new branch.

Before editing:
1. Read the issue description.
2. Identify the relevant files.
3. Summarize the intended change and ask for confirmation if the scope is ambiguous.

While editing:
- Make the smallest change that satisfies the issue.
- Do not refactor unrelated code.
- Follow existing project style.
- Add or update tests that fail before the fix and pass after it.

Before finishing:
- Run the relevant test subset.
- Run the formatter and linter.
- Provide a summary of changed files, tests run, and remaining risks.
- Do not commit unless explicitly instructed after review.

Teaching point:

Agent prompts need task boundaries, validation instructions, and permission control. The more autonomy the tool has, the more important the enforcement layer becomes.

Examples axis: zero-shot. No example classification is demonstrated; the five-way label set is the allowed-output enum, not a worked demonstration.

Additive techniques: chain-of-thought + structured-output.

• Chain-of-thought: “Think step by step in great detail. Show all your reasoning” is an explicit reasoning instruction.
• Structured-output: the five-way enum (UI / backend / database / infrastructure / unclear) constrains the output to one of a fixed set of allowed values.
• Not rubric-based: the enum is the allowed answers, not criteria for judging them.

Primary lever: the structured-output enum is the lever that actually helps — it forces a clean, machine-checkable label. The chain-of-thought instruction is present but counter-productive here: for a five-way classification it adds verbosity and false confidence without improving the label. The present lever worth keeping is the enum; the CoT is the technique to drop.

Why this may be inappropriate:

The task is a simple classification with ambiguity. A long visible reasoning chain is not necessary and may create false confidence. What matters is the label, confidence, and missing information.

Likely failure modes:

• Verbose answer.
• Rationalization of a weak guess.
• Overconfident classification.
• Hidden ambiguity gets transformed into an explanation.

Better prompt:

Classify the bug report as one of:
- UI
- backend
- database
- infrastructure
- unclear

Return:
- label
- confidence: low / medium / high
- one-sentence rationale
- what information would make the classification more certain

Teaching point:

Chain-of-thought is not automatically better. For many software tasks, a concise rationale plus uncertainty is more useful than detailed reasoning. The faithfulness caveat from §13.5 and §15.3 applies: visible reasoning is not an audit trail.