Feedback System

Overview

The SDAC feedback system collects append-only user ratings on AI Widget responses and supports two audited consumers: prompt tuning and grouped Linear issue automation. Prompt tuning can propose or apply a profile change; Linear automation uses the Feedback Planner to turn unprocessed evidence into review issues without changing a prompt.

Endpoints

Submit Feedback

POST /sdac/feedback

Called by the feedback widget through POST /api/ingestion/sdac/feedback after a user rates an agent response.

Request:

{
  "agentId": "sdac-coordinator-release",
  "conversationSk": 12345,
  "reportId": "b377a9be-9d4a-434a-9688-ba828b39c9f7",
  "userId": "jsmith@contoso.com",
  "sessionId": "sess-abc-123",
  "turnNumber": 3,
  "rating": 4,
  "category": "clarity",
  "comment": "Could use less jargon"
}

Field	Type	Required	Description
agentId	string	Yes	Agent that produced the response (e.g. sdac-coordinator-release)
conversationSk	number	Yes	ConversationSK from the chat session
reportId	string (UUID)	Yes	Cost report GUID
userId	string	Yes	Authenticated user ID
sessionId	string	Yes	Browser session ID
turnNumber	number	Yes	Which turn in the conversation (0-indexed)
rating	number	Yes	1-5 star rating
category	string	No	See Rating Categories below
comment	string	No	Free-text explanation, max 2000 chars

Response (200):

{
  "feedbackSk": 42,
  "success": true
}

Behaviors:

• Append-only -- every accepted submission inserts a new evidence row. There is no user/turn uniqueness constraint or update-in-place behavior.
• At the direct runtime contract, rating is the only required feedback field beyond the identifiers; category and comment are optional.
• The current widget UI is intentionally stricter: it submits thumbs up/down as rating 5/1 and requires a category plus a non-empty comment.
• The widget sends feedbackScope, but the current runtime Zod schema does not declare it, so it is stripped and not persisted.

Feedback Stats

GET /sdac/feedback/stats/{agentId}

Aggregated feedback metrics for a given agent. Use this for a quality dashboard or sparkline display.

Example:

GET /sdac/feedback/stats/sdac-coordinator-release

Response (200):

{
  "agentId": "sdac-coordinator-release",
  "totalCount": 10,
  "avgRating": 2.8,
  "ratingDistribution": {
    "1": 2,
    "2": 3,
    "3": 2,
    "4": 1,
    "5": 2
  },
  "categoryBreakdown": {
    "accuracy": 3,
    "clarity": 2,
    "relevance": 1,
    "helpfulness": 2,
    "tone": 1
  }
}

Rating Categories

Value	When to Use
accuracy	Agent gave incorrect information
clarity	Response was confusing or hard to read
relevance	Agent answered the wrong question or drifted off-topic
helpfulness	Found issues but gave no guidance on fixing them
tone	Condescending, too casual, or otherwise inappropriate
other	Catch-all for anything else

Category is optional. A simple 1-5 star widget works without it.

That optionality describes the runtime API. The current widget requires a category and comment before enabling submission.

Admin Endpoints

Trigger Prompt Tuning

POST /admin/sdac/tune-prompt

Runs the prompt tuning workflow and returns a flattened response. See Prompt Tuning Workflow for the full pipeline.

Request:

{
  "targetAgentId": "sdac-coordinator-release",
  "daysBack": 30,
  "minFeedbackCount": 5,
  "triggeredBy": "admin-ui",
  "applyPrompt": false,
  "selectedFeedbackIds": [101, 102, 103]
}

Field	Type	Required	Default	Description
targetAgentId / agentId	string	Yes	--	Agent to tune; agentId is accepted for admin dashboard compatibility
daysBack	number	No	30	Days of feedback to analyze (max 90)
minFeedbackCount	number	No	5	Minimum feedback items required (min 1)
triggeredBy	string	No	admin-ui	Who triggered the run (user ID or label)
applyPrompt	boolean	No	true	When false, return a draft suggestion for review
selectedFeedbackIds	number[]	No	--	Specific FeedbackSK rows to analyze instead of the whole feedback window

The admin dashboard supports selecting multiple feedback cards and generating one prompt suggestion from only those rows. If no feedback cards are selected, the workflow analyzes the normal daysBack window. The dashboard presents the result as current prompt, suggested prompt, diff, changes applied, and an Evidence tab showing the deterministic feedbackInsights that explain what actionable information came from the feedback.

Response (200):

{
  "success": true,
  "promptChanged": true,
  "gatedOut": false,
  "gateReason": null,
  "runId": 6,
  "executionTimeMs": 8200,
  "currentPrompt": "You are the current SDAC Coordinator...",
  "revisedPrompt": "You are the SDAC Coordinator...",
  "candidatePrompt": "You are the SDAC Coordinator...",
  "diff": "--- current prompt\n+++ suggested prompt\n@@\n-Old line\n+New line",
  "rationale": "Feedback showed recurring issues with...",
  "confidenceScore": 0.61,
  "confidence": 0.61,
  "changesApplied": [
    "Added row-level lookup rule for source/function questions (feedbackSK 101).",
    "Added current-report freshness rule (feedbackSK 102)."
  ],
  "feedbackSummary": "Common problems: accuracy errors...",
  "feedbackInsights": [
    {
      "feedbackSK": 101,
      "category": "accuracy",
      "rating": 3,
      "userRequest": "Are there any issues with source or function codes?",
      "feedbackComment": "This file had a position missing source and function codes.",
      "promptActions": ["Use row-level personnel lookup for exact source/function questions."],
      "evidencePreview": "No source or function code issues were flagged.",
      "createdAt": "2026-05-22T19:59:22.956Z"
    }
  ],
  "newVersionNumber": 20,
  "feedbackStats": {
    "totalCount": 10,
    "avgRating": 2.8,
    "categoryBreakdown": { "accuracy": 3, "clarity": 2 }
  }
}

Response states:

promptChanged	gatedOut	Meaning
true	false	Prompt was updated
false	true	Not enough feedback
false	false	Had feedback but no change needed

Tuning Run History

GET /admin/sdac/tune-prompt/runs

Param	Type	Required	Default	Description
agentId	string	No	all agents	Filter by agent
limit	number	No	20	Max rows returned (capped at 100)

Response (200):

{
  "runs": [
    {
      "runId": 6,
      "targetAgentId": "sdac-coordinator-release",
      "feedbackCount": 10,
      "confidenceScore": 0.61,
      "status": "applied",
      "triggeredBy": "admin@example.com",
      "createdAtUtc": "2026-02-10T18:37:35.000Z"
    }
  ]
}

Status	Meaning
applied	Prompt was changed and written to DB
no-change	Workflow ran but prompt was not updated
skipped	Gated out due to insufficient feedback
rolled_back	A previous applied run was manually reverted
error	Workflow encountered an error

Self-Improvement And Linear Automation

POST /admin/sdac/self-improve runs prompt tuning for one or more agents with unprocessed feedback. It defaults to a 30-day window, minimum one item, at most 10 agents, and marks consumed rows when a tuning audit run is created.

sdac-feedback-linear-automation is a registered scheduled workflow. It reads unprocessed feedback oldest-first, asks sdac_feedback-planner-agent for grouped actionable issue drafts, and creates issues in the RUU team / SDAC AI project only when Linear posting is configured. A successful non-dry run records a draft audit row before assigning ProcessedAtUtc and ProcessedRunId. Dry runs and disabled Linear posting do not consume rows.

Architecture

Data Retention

Data	Retention	Table
Feedback rows	90 days	SDAC.fact_UserFeedback
Tuning run logs	180 days	SDAC.log_PromptTuningRuns

Both tables carry ExpiresAtUtc values and indexed expiry columns. Cleanup is an operational retention responsibility; the table DDL itself does not delete rows automatically.

Code Location

• Routes: packages/domain-sdac/src/routes/sdac-feedback.routes.ts
• Store feedback tool: packages/domain-sdac/src/tools/store-feedback.tool.ts
• Read feedback tool: packages/domain-sdac/src/tools/read-feedback.tool.ts
• Prompt tuning workflow: packages/domain-sdac/src/workflows/
• Feedback/Linear automation: packages/domain-sdac/src/workflows/feedback-linear-automation.workflow.ts
• Feedback Planner: packages/domain-sdac/src/agents/sdac-feedback-planner-agent.ts