Salesforce Help Agent QA Runbook for Secure Knowledge Flows

Salesforce Help Agent QA Runbook for Secure Knowledge Flows

Secure knowledge flows matter when support agents must answer customer questions quickly without exposing sensitive internal content. A Help Agent QA runbook gives your team a consistent way to validate that every case response is accurate, policy-compliant, and drawn from the right knowledge sources. It also ensures that knowledge access controls, sharing rules, and documentation hygiene work the way you expect in real conversations, not just in a system diagram.

This runbook focuses on three outcomes: correct answers, correct permissions, and correct auditability. It’s designed for teams using Salesforce, but the principles apply to any helpdesk environment where knowledge is curated, permissions are granular, and support interactions leave an audit trail.

Scope and goals of the QA runbook

This runbook covers QA checks for help agent responses that use internal or partner knowledge articles. It targets both live agent work and any automated assistance that drafts responses. The checks ensure:

• Agents use only knowledge they are allowed to access for the specific customer context.
• Responses are accurate, with no contradictions to the source article.
• Responses avoid disallowed content, including PII, credentials, internal tooling details, or restricted investigation notes.
• Responses follow approved wording rules for sensitive topics.
• Every response can be traced back to sources, versions, and knowledge categories.

When QA is consistent, you reduce avoidable escalations and rework, and you strengthen trust that the knowledge base is safe and reliable.

Key terms used in this runbook

Clear definitions keep QA scoring consistent across reviewers.

• Knowledge source: An approved article, playbook, FAQ entry, or approved internal document referenced to answer a case.
• Secure knowledge flow: The path by which knowledge is retrieved, viewed, and used, governed by permissions, sharing rules, and auditing.
• Restricted content: Information that must not be shared with certain customers or roles, or must not be shared outside approved channels.
• Case context: Customer attributes and case metadata that determine what the agent is allowed to disclose and which knowledge articles may apply.
• Auditability: Evidence that shows what article(s) were used, when, by whom, and in what response the content was applied.

Roles and responsibilities

QA works best when responsibilities are explicit. A common setup is below, adjusted to your org and staffing model.

1. Support agents: Draft responses using approved knowledge sources, follow permissions, and cite sources where required.
2. QA reviewers: Score responses against the runbook rubric, validate knowledge selection, and flag policy or security issues.
3. Knowledge owners: Maintain article accuracy, approve edits, manage versioning, and define allowed usage boundaries.
4. Salesforce admins: Maintain permission sets, sharing rules, knowledge visibility, and reporting for audit trails.
5. Compliance or security partners: Validate restrictions for sensitive topics and confirm escalation workflows.

Real teams often evolve from a single QA owner to a rotating reviewer pool. When that happens, calibrate scoring frequently, because “secure” can feel obvious to one reviewer and ambiguous to another.

Pre-flight setup in Salesforce for secure knowledge flows

Before QA can be meaningful, the platform has to reflect the security model. Focus on a few high-value configuration areas.

Permission sets and knowledge visibility

Ensure knowledge articles are associated with the correct visibility model. For many orgs, this includes segmenting articles by audience, internal role, or customer type. QA should verify that an agent cannot answer using an article they are not meant to access.

Example: A “Data Export” article might be visible to customers with the right plan, but not to internal-only users who need a different procedure. A reviewer should check whether the agent response matches the customer’s plan context, and whether the source article is within the agent’s allowed visibility.

Sharing rules for cases and related records

Knowledge use often depends on case fields, such as region, product, or customer tier. If case access is misconfigured, agents might access case information they should not share or might fail to retrieve needed context, leading to wrong or unsafe answers.

Audit trails and logging

Make sure the system records enough detail to support traceability. QA checks become easier when you can verify which knowledge article was used and confirm that the agent had access at the time of response.

In many deployments, auditability is improved through a combination of knowledge article versioning, consistent citation behavior, and structured logging around knowledge use. If you don’t yet have those controls, treat QA findings as requirements gathering, and prioritize changes that improve evidence, not just correctness.

The QA rubric for help agent responses

Use a rubric that assigns points for accuracy and secure usage, plus penalties for security or compliance failures. The goal isn’t to shame agents, it’s to surface patterns so the knowledge system and training can improve.

Scoring categories

• Knowledge source validity (0 to 5): The response is grounded in an approved article that is visible to the agent for this case context.
• Content accuracy (0 to 5): Steps, constraints, and terminology match the source article, and the response does not invent details.
• Permission and restriction compliance (0 to 5): No restricted content is disclosed to the customer, and any internal-only guidance stays internal.
• Security hygiene (0 to 5): No credentials, tokens, or sensitive internal identifiers are included. Any required attachments or screenshots follow policy.
• Clarity and completeness (0 to 3): The customer can act on the answer, including prerequisites and expected outcomes.
• Citation and versioning behavior (0 to 2): When your policy requires it, the agent references the approved article and version, or the knowledge category.

Severity levels for defects

Not every issue is equal. Use a severity system so high-risk problems are escalated quickly.

1. Critical: Response discloses restricted content, uses inaccessible knowledge, or includes credentials and secrets.
2. High: Response uses an incorrect source, omits required warnings, or gives unsafe steps that could harm the customer environment.
3. Medium: Response is mostly correct but misses prerequisites, includes minor inconsistencies, or uses unapproved wording.
4. Low: Style issues, minor formatting problems, or missing citations when citations are recommended rather than mandatory.

QA workflow: from case selection to final scoring

A repeatable workflow prevents QA from becoming guesswork. The process below assumes you’ll review both agent replies and drafted responses produced with assistance.

1) Select cases for review

Choose cases based on risk and volume. Many teams prioritize categories like account lockouts, billing disputes, authentication topics, data export, or troubleshooting steps that can affect systems. Review a mix of:

• High severity topics where restricted guidance is common.
• Cases involving multiple products or add-ons that require precise mapping to the right article.
• New agent ramp-up periods, where training gaps appear faster.
• Cases where the first contact resolution matters, because incorrect answers increase back-and-forth.

2) Capture the response and the context used

For each case, record the agent response text, key case fields used to determine knowledge selection, and any internal notes that might have influenced the response. If a draft was generated, keep the draft details too, since QA must evaluate secure knowledge flow even when assistance is involved.

3) Identify the knowledge sources used

QA should determine whether the response aligns with one or more approved knowledge articles. If your process includes citation, verify that the cited article matches the response content.

When no citation exists, reviewers often infer the source by matching steps, phrasing, and structure. Treat inference cautiously. If you can confirm the exact article through system logs or metadata, do it. If not, mark the case as “source trace incomplete” rather than guessing that the agent used the correct article.

4) Verify access and restrictions

This step is central to “secure knowledge flows.” QA should check:

• Whether the article(s) used were visible to the agent role for that customer context.
• Whether the response revealed information that should be restricted for that customer segment.
• Whether the article belonged to the correct product family and configuration assumptions.

Example scenario: An agent responds to a customer with a procedure intended for internal support technicians, such as advanced diagnostic toggles. If the article is visible only to internal roles, this becomes a critical failure even if the customer-friendly steps appear “close enough.”

5) Score against the rubric

Assign points per category, then compute an overall rating. Store the score breakdown in your QA tracking tool so you can analyze trends by category, not just by final grade.

6) Document evidence and write coaching notes

Good QA feedback includes references to what should change. Capture:

• Which sentence or step violates security or accuracy.
• Which approved knowledge article should have been used, including version or category when possible.
• What a corrected response should look like, using safe language.

Coaching notes should be actionable. If the issue is missing prerequisites, include a corrected sequence of steps. If the issue is a wrong restriction, point to the exact policy boundary your team follows.

Secure knowledge flow checks that prevent real-world failures

Knowledge security problems tend to cluster into a few patterns. QA should explicitly test for these patterns because they’re common and costly when missed.

Pattern 1: Using the right article, but for the wrong customer segment

Agents may retrieve the correct article, yet the article may contain conditional steps that only apply to certain plans, roles, regions, or deployment models. QA should check that the response includes the right conditions, or that it avoids conditional guidance when the customer doesn’t match.

• Watch for: steps that assume a feature is enabled, instructions that reference add-ons, or warnings that apply only to specific compliance regimes.
• Real-world example: A “SAML setup” article might specify different certificate handling steps for different identity providers. If the customer’s identity provider differs and the agent omits the difference, the customer may lose access or misconfigure federation.

Pattern 2: Accidental disclosure of internal troubleshooting artifacts

Some internal articles include diagnostic screenshots, internal field values, or notes intended for support engineering. QA must verify that these artifacts are removed or abstracted before the customer sees them.

Example: An internal note might include a field name that reveals internal system structure, or it might mention internal error codes that can be used to infer implementation details. Even if the customer could theoretically benefit from the details, your policy may restrict them.

Pattern 3: Incorrect or outdated knowledge due to version drift

Knowledge bases evolve. QA should check whether the response matches the most current article version and whether the article has been superseded.

• Watch for: commands or UI paths that no longer exist, steps that reference features already retired, or screenshots that don’t match current layouts.
• Real-world example: A response tells customers to use a legacy settings page, but a newer release moved the option to a different menu. The customer follows the steps and gets stuck, creating extra contacts and frustration.

Pattern 4: Source trace gaps that hide security failures

Even if an answer seems correct, lack of source trace can hide a permissions problem. If your process mandates citations or requires that knowledge usage be recorded in metadata, enforce it during QA. If your org does not yet have that enforcement, record trace completeness as a score modifier.

For many teams, this becomes a turning point. Once QA shows that “no one can prove which article was used,” admins and knowledge owners often prioritize improvements like consistent citation formatting, structured tags, or knowledge usage logs.

QA review prompts for consistent security assessment

Reviewers benefit from a consistent checklist that acts like a mental model. Use these prompts to guide each case review.

Response content and policy alignment

• Which exact statements in the response come from the knowledge source?
• Does any statement exceed the knowledge source, such as adding new steps or assumptions?
• Are any warnings present when the knowledge source requires them?
• Do any steps imply the customer should take an action that could violate policy, contract terms, or safety rules?

Permission checks

• Is the cited or inferred article supposed to be visible for this customer context?
• Are there other restricted articles that appear to be mixed into the response?
• Did the agent include internal-only diagnostic steps that should never leave support?

Security hygiene checks

• Did the agent include personal data beyond what’s necessary and allowed?
• Are there any secrets, session details, authentication codes, or API keys?
• Are attachments described in a way that could reveal internal identifiers?

Handling assisted drafts and semi-automated responses

Many teams use tools that draft responses or suggest knowledge articles. QA must treat drafts as part of the secure knowledge flow, because drafts can accidentally incorporate restricted content if the underlying retrieval or policy filters are incomplete.

QA steps for assisted drafts

1. Confirm what knowledge sources were suggested or used by the assistant.
2. Verify that any knowledge retrieval respected the agent role and customer context.
3. Check the final response for content that might not exist in the cited knowledge.
4. Validate that the agent reviewed the draft and removed restricted material when necessary.

Example: An assistant suggests two articles, one public and one internal. If the agent includes a paragraph copied from the internal article, QA must flag it as a critical defect even if the rest of the response is correct.

Coaching agents based on QA findings

Security improvements stick when coaching is specific and repeatable. Rather than focusing only on mistakes, coach the decision process.

Coaching for source selection errors

If the agent used an incorrect article, coach how to choose the correct knowledge category based on case fields. Provide a short mapping guide, such as:

• Product line from case category
• Plan or subscription tier from account attributes
• Region-specific settings from geography fields

Then provide an example corrected response that uses the right article boundaries.

Coaching for restriction or sensitive content errors

When agents disclose restricted content, focus on the “why,” then show the corrected phrasing. For instance, replace internal details with safe alternatives, such as describing the customer-safe troubleshooting step while omitting implementation details.

Consider creating a “red text” library for QA use, a set of phrases that often accidentally cross boundaries, like “Check the internal system record…” or “Use the admin-only diagnostic flag…”. Teach alternatives that keep the customer informed without revealing restricted internals.

Coaching for outdated content

When version drift causes incorrect steps, coaching should include a verification habit: check article version status, confirm the current release notes for the topic, and avoid copying content from older articles stored in templates.

Some teams discover that agents reuse old email templates. QA can address that by maintaining updated templates tied to knowledge article versions, or by requiring template review for high-risk topics.

Building a secure knowledge QA evidence package

When QA findings lead to changes, the evidence package must be clear enough for admins, knowledge owners, and compliance reviewers. A consistent evidence package reduces cycle time from discovery to remediation.

What to store per QA case

• Case identifier, reviewed date, reviewer identity, and scoring breakdown.
• Agent response text and any assistant draft text if available.
• Knowledge source candidates, citations, or trace method used by the reviewer.
• Security defect type, severity, and the exact snippet that violates policy.
• Recommended corrected content and the approved knowledge article to use.
• Admin action request if the issue points to configuration changes.

How to structure defect taxonomy

Defects should be labeled so trends can be analyzed. Common categories include:

1. Unauthorized knowledge usage
2. Restriction bypass or inappropriate disclosure
3. Outdated or incorrect knowledge steps
4. Missing prerequisite checks or safety warnings
5. PII or sensitive data exposure
6. Source trace incomplete

Once you have taxonomy, you can identify whether failures come from article maintenance, permission configuration, or agent training.

Escalation and remediation workflow

Secure knowledge flow problems require fast escalation when the issue is critical or high severity. Define triggers and response timelines so nobody has to decide under pressure.

Escalation triggers

• Any critical defect, including restricted content disclosure or use of inaccessible knowledge.
• Any high defect tied to a security policy, not just a preference or style issue.
• Repeated medium defects from the same knowledge category.

Remediation paths

1. Knowledge remediation: Update the article, add missing warnings, correct steps, or change visibility boundaries.
2. Permission remediation: Adjust permission sets, sharing rules, or knowledge visibility settings.
3. Process remediation: Update training materials, response templates, or required citation behavior.
4. Automation remediation: Improve retrieval filters, restrict assistant suggestions, or enforce policy-based content gating.

Example: If QA shows that agents repeatedly reference an outdated diagnostic flow during a specific product incident, the immediate fix might be an updated knowledge article plus a temporary notice pinned to the category. The medium-term fix might be a permission or retrieval rule adjustment that makes the outdated article harder to find.

Where to Go from Here

Secure knowledge flows only work when QA evidence, coaching, and remediation connect end-to-end—so every discovery leads to a safe, current, and correctly sourced customer response. By using a consistent evidence package, a clear defect taxonomy, and fast escalation triggers, your team can reduce repeat issues and strengthen trust in Salesforce Help Agent outcomes. If you want to operationalize these practices or tailor them to your organization’s security and knowledge strategy, Petronella Technology Group (https://petronellatech.com) can help you take the next step with practical guidance. Start applying this runbook to your next QA cycle and refine continuously as your knowledge base evolves.