Building a Conversational Agent for job aids (RAG Use Case)
What if you could turn dense documentation into an always-on assistant that provides quick, contextual answers to complex questions? An agent that lives and breathes the domain and shares always up to date information with its users. That was the idea behind creating an AI-powered conversational agent that we created at a large Utility customer— and with the Microsoft Co-Pilot studio, this was a task made super easy.
Step 1: Focus on the Need
Every project begins with a problem to solve. For this conversational agent, the challenge was clear: team members needed reliable, on-demand answers to questions without the friction of searching through countless documents.
The Question:
How can we reduce the time spent looking for information and increase the confidence in the answers provided?
The solution had to:
Understand the nuances of specific processes.
Deliver clear, accurate answers based on trusted documentation.
Be intuitive and fast enough for everyday use.
The first step was to describe the agent’s purpose. Microsoft Co-Pilot Studio provided an intuitive interface for this, as seen in the "Describe Your Agent" screenshot. Using everyday language, I clarified the agent’s role to help users navigate their job aids efficiently.
“I’d like to create an Agent that can read up all the job aids from specific folders and answer business questions around how the process works, the use of applications and point to relevant job aids where necessary”
Step 2: Build with Simplicity
Microsoft Co-Pilot Studio made the process refreshingly straightforward. The interface guided me through setting up the agent in just a few steps:
Name and Purpose: I named the agent "JobAidAgent" and defined its role, as seen in the "Agent Details" screenshot.
Knowledge Integration: I uploaded the documents and configured the agent to use them as the source of truth.
Response Confidence: I instructed the agent to always provide a confidence score, ensuring users could trust its responses.
The user-friendly platform made it easy to create a robust conversational interface without extensive technical expertise.
“Enter the following data or whatever meets your needs.”
Step 3: Add the Knowledge Base
Instead of diving into complex integrations or custom code, With Co-Pilot I could simply upload a collection of job aids and process documents. These became the foundation of the agent’s knowledge.
The Approach:
Microsoft Co-Pilot Studio allowed me to upload documents directly to build the agent’s knowledge base. The "Add Knowledge" screenshot shows how I could seamlessly include files
By enabling AI to use these documents, the agent was equipped to retrieve and synthesize information effectively.
“Upload all the files you need to simply link to an existing SharePoint link”
Step 4: Test, Iterate, Refine
The real magic happened during testing. Co-Pilot Studio’s built-in testing environment allowed me to simulate questions like, "How do I update a revenue class?" The agent’s responses were not only accurate but structured step-by-step, as shown in the "Test Your Agent" screenshot:
Log in to the system with your credentials.
Navigate to the "Revenue Class Management" section.
Select and edit the relevant revenue class.
Make necessary changes and save updates.
This iterative process revealed areas for refinement, such as improving the agent’s handling of ambiguous queries or prompting for more context when needed.
“And begin to test your Agent. Try many variations to fully test how the model is working and then publish the model”
Lessons Learned
This project reinforced an important lesson: simplicity scales. By starting with a clear focus and leveraging tools like Microsoft Co-Pilot Studio, I avoided unnecessary complexity while delivering immediate value.
Challenges Overcome:
Overlapping Contexts: Clear segmentation of topics ensured the agent retrieved the most relevant information.
Ambiguity in Queries: Training the agent to ask clarifying questions improved user satisfaction.
Knowledge Gaps: Escalation workflows ensured unresolved questions didn’t become roadblocks.
