On this page
Key takeaways
- Ten questions separate a specific, checkable agentic learning claim from a polished demo, and none of them require inside knowledge of any vendor’s technology.
- Strong answers from an agentic learning vendor name actual steps, actual permission checks, and actual data policies, rather than general reassurances.
- An agentic learning vendor should be able to draw a clean line between what’s generally available, what’s in early access, what’s pilot-only, and what’s still roadmap.
- A demo environment built to impress can hide behavior that shows up only when the agent runs against your own content and your own learner permissions.
Ten questions separate an agentic learning claim you can trust from a demo built to impress. None of them require inside knowledge of any vendor’s technology. They ask a vendor to be specific about what the agent actually does with your content, your permissions, and your data. Bring all ten into your next evaluation, whichever platform you’re evaluating.
What does the agent complete after a learner asks for help?
Ask the vendor to narrate the user-visible steps behind a single request: retrieval, context use, tutoring, grading, review, or recommendation. You don’t need proprietary implementation detail, just a clear account of what happens between a learner’s question and the answer they receive.
A vague answer here is the first sign a vendor hasn’t mapped their own system. A specific one names each step and where it happens.
Which content can the agent retrieve?
Test the actual pages, courses, videos, and assessments your program depends on. Ask what extraction or indexing work each format requires before the agent can search and interpret it.
A platform built only for clean text pages will behave differently against your video library than one built to extract from video and slides too. Ask for that difference directly.
How are learner permissions enforced?
Confirm that retrieval respects catalog access, groups, roles, sites, and restricted content at the moment of the request, not just at content assignment. A learner should only ever receive what their existing access rules already allow.
A permission check that runs once, at assignment, behaves differently from one that runs at the moment of the request. Only the second reliably keeps a learner from seeing something they no longer should.
What learner context is used today?
Ask which current capabilities use course progress, catalog access, assessment results, language, enrollment, or other attributes, and separate what’s live now from what’s still roadmap. Vendors should be able to draw that line clearly.
If a vendor can’t separate live from planned on the spot, treat every context claim in the pitch as unconfirmed until they can.
What happens when the content can’t answer the question?
Watch whether the system acknowledges the gap and shows available sources, or invents an unsupported answer instead. Showing a safe next step when it can’t answer is the sign of a system built on retrieval, not pure generation.
Test this directly. Ask about something you already know isn’t in the catalog and watch what comes back.
How does the system handle conflicting content?
Ask what a learner sees when two retrievable sources give different instructions, and how your content team corrects the underlying conflict once it’s found. A good answer names both the learner-facing behavior and the content-operations fix.
If the vendor only has an answer for one of those two parts, ask directly about the other before you move on.
How do grading and assessment review work together?
Use an open-ended assessment example to see how grading instructions, written feedback, overall analysis, and guided remediation appear to the learner from submission to remediation. Two connected capabilities should read as one experience, not two tools bolted together.
Walk through one submitted response from score to remediation in a single sitting. If the handoff between grading and review reads as one continuous conversation to you as the evaluator, it will to the learner too.
Which controls are available to administrators?
Confirm how each capability is enabled, which audiences can receive it, and what existing permissions govern the experience once it’s on. Feature enablement should be your decision, not something the platform decides on your behalf.
Ask to see the actual admin screen, not a description of one.
What data is sent to a model provider?
Ask what leaves the platform, how long it’s retained, whether your content trains the model, and which contractual or technical protections apply. Get this in writing, not just as a verbal answer in a demo.
This is the question with the most variation across vendors, so treat it as a contract term to negotiate, not a detail to confirm later.
Which capabilities are available now?
Document what’s generally available, available through a labs or early-access program, limited to a pilot, or still on the roadmap. A vendor who draws this line clearly is easier to build a rollout plan around than one who blurs it.
A roadmap slide is not a capability. Hold the vendor to whichever side of that line they claim a feature sits on.
What a strong answer looks like across all ten
The strongest signal isn’t a polished answer to any single question. It’s whether a vendor treats all ten as reasonable to ask, and answers with specifics from your own content and permissions rather than a demo environment built to impress.
Bring this list into your next evaluation, whichever platform you’re evaluating. If you want to run it against Intellum’s own learning management system, ask your Intellum contact for a walkthrough using your own catalog and permissions instead of a generic demo.
