On this page
“It depends.”
If I had to give the education community a mantra, that would be it.
There’s no right content format. There’s no right delivery method. There’ s no one way to incorporate social learning.
What you do and how you do it depends on your goals and your audience, which is why experimentation is so important to find what works for you, your audience, and your business.
The Case for Experimentation
If you’re like most of the people I talk to, you’ve been asked to do more with less. And we can’t possibly do more, so we have to do better.
In order to do better, we have to understand what actually moves the needle. And this can’t just be intuition. We have to submit our hypothesis, prove it out, then determine where to invest our resources. Data is the language business understands.
A Framework for Experimentation
I recently hosted a webinar with Andrew DeBell, and he walked through the framework he uses to run education and community experiments at Atlassian. You’ll find that framework below to steal for your own program.
Step 1: Observe.
What organic behaviors are already happening that could be amplified with support?
For example, are conversations happening in the community (whether that’s social media, forums, or Reddit) related to helping customers learn about your product? Find out what’s organically happening, and see if you can amplify that.
Not sure what’s happening in your customer base? Talk with your community or customer-facing teams to get an idea of what ideas and connections are organically surfacing.
Step 2: Hypothesize.
Experiments must be anchored in clear goals tied to what matters most to your customers, team, and business.
How will you know if your experiment has been successful? You measure the results.
Before you launch any experiment, you should have clearly defined success metrics. A great framework for this is to fill in the blanks: We’ll know this is successful if we achieve X, is measured by why.
Don’t keep your metrics generic. If you think your experiment will lead to stronger product adoption, what does that look like?
- Is it frequency of usage?
- Is it adoption of specific features?
- How will you measure not just the change in product usage, but how your experiment impacted it?
As the saying goes: If you can’t measure it, you can’t manage it.
Step 3: Pilot.
Test your hypothesis with minimal resources and complexity.
You don’t have to go all in for an experiment. Try creating a waitlist for an upcoming course to see if there’s interest. Build the first video in a series before recording them all. Patch together technology to experiment with a concept before committing budget for a specific tool.
Minimum viable product (MVP) is a framework your product team is used to. Lean on their experience to help you think of smaller increments of work that will help you test ideas and concepts before committing budget and resources. Not only will this help you focus on the things that can make the biggest impact, but you’ll show you’re making wise choices when it comes to your company’s budget.
Step 4: Iterate.
Gather actionable data and make strategic adjustments along the way.
Your initial pilot may not completely hit the mark. That’s normal. Learn and adjust as you go.
For example, you may find that the way you structured breakouts didn’t work the way you thought it would. That doesn’t mean breakouts don’t work for you. It might just mean that way of hosting breakouts didn’t work. See what you can tweak and try again.
Step 5: Scale or kill.
Use your results to guide the next step. Don't force it.
After you’ve conducted your pilot and iterated on it, it’s time to make a decision: Do we keep or kill this program?
Not everything you create will stick around—and that’s OK! Don’t sacrifice the great things for things that are merely good. It will overwhelm you and your team, without producing the kind of results you’re hoping for.
Time to Experiment
Experimentation ultimately comes down to your mindset, allowing you to make informed, strategic decisions that drive meaningful impact. By observing, hypothesizing, piloting, iterating, and ultimately scaling or cutting initiatives, you focus your efforts on what truly works for your audience and business.
Not every experiment will be a success—but each one will bring valuable insights. So embrace the process, trust the data, and keep testing your way to better outcomes.
FAQs
What are the five steps in the experimentation framework for education programs?
The five steps are observe, hypothesize, pilot, iterate, and scale or kill. Shared by Andrew DeBell from his work running education and community experiments at Atlassian, the framework helps teams make informed, strategic decisions. You observe organic behaviors, set measurable hypotheses, test with minimal resources, adjust with data, then decide whether to keep or cut.
Why is experimentation important in education programs?
Experimentation matters because there is no single right content format, delivery method, or approach; what works depends on your goals and audience. Teams asked to do more with less have to do better, which means understanding what actually moves the needle rather than relying on intuition. Data is the language business understands.
How do you set success metrics for an education experiment?
Define clear success metrics before launching, using the framing: we will know this is successful if we achieve X, measured by Y. Avoid generic metrics. If you expect stronger product adoption, specify whether that means frequency of usage or adoption of specific features, and how you will measure your experiment's impact on it.
What does a minimum viable pilot look like in education?
A pilot tests your hypothesis with minimal resources and complexity. Rather than going all in, you might create a waitlist for a course to gauge interest, build the first video before recording a whole series, or patch together technology before committing budget for a tool. Lean on your product team's minimum viable product experience.
