AI Quiz Explanations: How Rocketseat Automates Them

See how Rocketseat generates AI quiz explanations with human review, OpenRouter routing, and Zod schema validation before students ever see them.

How AI Quiz Explanations Work in the Rocketseat Platform

AI quiz explanations at Rocketseat are pre-generated in the admin panel, reviewed by the education team, saved to the database, and read back to the student as a plain database lookup. Nothing about the student's answer triggers a live language-model call, which keeps latency and cost flat as usage grows.

Rocketseat is a Brazilian programming education platform; the case was presented on 2026-03-26 in a live stream by Levi Santiago, a backend developer on the platform team, and Thiago, the education area coordinator. The company runs a catalog of 23 learning tracks and close to 100 active course titles, and every course carries at least one quiz.

The feature moves explanation writing out of the student request path. A student answers a question, the server validates correctness, and the explanation arrives from a row that was written days or weeks earlier. That split is the whole architecture in one sentence, and it is what makes the rest of the design possible.

The student-facing side of the change is a dedicated quiz page. Santiago showed it on stream: the quiz used to open in a small modal with the rest of the platform visible around it, and the platform replaced that with a full page so the student can concentrate. The explanation panel sits beside the question as the student works through it, and it tries to explain why an option is wrong without simply handing over the correct one.

Why Generate Explanations Ahead of Time Instead of Per Answer

Generating on demand was rejected because it puts a non-deterministic model inside a request that many students can make at the same time. Santiago described the concern directly: several students answering a recently released quiz would mean repeated model calls, each producing slightly different text.

Two problems follow from per-answer generation. The first is scale, because the number of calls grows with student traffic rather than with content. The second is consistency, since the same wrong option could receive a different explanation for different students on the same day.

The platform also treats language models as unreliable narrators. Santiago noted that models generate text well and follow style instructions well, but they are not deterministic and can produce an answer the team did not intend. A student-facing explanation for a graded question is not a place to accept that variance.

Pre-generation turns an unbounded runtime problem into a bounded editorial task. The education team generates explanations once per quiz, checks them, and the published result is fixed until someone edits it.

The Generation Pipeline: Streaming, Structured Output, and Schema Validation

The generation service calls a model through the Vercel SDK, a TypeScript toolkit for building LLM features, which the team uses to switch between providers without rewriting integration code. The SDK is the abstraction layer; the request still goes to one specific model.

Rocketseat routes those calls through OpenRouter, a service that exposes many models behind one API. Santiago listed the appeal plainly: one place to pick a model, and one dashboard showing what was spent and on which model. OpenRouter also serves as the gateway the team uses to watch monthly spend.

The response is forced into JSON so the pipeline never has to parse prose. Each question arrives with its options, and each option carries its own explanation, which maps directly onto how the quiz data is stored. Requesting plain JSON rather than a friendly narrative answer also removes the room for the model to pad the response with commentary the pipeline would have to Stripe out.

Validation happens with Zod, a TypeScript schema validation library. The schema is passed through the SDK so the model must return data matching that shape before anything is written to the database. Malformed output fails validation instead of reaching a student.

The prompt carries the language and tone the explanation must use (educational, tied to the specific option), the input format, the required JSON output shape, and any extra rule the reviewer added. Once a valid response comes back, the pipeline structures it and writes the explanations to the database, where the student flow picks them up.

What the Education Team Sees Before Publishing

Inside the admin area, a quiz shows its questions with correct and incorrect options marked but no explanation attached. One button generates explanations for every option, and the review step follows immediately.

The reviewer can edit any explanation in place, regenerate a single one, or export the set for review elsewhere. The team can also append an extra system instruction for a specific generation, such as asking for a shorter explanation, and regenerate with that rule applied. Santiago demonstrated this by asking for a maximum of three lines and receiving a two-line result.

A quiz in the admin can also be authored manually: a person on the education team can write an explanation by hand instead of generating one. The generation button is the fast path, not the only path, and human control stays on the publishing side either way.

The knowledge that explanations exist and who approves them is a useful edge for anyone building something similar. The human review is not a formality; it is the reason the feature can ship at all, because someone has to confirm the generated text matches the specific option a student selected.

The education team also owns the separate tool that drafts the quiz questions themselves from lesson transcripts, so question writing and explanation writing are two systems rather than one. Santiago said the question-generation project was built by the education team and integrates with the explanation service as a distinct component. Thiago described what that replaced: a year earlier, a developer would watch the whole recorded lesson, take notes, and hand-build the questions from the video. Transcripts plus system prompts cut that to a fraction, though the older manual route still exists for cases that need it.

Batching Test: 50 Explanations at Once Versus Five at a Time

The team tested two shapes before writing production code. A quiz with ten questions and five options each needs 50 explanations, so the first test sent all 50 in a single request to a stronger model, and the second sent five at a time per question using a simpler model.

The single-batch run did not fail on cost. Quality was the problem: some outputs were not explanatory enough, and some stopped partway through, which the team read as the model giving up on a long task.

Sending five at a time produced noticeably better explanations and fewer failures. In the smaller-batch test the team struggled to find an explanation that was cut off or that ignored the instructions it had been given.

Cost stayed roughly comparable between the two approaches, according to Santiago. Many small calls cost less per call but add up to about the same total, so the decision was driven by output quality rather than by price.

The practical rule that came out of the test is to match task granularity to model capability. A small model asked to produce 50 short explanations in one pass has to hold a large task in view; the same model asked for five has a much smaller job to finish.

The team ran all of this as a proof of concept first, in a small throwaway project built quickly with AI, before wiring anything into the real platform. The explanation service was integrated afterward, which let the team validate cost and output quality before committing engineering time to the full implementation.

Cost Controls, Observability, and What the Team Watches

Spend control in this project comes from where generation runs, not from a hard spending cap. Explanations are produced in the admin area during quiz authoring, so the volume of calls tracks how much content the education team publishes rather than how many students are studying.

OpenRouter's dashboard gives the team per-model and monthly visibility into what was spent. Santiago said a separate internal project is adding broader AI observability, including token usage and the option to drop to a smaller model when cost rises, because that future project will generate content on demand. Those future projects will need the same validation step the quiz feature has, since the team does not trust model output without a check, and they will also need a cost view because nobody wants to find out after the fact that a project was burning through the budget.

Model selection is treated as a two-step decision. Planning work goes to a more capable model, while execution against an already detailed plan can go to a cheaper one. Santiago mentioned Claude models for planning and execution, GPT models through the Codex path, and Kimi K2.5 as a low-cost option that the team found useful for both planning and execution.

Routing Each Step to the Right Model

The comparison below reflects what Santiago described about how the Rocketseat platform team assigns work to models and tools. It covers roles in their workflow, not a scored evaluation.

Role in the workflow	Tool or model family	Reported reason
Agent orchestration	OpenCode	Plan first, then execute; team uses it for feature planning and bug fixing
Planning complex work	Claude models	Santiago said they are strong for planning
Execution of a finished plan	GPT Codex models, Kimi K2.5	Lower cost once the plan is detailed
Generating quiz explanations	Routed through OpenRouter	One API for many models plus a spend dashboard
Editor	Cursor	Autocomplete only; the team does not rely on its built-in chat

Santiago framed planning as the part where model choice matters most, and execution as the part where a cheaper model is usually sufficient because the plan already carries the reasoning. He did not draw a hard line between the model families: he noted that some models perform similarly and that the Claude models stood out mainly for planning work.

The team does not lock itself to one tool. Santiago has a personal ChatGPT subscription and uses it for brainstorming and rough pre-plans to avoid burning the monthly usage limit that the OpenCode seat comes with, then takes the refined idea into OpenCode to produce the real plan. He also mentioned testing other editors, such as Antigravity, and other developers on the team use Claude Code and Cursor.

The People and Process Behind the Team

The platform work described here is small by team standards. Santiago is a mid-level backend developer with more than six years of experience, split earlier across frontend, DevOps, and blockchain work that involved building tokens and NFTs. He has a degree in computer science and a master's in distributed systems, where he also studied machine learning, and he joined Rocketseat about a year ago after being a student and community ambassador since 2018.

The platform team has roughly three backend developers plus a frontend developer, and it reports to a product manager, with a product designer owning the design work. That small team maintains a catalog of 23 learning tracks and close to 100 active courses partly because of how much of the routine work is handed to AI tools.

A backend developer on this team reads the Figma designs during refinement. Santiago said the layout determines which backend routes have to exist, including cases where a route needs to be created or modified, so backend work depends on what the design shows happening on screen.

Santiago's view of the wider AI shift is that the job moves from typing the code to making the architectural decisions, with the model handling the mechanical parts. He described starting out skeptical of AI in 2022, when ChatGPT's first version launched, worrying it would make him complacent about writing applications from scratch, and ending up on the opposite conclusion.

Why the Planning Step Comes Before the Code

Rocketseat's own workflow asks the agent to write a plan as a Markdown file before any code is generated, then has a human read that plan. Santiago described reviewing the plan and correcting it when the agent proposed rebuilding something that already existed, or when it skipped a component the codebase already used. The fix is to edit the plan and ask for it again before any code is written.

Long plans are split into phases for the same reason the explanation generation was split into five-option calls. If a plan is too large for one execution pass, the agent can lose track of it, so the team asks for phases and runs them one at a time, checking each result before starting the next.

Context is scoped rather than dumped. For a page that shares nothing with the rest of the system, Santiago said the team passes the old page as an example of the pattern to follow instead of the whole codebase. The size of the context is a decision, not a default, and too much context can be as unhelpful as too little.

None of this removes the human from the loop. The plan is validated, the execution is watched, and the output is checked, which is the same control pattern the quiz explanation feature uses at the content level.

Why the Time Saving Matters More Than the API Bill

Thiago framed the project as the kind of AI use that does produce a return: work that used to take a person 40 to 50 minutes can be cut by 90% to 100% of that time on the generation side, because the model writes the first draft and a human only reviews it.

The direct API cost is small next to that saved time, and the cost is stable because generation happens per quiz rather than per student. If you are trying to decide whether a similar feature is worth building, the comparison that matters is the human hours per quiz against the model spend per quiz, not the model spend against zero.

For general background on how these tools are used in production teams, see the Rocketseat source video and the Rocketseat platform, and compare it with published guidance on structured output and tool calling from OpenAI. If you want to build the same stack yourself, Crazystack Typescript is a Brazilian TypeScript-focused course program worth looking at, and the broader Crazystack catalog covers the tooling around it.

Frequently Asked Questions

Are the quiz explanations generated at the moment a student answers?

No. They are generated in the admin area when a quiz is authored, reviewed by the education team, and stored in the database. The student request only reads the stored row, so no model call happens while the student waits.

Why does Rocketseat use OpenRouter instead of calling a provider directly?

OpenRouter exposes many models behind one API and provides a dashboard showing spend per model and per month. Santiago said the team also tested direct provider clients through the Vercel SDK, and chose routing for the flexibility and the cost visibility.

What stops the model from returning malformed output?

The service requests JSON and validates the response against a Zod schema defined in TypeScript. If the structure does not match, the output fails validation before it can be saved, so an unexpected shape never reaches a student.

Is human review required for every explanation?

Yes, every explanation is approved before the quiz is published, and reviewers can edit or regenerate any individual explanation. Santiago said the team cannot fully trust a language model's output for graded content, so approval stays manual. The education team can also write an explanation by hand if it prefers.

Did generating 50 explanations in one call cost more than generating five at a time?

Cost was roughly comparable, according to Santiago. The deciding factor was quality: the single large batch produced weaker explanations and occasional incomplete answers, while smaller batches produced better results.

Which model do they use for the explanations?

The choice is deliberately flexible because the call goes through OpenRouter. Santiago said the team tested stronger and cheaper models during the proof of concept and weighs cost against output quality for each type of task.

What did the team learn about passing a large plan to a coding agent?

Large plans can cause the agent to lose track of the work, so the team asks for the plan to be divided into phases, then executes one phase at a time and checks the result before moving on.

Does a backend developer at Rocketseat look at design files?

Yes. Santiago said he reviews the Figma designs during refinement sessions because the layout determines which backend routes have to exist, including cases where a route needs to be created or modified.

Was the quiz experience itself changed at the same time?

Yes. The platform moved the quiz out of a small modal into a dedicated page so students could focus, and the explanation panel sits alongside the question as they work through it.

Do they use AI for anything other than explanations?

Yes. Santiago said the team uses OpenCode for planning and executing everyday feature work and bug fixes, and AI is used to build small internal tools and quick proof-of-concept projects before real implementation. The question-generation system for quizzes was built by the education team with AI assistance and runs as a separate service.

Turning this setup into your own content

The interesting move in this case is not the model choice, it is the decision to generate once and review once, so that the expensive, non-deterministic part never runs inside a student's request. If you have the same kind of hard-won detail sitting in a YouTube video, whether it is an architecture walkthrough, an interview, or a lesson you recorded, that material can become a written article instead of staying buried in the video. Skala Blog takes a YouTube link, transcribes the video, and generates the article from what was said.