Generating Diagnostic Assessments for STEM With AI
A STEM diagnostic assessment is a pre-unit, ungraded probe, given before a single lesson on a topic this year, that surfaces a student's existing physical or design intuitions before instruction has to work against them unknowingly. Its job isn't checking whether today's teaching landed. It's finding out what a student already believes about forces, ratios, or design trade-offs, before a class has heard a word of formal instruction.
Quick Answer: STEM already has a validated, published diagnostic instrument for one half of its territory, the Force Concept Inventory (Hestenes, Wells, & Swackhamer, 1992), but full administration is out of reach for a single classroom's weekly use. The realistic pre-unit tool is a short, open-ended, ungraded probe modeled on Angelo and Cross's Background Knowledge Probe (1993), aimed separately at STEM's two defining item families: quantitative compute intuitions and engineering-design trade-off intuitions.
This guide covers:
- Why a STEM diagnostic's "before any instruction this year" timing differs from a mid-lesson STEM formative check, not an earlier version of one
- Why STEM already has a validated diagnostic instrument, and why its scope still leaves half of STEM's diagnostic territory uncovered
- Fresh, worked pre-unit prompts for two documented beliefs, one about density, one about structural design, covering both item families the STEM quiz guide treats as genuinely integrated
- A step-by-step AI workflow, a tools comparison, pitfalls, and FAQ
This guide sits alongside How to Create Spanish Quizzes With AI in this site's subject-specific series, where each entry looks for the format's different design problem in that subject.
Why "Before Any Instruction This Year" Is a Different Design Problem
A STEM diagnostic and a STEM formative check fail for different reasons: they sit on opposite sides of instruction, not different points on one timeline. The formative-assessment guide for this subject found a compressed, 30-90-second in-lesson prompt only survives for one of STEM's two integrated item types, and only in a narrowed form. A diagnostic doesn't inherit that constraint, since it isn't racing a clock inside a live lesson.
What a Formative Check Gets to Assume
A mid-lesson formative check can assume a concept was just introduced this period. The STEM formative guide found a compute item survives compression only when the correct direction contradicts a student's default "more of A, more of B" reasoning, and an engineering trade-off item barely survives at all, since stripping its numbers removes the only thing that makes it decidable.
What a Diagnostic Can't Assume
A diagnostic given before a unit's first lesson has no current lesson, and no clock, to work against. It doesn't need to pass a removal test built for a graded item, because it isn't measuring performance yet. It's measuring what belief was already sitting there, a prior grade, a documentary, a dinner-table conversation, or nothing at all.
| Design Question | Formative Check (mid-instruction) | Diagnostic Assessment (pre-instruction) |
|---|---|---|
| Timing | During the current lesson, after a concept has already been introduced | Before the unit's first lesson this year |
| What it measures | Whether today's teaching on one compute or design concept is landing | What a student already believes, across both compute and design intuitions |
| Best prompt shape | Concept-gated directional prediction, or a narrow trade-off prerequisite check | Open self-report on an existing belief, no clock, no removal test |
| STEM example | "Is the force the truck exerts on the car bigger, smaller, or the same?" (mid-lesson) | "What do you believe determines whether something floats or sinks?" (pre-unit) |
| What a wrong response means | A gap in today's specific instruction | A belief the whole unit needs to actively target |
A diagnostic that copies a formative check's narrow, single-concept framing throws away the one thing it exists to gather: an honest, broad read on beliefs nobody has taught against yet this year.
Why the Distinction Isn't Just Semantics
The National Research Council's framework for K-12 science and engineering education makes the case directly: instruction that doesn't identify and engage a student's prior conceptions often fails to change them, however clearly the new material is presented (National Research Council, 2012). Its earlier synthesis of learning-science research reaches the same conclusion: preconceptions left unconfronted often persist underneath new material (National Research Council, 2000). A forces unit taught generically, without knowing which beliefs a class holds, risks exactly that outcome.
STEM Already Has a Validated Instrument — Here's Why It Isn't the Full Answer
STEM, like biology, already has a real, published, validated diagnostic tool for part of its territory, and being honest about that instrument's scope is the right starting point. The physics quiz guide already builds classroom distractors from the same research base.
The Force Concept Inventory
Physicists David Hestenes, Malcolm Wells, and Gregg Swackhamer built the Force Concept Inventory (FCI), a 30-item multiple-choice instrument whose wrong-answer options are drawn from documented misconceptions about force and motion, not invented guesswork (Hestenes, Wells, & Swackhamer, 1992). Its item pool traces to years of interview-based research into how students reason about mechanics before formal instruction (Halloun & Hestenes, 1985).
- FCI's distractors work as diagnosis, not just wrong answers, because each maps to a specific, previously catalogued alternate conception about force, motion, or trajectories.
- A student who picks a wrong option is revealing which non-Newtonian model they're reasoning from, not guessing at random.
- Unlike a teacher-built probe, FCI went through iterative interview development and large-scale statistical validation before publication (Halloun & Hestenes, 1985).
Where the Precedent Doesn't Fully Transfer
That validation process is exactly what a single teacher can't replicate by next Monday, and FCI's scope is narrower than STEM's full diagnostic territory besides. FCI is normed for a full mechanics unit's pre/post comparison, not a quick temperature check before a Tuesday lesson, and hand-scoring 30 items before a unit starts is a heavier lift than most pacing calendars allow.
More importantly, FCI covers only one of the two item families that make STEM assessment distinct. It catalogs force-and-motion misconceptions, the compute-intuition side of the hub's defining item types, but it predates NGSS's engineering-design strand and says nothing about design trade-off reasoning (NGSS Lead States, 2013). What a teacher-built diagnostic can honestly borrow from FCI isn't the instrument, it's the principle: anchor every prompt to a misconception already documented by real research, rather than guessing at a plausible wrong belief.
The Background Knowledge Probe: A Lighter, Real Precedent
Classroom-assessment researchers Thomas Angelo and Patricia Cross documented a technique built specifically for this moment: the Background Knowledge Probe, a short, open-ended, ungraded prompt given before a unit to surface what students already know or believe (Angelo & Cross, 1993). It needs no pre-built answer key, no validation study, and no restriction to physics, just a well-aimed open question.
| Force Concept Inventory (Hestenes, Wells, & Swackhamer, 1992) | Background Knowledge Probe (Angelo & Cross, 1993) | |
|---|---|---|
| Format | 30-item validated multiple-choice | Short, open-ended, self-report |
| Domain covered | Force and motion only | Any documented belief, compute or design |
| Development cost | Years of interviews and statistical validation | A teacher writes it in minutes |
| Best use | A full mechanics unit's formal pre/post comparison | A single class's pre-unit temperature check |
| Realistic for next Monday? | Rarely, at full scale | Yes |
| Shared principle | Anchor every item to a documented belief, by name | Same |
Two STEM Beliefs Worth Surfacing Before a Single Lesson
A pre-unit STEM diagnostic has to sample both of the hub's genuinely integrated item families, not just the one FCI covers, since a class can hold naive compute intuitions and naive design intuitions independently. The STEM quiz guide's removal test defines those two families for a graded item; a diagnostic targets the raw belief underneath each, before instruction narrows toward one target.
Does Something Heavier Always Sink?
Cognitive-development researchers Carol Smith, Susan Carey, and Mary Wiser documented that children, and many adolescents, don't spontaneously differentiate weight from density before formal instruction forces the distinction (Smith, Carey, & Wiser, 1985). Left unprobed, this shows up as a mass-only heuristic: heavier things sink, lighter things float, with no room yet for the mass-to-volume ratio that actually governs buoyancy.
A pre-unit probe needs no data table or formula, just an open question broad enough to let the mass-only heuristic surface unprompted:
"Before we start this unit, write down what you already believe: if Object A weighs more than Object B, will A always sink while B floats? Explain your thinking, using any example you want, even a made-up one."
A student who writes "yes, because the heavier one has more weight pulling it down" has revealed the mass-only heuristic without a single distractor being built. A large steel cargo ship floating while a small steel bolt sinks is a ready-made case for the unit's first lesson, once the diagnostic confirms how widespread the belief is.
Is a Wider Bridge Deck Always Stronger?
Engineering-education research documents a parallel heuristic on the design side of STEM: novice designers tend to equate more material, or a bigger structure, with a stronger one, without weighing the trade-offs a real design has to satisfy. The Framework underlying NGSS names this gap directly: most students arrive with a model of engineering as building or tinkering, rather than a systematic weighing of criteria and constraints (National Research Council, 2012).
Christine Cunningham and William Carlsen's review of precollege engineering-education research describes the same default. Absent instruction, a "bigger or heavier equals stronger" heuristic tends to substitute for genuine reasoning about cost, weight, and load together (Cunningham & Carlsen, 2014).
"Before we design anything, what do you believe: if two bridges are built from the exact same material, is the one with a wider deck always the stronger bridge? Explain your thinking."
A student who answers "yes, wider is always stronger" is applying the heuristic Cunningham and Carlsen describe. A student who answers "it depends, because a wider deck also weighs more and needs more support underneath it" is already reasoning about trade-offs the unit hasn't formally taught, exactly the student a diagnostic exists to identify early.
| Documented Belief | Named Source | Pre-Unit Prompt Shape |
|---|---|---|
| Weight alone determines whether something sinks or floats (mass-only heuristic) | Smith, Carey, & Wiser (1985) | "Will a heavier object always sink while a lighter one floats?" |
| More material or a bigger structure is always the stronger design | National Research Council (2012); Cunningham & Carlsen (2014) | "Is a wider bridge deck always the stronger bridge?" |
A Grade-Calibrated Illustration
Say you're about to open a grade 8 unit blending density and structural design, a common pairing in a physical-science or STEM elective. Before a single lesson, you could hand out both open prompts above on one half-sheet, unscored, with instructions that "I don't know yet" is a completely acceptable answer.
- Reading responses that evening beats building a full pre-test with an answer key for either concept.
- It shows which heuristic, mass-only reasoning or bigger-equals-stronger, is more widespread in this class.
- It flags students who already reason about ratios or trade-offs correctly, so the opening days don't reteach what they understand.
A grade 3 or 4 class opening a simpler floating-and-sinking unit can run a lighter version: "Do you think a heavier toy boat will always sink faster than a lighter one?" Younger students won't name density, but their answer still reveals which model the class defaults to.
A Step-by-Step AI Workflow for Building a STEM Diagnostic
Building an honest pre-unit STEM diagnostic starts with naming the specific belief you're probing, in one of the two families, not the general unit topic, since a vague prompt collects vague, ungradeable answers.
- Name the specific belief, not the unit topic. "Buoyancy" isn't specific enough; "believes weight alone determines sinking" is.
- Identify which family the belief belongs to, a compute intuition like density, or a design-trade-off intuition like structural strength, so one diagnostic samples both instead of defaulting to whichever a teacher already has research for.
- Check whether the belief is already documented, in research like Smith, Carey, & Wiser (1985) or Cunningham & Carlsen (2014), rather than guessing at what might trip students up.
- Write each prompt as an open self-report question and draft several at once, one per targeted belief across both families, so the diagnostic samples broadly instead of narrowing to one item type.
- Set your class profile's grade level, so the wording matches what students can express, even when the belief is more sophisticated than their vocabulary.
- Anchor domain coverage to your state's science standard or the relevant NGSS core ideas (NGSS Lead States, 2013), so the diagnostic samples the unit's real scope.
- Sort responses into three buckets before planning instruction: reasons correctly already, names the documented heuristic, or unsure, and let that sort, not a score, shape the first week's lesson plan.
A Sample Prompt
Grade 8 STEM elective, opening a unit on density/buoyancy then
structural design. Generate four short, open-ended, ungraded
pre-unit prompts, two per belief:
(1) weight alone determines whether an object sinks or floats
(2) a wider or bigger structure is always the stronger one.
One or two sentences each, answerable in under three minutes,
explicitly ungraded, with permission to answer "I don't know."
You could use EduGenius to hold a class profile's grade level while drafting open pre-unit prompts anchored to named STEM misconceptions across both families, then export the finished probe as a one-page handout or a slide.
Tools for Building STEM Diagnostics
A tool for this job needs open-ended prompt generation across both compute and design intuitions, not just a scored answer key, since a diagnostic's value comes from the belief a student reveals, not a grade.
| Tool | Open-Ended Prompt Support | Anchors to Named Misconceptions | Covers Both Item Families | Starting Price |
|---|---|---|---|---|
| EduGenius | Yes, via open-response format selection | Yes, if the misconception is named in the prompt | Yes, if both are requested explicitly | 25 welcome credits free; Starter $7.99/month (500 credits); Professional $15.99/month (1,000 credits) |
| General-purpose AI chat tools | Yes, if explicitly requested | Only if the source is supplied in the prompt | Only if both are requested | Varies by provider |
| Published concept inventories (e.g., FCI) | No, fixed instrument | Yes, at full psychometric rigor | No, force and motion only | Free to administer; scoring guide required |
| Paper Background Knowledge Probe | Yes, inherently | Only as well as the teacher designs it | Yes, if the teacher writes both | No cost |
No tool replaces reading the raw responses. The general diagnostic-speed guide covers a separate bottleneck, verifying what a prior teacher actually taught, in any subject. The STEM formative-check guide covers a related but distinct job: catching whether today's teaching is landing after instruction has started, not surfacing a belief before it.
Pro Tips for STEM Diagnostics
- Ask AI to name which documented belief each prompt targets, the same way the STEM quiz guide asks for a named, sourced dependency behind an item. An untraceable prompt is a guess, not a diagnostic.
- Deliberately sample both families in one sitting. A class strong on density reasoning can still default to a bigger-equals-stronger design heuristic, and you won't know which until you ask both.
- Keep every prompt genuinely ungraded, and say so out loud. A student who suspects a "what do you believe" question is secretly scored answers what they think you want, not what they think.
- Reuse the same probe next year, and compare which beliefs recur. A heuristic that shows up every year earns a dedicated first lesson.
- Pair a pre-unit probe with a concrete real case, like a floating steel ship or a bridge that failed for a documented structural reason, so day one answers the belief the diagnostic surfaced.
What to Avoid
Scoring a Diagnostic Like a Quiz
Grading a "what do you believe" response defeats its purpose and teaches students to hide their real thinking next time. Read it as a starting point for instruction, never a grade in the book.
Writing a Vague Prompt With No Named Belief Behind It
"What do you know about forces?" collects scattered, hard-to-sort answers. "Will a heavier object always exert more force than a lighter one pushes back?" targets one misconception directly and sorts cleanly.
Copying the Formative Check's Compressed, Single-Concept Format
The STEM formative-check guide found a fast in-lesson prompt barely survives compression for a trade-off item, and only partly for a compute item. A diagnostic doesn't need to fit that clock, and copying its narrowed shape throws away the breadth a pre-unit check exists to gather.
Treating an AI-Drafted Probe as a Validated Instrument
An AI-drafted open prompt anchored to a real, named misconception is a reasonable classroom tool. It is not the Force Concept Inventory, and presenting it as psychometrically equivalent overstates what a quick, honest check can claim (Hestenes, Wells, & Swackhamer, 1992).
Key Takeaways
- A diagnostic happens before instruction; a formative check happens during it. The formative guide checks whether today's teaching is landing; a diagnostic checks what a student already believes before any teaching starts this year.
- STEM already has a validated diagnostic instrument for force and motion, the 30-item Force Concept Inventory (Hestenes, Wells, & Swackhamer, 1992), but full administration is out of reach for a Monday-morning check.
- FCI covers only one of STEM's two defining item families. It says nothing about engineering-design trade-off intuitions, which need their own documented anchor (National Research Council, 2012; Cunningham & Carlsen, 2014).
- The Background Knowledge Probe is the realistic model for an AI-assisted pre-unit diagnostic: lightweight, open-ended, ungraded, and not restricted to one domain (Angelo & Cross, 1993).
- Two documented beliefs are worth surfacing before a STEM unit opens: a mass-only heuristic for sinking and floating (Smith, Carey, & Wiser, 1985) and a bigger-equals-stronger heuristic for structural design (Cunningham & Carlsen, 2014).
- A diagnostic should sample broadly across both item families, the opposite move from what the formative-check guide found works mid-lesson.
- AI speeds up drafting misconception-anchored prompts; a human still reads the responses and decides which belief is widespread enough to shape the first week of instruction.
Frequently Asked Questions
What's the difference between a diagnostic assessment and a formative assessment in STEM?
A diagnostic happens before a unit starts and surfaces what a student already believes, like assuming a heavier object always sinks. A formative check happens during instruction and checks whether that day's teaching is landing in real time, a different question with a much tighter time budget.
Is there already a validated diagnostic test for STEM misconceptions?
Yes, for force and motion: the Force Concept Inventory is a 30-item, psychometrically validated multiple-choice instrument (Hestenes, Wells, & Swackhamer, 1992). It doesn't cover engineering-design trade-off reasoning, and its scoring burden makes it impractical as a routine pre-unit check even within its own domain.
Should a STEM diagnostic be multiple-choice or open-ended?
Open-ended usually works better for a classroom-level pre-unit diagnostic. A validated instrument like the Force Concept Inventory needs years of research to build properly; a short, open self-report prompt surfaces the same kind of belief across compute and design territory without that investment (Angelo & Cross, 1993).
How is this different from the general diagnostic-assessment speed guide on this site?
That guide solves a different bottleneck: verifying what a prior teacher actually taught, before writing a prerequisite item. This guide covers which STEM beliefs, compute intuitions and design intuitions alike, are worth surfacing before instruction.
Sources
- Angelo, T. A., & Cross, K. P. (1993). Classroom Assessment Techniques: A Handbook for College Teachers (2nd ed.). Jossey-Bass.
- Cunningham, C. M., & Carlsen, W. S. (2014). Precollege engineering education. In N. G. Lederman & S. K. Abell (Eds.), Handbook of Research on Science Education, Volume II. Routledge.
- Halloun, I. A., & Hestenes, D. (1985). The initial knowledge state of college physics students. American Journal of Physics, 53(11), 1043-1055.
- Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. The Physics Teacher, 30(3), 141-158.
- National Research Council. (2000). How People Learn: Brain, Mind, Experience, and School (Expanded ed.). National Academies Press.
- National Research Council. (2012). A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. National Academies Press.
- NGSS Lead States. (2013). Next Generation Science Standards: For States, By States. National Academies Press.
- Smith, C., Carey, S., & Wiser, M. (1985). On differentiation: A case study of the development of the concepts of size, weight, and density. Cognition, 21(3), 177-237.