Generating Diagnostic Assessments for Computer Science With AI
No validated, misconception-derived concept inventory exists for computer science's four non-code CSTA strands — Computing Systems, Networks & the Internet, Data & Analysis, and Impacts of Computing — at any grade band, not just K-9. That's a genuinely different finding from every other subject this diagnostic-assessment series has covered, and it's worth stating precisely instead of rounding it to the nearest prior answer.
Quick Answer: This series has found a clean yes (biology), a clean no (music), an access failure (coding's FCS1/SCS1), and a construct mismatch (ESL's WIDA Screener). CS's non-code strands are a fifth shape: a genuine, thin-but-real, and fast-growing body of misconception research exists — internet "mental models" studies, AI-misconception interview studies — but almost none of it has crystallized into a validated instrument yet. The one K-12-validated CS concept inventory that does exist targets the strand this guide excludes. What functions as a diagnostic here is a teacher-built probe, scoped to one of the newer, thinner citations below.
This guide covers:
- Why this question needed real research, not a reflex "yes" or "no" borrowed from a sibling entry
- The honest, five-shape answer, checked against Bebras, the Middle Grades CS Concept Inventory, and a brand-new 2024 AI literacy instrument
- What actually exists strand by strand — and why Impacts of Computing has more going for it than the other three
- A step-by-step AI workflow for building a diagnostic from a thinner citation base, a tools comparison, and the mistakes that overstate what this research supports
This entry joins four earlier ones in the series: the biology diagnostic guide, where a true CINS-equivalent transfers straight into a classroom; the music diagnostic guide, where no comparable instrument exists at all; the coding diagnostic guide, where a real instrument exists but fails a K-9 teacher on access, grade band, and format; and the ESL diagnostic guide, where a reachable K-12 instrument exists but measures the wrong construct. None of those four shapes fits cleanly here.
What a CS Diagnostic Is Actually Asking
A pre-unit computer science diagnostic asks what a student already believes about a non-code CSTA strand before this year's first lesson on it — a different question from a scored quiz item or a mid-lesson formative check. The origin CS quiz guide established that four of the five CSTA strands (CSTA, 2017) don't touch a code editor at all. The CS formative guide established how those same four strands compress under a five-minute clock. A diagnostic runs before either check has anything taught yet to measure against.
Three CS Instruments, Three Different Jobs
| Instrument | Timing | Core Question | Typical Format |
|---|---|---|---|
| Scored quiz item | After a concept is taught | Can the student now apply it? | Selected-response, self-grading |
| Formative check | Mid-lesson | Is today's lesson landing right now? | Fast, single-answer |
| Pre-unit diagnostic | Before the unit's first lesson this year | What does the student already believe? | Open, ungraded, teacher-read |
A wrong answer on a diagnostic isn't a gap in fresh teaching — there's no fresh teaching yet. It's a pre-existing belief, sometimes built from a game, a sibling, or a prior year's class, and it's exactly the kind of signal the coding diagnostic guide's pre-unit aliasing probe was built to surface for a different strand entirely.
Why the CINS/FCI Question Still Needs Asking Here
A validated concept inventory changes what a diagnostic can responsibly claim. If one exists and reaches a K-9 classroom, a diagnostic can borrow validated items instead of guessing at what's worth probing. That's the exact question this series has now asked of five subjects, and CS's non-code strands are the first to answer it with something other than a version of a prior answer.
Does Computer Science Have a CINS/FCI-Equivalent Diagnostic? A Fifth Shape
Checking CS's non-code strands against the four prior shapes in this series — and against Bebras, the field's best-known K-12 informatics assessment — turns up something none of them predicted: real misconception research exists, but it hasn't consolidated into an instrument the way biology's or even coding's has. That's not the same failure mode as any earlier entry.
Where the Four Prior Shapes Don't Fit
- Not biology's clean yes. No instrument here transfers straight into a classroom the way the Conceptual Inventory of Natural Selection does (Anderson, Fisher, & Norman, 2002).
- Not music's clean no. Unlike notation and theory, real, citable, growing misconception research exists for at least two of these four strands — it just isn't validated as a scored instrument yet.
- Not coding's access failure. Coding's FCS1/SCS1 are real instruments a K-9 teacher can't reach. For three of CS's four non-code strands, there's no comparable instrument to be locked out of anywhere, at any grade band.
- Not ESL's construct mismatch. WIDA Screener passes every practical bar and measures the wrong thing (WIDA, 2025). CS's near-miss instrument, covered next, is both narrower than that and barely a year old.
Bebras: Broad, K-12, and Still Not a Diagnostic
Bebras is the closest thing to a strand-spanning, K-12, internationally used informatics assessment, run annually for students roughly age six through eighteen. It looks, on the surface, like it could be the instrument this question is hunting for. It isn't, for two separable reasons.
Bebras items were never built from mined student wrong answers the way CINS's or FCS1's distractors were — they're written by task-setters for engagement and difficulty tiering, not derived from documented misconceptions. And its own construct validity is disputed: researchers testing Bebras scores against programming-course grades found only a moderate correlation and concluded it's "not very likely that CT measures can be derived from the Bebras test as it is currently designed" (Araujo, Santos, Andrade, Guerrero, & Dagienė, 2017).
Bebras is a genuinely valuable enrichment and competition tool. It was never validated as a misconception-diagnostic instrument, and treating a strong or weak Bebras score as evidence of a specific belief overstates what the test was built to do.
The One K-12-Validated CS Concept Inventory Targets the Excluded Strand
Here's the finding that sharpens the whole picture. A real, validated, K-12-appropriate CS concept inventory does exist: the Middle Grades Computer Science Concept Inventory (MG-CSCI), a 24-item instrument validated on 457 students ages 11-13 using both classical test theory and item response theory (Rachmatullah, Akram, Boulden, Mott, Boyer, Lester, & Wiebe, 2020).
It measures variables, loops, conditionals, and algorithms — Algorithms & Programming, the one strand this guide and the origin CS hub deliberately exclude, already covered by the coding diagnostic guide.
A 2014 review of CS concept inventories found the field's early instrument-building work concentrated almost entirely on programming (Taylor, Zingaro, Porter, Webb, Lee, & Clancy, 2014), and a 2023 systematic follow-up counting 65 CI-related papers found that concentration still holding, even as the overall number of CS concept inventories grows (Ali, Ghosh, Rao, Dhegaskar, Jawort, Medler, Shi, & Dasgupta, 2023).
| Candidate | K-12 Validated? | Misconception-Derived Distractors? | Covers a Non-Code Strand? | Verdict |
|---|---|---|---|---|
| MG-CSCI (Rachmatullah et al., 2020) | Yes, ages 11-13 | Yes, IRT-validated | No — Algorithms & Programming | Real instrument, wrong strand for this guide |
| Bebras | Yes, ages ~6-18 | No — task-setter written | Partially, across mixed content | Enrichment tool, not a diagnostic |
| FCS1/SCS1 (Tew & Guzdial, 2010; Parker et al., 2016) | No — university CS1 | Yes | No — Algorithms & Programming | Already ruled out by the coding guide |
| AI Literacy Concept Inventory (Zhang, Perry, & Lee, 2024) | Yes, middle school | Partially — expert-panel + interviews | Adjacent to Impacts of Computing only | Closest near-miss; covered below |
That table's honest upshot: every K-12-validated instrument this research turned up either targets the wrong strand or covers only a narrow slice of one strand. Nothing validated exists yet for Computing Systems, Networks & the Internet, or Data & Analysis specifically.
What Actually Exists, Strand by Strand
Since no single instrument answers this question, the honest move is the same one the ESL diagnostic guide made for interlanguage patterns: check what real, citable research exists per strand, even where it falls short of a validated test. The four strands turn out not to be equally thin.
Computing Systems: Real Research, but Not at This Grade Band
Documented misconceptions about how computers actually work do exist in the literature. Researchers Pamplona, Medinilla, and Flores (2013) catalogued specific student misconceptions about concurrency, interrupts, and context switches in an operating-systems course. That research targets university-level online coursework, not a K-9 classroom, leaving this strand with real citations that don't yet reach down to grade band.
Networks & the Internet: The Strongest Non-Code Research Base
This strand has the richest, most K-9-relevant citation trail of the four. Researchers Diethelm, Wilken, and Zumbrägel (2012) documented specific, recurring secondary-student misconceptions about how the internet moves data. A 2023 systematic review synthesizing 27 studies and over 2,200 children ages 3 to 15 found consistent, age-linked patterns — younger children describing the internet as living inside a single central computer or "in the satellites" (Babari, Hielscher, Edelsbrunner, & Conti, 2023).
That review's age range, 3 to 15, is the closest match to a K-9 classroom that this entire research question has turned up anywhere in the series. It's still a synthesis of qualitative and mixed-methods studies, not a validated scored instrument — real evidence a teacher can build from, not an off-the-shelf test to hand out.
Data & Analysis: Assessed at the Teacher Level, Not the Student Level
Search this strand for a validated K-12 instrument and the closest hit is aimed at the wrong population. The Educators Data Literacy Self-Assessment (EDLSA) is a validated questionnaire measuring teachers' own perceived data literacy, not students' (Donate-Beby, García-Peñalvo, & Amo-Filva, 2024). Separate research has flagged that "data literacy" itself stays conceptually ambiguous across K-12 studies, with real disagreement over whether it means chart-reading or something broader — a definitional gap this strand shares with computational thinking, noted in the origin CS hub.
Impacts of Computing: The Youngest but Most Active Line of Research
This strand has the most recent activity of the four, almost all of it from the last two years. Researchers Mertala and Fagerlund (2024) documented three specific, named AI misconceptions in a qualitative study of 195 Finnish 5th and 6th graders: treating AI as a form of human cognition, as an anthropomorphic entity, or as a machine with pre-installed knowledge rather than trained knowledge.
- Non-technological AI: believing AI describes a human thought process, not a machine.
- Anthropomorphic AI: describing AI as a human-like being with its own will.
- Pre-installed AI: assuming an AI system's "knowledge" was built in, not learned from data.
That third pattern connects directly to algorithmic bias: a student who doesn't yet grasp that a system's outputs come from its training data has no foothold for understanding why biased training data produces a biased result. A separate interview study of secondary students found parallel, documented machine-learning misconceptions at an older grade band (Marx, Witt, & Leonhardt, 2024).
A 2025 international working group reported they are still in the item-development phase of building a validated AI concept inventory for non-experts, explicitly citing "hidden human labor," bias, and stereotyping as content areas needing coverage (Mannila, Henry, Bahr, Chytas, Connamacher, Müller, Opel, & Scholl, 2025). As of this writing, that instrument doesn't exist yet — it's being built in real time, which is itself the honest state of this strand.
A Step-by-Step AI Workflow for a Thinner-Evidence CS Diagnostic
Building a defensible non-code CS diagnostic starts by treating "documented" as a stricter bar than usual, since this strand's citation pool is newer and thinner than any other subject in this series.
- Pick the strand, then check what tier of evidence it has. Networks & the Internet and Impacts of Computing have real citable studies; Computing Systems and Data & Analysis mostly don't reach K-9 yet.
- Name the exact belief, not the topic, the same discipline the coding guide and the ESL guide both apply: "believes the internet lives inside one central computer," not "internet basics."
- Ask AI to cite which study a claimed pattern traces to, and verify it yourself — this strand has more invented-sounding-but-real findings, and more genuinely invented ones, than most subjects in this series.
- Choose an open, ungraded prompt format, never multiple-choice, since no validated selected-response item bank exists for these strands to draw from.
- Set your class profile's grade band, since the internet mental-model research spans ages 3-15 with different expected framings at each end.
- Sort responses into buckets, not scores: reasons correctly already, shows a documented pattern, or unsure — never averaged into a grade.
- Route an Impacts of Computing finding into how you frame the first lesson, not into a judgment about a student's character or intent.
You could use EduGenius to hold a class profile's grade level while drafting a small batch of open, belief-targeted prompts for a named strand, each labeled with the study it traces to, then export the finished probe as an ungraded one-page handout once every citation checks out.
A Grade-Calibrated Illustration
Say you teach a grade 3 class about to open a unit on how the internet connects devices. Before any lesson, an open prompt like "where do you think a video lives before it plays on your screen?" surfaces exactly the central-computer or satellite-based beliefs the Babari et al. (2023) synthesis documented, with no penalty for a belief nobody has taught yet.
A grade 7 class about to open a unit on how AI systems make decisions is ready for a parallel probe: "Does an AI chatbot know things because someone typed the answers in, or some other way — and why do you think that?" That question is built to surface the "pre-installed knowledge" pattern Mertala and Fagerlund (2024) documented, ahead of any lesson on training data.
Tools Compared for CS Diagnostics
| Tool | Drafts Open, Strand-Specific Prompts | Names the Study a Pattern Traces To | Distinguishes Diagnostic From Placement/Competition Test | Starting Price |
|---|---|---|---|---|
| EduGenius | Yes, via open-response format selection | Yes, if requested explicitly | Yes, if specified in the prompt | 25 welcome credits free; Starter $7.99/month (500 credits); Professional $15.99/month (1,000 credits) |
| General-purpose AI chat tools | Yes, if explicitly requested | Only if explicitly requested | Only if explicitly requested | Varies by provider |
| Bebras / MG-CSCI | No — fixed batteries, wrong construct or wrong strand | Not applicable | Built for competition or programming screening | Free (Bebras) / restricted research access (MG-CSCI) |
| Manual, teacher-written probe | Full control | Fully teacher-researched | Fully teacher-judged | No cost |
For the timing distinctions between a diagnostic and a fast in-class check, see Generating Formative Assessments for Computer Science With AI and Generating Exit Tickets for Computer Science With AI. For a gradable, post-instruction version of the same strand-specific item types, see Generating Short-Answer Quizzes for Computer Science With AI. The cross-subject speed problem — verifying what a prior teacher already covered — sits in How to Write Better Diagnostic Assessments Faster With AI.
Pro Tips for CS Diagnostics
- Treat "no validated instrument" as license to build carefully, not casually. A thinner evidence base raises the bar for verifying every claimed pattern, it doesn't lower it.
- Lean on Networks & the Internet and Impacts of Computing first. Those two strands have the richest citation trails; Computing Systems and Data & Analysis need more teacher judgment and less borrowed research.
- Watch the AI-CI and AICI-NE space. A genuinely validated AI-literacy instrument may exist within a year or two of this writing — worth rechecking before assuming none exists.
- Keep every probe explicitly ungraded, the same rule the coding and ESL diagnostic guides both enforce, so students answer honestly rather than strategically.
What to Avoid
Treating Bebras Performance as a Misconception Diagnosis
A low Bebras score flags a struggle with a puzzle-style item, not a specific, documented belief. Its distractors weren't built from mined student wrong answers the way a true concept inventory's were (Araujo, Santos, Andrade, Guerrero, & Dagienė, 2017).
Borrowing MG-CSCI Items for a Non-Code Strand
MG-CSCI is real and validated — for variables, loops, conditionals, and algorithms (Rachmatullah et al., 2020). Presenting its items as evidence about Computing Systems or Networks understanding tests something the instrument was never built to measure.
Citing an AI-Sounding "Study" Without Checking It
This strand's real citation pool is thin enough that a fabricated-sounding "study" can slip past a skim-read. Verify every claimed source the way the ESL guide already insists on for interlanguage patterns.
Scoring a Belief-Surfacing Probe Like a Quiz
Attach a grade to "where do you think a video lives," and a class stops answering honestly. That erases the exact signal a pre-unit diagnostic exists to collect.
Key Takeaways
- No validated concept inventory exists for CS's four non-code strands at any grade band — this is a fifth shape in the series, distinct from biology's yes, music's no, coding's access failure, and ESL's construct mismatch.
- The one K-12-validated CS concept inventory, MG-CSCI, targets Algorithms & Programming (Rachmatullah et al., 2020) — the strand this guide and coding's guide already cover, not the non-code strands.
- Bebras is a real, widely used K-12 informatics assessment, but not a diagnostic — its items aren't misconception-derived, and its own construct validity is disputed (Araujo et al., 2017).
- Networks & the Internet has the richest non-code research base, spanning ages 3-15 in one 2023 synthesis (Babari et al., 2023) — the closest match to a K-9 classroom in this entire question.
- Impacts of Computing is the youngest but most active line, with named AI misconceptions documented in 2024 (Mertala & Fagerlund; Marx, Witt, & Leonhardt) and a validated instrument still under construction as of 2025 (Mannila et al., 2025).
- A teacher-built, ungraded, citation-checked probe is still the honest path, scoped to one documented belief at a time, exactly as coding's and ESL's diagnostic guides both concluded for their own subjects.
Frequently Asked Questions
Is there a validated diagnostic test for computer science, like biology's CINS or physics's FCI?
Not for the non-code strands. A validated K-12 CS concept inventory exists, MG-CSCI (Rachmatullah et al., 2020), but it measures programming concepts — the strand this series' coding guide already covers, not Computing Systems, Networks, Data & Analysis, or Impacts of Computing.
Can Bebras be used to diagnose a specific computer science misconception?
No. Bebras items are written for engagement and difficulty tiering, not mined from documented student wrong answers, and research testing Bebras scores against programming grades found only a moderate correlation (Araujo, Santos, Andrade, Guerrero, & Dagienė, 2017). It's a strong enrichment tool, not a misconception-diagnostic instrument.
Which non-code CS strand has the most research-backed diagnostic content available right now?
Networks & the Internet, based on a 2023 synthesis of 27 studies covering children ages 3-15 (Babari, Hielscher, Edelsbrunner, & Conti, 2023) and earlier secondary-student research (Diethelm, Wilken, & Zumbrägel, 2012). Impacts of Computing is close behind, with newly documented AI misconceptions from 2024 research.
Can AI generate a real diagnostic for computer science's non-code strands?
Yes, for open, belief-targeted probes grounded in the studies above, once a prompt names the specific strand and pattern. It works best paired with a human check that any claimed study is real, given how thin and recent this particular citation base is compared to other subjects in this series.
Sources
- CSTA. (2017). CSTA K-12 Computer Science Standards.
- Tew, A. E., & Guzdial, M. (2010). Developing a validated assessment of fundamental CS1 concepts. SIGCSE '10 Proceedings, 97-101.
- Parker, M. C., Guzdial, M., & Engleman, S. (2016). Replication, validation, and use of a language independent CS1 knowledge assessment. ICER '16 Proceedings, 93-101.
- Rachmatullah, A., Akram, B., Boulden, D., Mott, B., Boyer, K., Lester, J., & Wiebe, E. (2020). Development and validation of the Middle Grades Computer Science Concept Inventory (MG-CSCI) assessment. Eurasia Journal of Mathematics, Science and Technology Education, 16(4).
- Taylor, C., Zingaro, D., Porter, L., Webb, K. C., Lee, C. B., & Clancy, M. (2014). Computer science concept inventories: past and future. Computer Science Education, 24(4), 253-276.
- Ali, M., Ghosh, S., Rao, P., Dhegaskar, R., Jawort, S., Medler, A., Shi, M., & Dasgupta, S. (2023). Taking stock of concept inventories in computing education: A systematic literature review. ICER '23 Proceedings.
- Zhang, H., Perry, A., & Lee, I. (2024). Developing and validating the artificial intelligence literacy concept inventory: An instrument to assess artificial intelligence literacy among middle school students. International Journal of Artificial Intelligence in Education.
- Araujo, A. L. S. O., Santos, J. S., Andrade, W. L., Guerrero, D. D. S., & Dagienė, V. (2017). Exploring computational thinking assessment in introductory programming courses. 2017 IEEE Frontiers in Education Conference.
- Diethelm, I., Wilken, H., & Zumbrägel, S. (2012). An investigation of secondary school students' conceptions on how the internet works. Koli Calling '12 Proceedings, 67-73.
- Babari, M., Hielscher, M., Edelsbrunner, P., & Conti, D. (2023). A literature review of children's and youth's conceptions of the internet. International Journal of Child-Computer Interaction, 37, 100595.
- Donate-Beby, B., García-Peñalvo, F. J., & Amo-Filva, D. (2024). Filling the gap in K-12 data literacy competence assessment: Design and initial validation of a questionnaire. Computers in Human Behavior Reports.
- Mertala, P., & Fagerlund, J. (2024). Finnish 5th and 6th graders' misconceptions about artificial intelligence. International Journal of Child-Computer Interaction, 39, 100630.
- Marx, E., Witt, C., & Leonhardt, T. (2024). Identifying secondary school students' misconceptions about machine learning: An interview study. WiPSCE '24 Proceedings.
- Mannila, L., Henry, J., Bahr, T., Chytas, C., Connamacher, H., Müller, B. C. N., Opel, S., & Scholl, A. (2025). Towards validating an artificial intelligence concept inventory for non-experts (AICI-NE): Common misconceptions and item development. ITiCSE-WGR '25 Proceedings.
- Pamplona, S., Medinilla, N., & Flores, P. (2013). Exploring misconceptions of operating systems in an online course. Koli Calling '13 Proceedings, 24-33.
- WIDA. (2025). WIDA Screener. Board of Regents of the University of Wisconsin System.
- Anderson, D. L., Fisher, K. M., & Norman, G. J. (2002). Development and evaluation of the conceptual inventory of natural selection. Journal of Research in Science Teaching, 39(10), 952-978.