The problem
In Peer InstructionAn in-class strategy where students first answer a question on their own, then discuss it with peers, revise, and resubmit., students write code on their own, discuss it in small groups, revise, and resubmit. It reduces failure rates and improves exam performance, but only when the discussions are productive. In one large astronomy course, 37.7% of peer discussions were unproductive, and instructors often don't find out until it's too late.
In a large programming lecture, the instructor faces dozens of groups producing code submissions, test results and chat messages at the same time. Existing tools fall short in one of two ways:
- Learning-analytics dashboards (e.g. VizProg, Codeopticon) show individual students' code in real time but ignore collaboration.
- Collaboration-analytics tools (e.g. Groupnamics, Pair-Up) show group dynamics, but only with fixed, predefined metrics. They can't show how discussion relates to progress on the coding task.
So the question is how to help an instructor notice which groups need help, and why, without getting buried in data.
Formative study
We first interviewed 4 experienced instructors of large programming courses, then built a technology probeA simple, functional system deployed to learn about users' needs in context, rather than to test a finished design. to test those ideas.
The probe ran unit tests on every code submission and used GPT-4 for two jobs: summarizing each group's conversation topic, and tagging every chat message with one of six categories drawn from social interdependence theoryA theory holding that effective collaboration involves 'promotive interaction': members actively encouraging and facilitating each other's progress toward shared goals.: help-giving, help-seeking, exchanging information, joint reflection, mutual encouragement, or off-topic.
Go deeperHow the activity score is computed
Each on-task message (categories 1–5) adds 1.0 to a student's activity score, and each off-task message adds 0.3. A group's score is the average of its members' scores. On the scatter plot, each group is a dot: x = unit-test pass rate, y = activity level.
8 instructors then compared the probe against a baseline without the discussion features, using a replay of a real lecture: 111 students in 37 groups.
With the baseline, participants described issues vaguely ("Syntax Error"). With the probe they could connect code to conversation ("returning not an int or wrong int from calculation"). Completion time didn't differ.
The probe also exposed a new problem: instructors had to keep every pattern in their heads.
— P6, formative studySometimes I forget some of the students' performance before that I can't tell if the pass rate was increased.
Which patterns mattered also depended on timing. A quiet group is fine while everyone is coding, but it becomes a problem once everyone else has submitted. This led to three design goals:
- Navigate across granularity. Move smoothly between class, group and individual views.
- Monitor without constant attention. Track patterns in real time so instructors don't have to.
- Guide attention in context. Interrupt only when it matters, based on what both instructors and students need right now.
How VizGroup works
VizGroup tracks five attributes during a session. Pass rate, activity level and code issues are tracked for individual students. Pass rate, activity level, conversation topic and team structure are tracked for groups. All panels are linked, so selecting data in one updates the others.
Three levels of the same class
Instructors zoom between three views of the class:
Each triangle is a group, placed by its average pass rate (x) and average activity (y). This view is for a quick read of the whole class: which groups are behind, and whether they're talking.
Groups are placed by their current conversation topic (x) and number of active members (y). Each group is drawn as a small network, with arrows showing who messaged whom; thicker arrows mean more messages. Dots are green when a student passes every test and pink when they don't. When a group's topic or participation changes, its glyph moves and flashes to catch the instructor's eye. To avoid clutter, only the eight lowest-scoring groups are expanded.
Each circle is a student, placed by pass rate (x) and activity (y). This view is for spotting students who are struggling or left out, as well as students who are doing well and could help their peers.
Go deeperHow conversation topics are found
Each group's conversation is embedded with an LLM (text-embedding-3-large), and the embeddings are clustered with K-means++. Each cluster gets a short summary. New summaries are compared against existing ones and merged when they match, so the topic list stays short and stable as the session goes on. Arrow thickness in the Structure view is min(A × 0.25, 2) + 1, where A is the student's activity level.
Trackers and alerts
To meet the second design goal (monitoring without constant attention), instructors can turn any pattern into a notification:
- A tracker charts an attribute over time, such as code issues, conversation topics or how many members are participating, as either a bar chart or a timeline.
- An alert fires when students or groups meet criteria the instructor sets. Alerts come in two kinds:
Every data point that matches the criteria gets its own "alarm clock." If a group stays stuck, say talking about syntax errors with a pass rate under 41% for more than 2 minutes, the instructor is notified about that group.
The alert fires when too many students fall into a region at once. For example: more than 10 students with a pass rate below 50% and activity between 0 and 3. That usually signals a class-wide misconception rather than one group's trouble.
The AI suggests what to watch
Writing good alert criteria in the middle of a lecture is hard, which is where the third design goal comes in. VizGroup suggests notifications in two ways:
- From the instructor's interactions. Clicking a group, brushing a region or expanding a topic produces a suggested alert prefilled with those values, which the instructor can then edit.
- From the class's history. An LLM pipeline reviews every group's submissions and chat logs and proposes alerts for the problems it judges most urgent.
Go deeperThe four-step suggestion pipeline
- Identify issues. For each group, the LLM reviews its submission history, conversation log, pass rate, activity, topic and participation, and names the specific problem, e.g. "no conversation despite submission attempts."
- Rank by severity. A list-wise rankingThe LLM ranks the whole list of candidates in one pass, rather than scoring each one separately or comparing them in pairs. This balances cost and quality. puts issues that need a TA above ones students can resolve themselves.
- Summarize the top 5. The LLM summarizes what the five most severe groups have in common, and which attributes (pass rate, topic, relevance, participation) are involved.
- Generate criteria. VizGroup computes the alert criteria from those groups: the set of values for categorical attributes, and the minimum and maximum for numeric ones.
The prompts use few-shot examples and AI chains; the full prompts are in the paper's appendix.
Evaluation
12 instructors with programming-teaching experience watched a live replay of the same 111-student peer-instruction session. Each used two of three versions: a baseline (interactive visualization only), VizGroup without suggestions, and the full VizGroup. They answered two quiz questions and did an open-ended monitoring task:
- Q1: find students who haven't passed and only sent off-topic messages.
- Q2: find groups with a specific difficulty, and track the ones stuck for two minutes.
Instructors were faster and more accurate
On Q2, most VizGroup users set an alert and moved on to look at other groups, while most baseline users had to keep watching the target groups themselves.
— P4It is not that hard for me to monitor a group of students, but it would be impossible if there are multiple groups to track.
Suggestions changed what instructors looked for
In the open-ended task, participants received about 18 suggestions each. Of the 13 kinds of notification suggested, 11 were accepted by at least one participant.
The suggestions also pulled instructors beyond their usual strategy. P3 began by saying "I care about the pass rate, not the type of code issues". They then accepted a suggested Code Issue alert because the students it flagged "made sense," and went on to create more Code Issue alerts themselves. Without suggestions, participants such as P5 tended to make several near-identical notifications.
Participants reported lower mental demand with suggestions (median 2.0 vs. 3.5 on the NASA-TLX scale) and lower time pressure (1.0 vs. 2.0, not significant).
Workload (NASA-TLX)
Mental demand
Temporal demand
Performance
Effort
Frustration
Show data table
| Question | Condition | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|---|
| Mental demand | No suggestions | 1 | – | 2 | 1 | – | 2 | – |
| VizGroup | 1 | 3 | 1 | – | 1 | – | – | |
| Temporal demand | No suggestions | 2 | 3 | – | – | 1 | – | – |
| VizGroup | 4 | 2 | – | – | – | – | – | |
| Performance | No suggestions | 3 | 2 | – | 1 | – | – | – |
| VizGroup | 2 | 4 | – | – | – | – | – | |
| Effort | No suggestions | 1 | – | 4 | – | 1 | – | – |
| VizGroup | 2 | 1 | 1 | – | 2 | – | – | |
| Frustration | No suggestions | 3 | 2 | – | – | 1 | – | – |
| VizGroup | 3 | 1 | 1 | 1 | – | – | – |
Ease of use and usefulness
Ease of use
Usefulness
Figure 12, redrawn. Counts of the 6 participants per condition are estimated from the paper's chart; hover or focus a segment for its count.
Show data table
| Question | Condition | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|---|
| Ease of use | No suggestions | – | – | 1 | – | 3 | 1 | 1 |
| VizGroup | – | – | 1 | – | 1 | 4 | – | |
| Usefulness | No suggestions | – | – | – | – | – | 4 | 2 |
| VizGroup | – | – | – | – | – | 4 | 2 |
Instructors also saw the alerts as a way to prepare before stepping in:
— P6The topics filter can help professors/TAs prepare beforehand to help students instead of just going up to them and asking what they're struggling on.
Limitations
- A simulated classroom. The study used real student data, but participants watched a replay rather than teaching live, so there was less pressure and distraction, and no real students to help.
- One session. All data came from a single peer-instruction session in one large course, which may not generalize to other courses or smaller classes.
- Five attributes. Notifications are limited to pass rate, activity, code issues, topic and team structure.
- LLM trust and scale. The suggestions raise questions of stability, transparency and trust, and very long chat or submission histories can exceed the model's context window.
Takeaways
Large classes don't have to mean less support for each group. VizGroup shows that an AI can do more than summarize: it can propose what to watch. Instructors stayed in control of every alert, but the suggestions widened what they paid attention to and reduced how much they had to remember. More broadly, this points to a model of human-AI collaboration in real-time analytics where the AI suggests and the person decides.
Cite this paper
@inproceedings{tang2024vizgroup,
author = {Tang, Xiaohang and Wong, Sam and Pu, Kevin and Chen, Xi and Yang, Yalong and Chen, Yan},
title = {VizGroup: An AI-Assisted Event-Driven System for Real-Time Collaborative Programming Learning Analytics},
booktitle = {Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology},
series = {UIST '24},
year = {2024},
location = {Pittsburgh, PA, USA},
publisher = {Association for Computing Machinery},
doi = {10.1145/3654777.3676347}
}