My topic considers how to support teachers as researchers using a learning platform for a social and emotional learning (SEL) course.

What insights should teachers be able to take from process data about student progress, behavior, conversations, and activities done in a collaborative online platform^[1]? This is the current current question I'm facing. Of course, closely followed by, what inferences CAN researchers make (automatic rules created) using the process data (available) in order to generate system messages to support teachers in making these insights about student progress, behavior, and activity?

This project is in support of a grant to support the development of middle-grade students' SEL skills. To do this, we took a SEL classroom intervention program that was designed for presentation in the classroom (including slide decks and scripts for teachers), and made them interactive for students. During the lesson, the teacher still presents the content from the front of the room, but students follow along on their screens, occasionally answering questions or chatting about the lesson content in small groups through the interface.

While the grant scope is fairly basic (have positive outcome effects following the intervention), it makes sense that we do as much as we can with the opportunity. So, since we're collecting data, and we have the opportunity to share that data with teachers, we made it a priority to essentially turn teachers into resarchers of their students -- NEW LEARNING! Here's an example of what teachers might be presented with following a lesson. Some of it may be useful, some of it may not be, and too many useless charts will likley be too much for teachers to look at.

Example statistics teachers may see following a SEL lesson.

The data that we collect from students we do our best to summarize for teachers at the end of the lesson. For example, presenting the on- and off-topic lines of chats in a stacked bar chart (one bar for each student), summing response selections for different questions in pie charts, and making everything easy to drill down to another level for further exploration. When presented with all of the statistical details above, what we heard from teachers is that they mostly want to know what students are saying in their groups (so we now present the full chat data as a dedicated page and as a drill-down feature from the stats page) and that they're surprised to see how insightful normally quiet students can be (one observation that they can make about a students SEL skills). Teachers are interested in what students are saying and relate that to their SEL skill development; so, how can we use AI to support them?

SEL is a rather hot topic in education lately, right up there with AI, and we want to blend them. When I say AI, I'm not talking about an agent or pedagogical tool, but more a traditional pattern recognition algorithm. Here's the challenge:

1. We don't have data yet. We can develop a theoretical model of what "good" or "poor" communication would look like, what keywords or actions might reflect engaged or disengaged students, but until we get that data, an compare it to a somewhat valid and reliable measure of what we're trying to investigate (whether it be SEL development or indicators of classroom engagement with the lesson), it's only conjecture and untested theory. This brings us to...
2. We aren't sure if the data we have is really that useful. We have chat data, but what is that data reflective of? We're ultimately trying to improve SEL skills; can we really tell if a student is improving from chat log data? The thing with process data seems to be is that you use it because you have it, not because it's particularly valid about what your'e trying to measure. Along these same lines...
3. We don't know if the patterns we hope to identify are valid representations of what we expect them to be. This program is for 5th and 6th graders. They misspell words, play with the interface, and have side conversations. We're a long way from being able to detect when a student is acting inappropriately or if they're of a lower reading/writing skill level, or somthing else.

I think it makes sense to start with the basics, which is theoretical modeling. If teachers want to know if a student is chatting in their groups or just screwing around (holding down different keys and typing a bunch of emojis in the chat), we can create system rules to detect junk or not junk witout every analyzing the content. Within each lesson, the teacher asks students to chat at least three times. So there's another rule - students should have at least three on-topic chats during a lesson (and ideally, few to no off-topic chats).

We'd also like to produce some feedback for students. Given the young age, we do not expect that the majority of students will be able to understand a bar chart, or any other statistical representaiton, so system messages would be a clear way of offering feedback. Something like this, perhaps:

Example system message for students based on expected communication rules.

Once we get the data, then maybe we can start to do something cool with this. I recently read a LinkedIn interview^[2] with researchers at Development Dimensions International (DDI), which focuses on workplace assessment, is that the best contribution the current iteration of generative (AI) modeling can offer is conversation and language analysis, and that's probably exactly what we need. While it's still pretty far off, I did come across this article on the variation ("personalities") across different LLMs^[3] which would be important if we were to create system generated summaries or other similar system-generated feedback.

[Update]

I have run into issues finding great sources of existing literature on building system-generated teacher feedback (so please link, if you know any!), but most of my research has turned up coding guides that have identified the types of feedback a teacher might give to students.

There is a good deal of literature on student feedback. Although, as noted above, the early-phase is likely to only cover the low-hanging fruit, such as responding an expected number of times during the lesson and having those responses not contain junk text (such as holding down a key on the keyboard), getting a measure of student engagement with lesson content is a high priority. To that end, when looking at the literature, the majority of student engagement measures are based on identifying rapid guessing^[4][5], which would not be applicable to this setting. Another paper^[6] introduced “nudges” to students when it didn’t seem like their responses were effortful, but this also required a time factor. So I’m still working on this as well. I have some ideas, and will present in the next update!