Khan Academy

Student Progress

This was a 6 month-long project where we rearchitected the Khan Academy teacher dashboard. The team consisted of myself, a product manager, and 3 engineers. We shipped the feature set in Fall 2018 for Back-to-School.

The Problem

In 2017, we introduced an assignments feature to help teachers target individualized content at their students. Teachers could assign any item of content (exercises, videos, articles, quizzes) with a due date to any group of students. There was no easy way for teachers to view individual student progress over time or find trends in their learning. Student progress also only existed for math and computer science classrooms.

The only way for teachers to see student assignment progress was by clicking X / Y.

Our goal was to give all teachers a way to view their students’ progress that fits into their current workflow and surfaces patterns in student understanding.

Initial Explorations & Research

We had a month to explore and research different directions to display student progress. We started by interviewing 10 non-math teachers to understand how they track student progress and growth. These are some of our high level findings:

Most importantly, we learned that teachers all define proficiency, competency, and mastery differently and want to be able to see student data in a way that matches their own model. They want the flexibility to be able to filter data by student, date range, unit, skill level, standard, and improvement. This is an early prototype I worked on to showcase how all these might work together:

In this prototype, orange means a student has gotten less than 70% on the last 10 questions about the skill.

At the same time, we explored some conceptual directions. In this one, we thought it would be fun if a classroom’s progress was displayed as a garden where each student’s learning was displayed as a plant! The bottom features a interactive timeline to view change over time.

We also brought together the entire team to brainstorm different visuals and forms of displaying student progress.

This was done over Google Slides to be remote friendly!

Narrowing the Scope

After this first round of exploration and research, we wanted to hone in on a direction to tackle first. Progress is complicated and we had so many features we wanted to implement. To help us decide, we showed a series of wireframes to 5 teachers and had them rank the effectiveness of each one.

In each of these examples, we used orange to indicate students with under 70% and green to indicate students with 100%. Based on our conversations, the teachers liked...

We decided that "Progress by Student" would be the most beneficial to more teachers because it allows for teachers to quickly see individual student performance at a glance. It also felt familiar to teachers because it looks like a grade book.

Iterating

At this point, we pulled in the engineers and started building. We recruited an initial pilot group of 30 teachers. Each week, we sent out a survey asking teachers to reflect on the new Progress feature and conducted short interviews with 2-3 teachers to get their reflections. This is where we started:

This is pretty similar to the original wireframe, but just visually redesigned to fit our style guide
We sent a Google forms survey on a weekly basis to collect feedback

17 teachers responded out of the 30 pilot teachers. In general, they loved this new feature because they could see all the information at once. One teacher said:

“I loved that it gave a quick overview of who is doing what work, and the %. This is SO HELPFUL for me as a teacher in many ways. I can see the progress of individual students, determine how to group students/differentiate quickly. This can also make it simpler for me when it comes to putting work in the grade book. It's amazing!”

Here's a summary of the rest of their feedback:

Based on the feedback from round 1, we decided to implement a date filter and content type filter to help teachers identify gaps easier. Some teachers assign videos, articles, and exercises, so they want to be able to filter to see only exercises for a week at a time.

In this week's survey, we asked teachers: "How might you use the Progress page to inform your instruction?" Here are some of their responses:

“It allows me to notice larger patterns of content that I need to reteach and fill in gaps. I also like that I can track how much of the AP Calculus curriculum that my students have practiced."

“I can use it to quickly check how a student is doing, if they are completing their work or struggling. It is extremely handy, as today I had a parent ask if her son had completed all of his missing work that I haven't graded yet, and I was able to quickly check which had been completed.”

“Next year, I might use the Progress page to guide small groups and reteaching within the class and RtI [Response to Intervention]. I might also use the Progress page to check for Mastery and readiness for tests.”

We also wanted to drill into when teachers use the progress page. We discovered that their preferences for the date filter were pretty varied validating the need for the Date filter.

The most popular request we had was an individual student report. We reprioritized our timeline to accomodate for this feature because we felt it was crucial to a teacher's workflow. Many teachers wanted to discuss scores with individual students or their parents without compromising other students' private information.

What We Shipped

After conducting many user interviews and surveys, we felt really confident about launching this to 100% of our users across the world for Back To School 2018! This was our final design.

We changed a few small things before launching including adding icons for easier readability, pagination for faster loads, and vertical student names.

What's next?

This feature was very successful and has an NPS of +51%! We want to continue exploring ways to make progress better for non-math classes, but this would involve a major strategy shift on many teams. In the future, we're going to explore more ways of incorporating classroom activity and trajectory into our progress tracking.

Khan Academy

In-Class

Review Game