Banana Split Without the Bowl

Below is an interview with Andy Brown, lead instructor for Udacity — the last of a four-part series on the future of higher education. These and related posts are now indexed at this blog’s Higher Ed page.

The traditional curriculum for the four-year, full-time college experience is like a banana split. At the bottom is a general education or core curriculum track (breadth). It’s not really what you’re interested in but it’s supposed to be good for you. In the middle is the really tasty stuff, the courses for a major (depth). Then there’s the cherry on top, some sort of thesis, project, or other experience. All of this is spaced out to occupy a minimum of eight semesters over four years. This paradigm of higher education has become extremely expensive and inconvenient for a lot of people in the USA. What’s the worst-performing component of this paradigm, and how do the folks at Udacity know that they can do better?

A lot of things don’t make a ton of sense in the traditional paradigm, but first let me mention one that does: sticking a bunch of passionate people who are super enthusiastic together in the same place at the same time makes learning a contagious experience—we cannot even come close to that online.

That said, one of the major dysfunctional aspects of the traditional paradigm comes from the social belief of what college is supposed to be or look like. For example, in the existing paradigm, the question “Why do I have to sit through four years of this before I can be seriously apply what I’ve learned?” doesn’t really have a good answer.

Ideally, people would learn as needed – don’t teach me anything until I really need to know it. But even in the best course on campus, you are not always teaching students something that every single one of them immediately needs to know. There are physical constraints that play into this. At traditional universities, there is finite space—only a certain number of students can learn at a given time, because there are only so many classrooms—and finite time on the part of students and teachers. Because of technology, all these one-time costs that are associated with college can now go away. We can continuously update our knowledge. The idea of lifelong learning becomes much easier.

Going back to the banana split example, the breadth aspect, if done right, is very important. One thing that worries me is that if knowledge becomes a liquid asset, and people can study only what they want—a computer programmer who studies nothing but computer science, for example—that person may have educated himself or herself in a particular skill but isn’t very flexible or creative in how that skill can be applied. The practice also sends certain messages, like “you’re not going to benefit from people older than you are.” This is not very good for society.

In terms of the tasty stuff or depth, there is a lot of room for pruning. A lot of curricula are pretty uninspired; they are based on assumptions on what a field of study should look like and archaic practices. People are told all sorts of things are important, but the content isn’t relevant to what the student needs, or the content isn’t delivered well. There’s a disconnect between what students are told and the reality they experience. Even with the cherry at the end, you see the effects of committee-driven curricular design.

Keep in mind that people care deeply about the social. They don’t want to spend four years of their lives not being social. The traditional campus experience is not just getting help learning the content prescribed by the instructor, it’s the incidental learning that occurs when you ask someone wiser than you for help with that content, and in the process you stumble across a conversation about something like politics. That’s where a tremendous amount of learning occurs.

As a political science professor, I have a really hard time collecting good data on what and how my students learn. For me a large sample is thirty-five students, a full class, which doesn’t even meet the bar for real statistical significance. This means it’s really difficult for me to make data-driven decisions about the effectiveness of my teaching. Meanwhile Udacity is collecting data on hundreds of thousands of students. What are some of the surprising things that you’ve discovered about how people learn, and what are the implications for teaching in a physical classroom?

Even with massive sample size, measuring learning is difficult to do and we don’t really know how to do it. There are proxies that we can use—ask a question, did the student get the correct answer?—but mainly I want to see that students are thinking. Does a student spend five seconds, twenty seconds, or a minute and a half on a question? That is perhaps a better proxy for learning.

One of the challenges that we face is retention on a short scale: how can I design a course so that in three minutes people are still in it, engaging with the content? Changing visual media a good thing, so is asking questions in the middle of lectures. With open-ended questions, where any response is marked correct, a response rate of 85-90 percent is really good. It means people are thinking about the questions.

We interpret all of this as a sign that students on Udacity are really well motivated. Keep in mind that there is a complete selection bias here—these are people finding another resource to learn from, online. If we define retention as the number of people who finish a course divided by the number who started it, we are reluctant to use retention as the sole indicator of anything significant, because the barrier to entry for a Udacity course is very low. It’s probably not as significant as with a traditional university, where entry costs are so much higher.

Udacity’s curriculum was initially limited to computer science-oriented subjects, which made a lot of sense. Student performance in programming and computation is much easier to assess quantitatively than, for example, 19^th century British literature. Your code either works or it doesn’t, but efficiently grading 50,000 essays on Jane Austen novels is really difficult. Yet now I see Udacity is branching out into subjects like psychology, business, and design. Why? What’s the value that Udacity courses in these areas can offer to people, when there’s no reliable means of assessing knowledge or skill acquisition?

You are correct; there is no reliable automated method of assessing stuff like essays on 19^th century British novelists. So what? Not all of these education problems need scalable automated solutions.

For example, in principle, you don’t need grading at all. With grades, if it’s above a certain number, you feel validated, if it’s below, you feel crappy. In the latter situation, you can spiral into failure. A much better situation is for students to be intrinsically-motivated, to get a rubric, and then evaluate their own work in comparison to that of others. Or get feedback from a human instructor. These are non-scalable methods of evaluation that are way more valuable than grading.

We don’t do the ideal in our courses—yet—and that’s one reason why we branched out, to experiment. The design course will be really fun. And the genomics class is without a doubt the class that so far has had the most thought put into it. But we are now more focused on the short and medium term, and moving back toward a computer science focus. Not all of our experiments worked well. What have we learned from these experiments? In many cases it’s been simply relearning the lessons of in-person teaching and bringing them to bear in an online environment. For example, at the beginning of every lesson, clearly articulate the objectives of what is to be learned. Align the assignments with the objectives. Provide good examples and a compelling story. Is the course creatively designed?

Even though a course from Udacity might lack some of the assessment mechanisms used for the traditional classroom, it still might very well provide people with a much better experience than what they can get in person. For example, one advantage of an online course with Udacity is that once you figure out how to do something, you can share that knowledge. Similarly, from our end with course design, once we figure out how to do something well in terms of design, we can easily propagate it.

I’ll give you an example of how Udacity might provide someone with a superior experience. How does a professor know what students in the lecture hall are getting from the professor’s lecture? What parts of the lecture are the most valuable to the students in terms of their learning? The professor doesn’t really know any of this. Now let’s put a one-way mirror between the professor and the students, so the students can freely leave when they’re bored, without being seen by the professor. We’ll record when the students leave. How many students do you think will be there at the end of the lecture?

You show the video to the professor. “A bunch of students left right there, that part of the lecture must have really sucked.” Even if you’re purely ego-driven, when you see people walking out of the room, that’s going to influence what and how you teach. People don’t do that kind of feedback in the physical environment, but we’re doing it online.