Case Study: Cornell University


In the interview below, Professor Kilian Q. Weinberger, explains his strategies for teaching and managing large scale machine learning classes.


Kilian Q Weinberger

Kilian Weinberger, Professor of Computer Science at Cornell University (Machine Learning)

As interest in Machine Learning intensifies, Computer Science departments across the nation are introducing more and more classes into their CS curriculum. Accommodating these new classes and the increased enrollment has placed a strain on the schools. At Cornell University, for example, the number of students taking CS4780/5780 (Machine Learning for Intelligent Systems) has increased to nearly 350.

Can you tell us about your background?

I’m originally from Germany. I did my undergraduate studies at Oxford University in England and received my Ph.D. from the University of Pennsylvania. Then, I spent almost three years at Yahoo Research in Santa Clara. I was faculty at Washington University in St. Louis for about five years before moving to Cornell University.

As a faculty member at Cornell, I teach Machine Learning for Intelligent Systems in the Computer Science department. This is an introductory class on machine learning for senior undergraduate or graduate-level students.

We have eight class projects ~ every topic they learn as a project needs to be completed either in Python or in Julia. And everything is automatically graded by the auto-grading software and it’s a substantial part of the student’s grade.

What about the student enrollment for your Machine Learning class?

I’ve been teaching Machine Learning here since 2015. And the number of students enrolled has steadily increased; now the number is up to 340, which equals the maximum size allowed by our classroom.

Can you describe the student development environment and some of your assignments?

We do everything in the Jupyter Notebook, hosted in Vocareum. Students begin with some starter codes that were given to them. Then they have to implement machine learning algorithms and interface with them. We then automatically run a set of unit tests, and that’s how the grading works. Typically, they have to pass all the unit tests.

One assignment is to create a facial recognition system. They get a database on faces, and they have to implement an algorithm that identifies each new face shown to it. Or, another assignment is to give them a set of baby names and they have to build a system that, given any name, can automatically identify if it’s a boy or a girl.

Here, the fun thing is they can come up with their features and algorithms. They might design a feature to decide how a particular part of a name gives it away. They might come up with many different ideas how to do this. And, at the end, there is a competition to see who has the best algorithm and Vocareum supports that through Leaderboards.

In fact, when you enabled our Leaderboard feature, you had about 60,000 submissions for just 300 students.

Yes, they kept refining their code to improve results. From a learning perspective, I think implementing these strategies really engages the students. I can see when they submit and re-submit until 4:00 a.m. They really get into it. And they’re very proud that they can get something that beats their peers. Sometimes I also put in my own submissions, and they are very excited to beat me.

How much help did you have to manage the course?

I had up to 16 people who helped with the grading, but a core of eight people helped get the assignments prepared. We set up two Vocareum accounts, one was the Sandbox and the other was the real one.

There was also a Piazza forum where students asked questions, and the TAs answered most of the questions. Many of these TAs were my students from previous years.

Last year, I wrote my own auto-grader. It had similar functionality as the one from Vocareum, but it wasn’t a web interface and not as slick, so people had to submit from the command line. As the classes grow, things inevitably break down right before a deadline, and almost 400 students start sending emails. So that’s the advantage of outsourcing it to Vocareum.

What do you think was the overall impact of using Vocareum on your classes?

One major advantage for me will be consistency – I’m not dependent on any server changes, and I don’t have to manage IT or computing resources.

Previously, one problem we had to deal with was that students might use Linux, Windows, Mac and other different computer platforms. The interesting part is, to use Vocareum, students are provided with a Notebook environment, and everybody has the exact same interface. Homework assignments are also now standardized, and hopefully, we can keep that framework for the next couple of years.

Before now, struggling with the architecture of the class was time-consuming. Also, there were constant uncertainties of something breaking right before assignments were due. From my perspective, just being able to depend on Vocareum and being confident that we have support whenever we need it is the main thing.

One piece of critical feedback was that students wanted to work together in teams, in a Google docs kind of collaboration, right?

Yes that’s the number one feature request. There is a lot of teamwork involved and they work together on these assignments. It is frustrating for them when only one person can access the code, or when they made changes and didn’t save them. I think you solved the latter issues over the course of the class.

Universities like Cornell have TAs and professors like you, but what can be done so that courses like machine learning can be taught by schools that don’t have the same resources? Do you have any thoughts on how to make it scale?

Right now, I think it’s still a little complicated. Sometimes people ask me for my teaching material and texts. There’s no textbook. If there were an easy way of actually cloning it, that would make it a lot easier to share.

Would you like to clone more than assignments – such as your instruction or whatever else you provide to the students?

There’s a course webpage that has every single lecture written up in HTML. In class, I hand that out because I don’t want the students to take notes while I’m lecturing — my class is a blackboard course. I used to give them handwritten notes, but now they are typed up and annotated.

For most lectures, I have a demo that I like to showcase. A lot of students asked me for the code. I imagine that we could deliver the code and HTML via Vocareum and they could look at the code and play with an algorithm. That would be a lot of fun. Everything would be integrated: HTML, demos, and assignments.

Going forward, how do you see cloud computing impacting education at Cornell?

More and more students from other areas are taking computer science classes now too because they need it in their research or they want to know about it. Classes such as networking or operating systems can also get very large. Computer science is just a very popular topic right now – so, I think the cloud could be an essential tool for teachers to handle this massive influx of new students.

This partnership with Cornell is just the beginning of what might be possible with large class sizes. Right now, for example, Vocareum is supporting an online master’s course in data science that has a thousand students using Jupyter Notebook at the same time. Moreover, several new exciting projects are in the works right now involving some of the most dynamic areas of computer science.