With all the coverage on CNN and Fox News lately, Deep Learning has quickly become a household term. Well, not quite, but Deep Learning is definitely all the craze these days for those of us steeped in Machine Learning and big data. We at Lab41 have been trying to find signal in noise over the past few months, and thought it worthwhile to share our “Unauthoritative Practical Getting Started Guide” with others looking to get started with Deep Learning. Before I go on, I must warn you this largely avoids the difficult Greek symbols and other maths underpinning this complex topic. If you’re looking for formal definitions and proofs, you’ll have to follow the links to resources peppered throughout this post. Now let’s get Deeply Learned!
Why Deep Learning?
Before we get started, we have to ask the most basic question: Why is Deep Learning so interesting?
Well, it’s partly because you can be so incredibly flexible and creative with the tasks you want these algorithms to complete:
- Andrej Karpathy’s blog post, “The Unreasonable Effectiveness of Recurrent Neural Networks,” provides Deep Learning code to automatically generate text from scratch that looks and reads like Shakespeare. The same code base also can do other impressive things, including generating software source code (in C) that looks like a real programmer wrote it. Pretty soon I’ll probably be out of a job.
- People also are starting to write music using neural networks, which lucky for me enabled the extension of my favorite music from the famous song “Let it Go” from Disney’s “Frozen”. It’s amazing; the music actually sounds pretty good, which is incredible since the algorithms did not require human intervention and still managed to learn the rhythm, tempo, and style of the song.
- Yarin Gal, a third-year Ph.D. student at Cambridge, was able to take famous pieces of art and extrapolate what the painting would have looked like if the artist had drawn more on the canvas.
These are but a sample of the creative examples posted to places like Hacker News on a seemingly weekly basis, which makes it very exciting how many domains and applications will be improved by this still-bourgeoning field. But the question we asked ourselves is how do we actually get started with understanding deep learning? Since we’re not Ph.D. Machine Learning candidates and don’t work for any of the Google/Facebook/Microsoft brain trusts in the field, we initially felt a bit uneasy about how to tackle the foundations and applications. Luckily for us, people like Karpathy, repos on GitHub, and several other amazing resources are out there if you know where to look. Let’s do just that…
How to Learn about Deep Learning?
As we initially approached this question, we realized that there are already a ton of online resources that can help you get started and we have aggregated many of these resources on our Github Wiki.
This is great news and shifts the focus from “finding resources” to “filtering good ones.” If you follow down this path, you’ll quickly find that one of the first resources everyone mentions is Andrew Ng’s Coursera class on machine learning (ML).
We found the lectures about basic ML concepts like linear and logistic regression to be useful if you don’t have much of a ML background. More specific to Deep Learning, we found the lectures on neural networks to be a great primer for understanding the basic concepts and architectures. Once we got a hang of some of the basic neural net lingo we found a lecture series that Quoc Le (from the Google Brain project) gave at Carnegie Mellon a few years ago to be very effective. That lecture series not only explains what neural networks are, but also shows examples of how a place like Google uses them and why we call this field “deep learning”.
After we watched these different lecture series’, we felt like we were getting a grasp of the field. However, we were still looking for something that tied the math and theory to a practical application in a cohesive way. One of the best resources we evaluated for putting this picture together were the notes by the aforementioned Andrej Karpathy for his class at Stanford “Convolutional Neural Networks for Visual Recognition”. His notes are tremendously clear and understandable (unlike many other deep learning texts we found) and explain how to think of complicated deep learning concepts in simpler terms. We really can’t emphasize enough how his notes (which led us to referring to him as Andrej the Jiant) illuminated so many difficult topics and helped us transition from “reading about” to “actually doing” Deep Learning.
Once we got a good sense for the overall picture with a combination of class notes and lecture videos, we were in a good position to start understanding recent academic literature in this space. We found the transition from more high-level resources like Andrew Ng’s class to more specific example based resources like Andrej Karpathy’s notes to be extremely valuable in solidifying our understanding of key concepts. As we alluded to earlier, the resources mentioned above represent only a few of the resources we used to get started. More detailed resources lists can be found on our Github Wiki, which includes deep learning textbooks and academic papers we’re reading that apply deep learning in interesting domains.
Next Steps
So where do we plan to go from here? We’ve seen great progress in the computer vision field with people getting higher and higher scores on ImageNet. This is great news for the field and for image-related applications. However, as a recent Nature article from Bengio, Hinton, and LeCun highlighted, Natural Language Processing could be the next frontier to benefit from Deep Learning architectures. Classes like Richard Socher’s at Stanford, “Deep Learning for Natural Language Processing” (the class notes and lectures are freely available!) only reinforce this notion and indicate the growing interest in this area.
Since text processing is central to so many of our challenges at Lab41, we wanted to use NLP as our first hands-on foray into the field. Stay tuned to our blog as we document our exploration of Deep Learning! Our journey will be documented in code and Wiki within our Github repository and with blog entries centered on the following topics:
- Deep Learning applied to text: Sentiment analysis vs. Traditional ML approaches
- Easy setup of development environment for Deep Learning using Docker and GPU’s
- Deep Learning framework comparison: Theano vs Caffe vs Torch vs What is a Keras?
Originally published at www.lab41.org on September 21, 2015.
