Here at Lab41, we have embarked on a new challenge focused on recommender systems and how they may assist a software developer or data scientist find new data, tools, and programs. The results of this challenge will also include guidance on deciding which recommender system algorithm to use on a new application, perhaps in the form of a recommender system recommender. This post will try to illuminate why both of these goals are so important, and how they may be possible.
By now you have likely heard how the volume of data being created on a daily basis is growing at an exponential rate. At the same time, new tools and computer programs are being developed to ingest and understand this data. For example, there are dozens of news feed readers that can summarize and display current news articles. In fact, the number of GitHub repositories, and Apple apps are also increasing at an exponential rate.

Credit: https://github.com/blog/1724-10-million-repositories
As a software developer or analyst, it is important to be able to determine which data may be most pertinent to the problem at hand, as well as determine the set of tools and applications to utilize. In large organizations, it can be particularly difficult to know the variety of data sources and tools that have been curated by your peers. For an analyst starting a new project, this is even truer. So how does an analyst choose among all the choices? Enter recommender systems.
Recommender systems have a successful history of being able to connect a person to items they may find desirable, be it a song on a playlist, the next movie to watch, or a product to purchase. There are two main classes of recommender systems out there, and a very large number of hybrid methods:
- Collaborative Filtering: Recommendations are based on other users’ preferences and behavior. The key to this algorithm is determining the similarity between users through their ratings history, and then identifying other items these similar users liked. For example, Alice is recommended items that Bob likes if Alice and Bob share similar tastes in items. Alternatively, from a user’s set of preferred items you can determine similar items through other users’ ratings. For example, people who like tea will often be recommended teapots, as those who like tea often also like teapots. With either of these method you do not need to determine the attributes of items and can have a higher diversity of recommendations, but you need a large number of user ratings and you may over recommend popular items.
- Content Based: Recommendations are based on the attributes (content features) of items. The attributes of a movie could include the movie’s genre, actors, producer, musical score, year, and/or topical words or keywords from its title and description. For example, Alice is recommended other action movies if she liked the latest James Bond film. With this method you can make recommendations with sparser datasets, and can easily describe why a particular item is being recommended, but the attributes of an item must be determined, and it can be easy to over fit a model.
Now, Lab41 is far from the first group to look into recommender systems which means that there are a number of very good resources to help us understand this technology. Here are a few introductory sources for a general overview of recommender systems:
- Introduction to Recommender Systems: A six week Coursera course from the University of Minnesota with hands-on projects, and around 100 hours of topical lectures, interviews and guest lectures with experts from both academia and industry.
- Recommender Systems Handbook (Ricci, F., Rokarch, L., Shapria, B., Kantor, P.B., Springer 2011): A comprehensive handbook on recommender systems including techniques, applications and evaluation of recommender systems, interacting with recommender systems, recommender systems and communities, and advanced algorithms.
- Recommender Systems: A 4 hour tutorial slide deck by Xavier Amatriain on recommender systems including lessons learned from the Netflix Prize challenge.
Another very helpful resource was attending RecSys 2015 as it focused the state of the art in recommender systems, and allowed us to connect with other researchers in the field. If you are in the field, I would highly recommend (*pun intended) going to RecSys 2016,which will be held in Boston in September.
One of the good nuggets of wisdom we have learned is that building a recommender system really depends on the type of application you are analyzing. For example, the best algorithm for a movie recommender is going to look substantially different than the best algorithm for a product recommender. And while there are some algorithm comparisons, it can be difficult to determine which algorithm to try first for your dataset.

But, what if we could create a recommender system recommender? Where a user could input some high level information about their data and be shown a variety of recommender system algorithms, and their potential performance on that dataset. To achieve this goal, we will be evaluating a wide range of recommender system algorithms against a wide range of applications on a variety of performance metrics. At the end of the day, there may not be a clear “best” recommender system, but perhaps the field of possibilities can be narrowed. And after this analysis, we will be well equipped to start to determine how to recommend data, tools, and programs to a software developer or data scientist.
As with many of our challenges, the Hermes challenge builds on some of the work the lab has completed in the past. For example, the Circulo project created a community detection evaluation framework which you can download or read more about. The Framework performs statistical analysis against partitions of a Graph resulting from the execution of a given community detection algorithm. The resulting quantitative measures can be used to drive experiments such as measuring algorithm efficacy against specific dataset types or comparing different algorithm execution results against the same dataset. In addition, two other challenges compared the performance of a structure/behavior algorithm to a content based algorithm. Dendrite was an exploration of ways to analyze, manipulate, version, and share extremely large graphs (GitHub, wiki). And Soft-Boiled analyzed algorithms for Geo-Inferencing Twitter Data (GitHub).
We’ll keep you all up to date on what we find during this exploration and lend any advice that we can. Future blog posts will outline the various datasets we found which are suitable for evaluating recommender system algorithms, descriptions of the algorithms themselves, and the results of our analysis. And of course, the work we complete will also documented on our GitHub repository.
Tags: recommender systems
Originally published at www.lab41.org on December 19, 2015.