Pinterest is a visual catalog with several billion pins, which are visual bookmarks containing a description, a link, and an image or a video. A major problem faced at Pinterest is to provide personalized, engaging, and timely recommendations from a pool of 3+ billion items to 200+ million monthly active users. Recommendations at Pinterest are a problem with a scale beyond the classical recommendation problems studied in the literature. Pinterest has a catalog of several billion pins that the recommender system can choose from. However, classical recommender systems that consider catalogs that only contain millions of items . In contrast, Pinterest recommends from a catalog of billions of items, which makes the recommendations problem much more challenging. A second important challenge is posed by the fact that recommendations have to be calculated on demand and in real-time. This real-time requirement (i.e., sub 100 millisecond latency per recommendation request) is crucial for two reasons:
(1) Users prefer recommendations responsive to their behavior, thus recommendations have to be computed on demand and in real-time so the system can instantaneously react to changes in user behavior and intent.
(2) Real-time requirement also brings a drastic change in the design of the entire system. For example, even if recommendations would only take one second to compute, such times are too long for the user to wait. In turn, this would mean that recommendations for all users would have to be precomputed and materialized on a daily schedule. Moreover, the total number of registered users is usually much larger than the number of daily active users, so a lot of time and resources would be wasted updating recommendations for inactive users.
Present work: Pixie. Here we present Pixie, a scalable real-time graph-based recommendation system deployed at Pinterest. Currently, pins recommended by Pixie represent more than 80% of all user engagement at Pinterest. In an A/B tests recommendations provided by Pixie increase per pin engagement by up to 50% higher compared to the previous Pinterest recommendation systems. Users at Pinterest view pins and curate them into collections called boards. This way a single pin can be saved by thousands of users into tens of thousands of deferent boards. For example, the same recipe pin could be saved by different users to several different boards such as recipes, quick to cook, vegetarian or summer recipes .This manual curation mechanism provides a great source for recommendations, because curation captures the multi-faceted relationships between objects. With hundreds of millions of users manually categorizing/classifying pins into boards we obtain an object graph of multi-faceted relationships between pins. Thus, we can think of Pinterest as a giant human curated bipartite graph of 7 billion pins and boards, and over 100 billion edges 1 . We use this bipartite graph of pins and boards to generate recommendations. As a user interacts with pins at Pinterest our method uses these pins to create a query set Q of pins each with its own weight. The query set is user-specific and changes dynamically . It can contain most recently interacted pins as well as pins from long time ago. Given the query Q, we then generate recommendations using Pixie Random Walk algorithm. Because the walk visits both boards as well as pins both types of objects can be recommended to the user. Furthermore, the algorithm is fast, scalable, and runs in constant time that is independent of the input graph size. Our novel Pixie Random Walk algorithm includes the following innovations, which are critical for providing high-quality recommendations: (1) We bias the Pixie Random Walk in a user-speciic way, for example, based on the topic and language of the user;
(2) We allow for multiple query pins with different importance weights, which allows us to capture entire context of the user’s previous behavior;
(3) Our method combines results from multiple independent random walks in such a way that it rewards recommendations that are related to multiple query pins. In combinations with (2) this leads to more relevant recommendations;
(4) Our Pixie Random Walk uses special convergence criteria which allows for early stopping and is crucial for achieving real-time performance and throughput; Last,
(5) our Pixie algorithm allows for recommending both pins as well as boards, which helps solving the cold-start problem.
To recommend fresh new pins Pixie irst recommends boards (rather than pins) and then serves the new pins saved to those boards. In addition, we also develop a graph curation strategy that further increases the quality of recommendations by 58%. This curation also lowers the size of the graph by a factor of six which further improves runtime performance of Pixie. Our Pixie algorithm has several important advantages. The algorithm offers the flexibility to dynamically bias the walk. For example, in Pixie we bias the walk to prefer to recommend content local to the language of the user, which boosts user engagement. Pixie allows for computing recommendations based on multiple query pins. Furthermore, we can vary the walk length to make trade-off between broader and narrower recommendations. For areas of Pinterest intended to provide unexpected, exploratory recommendations Pixie can walk farther in the graph for more diverse recommendations. On the other hand, to generate narrowly focused and topical recommendations, Pixie can use shorter walks. Pixie Random Walk also has several advantages over traditional random walks (or over simply counting the number of common neighbors): In classical random walks low degree nodes with fewer edges contribute less signal. This is undesirable because smaller boards (lower degree nodes) tend to be more topically focused and are more likely to produce highly relevant recommendations. In Pixie Random Walk we solve this by boosting the impact of smaller boards. And last, Pixie Random Walk is efficient and runs in constant time that is independent of the input graph size. Deployment of Pixie is facilitated by the availability of large RAM machines. In particular, we use a cluster of Amazon AWS r3.8xlarge machines with 244GB RAM. We it the pruned Pinterest graph of 3 billion nodes and 17 billion edges into about 120 GB of main memory. This gives several important beneits:
(1) Random walk does not have to cross machines which brings huge performance benefits;
(2) The system can answer queries in real-time and multiple walks can be executed on the graph in parallel; And,
(3) Pixie can be scaled and parallelized by simply adding more machines to the cluster. Overall, Pixie takes less than 60 milliseconds (99-percentile latency) to produce recommendations.
Today, a single Pixie server can serve about 1,200 recommendation requests per second, and the overall cluster is serving nearly 100,000 recommendation requests per second. Pixie is written in C++ and is built on top of the Stanford Network Analysis Platform (SNAP).
For example, “spicy tofu with coconut sauce” is recommended to me when I click on “easy thai shrimp soup”. Thai cuisine is very closely related to tofu, coconut sauce and spicy spice therefore it makes a great recommendation. Chances are that I’ll be interested in the thai tofu recipe if I am interested in the recipe of thai shrimp soup.