Creating a recommendation system using deep learning
Optimizing online advertisements with convolutional neural networks
A breif note on the goal of this project: during the process of learning more about deep learning, I was fascinated by how it could be utilized in recommendation systems. Part of my inspiration comes from reading more about how deep learning is currently utilized in the retail industry— for example as Stitch Fix has an amazing web site where they elaborate on algorithms that they use. As I was thinking about more potential applications for deep learning, I came up with an idea how great would it be to use it to help make online advertisements more relevant by having their images match or reflect page content. Brands, fashion magazines, ad agencies, and, of course, customers can all be benefited by better-targeted ads. With that being said, I set out to see if I can use deep learning and other techniques to create a better ad recommendation system that in return can help to increase profits for brands by reducing user cost of acquisition.
To get a better idea of how large of an issue this is, I will briefly dive into the online advertising market and its size. The online advertising market is massive. $280 billion dollars will be spent on online advertisements across the world this (2020) year alone. In 2019 more money was spent in the US on online advertisements than on all other ad types combined (print, newspaper, TV, and radio).
As of now, online advertisements are far from perfect. In preparation for this project, I scrolled through a few websites looking for what ads they showed.
The above article in People Magazine shows a linen skirt yet comes with ads for a lawnmower and weed killer (perhaps these marketing departments know something I don’t, but I believe more relevant clothing ads would be more relevant). Targeted ads generally perform better. Companies can get better conversion rates and make more profit if they are able to match ads to content.
To archive the goal of my project I decided to build an image recommendation system. To do this I scraped over 35,000 images from a clothing website (I chose ShopStyle website) in various categories such as dresses and tops using selenium and beautiful soup (please, find an example of my code on scraping using this link). I then uploaded them to Google Colab where I used Keras to train my model.
Analysis — building a model
For my analysis, I used the Transfer learning technique — in case you are unfamiliar with Transfer Learning, it generally takes features learned on one problem and then levers them on a new, similar problem. For example, the knowledge gained while learning to recognize mules could then be applied when trying to recognize donkeys. This technique comes in handy if the data set is fairly small for the model to be trained from scratch. To get a better grasp on deep learning and transfer learning in a particular couple of the resources that I found eminently useful could be found following this and that links. There are numerous of pre-trained models out there for a use such as ResNet, Inception, Inception-ResNet, Xception I ended up using VGG16 model architecture that was trained on ImageNet dataset (1.4 million labeled images and 1000 different classes). For this, I used the convolutional base of this model that holds generic feature maps (such as visual edges, colors, and textures) and added two fully connected layers and one dropout layer as a regularization technique to prevent overfitting of my model, and finished with a densely-connected classifier that predicts to which class/category of a clothing input image belongs. The full jupyter notebook with a code for building a model could be found on this link to my GitHub page.
# snipped of the code for building a modelconv_base = VGG16(weights='imagenet',
include_top=False, # excluding top layers for using only convolutional base
input_shape=(224, 224, 3))# Freeze convolutional layersfor layer in conv_base.layers:
layer.trainable = False
# Establishing a newly fully connected block
x = conv_base.output
x = Flatten()(x) # flatten from convolution tensor output
x = Dense(3500, activation='relu')(x)
x = Dense(2500, activation='relu')(x)
x = Dropout(0.5)(x)
predictions = Dense(5, activation='softmax')(x) # should match number of classes predicted
model_final = Model(inputs=conv_base.input, outputs=predictions)
model_final.compile(loss="categorical_crossentropy", optimizer='adam', metrics=["acc"])
By training this model on my dataset I archived an accuracy score of 90%.
Recommendation system
However, since I wanted to build a recommendation system for advertisement I wasn’t interested in just a rough prediction of the categories of clothing. To archive that, I used the intermediate layer of my model to extract features of each image of my dataset to present it as a set of vectors. To further find a similar few to the input image. For increasing the prediction power of my model I also decided to use the PCA dimension reduction technique to make my set of vectors less spars so that the number of features is reduced from 2500 to 10 components.
The workflow of my recommendation system looks as such: I resized and converted the input image into NumPy array, extracted features from the intermediate layer, transformed it into a less sparse vector using PCA, and found closest/similar vectors from my dataset to the input image by using pairwise distance method calculating the distance between the input vector and set of vectors in my dataset with cosine similarity metric.
To test my model how well it could be used for the recommendation system I ran a few article photos on my end and got pretty good recommendations for each photo! One of the examples is bellow (for more examples, please check my jupyter notebook for the recommendation system on my GitHub on this link):
From the uploaded photo (the center one) my model was able to identify that it was a dress and then gave suggestions of similar dresses from my data set by silhouette, fabric patterns, and even palette.
Playing around with the recommendation system and trying various photos from different articles I put together how magazine’s webpage ideally should look like.
Conclusion
Deep learning algorithms are very powerful tools and fun to work with. There are so much more for me to learn in the future. Despite going through some of the challenges during my analysis, I got pretty amazing results!
- I built a convolutional neural network model that can classify pieces of clothing by category
- I then built a recommendation system using feature extraction from this model
- The algorithm of this model can be applied for better ad recommendations on websites that will be beneficial for brands and ad agencies