Top Songs Sentiment Analysis

by Sarah Fox and Emily Powers

Overview

In this project, we want to use Billboard's Top 100 Songs from 1950-2012 to create a topic modeling algorithm to group songs by theme. From there, we want to analyze the lyrics of songs to find popular themes or genres. We ultimately want to use this information to determine the negative or positive connotation of songs and look at these trends over time.

Collaboration Plan

For the final Milestone project, Sarah Fox and Emily Powers will be working together. In order to collaborate, we will use Google Collab to simultaneously work on code and then download it as an ipynb file for submission. We plan to meet once a week to discuss progress and then meet over Zoom as needed. We will also schedule our meeting close to office hours so we can quickly address issues if they arise.

Background

The dataset we are interested in is a collection of song lyrics from the Billboard Top 100 songs from 1950 to 2012. Link to Dataset

We think this is an unique dataset because song data is usually related to its success like tracking its ranking on the Billboard chart; however, this data examines the content of the songs. Music has changed so much over that time period that it would be interesting to compare summary statistics on word usage in the 1950s and the 2000s. Through Natural Language Processing, we can find the most popular words overall and per decade. For our main question, we want to design a topic modeling algorithm to cluster each song by theme. This will tell us the most popular theme and expose other underlying genres or associations. Finally, the same algorithm can be used for sentiment analysis to classify songs as negative or positive. Our initial prediction is that songs have gotten more negative over time, but our project will give us a better understanding of the songs content.

About Our Dataset

We loaded in the Billboard Top 100 dataset and faced only one minor issue. The function read_csv identified 7 columns instead of 4 so we had to drop those columns which contained no data. The dataset contains 4 columns of Year, Artist, Song Name and Lyrics with 5148 rows. This is a great dataset for analysis because it is large and the lyrics column will be great for NLP analysis.

Exploratory Discussion

We want to use this data with Song Name, Arist, and Lyrics to find trends in these things over time. We want to explore average word counts and most common words over time to be able to create a NLP model of these lyrics and their meanings in songs. We also want to potentially explore more about the artist and verifying the Billboard rating of artists compared to other factors that could determine an artists success.

Hypothesis

Our hypothesis is that songs have gotten more negative in sentiment over time.

Data ETL

Exploratory Analysis

Average Number of Words over Time

We first thought it would be interesting to explore the average number of words per song over time. We found that the average number of words peaked in the early 2000s and has steeply declining since. This trend could signify that songs have gotten longer over time or specifc genres of music where words are sang faster, such as rap, could've gained in popularity around this peak time.

Top 10 Artists

We also wanted to explore who the best artists were. We counted the number of times an artist has a song in the top 100 and selected the top 10. With this information, we could potentially match to top selling artists during the same time period to see if the Billboard rankings match up with the top sales. We could take a direction to verify the validity of the Billboard rankings based on artists other achivements whether it be sales, awards, streams, etc. to see if the Billboard rankings have valuable meaning behind them.

If we pull in a list of Top 10 Best-Selling Artists we can see how well the rankings match up

source: https://ledgernote.com/blog/interesting/best-selling-artists-of-all-time/

We can see that 6/10 of the Top Artists match the Billboard Rankings. The artists the matched printed above. It seems the rankings are not based solely on sales which is expected, so the question becomes what other factors could we pull in to match these rankings better?

Songs Appearing on Chart over Multiple Years

Another avenue we wanted to explore was what songs were in the top charts over multiple years. This could give us insight to their popularity over fad songs that are only popular for a year. Are these songs considered to be classics? Should they be considered classics just because of their continuous ranking or do we have a more culture lens over "classic" songs?

We wanted to explore the most frequently occuring words in songs to set us up for NLP where we can begin to make conclusions about the themes behind these words. We removed common filler words (stopwords) that have little to no impact on determining the context of the lyrics to reduce the corpus size. We are looking for words with more theme significance or meaning behind them to be able to draw conclusions on song themes over time.

sources: https://stackoverflow.com/questions/22219004/how-to-group-dataframe-rows-into-list-in-pandas-groupby

https://stackoverflow.com/questions/952914/how-to-make-a-flat-list-out-of-a-list-of-lists

https://docs.python.org/3/library/collections.html#collections.Counter

https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html

We can see that the word "love" appears a lot. For further analysis, let's track its usage over time. In order to account for the increase in average song length over time, we used the proportion of the count of occurrences of 'love' out of the total number of lyrics for the year.

We can see that the usage of the word love peaked around 1980 but was at an all time low in the early 2000s. This is interesting because song length was the highest in the early 2000s, so there being such a low proportion of "love" is signifcant.

Model

With this data, we plan to use NLP to build our sentiment analysis. Sentiment analysis identifies the positivity or negativity of lyrics which can answer the question if sounds have gotten more negative over time. Each word is classfied as positive (good i.e. love) or negative (bad i.e. death) which can be used to predict the feeling of the song.

We used the Natural Language Toolkit (nltk) to run this experiment, sepcifically the Sentiment Intensity Analyzer.

To quantify the sentiment analysis we used VADER (Valence Aware Dictionary and Sentiment Reasoner) scores in conjunction with the Sentiment Intensity Analyzer. This method returns a dictionary with a positive, negative, netural, and compound score (normalization of positive, negative, and neutral scores) for each song. We used compound score which gives overall sentiment as well as the intensity of the emotion. The way songs are classfied are as follows:

Positive if compound >= 0.5

Neutral if -0.5 < compound < 0.5

Negative if -0.5 >= compound

source: https://www.analyticsvidhya.com/blog/2021/06/rule-based-sentiment-analysis-in-python/

Now that we have a VADER score for each song, we can group the songs by year and take the average score for that year so get a general idea of how positive or negative lyrics were for that year. We chose to use the mean VADER score for that year because that would give us the average sentiment of top 100 popular songs for that year.

We can see from the trendline that the compound sentiment score has gone down over time which means that according to the VADER score analysis songs have gotten more negative since 1950.

Additionally, we want to break down this compound score to see the trendlines for songs that were categorized as Positive over time (VADER score >= 0.5) and Negative over time (VADER score <= -0.5)

We can see a negative trendline for positive songs showing that their positive sentiment (even for just the songs that are categorized as positive) has decreased over time.

When looking at our negative sentiment songs, the negative proportion has increased over time suggesting songs have gotten more negative over time (not necessarily that there are more negative songs now but we are measuring the extent to which the lyrics have become negative over time).

As part of our Exploratory Data Analysis was tracking Top Artists, we were curious to see what the sentiment analysis of these top artists were.

Interestingly enough, all of the top artists have overwhelmingly more positive songs than negative or neutral ones. However, looking deeper into this chart, we can see that most of these artists were popular decades ago. Rihanna, a more current artist, shows far fewer positive songs than someone like Elvis Presley who peaked in popularity in the 50s and 60s.

Conclusion

Our original hypothesis with our sentiment analysis was that songs have gotten more negative over time. Through our NLP model we found this to be true. In order to verify these results, we visualized not only the compound scores but also the trends over time for the songs already categorized into being positive or negative. We calculated a simple regression on each of these plots and found the slopes of those lines supported our hypothesis.

There are a variety of inclinations that we had as to why this would be the case (psychological priming, cultural acceptance of negativity, remembering negative stimuli better than positive ones, etc). We could easily expand upon this study to see if we could attribute a cause to the phenomenon we observed of increased lyrical negativity.