Group members: Liam Keane, Will Shrestha, Griffin Momsen-Hudson, Will Hassell, Nithin Poreddy
Progress!
Since our last written update, we were able to gain access to Krlx’s song log database. Liam did a wee bit of digging and found the credentials for accessing the database hosting the song log, and we exported that SQL table into a large csv (there are around three-hundred thousand songs played since 2013). We have also organized and pruned that data into more manageable chunks.
Tools & Techniques
Alongside the SQL database and the Linux server, we used a few python scripts to parse the data into more manageable and organized chunks. We split the original csv into several accompanying files. The original contained the columns of: Id number, show title, song title, song artist, its album, whether it was a new song (yes/no), and finally a timestamp.
We had two python scripts to order songs and their frequencies and artists and their frequencies for each year since 2013. That way we can track Krlx’s musical tendencies from year to year. Even withut the help of additional software, we could pick out strange patterns and outliers (we’ll explore those later on in our project).
Organizing the Data
There were a few choices to make before actually performing analysis: what pieces of the data do we actually want to use and what can story can we tell with them? The album and ‘new song’ category were optional fields for Djs to fill out, so they would not be reliable source for data analysis. We knew we wanted to focus on Krlx music taste throughout the years, so we decided to narrow our focus on songs and their artists.
Problems
After writing a few scripts to mess with the data, we found there was quite a bit of Dj ‘error’. They would not capitalize many artist names, or they would use hyphens instead of normally space separated names. The best way we found to fix this problem was remove all elements with a frequency of five or less from the the artists and song data. It follows that typos have a lower frequency because Djs generally write the correct artist/song names and the typos will tend to be unique. Thankfully, this problem did not change our intended plans.
Deliverables
Our project is still on track. The next steps will be finding another data source to compare to (ie. the most popular songs/artists on U.S radio), and/or visualizing song and artist frequency for different years in Krlx. Firstly, we will explore easily available software (perhaps tools we have used in this class) to create these visualizations, or we may even explore some JavaScript data viz libraries.
Personal Messages:
Liam: I am following my outlined role in the charter by organizing and pruning the data for analysis and making sure our group remains on track. From here, I will look into how we will visualize our data; I’m leaning towards messing around with chart.js to make cool visuals.
Nithin: Now that we have the data, I can hopefully help with condensing the data down to only the points we are analyzing, if needed, and try and find visualization tools to model the data aesthetically while being easy to interpret.
Will H: I am going to be looking at other college/magazines lists of popular songs and artists to attempt to see if there are any similarities at all to those of KRLX over the years, as well as going through the data and finding some interesting outliers that could be fun to talk about in our project.
Will S: Now that we know what the data looks like, I will brainstorm some more specific ways we could progress with this project. I think I’m going to look at mostly what has changed and what hasn’t changed over the years that the song log covers for a start. Some things I might focus on are artist and song popularity per year and over several years.
Griffin: Now that we have the data, I can help look and see what potential trends in the broader musical landscape may correspond with, or may disagree with, our KRLX data. I can also help try to understand why certain songs were played a certain number of times, and more than we might expect.
6 thoughts on “Krlx Killa Beats – Final Project Update”
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
Nice job, everyone! This project looks really exciting, I look forward to seeing its final form. I’m really impressed with how you were able to get all of the data gathered and cleaned. It seems like it was a complex process, and am glad that it worked out effectively. Best of luck on the next steps!
I really admire how you guys are able to quickly find the exact data set needed for your project and the use of Python to create customized algorithms. For the spelling errors in the artist’s names, may I recommend you guys to use OpenRefine for a final check? It has helped me to find lots of subtle errors and by merging, the data wouldn’t have to be deleted like you guys have done so the final result would be more correct.
This is such a cool project! It’s awesome that you guys got exactly the perfect data set for your project. It’s also really cool that you guys are making use of several different programming languages and fitting them to your necessary needs. Great work on the project so far, you guys are doing awesome!
This is going to be really awesome to see! In my group’s project, our collective lack of programming skills has held us back in terms of how much data we can handle for our project, so I’m impressed with how you are using your team’s skills to tackle an enormous dataset and make sense of it. Great job!
I’m curious about your decision to just omit certain typo cases from your artists/song data. Would this be a case where you could just run the data through Open Refine and clean up any typos that the DJ’s might have made? Other than that, this looks really good!
I really like the project idea. Our project does not have much data organization. We are more focused on the process in how we effectively capture our data. It was cool reading about all the processes you are taking to organize your refine your data. Good job! I am excited to see the results of your project.