Music

Plan

I wanted to build out a project to test my analyst knowledge and build something to combine SQL, Python, Tableau, and Analysis. My aim was to uncover trends within a niche and personally intriguing market. I also wanted to play with the Spotify API and thus decided to analyze trends within the Electronic Music industry.

I wanted to limit my dataset so I decided to focus on artists who have performed at the Electric Daisy Carnival in Las Vegas, believing this selection would encompass both established and emerging talents in the electronic music scene with the barrier to entry that they are somewhat commercially popular. In doing so, I want to acknowledge that my analysis is constrained by various factors that, for a detailed understanding of these limitations and potential solutions, I will provide documentation on analysis limitations below.

I broke down the project into phases, with each phase building functionality on the previous one. Initially, I will develop a rudimentary (I studied accounting, not CS!) Python script to call the Spotify API and retrieve genres and other information for a list of artists. I will then compile a list of artists who have performed at EDC and scrape their year of performance and set time. (The significance of set time lies in its potential correlation with popular genres, but I’ve kept this out of scope for now.) While not entirely necessary due to the small dataset, I aim to populate this data into a database to gain practice. Following this, I will execute SQL queries to pull the final dataset into Tableau Public (Im cheap). The focus of the project is to generate a Tableau dashboard with metrics, with the main metric being the top genres by stage. Visually, I envision the dashboard as an interactive map of the festival, allowing users to filter the data by stage.

Step 1 – The backbone

import pandas as pd
from dotenv import load_dotenv
import os
import base64
from requests import post, get
import json

load_dotenv()

client_id = os.getenv("CLIENT_ID")
client_secret = os.getenv("CLIENT_SECRET")

def get_token():
    auth_string = client_id + ":" + client_secret
    auth_bytes = auth_string.encode("utf-8")
    auth_base64 = str(base64.b64encode(auth_bytes), "utf-8")

    url = "https://accounts.spotify.com/api/token"
    headers = {
        "Authorization": "Basic " + auth_base64,
        "Content-Type": "application/x-www-form-urlencoded"
    }
    data = {"grant_type": "client_credentials"}
    result = post(url, headers=headers, data=data)
    json_result = json.loads(result.content)
    token = json_result["access_token"]
    return token

def get_auth_header(token):
    return {"Authorization": "Bearer " + token}

def search_for_artist(token, artist_name):
    url = "https://api.spotify.com/v1/search"
    headers = get_auth_header(token)
    query = f"?q={artist_name}&type=artist&limit=1"
    query_url = url + query
    result = get(query_url,headers=headers)
    json_result = json.loads(result.content)["artists"]["items"]
    if len(json_result) == 0:
        print(f"No artist found for {artist_name}")
        return None
    return json_result[0]

def get_genres_by_artists(token, artist_id):
    url = f"https://api.spotify.com/v1/artists/{artist_id}"
    headers = get_auth_header(token)
    result = get(url, headers=headers)
    json_result = json.loads(result.content)["genres"]
    return json_result

def get_genres_for_artists_from_excel(input_file, output_file):
    token = get_token()
    data = pd.read_excel(input_file, usecols=['Artist'])
    genres_list = []

    for artist_name in data['Artist']:
        artist_result = search_for_artist(token, artist_name)
        if artist_result:
            genres = get_genres_by_artists(token, artist_result["id"])
            genres_list.append(", ".join(genres))
        else:
            genres_list.append("Not Found")

    data['Genres'] = genres_list
    data.to_excel(output_file, index=False)

if __name__ == "__main__":
    input_file = "/Users/devicente/Desktop/Projects/SpotifyAPI/EDC Artists.xlsx"
    output_file = "/Users/devicente/Desktop/Projects/SpotifyAPI/output.xlsx"
    get_genres_for_artists_from_excel(input_file, output_file)
    print("It worked!")

I wrote the following Python script to take a workbook of artists, access the Spotify API and return information about the artists. This current version only returns the artist’s genre as defined by Spotify. This is the first issue I can see in terms of limitation of analysis. Genres are subjective. It can be argued that in using Spotify’s API, it would perform a baseline determination of genres such that it is not left to the subjectivity of a single person.

Step 2: Database

Data regarding artist rosters is readily available in nice Excel sheets provided by the great people of reddit. So I modified an SEO bot to scrape threads using chat-GPT. No need to over complicate the easy parts. I pulled the rosters from threads and spot checked the rosters (we may be losing a few last minute artist changes, but more on that later). We do want to over-complicate the fun stuff so over lunch I quickly sketched out a Entity Relationship Diagram for my data:

And here it is beautified:

Some considerations on this ERD draft. While I have experience in SQL, I don’t typically build databases from scratch. A few things I will look into that may already identify themselves as I am bringing this into postgreSQL.
-Probably need a relationship between EDC and stage
-Probably need to reconfigure date and year for EDC
-Probably could share some kind of composite key for Set

Need to combine data to have a better visual. Feel like – while this will work, we are losing clarity/flexibility/efficency.

Step 3: API Results

As anticipated, pushing my artist list through the Spotify API has returned some… interesting results. In my research I found the following from the man himself, Glenn McDonald, Spotify’s former data alchemist (I’m a huge fan Glenn, can’t wait for the book to drop!):

The genre tags you see in the API are for artists who we (Spotify) consider representative for those genres. Internally we have a larger mapping of less-representative artists to their probably-relevant genres…. Genre assignments are a combination of human curation and algorithmic extrapolation, so in most genres it’s quite possible for artists to qualify as representative (and thus appear in Sound or Intro playlists) based on listening patterns.

Glenn McDonald, 2022

Limiting the data to the 2024 EDC artist list, of the 236 Artists, 43 are not assigned a genre. The missing genre information for 43 out of 236 artists presents a significant challenge for accurate data analysis and interpretation. While utilizing the API was intended to reduce biases, the gaps in data re-introduces unanticipated biases and limitations. Implementing an effective mitigation strategy and acknowledging these gaps may help in managing the impact and improve the the derived results… if the data was perfect beyond these holes.

Step 4: Pressing forward (for Database Practice)

Despite the data issues above, I want to get the database practice in. I used PostgresSQL to build my database. Review my PostgresSQL ERD (wonky lines included!) and my SQL to populate the data. I want to provide a write up on the lessons learned in this practice, such as how we change methodology from the sketch above and how i’m not sure if we are implementing best practices as it relates to constraints. As mentioned in the previous chapter, project will be on hold until October while I file my 3rd busy season client (always the auditors-maid, never the Analyst).

CREATE TABLE Artists (
    ArtistID INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    ArtistName VARCHAR(100) NOT NULL
);

-- Create Genres Table
CREATE TABLE Genres (
    GenreID INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    GenreName VARCHAR(100) NOT NULL
);

CREATE TABLE ArtistGenres (
    ArtistID INT,
    GenreID INT,
    PRIMARY KEY (ArtistID, GenreID),
    FOREIGN KEY (ArtistID) REFERENCES Artists(ArtistID),
    FOREIGN KEY (GenreID) REFERENCES Genres(GenreID)
);

CREATE TABLE Stages (
    StageID INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    StageName VARCHAR(50) NOT NULL
);

CREATE TABLE Sets (
    SetID INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    ArtistID INT,
    StageID INT,
    SetTime TIME,
    Date DATE,
    FOREIGN KEY (ArtistID) REFERENCES Artists(ArtistID),
    FOREIGN KEY (StageID) REFERENCES Stages(StageID)
);

INSERT INTO Stages (StageName) VALUES
('Stereobloom'),
('Kinetic Field'),
('Neon Garden'),
('Wasteland'),
('Basspod'),
('Circuit Grounds');

INSERT INTO Artists (ArtistName) VALUES
('Abana'), ('Acraze').....

INSERT INTO Genres (GenreName) VALUES
('acid house'), 
('afro house'), 
('album rock'), 
('arab electronic').......

INSERT INTO Sets (ArtistID, StageID, SetTime, Date) VALUES
((SELECT ArtistID FROM Artists WHERE ArtistName = 'Abana'), (SELECT StageID FROM Stages WHERE StageName = 'Stereobloom'), '04:45', '2024-05-19'),
((SELECT ArtistID FROM Artists WHERE ArtistName = 'Acraze'), (SELECT StageID FROM Stages WHERE StageName = 'Kinetic Field'), '04:13', '2024-05-18').......