Fred and Loathing on The Internet

Nothing Too Fancy. Just a Data Shed

2023-Jul-19

Updates involved some fixes to my hacky hard-coded URL paths to the Spotify web player. Some of the links that go to /tracks/ fail because Spotify sometimes appends another code after the track ID; to distinguish different recordings of the same song name. My links fail because I do not use that extra code to make my links. I am not too concerned about it.

All the progress I had hoped to make from unleashing Spark 3.4 running in local mode on a Linux VM using all my 16 CPU threads on my paltry three years of Spotify listening data. This course ran aground when the JDBC connector auto infers my Sqlite data to the correct ‘date’ type. However, no matter what action is performed on that column results in a Py4j error leading to a java.parser error saying it does not like the first YYYY-MM-DD entry. I try to give it a custom schema, cast it as string, and give some date formatting options to the spark reader. Me and ChatGPT ran out of ideas that don’t involve looking into the JDBC and JAR file I used to let Spark connect to my Sqlite database. If I drop those date columns I can do my normal pySpark stuff, but the date column is essential for all but one of my tables…

Instead of trying to solve that date inference problem, I figured I’d throw at it some data I have yet to touch using my previous non-Spark methods: my Spotify playlists. I have a protocol for assembling playlists of new music that usually cover twelve weeks. I could make a ETL>Analytics process going from the API call for one playlist ID that results in a cleaned up table describing the playlist.

The introduction to Bullit with Steve McQueen is captivating. I love the score and the work of that movie’s composer. Seeing it used with daft cinematography gave me the good kind of chills. I was too sleepy to catch the movie’s particulars. Steve McQueen looked to be a cop driving a Mustang around San Francisco’s hilly streets. I do remember a particularly long post-coital kiss between McQueen and his comely girlfriend. I know it was a favorite of my father’s so I figured I will give it a more thorough look.