Rage is a popular all-night Australian music video program broadcast on ABC1 on Friday nights, Saturday mornings and Saturday nights. It was first screened on the weekend of Friday, 17 April 1987.
Since 1998 rage has posted all their episode playlists online. So I created a website that combines these playlists with YouTube allowing you to travel back through time and re-experience rage’s recent history.
Please note: all the matching of track to video is done programmatically so some tracks particularly older ones may not be available (depending on whether someone has uploaded them).
Behind the Scenes (v2 – 2020)
The original version of the site was written using the Lithium PHP Framework in 2012. The site ran well until 2019 when a number of issues cropped up. So I had a plan in mind for a re-envisioned version to address the following issues:
- Over the years Google has been reducing the number of requests that can be made to the Youtube search API. When the site was first developed the project had millions of requests in quota. Now it is down to 100s per month. I tried contacting the black hole that is Google support with no luck. So I needed an alternative to using the Youtube search API.
- The ABC website scraper needs to be rewritten since the ABC site has changed.
- I wanted to make the website static, and have it connect to a serverless backend to avoid having to host and maintain a server.
Given these requirements I decided on the following plan:
- Write a new API backend hosted on Cloudflare workers.
- Dump the playlist data stored in MySQL into flat files (JSON).
- Use scheduled Github workers to scrape the ABC website and update the data directory.
- Convert the frontend to be a static single page application (SPA).
The project is broken into three parts; the API backend, the static frontend SPA, and the ABC scraper script run via Github workers.
For the backend API my initial plan was to write this using Rust (https://www.rust-lang.org/). Cloudflare workers have good support for Rust and I enjoy working with the language.
However I ran into some issues writing a YouTube search scraper and given that there was already a NodeJS package for this purpose it didn’t seem like a great idea to re-invent the wheel. So I ended up writing the final version using Typescript.
The logic is fairly simple when a request is made for a specific music video, it will first check for a cached result stored in Cloudflare KV storage. If the cached data cannot be found it will then go off and grab the data from YouTube.
There is also another endpoint that proxies the data from Github, providing the correct CORS headers for the frontend.
All in all it was fun to re-visit the project.
Checkout the interview with ABC about the site for more details https://web.archive.org/web/20150402201331/https://open.abc.net.au/explore/31540
Source code and data
Scraping the data (particularly the 90s web pages) wasn’t easy due to broken and inconsistent markup so I have posted a database dump in case anyone else wants to play with the data.
Approx. Play Time: 1 year, 173 days, 5 hours, 46 mins
https://github.com/patrickgalbraith/rageagain (Source code and data repo)
17th January 2020
- RAGEAgain is back after a long hiatus!
- Fixing the site required a complete rewrite. Source code is now available here https://github.com/patrickgalbraith/rageagain.
12th June 2017
- Adjusted Youtube search algorithm to ignore Aria chart information and de-rank reaction and karaoke videos
4th March 2013
- Migrated to new sever (initial load should be significantly faster)
- Finally implemented listing by special
- Updated to jQuery 1.9.1
3rd September 2012
- Updated to include latest playlists
- New special playlist; Top 200 which plays the most played tracks in ascending order (i.e. most played first).
- Individual playlist/tracks are now bookmarkable and browser back button works
- Keyboard shortcuts (spacebar = pause/play, arrow left = prev, arrow right = next)
- Note: If you click on the video you may need to click outside of video for the keyboard events to trigger again. This is because the video will capture any keyboard events.
- Added anchor links to quickly skip to specific year.
- Added warning for users on unsupported/outdated browsers.
- Updated jquery to 1.8.1 (mainly fixes player not appearing in Internet Explorer 8/9)
- Fixed issue with tracklist being capped at 200 which caused the first tracks to not be played.
- Massive database overhaul
- Fixed broken 1999 playlists
- Added 30+ missing/incomplete playlists
- Removed a few duplicate playlists
- Removed all duplicate tracks (some tracklists had been entered twice)
- Fixed tracks containing unescaped html entites and tags
29th August 2012
First public release
28th July 2012