Patrick Galbraith

Web developer - Adelaide, Australia

RAGEagain 89

Re-experience ABC TV's RAGE again...

rageagain.com
www.rageagain.com

 

Rage is a popular all-night Australian music video program broadcast on ABC1 on Friday nights, Saturday mornings and Saturday nights. It was first screened on the weekend of Friday, 17 April 1987.

Since 1998 rage has posted all their episode playlists online. So I created a website that combines these playlists with YouTube allowing you to travel back through time and re-experience rage’s recent history.

Please note: all the matching of track to video is done programmatically so some tracks particularly older ones may not be available (depending on whether someone has uploaded them).

Behind the Scenes (v2 – 2020)

The original version of the site was written using the Lithium PHP Framework in 2012. The site ran well until 2019 when a number of issues cropped up. So I had a plan in mind for a re-envisioned version to address the following issues:

  • Over the years Google has been reducing the number of requests that can be made to the Youtube search API. When the site was first developed the project had millions of requests in quota. Now it is down to 100s per month. I tried contacting the black hole that is Google support with no luck. So I needed an alternative to using the Youtube search API.
  • The ABC website scraper needs to be rewritten since the ABC site has changed.
  • I wanted to make the website static, and have it connect to a serverless backend to avoid having to host and maintain a server.

Given these requirements I decided on the following plan:

  1. Write a new API backend hosted on Cloudflare workers.
  2. Dump the playlist data stored in MySQL into flat files (JSON).
  3. Use scheduled Github workers to scrape the ABC website and update the data directory.
  4. Convert the frontend to be a static single page application (SPA).

The project is broken into three parts; the API backend, the static frontend SPA, and the ABC scraper script run via Github workers.

For the backend API my initial plan was to write this using Rust (https://www.rust-lang.org/). Cloudflare workers have good support for Rust and I enjoy working with the language.

However I ran into some issues writing a YouTube search scraper and given that there was already a NodeJS package for this purpose it didn’t seem like a great idea to re-invent the wheel. So I ended up writing the final version using Typescript.

The logic is fairly simple when a request is made for a specific music video, it will first check for a cached result stored in Cloudflare KV storage. If the cached data cannot be found it will then go off and grab the data from YouTube.

There is also another endpoint that proxies the data from Github, providing the correct CORS headers for the frontend.

All in all it was fun to re-visit the project.

Checkout the interview with ABC about the site for more details https://web.archive.org/web/20150402201331/https://open.abc.net.au/explore/31540

Source code and data

Scraping the data (particularly the 90s web pages) wasn’t easy due to broken and inconsistent markup so I have posted a database dump in case anyone else wants to play with the data.

Playlists: 1,694
Tracks: 201,316
Approx. Play Time: 1 year, 173 days, 5 hours, 46 mins

https://github.com/patrickgalbraith/rageagain (Source code and data repo)

Changelog

17th January 2020

Features

12th June 2017

Features

  • Adjusted Youtube search algorithm to ignore Aria chart information and de-rank reaction and karaoke videos

4th March 2013

Features

  • Migrated to new sever (initial load should be significantly faster)
  • Finally implemented listing by special
  • Updated to jQuery 1.9.1

3rd September 2012

Features

  • Updated to include latest playlists
  • New special playlist; Top 200 which plays the most played tracks in ascending order (i.e. most played first).
  • Individual playlist/tracks are now bookmarkable and browser back button works
  • Keyboard shortcuts (spacebar = pause/play, arrow left = prev, arrow right = next)
    • Note: If you click on the video you may need to click outside of video for the keyboard events to trigger again. This is because the video will capture any keyboard events.
  • Added anchor links to quickly skip to specific year.
  • Added warning for users on unsupported/outdated browsers.
  • Updated jquery to 1.8.1 (mainly fixes player not appearing in Internet Explorer 8/9)

Bug Fixes

  • Fixed issue with tracklist being capped at 200 which caused the first tracks to not be played.
  • Massive database overhaul
    • Fixed broken 1999 playlists
    • Added 30+ missing/incomplete playlists
    • Removed a few duplicate playlists
    • Removed all duplicate tracks (some tracklists had been entered twice)
    • Fixed tracks containing unescaped html entites and tags

29th August 2012

First public release

28th July 2012

Private release

COMMENTS

Leave a reply

  1. Another one..

    Meant to play Taxiride – Everywhere you go

    Instead is plays Crowded House.

    Bad coding.

    • Hi Tim,

      It looks like an issue with the Top 50 songs because the track name contains the chart position information. I have updated the algorithm to remove this information and adjusted the search ranking.

      Hopefully it is working properly again.

      – Patrick

  2. Michael said

    Hi,

    Is there any way to download the specials and save a copy?
    Many years ago when Michael Hutchence died there was a RAGE special with heaps of INXS songs. I taped it on VHS and then years later taped over it. Haven’t heard the end of from the MRS ever since.
    I see in your listing by special the INXS special is there so was hoping I could get a copy to end this saga.
    Thanks and love the site.

  3. Dan John said

    Hi Patrick,

    I absolutely love this site but was just wondering how often it gets updated or when you were going to run the scraper again because there’s been some fantastic guest programmers lately. Also, I was wondering why it will briefly play the beginning of the previous song when moving onto the next in the list. I wasn’t sure if it was just on my computer or was something that needed tweaking

    Once again,
    Cheers for the spanking site
    Dan

  4. Justin said

    Hey Patrick,

    You haven’t updated Rage Again in a while,

    how’s everything going?

    Are you going to update this into the future?

  5. Thanks for this service – I use it every weekend. Any chance you will update the site? Seems to have stalled in May.

  6. Frank said

    Hi Patrick thanks for all your work on this, I really enjoy it :)

  7. Daniel said

    Hi Patrick, awesome website. Just wanted to let you know that the top 200 link appears to be broken.

    Thanks!

  8. Dan John said

    Absolutely love this site but am devastated by it’s inaction lately. If you are too time poor I would be more than happy to take over the reins with a little tutoring so we can get this magnificent site up and running again for alll the multitude of fans out there

  9. Tash said

    PLEASE get this site up and running again, i know it was working last year and it was the BEST site to rewatch the specials! as abc iview dosent even replay half the music or the specials anymore, just a random dump of 5ish songs from that date… I really hope it gets back up and running, even just for the old episodes.

  10. I was planning on opening the site up to contributions. However I ran into two major issues.

    1-Over the years Google has been reducing the number of requests that can be made to the Youtube search API. When the site was first developed I had millions of requests in my account quota. Now it is down to 100s per month. I’ve tried contacting the black hole that is Google support with no luck. So I need an alternative to using the Youtube search API. I have one in mind but it requires a bit of work to integrate.

    2-The scraper needs to be rewritten since the ABC site has changed.

    Site is back online after a complete rewrite.

  11. Dan John said

    Hi Patrick,
    I was wondering if there was something I needed to do with my MacBook pro because every time choose a date or a special it just plays the Rage theme with all the songs scrolling really fast in the top right hand corner with playing a single song, or is the site down again

Leave a Reply to Tim Cancel reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

PINGBACKS