Simplest way

The easiest way to upload a file is to base64 encode the contents and pass it to the create method. Unfortunately this method is limited to around 35MB.

const uploadContentVersion = (fileName: string, file: Buffer): Promise<jsforce.RecordResult> =>
  connection.sobject('ContentVersion').create({
    PathOnClient : fileName,
    VersionData : file.toString('base64')
  })

Multipart form data

The trick here is that you need to encode the request correctly as per the example in the documentation.

Example request body

--boundary_string
Content-Disposition: form-data; name="entity_content";
Content-Type: application/json
 
{  
    "Name" : "Marketing Brochure Q1 - Sales",
    "Keywords" : "sales, marketing, first quarter"
}
 
--boundary_string
Content-Type: application/pdf
Content-Disposition: form-data; name="Body"; filename="2011Q1MktgBrochure.pdf"
 
Updated document binary data goes here.
 
--boundary_string--

Using the nodejs request library

The popular nodejs `request` library provides a formData option which can be used like below:

import request from 'request'
import jsForce from 'jsforce'

const uploadContentVersion = (metadata: ContentVersionMetadata, file: Buffer): Promise<jsForce.RecordResult> =>
  new Promise((resolve, reject) => {
    const connection: jsForce.Connection = // ...create jsForce connection

    request.post({
      url: connection.instanceUrl + '/services/data/v49.0/sobjects/ContentVersion',
      auth: {
        bearer: connection.accessToken
      },
      formData: {
        entity_content: {
          value: JSON.stringify(metadata),
          options: {
            contentType: 'application/json'
          }
        },
        VersionData: {
          value: file,
          options: {
            filename: metadata.PathOnClient,
            contentType: 'application/octet-stream'
          }
        }
      }
    }, (err, response) => {
      if (err)
        reject(err)

      resolve(JSON.parse(response.body))
    })
  })

Example using `form-data` library

You could also create the formData object manually like this.

  const formData = new FormData()

  formData.append('entity_content', JSON.stringify(metadata), { contentType: 'application/json' })
  formData.append('VersionData', file, {
    filename: metadata.PathOnClient,
    contentType: 'application/octet-stream'
  })

  // then make request using formData object

Full example attaching to Case

export const addFileToCase = async (caseId: string, fileName: string, file: Buffer): Promise<string> => {
  // We use the helper method created previously
  const uploadResult = await uploadContentVersion(file, {
    PathOnClient: fileName,

    // For updates you can also add the following params
    // ContentDocumentId: '',
    // ReasonForChange: ''
  })

  if (!uploadResult.success) {
    throw new Error(`Failed to upload file for case "${caseId}"`)
  }

  const contentDocument = await connection.sobject<{
    Id: string,
    ContentDocumentId: string
  }>('ContentVersion').retrieve(uploadResult.id)

  const linkResult = await connection.sobject<TRes>('ContentDocumentLink').create({
    ContentDocumentId: contentDocument.ContentDocumentId,
    LinkedEntityId: caseId,
    ShareType: 'V'
  })

  if (!linkResult.success) {
    throw new Error(`Failed to link content document for case "${caseId}"`)
  }

  return uploadResult.id
})

/// <summary>
/// Factory for creating ObjectInfo objects for ObjectInfoProvider.
/// </summary>
public class ObjectInfoFactory { ... }

“Factory for creating ObjectInfo objects for ObjectInfoProvider.” difficult to say what that means since it could be interpreted different ways. But wouldn’t it be great if we could see what the code does. Maybe we could even search the codebase and see how it is being used in other parts of the library.

Read the source Luke…

In my opinion one of the most beneficial things you can do is read the source code. Documentation is hard and there are many times I have worked with libraries where the documentation and examples are not up-to-date. Not to mention subtle unexpected behaviors that occur when working with other people’s code.

Reading source code has served me well over the years and as a result I have made a number of humble contributions to open source projects and had fun reading the code for projects like Chromium (Chrome Browser), Samba (SMB/CIFS), FFMPEG, and so on. I find that different ecosystems tend to come with an associated culture. Working more closely with Microsoft .NET recently I have found a bit of a new roadblock… the source code is not always readily available.

I tried to picture clusters of information as they moved through the computer. What did they look like? Ships, motorcycles. With the circuits like freeways. I kept dreaming of a world I thought I’d never see. And then one day, I got in.
Kevin Flynn – Tron Legacy

DnSpy (https://github.com/0xd4d/dnSpy) is a debugger and .NET assembly editor. You can use it to edit and debug assemblies even if you don’t have any source code available. Below I have an example of what it looks like viewing a dll from the DotNetNuke project.

That project (DotNetNuke) is open source so we don’t necessarily need to decompile it but it’s just an example. This is a tool I have used extensively in the last few months and has made working with closed source legacy code a lot less painful.

It is also possible to use dnSpy to patch dll’s. There are two main ways to do this you can edit the decompiled C# source code. However doing so will require recompilation and I have found that this seldom works. The other option is to edit the IL code of a method or property. This is a much better way as long as you don’t mind rolling up your sleeves and reading through the CIL instructions. Honestly it’s not that hard, at least it’s easier than using a hex editor.

In this follow up post I will look at how I created my custom Zombicide spawn card deck. First of all building the spawn card deck is a little trickier for two reasons. First is that I couldn’t find a digital source of all the cards such as the companion app that I used for the equipment deck. Secondly because each spawn card is uniquely numbered and there are 144 of them.

Guillotine Games has released high-res spawn card templates. However the cards are essentially blank. Not being one to give up that easily I decided there had to be a way to generate the spawn cards if I had all the card data. Fortunately are some digging online and cross checking with my own sets I managed to put together a spreadsheet containing all the data (this is included in the download).

Generating the images

Photoshop has a undocumented feature called Variables & Data Sets. This allows you to define layers as variables of two types. There are text replacement layers which will replace the text in the layer with the text in the data set, and there are visibility layers that will be visible or invisible based on the data. As you can see in the image below I created a spawn card template with all the layer variables defined. Then it was just a matter of writing a simple program to translate the source data found in the spreadsheet into a data set that Photoshop could use which you can see a sample of below.

number,berserk_bg,berserk_icon,toxic_bg,toxic_icon,blue_text,blue_walker,blue_runner,blue_fatty,blue_abomination,blue_dogz,yellow_text,yellow_walker,yellow_runner,yellow_fatty,yellow_abomination,yellow_dogz,orange_text,orange_walker,orange_runner,orange_fatty,orange_abomination,orange_dogz,red_text,red_walker,red_runner,red_fatty,red_abomination,red_dogz,ft_just_when_you,ft_sewer,ft_cut_them_down,ft_frag_time,ft_bite_me,ft_its_time
1,false,false,false,false,,false,false,false,false,false,2x,false,true,false,false,false,2x,false,true,false,false,false,5x,true,false,false,false,false,false,false,false,false,false,true
2,false,false,false,false,1x,true,false,false,false,false,1x,true,false,false,false,false,6x,true,false,false,false,false,7x,true,false,false,false,false,false,false,false,false,false,true
3,false,false,false,false,1x,true,false,false,false,false,1x,false,true,false,false,false,1x,false,false,true,false,false,4x,false,true,false,false,false,false,false,false,false,false,true
4,false,false,false,false,2x,true,false,false,false,false,3x,true,false,false,false,false,4x,true,false,false,false,false,2x,false,true,false,false,false,false,false,false,false,false,true

With the setup finished all 144 spawn card were generated based on the data set. The only issue was that the card number found in the top right was not on an angle and prefixed with the hash (#) symbol. I fixed that by using a custom Photoshop action which rotated the layer then aligned the hash icon to the left of the layer and then pushed it left a bit. You can see an example of the final cards below.

I printed my deck through PrinterStudio which for a 160 card deck came to $10.76. This gave me 16 cards to use for custom cards after re-printing all the base cards. I build some nasty custom cards based on the Hell On Earth set by WakkaWakkatronic http://boardgamegeek.com/filepage/86115/hell-earth-zombie-spawn-cards. I found these are a great way to mix up the game. How about spawning 15 walkers, or 8 runners at once! I also made some custom Lost Zombivor spawn cards which means I can play with the new Lost Zombivor rules. Both custom sets are included in the download.

Here is the deck type I used http://www.printerstudio.com/personalized/custom-blank-playing-cards-mini-size.html. The great thing is that once you have reprinted the base sets it is easy to print additional custom cards at the same size.

The finished product

Lately I’ve been playing a lot of the awesome hobby that is board games. The last few years have seen a renewed interest in board games likely aided by the internet and its ability to provide crowd funding for a range of new and exciting games. One of these is Zombicide which was funded in 2012 and raised a respectable $780k.

Zombicide is a thematic co-op game for 1 to 6 players. Each player controls between one (for 6 players) and four (solo game) survivors, humans trapped in a zombie-infested town.

Basically lots of dice rolling, zombie minis, crazy weapons… what more could you ask for in a board game.

…

How about more crazy weapons and equipment!

With that thought began my quest for the ultimate Zombicide equipment deck.

Sourcing the images

There are a number of custom card printers but none that I found can print the mini-card size that Zombicide uses so the only alternative is to re-print the entire base deck in a slightly different size. Which sounds like a bad idea if it wasn’t for the fact that you can easily and cheaply print cards through a number of online printers such as PrinterStudio and ArtsCow but more about that later.

Guillotine Games has released high-res basic card templates and some of the cards in high-res on their website. However not all of the cards are available online in digital format. After a whole lot of searching online I managed to find that there is a Zombicide companion app.

After extracting all of the equipment card image assets from the app. I now had all 52 unique equipment cards from Season 1, 2 & Toxic City Mall. Time to make some custom cards.

Creating custom cards

Unfortunately I wasn’t able to find a equipment card photoshop template that contains all the features I needed so I build a new one which I have included in the download. Using this template I created a bunch of custom equipment cards. Here is a sample of some of the custom cards mixed in with the base cards.

For reference the fonts used are:

• Crackhouse
• ITC Avant Garde Gothic Std – Bold Condensed
• Helvetica Neue LT Std – Extra Black Condensed
• Eurostile Bold Condensed

Printing the deck

In order to re-print the equipment decks you will need to print the following.

Edit: See this document for updated list of equipment:
https://docs.google.com/spreadsheets/d/1XLGrwFbj3wk0c37scQS-GoIr4dh_YDeJpOjYveWdH74/edit#gid=1143787238

That works out to 173 equipment cards and 46 wound cards. However you won’t need 46 wound cards so I recommend you print around 16-24 wound cards. I printed my deck through PrinterStudio which for a 224 card deck came to $14.60 (Note: these prices have now changed, you may need to shop around or wait for a sale). This gave me 27 cards to use for custom cards after re-printing all the base cards. Here is the deck type I used http://www.printerstudio.com/personalized/custom-blank-playing-cards-mini-size.html.

The finished product

SCRIPT1004: Expected ';'
File: example.js, Line: 9, Column: 35

And the line it was pointing to looked like this.

var test = 1; //@todo: add new feature

Well it turns out that if conditional compilation (http://msdn.microsoft.com/en-us/library/ie/121hztk3(v=vs.94).aspx) is turned on using /*@cc_on @*/ in any script. Using the @ symbol anywhere at the start of a comment causes Internet Explorer to throw a syntax error.

The minimum reproducable test case is:

/*@cc_on @*/
var test = 1; //@todo: add new feature

Which can be fixed by putting a space in front of the @ symbol like this.

/*@cc_on @*/
var test = 1; // @todo: add new feature

This becomes an issue in larger codebases where you cannot know if other scripts have turned conditional compilation on at all. As far as I know there isn't a workaround. However you can show a console message warning that conditional compilation is on like this.

/*@if (@_jscript) if ('console' in self && 'log' in console) console.log('Conditional compilation is turned on!'); @end @*/

Keep in mind however that a syntax error caused by the @ symbol will trigger before the above runs.

Faux-Mustache

So the simplest approach to converting the template and data into the output we need is to just use String.prototype.replace(). Note I am using mustache/handlebars syntax for the template syntax in this section. None of the template variables escape html so don’t use in production with unsafe data.

var tpl    = "<p>Is this a template? {{test}}.</p>";
var output = tpl.replace('{{test}}', 'Yes it is!');

However this will only work if we have a single {{test}} template variable in our template because String.prototype.replace() only replaces the first item. You can use the third flags parameter to do a global replace using a string input but it is non-standard. So we have to use a regex instead.

var tpl    = "<p>Is this a template? {{test}}.</p>";
var re     = /\{\{test\}\}/gmi;
var output = tpl.replace(re, 'Yes it is!');

This works and for really simple applications you could easily just use this method instead of a template engine. But the data that we want to replace is hard-coded which means we can’t update it very easily and we need a new replace call for each additional variable. So what we want is to be able to pass in any data as properties of an object like this.

var Template = function(tpl, data){
    return tpl.replace(/\{\{(.*?)\}\}/gmi, function(match, p1) {
        var prop = p1.trim();
        return data.hasOwnProperty(prop) ? data[prop] : '';
    });
};

var tpl    = "<p>Is this a template? {{test}} {{test2}}.</p>";
var data   = {
    test: 'Yes',
    test2: 'it is!'
};
var output = Template(tpl, data);

This is better but it still has some issues. Most notably we have no way to access descendant properties. It would be great to be able to do this {{user.name}}. So lets make it happen.

var Template = function(tpl, data){
    return tpl.replace(/\{\{(.*?)\}\}/gmi, function(match, p1) {
        var prop = p1.trim().split('.'),
            obj  = data;
        while(prop.length && typeof(obj) != 'undefined')
            obj = obj[prop.shift()]
        return typeof(obj) != 'undefined' ? obj : '';
    });
};

var tpl    = "<p>Hi {{user.name}}! You are from {{user.location}}.</p>";
var data   = {
    user: {
        name: 'Pat',
        location: 'Australia'
    }
};
var output = Template(tpl, data);

Great so that works pretty well and it’s only nine lines of code to boot. However there are still two big features missing and they are template compilation and the ability to work with arrays.

Compilation

Template compilation allows us to avoid re-executing the regex and looping through every property every time we use the template. So lets look at how we could update the previous example to return a compiled function.

var Template = function(tpl){
    var cursor = 0,
        code = "var r=[];\n", 
        re = /\{\{(.*?)\}\}/gmi;
    
    var add = function(line, tag) {
        if(tag) {
            code += "r.push(data."+line+");\n";
        } else {
            code += line != '' ? 'r.push("' + line.replace(/"/g, '"') + "\");\n" : '';
        }
    };
    
    while(match = re.exec(tpl)) {
        add(tpl.slice(cursor, match.index));
        add(match[1], true);
        cursor = match.index + match[0].length;
    }
    
    add(tpl.substr(cursor, tpl.length - cursor));
    code += 'return r.join("");';

    try {
        result = new Function('data', code); 
    } catch(err) {
        console.error("'" + err.message + "'", " in \n\nCode:\n", code, "\n");
    }

    return result;
};

var tpl    = "<p>Hi {{user.name}}! You are from {{user.location}}.</p>";
var data   = {
    user: {
        name: 'Pat',
        location: 'Australia'
    }
};
var compiledTemplate = Template(tpl);
var output = compiledTemplate(data);

So now we have a function we can call to parse the template with different data whenever we need to. Note the above is based on the blog post by John Resig (jQuery) here and the template engine from Krasimir Tsonev here.

How it works is that it generates a new Function() that looks like this:

function(data) {
    var r=[];
        
    r.push("<p>Hi ");
    r.push(data.user.name);
    r.push("! You are from ");
    r.push(data.user.location);
    r.push(".</p>");

    return r.join(""); 
}

Lets update it to work with arrays. Ideally it would be great to be able to use a template like this {{#items}}{{.}}{{/items}} which loops through an array of items just like you would when using mustache.

var Template = function(tpl, data){
    var cursor = 0,
        code = "var r=[];\n", 
        re = /\{\{(.*?)\}\}/gmi;
    var add = function(line, tag) {
        if(tag) {
            if(line.charAt(0) === '#') {
                code += "for(var i=0, len=data."+line.slice(1)+".length; i<len; i++){\n";
                code += "var el=data."+line.slice(1)+"[i];\n";
            } else if (line.charAt(0) === '/') {
                code += "};\n";
            } else {
                code += line === '.' ? "r.push(el || data);\n" : "r.push(data."+line+");\n";
            }
        } else {
            code += line != '' ? 'r.push("' + line.replace(/"/g, '"') + "\");\n" : '';
        }
    };
  
    while(match = re.exec(tpl)) {
        add(tpl.slice(cursor, match.index));
        add(match[1], true);
        cursor = match.index + match[0].length;
    }
    
    add(tpl.substr(cursor, tpl.length - cursor));
    code += 'return r.join("");';
    try { result = new Function('data', code); }
    catch(err) { console.error("'" + err.message + "'", " in \n\nCode:\n", code, "\n"); }
    return result;
};

var tpl    = "<p>Hi {{user.name}}! You like:</p>"+
             "<ul>{{#user.likes}}<li>{{.}}</li>{{/user.likes}}</ul>";
var data   = {
    user: {
        name: 'Pat',
        likes: ['Code', 'Boardgames', 'Cooking']
    }
};
var output = Template(tpl)(data);

So as you can see this is a little more complex. Of course it is still missing a lot of features but the point isn’t to create a fully featured template engine (plenty of those exist).

Unfortunately it isn’t possible to re-create all of the mustache features using this approach simply because of the ambiguity in the way mustache handles objects, arrays, and variables with the hash (#) prefix. The alternative is to use something like this http://jsfiddle.net/a9o9Lw0x/1/ however this will likely perform a lot worse since we need to dynamically handle the block (#) tags and at that point you may as well use a proper template engine.

Embedded Javascript (EJS)

Now that we have template compilation it is fairly trivial to make a Embedded Javascript (EJS) style template engine. Let’s have a look at how it is done.

var Template = function(tpl){
    var cursor = 0,
        code   = "with(data) { var r=[];\n", 
        re     = /<%(.*?)%>/gmi;
    
    var add = function(line, tag) {
        if(tag) {
            code += line.charAt(0) === '=' ? "r.push("+line.slice(1).trim()+");\n" : line+"\n";
        } else {
            code += line != '' ? 'r.push("' + line.replace(/"/g, '"') + "\");\n" : '';
        }
    };
    
    while(match = re.exec(tpl)) {
        add(tpl.slice(cursor, match.index));
        add(match[1], true);
        cursor = match.index + match[0].length;
    }
    
    add(tpl.substr(cursor, tpl.length - cursor));
    code += 'return r.join(""); }';
    try { result = new Function('data', code); }
    catch(err) { console.error("'" + err.message + "'", " in \n\nCode:\n", code, "\n"); }
    return result;
};

var tpl    = "<p>Hi <%= user.name %>! You are from <%= user.location %>.</p>"+
             "<ul>"+
             "<% for(var i=0; i < user.likes.length; i++) { %>"+
                 "<li><%= user.likes[i] %></li>"+
             "<% } %>"+
             "</ul>";
var data   = {
    user: {
        name: 'Pat',
        location: 'Australia',
        likes: ['Code', 'Boardgames', 'Cooking']
    }
};
var output = Template(tpl)(data);

Well that’s it for the rundown on how to create a simple template engine.

What?
This happens because after the first loop $item maintains the reference to 3.

Of course PHP does not have block scoping so we then assign that reference to the current loop item in the second loop overwriting the initial value.

The fix. Always unset the item after the closing brace.

foreach($items as &$item) {
    echo $item;
} unset($item);

Or put it in a self-invoking function.

call_user_func(function() use (&$items) {
    foreach($items as &$item) {
        echo $item;
    }
});

Basically drawBezierSplit() allow you to draw a section of a quadratic bezier curve. It uses a quadratic bezier curve (quadraticCurveTo) because this is the default curve type built into the AS3 drawing API. For reference the HTML5 canvas API uses cubic curves by default.

/**
 * Animates bezier-curve
 * 
 * @param ctx       The canvas context to draw to
 * @param x0        The x-coord of the start point
 * @param y0        The y-coord of the start point
 * @param x1        The x-coord of the control point
 * @param y1        The y-coord of the control point
 * @param x2        The x-coord of the end point
 * @param y2        The y-coord of the end point
 * @param duration  The duration in milliseconds
 */
function animatePathDrawing(ctx, x0, y0, x1, y1, x2, y2, duration) {
    var start = null;
    
    var step = function animatePathDrawingStep(timestamp) {
        if (start === null)
            start = timestamp;
        
        var delta = timestamp - start,
            progress = Math.min(delta / duration, 1);
        
        // Clear canvas
        ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height);
        
        // Draw curve
        drawBezierSplit(ctx, x0, y0, x1, y1, x2, y2, 0, progress);
        
        if (progress < 1) {
            window.requestAnimationFrame(step);
        }
    };
    
    window.requestAnimationFrame(step);
}

/**
 * Draws a splitted bezier-curve
 * 
 * @param ctx       The canvas context to draw to
 * @param x0        The x-coord of the start point
 * @param y0        The y-coord of the start point
 * @param x1        The x-coord of the control point
 * @param y1        The y-coord of the control point
 * @param x2        The x-coord of the end point
 * @param y2        The y-coord of the end point
 * @param t0        The start ratio of the splitted bezier from 0.0 to 1.0
 * @param t1        The start ratio of the splitted bezier from 0.0 to 1.0
 */
function drawBezierSplit(ctx, x0, y0, x1, y1, x2, y2, t0, t1) {
    ctx.beginPath();
    
    if( 0.0 == t0 && t1 == 1.0 ) {
        ctx.moveTo( x0, y0 );
        ctx.quadraticCurveTo( x1, y1, x2, y2 );
    } else if( t0 != t1 ) {
        var t00 = t0 * t0,
            t01 = 1.0 - t0,
            t02 = t01 * t01,
            t03 = 2.0 * t0 * t01;
        
        var nx0 = t02 * x0 + t03 * x1 + t00 * x2,
            ny0 = t02 * y0 + t03 * y1 + t00 * y2;
        
        t00 = t1 * t1;
        t01 = 1.0 - t1;
        t02 = t01 * t01;
        t03 = 2.0 * t1 * t01;
        
        var nx2 = t02 * x0 + t03 * x1 + t00 * x2,
            ny2 = t02 * y0 + t03 * y1 + t00 * y2;
        
        var nx1 = lerp ( lerp ( x0 , x1 , t0 ) , lerp ( x1 , x2 , t0 ) , t1 ),
            ny1 = lerp ( lerp ( y0 , y1 , t0 ) , lerp ( y1 , y2 , t0 ) , t1 );
        
        ctx.moveTo( nx0, ny0 );
        ctx.quadraticCurveTo( nx1, ny1, nx2, ny2 );
    }
    
    ctx.stroke();
    ctx.closePath();
}

/**
 * Linearly interpolate between two numbers v0, v1 by t
 */
function lerp(v0, v1, t) {
    return ( 1.0 - t ) * v0 + t * v1;
}

document.addEventListener('DOMContentLoaded',function(){
    var docCanvas = document.getElementById('canvas'),
        ctx = docCanvas.getContext('2d');

    animatePathDrawing(ctx, 0, 100, 150, -50, 300, 100, 5000);
});

Note: Click on the result tab to restart the animation.

Credit to Andre Michelle (andre-michelle.com) for the original inspiration.

Rage is a popular all-night Australian music video program broadcast on ABC1 on Friday nights, Saturday mornings and Saturday nights. It was first screened on the weekend of Friday, 17 April 1987.

Since 1998 rage has posted all their episode playlists online. So I created a website that combines these playlists with YouTube allowing you to travel back through time and re-experience rage’s recent history.

Please note: all the matching of track to video is done programmatically so some tracks particularly older ones may not be available (depending on whether someone has uploaded them).

Behind the Scenes (v2 – 2020)

The original version of the site was written using the Lithium PHP Framework in 2012. The site ran well until 2019 when a number of issues cropped up. So I had a plan in mind for a re-envisioned version to address the following issues:

• Over the years Google has been reducing the number of requests that can be made to the Youtube search API. When the site was first developed the project had millions of requests in quota. Now it is down to 100s per month. I tried contacting the black hole that is Google support with no luck. So I needed an alternative to using the Youtube search API.
• The ABC website scraper needs to be rewritten since the ABC site has changed.
• I wanted to make the website static, and have it connect to a serverless backend to avoid having to host and maintain a server.

Given these requirements I decided on the following plan:

1. Write a new API backend hosted on Cloudflare workers.
2. Dump the playlist data stored in MySQL into flat files (JSON).
3. Use scheduled Github workers to scrape the ABC website and update the data directory.
4. Convert the frontend to be a static single page application (SPA).

The project is broken into three parts; the API backend, the static frontend SPA, and the ABC scraper script run via Github workers.

For the backend API my initial plan was to write this using Rust (https://www.rust-lang.org/). Cloudflare workers have good support for Rust and I enjoy working with the language.

However I ran into some issues writing a YouTube search scraper and given that there was already a NodeJS package for this purpose it didn’t seem like a great idea to re-invent the wheel. So I ended up writing the final version using Typescript.

The logic is fairly simple when a request is made for a specific music video, it will first check for a cached result stored in Cloudflare KV storage. If the cached data cannot be found it will then go off and grab the data from YouTube.

There is also another endpoint that proxies the data from Github, providing the correct CORS headers for the frontend.

All in all it was fun to re-visit the project.

Checkout the interview with ABC about the site for more details https://web.archive.org/web/20150402201331/https://open.abc.net.au/explore/31540

Source code and data

Scraping the data (particularly the 90s web pages) wasn’t easy due to broken and inconsistent markup so I have posted a database dump in case anyone else wants to play with the data.

Playlists: 1,694
Tracks: 201,316
Approx. Play Time: 1 year, 173 days, 5 hours, 46 mins

https://github.com/patrickgalbraith/rageagain (Source code and data repo)

Changelog

17th January 2020

Features

• RAGEAgain is back after a long hiatus!
• Fixing the site required a complete rewrite. Source code is now available here https://github.com/patrickgalbraith/rageagain.

12th June 2017

Features

• Adjusted Youtube search algorithm to ignore Aria chart information and de-rank reaction and karaoke videos

4th March 2013

Features

• Migrated to new sever (initial load should be significantly faster)
• Finally implemented listing by special
• Updated to jQuery 1.9.1

3rd September 2012

Features

• Updated to include latest playlists
• New special playlist; Top 200 which plays the most played tracks in ascending order (i.e. most played first).
• Individual playlist/tracks are now bookmarkable and browser back button works
• Keyboard shortcuts (spacebar = pause/play, arrow left = prev, arrow right = next)
• Note: If you click on the video you may need to click outside of video for the keyboard events to trigger again. This is because the video will capture any keyboard events.
• Added anchor links to quickly skip to specific year.
• Added warning for users on unsupported/outdated browsers.
• Updated jquery to 1.8.1 (mainly fixes player not appearing in Internet Explorer 8/9)

Bug Fixes

• Fixed issue with tracklist being capped at 200 which caused the first tracks to not be played.
• Massive database overhaul
• Fixed broken 1999 playlists
• Added 30+ missing/incomplete playlists
• Removed a few duplicate playlists
• Removed all duplicate tracks (some tracklists had been entered twice)
• Fixed tracks containing unescaped html entites and tags

29th August 2012

First public release

28th July 2012

Private release

Installation (iPad)

1. Add this page as a bookmark
2. Select and copy the Javascript in the textarea below.
3. Go to your bookmarks and tap “edit”. Then tap the new bookmark you just made of this page.
4. Edit the name and paste the Javascript in the field for the URL.
5. Click done.

javascript:(function(){var%20objectTags=document.getElementsByTagName(‘object’);var%20objectTagsArr=new%20Array();for(var%20i=0;i<objectTags.length;i++){objectTagsArr[i]=objectTags[i];}for(i=0;i<objectTagsArr.length;i++){var%20objectTag=objectTagsArr[i];if(objectTag.data.indexOf('http://www.youtube.com/')>=0){var%20youtubeID=objectTag.data.match(/(?:youtube\.com\/(?:[^\/]+\/.+\/|(?:v|e(?:mbed)?)\/|.*[?&]v=)|youtu\.be\/)([^’&?\/ ]{11})/i)[1];var%20newIframeElem=document.createElement(‘iframe’);newIframeElem.class=’youtube-player';newIframeElem.type=’text/html';newIframeElem.width=640;newIframeElem.height=385;newIframeElem.src=’http://www.youtube.com/embed/’+youtubeID;newIframeElem.frameborder=0;var%20newElem=objectTag.parentNode.replaceChild(newIframeElem,objectTag);}}})();

Source code with formatting

javascript:(function(){
	var objectTags = document.getElementsByTagName('object');
	var objectTagsArr = new Array();
	
	for (var i = 0; i < objectTags.length; i++) {
		objectTagsArr[i] = objectTags[i];
	}
	
	for (i = 0; i < objectTagsArr.length; i++) {
		var objectTag = objectTagsArr[i];
		if(objectTag.data.indexOf('http://www.youtube.com/') >= 0) {
			var youtubeID = objectTag.data.match(/(?:youtube\.com\/(?:[^\/]+\/.+\/|(?:v|e(?:mbed)?)\/|.*[?&]v=)|youtu\.be\/)([^'&?\/ ]{11})/i)[1];
			
			var newIframeElem = document.createElement('iframe');
			newIframeElem.class = 'youtube-player';
			newIframeElem.type = 'text/html';
			newIframeElem.width = 640;
			newIframeElem.height = 385;
			newIframeElem.src = 'http://www.youtube.com/embed/'+youtubeID;
			newIframeElem.frameborder = 0;
			
			objectTag.parentNode.replaceChild(newIframeElem, objectTag);
		}
	}
})();