SeanColombo.com

My little corner of the internet.

API Explorer for MediaWiki

Api Explorer: As part of my work trying to make the Wikia API more accessible to developers, I’ve created a basic version of an API Explorer.

Yo dawg: the entire API Explorer is actually written in javascript using the MediaWiki API to build the documentation about the API. Introspection win!

Open source: There is no reason this should be Wikia-specific, so I contributed it upstream to MediaWiki. You can find more info on the ApiExplorer Extension page on MediaWiki.org.

Future plans: when there is time, I’d like to make the MediaWiki API return its example URLs also. When that is working, hopefully we can make the API Explorer let the user issue those example requests and see the results live (and modify the URLs to issue new test requests).

Don’t forget to follow @WikiaAPI on twitter for more updates!

Image Compression at Wikia

At Wikia, we have a ton of images and we serve them up constantly on our 1 billion+ pageviews per month. Since we weren’t compressing our images, this left a fairly big potential area for improvement.

Stating what is probably obvious: having lossless image compression gives smaller filesizes which means lower bandwidth costs, but more importantly, it gives a speedup in the page load times since the user doesn’t have to spend as much time downloading the images.

Compressing user-generated images

Since I’d recently seen smush.it (which is built into Google Page Speed) give me a “lossless” image that looked way worse than the original, it didn’t seem right to just bulldoze the images that user uploaded. Instead, it seemed best to use a bot to upload better versions. If it turned out that one of our lossless compression algorithms actually hurt the image quality, the community could roll it back quite easily.

This means that we won’t save file-storage space (because we’ll actually keep the old version also), but we still get the other benefits.

Research!

There are a ton of tools for image compression floating around and it wasn’t clear from secondary-research (ie: googling) which were the best. So I decided to do some primary research. Side-note: A few years ago at a previous startup (which was later acquired by LinkedIn) I did some similar reasearch on PNG compression with a much smaller dataset but with more compressors.

This all started as a Wikia Hackathon project in which I wrote a bot which could download an image, compress it and re-upload it. There was a lot of buy-in for this idea, so my first tests were built from that script. I chose 10 wikis and used the Wikia API (via the Perl MediaWiki API library that I co-author) to find 100 images from each wiki. I compress PNGs and JPGs and ignore the others.

The raw data can be found on ImageBot’s User Page. But here are some takeaways:

  • We could save 11.10% across all images
  • pngcrush compressed more than optipng (when pngcrush was given a good long time to do its best methods) and jpegoptim compressed more than jpegtran.
  • But it really doesn’t matter which were better because BOTH was better than either. Meaning that if we compressed using both methods, then chose the best compression, that result was better. pngcrush saved 20.89% on average but always choosing the smallest PNG of the two, resulted in 23.83% savings. Similarly, jpegoptim was 5.40% on average, but JPGs in general compressed to 5.94% when using the better method for each image.

How to make (and use) a custom SASS function

The SASS (Syntactically Awesome StyleSheets) language is pretty neat. For my first use of SASS, I realized that one of the requirements for our system wasn’t easily supported by the language… but SASS is written in Ruby which is really easy to extend. The docs even mentioned the ability to create custom functions. However, I couldn’t find any docs on how to actually create and use a custom SASS function, so I figured I’d give a quick tutorial here of what I learned. This method is probably obvious to hardcore Ruby users, but I’d never used Ruby before. Turns out it’s pretty quick to learn. If you want to do the Matrix thing and pump the whole language into your brain like Neo in a ghetto dentist-chair, check out this Ruby crash course.

NOTE: This tutorial is targeted primarily at people using SASS via the command line (for PHP, Java, etc.), not as a Rails module.

My example: getting values from the sass command-line

I’m using SASS in a PHP environment (rather than Rails) and due to unique requirements, I need to be able to configure certain values in .scss at ‘compile’ time (referring to when the .scss is being compiled into .css).

One simple trick would be to simply write out a .scss file containing the key-value pairs. Unfortunately, the system I need to use SASS for already has tens of millions of page-requests per day so disk-writes would be a huge bottleneck (because disk i/o – even on solid state disks – is slow compared to many other methods). Custom SASS functions provided the perfect opportunity to completely skip this step. And yes: pre-generating all of the CSS files at code-deployment is out of the question because the number of possible stylesheets we need to support is intractably large.

The custom function

To create your custom SASS function, make a ruby file. We’ll call it sass_function.rb in this example. In the file, you need to define your function and then insert your module into SASS. Behold!


require 'sass'

module WikiaFunctions
        def get_command_line_param(paramName, defaultResult="")
                assert_type paramName, :String
                retVal = defaultResult.to_s

                # Look through the args given to SASS command-line
                ARGV.each do |arg|
                        # Check if arg is a key=value pair
                        if arg =~ /.=./
                                pair = arg.split(/=/)
                                if(pair[0] == paramName.value)
                                        # Found correct param name
                                        retVal = pair[1] || defaultResult
                                end
                        end
                end
		begin
			Sass::Script::Parser.parse(retVal, 0, 0)
		rescue
			Sass::Script::String.new(retVal)
		end
        end
end

module Sass::Script::Functions
  include WikiaFunctions
end

The particular SASS function in this example takes in name/value pairs defined on the sass command line and returns them if they’re there (or an optional default otherwise).

Calling SASS

Since this tutorial assumes that you’re using sass from the command-line, you’ll have to tweak the command a little bit to tell ruby to use your new module. Here is a simple example:
sass unicorn.scss unicorn.css -r sass_function.rb
That doesn’t make use of the awesomeness of our new, command-line parsing function though! So here is an example that WOULD make use of it:
sass unicorn.scss unicorn.css logoColor=#6495ED -r sass_function.rb

So now we have a function capable of reading the command-line and a command-line with some useful information in it. Now all that’s needed is some SASS code (.scss) to make use of all of that.

In this example, we’ll set the “logo” element to have a background-color that we get from the command-line (and default to white if no matching value is passed in on the command-line). Remember: this would go in SASS code such as unicorn.scss


$logoBackgroundColor: get_command_line_param("logoColor", "white");

#logo{
   background-color: $logoBackgroundColor;
}

Now we have all of the pieces:

  1. The custom SASS function (called get_command_line_param()) in sass_function.rb
  2. The code in unicorn.scss to use our function to set the style by command-line info.
  3. The command-line needed to include our custom code and to set the logoColor value.

So if we run
sass unicorn.scss unicorn.css logoColor=#6495ED -r sass_function.rb
we will have a unicorn.css which contains something like:


#logo{
   background-color: #6495ED;
   }

Just what we were going for! If you try this out, let me know in the comments if it worked for you or if you have any questions.

Best of luck!

Special thanks to Nathan Weizenbaum for pointing me down the right path with this stuff. Updated on 20100729 to change the return-value of the function based on the helpful comments below. Thanks!

Open sourcing a MediaWiki bot framework

On my last post I asked what my readers wanted me to write about and all of the responses I got on the post or in person had the “how to write a MediaWiki bot in 10 minutes or less” at the top of the list.

I have that post mostly written, but in order to make that whole process easier, I’ve finally made the bot framework that I now use to be open sourced and easily accessible online.

Background

I used to use custom scripts for my bot, but this summer when LyricWiki transitioned over to Wikia, they all broke. My scripts pre-dated the MediaWiki API so they had depended on screen-scraping which no longer worked when we switched to Wikia’s skins which had a completely different layout.

When I had to get my bots running again, I looked at a few Perl frameworks for connecting to the MediaWiki API, and the one that seemed to have significantly less bugs than the others was a perl module by CBM.

Over the months, I’ve realized that there was some functionality that wasn’t implemented yet but which I needed – deleting pages, issuing purges, finding all templates included on a page – so I updated to the module. I tried to get access to the MediaWiki Tool Server where the project is currently hosted, but they must be really busy because they haven’t replied to the JIRA issue (request for an account) and it’s been months.

Since it has become quite a waiting game, I decided to just fork the project. Hopefully CBM will want access to the repository and we can just keep working on it together. Regardless, I’ve created all of the usual suspects for a project such as this (see next section).

Project links

So, without further delay, here are the beginnings of the Perl MediaWiki API

The links (especially the wiki) need a lot of work before it becomes obvious how to quickly get set up and use the module. The next blog post will take care of that!

However, if you’re curious & are already comfortable with Perl (and to some extent MediaWiki), you can jump right in. Let me know if you have any feedback. Thanks!

When to release a software project

This is something I’ve thought about for a long time and I think I’ve finally come up with a succinct answer: as soon as possible with the minimum set of features that still lets the project accomplish its core goals.

Before we get into this full-steam, there are obviously exceptions such as big title video games where the consumer expectation is to be delivered a single piece of completed software that never needs an update. This post is aimed primarily at web-apps but will also apply to most other applications (including indie video games).

Who cares?

This may seem like a very minor issue or a personal preference, but I really don’t think it is. As a developer or as someone who knows developers, you probably realize that the vast majority of software never gets finished. Furthermore, too much of it is poorly written. When you choose to release a product and how you get there are both major factors in addressing these concerns.

My recommendations

Bullets are fast, lets use them!

  • Figure out what makes your product concept compelling to users. If you can write this in one sentence, that would be good. Here’s an example you may recognize: LyricWiki is a free site which is a source where anyone can go to get reliable lyrics for any song, from any artist, without being hammered by invasive ads. Even if you don’t word it as a sentence (which you really should figure out at some point*) for now, just make sure you have the main features: {free, reliable lyrics (wiki-editable), good coverage, no invasive ads}.
  • Even if you want your product to have a bunch of tangential-features which you think will make it awesome, make a list of only the features that are mandatory to meet your project’s core goals.
  • When you’re writing a feature, write it right. While I think you should cut back on what you implement, my experience has been that you almost never get to go back and really polish features as much as you’d like to after-the-fact. Do it right the first time. Also, if you ever do get the chance to go back you won’t remember the code quite as well as when you’re writing it the first time.
  • Release it! As soon as you hit your minimum goals, don’t hesitate… put it live! If you have friendly users asking you for fixes and new features, that will push you to continue. You probably created the project because you wanted to make something that would be used… so people actually using the product and wanting to use it more is one of the best forcing-functions to get you to keep working. It will be an even better motivator if you actually use your product yourself because you will quickly start to yearn for new features or bugfixes.

*: A full sentence like this should guide your decisions as the project grows to keep it from bloating. Also, when people social-bookmark, blog about, or tweet your site they will quite frequently just paste this sentence. Having it as the first and most prominent sentence helps. Also, people are going to ask you “what is your site?” in loud, crowded rooms dozens of times. If your project is successful, you will literally have to describe your project hundreds of times. My current loud-environment elevator-pitch for LyricWiki is: “it’s like Wikipedia for song lyrics… called LyricWiki.” This evolved primarily because apparently people couldn’t hear my enunciation of “LyricWiki” in a loud room unless I had pre-prepped their brains for both lyrics and wikis.

Quick Tip: Do huge MySQL queries in batches when using PHP

When using PHP to make MySQL queries, it is significantly better for performance to break one extremely large query into smaller queries. In my testing, there was a query which returned 1 million rows and took 19,275 seconds (5 hours, 20 minutes) to traverse the results. By breaking that query up into a series of queries that had about 10,000 results, the total time dropped to 152 seconds… yeah… less than 3 minutes.

While MySQL provides LIMIT and/or OFFSET functionality to do batching, if you have numeric id’s on the table(s) you’re querying, I’d recommend using your own system for offsets (code example below) rather than the default MySQL functionality since the hand-rolled method is much faster. See table in next section for performance comparisons.

Timing table

I’ll provide some example code below to show you how I did the batching (based on potentially sparse, unique, numeric ids). To start, here is a table of query-result-size vs. the total runtime for my particular loop. All timings are for traversing approximately 1,000,000 rows.

Query Batch Size Handrolled method MySQL “LIMIT” syntax
1,000,000+ 19,275 seconds 19,275 seconds
10,000 152 seconds 1,759 seconds
5,000 102 seconds 1,828 seconds
1,000 43 seconds ?
750 40 seconds ?

At the end, it was pretty clear that no more data was needed to continue to demonstrate that the LIMIT method was slow. Each one of those runs was taking about half an hour and about halfway through the 1,000 row test for the LIMIT method, it started causing the database to be backed up. Since this was on a live production system, I decided to stop before it caused any lag for users.

Example Code

This code is an example of querying for all of the pages in a MediaWiki database. I used similar code to this to make a list of all of the pages (and redirects) in LyricWiki. In the code, you’ll notice that the manual way I do the offsets based on id instead of using the MySQL “LIMIT” syntax doesn’t guarantee that each batch is the same size since ids might be sparse (ie: some may be missing if rows were deleted). That’s completely fine in this case and there is a significant performance boost from using this method. This test code just writes out a list of all of the “real” pages in a wiki (where “real” means that they are not redirects and they are in the main namespace as opposed to Talk pages, Help pages, Categories, etc.).


< ?php

$QUERY_BATCH_SIZE = 10000;
$titleFilenamePrefix = "wiki_pageTitles";

// Configure these database settings to use this example code.
$db_host = "localhost";
$db_name = "";
$db_user = "";
$db_pass = "";

$db = mysql_connect($db_host, $db_user, $db_pass);
mysql_select_db($db_name, $db);

$TITLE_FILE = fopen($titleFilenamePrefix."_".date("Ymd").".txt", "w");
$offset = 0;
$done = false;
$startTime = time();
while(!$done){
	$queryString = "SELECT page_title, page_is_redirect FROM wiki_page WHERE page_namespace=0 AND page_id > $offset AND page_id < ".($offset+$QUERY_BATCH_SIZE);
	if($result = mysql_query($queryString, $db)){
		if(($numRows = mysql_num_rows($result)) && $numRows > 0){
			for($cnt=0; $cnt < $numRows; $cnt++){
				$title = mysql_result($result, $cnt, "page_title");
				$isRedirString = mysql_result($result, $cnt, "page_is_redirect");
				$isRedirect = ($isRedirString != "0");
				if(!$isRedirect){
					fwrite($TITLE_FILE, "$title\n");
				}
			}
			$offset += $QUERY_BATCH_SIZE;
			print "\tDone with $offset rows. \n";
		} else {
			$done = true;
		}
	}
	mysql_free_result($result);
}
$endTime = time();
print "Total time to cache results: ".($endTime - $startTime)." seconds.\n";
fclose($TITLE_FILE);

?>

Hope that helps!

Old, quick app – “Runner’s Calculator”

Thanks to Google Alerts, I ran into an old web-application I made. It was a simple Runner’s Calculator and it was one of my first little standalone PHP programs. It’s literally from the summer of 2004 (ancient history!) before the term “AJAX” was even coined, so you actually have to press a submit button to get the results (the horror!). Old programs are funny.

I had thrown it together while prepping for a 5k that my co-op company was running in that summer. Given the recent upturn in the weather, I figured it be appropriate to dust it off in so it could get some use on the interwebs:

Runner’s Calculator

My personal favorite use is the “Generate Time Table” feature… plop in how far you’re going to be racing and approximate upper and lower bounds on how long you think you might take & it will tell you the splits for a bunch of different times in that range.

Enjoy! :)

“What gets measured gets managed”

That quote is a relatively common business adage which is attributed to a ton of different people – so many that I’ll just consider it public domain. The reason for this post is just that I’d like to draw attention to it and maybe as you read this post, you can think of how this can be used to help your own productivity and success.

As with any business advice, this tends to be more digestible with some anecdotes so I’ll give a couple of examples of how this has helped me lately with running LyricWiki.org.

Example Uno: Server Uptime

Without a doubt, the biggest problem with LyricWiki up until recently had been uptime. The site was constantly slow or completely unavailable from its inception. The reason is that we were always short on servers since the company had minimal capital, and setting up new servers took a great deal of time and fell on my shoulders during a time in my life where I always had at least one fulltime responsibility other than LyricWiki.

When I cut back my other job a bit to give me more time to devote to LyricWiki, one of the first things I set out to fix was the reliability of the site. In order to know if it was improving, I would need to know how long the site was down each month so I could track whether the number of minutes was going up or down.

I created a small spreadsheet to track outages, and each time the site went down I logged when the outage was, the duration (in minutes) of the outage, which server(s) had the problems, the apparent cause and whether or not the cause was resolved. Much to my surprise, I didn’t really need to graph this over the months. After the first couple of days, it became very apparent that there was one huge problem still lingering and it would be worth my time to automate a fix to it instead of responding ad-hoc each time the problem cropped up.

I whipped up the code to solve the problem while I was on a layover in Philadelphia, uploaded it when I got home – and the site stayed up for almost a month straight! That’s pretty huge. I don’t think that had ever happened in LyricWiki history until just now. Very cool.

Example Dos: Intractable To-do Lists

A major personal time-management problem I had recently was that I couldn’t tell if I was just spinning my wheels or if I was actually making progress on my backlog of LyricWiki tasks. It felt as though I was getting emailed so much work that I barely ever got to break out into the tasks on my talk page let alone the “Mid-term Actions” list I made which was my conscious plan of how to make LyricWiki rock your socks off in the mid-term.

To solve this problem of not having a grasp on my tasks, I whipped up a little widget which you can see in the side-column of the blog, near the bottom. The results were much better than I expected: I actually instantly felt like I was in more control, I felt comfortable removing tasks that were duplicated across lists, and I can finally tell when I’m moving fast enough to make forward progress.

The widget tracks the size of all of my various to-do lists and updates that number hourly. There are two reasons for the caching: 1) checking the lists requires hitting the sites and that takes at least 3 seconds total 2) if it updated in real-time, I’d sit there refreshing the thing all day!

The widget actually has some neat hidden features. Here is the to-do list widget on its own page which also tracks the number of tasks I’ve had at the end of each day and charts my progress (using the Google Charts API). If anyone is interested, maybe I’ll write another post where I give out the source-code to the widget.

Conveniently, the widget is narrow enough to both fit in a blog sidebar and be displayed on a smartphone. It’s currently the homepage on my blackberry and I don’t see that changing any time soon!

Conclusion

The quote “what gets measured gets managed” always seems to ring true for me. I think of it again and again – usually after-the-fact when I’ve just saved time by being OCD about something. I strongly recommend that you take a second (right now) and think of an area of your life or your work which you feel isn’t getting sufficient attention and consider tracking meaningful statistics about it. Please share any similar successful experiences in the comments!

How to restore WordPress categories

After upgrading to the latest version of WordPress, the text of all of the categories were deleted. It appears this may be my fault for not disabling all of the plugins before doing the upgrade (oops). They’re back now though. It was somewhat annoying to have to do, but as far as “data loss error”s go, it wasn’t very bad at all.

I noticed a bunch of other people online had similar problems but nobody mentioned a solution so I figured I might as well throw up a quick description of what I did.

I used google to find a cached version of my page which listed all of the categories in the side-bar. Now I had all of the categories but I didn’t know which id in the database matched which category. I found that out by running this query:
SELECT post_title, term_taxonomy_id FROM wp_posts,wp_term_relationships WHERE wp_term_relationships.object_id=wp_posts.ID ORDER BY term_taxonomy_id;
Which shows what posts were assigned to each category. After reading through the list and figuring it out, I was able to fix the text by typing queries along the lines of:
UPDATE wp_terms SET name='Motive Force', slug='motive-force' WHERE term_id=8;

Since I brushed over it above:
To find a cached version of your site on google, just search for “site:YOURDOMAINNAMEHERE.com”, then click on “Cached” next to any of the results. They clear those out after a while, so don’t put it off!

Quick Tip: Beep when a long process is complete over SSH

When you’re running a long command on a terminal over SSH, you may end up wasting a great deal of time checking back repeatedly to see if the process is complete. A quick alternative would be to make the shell beep when it’s complete. Assuming you are running a script called “longScript.sh”, then simply typing a line like:
> longScript.sh; printf \\a
Will cause most SSH clients to beep after longScript.sh is finished running.

Hope that helps!