I have that post mostly written, but in order to make that whole process easier, I’ve finally made the bot framework that I now use to be open sourced and easily accessible online.

Background

I used to use custom scripts for my bot, but this summer when LyricWiki transitioned over to Wikia, they all broke. My scripts pre-dated the MediaWiki API so they had depended on screen-scraping which no longer worked when we switched to Wikia’s skins which had a completely different layout.

When I had to get my bots running again, I looked at a few Perl frameworks for connecting to the MediaWiki API, and the one that seemed to have significantly less bugs than the others was a perl module by CBM.

Over the months, I’ve realized that there was some functionality that wasn’t implemented yet but which I needed – deleting pages, issuing purges, finding all templates included on a page – so I updated to the module. I tried to get access to the MediaWiki Tool Server where the project is currently hosted, but they must be really busy because they haven’t replied to the JIRA issue (request for an account) and it’s been months.

Since it has become quite a waiting game, I decided to just fork the project. Hopefully CBM will want access to the repository and we can just keep working on it together. Regardless, I’ve created all of the usual suspects for a project such as this (see next section).

Project links

So, without further delay, here are the beginnings of the Perl MediaWiki API

• Perl MediaWiki API SVN Repository
• Perl MediaWiki API Wiki
• Perl MediaWiki API Trac
• Perl MediaWiki API Ohloh project page

The links (especially the wiki) need a lot of work before it becomes obvious how to quickly get set up and use the module. The next blog post will take care of that!

However, if you’re curious & are already comfortable with Perl (and to some extent MediaWiki), you can jump right in. Let me know if you have any feedback. Thanks!

I’m in the mood to do some blogging in the next couple of days but have more ideas than time. What would YOU find most interesting? I’m thinking along the lines of either analyzing LyricWiki statistics or doing quick tutorials (“How to Write a MediaWiki Bot in 10 Minutes or Less”, or something similar).

Here were some ideas of stats I could do. They each take a decent amount of time, so please let me know which ones you are most interested in:

• Views / #Songs by Genre
• Views / #Pages by Language
• Views / #Pages by Publisher
• Infographic of Lables/Publishers in the Music Industry, how they relate to each other, and their prevalence in the market.
• Our prevalence in a country vs. it’s prevalence online
• Impact on page-views of being SOTD / AOTW / FMOM vs. not.
• Views from songs that were on the iTunes Top 100 during the month vs. those that weren’t. Views/page for that same group.
• Views by page-age and views/page by page-age.
• Views by page freshness (last touched) and views/page by page freshness. Include histogram of freshness across all pages.

Let me know in the comments what you want to see! Those stats, the tutorial mentioned, or anything else are all fair game. Since I don’t have many readers yet, if you comment then I’ll probably do the post you’re asking for.

While MySQL provides LIMIT and/or OFFSET functionality to do batching, if you have numeric id’s on the table(s) you’re querying, I’d recommend using your own system for offsets (code example below) rather than the default MySQL functionality since the hand-rolled method is much faster. See table in next section for performance comparisons.

Timing table

I’ll provide some example code below to show you how I did the batching (based on potentially sparse, unique, numeric ids). To start, here is a table of query-result-size vs. the total runtime for my particular loop. All timings are for traversing approximately 1,000,000 rows.

At the end, it was pretty clear that no more data was needed to continue to demonstrate that the LIMIT method was slow. Each one of those runs was taking about half an hour and about halfway through the 1,000 row test for the LIMIT method, it started causing the database to be backed up. Since this was on a live production system, I decided to stop before it caused any lag for users.

Example Code

This code is an example of querying for all of the pages in a MediaWiki database. I used similar code to this to make a list of all of the pages (and redirects) in LyricWiki. In the code, you’ll notice that the manual way I do the offsets based on id instead of using the MySQL “LIMIT” syntax doesn’t guarantee that each batch is the same size since ids might be sparse (ie: some may be missing if rows were deleted). That’s completely fine in this case and there is a significant performance boost from using this method. This test code just writes out a list of all of the “real” pages in a wiki (where “real” means that they are not redirects and they are in the main namespace as opposed to Talk pages, Help pages, Categories, etc.).

< ?php

$QUERY_BATCH_SIZE = 10000;
$titleFilenamePrefix = "wiki_pageTitles";

// Configure these database settings to use this example code.
$db_host = "localhost";
$db_name = "";
$db_user = "";
$db_pass = "";

$db = mysql_connect($db_host, $db_user, $db_pass);
mysql_select_db($db_name, $db);

$TITLE_FILE = fopen($titleFilenamePrefix."_".date("Ymd").".txt", "w");
$offset = 0;
$done = false;
$startTime = time();
while(!$done){
	$queryString = "SELECT page_title, page_is_redirect FROM wiki_page WHERE page_namespace=0 AND page_id > $offset AND page_id < ".($offset+$QUERY_BATCH_SIZE);
	if($result = mysql_query($queryString, $db)){
		if(($numRows = mysql_num_rows($result)) && $numRows > 0){
			for($cnt=0; $cnt < $numRows; $cnt++){
				$title = mysql_result($result, $cnt, "page_title");
				$isRedirString = mysql_result($result, $cnt, "page_is_redirect");
				$isRedirect = ($isRedirString != "0");
				if(!$isRedirect){
					fwrite($TITLE_FILE, "$title\n");
				}
			}
			$offset += $QUERY_BATCH_SIZE;
			print "\tDone with $offset rows. \n";
		} else {
			$done = true;
		}
	}
	mysql_free_result($result);
}
$endTime = time();
print "Total time to cache results: ".($endTime - $startTime)." seconds.\n";
fclose($TITLE_FILE);

?>

Hope that helps!