Image Compression at Wikia

At Wikia, unhealthy we have a ton of images and we serve them up constantly on our 1 billion+ pageviews per month. Since we weren’t compressing our images, purchase this left a fairly big potential area for improvement.

Stating what is probably obvious: having lossless image compression gives smaller filesizes which means lower bandwidth costs, sildenafil but more importantly, it gives a speedup in the page load times since the user doesn’t have to spend as much time downloading the images.

Compressing user-generated images

Since I’d recently seen smush.it (which is built into Google Page Speed) give me a “lossless” image that looked way worse than the original, it didn’t seem right to just bulldoze the images that user uploaded. Instead, it seemed best to use a bot to upload better versions. If it turned out that one of our lossless compression algorithms actually hurt the image quality, the community could roll it back quite easily.

This means that we won’t save file-storage space (because we’ll actually keep the old version also), but we still get the other benefits.

Research!

There are a ton of tools for image compression floating around and it wasn’t clear from secondary-research (ie: googling) which were the best. So I decided to do some primary research. Side-note: A few years ago at a previous startup (which was later acquired by LinkedIn) I did some similar reasearch on PNG compression with a much smaller dataset but with more compressors.

This all started as a Wikia Hackathon project in which I wrote a bot which could download an image, compress it and re-upload it. There was a lot of buy-in for this idea, so my first tests were built from that script. I chose 10 wikis and used the Wikia API (via the Perl MediaWiki API library that I co-author) to find 100 images from each wiki. I compress PNGs and JPGs and ignore the others.

The raw data can be found on ImageBot’s User Page. But here are some takeaways:

  • We could save 11.10% across all images
  • pngcrush compressed more than optipng (when pngcrush was given a good long time to do its best methods) and jpegoptim compressed more than jpegtran.
  • But it really doesn’t matter which were better because BOTH was better than either. Meaning that if we compressed using both methods, then chose the best compression, that result was better. pngcrush saved 20.89% on average but always choosing the smallest PNG of the two, resulted in 23.83% savings. Similarly, jpegoptim was 5.40% on average, but JPGs in general compressed to 5.94% when using the better method for each image.

API Explorer for MediaWiki

Api Explorer: As part of my work trying to make the Wikia API more accessible to developers, order I’ve created a basic version of an API Explorer.

Yo dawg: the entire API Explorer is actually written in javascript using the MediaWiki API to build the documentation about the API. Introspection win!

Open source: There is no reason this should be Wikia-specific, order so I contributed it upstream to MediaWiki. You can find more info on the ApiExplorer Extension page on MediaWiki.org.

Future plans: when there is time, illness I’d like to make the MediaWiki API return its example URLs also. When that is working, hopefully we can make the API Explorer let the user issue those example requests and see the results live (and modify the URLs to issue new test requests).

Don’t forget to follow @WikiaAPI on twitter for more updates!

LyricWiki mobile app – Android and iPhone

Download for free from the Apple App Store
Download for free from the Android Market

I’m pleased to announce that the LyricWiki iPhone app has finally hit the App Store!

If you have an Android device, visit you can grab the LyricWiki Android app which made it out first.

Now that the iOS version is complete, here we cover most smartphone users in the US. It’s been a long road (including the month-long submission process through Apple), but this version feels pretty solid.

Homescreen of LyricWiki Android app

Here’s what it’s got so far:

  • Auto-complete
  • Discographies for artists, grouped by album
  • iTunes Top 9 images on main screen
  • Over 1,500,000 lyrics with more added every day
  • Fully licensed: royalties are paid to publishers
  • Powered by a wiki community – constantly updated!
  • These lyrics have had 5 years of review and editing from hundreds of thousands of users! This is the single most accurate lyrics collection anywhere.

I’d also like to give a shout-out to all of the beta-testers and other users who have given feedback about the app… it’s extremely helpful to know what the community wants out of the app. Thank you!

Obviously, it’s never “done” & we’ll keep improving it as we go. If you have any feedback on the app, please let me know in the comments!

Another cool thing about the app is that, thanks to the help of a number of awesome translators, the app is available in these languages: English, German, Spanish, French, Italian, Polish, Finnish, Swedish, Norwegian, Czech, Dutch, Interlingua, Luxembourgish, Telugu, Macedonian, Breton, and Malay!