Quick tip: Perl version of PHP’s in_array()

There are a million and one ways to do this in Perl, sick but here’s a fairly readable way that you can copy/paste into your Perl scripts:


####
# From http://www.seancolombo.com
# Equivalent to PHP's in_array.  If the first element is in the array
# passed in as the second parameter, then the sub-routine returns non-zero.
# If the element is not in the array, then the sub-routine returns zero.
####
sub in_array{
my $retVal = 0;
my $val = shift(@_);
foreach my $curr (@_){
if($curr eq $val){
$retVal = 1;
last;
}
}
return $retVal;
} # end in_array()

Example usage:


if(in_array($needle, @haystack)){
print "Found it!
";
}

Been using this for years & it’s made my life a bit easier. Hope that’s useful to someone!

Quick tip: clone of PHP’s microtime() function as a Perl subroutine.

Refer to the PHP manual for how the function is supposed to work. The short version is that you call “microtime(1)” to get this perl subroutine to return a float-formatted value containing the seconds and microseconds since the unix epoch.


# Clone of PHP's microtime. - from http://seancolombo.com
use Time::HiRes qw(gettimeofday);
sub microtime{
my $asFloat = 0;
if(@_){
$asFloat = shift;
}
(my $epochseconds, buy  my $microseconds) = gettimeofday;
if($asFloat){
while(length("$microseconds") < 6){
$microseconds = "0$microseconds";
}
$microtime = "$epochseconds.$microseconds";
} else {
$microtime = "$epochseconds $microseconds";
}
return $microtime;
}

This is public domain, use it as you’d like. Please let me know if you find any bugs.

Hope it helps!

Quick tip: how to save your phone after dropping it in (boiling) water

Here is an excerpt from an email I sent last night. If you don’t know me and just want to know how to save your phone, information pills feel free to jump over it:

So while I was boiling my pasta tonight, apparently my body had the urge to include some additional ingredients for once as I found myself inadvertently add my Blackberry to the mix.

Whilst boiling, it cried out to me “Heathenous traitor! Spare me!” ..Whatever, don’t call me names then expect my help. Also, I eat cows and chickens with a clear conscience, I can eat a phone too. Then in a last-ditch effort it blurted “Without me, thou shall never text cute girls again!” I immediately decided that blackberries might not go very well with pasta anyway, so I removed it and pulled the battery out.

It is currently getting the Lazarus Treatment in a closed up shoebox with 14 ounces of desiccant (a 60 day supply for a closet). Hopefully I made the right choice, because after the desiccant, the Blackberry will be all dry and flavorless anyway.

Like most other fruit, I would imagine that Blackberries are fat-free and low in calories but I have a hard time envisioning that the contain many vitamins or antioxidants.

Do you have any idea of the Nutrition Facts or whether I made the right decision?
– Sean

In hopes that it may help others out, here are some things you can do to save your phone if you drop it in water…

  • If you are fortunate enough to read this BEFORE your phone goes for a swim: remember to take out the battery as quickly as possible. Don’t even bother shutting the phone off if you have a removable battery. In the case of iPhones or other devices where the battery isn’t removable, turn those off as fast as possible. If you remove the battery or at least get the phone off before it dies (short circuits) on its own, you have a good chance of saving it. If you have a sim card, it’s best to yank that too, but it can wait until the battery is out.
  • Alright, assuming your phone is chilling without a battery – the next step is to dry it off, shake it to get the water out, etc.
  • The next part is a great idea that I was given via twitter from my friend Nick: go find a desiccant to suck all of the moisture out of the phone. You can grab some at Home Depot or Lowes (which is where I found it). An example of a desiccant is DampRid (link has a picture).
  • Using a desiccant to dry a cellphone (click to enlarge)

    Using a desiccant to dry a cellphone (click to enlarge)

    Leave the phone open (battery out and cover off) and put it in a small container with the desiccant for as long as you can comfortably stand not having your phone. In my case, the phone was in the box for 8 hours and it came out just fine. If your phone was submerged longer or didn’t shake out as well, it might take longer. If you’re patient, I’d recommend 24 hours but I just made that number up so take it with a grain of salt. ;) I had happened to buy some new shoes yesterday so I had the box right there… since the DampRid I got was a hanging variety, I hung it up in there (see pic to the right) and then closed the lid and let it sit.
  • Pop that battery back in and cross your fingers!

I was fortunate enough to have my phone survive. The $9 for the desiccant might have been the factor that saved me from having to replace a $400 phone (I’m all out of free-upgrades!).

Although my BlackBerry (and most phones) are fortunate enough to have a removable battery, don’t despair if you have an iPhone: the unmistakable John Hann reports of having tried to use his iPhone to season his baby-stew and it was also able to survive by being shut off and allowed to dry.

Hope that helps someone!

Quick Tip: Do huge MySQL queries in batches when using PHP

When using PHP to make MySQL queries, stomatology it is significantly better for performance to break one extremely large query into smaller queries. In my testing, there was a query which returned 1 million rows and took 19,275 seconds (5 hours, 20 minutes) to traverse the results. By breaking that query up into a series of queries that had about 10,000 results, the total time dropped to 152 seconds… yeah… less than 3 minutes.

While MySQL provides LIMIT and/or OFFSET functionality to do batching, if you have numeric id’s on the table(s) you’re querying, I’d recommend using your own system for offsets (code example below) rather than the default MySQL functionality since the hand-rolled method is much faster. See table in next section for performance comparisons.

Timing table

I’ll provide some example code below to show you how I did the batching (based on potentially sparse, unique, numeric ids). To start, here is a table of query-result-size vs. the total runtime for my particular loop. All timings are for traversing approximately 1,000,000 rows.

Query Batch Size Handrolled method MySQL “LIMIT” syntax
1,000,000+ 19,275 seconds 19,275 seconds
10,000 152 seconds 1,759 seconds
5,000 102 seconds 1,828 seconds
1,000 43 seconds ?
750 40 seconds ?

At the end, it was pretty clear that no more data was needed to continue to demonstrate that the LIMIT method was slow. Each one of those runs was taking about half an hour and about halfway through the 1,000 row test for the LIMIT method, it started causing the database to be backed up. Since this was on a live production system, I decided to stop before it caused any lag for users.

Example Code

This code is an example of querying for all of the pages in a MediaWiki database. I used similar code to this to make a list of all of the pages (and redirects) in LyricWiki. In the code, you’ll notice that the manual way I do the offsets based on id instead of using the MySQL “LIMIT” syntax doesn’t guarantee that each batch is the same size since ids might be sparse (ie: some may be missing if rows were deleted). That’s completely fine in this case and there is a significant performance boost from using this method. This test code just writes out a list of all of the “real” pages in a wiki (where “real” means that they are not redirects and they are in the main namespace as opposed to Talk pages, Help pages, Categories, etc.).


< ?php

$QUERY_BATCH_SIZE = 10000;
$titleFilenamePrefix = "wiki_pageTitles";

// Configure these database settings to use this example code.
$db_host = "localhost";
$db_name = "";
$db_user = "";
$db_pass = "";

$db = mysql_connect($db_host, $db_user, $db_pass);
mysql_select_db($db_name, $db);

$TITLE_FILE = fopen($titleFilenamePrefix."_".date("Ymd").".txt", "w");
$offset = 0;
$done = false;
$startTime = time();
while(!$done){
$queryString = "SELECT page_title, page_is_redirect FROM wiki_page WHERE page_namespace=0 AND page_id > $offset AND page_id < ".($offset+$QUERY_BATCH_SIZE);
if($result = mysql_query($queryString, $db)){
if(($numRows = mysql_num_rows($result)) && $numRows > 0){
for($cnt=0; $cnt < $numRows; $cnt++){
$title = mysql_result($result, $cnt, "page_title");
$isRedirString = mysql_result($result, $cnt, "page_is_redirect");
$isRedirect = ($isRedirString != "0");
if(!$isRedirect){
fwrite($TITLE_FILE, "$title
");
}
}
$offset += $QUERY_BATCH_SIZE;
print "	Done with $offset rows. 
";
} else {
$done = true;
}
}
mysql_free_result($result);
}
$endTime = time();
print "Total time to cache results: ".($endTime - $startTime)." seconds.
";
fclose($TITLE_FILE);

?>

Hope that helps!

Quick Tip: Generate Random Numbers in a MySQL Query

Noticed a dearth of Google articles about this so I figured I’d post my solution…

If you need to generate a random number inside of a MySQL query (which is useful for making fake test-data), there you can do it as follows:

SELECT floor(rand() * 10) as randNum;

Which would generate a random integer from 0 to 9 inclusive. Just change the 10 to the number one higher than you want to generate. The important part is just the “floor(rand() * THE_EXCLUSIVE_UPPER_BOUND)”.

The full explanation is that rand() will generate a random floating point number that is greater than or equal to 0 but less than 1. After you multiply that number by your upper-bound, sales floor() gets rid of everything after the decimal point.

Hope that helps someone!