MySQL defaults to “latin1″ as its character set, but at some point, most people want to migrate to “utf8″. I realize that there are dozens of posts about how people handled this, and yet, not a single one of those worked completely for me.

I wanted MySQL to use “utf8″ for the character set and “utf8_unicode_ci” (not “utf8_general_ci”) for the collation, and I wanted it all to work on RDS with ActiveRecord on Rails and Rack.

Here are my steps:

1. use mysqldump to extract the old data as “latin1″
2. use sed to replace “latin1″ with “utf8″ in the dump file
3. create the new database with the right parameters: character set utf8 collate utf8_unicode_ci
4. use mysql --default-character-set=utf8 to pipe the converted dump into the new database

Here is my code:

# Dump the old database as latin1, because ironically, mysqldump defaults to utf8.
mysqldump --default-character-set=latin1 db > db.dump

# If you need to convert a MySQL dump from one character set to another, use iconv.
iconv -f LATIN1 -t UTF-8 < db.dump > db.dump

# If you've been running mysqldump without parameters on a latin1 instance, you can convert the dump from UTF8 to latin1 to correct it.
iconv -f UTF-8 -t LATIN1 < db.dump > db.dump

# Rewrite the dump to say 'utf8' and 'utf8_unicode_ci' in all the right places.
sed -e 's/SET NAMES latin1/SET NAMES utf8/g' -i db.dump
sed -e 's/CHARSET=latin1/CHARSET=utf8 COLLATE=utf8_unicode_ci/g' -i db.dump

# Create a new database with the correct parameters.
create database db character set utf8 collate utf8_unicode_ci;
# Verify it.
show create database db;

# Pipe the converted database dump into MySQL.
mysql -h hostname --default-character-set=utf8 -u root -p db < db.dump

To verify the character set and collation, you can always query the MySQL variables:

show variables like 'collation%';
show variables like 'character%';

Amazon RDS

For those using Amazon’s AWS RDS for their MySQL instance, you have to create a parameter group with “utf8″ values. I’d guess you could just modify the current parameter group then apply it, but I haven’t verified that.

# Create a parameter group.
rds-create-db-parameter-group utf8 -e mysql5.1 -d utf8

# Modify the parameter group's values
rds-modify-db-parameter-group utf8 \
    --parameters="name=character_set_server, value=utf8, method=immediate" \
    --parameters="name=character_set_client, value=utf8, method=immediate" \
    --parameters="name=character_set_results,value=utf8,method=immediate" \
    --parameters="name=collation_server, value=utf8_unicode_ci, method=immediate" \
    --parameters="name=collation_connection, value=utf8_unicode_ci, method=immediate"

# Check the parameter group's values.
rds-describe-db-parameters utf8 --source=User

# Push this new parameter group to your instance.
rds-modify-db-instance rds-db --db-parameter-group-name utf8

# Reboot the instance (necessary: http://aws.amazon.com/articles/Amazon-RDS/2935).
rds-reboot-db-instance rds-db

ActiveRecord in Rails and Rack

ActiveRecord supports an :encoding option in its parameters for ActiveRecord::Base.establish_connection. The option tells the connection to execute SET NAMES as soon as the connection is established, thereby telling the server what the character set the client wants.

However, I also wanted to specify the collation. When I added :encoding to ActiveRecord::Base.establish_connection, collation_connection (from the MySQL variables, not connection.collation) remained as “utf8_general_ci”. Some people have indicated that you can specify :collation in database.yml for the establish_connection call, but that never worked for me. I think MySQL bug #34980 prevented it. Others indicated that you can simply add ActiveRecord::Base.connection.execute("set collation_connection='utf8_unicode_ci'") at the bottom of environment.rb for Rails; that also never worked for me. To specify the collation in Rails, I used a before_filter in application_controller.rb. See my code below.

# Rack: add a statement right after establishing the connection.
ActiveRecord::Base.establish_connection(
  :adapter  => "mysql",
  :host     => "host",
  :username => "username",
  :password => "password",
  :database => "database",
  :encoding => "utf8",
  :reconnect => true
)
ActiveRecord::Base.connection.execute("SET collation_connection='utf8_unicode_ci'");

# Rails: add a before_filter in application_controller.rb
before_filter :set_database_collation
def set_database_collation
  ActiveRecord::Base.connection.execute("set collation_connection='utf8_unicode_ci'")
end

Pointers

I pieced my steps together from the following helpful links:

• iconv
• UTF8 on RDS
• Official AWS steps
• Derek Sivers converting CD Baby
• RDS Parameter group
• UTF8 through the stack
• UTF8 with Rails
• Character Hell in MySQL
• MySQL character set

First, I wanted to see how fast, so I wrote a short bash script that sleeps for increasing periods of time. I found that the connection timed out after five minutes.

let time=0
while true; do
  let time=time+10
  echo timeout is $time seconds
  sleep $time
done

I tried creating a ~/.ssh/config file and telling the connection to stay alive: TCPKeepAlive yes. No luck, because the default time period is two hours.

Instead, I ignored the TCP connection’s state and told the SSH client to send a null packet every twenty seconds to keep the connection alive. The SSH client times out if it doesn’t receive a server response after ten packets.

# ~/.ssh/config
TCPKeepAlive no
ServerAliveInterval 20
ServerAliveCountMax 10

Thanks to Andy Skelton for making this tidbit googleable.

Frankly, I physically removed the left Command key after that to prevent my muscle memory from using it. That worked quite well.

# crontab allows variables.
PATH=/home/user/.gems/bin:/usr/local/bin:/usr/bin:/bin
GEM_PATH=/home/user/.gems:/usr/lib/ruby/gems/1.8
@hourly sh -c '/usr/bin/ruby /home/user/script.rb'

# sudo can preserve the environment with '-E'.
sudo -E ruby /home/user/script.rb

“Nothing about the iPad is obviously revolutionary, but it didn’t need to be: the iPhone OS and iPhone hardware are already revolutionary.

Apple already reinvented John’s mobile computing and my input mechanics and novice usability in 2007 with the iPhone. We’ve had the truly magical and revolutionary product this entire time, but we take it for granted now, and we’ve forgotten how awesome it already is.”
- Marco Arment from Tumblr and Instapaper

“The iPad as a particular device is not necessarily the future of computing. But as an ideology, I think it just might be. In hindsight, I think arguments over ‘why would I buy this if I already have a phone and a laptop?’ are going to seem as silly as ‘why would I buy an iPod if it has less space than a Nomad?’”
- Steven Frank from Panic

Not exactly filling the void with an iPad post, particularly one that’s two weeks late to the party and starts with what smarter people have already said. I blame Things for not being more insistent.

I agree that Apple’s iPad is the next step in computing: simplified computing. Yes, it has its shortcomings (inherent to a 1.0 release, which the iPhone had only three years ago), but it keeps with the design decisions that made the iPhone/iPod Touch ridiculously popular.

To illustrate the need, let me tell you about the “Check disc” incident. Over Christmas, my Dad subscribed to the New York Times Crossword Puzzle. The NYT doesn’t have an online puzzle component, so he had to download Across Lite, a third party Windows app. He called me after using it for a couple days about a “disc” issue. Whenever he tried to save the day’s puzzle, the app displayed, “Cannot write to file. Check disc.” It was such a weird error that I wanted to see it for myself. With VMWare Fusion on my Mac, I installed Across Lite and got the same error within a minute. I didn’t actually figure out what the problem was (read-only directory?); instead, I simply told my Dad how to use the “Save As…” option to save the puzzle to the Desktop. Problem solved.

But the larger problem is not solved. The incident is a perfect example of the abstraction problems that current computing environments have. They want to deal in apps and documents and photos, but they still break these abstractions daily. Windows gives me app shortcuts available from the Start menu, but those are a mediocre hack to abstract away the monstrous Program Files folder. Mac apps are nicely treated as atomic objects (even though they are folders), but people still run them out of read-only DMGs, leading to equally confusing error messages. All these semi-porous abstractions lead to a lot of unnecessary confusion.

The iPhone OS enforces the abstraction to a much better degree, and we’ve seen the results: over two billion apps downloaded on seventy million devices in eighteen months. All these little apps are islands of data, a per-app version of the Newton’s data soup platform. The iPad simply extends this OS to a larger surface.

I want one.

WPtouch is very simple to install and instantly turned my blog into an iPhone app on Mobile Safari. Check out the Photos screenshot. It uses the flickRSS plugin to generate a Photos page. Really neat. And, it’s free!

I’m pretty happy with the iteration.

Aaron Johnson posted a great analogy: “We need better distilleries, not better harvesters.” It’s simple to collect more information; we’ve gotten very good at that. The next step is distilling that information down into something easily consumable. That’s where companies like FriendFeed (for RSS feeds) and Xobni (for email) come in. They’re trying to solve the problem of filtering. How do we want email prioritized? How do we see the most important items first? While those guys refine their filtering systems, the best solution is to just ask yourself, “What do I actually care about?” and let the rest go. At least that’s what I tell myself when I stare at the 173 feeds in my Google Reader.

I spent five minutes with the woman behind the gas station register, going back and forth about what exactly I did. Finally, she looked at my receipt and looked at her printout of credit card charges. No match. The gas pump machine had given me a receipt, but it hadn’t actually charged my card. I had no idea those two processes weren’t an atomic transaction.