Here’s a quick little hack to get MySQL 2.8.1 using UTF-8 in Rails 2.3.4 and Ruby 1.9.1.

Filename: lib/mysql_utf8.rb

Put that snippet in your Rails project (I used lib/mysql_utf8.rb) and load it in your environment.

That's all. Now your queries should use Unicode:

$ ./script/console
Loading development environment (Rails 2.3.4)
>> u = User.find(1)
=> #<User id: 1, name: "Test", ...>
>> u.name.encoding
=> #<Encoding:ASCII-8BIT>
>> require 'lib/mysql_utf8'
=> []
>> u = User.find(1)
=> #<User id: 1, name: "Test", ...>
>> u.name.encoding
=> #<Encoding:UTF-8>

Note that if you have BLOB types in MySQL you’ll need to force_encoding back to another type. You could do this in your model.

So here we go, the issue:

You’re using Rails, and you see this:

invalid byte sequence in US-ASCII

You might also see:

incompatible character encodings: ASCII-8BIT and UTF-8

That happens all the time when trying to get YARD to parse Rails source. It also happens when your ERB templates and your DB data have mismatched encodings.

The Problem

Ruby has multiple default encoding values: internal (the default encoding for new String objects), external (the default encoding for file data), and script (the default encoding of the content of Ruby scripts). When dealing with IO, you need to make sure both your internal and external encodings match up, especially since Ruby defaults to ASCII-7BIT and not UTF-8. The “script” encoding problems were covered in my last blog post on the subject. 

The Fix

All you need to do is tweak your environment variables (your region / language may differ):

$ export LANG=en_US.UTF-8

Now Ruby will use UTF-8 for your external encodings:

$ LANG=en_US.ASCII-7BIT irb19
>> __ENCODING__
=> #<Encoding:US-ASCII>
$ LANG=en_US.UTF-8 irb19
>> __ENCODING__
=> #<Encoding:UTF-8>

Note, this doesn’t cover your default_internal, but usually this will be handled for you. And if you can’t set your ENV, you can set this stuff right in Ruby:

Encoding.default_internal, Encoding.default_external = ['utf-8'] * 2

This sets both your default internal and external encodings.

Now when Rails (well, ERB) tries to read files on disk, it will default to UTF-8 rather than ASCII, and your UTF-8 data from the DB will work just fine.

If you have this problem the other way around (your DB is ASCII), just reverse everything I said.

invalid multibyte char (US-ASCII)

- OR -

CompatibilityError: incompatible encoding regexp match (Windows-31J regexp with UTF-8 string)

The errors themselves are relatively self-explanatory. Ruby 1.9 is far more Unicode aware than 1.8, and this error happens when have some Unicode (usually UTF-8) in one of your files. What you have to do to fix these, however, is not always as straightforward.

Well, after some time pulling my hair out, I’ve figure out that the solutions to these issues are actually quite simple.

Invalid multibyte char (encoding here)

If you get this issue, add the following to the top of each exploding file (below the shebang if there is one):

# encoding: utf-8

Note: You can also use "coding:" or even the Emacs style -*- encoding: utf-8 -*- but I like the simple term, ‘encoding’. Also note that you might need to replace ‘utf-8′ wth your specific encoding if it’s something else.

This should resolve the issue.

Basically, Ruby by default assumes that every file is encoded as US-ASCII, and so when it reads UTF-8 (or any Unicode) it freaks out because it is beyond the 7-bit encoding. You have to tell it that the file is encoded as utf-8 by listing it as we did at the top of the file. Yes, it’s a little anal, but I’m sure we’ll get used to it.

Incompatible encoding regexp match

This issue is a little bit hairier. I had this problem with the BlueCloth 1.0.0 gem (on line 972 of bluecloth.rb). It turns out that there are a few switches that turn on specific encodings, and for some reason BlueCloth turns on the //s switch which enables the SJIS encoding (maybe John Gruber wanted //m for multiline?). Ruby 1.8 didn’t mind having this on, but 1.9 freaks out. Moral of the story, when you see this error, check your Regexp switches.

1. Hash entries are Ordered!

h = { cat: 1, zebra: 2, dog: 3 }
h.delete(:zebra)
h[:monkey] = 4
h.each {|key, val| puts "#{key}: #{val}" }

# Guaranteed to print:
# cat: 1
# dog: 3
# monkey: 4

I’ve been praying for this for who knows how long. Everyone knows hashes don’t maintain order internally, but much of Ruby development makes use of Hashes as dictionaries because of the simplistic syntax. The problem is that dictionaries, unlike straight hash objects, usually require order to be maintained. There are tons of places in YARD where I had to resort to nested Arrays (eg. [[key, val], ...]) to maintain order of a set of associated objects… I’ve even written countless OrderedHash classes to solve this problem. This is definitely the biggest feature in 1.9 to me.

PS. You might have noticed the Pythonish {a: 1} syntax— yep, that’s new too, but it’s not quite cool enough to make it’s own item in our list but it’s really convenient to type.

2. Symbols are now Intuitively Comparable

YARD calls #to_s on about a million things. Last I checked, Rails does too. This is because internally we try to keep everything stored as a Symbol for obvious efficiency. The problem is that when writing API’s we’re usually allowing developers to pass in a String or a Symbol, which means comparison is always annoying. It looks like the Ruby devs looked at a lot of this code, puked all over their keyboards, and cleaned up both their keyboards and the code to deal with this. We can now do things like:

:hello =~ /e/      # => 1
:hello === "hello" # => true
:hello[1] == "e"   # => true

3. Enumerations on a Hash return a Hash

For the same reasons as above, this is awesome. Performing a #select on a hash was always a pain in Ruby 1.8 because you would end up with an Array… totally not a Hash. This inconsistency has been dealt with and we can now do:

{a:1,b:2,c:3}.reject {|k,v| k == :b }
=> {:a=>1, :c=>3}

4. Proper Unicode Support

Let’s not get too far into this list without giving credit for the awesome work Ruby 1.9 has done in fixing the Unicode support. In short, it is now possible to do things like:

"Hi!".encode("utf-16be") # => "\x00H\x00i\x00!"
File.read('test.txt', encoding: 'utf-8').encoding

We also get Enumerations like #each_char that iterate properly over such strings. There’s plenty more goodies regarding Unicode, but I still don’t know all of the details. Interestingly enough, I ran into a Ruby Unicode problem the other day, so I think I’ll be revisiting the problem with these new tools soon and write about what happened.

5. Regexps get Look-ahead/Look-behind

Probably the most powerful feature of regular expressions have been missing from Ruby for the longest time. I can’t remember any specific times when I hit this limitation, but I imagine it will simplify a lot of hacky code attempting to work around the feature omission in 1.8.

6. Object #tap

This is an easy one because anyone could have implemented it, but it’s a nice way to silently hook into a method call chain. The Ruby 1.9 docs give a cool example of this, but I think a better example is when dealing with method calls that aren’t chainable like Hash#delete:

# In 1.8 we need to do this because #delete returns nil:
h = {:a => 1, :b => 2, :c => 3}
h.delete(:b)
h.each {|k,v| puts k }

# In 1.9 we can do this in one line:
{a:1,b:2,c:3}.tap {|o| o.delete(:b) }.each {|k,v| puts k }

7. Fibers are lighter Threads

Still have yet to dig into this one, but Dave Thomas covers the new Fiber class. In short, they’re basically lambdas encapsulated in threads— but of course it’s not the implementation but the way you use these guys that makes them quite elegant. I can think of a few things right off the top of my head that involve the quick creation of threads from inline storable procedures.

8. New Hash Key Syntax

UPDATE: Okay, I wasn’t going to put this in the list until I realized how the syntax can be applied to method calls. Consider the following 1.8 code:

def open(filename, opts = {}) end

open('test.txt', :access => :read, :close => true)

The same code in 1.9 can be called as:

def open(filename, opts = {}) end

open('test.txt', access: :read, close: true)

That is cool.