1. Support for documenting native Ruby C code
2. Incremental file parsing and HTML generation
3. Improved yri tool with support for linking gems

Support for Documenting Native Ruby C Code

This one was certainly a long time coming. YARD has always planned to support native Ruby code. This was probably the last "big" feature that RDoc could do and YARD could not. Now, YARD can.

The best part of this feature is that YARD can now parse Ruby’s core codebase and the stdlib, meaning yardoc.org can now host Ruby-core docs. And it now does: http://yardoc.org/docs/ruby-core. The stdlib will be added soon.

Incremental file parsing and HTML generation

One annoying thing about documenting code with YARD for larger projects is the time it takes to generate HTML. This makes the documentation development cycle slow for previewing small documentation changes. YARD 0.5.0 introduces incremental parsing and HTML generation to parse and generate HTML for only the files that were modified since the last running of yardoc, which makes for super-fast previews. To use the incremental parsing feature, simply add -c or --use-cache and YARD will use the available .yardoc cache.

$ yardoc --use-cache

Improved yri tool with support for linking gems

The yri tool that bundled with YARD prior to this release only worked for the .yardoc file in the local directory. In YARD 0.5.0, yri will now perform a lookup on all installed gems on your system. To use this, you’ll first need to build the .yardoc files for your gems, and you can do this with yardoc --build-gems. After that, you can use yri as normal:

$ yri JSON#load

More?

You can see more new toys in the What’s New document on yardoc.org. You can also install YARD:

$ gem install yard

Enjoy!

Here’s a quick little hack to get MySQL 2.8.1 using UTF-8 in Rails 2.3.4 and Ruby 1.9.1.

Filename: lib/mysql_utf8.rb

Put that snippet in your Rails project (I used lib/mysql_utf8.rb) and load it in your environment.

That's all. Now your queries should use Unicode:

$ ./script/console
Loading development environment (Rails 2.3.4)
>> u = User.find(1)
=> #<User id: 1, name: "Test", ...>
>> u.name.encoding
=> #<Encoding:ASCII-8BIT>
>> require 'lib/mysql_utf8'
=> []
>> u = User.find(1)
=> #<User id: 1, name: "Test", ...>
>> u.name.encoding
=> #<Encoding:UTF-8>

Note that if you have BLOB types in MySQL you’ll need to force_encoding back to another type. You could do this in your model.

So here we go, the issue:

You’re using Rails, and you see this:

invalid byte sequence in US-ASCII

You might also see:

incompatible character encodings: ASCII-8BIT and UTF-8

That happens all the time when trying to get YARD to parse Rails source. It also happens when your ERB templates and your DB data have mismatched encodings.

The Problem

Ruby has multiple default encoding values: internal (the default encoding for new String objects), external (the default encoding for file data), and script (the default encoding of the content of Ruby scripts). When dealing with IO, you need to make sure both your internal and external encodings match up, especially since Ruby defaults to ASCII-7BIT and not UTF-8. The “script” encoding problems were covered in my last blog post on the subject. 

The Fix

All you need to do is tweak your environment variables (your region / language may differ):

$ export LANG=en_US.UTF-8

Now Ruby will use UTF-8 for your external encodings:

$ LANG=en_US.ASCII-7BIT irb19
>> __ENCODING__
=> #<Encoding:US-ASCII>
$ LANG=en_US.UTF-8 irb19
>> __ENCODING__
=> #<Encoding:UTF-8>

Note, this doesn’t cover your default_internal, but usually this will be handled for you. And if you can’t set your ENV, you can set this stuff right in Ruby:

Encoding.default_internal, Encoding.default_external = ['utf-8'] * 2

This sets both your default internal and external encodings.

Now when Rails (well, ERB) tries to read files on disk, it will default to UTF-8 rather than ASCII, and your UTF-8 data from the DB will work just fine.

If you have this problem the other way around (your DB is ASCII), just reverse everything I said.

In this first article I’m going to implement a standard Markaby-style DSL that takes declarative Ruby method calls and translates them into HTML. The example only really highlights one metaprogramming trick, namely passing a block to instance_eval to have code executed in the context of an arbitrary object. The rest is just fluff, so it should be simple enough to follow.

First off, if you haven’t already, you’ll probably want to pull an updated version of TextMate’s support files from the svn repository to get some necessary 1.9 compatibility fixes.

$ cd ~/Library/Application\ Support/TextMate/
$ svn co http://macromates.com/svn/Bundles/trunk/Support

You’ll need to clone my Git.tmbundle repository into the Bundles directory to get most of the changes I made to his bundle to deal with the basic Ruby 1.9 issues.

After that, you should get most of the basic functionality of the bundle, but you’ll probably notice that if you try to compare branches, revisions, or any of the more fun stuff, you’ll get the following error:

ArgumentError: An object in the argument tree could not be converted
    from (irb):17:in `to_plist'
    from (irb):17
    from /usr/local/bin/irb:12:in `<main>'

This is because TextMate ships with a native plist library in its support directory (the one we checked out above) that isn’t quite Ruby 1.9 compatible. We’re lucky though, because an updated version of the library, osx-plist (written by Kevin Ballard) is on GitHub, so let’s pull and build it.

$ cd ~/Library/Application\ Support/TextMate/Support/lib/osx
$ git clone git://github.com/kballard/osx-plist.git
$ cd osx-plist/ext/plist
$ ruby extconf.rb && make
$ cp plist.bundle ../../../

Assuming you have TextMate using Ruby 1.9, you should now be able to compare revisions and branches, and do pretty much anything with the awesome Git.tmbundle!

Most blogs and articles I’ve seen that try to explain continuations get it wrong. They bore you with low level details that confuse you in the first place, or scare you by starting with the premise that “it’s complicated, but I’ll try to make you understand”. I’m not going to do either, I’m actually going to explain continuations in two words, and I want you not to laugh, cry, or vomit on yourself when you read them:

GLORIFIED GOTOs

Oh no, I just said a dirty word, didn’t I. You may have your own irrational fears of the goto statement, but you probably know what they are and how they work:

#include <stdio.h>
int main() {
  int i = 0;
  printf("%d\n", i);
  goto some_label;
  i = 1;
  some_label:
  printf("%d\n", i);
  return 0;
}

The above snippet of working C shows how we can jump to different points in a function. It prints 0 twice instead of 0 followed by 1, as you would expect. Language evolution has mostly deprecated this form of programming through method calls and, more intricately, exception handling. These new practices are really just syntactic sugar on the old goto concept. Hopefully all those who scoff at any mention of goto will one day sit down and realize that they’re a central part of how computers work, but I’m getting off topic. The point is, whether you use gotos or not, they’re extremely simple to understand. In fact, they’re exactly as simple as continuations; let’s re-write the above using those “scary” guys:

def main
  i = 0
  callcc do |label| # callcc gives us ‘label’, a continuation object
    puts i
    label.call # this is our "goto" statement
    i = 1      # we skip this completely
  end          # this is where our "label" would otherwise sit
  puts i
end

To those in the know, this doesn’t really show off the full-extent of callcc’s powers (that comes later), but you can look at "label.call" as the equivalent of "goto label", and the end of the callcc block is where the label would be. That’s basically how continuations work.

If you’re having trouble parsing the above: the little trick to understanding the code is simply realizing that the block that callcc yields is executed immediately, once, and only once. It’s simply there so you can do any one-time initialization with the continuation you’re creating. The end of the block is where any subsequent calls to the yielded continuation object will go, not the block itself.

NOW YOU KNOW.

So what, is it really that simple? Continuations are another way to write gotos? The answer is: pretty much. There are actually only two functional difference that makes them even more powerful than the above C-style gotos, but one that makes them a little less-so.

More Powerful, How?

1. They’re not local to your method. Simply put, we can’t do the following in C:

// a() is called by main()
void a() {
  printf("hello world\n");
  label1:
  printf("then you say...\n");
  b();
}
void b() {
  printf("then I say...\n");
  goto label1;
}

This should print "hello world" followed by "then you say", "then I say", in a never-ending loop. Problem is, it won’t compile. Without getting into the nitty-gritty of why C can’t do this, I’ll just simply show how we can actually get away with it using continuations:

def a
  puts "hello world"
  callcc {|cc| $label1 = cc } # pretend this says "label1:"
  puts "then you say..."
  b
end

def b
  puts "then I say"
  $label1.call # pretend this says "goto label1"
end

As the comments indicate, this is almost a one-to-one translation of the C code above. Our continuation object is set to a global variable, essentially making it a “global label” that can be goto’d by running the method #call (picture it as “label.goto” instead of #call).

2. They maintain stack-frame state. That’s the reason why labels in C are local to a function, because you can’t just jump between functions without switching stack frames. Actually, there are tricks in C to do just that (namely longjmp and setjmp to jump to arbitrary addresses and continue execution which is actually how continuations are done in C), but we’re focusing on Ruby here, right? Basically, the continuation object yielded from callcc (the {|cc| thing}) contains a snapshot of our stack frame at the point it was yielded at (remember, the block is only yielded that first time). This allows us to jump between methods in a class, between classes, and jump back from a nested call when we are arbitrarily deep in a stack frame.

Anyone Up For A Quick Real World Example?

The last situation is the one with some real world usage. Imagine we’re writing a web framework with a Rails-style before_filter and we want the ability to halt the execution during a filter and jump all the way back up to the router code to find the next matching route. Sinatra (sort-of) does this with the `pass` method, and though it probably doesn’t use continuations to achieve the result, the concept here is a perfect example of wanting to jump back through our stack frames arbitrarily. We could be 3 method calls deep, or we could be 20 method calls deep, but we need to “go to” a specific point in our program execution. Usually, people implement this with try/catch style exception handling (RouteAbortError maybe), but continuations may be a little cleaner depending on the scenario (we might have a method handling the exception 3 method calls up, but we want to jump 7 method calls up to the initial point in code).

Less Powerful, How?

There’s one thing you may have noticed from the initial example, it’s that jumping forward looks different from the example where we jumped backwards. The limitation is that continuations can’t actually jump forwards, at least not between method calls. The reason is that in C, labels are compiled into the program statically, but in Ruby, the continuation objects are created at runtime. This means that we’d need to execute that callcc call in our future method to generate the continuation, but we can’t run code that hasn’t yet been run. In short, continuations are great for jumping back in time, not quite so for jumping forwards.

Final Correction

So I initially called continuations “glorified gotos”. Hopefully this got you to understand the simple concept behind their use; but now I should fess up and modify the definition just a little bit to be slightly more accurate. Instead of glorified gotos, think of continuations as:

GLORIFIED, STATEFUL, BACKWARDS JUMPING, GOTOs

Now that you know what the fuss is about, you can decide to hate continuations all you want.

invalid multibyte char (US-ASCII)

- OR -

CompatibilityError: incompatible encoding regexp match (Windows-31J regexp with UTF-8 string)

The errors themselves are relatively self-explanatory. Ruby 1.9 is far more Unicode aware than 1.8, and this error happens when have some Unicode (usually UTF-8) in one of your files. What you have to do to fix these, however, is not always as straightforward.

Well, after some time pulling my hair out, I’ve figure out that the solutions to these issues are actually quite simple.

Invalid multibyte char (encoding here)

If you get this issue, add the following to the top of each exploding file (below the shebang if there is one):

# encoding: utf-8

Note: You can also use "coding:" or even the Emacs style -*- encoding: utf-8 -*- but I like the simple term, ‘encoding’. Also note that you might need to replace ‘utf-8′ wth your specific encoding if it’s something else.

This should resolve the issue.

Basically, Ruby by default assumes that every file is encoded as US-ASCII, and so when it reads UTF-8 (or any Unicode) it freaks out because it is beyond the 7-bit encoding. You have to tell it that the file is encoded as utf-8 by listing it as we did at the top of the file. Yes, it’s a little anal, but I’m sure we’ll get used to it.

Incompatible encoding regexp match

This issue is a little bit hairier. I had this problem with the BlueCloth 1.0.0 gem (on line 972 of bluecloth.rb). It turns out that there are a few switches that turn on specific encodings, and for some reason BlueCloth turns on the //s switch which enables the SJIS encoding (maybe John Gruber wanted //m for multiline?). Ruby 1.8 didn’t mind having this on, but 1.9 freaks out. Moral of the story, when you see this error, check your Regexp switches.

1. Hash entries are Ordered!

h = { cat: 1, zebra: 2, dog: 3 }
h.delete(:zebra)
h[:monkey] = 4
h.each {|key, val| puts "#{key}: #{val}" }

# Guaranteed to print:
# cat: 1
# dog: 3
# monkey: 4

I’ve been praying for this for who knows how long. Everyone knows hashes don’t maintain order internally, but much of Ruby development makes use of Hashes as dictionaries because of the simplistic syntax. The problem is that dictionaries, unlike straight hash objects, usually require order to be maintained. There are tons of places in YARD where I had to resort to nested Arrays (eg. [[key, val], ...]) to maintain order of a set of associated objects… I’ve even written countless OrderedHash classes to solve this problem. This is definitely the biggest feature in 1.9 to me.

PS. You might have noticed the Pythonish {a: 1} syntax— yep, that’s new too, but it’s not quite cool enough to make it’s own item in our list but it’s really convenient to type.

2. Symbols are now Intuitively Comparable

YARD calls #to_s on about a million things. Last I checked, Rails does too. This is because internally we try to keep everything stored as a Symbol for obvious efficiency. The problem is that when writing API’s we’re usually allowing developers to pass in a String or a Symbol, which means comparison is always annoying. It looks like the Ruby devs looked at a lot of this code, puked all over their keyboards, and cleaned up both their keyboards and the code to deal with this. We can now do things like:

:hello =~ /e/      # => 1
:hello === "hello" # => true
:hello[1] == "e"   # => true

3. Enumerations on a Hash return a Hash

For the same reasons as above, this is awesome. Performing a #select on a hash was always a pain in Ruby 1.8 because you would end up with an Array… totally not a Hash. This inconsistency has been dealt with and we can now do:

{a:1,b:2,c:3}.reject {|k,v| k == :b }
=> {:a=>1, :c=>3}

4. Proper Unicode Support

Let’s not get too far into this list without giving credit for the awesome work Ruby 1.9 has done in fixing the Unicode support. In short, it is now possible to do things like:

"Hi!".encode("utf-16be") # => "\x00H\x00i\x00!"
File.read('test.txt', encoding: 'utf-8').encoding

We also get Enumerations like #each_char that iterate properly over such strings. There’s plenty more goodies regarding Unicode, but I still don’t know all of the details. Interestingly enough, I ran into a Ruby Unicode problem the other day, so I think I’ll be revisiting the problem with these new tools soon and write about what happened.

5. Regexps get Look-ahead/Look-behind

Probably the most powerful feature of regular expressions have been missing from Ruby for the longest time. I can’t remember any specific times when I hit this limitation, but I imagine it will simplify a lot of hacky code attempting to work around the feature omission in 1.8.

6. Object #tap

This is an easy one because anyone could have implemented it, but it’s a nice way to silently hook into a method call chain. The Ruby 1.9 docs give a cool example of this, but I think a better example is when dealing with method calls that aren’t chainable like Hash#delete:

# In 1.8 we need to do this because #delete returns nil:
h = {:a => 1, :b => 2, :c => 3}
h.delete(:b)
h.each {|k,v| puts k }

# In 1.9 we can do this in one line:
{a:1,b:2,c:3}.tap {|o| o.delete(:b) }.each {|k,v| puts k }

7. Fibers are lighter Threads

Still have yet to dig into this one, but Dave Thomas covers the new Fiber class. In short, they’re basically lambdas encapsulated in threads— but of course it’s not the implementation but the way you use these guys that makes them quite elegant. I can think of a few things right off the top of my head that involve the quick creation of threads from inline storable procedures.

8. New Hash Key Syntax

UPDATE: Okay, I wasn’t going to put this in the list until I realized how the syntax can be applied to method calls. Consider the following 1.8 code:

def open(filename, opts = {}) end

open('test.txt', :access => :read, :close => true)

The same code in 1.9 can be called as:

def open(filename, opts = {}) end

open('test.txt', access: :read, close: true)

That is cool.