1. A Local Documentation Server

RubyGems has gem server to serve static docs of your gems. The problem here, again, is that to get these static .html files we need to force our users to generate docs when the gem installs, and this is often a painfully slow process (especially with behemoths like Rails). YARD takes a different approach to this, only parsing out the source and saving an intermediate dump file when a gem installs. This is pretty fast (~2 seconds under 1.9 for YARD’s codebase). Not only can a developer use this data to their heart’s content, but it allows YARD to generate HTML/PDF/LaTeX on demand, and that’s exactly what 0.6 is going to do.

The yard server command coming in 0.6 will generate HTML documentation for gems on demand from the parsed source (perhaps even parsing on demand). This means your CPU will never be used to generate static files that you never look at. If you only need to access 10% of the documentation of 1% of your gems, YARD will only generate that 0.1%. Big win.

Not to mention you get all the benefits a YARD template provides, including full class hierarchy, less obtrusive layouts, markdown support, and even full method/class name searching with the potential of having full-text searching in the future, since this is all running in a Webrick/Rack adapter.

All of this stuff is currently being ported from the existing yardoc.org/docs site, so it will basically look like that. The existing code is in the docserver branch of YARD’s github and is being tracked as gh-139.

2. Method Listing by Groups

We’ve seen this in Apple’s docs, doxygen docs and a few others. The ability to organize methods in a large class by functionality rather than alphabet is often far more helpful to your users. YARD 0.6 will be introducing a @group freeform tag to identify methods under a specific organizational header which will be used in the templates. Still working on the exact syntax of this one, trying to make it as easy to use and remove as much duplication as possible. This feature is being tracked as gh-143.

3. Generate Documentation for Single File Scripts

RDoc has a --one-file option that is occasionally used for small Ruby scripts. YARD has always lacked this option. Though it’s kind of simple to roll your own with templates, 0.6 will have built-in templates for generating a single HTML document. Not really sure what it’s going to look like just yet, but if you have ideas or want to help out, the feature is being tracked as gh-115.

4. New yard CLI executable

YARD is growing into a tool that does much more than just parse source. This means that having a single executable is not enough, and spreading out the functionality over multiple executables is just messy. YARD currently uses the yard-graph tool to generate Graphviz dot files. In YARD 0.6, all of these tools, including the new documentation server, will all be merged into a single executable called yard with respective subcommands (doc, graph, server, etc.). The goal here is to open this up for plugins to add commands just like they do in RubyGems without resorting to new executable names. And don’t worry, the yardoc executable will probably stick around for legacy support. This feature is being tracked as gh-140.

5. Improved Template APIs

If you’ve ever written or overridden a template in YARD you know that it can be hard to figure out what section to override with the sections command. YARD has a sections.place(:foo).before(:bar) idiom, but it does not help with subsections. In 0.6, the sections will be redesigned from being Array objects to a more proper tree-like structure in order to handle accessing subsections. Now instead of:

def init
  sections.first.last.place(:foo).before(:bar)
end

You will have the ability to do the following:

def init
  sections.first.subsections.place(:foo).before(:bar)
  # or even:
  sections.place(:foo).before_first(:bar)
end

The last method will look for any "bar" inside sections or subsections within the tree. All in all, this should make it more effective to override a template. This feature is being tracked as gh-58.

6. Documentation for DSL Handlers and Other Use Cases

YARD packs in a lot of functionality, and many users have complained that they had to dig into source to find out how to do some relatively simple things like writing handlers for DSL syntaxes. Although YARD does have documentation on many things, including writing handlers in general, there are few "use-case specific" documents that describe how to do specific things with YARD. 0.6 will introduce improved documentation for some of these use cases, most importantly extending YARD for custom DSL syntaxes. It's possible this may start as a project in the Github Wiki as a set of how-to guides.

So there they are, the major new features coming to 0.6. Let me know if you have any suggestions about other features. And if you have time to donate to the project that would be great, see the above Github issue links and help out. Money would also be appreciated; I wrote YARD because I want to improve docs in the community, not because I want money, but I need to eat. You can donate below:

The release plan for YARD 0.6 is: "soon". It's hard to say when it will be out, but things are certainly rolling in that direction.

Update: 04/02/2010 – Thanks to Martin Luder who pointed out that you can in fact do visibility in Eiffel without the inheritance “export” syntax. Updated the example.

I’ve used Eiffel a couple of times. I’ve even blogged about it once. There’s one really cool feature in Eiffel that lives below the whole DbC radar. Peculiarly, the feature itself deals with "features". Basically, Eiffel allows you to decide which classes can see which "features" (in Ruby features are basically methods) on an individual basis. Therefore, Eiffel has no need for a half-baked "public", "protected", "private" solution to the visibility problem because by exporting features you have extreme control over exactly how public, protected or private a feature can be. Here’s what it looks like:

We Rubyists like to call ourselves engineers. For the most part, there are a lot of great engineering practices being applied in the realm of testing. Continuous integration, acceptance testing and other automated testing are some pretty advanced techniques that are not so well-practiced in other communities. We should all be proud of ourselves in this respect.

But you can’t just do testing right and call yourself an all-around engineer. There are still plenty of engineering concepts that Rubyists ignore or completely reject but can prove pretty useful for a well structured project. Techniques like the use of design patterns, metrics, traceability and estimation are all important but seem to be completely ignored by most Rubyists I know. I’m going to cover these all in parts over the next few weeks starting with design patterns, because it’s the easiest to cover.

Note that I can’t teach you patterns. You probably know most of them anyway. If not, please visit Wikipedia or buy Design Patterns. I just want to point out that you should pay more attention to them. And here’s why:

Denial is a River

The issue here is that although many Rubyists make use of design patterns every day, some of them (who shall remain unnamed) refuse to accept that they are actually making use of patterns, citing that the names of these patterns imply a complexity that is not actually being expressed. The "complexity" here usually comes from the fact that these patterns are mostly applied in static languages (like Java) where design patterns are occasionally more difficult to implement due to all the extra moving parts, compilation and explicit typing. None of this makes these patterns any less effective or any more restricted to Java (or C++, or …). Patterns are still patterns whether you acknowledge them or not, and they are not defined by the LOC or number of classes they require in language X. I mean, come on, Rails used to have "MVC" and "ActiveRecord" plastered on the website to sell its new web development technique, two patterns ripped right out of P of EAA. Now MVC is a household name. Rails is the ultimate testament that a pattern can be simple to implement in a simple language. Refusing to admit that patterns are used hurts your development team and the community, because patterns (simply identifying them) serve a very important purpose.

Why Patterns Matter

Pat Maddox brought this up at his MWRC2010 talk a few weeks ago. I forget his exact wording, but the paraphrasing goes a little something like: "patterns convey meaning". This is entirely true. I think it was Pat who also brought up the idea of metaphors in Agile development at a MWRC lunch. Metaphors in Agile are the idea that the team decides on a metaphor that expresses some high level concept of the project, and they use this metaphor exclusively to convey this concept. This saves time because usually these metaphors are one or two words. It also keeps everyone on the same page, because they all know (or can look up) what the metaphor really means. The differences between two metaphors can be subtle but have large impact on the functionality of a system. For example, consider the metaphors "email" and "instant messaging". Although emails can be short and instant and instant messages can be long and very email-like, we already know (mostly from experience) that the functionality of these two systems involve some significantly different user interactions. If your client wanted "email" and got "instant messaging", they would notice the difference and probably complain.

Well, design patterns are just like these metaphors, but they are meant for code, not UI. Just like metaphors, they convey a very specific meaning in one or two words. This means they save everyone time. They can also be looked up, because there is a pretty good set of standard design patterns (GoF, Fowler) that people can look up and immediately understand. These patterns can have very subtle differences depending on the terminology used. For instance, consider the factory pattern versus the builder pattern. Both are concerned with creating objects, but a builder is also concerned with the assembling of the object. This is a relatively subtle but pretty noticeable difference. Telling someone you are using a "builder pattern" should immediately ring a bell in their heads saying, "oh, it’s not just a factory".

These patterns also come with another interesting benefit: authority. We know these patterns work, because they’ve been running in production code for decades. There is no argument about whether the factory pattern works, because it does. We know this. When someone proposes a design and tells you "I’m going to use X because it’s worked in hundreds of previous projects that have had this same problem," that’s real engineering.

A Tiny Case Study

But back to the topic of saving time: just the other week James Golick blogged about how he writes Rails apps. Now, James is a really smart guy, and he’s way ahead of the curve when it comes to good engineering. But really, his whole blog post could have been replaced with the statement: "Here’s how I do Rails apps: I don’t create domain objects directly in my controller. Instead I create all of my domain objects through the Command pattern. <example here>" End of blog post.

Now of course this is an exaggeration, there’s some explanation of how this relates to his tests, coupling and logging. But a lot of this discussion and assertion that it’s smart is also unnecessary when we describe the design through the command pattern. We know what a command pattern does, what problems it solves, and why it is used. We know that it’s meant to solve coupling issues, help implement SRP properly and comes with the implicit benefit of composability and dependency injection. More importantly, we know there’s nothing heretical about his methods at all; he’s using an authoritative design used by many others in this situation. No justification necessary, James.

Some to Get You Started

Now I know I said I wouldn’t teach you patterns, but here are a few you should definitely look into if you’re developing any kind of client-server architecture these days:

• Adapter
• Command
• Decorator
• Facade
• Factory
• Strategy
• Unit of Work

If you’ve never heard of any of these, look them up now. I use most of these daily, and you should be using them as well (you probably already are). Being aware of their existence can help you rethink some of your designs and find ways to make them much simpler by applying design patterns that have worked many times before. More importantly, when you review your current architecture (as any good engineer periodically does), having the ability to associate your design with existing, working, authoritative patterns should certainly give you some confidence in your system.

Last night I thought of a really neat trick to solve this long-running “to alias_method_chain or not to alias_method_chain” feud. It’s completely transparent, meaning it requires absolutely no modification to your original code. More importantly, it essentially changes Ruby’s inheritance model to what I would consider the “correct” model, with respect to mixin behaviour.

A Little Background

The issue that people bump their heads against can be expressed by the following Ruby example. Say I’m using a library that defines some class ‘A’:

class A
  def foo; "foo" end
end

Being the Rubyist that I am, I want to be able to extend this method and add my own behaviour, but without wiping out the original. This is where alias_method_chain traditionally comes in. I would normally just rename the method and call it myself from my overridden method. The problems with that have been discussed ad infinitum, and people have realized that mixing in a module is proper OO, much cleaner, maintains inheritance, etc.. so let’s do that:

module B; def foo; super + "bar" end end
class A; include B end

Problem! You probably noticed this. "But ‘foo’ is defined directly in A," you say, "you can’t override it with a mixin!"

This is the ultimate problem with Ruby’s inheritance model, in my opinion. Because mixins always take lower priority than methods defined directly in a class body, A#foo can not be overridden by mixin inclusion. Moreover, because mixins take lower priority than methods, A#foo is now violating encapsulation, but to see this we will need to modify our example slightly.

What is Proper Encapsulation, Anyway?

Encapsulation is the idea that only a component knows about its internal structure. But structure in OOP implies your inheritance tree as much as it implies your data. Suppose our ‘A’ class inherited some class ‘X’ and the A#foo method we’ve been calling really calls the super method X#foo:

class X
  def foo; "foo" end
end
class A < X; def foo; super end end

When our class ‘A’ calls super, it expects that super should be class ‘X’, because this is its internal structure as defined by A. By this standard, mixing in another module should not change A’s internal expectations. Mixing in a module should create the following encapsulation, where B encapsulates A:

This is what mixing in a module after A has been created should look like. However, Ruby does not wrap our ‘A’ class, instead it injects itself under A:

Here we have modified the "internal structure" of A, violating proper encapsulation. Whether or not this is "right" or "wrong" depends on how you view the concept of re-opening a class in Ruby. However, pragmatically speaking, this behaviour causes a lot of problems with the way people tend to write code in Ruby-land (and, in general, many languages). Firstly, it requires that developers design for the possibility that a module would be inserted into itself rather than wrapped around, which means to properly accept mixins you should always call "super" in all of your methods since you never know what your inheritance tree will look like. Secondly, it requires you to opt-in to allowing a method to be overridden by a mixin by performing very explicit code transformations that make your code harder to document and add needless complexity. I’m referring to the ClassMethods and InstanceMethods module idioms that Rails likes to use. These are problematic because most documentation tools will show these methods as being "inherited" by the class (if at all), and although this is technically correct, these methods now need to be addressed and looked up as ClassName::InstanceMethods#method_name which is unintuitive for the user. Of course this is in addition to actually organizing your code in this manner, which is, in general, an unintuitive step. These are all workarounds to deal with the failings of a proper inheritance model.

Solving Encapsulation Transparently

So how can we solve the inheritance/encapsulation problem and do so with as small a footprint as possible? This is what I came up with last night:

class Object
  def self.method_added(name)
    return if name == :initialize
    const_set(:InstanceMethods, Module.new) unless defined?(self::InstanceMethods)
    meth = instance_method(name)
    self::InstanceMethods.send(:define_method, name) {|*args, &block| meth.bind(self).call(*args, &block) }
    remove_method(name)
    include self::InstanceMethods
  end
end

This simple hook transparently defines all new methods inside an anonymous inner module, essentially doing this source transformation for us on the fly. As mentioned, it requires no modification to the source code you’re trying to modify, simply add this code before including the rest. For instance, with the above hook, we can now do the following with our original example:

class A; def foo; "foo" end end
module B; def foo; super + "bar" end end

A.new.foo # => "foo"
A.send(:include, B)
A.new.foo # => "foobar"

Note that we need not change the code from our original example and our mixed in module now takes precedence over the explicitly defined A#foo.

Hold on There, Cowboy

"But that looks so inefficient!" you scream. For those who noticed, you will see that we’ve been replacing every method with a delegate Proc object that performs quite a lot of crap before dispatching another method call. Yes, it’s inefficient. There is, however, a better way to do this, but it involves native code. I’ve implemented the native version for 1.9.x, but I’ll spare you the details and code. If you care, you can check the github link (see below).

The Metamorph Gem (1.9 Only)

In 1.9 you can do a gem install metamorph to get this gem. It’s a native gem, so it requires a compile. For 1.8 (or Windows machines) you can use the above non-native version, but it will probably be slower.

The code is on github at: http://github.com/lsegal/metamorph, so if you want to make it work in 1.8, feel free.

PS. It’s not that it’s impossible to implement (efficiently) in 1.8, it’s simply that I don’t personally care about your legacy Rubies the same way DHH never cared about your legacy DBs. Oddly enough, people seem to love their legacy Rubies and still manage to hate legacy databases, but that’s a rant for another day.

I love visualization toolkits, so when I found out about the JS visualization toolkit InfoVis, I had to play with it. Generally the first thing that comes to mind when visualizing data is class model relationships, so I figured I’d write another snippet to export data from YARD into a JS-friendly format and see how it looked. This time, I’ll spare the show-and-tell and just do the showing, since the code to export this data is fairly trivial.

I basically used the few examples from the site but plugged in class relationship data from Ruby libraries to get the following visualization of YARD in particular:

Visualization 1: Simple Tree View

Pretty simple, we can also make it look a little fancier:

Visualization 2: Polar Tree View

Visualization 3: Coupling

It then occurred to me that visualizations like these (weighted graphs in particular) would be a good way to visualize class coupling in your library. I experimented with the concept by generating a weighted graph, but it seems InfoVis’ RGraph class (which they demo a weighted graph with) isn’t really that great at really visualizing the weights, also it doesn’t seem to support weighted graphs. The end result shows some interesting data, but it’s not exactly great to really view coupling.

Note: FF doesn’t seem to like this one. Don’t be surprised if it doesn’t work.

Those are the visualizations I played with. There’s a nicer version here that lets you see visualizations for Ruby’s core lib and Rails in addition to YARD’s, plus it has some extra niceties like showing methods in the current highlighted class on the side (not enough space to show it on my blog). Check it out.

Thanks to Josh Martin for pointing out InfoVis today, by the way.

1. Support for documenting native Ruby C code
2. Incremental file parsing and HTML generation
3. Improved yri tool with support for linking gems

Support for Documenting Native Ruby C Code

This one was certainly a long time coming. YARD has always planned to support native Ruby code. This was probably the last "big" feature that RDoc could do and YARD could not. Now, YARD can.

The best part of this feature is that YARD can now parse Ruby’s core codebase and the stdlib, meaning yardoc.org can now host Ruby-core docs. And it now does: http://yardoc.org/docs/ruby-core. The stdlib will be added soon.

Incremental file parsing and HTML generation

One annoying thing about documenting code with YARD for larger projects is the time it takes to generate HTML. This makes the documentation development cycle slow for previewing small documentation changes. YARD 0.5.0 introduces incremental parsing and HTML generation to parse and generate HTML for only the files that were modified since the last running of yardoc, which makes for super-fast previews. To use the incremental parsing feature, simply add -c or --use-cache and YARD will use the available .yardoc cache.

$ yardoc --use-cache

Improved yri tool with support for linking gems

The yri tool that bundled with YARD prior to this release only worked for the .yardoc file in the local directory. In YARD 0.5.0, yri will now perform a lookup on all installed gems on your system. To use this, you’ll first need to build the .yardoc files for your gems, and you can do this with yardoc --build-gems. After that, you can use yri as normal:

$ yri JSON#load

More?

You can see more new toys in the What’s New document on yardoc.org. You can also install YARD:

$ gem install yard

Enjoy!

Here’s a quick little hack to get MySQL 2.8.1 using UTF-8 in Rails 2.3.4 and Ruby 1.9.1.

Filename: lib/mysql_utf8.rb

Put that snippet in your Rails project (I used lib/mysql_utf8.rb) and load it in your environment.

That's all. Now your queries should use Unicode:

$ ./script/console
Loading development environment (Rails 2.3.4)
>> u = User.find(1)
=> #<User id: 1, name: "Test", ...>
>> u.name.encoding
=> #<Encoding:ASCII-8BIT>
>> require 'lib/mysql_utf8'
=> []
>> u = User.find(1)
=> #<User id: 1, name: "Test", ...>
>> u.name.encoding
=> #<Encoding:UTF-8>

Note that if you have BLOB types in MySQL you’ll need to force_encoding back to another type. You could do this in your model.

Barracuda is my latest experiment. I wanted to toy around with the new features of Snow Leopard, and as far as I’ve seen, there are no CUDA or OpenCL wrapper libraries out there for Ruby.

OpenCL, if you don’t know, is an “open” version of CUDA, which is NVIDIA’s proprietary architecture for GPGPU programming. Okay, acronyms out of the way, this means that OpenCL is a way to run small but heavy computations in parallel on your GPU’s hundred or so cores. There’s a great demo of why this is cool at MacResearch (skip to the end of the video).

I should point out that while the library is called Barracuda (did you get it?), there’s currently no CUDA support. I want to add support for CUDA down the road after I figure out what’s involved, though.

Show and Tell Time

Anyway, long story short, I wrote a very basic wrapper for OpenCL which currently only supports signed integers and floats, but will hopefully add some more functionality soon. I know you’re all dying for a demo, so, here’s a benchmark of some integer to float related computation in Ruby, versus the same computation using Barracuda:

require 'barracuda'
require 'benchmark'

include Barracuda

prog = Program.new <<-eof
  __kernel sum(__global float *out, __global int *in, int total) {
    int i = get_global_id(0);
    if (i < total) out[i] = ((float)in[i] + 0.5) / 3.8 + 2.0;
  }
eof

arr = (1..3333333).to_a
input = Buffer.new(arr)
output = OutputBuffer.new(:float, arr.size)

Benchmark.bmbm do |x|
  x.report("cpu") { arr.map {|x| (x.to_f + 0.5) / 3.8 + 2.0 } }
  x.report("gpu") { prog.sum(output, input, arr.size) }
end

The way OpenCL works is you compile a C-like program on the GPU, then run it on the GPU with your data input and get some output; it’s a standard Pipes and Filters architecture. The compilation of the program happens through LLVM, meaning it doesn’t require GCC, so this is not like running RubyInline. Also we’re running these programs on like 200 cores, so there’s that part too. Anyway, “where are the benchmark results?” you must be yelling right now, so here:

Rehearsal ---------------------------------------
cpu   5.510000   0.250000   5.760000 (  5.767632)
gpu   0.630000   0.260000   0.890000 (  0.930966)
------------------------------ total: 6.650000sec

          user     system      total        real
cpu   3.860000   0.020000   3.880000 (  3.931644)
gpu   0.310000   0.040000   0.350000 (  0.366625)

Okay, it’s only about a 10x speed increase. That’s not too impressive. Keep in mind though, (x + 0.5) / 3.8 + 2.0 is a freaking simple computation. I dare someone to hook this up to some Hadoop map-reduce and see some real world results, because they’d probably be much more impressive.

Wanna Try It?

The catch: you can install Barracuda if you’re on OSX 10.6 (Snow Leopard) or have a way to get OpenCL on your machine and hack up the Makefiles. I’d like to add support for other environments but I just finished this build.

If you have the right stuff you can grab it right from rubyforge:

sudo gem install barracuda

Or visit the git repo:

http://github.com/lsegal/barracuda