I love visualization toolkits, so when I found out about the JS visualization toolkit InfoVis, I had to play with it. Generally the first thing that comes to mind when visualizing data is class model relationships, so I figured I’d write another snippet to export data from YARD into a JS-friendly format and see how it looked. This time, I’ll spare the show-and-tell and just do the showing, since the code to export this data is fairly trivial.

I basically used the few examples from the site but plugged in class relationship data from Ruby libraries to get the following visualization of YARD in particular:

Visualization 1: Simple Tree View

Pretty simple, we can also make it look a little fancier:

Visualization 2: Polar Tree View

Visualization 3: Coupling

It then occurred to me that visualizations like these (weighted graphs in particular) would be a good way to visualize class coupling in your library. I experimented with the concept by generating a weighted graph, but it seems InfoVis’ RGraph class (which they demo a weighted graph with) isn’t really that great at really visualizing the weights, also it doesn’t seem to support weighted graphs. The end result shows some interesting data, but it’s not exactly great to really view coupling.

Note: FF doesn’t seem to like this one. Don’t be surprised if it doesn’t work.

Those are the visualizations I played with. There’s a nicer version here that lets you see visualizations for Ruby’s core lib and Rails in addition to YARD’s, plus it has some extra niceties like showing methods in the current highlighted class on the side (not enough space to show it on my blog). Check it out.

Thanks to Josh Martin for pointing out InfoVis today, by the way.

You can now use yri with Ruby core classes. This is relatively old news (as of yesterday) since YARD can now parse core classes, but Ruby source is generally not available on all installs, so building the .yardoc database is not always an option.

In YARD 0.5.1 (released today), instead of downloading and parsing the Ruby source manually, you can now download a pre-built gem containing the .yardoc files for a bunch of Ruby versions (1.8.6, 1.8.7, 1.9.1). Simply use:

$ gem install yard-doc-core

You should now be able to use something like yri String#split. By default this will install docs for 1.9.1, but if you need docs for a specific version of Ruby, add -v VERSION to your gem install command:

$ gem install yard-doc-core -v 1.8.7

The stdlib package (yard-doc-stdlib) will be coming soon.

1. Support for documenting native Ruby C code
2. Incremental file parsing and HTML generation
3. Improved yri tool with support for linking gems

Support for Documenting Native Ruby C Code

This one was certainly a long time coming. YARD has always planned to support native Ruby code. This was probably the last "big" feature that RDoc could do and YARD could not. Now, YARD can.

The best part of this feature is that YARD can now parse Ruby’s core codebase and the stdlib, meaning yardoc.org can now host Ruby-core docs. And it now does: http://yardoc.org/docs/ruby-core. The stdlib will be added soon.

Incremental file parsing and HTML generation

One annoying thing about documenting code with YARD for larger projects is the time it takes to generate HTML. This makes the documentation development cycle slow for previewing small documentation changes. YARD 0.5.0 introduces incremental parsing and HTML generation to parse and generate HTML for only the files that were modified since the last running of yardoc, which makes for super-fast previews. To use the incremental parsing feature, simply add -c or --use-cache and YARD will use the available .yardoc cache.

$ yardoc --use-cache

Improved yri tool with support for linking gems

The yri tool that bundled with YARD prior to this release only worked for the .yardoc file in the local directory. In YARD 0.5.0, yri will now perform a lookup on all installed gems on your system. To use this, you’ll first need to build the .yardoc files for your gems, and you can do this with yardoc --build-gems. After that, you can use yri as normal:

$ yri JSON#load

More?

You can see more new toys in the What’s New document on yardoc.org. You can also install YARD:

$ gem install yard

Enjoy!

Chrome is an awesome browser. I mean, it’s WebKit, it’s V8, it’s super fast. My problem with Chrome, as with every single Google product ever, is the UI. It’s extremely opinionated, and imposes a non-standard UI that simply rubs me the wrong way

Google likes to break rules. Unfortunately the best UI’s are ones that follow many rules and break only the ones that really need breaking. Apple has an OS that touts UI uniformity to the near extremes, and this works extremely well among reviews and critics alike. Google should be taking advice from companies like these, not attempting to blaze their own trail; they certainly don’t have the credibility in UX to set any trends. So what are my problems? There are three, and they are as follows:

Problem 1: The Tabs

Chrome has this obsession with putting tabs on top of the address bar. While this is an interesting UI choice, there’s no real benefit, and there are a few noticeable losses. Firstly, you lose out on long titles, even when the tab is active, because Chrome completely ignores the standard Windows/OSX titlebar concept (yes, it sucks on OSX too):

The only way to get the full title is to hover over the tab, waiting there for your mouse to show you something you could have read three times by now. Sure, this is really a minor issue in itself, but what’s the real benefit? The aesthetics are not nice enough to justify a breaking of the rules here. Simply put, this UI is being tailored to their Chrome OS, not to my OS. In OSX it looks even more broken, because of the dead gray space and window control buttons on the left side:

I should also point out, Apple tried “tabs on top” in Safari 4.0 beta and then reverted the change. Google should have taken the hint.

You may have also noticed that 10-12 pixels of dead vertical space above the tabs that don’t go away (more pronounced in Vista/7). We’ll get back to that one later.

Problem 2: No Dedicated Search Bar

Yes, the address bar doubles as a search bar. I get that. But it’s not what I want. There is something beneficial about being able to have your search terms separated from your current address:

This matters when I delve into long searches spanning multiple search results or pages, because I still have my search terms safely stowed away. I might realize, “hey, those aren’t quite the right keywords”, or even, “wow, this page is awesome, what did I search to get here?” – all that information will be available without digging in my history. It’s also great if I want to start back at square one, since I don’t have to re-type my search term or navigate backwards. Oh, and I didn’t even begin to mention the search extension support you get for free with a dedicated search bar.

You might like a combined address/search bar. I don’t. The thing is, there should be extensibility support for those of us that don’t. Unfortunately, Google’s extensions have no provisions to make any modifications to the UI besides horribly ugly, useless and limited theming support. Again, there’s no real justification for changing the standard browser layout here except the extremely minimal benefit of extra simplicity, and there at least a few drawbacks.

Problem 3: No Dedicated Status Bar

This one’s probably controversial, but I want a dedicated status bar too. I like having a status bar. It shows me when my page is loading, it tells me what I’m hovering over, but best of all, it does it without being visually disruptive.

The justification for this one is touted as “Look Ma, now you have 12 extra pixels of vertical space!!”. Great, I’ll add it back to the 12 pixels of vertical space you wasted with the tabs:

I just don’t buy this. Websites aren’t optimized for vertical space. In what situation have you thought, “damn, I wish I just had twelve more pixels so I could read that last line of text, now I have to scroll!”?

But even if you actually are like that, status bars have actual functional benefits beyond giving you the URL of a link. My Firefox status bar has plenty of “status”-like information for me:

Sure, the “Done” is somewhat pointless, but the rest of it is not. Plus, I imagine as Chrome extensions become more prevalent, they will need somewhere to put their “status information” too. The idea that the only use of a status bar is to show hovered links is a naive way of looking at the UI, and it shows you really don’t understand what the purpose of a status bar is in the first place.

They Would Fix It, But They Can’t

As mentioned before, the extension support in Chrome makes no provisions for modifying the UI. It might just be a product of its new-ness, but I’ve yet to see any official response from Google on these issues, so I’m doubtful anything will be done. In fact, from the stance they took on justifying the UI when it initially came out, I’d say these UX features are steadfast design decisions and are unlikely to change. Until then, I’m “stuck” with the slower, more bloated, Firefox (on Windows; Safari serves me just fine on OSX).

Since I don’t write many Rails apps, I figured I’d select some of the most popular open source rails apps out there and generate ER diagrams for those. It turned out to be an interesting exercise in visualizing database structures of some of the projects people use day to day.

The projects I chose were: Instiki, Gemcutter, Seed, Radiant, Mephisto, Insoshi and Spree. The results range from awesome to scary, but you can judge for yourselves.

Results

Instiki

Nice and simple. Maybe too simple?

Gemcutter

Here’s a nice one. Sufficiently complex but still easy to follow. I should point out at this point that dashed lines refer to has-many-through relationships (with the attr name as the arrow label.

Seed

This is probably the cleanest one. The use of STI in the models really makes this look nice.

Radiant

Simple, but those recursive relationships look a little messy, don’t they? I guess it would look better if the layout was done right. Note that those labels again refer to the attribute name.

Mephisto

Manageable.

Insoshi

Now we’re getting crazy. yUML is starting to fail on us.

Click the image to see the madness

Spree

This one isn’t exactly fair. I think the amount of complexity here is a little out of the scope of a single UML diagram. It’s fun to look at, though.

Click the image to see the madness

How?

All of these diagrams were generated automatically with a simple YARD handler and piped into yUML. The code is on gist:

SPEC = Gem::Specification.new do |s|
  ...
  s.has_rdoc = 'yard'
end

If you use BlueCloth or another lib to do your markdown processing, you’ll need to add that as a dependency to your gem.

And with that, YARD will generate docs when your gem is installed if YARD is on the user’s system. This works with RubyGems 1.3.1+. Best part of all, it integrates perfectly seamlessly with your RubyGems doc server. gem server will show YARD docs for the rdoc link of your project.

Comments

The Use Case

To show how it’s done, we’ll be implementing the following use case: we have a set of tests (RSpec syntax) that we want integrated and mentioned in our documentation. For those who haven’t used it, RSpec allows use to write our tests in the format:

describe ClassName do
  describe '#method' do
    it "should behave like this" { ... }
    it "should behave like that" { ... }
  end
end

This syntax makes it very easy to integrate this information into our docs. We would want our ClassName#method method to list the following “Specifications”:

• should behave like this
• should behave like that

YARD does this in a two step process: the first is data processing (where source code is parsed) and the second is the template rendering (when our HTML docs are generated). I’ll be listing the code for the data processing but we’ll be mostly skimming over it as the focus of this article is on the templates.

Processing the Specification Data

The crux of our plugin is the following code. It parses through our RSpec tests, grabs and stores the necessary information for our templates:

File listing: rspec_plugin.rb

YARD::Templates::Engine.register_template_path Pathname.new(YARD::ROOT).join('..', 'templates_custom')
YARD::Parser::SourceParser.parser_type = :ruby18

class RSpecDescribeHandler < YARD::Handlers::Ruby::Legacy::Base
  MATCH = /\Adescribe\s+(.+?)\s+(do|\{)/
  handles MATCH

  def process
    objname = statement.tokens.to_s[MATCH, 1].gsub(/["']/, '')
    obj = {:spec => owner ? (owner[:spec] || "") : ""}
    obj[:spec] += objname
    parse_block :owner => obj
  end
end

class RSpecItHandler < YARD::Handlers::Ruby::Legacy::Base
  MATCH = /\Ait\s+['"](.+?)['"]\s+(do|\{)/
  handles MATCH

  def process
    return if owner.nil?
    obj = P(owner[:spec])
    return if obj.is_a?(Proxy)

    (obj[:specifications] ||= []) << {
      :name => statement.tokens.to_s[MATCH, 1],
      :file => parser.file,
      :line => statement.line,
      :source => statement.block.to_s
    }
  end
end

The first line registers our custom template path for when we override our templates (later on). This can also be specified at the command line (we will see that method too). We also have to set the parser to use the legacy parser API. We can write this to support the Ruby-1.9-only parser as well, but I want to keep it universal and simple (the legacy API works in both 1.8 and 1.9). There is also a command line switch to enable the legacy parser.

It’s likely that you can get a general sense of what the rest of the code does. Basically, it builds up the object name whenever it reads a “describe” statement, and attaches any “it” statements to the object that it’s currently pointing to. We can now access these specifications in our templates from the specifications attribute we set on the object.

This file will be loaded into yard when we call yardoc and the data will be collected and rendered properly. We will see the exact command arguments later.

A Crash Course in the Templates API

The new template system in YARD allows for users to non-intrusively modify existing templates by (literally) “mixing in” features from other templates and overriding behaviour at a very granular level. In our case, we want to inject a segment of HTML (or plaintext) in the middle of the existing YARD templates, but we don’t want to directly modify the templates.

Each template is represented by a directory inside one of our template paths. It is also represented by a logical module automatically defined in code for us. For instance, the template at the path “default/method” will be managed by the module YARD::Templates::Engine::Template__default_method. We can use this module to define helpers and most importantly mix in (or inherit) other templates in order to override behaviour.

Inside each directory sits an optional setup.rb file which defines the body of our logical module. This is where we can include (literally) other templates. This is also where we define our “sections”. Sections (kind of like partials) are what make up the template. A setup.rb file will generally have an #init method which calls sections to set the list of sections the template uses. A section can either be a method, .erb file or another template. Basically, when our template is rendered, the resulting String will be the result of running each individual section. A standard setup.rb file looks like:

def init
  sections :section1, :section2, ...
end

def optional_helper_method; ... end

If section1 and section2 are not defined methods, they will be rendered as erb files with the filenames of section1.erb and section2.erb respectively.

You might now see how we can override and add custom data by “inheriting” such a template and then modifying the sections list like so:

def init
  super
  sections.push :mysection
end

We could also insert our section into the middle by using some YARD extensions to the Array class:

def init
  super
  sections.place(:mysection).before(:section2)
end

This is essentially what we will be doing with our RSpec templates.

Creating the Templates

Using that basic overview above, we will be modifying the "default/method_details" template in YARD by overriding it in our own custom template path, inserting our section in the init method and adding the necessary .erb file. The directory structure will look like this:

You can see our setup.rb file and specs.erb files. Notice that we are defining the template for multiple formats, html and text. YARD looks for the template at “template/FORMAT” to support multiple formats for a template. As mentioned above, any directory, including subdirectories, is a template. In addition, any subdirectory will automatically inherit the parent directory template, which is incidentally where our setup.rb resides. This allows us to define the sections used by all our formats at once.

One extra trick not mentioned in the previous section is that we are following the exact directory structure of the YARD templates themselves. Whenever a template is found in another template path with a matching directory structure, it is also automatically included (or inherited from). So by registering our custom template path and following the same structure, we are implicitly inheriting all of the templates and have the ability to override them with classic Ruby OOP.

Okay, enough with the confusion, here is the code for the three files in the above diagram:

File listing: setup.rb

def init
  super
  sections.last.place(:specs).before(:source)
end

You may notice this is almost exactly what we saw in the crash course. We override an included template, which in our case is the template sitting in YARD’s templates/default/method_list.

Now for the erb files:

Fie listing: templates_custom/default/method_details/html/specs.erb

<% if object[:specifications] %>
<div class="tags">
  <h3>Specifications:</h3>
  <ul class="see">
    <% for spec in object[:specifications] %>
      <li><%= spec[:name] %>
        <div class="source_code">
          <table>
            <tr>
              <td>
                <pre class="lines">

<%= spec[:source].split("\n").size.times.to_a.map {|i| spec[:line] + i }.join("\n") %></pre>
              </td>
              <td>
                <pre class="code"><span class="info file"># File '<%= h spec[:file] %>', line <%= spec[:line] %></span>

<%= html_syntax_highlight format_source(spec[:source]) %></pre>
              </td>
            </tr>
          </table>
        </div>
      </li>
    <% end %>
  </ul>
</div>
<% end %>

To support the plaintext format (used by the yri tool), we can add the following:

File listing: templates_custom/default/method_details/text/specs.erb

<% if object[:specifications] %>

Specifications:
---------------

<% for spec in object[:specifications] %>
<%= indent wrap("- " + spec[:name]) %>
<% end %>
<% end %>

Both templates are fairly straightforward. Let’s use them!

Putting it All Together

We now have our rspec_plugin.rb and our templates_custom directory sitting somewhere. We can generate YARD documentation for our code and specs using the command

$ yardoc -e rspec_plugin 'lib/**/*.rb' 'spec/**/*.rb'

This loads our rspec_plugin.rb file and uses the handler to parse our specs (after parsing our regular code). It then uses the registered template path (templates_custom, defined in our .rb file) to find our overridden templates.

I mentioned earlier that we can specify the template path in the command line and tell YARD to use the legacy parser. Here’s how:

$ yardoc --legacy -e rspec_plugin -p templates_custom 'lib/**/*.rb' 'spec/**/*.rb'

Our docs should now have an extra section in them. Here’s what YARD’s YARD::Templates::Engine.render method looks like with specifications:

Packaging as a Plugin

Until now we’ve been manually loading our ruby script ‘rspec_plugin.rb’ whenever we call yardoc. This is simple for per-project customization, but when you want to use multiple customizations, it becomes difficult. YARD 0.4.x adds support for plugins by auto-loading any gem starting with the prefix yard-. This means we can create a gem for our code, install it and have YARD load our plugin when it boots up. For example, this plugin can be found by running the following command

$ sudo gem install yard-rspec --source http://gemcutter.org

Now, whenever you run yardoc or yri, YARD will automatically document RSpec code. Just remember to add spec/**/*.rb to your file path. We can now see our text templates in action:

$ yri ClassName#method

Conclusion

This method of template customization in YARD can be applied to many other situations and is what makes it far more powerful than RDoc when it comes to properly documenting Ruby code. The other great part about YARD templates is that multiple plugins can make independent modifications to a template with very little conflict. This RSpec plugin can be used in tandem with your own custom plugins and you wouldn’t have to rewrite either one in order to use them together.

Now go, create your own awesome templates!

Source code for the yard-rspec plugin can be found on GitHub at http://github.com/lsegal/yard-spec-plugin

Here’s a quick little hack to get MySQL 2.8.1 using UTF-8 in Rails 2.3.4 and Ruby 1.9.1.

Filename: lib/mysql_utf8.rb

Put that snippet in your Rails project (I used lib/mysql_utf8.rb) and load it in your environment.

That's all. Now your queries should use Unicode:

$ ./script/console
Loading development environment (Rails 2.3.4)
>> u = User.find(1)
=> #<User id: 1, name: "Test", ...>
>> u.name.encoding
=> #<Encoding:ASCII-8BIT>
>> require 'lib/mysql_utf8'
=> []
>> u = User.find(1)
=> #<User id: 1, name: "Test", ...>
>> u.name.encoding
=> #<Encoding:UTF-8>

Note that if you have BLOB types in MySQL you’ll need to force_encoding back to another type. You could do this in your model.

So here we go, the issue:

You’re using Rails, and you see this:

invalid byte sequence in US-ASCII

You might also see:

incompatible character encodings: ASCII-8BIT and UTF-8

That happens all the time when trying to get YARD to parse Rails source. It also happens when your ERB templates and your DB data have mismatched encodings.

The Problem

Ruby has multiple default encoding values: internal (the default encoding for new String objects), external (the default encoding for file data), and script (the default encoding of the content of Ruby scripts). When dealing with IO, you need to make sure both your internal and external encodings match up, especially since Ruby defaults to ASCII-7BIT and not UTF-8. The “script” encoding problems were covered in my last blog post on the subject. 

The Fix

All you need to do is tweak your environment variables (your region / language may differ):

$ export LANG=en_US.UTF-8

Now Ruby will use UTF-8 for your external encodings:

$ LANG=en_US.ASCII-7BIT irb19
>> __ENCODING__
=> #<Encoding:US-ASCII>
$ LANG=en_US.UTF-8 irb19
>> __ENCODING__
=> #<Encoding:UTF-8>

Note, this doesn’t cover your default_internal, but usually this will be handled for you. And if you can’t set your ENV, you can set this stuff right in Ruby:

Encoding.default_internal, Encoding.default_external = ['utf-8'] * 2

This sets both your default internal and external encodings.

Now when Rails (well, ERB) tries to read files on disk, it will default to UTF-8 rather than ASCII, and your UTF-8 data from the DB will work just fine.

If you have this problem the other way around (your DB is ASCII), just reverse everything I said.