Customizing YARD Templates

I usually sift over this little snippet of an extension when I give talks on YARD. I haven’t written it down until now because the templating system was pretty much up in the air for the last year or so. Now that the new templating system in YARD is out, I can finally explain in detail how simple it is to customize the existing YARD templates, or even create your own.

The Use Case

To show how it’s done, we’ll be implementing the following use case: we have a set of tests (RSpec syntax) that we want integrated and mentioned in our documentation. For those who haven’t used it, RSpec allows use to write our tests in the format:

describe ClassName do
  describe '#method' do
    it "should behave like this" { ... }
    it "should behave like that" { ... }
  end
end

This syntax makes it very easy to integrate this information into our docs. We would want our ClassName#method method to list the following “Specifications”:

• should behave like this
• should behave like that

YARD does this in a two step process: the first is data processing (where source code is parsed) and the second is the template rendering (when our HTML docs are generated). I’ll be listing the code for the data processing but we’ll be mostly skimming over it as the focus of this article is on the templates.

Processing the Specification Data

The crux of our plugin is the following code. It parses through our RSpec tests, grabs and stores the necessary information for our templates:

File listing: rspec_plugin.rb

YARD::Templates::Engine.register_template_path Pathname.new(YARD::ROOT).join('..', 'templates_custom')
YARD::Parser::SourceParser.parser_type = :ruby18

class RSpecDescribeHandler < YARD::Handlers::Ruby::Legacy::Base
  MATCH = /Adescribes+(.+?)s+(do|{)/
  handles MATCH

  def process
    objname = statement.tokens.to_s[MATCH, 1].gsub(/["']/, '')
    obj = {:spec => owner ? (owner[:spec] || "") : ""}
    obj[:spec] += objname
    parse_block :owner => obj
  end
end

class RSpecItHandler < YARD::Handlers::Ruby::Legacy::Base
  MATCH = /Aits+['"](.+?)['"]s+(do|{)/
  handles MATCH

  def process
    return if owner.nil?
    obj = P(owner[:spec])
    return if obj.is_a?(Proxy)

    (obj[:specifications] ||= []) << {
      :name => statement.tokens.to_s[MATCH, 1],
      :file => parser.file,
      :line => statement.line,
      :source => statement.block.to_s
    }
  end
end

The first line registers our custom template path for when we override our templates (later on). This can also be specified at the command line (we will see that method too). We also have to set the parser to use the legacy parser API. We can write this to support the Ruby-1.9-only parser as well, but I want to keep it universal and simple (the legacy API works in both 1.8 and 1.9). There is also a command line switch to enable the legacy parser.

It’s likely that you can get a general sense of what the rest of the code does. Basically, it builds up the object name whenever it reads a “describe” statement, and attaches any “it” statements to the object that it’s currently pointing to. We can now access these specifications in our templates from the specifications attribute we set on the object.

This file will be loaded into yard when we call yardoc and the data will be collected and rendered properly. We will see the exact command arguments later.

A Crash Course in the Templates API

The new template system in YARD allows for users to non-intrusively modify existing templates by (literally) “mixing in” features from other templates and overriding behaviour at a very granular level. In our case, we want to inject a segment of HTML (or plaintext) in the middle of the existing YARD templates, but we don’t want to directly modify the templates.

Each template is represented by a directory inside one of our template paths. It is also represented by a logical module automatically defined in code for us. For instance, the template at the path “default/method” will be managed by the module YARD::Templates::Engine::Template__default_method. We can use this module to define helpers and most importantly mix in (or inherit) other templates in order to override behaviour.

Inside each directory sits an optional setup.rb file which defines the body of our logical module. This is where we can include (literally) other templates. This is also where we define our “sections”. Sections (kind of like partials) are what make up the template. A setup.rb file will generally have an #init method which calls sections to set the list of sections the template uses. A section can either be a method, .erb file or another template. Basically, when our template is rendered, the resulting String will be the result of running each individual section. A standard setup.rb file looks like:

def init
  sections :section1, :section2, ...
end

def optional_helper_method; ... end

If section1 and section2 are not defined methods, they will be rendered as erb files with the filenames of section1.erb and section2.erb respectively.

You might now see how we can override and add custom data by “inheriting” such a template and then modifying the sections list like so:

def init
  super
  sections.push :mysection
end

We could also insert our section into the middle by using some YARD extensions to the Array class:

def init
  super
  sections.place(:mysection).before(:section2)
end

This is essentially what we will be doing with our RSpec templates.

Creating the Templates

Using that basic overview above, we will be modifying the "default/method_details" template in YARD by overriding it in our own custom template path, inserting our section in the init method and adding the necessary .erb file. The directory structure will look like this:

You can see our setup.rb file and specs.erb files. Notice that we are defining the template for multiple formats, html and text. YARD looks for the template at “template/FORMAT” to support multiple formats for a template. As mentioned above, any directory, including subdirectories, is a template. In addition, any subdirectory will automatically inherit the parent directory template, which is incidentally where our setup.rb resides. This allows us to define the sections used by all our formats at once.

One extra trick not mentioned in the previous section is that we are following the exact directory structure of the YARD templates themselves. Whenever a template is found in another template path with a matching directory structure, it is also automatically included (or inherited from). So by registering our custom template path and following the same structure, we are implicitly inheriting all of the templates and have the ability to override them with classic Ruby OOP.

Okay, enough with the confusion, here is the code for the three files in the above diagram:

File listing: setup.rb

def init
  super
  sections.last.place(:specs).before(:source)
end

You may notice this is almost exactly what we saw in the crash course. We override an included template, which in our case is the template sitting in YARD’s templates/default/method_list.

Now for the erb files:

Fie listing: templates_custom/default/method_details/html/specs.erb

<% if object[:specifications] %>
<div class="tags">
  <h3>Specifications:</h3>
  <ul class="see">
    <% for spec in object[:specifications] %>
      <li><%= spec[:name] %>
        <div class="source_code">
          <table>
            <tr>
              <td>
                <pre class="lines">


<%= spec[:source].split("n").size.times.to_a.map {|i| spec[:line] + i }.join("n") %></pre>
              </td>
              <td>
                <pre class="code"><span class="info file"># File '<%= h spec[:file] %>', line <%= spec[:line] %></span>

<%= html_syntax_highlight format_source(spec[:source]) %></pre>
              </td>
            </tr>
          </table>
        </div>
      </li>
    <% end %>
  </ul>
</div>
<% end %>

To support the plaintext format (used by the yri tool), we can add the following:

File listing: templates_custom/default/method_details/text/specs.erb

<% if object[:specifications] %>

Specifications:
---------------

<% for spec in object[:specifications] %>
<%= indent wrap("- " + spec[:name]) %>
<% end %>
<% end %>

Both templates are fairly straightforward. Let’s use them!

Putting it All Together

We now have our rspec_plugin.rb and our templates_custom directory sitting somewhere. We can generate YARD documentation for our code and specs using the command

$ yardoc -e rspec_plugin 'lib/**/*.rb' 'spec/**/*.rb'

This loads our rspec_plugin.rb file and uses the handler to parse our specs (after parsing our regular code). It then uses the registered template path (templates_custom, defined in our .rb file) to find our overridden templates.

I mentioned earlier that we can specify the template path in the command line and tell YARD to use the legacy parser. Here’s how:

$ yardoc --legacy -e rspec_plugin -p templates_custom 'lib/**/*.rb' 'spec/**/*.rb'

Our docs should now have an extra section in them. Here’s what YARD’s <a href="http://yardoc.org/docs/yard/YARD/Templates/Engine.render">YARD::Templates::Engine.render</a> method looks like with specifications:

Packaging as a Plugin

Until now we’ve been manually loading our ruby script ‘rspec_plugin.rb’ whenever we call yardoc. This is simple for per-project customization, but when you want to use multiple customizations, it becomes difficult. YARD 0.4.x adds support for plugins by auto-loading any gem starting with the prefix yard-. This means we can create a gem for our code, install it and have YARD load our plugin when it boots up. For example, this plugin can be found by running the following command

$ sudo gem install yard-rspec --source http://gemcutter.org

Now, whenever you run yardoc or yri, YARD will automatically document RSpec code. Just remember to add spec/**/*.rb to your file path. We can now see our text templates in action:

$ yri ClassName#method

Conclusion

This method of template customization in YARD can be applied to many other situations and is what makes it far more powerful than RDoc when it comes to properly documenting Ruby code. The other great part about YARD templates is that multiple plugins can make independent modifications to a template with very little conflict. This RSpec plugin can be used in tandem with your own custom plugins and you wouldn’t have to rewrite either one in order to use them together.

Now go, create your own awesome templates!

Source code for the yard-rspec plugin can be found on GitHub at http://github.com/lsegal/yard-spec-plugin