1. A Local Documentation Server

RubyGems has gem server to serve static docs of your gems. The problem here, again, is that to get these static .html files we need to force our users to generate docs when the gem installs, and this is often a painfully slow process (especially with behemoths like Rails). YARD takes a different approach to this, only parsing out the source and saving an intermediate dump file when a gem installs. This is pretty fast (~2 seconds under 1.9 for YARD’s codebase). Not only can a developer use this data to their heart’s content, but it allows YARD to generate HTML/PDF/LaTeX on demand, and that’s exactly what 0.6 is going to do.

The yard server command coming in 0.6 will generate HTML documentation for gems on demand from the parsed source (perhaps even parsing on demand). This means your CPU will never be used to generate static files that you never look at. If you only need to access 10% of the documentation of 1% of your gems, YARD will only generate that 0.1%. Big win.

Not to mention you get all the benefits a YARD template provides, including full class hierarchy, less obtrusive layouts, markdown support, and even full method/class name searching with the potential of having full-text searching in the future, since this is all running in a Webrick/Rack adapter.

All of this stuff is currently being ported from the existing yardoc.org/docs site, so it will basically look like that. The existing code is in the docserver branch of YARD’s github and is being tracked as gh-139.

2. Method Listing by Groups

We’ve seen this in Apple’s docs, doxygen docs and a few others. The ability to organize methods in a large class by functionality rather than alphabet is often far more helpful to your users. YARD 0.6 will be introducing a @group freeform tag to identify methods under a specific organizational header which will be used in the templates. Still working on the exact syntax of this one, trying to make it as easy to use and remove as much duplication as possible. This feature is being tracked as gh-143.

3. Generate Documentation for Single File Scripts

RDoc has a --one-file option that is occasionally used for small Ruby scripts. YARD has always lacked this option. Though it’s kind of simple to roll your own with templates, 0.6 will have built-in templates for generating a single HTML document. Not really sure what it’s going to look like just yet, but if you have ideas or want to help out, the feature is being tracked as gh-115.

4. New yard CLI executable

YARD is growing into a tool that does much more than just parse source. This means that having a single executable is not enough, and spreading out the functionality over multiple executables is just messy. YARD currently uses the yard-graph tool to generate Graphviz dot files. In YARD 0.6, all of these tools, including the new documentation server, will all be merged into a single executable called yard with respective subcommands (doc, graph, server, etc.). The goal here is to open this up for plugins to add commands just like they do in RubyGems without resorting to new executable names. And don’t worry, the yardoc executable will probably stick around for legacy support. This feature is being tracked as gh-140.

5. Improved Template APIs

If you’ve ever written or overridden a template in YARD you know that it can be hard to figure out what section to override with the sections command. YARD has a sections.place(:foo).before(:bar) idiom, but it does not help with subsections. In 0.6, the sections will be redesigned from being Array objects to a more proper tree-like structure in order to handle accessing subsections. Now instead of:

def init
  sections.first.last.place(:foo).before(:bar)
end

You will have the ability to do the following:

def init
  sections.first.subsections.place(:foo).before(:bar)
  # or even:
  sections.place(:foo).before_first(:bar)
end

The last method will look for any "bar" inside sections or subsections within the tree. All in all, this should make it more effective to override a template. This feature is being tracked as gh-58.

6. Documentation for DSL Handlers and Other Use Cases

YARD packs in a lot of functionality, and many users have complained that they had to dig into source to find out how to do some relatively simple things like writing handlers for DSL syntaxes. Although YARD does have documentation on many things, including writing handlers in general, there are few "use-case specific" documents that describe how to do specific things with YARD. 0.6 will introduce improved documentation for some of these use cases, most importantly extending YARD for custom DSL syntaxes. It's possible this may start as a project in the Github Wiki as a set of how-to guides.

So there they are, the major new features coming to 0.6. Let me know if you have any suggestions about other features. And if you have time to donate to the project that would be great, see the above Github issue links and help out. Money would also be appreciated; I wrote YARD because I want to improve docs in the community, not because I want money, but I need to eat. You can donate below:

The release plan for YARD 0.6 is: "soon". It's hard to say when it will be out, but things are certainly rolling in that direction.

I love visualization toolkits, so when I found out about the JS visualization toolkit InfoVis, I had to play with it. Generally the first thing that comes to mind when visualizing data is class model relationships, so I figured I’d write another snippet to export data from YARD into a JS-friendly format and see how it looked. This time, I’ll spare the show-and-tell and just do the showing, since the code to export this data is fairly trivial.

I basically used the few examples from the site but plugged in class relationship data from Ruby libraries to get the following visualization of YARD in particular:

Visualization 1: Simple Tree View

Pretty simple, we can also make it look a little fancier:

Visualization 2: Polar Tree View

Visualization 3: Coupling

It then occurred to me that visualizations like these (weighted graphs in particular) would be a good way to visualize class coupling in your library. I experimented with the concept by generating a weighted graph, but it seems InfoVis’ RGraph class (which they demo a weighted graph with) isn’t really that great at really visualizing the weights, also it doesn’t seem to support weighted graphs. The end result shows some interesting data, but it’s not exactly great to really view coupling.

Note: FF doesn’t seem to like this one. Don’t be surprised if it doesn’t work.

Those are the visualizations I played with. There’s a nicer version here that lets you see visualizations for Ruby’s core lib and Rails in addition to YARD’s, plus it has some extra niceties like showing methods in the current highlighted class on the side (not enough space to show it on my blog). Check it out.

Thanks to Josh Martin for pointing out InfoVis today, by the way.

You can now use yri with Ruby core classes. This is relatively old news (as of yesterday) since YARD can now parse core classes, but Ruby source is generally not available on all installs, so building the .yardoc database is not always an option.

In YARD 0.5.1 (released today), instead of downloading and parsing the Ruby source manually, you can now download a pre-built gem containing the .yardoc files for a bunch of Ruby versions (1.8.6, 1.8.7, 1.9.1). Simply use:

$ gem install yard-doc-core

You should now be able to use something like yri String#split. By default this will install docs for 1.9.1, but if you need docs for a specific version of Ruby, add -v VERSION to your gem install command:

$ gem install yard-doc-core -v 1.8.7

The stdlib package (yard-doc-stdlib) will be coming soon.

1. Support for documenting native Ruby C code
2. Incremental file parsing and HTML generation
3. Improved yri tool with support for linking gems

Support for Documenting Native Ruby C Code

This one was certainly a long time coming. YARD has always planned to support native Ruby code. This was probably the last "big" feature that RDoc could do and YARD could not. Now, YARD can.

The best part of this feature is that YARD can now parse Ruby’s core codebase and the stdlib, meaning yardoc.org can now host Ruby-core docs. And it now does: http://yardoc.org/docs/ruby-core. The stdlib will be added soon.

Incremental file parsing and HTML generation

One annoying thing about documenting code with YARD for larger projects is the time it takes to generate HTML. This makes the documentation development cycle slow for previewing small documentation changes. YARD 0.5.0 introduces incremental parsing and HTML generation to parse and generate HTML for only the files that were modified since the last running of yardoc, which makes for super-fast previews. To use the incremental parsing feature, simply add -c or --use-cache and YARD will use the available .yardoc cache.

$ yardoc --use-cache

Improved yri tool with support for linking gems

The yri tool that bundled with YARD prior to this release only worked for the .yardoc file in the local directory. In YARD 0.5.0, yri will now perform a lookup on all installed gems on your system. To use this, you’ll first need to build the .yardoc files for your gems, and you can do this with yardoc --build-gems. After that, you can use yri as normal:

$ yri JSON#load

More?

You can see more new toys in the What’s New document on yardoc.org. You can also install YARD:

$ gem install yard

Enjoy!

Since I don’t write many Rails apps, I figured I’d select some of the most popular open source rails apps out there and generate ER diagrams for those. It turned out to be an interesting exercise in visualizing database structures of some of the projects people use day to day.

The projects I chose were: Instiki, Gemcutter, Seed, Radiant, Mephisto, Insoshi and Spree. The results range from awesome to scary, but you can judge for yourselves.

Results

Instiki

Nice and simple. Maybe too simple?

Gemcutter

Here’s a nice one. Sufficiently complex but still easy to follow. I should point out at this point that dashed lines refer to has-many-through relationships (with the attr name as the arrow label.

Seed

This is probably the cleanest one. The use of STI in the models really makes this look nice.

Radiant

Simple, but those recursive relationships look a little messy, don’t they? I guess it would look better if the layout was done right. Note that those labels again refer to the attribute name.

Mephisto

Manageable.

Insoshi

Now we’re getting crazy. yUML is starting to fail on us.

Click the image to see the madness

Spree

This one isn’t exactly fair. I think the amount of complexity here is a little out of the scope of a single UML diagram. It’s fun to look at, though.

Click the image to see the madness

How?

All of these diagrams were generated automatically with a simple YARD handler and piped into yUML. The code is on gist:

I’m referring specifically to my blog, which was becoming a complete mess of ugliness and broken HTML. I decided to completely gut it, remove a bunch of useless crap and focus on making the content clean and easy to read. This is pretty much a warm-up exercise for the coming week of writing docs and putting out the new YARD site, but it’s something I should have done long ago. Hopefully this will give me some new incentive to write more (about YARD?)

It’s not a “blog” anymore

You may have noticed that comments are gone. Good. Comments are often useful, but to be honest, like many other “bloggers”, I’ve discovered they often don’t add much to the conversation (I rarely get comments on my blog anyway, so styling all that HTML is a waste of my time). A lot of people will say a site is “not a blog” if it does not allow a two-way conversation between the author and the reader. Fine, then I guess this is no longer a blog. My intention for writing articles is not to inherently start a conversation or get feedback. I write because I want to share, not because I need validation or care that you think I’m a dumbass. Don’t get me wrong, I still like when people point out errors and omissions or insert points of view, so if you notice anything send me an email or twitter me instead, we can have the discussion there. Better yet, reply with your own article. This keeps the conversation sane, civil and keeps the signal-to-noise ratio down. I’ve noticed this trend from other blogs I visit, and I’m starting to think the future of blogging will look a lot more like magazine articles than threaded conversations. Dialog is good, but it should be controlled so that it remains on topic and relevant.

I mentioned Twitter is a good way to reach me. I’ve actually been using it a lot for the past 6 months; a lot more than this blog. It’s no surprise that microblogging is way easier than writing long articles, so for the past few months my Twitter account has probably been a better source of blog content than my blog (I guess this is probably why people like Tumblr). Let’s be honest, I’m probably not going to start blogging much more than I already do, but I can keep up with Twitter pretty well. Because of this, I’ve incorporated my Twitter feed into my blog and it’s prominently displayed on the front page and in the sidebar on others. I didn’t just add it as a widget to fill whitespace. Instead, expect that feed to be a major source of content for this site. Though you should probably just follow me.

Anyway, enough about me. Hope you like the new design.

Using the raw data file that yardoc generates when it is run, I mocked up a quick tool called yard-graph which allowed me to visualize my project using Graphviz. Note that this is the same functionality that I described any developer could take advantage of in YARD’s very first release. Doing things like this is what YARD is all about. The results aren’t quite perfect due to both the incompleteness of the graph tool and the limitations of Graphviz, like how hierarchical mode is extremely strict about hierarchy and likes to make extremely wide graphs, but they’re highly suitable for my needs; actually, I would call them quite excellent for my needs. Take a look!

As an excercise, I also implemented the ability to show module dependencies (via include) and the public class API. This is what that looks like:

For those interested, this code lives in the diagram branch in Github (yea, I got a free invite and decided to try it out; I forgot / have no idea who invited me– please email me so I can thank you, by the way). After a rake install of the gem, you can run (in your root project directory, or maybe lib/ to ignore tests):

yard-graph --dependencies --empty-mixins --full | dot -T pdf -o diagram.pdf

This requires dot (part of Graphviz) and generates the diagrams with all the above options, module dependencies (--dependencies) and --full for the class/instance methods, constants and attributes info. The --empty-mixins flag shows empty modules as nodes instead of subgraphs (the delimiting boxes), since by default Graphviz ignores empty subgraphs. If you use modules as mixins and not just a way to namespace classes, you’ll want this on or you won’t see your modules.

I also highly suggest using a vector format for output like pdf as in the above example. Bitmap files can get really big. If you want proof, check out the full class diagram of the rails source I generated and imagine that as a bitmap.

Hopefully I can get some more practical uses of YARD up in the next coming weeks for people to see how awesome it is!