1. A Local Documentation Server

RubyGems has gem server to serve static docs of your gems. The problem here, again, is that to get these static .html files we need to force our users to generate docs when the gem installs, and this is often a painfully slow process (especially with behemoths like Rails). YARD takes a different approach to this, only parsing out the source and saving an intermediate dump file when a gem installs. This is pretty fast (~2 seconds under 1.9 for YARD’s codebase). Not only can a developer use this data to their heart’s content, but it allows YARD to generate HTML/PDF/LaTeX on demand, and that’s exactly what 0.6 is going to do.

The yard server command coming in 0.6 will generate HTML documentation for gems on demand from the parsed source (perhaps even parsing on demand). This means your CPU will never be used to generate static files that you never look at. If you only need to access 10% of the documentation of 1% of your gems, YARD will only generate that 0.1%. Big win.

Not to mention you get all the benefits a YARD template provides, including full class hierarchy, less obtrusive layouts, markdown support, and even full method/class name searching with the potential of having full-text searching in the future, since this is all running in a Webrick/Rack adapter.

All of this stuff is currently being ported from the existing yardoc.org/docs site, so it will basically look like that. The existing code is in the docserver branch of YARD’s github and is being tracked as gh-139.

2. Method Listing by Groups

We’ve seen this in Apple’s docs, doxygen docs and a few others. The ability to organize methods in a large class by functionality rather than alphabet is often far more helpful to your users. YARD 0.6 will be introducing a @group freeform tag to identify methods under a specific organizational header which will be used in the templates. Still working on the exact syntax of this one, trying to make it as easy to use and remove as much duplication as possible. This feature is being tracked as gh-143.

3. Generate Documentation for Single File Scripts

RDoc has a --one-file option that is occasionally used for small Ruby scripts. YARD has always lacked this option. Though it’s kind of simple to roll your own with templates, 0.6 will have built-in templates for generating a single HTML document. Not really sure what it’s going to look like just yet, but if you have ideas or want to help out, the feature is being tracked as gh-115.

4. New yard CLI executable

YARD is growing into a tool that does much more than just parse source. This means that having a single executable is not enough, and spreading out the functionality over multiple executables is just messy. YARD currently uses the yard-graph tool to generate Graphviz dot files. In YARD 0.6, all of these tools, including the new documentation server, will all be merged into a single executable called yard with respective subcommands (doc, graph, server, etc.). The goal here is to open this up for plugins to add commands just like they do in RubyGems without resorting to new executable names. And don’t worry, the yardoc executable will probably stick around for legacy support. This feature is being tracked as gh-140.

5. Improved Template APIs

If you’ve ever written or overridden a template in YARD you know that it can be hard to figure out what section to override with the sections command. YARD has a sections.place(:foo).before(:bar) idiom, but it does not help with subsections. In 0.6, the sections will be redesigned from being Array objects to a more proper tree-like structure in order to handle accessing subsections. Now instead of:

def init
  sections.first.last.place(:foo).before(:bar)
end

You will have the ability to do the following:

def init
  sections.first.subsections.place(:foo).before(:bar)
  # or even:
  sections.place(:foo).before_first(:bar)
end

The last method will look for any "bar" inside sections or subsections within the tree. All in all, this should make it more effective to override a template. This feature is being tracked as gh-58.

6. Documentation for DSL Handlers and Other Use Cases

YARD packs in a lot of functionality, and many users have complained that they had to dig into source to find out how to do some relatively simple things like writing handlers for DSL syntaxes. Although YARD does have documentation on many things, including writing handlers in general, there are few "use-case specific" documents that describe how to do specific things with YARD. 0.6 will introduce improved documentation for some of these use cases, most importantly extending YARD for custom DSL syntaxes. It's possible this may start as a project in the Github Wiki as a set of how-to guides.

So there they are, the major new features coming to 0.6. Let me know if you have any suggestions about other features. And if you have time to donate to the project that would be great, see the above Github issue links and help out. Money would also be appreciated; I wrote YARD because I want to improve docs in the community, not because I want money, but I need to eat. You can donate below:

The release plan for YARD 0.6 is: "soon". It's hard to say when it will be out, but things are certainly rolling in that direction.

You can now use yri with Ruby core classes. This is relatively old news (as of yesterday) since YARD can now parse core classes, but Ruby source is generally not available on all installs, so building the .yardoc database is not always an option.

In YARD 0.5.1 (released today), instead of downloading and parsing the Ruby source manually, you can now download a pre-built gem containing the .yardoc files for a bunch of Ruby versions (1.8.6, 1.8.7, 1.9.1). Simply use:

$ gem install yard-doc-core

You should now be able to use something like yri String#split. By default this will install docs for 1.9.1, but if you need docs for a specific version of Ruby, add -v VERSION to your gem install command:

$ gem install yard-doc-core -v 1.8.7

The stdlib package (yard-doc-stdlib) will be coming soon.

1. Support for documenting native Ruby C code
2. Incremental file parsing and HTML generation
3. Improved yri tool with support for linking gems

Support for Documenting Native Ruby C Code

This one was certainly a long time coming. YARD has always planned to support native Ruby code. This was probably the last "big" feature that RDoc could do and YARD could not. Now, YARD can.

The best part of this feature is that YARD can now parse Ruby’s core codebase and the stdlib, meaning yardoc.org can now host Ruby-core docs. And it now does: http://yardoc.org/docs/ruby-core. The stdlib will be added soon.

Incremental file parsing and HTML generation

One annoying thing about documenting code with YARD for larger projects is the time it takes to generate HTML. This makes the documentation development cycle slow for previewing small documentation changes. YARD 0.5.0 introduces incremental parsing and HTML generation to parse and generate HTML for only the files that were modified since the last running of yardoc, which makes for super-fast previews. To use the incremental parsing feature, simply add -c or --use-cache and YARD will use the available .yardoc cache.

$ yardoc --use-cache

Improved yri tool with support for linking gems

The yri tool that bundled with YARD prior to this release only worked for the .yardoc file in the local directory. In YARD 0.5.0, yri will now perform a lookup on all installed gems on your system. To use this, you’ll first need to build the .yardoc files for your gems, and you can do this with yardoc --build-gems. After that, you can use yri as normal:

$ yri JSON#load

More?

You can see more new toys in the What’s New document on yardoc.org. You can also install YARD:

$ gem install yard

Enjoy!

You can check out the docs over here but remember they’re not supposed to look nice yet.

And for those interested, the Subversion repository for YARD is at

svn co http://soen.ca/svn/projects/ruby/yard/trunk

sudo gem install yard

To run this tool against your Ruby code just type in yardoc in your root directory; a doc directory will be created with an index.html file you can run to see the results, providing it didn’t crash in the process. Note: It still is horribly buggy.

For those who don’t know (and haven’t clicked the above link to YARD’s homepage), YARD is a Ruby documentation tool meant to fix all the problems with RDoc. There are plenty, and I outlined them all on the YARD site, so read up. After you read up, close that page, because it’s horribly out of date.

Where I’ve been

This gem update was a long time coming. A long time thinking, and a long time pondering over what YARD would be and where it would go. I was very disappointed with the state of the code (read: parser) when I gave up, and I also got a little sidetracked over last summer with an internship at Apple, so I kind of abandoned it knowing I would eventually come back. Between having people actually ask me about my progress, and Merb’s docs using a format very similar to the one I created, I think it’s becoming increasingly clear that the time is either now or very soon.

And so, I had a chance to look at the yard source again recently and realized it actually works a lot better than I thought it did. I applied a few patches, and here we are.

But there is still plenty of work to be done.

I’ve actually known how I wanted to get to 1.0 for a while now. This week I bumped into a colleague of mine on the same flight as me to Seattle (well, he was going to Frisco for GDC, me to Redmond for a MS interview, more on that later) and we got to chatting about a bunch of projects we’ve been working ended up discussing how we plan our milestones for projects. I actually laid out my entire roadmap to him right there, so I figure I’ll make it more official and put it out on my blog for those interested in developments.

The Roadmap

I plan on dividing my releases in ’0.x.0 – 0.x.9′ segments to target one specific design feature all the way up to 1.0. There will be a number of iterations for each design feature, but for the most part each feature will be able to be developed independently/concurrently. That is to say, I plan on seeing through development of 0.3.x potentially before even finishing 0.2.x. As for what each exact release will be, that will depend on how development goes. I can’t get that detailed just yet.

0.2.0 – 0.2.9: Parser fixes / complete parser rewrite

The parser is horrid and needs to be fixed. On the other hand, every parser is horrid, so I’ll live with it the horridness. What I can’t live with is the bugginess. My only wish is that I define the grammar formally using some parser generator, that way I don’t have to look at the ugly part of the code. Ragel is currently my top choice, but Adrian Thurston personally recommended I look at Island parsing. There’s a lot of reading to do, and a lot of time to do it in. This code will need to change as Ruby does (going from 1.8 to 2.0) and there is a lot of testing that has to go through to get it working right. This iteration will probably take a long time. We can potentially use the Ruby lex file to do the lexing, but I don’t know much about hacking with that. Help.

0.3.0 – 0.3.9: Fixing the developer API

YARD holds itself on being an extensible / modular piece of code that could easily support the documentation of various Ruby DSL’s through plugins. The code is not as elegant as it could be, but there is a general framework in place that can be whipped up into something much more usable. Once this is done, YARD can start talking to framework developers to write YARD plugins and use Yardoc formatting. Merb would be a great place to start since they’re already going in that direction.

0.4.0 – 0.4.9: Revamping the templates

Currently YARD was prototyped with a basic Javadoc-style template just to show off some of the inheritance features it has over RDoc doc templates. It’s not meant to be pretty right now. That needs to change eventually, but should be easy once the API is tacked down. YARD can potentially provide a number of templates for different peoples’ tastes, from Javadoc (for JRuby guys) to a more Ruby feel, and everything in between. Output will not be limited to XHTML of course, and can be done in plaintext, man files, etc.

0.5.0 – 0.5.9: Bring it to the community / design review & feature requests

As much as I have my say in software I make, other people have their own. It’s important to hear them out. This will be a good time to get the name out and start getting users using the software and evangelizing the benefits of YARD / structured documentation. I’ll be able to see what people can and can’t live with from a high level, and make some changes before it’s too late. Documentation, tutorials, howtos and basic written word can be fleshed out at this point, to give people better ways to access YARD’s internals.

0.6.0 – 0.6.9: Extend on API with optimization focus on raw data storage, consider database adapters.

One of the other goals of YARD is to provide a raw format for all the information that YARD collects about your source code. This enables developers to perform analysis on documentation through auditing tools, or simply provide a way to serialize the documentation data to another format that may not directly be for human consumption: YAML, XML, etc. This would also mean providing a way for developers to pull the contents from the raw data file to a database, which could enable developers to write interactive documentation applications for their frameworks/code (what caboo.se is currently attempting to do– but manually, or even RailsLodge). This could even mean allowing YARD to store data directly to a database of choice using various SQL adapters.

0.7.0 – 0.7.9: Make YARD work with everyone’s code, up to 2.0

By this point the parser should be far superior to the current 0.2.1 state. Now it should be parsing Ruby’s source tree, Rails, Merb, etc. without issues. At this point we can pre-emptively start adding compatibility for Ruby1.9.x since this will need changes to the parser. Better now than after Ruby1.9.x is finalized.

0.8.0 – 0.8.9: Integrate YARD with Gems/Ruby

Currently when you release a gem you can have it generate RDoc. We’re going to want to change that– well, not change, just allow Yardoc generation. This will require some cross-patching, and a little convincing the gems guys that YARD is awesome. It should be at this point. Also, we want to have YARD generating Yardoc files for Ruby’s source tree when it installs at this point so that you can yri String, for instance, or link to the String class from your applications documentation.

This is where we would need to start converting Ruby’s RDoc format to Yardoc in preparation for the 1.0. This will take a while and extend all the way through the 0.9.x run as well.

0.9.0 – 0.9.9: Run a round of stability patches and general bugfixes, feature-set should be stable.

Have another iteration of community feedback, last minute minor changes to the source for small fixes, minor feature additions/removals, but nothing too grand. YARD should already be awesome and integrated at this point, right? We should also be continuing to convert Ruby docs here.

1.0!

Matz pulls YARD into Ruby as a standard module replacing RDoc. I can dream, right?

Help me dammit!

This project needs developers. Rewriting a parser isn’t easy, and that’s just one of the many steps involved. If anyone finds any shred of motivation to help in any way they can, pleasepleaseplease contact me [l s e g a l (a t) s o e n . c a]. I’m not usually the type to ask for help but it would be extremely healthy for this any project not to hinge on just one person. If there’s any set of features listed above that you think you can tackle and you actually want to see this get done, get in touch. Stuff like template design, coding, writing documentation, testing, or even just blogging about the project / making a screencast (when there’s something to screencast, of course), it all helps.