1. A Local Documentation Server

RubyGems has gem server to serve static docs of your gems. The problem here, again, is that to get these static .html files we need to force our users to generate docs when the gem installs, and this is often a painfully slow process (especially with behemoths like Rails). YARD takes a different approach to this, only parsing out the source and saving an intermediate dump file when a gem installs. This is pretty fast (~2 seconds under 1.9 for YARD’s codebase). Not only can a developer use this data to their heart’s content, but it allows YARD to generate HTML/PDF/LaTeX on demand, and that’s exactly what 0.6 is going to do.

The yard server command coming in 0.6 will generate HTML documentation for gems on demand from the parsed source (perhaps even parsing on demand). This means your CPU will never be used to generate static files that you never look at. If you only need to access 10% of the documentation of 1% of your gems, YARD will only generate that 0.1%. Big win.

Not to mention you get all the benefits a YARD template provides, including full class hierarchy, less obtrusive layouts, markdown support, and even full method/class name searching with the potential of having full-text searching in the future, since this is all running in a Webrick/Rack adapter.

All of this stuff is currently being ported from the existing yardoc.org/docs site, so it will basically look like that. The existing code is in the docserver branch of YARD’s github and is being tracked as gh-139.

2. Method Listing by Groups

We’ve seen this in Apple’s docs, doxygen docs and a few others. The ability to organize methods in a large class by functionality rather than alphabet is often far more helpful to your users. YARD 0.6 will be introducing a @group freeform tag to identify methods under a specific organizational header which will be used in the templates. Still working on the exact syntax of this one, trying to make it as easy to use and remove as much duplication as possible. This feature is being tracked as gh-143.

3. Generate Documentation for Single File Scripts

RDoc has a --one-file option that is occasionally used for small Ruby scripts. YARD has always lacked this option. Though it’s kind of simple to roll your own with templates, 0.6 will have built-in templates for generating a single HTML document. Not really sure what it’s going to look like just yet, but if you have ideas or want to help out, the feature is being tracked as gh-115.

4. New yard CLI executable

YARD is growing into a tool that does much more than just parse source. This means that having a single executable is not enough, and spreading out the functionality over multiple executables is just messy. YARD currently uses the yard-graph tool to generate Graphviz dot files. In YARD 0.6, all of these tools, including the new documentation server, will all be merged into a single executable called yard with respective subcommands (doc, graph, server, etc.). The goal here is to open this up for plugins to add commands just like they do in RubyGems without resorting to new executable names. And don’t worry, the yardoc executable will probably stick around for legacy support. This feature is being tracked as gh-140.

5. Improved Template APIs

If you’ve ever written or overridden a template in YARD you know that it can be hard to figure out what section to override with the sections command. YARD has a sections.place(:foo).before(:bar) idiom, but it does not help with subsections. In 0.6, the sections will be redesigned from being Array objects to a more proper tree-like structure in order to handle accessing subsections. Now instead of:

def init
  sections.first.last.place(:foo).before(:bar)
end

You will have the ability to do the following:

def init
  sections.first.subsections.place(:foo).before(:bar)
  # or even:
  sections.place(:foo).before_first(:bar)
end

The last method will look for any "bar" inside sections or subsections within the tree. All in all, this should make it more effective to override a template. This feature is being tracked as gh-58.

6. Documentation for DSL Handlers and Other Use Cases

YARD packs in a lot of functionality, and many users have complained that they had to dig into source to find out how to do some relatively simple things like writing handlers for DSL syntaxes. Although YARD does have documentation on many things, including writing handlers in general, there are few "use-case specific" documents that describe how to do specific things with YARD. 0.6 will introduce improved documentation for some of these use cases, most importantly extending YARD for custom DSL syntaxes. It's possible this may start as a project in the Github Wiki as a set of how-to guides.

So there they are, the major new features coming to 0.6. Let me know if you have any suggestions about other features. And if you have time to donate to the project that would be great, see the above Github issue links and help out. Money would also be appreciated; I wrote YARD because I want to improve docs in the community, not because I want money, but I need to eat. You can donate below:

The release plan for YARD 0.6 is: "soon". It's hard to say when it will be out, but things are certainly rolling in that direction.

In the software world, "engines" don’t have nearly as strict a definition. The term comes up a lot, but it is never strictly bound by a specific functionality. In software, we tend to consider an engine as anything that "powers" something else. The problem with that definition is that, in the right context, anything "powers" something else. It might allow for more abstraction (something we developers love), but it also makes an engine no more meaningful more than calling it a "component", or a "thing". At some point, it’s important to set a strict definition of where the responsibilities of one component end and another begin.

Blogs are Engines(?)

I often hear WordPress described as a "blogging engine". In the sense that it powers blogs by allowing people to post bits of text and media, sure. But how do we define the boundaries of WordPress as an engine? WordPress is also a full fledged blogging tool, something the end user touches directly. When we say "WordPress the engine", do we mean just the bits that handle blog posting, or are we talking about the admin interface too? In my opinion, calling WordPress an engine would be like calling a whole car the "engine" and saying the cupholders are plugins.

So does WordPress just consist of an engine inside of it, or is it all a big engine?

Games are Engines(?)

3D engines have the same definition problem. Back in the day, Id was the king of licensing 3D "engines". They were very simplistic, though still quite functional relative to our real world "engine" counterpart. They basically handled the OpenGL abstraction, but handled a bit of game logic too. Nowadays we have engines like the Far Cry engine, which not only do 3D graphics, the entirety of game logic, but they now come with map editors and modelers built in. We’re hardly talking about an engine anymore, this is practically a full development environment. Where is the line?

The Driver

We do have examples of words that have very explicit meanings. For instance, when we hear the word "drivers", we tend to understand that we’re talking about a kernel extension that interfaces a hardware component. The boundaries of a device driver is usually quite specific, and we know exactly where the component fits into our system. There are others, but we could use more of these guys.

What’s an Engine to You?

I think it’s interesting how much leeway we give our words (for the most part) in the software industry. We always like to shy away from committing to any assumptions whatsoever. This sounds nice, but perhaps our inability to strictly define component boundaries is why we have so much trouble defining what software is, and perhaps more so, why we have so much trouble managing software projects. It would be extremely helpful to project scope if we could say "I’m building X which means you will get functionality Y" without having to redefine "Y" each time.

invalid multibyte char (US-ASCII)

- OR -

CompatibilityError: incompatible encoding regexp match (Windows-31J regexp with UTF-8 string)

The errors themselves are relatively self-explanatory. Ruby 1.9 is far more Unicode aware than 1.8, and this error happens when have some Unicode (usually UTF-8) in one of your files. What you have to do to fix these, however, is not always as straightforward.

Well, after some time pulling my hair out, I’ve figure out that the solutions to these issues are actually quite simple.

Invalid multibyte char (encoding here)

If you get this issue, add the following to the top of each exploding file (below the shebang if there is one):

# encoding: utf-8

Note: You can also use "coding:" or even the Emacs style -*- encoding: utf-8 -*- but I like the simple term, ‘encoding’. Also note that you might need to replace ‘utf-8′ wth your specific encoding if it’s something else.

This should resolve the issue.

Basically, Ruby by default assumes that every file is encoded as US-ASCII, and so when it reads UTF-8 (or any Unicode) it freaks out because it is beyond the 7-bit encoding. You have to tell it that the file is encoded as utf-8 by listing it as we did at the top of the file. Yes, it’s a little anal, but I’m sure we’ll get used to it.

Incompatible encoding regexp match

This issue is a little bit hairier. I had this problem with the BlueCloth 1.0.0 gem (on line 972 of bluecloth.rb). It turns out that there are a few switches that turn on specific encodings, and for some reason BlueCloth turns on the //s switch which enables the SJIS encoding (maybe John Gruber wanted //m for multiline?). Ruby 1.8 didn’t mind having this on, but 1.9 freaks out. Moral of the story, when you see this error, check your Regexp switches.

I picked up this nifty little trick on a forum while searching for company in my misery over the "Windows 7 QuickLaunch". Apparently, "Toolbars" still exist in full force under Windows 7, and if you’ve ever messed with them before, you’ll know QuickLaunch was always just one of those folders. The simple way to resurrect the QuickLaunch bar is to:

1. Right click on the taskbar and go to Toolbars
2. Click "New Toolbar…"
3. Navigate to "%USERPROFILE%\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch"
4. Select that directory.

You technically don’t even need to use that specific directory— any empty directory will do (you can make one in a safe place).

It will show up on the right by default, but you can right click on the taskbar again and unlock the taskbar to move it back to the left.

Voila, ze Quick Launch, she lives!

This means I can completely avoid using the horribly unintuitive pinned icon behaviour that is supposedly "superior". Better yet, by using the old QuickLaunch, I even get back all that wasted horizontal space that comes with the pinned icons!

So Mail.app doesn’t know jack about subscriptions unless you know how to rub it the right way. After recently switching to IMAP I also discovered that it doesn’t like to use standardized IMAP folder names. You know, “Sent”, “Trash”, et al. If you’ve been annoyed by the inconsistency of having Mail.app trash your mail into directories you’ll never see again, or not recognize your Sent folder, you should know there’s a really simple solution.

Enter Property List Editor

Close Mail.app before editing the preferences file so it doesn’t overwrite the settings. You need to restart the app anyway, so shut it down now. I also suggest backing up the file if you’re not used to editing .plist files

1. Open up ~/Library/Preferences/com.apple.mail.plist with the Property List Editor
2. Navigate to the key named MailAccounts and open it up
3. In it are all your accounts indexed as 0, 1, 2, …
4. Find your IMAP accounts (check that the key AccountType is IMAPAccount)
5. Navigate to the keys SentMessagesMailboxName and TrashMailboxName and change them to Sent and Trash respectively

Voila, you’re all set. Save the plist file and start up Mail.app, it should now use the proper special folders (the real Trash folder, etc.) and the IMAP folders should disappear from the sidebar. Now when you delete something, it will actually go to your IMAP trash!

1. IE8 completely overwrites IE7 just like IE7 did with IE6. At least this time you can roll back, but you’d figure by now they’d have learned the simple software development principle of allowing applications to co-exist on a system. This is especially necessary when developers need to switch back and forth to compare and contrast behaviour.
2. IE8 replaces the very simple concept described in (1) with an "IE7 emulation" mode. Unfortunately, "IE7 emulation" mode only emulates browser rendering, not the horrible UI issues. In addition to that, switching between IE7 emulation mode requires the browser (and all other windows!) to be completely restarted. Now all developers need to do to see changes in behaviour is:
   1. Navigate to their website in IE7 emulation mode
   2. Switch IE7 emulation mode off
   3. Close the browser
   4. Open the browser
   5. Navigate to the website again
   6. Check & attempt to solve differences
   7. Switch IE7 emulation mode back on
   8. Close the browser
   9. Open the browser
   10. Navigate to the website again
   11. Make sure your solution didn’t break IE7

   And don’t even try to view them side by side– no, that would be too easy. What a great idea, Microsoft.

3. IE8 introduces the concept of "Activities" and "Slices". It lets you highlight text and perform a bunch of common tasks like searching (on Live) and mapping (on Live) by right clicking. Now I know this is Microsoft, but where the hell is Google Maps? You know, that better mapping website that pre-dates yours? These features are nice in theory, but everything here is so monopolistically targetted to their services only– meaning normal people get nothing out of it. Hopefully they add normal sites by the final release. Usually I would wait until the final to whine about third party functionality, but for some reason I don’t see this ever working with anything I actually use. And no, a Facebook search activity does not count.
4. IE8 "standards" completely break even the most basic of pages. I’m talking oldschool tabular layouts, and the part that broke was the layout. How do you break that stuff?
5. The horizontal scrollbar bar shows up on almost every single page. How the hell did a dead obvious bug like that get all the way to the beta?
6. Why do domain names show up in bold in the address bar and the rest of the equally important URL is grayed out? Since when is the only important part of a URL the domain name? I’m not even sure what the UI purpose of that is…
7. How do I disable Activities & Slices? I don’t want to see useless information in my popup menu and an even more useless popup button whenever I highlight text. I’ll assume it’s buried somewhere in the options… I see they’ve done no work to improve the horrible preferences dialog.

I’m not even going to talk about the standards (besides #4). The web development community has sufficiently hated on IE’s implementation of standards and there’s nothing left to say. I don’t necessarily share those opinions, but I also don’t care. When a non-developer visits your website they don’t care that you correctly implemented the CSS2.1 spec or that your XHTML validates and you have a graphic to prove it– they just want it to look normal. The second normal sites stop looking normal (#4) is when you’ve messed up and need to fix it.

All in all I’m pretty disappointed. Remember, I’m an IE user, and just listen to how pissed I sound. Needless to say I used the benefit of the rollback to IE7.

This beta makes me want to start using Safari or something. Don’t people make browsers to download and render webpages anymore? I don’t need Activities, I don’t need Slices, I don’t need Download Managers or Favourite Tagging Blogosphere Viewing Youtube Integration. I don’t even need tabs! Just render my damn HTML.

Using the raw data file that yardoc generates when it is run, I mocked up a quick tool called yard-graph which allowed me to visualize my project using Graphviz. Note that this is the same functionality that I described any developer could take advantage of in YARD’s very first release. Doing things like this is what YARD is all about. The results aren’t quite perfect due to both the incompleteness of the graph tool and the limitations of Graphviz, like how hierarchical mode is extremely strict about hierarchy and likes to make extremely wide graphs, but they’re highly suitable for my needs; actually, I would call them quite excellent for my needs. Take a look!

As an excercise, I also implemented the ability to show module dependencies (via include) and the public class API. This is what that looks like:

For those interested, this code lives in the diagram branch in Github (yea, I got a free invite and decided to try it out; I forgot / have no idea who invited me– please email me so I can thank you, by the way). After a rake install of the gem, you can run (in your root project directory, or maybe lib/ to ignore tests):

yard-graph --dependencies --empty-mixins --full | dot -T pdf -o diagram.pdf

This requires dot (part of Graphviz) and generates the diagrams with all the above options, module dependencies (--dependencies) and --full for the class/instance methods, constants and attributes info. The --empty-mixins flag shows empty modules as nodes instead of subgraphs (the delimiting boxes), since by default Graphviz ignores empty subgraphs. If you use modules as mixins and not just a way to namespace classes, you’ll want this on or you won’t see your modules.

I also highly suggest using a vector format for output like pdf as in the above example. Bitmap files can get really big. If you want proof, check out the full class diagram of the rails source I generated and imagine that as a bitmap.

Hopefully I can get some more practical uses of YARD up in the next coming weeks for people to see how awesome it is!