connect nothing, f. cried piecing together a puzzling world of ______. by Loren Segal.

So, recently I was looking at refactoring the current internal structure of YARD to improve the developer API and general stability improvements and I figured I would draw out the class diagram as a first step in figuring out what needed to stay and what needed to go. Problem was, having been away from the project for so long, I completely forgot what YARD looked like on the inside. I did not want to dig through the source; I wanted to see the project from afar to keep my design from being influenced by the current layout (in terms of implementation details). I really needed to visualize the project from a high level.

Using the raw data file that yardoc generates when it is run, I mocked up a quick tool called yard-graph which allowed me to visualize my project using Graphviz. Note that this is the same functionality that I described any developer could take advantage of in YARD’s very first release. Doing things like this is what YARD is all about. The results aren’t quite perfect due to both the incompleteness of the graph tool and the limitations of Graphviz, like how hierarchical mode is extremely strict about hierarchy and likes to make extremely wide graphs, but they’re highly suitable for my needs; actually, I would call them quite excellent for my needs. Take a look!

As an excercise, I also implemented the ability to show module dependencies (via include) and the public class API. This is what that looks like:

For those interested, this code lives in the diagram branch in Github (yea, I got a free invite and decided to try it out; I forgot / have no idea who invited me– please email me so I can thank you, by the way). After a rake install of the gem, you can run (in your root project directory, or maybe lib/ to ignore tests):

yard-graph --dependencies --empty-mixins --full | dot -T pdf -o diagram.pdf

This requires dot (part of Graphviz) and generates the diagrams with all the above options, module dependencies (--dependencies) and --full for the class/instance methods, constants and attributes info. The --empty-mixins flag shows empty modules as nodes instead of subgraphs (the delimiting boxes), since by default Graphviz ignores empty subgraphs. If you use modules as mixins and not just a way to namespace classes, you’ll want this on or you won’t see your modules.

I also highly suggest using a vector format for output like pdf as in the above example. Bitmap files can get really big. If you want proof, check out the full class diagram of the rails source I generated and imagine that as a bitmap.

Hopefully I can get some more practical uses of YARD up in the next coming weeks for people to see how awesome it is!

With just a few tweaks, I just ran successfully YARD over actionpack, activesupport, and activerecord with just one error (repeated in a few spots) in the template generation process. I’m starting to realize that the parser isn’t as bad as I thought– there was one noticeable parser issue, which will be fixed in the coming weeks. I am still, however, looking closely at the ruby2ruby and ruby_parser projects– ruby_parser is useless to yard until it starts supporting line numbers, but I’m waiting anxiously to see where Ryan Davis is going to get with it.

You can check out the docs over here but remember they’re not supposed to look nice yet.

And for those interested, the Subversion repository for YARD is at

svn co http://soen.ca/svn/projects/ruby/yard/trunk

It’s been almost a year now since the last release, but today I released a new version of my Ruby documentation tool, YARD 0.2.1. This release gives the tool a surprising amount of robustness for the little code I changed and should make it easier for people to play around with, since it can currently parse the following gems without exploding: merb-core, merb-more, datamapper and obviously yard. You can actually see them here, but don’t judge them for aesthetic merit– the templates were hacked together to show off the introspective power of the documentation, not the visual results. In fact, the only docs worth looking at are the yard docs, because without the Yardoc formatting, a lot of the point is lost. I believe it works with active_record too but I haven’t tried it recently. You can install YARD with a simple:

sudo gem install yard

To run this tool against your Ruby code just type in yardoc in your root directory; a doc directory will be created with an index.html file you can run to see the results, providing it didn’t crash in the process. Note: It still is horribly buggy.

For those who don’t know (and haven’t clicked the above link to YARD’s homepage), YARD is a Ruby documentation tool meant to fix all the problems with RDoc. There are plenty, and I outlined them all on the YARD site, so read up. After you read up, close that page, because it’s horribly out of date.

Where I’ve been

This gem update was a long time coming. A long time thinking, and a long time pondering over what YARD would be and where it would go. I was very disappointed with the state of the code (read: parser) when I gave up, and I also got a little sidetracked over last summer with an internship at Apple, so I kind of abandoned it knowing I would eventually come back. Between having people actually ask me about my progress, and Merb’s docs using a format very similar to the one I created, I think it’s becoming increasingly clear that the time is either now or very soon.

And so, I had a chance to look at the yard source again recently and realized it actually works a lot better than I thought it did. I applied a few patches, and here we are.

But there is still plenty of work to be done.

I’ve actually known how I wanted to get to 1.0 for a while now. This week I bumped into a colleague of mine on the same flight as me to Seattle (well, he was going to Frisco for GDC, me to Redmond for a MS interview, more on that later) and we got to chatting about a bunch of projects we’ve been working ended up discussing how we plan our milestones for projects. I actually laid out my entire roadmap to him right there, so I figure I’ll make it more official and put it out on my blog for those interested in developments.

The Roadmap

I plan on dividing my releases in ‘0.x.0 - 0.x.9′ segments to target one specific design feature all the way up to 1.0. There will be a number of iterations for each design feature, but for the most part each feature will be able to be developed independently/concurrently. That is to say, I plan on seeing through development of 0.3.x potentially before even finishing 0.2.x. As for what each exact release will be, that will depend on how development goes. I can’t get that detailed just yet.

0.2.0 - 0.2.9: Parser fixes / complete parser rewrite

The parser is horrid and needs to be fixed. On the other hand, every parser is horrid, so I’ll live with it the horridness. What I can’t live with is the bugginess. My only wish is that I define the grammar formally using some parser generator, that way I don’t have to look at the ugly part of the code. Ragel is currently my top choice, but Adrian Thurston personally recommended I look at Island parsing. There’s a lot of reading to do, and a lot of time to do it in. This code will need to change as Ruby does (going from 1.8 to 2.0) and there is a lot of testing that has to go through to get it working right. This iteration will probably take a long time. We can potentially use the Ruby lex file to do the lexing, but I don’t know much about hacking with that. Help.

0.3.0 - 0.3.9: Fixing the developer API

YARD holds itself on being an extensible / modular piece of code that could easily support the documentation of various Ruby DSL’s through plugins. The code is not as elegant as it could be, but there is a general framework in place that can be whipped up into something much more usable. Once this is done, YARD can start talking to framework developers to write YARD plugins and use Yardoc formatting. Merb would be a great place to start since they’re already going in that direction.

0.4.0 - 0.4.9: Revamping the templates

Currently YARD was prototyped with a basic Javadoc-style template just to show off some of the inheritance features it has over RDoc doc templates. It’s not meant to be pretty right now. That needs to change eventually, but should be easy once the API is tacked down. YARD can potentially provide a number of templates for different peoples’ tastes, from Javadoc (for JRuby guys) to a more Ruby feel, and everything in between. Output will not be limited to XHTML of course, and can be done in plaintext, man files, etc.

0.5.0 - 0.5.9: Bring it to the community / design review & feature requests

As much as I have my say in software I make, other people have their own. It’s important to hear them out. This will be a good time to get the name out and start getting users using the software and evangelizing the benefits of YARD / structured documentation. I’ll be able to see what people can and can’t live with from a high level, and make some changes before it’s too late. Documentation, tutorials, howtos and basic written word can be fleshed out at this point, to give people better ways to access YARD’s internals.

0.6.0 - 0.6.9: Extend on API with optimization focus on raw data storage, consider database adapters.

One of the other goals of YARD is to provide a raw format for all the information that YARD collects about your source code. This enables developers to perform analysis on documentation through auditing tools, or simply provide a way to serialize the documentation data to another format that may not directly be for human consumption: YAML, XML, etc. This would also mean providing a way for developers to pull the contents from the raw data file to a database, which could enable developers to write interactive documentation applications for their frameworks/code (what caboo.se is currently attempting to do– but manually, or even RailsLodge). This could even mean allowing YARD to store data directly to a database of choice using various SQL adapters.

0.7.0 - 0.7.9: Make YARD work with everyone’s code, up to 2.0

By this point the parser should be far superior to the current 0.2.1 state. Now it should be parsing Ruby’s source tree, Rails, Merb, etc. without issues. At this point we can pre-emptively start adding compatibility for Ruby1.9.x since this will need changes to the parser. Better now than after Ruby1.9.x is finalized.

0.8.0 - 0.8.9: Integrate YARD with Gems/Ruby

Currently when you release a gem you can have it generate RDoc. We’re going to want to change that– well, not change, just allow Yardoc generation. This will require some cross-patching, and a little convincing the gems guys that YARD is awesome. It should be at this point. Also, we want to have YARD generating Yardoc files for Ruby’s source tree when it installs at this point so that you can yri String, for instance, or link to the String class from your applications documentation.

This is where we would need to start converting Ruby’s RDoc format to Yardoc in preparation for the 1.0. This will take a while and extend all the way through the 0.9.x run as well.

0.9.0 - 0.9.9: Run a round of stability patches and general bugfixes, feature-set should be stable.

Have another iteration of community feedback, last minute minor changes to the source for small fixes, minor feature additions/removals, but nothing too grand. YARD should already be awesome and integrated at this point, right? We should also be continuing to convert Ruby docs here.

1.0!

Matz pulls YARD into Ruby as a standard module replacing RDoc. I can dream, right?

Help me dammit!

This project needs developers. Rewriting a parser isn’t easy, and that’s just one of the many steps involved. If anyone finds any shred of motivation to help in any way they can, pleasepleaseplease contact me [l s e g a l (a t) s o e n . c a]. I’m not usually the type to ask for help but it would be extremely healthy for this any project not to hinge on just one person. If there’s any set of features listed above that you think you can tackle and you actually want to see this get done, get in touch. Stuff like template design, coding, writing documentation, testing, or even just blogging about the project / making a screencast (when there’s something to screencast, of course), it all helps.

Man oh man am I happy. I recently discovered Merb, a more lightweight thread-safe alternative to Ruby on Rails. I might devote an entire post to this framework because there’s a lot to be said, but I want to get this off my chest.

I need a mini-framework

I’ve been looking at all these different frameworks to find something that can run more efficiently on my tiny little ghetto server, because I don’t have the resources to run these memory hogging rails apps for my relatively tiny websites. I really need a small framework for the kind of apps I make. You know, pull out a quick blog, wiki, or other, with no DB access at all. What about Camping? No, that’s a little too small. I settled on Merb. Merb is considerably bulkier than Camping, and for a bit I was wondering if even it was too much, until now…

The Merb guys have just released a developer version 0.9.0 which has some neat changes, but what definitely caught my eye was (from the last few lines of the post) the –very-flat option for the application template generator. No need to explain, I’ll just show you how awesome it is:

titanium:merb jinx$ merb-gen blog --very-flat
Fri, 15 Feb 2008 22:57:23 GMT ~ Not Using Sessions
RubiGen::Scripts::Generate
      create  /blog.rb
      create  /README.txt

Now, if we load up the one file created for our project:

Merb::Router.prepare do |r|
  r.match('/').to(:controller => 'blog', :action =>'index')
end
 
class Blog < Merb::Controller
  def index
    "hi"
  end
end
 
Merb::Config.use { |c|
  c[:framework]           = {},
  c[:session_store]       = 'none',
  c[:exception_details]   = true
}

BAM!

Imagine that? That’s all there is to our app. That’s the entire source code– program logic, configuration and support files. It’s fully executable as its own application with the command merb -I blog. Microframework indeed. Granted, it’s pulling all of the ‘merb-core‘ gem, but that’s pretty small. This code is super portable (in the put-it-on-a-usb-key sense, not the platform-to-platform sense) and really quick to develop with. Step aside, Camping.

Granted, I don’t think I’ll use this method of development since it’s the same reason I won’t use Camping, but it’s nice to know I can drop down to the really simple level if I need to just prototype one quick "one-button" app and still have room to grow it out. The fact that Merb can actually do this is what’s most mindblowing. It’s a true testament to the modularity and extensibility of the framework’s design. Getting Rails to run without ActiveRecord is a pain enough, let alone pulling out everything but routing and controllers.