connect nothing, f. cried piecing together a puzzling world of ______. by Loren Segal.

This is a little journal/tip style entry about how to get Mail.app to use normal IMAP folder names so as to not confuse your other clients on other machines (Thunderbird, etc.).

So Mail.app doesn’t know jack about subscriptions unless you know how to rub it the right way. After recently switching to IMAP I also discovered that it doesn’t like to use standardized IMAP folder names. You know, “Sent”, “Trash”, et al. If you’ve been annoyed by the inconsistency of having Mail.app trash your mail into directories you’ll never see again, or not recognize your Sent folder, you should know there’s a really simple solution.

Enter Property List Editor

Close Mail.app before editing the preferences file so it doesn’t overwrite the settings. You need to restart the app anyway, so shut it down now. I also suggest backing up the file if you’re not used to editing .plist files

1. Open up ~/Library/Preferences/com.apple.mail.plist with the Property List Editor
2. Navigate to the key named MailAccounts and open it up
3. In it are all your accounts indexed as 0, 1, 2, …
4. Find your IMAP accounts (check that the key AccountType is IMAPAccount)
5. Navigate to the keys SentMessagesMailboxName and TrashMailboxName and change them to Sent and Trash respectively

Voila, you’re all set. Save the plist file and start up Mail.app, it should now use the proper special folders (the real Trash folder, etc.) and the IMAP folders should disappear from the sidebar. Now when you delete something, it will actually go to your IMAP trash!

I’m usually a fan of IE (What, did someone just admit to that??), in fact, my primary browser on my Windows machine is not Firefox, but IE7, which I’ve used and has perfectly suited my needs since the first beta. Actually, I liked the first beta a lot more than the final release, so when I found out that Microsoft released beta 1 for IE8 I immediately went to download it. In retrospect, I should have did a little research first (there’s actually little information on IE8 right now), maybe I would have found out that:

1. IE8 completely overwrites IE7 just like IE7 did with IE6. At least this time you can roll back, but you’d figure by now they’d have learned the simple software development principle of allowing applications to co-exist on a system. This is especially necessary when developers need to switch back and forth to compare and contrast behaviour.
2. IE8 replaces the very simple concept described in (1) with an "IE7 emulation" mode. Unfortunately, "IE7 emulation" mode only emulates browser rendering, not the horrible UI issues. In addition to that, switching between IE7 emulation mode requires the browser (and all other windows!) to be completely restarted. Now all developers need to do to see changes in behaviour is:
   1. Navigate to their website in IE7 emulation mode
   2. Switch IE7 emulation mode off
   3. Close the browser
   4. Open the browser
   5. Navigate to the website again
   6. Check & attempt to solve differences
   7. Switch IE7 emulation mode back on
   8. Close the browser
   9. Open the browser
   10. Navigate to the website again
   11. Make sure your solution didn’t break IE7

   And don’t even try to view them side by side– no, that would be too easy. What a great idea, Microsoft.

3. IE8 introduces the concept of "Activities" and "Slices". It lets you highlight text and perform a bunch of common tasks like searching (on Live) and mapping (on Live) by right clicking. Now I know this is Microsoft, but where the hell is Google Maps? You know, that better mapping website that pre-dates yours? These features are nice in theory, but everything here is so monopolistically targetted to their services only– meaning normal people get nothing out of it. Hopefully they add normal sites by the final release. Usually I would wait until the final to whine about third party functionality, but for some reason I don’t see this ever working with anything I actually use. And no, a Facebook search activity does not count.
4. IE8 "standards" completely break even the most basic of pages. I’m talking oldschool tabular layouts, and the part that broke was the layout. How do you break that stuff?
5. The horizontal scrollbar bar shows up on almost every single page. How the hell did a dead obvious bug like that get all the way to the beta?
6. Why do domain names show up in bold in the address bar and the rest of the equally important URL is grayed out? Since when is the only important part of a URL the domain name? I’m not even sure what the UI purpose of that is…
7. How do I disable Activities & Slices? I don’t want to see useless information in my popup menu and an even more useless popup button whenever I highlight text. I’ll assume it’s buried somewhere in the options… I see they’ve done no work to improve the horrible preferences dialog.

I’m not even going to talk about the standards (besides #4). The web development community has sufficiently hated on IE’s implementation of standards and there’s nothing left to say. I don’t necessarily share those opinions, but I also don’t care. When a non-developer visits your website they don’t care that you correctly implemented the CSS2.1 spec or that your XHTML validates and you have a graphic to prove it– they just want it to look normal. The second normal sites stop looking normal (#4) is when you’ve messed up and need to fix it.

All in all I’m pretty disappointed. Remember, I’m an IE user, and just listen to how pissed I sound. Needless to say I used the benefit of the rollback to IE7.

This beta makes me want to start using Safari or something. Don’t people make browsers to download and render webpages anymore? I don’t need Activities, I don’t need Slices, I don’t need Download Managers or Favourite Tagging Blogosphere Viewing Youtube Integration. I don’t even need tabs! Just render my damn HTML.

So, recently I was looking at refactoring the current internal structure of YARD to improve the developer API and general stability improvements and I figured I would draw out the class diagram as a first step in figuring out what needed to stay and what needed to go. Problem was, having been away from the project for so long, I completely forgot what YARD looked like on the inside. I did not want to dig through the source; I wanted to see the project from afar to keep my design from being influenced by the current layout (in terms of implementation details). I really needed to visualize the project from a high level.

Using the raw data file that yardoc generates when it is run, I mocked up a quick tool called yard-graph which allowed me to visualize my project using Graphviz. Note that this is the same functionality that I described any developer could take advantage of in YARD’s very first release. Doing things like this is what YARD is all about. The results aren’t quite perfect due to both the incompleteness of the graph tool and the limitations of Graphviz, like how hierarchical mode is extremely strict about hierarchy and likes to make extremely wide graphs, but they’re highly suitable for my needs; actually, I would call them quite excellent for my needs. Take a look!

As an excercise, I also implemented the ability to show module dependencies (via include) and the public class API. This is what that looks like:

For those interested, this code lives in the diagram branch in Github (yea, I got a free invite and decided to try it out; I forgot / have no idea who invited me– please email me so I can thank you, by the way). After a rake install of the gem, you can run (in your root project directory, or maybe lib/ to ignore tests):

yard-graph --dependencies --empty-mixins --full | dot -T pdf -o diagram.pdf

This requires dot (part of Graphviz) and generates the diagrams with all the above options, module dependencies (--dependencies) and --full for the class/instance methods, constants and attributes info. The --empty-mixins flag shows empty modules as nodes instead of subgraphs (the delimiting boxes), since by default Graphviz ignores empty subgraphs. If you use modules as mixins and not just a way to namespace classes, you’ll want this on or you won’t see your modules.

I also highly suggest using a vector format for output like pdf as in the above example. Bitmap files can get really big. If you want proof, check out the full class diagram of the rails source I generated and imagine that as a bitmap.

Hopefully I can get some more practical uses of YARD up in the next coming weeks for people to see how awesome it is!

With just a few tweaks, I just ran successfully YARD over actionpack, activesupport, and activerecord with just one error (repeated in a few spots) in the template generation process. I’m starting to realize that the parser isn’t as bad as I thought– there was one noticeable parser issue, which will be fixed in the coming weeks. I am still, however, looking closely at the ruby2ruby and ruby_parser projects– ruby_parser is useless to yard until it starts supporting line numbers, but I’m waiting anxiously to see where Ryan Davis is going to get with it.

You can check out the docs over here but remember they’re not supposed to look nice yet.

And for those interested, the Subversion repository for YARD is at

svn co http://soen.ca/svn/projects/ruby/yard/trunk

It’s been almost a year now since the last release, but today I released a new version of my Ruby documentation tool, YARD 0.2.1. This release gives the tool a surprising amount of robustness for the little code I changed and should make it easier for people to play around with, since it can currently parse the following gems without exploding: merb-core, merb-more, datamapper and obviously yard. You can actually see them here, but don’t judge them for aesthetic merit– the templates were hacked together to show off the introspective power of the documentation, not the visual results. In fact, the only docs worth looking at are the yard docs, because without the Yardoc formatting, a lot of the point is lost. I believe it works with active_record too but I haven’t tried it recently. You can install YARD with a simple:

sudo gem install yard

To run this tool against your Ruby code just type in yardoc in your root directory; a doc directory will be created with an index.html file you can run to see the results, providing it didn’t crash in the process. Note: It still is horribly buggy.

For those who don’t know (and haven’t clicked the above link to YARD’s homepage), YARD is a Ruby documentation tool meant to fix all the problems with RDoc. There are plenty, and I outlined them all on the YARD site, so read up. After you read up, close that page, because it’s horribly out of date.

Where I’ve been

This gem update was a long time coming. A long time thinking, and a long time pondering over what YARD would be and where it would go. I was very disappointed with the state of the code (read: parser) when I gave up, and I also got a little sidetracked over last summer with an internship at Apple, so I kind of abandoned it knowing I would eventually come back. Between having people actually ask me about my progress, and Merb’s docs using a format very similar to the one I created, I think it’s becoming increasingly clear that the time is either now or very soon.

And so, I had a chance to look at the yard source again recently and realized it actually works a lot better than I thought it did. I applied a few patches, and here we are.

But there is still plenty of work to be done.

I’ve actually known how I wanted to get to 1.0 for a while now. This week I bumped into a colleague of mine on the same flight as me to Seattle (well, he was going to Frisco for GDC, me to Redmond for a MS interview, more on that later) and we got to chatting about a bunch of projects we’ve been working ended up discussing how we plan our milestones for projects. I actually laid out my entire roadmap to him right there, so I figure I’ll make it more official and put it out on my blog for those interested in developments.

The Roadmap

I plan on dividing my releases in ‘0.x.0 - 0.x.9′ segments to target one specific design feature all the way up to 1.0. There will be a number of iterations for each design feature, but for the most part each feature will be able to be developed independently/concurrently. That is to say, I plan on seeing through development of 0.3.x potentially before even finishing 0.2.x. As for what each exact release will be, that will depend on how development goes. I can’t get that detailed just yet.

0.2.0 - 0.2.9: Parser fixes / complete parser rewrite

The parser is horrid and needs to be fixed. On the other hand, every parser is horrid, so I’ll live with it the horridness. What I can’t live with is the bugginess. My only wish is that I define the grammar formally using some parser generator, that way I don’t have to look at the ugly part of the code. Ragel is currently my top choice, but Adrian Thurston personally recommended I look at Island parsing. There’s a lot of reading to do, and a lot of time to do it in. This code will need to change as Ruby does (going from 1.8 to 2.0) and there is a lot of testing that has to go through to get it working right. This iteration will probably take a long time. We can potentially use the Ruby lex file to do the lexing, but I don’t know much about hacking with that. Help.

0.3.0 - 0.3.9: Fixing the developer API

YARD holds itself on being an extensible / modular piece of code that could easily support the documentation of various Ruby DSL’s through plugins. The code is not as elegant as it could be, but there is a general framework in place that can be whipped up into something much more usable. Once this is done, YARD can start talking to framework developers to write YARD plugins and use Yardoc formatting. Merb would be a great place to start since they’re already going in that direction.

0.4.0 - 0.4.9: Revamping the templates

Currently YARD was prototyped with a basic Javadoc-style template just to show off some of the inheritance features it has over RDoc doc templates. It’s not meant to be pretty right now. That needs to change eventually, but should be easy once the API is tacked down. YARD can potentially provide a number of templates for different peoples’ tastes, from Javadoc (for JRuby guys) to a more Ruby feel, and everything in between. Output will not be limited to XHTML of course, and can be done in plaintext, man files, etc.

0.5.0 - 0.5.9: Bring it to the community / design review & feature requests

As much as I have my say in software I make, other people have their own. It’s important to hear them out. This will be a good time to get the name out and start getting users using the software and evangelizing the benefits of YARD / structured documentation. I’ll be able to see what people can and can’t live with from a high level, and make some changes before it’s too late. Documentation, tutorials, howtos and basic written word can be fleshed out at this point, to give people better ways to access YARD’s internals.

0.6.0 - 0.6.9: Extend on API with optimization focus on raw data storage, consider database adapters.

One of the other goals of YARD is to provide a raw format for all the information that YARD collects about your source code. This enables developers to perform analysis on documentation through auditing tools, or simply provide a way to serialize the documentation data to another format that may not directly be for human consumption: YAML, XML, etc. This would also mean providing a way for developers to pull the contents from the raw data file to a database, which could enable developers to write interactive documentation applications for their frameworks/code (what caboo.se is currently attempting to do– but manually, or even RailsLodge). This could even mean allowing YARD to store data directly to a database of choice using various SQL adapters.

0.7.0 - 0.7.9: Make YARD work with everyone’s code, up to 2.0

By this point the parser should be far superior to the current 0.2.1 state. Now it should be parsing Ruby’s source tree, Rails, Merb, etc. without issues. At this point we can pre-emptively start adding compatibility for Ruby1.9.x since this will need changes to the parser. Better now than after Ruby1.9.x is finalized.

0.8.0 - 0.8.9: Integrate YARD with Gems/Ruby

Currently when you release a gem you can have it generate RDoc. We’re going to want to change that– well, not change, just allow Yardoc generation. This will require some cross-patching, and a little convincing the gems guys that YARD is awesome. It should be at this point. Also, we want to have YARD generating Yardoc files for Ruby’s source tree when it installs at this point so that you can yri String, for instance, or link to the String class from your applications documentation.

This is where we would need to start converting Ruby’s RDoc format to Yardoc in preparation for the 1.0. This will take a while and extend all the way through the 0.9.x run as well.

0.9.0 - 0.9.9: Run a round of stability patches and general bugfixes, feature-set should be stable.

Have another iteration of community feedback, last minute minor changes to the source for small fixes, minor feature additions/removals, but nothing too grand. YARD should already be awesome and integrated at this point, right? We should also be continuing to convert Ruby docs here.

1.0!

Matz pulls YARD into Ruby as a standard module replacing RDoc. I can dream, right?

Help me dammit!

This project needs developers. Rewriting a parser isn’t easy, and that’s just one of the many steps involved. If anyone finds any shred of motivation to help in any way they can, pleasepleaseplease contact me [l s e g a l (a t) s o e n . c a]. I’m not usually the type to ask for help but it would be extremely healthy for this any project not to hinge on just one person. If there’s any set of features listed above that you think you can tackle and you actually want to see this get done, get in touch. Stuff like template design, coding, writing documentation, testing, or even just blogging about the project / making a screencast (when there’s something to screencast, of course), it all helps.

Or, why your project doesn’t need Git

Geeks love new things. If geeks were frogs, gadgets would be lillypads. Here’s a diagram:

Frogs Geeks love to jump from lillypad to lillypad as soon as they see a new one. This is why I’m not all that surprised that people (at least in the Ruby community) are quickly ditching the Subversion ship that’s gotten so much attention over the last few years. While surprised I am not, I am slightly disappointed in people for leapfrogging to a new technology so quickly before it’s proven to really be as cool as it sounds.

Important note: I’ve only been using Git for about a week now, so many of the technical details I provide about Git may will be wrong. Please correct me. However, none of what I’m about to say has anything to do with the technical aspects of git. Having seen the way git is used in certain projects makes me wonder what people think they’re really getting from distributed source control management. Also note that I’m not discussing Linux development here– I know nothing about it, nor do I care. The fact that git works great for Linus doesn’t mean it works for every other open source project.

Git is a technical masterpiece; but not all technical masterpieces are useful to you.

Git has some really nice features. Branching is effortless and hidden from the user. Merging is nice, conceptually, though resolving conflicts seems like a pain compared to Subversion. The speed/size optimizations are a must, and I sure hope the svn guys get their act together.

Note however that none of what I just credited Git for has anything to do with its decentralizededness (dictionary, please). All of this could be implemented in Subversion without changing your workflow.

Programming is not quite like editing Wikipedia

The supposed advantage to decentralized SCM’s is that anyone can contribute code just by running a git clone and then making and sharing changes. Everybody has "their own branch" that they could develop on. But the truth is that anyone who tells you this is simply giving you a false sense of reality.

In real life, you don’t just download the repository, make a change and get a guarantee that your work will be merged back into the main branch for everyone to use (you made the change using SCM so you could share it, after all). In real life, projects are well guarded from the outside world and have a few gatekeepers known as maintainers. These people are usually the project owners/creators. In real life, you deal with having to convince these maintainers that your code deserves to be in the main branch.

The only advantage to git is that once you deal with all the politics, you can theoretically have the maintainers merge your Git branch with theirs really easily– though in reality most projects still would rather take patches using ticketing systems. In reality, workflow trumps technology.

Your workflow is the ultimate bottleneck.

To really understand why decentralized SCM is a complete waste for 90% of your projects, you must first step back and look at how you work. Let’s describe your average open source project:

1. Most open source projects are small. Not everything is Gnome/wine/Linux.
2. Most projects I see using git have about 10-12 active developers, with about 3-5 active committers. It can easily be fewer.
3. More specifically, these projects have far fewer developers than users. Most of the people who download the source only do so to compile it– never to edit.
4. Sometimes there is only one main committer with one or two backups. Watch project timelines, you’ll see only a handful of names– the rest will be your odd patch.
5. 99.99% of all open source project inevitably make one official release for each set of changesets.
6. Such a release is usually hosted in one centralized location with maybe a few mirrors strictly for distribution’s sake.

Is anyone coming to a scary realization here? As decentralized as you attempt to make your project, you will always run into a single point of failure: your workflow.

I’m currently watching merb-core development as an example of one of these projects and I’ve noticed that the workflow is essentially equivalent to one with a centralized repository. Someone will submit a patch and have it committed by one of three main committers. If your patch doesn’t make it to the main branch, you’re simply out of luck. Sure, you could use your patch locally, but you could do this with any body of source code whatsoever, .git, .svn or .tar.gz. This is really no different from Subversion to anyone outside the core development team.

If your project has only one or two active committers or falls under any of the above categories, do yourself a favour and don’t waste your time installing git on your server. You won’t be benefiting from its features because your project and workflow will not have changed.

So who really benefits from distributed source control?

Core developers do. The truth is, git isn’t as great as a DVCS as it is a private whiteboard for "pre-commits" to the main repository. Git can make it a lot easier to pass around changes before finalizing them which would mean less broken builds on the main repository. That’s a good thing, and almost worth a two-tier setup (see diagrams below). But really, this is nothing Subversion cannot do with almost as little effort.

Why Subversion can do what Git does

This is what a git development workflow normally looks like:

The outer repository in this diagram is bundled with a release server (web server, most likely) and ticketing system for patches. The "git" blocks are machines with individual branches for each core developer (abstracted from their physical machines in case they use github or something). I didn’t draw all the connecting lines, but enough to show where the bottleneck lies. More importantly, that you’re not really using git as a decentralized development platform (sorry).

Now lets try this setup with Subversion:

Notice that the Tier 1 workflow does not change. Instead, imagine a subversion repository (it doesn’t have to be the same physical repository) where each developer has their own branch and "publishes" their changes by committing to that branch. This development workflow is 100% equivalent to using, say, github, to share changes. Literally– it’s exactly the same. No, really, it is. In this scenario, Joe can merge Larry’s changes by simply– merging them into his branch. When a final release is made, the few maintainers will merge the code that they got from other branches back into trunk, potentially tagging the changeset.

In fact, not only is this workflow exactly the same as Git’s, but it has a side effect that nearly makes it more powerful than using Git: in an optimistic development environment, the maintainers could give out write access for branches to people from the outside world. I could get "Loren’s branch", and start developing my changes in my own little sandbox. This would be similar to git, but the visibility of my code would be much higher in that the core developers would be able to keep tabs on changes that non-core developers are making without having me ping them about it *. I would no longer be a second-class citizen with my own git repository far off at some URI in the public world (see git diagram), but instead I would be developing in the same location where the core team is. I have no clue why people think this is a bad thing.

* To be fair, git could do what I just described, but the developers would need to manage links to all the outsider repositories / track them all with relatively complex, currently-non-free-or-very-private software (github being one). The infrastructure for doing this in Subversion is built-in and implicit.

In summary, group me with all the other people who are skeptical of distributed source control management, please.

Microsoft? Boring– oh wait, free pizza, chance to win XBox? I’m in.

This is a follow-up post to the on-going set of QTrax articles I’ve been posting. If you don’t know what QTrax is, start here or here. If you don’t know what Songbird is, start here.

A Quick Comparison of QTrax & Songbird

For those interested, this is a side by side comparison of QTrax vs. Songbird in the application, which still isn’t allowing downloads but it’s easy to see how it will work in both. Let’s get right to the screenshots:

You can see that the only major difference between Songbird and QTrax is that QTrax re-skinned the play controls to give more room at the top for banner ads. Looking further into the app you also notice that they did a similar job with the library screen to make room on the right for ads as well. But with Songbird working perfectly on the same service (so far), there must be something that the QTrax app must do to deal with this, otherwise, why do I need to deal with the ads when I could just, not?

Another advantage to just using Songbird is that Songbird actually works on OS X, Linux, and other platforms. Why the hell doesn’t QTrax?

Oh, and even if they have something up their sleeves, it’s all right there exposed in the app, since they used XUL in order to write their Mozilla-like extensions…

Removing QTrax Ads

The QTrax service isn’t even functional yet and I’ve already managed to remove their ads. That doesn’t say much for this company’s ability to succeed with this ad-based business model. Frankly, if I was an advertiser or record label, I would probably back out of the deal as quick as possible if I saw this.

Like I said in the previous article, QTrax is really just a bunch of Firefox/Songbird extensions plugged into Songbird without any modification to the source (that I’m aware of). I’d love to dig deeper and confirm that the source has not been touched, but that will have to wait. For now, we can easily just dive right into the XUL that they used to right their ad-adding skin and take it right out.

Let me first point out that there are about 500 different ways to disable the ads in this thing. The simplest would be to just remove the extension. I’m only showing one more complex one just to show that even if the extension added something really cool, we could still work around it.

But here we go. Just load up the extension folder in the QTrax program files folder and go to any of the .xul, .js, or .html files and mess around. You can pretty much undo whatever you don’t like and keep what you do.

I’m waiting to see if QTrax actually thought this through somehow on the server-end… but even then, it would be very difficult unless they actually modified the binaries. I can’t tell if QTrax is really dumb in not realizing how easy they made it to subvert their ads, or really sneaky in trying to make an app that will secretly not be a pain in the ass to use, unbeknown to the labels.

Update: I posted a quick side by side comparison of Songbird and QTrax if you want to see some screenshots of “both” applications in action.

After playing with the anticipated QTrax beta, I have quickly discovered that QTrax.. is Songbird. Not just based on.. is.

(more…)

At midnight tonight, QTrax will be unveiling what can potentially be a radically new business model for downloading music online. To be fair though, the business model is in no way innovative. Offsetting costs of membership-free sites with banner ads is about the oldest trick in the book. The difference in this case is the fact that the major labels are actually on board with this idea, as Wired recently reported.

This actually looks pretty exciting. You can still hear the loud roar of those people against DRM claiming that this is not progress, but I think we should see how DRM would actually be able to limit the portability of music that has already been made freely available to anyone over that network. Considering that they plan on making iPod compatible in the near future, the music provided by this service should not only be able to move from machine to machine (by sharing it, at worst case), but from music player to music player. Without having seen the application in action, I can’t imagine how DRM in this context could do much harm.

Update 1 (01/28/08 1:00 AM): Seems like QTrax might be in some hot water with some final attempts to start the service, as reported by Silicon Valley Insider. At 9:57pm, QTrax was still in discussion with WMG trying to finalize the deal. This might explain why there is still no beta available to download. Will they be able to pull this off?

Update 2 (01/28/08 2:00 AM): The guys over at GameDrift seem just as anxious as me to get my hands on this stuff and play. They claim that the launch has been moved to midnight PST (instead of EST) and are going to be waiting to report on this live, so check it out.

Update 3 (01/28/08 12:00 PM):

“The future of Qtrax, a new file-sharing website which promised legal downloads of over 25m music tracks, is in doubt today after three of the four major record labels denied giving the site permission to use their music.” - source: Guardian Unlimited Music

Update 4 (01/28/08 6:13pm): The guys over at GameDrift did a little digging and found a link to the 0.2 beta version of QTrax. Keep in mind, downloading from the service is still disabled, but if you want to check it out, you can either install or check with GameDrift for their overview.

Update 5 (01/28/28 10:20pm): QTrax officially released the software from this download link (link from the download page). The site is currently acting up, but hopefully downloads will begin when it starts functioning again.