a glob of nerdishness

Our little two man company just got back from our second trip to attend Seattle Xcoders. Once again, we had great conversations with friends and developers there. Among ourselves on the way there and back, we talked about many hopes and ideas for our company’s future. We discussed the temptation to let overlofty ideals about how software “should be” hamper the smaller goals that must first be achieved along the way. And we imagined ourselves writing great software that helps people do great things, built on top of a great platform.

Then we came back to this bad news. Apple have decided not to reward yet another developer’s hard work, this time on a very legitimate iPhone application. When the iPhone developer’s kit was announced, developers were warned that certain limitations would be enforced — Apple wouldn’t allow apps that were (for example) malicious, violated users’ privacy, “illegal” and other such things to be sold. This generated some immediate speculation about how Apple would use this power they’d reserved. After months of having to work under an NDA that would forbid developers from learning this new operating system as a community, now comes this confirmation of a once-latent fear. It’s a tough struggle to turn ideas, time, money and other limited resources into a product that will improve user’s lives. Now Apple are using their control in a way that makes the expected return on this investment even lower, especially for competitors of Apple or Apple’s partners. They’ve set up a game where they are the only ones who can’t lose, unless customers and third party developers stop playing.

Already one talented developer has announced that he will not be writing any more iPhone applications until the rules are made clear to all contestants and the referees are no longer hidden behind one-way glass. Apple claim they are still “processing” our application into this Secret Sharecropper Society. Should we email them and tell them not to bother, that we don’t want any share in this? If we believe Apple’s market is unnecessarily risky for us (it is) and if we believe Apple’s censorship will hurt our customers (it will) and if we believe this kind of human-centralized control will end up harming everyone everywhere (it may) — if we believe all that — is it hypocritical of us to still want to make iPhone software?

Worse, it is also Apple who control the equally amazing desktop platform for which we have been writing software since before January. What if this is also the future Apple intend for Mac development? Will we be legally obligated to learn and grow alone and leave others to do the same? Will we be unable to properly test our code in real world conditions? Will we have no choice but to give a cut of all our revenue to yet another bureaucracy that thinks it knows what’s best? And after all that, will we occasionally have the freedom to sell our work simply taken away? If so, we’re making a very bad investment.

What should we do? In a number of countries, the World Wide Web still has many of the freedoms and opportunities that the Wild Wild West is rumored to have once held. But just as many covered wagons never made it, and the ones that did learned to do without the modern conveniences they once enjoyed, we like the comfort of the desktop and our software is better for having the plumbing and wiring a web cabin just doesn’t provide. If we stay on Apple’s good side, we might be able to make a lot of money. And except for the ideals we’d have to disregard, it would be honest money too.

The first version of our software will not do all we think it should do for our paying customers. We don’t intend to rip anybody off, and it’s not because we disrespect our users. It’s because we are learning, at least with software development, that sometimes it is impossible to get to where we want to be without making some practical compromises along the way. We’ll always try to avoid doing things the wrong way, but sometimes we don’t yet have the ability to do them the best way either. I wonder if more of life — or at least this business struggle we’ve found ourselves a part of — I wonder if it’s okay to invest in a system we don’t fully agree with, at least until we have the opportunity to trade in for something better? Our company would be grateful for your advocacy to the one who is really in control on our behalf, and I would appreciate reading your thoughts about these “compromises” as well.

[Editors note: I'd been editing this article for a few days, and then John Gruber sent everybody to a similar article on 5thirtyone instead. I'm not jealous or anything — this glob is too typographically atrocious to merit a link from DF at present — but I still wanted to put this up for my own reference.]

Google recently rolled out free IMAP support to Gmail users. This is a neat gesture, but they twisted the IMAP protocol so that it works in The Way of the One True Algorithm. In their own words, “we’d like to make your IMAP experience match the Gmail web interface as much as possible”. Fortunately, Apple’s Mail provides the tools necessary to work around most of this Google IMAP “experience”.

The skinny

1. Set Mail.app to work with Gmail
2. Google has recommended settings. Ignore them. Well, DO uncheck “Store sent messages on the server”, unless you are using a non-Google SMTP server. But don’t uncheck “Store deleted messages on the server” or “Store junk messages on the server”.
3. Map Gmail IMAP Folders to Mail.app Default Folders:
   

   Use mailbox	For Mail’s…
   [GMAIL]/Trash	Trash
   [GMAIL]/Spam	Junk
   [GMAIL]/Drafts	Drafts
   [GMAIL]/Sent Mail	Sent, only if you are using a non-Gmail SMTP server

4. Google has a table showing what actions in your email client do to your Gmail. Read them, but realize half of them are wrong or irrelevant with the way Mail.app is now set up. Here’s some corrections:
   

   If you want Gmail to…	Do this in mail
   Apply a star to a message.	Flag the message.
   Apply a label to a message.	Copy the message to the corresponding folder.
   Remove a label from a message.	Move the message to “[GMAIL]/All Mail”. Don’t delete the message.
   Undelete a message.	Move it to “[GMAIL]/All Mail” or another label.
   Make “All Mail” not match what Mail.app shows	Delete a message from “[GMAIL]/All Mail”. Don’t do this!

5. As you use it, sometimes things get out-of-sync for a bit since Gmail is changing folder contents behind the scenes in ways that Mail.app doesn’t expect. If you want to make sure that what you see in Mail reflects the way things are in Gmail right away, use the “Synchronize All Accounts” item in the “Mailbox” menu.

The key to understanding how Google changed IMAP is to realize that the folders it presents are never locations in the sense that folders usually are, which is the sense that Mail.app treats them. Every Gmail IMAP folder is just a “Label”, a tag. Because of this, you may find (or place) copies of the same message in multiple folders. To Mail.app, these each look like individual messages with an identity of their own. To Gmail, they are just differing representations of one underlying object, which can only be deleted via the Trash.

The gory details

So what’s the problem with just doing it the way Google says to? Having folders represent labels means that Mail.app’s “Delete” button won’t work like it does for with normal IMAP accounts. When you “delete” a message in Mail, it removes it from whatever folder it was in and puts it in a deleted items folder. Since it doesn’t know about Gmail’s Trash folder, it creates a new one named “Deleted Messages” and moves your deleted items there. This is a problem, because to Gmail you’re just removing the “Inbox” tag and adding a tag called “Deleted Messages”. You haven’t really deleted the item, and it will still show up in “[GMAIL]/All Mail” and any other Label folders it was in. Then when you empty Mail’s trash, Gmail just sees you removing the “Deleted Messages” label and the message lives on, even if orphaned.

Deleting

To actually delete the underlying message, you must place a representation of it into the “[GMAIL]/Trash” folder-aka-label. (That’s what the mapping in step 3 is about.) While it hangs out there, the message will be hidden from all the other label folders it was in. If you move it back to another “folder”, it will reappear in all the Labels it previously had. There is one catch, though.

Not deleting, when we don’t mean to

What if we want to keep our message, but just remove a particular Label? If we hadn’t told Mail to use the “[GMAIL]/Trash” folder for storing deleted messages, we could just delete a message from the corresponding folder to clear that label. But if we do that now, Gmail will get not only a message saying “remove this message from Label” but also “add this message to [GMAIL]/Trash”. This will cause the message to be hidden from ALL labels, and when we empty the trash it will disappear for good. So we can’t do that, despite Google’s suggestion. Instead we move it, which sends two messages to the server: “remove this message from Label” (thus accomplishing our goal) and “add this message to the [GMAIL]/All Mail folder” (where it probably already is anyway). The same trick can be used to undelete a message as well.

When to move, when to copy

When copying message within Mail.app, Gmail is smart about maintaining only a single underlying identity. This is important, because to add a label you can’t really “Move” a message from one folder to another, because that would also remove the label you moved it from. So, generally, to add a label to a message copy it instead of moving by holding down the command key while you drag. If you do want to remove the Inbox label (for example), then by all means do move instead of copy.

Regarding “Inbox” and “All Mail”

Both “Inbox” and “All Mail” are just tags. If you remove an item from either, it stays on Google’s server (unless you move a “copy” from any folder into “[GMAIL]/Trash” or “[GMAIL]/Spam”, which we’ve set up Mail to do). There seems to be a discrepancy between “All Mail” IMAP folder and the “All Mail” view online: if you delete messages from the folder in Mail.app it still shows up in the web interface.

In thinking about wrapping up…

If you’ve got any other questions, tips or corrections feel free to leave them in the comments. Or in the comments on my new arch-nemesis’s article. But keep in mind, when I find out which friend in Omsk told friend in Tomsk the results of my research, there will be great suffering in Guilder. (Kidding, kidding!)

As reported by Gus Mueller, remembered by Fraser Speirs and reverse engineered by dragging messages from Mail.app into a rich TextEdit document: Mail.app now supports permalink URLs to messages.

The URLs are just “message:” followed by the Message-Id, which should be URL encoded. So if your Message-Id is “<abc%20071105@sender.org>” — the angle brackets are considered part of the Message-Id — the URL becomes “message:%3Cabc%2520071105@sender.org%3E”, although “message:<abc%2520071105@sender.org>” will also work from Safari’s address bar.

This is basically just the mid: URL scheme with a different scheme name. Why they used “message” instead of “mid” is strange, especially since on the Flying Meat forum there’s a discussion of links provided by an add-on called MailTags that use a similar URL scheme. These have an extra “//” before the Message-Id, and make Mail grumble that “No associated application could be found”. Update: On second try the extra “//”s seem to work as well, so the real question is why MailTags, not Apple, didn’t use the “mid:” form.

To get the Message-Id, select “Long headers” or “Raw Source” in Mail’s “View > Message” menu. You can also drag a message from Mail into a rich text field to get the hyperlink. Enjoy!

Update: John Gruber has put together a more definitive article on this topic, though he doesn’t make mention of the (seemingly historical) “mid:” URL scheme.

I’ve been using Leopard for just over a week now, sans Time Machine even, and it’s good stuff. For the record, a lot of the glitches seem to disappear after one reboot. I’m getting used to Spaces, though I still wish there were some little onscreen indication of which space I’m currently in. And I already might be attached to the glitz of the Dock/menubar.

• Top feature so far: Xcode 3.0, hands down.
• Top little fix: No intermediate .tar file from .tar.gz extraction.
• Coolest spectacle: iChat switching between local and shared screens.

Over and out.

Editor’s note: This is the only Mac or technology blog that my lovely wife reads, but within five minutes of looking at her new desktop, she was begging to make her disgust known publicly. Here are her notes, with only a few edits for clarity. If you thought developers (I’ve added cross-references) were just being curmudgeonly about some of Leopard’s new “features” , take a look at what a customer thinks.

Here’s what bugged me:

-The side dock is ugly. The dark box makes the screen look more cluttered. [I like the curved popping out folders], but they do not do that from the side. cf
-the semi-transparent top bar is unnecessary, sometimes annoying. cf
-there are obvious glitches (Camino didn’t hide when I told it to, etc.) Hopefully these things will be fixed soon.
-the new icons for “Pictures”, “Desktop”, “Documents” folders, etc. are indistinct. What were they thinking. cf

In summary, side dock users have been slighted and it seems that many changes (such as the top bar, and the rounded-edge pop-up menus) are just for the sake of having a change, and do not represent any true improvement.

Time Machine and other things may be great, but hopefully I will not be using time machine on a daily basis. I do have to look at the dock on a daily basis. Boo.

What was kind of cool:

I do like the calculator and dictionary in spotlight (hopefully I’ll remember to use them!) and the make-your-own widget from the web. And that I can use formatting such as bold while in Safari on blogger. Easily adding phone numbers and appointments to Address Book and iCal [from Mail] was impressive and might be useful.

On Friday, we upgraded Hannah’s Macbook to Leopard, and then packed for a funeral back in the Midwest. The Developer Tools stayed off, and I spent the available portions of the weekend as a regular user. Ignoring all the under-the-hood improvements, I’m not sure if I should be eager or worried for the future of OS X. I came back home disappointed in Leopard.

I’m not trying to say that I, and the other 2 million users who went out and bought Leopard last weekend, should have stuck with Tiger. Leopard is (or will be, after a few patches) a good upgrade for current Mac users. Combined with some gorgeous hardware, it offers a very tantalizing package to first time buyers.

The trouble is, the best improvements are mostly all under the hood. They’re invisible as they should be, and you can’t queue many end users willing to pay for a developer API update. Time Machine is the one killer feature, and while many the other additions are sure to be addicting, Apple needed a larger swatch of gateway drugs to make the sale. Which is probably why every other headline feature seems thrown in just to make Leopard look more valuable.

Finder is improved and QuickLook is handy — the latter being a good way to encourage third party developers to pitch in with the former — but nothing truly revolutionary. The Network folder is still present and still rubbish.

Stacks seemed like a cool feature, but it turns out those aren’t actually new. The “scoliosis” view can disturb physical therapy majors, and folks who used Tiger’s stack feature will miss the functional downgrades.

iChat’s cheesy effects appeal to…I dunno. People who enjoy getting into Photo Booth with friends but would rather not be in the same photo, or even the same “booth”, as said friends?

Spaces has so far been more confusing than helpful. I’m not sure if the typical end-user will even bother to turn it on.

The myriad minor improvements throughout the OS — faster Spotlight, no network share beachball of death — are currently outweighed by all the first-release glitches — countless little window manager, desktop and Finder bugs that come and go, rapid battery discharge with the lid closed, broken Safari web archives.

Of course, many of these issues should be resolved in forthcoming software updates. Perhaps some of the eleventh-hour feature removals will also find their way back in. Then users can start getting attached to another great release, one that enables some great third-party software. I wonder, though, what Apple is planning 12–18 months down the road for 10.6. Will they expect users to fall for bells, whistles and fixes, or do they have at least one more killer feature up their sleeve?

AppleInsider has been posting a great series called “Road to Mac OS X Leopard” this month. The journalism shines not in the actual Leopard bits, but in the historical lead-up to each:

• QuickTime, iTunes, and Media Features
• System Preferences
• Parental Controls and Directory Services
• Collaborative Info Sharing Services
• Dashboard, Spotlight and the Desktop
• Safari 3.0
• iCal 3.0
• iChat 4.0
• Mail 3.0
• Time Machine
• Spaces
• Dock 1.6
• Finder 10.5
• Dictionary 2.0
• Preview 4.0

I’ll update this list as necessary. Tip: as elsewhere on teh adverwebs, the print version gives you the current article on a single page.

My favorites have been Safari (though sans Vannevar Bush and Ted Nelson), the Desktop, and Collaborative Services.

On OS X, you use otool instead of ldd to view required shared libraries of an executable. An interesting browse, proofread by slashdot last year, providing further reading as well. (See also the Firefox Poster from Source Code on that blog.)

I’m looking forward to Leopard’s upcoming release, but part of me is already wondering what exciting features 10.6 might bring. While I can’t say there’s anything I personally resent in Leopard (except perhaps the iChat special effects), nothing is particularly compelling either. Nothing on the outside, that is. Under the hood is an entirely different story, explained quite in depth by the 3 part Developer’s-eye view of Leopard series at InfoWorld.

As an aspiring developer, I’m eager for the budget-friendlier Leopard start kit, even if it doesn’t include a hardware discount or technical support.

Just found out that Will Thimbleby has a blog. No deluge of posts there yet, but it’s in my feed reader.

Will has created all sorts of neat apps like Lineform, RecDit and, my personal favorite, MacSword. While I’ve been using the latter since I got my Mac mini in the spring of 2005, I didn’t know much about Will’s other work until I read an article about Lineform’s development on Apple.com. He seems like an all-around brilliant chap who I’d love to meet, should I ever make it to the UK.