On tagging

OpenMeta is the most useful tool in my tagging toolbox. It allows me to attach tags to anything with a filesystem representation: websites, documents, photos, source code, emails and more. Before OpenMeta we had to make do with whatever fields Spotlight would search, which was Finder comments on files, and really no options on emails and other items accessed outside of Finder.

Now that we have OpenMeta¹ and a plethora of tools (I highly recommend Tags.app) to work with, tagging is a real option. We just have to figure out the best way to make use of it.

Things I’ve learned

The goal of tagging is to associate files into groups that wouldn’t be feasible using a folder hierarchy. As I’ve mentioned before, I still use a shallow hierarchy, especially for grouping project files together. The idea of “one big pile” is as frightening to me as it is to most people. I use tags to create additional groups that might not otherwise be possible, bringing together files and other objects that might be related in ways that a folder reflects.

The primary problem you run into is that tag collections get out of control, and navigating them isn’t any faster than drilling through folders and mailboxes. This happens because multiple forms of a tag are used, or long after the object is tagged the original associations are forgotten. I’ve learned this the hard way, and on multiple occasions.

There are two basic approaches to tagging:

1. Tag with anything and everything you could possibly associate with the object (file, email, photo, etc.). It’s a free-association tag-for-all that results in a huge collection of tags. This is navigable, but often results in messy tagging.
2. Tag sparingly, only using tags that will be easy to associate other objects with. Anything that is searchable in text or other metadata is left out of the tags. This is the method I’ve come to use.

What I’ve learned is to do follow three obvious-sounding rules: tag intuitively, intelligently, and consistently.

Intuitively

Drill down

I typically limit myself to three levels of tags. Top-level tags encompass an overall topic or broad scope that the object falls into. For example, if it’s a website, I tag it with the primary keyword that led me to the page, or the subject I was interested in when I followed the link. If it’s a project you’re working on, a client’s name is probably the right choice for a top-level tag. Tags such as “email,” “communication” or “research” aren’t top-level tags. If used at all, those are third-tier.

Second-tier tags divide the group into subsections. If your top-level tag is a client name, then second-tier tags would include a project name or other unique identifier that can be shared across all related files. If you consider your top-level tag specific enough for narrowing a project down in a future search, just skip to third-tier tags.

Third-tier tags are where you begin to cross-pollinate the tag groups. These tags are going to be common tags across multiple top-level groups. Tags like “research,” “approved,” “communication,” etc. can make sense here. This comes down to personal style. The only real rule is that they have to be tags you’ll consistently use elsewhere within other scopes. They’re what make this different from just using a folder hierarchy.

This pattern allows you to view your tag groups in a drill down fashion. If you tag with two or three levels, it increases the ease of browsing in various situations. If you were always going to look for a file as CSS3, you should just use a folder. It’s nice to be able to browse for all your web design articles, but maybe you’re just interested in the CSS articles at the moment. A search for bookmarks tagged “tutorial” and “CSS3” will quickly produce all of the relevant articles you’ve found in your web browsing.

Plus, you can cross groups: an article that covers html, javascript and CSS would get tags for all three, assuming I would be interested in all subjects in the future. If I dealt only in CSS, I wouldn’t bother tagging with the other topics, it would just pollute my tag collection.

Choose tags that will be useful next year

“What will I search for when I’ve forgotten this item exists?” It’s not always an easy question to answer. The basic rule is to take what first comes to mind and ask yourself that question. Is the first association you make related to current circumstances or events? You might not have that association in a year. You want to use tags that come to mind easily, but double-check yourself to make sure it will be as front-of-mind when the time comes to search.

For example, if it’s a website, tag it with the primary keyword that led you to the page, or the subject that you were interested in when you followed the link. If it’s a project you’re working on, a client’s name and the scope or project name are probably the right choices for top-level tags.

There are a lot of edge cases. You get better at it after you’ve had to deal with your own tags for a few years.

Intelligently

These are common sense rules that make tag collections work. You probably know them already (or have figured them out if you’ve been tagging for a while).

1. Don’t tag with data already available

   Unless your system has no other means of searching metadata, you can always add things like dates, filetypes and even content to your search to find specific files within a tag group.

2. Use lowercase tags. Always

   It gets messy if you’re inconsistent, and autocomplete will almost always substitute the first capital letter it comes across in a completion, resulting in your previously unused tag now being capitalized.

3. Don’t use “flagged” or other time sensitive tags

   This one needs some explanation. Unless you are religious about removing “tickler” tags as you go, find another way to denote importance of the tagged element. Finder labels work well for files, and you can keep a “Current” folder with things that need to be attended to. Once you’ve handled whatever needs to be handled, you file it (or send it to a script that will).

   I know this from experience. I tend to be pretty good at reviewing my systems, but the “flagged” and “important” tags got out of hand quickly. Pretty soon I was looking back at flagged files, emails, photos and bookmarks and I really had no idea why they were important anymore.

   In a best-case scenario, you’re using a project/task management solution that allows links. If I need to reply/follow-up on an email later, I drag it into my task manager and create a new task. That gets the flag, the due date, and any notes I need to remember why. I could tag/flag the email as well, but then I’d have to untag it later. This way I just check off the task. Side benefit: assuming you don’t delete completed tasks, you can find it later and follow the link as an easy bookmark.

Consistently

If a tag is used only once, you’re wasting your time. You could have found that file, page, calendar entry, etc. with Spotlight. You didn’t need a tag to do that.

Keep a list of common tags if you need to, at least until they become second-nature. Most of the tagging applications will show you common/recently-used tags when you’re tagging. The best thing in the world can be autocomplete, assuming your tag collection isn’t already a polluted mess. Autocomplete in tag dialogs means you don’t have to struggle to remember casing, plural forms, etc., just go with what you did before.

I generally try to avoid capital letters, hyphenation and plural forms. I’ll use multi-word tags frequently, but without punctuation or intercaps.

Exceptions

There are some obvious exceptions to these rules in my system. For example, I keep a list of things I want using a “tobuy” tag. I have to manually untag these things as I buy them or decide I don’t want them anymore. The tag works really well with Smart Folders in Finder, though, and I don’t have trouble keeping up with it. It’s not an area that changes rapidly; just a folder I can browse when I have some spending money and a shopping urge. On these occasions I just untag “tobuy” items as I go through them. It never gets out of hand and it never takes long to update. Unlike the “important” tag I used to use in all of my projects, “tobuy” is basically a top-level tag that I always search for across all topics and groups.

I also sometimes use questionable tags for scripting purposes. I prefer client→project folders to their redundant tag counterparts, mostly just because I’m going to separate those anyway for the sake of filing sanity. I’ll use client/project tags on files on my Desktop, though, and my scripts will automatically sort those tagged files based on their tags and subtags. It’s a bit of a complex system, but it lets me use my Desktop as a general bucket for everything I’m working on, and have the various elements from multiple projects all neatly filed at the end of the day. The filing tags can be removed by the script after they’ve found their home.

There are plenty more exceptions. I don’t think any two people’s tagging systems will be identical. It’s the beauty of tagging: you can build a system that works for you. It doesn’t have to be rigid, but some general, self-imposed rules can definitely make it a more useful process.

Related posts:

1. On sorting, tagging and other nerdery
2. AutoTag2: smarter tagging for TextMate and WordPress
3. A Bash function for Markdown bloggers

Originally posted on BrettTerpstra.com at Some suggestions for better tagging

I forget a lot of things. I forget how I did something, or where I found an answer, or where I put my glasses. I use technology to help with that to some extent, and Notational Velocity/nvALT are excellent at helping me find previous notes, assuming I bothered to take them.

I keep a log in VoodooPad, especially when I’m up late enough at my computer that I might not remember what I was working on in the morning. That’s handy, the search is excellent, and gathering tagged entries together on one page automatically is sweet. After a couple of years of doing it religiously, though, I’ve found that I rarely reference anything beyond the previous day or two. I rarely go back because the things I log aren’t “knowledge”, just activities. My logging system lacks the details I need to remember exactly how I did what I did. That’s my own fault, of course, but I needed a system that would build an environment-agnostic knowledgebase for remembering all the things I tend to forget.

Thus, I’ve been playing with an idea. I’m building a collection of plain-text notes (primarily using nvALT with notes saved to a Dropbox folder) where the title of the note is the question I think I would ask myself long after I’ve forgotten the answer. The content of the note is the answer in its briefest form, including commands, filenames, urls and whatever is necessary to get me back on the right track.

I can query these notes quickly from the command line (qq where glasses or qq best h.264 settings), as well as from LaunchBar and Alfred. Adding new notes can be done just as easily from the same sources. The text files are stored among all of my nvALT notes (with a special prefix to easily separate them), so it automatically benefits from Notational Velocity-style searching and editing.

One of the reasons I prefer to store my nvALT notes as plain text files is that I have access to them from Spotlight and mdfind (man page). Spotlight allows me to query a little more intricately, separating tags from titles and titles from content in a way that nvALT currently can’t. mdfind is the crux of this system. You can store all you want in plain text and feel great about it, but the ease of querying any information system is the measure of its worth.

I like tagging. I’ve found that tagging isn’t the solution for my forgetfulness, though. It’s been extremely helpful in tracking projects across multiple folders, but not so much in dealing with aged notes. I’m as likely to forget a tag I associated as I am anything else. Extra, more verbose text seems to be the trick. Instead of piling on a bunch of tags, I set my scripts up to search–but not output–anything inside of a @(extra text tag). Then I can add extra verbosity and keywords that I think might help the search later… without fouling up my tag database.

While there’s definite GTD-style comfort in knowing I have my discoveries and important notes in a portable, syncable system, the process of thinking through the syntax of the question/filename is actually helping me remember things better to begin with. It’s a conscious effort to connect neurons in a way that keeps the information more readily accessible to my feeble memory. I’ve only been doing it for a month, though, so I’m curious to see how the knowledgebase looks and functions in a year.

I’ve put the whole system and installation/usage/best practices up on GitHub. There’s a command line tool (bash), a pair of LaunchBar Actions and two standalone Alfred extensions for adding and querying the notes. As I use it more, I’ll probably come up with more ways to interface with it. Right now I’m thinking about an AppleScript/Mail Rule-based system for adding and querying notes via a GMail alias. More to come, pending determination that the new method is worth the fiddling time in the end. I do love having universal access to any bucket I use, though…

If you’ve ever forgotten where you put something important or solved a problem and later forgotten the solution, check out the QuickQuestion GitHub page. The Readme and up-to-date files are also available as a direct download below.

Related posts:

1. Quick tip: running nvALT after a Notational Velocity update
2. Taking the Markdown to Evernote service further
3. nvALT 2.2 public beta

Originally posted on BrettTerpstra.com at Forget about it. Or not.

Thinking about it for a minute, I realized I really did miss the functionality that AutoTag had given me. While I’ve toyed around with blogging from Vim, I still primarily use TextMate, not least because of the Blogsmith Blogging Bundle. Until that kind of functionality is ported to Vim, I’m happy right here. Thus, I spent my Sunday afternoon reviving the AutoTag bundle.

If you use TextMate to blog on a WordPress site, you’ll want to see this.

There are two parts to this: a WordPress plugin that adds an XML-RPC call for getting all of the tags in use on the blog, and a small TextMate bundle for making good use of it. The TextMate bundle cribs off of my previous AutoTag project, and borrows most of the remote functionality from the original TextMate blogging bundle. Why reinvent the wheel? I’m bundling both together in the download, so if you want to use the WordPress plugin to port the functionality elsewhere (say, Vim?), you can just pull out the parts you need.

[Skip to download]

AutoTagging in TextMate

What the bundle does (short version) is find the most appropriate tags for your content based on tags you’ve already used before. This helps you easily keep your tag collection trim and useful. No disparate tags because of inconsistent capitalization, plurals or conjugation. Things that are related stay related. This can be especially difficult blogging from TextMate because you can’t quickly see “all tags” the way you can from the WordPress editor. This solves the problem.

Install the WordPress plugin

Standard install, like any other (manually-installed) plugin:

• Upload the gettags-rpc folder from the zip download to your wp-content/plugins folder on your server.
• Open your admin page and go to the Plugins panel.
• Activate the GetTags-RPC plugin in the plugin list.

If the plugin activates without error, you should be ready to go. It’s a very simple plugin without a lot of error checking, so let me know if you have any trouble with it.

Install the AutoTag bundle

To install AutoTag in TextMate, double click on the bundle in the download zip file. It should open TextMate, pop up the Bundle Editor and show you the installed bundle and its commands. Done.

Assuming you have AutoTag installed in TextMate and the GetTags-RPC plugin installed in WordPress, you’re ready to go. Well, you’ll need to have your blog set up (Bundles > Blogging > Setup Blogs) in TextMate, but we’ll assume you’ve done that by now.

Using the commands

The AutoTag commands can be triggered in two ways. First, you can just put your cursor one space after the colon in the “Keywords:” line of the template headers. If you don’t have that line in your template, you can just type “Keywords: ” on a line before the end of the headers, or add it permanently by editing the template in the Bundle Editor. Hit tab from there and you’ll get a menu of the three commands in the bundle. You can trigger that from the same point even if you already have tags on the line; it will add appropriate commas automatically. Second, you can use Control-Option-T to get the same menu at any point in the file, although I’m not sure why you’d call it from anywhere other than the “Keywords” line.

Commands

AutoTag: This is the primary command, or at least my favorite in the bundle. It’s only good when you run it after you write your post; it doesn’t have anything to work with on a blank template. Finish your post (or open an existing one) and trigger the command. It will grab all existing tags on your blog and compare them to your text, resulting in a dialog with a list of the best matches ranked by relevance. If you’re curious, here’s what it does when processing:

• Strips out all punctuation from the content of your post and creates an array of words
• Uses the Porter Stemming algorithm to get the stem of each word. This makes plurals, adjective, adverb and verb forms all represent the same word.
• Compares the list of stemmed words to the stems of each tag returned by your blog, keeping matches.
• Measures the frequency of each match in the content and sorts the list accordingly.

Common Words: This command ignores the tags on your blog and just gives you the most common words from your post for selection as tags. It’s a good way to figure out what new tags might be worth adding. Again, it’s not very useful if you don’t have your content written yet.

All Tags: If you’re not finding the AutoTag results satisfactory on a given post, or you want to tag before you have any content, this command will simply return your entire list of available tags. You can use it to check what form or casing you used on an existing tag, and it can be run at any time.

That’s it. If you use WordPress and TextMate together, I know you’re going to love this. Let me know how it goes! By the way, I’ll probably (eventually) build similar functionality right into the WordPress editor, and I’d love to see this ported to Vim, too. Any Vim plugin writers want a challenge?

Download

Related posts:

1. Markdown QuickTags: WordPress plugin for Markdown lovers
2. TextMate as WriteRoom, for free
3. Getting back into real TextMate blogging

Originally posted on BrettTerpstra.com at AutoTag2: smarter tagging for TextMate and WordPress

The download is at the end of this post, but I highly suggest reading at least the setup section before you get all crazy with it.

I wrote a script a while back that would watch for changes in my Delicious bookmarks and mirror new ones to .webloc files on my drive, applying OpenMeta tags so that they were locally searchable with Spotlight and other OpenMeta applications. I stopped using Delicious, though, and started using Pinboard, which I mentioned recently that I adore. In the process of revamping the script it became a little more “involved” than the original.

This script is for people who want to take advantage of Pinboard–with its full text search, easy privacy settings, accessible API, etc.–yet still want to be able to search their bookmarks in local Spotlight (and similar) searches. While it has the option to save bookmarks with a certain tag as searchable PDF files, it doesn’t attempt to replicate the full spectrum of Pinboard features. It’s just a way to make your remote bookmarks locally searchable, available system-wide and OpenMeta compatible.

I toyed around for a long time with using safaribookmark files instead of webloc files. They let you store a larger preview image, and you can include full text from websites within the XML of the file. Lots of possibilities there. For many reasons, I decided to stick with these little webloc files. If I want fancier images and web text, I’ll use Webbla, and if I want comprehensive full text search I’ll use HistoryHound, both excellent programs in their own right. I want OpenMeta and simplicity, though. If I know I’m looking for a bookmark from Pinboard, I can just go to Delibar and do some searching. The goal is to be able to include my web discoveries in larger searches on my Mac.

Setup and Features

First, put the script (download below) somewhere you can leave it, preferably somewhere in your system path. That’s not a huge deal, though, because you’ll be supplying an absolute path in most automation cases anyway. Once you have it situated, open Terminal and run chmod a+x /path/to/your/script.rb. Now you can run the script from the command line to configure and test.

When you run the script the first time (do it from the command line with /path/to/script/getpinboard.rb), it puts a configuration file at ~/getpinboard.yaml. It will let you know exactly where it is, and will automatically open it in your text editor. You must edit the configuration settings before you’re ready to run it again. The configuration has options for all of the main features of the script, so these instructions are also going to be the tour. You can edit any of these options at any time. Note that the next time you run the script it will pull in up to 500 of your bookmarks, starting with the oldest. If you decide you didn’t like a setting, you may want to trash those files and the database and start over. Try not to let that happen.

Configuration options

user and password (string)

Set these to your Pinboard credentials

dateformat (string)

Leave this as ‘US’ if your local date format is mm-dd-yyyy. Set it to ‘UK’ if your date format is dd-mm-yyyy.

target (absolute path)

This is where the webloc files will be collected. It works great with a Dropbox folder, but put it anywhere you like. On my system, I have my ~/Library/Caches/Metadata/Tags/Bookmark folder (where Tags.app stores its tagged bookmarks) symlinked to ~/Dropbox/Sync/Bookmark. That Dropbox folder is my target for the script, so I’m saving my Pinboard bookmarks to my Tags folder and still syncing them (and their OpenMeta tags) to my other computers. Further Tags integration will be covered at the end of the options.

If a folder specified in the config is missing, the script will attempt to create it.

db_location (absolute path)

This is the location of the bookmarks database. The filename will be bookmarks.stash, and it’s perfectly fine for it to exist in the same folder as you set for your TARGET.

pdf_location (absolute path)

If the PDF_TAG below isn’t set to false, this is where PDFs of bookmarks with that tag will be created. This requires the latest version of Paparazzi! (which does run fine on Snow Leopard).

tag_method (integer 0–2)

This determines how the OpenMeta tags will be applied. Use 0 to disable, 1 for Tags.app or 2 for the OpenMeta CLI utility.

always_tag (string)

I like to mark my tags which come from bookmark services for top-level grouping. This setting defaults to “pinboard”, but you can change it to anything (or leave it blank).

update_tags_db (boolean)

If you’re using Tags.app, you know you can tag a web page with it and it will remember the tags next time you visit that address. That doesn’t work with external tools, though, because Tags keeps a separate database for those links. Setting this to true will let the script update the Tags database and keep everything in sync.

create_thumbs (boolean)

If set to true, this feature will add custom icons to your webloc files using a screengrab of the website and the website’s favicon. It looks great in icon and CoverFlow views.

This is another external requirement. To get thumbnails, you must have setWeblocThumb, a free utility for doing just that. The utility must be located at /usr/local/bin/setWeblocThumb.

Note that creating thumbs takes a typical 4-8k webloc file and makes it around 160k average. My bookmarks folder has nearly 200MB of bookmarks in it, tagged and thumbnailed. That’s ok with me, but if you want to keep it small and agile, skip the thumbs.

pdf_tag (string)

The string defined here will determine which bookmarks, if any, are also saved as searchable PDF files. Just use it as a tag and Paparazzi! will download the url in the background to the location you set above.

debug (boolean)

Leave this off (false), unless you need a little more info about what’s going on. It will use Growl and STDOUT to display progress if enabled.

gzip_db (boolean)

I can’t imagine the database file that this generates being large enough to worry about size, but this option will cut the disk space it requires significantly. I leave it off, but it’s your choice.

Optional additions

As mentioned above, if you want to create thumbnails for your webloc files from screenshots of the web page, you’ll need setWeblocThumb, a free utility for doing just that. Its functionality is included in the script, just install the utility and make sure thumbnailing is enabled in the config. The script expects the utility to be located at /usr/local/bin/setWeblocThumb.

If you want the option to save bookmarks with a certain tag as fully-searchable PDF files, you’ll need the latest (I use the term loosely) version of Paparazzi!.

You’ll also probably want Growl installed. I can’t recall if the command line utility growlnotify is set up by default, but that’s what the script uses to send notifications. It will live if you don’t have it, but it generally won’t try to communicate by any other channels when debugging is turned off.

Running the script

There are a couple of options for automating the script. You can have it run at regular intervals; it stores its last update time and compares it with the Pinboard server before it bothers downloading anything. Once you’re up-to-date on your sync, you could run it every 15–30 minutes without any trouble. The easiest way to do that is with launchd, and the easiest way to do that is with Lingon. If you don’t already have it, grab it from the Mac App Store. It’s worth the five bucks. Use the wizard to set up a schedule and run the script.

What I do is set up Hazel to watch the database file for Delibar. Delibar is my favorite app for bookmarking and searching my online bookmarks. It works wonderfully with Pinboard, and I can’t recommend it highly enough. I can hit a key when I’m on a website in any browser and be able to quickly comment, tag and save the page (either privately or publicly) using the same Cocoa interface every time. Anyway, Delibar keeps its database in ~/Library/Application Support/Delibar and the file is named DelibarDB.xml. I simply watch for changes since the last match, and then run the script when one is found. I’m sure you could accomplish something similar with Webbla, or even one of the browser plugins if it modified a local store at all when you add the bookmark.

You could resort to cron, or run it manually once in a while, I suppose. It’s far handier to have it out of mind, though, and just have your bookmarks show up in OpenMeta and Spotlight searches within minutes of bookmarking them.

Tips

1. If you imported a ton of bookmarks from Delicious and haven’t spent a lot of time “weeding” them, you’ve probably got a lot of dead links that you’d be better off not downloading. Here’s a great solution: stale.py. It’s a Python script that you run locally, and it will traverse your entire collection of links and test them for error responses. You can run it in test mode first, and then turn on the delete mode to get rid of the dead ones. Instructions are at the bottom of the GitHub page.
2. Use descriptions and tags on things you want to make sure you can find. Clip some text out of the web page or write yourself a note in the description field. These notes are transferred by the script to your Spotlight Comments for the webloc file, making them instantly searchable, in addition to the convenience of tag search.
3. Don’t be shy about saving PDFs. If a page is a tutorial that you know you’ll need to reference, just go for it. They don’t take up much space, they can be annotated easily (seriously, have you tried Skim?), and they allow for full text search locally.
4. Use Choosy. With a local store of Spotlight-searchable bookmarks synced with Dropbox, and Choosy to determine what browser you open them in, you have cross-browser, cross-machine support for your entire bookmark collection.
5. The timestamp of the last check is stored in your user’s preference files using the defaults command. For the purposes of debugging, it’s sometimes useful to set that back a bit and force it to update. Just use getpinboard.rb -r to set it back 24 hours, or use a number after the –r to specify a number of days to revert.

Notes

The script is a one-way sync. All of the pieces are there to start working on a system that would update Pinboard with local deletions and insertions, but I just don’t have a need for it. At that point, I might as well make a dedicated application with an SQLite database to do all of this, and that gets way outside the scope of what I started here. Anyway, the only reason I’m not using Webbla for more of this is that it’s sandboxed from the rest of the system, and this script can work in tandem with Webbla to fix that.

Also, the local database created by this script is essentially a text dump, and it can be easily read into a Ruby script and manipulated. It serves as a good backup of all of your Pinboard info, and has some other possibilities as well. Here’s a quick demo script for outputting an HTML file, and a little tweaking could make it output a format that Safari, Firefox and Chrome can read to import bookmarks. All of the keys and values are there, so you can sift and sort any way you want. Lots of fun to be had, for the adventurous (or easily distracted).

Uninstalling

If you need to uninstall the script, remove it from whatever you’re using to schedule its activity, delete the script and locate two files:

• ~/getpinboard.yaml (in your User’s home folder)
• ~/Library/Preferences/com.brettterpstra.PinboardTagger.plist (in your User’s Library/Preferences folder)

Hope the script comes in useful for somebody, feel free to let me know if you have any trouble with it.

Download

There is a repo on GitHub, feel free to fork and send pull requests.

Related posts:

1. Delicious bookmarks and OpenMeta tags
2. Delicious, Spotlight and OpenMeta tags, revisited
3. Bookmarklet: Read Pinboard links with Instapaper Text

Originally posted on BrettTerpstra.com at Mirror your Pinboard bookmarks with OpenMeta tags

Since posting my latest desktop, I’ve received a few questions about how I keep my Mac’s Desktop icons under control. I thought I’d answer that question before I got around to posting my whole GeekTool setup. Beyond hiding disk icons and making the icons smaller in Finder’s Desktop view settings, I have a “system.” My filing system is simple to use, but a little complicated to explain (and set up), so it gets a whole post to itself. I doubt many will follow directly in my footsteps, but you asked…

Like any so-called system…

The basic idea is that my Desktop is an inbox. During the day, I save just about everything I work on to my Desktop. Like any system I’ve ever used, the secret is actually doing the “reviews”. At the end of each day I do a scan and file what obviously has a place (I have some neat tricks for speeding that part up, but I’ll get there in a bit). Things that represent new projects or that don’t fit into an existing folder/tag setup take further action; if there are enough related items to represent a project or topic, they get collected into a new Desktop folder which is descriptively named. The new folders will generally stay on the Desktop until the weekend, when I do a Weekly Review. If a file is singular and has no context, it stays on the Desktop and waits for company. If a file sits alone for a few days, it usually ends up being tossed into a “pile” of similar filetypes. Let Spotlight sort ‘em out.

In the Weekly Review, I go through the new folders on the Desktop and determine what type of information they represent. Is it a project? A client? Reference material? I have a folder hierarchy based on these types, the top levels being Work, Code, Reference and File Cabinet. Each breaks down a couple more levels. If a Desktop folder can be filed into one of these categories, it’s moved there. Not before it’s tagged, though.

Tags, tags, tags

I use OpenMeta tags to sort and search all of my data. I primarily use Tags from Case Apps, Default Folder X and Spotlight, but make frequent use of HoudahSpot and the command line openmeta utility as well (among many other OpenMeta-compatible apps). I don’t stake the life of my data on the continued availability or stability of the OpenMeta system, but in combination with a consistent, shallow folder hierarchy, I know where things are, even without the convenience of tags. Tags make it easy to group files and work with those groups, but there’s more than enough metadata in the filesystem to make any information easy to locate with Spotlight or other mdfind tools.

The beauty of tagging, as you may know, is that you can easily assign multiple categorizations and topics to each item, rather than just having them exist at one location which defines it as a type or part of a static collection. I still use the shallow hierarchy of folders that drill down to individual projects and topics, so it’s not a “one pile” deal. I couldn’t function like that; it’s difficult to weed, and if metadata is lost, so is the file, essentially. I use folders to maintain filesystem sanity. I use tags and other metadata to maintain my sanity.

I also use tags on iCal events, emails, web sites and other elements of my system. They’re easy to filter in searches and allow me to make a project “folder” that contains more types of data than Finder would normally make convenient.

Tags, to me, are like mind maps, in that they work the way my brain does. Each tag creates a “nodal connection” to other items sharing the tag, and neural networks are formed by related tags. Things aren’t always directly connected, but the nodes make it possible to track things down after you’ve forgotten about them. They’re useful in the short term for gathering projects together and categorizing, but even more useful after you’ve forgotten you ever had a file and then suddenly need it.

What’s in a (tag) name?

The secret, which I’ve learned from a few years of doing this, is to tag intelligently, sparingly and consistently. Being liberal with tags like “important” or “flagged” just means more upkeep and reviews that have to be done to keep the tags relevant. While tags like “neat” or “supercool” seem silly to me, I use one “inspiration” tag on anything that I’m saving just because it’s, well, inspirational. I have a Smart Folder in Finder that pulls up everything with that tag, and then I can filter the files based on date, type, etc. If I add additional tags, they are pretty general, such as “color,” “webdesign,” or “productivity.” These are big buckets I can dive into when I need an idea, or a kick in the pants. The tags on project and research files tend to be more specific, and I can drill down to precise sets of files by typing a few words in Tags or HoudahSpot.

To keep the cruft down, I use tags which aren’t already defined by other metadata like the creation or modified date, the filetype, the filename, etc. I do tag files with the name of their parent project or topic, even though it’s often redundant to the folder hierarchy. This allows me to tie projects together across multiple folders, disks or even servers. Sometimes a file is more strongly tied to a topic than it is to an individual project. Say it’s a general reference item or a stock photo that I gathered while working on a specific project, but that I know I’ll need again. I put it into a reference folder or photo collection, and tag it with the related project. It’s easy to view it in the context of the project with a tag search, but part of a larger collection of similar items at the same time.

When in doubt, I usually choose tags based on the first thought that pops into my head. Chances are, when I’m looking for it again in a year, I’ll have the same thought. It doesn’t always work, but I’ve had a pretty good success rate over the last couple of years. As often as possible I try to reuse tags; a tag that reveals one or two files is a useless tag. My “Yearly Review” includes weeding out tags that have very few associated files, or that could be merged into more common tags.

Is this future-proof?

I know I started by saying I don’t depend solely on tags, and then went on to describe a system that sounds pretty tag-heavy. Given the somewhat unorthodox nature of OpenMeta tags, and Apple’s power to pull the rug out from under them at any time, it kind of seems like a disaster waiting to happen, right? The fact of the matter is that OpenMeta tags are well preserved by design, backed up¹, and can be re-associated with their files through various means, should the system ever fail. It’s really pretty solid.

The tags don’t translate to iOS, though, so that’s taken some extra thought in some cases. The tags I use in Simplenote/Notational Velocity, for example, are inline in the file as a special character followed by the tag name. I can set up Spotlight searches to search for a keyword with and without that prefix, so it doesn’t really add complexity to the system. Searching in Simplenote on my iPhone/iPad is then much easier. Spotlight and apps like Simplenote also search text content and filenames, and iOS apps don’t “talk” to each other very well anyway, so it’s not a huge deal at this point.

The cool part

Tagging files with a project name has an added benefit for me. I’ve built a system using some Ruby scripts, the CLI tool for OpenMeta, the mdfind command, Dropzone, Hazel and Tags.app. This is the truly “complicated” part. The system took a while to set up, and there was some tweaking early on, but it’s solid now and I rarely have to think about it. It just works.

When I file those Desktop folders each week, I add a special supertag to define them as “targets.” They get a unique project name or combination of names, and the target tag. Then they’re filed in parent folders which are also targets with unique names. The second level of target can have one more child level of targets to separate sub-projects or file types. These tag names do not have to be unique system-wide, only within their parent target tag. Within any of those folders, the files are tagged and finding them rarely actually requires going into the folders.

Now, during the daily review, I just use Tags.app to quickly tag files on the Desktop with whatever is appropriate, plus the name of the project or topic they’re associated with, and any subfolders I want them to reach. Then I can let Hazel pick them up, or drag them straight to my Filer destination in Dropzone. Both run a script which gathers all of the target folders on the system and starts drilling down to find the best matching tag or series of tags. Then, it moves the tagged file into the best-match folder, logs the action to a text file, and gives me a quick Growl notification letting me know what it decided.

That’s the system. I’m sure I left some questions unanswered, so feel free to ask again in the comments. As I said in the beginning, I don’t know if this kind of setup makes sense to other Mac users, but it’s served me well for years. I don’t lose files, I keep my filesystem tidy, and I stay relatively sane.

Here’s a mind map of the elements that went into this article. It seemed to make a nice summary of my long-winded diatribe.

Related posts:

1. Some suggestions for better tagging
2. AutoTag2: smarter tagging for TextMate and WordPress
3. A Bash function for Markdown bloggers

Originally posted on BrettTerpstra.com at On sorting, tagging and other nerdery

Why would you want to do this?

First, it adds Spotlight search abilities to your Delicious bookmarks, and, if you use OpenMeta tags, it integrates a tag-based search into any OpenMeta application. The second half of that is, in my case, I bookmark in different ways for different purposes, and it’s nice to have everything integrate into a full search in the end. For me, that search is primarily tag-based, thus the OpenMeta tagging. When I want to share a link, I use Delicious (via Delibar), and when I just want to save it for my own reference, I use OpenMeta (via Tags.app). This script is part of a system which blends the two, removing the need for redundant tagging and bringing my Spotlight searches together.

Read on for the new script, and installation instructions.

At the top of the script, you’ll find a section that looks like this:

-- CONFIG
-- delicious username
property _user : "username"
-- delicious password
property _pass : "password"
-- POSIX path to *existing* folder, manually create the folder if needed or point to one that does exist
property targetFolder : "/Users/OSXUSERNAME/Library/Caches/Metadata/Delicious"
-- use Tags.app to apply OpenMeta tags based on Delicious tags
property useTagsApp : true
-- add thumbnails using http://hasseg.org/setWeblocThumb/
property useWebLocThumb : true
-- END CONFIG

You’ll need to set these properties to your own information, but after that the script should take care of itself. Username and password are your login info for Delicious.com. Obviously, it won’t work without those. The targetFolder property should be set to a folder that already exists or that you just created for the purpose. The script will not create a folder that doesn’t exist. I recommend creating one in ~/Library/Caches/Metadata/Delicious, and just substituting your user name in the targetFolder config.

The last two configuration options determine whether posts are tagged with Tags.app and/or thumbnailed with setWeblocThumb. I recommend doing both, but if you have other purposes for the script, feel free to set these to false. In order to use either option, you’ll need to have the appropriate application installed. Tags is a $29US application that makes tagging everything from URL’s to iPhoto pictures, as well as files and folders, a breeze. Worth every penny, in my opinion. setWeblocThumb is a free utility that captures thumbnail images of the bookmarked page and applies it to the webloc file. It makes for very nice presentation of search results and bookmark folders.

The last thing you’ll probably want to alter is the first line after the end of the config. It sets the ‘date from’ parameter default to 1 hour prior, in Central Standard Time. If you’re not stuck in the middle of the country, you’ll want to edit the –v+4H part to a + or — number more indicative of your own GMT offset. It’s not a huge deal, this current script will grab everything after this date on the first run, but will record the date of the last bookmark it finds and use that for the next run, and so on. If you want to get all of your Delicious bookmarks on the first run, set this to something long, long ago.

I’ve smartened the script up a bit from the previous version, and solved some of the errors that caused the bookmark save to fail on certain filenames. It also checks for existing bookmarks before writing new ones to disk, avoiding duplicates.

As with the last script, you can use Lingon to set up an hourly run for the script in launchd, or even just set it as an alarm in iCal and use that to schedule it, but I’ve found a better solution. I use Delibar for bookmarking, and it writes out to an xml file in ~/Library/Application Support/Delibar. I used Lingon to set up a launchd process that watches for changes in that file and runs the script. Seems to work really well, at least so far. However you choose to run it, it will keep storing the latest date at the time of retrieval, so you can even just run it once a day (or week, or month) and it will find just the newest bookmarks.

Without further ado, here’s the script in its entirety, commented for readability:

Updated: Thanks to Matteo from Shiny Frog for tipping me off to the fact that the API rate limiting I was running into was due to the lack of a user agent. This has been added to the curl call in the script, testing will be much smoother now!

#!/usr/bin/osascript
-- CONFIG
-- delicious username
property _user : "username"
-- delicious password
property _pass : "password"
-- POSIX path to *existing* folder, manually create the folder if needed or point to one that does exist
property targetFolder : "/Users/OSXUSERNAME/Library/Caches/Metadata/Delicious"
-- use Tags.app to apply OpenMeta tags based on Delicious tags
property useTagsApp : true
-- add thumbnails using http://hasseg.org/setWeblocThumb/
property useWebLocThumb : true
-- END CONFIG

set _date to do shell script "date -v+4H '+%Y-%m-%dT%H:%M:%SZ'"
-- returns a date 1 hour ago (CST) in the format required by the twitter API.
-- you can adjust the date based on your time zone and the frequency that you run
-- the script by editing the -v+4H part of the date command (e.g. -v-1H for GMT - 1 hour)
-- Note that the Delicious dates run on GMT, so you have to account for your time offset
-- For me (CST), this means -v+4H if I want to go back one hour
try
    set _fromdt to do shell script "defaults read com.brettterpstra.getdelicious fromdt"
    -- if the last update time is stored in defaults, use it
on error
    set _fromdt to _date
    -- otherwise, use the date we just set
end try

set bookmarks to do shell script "curl -A \"OpenMetaTagger/1.1\" https://" & _user & ":" & _pass & "@api.del.icio.us/v1/posts/all?fromdt=" & _fromdt

-- gets the bookmarks starting with the date specified in _date
set _folder to POSIX file targetFolder as alias
-- turns the POSIX path in CONFIG into an AppleScript alias
set _output to ""

tell application "System Events"
    set xmlDocument to make new XML data with data bookmarks
    -- set xmlDocument to contents of XML file "Users:ttscoff:Desktop:deliciousbookmarks.xml"
    -- create the XML object
    set {astid, AppleScript's text item delimiters} to {AppleScript's text item delimiters, " "}
    -- store the current delimiter and set the new one to a space for breaking up the tags
    repeat with _post in XML elements of XML element 1 of xmlDocument
        set _url to value of XML attribute "href" of _post as string
        set _tags to text items of (value of XML attribute "tag" of _post as string)
        set end of _tags to "delicious"
        set _name to value of XML attribute "description" of _post as string
        set _title to do shell script "echo \"" & _name & "\"" & "|tr -cs /[^A-Za-z]/ ' '"
        -- strips some bad characters from the page title, solves some errors
        set _date to value of XML attribute "time" of _post as string
        try
            tell application "Finder"
                set AppleScript's text item delimiters to ""
                if not exists alias ((_folder & _title as string) & ".webloc") then
                    set webloc to make new internet location file at _folder to _url with properties {name:_title}
                    -- makes a .webloc file in our target folder
                    set _output to _output & "Grabbed " & _title
                    set AppleScript's text item delimiters to " "
                    if useTagsApp then
                        tell application "Tags" to apply tags _tags to files {POSIX path of (webloc as string)}
                        -- adds OpenMeta tags to our newly created file
                        set _output to _output & " and tagged with " & _tags as string
                    end if
                    if useWebLocThumb then
                        do shell script "/usr/local/bin/setWeblocThumb " & quoted form of (POSIX path of (webloc as string))
                        -- use setWeblocThumb to add a thumbnail to our webloc file
                        -- http://hasseg.org/setWeblocThumb/
                        set _output to _output & ", applied thumbnail"
                    end if
                    set _output to _output & "." & return
                end if
            end tell
        on error myError
            set _output to _output & "Error " & myError & return
        end try
        set {_url,_title,_tags,_name,_time} to {null,null,null,null,null}
    end repeat
    set AppleScript's text item delimiters to astid
    -- restore the delimiters
end tell
do shell script "defaults write com.brettterpstra.getdelicious fromdt " & _date
-- store the date of the last bookmark in system defaults
return _output

Open this script in your Script Editor

Related posts:

1. Delicious bookmarks and OpenMeta tags
2. Mirror your Pinboard bookmarks with OpenMeta tags
3. Save Safari tabs to Instapaper

Originally posted on BrettTerpstra.com at Delicious, Spotlight and OpenMeta tags, revisited

Unfortunately, one of my favorite apps right now, Delibar, doesn’t integrate with Spotlight or OpenMeta, despite the fact that it would be relatively easy to do. Delibar is an excellent (and sexy) menubar application for creating and searching Delicious (and Pinboard) bookmarks. A similar app, Pukka, pulls off the Spotlight part, but I still wanted OpenMeta tags that matched my Delicious tags for local searching.

I also use Tags.app for most of my tagging. OpenMeta tags allow you to group and classify files with simple tags, and provide a Spotlight-compatible way to search more intelligently. When you tag a website with Tags, it creates a webloc file in your metadata cache and applies the OpenMeta tags to it. I figured a similar approach would be feasible using AppleScript, and it seems to be working out great. Read on to see the script I’m using, and be sure to let me know if you improve on it!

What I’m doing is using curl with the Delicious API, and downloading everything that’s been bookmarked within the last hour. I’m running it with launchd every hour, so I don’t want to grab more than that every time it runs. It parses out the title, tags and url for the bookmark, saves it to a .webloc file, and tags it using Tags.app’s AppleScript commands. If you’re not running Tags, it could be modified pretty easily to work with the openmeta command line application.

I’m leaving the target folder up to the user, so when you define it in the CONFIG section, make sure it’s pointing to an existing folder. The only other config properties are your Delicious username and password (sent securely). As noted in the comments in the script, you can adjust the time/date it retrieves from in the first line after the CONFIG section.

The script is pretty well commented, so I won’t rewrite the whole thing here. Take a look, and then I’ll show you how I installed it.

-- CONFIG
property _user : "username" -- delicious username
property _pass : "password" -- delicious password
property targetFolder : "/Users/ttscoff/Library/Caches/Metadata/Delicious" -- POSIX path to existing folder
-- END CONFIG

set _date to do shell script "date -v-1d '+%Y-%m-%dT%H:%M:%SZ'"
-- returns a date 1 day ago in the format required by the twitter API.
-- you can adjust the date based on the frequency that you run the script
-- by editing the -v-1d part of the date command (-v-1H for 1 hour)
-- Note that the Delicious dates run on GMT, so you have to account for your time offset
-- For me, this means -v+4H if I want to go back one hour

set bookmarks to do shell script "curl https://" & _user & ":" & _pass & "@api.del.icio.us/v1/posts/all?fromdt=" & _date
-- gets the bookmarks starting with the date specified in _date

set _folder to POSIX file targetFolder as alias
-- turns the POSIX path in CONFIG into an AppleScript alias

tell application "System Events"
    set xmlDocument to make new XML data with data bookmarks
    -- create the XML object
    set {astid, AppleScript's text item delimiters} to {AppleScript's text item delimiters, " "}
    -- store the current delimiter and set the new one to a space for breaking up the tags
    repeat with _post in XML elements of XML element 1 of xmlDocument
        set _url to value of XML attribute "href" of _post as string
        set _tags to text items of (value of XML attribute "tag" of _post as string)
        set end of _tags to "delicious"
        set _title to value of XML attribute "description" of _post as string
        try -- it fails on some files, the try block lets us keep moving if we hit a problem
            tell application "Finder" to set webloc to make new internet location file to _url at _folder with properties {name:_title}
            -- makes a .webloc file in our target folder
            tell application "Tags" to apply tags _tags to files {POSIX path of (webloc as string)}
            -- adds OpenMeta tags to our newly created file
        end try
    end repeat
    set AppleScript's text item delimiters to astid
    -- restore the delimiters
end tell

Open this script in your Script Editor

Installing

The first thing I did was make a version of the script that excluded the date parameter (fromdt) in the curl call in order to download my entire history and process it so I was up-to-date. This took a while, but didn’t seem to cause any problems. The Delicious API is pretty quick to throttle you, so download the whole bunch in one API call or you’ll be in trouble.

After editing the config options, I saved the file as an Application Bundle (see addendum below) in my own apps directory (~/Applications). It doesn’t matter where you put it (or what you name it), as long as you know the path to get there. Then I set up a launchd plist to run it every 3600 seconds (hour). I highly recommend Lingon for editing all things launchd. It will make sense when you get there.

Addendum

In reference to the install procedure above, I’ve actually had more luck saving it as a text file to a scripts folder and running it with osascript through launchd. Here’s my launchd plist, which you can edit and load through the “Expert” section of Lingon’s editor.

Also, as I just added into the code comments above, I didn’t originally account for the fact that Delicious dates are in GMT, and my offset is –5 right now, so I have to use –v+4H to get 1 hour back, not –v-1H.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.brettterpstra.GetDelicious</string>
    <key>Nice</key>
    <integer>8</integer>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/osascript</string>
        <string>/Users/ttscoff/scripts/getdelicious</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>StartInterval</key>
    <integer>3600</integer>
</dict>
</plist>

Areas for improvement:

• Currently errors out on some bookmarks for unknown reasons
• Could have a first-run setting to automatically download and tag ALL bookmarks
• Could set the created date of the webloc file based on the date of the bookmark from Delicious
• Could provide more feedback or logging

Related posts:

1. Delicious, Spotlight and OpenMeta tags, revisited
2. Mirror your Pinboard bookmarks with OpenMeta tags
3. Natural Language Date Service update

Originally posted on BrettTerpstra.com at Delicious bookmarks and OpenMeta tags