
Delicious, Spotlight and OpenMeta tags, revisited
I’ve taken a little time to improve the script I’d posted previously. It saves bookmarks you add to Delicious as webloc (Safari bookmark) files on your local hard drive, optionally with thumbnails and OpenMeta tags.
Why would you want to do this?
First, it adds Spotlight search abilities to your Delicious bookmarks, and, if you use OpenMeta tags, it integrates a tag-based search into any OpenMeta application. The second half of that is, in my case, I bookmark in different ways for different purposes, and it’s nice to have everything integrate into a full search in the end. For me, that search is primarily tag-based, thus the OpenMeta tagging. When I want to share a link, I use Delicious (via Delibar), and when I just want to save it for my own reference, I use OpenMeta (via Tags.app). This script is part of a system which blends the two, removing the need for redundant tagging and bringing my Spotlight searches together.
Read on for the new script, and installation instructions.
At the top of the script, you’ll find a section that looks like this:
-- CONFIG
-- delicious username
property _user : "username"
-- delicious password
property _pass : "password"
-- POSIX path to *existing* folder, manually create the folder if needed or point to one that does exist
property targetFolder : "/Users/OSXUSERNAME/Library/Caches/Metadata/Delicious"
-- use Tags.app to apply OpenMeta tags based on Delicious tags
property useTagsApp : true
-- add thumbnails using http://hasseg.org/setWeblocThumb/
property useWebLocThumb : true
-- END CONFIG
You’ll need to set these properties to your own information, but after that the script should take care of itself. Username and password are your login info for Delicious.com. Obviously, it won’t work without those. The targetFolder property should be set to a folder that already exists or that you just created for the purpose. The script will not create a folder that doesn’t exist. I recommend creating one in ~/Library/Caches/Metadata/Delicious, and just substituting your user name in the targetFolder config.
The last two configuration options determine whether posts are tagged with Tags.app and/or thumbnailed with setWeblocThumb. I recommend doing both, but if you have other purposes for the script, feel free to set these to false. In order to use either option, you’ll need to have the appropriate application installed. Tags is a $29US application that makes tagging everything from URL’s to iPhoto pictures, as well as files and folders, a breeze. Worth every penny, in my opinion. setWeblocThumb is a free utility that captures thumbnail images of the bookmarked page and applies it to the webloc file. It makes for very nice presentation of search results and bookmark folders.
The last thing you’ll probably want to alter is the first line after the end of the config. It sets the ‘date from’ parameter default to 1 hour prior, in Central Standard Time. If you’re not stuck in the middle of the country, you’ll want to edit the –v+4H part to a + or — number more indicative of your own GMT offset. It’s not a huge deal, this current script will grab everything after this date on the first run, but will record the date of the last bookmark it finds and use that for the next run, and so on. If you want to get all of your Delicious bookmarks on the first run, set this to something long, long ago.
I’ve smartened the script up a bit from the previous version, and solved some of the errors that caused the bookmark save to fail on certain filenames. It also checks for existing bookmarks before writing new ones to disk, avoiding duplicates.
As with the last script, you can use Lingon to set up an hourly run for the script in launchd, or even just set it as an alarm in iCal and use that to schedule it, but I’ve found a better solution. I use Delibar for bookmarking, and it writes out to an xml file in ~/Library/Application Support/Delibar. I used Lingon to set up a launchd process that watches for changes in that file and runs the script. Seems to work really well, at least so far. However you choose to run it, it will keep storing the latest date at the time of retrieval, so you can even just run it once a day (or week, or month) and it will find just the newest bookmarks.
Without further ado, here’s the script in its entirety, commented for readability:
Updated: Thanks to Matteo from Shiny Frog for tipping me off to the fact that the API rate limiting I was running into was due to the lack of a user agent. This has been added to the curl call in the script, testing will be much smoother now!
#!/usr/bin/osascript
-- CONFIG
-- delicious username
property _user : "username"
-- delicious password
property _pass : "password"
-- POSIX path to *existing* folder, manually create the folder if needed or point to one that does exist
property targetFolder : "/Users/OSXUSERNAME/Library/Caches/Metadata/Delicious"
-- use Tags.app to apply OpenMeta tags based on Delicious tags
property useTagsApp : true
-- add thumbnails using http://hasseg.org/setWeblocThumb/
property useWebLocThumb : true
-- END CONFIG
set _date to do shell script "date -v+4H '+%Y-%m-%dT%H:%M:%SZ'"
-- returns a date 1 hour ago (CST) in the format required by the twitter API.
-- you can adjust the date based on your time zone and the frequency that you run
-- the script by editing the -v+4H part of the date command (e.g. -v-1H for GMT - 1 hour)
-- Note that the Delicious dates run on GMT, so you have to account for your time offset
-- For me (CST), this means -v+4H if I want to go back one hour
try
set _fromdt to do shell script "defaults read com.brettterpstra.getdelicious fromdt"
-- if the last update time is stored in defaults, use it
on error
set _fromdt to _date
-- otherwise, use the date we just set
end try
set bookmarks to do shell script "curl -A \"OpenMetaTagger/1.1\" https://" & _user & ":" & _pass & "@api.del.icio.us/v1/posts/all?fromdt=" & _fromdt
-- gets the bookmarks starting with the date specified in _date
set _folder to POSIX file targetFolder as alias
-- turns the POSIX path in CONFIG into an AppleScript alias
set _output to ""
tell application "System Events"
set xmlDocument to make new XML data with data bookmarks
-- set xmlDocument to contents of XML file "Users:ttscoff:Desktop:deliciousbookmarks.xml"
-- create the XML object
set {astid, AppleScript's text item delimiters} to {AppleScript's text item delimiters, " "}
-- store the current delimiter and set the new one to a space for breaking up the tags
repeat with _post in XML elements of XML element 1 of xmlDocument
set _url to value of XML attribute "href" of _post as string
set _tags to text items of (value of XML attribute "tag" of _post as string)
set end of _tags to "delicious"
set _name to value of XML attribute "description" of _post as string
set _title to do shell script "echo \"" & _name & "\"" & "|tr -cs /[^A-Za-z]/ ' '"
-- strips some bad characters from the page title, solves some errors
set _date to value of XML attribute "time" of _post as string
try
tell application "Finder"
set AppleScript's text item delimiters to ""
if not exists alias ((_folder & _title as string) & ".webloc") then
set webloc to make new internet location file at _folder to _url with properties {name:_title}
-- makes a .webloc file in our target folder
set _output to _output & "Grabbed " & _title
set AppleScript's text item delimiters to " "
if useTagsApp then
tell application "Tags" to apply tags _tags to files {POSIX path of (webloc as string)}
-- adds OpenMeta tags to our newly created file
set _output to _output & " and tagged with " & _tags as string
end if
if useWebLocThumb then
do shell script "/usr/local/bin/setWeblocThumb " & quoted form of (POSIX path of (webloc as string))
-- use setWeblocThumb to add a thumbnail to our webloc file
-- http://hasseg.org/setWeblocThumb/
set _output to _output & ", applied thumbnail"
end if
set _output to _output & "." & return
end if
end tell
on error myError
set _output to _output & "Error " & myError & return
end try
set {_url,_title,_tags,_name,_time} to {null,null,null,null,null}
end repeat
set AppleScript's text item delimiters to astid
-- restore the delimiters
end tell
do shell script "defaults write com.brettterpstra.getdelicious fromdt " & _date
-- store the date of the last bookmark in system defaults
return _output

Hi Everything worked out fine except that extended characters in the bookmark title did not survive the download. It worked with all tags. For example: the text väder < vder, the tag väder < väder
Ah… I have a regular expression in there that just strips out all non-alphanumeric characters to avoid problems, but it’s not unicode-aware. I’ll see if I can modify that easily to handle some more interesting characters :).
I too, look for elegant solutions to complex problems.
the problem: using spotlight to find tagged files found in afp shared servers.
I have been up all night and I have to tell you. I’am beat.
So… I’am looking for the ‘elegant solution’
(btw I have tried all flavors of mdutil, mdimport — mounting drives all kinds of ways, etc…)
ideas… I thinking about writing an app to search tags in afp folders
Aloha p
This looks awesome. I’d like to adapt it for Pinboard, but I’m a bit shy about modding scripts; any clues about how to do that most efficiently would be welcome ;)
It looks like the Pinboard API is identical to Delicious, so you can just replace “api.del.icio.us/” in the line:
with “api.pinboard.in/” and adjust your username/password accordingly. Haven’t tested it, but this page indicates that should work: http://pinboard.in/howto/#api
Follow-up to the above response: the Pinboard API doesn’t honor the fromdt parameter, so this script doesn’t work very well when flipped over to that endpoint. I’ll try to put together an alternative soon, though the Pinboard devs promised they’d add the parameter. We’ll see which happens first :).
This is exactly what I wanted. Thanks for posting this Brett.
I had a few issues though so I modified it quite a bit. Mainly issues with non-roman characters. It lets through more characters only concerning itself with colons, slashes and leading dots and it cleans up the spacing all done with sed. Did a lot of Googling since I’m not an AppleScript/shell expert.
Other improvements/changes:
Creates a log file into ~/Library/Logs with the option to show general messages and/or errors.
Uses openmeta from the command line instead of Tags. I own a copy of Tags but I preferred this approach. Tags will pickup whatever you set through the shell environment.
When setWeblocThumb retrieves a thumbnail it will flag the Finder when the server can’t be reached (dead links) by setting a color label index of 7 which is Gray by default.
Option to prefix the tags when carried over from Delicious. I like to keep them separated from my local tags so I set in this option. Defaults to “delicious:”.
Saves the notes from Delicious bookmarks into the Finder comments for the .webloc file. Can be switched on/off.
Fixes how the date of last scan is stored. It think there was a bug in there.
Zip file containing an Automator .workflow file & .applescript. Hope you find it useful.
In Snow Leopard there is a major issue with extended attributes. User space extended attributes are wiped out on save with standart Cocoa applications like TextEdit, Preview etc. OpenMeta apprently circumvent this by copying back the information from Apple domain space to its own user space.
The situation is pretty sad. I had a couple of tools adding metadata to PDF and text files. But these are delete whenever the file is edited with a Cocoa application.
So beware, xattr are broken on 10.6.
Hi Brett,
I’ve been running your script once a week for a couple months now, but for whatever, the last week’s of bookmarks were created (webLocs) but the tags weren’t applied. I’m not sure if this is a result of the most recent Mac update (which has broken a couple other, totally unrelated things on my system).
Anyone else encountering this issue?
Grant
I made a full switch to Pinboard recently. I had to stop running this script because Pinboard doesn’t honor the fromdt parameter (so it sends everything every time) and I haven’t had a chance to add a full XML parser to the script yet. So, long story short, I don’t know. I’ll do some testing and post back here if I find anything similar.
Very cool script. It solves an ongoing problem that has been bugging me for some time. I tried to run the script after editing my user name, password and creating the target folder directory but keep coming up with the following error
error “curl: (3) [globbing] error: bad range specification after pos 27 ” number 3
I am not a programmer and am not sure what to make of the error.
Thanks
I’m afraid I have no idea, offhand, what that error would be caused by. It may be failing on some strange characters… unfortunately, I’ve stopped directly supporting this script and have moved on to the (far more advanced) Pinboard version. I may make that one backwards-compatible with Delicious eventually.