Triggering tasks remotely with notifyutil and launchd

Let me start this by saying that I’m fully aware that there are multiple ways to accomplish this feat, and that my choice may not have been the perfect one. It happens to work around some frustrations I found with other solutions, but I kind of got on a kick of really wanting to make this work. It works (though it killed my Sunday). It’s pretty complex to describe, though, so I’ll probably end up leaving some holes in the story. If something doesn’t make sense, point it out in the comments and I’ll fill in the blanks.

The final summary, early

Here’s the summary of the system I’m using to build my Jekyll blog by having scripts in protected shells set up the necessary environment and run Rake tasks from them.

• A launchd job for each trigger
• A notifyutil key for each trigger
• each job calls the same script with a different argument
  • The first instance of the script that opens creates a shared id.running key for determining if a job is running (0 = unset, 1 = no job running, 2 = job running)
  • The script argument sets a unique notification key for the instance and assigns a set of tasks for that trigger
  • The script launches notifyutil -1 with the unique key, causing it to wait for a notification to be posted to that key before continuing
  • When that key is posted to, notifyutil -1 yields and the script continues:
    • If the running.id state is 2, it pauses and polls for the current job to finish (id.running = 1)
    • It sets the id.running key to 2 to indicate that its own job has started
    • Performs the tasks for that key
    • Remove the id.running notification
    • Loop returns to waiting for a signal

A few details

launchd (man page)

A system built into OS X that handles scheduled, constant and triggered tasks for the system. It’s easy to add your own tasks and have run in the background as any user.

notifyd/notifyutil (man page)

The notification system in OS X is an easy way to post “messages” between shells and across users. It registers “keys” such as foo.bar and allows you to listen to them, post values to them and query their value.

notifyutil basics:

You can name your keys whatever you like, you just have to know the name of it to call it later. For my Jekyll listeners I’ve been using a convention like “jekyll.deploy” and “jekyll.generate”. Easy to remember.

Start a listener for a key with notifyutil -1 key.name where the -1 can be any number and it will wait for that number of posts before yielding.

An active key can have a state set using notifyutil -s key.name 123, where 123 is the numeric value to assign. I use these to mark whether jobs are running or idle, with 0 being “inactive,” 1 being “listening,” and 2 being “running.” You can query the value of the key with notifyutil -g key.name.

Why in the…

I wanted to be able to control more aspects of my computer remotely than standard methods would allow. I pulled a few hairs out trying to solve various problems, and this is what it came down to.

The reason I wanted to do this with launchd is because it provides the ability to run scripts as any user (including root), manages keeping the listeners alive automatically, and runs invisibly and without additional hooks and setup requirements. notifyd simply provided me with the easiest way to communicate with these processes.

With tools like LaunchControl, setting up launchd jobs and confirming their status is really simple. Yes, I’ve come to prefer LaunchControl over (even the latest incarnation) of Lingon. LaunchControl can set up jobs as root (Lingon X can, too, now), quickly load and unload agents, edit every known key with a helpful wizard (with expert mode for adding your own) and — my favorite feature — shows you at a glance what’s loaded, what’s running and what has errors (with descriptive info).

The launchd and notifyutil combination isn’t perfect, at least not in my current implementation. Using the methods I currently have set up, I can’t pass anything to the daemon; it just keeps the listener scripts running in the background to handle the notifications. If I expand on this idea, the first thing to do is hook up sockets that I can read and write data to so that I can control everything from one central script and pass arguments through the launchd agent.

I’ve been able to automate the generation and loading of the plist files it uses, so adding jobs is a bit faster now. It’s still convoluted. If I end up wanting more than a few triggers, I’m going to have to centralize everything better.

Triggering

The jobs are now triggered by anything that can run a notifyutil command from any user on the system. Say I have a script that is listening to brett.whatnot, I’d just call notifyutil -p brett.whatnot from any tool that can run a shell command.

SSH/shell

Run notifyutil (or an alias) from any script, via local or remote SSH sessions. Yes, in this case you could also use screen, nohup and/or at to perform the tasks, so it’s not the most useful example.

BetterTouchTool/EventScripts

AppleScript actions called from BetterTouchTool Remote can simply run do shell script "notifyutil -p [key]" to trigger any action controlled by the launchd agents. This means that even though those scripts run in a protected shell as the login user, they don’t need any special permissions to trigger scripts that load in their own shell and perform actions you wouldn’t otherwise be able to from the triggering application.

Web interface

Running a Internet-accessible website securely more or less precludes the ability to access the system in the ways most of my scripts would need to. Actions that I can trigger via PHP or Ruby exec calls are run as the _www user and there are endless headaches involved with trying to load an environment and run scripts outside of the web folder. This frustration was the impetus for the whole project, and now I have a system that I can quickly build web interfaces for while still maintaining a modicum of security¹.

REST API

You can also build a simple API for the same setup as above, which lets you trigger scripts from applications other than a web browser. Even a quick curl command from a remote machine can set things in motion.

Cocoa applications

You can do this from compiled languages like C and Objective-C, too, of course. For now I’m just using notifyutil, but the notifyd system could have some fun possibilities for simple automation from applications, too.

How I’m using it

I built a remote trigger system for builds and deploys of my Jekyll blog. It currently runs on my home machine (with a static IP and externally-accessible ports set up). It could easily be moved to the server side of my blog, too (which runs on a Mac mini), if I change my setup and require that.

YouTube Video

As stated above, the system uses launchd to keep four instances of a script running in the background (sample), each with different parameters: preview², generate, deploy and a combination generate/deploy task. Each one builds a unique notifyd listener key based on its name and begins waiting for posts to that key. When a given notification key is posted, the associated instance of the script runs the necessary Rake tasks.

I have AppleScripts set up in BTT Remote for each task, and each script is just one do shell script line that runs a notifyutil -p jekyll.task command. I also have a full web interface that uses a combination of front and back-end scripting to provide feedback and prevent things like job overlaps and accidentally deploying a preview site.

In addition to the feedback provided by the web interface, I added the Pushover API to the Rakefile, and using my beengone utility I can push notifications to my iOS devices when a task finishes and the computer has been idle for more than 5 minutes. That way, no matter where I trigger from, I can get a progress report/confirmation on my phone³.

It’s complicated.

This was partly necessary to reach my goal, and partly just mad science gone awry. The end result is stable and perfectly suited to me, but I’m pretty sure it would be neither if I handed it as-is to anyone else. Hopefully this proof-of-concept is inspirational to others in some fashion.

I’m not sharing any of the code for this except by specific requests from people who have most of the system in hand first. If this is something you actually want to implement, have a go at it and let me know if there are specific hanging points that I might have already solved and forgotten to mention.