Welcome to the lab.

Gather, Shortcuts, Bookmarklets, and Paywalled Content

Gather is constantly improving at this point, and a lot of its rough edges have already been polished. I’m loving all the feedback I’m getting, both for fixing bugs and for adding new features. It’s hard to test something like Gather on every possible permutation of a web layout, so I appreciate hearing about edge cases (even if I do have to weigh how much effort they’re worth).

I’d recommend upgrading to the latest version before trying anything in this post. As of this writing, that’s v2.0.33. If you installed via Homebrew, you’re just a brew upgrade gather-cli away. Otherwise, please download the PKG file and update.

If you have Homebrew installed but ran into errors trying to install gather-cli, try running sudo xcode-select -s /Applications/Xcode.app/Contents/Developer. It fixes 90% of the issues I’ve seen.

Pesky Paywalls

So anyway, one common thing that people want to clip from the web is paywalled content, which makes sense as it’s harder to search and more convenient to have in your own personal, local notes. But Gather can’t access a web page that isn’t public. You can select all the text, copy it, then run Gather with gather --paste --html to convert it, but it’s never perfect. The better solution is bookmarklets.

Gather to nvUltra

I got a request to add a --title-only option to Gather. Makes sense; when you’re using it to save Markdown to notes, you’d want the title of the page to be the title of the note. You could parse that out of the Markdown version in several ways, but in some cases it might be easier to just call gather --title-only URL and get an undecorated string back. So that’s been added in v2.0.29.

That got me to thinking, though… I clip all my notes to nvUltra. I’ve been trying to convince Fletcher that we should build a web clipper into nvUltra, but until then I’ve been using Gather to create Markdown notes out of web pages I want to reference. So, for purely selfish reasons, I’ve added a couple of commands to Gather that will markdownify a page and create a titled note all in one fell swoop (using nvUltra’s URL handler). I even added it to a Shortcut that I can trigger with LaunchBar. Now I’m a few keystrokes away from turning any page into a note in nvUltra. See the notes on the project page if you happen to be on the nvUltra beta and would like to try this. I’ll even throw in a link to the Shortcut.

Since it’s a compatible URL handler, I also added options for creating Notational Velocity/nvALT links/notes. If this sounds great but you’re not into nvALT/nvUltra, let me know if there’s a note platform you use with a url handler that I could also incorporate. I’m open to making your life simpler.

By the way, if you’re not using Gather via Homebrew, you really should be. Imagine how easy your life would be right now if all you had to do was run brew upgrade gather-cli instead of locating, downloading, and clicking all the install screens for the update…

Gather on Homebrew

I considered calling this post “Gather round for a brew” but I says to myself, I says “dammit, Brett, you’re better than that.”

I knew the best way to distribute my new Gather tool was going to be via Homebrew. No worries about notarization, installing to the wrong folder, or requiring finicky steps from users. But the Homebrew documentation is… intense. I got a little scared at first, but then I found an NSHipster tutorial that made the process super simple. Like five, ten minutes max, plus another five to add it to the build automation for updates. So, as of today, Gather is available for installation via brew.

To install via Homebrew, just run two commands:

brew tap ttscoff/thelab
brew install gather-cli

Done. You’ll have a working copy of gather that you can keep updated using Homebrew. It’s a way better solution than the Package installers, though that option will still be available for those who aren’t brew-savvy.

To learn more about Gather, visit the project page. Oh, and let me know if my brilliant solution doesn’t work for some stupid reason. It’s been one of those weeks.

PopClip WebMarkdown fix and other codesigning adventures

So the first project I worked on with the libraries that went into Gather was the WebMarkdown extension for PopClip. It worked great… for me. I suffered from the all-too-common hacker mistake summed up as “well, it works on my machine.”

I codesigned the binary embedded in the extension, but when you download a file from a website, it adds a quarantine bit that requires what Apple calls “notarization,” and you can’t notarize a single binary (or a zipped package like a PopClip extension). You have to distribute it as a notarized package file and installer, which is what I’m doing with Gather itself now.

A .pkg isn’t really a good option for a PopClip extension, though, so I found another workaround. Without the binary the extension should install without error just by double clicking. When you select some text and click WebMD, it looks for the gather CLI in /usr/local/bin on your machine. If it doesn’t find it there, it offers to download and run the Gather installer for you. You can just hit the OK buttons, and when it’s done, the new, improved WebMarkdown extension should work for you.

Please give it a shot!

Brett's PopClip Extensions v1.41

A few PopClip extensions for Markdown writing and other useful tools

Published 11/30/14.

Updated 09/02/22. Changelog

DonateMore info…

Read2Text is now Gather, your new Markdownifier

As promised, I’ve updated Read2Text as a standalone binary using the Swift versions of Readability and HTML2Text. In the process I made it a whole new tool called Gather. You may recall that name from a little utility I made a while back. This does the exact same thing, just without the GUI, and I liked that name enough to revive it.

I designed it to be flexible and easy to drop into any kind of workflow, so you can use it in Services and Shortcuts, PopClip extensions, LaunchBar Actions, Alfred, Raycast, whatever floats your boat. It can take URLs, raw HTML, even rich text from web page copies. It can arguments, piped input via STDIN, and even pull urls and HTML data out of environment variables. Whatever you want to do with it, it should be flexible enough to handle.

And it does a better, more consistent job than Marky (my API-based markdownifier) ever did.

What Is This?

If you’ve never seen this tool before, it allows you to turn any URL into Markdown text with comments, ads, etc. stripped out, and most Markdown-compatible elements properly converted. To use it, just run gather https://yoururl.com on the command line. It will output the result to STDOUT, so you can add |pbcopy to the end of it to clip it directly to the clipboard. The tool has a bunch more options that I’ve detailed in the README.

And yes, I know PopClip and some other tools have some HTML-to-Markdown conversion built in already. What sets Gather apart is its ability to locate the important content of a web page, its handling of some advanced things like tables and definition lists, advanced handling of highlighted code blocks, and other products of my weekend obsessions.

PopClip extensions: WebMarkdown fix

So Apple stopped shipping Python with macOS, which they warned quite a while ago they were going to do. Unfortunately, I still have a good number of apps and scripts that rely on Python scripts in the background, meaning people who haven’t installed the Command Line Tools are unable to use some of my tools. I’ve slowly been working to rectify this, but it’s going to take some time.

The Python script I’m most reliant on is Aaron Swartz’ html2text, used for converting HTML markup to Markdown text. Read2Text uses it, the PopClip extension WebMarkdown uses it, even Marked uses it for rendering Scrivener documents. So my first fix is for the WebMD extension. If it works for everyone, I’ll know I’m on the right track to solve this. (Then I have to port some Ruby scripts, but one thing at a time.)

The WebMarkdown extension lets you select text on a web page, and then select “WebMD” in PopClip to have a Markdown version of the selected text placed in your clipboard.

If you’re interested in the nitty gritty, coming to my rescue is Shahaf Levi and his Swift port of the html2text and Readability scripts. With those in hand I’m able to compile command line utilities as binaries that can be drop-in replacements for the various incarnations of the html2text script I’ve used. I made a very simple one for WebMarkdown that takes raw HTML from the POPCLIP_HTML environment variable and converts it to Markdown. No Readability, no STDIN/argument input, just read the variable and output the result.

Please help me test. Download below, unzip, and locate the WebMarkdown extension in Finder. Double click it to install. If you already had the previous version installed, it will be updated.

If you haven’t tried it or it wasn’t working for you previously, give it a shot and let me know how it goes. If people run into issues, I’ll update this post with fixes.

I’ll also be publishing a replacement for Read2Text very soon.

Brett's PopClip Extensions v1.41

A few PopClip extensions for Markdown writing and other useful tools

Published 11/30/14.

Updated 09/02/22. Changelog

DonateMore info…

Hook and SearchLink

In case you’ve missed it previously, SearchLink is my macOS Service for doing web searches and inserting Markdown links in your writing without ever switching to your browser. I use it daily, and it’s been solid enough that it hasn’t changed much over the last year.

This morning I added Hook integration to SearchLink. I borrowed some code from the Hook command line tool I created and added a new !hook search type to SearchLink. Now you can type something like [Bunch Notes](!hook) and get back a link to a local file, email, url handler, or web address.

[Hook Notes](!hook)

=> [Hook Notes](x-nvultra://open?notebook=/Users/ttscoff/Library/Mobile%20Documents/9CR7T2DMDG~com~ngocluu~onewriter/Documents/nvALT2.2&note=Hook%20notes.md)

Combined with the existing Pinboard search, the Hook search makes it easy to pull up anything you’ve ever bookmarked using either of those tools. Unlike the Pinboard search, which requires an API key, no additional setup is required for searching Hook. As long as you have Hook installed, !hook searches will work. Hopefully you’ll find it useful!

Download the latest version below or visit the SearchLink project page for more info.

SearchLink v2.2.23

Generate Markdown links from web searches without leaving your editor.

Published 11/10/14.

Updated 09/19/22. Changelog

DonateMore info…