I’ll admit I’m a bit perturbed by the disappearance of all of the decent web search APIs (Yahoo, Google, Live.net, Bing went paid…). Google Ajax Search is still active, I think, but I don’t like working with it from the desktop. So I’m back to scraping, though this is some pretty light scraping.
As a result of constantly disappearing APIs, my auto-linking tools keep breaking. So I built yet another, for however long it will last.
I’m using the backslash feature on DuckDuckGo to do this. That feature lets you start any query with a “" and will automatically redirect to the first result. So I’m taking the redirect page in as a variable and scraping it for the target link.
I’ve updated the Markdown Service Tools “Auto-link Web Search” Service to use this. This service lets you select text, search the web and replace it with a Markdown link to the first result. You can download the updated version on the project page.
Here’s the code. It’s as simple as can be, and you’re free to modify it and change the output format (or whatever you like):
#!/usr/bin/ruby
require 'net/http'
def e_url(string)
string.gsub(/([^a-zA-Z0-9_.-]+)/n) do
'%' + $1.unpack('H2' * $1.size).join('%').upcase
end
end
def do_search(phrase)
res = Net::HTTP.get(URI.parse("http://duckduckgo.com/?q=%5C#{e_url(phrase)}"))
match = res.match(/window.location.replace\('(.*?)'\)/)
return match.nil? ? false : match[1]
end
ARGF.each do |input|
res = do_search(input.strip)
print %Q{[#{input.strip}](#{res.strip})}
end
