Ok, I made something interesting (to me) last night. It’s probably not worth taking the time to write up, but someday someone might find it and think it useful. I apologize for the messiness of the code, if I take this further and clean it up, I’ll update this post.

I was working on a few text-analysis features for Marked and decided I wanted to be able to show repeated words on a per-paragraph basis. The following is the experiment I did as a proof-of-concept. I decided that I wanted to do this in JavaScript/jQuery for various reasons, so my existing Ruby scripts were mostly useless. Thus, I devised a way to handle it entirely in a WebKit browser.

Demo

For the purposes of demonstration, I set up a single-page, dynamic version of this. You can enter any article URL and see the processing take place. Hover over a bolded word to see where it repeats in the paragraph. Note that it’s pulling through a proxy of Marky the Markdownifier, and that some markup will return blanks. Obviously, it also helps to have a lot of text in the article you’re analyzing. Hard up for ideas? Try this, or this. Give it a few seconds to load, it pulls in the content in the background and I haven’t put a progress indicator on it yet.

Breakdown

I started with a Porter Stemmer using a script from tartarus.org. Stemming allows you to break a word down to a root form, so that all variations of a word can be boiled down and plurals, conjugations and various anomalous representations of a word will all match each other. From there, I do a frequency check to find word roots used more than once within the block being processed, creating an array of the repeated words. Then I re-parse the block, one word at a time, adding some markup to words whose root is found in the previously-created array.

Here’s the main script with a few comments. Until I get this prettied up, I won’t go into a step-by-step. Feel free to lift and improve as you like. Remember to include the porter-stemmer script before this script. Oh, and because I’m lazy, you’ll need to include jQuery as well. The in_array function is stolen from php.js.

// from php.js
function in_array (needle, haystack) {
  for (key in haystack) {
    if (haystack[key] == needle) {
      return true;
    }
  }
  return false;
}
 
// short, common words to skip when counting
var stopwords = ['1','2','3','4','5','6','7','8','9','0','one','two','three','four','five','about','actually','always','even','given','into','just','not','Im','thats','its','arent','weve','ive','didnt','dont','the','of','to','and','a','in','is','it','you','that','he','was','for','on','are','with','as','I','his','they','be','at','one','have','this','from','or','had','by','hot','but','some','what','there','we','can','out','were','all','your','when','up','use','how','said','an','each','she','which','do','their','if','will','way','many','then','them','would','like','so','these','her','see','him','has','more','could','go','come','did','my','no','get','me','say','too','here','must','such','try','us','own','oh','any','youll','youre','also','than','those','though','thing','things'];
 
// takes the text of a paragraph element as input
// returns marked up text with repeated words in 'b' tags with a class matching their "stemmed" root
function checkWords(input) {
 
  var words = input.split(' ');
  var wordcount = {};
 
  // build an object to count word frequency
  $.each(words,function(i){
    thisWord = String(this).replace(/[\/\\]/,' ').replace(/’/g,"'").replace(/[^a-z' ]/gi,'').toLowerCase();
    if (!in_array(thisWord,stopwords)) {
      var word = stemmer(thisWord);
      if (wordcount[word] > 0 && word.length) {
        wordcount[word] += 1;
      } else {
        wordcount[word] = 1;
      }
    }
  });
 
  // convert the object to an object array
  // include only words repeated more than once within the paragraph
  var topwords = new Array();
  $.each(wordcount,function(w,i){
    if (i > 1)
      topwords.push({'word':w,'freq':i});
  });
 
  // convert the object array to a flat array
  topwordsArr = new Array();
    $.each(topwords,function(i) {
    topwordsArr.push(String(this['word']));
  });
 
  // re-parse the output, marking up repeated words based on their stems
  var output = '';
  $.each(words,function(w) {
    var aWord = String(this);
    var stripWord = stemmer(aWord.replace(/[\/\\]/,' ').replace(/’/g,"'").replace(/[^a-z' ]/gi,'').toLowerCase());
    if (in_array(stripWord,topwordsArr))
      output += ' <b class="'+stripWord+'">'+aWord+'</b>';
    else
      output += ' '+aWord;
  });
  return output;
}
 
(function($){
  // grab common top-level elements
  grafs = $('p,ul,ol,blockquote,h1,h2,h3,h4,h5,h6,pre code',$('#content'));
  // navigate each element found
  $.each(grafs,function(a,g){
    // if it's a paragraph, we'll process it
    if (grafs[a].tagName == "P") {
      $('#work').append($('<p>').html(checkWords($(grafs[a]).text())));
    // if not, we just stick it back into the DOM
    } else {
      $('#work').append(grafs[a]);
    }
  });
  // set up hover listeners on the 'b' elements
  // the class is pulled from the hovered element
  // all similar words are highlighted on hover
  $('b','#work').hover(function(){
    var thisClass = this.className;
    $('.'+thisClass).addClass('highlight');
  },function(){
    $('.highlight').removeClass('highlight');
  });
})(jQuery);

This little bit of css will add the highlighting and a nice fade in compatible browsers:

b {
  font-weight:bold;
  -webkit-transition:color .2s ease-in-out;
  -moz-transition:color .2s ease-in-out;
  -o-transition:color .2s ease-in-out;
  transition:color .2s ease-in-out;
}
.highlight { color:rgba(207, 95, 205, 1);}

If I decide to include this in Marked, it will definitely get some revamping. Like I said… proof-of-concept. Check out the demo, though, it’s kind of neat.

Related posts:

1. A simple but handy Bash function: console
2. Ampersands & Javascript
3. Quick Tip: This command line trick is tops

Originally posted on BrettTerpstra.com at Word repetition checking with JavaScript

I had an interesting idea this morning. At least I find it interesting, but I haven’t slept much lately. Either way, here it is: in Markdown, if you misname a reference link, forget to fill one in or have a malformed URL, your broken Markdown shows up in your output. Wouldn’t it be nice if your preview highlighted those for you before you went to publish?

Marked 1.3, which is coming along very nicely, has a few JavaScripts built in to the preview (which can be turned off in preferences). It provides a table of contents based on headers in your document, smooth scrolling, tool tips to show you where external links will go and, as of this morning, highlighting of broken links.

The code is simple. This version is jQuery, just because that was convenient, but you can pull this off in plain old JavaScript with barely any extra code:

// Scan for and highlight malformed/incomplete Markdown links
$('#wrapper').html($('#wrapper').html().replace(/(\[.*?\][\[\(].*?[\]\)])/g,
"<span style=\"color:rgba(152, 50, 63, 1)\">$1</span>"));

It just scans for Markdown-style links ([text][title] or [text](link)) that are still in your text after converting to HTML. If it finds them, it highlights them in red. Just thought I’d share.

By the way, in addition to the JavaScript fun, the next version of Marked has multiple custom styles (unlimited), can open any text file with any extension, can load MathJax for MathML rendering and much more. I’ll keep you posted on its release, but it should be soon.

Related posts:

1. Auto-convert your inline Markdown links to references
2. Markdown QuickTags 0.7
3. Some Chrome love for the Markdown Service Tools

Originally posted on BrettTerpstra.com at Catching Markdown mistakes

Another quick experiment. Honestly, I don’t use pluggable functions in jQuery nearly as much as I should, so this is really just a brain exercise to get myself used to it. This one, called Pully, lets you specify a selector to have its contents cloned and inserted as a pull quote in your text.

It seems to me that the average HTML pull quote, when you look at it from a source code or screen reader viewpoint, is redundant and/or confusing. They really only make sense when they’re visually styled. My thought is to use semantic markup within a block of content to emphasize pull-quotable text, and then unobtrusively create the pull quote from that text when it’s of use. I could be operating under faulty assumptions, but it makes sense to me tonight.

Pully has two options, newclass and prependtoparent. newclass determines the class that is added to the inserted pull quote element, and defaults to ‘pullyquote’. prependtoparent determines whether the pull quote element is inserted “inline” (directly before the selected source element) or moved up in the DOM and prepended to a parent element. It defaults to false (inline) and can be set to any element found upstream in the DOM from the source.

In its simplest usage, you can take a paragraph which contains a span with a class of .pullquote (for example) and call it like this:

$(document).ready(function() {
	$('.pullquote').pully()
});

Here’s an example of Pully being run with all options. This call will take an <em> block inside of a <strong> element and turn it into the pullquote. It will insert the new element at the beginning of the enclosing paragraph and apply the class “ppullquote” to it:

$(document).ready(function() {
	$('p strong>em').pully({'prependtoparent':'p','newclass':'ppullquote'});
});

You’ll need to provide your own CSS. Unless otherwise specified, your pullquote will have a class of “pullyquote,” to which you can apply whatever styles are most fitting. I borrowed some styles from type-a-file for the demo, the pertinent portions are included in the main source of the demo page.

You can download the plugin here, and view it in action here. Feel free to modify, repair or improve on the idea.

Related posts:

1. Desktop Quotes Geeklet
2. Antique 1.6, final release
3. Quick Tip: Extracting Mac App Store reviews as text

Originally posted on BrettTerpstra.com at Pully: jQuery plugin for automatic pull quotes

The concept is exactly the same: tap the main text field to clear it, type or paste your text, then tap the prompt button to open the prompt view. A single tap will start scrolling, a double tap will exit, and dragging up or down will allow you to fast forward or rewind. On the iPhone, it actually works best in landscape mode, allowing the words to be big enough to see while still fitting enough on the screen that it’s not ridiculously slow. I haven’t added any speed controls to it, which I may do at some point. For now, it’s on and off.

Point your mobile browser to http://brettterpstra.com/share/promptdown/mobile.html and give it a try. Tap the plus button (on the iPhone) or the “send to” icon (on the iPad) to add the app to your homescreen and be able to use it full-screen. This version actually works well in Safari and Firefox on the desktop, too.

Neat.

Prompt screen on iPad

Edit screen on iPhone 4

Related posts:

1. Look, I made you a Markdown teleprompter
2. iOS App Giveaway: WriteUp
3. iOS App Review: WriteUp

Originally posted on BrettTerpstra.com at PromptDown for iOS: Mobile Markdown Teleprompter

PromptDown Edit Screen

PromptDown Prompt Screen

Be sure to check out version 2, with iOS support (iPhone/iPad)!

Here’s my stupid trick of the day: a Markdown teleprompter called PromptDown (I see what you did there…). The idea was to create a teleprompter for recording screencast voiceovers that was cheap, fast and worked on plain text. I wanted to provide a non-WYSIWYG form field for typing or pasting which still preserved paragraph breaks, so Markdown (via Showdown) was an ideal solution. I also included the jQuery “tabby” plugin so you could use tabs in the textarea without jumping out of it, which is nice when editing Markdown, depending on your formatting habits. The rest is just CSS and quick-and-dirty tricks.

How to use it

It takes input in standard Markdown format and, when you press the “Prompt” button or hit Shift-Enter, it blacks the screen and renders your Markdown in large, contrasting text. Then you can hit Spacebar or Enter to start and stop scrolling. Stopping auto-scroll and using your mouse’s scroll wheel will allow you to navigate. Escape (or a click on the background) will take you back to the editor.

There are a few more tricks listed in the text that initially populates the text area when the page loads. Among them is a note that entering a horizontal rule in Markdown syntax will give you a bright yellow marker at that point in your script, which is handy for longer work. The easiest format for a horizontal rule, for me, is - - -, three dashes with spaces between them on their own line.

Why I made it

Why did I spend an hour and a half making a cheap teleprompter with very few features? Because I just wanted to scroll some text, and the market for Mac-based teleprompters seemed to start around $65 and go up from there. I’m not one to complain about a price when there’s appropriate value, but I’m assuming that those $65–250 apps were doing magical things I just didn’t need. This is my solution, as hackish as it may be. As an aside, I used to have a license for VideoCue, which I loved, but I never upgraded it and now I’ve lost both it and my older version of VideoCue. Drat.

Do it yourself

Feel free to use it online, or feel free to borrow the code and build your own locally. It doesn’t require anything unusual, and will run right from a folder on your desktop if you point your browser there. I should note that it works much better in Safari 5 than in Firefox, at least in my quick testing (and hopefully not-at-all in Internet Explorer… I’m still carrying a grudge). I’m not really going for a fully cross-browser, all-purpose solution here. It fills a need for me, and if someone wants it to do something else, they can have at it. Here’s a zip of the code as of this evening: PromptDown.zip. Try out the online version at http://brettterpstra.com/share/promptdown.

Related posts:

1. PromptDown for iOS: Mobile Markdown Teleprompter
2. Markdown QuickTags makes 1000 people happier and more attractive
3. Markdown QuickTags: WordPress plugin for Markdown lovers

Originally posted on BrettTerpstra.com at Look, I made you a Markdown teleprompter

Video preview

httpv://www.youtube.com/watch?v=KTcISVU4Nf4

YouTube Link…

Setting up your web server

To run it, you need to run your local web server as your own user. This is, of course, a major security risk and completely inadvisable if your local web server is open to outsiders in any way. Mine’s not, so I didn’t put too much time into figuring out a more secure way to do this. If you have any clues in that area, let me know and I’ll post them.

To run your local web server as a different user, you need to edit /etc/apache2/httpd.conf. You’ll need to edit it as root, so use sudo to launch your text editor of choice (e.g. sudo vi /etc/apache2/httpd.conf). If you, like me, just use TextMate’s mate command, you’ll be prompted for a password when you save.

Locate the lines:

User _www
Group _www

Change them to:

User yourusername
Group staff

You’ll also need to enable PHP, if you haven’t already. If you haven’t, locate the line that starts with:

#LoadModule php5_module

Just remove the hashmark at the beginning (#) to turn PHP on.

Now, at the command line, type sudo apachectl graceful to restart the server under the new user. If everything is in place, you’ll be able to run HomeControl without a hitch now. Turn on Web Sharing in System Preferences > Sharing to keep the web server running through reboots and logouts.

Installing HomeControl

Just download the zip file at the end of this post and unzip it into your ~/Sites folder. If you prefer to have it elsewhere, or have a custom folder set up for Apache, it will run just fine in any folder, as long as you can access it from a web browser on the local network.

Next, point your web browser to the folder. If you put the ‘homecontrol’ folder directly in ~/Sites, you should be able to reach it at the url http://computer_name.local/~Username/homecontrol. You need to know the name of the computer it’s on (set in System Preferences > Sharing, at the top), and your username on that system, substituting each in the appropriate place in the url.

Once you’ve loaded the page and tested it out, use the “+” icon at the bottom of Mobile Safari’s web browser to add an icon to your home screen which will take you directly there in the future, and will run the app full-screen.

Customizing HomeControl

Since I know not everybody who wants to try this out is going to be ready to hack into the jQuery and PHP, I made quite a few bits of the app modifiable with simple HTML edits. If you’re comfortable with that, you can change the list of applications and add iTunes features quite easily.

Application launcher

• Locate the div with the id “apps”
• Find the unordered list (<ul>) inside of it
• Edit or copy and edit existing lines by changing only the name of the application in the first span
• Use the exact displayed name of the application you want to control, and HomeControl will pass it to the PHP exec function

iTunes functions

• Locate the div with the id “itunes”
• Find the unordered list inside of it with the id “itunescmd”
• Add new features by copying an existing line and changing the rel attribute and the text of the link
  • The rel attribute passes the actual iTunes AppleScript command, which will be appended to a ‘tell application "iTunes" to’ line and passed to osascript
  • The text of the link is the title which will appear in the menu

iTunes EQ Presets

• Locate the div with the id “ituneseq”
• Find the unordered list inside of it
• Edit the text of the links in the list with the exact title of the preset you want to control
• Add or remove list items as desired

Startup screen and icon

• Edit homecontrolicon.png in the homecontrol folder to customize the app’s icon on the homescreen.
• Edit hc_startup.png in the homecontrol folder to customize the startup screen.
• If you’ve installed the web app, delete the icon and reinstall from Mobile Safari to see the new images.

Advanced customization

If you’re handy with the jQuery, the PHP and the HTML, you can do a lot with the examples in the code. It’s very much a proof-of-concept, and intended to be a jumping point for more experimentation, so have at it.

The app currently uses the jQTouch library, but mostly for the CSS. It has very few dependencies on the API, so it should be a relatively trivial matter to switch to a different library. It does rely quite heavily on jQuery, though.

The main PHP/osascript calls are in functions.php. All functions are called using jQuery’s $.get function, which makes Ajax calls to the functions.php file in the background and receives updates on completion.

Have fun, hopefully my horrible waste of time will be someone else’s inspiration…

Main Screen

Applications Screen

iTunes controls

Download

Related posts:

1. PromptDown for iOS: Mobile Markdown Teleprompter
2. The Mac and iOS mind mapping app extravaganza
3. Look, I made you a Markdown teleprompter

Originally posted on BrettTerpstra.com at HomeControl: Local Mac control for iPhone

For a quick “how-to” on using the Antique (or any) Safari Reader hack, see the related article in the HowTo section.

This will be the last release of Antique, I think, barring a few minor fixes (follow @ttscoff for updates on this and other projects). You can download it here. The code is completely open source, if anyone wants to continue the project feel free (credit would be swell, where it’s due). I’ll be putting my free time into Instapaper Beyond and other more “legitimate” projects now (although I really would like to do a nice, high-contrast, Helvetica version…). In deference to Faruk Ateş, I’d like to clearly state that this is a hack, and you run a risk (albeit very minimal) of breaking your Safari install if you don’t know what you’re doing. I also won’t be posting further simplified instructions for installation, but they’re not hard to find. For advanced users, you simply need to know that you’re replacing Reader.html in Contents/Resources.

The coolest changes, in my opinion, are an amped-up Widon’t for headlines, and improved image linking. On the Widon’t end, if a title exceeds the width of the headline area, it’s actually split in the middle to provide more equal-length lines. It’s neat-o. And for the images: I had it only lightboxing images which were too large for the style and which weren’t already linked somewhere else, resulting in the user never knowing what would happen when they clicked an image. Now, if an image is linked to an external site, that link is extracted and placed on a line below the image. If an alt or title attribute are present in the image or the link, they’re preferred over the source attribute for naming the link, which should provide a decent idea of where you’re going, in most cases. Images which are linked, then, are always linked to the Colorbox effect, and will never take you out of the Reader page. Ah, consistency.

Here’s what else is new in this version.

• I swapped out the scrollbar images, shifted the colors a little, and included them in the HTML file.
• Some style changes, including a darker background and slightly heavier gradient
• Massively fixed up the ampersand code. Also shifted ampersands 1 pixel left for better kerning within non-italicized text.
• Improved smashing of empty paragraphs in bad markup to avoid huge spaces between paragraphs.
• Fixed a CSS mistake I had introduced which caused the background and drop shadow to be visible before the content slid up from the bottom.
• Set up an onscroll check for new elements. Every time the page is scrolled, Antique checks for new elements (i.e. additional pages that were inserted after initial load) and applies the default functions to them.
• did I mention 12px Droid Sans Mono for code blocks? 12px seems reasonable, and doesn’t change when zooming/shrinking the rest of the text.

Are there too many buttons in the HUD for your taste? They’re there as an example, but you can remove them easily by commenting out the “button” lines at the bottom of the Reader.html file. The HUD will adjust size and position automatically based on the number of buttons you have in there, so just remove the ones that aren’t of use to you.

If you’re playing with Reader styles, here’s a tip: you can set up a “sandbox” page to speed up development (at least with css). Once Reader has loaded a page that has elements you want to style, right click and choose “Inspect Element” (obviously, you have to have the developer menu enabled in preferences). Then, right click on the doctype declaration in the element inspector and choose “Copy as HTML.” That will give you the full HTML of the rendered document, which you can save to a new HTML file. Extract the inline stylesheets and link them into external stylesheets, and you can edit in CSSEdit or Firebug. It’s handy, given that, in order to change the live styles, you have to restart Safari every time you edit Reader.html.

I had started working on smart quotes and other javascript-based typography changes, but decided it wasn’t worth the time to try to come up with regular expressions that covered every possible circumstance. If anyone wants to pick up where I left off, I’d love to see what you come up with.

For a quick “how-to” on using the Antique (or any) Safari Reader hack, see the related article in the HowTo section.

Related posts:

1. Antique is back, baby
2. Antique Safari Reader hack update with Evernote goodness
3. Safari Reader Antique hack

Originally posted on BrettTerpstra.com at Antique 1.6, final release

I set out on a search for an easy, all-purpose solution, but nothing worked as well as I wanted it to. So I did a little scavenging and put together a couple of functions that do the job pretty well. They use jQuery, but could just as easily be done with pure Javascript, if you really wanted to.

Here are the two functions I’m using in the latest (unreleased, at this moment) version of Antique:

function fixAmpersands() {
    // entity encodes only ampersands that aren't already encoded
    $("h1,h2,h3,p:contains('&')", document.body).each(function() {
        if( this.nodeType == 3 ) {
            // regex from Stack Overflow http://bit.ly/aJZVCG
            $(this).html( $(this).html().replace( /&(?![a-zA-Z]{2,6};|#[0-9]{2,4};)/g, "&" ) );
        }
    });
}
function dressUpAmpersands() {
    // adds a span with class 'amp' to the entity-encoded ampersands
    $("h1,h2,h3,p:contains('&')", document.body).each(function() {
        $(this).html( $(this).html().replace( /&amp;/g, "<span class='amp'>&amp;</span>" ) );
    });
}
$(document).ready(function() {
    fixAmpersands();
    dressUpAmpersands();
});

They seem to work pretty well, so far. I accompany them with some CSS I nicked from SimpleBits and Patrick Haney:

.page span.amp { 
    font-family: Baskerville, Palatino, Constantia, "Book Antiqua", "URW Palladio L", serif;
    font-style: italic;
}

In my situation, it doesn’t really matter if ampersands are captured within code blocks, but just to keep styling consistent, I also force any .amp-wrapped ampersands within pre and code blocks to match the pre/code styling:

#article .page pre,#article .page code,.page code span.amp, .page pre span.amp {
    font-family: 'Droid Sans Mono', 'Andale Mono', 'Courier New', Courier, monospace !important;
    font-style: normal;
}

Of course, this is all really better handled server-side with, say, PHP, but if you’re looking for spicy typography and are limited to client-side solutions, give it a shot! I’m running these right before my rather wicked new version of widon’t which splits long titles into nearly equal-length lines… more on that soon.

Related posts:

1. Word repetition checking with JavaScript
2. Catching Markdown mistakes
3. Antique 1.6, final release

Originally posted on BrettTerpstra.com at Ampersands & Javascript

Since the point of Clippable was to deal better with things like code blocks (especially for saving snippets to Evernote), it now removes the toolbar and line numbers from SyntaxHighlighter blocks. It also looks for another common technique: converting lines in code to an ordered list inside of a pre block. This is just blotted out with CSS now. Those are the only two highlighting methods it targets at the moment, but I’ll tackle more as I run into them.

If you already have the bookmarklet installed, you’re already benefitting from these changes (the bookmarklet calls the source scripts on my server, so it is, in essence, automatically updated). If not, just cruise over to the Clippable page and grab it!

Related posts:

1. Clippable updated, goes mobile
2. Clippable
3. Clippable to Evernote Snow Leopard Service

Originally posted on BrettTerpstra.com at Clippable updated to remove source code line numbers

So my modification of the Readability bookmarklet kind of snuck out before it was ready, but it’s my own fault. Now I’m scrambling a little to make it more presentable and less of a straight-up hack of the excellent original. I wanted to make a few things clear about my goals and purpose on this one.

First the entire project was really a subset of my attempt at a better Evernote clipper for Snow Leopard. One which allowed me to preserve code formatting and automatically remove comments and ads from the post, in a smarter fashion than the current Safari clipper does. I built it as a System Service and run it with a hotkey. You can download it and try it out, if you like. I’ll make a more accessible version with instructions shortly.

This is why I removed the formatting options from the bookmarklet… Evernote was going to strip all of that out anyway. Originally, I was just using the code to strip out ads and find the meat. The modifications to preserve code blocks, movies, etc. were simply working toward the “perfect” Evernote clip.

It works for what it is, but wasn’t really intended to be used without the Evernote Service. If I get enough feedback, and no cease and desist orders from the original creators, I’ll continue to modify it. One thing you can certainly do to help is provide me with URL’s to pages it fails on; the more scenarios I can study, the smarter I can make it.

Related posts:

1. Taking the Markdown to Evernote service further
2. Why a plain-text nerd uses Evernote
3. Evernote Site Memory tagger for WordPress

Originally posted on BrettTerpstra.com at Readability2 leaks out