以前研究过这个问题，最后也实现了这个效果，贴下部分代码（很早以前写的，当时还没养成好的习惯，所以代码质量很差，题主主要参考思路即可）：

 Tooltip.prototype.show = function() {

    elTopLocation = this.getBoundingClientRect().top;
    elBottomLocation = window.innerHeight - this.getBoundingClientRect().bottom;
    elLeftLocation = this.getBoundingClientRect().left;
    elRightLocation = window.innerWidth - this.getBoundingClientRect().right;

    if(elLeftLocation < minVerticalDistance) {
      $(this).addClass('hint hint--right');
      this.direction = 'right';
    } else {
      if(elRightLocation < minVerticalDistance) {
        $(this).addClass('hint hint--left');
        this.direction = 'left';
      } else {
        if(elBottomLocation < minHorizonDistance) {
          $(this).addClass('hint hint--top');
          this.direction = 'top';
        } else {
          $(this).addClass('hint hint--bottom');
          this.direction = 'bottom';
        }
      }
    }
  };

大致讲下思路，先实现上下左右四种浮动效果，然后计算出元素相对于浏览器窗口的相对位置进行判断，继而选择特定的效果。
接受 @公子 批评，修改代码如下：

 Tooltip.prototype.show = function() {

    elTopLocation = this.getBoundingClientRect().top;
    elBottomLocation = window.innerHeight - this.getBoundingClientRect().bottom;
    elLeftLocation = this.getBoundingClientRect().left;
    elRightLocation = window.innerWidth - this.getBoundingClientRect().right;

    var direction;
    direction = elLeftLocation < minVerticalDistance ? 'right' : 'left';
    direction = elBottomLocation < minHorizonDistance ? 'top' : 'bottom';

    $(this).addClass('hint hint--'+direction);
    return this.direction = direction;
};

下次再也不敢贴奇葩代码了。