You've got two vectors, with n coordinates (a1, a2, a3 ... an) and (b1, b2, b3 ... bn). Their dot product is a1*b1+a2*b2+a3*b3+...+an*bn.
Dot product of two vectors, as you could see, is the orthogonal projection of one vector on the other, if one of them has a length of one. If not, it's that much longer. Therefore that R right angle vector in the tutorial should be one unit long, so that (R.F) would exactly be the amount of force that rotates the bone.
You don't calculate force, here, unless you want stuff to move on their own, due to gravity, or something, rather just apply some kind of force. Most appropriate would be to make the force proportional to the distance, (or probably square root of the distance)
Good thing that I do know a little bit about physical simulations, so adding forces/restraints in to the equation makes it a lot simpler for me.