Rectangles contain a large amount of inherent symmetry (or to put it another way, although they contain four coordinates they only actually contain 4 ordinates worth of information). The algorithm used takes advantage of the symmetry of the geometric situation to optimize performance by minimizing the number of line intersection tests.