Why does this happen?

Its just chunk errors! You can use em for awesome building advantages!

This probably used to be the point up to where the world got generated in a previous minecraft version.

Whenever a major update of minecraft occurs (for example 1.2.5 to 1.3.1) the world gets expanded with newly generated terrain, starting from the point where it had ended. It often happens that the old border and new land don't fit well, which is what happened in the screenshot you posted.

It could be, but I highly doubt it as I have had it MANY times in single player worlds, including starting off right in a chunk error biome!

Single player worlds tend to be generated over many versions as well as time goes on. There have been major changes in the world generation routines several times over. I will side with Landr0 on this one and say it was generated with a different world generation coding,

Yup. With each update world generation is tweaked, so when new land is generated, it creates these epic cliffs. Great place to build a house, like I did.

But its so ugly! Haha.