
I suppose the metaphor is more about isolation than any specific image (in my mind this suggests more "tower" and less "ivory"). In fact, according to my not very reliable Internet source, the metaphor alludes to a passage from the Songs of Solomon that suggests purity. The same source suggests that in the nineteenth-century, the phrase could have been used to refer to towers in Oxford. Now this I believe.
To the extent that the metaphor suggests isolation from the world, I think it is valuable because it helps us question our own academic practices. Yet, it is also valuable to remember that it's not all or nothing - the ivory tower and the world can be related in complicated ways.