People having difficulties setting the --with-libdir option needs to set it like this :
--with-libdir=lib64
That option works fine. Doing --with-libdir=/usr/lib64 doesn't work because the configure script prepends the --prefix option before the --with-libdir option. So, doing --with-libdir=/usr/lib64 makes the configure script to look for libs in /usr/usr/lib64, which is wrong of course, and not in /usr/lib64.