Summary: Google it. Do it. Perhaps most notiably, pay attention to “visual vocabulary” and a “taxonomy”. Think them through and then stick to them.
Backstory: Most of my current work is on maintaining an excellent and widely used existing application. But before that, I was given the opportunity to create and maintain a far-reaching custom CMS as a Federal employee for the agency which employed me.
That opportunity was exhaustive in its breadth and depth, and its ultimate success. Most notably, in the beginning, was the guidance and funding to address the key ingredient: Information Architecture.
This started with a copy of the most fundamental book on the subject, Information Architecture: For the Web and Beyond by Louis Rosenfeld, Peter Morville & Jorge Arango. It continued with attendance at a conference entirely devoted to the subject (circa 1993). It was completed with the help of a skilled team on the subject, but sad to say I only remember the lead’s name was Steve and they worked of Manhattan. Great work there included videos in a controlled environment of an array of users using the test site to do their research.