Picture an onion. Or, if you prefer space to produce, think of a solar system. Your logo is in the center, and everything else spins off of it. In fact, at least for the first couple of layers, everything should also be inspired by your logo.
Center: Logo
Ring 1: Colors and Primary Fonts
Ring 2: Secondary Fonts, Secondary Colors, Tagline
Ring 3: Logo Placement (Stationary), Logo Placement (Presentations), Logo Placement (Promotional Materials), Logo Placement (Electronic Communications), Document Fonts, Type Treatments, Secondary Image Marks and Icons
Ring 4: Photographic Guidelines, Graphic Element Guidelines, Editorial Policies, Catch Phrases…
…and so on. 
As you can see, as the rings move out, more is added. While visuals are the primary elements in branding, sounds (music, voice and tonals) and even smells can also become a part of a brand. The more rings, the more detailed the brand becomes.
Not every brand is super-detailed, but the heavy hitters definitely are. For example, think about Coke®. Coke has a super-detailed, highly regimented brand standard. But… if I say “Coke Red” just about everyone know exactly what shade of red I’m talking about.
So should your brand be super detailed? Well, that’s for you to decide. However, no matter how simple you decide your brand should be, you should build it in an organized way (remember, think onion or solar system) so that should you decide to expand on it, the structure is already there.