How creativity works
I read this article which is an excerpt from a book and explains, partly using the example of Bob Dylan, how insight can be gained and a creative process kickstarted. The main points summed up are: There needs to be a frustrating impasse first and this impasse can artificial, eg in the form of restrictions placed on the creative piece, that force you to think in different ways.
That aside I knew that long ago, as did anyone who ever played with LEGO. When you limit yourself to, say, using bricks of only one(two tops) color that's when best shit gets made.
Actually, just yesterday I was listening to a talk John Cleese gave a couple decades ago on creativity. BoingBoing or io9 or some tech website had it up.
Putting an artificial restriction on your thinking to prompt new ways of thinking about something is what Douglas Hofstadter always talks about, especially in Le Ton Beau de Marot (where the artificial restrictions on poetry, meter, rhythm, rhyme force a person to be more creative in writing it), but all his books talk about creativity some, Godel Escher Bach, The Mind's Eye, Fluid Concepts, Metamagicalsomething...
I think the coolest work around creativity and how concepts form is being done by the Cognitive Linguistics guys, Fauconnier, Turner, Lakoff, Jackendoff a little, etc... Anyway the punchline there is conceptual blending... Take a concept and then either put it into a wildly different context or apply a metaphor to it from some other, much different part of life, and the mind is very good at blending the two domains into this new domain that starts taking a life of its own as this new, creative thing. Anytime people say, Z is like X meets Y... Actually kind of cliche, you'd probably want to dissociate Z from X and Y as soon as you get there. But thinking about X meets Y is still a way to think about getting there.
Cleese's idea of an open playful and purposeless mode which allows natural creativity to flow at first seems to contradict the theory of an impasse that spurns creativity.
But looking at the example of his Bobness again, he wasn't trying to overcome that impasse he had met. Instead he had resigned not to make music anymore. And it was this purposeless mode that enabled him to come up with something completely new. Apparently sometimes the only way to win is not to play.