Building Blind

I wonder how world-building works for other writers.  Right now, I'm working on a 12-year-old girl in the Midwest in the early 90s and a fantasy world trying to find a comparable historical time period, so I can do some research.  Since the story about the young girl is semi-autobiographical, you'd think it would be much easier, but my memory is so poor, and I hate to fictionalize if I can factualize.  As a result, I've been thinking more about the fantasy world. 

Most writers I know talk about an image being the first thing that happens to them, and the world grows from there.  I don't have that option because, for whatever reason, my mental lens cap is on, so I can't visualize like that.  Images don't come to me; words do.  All of my major advances in building this world (Napkin Epic World in honor of its humble beginnings in college) have come in the form of either dialogue explaining some action or me wondering what led to certain actions I know are part of the plot.  They just are.  It's my job to find out why, I guess.

What's your catalyst when it comes to world-building/writing?  Do you just write in your world and let the details come, or do you plan/plot things out carefully in advance?