I grew up in the Midwest and struggled with a similar guilt problem. It was cloudy most of the time, so it was a special occasion when the sun came out; I kicked myself for not leaving the house (never wanted to). However, I do love the sun.
I now live in southern Arizona where the sun shines 310 days per year, so if I do feel like getting out there, chances are it will be sunny, and I won't be missing out if I hide in my hole another day.