The rubber on winter tires is 'softer' than on all-seasons, which helps stop and 'grip' in icy or snowy conditions. The tread on winter tires is also more aggressively 'cut' to better 'cut into' and grip snow. That's why winter tires are usually more noisy than all seasons. The greater 'cut' edges hitting hard asphalt is what makes all of the noise.
So, to answer your question... The main issue with using winter tires year round is that they will wear out A LOT quicker in warmer temperatures (ideal temperature for winter tires is below 7 degrees celsius) probably to the point that they'll be useless as 'winter tires' next season. If you do run winters in the warmer months, you'll probably notice them getting quieter as more and more of those 'grip' edges get worn away.
If you can only afford one set of tires, it's better to go with all-seasons. Not strictly as good as winters in ice and snow, but wear much better during the warmer months.
Cheers