To all viewers but yourself, what matters is the product: the finished artwork. To you, and you alone, what matters is the process: the experiences of shaping that artwork. The viewers' concerns are not your concerns (although it'd dangerously easy to adopt their attitudes.) Their job is wahtever it is: to be moved by art, to be entertained by it, to make a killing off it, whatever. Your job is to learn to work on your work.
Art & Fear: Observations on the Perils (and Rewards) of Artmaking (ed. Image Continuum Press, 2001) - ISBN: 9780961454722