Update, September 4th 2017:
So I got a little curious and wanted to try and get an idea of what 1 acre of cotton means in yield, also what sort of global power demand producing cotton fiber from the ground up looks like.
There are something like 167 million acres of cotton planted worldwide, that's more than 261,000 square miles of cotton. It looks like 29 million tons of cotton are produced a year right now, that's 133,195,950 bales. That's roughly 1.25 bales per acre, so 272kg pounds of cotton per acre. A quick google search shows 6-9.5 ounces per t-shirt, giving you an estimate of 1238 t-shirts per acre.
So if we look at averages, you need a season and an acre of land to make a bit over 272kg of t-shirts. For that 272kg of t-shirts, you'd also need about 1.4 million gallons of water, 122kg of fertilizers, 4.3kg of pesticides, and approximately 4534 kWh of electricity (average American household in 2015 had an electricity consumption of about 10,812 kilowatt-hours). That means the 167 million acres of cotton being grown worldwide uses about 757 TWh of electricity to make cotton fibers. Total world electricity consumption was 19,504 TWh in 2013, that's like 3.8% of the world's power consumption just to get from an open field to spun cotton fibers.
Insanity.
Update, July 3rd 2018:
There's a problem with polyester. Polyester itself is a source of pollution.
First, the manufacturing process results in a host of carcinogenic compounds being used (and in parts of the world with lax regulation, being released into the environment) as well as the fact that during the washing process, you are introducing microplastics/microfibers to the environment. One article states that as much as 85% of human-made materials found on the shoreline are microfibers like nylon and acrylic, which are ubiquitous fibers for textile production. Ouch!
While the danger of microplastics as environmental contaminants is still widely unknown, more and more studies are being conducted to see the extent of damage already done as well as future implications. One fact for sure, though, is we are creating a new geological age which some geologists wish to call the 'Anthropocene' epoch, which includes changes such as anthropogenic climate change and the introduction of synthetic materials into not just landfills but ecosystems the world over. This is concerning.
We also have the fact that 70 billion barrels of oil are used each year to manufacture just polyester! Insanity! With an ever-growing population, what are we to do? Cotton isn't a realistic answer, synthetic and most (if not all?) artificial fibers have long-term environmental ramifications due to the way they break down and linger. Sure, we have discovered that mealworms can digest some synthetic materials into environmentally safe products, but that will only help a fraction of a percent in dealing with this problem in any realistically scalable attempt.
Yet another problem we need to start thinking long and hard about.