At one time, there were environmentalists and thinkers that were challenging the concept of growth in the context of being on a single planet with limited resources. Although a little before my time, I am thinking of Barry Commoner and Frances Lappe (who wrote "Diet for a Small Planet" and similar thinkers. This movement still seems alive but more individual.
Personally, I have been influenced for a number of years by a book probably no one has heard of called "The Entropy Law and the Economic Process" by Nicolas Roegen. Although I dont agree with all of this book, there are things in this book that I believe are spot on in analyzing our economic problems and the problems we would face if we really address the fact that unlimited growth is not the answer.
In particular, he points out that our reliance on pricing by work and energy used rather than by entropy gained through an activity is wrong. Energy is always conserved whether using solar, oil, or nuclear as the source of work. The main difference between the sources is the entropic costs of various energy sources. Our polution and resource issues that you reference and the problems with conceptualizing unlimited growth relate to entropy and not energy that is used in using limited resource endlessly.
He also analyzes the costs of reducing growth (to reduce entropic costs) on our need for labor. I am not sure about his solutions but I think his analysis of our problems are again directly on point.