The solar panels in my area are often in empty fields that could have been forests, farms, or houses with panels on the roofs. The panels still contain minerals mined from the earth, which has an impact. Broken panels will still have to be disposed of safely and ideally recycled, either way creating an impact.
If we let the rules of captialism continue to control our mindset of course we will continue to use more and more energy, just like companies expect more and more growth and profits.