Yes, that's the short of it. Each pixel needs its own wires, readout, and processing chain, and resources are limited on the spacecraft. The cryostat (instrument that keeps the pixels cold) only has so much cooling capacity and all the wires add thermal load.
Future missions are planned with more pixels (Take a look at the EASA Athena mission and its X-IFU instrument), and to reach that goal they are using multiplexing methods to allow more pixels to run on fewer wires.
NASA/JAXA’s XRISM Mission Captures Unmatched Data With Just 36 Pixels ( science.nasa.gov )