In Taiwan we just call it 總統府, which translates to "the presidential office". It's true that it is a Japanese-colonial-era building, but I only ever heard it referenced in historical context, like in a tour or a textbook.
Fun fact: there are also some Spanish and Dutch buildings remaining in Taiwan. They were the first official colonizers before the Ming-dynasty took over.