First, don't conflate Israel with Jews. It is a government not a religious authority, and using Judaism as a veil is deeply antisemitic.
Second, this is an infantile understanding of colonization. Shall we give Crimea back yo to the Greeks? They inhabited it 3,000 years ago. Clearly not, and if Greece invaded and exploited the land and the people we'd call them colonists.
This blood and soil logic is exactly what european nationalists use to expell Jews. Comically such advocates of this logic also rarely support returning land to indigenous peoples in the US...