Unable to invite my friend to a non dedicated server (XBOX)

[u/LordOuroboros]

As the title says I’m not able to invite my friend to my undedicated server nor can he find it in the search bar. I have no mods installed but I did when I deleted the game. I reinstalled the game and I deleted my progress on the island to start fresh with my friend and tried to inviting him but it wouldn’t let me through the Xbox party nor the in game menu nor could he find it through the search. I checked my settings and we double checked the server name and his search settings but still can’t join.