Unique housing development under threat as the games come to town