You don't need to build complete sets. You can often build three walls of a room, as the fourth wall may not be seen.
In the directors view, even the walls you have built are shown at only a low height, so that you can more easily see the action.
Remember, what you see through the camera is what your watchers will see.