I previously had the render distance set to 200 which caused framerate to drop when looking at the maze.
This render distance isn't required to experience the game so I shortened it to 50 and added a fog.

I've know of this problem for a while and I'm sorry I haven't fixed it, even my PC lags when looking the wrong way.
Now the game should be playable without frame drops.