Immersive Navigation in XR

This project explores new ways to navigate XR environments. The focus is on creating a simple, intuitive, and immersive user experience. At the heart of this experience is “Cosmo,” a voice-controlled assistant powered by OpenAI’s ChatGPT API. Cosmo helps users move through the virtual solar system and delivers factual and fun information on request.

The project was developed as part of an independent study at the University of Applied Sciences Potsdam, supervised by Prof. Reto Wettach. The setup allowed for a high degree of creative freedom while following an academic process.

Core Features

• Voice-controlled navigation using AI integration.
• Natural language interaction with a responsive assistant.
• Smooth motion through space without teleportation.
• Dynamic feedback based on proximity to planets.
• Flexible expansion possibilities for future development.

Interactive Behaviors

Beyond basic navigation, several subtle interactive elements were added to enhance the immersive experience:

• Planet rotation: When the user arrives at a planet, it begins to slowly rotate, adding life and motion to the environment.
• Contextual stop: Once the user leaves the planet’s proximity, the rotation automatically stops to signal disengagement.
• Smooth transitions: Movement between planets follows a curved trajectory for a natural, floating sensation in space.
• Arrival messaging: On arrival, the assistant says where you are and invites questions, but only provides facts upon user request — allowing for more user-driven interaction.
• Zone detection: Each planet has an invisible collider zone that detects when the player enters or exits, updating the current state of the system dynamically.

Challenges and Learnings

One of the biggest challenges was ensuring that navigation commands such as “bring me to the next planet” are correctly recognized and executed through the AI. It took a great deal of iteration to make this interaction reliable and natural. Because ChatGPT cannot execute commands itself, the logic had to be implemented in a way that bridges voice recognition, response parsing, and actual navigation in Unity.

Creating a seamless communication between multiple scripts was another complex task. The experience taught me how important clean architecture is when working with multiple interacting systems — from AI to animation to user interface and voice output.

I also learned how powerful Unity can be when it comes to experimenting with creative XR ideas. The project opened many doors in terms of what could still be added or improved — from more refined dialogue handling to integrating eye tracking or gesture controls in the future.

What I Gained

This was my first time building something this technical from scratch. I gained hands-on experience in working with APIs, developing AI prompts, and debugging complex system behaviors. I also realized how detailed and iterative this kind of work can be: fixing one problem often causes a ripple effect that impacts other parts of the project.

Despite these challenges, I found voice navigation to be a particularly enjoyable aspect to work with — and something that holds a lot of potential for future XR applications.

Future Vision

If continued, this project could be expanded with more intelligent interactions, additional content, and advanced sensing technologies. The modular structure makes it possible to build on what’s already working — and bring Cosmo even closer to a truly helpful and intuitive guide through immersive environments.