Virtual reality is only for tech enthusiasts? Get ready to see fish, mice and fruit flies join the VR craze with scientists from Austria and Germany
For generations, scientists and sci-fi fans have inspired each other to (boldly) go where no one has gone before to better understand the world around them. Now, real-world science has taken another page right out of the science fiction handbook, creating an immersive, virtual reality “holodeck” for animals.
You might remember the holodeck as one of the most iconic features of Star Trek: The Next Generation, a high tech VR arena that could create any imaginable world in the blink of an eye. Now, scientists in Vienna and Freiburg have developed a holodeck-like environment to help us understand how visual cues influence behavioral choices in animals.
The platform, called FreemoVR, projects an environment within the animal arena (like an ocean floor) and then precisely follows the animal with multicamera tracking software developed by the Straw Lab. Most importantly, FreemoVR adapts to the viewpoint of the animal in real-time according to its behavior. This means that, from the perspective of the animal, it is moving through the virtual world and not toward a at screen.
Using VR to study behavior is nothing new. Model organisms have been maneuvering their way through video game mazes for years and this work has been incredibly helpful for our understanding of the brain. But until now these models have required the animals to be partially or fully immobilized. Ultimately this restriction influences the way the animal would behave compared to the natural environment. The advantage of FreemoVR is that there are no strings attached. Here, animals can explore the environment without physical restriction or additional visual aids (think: Google glasses for mice) that might change the way they interact with their surroundings.
Proof-of-concept tests with FreemoVR have been published in Nature Methods for three of the most common model organisms: fish, fruit flies and mice. The team tracked the animals as they swam, flew or ran around the virtual objects and situations and compared this with the animal’s response to the real thing. Using this technique, the team could make mice afraid of virtual heights and make fruit ies and zebra fish avoid a virtual pole in a way that mirrored their responses to real-world objects.
Why is this important? Let’s take fish, for example. Until now, if you wanted to see how a fish would interact with an object you would have needed to physically move the object or the fish.
“It is very difficult to separate the effect of this movement from the experiment. Now you can project something, like a plant, and then take it away instantaneously. It gives you a real feedback loop for your experiment,” says Prof. Kristin Tessmar-Raible from the Max F. Perutz Laboratories in Vienna. Her lab is studying how light and time influence behavior and uses FreemoVR for zebrafish and medaka fish.
FreemoVR can be adapted for each type of animal, which might mean that a different arena style or number of cameras may be used to capture its unique challenges (for a home experiment, try tracking a fruit fly with your eye). In FreemoVR for fish, the virtual environment is projected directly on the fishbowl. Four cameras track the fish, measuring disturbances in near-infrared light. Using strong contrasting backgrounds like a checkerboard, it is possible to detect preferences in fish who choose to swim close to light or dark squares, which might give insight into how they perceive light.
“In general, this system opens up a lot of new research opportunities to address questions related to animal biology. For example, we have no idea why fish have photoreceptors in the brain. What are they doing there and how does this affect their perception of the world?” says Kristin.
Taking it one step closer to sci-fi, FreemoVR also allows you to instantaneously switch the environment or the object in mid-experiment to see how the animal reacts when it is suddenly “teleported” to a new virtual location. The team tested this by creating two portals where the fish could decide between a checkerboard world and a plant world. In another scenario, they added a twist of swimming together with virtual fish or a swarm of video game “space invaders” to see how the animal’s social responsiveness influenced its decisions.
Although FreemoVR was able to convince the animals in the tests, it still has some kinks to work out. Unlike a Star Trek holodeck, the cues are still only visual and the animal’s interactions with the arena itself may influence its behavior.
It is also limited by the number of images shown per second on consumer marketed display units (like projectors, monitors) which currently peak at 120 per second. Animals that are able to process images at a greater rate per second than the VR display unit is able to produce would see the illusion like a slowed-down movie reel. The team already tested this by artifcially prolonging the time between images. They found that the fruit flies, which can very quickly process images, stopped interacting with the virtual object like they would a natural one. But this limitation should subside with advances in consumer electronics.
Trekkies turn Entrepreneurs
FreemoVR is an open-source software and for the tech savvy behavioral biologist, it is a great way to incorporate freely moving animals in future experiments. It is also scalable to fit your model organism of interest. For biologists who might not have spent enough time in coding classes, things just got easier. A startup, Loopbio, founded by the lead authors on the publication, Dr. John Stowers and Max Hofbauer, is now helping laboratories around the world set up their own FreemoVR systems.
“The flexibility of the system is what makes it so special. You can have an idea for a new object or experiment and within 20 minutes you have designed what you need and are ready to start,” says Max, CEO of Loopbio. The company is providing gear to set up FreemoVR to labs that want to start investigating complex behavioral questions – and who knows? They may be writing the script for the next generation of iconic sci-fi .