A glitch in the Matrix: Using virtual reality to understand how fish predict the future.

A new virtual reality system allows us to manipulate the sensory environment of adult zebrafish at will while simultaneously analyzing neural activity. This approach can be used to explore how the brain processes complex sensory inputs and how predictions control behaviors.
A glitch in the Matrix: Using virtual reality to understand how fish predict the future.

When it comes to studying higher brain functions like learning, social behavior and cognitive behaviors, zebrafish are an appealing not-too-small, not-too-big, but ‘just right’ model organism. Their brains are small enough to perform high-resolution calcium imaging microscopy of neuronal activity, and to reconstruct neuronal wiring diagrams of many brain areas. Larger vertebrates, such as mice, have brains so big it is highly challenging to optically access a significant proportion of neurons of interest, while smaller organisms like C. elegans or larval zebrafish have a more limited range of behaviors to study.

However, the range of questions we have been able to answer about complex behavior in zebrafish has been limited up to this point by existing experimental methods. The fish need to be somehow head-fixed, so that a microscope can record their brain activity, but they must also remain free enough to ‘behave’ authentically in response to an environment that we can experimentally control to stimulate the behavior of interest.  

Researchers have previously been able to embed larval zebrafish in agarose and use a multiphoton microscope to image neuronal activity evoked by different sensory stimuli. The combination of this approach with electron microscopic reconstructions of neuronal circuits has enabled us, for example, to discover precise wiring patterns among neurons that help distinguish similar smells. In the future, it would be interesting to understand how neuronal networks control complex behaviors like associative learning or social interactions. But such complex behaviors are not yet developed at larval stages, and older zebrafish cannot be restrained in agarose anymore.

In a paper that has been published this month in Nature Methods, we now present a virtual reality (VR) experimental set up that can resolve this conundrum. In this setup, adult zebrafish are fixed in place by a head bar under a multiphoton calcium imaging microscope. An infrared camera recording the fish’s swimming action provides feedback to the virtual environment, so that it can be updated in response to the fish’s movements. The fish thus gets the impression that it is moving as expected through this virtual environment. This experimental setup allows us to measure neuronal activity during behavior.

Fig. 1 - The Virtual Reality arena. The fish is positioned between the posts under the microscope objective, where it can swim as it is shown the VR projection. The illumination by IR LEDs is invisible to the fish and allows filming of the swimming behavior without interfering with the VR. The video is analyzed in real time to predict the swimming trajectory of the fish in order to update the projection of VR environment.

Fish in the VR showed regular swimming patterns and were attracted to animations of other zebrafish. These basic observations confirmed that fish in the VR behaved normally and reacted to complex stimuli. We then took advantage of our ability to manipulate the VR in ways that we can’t manipulate a real fish tank environment. Usually, when the fish turns in one direction, the visual image of the environment on the retina moves in the opposite direction. Here, we programmed the VR system so that when the fish turned, the environment turned the same way for a few seconds.

Fig. 2 – As the Head-fixed Normal coupling fish turns to the right, the VR environment it sees between the star and the square moves the opposite way, in the direction of the arrow, as it would in reality.  In the reversed coupling experiment, as the fish swims right, the VR environment it sees between the star and the square moves in the same direction as the fish, the opposite of what would happen in reality.

During reversed sensory-motor coupling, a substantial fraction of neurons in the forebrain showed prominent responses. This activity did not represent classical “sensory responses” or “motor responses” but it represented the mismatch between predicted and actual consequences of motor behavior. Based on experiments performed in mammals it has been suggested that the brain uses an internal model of the world to predict the sensory consequences of its own actions. Such a model must be learned and continuously updated based on experience. The finding that mismatches between the expected and the real sensory input during motor behaviors elicit strong neuronal responses in the forebrain support the notion that internal models of the world are not unique to mammals, but are also prominent in fish. Our findings thus provide a starting point to explore how a model of the world is implemented in the neuronal circuitry of the brain and how it is modified by experience. This will ultimately be important when asking whether neurological conditions may involve mismatches between predicted and actual consequences of behaviors that do not normally occur in healthy brains. It also shines light on the fundamental question of how our brain represents the world. 

We can also use the VR to study other complex behaviors. For example, would a fish attempt to join two virtual fish to its left, or one on its right? We might expect it to join the two on the left, and form a shoal for safety from predators. But what if they were made to appear further away? Such social situations wouldn’t be possible to orchestrate in a real fish tank, but this experimental set up makes this, and countless other scenarios, possible.

We are also keen to answer some key questions about the interplay between different types of neurons. Recent work indicates that during learning, both excitatory and inhibitory neurons change their interactions with other neurons. But which type of neurons are in the driver’s seat? Do excitatory neurons change first and inhibitory neurons follow, or vice versa? Or do all of neurons change in a concerted fashion? Up until now, we have primarily seen the end point of learning, but the VR now gives us the chance to follow behaviors and neuronal mechanisms in real time during a learning process, thus providing new opportunities to study how the activity of neurons translates into functions and thoughts in the brain. 


Huang, K., Rupprecht, P., Frank, T. et al. A virtual reality system to analyze neural activity and behavior in adult zebrafish. Nat Methods (2020).


Please sign in or register for FREE

If you are a registered user on Springer Nature Protocols and Methods Community, please sign in