Using Convolutional Neural Networks to Classify Magnetoencephalography(MEG) Data into Musicians and Non-Musicians


In Winter 2019, I took a Deep Learning course taught by Prof. Honglak Lee of the Computer Science and Engineering department at the University of Michigan. I worked with Kunyi Lu, Mingyu Yang, Congrong Zhou and Jeffrey Dominic on using deep learning to classifying Magnetoencephalography (MEG) data of brains of Musicians and Non-Musicians. We worked with Dr. Kanad Mandke, a research associate at the University of Cambridge in UK on this project.  A detailed paper is being written as this is the first known attempt at classifying brain scans of musicians and non-musicians in resting state.

In this project, we seek to classify the brain MEG scans of musicians and non-musicians,
taken while subjects are in a resting state. Due to the superior practice and skill developed by musicians over years, their brain networks have different characteristics from non-musicians. Understanding these characteristics would help neuroscientists in understanding the process of neuroplasticity. However, it is not a easy task since traditional univariate analysis is not useful when there isn’t a prior model (linear) available. To solve this problem, we develop two Convolutional Neural Networks (CNN) based models that produces reasonable test accuracy. Besides, our methods are capable to find key brain regions distinguish musicians from non-musicians. This is the first known attempt at classification of resting state brain network profiles of musicians and non-musicians.
The dataset we used is publicly available as part of the Open MEG Archive (OMEGA). We had the wonderful opportunity to work with Dr. Kanad Mandke, a research associate at the University of Cambridge. Dr. Mandke during his doctoral research at the University of Nottingham (2014-18) had used the data to identify multi-layer, resting state network differences between musicians and non-musicians.

Magnetoencephalography Data

The OMEGA data consists of 1D time domain scans of 62 participants: 31 musicians and 31 non-musicians. Each scan has a length of 5 minutes, and was taken while subjects were in a resting state with a sampling rate of 600Hz. The MEG signals are split
into the 5 standard frequency bands: delta, theta, alpha, beta and gamma, followed by leakage correction and Hilbert envelope computation.

A 1D Time series data of an MEG scan for 5 minutes.


Traditionally, different parts of the brain are analysed using mass-univariate analysis where the underlying assumption is that the correlation between different regions in the brain is not informative. In such analysis, data is fit to a model using a least squares approach or similar approaches, and an estimate of the parameters of the model is obtained. The covariance across different regions in the brain is assumed to be uncorrelated noise in this analysis. This kind of analysis for feature selection suffers from 2 limitations:
  1. A linear model is assumed
  2. Noise is assumed to be uncorrelated.

When studying the brain connectivity profiles of musicians and non-musicians, these assumptions may not be appropriate for performing a classification between the 2 groups. This is because we do not have a linear model with which we can extract the features for the brain connectivity profiles. In addition, there may be distributed activity in the brain that would require multivariate analysis approach. Therefore, the subtle differences that may exist between the connectivity profiles of musicians and non-musicians might easily be overlooked using a classical feature selection approach.

In order to capture these subtle differences, we chose to implement a CNN to perform the classification task as it does not require any assumptions of the model and would be able to capture the neural activity across different regions in the brain. Furthermore, CNN based classification is robust to noise in data as opposed to univariate analysis.


Adjacency Matrix

Since the original time series data is long (5*30*600 = 90000) and not convenient for CNN, we also generated adjacency matrix to show the connectivity of all 78 regions from the time series data. An adjacency matrix refers to a square matrix used to represent a finite graph. The elements of the matrix indicate whether pairs of vertices are adjacent or not in the graph. In our special case of the finite graph, the adjacency matrix is a matrix with zeros on its diagonal. In order to obtain the adjacency matrix, we stack the time-series data of all the regions (78 in total) per subject together and compute pairwise linear correlation coefficients between each pair of columns of the preprocessed data . Thus, we transform a 5*90000 input into 5 * 78 * 78 , which is better for CNN implementation.


adjacency matrix

Adjacency Matrix obtained using last 2 minutes of the MEG scan

Graph Based Deep Learning Model:


Our graph based model

To classify between musicians and non-musicians, we formulated it as a graph classification problem. As we know, brain connectivity profile of a musician is different from that of a non-musician, which can be leveraged as a key feature to solve this classification problem. Therefore, we built a graph based deep learning network.

The input of the model is a graph representation of a brain: adjacency matrix. At first, we used an atlas that divides the brain into 78 regions. Then, we computed correlation between each two regions. To represent the brain as a graph, we treated each region as a node and treated the correlation as the weight of an edge. After getting the adjacency matrix, we used Node2Vec algorithm to embed each node in one graph to a 1-d dimension vector. Therefore, we can derive a 78 d_{embedding} feature map from a 78 * 78 adjacency matrix, where d_{embedding} denotes the dimension of 1-d embedding vector. For the purpose of data augmentation, we applied filters to the time series signals. For one person’s brain, we can use each frequency band to compute an adjacency matrix. Therefore, each person’s brain can formulate several adjacency matrices corresponding to the number of filters(we have 5 frequency bands in our project). To integrated the information in all frequency bands, we used an average pooling layer among the frequency band dimension for all feature maps from one person’s brain. Eventually, we passed the feature map to one convolution layer, two fully connected layers and one Sigmoid layer. The output is the probability of classifying to musicians. If the probability is larger than 0.5, the classification result is a musician. Otherwise, the classification result is a non-musician.

Time Series Classification

Although an adjacency matrix presents useful connectivity information that can help with our task, it is inherently static data. Our brains are highly dynamic systems and, as a result, the oscillations of brain regions are also highly dynamic. After a discussion with Dr. Mandke, we learned that two different brain regions generally oscillate out of phase with each other. However, when these regions communicate with each other, their oscillations rapidly synchronize and become in phase. Such changes in neural oscillations are extremely important to account. Consequently, what may look like noise in an MEG signal is actually very important data that could greatly help with classification. We used the famous EEGNet model for implementation.

Since the raw time series data captures these changes, we experimented with using this data directly for classification.

Each 5-minute MEG signal is split into five frequency bands: delta, theta, alpha, beta, and gamma. We downsampled the time series data by different sampling rates to produce input data with equal length. The data is formatted as a 5 x 78 x T tensor, where T is the length of the data, and passed to the EEGNet. 



Following are the results of the accuracy we got for different models. A paper currently is being written and which will include thorough details on analysis of the same.

Screen Shot 2019-06-30 at 5.31.06 PM.png


Screen Shot 2019-06-30 at 5.32.08 PM.png


This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s