Researchers have developed a smartphone app that uses machine learning to accurately determine body composition – which is linked to a higher risk of heart disease, stroke and related conditions – from photographs.

The researchers, from the University of Cambridge, developed the smartphone app called 3D BodyShape, which is currently a research prototype and will soon be available for iOS and Android devices. The app incorporates a machine-learning algorithm, which was trained on detailed medical imaging data from more than 12,000 adults.

From just four smartphone photographs, the app constructs a three-dimensional model of the human body from which it can determine body composition – how and where fat and muscle tissues are distributed across the body with a high level of accuracy.

The results are reported in the journal npj Digital Medicine.

Body composition is a strong predictor of diabetes and heart disease. Those with an ‘apple-shaped’ body, or with low muscle mass in the upper arms and lower legs, are at higher risk of metabolic conditions such as diabetes, heart disease, stroke and certain cancers.

The researchers say that while the app is not intended as a substitute for medical care, it could give individuals the ability to monitor changes in their health and associated risks, and make necessary lifestyle adjustments.

Cardiovascular disease – such as heart disease and stroke – is the leading cause of death worldwide, and many of these deaths are associated with poor metabolic health. One third of the world’s adult population suffers from conditions related to metabolic health, including obesity.

“Poor metabolic health is highly correlated with body shape and composition – how and where fat and muscle is distributed on the body, but it’s also strongly correlated with poor quality of sleep and high stress levels,” said Professor Roberto Cipolla from Cambridge’s Department of Engineering, who led the research. “Genetics also plays a role, but many diseases are preventable with lifestyle and behaviour.”

Body mass index, or BMI, is commonly used to estimate obesity, but since BMI does not distinguish between fat and muscle, it’s not as strong an indicator of health as measurement methods that do make this distinction

“In clinical assessments, we use more accurate methods of body composition, but these are expensive and only available in research or healthcare facilities, and so are not suitable for regular health monitoring,” said co-author Dr Emanuella De Lucia Rolfe from the Medical Research Council (MRC) Epidemiology Unit. “This new tool has the potential to provide similar information for everybody with a smartphone whenever they want it and for free.”

The researchers used more than 20,000 DXA (dual energy X-ray absorptiometry) scans from 12,000 Fenland Study participants, showing detailed body composition and how it changed over time. The study contains weight, height, hip and waist ratios, as well as body fat percentage, and several other health parameters for every participant, who were between 30 and 65 years of age when they joined the study.

This dataset was used as training data for a machine learning algorithm to determine body composition from 3D body shape alone. The researchers converted the two-dimensional DXA images into three-dimensional body shapes, or avatars.

The avatars were then used to train a deep learning network to predict body composition, including visceral abdominal fat and skeletal muscle. Results showed that the model was able to accurately predict body composition, and changes over time, for Fenland Study participants.

The app uses computer vision algorithms to convert four mobile phone photographs (front, back, left and right) into a 3D body shape avatar, and then into body composition. All the processing is done on the phone and no images or data are shared. Results from the app also predicted body composition with high accuracy.

“The Fenland study predominantly includes White Europeans, and we now need to test this method in other populations,” said De Lucia Rolfe. “The distribution of body fat and muscle varies by age and ethnicity, and we need to check how well we are picking these differences up with this 3D body shape method. This is our next project.”

Cipolla says that the app could be useful for people to determine their own risk and monitor their health over time, enabling them to make any necessary lifestyle changes – such as diet, physical exercise, sleep and stress management – to stay healthy and lower their risk.

“The utility of this tool may even extend beyond health care encounters,” said co-author Dr Soren Brage, also from the MRC Epidemiology Unit. “It would be better if we did not need to see our doctor in the first place, or at least less often, and being physically active plays an important role in that. If providing easily measurable health status indicators such as body fitness and fatness can motivate people to start an exercise programme and keep them engaged by allowing them to monitor impact on their health and track their progress, these tools will save lives and taxpayers’ money.”

“The app could be a form of pre-screening before seeing a doctor,” said Cipolla. “Machine learning is improving to the point where it can help make tangible improvements to peoples’ lives, and it’s gratifying if we can be one small part of that.” 

The research was supported in part by the Medical Research Council (MRC), part of UK Research and Innovation (UKRI), Wellcome, the National Institute for Health Research (NIHR), and the NIHR Biomedical Research Centre.

Reference:

Chexuan Qiao, Emanuella De Lucia Rolfe et al. ‘Prediction of total and regional body composition from 3D body shape.’ npj Digital Medicine (2024). DOI: 10.1038/s41746-024-01289-0