New computer model automatically, aesthetically crops photos

UNIVERSITY PARK, Pa. — Computers can now automatically crop photos to capture the most interesting part in an aesthetically pleasing manner, thanks to researchers at Penn State.

“When manually cropping an image, humans identify the most interesting or eye-catching part of the photo — even if it is off-centered — and crop the photo to frame it, but existing tools crop photos down the middle of the image, with no consideration of whether that section contains a visually significant part of the photo,” said Jia Li, professor of statistics, who, along with her colleague James Wang, professor of information sciences and technology, recently received U.S. patent No. 10,186,040 for the model. “We have solved this problem by creating a statistical model that quickly identifies the visual significance of each segmented part of a photo and crops the overall photo in an aesthetically pleasing manner.”

According to the researchers, the software could be used in smartphones, on social media and on internet browsers, where photos traditionally have been automatically cropped in the middle, often missing the main subject or yielding an aesthetically displeasing result.

The team modeled the amount of time that a human eye is likely to linger on a particular segment of an image using a Markov chain, which is a statistical model. The attention paid to any segment depends on its individual attributes such as color, texture and brightness, as well as its spatial relationship with other segments. The researchers encoded all of these different types of information within the parameters of the Markov chain. The statistical model then ranks the segments, with the highest significance assigned to the segment that retains the eye the longest. This ordering helps the model to determine the best location to crop. The decision also takes into account the aesthetic quality of the cropped image.

“Spatial arrangement of the segments in an image plays a major role in determining significance,” said Li. “The model considers how the segments interact spatially, without assuming that the geometric center of the image is necessarily more important. For example, two segments with the same appearance can have rather different significance scores depending on their placement with respect to the rest of the picture.”

According to Li and Wang, the model works quickly to crop photos and requires very little computing power.

“Our code takes around 50 milliseconds to process an image of 400 by 300 pixels on a common desktop computer or laptop,” said Wang. “As a result, our method is suitable for devices with limited memory and computing power.”

The team hopes to market its method to internet search engine and mobile phone companies, among others.

“By cropping photos in a more aesthetically pleasing manner, a number of photo-related applications can be improved, especially when the screen size is small, such as in phones, watches or tablets,” said Wang. “In addition, the model may help visual ads companies improve their click-through rates.”

###