�鶹��ý

, assistant professor of computer science in Pitt's �鶹��ý of Computing and Information, recently received an to develop a framework to train computer vision models for the detection of objects from weak, naturally-occurring supervision in the form of text or speech language and additional multimodal signals.

The framework considers dynamic settings, where humans interact with their visual environment and refer to the encountered objects, such as: “Carefully put the tomato plants in the ground,” and “Please put the phone down and come set the table,” as well as captions written for a human audience to complement an image, like news article captions.

The project will benefit society by exploring novel avenues for overcoming this challenge and reducing the need for expensive and potentially unnatural crowdsourced labels for training.