Multimodal Learning

Combining vision with other modalities, from generative sketch synthesis to molecular property prediction.

(Lim et al., 2024) (Tirupattur et al., 2018) (Adak et al., 2025) (Baranwal et al., 2026) (McIntosh et al., 2020) (Sharma et al., 2021) (Quintanilla et al., 2020)

References

2026

MolSight: Molecular Property Prediction with Images

Aaditya Baranwal, Akshaj Gupta, Yogesh S Rawat, and Shruti Vyas

arXiv preprint arXiv:2605.10157, 2026

arXiv Bib

@article{baranwal2026molsight,
  title = {MolSight: Molecular Property Prediction with Images},
  author = {Baranwal, Aaditya and Gupta, Akshaj and Rawat, Yogesh S and Vyas, Shruti},
  journal = {arXiv preprint arXiv:2605.10157},
  year = {2026},
}

2025

MolVision: Molecular Property Prediction with Vision Language Models

Deepan Adak, Yogesh Singh Rawat, and Shruti Vyas

In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2025

NeurIPS arXiv Bib Supp Code Website

@inproceedings{adak2025molvision,
  title = {MolVision: Molecular Property Prediction with Vision Language Models},
  author = {Adak, Deepan and Rawat, Yogesh Singh and Vyas, Shruti},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track},
  year = {2025},
}

2024

AirSketch: Generative Motion to Sketch

Hui Xian Grace Lim, Xuanming Cui, Ser-Nam Lim, and Yogesh S Rawat

In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

NeurIPS Bib

@inproceedings{lim2024airsketch,
  title = {AirSketch: Generative Motion to Sketch},
  author = {Lim, Hui Xian Grace and Cui, Xuanming and Lim, Ser-Nam and Rawat, Yogesh S},
  booktitle = {The Thirty-eighth Annual Conference on Neural Information Processing Systems},
  year = {2024},
}

2021

NoisyActions2M: A Multimedia Dataset for Video Understanding from Noisy Labels

Mohit Sharma, Raj Patra, Harshal Desai, Shruti Vyas, Yogesh Rawat, and Rajiv Ratn Shah

In ACM International Conference on Multimedia in Asia, 2021

MMAsia Bib PDF Website

@inproceedings{sharma2021noisyactions2m,
  title = {NoisyActions2M: A Multimedia Dataset for Video Understanding from Noisy Labels},
  author = {Sharma, Mohit and Patra, Raj and Desai, Harshal and Vyas, Shruti and Rawat, Yogesh and Shah, Rajiv Ratn},
  booktitle = {ACM International Conference on Multimedia in Asia},
  year = {2021},
}

2020

Visual-textual Capsule Routing for Text-based Video Segmentation

Bruce McIntosh, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah

In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020

CVPR Bib PDF

@inproceedings{mcintosh2020visual,
  title = {Visual-textual Capsule Routing for Text-based Video Segmentation},
  author = {McIntosh, Bruce and Duarte, Kevin and Rawat, Yogesh S and Shah, Mubarak},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages = {9942--9951},
  year = {2020},
}

Adversarial Learning for Personalized Tag Recommendation

Erik Quintanilla, Yogesh Rawat, Andrey Sakryukin, Mubarak Shah, and Mohan Kankanhalli

IEEE Transactions on Multimedia, 2020

TMM Bib

@article{quintanilla2020adversarial,
  title = {Adversarial Learning for Personalized Tag Recommendation},
  author = {Quintanilla, Erik and Rawat, Yogesh and Sakryukin, Andrey and Shah, Mubarak and Kankanhalli, Mohan},
  journal = {IEEE Transactions on Multimedia},
  year = {2020},
}

2018

ThoughtViz: Visualizing Human Thoughts Using Generative Adversarial Network

Praveen Tirupattur, Yogesh Singh Rawat, Concetto Spampinato, and Mubarak Shah

In Proceedings of the 2018 ACM on Multimedia Conference, 2018

ACM MM Bib

@inproceedings{tirupattur2018thoughtviz,
  title = {ThoughtViz: Visualizing Human Thoughts Using Generative Adversarial Network},
  author = {Tirupattur, Praveen and Rawat, Yogesh Singh and Spampinato, Concetto and Shah, Mubarak},
  booktitle = {Proceedings of the 2018 ACM on Multimedia Conference},
  pages = {950--958},
  year = {2018},
  organization = {ACM},
}