Nowadays media overload is a pretty common scenario all around the world. The prevalence of media overload grants both individuals and governmental entities the ability to shape public opinions, highlighting the need to deploy effective fake news detection methods. In this paper, we suggest a novel model named GraMuFeN, for detecting fake news that has been posted by users on Twitter and Weibo. This model has been designed to detect fake news using both textual and image data accompanying each piece of news. We utilize Graph Convolution Neural Networks (GCN) as the text encoder and Convolutional Neural Networks (CNN) as the image encoder with the help of Supervised Contrastive Loss aiming to develop a model much lighter in terms of trainable parameters and easier to train while having a higher performance compared to previous works. Our evaluations on two different benchmarks show a promising 10% improvement in micro f1 score and a 50% reduction in terms of the model’s trainable parameters.