After college Shergill formed a band named Kaffir. The band played in some competitions and college festivals before parting ways.Shergill initially composed jingles for advertisement agencies such as Yamaha RX-T motorbikes and Times FM.He had an unsuccessful stint with Sony Music and Tehelka before finally releasing his debut album Rabbi in 2004 under Phat Phish Records. Relying on word-of-mouth publicity and a music video, he had a chart topper song “Bulla Ki Jaana”. Most of the songs in the album were composed and written by Shergill himself except for “Bulla ki Jana” based on the poetry of 18th century Muslim Sufi mystic Baba Bulleh Shah, “Heer” from Heer by Waris Shah and “Ishtihar” by Shiv Kumar Batalvi. Shergill had one song ‘Dilli’ in the Hindi movie, Delhii Heights.
In October 2008, Shergill released his second album Avengi Ja Nahin. under Yashraj Music.The album contains nine songs and deals with issues like communal violence, social responsibility and the need for “collective morality”. He also appeared in MTV Unplugged (India) in 2011. He lent his voice to Yash Chopra’s 2012 romantic film Jab Tak Hai Jaan singing the leading number Challa composed by A. R. Rahman and the lyrics penned by Gulzar. In March 2012, he released his third album