Jimmy Cargill was born in Arbroath, Scotland. He played junior football for Arbroath Roselea and Arbroath Woodside, and made an appearance for Arbroath in the 1933-34 Division Two season as a trialist, making a good impression in a 4–2 win against Dundee United.
Ahead of the 1934–35 season, he went south and signed professional forms with Nottingham Forest of the English Second Division. He made only 10 league appearances in two years before dropping down a division to join Brighton & Hove Albion, for whom he scored 20 goals from 70 appearances in all competitions.
He moved on to Barrow, and made his Third Division North debut on 2 September 1939, playing “a capital game” and scoring in a 2–2 draw with Bradford City. War was declared the following day and league football was abandoned for the duration.