Born in Cardiff, Peter Sayer began his career at his hometown club Cardiff City making his debut against Hull City in September 1974. He went on to play a further nine matches in his first season before suffering a broken ankle in a match against Southampton which kept him out for nearly a year, returning at the end of the following season to play a handful of games as Cardiff clinched promotion to Division Two. He established himself in the side the following year and helped the club to avoid relegation, as well as scoring the only goal of the game when Cardiff knocked Tottenham Hotspur out of the FA Cup in 1977.
Peter left Cardiff midway through the 1977–78 season for £100,000 to sign for Brighton & Hove Albion. He helped the Seagulls win promotion to Division One in his first year but, after starting the 1978–79 season as a member of the first-team, Sayer found himself replaced in the side by Mark Lawrenson. He remained at the Goldstone until the end of the season when he was sold to Preston North End for £85,000.
After finding it difficult to break into the side, he was loaned back to Cardiff in September 1981, playing four times and scoring during a 3–2 win over Luton Town. He eventually ended his league career with Chester City.