Harroun was born in Spartansburg, Pennsylvania to a carpenter father. He served on a U.S. Navy coaler during the Spanish-American War before entering the automobile industry at its very beginning. In 1903, he participated in an attempt to set a land speed record with a drive from Chicago to New York, covering the distance non-stop in shifts with four other drivers to establish a record of 76 hours. In October 1904, the same team re-set the record at 58 hours and 35 minutes, a mark that stood for nearly two years.
Harroun is known to have started at least 60 AAA-sanctioned races between 1905 and 1911, winning 19. He drove primarily for the Indianapolis-based automaker Marmon from 1909 to 1911, although earlier records show cars he built himself — referred to as "Harroun Custom" and "Harroun Sneezer." His victories in this period included a 100-mile race at the Atlanta Motordrome in 1910, the 1910 Wheeler-Schebler Trophy Race and the Remy Grand Brassard Race at Indianapolis Motor Speedway, and numerous victories at Churchill Downs and Latonia Race Track. He won eight races in total at Indianapolis Motor Speedway, the second-highest total by any driver in the track's hundred-year history.
In 1927, the AAA Contest Board retroactively assigned points to pre-championship-era seasons, and Harroun was designated the national champion for 1910 under this revised framework. Historians generally regard these retroactive designations as unofficial.
The 1911 Indianapolis 500 was Harroun's most famous race and the culmination of his driving career. He had effectively come out of retirement to compete, partnering with his employer Marmon to field the Marmon Wasp — an innovative single-seater design he co-created with Howard Marmon. The car was revolutionary for being the first purpose-built open-wheel single-seat racing car to compete at the highest level.
Harroun's decision to race without a riding mechanic — required by the rules to serve as a second set of eyes — created pre-race controversy. In place of the mechanic, Harroun fitted what would now be called a rear-view mirror. Officials ultimately permitted this solution, and Harroun went on to win at an average speed of 74.602 mph, leading the race for significant stretches across 200 laps. The car — a bright yellow number 32 on Firestone tires — is preserved today at the Indianapolis Motor Speedway Hall of Fame Museum.
The 1961 Indianapolis 500, the race's 50th anniversary, was won by A.J. Foyt. Both Harroun and Foyt appeared together on the television program I've Got a Secret, their shared secret being their respective Indianapolis victories.
After retiring from racing following the 1911 season, Harroun continued engineering work for Marmon and later for the Maxwell racing team. In 1916, he founded the Harroun Motor Car Company in Wayne, Michigan, though the venture folded after World War I. He later joined Lincoln Products and continued working in the automotive industry until retiring at age 79.
Harroun lived to 89, dying on January 19, 1968, just one week after his 89th birthday. He was inducted into the Indianapolis Motor Speedway Hall of Fame in 1952, the Motorsports Hall of Fame of America in 2000, and the Michigan Motorsports Hall of Fame in 2010. A street in Wayne, Michigan is named in his honor.