Hylton was born on August 26, 1934, to a farming family near Roanoke, Virginia, one of thirteen children. He came to motorsport initially as a mechanic, joining Rex White's team in the late 1950s. Working alongside White and Louis Clements, Hylton was part of the crew that won 26 races and the 1960 NASCAR Grand National championship. He moved to a crew chief role with the Ned Jarrett and Bondy Long team in 1964, a year in which they won 14 races, and again in 1965 when Jarrett claimed the Grand National championship with twelve victories.
Hylton made his first Grand National start on July 8, 1964, at Old Dominion Speedway in Manassas, Virginia, finishing 19th. He committed fully to driving in 1966, immediately finishing second in the season points standings — the best points debut by any rookie in series history — and winning the NASCAR Rookie of the Year award. He finished 1,950 points behind champion David Pearson that year.
In 1967, driving Dodges for owner Bud Hartje, Hylton again finished second in points, this time to Richard Petty despite Petty winning 27 races while Hylton won none. Over those two seasons Hylton compiled 46 top-five finishes in 87 starts through sheer consistency, fielding modest equipment in an era dominated by factory-backed operations.
From 1968 onward Hylton ran as both driver and car owner in the No. 48, a dual role he maintained for the rest of his life. He reached victory lane for the first time on March 1, 1970, at the Richmond 500. His second and final Cup win came at the Talladega 500 on August 6, 1972, under unusual circumstances: Goodyear had supplied a high-speed superspeedway tire that proved prone to shredding over long distances. Unable to afford the new compound, Hylton ran older rubber alongside similarly positioned competitor Ramo Stott, and as other cars required unscheduled stops the pair moved to the front. Hylton led 106 of 188 laps and won by less than a second over Stott.
He competed on the full Grand National and Winston Cup schedule through 1982, finishing in the top ten in points in ten of the twelve seasons between 1966 and 1977. He ranked in the championship standings as follows: second in 1966, 1967, and 1971; third in 1972 and 1975.
After stepping back from driving in 1982, Hylton continued as a team owner until 1993, fielding cars for Canadian driver Trevor Boys among others. He later competed extensively in the ARCA Racing Series into his mid-seventies, and repeatedly attempted to qualify for Daytona 500 events in the mid-2000s and early 2010s. In 2011 at Darlington he set a record as the oldest driver to start a race in NASCAR's top three series at age 76.
On April 28, 2018, Hylton, his son James "Tweet" Hylton Jr., and crew chief Terry Strange were returning from an ARCA race at Talladega when their hauler was involved in a traffic accident on Interstate 85 near Carnesville, Georgia. Both Hylton and his son died in the crash; Strange survived with severe injuries.
Hylton's career stands as a testament to consistency over firepower. Competing as a self-funded owner-driver against factory-backed teams, he finished second in the championship three times and accumulated more than 300 top-ten results across two decades of racing. His record points finish as a rookie remains unmatched in the sport's history.