Hylton was born on August 26, 1934, on a family farm near Roanoke, Virginia, one of thirteen children. He began his association with motorsport in the late 1950s as a mechanic rather than a driver, joining Rex White's operation. That team, with Hylton contributing as a mechanic alongside Louis Clements, won 26 races and the 1960 NASCAR Grand National championship. In 1964, Hylton moved to the crew chief role for the Ned Jarrett and Bondy Long team, which won 14 races that season and claimed the 1965 Grand National championship with twelve additional wins.
Hylton made his Grand National driving debut on July 8, 1964, at the Old Dominion 400 in Manassas, Virginia, finishing 19th. His breakthrough came in 1966, when he finished second in the championship standings — just 1,950 points behind David Pearson — and won the series Rookie of the Year award. He finished second in points again in 1967, this time behind Richard Petty despite having no wins compared to Petty's 27, and repeated a second-place championship finish in 1971. Hylton holds the record for the highest points finish by a NASCAR rookie.
Racing as both driver and car owner from 1968 onward, Hylton found his way to victory lane for the first time on March 1, 1970, at the Richmond 500, driving the No. 48 Ford.
His second and final Winston Cup win came at the Talladega 500 on August 6, 1972, under notable circumstances. Goodyear had supplied teams with a new superspeedway tire designed for the extreme speeds at Talladega, but the compound proved to shred over long distances. Hylton's team could not afford the new tires and ran the older specification. As competing teams were forced to pit for tire changes, Hylton and fellow independent Ramo Stott, who also ran the older tires, worked through the field. Hylton won by less than a second, leading 106 of 188 laps and earning $24,865.
Hylton drove the full NASCAR schedule through 1982, finishing in the top ten in the championship standings in ten of the twelve seasons between 1966 and 1977. He continued as a car owner in the Winston Cup through 1993.
Hylton's later career became notable for his sustained attempts to continue racing at an advanced age. In 2006, he returned to compete in the Busch Series at Milwaukee, making him the oldest driver to start in a top-level NASCAR event. He attempted to qualify for the Daytona 500 multiple times in his seventies, coming close in 2007 before a clutch problem cost him in the qualifier. In 2011, at the age of 76, Hylton set a record as the oldest driver in history to start a NASCAR race in any of the top three divisions, competing in the ARCA series at Darlington. He holds the additional mark of being the oldest driver to finish a race in NASCAR's top three series, achieved at Daytona in the Nationwide Series in 2008 at age 73.
On April 28, 2018, Hylton and his son James "Tweet" Hylton Jr. were killed when the team's hauler was involved in a traffic accident on Interstate 85 near Carnesville, Georgia, while returning from an ARCA race at Talladega. Crew chief Terry Strange survived the crash with severe injuries. Hylton was 83 years old.
James Hylton's career represents a rare combination of longevity, consistency, and competitive achievement maintained almost entirely as an independent owner-driver with limited resources. His three second-place championship finishes without the backing of major factory teams or large-budget sponsors placed him among the most accomplished independents in NASCAR history.