Japan s top-ranked player reeled off eight consecutive games at one stage to power into the quarter-finals.
It all seemed to be going to plan for second seed Williams when she broke the 19-year-old Osaka in the fifth game of the opening set.
But then the American s first serve deserted her when serving for the set at 5-4.
Two double faults and a series of fierce ground strokes from Osaka which unerringly kissed the lines enabled the world number 64 to break twice and take the set 7-5.
“She played well,” a disappointed 37-year-old Williams told reporters.
“You know, I made a few errors at 5-4 and after that she played pretty flawless. I can only give her credit.”
Osaka, who made headlines when she knocked Angelique Kerber out of the US Open in the first round last month, continued in the same vein at the start of the second set and raced into a 5-0 lead.
“I felt like I played offensively and hit really deep hard balls but she had the luck today and could return those balls even harder and deeper,” said Williams.
Williams briefly rallied to 5-2, but after a pep talk from Osaka s coach, the Japanese refocused and closed out the match on her serve at the second time of asking after an hour and 24 minutes.
“Venus is someone I ve respected and admired,” said Osaka, who was not born when Williams reached her first Grand Slam final at the 1997 US Open.
“I ve grown up watching her. Even though she someone I admire, it s just another opponent at the end of the day so I tried to focus hard.”
Earlier, the seventh seed Daria Gavrilova battled into the quarter-finals with a 6-1, 2-6, 6-2 win over American Shelby Rogers.
Rogers and Gavrilova renewed their rivalry at Victoria Park after setting a record for the longest-ever women s singles match last month, with a three hours and 33 minute epic at the US Open.
“I was really motivated,” said world number 22 Gavrilova after avenging her Flushing Meadows defeat. “She has such a great serve I knew I had to take my chances on her second serve.”
Rogers had come out on top 7-6, 4-6, 7-6 in their marathon New York encounter but on this occasion Gavrilova raced to victory in the first set 6-1 in just 26 minutes.
Rogers, ranked 55, hit back immediately to take the second 6-2 in a scrappy encounter littered with errors.
With both players coming under pressure on their serve in the final set, it seemed a matter of who would crack first.
And at 2-2 and 0-30 it was Rogers who blinked, serving back-to-back double faults to hand the Russian-born Australian the crucial break.
Another followed and the Australian closed out the match in an hour and 44 minutes.
Sam Stosur ensured there would be two Australians in the last eight when she came from a set down to oust fourth seed Agnieszka Radwanska in the final match of the evening 3-6, 6-4, 6-0.
The Polish world number 18 started strongly and was a set and break up before Stosur struck back to level at 3-3 in the second.
From then on the Australian constantly pressured the Radwanska serve, and a second break in the ninth game of the second set also broke the Pole s resolve.
Stosur, the 2011 US Open champion, raced through the final set to love to take her career record against Radwanska to four wins in five meetings.
“I feel I played really well and aggressively,” said Stosur. “Everything I tried came off. Really happy to be in the quarter-finals.”