The win came in the first leg of their Champions League last-16 tie.
Substitute Marko Arnautovic scored the winner in the closing stages, firing home the rebound of a Lautaro Martinez shot that Atletico keeper Jan Oblak had parried.
Austria forward Arnautovic, who came on at half time after Marcus Thuram took a knock, had squandered three good opportunities to score before he found the net and he was not the only one to miss chances.
“I have a very good feeling after the match about the performance the lads gave against a physical, skilful team,” Inzaghi told Mediaset television.
“There is regret about having won by just one goal.
“We have won the first half [of the tie], but the return leg will be very difficult.”
ANSA