Swansea's final home game of the season looked to be heading for a goalless draw until the Saints scored in injury time.
Ashley Williams sent the ball soaring into the air, and Rickie Lambert got above Michel Vorm to head home.
"We are disappointed with the final result, we probably would have taken 0-0 after a gritty performance, but it is really disappointing to concede like that at the end," said the left-back.
"It's not as if we can stand here and say we were beaten by a moment of brilliance.
"It was a sloppy goal to concede and very preventable.
"It was probably a typical end-of-season game. I don't think there was a lack of effort, both teams played well and created chances.
"We probably had the better chances but we couldn't take them.
"It is frustrating when a team keeps the ball and restricts you from getting it down as much as you would like.
"But we worked really hard and I thought they didn't really create that many clear-cut chances."