Japan scored the only try of the game, a brilliant 90-metre move finished off by Keito Shigeno - in front of a record crowd of 33,073
Indiscipline cost Japan as all Scotland's points came from penalties - three from Henry Pyrgos and four from Greig Laidlaw.
Scotland head coach Vern Cotter praised the character of his side to "grind out" the win despite an underwhelming performance in Tokyo.
"If we can have frustrating days and a win at the end of them all the time, I will be happy," Cotter said.
"It was a game that in the context of the end of a season and a long year, to get the win, to grind it out, I am happy.
"I am happy with the character, the way the team applied pressure and finally got the result. It could have gone the other way.
"We weathered a very difficult first half, they flew into us and lifted the intensity from last week, we felt that. It took us a while to wear them down, but we are happy with the result.
"No coach is ever going to be totally happy and I know we will find parts we need to improve on. It will be a focus next time we come together."