Four* (NYT)
There have been several less public visits to the historic naval base:
Yoshida (1951)

Yoshida (center right) in Honolulu in Aug. 31, 1951. From AP.
"Japanese Leader’s Pearl Harbor Visit May Not Be a First, After All", New York Times:
Prime Minister Shigeru Yoshida stopped in Hawaii in 1951 on his way home to Tokyo from San Francisco, where he had signed a treaty to normalize relations between Japan and most of the victorious Allies of World War II.
During his brief time on the island of Oahu, he paid a formal visit to the National Memorial Cemetery of the Pacific, a mountaintop shrine dedicated to American war dead, and, it now appears, made a less public stop at Pearl Harbor.
A search of Japanese newspaper archives turned up a 1951 dispatch from the daily Yomiuri Shimbun. The newspaper reported that the premier had indeed gone to the American naval base at Pearl Harbor.
No ceremonies or other public events appear to have taken place while Mr. Yoshida was at the base.
Hatoyama (1956) and Kishi (1957)

Hatoyama at Pearl Harbor, taken from a Hawaii Hochi article in Oct. 30, 1956. From the Yomiuri Shinbun.

Hawaii Hochi article, June 29, 1957, stating that Kishi has visited Pearl Harbor. Notice text highlighted. From the Yomiuri Shinbun.
"2 more Japanese PMs ‘visited Pearl Harbor’", Yomiuri Shinbun:
The Hawaii Hochi, a local Japanese-language daily, said Thursday that its past reports show then Prime Ministers Ichiro Hatoyama and Nobusuke Kishi visited Pearl Harbor in 1956 and 1957, respectively.
Hatoyama visited Honolulu in October 1956 after he signed the Japan-Soviet joint declaration in Moscow.
Kishi stayed in Honolulu two nights in June 1957 after a meeting with then U.S. President Dwight Eisenhower, according to the past reports.
The visits by Hatoyama and Kishi are believed to be official ones, as they received ceremonial welcomes, including gun salutes, according to the Hawaii Hochi.
Abe (2016)

Abe and Obama at the USS Arizona Memorial, Pearl Harbor on Dec. 27, 2016. From the BBC.
"Japan PM Shinzo Abe offers Pearl Harbor condolences", BBC:
Japanese PM Shinzo Abe has visited the US naval base at Pearl Harbor, where he offered "sincere and everlasting condolences" to the victims of Japan's attack on the base 75 years ago.
Mr Abe was accompanied by US President Barack Obama, making the visit the first by the leaders of both countries.
Further notes
The Pompitous makes a note that other Japanese officials may have visited Pearl Harbor unofficially. This would include Yoshida's. In addition, there may be other visits which I will have missed and that have only been covered by local or Japanese newspapers.
This answer will change as future Japanese officials may choose to visit Pearl Harbor, so this list isn't final.