Local authorities in Japan have demanded the removal of a monument in memory of the tens of thousands of labourers forcibly recruited from the Korean peninsula during the second world war.

The monument, dedicated to Koreans who died after being brought to Japan to work in coalmines and factories amid a wartime labour shortage, was erected by a friendship society in a public park in 2004.

The government in Gunma prefecture, north-west of Tokyo, has ordered it removed after what the Asahi Shimbun newspaper described as petitions from "anti-Korean" groups and individuals complaining that it was anti-Japanese and had become the focus of political activity in a publicly owned space.

Part of the inscription, written in Japanese and Korean, reads: "We hereby express our determination not to repeat the same mistake by remembering and reflecting on the historical fact that our country inflicted tremendous damage and suffering on Koreans in the past."

Many of the 600,000 ethnic Koreans living in Japan are the descendants of labourers who remained in the country after its defeat in 1945.

The Gunma assembly adopted a resolution to remove the monument after accepting criticism that it had been the focus for political activity during a memorial service in 2012. Prefectural authorities said they would refuse to extend the monument's 10-year licence if the friendship society failed to remove it voluntarily.

Giichi Tsunoda, a former upper house MP and representative of the group that commissioned the monument, said the officials had acted unreasonably. "The gathering is a memorial event and the prefectural government's decision to remove the monument is tantamount to abusing its authority," Tsunoda said. "Its conclusion that the gathering was politically motivated is also arbitrary."

Commemorating the use of forced labour is causing similar controversy in other parts of Japan. In Nagasaki, a row has erupted over a proposed monument to Korean victims of the atomic bombing of the city in August 1945. A draft text of the epitaph reportedly condemns imperial Japan for its use of slave labour.

The controversies are being played out against a rise in anti-Korean sentiment in Japan, fuelled by disputes between Tokyo and Seoul over territory and Japan's conduct during its 1910-1945 occupation of the Korean peninsula.

The UN human rights committee has called on Japan to do more to clamp down on hate speech directed at Koreans during demonstrations in Tokyo and other cities. The committee, which noted that there were more than 360 such demonstrations and speeches last year, mainly in Korean neighbourhoods in Tokyo, is expected to make recommendations to Japan on Wednesday, possibly including the introduction of legislation against hate speech.

On Wednesday, officials from both countries met in Seoul to discuss Japan's use of as many as 200,000 mainly Korean and Chinese women as sex slaves before and during the war. Japan recently ruled out a revision to a 1993 official apology, but suggested that the statement was the result of a political compromise and not an accurate reflection of Japanese involvement in wartime sex slavery.

The prime minister, Shinzo Abe, is one of several prominent conservative politicians who have questioned claims that the imperial Japanese army coerced the women – euphemistically referred to as comfort women – into working in frontline brothels.