Vien Co Mat - Tam Toa (the Secret Affairs Institute) was originally located in Phu Xuan Residence which was constructed by Lord Nguyen Phuc Khoat in 1738. In 1754, it was renamed and known as the culture and politics center in the South of Vietnam of Nguyen Lords till 1775. It was occupied by Trinh force (1775-1786) and became the capital of Tay Son Dynasty from 1786 to 1801.

In 1802, Emperor Gia Long was enthroned. After Phu Xuan Residence was demolished, the house of Dam Prince (the name of Emperor Minh Mang in his early days) was built in the area. In 1816, Dam Prince became the Crown Prince and moved to the East of the Imperial City. Prince Nguyen Phuc Chan (the younger brother of Emperor Minh Mang) and his eldest son - Nguyen Phuc Thien Khue then lived in this place. In 1839, Giac Hoang Pagoda which was one of the 20 places of interest at the time was constructed in this area.

After Giac Hoang Pagoda was demolished, the Secret Affairs Institute where French colonialists and Nguyen dynasty discussed was constructed in the area and completed in 1903. Entire Secret Affairs which is surrounded by brick walls has 2 floors and 3 compartments constructed in Western European style. While its upper floor has 5 moon-shaped gates, the lower floor has 7 moon-shaped gates. There are also 3 screens and 6-sided moon-shaped gates. The two-sided houses have 15 compartments and 2 wings.

Ever since its architecture and function have been changed.

From 1955 to 1975, the two rows of sided houses were the offices of local justice agencies. The main house (Vien Co Mat - the Secret Affairs Institute) served as the place where first-instance and superior courts took place. From 1975 to 1976, Uy ban Quan Tri Thua Thien Hue based and worked in this area. From 1976 - 1989, the Secret Affairs Institute became the office of the Party Committee of Binh Tri Thien, then the Party Committee of Thua Thien Hue province from 1989 to 2000. Since October 2000, it was under the management of Hue Monuments Conservation Center.

The Secret Affairs Institute (Vien Co Mat  - Tam Toa) was ranked as National Relic on Decision No. 99/2004/QD-BVHTT dated 15/12/2004.

In the process of formation and development of the Nguyen Dynasty, from the time of the lords to the time of the kings, the present land of Tam Toa has undergone many times of construction and renovation with architectural and functional features. different uses and names. Here are the 5 most important changes in terms of function and architectural face of this area.

1. 1738 - 1801: this was the center of the capital of the Nguyen lords, then the capital of the Tay Son Dynasty

2. Early nineteenth century: this site was built as the residence of Prince Dam (later to become King Minh Mang)

3. 1839 - 1899: Giac Hoang Pagoda

4. 1903 - 1945: Privy Institute

5. 1945 - 2000: Headquarters of a number of public agencies