Prince Charles rarely visits the Cabinet Office, the heart of U.K.'s government operations. But when he did come by recently, he could not help but crack a joke about Brexit.

The Prince of Wales was with his wife, Camilla, the Duchess of Cornwall, during his rare visit to the center of the government on Thursday, Feb. 13, and they were given a tour of the Cabinet Office. While there, however, Prince Charles apparently said amusingly that he was not quite sure it would be the most sensible day to visit and talk to Prime Minister Boris Johnson.

Johnson was shuffling his cabinet during the royal visit as a result of Brexit. The heir to the throne said that he didn't want to get in the way of the PM's busy day, but apparently, the royals were told it was alright to drop in.

Prince Charles then thanked the people working at the Government Office. He added that he was very aware of their hard work, especially in the last two years, when the government was setting up to leave the European Union. The royal said, once again in jest, that he hopes the staff is now having a better sleep since Brexit is done, and there's less stress at work.

The royals also spotted the PM's famous dog, Monty. The prince asked and wondered if the pup has had his lunch.

Prince Charles stayed behind the Cabinet Office to have private meetings with several government officials. Camilla left earlier than her husband, but not before she signed autographs for the staff at the government office.

Meanwhile, earlier that day, Prince Charles and Camilla also visited the Tower of London in honor of the 535 years since the creation of Yeoman Warders. They also joined the British Tourist Authority as it celebrates its 50th year of the VisitBritain/VisitEngland initiative.

Prince Charles was reunited with the coronet (small crown) he wore during his investiture in 1969 as the Prince of Wales. Designed by Louis Osman, the coronet is made of 24-k gold with emeralds and 13 encrusted square diamonds. The jewels formed the prince's star sign of Scorpio, while the crown had the engraving of the Prince of Wales' insignia in the center.

Prince Charles was only 20 years old during his investiture ceremony, where Queen Elizabeth also bestowed his sword, ring, golden rod, and mantle, aside from the coronet.