They say legal proceedings will now be conducted in another country.
Afghan MPs have demanded that the man be tried in Afghanistan. Correspondents say that scenario is very unlikely.
Presidential aides say Kabul was informed at the highest level about the moving of the US soldier from Kabul to Kuwait.
There have been demonstrations demanding an Afghan trial for the man. Protesters also warned Mr Karzai not to sign a strategic partnership with the Americans.
There have been calls for the soldier’s trial to be held in Afghanistan and be made public. But Kabul signed an agreement with the US and other Nato members under which soldiers will be tried in their home countries, not Afghanistan.
Mr Panetta has said that if found guilty, he could face the death penalty.
Officials said the soldier had completed several tours in Iraq but was on his first tour of duty in Afghanistan.