Uninstall Visual Studio extension does not work

I've created a Visual Studio extension which will be installed as part of a MSI. In WiX I use theVsixPackage element in order to install the VSIX. The installation works fine and the extension is installed properly in VS but when I run the uninstaller the extension is not removed.

Looking in the MSI log file I see that WiX just incorporates the vsixinstaller in order to install the VSIX but fails on the uninstall. I managed to narrow down the problem into a simple VS extension test project and by running the vsixinstaller commands myself:

VSIX installation:

vsixinstaller /skuName:Ultimate /skuVersion:12.0 VSIXProject1.vsix /admin

Log:

3/4/2015 10:35:02 AM - Microsoft VSIX Installer
3/4/2015 10:35:02 AM - -------------------------------------------
3/4/2015 10:35:02 AM - Initializing Install...
3/4/2015 10:35:03 AM - Extension Details...
3/4/2015 10:35:03 AM -  Identifier      : VSIXProject1.Foo.760a65de-0e2d-4849-82e8-474e40d6a614
3/4/2015 10:35:03 AM -  Name            : VSIXProject1
3/4/2015 10:35:03 AM -  Author          : Foo
3/4/2015 10:35:03 AM -  Version         : 1.0
3/4/2015 10:35:03 AM -  Description     : Empty VSIX Project.
3/4/2015 10:35:03 AM -  Locale          : en-US
3/4/2015 10:35:03 AM -  MoreInfoURL     :
3/4/2015 10:35:03 AM -  InstalledByMSI  : False
3/4/2015 10:35:03 AM -  SupportedFrameworkVersionRange : [4.5,)
3/4/2015 10:35:03 AM -
3/4/2015 10:35:03 AM -  Supported Products :
3/4/2015 10:35:03 AM -      Microsoft.VisualStudio.Pro
3/4/2015 10:35:03 AM -          Version : [12.0]
3/4/2015 10:35:03 AM -
3/4/2015 10:35:03 AM -  References      :
3/4/2015 10:35:03 AM -
3/4/2015 10:35:24 AM - The extension with ID 'VSIXProject1.Foo.760a65de-0e2d-4849-82e8-474e40d6a614' is not installed to Microsoft Visual Studio Ultimate 2013.
3/4/2015 10:37:14 AM - The following target products have been selected...
3/4/2015 10:37:14 AM -  Microsoft Visual Studio Ultimate 2013
3/4/2015 10:37:14 AM -
3/4/2015 10:37:15 AM - Beginning to install extension to Microsoft Visual Studio Ultimate 2013...
3/4/2015 10:37:16 AM - Install to Microsoft Visual Studio Ultimate 2013 completed successfully. The extension has been installed to C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 12.0\\COMMON7\IDE\EXTENSIONS\HH5OTBMV.V2J\

VSIX uninstall:

vsixinstaller /u:"VSIXProject1.Foo.760a65de-0e2d-4849-82e8-474e40d6a614" /admin

Log:

3/4/2015 10:39:48 AM - Microsoft VSIX Installer
3/4/2015 10:39:48 AM - -------------------------------------------
3/4/2015 10:39:48 AM - Initializing Uninstall...
3/4/2015 10:39:51 AM - Found installed product - Microsoft Visual Studio Ultimate 2013
3/4/2015 10:39:51 AM - Extension Details...
3/4/2015 10:39:51 AM -  Identifier      : VSIXProject1.Foo.760a65de-0e2d-4849-82e8-474e40d6a614
3/4/2015 10:39:51 AM -  Name            : VSIXProject1
3/4/2015 10:39:51 AM -  Author          : Foo
3/4/2015 10:39:51 AM -  Version         : 1.0
3/4/2015 10:39:51 AM -  Description     : Empty VSIX Project.
3/4/2015 10:39:51 AM -  Locale          : en-US
3/4/2015 10:39:51 AM -  MoreInfoURL     :
3/4/2015 10:39:51 AM -  InstalledByMSI  : False
3/4/2015 10:39:51 AM -  SupportedFrameworkVersionRange : [4.5,)
3/4/2015 10:39:51 AM -
3/4/2015 10:39:51 AM -  Supported Products :
3/4/2015 10:39:51 AM -      Microsoft.VisualStudio.Pro
3/4/2015 10:39:51 AM -          Version : [12.0]
3/4/2015 10:39:51 AM -
3/4/2015 10:39:51 AM -  References      :
3/4/2015 10:39:51 AM -
3/4/2015 10:39:51 AM - Uninstalling 'VSIXProject1', version 1.0.
3/4/2015 10:39:51 AM - System.IO.IOException: The process cannot access the file because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.__Error.WinIOError()
   at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)
   at Microsoft.VisualStudio.ExtensionManager.ExtensionManagerService.AtomicallyDeleteFiles(IEnumerable`1 filePaths, Boolean justMarkForDeletion)
   at Microsoft.VisualStudio.ExtensionManager.ExtensionManagerService.DeleteDiscoverableFiles(IInstalledExtension extension)
   at Microsoft.VisualStudio.ExtensionManager.ExtensionManagerService.UninstallInternal(IInstalledExtension extension, Boolean forceDelete)
   at Microsoft.VisualStudio.ExtensionManager.ExtensionManagerService.Uninstall(IInstalledExtension extension)
   at VSIXInstaller.App.UninstallExtensionFromSKU(String vsixID, SupportedVSSKU sku)

The error message says that there's a file in use but I cannot determine what's causing this. Visual Studio is not open during the uninstall and I tried using the tool 'unlocker' to see if there're other processes causing a file lock which there aren't.

Also note that the extension is installed using the /admin flag which means that the extension is installed to <Visual Studio installation folder>\Common7\IDE\Extensions. This is needed since my installer runs with elevated privileges as well.

When I run vsixinstaller without /admin I don't get this problem. So it looks like some kind of user rights issue. On my machine I'm logged in as local admin.