Hosting an XML editor does not reload in Visual Studio properly

I have created a custom IVsEditorFactory to host an XML editor with a WPF control. I found a few examples all using an IVsInvisibleEditorManager like the following:

        Dim textBuffer As IVsTextBuffer

        If punkDocDataExisting = IntPtr.Zero Then
            Dim invisibleEditorManager = CType(Me.mServiceProvider.GetService(GetType(SVsInvisibleEditorManager)), IVsInvisibleEditorManager)

            Dim invisibleEditor As IVsInvisibleEditor = Nothing
            ErrorHandler.ThrowOnFailure(invisibleEditorManager.RegisterInvisibleEditor(pszMkDocument, Nothing, CUInt(_EDITORREGFLAGS.RIEF_ENABLECACHING), Nothing, invisibleEditor))

            Dim docDataPointer = IntPtr.Zero
            ErrorHandler.ThrowOnFailure(invisibleEditor.GetDocData(1, GetType(IVsTextBuffer).GUID, docDataPointer))

            textBuffer = CType(Marshal.GetObjectForIUnknown(docDataPointer), IVsTextBuffer)
        Else
            textBuffer = TryCast(Marshal.GetObjectForIUnknown(punkDocDataExisting), IVsTextBuffer)

            If textBuffer Is Nothing Then ErrorHandler.ThrowOnFailure(VSConstants.VS_E_INCOMPATIBLEDOCDATA)
        End If

        ErrorHandler.ThrowOnFailure(textBuffer.SetLanguageServiceID(Constants.XmlLanguageServiceGuid))

        Dim componentModel = CType(Me.mServiceProvider.GetService(GetType(SComponentModel)), IComponentModel)

        Dim editorAdapterFactory = componentModel.GetService(Of IVsEditorAdaptersFactoryService)

        Dim codeWindow = editorAdapterFactory.CreateVsCodeWindowAdapter(Me.mOleServiceProvider)

        ' Disable the splitter which causes a crash
        Dim initView = {New INITVIEW}
        ErrorHandler.ThrowOnFailure(CType(codeWindow, IVsCodeWindowEx).Initialize(CUInt(_codewindowbehaviorflags.CWB_DISABLESPLITTER), VSUSERCONTEXTATTRIBUTEUSAGE.VSUC_Usage_Filter, "", "", 0, initView))

        ErrorHandler.ThrowOnFailure(codeWindow.SetBuffer(CType(textBuffer, IVsTextLines)))

        Dim textView As IVsTextView = Nothing
        ErrorHandler.ThrowOnFailure(codeWindow.GetPrimaryView(textView))

        Dim textViewHost = editorAdapterFactory.GetWpfTextViewHost(textView)

        If textViewHost Is Nothing Then Return VSConstants.VS_E_INCOMPATIBLEDOCDATA

        RemoveParentControl(textViewHost.HostControl)

        Dim editor As New HostEditorTestPane(codeWindow, textViewHost)
        ppunkDocView = Marshal.GetIUnknownForObject(editor)
        ppunkDocData = Marshal.GetIUnknownForObject(textBuffer)
        pbstrEditorCaption = ""

This works on a custom test file when I create and open a project and select the test file. My custom editor opens and intellisense works for the XML editor.

However, if I open a file in a project and then close Visual Studio, when I reopen the project, it attempts to restore the opened file but it fails on the following line:

Dim textViewHost = editorAdapterFactory.GetWpfTextViewHost(textView)

textViewHost is null. By returning VSConstants.VS_E_INCOMPATIBLEDOCDATA Visual Studio then asks to close the file, if I select Yes, it attempts to reopen the file again, but this time punkDocDataExisting is IntPtr.Zero and it successfully opens.

A slight workaround but that means an annoying message whenever Visual Studio tries to restore a workspace, and also fails to restore other files after. Any idea how to fix this?

Also, is there any alternative to using IVsInvisibleEditorManager? The original code had a comment about it being easier, but I do know how to get the content type, language service GUID, etc. But when I try to use other methods to create the editor, intellisense and highlighting does not work correctly.

I created a project and uploaded it here to test https://github.com/manuelxmarquez/HostEditorTest