Direct DTE access: should we try to limit its use?

Hi to all,
I started some days ago, for the first time, the exploration of VS extensibility world through packages. On a recent thread I've been suggested for the most part to try to avoid DTE due to its overhead.
So, I started to work on my project by following this rule until I realized how much is hard and how many work requires to access every element of the ide without DTE. The forum lacks of sample codes while has plenty of samples codes that refers to DTE since seems to be the most common method used from people to access the ide.
Let me start with some code...

For example in my project I need to access the debugger in order to get notification when it enter in break mode.
Through DTE is very easy and with a few lines of code the job is done:

dte = (DTE)GetService(typeof(SDTE));
dte.Events.DebuggerEvents.OnEnterBreakMode += debugEvents_OnEnterBreakMode;
void debugEvents_OnEnterBreakMode(dbgEventReason Reason, ref dbgExecutionAction ExecutionAction)
{
}

Without DTE things becomes much more harder because I have to implement IVsDebugger interface. This one is the only code sample I found in all over the web and even if it works, it's still a basic implementation which lacks of functionalities and elements already available in DTE. So, I don't want to imagine how many other code it would be required for a more complete implementation.

var h = new DebuggerEventsSink();
h.DebugModeChange += h_DebugModeChange;

public sealed class DebuggerEventsSink : Microsoft.VisualStudio.PlatformUI.DisposableObject, IVsDebuggerEvents
        {
            public enum DebugMode { Run, Break, Design }
            public class DebugModeChangedEventArgs : EventArgs
            {
                public readonly DebuggerEventsSink Source;
                public readonly DebugMode Action;

                public DebugModeChangedEventArgs(DebuggerEventsSink source, DebugMode action)
                {
                    this.Source = source;
                    this.Action = action;
                }
            }
            public IVsDebugger debugger;
            private uint debugEventsCookie = VSConstants.VSCOOKIE_NIL;

            private readonly DebugModeChangedEventArgs runEventArgs;
            private readonly DebugModeChangedEventArgs breakEventArgs;
            private readonly DebugModeChangedEventArgs designEventArgs;
            public event EventHandler<DebugModeChangedEventArgs> DebugModeChange = (sender, args) => { };

            public DebuggerEventsSink()
            {
                runEventArgs = new DebugModeChangedEventArgs(this, DebugMode.Run);
                breakEventArgs = new DebugModeChangedEventArgs(this, DebugMode.Break);
                designEventArgs = new DebugModeChangedEventArgs(this, DebugMode.Design);

                Initialize();
            }

            private void Initialize()
            {
                debugger = Package.GetGlobalService(typeof(IVsDebugger)) as IVsDebugger;
                ErrorHandler.ThrowOnFailure(debugger.AdviseDebuggerEvents(this, out debugEventsCookie));
            }

            protected override void DisposeManagedResources()
            {
                ThreadHelper.Generic.Invoke(() =>
                {
                    if (this.debugEventsCookie != VSConstants.VSCOOKIE_NIL &&
                        this.debugger != null)
                    {
                        ErrorHandler.CallWithCOMConvention(() => this.debugger.UnadviseDebuggerEvents(this.debugEventsCookie));
                        this.debugEventsCookie = VSConstants.VSCOOKIE_NIL;
                    }
                });

                base.DisposeManagedResources();
                //this.DisposeManagedResources();
            }

            int IVsDebuggerEvents.OnModeChange(DBGMODE dbgmodeNew)
            {
                switch (dbgmodeNew)
                {
                    case DBGMODE.DBGMODE_Run:
                        DebugModeChange(this, runEventArgs);
                        break;
                    case DBGMODE.DBGMODE_Break:
                        DebugModeChange(this, breakEventArgs);
                        break;
                    case DBGMODE.DBGMODE_Design:
                        DebugModeChange(this, designEventArgs);
                        break;
                }

                return VSConstants.S_OK;
            }
        }

Later, I realized that I need to access to the active document to change the cursor position, access breakpoints, execute some commands and so on. Without even leaving the ide for checking online references, DTE immediately suggest me what I have to write to obtain what I want:

((TextSelection)dte.ActiveDocument.Selection).LineDown(false, 4);
var total = dte.Debugger.Breakpoints.Count;
dte.Debugger.SetNextStatement();
dte.ExecuteCommand("Debug.SetNextStatement");

Now, I don't want to imagine how many researches and how many code I will have to write in order to do the same without DTE, but giving the headaches I got only for the debugger code, it will be a long journey.

I opened this thread to understand why we should try to avoid DTE and spend lots of time to write code when we can have all what we need easily and fast.
I want to know your opinions. When can be used safely, when should be avoided, if it apply only to specific parts  and if there is at least some sort of query that can return the same specific object of the tree without a direct access.

Thanks