I Have Had This Argument About a Zillion Times
I think that (non-classical) liberals and libertarians see the problem of "special interests" differently. Liberals view special interests as exogenous to the policy process. You have to overcome special interests to create good policy. Libertarians see special interests as endogenous. Policy is what creates them.
Yep, I have had this argument about a million times with liberals. Liberals will argue that government power is neutral to positive, and that it is private action corrupting government, and this corruption can be avoided if private action is aggressively policed (including campaign spending limits, etc). Example: If Wall Street money could be taken out of politics then financial regulation would work.
I argue that money in politics are a result of the stakes that we have put on the table -- the more power we give to government to reallocate wealth, the more money will be spent to have such decisions made in one's favor. In the age old question of whether a bribe is more the fault of the politician that demanded it or the private party that offered it, I would answer that the fault is with the system that gives the politician enough power to make such a bribe pay. And increasing the government's power to limit private involvement in politics (e.g. via campaign spending limits) only makes the government power problem worse.