Voters are the ones who put people in Congress, so voters and their needs should be the focus of members of Congress. Our representatives should not give excessive attention to groups that have a narrow interest and deep pockets.
Special interest groups wield tremendous power over Congress. They are organized, well-staffed, well-funded, and they know how to influence an election. They offer their support to politicians who are willing to return the favor, no matter what may be right for their constituents. Because these politicians are always looking to the next election, they make sure to keep these organizations happy so they can count on their support in the future.
With term limits, politicians will be stripped of the ability to remain in power for very long. This means that special interest groups will lose their capacity to hold their politicians accountable, thus freeing our representatives to do the right thing, rather than what will keep the support of special interest groups.