As an investor I can't help but feel good about getting cash back through share buybacks.
However, I do appreciate concerns about having sufficient cash on hand to invest in the business, to pursue opportunistic transactions, and to get through a rainy day. The market is happy to take the money at any time, but is not likely to eagerly reinvest when the company needs the cash. We tried this model in Canada, with income trusts (pay out all your cash, then go back to the market for more money), but the government shut down the trade before it ever went through a real test of faith.
There needs to be confidence that management is going to make good use-of-cash decisions over time--or else why invest in the company at all? So the discussion should pivot from buybacks as the devil, to the proper incentive structure for managements. It's easy to see how management could use buybacks to maximize personal compensation: juice up EPS based bonuses, raise the value of stock options, and make the company a less attractive takeover target (where management would be fired). It is less clear how to fix the situation.
See Aswath Damodaran for an in-depth look at buybacks.