Is this a good idea?!?

It's a great idea. Balfour won't give you any real stats unless he becomes a closer. Millwood is hot and you can ride out his streak. That will still give you more than Balfour. Millwood has had too many seasons with high ERA and WHIP so when he starts getting bad, address the problem.