Can't Have Your Freedom Cake and Eat it Too

I have no issues if 2 gay people want to get married. I don't get hung up on the notion that marriage is only between a man and a woman because marriage is a human construct; the only restrictions in place are the ones we place on it.

The most common reason I've seen people give to restrict gay marriages is because of religion. Unfortunately for me that oppositional basis doesn't hold any water because while this country may have Christian-based underpinnings the Constitution was purposely crafted as a secular document, a document which, among other things, reaffirms our right to freedom of association. And that's the point at which I find some supporters of gay rights want to have their cake and eat it too.

Some gay rights supporters want the government to intervene and protect the right to freely associate (marry in this case), but in the same breath also want the government to intervene and prevent others from exercising their right to freely associate. What do I mean? Two particular incidents that immediately come to mind are the lawsuits surrounding a photographer that refused to service a gay couple and more recently a baker that wouldn't bake a cake for a gay couple's wedding. In both cases the same-sex couples sued the businesses, asking the government to violate the rights of those business owners to decide who they transact with instead of the couples simply taking their business elsewhere.

If a neo-nazi had asked the baker to bake a cake with a swastika on it and the baker refused because he was Jewish, would the neo-nazi have filed suit? If so, would he have won? There is effectively no difference between the two instances of discrimination, and yet somehow people have a problem with one and not another.
If a man (or woman) has the right to choose whom he marries, a business owner must also have the right to chose whom he serves. You cannot honestly argue for the first while actively working against the second. Where I come from they call that hypocrisy.