Start with this intuition: all adulterous actions are intrinsically wrong. Therefore, any action that is intrinsically just like an adulterous action is also wrong. Now the intrinsic character of a successful action is defined by the intentions or action plan—by what the end is and how it is intended to be achieved. Many cases of adultery do not, however, involve an intention to commit adultery. Sam knows that sex with Suzy would be adulterous, but he need not intend the sex qua adulterous. He might intend it qua pleasant or qua unitive-with-Suzy. The distinction is important. There are cases of adultery where there is an intention to commit adultery as such, as when Sam intends to make Suzy's husband a cuckold or make his own wife jealous. Such malicious cases are, ceteris paribus, morally worse than run-of-the-mill adultery done for the sake of pleasure or union.
Thus the intentions in run-of-the-mill adultery are the same as those in typical cases of fornication—to share pleasure with this person, to unite with this person, etc. If adultery is intrinsically wrong, so will these typical cases of fornication be. (And I don't think there are any atypical permissible cases of fornication, either.)
Moreover, so will cases of sexual activity within marriage when the intentions are the same kinds of intentions that typical adulterers and fornicators have. Thus, if Sam's intention is simply to share pleasure with Tamara, he is doing intrinsically the same thing as when he commits adultery with Suzy, even if Tamara happens to be his wife and Suzy doesn't. Thus, if there is to be an intrinsic difference between marital activity and adultery, the marital activity must involve intentions that adulterers cannot have, properly marital intentions such as to unite maritally with Tamara or at least to share pleasure with his own wife, Tamara. It is clear, thus, that it is possible to do something that is relevantly like adultery with one's spouse. Is this why, perhaps, when Jesus said that the man who looks lustfully at a woman has committed adultery with her in heart, he did not limit his remarks to the case of the married man or the married woman? Taking his remarks literally, to look lustfully at one's spouse is to commit adultery with her in the heart. Lust is an essentially non-marital attitude.
Now, for every possible kind of action, there are negative conditions that the intentions have to satisfy. For instance, so that a financial transaction, a hammering of a nail, a drinking of a cup of coffee, a sexual act or an act of teaching be permissible, it must not be done for a malicious ulterior end. However, sexual relations, unlike the hammering of a nail, must satisfy a positive condition on their intentions to be permissible. The intentions must be marital—of a sort that could not be satisfied outside of a marriage.
It is sometimes said that there is something wrong with a couple that stays together only because of their marriage vows rather than because they like each other. Be that as it may, if my above are right, there is something wrong with a couple that stays sexually together only because they like each other. The fact of being married needs to enter into their reasons.