There's a pretty good chance that my PCs will enter a team-battle tournament soon. One of the characters that they might fight has an ability that would give disadvantage to attack someone other than him.

When do I tell the player that their attack would have disadvantage? For context, to save time, we'll often say, "Three attacks on guy X." Should I stop him after the first one and let him know it's at disadvantage, or after he's committed to attacking?

When do I let my players know their roll will be affected?

Let them know after the first attack at latest

By the rules, each attack can have a separate target, and don't need to be called out before the first attack. Therefore, your players have the right to switch the target after their first attack for any reason --- the first target may have died, it may have turned out to impose disadvantage, it may have had a damage resistance, or whatever. "Three attacks on [target]" is, unless explicitly ruled otherwise, just a convenience thing and should not be regarded as a mechanical commitment.

"Okay," you might say, "but should I rule it that way then?" My answer is no, you shouldn't. First of all, penalizing your players for using a shorthand is not cool and will feel like antagonizing them for no proper reason. It is a convenience thing and depriving them of that convenience for no reason will not serve any practical purpose.

Likewise, you could rule that regardless of the wording, all attacks made will have to target the same creature, but that reduces the tactical agency the players have (in layman's terms, they have less interesting choices to make) and has the potential to slow down combat encounters significantly. In all fairness, it's more likely to increase metagaming than reduce it, since the players will have to consider eg. whether they want to use their three attacks at a creature that's likely to go down in just one hit.

Finally, one might consider the option of not letting the players know they're rolling at a disadvantage at all (and rolling the second die in secret), but I seriously recommend against this. It will just lead to clumsy and confusing gameplay, put more stress on the GM as they have to remember to roll after player attacks and is very transparently "unusual" to the players. Metagaming shouldn't even be a concern here --- surely the characters, after attempting an attack, should realize that something is making hitting the target more difficult than usual?

In general, intelligent creatures in D&D5 know when they're under an effect, as long as that effect has perceptible signifiers.

See PH page 204 (which is about magic, but the principle establishes the basic principle): That people under a spell effect don't know about it UNLESS IT HAS A PERCEPTIBLE EFFECT. Which means if it does have such an effect, they should know about it.

This is a core principle of all roleplay, not just D&D -- players should know about stuff their characters would be able to perceive. Yes, I know this is basic, but trust me, I'm going somewhere with this.

This means that while you don't have to tell players explicitly how a "marked"-style ability (which generally signifies that the character in question is positioning themselves in a way that will let them counter-attack if the target attacks anyone except the) works, you are being unfair if you don't give them the basic description that will let them react intelligently.

If you look back at 4th edition (where the "marked" condition originated), the purpose of the "marked" concept (which, no, isn't an explicit condition in 5th edition, but is still present in concept) isn't to let a character -- PC or NPC -- do extra damage. Instead it's to help a character designed to take hits to do their job; to prevent their allies from taking damage by "persuading" foes to attack their higher defenses instead.

This means that if you rule that characters who are targeted by such a defensive ability have no awareness of it, you're letting your monsters (and PCs who take similar abilities) do extra damage, but a the expense of being able to accomplish their role.

So instead, it's better overall to give people descriptions of what's going on that will allow them to make meaningful choices; whether to attack the person who is, say "watching you closely, and seems ready to attack if you take your eyes off them for a moment", or to ignore the damage in order to get the foe out of position and maybe take down a higher value target. Similarly, it's better to play monsters that care about their well-being and have enough perceptual awareness to usually respect marks, so that PCs who take abilities like this (like Sentinel) can often do their jobs of protecting allies.

Of course, particularly if the clues here are present, but not obvious, it's entirely reasonable to ask for a Perception (or Insight, in some cases) check and base how much description you give a player based on that roll. It's also reasonable (preferrable, even) to rule that some monsters (or other opponents) either don't notice a character using a feat or ability like this -- or simply don't care.

This is all, BTW, assuming that this is an ability with a trigger. In cases where the character has a penalty, they should have a general idea of why the penalty exists. If it's someone interfering with them physically, they'll certainly know who that is. If instead it's a magical effect messing with their aim, they'll know that, say, their attack got blocked by something invisible, or that a shot that was on target got deflected by something unseen.

Now, regarding your final question, this has a very clear answer. Every attack a character makes during their turn is sequenced and is resolved before they make another one; even if the player says "I attack the orc three times," the three attacks are really separate. So if they get new information after the first attack (like the attack gaining disadvantage or the orc falling over dead) then they should find that out immediately and get an opportunity to change their mind about the other attacks. Similarly, if the player says something like "I move away from the giant and attack the statue in the corner." their attack isn't necessarily wasted if the giant knocks them prone before they can get away, because the movement is separate from their attack. So once they're on the floor, they can change their mind and decide to attack whatever's in range, rather than swing futilely against the sui-distant statue.

Make it interesting, at least

If my guess is correct you're talking about something similar to Goading Strike from the Battle Master?

I'd give the party some information the instant it happened, and if they still didn't get it, I'd inform them the mechanics after they made an attack at disadvantage.

For example:

The enemy knight with full armor and his rapier and dagger take an attack on you, he taunts laughing, "Are you an utter fool? You leave your guard WIDE open, Har har har, this is like stabbing fish in a barrel!"

Whom ever he targets with this effect, I'd also tell them that, the words of the enemy seem to echo in their ears, making them feel annoyed and distracted. IF they still decided to attack a different enemy, even after I've told them how annoying the knight was, then I'd tell them on making the first attack, it is at disadvantage and again remind them the taunting/goading words of the 2 weapon knight and probably explain the mechanics I was using.

Honestly just make it something that feels organic and they'll probably like it. Imagine that these players haven't fought someone like this before, or maybe they have, how does that change things?

I'd use this in reverse also, for example a random animal probably wouldn't know to attack casters who are concentrating on spells. They'd probably attack the fighters/barbarians directly in front of them.