Yubikeys/U2F in FIDO2 mode, a super high-level primer:

REGISTERING YOUR KEY

You give it a name (This is human-friendly, for you, and helps you know which key you’re using if you have multiple keys)
The Yubikey generates a public+private keypair unique to this server
Tye Yubikey provides its serial number and a “Key Handle” to the server

Serial number: So you can have more than one YK
Key Handle: Is a hash of the requesting origin (that server) the generated pub-private keypair; the private key is encrypted on-key with the YK’s master key before sending.

USING YOUR KEY

Your client (e.g, Browser but this can be SSH too!) initiates a login request to the server
The server asks for your primary login info first (e.g, Username+Password)

The server sends the stored Key Handle for the key it wants you to touch, along with a secret value it generated

It uses its secret value, hashes the origin issuing the auth request, and compares that hash to the Key Handle the server sent.
This prevents phishing or “push-MFA-auth” style attacks from remote origins; the attacker would have to know your secret value and issue the attempt from the valid origin

On succeeding at validating the hash, a signing operation is initiated.
Your YK waits for you to touch it to prove human presence
On being touched, the YK uses its unique-to-this-origin private key to sign the secret value the server generated
Your YK provides its serial number and the returned secret back to the browser
Your browser forwards the secret value and serial number back to the server
The server looks up the serial number of the key to determine which public key to use to validate you
The server validates the signature of its secret value by using the previously stored public key belonging to that YK
If the signature is valid and the secret value matches, your auth proceeds

More specs at: