The Cryptographic Challenge is used for authentication in TLS as well as other places, e.g. SixToken . In TLS it can be used both to authenticate the server to the client and optionally to authenticate the client to the server (two separate crypto challenges in opposite directions).
The basic idea is as follows (for server to client authentication)
- The server sends its server certificate to the client
- The client validates the server’s server certificate:
- checks digital signature on cert using CA’s cert
- checks trust chain of server cert to a trusted root
- checks for expiration
- checks revocation status
- checks if Server Authentication usage flag is present
- compares FQDN of connected server with SubjectDN CN field in cert
- The client extracts the server’s public key from its server certificate
- The server creates a short random plaintext (e.g. 16 character alphanumeric string) as the random string
- The client encrypts the random string with the server’s public key (and appropriate asymmetric key algorithm, e.g. RSA), to produce the challenge string, and sends it to the server
- The server uses its private key to decrypt the challenge to produce the challenge response, which is returned to the client
- The client compares the challenge response with the original random string. If they match that proves the server has possession of the private key corresponding to its server certificate. This strongly authenticates the server to the client.
If the second phase of the TLS handshake is done to accomplish client to server authentication, the above process is repeated with client and server roles reversed.
This crypto challenge is performed before the symmetric session key is exchanged and encryption is enabled (it is done in plaintext). A hacker can capture everything sent back and forth without compromising the process.
This screen capture shows how this works: