Code walkthrough of openssl server side implementation

July 25, 2011

Openssl is a free software implementation of the SSL protocol. It is used in a wide variety of products including web application firewall, SSL accelerators etc. However, there is little to no documentation of the source code. This article explains the openssl code that implements the server side of the SSL protocol. More specifically, we look at the routines that parse and validate the first message i.e. CLIENT HELLO that is sent to the server by the client.We look at version 3 of the protocol.The reader is assumed to have familiarity with the SSL protocol and network programming.

Similar to a server that implements a socket, the SSL server also listens for an incoming SSL connection from the client. The code that processes an incoming connection is implemented under the routine ssl3_accept .

1. int ssl3_accept(SSL *s) 

This is a state machine that is implemented as a large switch statement.

32. switch (s->state) 33. { 34. case SSL_ST_RENEGOTIATE: 35. s->new_session=1; 36. /* s->state=SSL_ST_ACCEPT; */ 37. 38. case SSL_ST_BEFORE: 39. case SSL_ST_ACCEPT: 40. case SSL_ST_BEFORE|SSL_ST_ACCEPT: 41. case SSL_ST_OK|SSL_ST_ACCEPT: 

The first state is SSL_ST_RENEGOTIATE. This corresponds to the renegotiate state which essentially is reestablishment of the SSL connection with different parameters over an already established connection. At present, we skip this and continue our examination of scenario when a new client connections comes in. This is indicated by the state SSL_ST_BEFORE.

43. s->server=1; 

Set a flag indicate that this is a server.

44. if (cb != NULL) cb(s,SSL_CB_HANDSHAKE_START,1); 

Openssl provides a way for a user to add its own code in every state. This can be used as a debugging tool. If a callback function has been assigned then openssl calls that function with the SSL_CB_HANDSHAKE_START.

46. if ((s->version>>8) != 3) 47. { 48. SSLerr(SSL_F_SSL3_ACCEPT, ERR_R_INTERNAL_ERROR); 49. return -1; 50. } 

Examine the ssl version. If this version is not 3, then we should not be in ssl version 3 routines. Flag the error as internal error using SSLerr routine and return.

51. s->type=SSL_ST_ACCEPT; 

Set the type to SSL_ACCEPT.

53. if (s->init_buf == NULL) 54. { 55. if ((buf=BUF_MEM_new()) == NULL) 56. { 57. ret= -1; 58. goto end; 59. } 60. if (!BUF_MEM_grow(buf,SSL3_RT_MAX_PLAIN_LENGTH)) 61. { 62. ret= -1; 63. goto end; 64. } 65. s->init_buf=buf; 66. } 

If init_buf is null, then create a buffer of size equivalent to the maximum record size of SSL.This record size is 16KB at present.

68. if (!ssl3_setup_buffers(s)) 69. { 70. ret= -1; 71. goto end; 72. } 73. 74. s->init_num=0; 75. 

Set up buffers to hold the reading and writing data. The size of these buffers is equivalent to the maximum size of SSL packet. This size includes the record size + the additional padding upto 2048 bytes needed for encryption + SSL header length. Also set init_num to 0. This is used later to keep track of the bytes that are read from the wire.

76. if (s->state != SSL_ST_RENEGOTIATE) 77. { 78. /* Ok, we now need to push on a buffering BIO so that 79. * the output is sent in a way that TCP likes :-) 80. */ 81. if (!ssl_init_wbio_buffer(s,1)) { ret= -1; goto end; } 82. 83. ssl3_init_finished_mac(s); 84. s->state=SSL3_ST_SR_CLNT_HELLO_A; 85. s->ctx->stats.sess_accept++; 86. } 

If we are not renegotiating SSL (as in this case), set up a buffering BIO. This essentially means that whenever the SSL data is written to the network socket, it will be buffered until a certain number of bytes are accumulated.
The message authentication code i.e. MAC is also set up for later use.
The state is set to SSL3_ST_SR_CLNT_HELLO_A. This essentially means that we will be reading the SSL Client Hello next
The number of accepted SSL sessions is increased by 1.
The rest of the lines in this case statement refer to the SSL renegotation scenario. We skip this as we are discussing the more common scenario of intial SSL connection establishment.

107. break; } 

Control now goes out of switch statement. Since we are looping forever, we reenter the switch statement with state as SSL3_ST_SR_CLNT_HELLO_A.

127. case SSL3_ST_SR_CLNT_HELLO_A: 128. case SSL3_ST_SR_CLNT_HELLO_B: 129. case SSL3_ST_SR_CLNT_HELLO_C: 130. 131. s->shutdown=0; 132. ret=ssl3_get_client_hello(s);9821210951 133. if (ret <= 0) goto end; 134. s->new_session = 2; 135. s->state=SSL3_ST_SW_SRVR_HELLO_A; 136. s->init_num=0; 137. break; 

In this state, we read the SSL client hello. If there is an error we get out of the loop else change the state to SSL3_ST_SW_SRVR_HELLO_A, reset init_num to 0 and come out of the switch statement.

728 int ssl3_get_client_hello(SSL *s) 

We now discuss the implementation of the routine ssl3_get_client_hello. This is in the file s3_srvr.c

747 if (s->state == SSL3_ST_SR_CLNT_HELLO_A) 748 { 749 s->state=SSL3_ST_SR_CLNT_HELLO_B; 750 } 751 s->first_packet=1; 752 n=s->method->ssl_get_message(s, 753 SSL3_ST_SR_CLNT_HELLO_B, 754 SSL3_ST_SR_CLNT_HELLO_C, 755 SSL3_MT_CLIENT_HELLO, 756 SSL3_RT_MAX_PLAIN_LENGTH, 757 &ok); 758 759 if (!ok) return((int)n); 760 s->first_packet=0; 761 d=p=(unsigned char *)s->init_msg; 

If the state is SSL3_ST_SR_CLNT_HELLO_A, we set it to SSL3_ST_SR_CLNT_HELLO_B and read the SSL client hello message. The message is read in s->init_msg variable. ssl_get_message is a pointer to a function and is member of the structure ssl_method_st. For SSL version 3 protocol, this points to ssl3_get_message. ssl3_get_message is implemented in s3_both.c.
We will skip the details of this function for now. At present, it is sufficient to know at the successful execution of this function, a well formed Client Hello is in the init_msg. Internally, ssl3_get_message is used by s->init_num to keep track of the bytes read on the wire.

765 s->client_version=(((int)p[0])<<8)|(int)p[1]; 766 p+=2; 767 if ((s->version == DTLS1_VERSION && s->client_version > s->version) | 769 (s->version != DTLS1_VERSION && s->client_version < s->version)) 770 { 771 SSLerr(SSL_F_SSL3_GET_CLIENT_HELLO, SSL_R_WRONG_VERSION_NUMBER); 772 if ((s->client_version>>8) == SSL3_VERSION_MAJOR) 773 { 774 /* similar to ssl3_get_record, send alert using remote version number */ 775 s->version = s->client_version; 776 } 777 al = SSL_AD_PROTOCOL_VERSION; 778 goto f_err; 779 } 

Examine the SSL client version. If it is incorrect, then the statement goto f_err is executed which essentially sends an SSL alert terminating the connection.

796 /* load the client random */ 797 memcpy(s->s3->client_random,p,SSL3_RANDOM_SIZE); 798 p+=SSL3_RANDOM_SIZE; 

The client random is copied from the pointer p to s->s3->client_random. This random is used in later computations.

801 j= *(p++); 812 if ((s->new_session && (s->options & SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION))) 813 { 814 if (!ssl_get_new_session(s,1)) 815 goto err; 816 } 817 else 818 { 819 i=ssl_get_prev_session(s, p, j, d + n); 820 if (i == 1) 821 { /* previous session */ 822 s->hit=1; 823 } 824 else if (i == -1) 825 goto err; 826 else /* i == 0 */ 827 { 828 if (!ssl_get_new_session(s,1)) 829 goto err; 830 } 831 } 832 833 p+=j; 

The pointer p is incremented. If this is a new session, then a new session id is created and returned. Otherwise, attempt is made to obtain a session info from the session cache. s->hit is set to 1 to indicate that this is a cache hit.

908 /* If it is a hit, check that the cipher is in the list */ 909 if ((s->hit) && (i > 0)) 910 { 911 j=0; 912 id=s->session->cipher->id; 913 914 #ifdef CIPHER_DEBUG 915 printf("client sent %d ciphers\n",sk_num(ciphers)); 916 #endif 917 for (i=0; i<sk_SSL_CIPHER_num(ciphers); i++) 918 { 919 c=sk_SSL_CIPHER_value(ciphers,i); 920 #ifdef CIPHER_DEBUG 921 printf("client [%2d of %2d]:%s\n", 922 i,sk_num(ciphers),SSL_CIPHER_get_name(c)); 923 #endif 924 if (c->id == id) 925 { 926 j=1; 927 break; 928 } 929 } 930 /* Disabled because it can be used in a ciphersuite downgrade 931 * attack: CVE-2010-4180. 934 */ 935 #if 0 936 if (j == 0 && (s->options & SSL_OP_NETSCAPE_REUSE_CIPHER_CHANGE_BUG) && (sk_SSL_CIPHER_num(ciphers) == 1)) 937 { 938 /* Special case as client bug workaround: the previously 9 used cipher may 939 * not be in the current list, the client instead might be trying to 940 * continue using a cipher that before wasn't chosen due to server 941 * preferences. We'll have to reject the connection if the cipher is not 942 * enabled, though. */ 943 c = sk_SSL_CIPHER_value(ciphers, 0); 944 if (sk_SSL_CIPHER_find(SSL_get_ciphers(s), c) >= 0) 945 { 946 s->session->cipher = c; 947 j = 1; 948 } 949 } 950 #endif 951 if (j == 0) 952 { 953 /* we need to have the cipher in the cipher 954 * list if we are asked to reuse it */ 955 al=SSL_AD_ILLEGAL_PARAMETER; 956 SSLerr(SSL_F_SSL3_GET_CLIENT_HELLO,SSL_R_REQUIRED_CIPHER_MISSING); 957 goto f_err; 958 } 

This addresses the case where there is a session hit. The client’s ciphers are looked into to ensure that the cipher associated with the previous session is in the list of ciphers sent by the client.

1050 if (!s->hit) 1051 { 1052 #ifdef OPENSSL_NO_COMP 1053 s->session->compress_meth=0; 1054 #else 1055 s->session->compress_meth=(comp == NULL)?0:comp->id; 1056 #endif 1057 if (s->session->ciphers != NULL) 1058 sk_SSL_CIPHER_free(s->session->ciphers); 1059 s->session->ciphers=ciphers; 1060 if (ciphers == NULL) 1061 { 1062 al=SSL_AD_ILLEGAL_PARAMETER; 1063 SSLerr(SSL_F_SSL3_GET_CLIENT_HELLO,SSL_R_NO_CIPHERS_PASSED); 1064 goto f_err; 1065 } 1066 ciphers=NULL; 1067 c=ssl3_choose_cipher(s,s->session->ciphers, 1068 SSL_get_ciphers(s)); 1069 if (c == NULL) 1070 { 1071 al=SSL_AD_HANDSHAKE_FAILURE; 1072 SSLerr(SSL_F_SSL3_GET_CLIENT_HELLO,SSL_R_NO_SHARED_CIPHER); 1073 goto f_err; 1074 } 1075 s->s3->tmp.new_cipher=c; 1076 } 

This addresses the case when there is no session hit. ssl3_choose_cipher compares the ciphers in the server list and in the client list. The first match is chosen as the cipher to be used in the new session. Its value is stored in tmp.new_cipher.


At this stage, we have the following
client_random – random number to be used in in later calculations.
cipher_list – our prefered list of ciphers
ciphers – the clients prefered list of ciphers
compression – basically ignored right now
ssl version is set – sslv3
s->session – The ssl session has been setup.
s->hit – session reuse flag
s->tmp.new_cipher – the new cipher to use.

The client hello message has now been successfully parsed and validated. The contents of that are stored in the variables mentioned above. The next step is for the server to send the Server Hello, the certificate that it is configured with and the Server Hello Finish. We look at this in the next post.



Notes on common criteria certification

June 20, 2011

As an IT security/systems engineer, you will often come across terms such as common criteria, ITSEC and FIPS, especially while dealing with federal agencies. It is the reason why vendors such as Juniper are willing to put several thousand dollars to achieve certification for their products.

There is a wealth of information on the web but most sources scratch only the surface of this issue. Here are some notes on common criteria that I have gleaned from various links.

The Common Criteria for Information Technology Security Evaluation ( is an international standard (ISO/IEC 15408) for computer security certification.

Under the Common Criteria model, an evaluation is carried out on a product and it is assigned an Evaluation Assurance Level (EAL). The thorough and stringent testing increases in detailed-oriented tasks as the assurance levels increase. The Common Criteria has seven assurance levels. The range is from EAL1, where functionality testing takes place, to EAL7, where thorough testing is performed and the system design is

The different EAL packages are listed below:
• EAL1 Functionally tested
• EAL2 Structurally tested
• EAL3 Methodically tested and checked
• EAL4 Methodically designed, tested, and reviewed
• EAL5 Semiformally designed and tested
• EAL6 Semiformally verified design and tested
• EAL7 Formally verified design and tested

When a system is “formally verified,” this means it is based on a model that can be mathematically proven.

The Common Criteria uses protection profiles in its evaluation process. This is a mechanism used to describe a real-world need of a product that is not currently on the market. The protection profile contains the set of security requirements, their meaning and reasoning, and the corresponding EAL rating that the intended product will require. The protection profile describes the environmental assumptions, the objectives, and the functional and assurance level expectations. Each relevant threat is listed along with how it is to be controlled by specific objectives. The protection profile also justifies the assurance level and requirements for the strength of each protection mechanism.The protection profile provides a means for a consumer, or others, to identify specific security needs; this is the security problem to be conquered. If someone identifies a security need that is not currently being addressed by any current product, that person can write a protection profile describing the product that would be a solution for this real-world problem. The protection profile goes on to provide the necessary goals and protection mechanisms to achieve the required level of security, as well as a list of things that could go wrong during this type of system development. This list is used by the engineers who develop the system, and then by the evaluators to make sure the engineers dotted every i and crossed every t.
The Common Criteria was developed to stick to evaluation classes but also to retain some degree of flexibility. Protection profiles were developed to describe the functionality, assurance, description, and rationale of the product requirements

Like other evaluation criteria before it, the Common Criteria works to answer two basic questions about products being evaluated: what does its security mechanisms do (functionality), and how sure are you of that (assurance)? This system sets up a framework that enables consumers to clearly specify their security issues and problems; developers to specify their security solution to those problems; and evaluators to unequivocally determine what the product actually accomplishes.A protection profile contains the following five sections

• Descriptive elements Provides the name of the profile and a description of
the security problem to be solved
• Rationale Justifies the profile and gives a more detailed description of the
real-world problem to be solved. The environment, usage assumptions, and
threats are illustrated along with guidance on the security policies that can be
supported by products and systems that conform to this profile
• Functional requirements Establishes a protection boundary, meaning the
threats or compromises within this boundary to be countered. The product or
system must enforce the boundary established in this section
• Development assurance requirements Identifies the specific requirements
the product or system must meet during the development phases, from design
to implementation
• Evaluation assurance requirements Establishes the type and intensity of the

The evaluation process is just one leg of determining the functionality and assurance of a product. Once a product achieves a specific rating, it only applies to that particular version and only to certain configurations of that product. So if a company buys a firewall product because it has a high assurance rating, the company has no guarantee the next version of that software will have that rating. The next version will need to go through its own evaluation review. If this same company buys the firewall product and installs it with configurations that are not recommended, the level of security the company was hoping to achieve can easily go down the drain. So, all of this rating stuff is a formalized method of reviewing a system being evaluated in a lab. When the product is implemented in a real environment, factors other than its rating need to be addressed and assessed to ensure it is properly protecting resources and the environment

Web application firewall

June 2, 2011

An application firewall (or web application firewall) is a device that prevents web (or L7) attacks.

This is achieved by the device sitting between the application (or web) server and the user-side client, thereby “intercepting” all user data entering or leaving the application server. The intercepted traffic is examined against various rules in order to attempt to judge whether the data is “good” or “bad.” For example, data entered by a user can be examined to see if it contains potentially dangerous characters, such as SQL commands that could lead to an SQL insertion attack. When attacks are detected, the application firewall can take whatever remedial or evasive action its owner deems appropriate, ranging from simply disconnecting the current application session to more sophisticated approaches such as shunting the session to a “honeypot” system that is specially instrumented to gather details of attack methodologies. Running the application firewall in a separate process and memory space from the application further protects the applications, business logic, and data.

Naturally, in order for an application firewall to be able to make reasonable assessments of the application data, it would need to have some advance knowledge. Some products accomplish this by way of a database of known attacks—either via signatures of actual attack data, behavioral patterns, or some hybrid of both. Other products observe the normal behavior of the application(s) that they are to protect, preferably in a controlled environment, and then attempt to ensure that all “live” data conforms to similar content, size, and structure.
This page describes more about the web application firewall. It also lists the vendors for such devices
Nowadays, such a device will also have other features like load balancing, SSL acceleration etc.

Books for SSL

May 21, 2011

If you wish to know more about the SSL internals, then Eric Resocorla’s book SSL and TLS: Designing and Building Secure Systems is the golden standard.
If you are unfortunate enough to have to deal with SSL implementations, chances are that you will come across openssl, an open source implementation of SSL. Over the years, openssl has become quite complex. The book, Network security with openssl will help in getting an understanding of openssl. This book however, is becoming dated as new versions of openssl come out.

Certificate Revocation

April 20, 2011

An SSL certificate can be revoked by the issuing authority (also known as trusted certificate authority). This means that the issuing authority no longer considers the certificate as valid. This can happen, for example, if the issuing authority believes that the private key associated with the certificate has been compromised.

Once the certificate is revoked, it can be added to a list which contains all the certificates that have been revoked by the issuing authority. This list is known as the certificate revocation list or CRL.

Once a SSL server (or client) receives a certificate from the other party, it checks if the certificate has been revoked. It can do this by seeing the CRL of the issuing authority and checking if the list contains the certificate that it has received.

Over time, the list of certificate can become quite large. Downloading it  frequently to know the latest update can decrease performance. Another protocol, OCSP (online certificate status protocol) can be used to work around this issue. OCSP is described in RFC 2560

Some often used Openssl commands

March 18, 2011

This is a ready list of the openssl commands that I use frequently

For connecting to the openssl server:
openssl s_client -connect <Server-IP> -cipher
For connecting to the openssl server with client certificate
openssl s_client -connect <Server-IP> -cert <certificate path> -key <private key> 
For the server to be ready to accept connections
openssl s_server -cert <certificate path> -key <private path> -accept 443 -WWW
For seeing the plaintext contents of the SSL certificate
openssl x509 -in cert.pem -noout -text

SSL for beginners

March 14, 2011

I often have to explain SSL to the systems and sales engineers in the organization.

Here is a short version of what I usually end up saying.
Let’s say Bob and Alice wish to talk securely. The usual requirements would be that
1. No one should be able to eavesdrop on the conversation. This is fulfilled by encryption.
2. If both Bob and Alice cannot see each other, then both of them should be assured that they are talking to the right person. This is fulfilled by authentication.
3. No one should be able to modify the message that they send to each other. This is fulfilled by using message digests.
SSL stands for Secure sockets layer. It uses separate mechanisms to handle encryption, authentication and message digests.

Encryption is a way in which one can mathematically change the message to make it look like a random string. In other words, if Alice sends “Hello Bob” to Bob, the encrypted message may look like “xwygh” i.e. a random English message.
Authentication can be used by Alice / Bob to prove that they are indeed who they claim to be. One of the ways in which they can be achieved is to use a certificate. This certificate is issued by someone whom both Alice and Bob trust. In SSL parlance, this entity is called as the trusted certificate authority.Thus, when Alice presents a certificate and claims that it is signed by a trusted certificate authority that is also trusted by Bob, then Bob can be assured that Alice is indeed whom she claims to be.
Message digest is a short version of the message (or mathematically speaking, a hash of the message). The important property of this version is that the message cannot be recovered if someone looks at this version(again, mathematically speaking, this is a one way hash). So when Alice sends a message “Hello, Bob) to Bob, she also sends this version along with it. When Bob receives the message, he will again create a message digest over the message he received and compare the two digests. If someone modifies the message to “Bye, Bob”, then the message digest that Bob creates will be different from the message digest that Bob receives. This will indicate to Bob that the message has been modified.

Cloud based WAF services

January 2, 2011

As web application firewall becomes commoditized, the industry is moving towards cloud based solution for small and medium based business. Akamai, Incapsula and Savvis are the main contenders today.

A cloud based service would require double the bandwidth because the traffic goes from the client to the WAF and then back again. This, however, is not a problem for small and medium sized businesses which these cloud based solution target.