HTTP/1.1 Client Implementation: Buffered Header Reading (AXIS2C-1480)¶
Overview¶
This document describes the buffered HTTP header reading implementation in http_client.c, which replaces the legacy byte-by-byte reading approach. The implementation uses a decorator pattern with polymorphic streams to efficiently parse HTTP headers while preserving body data integrity.
Implemented Transports and Stream Types¶
The axutil_stream_t polymorphism supports multiple transport implementations:
Stream Types (axutil_stream_type_t)¶
| Type | Description | Implementation |
|---|---|---|
AXIS2_STREAM_BASIC |
In-memory buffer | axutil_stream_create_basic() |
AXIS2_STREAM_FILE |
File I/O wrapper | axutil_stream_create_file() |
AXIS2_STREAM_SOCKET |
TCP socket (plain HTTP) | axutil_stream_create_socket() |
AXIS2_STREAM_MANAGED |
Custom streams with self-cleanup | Prepend stream, SSL stream |
Client-Side Transports¶
┌─────────────────────────────────────────────────────────────────────────────┐
│ axis2_http_client_send() │
│ │
│ Protocol Detection: axutil_url_get_protocol(client->url, env) │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ "http" │ │ "https" │ │
│ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌────────────────────────┐ ┌────────────────────────────┐ │
│ │ axutil_stream_create │ │ axutil_stream_create_ssl │ │
│ │ _socket() │ │ (OpenSSL) │ │
│ │ │ │ │ │
│ │ stream_type=SOCKET │ │ stream_type=MANAGED │ │
│ │ Uses: recv()/send() │ │ Uses: SSL_read/SSL_write │ │
│ └────────────────────────┘ └────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
TLS/SSL Support (AXIS2_SSL_ENABLED)¶
When compiled with --enable-openssl, HTTPS is supported via OpenSSL:
/* ssl_stream.c - SSL stream implementation */
typedef struct ssl_stream_impl
{
axutil_stream_t stream; /* Base stream (polymorphic) */
axutil_stream_type_t stream_type;
SSL *ssl; /* OpenSSL connection */
SSL_CTX *ctx; /* OpenSSL context */
axis2_socket_t socket; /* Underlying TCP socket */
} ssl_stream_impl_t;
/* Polymorphic read delegates to SSL_read */
int AXIS2_CALL axis2_ssl_stream_read(axutil_stream_t *stream,
const axutil_env_t *env,
void *buffer, size_t count)
{
ssl_stream_impl_t *impl = (ssl_stream_impl_t *)stream;
return SSL_read(impl->ssl, buffer, (int)count);
}
SSL Features:
- Server certificate validation
- Client certificate authentication (mutual TLS)
- Hostname verification (client->validate_ssl_hostname)
- Proxy CONNECT tunneling for HTTPS through HTTP proxy
Alternative: libcurl Transport (AXIS2_LIBCURL_ENABLED)¶
When compiled with --with-libcurl, an alternative HTTP client using libcurl is available:
/* libcurl_stream.c - libcurl-based stream */
typedef struct libcurl_stream_impl
{
axutil_stream_t stream;
axutil_stream_type_t stream_type;
axis2_char_t *buffer; /* Response buffer from curl */
int size;
int read_len;
} libcurl_stream_impl_t;
libcurl advantages: - Built-in HTTP/1.1 and HTTP/2 support - Automatic redirect handling - Connection pooling and keep-alive - Platform SSL/TLS (not just OpenSSL)
Server-Side Stream Adapters¶
Different server environments require custom stream adapters:
| Server | Stream Adapter | Description |
|---|---|---|
| Apache httpd | apache2_stream.c |
Wraps Apache bucket brigades |
| CGI | axis2_cgi_stream.c |
stdin/stdout for CGI |
| Standalone | simple_http_svr_conn.c |
Direct socket I/O |
Raw TCP Transport¶
A non-HTTP TCP transport exists for direct SOAP-over-TCP:
/* tcp_transport_sender.c */
typedef struct axis2_tcp_transport_sender_impl
{
axis2_transport_sender_t transport_sender;
int connection_timeout;
int so_timeout;
} axis2_tcp_transport_sender_impl_t;
This sends raw SOAP XML without HTTP framing - useful for embedded systems or custom protocols.
Stream Composition (Decorator Pattern)¶
The prepend stream demonstrates how streams can be composed:
┌─────────────────────────────────────────────────────────────────┐
│ Application Code │
│ axutil_stream_read(stream, ...) │
└─────────────────────────────┬───────────────────────────────────┘
│
┌───────────────┼───────────────┐
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Prepend │ │ SSL │ │ Socket │
│ Stream │ │ Stream │ │ Stream │
│ (MANAGED) │ │ (MANAGED) │ │ (SOCKET) │
└──────┬───────┘ └──────┬───────┘ └──────────────┘
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ SSL Stream │ │ Socket │
│ or Socket │ │ │
└──────────────┘ └──────────────┘
The prepend stream can wrap either: - A plain socket stream (HTTP) - An SSL stream (HTTPS) - Potentially a libcurl stream
This composition is transparent to the caller - they just see a single axutil_stream_t*.
The Problem: Byte-by-Byte Reading¶
The original implementation read HTTP response headers one byte at a time:
/* BEFORE: Legacy byte-by-byte implementation (slow) */
while((read = axutil_stream_read(client->data_stream, env, tmp_buf, 1)) > 0)
{
tmp_buf[read] = '\0';
strcat(str_status_line, tmp_buf);
if(0 != strstr(str_status_line, AXIS2_HTTP_CRLF))
{
end_of_line = AXIS2_TRUE;
break;
}
}
Problems with this approach:
- Excessive syscalls: Each byte required a recv() system call
- Poor performance: System call overhead dominated processing time
- String operations: strcat() and strstr() on every byte
- No buffering: Could not take advantage of TCP receive buffers
The Solution: Chunked Reading with Prepend Stream¶
Architecture Flow¶
┌─────────────────────────────────────────────────────────────────┐
│ HTTP Client Code │
│ axis2_http_client_receive_header() │
│ │
│ 1. Read 4KB chunks from socket │
│ 2. Scan for CRLF in memory (fast) │
│ 3. Parse status line and headers │
│ 4. Detect end of headers (\r\n\r\n) │
│ 5. Leftover data = body bytes read during header parsing │
└─────────────────────────────┬───────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Prepend Stream (Decorator) │
│ AXIS2_STREAM_MANAGED │
│ │
│ ┌─────────────────┐ ┌─────────────────────────────────┐ │
│ │ Leftover Data │ -> │ Underlying Socket Stream │ │
│ │ (body bytes) │ │ (or SSL stream) │ │
│ │ [H,e,l,l,o] │ │ │ │
│ └─────────────────┘ └─────────────────────────────────┘ │
│ │
│ read() behavior: │
│ 1. First returns buffered leftover data │
│ 2. Then delegates to underlying stream │
│ 3. Seamless to caller - just looks like one stream │
└─────────────────────────────┬───────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Body Reading Code (unchanged) │
│ axis2_http_simple_response_get_body_bytes() │
│ │
│ Calls axutil_stream_read() - unaware of prepend wrapper │
│ Receives complete body: leftover + remaining from socket │
└─────────────────────────────────────────────────────────────────┘
The Decorator Pattern Implementation¶
The prepend stream wraps the underlying socket/SSL stream transparently:
typedef struct axis2_prepend_stream_impl
{
axutil_stream_t stream; /* Must be first - allows casting */
axutil_stream_t *underlying; /* Original socket/SSL stream */
axis2_char_t *prepend_data; /* Leftover data from header reading */
int prepend_pos; /* Current position in prepend buffer */
int prepend_len; /* Total length of prepend data */
} axis2_prepend_stream_impl_t;
Polymorphic read operation:
static int AXIS2_CALL
axis2_prepend_stream_read(axutil_stream_t *stream, const axutil_env_t *env,
void *buffer, size_t count)
{
axis2_prepend_stream_impl_t *impl = (axis2_prepend_stream_impl_t *)stream;
int total_read = 0;
/* Phase 1: Return buffered leftover data first */
if (impl->prepend_data && impl->prepend_pos < impl->prepend_len)
{
int available = impl->prepend_len - impl->prepend_pos;
int to_copy = (available < (int)count) ? available : (int)count;
memcpy(buffer, impl->prepend_data + impl->prepend_pos, to_copy);
impl->prepend_pos += to_copy;
total_read = to_copy;
if (total_read >= (int)count)
return total_read; /* Request fully satisfied from buffer */
buffer += to_copy;
count -= to_copy;
}
/* Phase 2: Delegate to underlying stream for remainder */
if (impl->underlying && impl->underlying->read)
{
int underlying_read = impl->underlying->read(impl->underlying, env, buffer, count);
if (underlying_read > 0)
total_read += underlying_read;
}
return total_read;
}
Algorithm: Buffered Header Parsing¶
#define AXIS2_HTTP_READ_BUFFER_SIZE 4096
while(!end_of_headers)
{
/* Step 1: Read chunk from socket (one syscall for up to 4KB) */
if(buf_pos >= buf_len)
{
bytes_read = axutil_stream_read(client->data_stream, env,
read_buffer, AXIS2_HTTP_READ_BUFFER_SIZE);
buf_pos = 0;
buf_len = bytes_read;
}
/* Step 2: Scan buffer in memory (no syscalls) */
while(buf_pos < buf_len && !end_of_headers)
{
char c = read_buffer[buf_pos++];
/* Build current line character by character */
str_line[str_line_len++] = c;
/* Check for CRLF (end of line) */
if(str_line_len >= 2 &&
str_line[str_line_len-2] == '\r' &&
str_line[str_line_len-1] == '\n')
{
if(str_line_len == 2) /* Empty line = end of headers */
end_of_headers = AXIS2_TRUE;
else
process_header_line(str_line);
str_line_len = 0; /* Reset for next line */
}
}
}
/* Step 3: Handle leftover body data */
if(buf_pos < buf_len)
{
int leftover_len = buf_len - buf_pos;
char *leftover_data = AXIS2_MALLOC(env->allocator, leftover_len);
memcpy(leftover_data, read_buffer + buf_pos, leftover_len);
/* Wrap in prepend stream for transparent body reading */
body_stream = axis2_prepend_stream_create(env, leftover_data,
leftover_len, client->data_stream);
}
Security Features¶
The implementation includes comprehensive security hardening against malicious HTTP responses.
Security Test Coverage¶
| Attack Vector | Test | Protection |
|---|---|---|
| Buffer overflow | 100KB header line | Truncation at AXIS2_HTTP_HEADER_LENGTH (4096) |
| Resource exhaustion | 10,000 headers | Headers stored, no limit enforced (future work) |
| Header injection | Embedded CRLF in value | Parsed as separate headers (logged) |
| Null byte injection | \0 in header |
C string truncation (safe) |
| Malformed status | INVALID RESPONSE |
Returns -1, no crash |
| Incomplete headers | Truncated response | Returns 0 (server shutdown) |
| CR-only line endings | \r without \n |
Timeout/failure (strict CRLF) |
| LF-only line endings | \n without \r |
Timeout/failure (strict CRLF) |
| Negative Content-Length | -1 |
Parsed as-is (caller handles) |
| Huge Content-Length | 999999999999 |
No pre-allocation, returns -1 |
| Slowloris | 1 byte/ms delivery | 5 second timeout protection |
Security Test Examples¶
Long Header Protection:
TEST(security_long_header_line)
{
/* 100KB header - potential buffer overflow attempt */
std::string long_value(100 * 1024, 'X');
std::string response = "HTTP/1.1 200 OK\r\n"
"X-Malicious-Header: " + long_value + "\r\n"
"Content-Length: 0\r\n\r\n";
/* Should not crash - truncates at AXIS2_HTTP_HEADER_LENGTH */
int status = send_request_and_get_status(env, port);
ASSERT_TRUE(status == 200 || status <= 0); /* Success or graceful failure */
}
Malformed Status Line Protection:
TEST(security_malformed_status_line)
{
/* Invalid HTTP response - potential crash vector */
const char *response = "INVALID GARBAGE DATA\r\n\r\n";
int status = send_request_and_get_status(env, port);
ASSERT_EQ(status, -1); /* Graceful failure, no crash */
}
Security Code Paths¶
Header length limit:
if(str_line_len < AXIS2_HTTP_HEADER_LENGTH - 1)
{
str_line[str_line_len++] = c;
}
else
{
AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
"Header line exceeds maximum length %d", AXIS2_HTTP_HEADER_LENGTH);
/* Continue scanning for line end, but truncate data */
}
NULL response protection:
/* Handle case where status line parsing failed - response will be NULL */
if(!client->response)
{
AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
"Failed to create HTTP response - status line may be malformed");
if(status_line)
axis2_http_status_line_free(status_line, env);
return -1; /* Graceful failure instead of NULL dereference */
}
Performance Comparison¶
| Metric | Byte-by-Byte | Buffered (4KB) | Improvement |
|---|---|---|---|
| Syscalls per 1KB headers | ~1000 | ~1 | 1000x fewer |
strstr() calls |
~1000 | 0 | Eliminated |
strcat() calls |
~1000 | 0 | Eliminated |
| Memory scans | O(n²) | O(n) | Linear |
| Typical header parse time | ~5ms | ~0.1ms | 50x faster |
Stream Type Usage¶
The implementation leverages AXIS2_STREAM_MANAGED for the prepend stream:
impl->stream.stream_type = AXIS2_STREAM_MANAGED;
impl->stream.read = axis2_prepend_stream_read;
impl->stream.write = axis2_prepend_stream_write;
impl->stream.skip = axis2_prepend_stream_skip;
/* Store cleanup info in base stream fields */
impl->stream.buffer_head = prepend_data; /* Freed by MANAGED handler */
impl->stream.fp = (FILE *)underlying; /* Freed as stream by MANAGED handler */
The AXIS2_STREAM_MANAGED type in axutil_stream_free() handles cleanup:
case AXIS2_STREAM_MANAGED:
if(stream->buffer_head)
AXIS2_FREE(env->allocator, stream->buffer_head);
if(stream->fp)
axutil_stream_free((axutil_stream_t *)stream->fp, env);
AXIS2_FREE(env->allocator, stream);
break;
Bug Fixes Included¶
1. Empty Body Returns FAILURE (Fixed)¶
Problem: axis2_http_client_send() returned AXIS2_FAILURE for requests with no body (GET, or POST with Content-Length: 0).
Fix:
else
{
/* No body to send (e.g., GET or POST with Content-Length: 0).
* Headers were already written successfully, so return SUCCESS.
*/
status = AXIS2_SUCCESS;
}
2. NULL Pointer Crash on Malformed Status (Fixed)¶
Problem: Malformed HTTP status lines caused NULL pointer dereference in axis2_http_simple_response_set_body_stream().
Fix: Added NULL check before using client->response:
if(!client->response)
{
AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
"Failed to create HTTP response - status line may be malformed");
return -1;
}
Files Modified¶
| File | Changes |
|---|---|
src/core/transport/http/sender/http_client.c |
Buffered reading, prepend stream, bug fixes |
util/src/stream.c |
AXIS2_STREAM_MANAGED cleanup support |
test/core/transport/http/test_axis2c_1480.cc |
15 security/functional tests |
test/core/transport/http/Makefile.am |
Build configuration for new tests |
Related Issues¶
- AXIS2C-1480: Buffered header reading (main issue)
- AXIS2C-1568: Timeout detection for keep-alive retry logic
- CVE Prevention: Security hardening against malicious responses
References¶
- HTTP/1.1 RFC 7230 - Message Syntax
- OWASP HTTP Response Splitting
- CWE-113: Improper Neutralization of CRLF
