OpenSSL: Made cert hostname check conform to RFC 6125

This change replaces RFC 2818 based hostname check in OpenSSL build with
RFC 6125 [1] based one.

The hostname check in RFC 2818 is ambiguous and each project implements
it in the their own way and they are slightly different. I check curl,
gnutls, Firefox and Chrome and they are all different.

I don't think there is a bug in current implementation of hostname
check. But it is not as strict as the modern browsers do. Currently,
curl allows multiple wildcard character '*' and it matches '.'. (as
described in the comment in ssluse.c).

Firefox implementation is also based on RFC 2818 but it only allows at
most one wildcard character and it must be in the left-most label in the
pattern and the wildcard must not be followed by any character in the
label.[2] Chromium implementation is based on RFC 6125 as my patch does.
Firefox and Chromium both require wildcard in the left-most label in the
presented identifier.

This patch is more strict than the current implementation, so there may
be some cases where old curl works but new one does not. But at the same
time I think it is good practice to follow the modern browsers do and
follow the newer RFC.

[1] http://tools.ietf.org/html/rfc6125#section-6.4.3
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=159483

1 files changed, 37 insertions, 27 deletions

diff --git a/lib/ssluse.c b/lib/ssluse.c
index 74563c7ee..8652cbd7c 100644
--- a/lib/ssluse.c
+++ b/lib/ssluse.c
@@ -1048,40 +1048,50 @@ static int asn1_output(const ASN1_UTCTIME *tm,
  * E.g.
  *  "foo.host.com" matches "*.host.com".
  *
- * We are a bit more liberal than RFC2818 describes in that we
- * accept multiple "*" in pattern (similar to what some other browsers do).
- * E.g.
- *  "abc.def.domain.com" should strickly not match "*.domain.com", but we
- *  don't consider "." to be important in CERT checking.
+ * We use the matching rule described in RFC6125, section 6.4.3.
+ * http://tools.ietf.org/html/rfc6125#section-6.4.3
  */
 #define HOST_NOMATCH 0
 #define HOST_MATCH   1
 
 static int hostmatch(const char *hostname, const char *pattern)
 {
-  for(;;) {
-    char c = *pattern++;
-
-    if(c == '\0')
-      return (*hostname ? HOST_NOMATCH : HOST_MATCH);
-
-    if(c == '*') {
-      c = *pattern;
-      if(c == '\0')      /* "*\0" matches anything remaining */
-        return HOST_MATCH;
-
-      while(*hostname) {
-        /* The only recursive function in libcurl! */
-        if(hostmatch(hostname++,pattern) == HOST_MATCH)
-          return HOST_MATCH;
-      }
-      break;
-    }
-
-    if(Curl_raw_toupper(c) != Curl_raw_toupper(*hostname++))
-      break;
+  const char *pattern_label_end, *pattern_wildcard, *hostname_label_end;
+  int wildcard_enabled;
+  size_t prefixlen, suffixlen;
+  pattern_wildcard = strchr(pattern, '*');
+  if(pattern_wildcard == NULL) {
+    return Curl_raw_equal(pattern, hostname) ? HOST_MATCH : HOST_NOMATCH;
+  }
+  /* We require at least 2 dots in pattern to avoid too wide wildcard
+     match. */
+  wildcard_enabled = 1;
+  pattern_label_end = strchr(pattern, '.');
+  if(pattern_label_end == NULL || strchr(pattern_label_end+1, '.') == NULL ||
+     pattern_wildcard > pattern_label_end ||
+     Curl_raw_nequal(pattern, "xn--", 4)) {
+    wildcard_enabled = 0;
+  }
+  if(!wildcard_enabled) {
+    return Curl_raw_equal(pattern, hostname) ? HOST_MATCH : HOST_NOMATCH;
+  }
+  hostname_label_end = strchr(hostname, '.');
+  if(hostname_label_end == NULL ||
+     !Curl_raw_equal(pattern_label_end, hostname_label_end)) {
+    return HOST_NOMATCH;
+  }
+  /* The wildcard must match at least one character, so the left-most
+     label of the hostname is at least as large as the left-most label
+     of the pattern. */
+  if(hostname_label_end - hostname < pattern_label_end - pattern) {
+    return HOST_NOMATCH;
   }
-  return HOST_NOMATCH;
+  prefixlen = pattern_wildcard - pattern;
+  suffixlen = pattern_label_end - (pattern_wildcard+1);
+  return Curl_raw_nequal(pattern, hostname, prefixlen) &&
+    Curl_raw_nequal(pattern_wildcard+1, hostname_label_end - suffixlen,
+                    suffixlen) ?
+    HOST_MATCH : HOST_NOMATCH;
 }
 
 static int