Fork me on GitHub

URI.js

Understanding URIs

Uniform Resource Identifiers (URI) can be one of two things, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). You likely deal with URLs most of the time. See RFC 3986 for a proper definition of the terms URI, URL and URN

URNs name a resource. They are (supposed to) designate a globally unique, permanent identifier for that resource. For example, the URN urn:isbn:0201896834 uniquely identifies Volume 1 of Donald Knuth's The Art of Computer Porgramming. Even if that book goes out of print, that URN will continue to identify that particular book in that particular printing. While the term "URN" technically refers to a specific URI scheme laid out by RFC 2141, the previously-mentioned RFC 3986 indicates that in common usage "URN" refers to any kind of URI that identifies a resource.

URLs locate a resource. They designate a protocol to use when looking up the resource and provide an "address" for finding the resource within that scheme. For example, the URL http://tools.ietf.org/html/rfc3986 tells the consumer (most likely a web browser) to use the HTTP protocol to access whatever site is found at the /html/rfc3986 path of tools.ietf.org. URLs are not permanent; it is possible that in the future that the IETF will move to a different domain or even that some other organization will acquire the rights to tools.ietf.org. It is also possible that multiple URLs may locate the same resource; for example, an admin at the IETF might be able to access the document found at the example URL via the ftp:// protocol.

URLs and URNs in URI.js

The distinction between URLs and URNs is one of semantics. In principle, it is impossible to tell, on a purely syntactical level, whether a given URI is a URN or a URL without knowing more about its scheme. Practically speaking, however, URIs that look like HTTP URLs (scheme is followed by a colon and two slashes, URI has an authority component, and paths are delimited by slashes) tend to be URLs, and URIs that look like RFC 2141 URNs (scheme is followed by a colon, no authority component, and paths are delimited by colons) tend to be URNs (in the broad sense of "URIs that name").

So, for the purposes of URI.js, the distinction between URLs and URNs is treated as one of syntax. The main functional differences between the two are that (1) URNs will not have an authority element and (2) when breaking the path of the URI into segments, the colon will be used as the delimiter rather than the slash. The most surprising result of this is that mailto: URLs will be considered by URI.js to be URNs rather than URLs. That said, the functional differences will not adversely impact the handling of those URLs.

Components of an URI

RFC 3986 Section 3 visualizes the structure of URIs as follows:

URL:      foo://example.com:8042/over/there?name=ferret#nose
          \_/   \______________/\_________/ \_________/ \__/
           |           |            |            |        |
        scheme     authority       path        query   fragment
           |   _____________________|__
          / \ /                        \
URN:      urn:example:animal:ferret:nose

Components of an URL in URI.js

              origin
       __________|__________
      /                     \
                         authority
     |             __________|_________
     |            /                    \
              userinfo                host                          resource
     |         __|___                ___|___                 __________|___________
     |        /      \              /       \               /                      \
         username  password     hostname    port     path & segment      query   fragment
     |     __|___   __|__    ______|______   |   __________|_________   ____|____   |
     |    /      \ /     \  /             \ / \ /                    \ /         \ / \
    foo://username:password@www.example.com:123/hello/world/there.html?name=ferret#foo
    \_/                     \ / \       \ /    \__________/ \     \__/
     |                       |   \       |           |       \      |
  scheme               subdomain  \     tld      directory    \   suffix
                                   \____/                      \___/
                                      |                          |
                                    domain                   filename

In Javascript the query is often referred to as the search. URI.js provides both accessors with the subtle difference of .search() beginning with the ?-character and .query() not.

In Javascript the fragment is often referred to as the hash. URI.js provides both accessors with the subtle difference of .hash() beginning with the #-character and .fragment() not.

Components of an URN in URI.js

    urn:example:animal:ferret:nose?name=ferret#foo
    \ / \________________________/ \_________/ \ /
     |               |                  |       |
  scheme       path & segment         query   fragment
    

While RFC 2141 does not define URNs having a query or fragment component, URI.js enables these accessors for convenience.

URLs - Man Made Problems

URLs (URIs, whatever) aren't easy. There are a couple of issues that make this simple text representation of a resource a real pain

Parsing (seemingly) invalid URLs

Because URLs look very simple, most people haven't read the formal specification. As a result, most people get URLs wrong on many different levels. The one thing most everybody screws up is proper encoding/escaping.

http://username:pass:word@example.org/ is such a case. Often times homebrew URL handling misses escaping the less frequently used parts such as the userinfo.

http://example.org/@foo that "@" doesn't have to be escaped according to RFC3986. Homebrew URL handlers often just treat everything between "://" and "@" as the userinfo.

some/path/:foo is a valid relative path (as URIs don't have to contain scheme and authority). Since homebrew URL handlers usually just look for the first occurence of ":" to delimit the scheme, they'll screw this up as well.

+ is the proper escape-sequence for a space-character within the query string component, while every other component prefers %20. This is due to the fact that the actual format used within the query string component is not defined in RFC 3986, but in the HTML spec.

There is encoding and strict encoding - and Javascript won't get them right: encodeURIComponent()

Top Level Domains

The hostname component can be one of may things. An IPv4 or IPv6 address, an IDN or Punycode domain name, or a regular domain name. While the format (and meaning) of IPv4 and IPv6 addresses is defined in RFC 3986, the meaning of domain names is not.

DNS is the base of translating domain names to IP addresses. DNS itself only specifies syntax, not semantics. The missing semantics is what's driving us crazy here.

ICANN provides a list of registered Top Level Domains (TLD). There are country code TLDs (ccTLDs, assigned to each country, like ".uk" for United Kindom) and generic TLDs (gTLDs, like ".xxx" for you know what). Also note that a TLD may be non-ASCII .香港 (IDN version of HK, Hong Kong).

IDN TLDs such as .香港 and the fact that any possible new TLD could pop up next month has lead to a lot of URL/Domain verification tools to fail.

Second Level Domains

To make Things worse, people thought it to be a good idea to introduce Second Level Domains (SLD, ".co.uk" - the commercial namespace of United Kingdom). These SLDs are not up to ICANN to define, they're handled individually by each NIC (Network Information Center, the orgianisation responsible for a specific TLD).

Since there is no central oversight, things got really messy in this particular space. Germany doesn't do SDLs, Australia does. Australia has different SLDs than the United Kingdom (".csiro.au" but no ".csiro.uk"). The individual NICs are not required to publish their arbitrarily chosen SLDs in a defined syntax anywhere.

You can scour each NIC's website to find some hints at their SLDs. You can look them up on Wikipedia and hope they're right. Or you can use PublicSuffix.

Speaking of PublicSuffix, it's time mentioning that browser vendors actually keep a list of known Second Level Domains. They need to know those for security issues. Remember cookies? They can be read and set on a domain level. What do you think would happen if "co.uk" was treated as the domain? amazon.co.uk would be able to read the cookies of google.co.uk. PublicSuffix also contains custom SLDs, such as .dyndns.org. While this makes perfect sense for browser security, it's not what we need for basic URL handling.

TL;DR: It's a mess.

The Query String

PHP (parse_str()) will automatically parse the query string and populate the superglobal $_GET for you. ?foo=1&foo=2 becomes $_GET = array('foo' => 2);, while ?foo[]=1&foo[]=2 becomes $_GET = array('foo' => array("1", "2"));.

Ruby's CGI.parse() turns ?a=1&a=2 into {"a" : ["1", "2"]}, while Ruby on Rails chose the PHP-way.

Python's parse_qs() doesn't care for [] either.

Most other languages don't follow the []-style array-notation and deal with this mess differently.

TL;DR: You need to know the target-environment, to know how complex query string data has to be encoded

The Fragment

Given the URL http://example.org/index.html#foobar, browsers only request http://example.org/index.html, the fragment #foobar is a client-side thing.