diff options
| -rw-r--r-- | docs/TheArtOfHttpScripting | 345 |
1 files changed, 345 insertions, 0 deletions
diff --git a/docs/TheArtOfHttpScripting b/docs/TheArtOfHttpScripting new file mode 100644 index 000000000..bf065899c --- /dev/null +++ b/docs/TheArtOfHttpScripting @@ -0,0 +1,345 @@ +Author: Daniel Stenberg <daniel@haxx.se> +Date: August 7, 2000 +Version: 0.2 + + The Art Of Scripting HTTP Requests Using Curl + ============================================= + + This document will assume that you're familiar with HTML and general + networking. + + The possibility to write scripts is essential to make a good computer + system. Unix' capability to be extended by shell scripts and various tools to + run various automated commands and scripts is one reason why it has succeeded + so well. + + The increasing amount of applications moving to the web has made "HTTP + Scripting" more frequently requested and wanted. To be able to automatically + extract information from the web, to fake users, to post or upload data to + web servers are all important tasks today. + + Curl is a command line tool for doing all sorts of URL manipulations and + transfers, but this particular document will focus on how to use it when + doing HTTP requests for fun and profit. I'll assume that you know how to + invoke 'curl --help' or 'curl --manual' to get basic information about it. + + Curl is not written to do everything for you. It makes the requests, it gets + the data, it sends data and it retrieves the information. You probably need + to glue everything together using some kind of script language or repeated + manual invokes. + +1. The HTTP Protocol + + HTTP is the protocol used to fetch data from web servers. It is a very simple + protocol that is built upon TCP/IP. The protocol also allow information to + get sent to the server from the client using a few different methods, as will + be shown here. + + HTTP is plain ASCII text lines being sent by the client to a server to + request a particular action, and then the server replies a few text lines + before the actual requested content is sent to the client. + + Using curl's option -v will display what kind of commands curl sends to the + server, as well as a few other informational texts. -v is the single most + useful option when it comes to debug or even understand the curl<->server + interaction. + +2. URL + + The Uniform Resource Locator format is how you specify the address of a + particular resource on the internet. You know these, you've seen URLs like + http://curl.haxx.se or https://yourbank.com a million times. + +3. GET a page + + The simplest and most common request/operation made using HTTP is to get a + URL. The URL could itself refer to a web page, an image or a file. The client + issues a GET request to the server and receives the document it asked for. + If you isse the command line + + curl http://curl.haxx.se + + you get a web page returned in your terminal window. The entire HTML document + that that URL holds. + + All HTTP replies contain a set of headers that are normally hidden, use + curl's -i option to display them as well as the rest of the document. You can + also ask the remote server for ONLY the headers by using the -I option. + +4. Forms + + Forms are the general way a web site can present a HTML page with fields for + the user to enter data in, and then press some kind of 'OK' or 'submit' + button to get that data sent to the server. The server then typically uses + the posted data to decide how to act. Like using the entered words to search + in a database, or to add the info in a bug track system, display the entered + address on a map or using the info as a login-prompt verifying that the user + is allowed to see what it is about to see. + + Of course there has to be some kind of program in the server end to receive + the data you send. You cannot just invent something out of the air. + + 4.1 GET + + A GET-form uses the method GET, as specified in HTML like: + + <form method="GET" action="junk.cgi"> + <input type=text name="birthyear"> + <input type=submit name=press value="OK"> + </form> + + In your favourite browser, this form will appear with a text box to fill in + and a press-button labeled "OK". If you fill in '1905' and press the OK + button, your browser will then create a new URL to get for you. The URL will + get "junk.cgi?birthyear=1905&press=OK" appended to the path part of the + previous URL. + + If the original form was seen on the page "www.hotmail.com/when/birth.html", + the second page you'll get will become + "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK". + + Most search engines work this way. + + To make curl do the GET form post for you, just enter the expected created + URL: + + curl "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK" + + 4.2 POST + + The GET method makes all input field names get displayed in the URL field of + your browser. That's generally a good thing when you want to be able to + bookmark that page with your given data, but it is an obvious disadvantage + if you entered secret information in one of the fields or if there are a + large amount of fields creating a very long and unreadable URL. + + The HTTP protocol then offers the POST method. This way the client sends the + data separated from the URL and thus you won't see any of it in the URL + address field. + + The form would look very similar to the previous one: + + <form method="POST" action="junk.cgi"> + <input type=text name="birthyear"> + <input type=submit name=press value="OK"> + </form> + + And to use curl to post this form with the same data filled in as before, we + could do it like: + + curl -d "birthyear=1905&press=OK" www.hotmail.com/when/junk.cgi + + This kind of POST will use the Content-Type + application/x-www-form-urlencoded and is the most widly used POST kind. + + 4.3 FILE UPLOAD POST + + Back in late 1995 they defined a new to post data over HTTP. It was + documented in the RFC 1867, why this method sometimes are refered to as + a rfc1867-posting. + + This method is mainly designed to better support file uploads. A form that + allows a user to upload a file could be written like this in HTML: + + <form method="POST" enctype='multipart/form-data' action="upload.cgi"> + <input type=file name=upload> + <input type=submit name=press value="OK"> + </form> + + This clearly shows that the Content-Type about to be sent is + multipart/form-data. + + To post to a form like this with curl, you enter a command line like: + + curl -F upload=@localfilename -F press=OK [URL] + + 4.4 HIDDEN FIELDS + + A very common way for HTML based application to pass state information + between pages is to add hidden fields to the forms. Hidden fields are + already filled in, they aren't displayed to the user and they get passed + along just as all the other fields. + + A similar example form with one visible field, one hidden field and one + submit button could look like: + + <form method="POST" action="foobar.cgi"> + <input type=text name="birthyear"> + <input type=text name="person" value="daniel"> + <input type=submit name="press" value="OK"> + </form> + + To post this with curl, you won't have to think about if the fields are + hidden or not. To curl they're all the same: + + curl -d "birthyear=1905&press=OK&person=daniel" [URL] + +5. PUT + + The perhaps best way to upload data to a HTTP server is to use PUT. Then + again, this of course requires that someone put a program or script on the + server end that knows how to receive a HTTP PUT stream. + + Put a file to a HTTP server with curl: + + curl -t uploadfile www.uploadhttp.com/receive.cgi + +6. AUTHENTICATION + + Authentication is the ability to tell the server your username and password + so that it can verify that you're allowed to do the request you're doing. The + basic authentication used in HTTP is *plain* *text* based, which means it + sends username and password only slightly obfuscated, but still fully + readable by anyone that sniffs on the network between you and the remote + server. + + To tell curl to use a user and password for authentication: + + curl -u name:password www.secrets.com + + Sometimes your HTTP access is only available through the use of a HTTP + proxy. This seems to be especially common at various companies. A HTTP proxy + may require its own user and password to allow the client to get through to + the internet. To specify those with curl, run something like: + + curl -U proxyuser:proxypassword curl.haxx.se + + If you use any one these user+password options but leave out the password + part, curl will prompt for the password interactively. + + Do note that when a program is run, its parameters are possible to see when + listing the running processes of the system. Thus, other users may be able to + watch your passwords if you pass them as plain command line options. + +7. REFERER + + A HTTP request has the ability to feature a 'referer' field, which can be + used to tell which URL that causes the client to get this particular + resource. Some programs/scripts check the referer field of requests to verify + that this wasn't arriving from an external site or unknown page. While this + is a stupid way to check something so easily forged, many scripts still do + it. Using curl, you can put anything you want in the referer-field and thus + more easily being able to fool the server into serving your request. + + Use curl to set the referer field with: + + curl -e http://curl.haxx.se daniel.haxx.se + +8. USER AGENT + + Very similar to the referer field, all HTTP requests may set the User-Agent + field. It names what user agent (client) that is being used. Many + applications use this information to decide how to display pages. Silly web + programmers try to make different pages for users of different browsers to + make them look the best possible for their particular browsers. They usually + also do different kinds of javascript, vbscript etc. + + At times, you will see that getting a page will curl will not return the very + same page that you see when getting the page with your browser. Then you know + it is time to set the User Agent field to fool the server into thinking + you're one of those browsers. + + To make curl look like Internet Explorer on a Windows 2000 box: + + curl -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL] + + Or why not look like you're using Netscape 4.73 on a Linux (PIII) box: + + curl -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL] + +9. REDIRECTS + + When a resource is requested from a server, the reply from the server may + include a hint about where the browser should go next to find this page, or a + new page keeping newly generated output. The header that tells the browser + to redirect is Location:. + + Curl does not follow Location: headers by default, but will simply display + such pages in the same manner it display all HTTP replies. It does however + feature an option that will make it attempt to follow the Location: pointers. + + To tell curl to follow a Location: + + curl -L www.sitethatredirects.com + + If you use curl to POST to a site that immediately redirects you to another + page, you can safely use -L and -d/-F together. Curl will only use POST in + the first request, and then revert to GET in the following operations. + +10. COOKIES + + The way the web browsers do "client side state control" is by using + cookies. Cookies are just names with associated contents. The cookies are + sent to the client by the server. The server tells the client for what path + and host name it wants the cookie sent back, and it also sends an expiration + date and a few more properties. + + When a client communicates with a server with a name and path as previously + specified in a received cookie, the client sends back the cookies and their + contents to the server, unless of course they are expired. + + Many applications and server use this method to connect a series of request + into a single logical session. To be able to use curl in such occations, we + must be able to record and send back cookies in the way that the web + application expects them. The same way browsers deal with them. + + The simplest way to send a few cookies to the server when getting a page with + curl is to add them on the command line like: + + curl -b "name=Daniel" www.cookiesite.com + + + Cookies are sent as common HTTP headers. This is practical as it allows curl + to record cookies simply by recording headers. Record cookies with curl by + using the -D option like: + + curl -D headers_and_cookies www.cookiesite.com + + Curl has a full blown cookie parsing engine built-in that comes to use if you + want to reconnect to a server and use cookies that were stored from a + previous connection (or handicrafted manually to fool the server into + believing you had a previous connection). To use previously stored cookies, + you run curl like: + + curl -b stored_cookies_in_file www.cookiesite.com + +11. HTTPS + + There are a few ways to do secure HTTP transfers. The by far most common + protocol for doing this is what is generally known as HTTPS, HTTP over + SSL. SSL encrypts all the data that is send and received over the network and + thus makes it harder for attackers to spy on sensitive information. + + SSL (or TLS as the latest version of the standard is called) offers a + truckload of advanced features to allow all those encryptions and key + infrastructure mechanisms ecnrypted HTTP requires. + + Curl supports enscrypted fetches thanks to the freely available OpenSSL + libraries. To get a pafe from a https server, simply run curl like: + + curl https://that.secure.server.com + + 11.1 CERTIFICATES + + In the HTTPS world, you use certificates to validate that you are the one + you you claim to be, as an addition to normal passwords. Curl supports + client-side certificates. All certificates are locked with a PIN-code, why + you need to enter the unlock-code before the certificate can be used by + curl. The PIN-code can be specified on the command line or if not, entered + interactively when curl queries for it. Use a certificate with curl on a + https server like: + + curl -E mycert.pem https://that.secure.server.com + +12. REFERENCES + + RFC 2616 is a must to read if you want in-depth understanding of the HTTP + protocol. + + RFC 2396 explains the URL syntax + + RFC 2109 defines how cookies are supposed to work. + + http://www.openssl.org is the home of the OpenSSL project + + http://curl.haxx.se is the home of the cURL project |