INTERNALS: absorbed docs/LIBCURL-STRUCTS

1 files changed, 233 insertions, 1 deletions

diff --git a/docs/INTERNALS b/docs/INTERNALS
index b134260c3..4cd63b4e2 100644
--- a/docs/INTERNALS
+++ b/docs/INTERNALS
@@ -37,6 +37,7 @@ Table of Contents
  - [hostip.c explained](#hostip)
  - [Track Down Memory Leaks](#memoryleak)
  - [`multi_socket`](#multi_socket)
+ - [Structs in libcurl](#structs)
 
 <a name="intro"></a>
 curl internals
@@ -848,6 +849,235 @@ Track Down Memory Leaks
   only `fd_sets` and that is no longer good enough. The changes done to c-ares
   are available in c-ares 1.3.1 and later.
 
+<a name="structs"></a>
+Structs in libcurl
+==================
+
+This section should cover 7.32.0 pretty accurately, but will make sense even
+for older and later versions as things don't change drastically that often.
+
+## SessionHandle
+
+  The SessionHandle handle struct is the one returned to the outside in the
+  external API as a "CURL *". This is usually known as an easy handle in API
+  documentations and examples.
+
+  Information and state that is related to the actual connection is in the
+  'connectdata' struct. When a transfer is about to be made, libcurl will
+  either create a new connection or re-use an existing one. The particular
+  connectdata that is used by this handle is pointed out by
+  SessionHandle->easy_conn.
+
+  Data and information that regard this particular single transfer is put in
+  the SingleRequest sub-struct.
+
+  When the SessionHandle struct is added to a multi handle, as it must be in
+  order to do any transfer, the ->multi member will point to the `Curl_multi`
+  struct it belongs to. The ->prev and ->next members will then be used by the
+  multi code to keep a linked list of SessionHandle structs that are added to
+  that same multi handle. libcurl always uses multi so ->multi *will* point to
+  a `Curl_multi` when a transfer is in progress.
+
+  ->mstate is the multi state of this particular SessionHandle. When
+  `multi_runsingle()` is called, it will act on this handle according to which
+  state it is in. The mstate is also what tells which sockets to return for a
+  specific SessionHandle when [`curl_multi_fdset()`][12] is called etc.
+
+  The libcurl source code generally use the name 'data' for the variable that
+  points to the SessionHandle.
+
+  When doing multiplexed HTTP/2 transfers, each SessionHandle is associated
+  with an individual stream, sharing the same connectdata struct. Multiplexing
+  makes it even more important to keep things associated with the right thing!
+
+## connectdata
+
+  A general idea in libcurl is to keep connections around in a connection
+  "cache" after they have been used in case they will be used again and then
+  re-use an existing one instead of creating a new as it creates a significant
+  performance boost.
+
+  Each 'connectdata' identifies a single physical connection to a server. If
+  the connection can't be kept alive, the connection will be closed after use
+  and then this struct can be removed from the cache and freed.
+
+  Thus, the same SessionHandle can be used multiple times and each time select
+  another connectdata struct to use for the connection. Keep this in mind, as
+  it is then important to consider if options or choices are based on the
+  connection or the SessionHandle.
+
+  Functions in libcurl will assume that connectdata->data points to the
+  SessionHandle that uses this connection (for the moment).
+
+  As a special complexity, some protocols supported by libcurl require a
+  special disconnect procedure that is more than just shutting down the
+  socket. It can involve sending one or more commands to the server before
+  doing so. Since connections are kept in the connection cache after use, the
+  original SessionHandle may no longer be around when the time comes to shut
+  down a particular connection. For this purpose, libcurl holds a special
+  dummy `closure_handle` SessionHandle in the `Curl_multi` struct to use when
+  needed.
+
+  FTP uses two TCP connections for a typical transfer but it keeps both in
+  this single struct and thus can be considered a single connection for most
+  internal concerns.
+
+  The libcurl source code generally use the name 'conn' for the variable that
+  points to the connectdata.
+
+## Curl_multi
+
+  Internally, the easy interface is implemented as a wrapper around multi
+  interface functions. This makes everything multi interface.
+
+  `Curl_multi` is the multi handle struct exposed as "CURLM *" in external APIs.
+
+  This struct holds a list of SessionHandle structs that have been added to
+  this handle with [`curl_multi_add_handle()`][13]. The start of the list is
+  ->easyp and ->num_easy is a counter of added SessionHandles.
+
+  ->msglist is a linked list of messages to send back when
+  [`curl_multi_info_read()`][14] is called. Basically a node is added to that
+  list when an individual SessionHandle's transfer has completed.
+
+  ->hostcache points to the name cache. It is a hash table for looking up name
+  to IP. The nodes have a limited life time in there and this cache is meant
+  to reduce the time for when the same name is wanted within a short period of
+  time.
+
+  ->timetree points to a tree of SessionHandles, sorted by the remaining time
+  until it should be checked - normally some sort of timeout. Each
+  SessionHandle has one node in the tree.
+
+  ->sockhash is a hash table to allow fast lookups of socket descriptor to
+  which SessionHandle that uses that descriptor. This is necessary for the
+  `multi_socket` API.
+
+  ->conn_cache points to the connection cache. It keeps track of all
+  connections that are kept after use. The cache has a maximum size.
+
+  ->closure_handle is described in the 'connectdata' section.
+
+  The libcurl source code generally use the name 'multi' for the variable that
+  points to the Curl_multi struct.
+
+## Curl_handler
+
+  Each unique protocol that is supported by libcurl needs to provide at least
+  one `Curl_handler` struct. It defines what the protocol is called and what
+  functions the main code should call to deal with protocol specific issues.
+  In general, there's a source file named [protocol].c in which there's a
+  "struct `Curl_handler` `Curl_handler_[protocol]`" declared. In url.c there's
+  then the main array with all individual `Curl_handler` structs pointed to
+  from a single array which is scanned through when a URL is given to libcurl
+  to work with.
+
+  ->scheme is the URL scheme name, usually spelled out in uppercase. That's
+  "HTTP" or "FTP" etc. SSL versions of the protcol need its own `Curl_handler`
+  setup so HTTPS separate from HTTP.
+
+  ->setup_connection is called to allow the protocol code to allocate protocol
+  specific data that then gets associated with that SessionHandle for the rest
+  of this transfer. It gets freed again at the end of the transfer. It will be
+  called before the 'connectdata' for the transfer has been selected/created.
+  Most protocols will allocate its private 'struct [PROTOCOL]' here and assign
+  SessionHandle->req.protop to point to it.
+
+  ->connect_it allows a protocol to do some specific actions after the TCP
+  connect is done, that can still be considered part of the connection phase.
+
+  Some protocols will alter the connectdata->recv[] and connectdata->send[]
+  function pointers in this function.
+
+  ->connecting is similarly a function that keeps getting called as long as the
+  protocol considers itself still in the connecting phase.
+
+  ->do_it is the function called to issue the transfer request. What we call
+  the DO action internally. If the DO is not enough and things need to be kept
+  getting done for the entire DO sequence to complete, ->doing is then usually
+  also provided. Each protocol that needs to do multiple commands or similar
+  for do/doing need to implement their own state machines (see SCP, SFTP,
+  FTP). Some protocols (only FTP and only due to historical reasons) has a
+  separate piece of the DO state called `DO_MORE`.
+
+  ->doing keeps getting called while issuing the transfer request command(s)
+
+  ->done gets called when the transfer is complete and DONE. That's after the
+  main data has been transferred.
+
+  ->do_more gets called during the `DO_MORE` state. The FTP protocol uses this
+  state when setting up the second connection.
+
+  ->`proto_getsock`
+  ->`doing_getsock`
+  ->`domore_getsock`
+  ->`perform_getsock`
+  Functions that return socket information. Which socket(s) to wait for which
+  action(s) during the particular multi state.
+
+  ->disconnect is called immediately before the TCP connection is shutdown.
+
+  ->readwrite gets called during transfer to allow the protocol to do extra
+  reads/writes
+
+  ->defport is the default report TCP or UDP port this protocol uses
+
+  ->protocol is one or more bits in the `CURLPROTO_*` set. The SSL versions
+  have their "base" protocol set and then the SSL variation. Like
+  "HTTP|HTTPS".
+
+  ->flags is a bitmask with additional information about the protocol that will
+  make it get treated differently by the generic engine:
+
+  - `PROTOPT_SSL` - will make it connect and negotiate SSL
+
+  - `PROTOPT_DUAL` - this protocol uses two connections
+
+  - `PROTOPT_CLOSEACTION` - this protocol has actions to do before closing the
+    connection. This flag is no longer used by code, yet still set for a bunch
+    protocol handlers.
+  
+  - `PROTOPT_DIRLOCK` - "direction lock". The SSH protocols set this bit to
+    limit which "direction" of socket actions that the main engine will
+    concern itself about.
+
+  - `PROTOPT_NONETWORK` - a protocol that doesn't use network (read file:)
+
+  - `PROTOPT_NEEDSPWD` - this protocol needs a password and will use a default
+    one unless one is provided
+
+  - `PROTOPT_NOURLQUERY` - this protocol can't handle a query part on the URL
+    (?foo=bar)
+
+## conncache
+
+  Is a hash table with connections for later re-use. Each SessionHandle has
+  a pointer to its connection cache. Each multi handle sets up a connection
+  cache that all added SessionHandles share by default.
+
+## Curl_share
+  
+  The libcurl share API allocates a `Curl_share` struct, exposed to the
+  external API as "CURLSH *".
+
+  The idea is that the struct can have a set of own versions of caches and
+  pools and then by providing this struct in the `CURLOPT_SHARE` option, those
+  specific SessionHandles will use the caches/pools that this share handle
+  holds.
+
+  Then individual SessionHandle structs can be made to share specific things
+  that they otherwise wouldn't, such as cookies.
+
+  The `Curl_share` struct can currently hold cookies, DNS cache and the SSL
+  session cache.
+
+## CookieInfo
+
+  This is the main cookie struct. It holds all known cookies and related
+  information. Each SessionHandle has its own private CookieInfo even when
+  they are added to a multi handle. They can be made to share cookies by using
+  the share API.
+
 
 [1]: http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
 [2]: http://curl.haxx.se/libcurl/c/curl_easy_init.html
@@ -860,4 +1090,6 @@ Track Down Memory Leaks
 [9]: http://curl.haxx.se/libcurl/c/curl_multi_setopt.html
 [10]: http://curl.haxx.se/libcurl/c/CURLMOPT_TIMERFUNCTION.html
 [11]: http://curl.haxx.se/libcurl/c/curl_multi_perform.html
-[12]: http://curl.haxx.se/libcurl/c/curl_multi_fdset.html
\ No newline at end of file
+[12]: http://curl.haxx.se/libcurl/c/curl_multi_fdset.html
+[13]: http://curl.haxx.se/libcurl/c/curl_multi_add_handle.html
+[14]: http://curl.haxx.se/libcurl/c/curl_multi_info_read.html