Web server

Arc includes a web server with several interesting features, including a continuation-based design. The web server includes many different operations to create forms and links with associated continuations.

The Arc distribution comes with sample web applications including a new site (news.arc) and a blog (blog.arc).

Running the server

The server can be started simply with
arc>(serve 8080)

However, it is generally better to start the server in a separate thread, so the Arc REPL can be used. This allows the web server to be modified while it is running.

arc> (thread (serve 8080))
A handler can be associated with a URL using the defop macro; this defines a handler function, taking the req parameter:
(defop hello req (prn "Hello world!"))
The resulting page can be accessed at http://localhost:8080/hello.

The Arc web server is rather fragile and platform-dependent. If you encounter problems, the easiest solution is to use the unofficial Anarki version, which has multiple patches.

To define a top-level page (http://localhost:8080), use the page name || (the MzScheme representation of the empty symbol, between quoting vertical bars).

(defop || req (pr "This is the home page."))

To redirect a page to a different page, the defopr macro is used, with a function that outputs the name of the target page. For example, to redirect http://localhost:8080/index.html to the earlier "hello" page:

(defopr index.html req (prn "hello"))

The req parameter receives a table that has the field ip holding the IP address of the client, and potentially cooks holding the cookies, and args holding a list of key value pairs from the URL's query string.

The defop-raw and defopr-raw macros let the handler function add HTTP headers. The handler must then output a blank line followed by the HTML content or redirect path. One use of this is to add cookies to the headers.

 (defop-raw bar (str req) (w/stdout str
   (prn "Set-Cookie: mycookie=42")
   (prn)
   (prn (req 'ip)) (br) (prn (req 'cooks)) (br) (prn (req 'args))))
On the second reload (after the cookie gets assigned), http://localhost:8080/bar?x=1&y=2&z will display:
127.0.0.1
((mycookie 42))
((x 1) (y 2) (z ))
This illustrates how the handler can access the client's IP address, the cookies, and the URL query parameters. The handle does not have access to other HTTP headers.

This functionality can be used to implement a simple form and form handler; the form is at http://localhost:8080/myform. This example uses some of the HTML functions.

(defop myform req (form "myhandler" (single-input "Enter:" 'foo 10 "Submit")))
(defop myhandler req  (prn "You entered") (prbold (alref (req 'args) "foo")))

Several types of output functions are supported by Arc. The simplest is a function that outputs HTML. A function can also optionally output additional HTTP headers. Arc also supports redirect functions that can perform server-side operations and then redirect the browser to a new page; these functions can optionally output additional headers. Finally, Arc has partial support for asynchronous functions, which don't return any response to the request.

The following table shows the macros for generating arbitrary server-side handlers of the given types.
 HTMLHeaders + HTMLRedirectHeaders + Redirect
Macro defop defop-raw defopr defopr-raw

Continuations

The previous section showed how to implement basic HTML and redirect handlers in Arc. Arc's web server also provides continuation support, which is typically used for handlers. That is, links and forms can be assigned a function that will be executed on the server if the link or form is clicked. (The continuation would be called a callback in some languages.)

The continuations used by the web server are explicit function. (The web server does not use ccc first-class continuations.) The functions take a request object as argument and print the appropriate response. By using closures, the functions can maintain state across requests; the state will be included in the closure when the function is defined, and the function can access the state when it is later invoked.

The web server includes many operations to associate continuation functions and fnids with links and forms. The web server contains operations to generate links and forms of the various types, as shown in the following table.

The previous example can be implemented with continuations as follows:

(defop myform2 req (aform myfunc (single-input "Enter:" 'foo 10 "Submit")))
(def myfunc (req) (prn "You entered") (prbold (alref (req 'args) "foo")))
Generally, the continuation function is defined within the original form definition, so a closure can be created. (In this case, the closure is unnecessary, as no state from the first operation is preserved.)
(defop myform2 req (aform
  [do (prn "You entered") (prbold (alref (_ 'args) "foo"))]
  (single-input "Enter:" 'foo 10 "Submit")))

Internally, the web server associates a token called the fnid with each instance of a link or form, and records the continuation function for each token. Periodically, old fnids are deleted.

The following table illustrates the operations to generate links, URLs, or forms, for the four different types of output functions. (In practice, both arform and arformh give access to the headers.)
 HTMLHeaders + HTMLRedirectHeaders + Redirect
Link generation w/link, w/link-if, onlink, linkf w/rlink, rlinkf
URL generation flink, url-for rflink
Form generation aform, timed-aform aformh arform arformh

Different operations use one of five different mechanisms to specify the continuation function. The function may be specified as an expression, a function of one variable (the request object), a request parameter and body, output stream and request parameters and a body, or a body alone. The documentation should be consulted to see which mechanism goes with which function.

Explanation of the "Arc Challenge" solution

The Arc Challenge posed the problem of implementing a simple web function. Under the URL "/said", it provides a form with input field. When submitted, the form goes to a page with a link "click here". The link leads to a page with the text "You said: " followed by the original input.

The solution in Arc is as follows:

(defop said req
  (aform [w/link (pr "you said: " (arg _ "foo"))
           (pr "click here")]
    (input "foo") 
    (submit)))
The solution may be easier to understand if the continuation functions are made explicit:
(defop said req
  (aform form-continuation
    (input "foo") 
    (submit)))

(def form-continuation (req) 
  (w/link 
      (pr "you said: " (arg req "foo")) ;link continuation expression
    (pr "click here")))
The said operation creates a form that when submitted executes the form-continuation function. The continuation function creates a link that will execute the link continuation expression. The important thing to notice is the link continuation expression is defined inside form-continuation, which results in a closure with the req object from form-continuation. That is, when the link continuation executes, potentially a long time after form-continuation completes, it will still have the state. This illustrates how closures can be used to carry state from one request to another. The form can be executed multiple times, and each execution will have a unique state in the link continuation closure.

Web server

defop name req-param [body ...]
Defines a HTML response handler for the URL path name. The handler body takes one parameter, the req object and should output to stdout.
>(defop foo req (prn "Hello"))
#<procedure: gs2265>
defop-raw name (outstream-param req-param) [body ...]
Defines a HTML response handler for the URL path name. The handler body takes two parameters: the output stream and the req object. The handler can output additional HTTP headers.
>(defop-raw fooraw (str req)
  (w/stdout str (prn "X-Arc: foo") (prn) (prn "Hello")))
#<procedure: gs2271>
defopr name req-param [body ...]
Defines a redirect response handler for the URL path name. The handler body takes one parameter, the req object and should output to stdout.
>(defopr bar req (prn "foo"))
#<procedure: gs2282>
defopr-raw name (outstream-param req-param) [body ...]
Defines a redirect response handler for the URL path name. The handler body takes two parameters: the output stream and the req object. The handler can output additional HTTP headers.
>(defopr-raw baz (str req)
  (disp "Set-Cookie: this=that\n" str) "foo")
#<procedure: gs2291>
aform f [body ...]
Generates a form from body. f is a continuation function to execute when the form is submitted.
>(defop af rq (aform (fn (req) (prn req)) (submit)))
#<procedure: gs2301>
fnform f bodyfn [redir]
Generates a form from a body function. f is a continuation function to execute when the form is submitted.
>(defop af rq (fnform (fn (req) (fn (_) (prn req))) (submit)))
#<procedure: gs2313>
timed-aform lasts f [body ...]
Generates a form. The fnid has a lifetime of lasts seconds.
>(defop taf rq (timed-aform 10 (fn (req) (prn req)) (submit)))
#<procedure: gs2324>
timed-arform lasts f [body ...]
Generates a form. Like timed-aform except with a redirect. New in arc3.
>(defop taf rq (timed-arform 10 (fn (req) (prn req)) (submit)))
#<procedure: gs2335>
arform f [body ...]
Generates a form that goes to a redirect. The callback function f must return the redirect target.
>(defop arf rq (arform (fn (req) "foo") (prn "Click:") (submit)))
#<procedure: gs2346>
aformh f [body ...]
Generates a form with additional headers.
>(defop afh rq (aformh
  (fn (req) (prn "Set-Cookie: a=b\n") (prn req))
  (prn "Click:") (submit)))
#<procedure: gs2357>
arformh f [body ...]
Generates a form that goes to a redirect with additional headers.
>(defop arfh rq (arform (fn (req) (prn "Set-Cookie: a=b") "foo")
  (prn "Click:") (submit)))
#<procedure: gs2368>
url-for fnid
Generates a URL linking to the given fnid, using fnurl*.
>(url-for '1234)
"/x?fnid=1234"
afnid f
Generate anaphoric fnid. The continuation function f can access the fnid as the variable it. f must print a blank line before the HTML content.
>(afnid (fn (req) (prn "\nFnid is " it)))
ApzY33zZEd
flink f
Generates a URL that will run the continuation function f. It creates a fnid to do this.
>(flink (fn (req) (prn "hi")))
"/x?fnid=DltGUEIJsK"
rflink f
Generates a URL that will run the redirect continuation function f. It creates a fnid to do this.
>(rflink (fn (req) (prn "Header: x") "foo"))
"/r?fnid=wzWRwQ3LcI"
w/link expr [body ...]
Wraps body in a link to continuation expression expr.
>(defop wl req (w/link (prn "You clicked.") (prn "Click here")))
#<procedure: gs2386>
w/link-if test expr [body ...]
If test is true, wraps body in a link to continuation expression expr. Otherwise, just displays body without the link.
>(defop wli req (w/link-if t (prn "You clicked.")
  (prn "Click here")))
#<procedure: gs2398>
w/rlink expr [body ...]
Wraps body in a link to redirect continuation expression expr.
>(defop wrl req (w/rlink "foo" (prn "Click here")))
#<procedure: gs2410>
onlink text [body ...]
Creates a link with contents text that will execute continuation expression body. Note that onlink reverses the roles of the arguments compared to w/link.
>(defop ol req (onlink "Click here" (prn "You clicked.")))
#<procedure: gs2422>
onrlink text [body ...]
Creates a rlink with contents text that will execute continuation expression body. New in arc3.
>(defop ol req (onrlink "Click here" (prn "You clicked.")))
#<procedure: gs2434>
linkf text parms [body ...]
Creates a link with contents text to run the continuation function specified by params and body. Params should be a single argument such as (req).
>(defop lf rq (linkf "Click here" (req) (prn "Your request: " req)))
#<procedure: gs2446>
rlinkf text parms [body ...]
Creates a link with contents text to run the redirection continuation function specified by params and body. Params should be a single argument such as (req).
>(defop rlf rq (rlinkf "Click here" (req) "foo"))
#<procedure: gs2457>
arg req key
Looks up the URL query value for key in req.
>(let req (obj args '((foo 1)))
  (arg req 'foo))
1
serve [port]
Start server, by default on port 8080.
>(serve 8080)

srvlog type [args ...]
Records timestamp and args in the server log type.
>(srvlog 'blog "Stuff to log")
Error: open-output-file: cannot open output file
  path: /pa
th/to/doc/arc/logs/blog-2018-08-26
  system error: No such f
ile or directory; errno=2

defbg id sec [body ...]
Creates a background thread with the given id tha will run the body every sec seconds. Any existing thread with the same id is terminated. New in arc3.

Web server configuration variables

arcdir*
Parent directory for server data.
>arcdir*
"arc/"
logdir*
Directory to hold server logs.
>logdir*
"arc/logs/"
staticdir*
Directory to hold static files. New in arc3.
>staticdir*
"static/"
static-max-age*
Maximum age for static files; used in Cache-Control header. New in arc3.
>static-max-age*
nil
max-age*
Table of maximum age by operation; used in Cache-Control header. New in arc3.
>max-age*
#hash()
quitsrv*
Flag to shut down server if set to true.
>quitsrv*
nil
breaksrv*
Flag to enable stopping server with control-C.
>breaksrv*
nil
srv-noisy*
Flag to enable debug logging in the server.
>srv-noisy*
nil
threadlife*
Maximum time (in seconds) for a thread to handle a request before being killed.
>threadlife*
30
throttle-ips*
Table of IP addresses that should be throttled; they will be delayed by up to throttle-time* seconds.
>throttle-ips*
#hash()
throttle-time*
Maximum number of seconds to delay requests from IP addresses in throttle-ips*
>throttle-time*
Error: _throttle-time*: undefined;
 cannot reference an iden
tifier before its definition
  in module: top-level
  intern
al name: _throttle-time*

ignore-ips*
IP addresses to ignore. New in arc3.
>ignore-ips*
#hash()
spurned*
Count of ignored IP addresses by address. New in arc3.
>spurned*
#hash()
req-limit*
Maximum number of requests in req-window* seconds before triggering DoS attack detection. New in arc3.
>req-limit*
30
req-window*
Time window in seconds for triggering throttling. New in arc3.
>req-window*
10
dos-window*
Time window in seconds for triggering DoS attack detection. New in arc3.
>dos-window*
2
opcounts*
Table counting number of times each operation executed. New in arc3.
>opcounts*
#hash()

Web server internals

The key stages in the life cycle of a request are:

The URL "/deadlink" displays a "dead link" message. The URL "/" displays a welcome message. The URL "/topips" displays the user IP addresses with the heaviest server use.

The table below provides some internal implementation details. The table shows which URL is assigned to each request type, and which macro is used to implement it. Somewhat surprisingly, that the HTML functions with and without headers both use the same underlying URL and implementation; the higher-level macros simply don't provide headers if no headers are desired.
 HTMLHeaders + HTMLRedirectHeaders + RedirectAsynchronous
Path variable fnurl* fnurl* rfnurl* rfnurl2* jfnurl*
URL Path /x /x /r /y /a
Implementation macro defop-raw x defop-raw x defopr r defopr-raw y defop-raw a
The following table lists the operations and variables in srv.arc that are generally for internal use.

serve1 [port]
Start server to handle a single request.
>(serve1 8080)
 
ensure-srvdirs
Create directories for arcdir* and logdir* if necessary.
>(ensure-srvdirs)
 
handle-request s [breaksrv]
Handles a request. It accepts a connection on socket s and starts a thread running handle-request-thread. If breaksrv is true, ^C will break out of the server.
handle-request-thread i o ip
The core code to handle a request. The arguments are the input port, the output port, and the user's IP address. It reads the header, calls parseheader, srvlog, respond (get) or handle-post (post) or respond-err. Then executes harvest-fnids when done.
handle-post i o op args n cookies ip
Handles a POST action. The arguments are the input port, the output port, the path, the arguments, the content-length, the cookies, and the user's IP address. Collects POST data and calls respond.
respond str op args cooks ip
Responds to a GET or POST request. str is the output stream, rest from parseheader. If op is in redirector*, it does a redirect. If op is in srvops* otherwise, it prints header* and executes the associated function. Otherwise, a static-filetype is copied with type-header*. Otherwise, respond-err.
save-optime name elapsed
Updates optimes* with the time taken to run name.
>(save-optime "foo" 10)
(10)
static-filetype sym
Returns a filetype symbol for a static file.
>(static-filetype "foo.gif")
gif
respond-err str msg [args ...]
Generates an error response page containing msg and args.
>(respond-err (stdout) "Bad news")

parseurl s
Parses an action and URL. Returns a list of type (get or post), op (the path), and an association list of args.
>(parseurl "GET /x?foo=1&bar")
(get x (("foo" "1") ("bar" "")))
parseheader lines
Parses a request header. Returns a list of type, op, args, content-length (for post), and cookies.
>(parseheader '("POST /foo?a=b" "Cookie: bar=baz"
     "Content-Length: 42"))
(post foo (("a" "b")) 42 (("bar" "baz")))
parsecookies s
Parse a HTTP cookies header. Returns an association list of name/value pairs.
>(parsecookies "Cookie: name1=val1;name2=val2")
(("name1" "val1") ("name2" "val2"))
parseargs s
Parses args part of URL. Returns an association list of key/value pairs.
>(parseargs "x=a+b&y=%c3%e9&z")
(("x" "a b") ("y" "��") ("z" ""))
reassemble-args req
Creates the URL query string from the args in req. The arguments are not URL-encoded, so they must not contain any special characters.
>(let req (obj args '(("x" "foo") ("y" "bar")))
  (reassemble-args req))
"?x=foo&y=bar"
new-fnid
Generates a random unused fnid symbol..
>(new-fnid)
OZ3W6XFovO
fnid f
Generates a fnid with continuation function f.
>(fnid (fn (req) (prn "hi")))
h74MwrIYrF
timed-fnid lasts f
Generates a fnid that will expire after lasts seconds.
>(timed-fnid 100 (fn (req) (prn "hi")))
R6cceNyVpi
harvest-fnids [max]
If fns* has more than max fnids, purges any expired timed-fnids* and purges the oldest fnids* (10% of max are purged).
>(harvest-fnids 1000)
nil
fnid-field id
Generates a hidden field assigning id to fnid.
>(fnid-field 'abc123)
<input type=hidden name="fnid" value="abc123">
unique-id [length]
Generates a unique random alphanumeric id. The minimum length is 5, and the default length is 8. The table unique-ids* holds all generated ids to ensure uniqueness.
>(unique-id)
32o5uGIQ
memodate
Returns current date. Uses memoization for efficiency; dates are cached for 60 seconds. Only works on some platforms.
fns*
Table mapping from fnid to continuation function.
fnids*
List of fnids without an explicit lifetime.
timed-fnids*
List of entries for fnids with an explicit lifetime. Each entry is a list of fnid, time, lasts for fnids with an expiration time.
fnurl*
URL path for a fnid continuation.
>fnurl*
"/x"
rfnurl*
URL path for a fnid continuation redirect.
>rfnurl*
"/r"
rfnurl2*
URL path for a fnid continuation raw redirect.
>rfnurl2*
"/y"
jfnurl*
URL path for an asynchronous fnid continuation.
>jfnurl*
"/a"
unique-ids*
Table of ids generated by unique-id.
unknown-msg*
Message for an unknown operator.
>unknown-msg*
"Unknown."
rdheader*
Redirect header
>rdheader*
"HTTP/1.0 302 Moved"
srvops*
Holds operations defined by the defop macros. Used by respond to map the URL path to the handler.
redirector*
Table with entries for names that are 'redirectors'. These will return rdheader* and the new location.
>(keys redirector*)
(bar r baz y)
optimes*
Table of the last 1000 elapsed times (in ms) for operations.
header*
HTTP header
>header*
"HTTP/1.1 200 OK\nContent-Type: text/html; charset=utf-8\nConnection: close"
type-header*
Table of return HTML headers for static files. Renamed from srv-header* in arc3.
>(keys type-header*)
(jpg png gif text/html)
requests*
Counter of the total number of requests handled.
>requests*
0
requests/ip*
Table mapping from IP address to the total number of requests associated with that IP address.
>requests/ip*
#hash()
dead-msg*
Message when a link has an expired fnid.
>dead-msg*
"\nUnknown or expired link."
bgthreads*
Table of background threads.
>bgthreads*
#hash()
pending-threads*
List of pending background threads.
>pending-threads*
Error: _pending-threads*: undefined;
 cannot reference an id
entifier before its definition
  in module: top-level
  inte
rnal name: _pending-threads*

req-times*
Table of requests by IP addresses for DoS attack detection. New in arc3.
>req-times*
#hash()
handle-request s [life]
Implements handle-request without breaksrv. New in arc3.
log-request type op args cooks ip t0 t1
Logs a request. New in arc3.
gen-type-header ctype
Generates response header for the type. New in arc3.
>(gen-type-header "gif")
"HTTP/1.0 200 OK\nContent-Type: gif\nConnection: close"
timed-aform2 genurl lasts f [body ...]
timed-aform with ignored genurl argument.
logfile-name type
Creates logfile name for the given type. New in arc3.
>(logfile-name "gif")
"arc/logs/gif-2018-08-26"
sortable table comparison-fn
Sorts a table. Default comparison is >. New in arc3.
>(sortable (obj 'a 5 'b 2 'c 4))
(((quote a) 5) ((quote c) 4) ((quote b) 2))
new-bgthread id f sec
Creates a background thread with the given id tha will run f every sec seconds. Any existing thread with the same id is terminated. New in arc3.

Copyright 2008 Ken Shirriff.