Libwww is a highly modular, general-purpose client side Web API written in C
for Unix and Windows (Win32) (and I know
that it runs on Macs too). It's well suited for both small and large
applications, like browser/editors, robots, batch tools, etc. Pluggable
modules provided with libwww include complete HTTP/1.1 (with caching,
pipelining, PUT, POST, Digest Authentication, deflate, etc), MySQL logging,
FTP, HTML/4, XML (expat), RDF (SiRPAC), and much more. The purpose of libwww
is to serve as a testbed for protocol experiments.