Sarracenia Documentation

Documentation comes in two varieties: Guides and Manual Pages. The Guides provide the overall idea of how to use the tools, with description of several components, use cases, and some sample configurations and runs. The guides provide "Getting started" type information, describing the purpose, in general, of the various components needed for various jobs.

The Manual Pages, on the other hand, are reference material. For each component or format, the manual page provides a complete reference that shows all the options that exist for that component, and also provides examples of usage. To learn the application, it is best to start with Guides. The Guides are also best read in order, as knowledge of each guide assumes one is familiar with the previous ones.

Once an initial configuration is established and the overall picture is understood, then the reference materials provides quick answers to specific questions.

Guides

User-Facing Components

Administrative Components

Formats/Protocols

  • sr_post(7) - the format of postings. Posted by watch and post, consumed by subscribe.
  • sr_report(7) - the format of report messages. Sent by consumers, for sources to measure reach.
  • sr_config(7) - reference for options used by many components.

Glossary

Sarracenia documentation uses a number of words in a particular way. This glossary should make it easier to understand the rest of the documentation.

Source

Someone who wants to ship data to someone else. They do that by advertise a trees of files that are copied from the starting point to one or more pumps in the network. The advertisement sources produce tell others exactly where and how to download the files, and Sources have to say where they want the data to go to.

Sources use programs like sr_post.1, sr_watch.1, and sr_poll(1) to create their advertisements.

Subscribers

are those who examine advertisements about files that are available, and download the files they are interested in.

Subscribers use sr_subscribe(1)

Post, Notice, Notification, Advertisement, Announcement
These are AMQP messages build by sr_post, sr_poll, or sr_watch to let users know that a particular file is ready. The format of these AMQP messages is described by the sr_post(7) manual page. All of these words are used interchangeably. Advertisements at each step preserve the original source of the posting, so that report messages can be routed back to the source.
Report messages
These are AMQP messages (in sr_report(7) format) built by consumers of messages, to indicate what a given pump or subscriber decided to do with a message. They conceptually flow in the opposite direction of notifications in a network, to get back to the source.
Pump or broker
A pump is a host running Sarracenia, a rabbitmq AMQP server (called a broker in AMQP parlance) The pump has administrative users and manage the AMQP broker as a dedicated resource. Some sort of transport engine, like an apache server, or an openssh server, is used to support file transfers. SFTP, and HTTP/HTTPS are the protocols which are fully supported by sarracenia. Pumps copy files from somewhere, and write them locally. They then re-advertised the local copy to it's neighbouring pumps, and end user subscribers, they can obtain the data from this pump.
Dataless Pumps
There are some pumps that have no transport engine, they just mediate transfers for other servers, by making messages available to clients and servers in their network area.
Dataless Transfers
Sometimes transfers through pumps are done without using local space on the pump.
Pumping Network
A number of interconnects servers running the sarracenia stack. Each stack determines how it routes stuff to the next hop, so the entire size or extent of the network may not be known to those who put data into it.
Network Maps
Each pump should provide a network map to advise users of the known destination that they should advertise to send to.