Failure Handling

there are many failure scenarios in any distributed system big peer leans heavily on a durable transaction log for many failure scenarios, and replicated copies of data for many others safety in big peer (bad things never happen) has been discussed at length above, in ust, and transitions, and how causally consistent reads occur liveness, however, depends on every replica contributing to the ust the ust (and gc timestamp) are calculated from gossip from every node if any node is down, partitioned by the network, slow, or in some other way broken, it impacts the progress of the cluster yes, big peer can still accept writes, and serve (ever staler) reads, but the ust won't rise, transitions won't finish and gc will stall (leading to many versions on disk ) it is possible in future that we make some changes to how the ust is calculated, and use a quorum of nodes from a partition, or the single highest maximum transaction from a partition the trade off being that query routing becomes more complex, and in the event that the node that set the ust high then becomes unavailable something has to give in terms of consistency these trade offs are mutable, we can re visit them, we favoured safety in the current iteration of the design if some nodes are keeping the ust down, and slowing or halting progress, the bad node(s) can be removed bad nodes in this first iteration of big peer we have the blunt, expedient tool available to us of killing and removing a node that is bad the process is simple update the current config to remove the offending node(s) and signal the remaining servers they will immediately no longer route to that node, use that node in their calculations, or listen to gossip from that node this is fine as long as at least one node in each partition of the map is left standing as soon as the offending nodes are removed the data is under replicated at once add replacement nodes by performing a transition as above for example, imagine p1r2 has become unresponsive remove it from the current config, create a next config with a new server to take the place of p1r2 , store the configs in the strongly consistent metadata store, and signal the nodes the new node will begin to consume transactions and backfill, and the ust will rise etc missing and lost data as described in the first backfill section, it is possible, with a long network incident and a short log retention policy, that some transactions are missed if all the replicas for a partition miss some intersecting subset of transactions, that data has been missed, and it is lost this should never happen if it does, we don't want to throw away the big peer cluster, and all the good data progress must still be made in this case each replica of the partition understands from the intervalmaps that some transaction t has been missed after doing a strongly consistent read of the metadata store, to check that no server in the next config exists that may have the data, the replicas agree unilaterally to pretend that really they did store this data, and they splice it into their intervalmaps the ust rises, and progress is made it is essential to understand this is a disaster scenario, and not business as usual, but disasters happen, and they should be planned for we do everything we can to never lose data, including a replicated durable transaction log with a long retention policy document cache in order to not be required to query big peer for all data requested by small peers, all the time, the subscription server maintains a sophisticated, causally consistent, in memory cache of a data the data it chooses to cache is based on the subscriptions of the small peers connected to it by routing devices to a subscription server by appid, we increase the likelihood that the devices have an overlapping set of subscriptions and share common data in the cache the document cache takes data from the mesh clients and puts it on the log as transactions it also consumes from the log, so that it can keep the data in the cache up to date, without querying big peer any documents that it observes on the log, that are potentially of interest to the cache, must first be "upqueried" from big peer to populate the cache as a cache becomes populated upqueries decrease in size and number as clients disconnect, if any data is no longer required in the cache, it is eventually garbage collected away