Sync and Replication Concepts

the realtime end user transactional data generated by your app can be disseminated across devices through three different methods centralized — a small peer device establishes a direct internet enabled connection to the big peer cloud deployment decentralized — a small peer device establishes a mesh network connection with nearby small peer devices using any and all communication transport types available to them by default hybrid — if one or multiple small peer devices connected in the mesh network gain access to the internet, not only do those devices with internet access upload their local database to the big peer cloud, but each and every nearby offline small peer device as well for instructions on enabling and disabling transport types, see docid\ q3jydbwoo rksjdqkg0 comparing sync and replication data sync involves bidirectional data transmission with the big peer cloud deployment, where the big peer both sends and receives updates from connected peers as opposed to data replication , in which data is partitioned into smaller, more manageable pieces, that are then unidirectionally mirrored across the mesh network syncing large documents can significantly impact network performance caution is advised when handling large binary data, such as a high resolution image or video exceeding 50 megapixels; a deeply embedded document; or a very large document instead of storing files exceeding 250kb directly within a document object, carefully consider using attachments for more information, see docid\ rb3wd2jhenojpw8xuirfy the following graphic illustrates the bidirectional sync process and the unidirectional data egress replication process syncing with big peer the big peer supports both bidirectional sync and unidirectional replication before you can establish a peer to peer mesh network and exchange data directly with nearby peers, you must connect to the big peer at least once to set up your communication infrastructure when you connect to the big peer for the first time, you obtain your access credentials, and ensure you have the latest data and are up to date with the rest of the peers for more information, see docid\ hjfoifaqghorv3pcloln8 any small peer that has access to the internet automatically establishes a connection with the big peer to sync data across the big peer and, by way of multihop replication, all connected small peers for more information, see /#multihop replication , as follows sync groups to enhance performance by default, devices with the same app id automatically form a mesh network to connect with one another however, you can streamline replication processes, minimize unnecessary data transfer, and optimize resource usage by configuring distinct groups for instructions on setting up sync groups, see transport configurations > docid\ q3jydbwoo rksjdqkg0 example scenario for example, the following graphic illustrates two distinct restaurants identified by their location ids 7890 and 1234 when a peer establishes its identity with the big peer, it includes its location id as part of the authentication callback now, the two groups maintain isolated mesh networks replicating across peers the multiple replicas that result from a single partition are spread across virtual and relevant physical server nodes; that is, the big peer cloud deployment and local ditto stores of subscribing small peers by default, ditto does not automatically replicate data to peers instead, small peers use subscriptions and live queries to precisely indicate the data they're interested in once that data changes, only the delta matching their live query replicates to their local ditto store in simpler terms, replication involves a subscribing small peer selectively "pulling" data from peers, rather than remote small peers automatically "pushing" data to them as illustrated in the following graphic, if a small peer is not subscribed to any queries, no documents replicate to them greedy and selfish replication the big peer and small peer approach data sync in different ways the big peer uses a greedy sync model — automatically subscribing to everything and, when possible, replicating complete document objects across peers small peers use a selfish sync model — replicating only the data that they explicitly query for and that has undergone changes to their local ditto store multihop replication if at any time a small peer connected in the mesh gains access to the internet, all mesh generated data gets automatically exchanged with the big peer by way of multihop replication multihop replication is the process of passing data from one connected small peer to another connected small peer by way of intermediate “hops” along a given path for multihop replication to occur, all peers within the chain must observe the same data, as demonstrated in the following graphic the big peer and small peer approach data sync in different ways the big peer uses a greedy sync model — automatically subscribing to everything and, when possible, replicating complete document objects across peers small peers use a selfish sync model — replicating only the data that they explicitly query for and that has undergone changes to their local ditto store benefits with ditto's multihop replication technology, you achieve three significant advantages guarantee a consistent offline first sync of critical data, ensuring data remains consistent event when connectivity is intermittent enhance the overall responsiveness of your app, resulting in faster interactions reduce costs by optimizing data operations since, when feasible, you can read and write data to a nearby peer instead of traversing a very fragile, low‑bandwidth data source multihopping process the multihop replication process consists of the following first, a peer discovers which nearby peers are reachable the ditto router continuously responds to changes in the local mesh, such as peer disconnects and transport type limitations when a peer establishes a mesh network connection with another peer it first establishes encrypted internal tls tunnels, or ditto links , that nearby peers use for transmission ditto links for advanced security similar in function to a virtual private network (vpn), ditto links are virtual private connections that utilize the noise protocol to establish secure and protected connections the noise protocol is a cryptographic framework for advanced security protections for more information, see the official documentation at http //noiseprotocol org/ concurrency conflicts a challenge arises in offline scenarios when two or more peers make edits independently and the data values stored by each peer diverge over time referred to as conflict resolution , ditto's process of addressing concurrency conflicts involves a combination of a /#metadata database and /#guiding principles metadata database each peer individually maintains its own metadata database the metadata database serves as a storage repository of information essential in resolving conflicts that arise during the process of merging concurrent changes within this metadata database is the version vector the version vector manages the following essential details for a particular peer the current state of replication a record of the data previously sent and received the sequence of updates the timestamp of last write operation guiding principles ditto adheres to the following strategies to ensure that all peers ultimately reach the same value for a particular data item deterministic — as part of the strategy of eventual consistency, regardless of the order in which updates from different peers are received and merged, all peers ultimately converge to the same single value predictable and meaningful — instead of arbitrarily resolving conflicting registers to a predefined value, the resulting merge accurately represents the original input and some rational interpretation of that input the following scenario provides a walkthrough of the mechanics of version vectors, their role in determining merging behavior, and how different peers contribute to data replication eventual consistency example scenario the hlc uses the uint128 data type to represent the site id and 64bit timestamp in ditto; however, for simplicity, the following scenario uses basic string and number types instead local peer a document 123abc links to a version vector that indicates its locally stored document is currently at vector version 5 the most recent incoming remote peer b changes were incorporated and merged at version 1 the remote peer c changes were incorporated and merged at vector version 4 documentid "123abc" version vector { "a" 5, "b" 1, "c" 4 } local peer a receives document changes from remote peer b the incoming document's version vector indicates the remote peer b incoming version vector is value 4 the most recent incoming remote peer b changes were of version 1 ; a value less than the incoming document version vector value of 4 since 4 is greater than 1 , the local peer a determines that the changes are new and should be incorporated and merged in order to remain consistent documentid "123abc" version vector { "a" 5, "b" 1, //merge in {"b" 4} because 4 > 1 "c" 4 } hybrid logical clock each document in each peer contains a hidden metadata map of a site id and a hlc the hlc stands for a hybrid logical clock this hlc is used to determine whether a change has "happened before" it might be tempting to use physical clocks to resolve conflicts when attempting to merge concurrent data changes however, it's essential to know that even quartz crystal based physical clocks can skew forwards or backward in time almost every device regularly attempts to synchronize with an ntp synchronized clock server but even then, the round trip time from the request to the server's response adds additional variability in addition, there are limitations to nature and physics that will never allow two measurements of physical time to align precisely thus, these conditions led us to determine that physical clocks were not reliable in a distributed mesh network although we decided that we could not build a system that resolved conflicts based purely on physical time, we needed to preserve the notion of physical time as not to confuse users of collaborative applications however, each peer still needs a deterministic way to resolve conflicts in other words, each peer when sharing crdt deltas needs to always resolve conflicts exactly the same way this requirement still needs logical ordering this requirement led us to implement the version vector with a hybrid logical clock (often referred to as hlc) in ditto's distributed system, the hlc is a 64 bit timestamp comprised of 48 bits of a physical timestamp and 16 bits of a monotonically increasing logical clock clock accuracy ditto requires that all devices participating in the mesh have the time and date configured accurately it is recommended to synchronize the clock automatically with an internet time server this feature is built into most platforms including ios and android this will be sufficiently accurate for ditto’s needs if a device has a significantly incorrect clock, outcomes may include failing to connect to a peer because it thinks their certificate isn’t valid yet or has already expired unintuitive conflict resolution in certain situations when concurrently editing documents version vectors each peer has its own version of the documents that multiple peers can work on concurrently they keep track of their respective document changes in a data structure known as a version vector the version vectors that each document links to contain a snapshot of essential metadata, such as the given document's current state, as well as a record of the data previously sent and received incrementing version vectors when a peer directly updates its own locally stored document, the document version vector increments by a value of one determining newness local peers use document version vectors to accurately assess local and observed edits and ensure data consistency before a peer incorporates and merges incoming changes into its document state, the peer compares its existing version vector value with the receiving version vector value to determine whether the incoming changes are new (or old) relative to its current document state if the incoming version vector is a value greater than the existing version vector value, the incoming changes are new and must be incorporated and merged in order for its document state to remain consistent managing time based operations the version vector pinpoints the chronological order of events and determines the exact timing of the most recent write operation for a specific peer utilizing a combination of temporal timestamp and the hybrid logical clock ( hlc ) the temporal timestamp is a component associated with an individual peer that tracks the chronological order of its local events the hlc is a higher level component that combines both physical time and logical clocks to create a unified timestamp that tracks the chronological order of events for the entire distributed system