Data-Handling Essentials

this article provides a complete overview of the various data types you can use with your queries to retrieve, modify, and sync data in ditto in addition, you'll find an introduction to related concepts like conflict resolution strategies introduction in ditto, there's a clear distinction between traditional create , read , update , and delete (crud) database operations and data sync crud for crud operations, you interact with data stored locally in the ditto store by invoking a single operation on the store namespace once executed, ditto returns a response object encapsulating the results the following table provides a high level overview of the different ways you can perform crud in ditto operation description create using the insert statement, either insert a new document or update an existing document for a given document id read using the select statement , retrieve documents based on the specific criteria you pass as parameters in your function update using the update method, write changes to ditto delete using the evict method, remove local data from your store in addition, using a soft delete pattern , indicate data as deleted without physically removing it from ditto sync for syncing, you set up remote listeners, or sync subscriptions , to monitor changes to the data you're interested in when data changes, ditto automatically syncs the delta changes to your local ditto store observe in addition to setting up sync subscriptions to monitor mesh wide data changes, you can establish local listeners known as store observers to monitor changes in the ditto store operating locally a store observer is a dql query that runs continuously and, once ditto detects relevant changes, asynchronously triggers the callback function you defined when you set up your store observer for instance, when the end user updates their profile, display the profile changes to the end user in realtime documents and collections ditto stores data in structured json like document objects each document consists of human‑readable fields that identify and represent the information the document stores the following snippet provides an example of a basic json encoded document object ditto document { " id" "abc123", "make" "hyundai", "year" 2018, "color" "black" } a single document consists of one or more fields that self‑describe the data it encodes; each field is associated with a value item description 1 the name identifying the data 2 the value that holds the actual data to store a grouping of related documents is referred to as a collection think of a collection as a table in a relational database table — but with greater flexibility in terms of the data it can hold — and the documents in a collection like table rows in the following structure, the location field property is logically grouped with details about the car, such as its make , year , and color it contains nested fields also referred to as subfields , representing both the coordinates and the address where the car is located dedicated query language ditto query language (dql) is the platform's dedicated query language for defining criteria for local database operations and data sync with remote peers for instance, you use dql to perform various filter operations like traditional create , read , update , and delete (crud) and filter data for sync subscriptions dql is a familiar sql‑like syntax designed specifically for ditto's edge sync features that enable the platform's offline first capabilities dql offers features like reusable statements clear syntax advanced querying capabilities query statements overview the following table provides an overview of the types of statements you'll write in dql and enclose within your api method operations to specify data selection criteria ditto store execute — interact with data stored locally within the small peer device running your app (see docid\ fny xpterhpnussplv32w ) ditto store registerobserver — set up store listeners to monitor changes to data in the local ditto store based on specified query criteria (see docid\ fny xpterhpnussplv32w ) ditto sync registersubscription — set up subscriptions for syncing data from other peers based on their associated queries (see docid\ fny xpterhpnussplv32w ) ditto presence — enable end user functionality like mesh network monitoring, management, and transport optimization (see /#presence operations ) data types and scalar types when writing queries with dql, data formatting is categorized into two buckets scalar type a scalar data type can be a string , boolean , array , and any other basic primitive type in addition, a scalar type can be a json blob object capable of nesting multiple key value pairs functioning as a single unit data type an advanced type that guarantees conflict free resolution at merge and includes register , map , and attachment as illustrated on the right, each data type includes two components t he value to be stored — encoded using scalar types like string , boolean , and so on the field‑specific metadata that defines the enforced merge strategy in conflict resolution the default data type is register ; you'll use other data types in specific scenarios where appropriate specifying data types to use a dql type other than a register — the default data type in ditto — you must explicitly specify the type in your query; otherwise, ditto defaults to the register type as follows in dql syntax, enclose string literals within single quotes ( '' ); f or example 'blue' dql select from collection cars where features trim = 'standard' here is an example illustrating the same select statement query explicitly expressed as a map structure it specifies retrieval of the map structure storing the "features" collection with a "trim" field property set to a value of "standard" dql select from collection cars (features map) where features trim = 'standard' conflict resolution as the foundation of how ditto exposes and models data, these data types leverage conflict free replicated data type (crdt) technology to ensure that no data inconsistencies occur as a result of concurrent modifications ; that is, simultaneous edits made to the same data types in multiples local ditto stores all data types — register , map , and attachment — adhere to the causal consistency model when resolving concurrency conflicts the causal consistency model is a guarantee that if there is an operation that must happen before another operation — for example, events a and b, where b is a result of a — all peers agree upon and observe the same sequential order of these operations; as in, a always executes before b merge strategies in ditto's implementation, conflicts are automatically resolved, merged, and synced across peers without the need for coordination or validation from a centralized authority within this consistency model, there are two principles for guiding conflict resolution at merge last write wins merge strategy add wins merge strategy the following table provides a quick overview of the data types you can use to write queries, along with their merge strategies, a brief description, and a common usage scenario type merge strategy description use case register "last write wins" stores a single value and allows for concurrent updates store json‑compatible scalar subtypes, including a nested blob representing two or more fields as a single object map "add wins" contains a nested object consisting of any ditto type register , map , and attachment enable field level concurrent hierarchy within a ditto document attachment "last write wins" stores the token you use to retrieve the attachment reduce small peer resource usage by storing data that can be retrieved lazily ; as in, you fetch the data only when needed representing complex datasets when you want to embed a hierarchical structure to represent complex parent‑children data structures within a document in dql, you have the option to nest either of the following json blob functioning as a register map the decision between the two depends on your specific use case for instance, t o represent embedded values with dependencies, such as a gps coordinate along with its corresponding address, structure your data in a json object, as follows "location" { "coordinates" \[ 122 0308, 37 3318], "address" "123 main st, san francisco, ca 94105" } this is because, unlike a map , a json object functions as a single unit managing both the coordinate and address as a cohesive unit ensures that any changes made to one automatically update the other similar to the json object, t he array type in ditto acts as a register and therefore encapsulating values function as a single unit representing coordinates as an array , as demonstrated in the previous snippet, is the smart choice since its latitude and longitude values function as a unified entity, always changing simultaneously to represent embedded values with no dependencies; that is, you want the flexibility to update each key value pair independently, and structure your data as separate register fields in a map ditto document "features" { "trim" "standard" "speakers" 5 } to represent highly complex data structures in which you need to establish additional hierarchies, embed a map within another map , as follows syncing large documents can significantly impact network performance the decision to use deeply embedded maps in a single document or opt for a flat model depends on requirements, relationships between data, and tolerance for certain tradeoffs ditto document "owner" { "name" "john larson", "information" { "address" "2147 chestnut st, oakland, ca 94607", "phone" "555 555 5555", "updated" "june 2, 2023" } } distinctions by type the following graphic and corresponding table aim to demonstrate the distinct capabilities and versatility of each dql type the register data type functions as the default in ditto for scalar encoded values so any scalar encoded values, including embedded json objects and arrays , are assumed to be type register item type description 1 register a value can be any json encoded primitive type boolean , numeric , binary , string , array , and null 2 register a hierarchical data structure of multiple json encoded fields nested within a larger json object and serves as a single value 3 map a hierarchical data structure of two or more key value pairs encoded using any data type — register , attachment , or map 4 response object the response object returned after creating a new attachment you use the attachment response object within your app's code to retrieve and display the file to the end user, as appropriate, and to update or delete the file 5 attachment token the pointer that ditto uses to reference the large file's storage location when fetching you can use an attachment for any file type, including binary data of 50 megapixels or more, such as an mp4 file, or a large document object featuring complex hierarchical structures map data type conflicts an issue unique to map data types is the possibility for two offline peers to create a new document, in which one peer represents the field as an object ( map ), while the other peer represents the field as an array divergent types preventing merge the following snippets illustrate a scenario of a type level conflict unique to map types peer a creates the following new document json { "name" "bob jones", "address" { "street" "long road", "house number" 10298, "zip" "90210" } } while at the same time peer b creates the following new document json { "name" "bob jones", "address" \[ 10298, "long road", "90210" ] } managing conflicts update history because peer a and peer b use divergent data structures, combining an array with an object ( map ) is impossible rather than adhering to the default "last updated type" win principle, which can trigger a ping pong alteration of types between connected peers, as demonstrated in the following snippet, retain both values for the address field property by creating a data structure that accommodates both the object ( map ) and the array a ping pong alteration of types occurs when distinct peers repeatedly modify the data type of a specific field in response to each other's updates, leading to a forever loop of back‑and‑forth behavior { "name" "bob jones", "address" { "objectversion" { "street" "long road", "house number" 10298, "zip" "90210" }, "arrayversion" \[ 10298, "long road", "90210" ] } } store operations there are two ways of interacting with the ditto store operating locally on end user devices perform a one time execution of a create , read , update , delete (crud) operation against the store namespace asynchronously monitor local changes by setting up observers against the store namespace ditto store execute interact with data stored locally within the small peer device running your app on the right is a graphic illustrating crud operations on a small peer end ‑ user device with these one time operations, you perform a single action, such as modifying or retrieving data, on the ditto store at a given time once invoked, you must wait for the operation to complete before continuing to the next action in your code for example, the following single execution query, once called, uses the execute api method and a local select query to search within the local small peer store for data in a dataset named cars let result = await ditto store execute(query "select from cars")val result = ditto store execute("select from cars")const result = await ditto store execute("select from cars");dittoqueryresult result = (dittoqueryresult) ditto store execute( "select from cars", new continuation<>() { @nonnull @override public coroutinecontext getcontext() { return emptycoroutinecontext instance; } @override public void resumewith(@nonnull object o) { if (o instanceof result failure) { // handle failure } } } );var result = await ditto store executeasync("select from cars");auto result = ditto get store() execute("select from cars") get();let result = ditto store() execute("select from cars", none); creating to create a new document in your local ditto store, call insert following is an example of how to perform an insert operation using ditto's sdk await ditto store execute( query "insert into cars documents (\ newcar)", arguments \["newcar" \["color" "blue"]]);ditto store execute( "insert into cars documents (\ newcar)", mapof("newcar" to mapof("color" to "blue")))await ditto store execute( "insert into cars documents (\ newcar)", { newcar { color "blue" } });dittoqueryresult result = (dittoqueryresult) ditto store execute( "insert into cars documents (\ newcar)", collections singletonmap("newcar", collections singletonmap("color", "blue")), );var args = new dictionary\<string, object>(); args add("newcar", new { color = "blue" }); await ditto store executeasync( "insert into cars documents (\ newcar)", args);std map\<std string, std map\<std string, std string>> args; args\["newcar"] = {{"color", "blue"}}; auto result = ditto get store() execute( "insert into cars documents (\ newcar)", args) get();struct args { newcar car, } struct car { color string } // let args = args { newcar car { color "blue" to string() }, }; ditto store() execute( "insert into cars documents (\ newcar)", args); dart (beta) reading s earch for documents within your local ditto store by calling the execute api method on a select query as follows let result = await ditto store execute(query "select from cars")val result = ditto store execute("select from cars")const result = await ditto store execute("select from cars");dittoqueryresult result = (dittoqueryresult) ditto store execute( "select from cars", new continuation<>() { @nonnull @override public coroutinecontext getcontext() { return emptycoroutinecontext instance; } @override public void resumewith(@nonnull object o) { if (o instanceof result failure) { // handle failure } } } );var result = await ditto store executeasync("select from cars");auto result = ditto get store() execute("select from cars") get();let result = ditto store() execute("select from cars", none); dart (beta) establish a listener, known as a store observer , to watch local changes so you can respond immediately; for example, to observe updates to end user profiles here is how you set up a store observer in ditto let observer = ditto store registerobserver( query "select from cars"){ result in / handle change / };val observer = ditto store registerobserver("select from cars") { result > / handle change / };const changehandler = (result) => { // handle change } const observer = ditto store registerobserver( "select from cars", changehandler);dittostoreobserver observer = ditto store registerobserver( "select from cars", result > { // handle change } );// without arguments var result = await ditto store registerobserver( "select from cars", (result) => { // handle change }); // with arguments var result = ditto store registerobserver( "select from cars", (result) => { // handle change }); auto observer = ditto get store() register observer( "select from cars", \[&]\(queryresult result) { / handle change / });let observer = ditto store() register observer( "select from cars", none, move |result queryresult| { // handle change }) dart (beta) updating modify fields within one or more documents in your local ditto store for example, executing an update operation within the cars collection, changing the color to 'blue' and the mileage to 3001 in documents where the id field is '123' , as follows try await ditto store execute(""" update cars set color = 'blue' where id = '123' """);ditto store execute(""" update cars set color = 'blue' where id = '123' """)await ditto store execute(` update cars set color = 'blue' where id = '123'`)dittoqueryresult result = (dittoqueryresult) ditto store execute( "update cars set color = 'blue' where id = '123'", new continuation<>() { @nonnull @override public coroutinecontext getcontext() { return emptycoroutinecontext instance; } @override public void resumewith(@nonnull object o) { if (o instanceof result failure) { // handle failure } } } );await ditto store executeasync( "update cars set color = 'blue' where id = '123'");ditto get store() execute( "update cars set color = 'blue' where id = '123'") get();ditto store() execute( "update cars set color = 'blue' where id = '123'", none); dart (beta) deleting call the evict method to clear one or more documents from the local ditto store once invoked, the documents are no longer accessible by local queries, h o wever, they remain accessible from other peers connected in the mesh the following snippet shows how to write a basic evict operation to purge the document with an id field of '123' from the local ditto store try await ditto store execute(""" update cars set color = 'blue' where id = '123' """);ditto store execute(""" update cars set color = 'blue' where id = '123' """)await ditto store execute(` update cars set color = 'blue' where id = '123'`)dittoqueryresult result = (dittoqueryresult) ditto store execute( "update cars set color = 'blue' where id = '123'", new continuation<>() { @nonnull @override public coroutinecontext getcontext() { return emptycoroutinecontext instance; } @override public void resumewith(@nonnull object o) { if (o instanceof result failure) { // handle failure } } } );await ditto store executeasync( "update cars set color = 'blue' where id = '123'");ditto get store() execute( "update cars set color = 'blue' where id = '123'") get();ditto store() execute( "update cars set color = 'blue' where id = '123'", none); dart (beta) ditto store registerobserver set up store observers to monitor changes to data in the local ditto store based on specified query criteria a store observer is a continuous dql query — when a change to the store impacts the query results, it automatically triggers a callback that you can use to perform some action in your app store observers are useful when you want to monitor changes from your local ditto store and react to them immediately for instance, when observing end user profile updates, you can asynchronously display their changes to them in realtime sync operations establish sync subscriptions to monitor changes to the data you're interested in the queries defining your da ta criteria run on all remote stores subscribed to the same sync subscription when you subscribe to a collection, the subscription query is distributed to all relevant stores in the network each store executes the query locally and sends updates to the subscriber whenever changes match the subscription criteria ditto sync registersubscription following is a graphic illustrating how to set up a sync subscription to monitor changes to the data you're interested in first, y ou initiate a subscription request from your local ditto store to the remote peer then, when the data you're subscribed to changes on the remote peer, such as updates, deletions, or insertions, the remote peer automatically syncs the delta changes to your local ditto store over the mesh combining store observers with your sync subscriptions ensures that you receive updates from remote stores, enabling you to quickly respond in your app presence operations as ditto establishes connections and forms a mesh network using all available network transports on a device, each peer discovered in the mesh creates a presence graph through advertising and forming connections the presence graph is a data structure representing the current state of the mesh from a specific peer's point of view with ditto's presence api, you can implement the following end user functionality intercept and decide to accept or reject incoming connection requests retrieve and view the presence graph providing near realtime status of connected peers and network resources retrieve and view your local device's peer key or the peer key identifying a specific remote peer device input and view personal information about yourself, such as name and role view the personal information set by remote peers for more information, see docid\ vhjyukvy6 xgltdjbrns7 , docid\ ze725plpdadl4zthnx4ct , and docid\ mshpp 4athaemelokloyn patterns this topic provides an overview of lazy load retrieval, memory management, ping pong effect, and utilizing a map to sync concurrent changes understanding these patterns will help you make informed design decisions in your app lazy load retrieval to improve performance, instead of storing a file that encodes large amounts of binary data within a document, consider storing a reference to it in a separate, explicitly fetched object (token) known as an attachment with the attachment data type, you can implement lazy loading lazy loading is when you delay retrieval until necessary rather than aggressively fetching the data in anticipation of hypothetical future use this "on demand" retrieval pattern enhances performance by optimizing resource usage for a realworld usage scenario, see either the demo chat app for ios or android in the getditto > https //github com/getditto/demoapp chat/tree/main github repository for instance, in the https //github com/getditto/demoapp chat/tree/main/ios , you can see a savvy implementation of attachment with a full resolution avatar image from a collection named user memory management data storage management is essential for preventing unnecessary resource usage, which affects sync performance, battery life, and overall end user experience eviction is important for use cases like cabin crew apps where data from the last flight is not needed on the next flight ping pong effect the best approach to handle conflicts that result from two peers making concurrent offline edits and then later rejoining online depends on your specific requirements and use case following is an overview of best approaches for handling concurrency conflicts resolving concurrency conflicts — if you want to give priority to the "latest" change, use a register ditto's register type use last write wins semantics so the value written last always becomes the current value auditing concurrency conflicts — if you want to keep track of the changes made by different peers over time, use the map type to model your list of operations each write operation is independently tracked as a field value pair, with the field representing the unique identifier and the value storing only the specific changes made by a given peer prompting end users to choose — if you want your end users to resolve concurrency conflicts instead of ditto, use the map type inside of your document and prompt end users to select the value to replicate using a map for concurrent updates imagine a scenario in which two ditto stores, peer a and peer b, have the following document json { " id" "abc123", "color" "red", "make" "toyota", "mileage" 160000, "inspections" "\<very large map>" } peer a calls the upsert method to change the field value color\ red to color\ blue val initialdocument = mapof( " id" to "abc123", "color" to "blue" )upsert({ id "abc123", color "blue" }) a t the same time, peer b calls the update method to change the value of the mileage field findbyid("abc123") update(doc => { doc mileage incrememt(200) }) ditto store collection("cars") update(initialdocument)findbyid("abc123") update(doc => { doc mileage increment(200) }) when the changes replicate across the distributed peers, both changes merge resulting in both peer a and peer b ditto stores having a mileage increment of 200 and the color change to blue json { " id" "abc123", "color" "blue", "make" "toyota", "mileage" 160200, "smogreports" "\<very long json blob>" } antipatterns following is a table summarizing the risk associated with improperly implemented patterns pattern risks lazy load retrieval increased latency if resources are not efficiently preloaded potential resource contention from simultaneous requests memory management memory leaks that lead to increased memory consumption over time ping pong effect excessive network traffic or processing overhead reduced system responsiveness if components engage in a cycle of redundant operations using a map for sync race conditions causing data inconsistency