Schema Versioning

in a distributed environment, data structures, schemas, and documents organically undergo changes over time, which can lead to data inconsistencies in order to maintain data consistency and integrity throughout these changes, ditto offers built in schema versioning patterns and tools that you can incorporate into your workflow to make changes to data structures more effectively, adapt to new requirements, and ensure reliable and consistent data replication across distributed peers this topic provides strategies you can implement in your app to handle changes in schema and versioning /#creating schema versioning patterns /#forward compatibility /#backward compatibility /#supporting the latest version timing when to version your documents changing your schema is inevitable to ensure reliability over time, create your own schema versioning pattern for each ditto document creating schema versioning patterns a schema versioning pattern is a systematic approach to managing and handling changes to your schema over time by providing mechanisms for tracking and controlling your data structures as they inevitably evolve to ensure data consistency and reliability over time, create your own schema versioning pattern for each ditto document following is a list of various practices that you can apply in your schema versioning pattern practice description standard naming conventions establish consistent naming rules for elements in your schema field modifications handle changes to existing fields effectively field types manage the types of fields within your schema validation and transformation ensure data validation and transformation procedures are in place upgrade notifications implement a system to notify users about schema upgrades same version compatibility ditto's replication protocol is designed to be backward compatible backward compatibility means that eventually you will have the "couch device problem" (i e , a device that fell behind a couch) in other words, a device in your mesh may be offline for a significant amount of time before connecting back with other devices if the shape of your documents is significantly different on that device, there could be documents that do not conform with your new application code synchronizing with this "couch device" could cause other devices to crash unexpectedly in production if precautions aren't taken in your application schema some applications do not need backward or forward compatibility, which can simplify their business logic significantly if that sounds like your application, we recommend that you use a pattern where you change the name of the collection for each schema version of your application this enforces further that field types never change for example, you can use mycollection v{number} as a convention to specify the collection schema version your app will be listening to when a schema change is necessary, bump the number collections are very cheap to create in ditto, so this will scale even for applications that run for many years you could also only synchronize documents that come from schema versions that are the same as your current schema version dql select from your collection where schema version = 1 forward compatibility in a typical centralized database like postgresql, developers often focus on backward‑compatibility, where newer versions of the application can open old documents in a distributed system, you do not have central control of all modifications to data in an offline peer to peer mesh, it is difficult and sometimes impossible to control all versions of your application that are active in production environments because of these constraints, you need to not only think of backward‑compatibility, but also forward‑compatibility an application is forward compatible when existing code is able to read new data we can see https //developer mozilla org/en us/docs/web/guide/writing forward compatible websites to achieve forward‑compatibility of your database, you should never change the type of an existing field in other words, developers should only ever add new fields, and never remove or modify old fields you can ensure this by creating a controller that encapsulates ditto and is used across your application(s) to validate the field values and their associated types before upserting those values into the database backward compatibility older data could be very important, or it could not be it's your choice to decide what to do with these old documents you could accept (as is), reject (ignore), or migrate them to the new schema for example, here's a breaking version change where we add a new field and change the type of an old field app version 2 pseudocode private struct v1car { let id string let make string let model string let year string var version int init(doc dittodocument) { self id = doc\[" id"] stringvalue self make = doc\["make"] stringvalue self model = doc\["model"] stringvalue self year = doc\["year"] stringvalue self version = doc\["version"] intvalue } } private struct v2car { let id string let make string let model string let year int let hometown string var version int init( id string, make string, model string, year int, hometown string, version int) { self id = id self make = make self model = model self year = year self hometown = hometown self version = version } init(doc dittodocument) { self id = doc\[" id"] stringvalue self make = doc\["make"] stringvalue self model = doc\["model"] stringvalue self year = doc\["year"] intvalue self hometown = doc\["hometown"] stringvalue self version = doc\["version"] intvalue } } func decode(car dittodocument) > v2car { switch(car version) { case 1 { let oldcar = v1car(car) let migratedcar = v2car( id oldcar id, make oldcar make, model oldcar model, year int(oldcar year), hometown "n/a", version 2) return migratedcar } case 2 { return v2car(car) } } } app version 1 you also may want to ignore documents that come from incompatible applications pseudocode private struct v1car { let id string let make string let model string let year string var version int init(doc dittodocument) { self id = doc\[" id"] stringvalue self make = doc\["make"] stringvalue self model = doc\["model"] stringvalue self year = doc\["year"] stringvalue self version = doc\["version"] intvalue } } func decode(car dittodocument) > v1car { switch(car version) { case 1 let oldcar = v1car(car) return oldcar default // create default car item, or ignore document altogether return } } supporting the latest version https //docs ditto live/common/concepts/schemas#supporting the latest version when a new application version is detected, you can stop synchronizing you can detect that a new application version is available by querying for a schemaversion that is greater than the current version if a new version is detected, stop sync and tell the user they need to upgrade their app to the latest version const schemachangehandler = () => { // notify user to update to latest application version ditto stopsync() } ditto store registerobserver(` select from cars where schemaversion > \ currentversion`, { currentversion 1 }, schemachangehandler); this is a common pattern that many applications use for example, apple notes warns users that they are on an older version and will experience degraded features until they upgrade