MongoDB Connector

with the https //www mongodb com/docs/atlas/app services/sync/device sync deprecation/#std label device sync deprecation , ditto is the recommended replacement for synchronizing edge devices with mongodb refer to our docid 7yuzlks39mfi2dblajh54 for a comprehensive guide on moving from atlas device sync to ditto the mongodb connector is currently in private preview and is likey to evolve further, prior to its full release you can sign up for the https //ditto live/platform/mongodb connector waitlist to request access to the preview! the ditto mongodb connector, built in partnership with mongodb, provides seamless bidirectional synchronization between ditto apps and mongodb databases it allows you to extend your data in mongodb's developer platform to the edge via ditto's powerful edge synchronization capabilities, effectively bridging the gap between edge and cloud data management how it works the mongodb connector is provided as a managed service, running alongside the big peer that you are using for your ditto applications the big peer is a centralized cloud datastore and synchronization engine that is analogous to the atlas device sync service only data currently stored within the big peer will be replicated to/from individual small peers (devices), so purpose of the connector is to synchronize data as it changes between mongodb and the big peer to do this, the connector listens to changes to documents in both ditto and mongodb when it detects a change to a document it then performs conflict resolution between the two systems and updates the document in the target system with the new value this conflict resolution mechanism roughly models ditto's docid\ g71cudhi5isfkvfrvs4pt , ensuring that changes are not unexpectedly lost when mutations are made to the same documents in both mongodb and ditto simultaneously unlike other synchronization systems for mongodb, ditto's advanced conflict resolution (based on crdts) avoid the need to deploy extra backend services to handle writes and conflicts, just deploy a big peer and you have everything you need to integrate with mongodb setting up the connector to setup the connector, you first must carry out some steps within your mongodb cluster, then configure the connector via api to launch the connector pre requisites you need to have a mongodb cluster already setup, the minimum supported version for the connector is currently mongodb 7 0 13 and mongodb 8 0 0 you need to follow a number of steps to ensure that mongodb is ready to be used with the connector create mongodb database if your target database doesn't currently exist, then you must first create this within mongodb no special configuration is required for the created database an existing database can be used if one exists create mongodb collections all collections you wish to synchronize with ditto must exist within mongodb before setting up the connector each of these synchronized collections must have changestreampreandpostimages set to true this setting allows the connector to ensure causal consistency between both systems, and the connector will not start if this is not set for all collections when creating new collections, pass changestreampreandpostimages true to your creation command for pre existing collections, you can modify this configuration using collmod for example, using mongosh as user with appropriate permissions db runcommand({ collmod \<collectionname>, changestreampreandpostimages { enabled true }}) if you are using pre existing collections, existing data in these collections will not be synchronized unless they are modified after the connector has been setup create a mongodb database user the connector authenticates to mongodb using the username and password of a database user we recommend creating a dedicated database user for the connector to use within mongodb at a minimum the database user will require the readwrite permission for the specific database that you're looking to synchronize with ditto add ditto ips to mongodb allowlist when configuring the connector, you will have a set of three ips returned that represent the egress ips of your big peer you need to add these three ips to the allowlist of your mongodb cluster to allow the connector to access the cluster correctly note that these ips may differ between ditto apps depending on the underlying big peer that is hosting the app configuring the connector the connector is configured using the /v1/mongodb/link api provided by ditto's cloud service you need to provide the following information as part of the request app id the id of the ditto app that you wish to set the connector up for connection string the connection string, including the credentials of the created database user, that should be used to connect to mongodb collections the object specifying the collections and id mappings to use for the connector, see /#id mapping between systems for more details an example payload looks as follows { "app id" "1bff513e 4755 4b48 bbab 55c305f0b3d9", "connection string" "mongodb+srv //\<username> \<password>@beyondthebasics abcde mongodb net/test", "collections" { "collection1" { "fields" \[ "field1", "field2" ] }, "collection2" { "fields" \[ " id" ] } } } data modelling considerations both ditto and mongodb store their data in the form of json like documents, so are a natural fit together when building applications that run on the edge unlike integrations with other systems that rely on synchronizing to relational forms (e g sqlite), you do not need to remodel your data to use ditto and mongodb together however, there are a few considerations that you need to make when using ditto id mapping between systems due to security considerations in a peer to peer context, ditto's permission system (see docid\ hrv1c0ofsvhppljevx8r2 ) only has access to the unique and immutable id field ( id ) to use permissions effectively, most ditto applications use objects rather than strings as their id these sub fields with the id are then often duplicated in the document's body to facilitate pojo/dto like patterns in your application code for example you may choose to model an order (in a retail system) as follows { " id" { "location" "londonliverpoolstreet", "orderid" "7c0c20ed b285 48a6 80cd 6dcf06d52bcc" }, "location" "londonliverpoolstreet", "orderid" "7c0c20ed b285 48a6 80cd 6dcf06d52bcc", } this data model would allow you to effectively scope user's permissions down to orders originating from a specific location, so that stores cannot view other stores orders ids in mongodb, on the other hand, are commonly objectids , essentially uuids with a time based component; alternatively you may choose to use a single field value as the id (e g orderid ) it's very rare for ids in mongodb to be used in the same way as within ditto, therefore ids likely need to be handled differently in both systems the mongodb connector automatically bridges this difference in ids, so that both systems can converge, even if their primary keys differ it does this through the "id mapping" process, based on the configuration that you provide to the connector fields used in the id mapping must be immutable, always present, and a unique combination across all documents to achieve convergence between the two systems if fields used in the id mapping are missing from the document then the document will ignored by the connector and tracked as an unsync'ed document if id fields mutate or are duplicated across different documents, it will result in undefined behavior, such as duplicate ditto documents with different ids, or failure to synchronize the document between mongodb and ditto there are 3 modes for the id mapping 1 1 id (i e the same id is used in both ditto and mongodb) single document field multiple document fields 1 1 id this mode is used if the only field mapped for a given collection is id in that case the id fields used between ditto and mongodb will be identical however, some restrictions apply here null cannot be present in the id arrays cannot be present in the id no object fields within the id can start with a $ bson types in the id that aren’t direct ditto json types (e g decimal or datetime) aren't currently supported as they cannot be automatically mapped between systems single document field this mode is used if only a single (non id ) field is specified in the field mapping in this mode, all generated ditto ids will be set to the value of the selected field for example, with a field mapping of just the orderid field and the following mongodb document mongodb { " id" objectid("abc123"), "location" "londonliverpoolstreet", "orderid" "7c0c20ed b285 48a6 80cd 6dcf06d52bcc", "items" \[ "fries", "burger", "soda" ] } the connector would insert the following document into ditto { " id" "7c0c20ed b285 48a6 80cd 6dcf06d52bcc", "location" "londonliverpoolstreet", "orderid" "7c0c20ed b285 48a6 80cd 6dcf06d52bcc", "items" \[ "fries", "burger", "soda" ] } multiple document fields when the id mapping contains multiple fields, like the restaurants collection in the example above, the ditto id will become an object with the specified fields and corresponding values from the document note that the id mapping cannot refer to the id field itself in this case for example, with a field mapping of the location and orderid fields and the following mongodb document mongodb { " id" objectid("abc123"), "location" "londonliverpoolstreet", "orderid" "7c0c20ed b285 48a6 80cd 6dcf06d52bcc", "items" \[ "fries", "burger", "soda" ] } the connector would insert the following document into ditto ditto document { " id" { "location" "londonliverpoolstreet", "orderid" "7c0c20ed b285 48a6 80cd 6dcf06d52bcc" }, "location" "londonliverpoolstreet", "orderid" "7c0c20ed b285 48a6 80cd 6dcf06d52bcc", "items" \[ "fries", "burger", "soda" ] } mongodb datatypes mongodb stores data as bson, which includes rich datatypes such as native date objects, you can read more about the datatypes available in https //www mongodb com/docs/manual/reference/bson types/ ditto represents its data using json with a few key additional datatypes (e g integer), which does not have the full set of datatypes available in bson the connector will perform automatic mapping of bson datatypes to json when synchronizing data from mongodb to ditto the full set of mappings are listed below mongodb bson datatype ditto json datatype double number string string object object array array objectid string boolean boolean date string (iso8601 format) null null regular expression object (see https //www mongodb com/docs/manual/reference/mongodb extended json/#mongodb bsontype regular expression ) 32 bit integer integer 64 bit integer integer timestamp number (epoch timestamp) decimal128 string all bson datatypes that are not listed above will not be replicated between ditto and mongodb when replicating data from mongodb to ditto, the connector will store metadata allowing the connector to retain the mapping between datatypes when writing the same document back to mongodb in the case that a document is created within ditto, the document will be inserted into mongodb using only the basic json datatypes internal metadata in order to provide consistency across the two systems, the connector uses some internal metadata specifically this metadata tracks current mongodb sessions, to avoid circular writes between the two systems this is stored in the ditto connector sessions internal collection within your mongodb database documents that failed to synchronize from ditto to mongodb (for example they may have been rejected due to schema validation issues) these are stored in the ditto unsynced documents internal collection within your mongodb database you can use this collection to debug why certain documentrs are not synchronizing between systems, typically due to failed id mappings bson field mappings for documents, currently stored as a metadata field (prefixed with within the ditto document) current limitations the connector is still in preview, so it does not yet have all the functionality that is expected to be available at the ga release of the connector notably there are some key limitations with the connector, that will be addressed over the preview period the connector reads from mongodb change streams and so only performs ongoing replication between ditto and mongodb if you need to copy existing data over from mongodb to ditto you can do the following create a new database in your mongodb cluster add the necessary collections to your database setup the integration within ditto, ensuring it is running copy the data across from the existing database to the new database so that the connector will synchronize the data into ditto the connector only supports the use of the register datatype within ditto other datatypes such as map are not synchronized by the mongodb connector filtering of data to synchronize with mongodb is done at the collection level, filtering data within a collection to synchronize is not possible connection to a mongodb atlas cluster is routed via the internet rather than private networking options you should add the 3 provided ips during the setup process to your atlas project's allowlist to allow the connector to connect to the cluster it is not possible to change the configuration (e g add or remove a collection, change password) from a running connector you must delete the connector then create it again with the new configuration richer bson datatypes are not used by the connector when documents or fields are added on the ditto side, only json types will be used bson types inserted into mongodb will be retained by the connector