Database Migration — Quick Guide

Database Migration is a broad and complex term which is related to the movement of data from one database to similar/different database(s) on cloud or related to database change management (schema migration) or to upgrading a major version of database software being used. A physical transformation process may be required when there is a change in data format between source and target.

Standard Process involves 3 major phases and each phase involves its own steps to accomplish overall data migration.

Types of Database Migration

1. Schema Upgrade is an easy process as compared to the data migration in cloud and database software upgrade. This migration is performed by database/schema migration tools like Flyway and Liquibase to update or revert a database schema to some newer or older version. It controls different versions of schemas which is an essential part of software evolution in agile. Both support most of the currently used database engines, such as Postgres, Oracle, SQL Server, DB2, H2, MariaDB, and many others. They also support some cloud base database services like Amazon RDS. Both have rollback and change management support. Liquibase is an open-source database migration tool where some of its capabilities like rollback is supported in the paid version of Flyway.
2. Database Version Upgrade strategy depends on the benefits vs risks involved in the migration process. Migration to newer software releases is less complex as compared to a major version upgrade since it requires a new installation of the software. An upgrade plan needs to be prepared with a fallback mitigation plan beforehand to avoid disruption to the existing users and integrated applications.
3. Cloud Migration refers to the movement of on-premises databases to the cloud. It is a one-way journey that ends once all the information is transported to a target location. At times, this migration is simple and a bit straightforward in case data formats are the same in source and destination platforms. Otherwise, it is tougher, especially if target and source databases support different data formats and it requires ETL tools for a successful migration and refactoring of applications. Popular cloud Migration tools are AWS database migration service, Azure database migration service and BigQuery Data Transfer Service.

Cloud migration Strategies for database

Cloud Migration Strategy refers to the process of moving existing data to a new cloud storage location. There are typically two approaches where some organizations use both approaches at the same time due to cloud migration constraints in their migration strategy.

1. Lift and Shift move is a one-time movement required when there is no change in data-formats and databases. It is just a change in platform from on-premises to a cloud provider. For example, data lakes or warehouse data are best fit for Lift and Shift which can be done via dedicated network connections or by physically transferring the data.
2. Hybrid Model is another approach weighted toward the cloud. On-premises data stores may shift toward the cloud over time. It depends on factors like amount of data, network, data classification such as backup data, data warehouses or lakes, critical data and the amount of migration time which may take to complete this activity. Hybrid model itself is divided into one-way, two-way and zero downtime migrations based on the downtime expected for a migration.

One-way migration is the best option for a small-scale database which requires you to shut down your application for 2–5 days. In the meantime, the database is extracted and migrated to the destination database in the cloud. Once it finishes migration, it needs to be validated with the source database for no-data loss. After that a final cutover can be completed.

In the other case, Two-way migration requires minimal downtime and it is applicable for databases of any size. Information is extracted from the source database and is migrated to the target database while the source database is still up and running. Since during the entire migration time, source database is still up and running hence delta changes will need to be migrated before the final cutover. In the process, entire downtime is limited to a few hours when actual data synchronization is required between source and destination databases for all those delta changes. After all the changes have been transferred to the target database. Data validation is required before final cutover.

Near Zero-downtime strategy is required for mission critical databases that cannot have any downtime or a few minutes. It requires detailed planning and replication tools. You can use synchronous replication in case of zero downtime. Benefit of this approach is you can plan cutover anytime since source and target databases are always in sync with one another but it comes with a cost of application latency. Hence, asynchronous replication is mostly preferred, if a few minutes downtime can be afforded. This downtime is required for small delta changes which are left to be transferred to the target database. After that validation is performed before final cutover.

Database Migration Strategy

Tip: ETL tools will be recommended when there is a change in data formats between on-premises database and the cloud database. Popular tools are Xplenty, Talend and Stitch available in both free and paid variants. It requires much upfront agile team effort.

Integration and Validation

Migration, integration and validation go hand in hand to ensure continuous integration and validation in the cloud during the entire process.

Primary Checklist

• Application is running with proper network configurations
• Validate server configurations such as RAM, CPU and Disk Storage
• Basic cloud operations like start and stop instances

Integration and Validation Checklist

• Run Integration Tests
• Validate integration with third-party applications
• Sign-off from business stakeholders

Cutover

Cutover is the final step of cloud migration. In this phase, we redirect application traffic from on-premises environment to the target cloud environment. Some factors to consider while deciding a cutover strategy are as follows:

• Acceptable downtime for the application
• Preferred time slots for cutover when traffic is less or minimal
• Ratio of write vs read operations
• Database syncs and backup requirements
• Decide the suitable deployment strategy like Red-Black or Blue-Green

Tip: Red-Blue Deployment is the instant cutover to the new server from the existing server and Blue-Green deployment is the replacing existing server gradually with the new server.

Live migration cutover for mission-critical databases

It is preferred for mission-critical databases with zero downtime. In this approach, data is continuously replicated to the target followed by validation and integration testing at the destination while the application is still up and running. In the replication process, the source on-premises database and target cloud database are always in sync. Hence, application traffic is distributed between on-premises and in the cloud. We gradually increase the traffic to the cloud applications until all the traffic is directed to the new application, thus resulting in a cutover with zero downtime.

Live Migration Cutover by Blue-Green Deployment

Summary

Database migration is not a straight line for agile teams. It requires a good amount of planning and good migration strategy in place before initiating the process. Every migration strategy for application data must have a failure mitigation strategy beforehand. Picking up any strategy typically depends on the factors like amount of data, network, data classification such as backup data, data warehouses or lakes, critical data and the amount of migration time which may take to complete this activity. Hopefully, this guide will help you in your migrations. Good luck!