MySQL Database Service is a fully managed cloud service, 100% Developed, Managed and Supported by the MySQL Team.
This is the sixth episode of “Discovering MySQL Database Service“, a series of tutorials where I will show you, step by step, how to use MySQL Database Service and some other Oracle Cloud Infrastructure services.
Please also note that you can run this tutorial and thus try MySQL Database Service & the other Oracle Cloud Infrastructure services for free by starting your 30-day trial.
Episode 6 – Update the Private Subnet Security List
In the previous episode we’ve seen how to create a MySQL DB system from a MySQL Shell dump stored into an Oracle Cloud Infrastructure object storage bucket. One more step to our Discovering MySQL Database Service journey.
In this episode, we’ll introduce the concept of Security Lists a virtual firewall to control traffic at the network packet level.
Security Lists
A security list acts as a virtual firewall for an instance, with ingress and egress rules that specify the types of traffic allowed in and out. Each security list is enforced at the VNIC level. However, you configure your security lists at the subnet level, which means that all VNICs in a given subnet are subject to the same set of security lists. The security lists apply to a given VNIC whether it’s communicating with another instance in the VCN or a host outside the VCN.
The default security list comes with an initial set of stateful rules, which should in most cases be changed to only allow inbound traffic from authorized subnets relevant to the region that homes that VCN or subnet.
Note Your OCI user must be part of a group that has the right privileges in order to work with Security Lists. At least the following rules: Allow group SecListAdmins to manage security-lists in tenancy Allow group SecListAdmins to manage vcns in tenancy Please see: Working with Security Lists
Update the Security List
Go to the OCI console, in the menu, go to: Networking / Virtual Cloud Networks And select (click on) your VCN ( Demo_VCN in this example), and please check that you are in the right compartment.
Select (click on) the privatesubnet (Private Subnet-Demo_VCN in this example):
Select the Security List for the Private Subnet (Security List for Private Subnet-Demo_VCN in this example):
Click on Add Ingress Rules:
Now we are adding our ingress rules in order to be able to connect to the MDS instance from the VCN on ports 3306 (MySQL classic protocol) and 33060 (MySQL X protocol).
The information to provide are (please adapt to your context): + Source type: CIDR + Source CIDR: 10.0.0.0/16 + IP protocol: TCP + Source port range: All + Destination port range: 3306,33060
Then click on Add Ingress Rules.
You should end up with something like:
Well, that’s all for today! In this episode, we have seen how to allow traffic from the VCN to the MySQL Database Service instance on ports 3306 & 33060. We are now able to connect to our MySQL instance…
MySQL Database Service is a fully managed cloud service, 100% Developed, Managed and Supported by the MySQL Team.
This is the fifth episode of “Discovering MySQL Database Service“, a series of tutorials where I will show you, step by step, how to use MySQL Database Service and some other Oracle Cloud Infrastructure services.
Please also note that you can run this tutorial and thus try MySQL Database Service & the other Oracle Cloud Infrastructure services for free by starting your 30-day trial.
Episode 5 – Create a MySQL DB system from a MySQL Shell dump
In the previous episode we’ve seen how to export data from a MySQL instance to an Oracle Cloud Infrastructure bucket using the awesome MySQL Shell. One more step to our Discovering MySQL Database Service journey.
In this episode, we’ll see how these data can now easily be imported into a MySQL Database Service instance.
I would recommend to use the later because it is the easiest way to create and populate with your own data a MDS. So let’s see how easy it is!
MySQL Database Service
MySQL Database Service is a fully managed Oracle Cloud Infrastructure native service, developed, managed, and supported by the MySQL team in Oracle. Oracle automates all tasks such as backup and recovery, database and operating system patching, and so on.
You are responsible solely for managing your data, schema designs, and access policies.
Note Your OCI user must be part of a group that has the right privileges in order to work with MySQL Database Service. At least the following rules: Allow group to {SUBNET_READ, SUBNET_ATTACH, SUBNET_DETACH, VCN_READ, COMPARTMENT_INSPECT} in [ tenancy | compartment | compartment id ] Allow group to manage mysql-family in [ tenancy | compartment | compartment id ] Allow group to use tag-namespaces in tenancy Please see: Mandatory Policy Statements / Policy Details for MySQL Database Service
Create a MySQL DB system from a MySQL Shell dump
Go to the OCI console, in the menu, go to: Databases / MySQL
Select the right compartment (left of the screen – DBA in this example) then press Create MySQL DB System.
Provide DB System information & create administrator credentials
Check that you are in the right compartment, name your MySQL instance and enter a meaningful description (optional but recommended).
Select your instance type – Standalone in this example – then create the administrator credentials, usaername & password (see this like your root account, although you obviously will not have all the privileges because it’s a PaaS).
Let’s take a quick look at the different current types of MySQL DB systems:
Standalone: specifies a single-instance DB System.
High Availability: specifies a three-instance DB System containing one primary instance, and two secondary instances. See High Availability for more information.
HeatWave: configures a Standalone DB System with a HeatWave-compatible shape (MySQL.HeatWave.VM.Standard.E3) and 1TB of data storage, by default. For more information, see HeatWave.
You can also place your MySQL DB system in the Availability Domain of your choice if your region has several. Same logic for the Fault Domain.
Configure hardware
This is were you’re going to choose the “power” of your MySQL instance. Currently it starting from 1 to 64 CPU cores and 8 GB to 1 TB for the memory size.
In the Data Storage Size box you can enter in GB the amount of storage to allocate to the MySQL DB System for all data and log files.
Configure Backup Plan
You can specify the backup details ie enable/disable automatic backups, setup the retention period and the backup window.
We almost there 🙂 In fact, if you want to create an empty MySQL DB system you can push the Create button right now.
In order to create the PAR URL for our MySQL Shell dump stored in the bucket, we must click on Click here to create a PAR URL for an existing MySQL Shell dump file (@.manifest.json)
A new form appears where you need to
Select the right bucket (again check that you are in the right compartment), bucket in DBA
Select the MySQL Shell dump manifest file (@.manifest.json)
Specify an expiration time for the PAR
Brilliant! We have all the information to Create and set PAR URL.
So we can create our MySQL DB system that will contain our data. Click on Create.
After a while (mostly dependent on the quantity of data to import) your MySQL instance is up and running \o/
You then have access to diverse information. Among others, the endpoint which gives you some relevant ones from the connection point of view like the private IP Address and the default MySQL 8.0 ports 3306 & 33060 respectively for classic and X protocols.
Well, that’s all for today! In this episode, we have seen how to create a MySQL DB system from a MySQL Shell dump stored into an Oracle Cloud Infrastructure bucket.
MySQL Database Service is a fully managed cloud service, 100% Developed, Managed and Supported by the MySQL Team.
This is the fourth episode of “Discovering MySQL Database Service“, a series of tutorials where I will show you, step by step, how to use MySQL Database Service and some other Oracle Cloud Infrastructure services.
Please also note that you can run this tutorial and thus try MySQL Database Service & the other Oracle Cloud Infrastructure services for free by starting your 30-day trial.
Episode 4 – Dump your MySQL data into an Object Storage bucket
In the previous episode we’ve created our Virtual Cloud Network, that provides you with complete control over your cloud networking environment that we will create in this Discovering MySQL Database Service journey.
In this episode, we’ll see what is OCI Object Storage and how to export data from it to a MySQL instance using MySQL Shell. In fact, in the process of creating a MySQL DB system, this step is optional. But it is especially useful if you want to create a MDS instance with data from a MySQL server on premise or in any public/private cloud.
Because it is a plausible scenario, let’s see how to do that!
The workflow
The workflow is quite simple. We will export all or part of our MySQL instance data using MySQL Shell into an Oracle Cloud Infrastructure Object Storage. Then we will import the data from the bucket to a MySQL Database Service instance. Today we are going to see only the first part (red rectangle in the picture below).
The Oracle Cloud Infrastructure Object Storage service is an internet-scale, high-performance storage platform that offers reliable and cost-efficient data durability. The Object Storage service can store an unlimited amount of unstructured data of any content type, including analytic data and rich content, like images and videos.
With Object Storage, you can safely and securely store or retrieve data directly from the internet or from within the cloud platform. You can access data from anywhere inside or outside the context of the Oracle Cloud Infrastructure, as long you have internet connectivity and can access one of the Object Storage endpoints.
Create an Object Storage Bucket
Here as well, many ways to do that. The more convenient way is probably to use the OCI console.
In the menu, go to: Storage / Object Storage & Archive Storage / Buckets
Select the right compartment (left of the screen – DBA in this example) then press Create Bucket.
The bucket must have a name (Sakila_dump in this example – if you want to test with a sample database, Sakila database is available here).
The Default Storage Tier must be Standard – the default value.
Then you good to go!
Now let’s export the data to this fresh new created object storage bucket.
Export data to the bucket
You already get it, were going to use this amazing tool called MySQL Shell. But before we need to make some configuration for allowing MySQL Shell to easily connect to the OCI object storage service.
To this end, we will create an OCI CLI configuration file.
The CLI is a small-footprint tool that you can use on its own or with the Console to complete Oracle Cloud Infrastructure tasks. The CLI provides the same core functionality as the Console, plus additional commands. Some of these, such as the ability to run scripts, extend Console functionality.
To be clear, we will not use the CLI but its configuration file. The configuration file name and default location (on Linux) is /home/opc/.oci/config.
It should have the following information:
user: OCID of the user calling the API.
fingerprint: Fingerprint for the public key that was added to this user.
key_file: Full path and filename of the private key.
Awesome! Now we have all the parts to export our MySQL data with the MySQL Shell dump utilities.
MySQL Shell
MySQL Shell is an advanced client and code editor for MySQL. In addition to the provided SQL functionality, similar to the mysql text client, MySQL Shell provides scripting capabilities for JavaScript and Python and includes APIs for working with MySQL. X DevAPI enables you to work with both relational and document data. AdminAPI enables you to work with InnoDB Cluster.
In this article we are mainly interested in the utility part of MySQL Shell. MySQL Shell includes utilities for working with MySQL. To access the utilities from within MySQL Shell, use the util global object, which is available in JavaScript and Python modes, but not SQL mode. The util global object provides many functions like checkForServerUpgrade(), importJSON(), exportTable(), importTable(), loadDump(), util.dumpTables(), util.dumpInstance(), util.dumpSchemas(), …
In order to create a MySQL Database Service instance automatically populated with your data, you will use either util.dumpInstance() or util.dumpSchemas().
I’m going to use the later – dumpSchema() – but please keep in mind that the logic is the same for dumpInstance().
There are many relevant options. I will briefly describe them below (please RTFM for more details).
ocimds: Setting this option to true enables checks and modifications for compatibility with MySQL Database Service.
osBucketName: The name of the Oracle Cloud Infrastructure Object Storage bucket to which the dump is to be written.
osNamespace: The Oracle Cloud Infrastructure namespace where the Object Storage bucket named by osBucketName is located.
compatibility: Apply the specified requirements for compatibility with MySQL Database Service for all tables in the dump output, altering the dump files as necessary.
threads: The number of parallel threads to use to dump chunks of data from the MySQL instance. Each thread has its own connection to the MySQL instance.
dryRun: Display information about what would be dumped with the specified set of options, and about the results of MySQL Database Service compatibility checks (if the ocimds option is specified), but do not proceed with the dump. Setting this option enables you to list out all of the compatibility issues before starting the dump.
ociParManifest: Setting this option to true generates a pre-authenticated request for read access (an Object Read PAR) for every item in the dump, and a manifest file listing all the pre-authenticated request URLs. The pre-authenticated requests expire after a week by default, which you can change using the ociParExpireTime option.
ociParExpireTime: The expiry time for the pre-authenticated request URLs that are generated when the ociParManifest option is set to true. The default is the current time plus one week, in UTC format.
If your config file is not in the default location, you will also need ociConfigFile.
ociConfigFile: An Oracle Cloud Infrastructure CLI configuration file that contains the profile to use for the connection, instead of the one in the default location ~/.oci/config.
osBucketName is the name of the bucket we created previously – Sakila_dump in this article.
osNamespace is available in the OCI console. Open the menu and go to Storage / Object Storage & Archive Storage Click on the Bucket name (if you don’t see your bucket , check if you are in the right compartment). Namespace field is what your are looking for.
compatibility provides many types of modification. Please read carefully the documentation in order to use the correct setup for your needs. I’m using here: force_innodb, skip_invalid_accounts, strip_definers, strip_restricted_grants, strip_tablespaces.
If you have tables without primary keys (that is usually not a good thing) you can force the dump using the ignore_missing_pks compatibility mode. But you will not be able to use some advanced features like MDS High Availability, nor MDS HeatWave)
Another alternative is to use create_invisible_pks instead of ignore_missing_pks. Please check the documentation for the relevant details.
ociParManifest & ociParExpireTime are very useful in this context. They will allow you to create a Pre-Authenticated Requests (PAR) and setup an expiration time of your choice. Thus you will be able to load your data whenever it suits you.
To create or manage pre-authenticated requests, you need PAR_MANAGE permission to the target bucket.
Note While you only need PAR_MANAGE permission to create a pre-authenticated request, you must also have the appropriate permissions for the access type that you are granting. Details here
Connect to your (on-premise / private or public cloud) MySQL instance using MySQL Shell:
Well, that’s all for today! In this episode, we have seen how to export data from a MySQL instance to an Oracle Cloud Infrastructure bucket using this awesome tool called MySQL Shell. These data can now easily be imported in a MySQL Database Service instance.
MySQL Database Service is a fully managed cloud service, 100% Developed, Managed and Supported by the MySQL Team.
This is the third episode of “Discovering MySQL Database Service“, a series of tutorials where I will show you, step by step, how to use MySQL Database Service and some other Oracle Cloud Infrastructure services.
Please also note that you can run this tutorial and thus try MySQL Database Service & the other Oracle Cloud Infrastructure services for free by starting your 30-day trial.
In the previous episode we’ve created our compartment, the foundation stone, of the architecture that we will build during this Discovering MySQL Database Service journey.
In this episode, we’ll see what is a Virtual Cloud Network (VCN) and how to create one and use it.
Virtual Cloud Network (VCN)
Oracle Virtual Cloud Networks (VCNs) provide customizable and private cloud networks in OCI. Just like a traditional data center network, the VCN provides customers with complete control over their cloud networking environment. This includes assigning private IP address spaces, creating subnets and route tables, and configuring stateful firewalls. For more information please see VCNs and Subnets.
Create a Virtual Cloud Network
If you are a network expert, OCI provides you the tools and the granularity to create your VCN.
if, like me, your are not an expert and/or if you want to quickly create a VCN with all the relevant component, the VCN Wizard is the right feature for you.
Creates a regional public subnet with routing to the internet gateway. Instances in a public subnet may optionally have public IP addresses.
Creates a regional private subnet with routing to the NAT gateway and service gateway (and therefore the Oracle Services Network). Instances in a private subnet cannot have public IP addresses.
Sets up basic security list rules for the two subnets, including SSH access.
Brilliant! This is exactly what we need.
Select Create VCN with Internet Connectivity from the VCN Wizard. It creates a VCN with a public subnet that can be reached from the internet. Also creates a private subnet that can connect to the internet through a NAT gateway, and also privately connect to the Oracle Services Network.
To summarize we will have the following component:
VCN
Public Subnet
Private Subnet
Internet Gateway (IG)
NAT Gateway (NAT)
Service Gateway (SG)
You should provide a name for the VCN – Demo_VCN in this example.
Check that the compartment is the good one, the one we’ve created in the previous episode – DBA in this example.
The VCN and subnet default configuration should be fine.
After clicking on the Next button, you can review your configuration:
Click on Create when you are ready…
Et voilà!
You should end up with something that looks like:
Among others an IPv4 CIDR block (10.0.0.0/16) and 2 subnets (private in 10.0.1.0/24 and public in 10.0.0.0/24)
Well, that’s all for today! In this episode we have created an already configured Virtual Cloud Network for use with MySQL DB Systems.
MySQL Database Service is a fully managed cloud service, 100% Developed, Managed and Supported by the MySQL Team.
This is the second episode of “Discovering MySQL Database Service“, a series of tutorials where I will show you, step by step, how to use MySQL Database Service and some other Oracle Cloud Infrastructure services.
Please also note that you can run this tutorial and thus try MySQL Database Service & the other Oracle Cloud Infrastructure services for free by starting your 30-day trial.
In the previous episode we’ve introduced the different components that will be used during this Discovering MySQL Database Service journey.
In this episode, we’ll see what is a compartment, how to create one and use it in order to create a MySQL DB system.
Compartment
A compartment is a collection of related resources that can be accessed only by groups that have been given permission by an administrator in your organization.
Actually, compartment is a powerful Oracle Cloud Infrastructure feature for security that would allow you to organize and isolate your cloud resources.
Please note that it is not mandatory to create compartments, it means that all your assets would be in the root compartment (the default) but it is obviously a bad practice 🙂
Create a compartment
Create a compartment is very simple.
Assuming that you already setup your tenancy —Setting Up Your Tenancy — you can use the console menu :
Go to: Identity & Security and Compartments
Fill Name and Description fields
Then push Create Compartment button
Here we go! The compartment is now created.
Well, that’s all for today! In this episode, following the best practices, we have created our compartment where we will store our resources. Thus we are able to enforce security isolation and access control.
MySQL Database Service is a fully managed cloud service, 100% Developed, Managed and Supported by the MySQL Team.
This is the first episode of “Discovering MySQL Database Service“, a series of tutorials where I will show you, step by step, how to use MySQL Database Service and some other Oracle Cloud Infrastructure services.
Please also note that you can run this tutorial and thus try MySQL Database Service & the other Oracle Cloud Infrastructure services for free by starting your 30-day trial.
Like any series, in this episode I’m going to give you some context and set up the characters.
Open Source Software
As you may know, Open Source Software has been mainstream in the enterprise for quite some time. Open Source usage still increases and it also drives innovation (you can read this and this). This is particularly the case for Open Source Databases, like MySQL obviously.
MySQL
MySQL, The world’s most popular open source database
Most popular database (Jetbrains survey – Stackoverflow survey) means you’ll find experts, experienced DBA, skilled developers, tons of resources… to help you to grow your business.
Popular also means that MySQL is widely used and used in many industries that run various workloads.
MySQL is the ideal database if your use case is:
Cloud Native Apps
Move Existing Workloads
Hybrid Cloud Flexibility
SaaS Applications
MySQL is the right database for on-premise, cloud and hybrid architectures.
Oracle Cloud Infrastructure (OCI)
Oracle Cloud Infrastructure (OCI) is a deep and broad platform of public cloud services that enables customers to build and run a wide range of applications in a scalable, secure, highly available, and high-performance environment.
Some highlights of this Next-generation cloud infrastructure:
You can freely use Oracle Cloud Infrastructure services – including MySQL Database Service – by trying the Oracle Cloud Free Tier.
What’s included with Oracle Cloud Free Tier?
What’s included with Oracle Cloud Free Tier?
MySQL Database Service (MDS)
In Short: MDS = OCI + MySQL
Ok, please allow me to elaborate a little more 🙂
MySQL Database Service is a fully managed cloud service. Meaning that low-value tasks (no direct value for the business) are done by the MySQL team at OCI:
MDS is a fully managed cloud service
It is also worth noting that MDS is:
100% Developed by the MySQL team
100% Managed by the MySQL team
100% Supported by the MySQL Team, 24/7
100% Up to date with the latest security fixes
100% Built on MySQL EnterpriseEdition
100% Compatible with on-premises MySQL
100% Compatible with Oracle technologies
In terms of features, you will obviously find the common one, High Availability, Replication, Migration tools,… Learn more…
However it is exclusively in OCI that you will find HeatWave : the only MySQL cloud service with a massively-scalable integrated analytics engine. In other words, HeatWave is a MySQL cloud service with an integrated, high-performance, in-memory query accelerator that enables customers to run sophisticated analytics directly against their operational MySQL databases, eliminating the need for complex, time-consuming, and expensive data movement and integration with a separate analytics database. Learn more…
Last but not least, with MDS you’re going to save money!
Components
A DBMS is indeed part of a chain of components. In this series we will deal with different concepts and components. Below the main ones:
Tenancy A secure and isolated partition within Oracle Cloud Infrastructure (OCI) where you can create, organize, and administer your cloud resources. When you sign up for OCI, a tenancy is created for your company. Tenancy also refers to the root compartment that contains all of your organization’s compartments and other OCI resources.
Region A collection of availability domains located in a single geographic location.
Availability Domain One or more isolated, fault-tolerant Oracle data centers that host cloud resources such as instances, volumes, and subnets. A region contains one or more availability domains.
Fault Domain A logical grouping of hardware and infrastructure within an availability domain. Fault domains isolate resources during hardware failure or unexpected software changes.
Groups A collection of users who all need a particular type of access to a set of resources or compartment.
Policy An Identity and Access Management (IAM) document that specifies who has what type of access to your resources. Policy can refer to several types of documents: an individual statement written in the policy language, a collection of statements in a single named “policy” document, and the overall body of policies that your organization uses to control access to resources.
Compartment A collection of related resources that can be accessed only by groups that have been given permission by an administrator in your organization.
Virtual Cloud Network (VCN) A virtual version of a traditional network — including CIDRs, subnets, route tables, and gateways — on which your instance runs.
Security Lists A virtual firewalls for your Compute instances and other kinds of resources.
Object Storage An internet-scale, high-performance storage platform that offers reliable and cost-efficient data durability.
MySQL Database Service (MDS) A fully managed database service that lets developers quickly develop and deploy secure, cloud native applications using the world’s most popular open source database.
Bastion Provide restricted and time-limited secure access to resources that don’t have public endpoints and require strict resource access controls.
SSH Client Software program which uses the secure shell protocol to connect to a remote computer.
MySQL Shell An interactive Javascript, Python, or SQL interface supporting development and administration for the MySQL Server
MySQL Workbench A unified visual tool for database architects, developers, and DBAs that provides data modeling, SQL development, and comprehensive administration tools for server configuration, user administration, backup, and much more
Cloud Shell A web browser-based terminal accessible from the Oracle Cloud Console. It provides access to a Linux shell, with a pre-authenticated Oracle Cloud Infrastructure CLI, a pre-authenticated Ansible installation, and other useful tools.
In this series of articles, I assume that you already setup your tenancy. More information: Setting Up Your Tenancy
Well, that’s all for today! I have introduced you to the different characters of this Discovering MySQL Database Service series.
Not so much technical contents so far, but I swear there will be more next time 🙂
If you have (too) long running select queries it is probably because of lack of relevant indexes, problematic schema that lead to poor queries or inadequate hardware.
That said, sometime even if you doing it right, the query execution time could be too long regarding of what the application or your users expect. It is often true for reporting, real time analytics or BI queries.
If your application already use a MySQL database that is not MySQL Database Service, you can still use HeatWave, thanks to the Inbound Replication feature.
This is basically what you architecture will look like
Inbound Replication with MySQL Database Service
Your “problematic” select queries will be running on MDS (with HeatWave plugin enabled).
Create this architecture is pretty simple, this is the topic of this article…
Assumptions You have an application and a MySQL instance somewhere (on-premise / public | private cloud / on the moon, …), called 10.0.1.9 in this article. You’ve already created your HeatWave instance, called MDSHW(10.0.1.10) in this article.
Last but not least, you want to dramatically improve the response time of your select queries.
You can take advantage of the power of HeatWave, by setup a replication channel between your MySQL instance and a MySQL Database Service with a HeatWave cluster.
For simplicity, in this blog post the MySQL instance is on OCI. However this architecture is also relevant with the DB out of OCI. In that case you will need a VPN (e.g. Using OpenVPN with MySQL Database Service).
Also for simplicity, I’m using MySQL 8.0. However, the following architecture is also relevant with MySQL 5.7. To understand how to setup a replication channel from MySQL 5.7 to MySQL Database Service, you can read : Replicate from MySQL 5.7 to MySQL Database Service.
What is the plan?
Assuming we already have the application and a MySQL 8.0 database running, we will:
Create a dedicated replication user on the source
Create a dump of the MySQL instance
Load the dump into MySQL Database Service with HeatWave
Create a replication channel on MySQL Database Service with HeatWave
Enable a HeatWave cluster
Create a dedicated replication user on the source
Connect to the MySQL source instance (10.0.1.9), using MySQL Shell:
1
2
$
mysqlsh root@localhost:3306--sql
then create the replication user:
1
2
MySQL localhost:3306ssl SQL>
CREATE USER rpl@'10.0.1.%'IDENTIFIED BY'Rpl1234_'REQUIRE SSL;
with his relevant privileges:
1
2
MySQL localhost:3306ssl SQL>
GRANT REPLICATION SLAVE on *.*torpl@'10.0.1.%';
Simple!
Create a dump of the MySQL instance
If you think mysqldump (or mysqlpump) when you heard “dump” then it is time to upgrade your knowledge!!! 🙂
When using MySQL 8.0 or even 5.7, forget these 2 tools and please welcome MySQL Shellutilities and especially its load / dump tools – you’ll thank me later 😉
In this scenario the dump is stored locally. But MySQL Shell also allow you to store your dump into an OCI Object Storage Bucket. This could be a better alternative if your data set is large.
Super simple!
Load the dump into MySQL Database Service with HeatWave
Now it is the time to load the dump into MySQL Database Service with HeatWave (10.0.1.10).
MySQL Shell loadDump utility is even more amazing than you think! It allows you to start loading in parallel the dump even if it is not completed yet 🙂
Please note the updateGtidSet option. Documentation is available here.
Very simple!
Create a replication channel on MySQL Database Service with HeatWave
This step is even easier than the other 🙂 because most of the work will be done using the OCI console. I already described it in this article and you’ll find all the details in the documentation.
Below the main stages.
The feature we are looking for is named Channel. You can find it by clicking on “MySQL” then “Channel” and finally push the “Create Channel” button
Create a replication channel on MySQL Database Service
Then fill the form:
Create in Compartment: Choose the right Compartment
Source Connection – Configure Connection to the MySQL Source Hostname: 10.0.0.9
Username: rpl (from the replication user created on the source)
Password: Rpl1234_ (from the replication user created on the source)
SSL Mode: – set when create the replication user – Required (REQUIRED) Establish an encrypted connection.
Target – Configure the DB System target Select a DB System: MDSHW
Finally, push the Create Channel button… et voilà!
Wait for the Channel icon become green (ACTIVE)…
You can confirm the creation of the replication channel with the command SHOW REPLICA STATUS\G or run the following query using the replication channel name (default: replication_channel) :
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
MySQL10.0.1.10:3306ssl SQL>
SELECT
SERVICE_STATE,
HOST,
USER,
PORT,
CHANNEL_NAME
FROM performance_schema.replication_connection_configuration
The last stage covered in this article is how to enable the HeatWave cluster.
You must choose the number of node (2 minimum), it depends of the size of the data you want to put in HeatWave. The “Estimate Node Count” feature will help you.
Enable a HeatWave cluster
Then click Add HeatWave Cluster button.
What next?
There are some data preparation and finally load the data into the HeatWave cluster.
Up until now we have seen MDS (MySQL Database Service) and MySQL in Azure. As the Cloud technology keeps moving fast, I thought it would be a good idea to see how to set up a MySQL in Amazon , as a service, that is RDS.
Let’s get started then!
What we need is an Amazon subscription that is really easy to get one and there is also a Free Tier. The AWS Free Tier is available to you for 12 months starting with the date on which you create your AWS account. When your free usage expires or if your application use exceeds the free usage tiers, you simply pay standard, pay-as-you-go service rates.
For the specific session, I had root access. There is also the IAM user. AWS Identity and Access Management (IAM) is an AWS service that helps an administrator securely control access to AWS resources. IAM administrators control who can be authenticated (signed in) and authorized (have permissions) to use Amazon RDS resources.
Once inside the Amazon Subscription, we are seeing the Management Console:
From the Management Console, we are able to identify the RDS service and choose it:
Once inside the RDS service, we click on the Create Database:
From the Databases provided, we are choosing MySQL and Standard Create, as we would like to set up our configuration. With Easy Create enabled, you specify only the DB engine type, DB instance size, and DB instance identifier. Easy Create uses the default setting for other configuration options.
With Easy Create not enabled, you specify more configuration options when you create a database, including ones for availability, security, backups, and maintenance.
Now, on MySQL versions we have chosen the latest provided until now for Amazon, that is 8.0.21 :
If you noticed, there is also the choice to choose Production , Dev or Free Tier template. For Production Template, there are default options for High Availability and consistent performance. We have chosen the Free Tier, as this is for educational purposes.
We have also given a Database Identifier, database-1. Each DB instance has a DB instance identifier. This customer-supplied name uniquely identifies the DB instance when interacting with the Amazon RDS API and AWS CLI commands. The DB instance identifier must be unique for that customer in an AWS Region.
Moving on next are the admin’s credentials, the super user of the database:
The DB instance class determines the computation and memory capacity of an Amazon RDS DB instance. The DB instance class you need depends on your processing power and memory requirements. Apparently, since we are only testing this we have a basic tier:
db.t2 – Instance classes that provide a baseline performance level, with the ability to burst to full CPU usage.
Moving along with the rest of our options:
Amazon RDS provides three storage types: General Purpose SSD (also known as gp2), Provisioned IOPS SSD (also known as io1), and magnetic (also known as standard).
They differ in performance characteristics and price, which means that you can tailor your storage performance and cost to the needs of your database workload.
You can create MySQL, MariaDB, Oracle, and PostgreSQL RDS DB instances with up to 64 tebibytes (TiB) of storage. You can create SQL Server RDS DB instances with up to 16 TiB of storage. For this amount of storage, use the Provisioned IOPS SSD and General Purpose SSD storage types.
The following list briefly describes the three storage types:
General Purpose SSD – General Purpose SSD volumes offer cost-effective storage that is ideal for a broad range of workloads. These volumes deliver single-digit millisecond latencies and the ability to burst to 3,000 IOPS for extended periods of time. Baseline performance for these volumes is determined by the volume’s size.
Provisioned IOPS – Provisioned IOPS storage is designed to meet the needs of I/O-intensive workloads, particularly database workloads, that require low I/O latency and consistent I/O throughput.
Magnetic – Amazon RDS also supports magnetic storage for backward compatibility. We recommend that you use General Purpose SSD or Provisioned IOPS for any new storage needs. The maximum amount of storage allowed for DB instances on magnetic storage is less than that of the other storage types.
One the Availability & Durability section, you are seeing the option Multi-AZ deployment. Amazon RDS Multi-AZ deployments provide enhanced availability and durability for database instances, making them a natural fit for production database workloads. When you provision a Multi-AZ database instance, Amazon RDS synchronously replicates your data to a standby instance in a different Availability Zone (AZ).
The next section to check is that on of the Connectivity:
Be certain to choose the correct Virtual private cloud, as it cannot be changed after the database is created and set the proper access (IP-ranges) for your subnets.
Security groups control the access that traffic has in and out of a DB instance. Each VPC security group rule enables a specific source to access a DB instance in a VPC that is associated with that VPC security group. The source can be a range of addresses, or another VPC security group. By specifying a VPC security group as the source, you allow incoming traffic from all instances (typically application servers) that use the source VPC security group.
Moving on next, we have the Database options, we choose the DB name and we have the default parameter group & the option group. We will talk about these in more detail in a few minutes.
The backups sections is pretty straightforward, we may choose automatic ones and there is a retention period from 7 to 35 days.
We can set a specific window for the backups and we may enable the logs that we wish:
Here, we have chosen the error logs and the slow query log.
And now we are ready to create our database:
Please note that once the MySQL RDS is ready, it is the only time we are able to see the connection details and specifically the password, so we need to take a note of this.
Once our database is ready for use, we will see a notification at the top of the page:
Now, remember the default-group that we saw earlier while setting our MySQL RDS? Basically, a DB parameter group acts as a container for engine configuration values that are applied to one or more DB instances.
If you create a DB instance without specifying a DB parameter group, the DB instance uses a default DB parameter group. Each default DB parameter group contains database engine defaults and Amazon RDS system defaults based on the engine, compute class, and allocated storage of the instance. You can’t modify the parameter settings of a default parameter group.
So, if we try to edit the default parameter group, we will receive the following error:
Therefore, if we wish to edit and configure the parameter values, we will need to create a new parameter group and assign it to our database.
Pretty straightforward, we name our parameter group and add a description if we wish:
And we are able to edit it and save the changes:
To apply it on our MySQL instance, we need to modify and set the corresponding parameter group:
We may also modify our backup options and set a window that we didn’t do while creating the MySQL:
On the Events tab, we may see all the actions that have been performed on our instance:
We are also able to check the Recommendations regarding the performance and our queries if we have the enhanced monitoring is enabled:
And last but not least, on the Actions we may create a replica :
To sum up, this is how to set up a MySQL RDS and the options that we have.
Conclusion
We have seen MySQL PaaS on the top Cloud Providers so far and there is definitely much more to see on that department, so stay tuned!
I don’t need a hard disk in my computer if I can get to the server faster… carrying around these non-connected computers is byzantine by comparison.
If you are dealing with data, and you most probably are if you are reading this, one of your biggest fears would be not to be able to retrieve them. In a world where data actually surround us, it is critical to be able to retrieve them fast and with the best consistency.
Thus, it is always a good idea to have high availability settings in place to avoid loosing your data.
However, most of the times, we may wish or we may need to save the database and our data, and be a DBA-hero. Not an easy task, and it may be smoother to just perform a backup-restore. Sadly, this is not always the case.
So, this is what we will be facing in this article, we are going to see what to do when there is a data corruption in MySQL and the steps we need to perform to try saving our database.
Before any task or operation, it is always a good idea to copy the already existing data at an OS level:
First, let’s stop MySQL:
1
systemctl stop mysqld
And then:
1
cp-r/datadir//backup_db
Before attempting to bring back up MySQL, it is a good idea to execute an innochecksum to see the status of your database or of a specific table.
A few words on innochecksum, in case you are not familiar with it:
innochecksum prints checksums for InnoDB files. This tool reads an InnoDB tablespace file, calculates the checksum for each page, compares the calculated checksum to the stored checksum, and reports mismatches, which indicate damaged pages. It was originally developed to speed up verifying the integrity of tablespace files after power outages but can also be used after file copies. Because checksum mismatches cause InnoDB to deliberately shut down a running server, it may be preferable to use this tool rather than waiting for an in-production server to encounter the damaged pages.
So, in order to check a table, you may use a simple command like:
1
innochecksum table.ibd--log=/tmp/log.txt
The status will be logged on file /tmp/log.txt, if the file doesn’t exist innochecksum can do this for you. And, the output will be something like the following:
page::41354; log sequence number:first = ; second = Page:: uncorrupted
Innochecksum offers a variety of options, the most useful is to be able to choose a starting or an ending page, to save time:
Please be aware that innochecksum can be executed only on a Database that is not running. Otherwise, you are going to receive a lock error:
1
2
Error:Unable tolock file::table.ibd
fcntl:Resource temporarily unavailable
Once you have checked the tables and discovered the problematic table or tables, it is time to attempt to startup the server and see what we can do. Very important, on my.cnf it is imperative to add the following:
1
2
innodb_force_recovery=1
log_error_verbosity=3
log_error_verbosity: The log_error_verbosity system variable specifies the verbosity for handling events intended for the error log. I always prefer to have this set as it is very helpful to identify various issues and at the same time to be able to check the progress on the error_log file.
innodb_force_recovery: The crash recovery mode, typically only changed in serious troubleshooting situations. Possible values are from 0 to 6. If server comes up with a value of 3 and below then the data can be recovered, most probably. Always start with 1 and increase one by one, if the server is not coming up with your previous choice.
Now that we have these settings on our configuration, it is time to start-up the server and hope for the best:
1
systemctl start mysqld
If we are lucky and the server does manage to come up with a value among 1 to 3, it is time to take a full dump of the table(s) that are corrupted. If you are using 8.0.22 version and above, then you may take the dump via MySQL Shell:
1
MySQL localhost:33060+ssl JS>util.dumpTables("schema",["table"],"/tmp/dump_table");
Once the backup is completed successfully, it is time to drop the corrupted table:
1
mysql–uroot–p–execute=”DROP TABLE schema.table”
And then restore it from the dump we have taken:
1
mysql–uroot–p<table.sql
Once the restoration is done, we need to restart the MySQL server, but this time without the innodb_force_recovery parameter. Simply, edit your configuration file and mark it as a comment:
1
#innodb_force_recovery=1
Hopefully, this will work and our database will be up and running with all its’ data intact! Unfortunately, this is not always the case and the database won’t be able to come up for us to retrieve our files. In situations like these, we need to restore from an existing backup or from a replica, if they are available.
To avoid facing serious trouble with your data, condider having in place one (or better yet all) of the following:
Full backups ( at a daily,weekly,monthly base)
Incremental backups
Binlog backups
In sync replicas
Innodb cluster
Additionally, in cases of a system crash, it is always helpful to set certain parameters to maintain data consistency. Let’s view the most basic ones:
innodb_flush_log_at_trx_commit = 1 :
Controls the balance between strict ACID compliance for commit operations and higher performance that is possible when commit-related I/O operations are rearranged and done in batches. You can achieve better performance by changing the default value but then you can lose transactions in a crash. The default setting of 1 is required for full ACID compliance. Logs are written and flushed to disk at each transaction commit.
sync_binlog = 1 :
Enables synchronization of the binary log to disk before transactions are committed. This is the safest setting but can have a negative impact on performance due to the increased number of disk writes. In the event of a power failure or operating system crash, transactions that are missing from the binary log are only in a prepared state. This permits the automatic recovery routine to roll back the transactions, which guarantees that no transaction is lost from the binary log.
innodb_doublewrite = 1 :
The doublewrite buffer is a storage area where InnoDB writes pages flushed from the buffer pool before writing the pages to their proper positions in the InnoDB data files. If there is an operating system, storage subsystem, or unexpected mysqld process exit in the middle of a page write, InnoDB can find a good copy of the page from the doublewrite buffer during crash recovery.
relay_log_recovery = 1 :
If enabled, this variable enables automatic relay log recovery immediately following server startup. The recovery process creates a new relay log file, initializes the SQL thread position to this new relay log, and initializes the I/O thread to the SQL thread position. Reading of the relay log from the source then continues.
This global variable is read-only at runtime. Its value can be set with the –relay-log-recovery option at replica server startup, which should be used following an unexpected halt of a replica to ensure that no possibly corrupted relay logs are processed, and must be used in order to guarantee a crash-safe replica.
relay_log_purge = 1 :
Disabling purging of relay logs when enabling the –relay-log-recovery option risks data consistency and is therefore not crash-safe.
innodb_directories :
Another parameter that can assist with corrupted and crash-recovery tablespaces is innodb_directories. Tablespace discovery during crash recovery relies on the innodb_directories setting to identify tablespaces referenced in the redo logs. Innodb_directories can be defined on the start-up and they can be used when moving your tablespaces while the server is offline.
At this point, it is wise to mention that the recovery time is also depended on the innodb_log_file_size.
Generally, the combined size of the log files should be large enough that the server can smooth out peaks and troughs in workload activity, which often means that there is enough redo log space to handle more than an hour of write activity. The larger the value, the less checkpoint flush activity is required in the buffer pool, saving disk I/O.
Larger log files also make crash recovery slower.
Conclusion
MySQL does offer a stability, however it is really hard to avoid a corruption or to not face having your server crashing at some point. If safety measurements are in place, take a deep breath and dive in to save your data.
When you crash and burn, you have to pick yourself up and go on and hope to make up for it.
MySQL Replication is a very common topology, widely used in various architecture. People use it, among others, for High Availability, Read Scalability or Geographic Redundancy.
Another use case is to use MySQL Replication to seamlessly integrate a newer version of the server in your architecture. Let’s say you are running MySQL 5.7 then you can easily setup a 8.0 instance as a replica of your 5.7.
MySQL Replication topology from MySQL Enterprise Monitor
Plan
How to proceed?
Let’s split the problem in sub-parts:
Create a compute & a MDS instances
Create on the source a dedicated user to the replication
Dump the MySQL instance running on the OCI compute for being migrated to MDS
Load the dump in MDS
Create a replication channel on MDS (from OCI to MDS)
Requirement
Before starting, please verify that your source follows the requirements.
Current limitations of MySQL Database Service Inbound Replication
Only Row-based replication supported
Only GTID-based replication is supported
Multi-source replication is not supported
Replication filters are not supported
Changes to the mysql schema are not replicated and cause replication to stop
Source and Replica must run with the same lower_case_table_names to avoid data transfer problems arising from letter case of database or table names in cross-platform topology
The inbound applier runs under the privileges of the DB System’s admin user
You also need to have a running compute instance attached to a public subnet on the same VCN as the MySQL DB System : Creating a Compute Instance
Indeed MySQL 5.7 and MySQL Shell 8.0 are installed on this compute instance : Installing MySQL Shell on Linux. (obviously it works also if you are on Windows).
Create on the source a dedicated user to the replication
Assuming the prerequisites above are fine and the security lists are updated, we can now proceed.
Connect by SSH, to the compute instance where MySQL 5.7 is installed:
1
ssh-iid_rsa ubuntu@123.45.678.90
Note: If you are using Ubuntu (the case here) the user is ubuntu. For others GNU Linux systems, the user should be opc.
Connect to the MySQL 5.7 instance using MySQL Shell 8.0.23:
1
2
$mysqlsh--version
mysqlsh Ver8.0.23forLinux on x86_64-forMySQL8.0.23(MySQL Community Server(GPL))
1
$mysqlsh root@localhost:3306--sql
1
2
3
4
5
6
SQL>SELECT version();
+------------+
|version()|
+------------+
|5.7.33-log|
+------------+
Then create the replication user with his relevant privileges:
1
2
3
4
SQL>CREATE USER rplAdmin@'10.0.1.%'IDENTIFIED WITH'mysql_native_password'BY'Repl1234c@'REQUIRE SSL;
SQL>GRANT REPLICATION SLAVE on *.*torplAdmin@'10.0.1.%';
We can check the user:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
SQL>SELECT user,host,plugin FROM mysql.user WHERE user='rplAdmin';
Dump the MySQL instance running on OCI for being migrated to MDS
Now we want to put the current MySQL 5.7 data on the MDS instance.
In this context the best tools to do this job are MySQL Shell utilities. We will use the instance dump utility.
The backup output can be stored on a Object Storage Bucket – I will write an article on that. But today, let’s keep it simple. We’ll store the backup locally on the compute instance.
This strategy makes sense if you have enough free disk space on the compute. Also it will be faster to restore the data on MDS with this strategy than having the dump on the object storage.
A good practice when attempting to import data from MySQL 5.7 into MDS is to use MySQL Shell’s upgrade checker utility util.checkForServerUpgrade() to check the schemas and if needed proactively fix any compatibility issues identified.
1
2
3
SQL>\JS
JS>util.checkForServerUpgrade()
We’re good to go, so let’s create the backup directory:
1
$mkdir-p/home/ubuntu/backup/
Then dump our MySQL 5.7 instance using util.dumpInstance() :