AWS Storage Gateway

intro

 

Using Amazon AWS Storage Gateway :
AWS Storage Gateway is a service that connects an on-premises software appliance with cloud-based storage in the form of an iSCSI devices.
Storage Gateways Documentation – assume a scenario of Data Centre VM where we use S3 for storage solution.

There are 2 type of iSCSI devices:
1. Gateway-Cached Volume Solution :  create your storage volumes and mount them as iSCSI devices from your on-premises application servers – Data is stored on Amazon S3 and frequently accessed data is stored on the on-premises storage hardware.

2. Gateway-Stored Volume Solution : store all your data locally in storage volumes on your on-premises storage hardware. The gateway periodically takes snapshots as incremental backups and stores them in Amazon S3.

[Youtube Video on StorageGateway walkthrough on Windows]

AWS Storage Gateway uses two different hosting environments: VMware virtualization environment and an Amazon Elastic Compute Cloud (Amazon EC2) environment.

VMware virtualization environment
Amazon Elastic Compute Cloud (Amazon EC2) environment

 

working

Setting Up AWS Storage Gateway

Steps :
Click Storage Gateway from the AWS Consol
Click – Deploy a New Gateway on Amazon EC2
Click – Lauch Gateway AMI
Click – Select
Select – Lauch with EC2 Console
Click Accept Terms
One of the following AMI can be choosen

Region    ID
US East (Virginia)                               ami-29f27a40
US West (Oregon)                               ami-4847cc78
US West (Northern California)          ami-36b39373
EU West (Ireland)                               ami-04393670
Asia Pacific (Singapore)                     ami-4a94d618
Asia Pacific (Tokyo)                            ami-d941fbd8
South America (Sao Paulo)                ami-6526fe78

The instance type must be at least a Standard XL (m1.large) or the instance will not launch.

With default setup – 2 more EBS also needs to be added,  one for cache storage and one for upload buffer.

NOTE: For a gateway-cached setup, you can add up to 18 TB of storage comprised of up to 2 TB allocated to upload buffer and up to 16 TB allocated to cache storage.

Equip with Elasticache

Memcache ? and Its Facts !!

Fotolog, as they themselves point out, is probably the largest site nobody has ever heard of, pulling in more page views than even Flickr.
Fotolog has 51 instances of memcached on 21 servers with 175G in use and 254G available.

Memached is: A high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.The magic is that none of the memcached servers need know about each other. To scale up you just add more servers and the key hashing algorithm makes it all work out right. Memcached is not redundant, has no failover, and has no authentication. It’s simple server for storing and getting data, the complex bits must be implemented by applications.

Case Study

Amazon ElastiCache supports nodes with cache sizes ranging from 6 to 67 GB. A DNS name is assigned to each Cache Node when it is created.

Memcache Hasing

AWS Blog on Elasticache

Understanding Elasticache Internals

Why is memcached not recommended for sessions? Everyone does it!
If a session disappears, often the user is logged out. If a portion of a cache disappears, either due to a hardware crash or a simple software upgrade, it should not cause your users noticable pain. This overly wordy post explains alternatives. Memcached can often be used to reduce IO requirements to very very little, which means you may continue to use your existing relational database for the things it’s good at.

Like keeping your users from being knocked off your site.
In detail why we dont use memcache for sessions – http://dormando.livejournal.com/495593.html

What about the MySQL query cache?
The MySQL query cache can be a useful start for small sites. Unfortunately it uses many global locks on the mysql database, so enabling it can throttle you down. It also caches queries per table, and has to expire the entire cache related to a table when it changes, at all. If your site is fairly static this can work out fine, but when your tables start changing with any frequency this immediately falls over.

Memory is also limited, as it requires using a chunk of what’s directly on your database.

Can using memcached make my application slower?
Yes, absolutely. If your DB queries are all fast, your website is fast, adding memcached might not make it faster.

Memcache FAQ on Google Code

Elasticache Setup

launch-amazon-elasticache-cluster-1

Name: This is the Cache Identifier name and should be unique for an Amazon EC2 region(per account).

Node Type: Cache Capacity type with Memory and CPU. If you want 20 GB of distributed Cache you can choose either 3 Cache.M1.Large or 2 Cache.M1.Xlarge Node types. Usually users prefer Node types with more memory rather than High CPU node types (Not sure what kind of workload needs cache.c1.Xlarge capacity on memory hungry applications). Recently AWS has introduced Cache.M3.Class Node type which fits Memcached kind of use cases very well. The Node type cannot be modified after creating an Amazon ElastiCache Cluster, so please plan your base capacity in advance with some thought. To know more about ElastiCache deployment strategies refer URL: http://harish11g.blogspot.in/2012/11/amazon-elasticache-memcached-ec2.html

Number of Nodes: Number of Node types you want the Amazon ElastiCache Cluster to launch. There is a limit to Cache nodes you can launch per account. Please increase this limit using ElastiCache Limit Increase Request form.

Version: Memcached 1.4.5

Cache Port: The default port 11211 in which the Amazon ElastiCache node accepts connections.

Preferred Zone: It is the Preferred Amazon EC2 Availability Zone in which you want to launch the Amazon ElastiCache Cluster. It is recommended to keep the Web/App EC2 instances and Amazon ElastiCache nodes on same Availability zone for low latency processing. In case multi-AZ is applicable your architecture, standard Amazon EC2 Regional Data Transfer charges of $0.01 per GB in/out apply when transferring data between an Amazon EC2 instance and an Amazon ElastiCache Node in different Availability Zones of the same Region, you are only charged for the Data Transfer in or out of the Amazon EC2 instance.

The Cache security group will allow request access between your EC2 instances and ElastiCache Nodes. The Security group of your EC2 instances should be added in this ElastiCache Security group for opening the access. This setting applies to all the existing and new cache nodes inside the Amazon ElastiCache cluster. You can either create a new Amazon ElastiCache Security group or make changes in the default security group as well. In case you have Multitude of cache clusters with variety of EC2 tiers accessing them on various workflows I would strongly recommend creating your own cache security groups as best practice. Currently Amazon ElastiCache cannot be accessed outside Amazon EC2 (It does not make sense unless you have Gigabit NW to make productive use of Cache with low latency). Also in future, we can expect Amazon ElastiCache to work inside Amazon Virtual Private Cloud (VPC) network as well.
Cache Parameter Group: You can either create a new parameter group or use the default memcached parameter group. AWS provided default will suffice for most use cases. The parameters will be applied to all the cache nodes in the Amazon ElastiCache cluster.

what is an elasticache cluster node?
A cache node is the smallest building block of an Amazon ElastiCache deployment. It is a fixed-size chunk of secure, network-attached RAM. Each cache node runs an instance of the Memcached service, and has its own DNS name and port. Multiple types of cache nodes are supported, each with varying amounts of associated memory.

source

What is Configuration Endpoint ?

configurations_endpoint

To use Amazon ElastiCache you have to set up a cache cluster. A cache cluster is a collection of cache nodes. You choose the number and the type of nodes to match the performance needs of your application. In the past, if you changed the nodes in your cache cluster (for example, by adding a new node), you would have to update the list of node endpoints manually. Typically, updating the list of node endpoints involves reinitializing the client by shutting down and restarting the application, which can result in downtime (depending on how the client application is architected). With the launch of Auto Discovery, this complexity has been eliminated.

All ElastiCache clusters (new and existing!) now include a unique Configuration Endpoint, which is a DNS Record that is valid for the lifetime of the cluster. This DNS Record contains the DNS names of each of the nodes that belong to the cluster. Amazon ElastiCache will ensure that the Configuration Endpoint always points to at least one such “target” node. A query to the target node then returns endpoints for all the nodes in the cluster. To be a bit more specific, running a query means sending the config command to the target node. We implemented this command as an extension to the Memcached ASCII protocol (read about Adding Auto-Discovery to Your Client Library for more information).

You can then connect to the cluster nodes just as before and use the Memcached protocol commands such as get, set, incr, and decr. The Configuration Endpoint is accessible programmatically through the ElastiCache API, via the command line tools, and from the ElastiCache Console.

To take advantage of Auto Discovery, you will need to use a Memcached client library that is able to use this new feature. To get started, you can use the ElastiCache Cluster Client, which takes the popular SpyMemcached client and adds Auto Discovery functionality. We have a Java client available now (view source), which can be downloaded from the ElastiCache Console:

We plan to add Auto Discovery support to other popular Memcached client libraries over time; a PHP client is already in the works.

ElastiCache remains 100% Memcached-compatible so you can keep using your existing Memcached client libraries with new and existing clusters, but to take advantage of Auto Discovery you must use an Auto Discovery-capable client.

Do we need to install memcached on server?
You don’t need to have memcache installed, only the memcache pecl module in your php installation. Elasticache is a memcached server, nothing more nothing less. As long as you have the memcache pecl module installed in your php, the memcache option will be available on the W3TC dropdowns.

Installing memcache on PHP
You can install the pecl module with:
#pecl install memcache
OR on an apt based system like debian or ubuntu
#apt-get install php5-memcached

Installing Elasticache Cluster Client Module
source
#apt-get update
#apt-get install gcc g++ php5 php-pear

Download Amazon_Elasticache_cluster_client for PHP version from the Elasticache Management Console.

With root/sudo permission, add a new file memcached.ini under the directory /etc/php5/conf.d, and insert “extension=<absolute path to amazon-elasticache-cluster-client.so>” in it.
#echo “extension=<absolute path to amazon-elasticache-cluster-client.so>” > /etc/php5/conf.d/memcached.ini

Test it by running the below contents inside a .php file
<?php
error_reporting(E_ALL & ~E_NOTICE);

$mc = new Memcached();
$mc->addServer(“localhost”, 11211);

$mc->set(“foo”, “Hello!”);
$mc->set(“bar”, “Memcached…”);

$arr = array(
$mc->get(“foo”),
$mc->get(“bar”)
);
#uncomment below line if detailed results are necessary
#var_dump($arr);
echo “<br> foo = “.$mc->get(“foo”);
echo “<br> bar = “.$mc->get(“bar”);
?>

Observations
Here $mc is an object of the class Memcached() which is defined inside the php module “php5-memcached”.Hence the function addServer() also worked well where we add the endpoint of our Elasticache Cluster Nodes Endpoint/Configuration Endpoint and port number.
Output should be
foo = Hello!
bar = Memcached…
Which means that the memcache module is online.

Developer’s Part in Elasticache Implementation
Internal Working of the Elasticache and Application
Different Elasticache Scenarios

Convey this to the PHP Guy and Provide him with the Configuration Endpoint :
ElastiCache supports Auto Discovery—the ability for client programs to automatically identify all of the nodes in a cache cluster, and to initiate and maintain connections to all of these nodes. With Auto Discovery, your application does not need to manually connect to individual cache nodes; instead, your application connects to a configuration endpoint. The configuration endpoint DNS entry contains the CNAME entries for each of the cache node endpoints; thus, by connecting to the configuration endpoint, you application immediately “knows” about all of the nodes in the cluster and can connect to all of them. You do not need to hardcode the individual cache node endpoints in your application.

Choosing a Cache Node Type and the Number of Cache Nodes

The total memory capacity of your cache cluster is calculated by multiplying the number of cache nodes in the cluster by the capacity of each Node. The capacity of each cache node is based on the cache node type.
The number of cache nodes in the cache cluster is a key factor in the availability of your cache cluster. The failure of a single cache node can have an impact on the availability of your application and the load on your backend database while ElastiCache provisions a replacement for the failed cache node. The scale of this availability impact can be reduced by spreading your memory and compute capacity over a larger number of cache nodes, each with smaller capacity, rather than a fewer number of high capacity nodes.

In a scenario where you want to have 20GB of cache memory, you can set it up in one of the following ways:

Use 15 cache.m1.small cache nodes with 1.3 GB of memory each = 19.5 GB

Use 3 cache.m1.large cache nodes with 7.1 GB of memory each = 21.3 GB

Use 3 cache.c1.xlarge cache nodes with 6.6 GB of memory each = 19.8 GB

These options provide you with similar memory capacity, but different computational capacity for your cache cluster.

If you’re unsure about how much capacity you need, we recommend starting with one cache.m1.small cache node type and monitoring the memory usage, CPU utilization and Cache Hit Rate with the ElastiCache metrics that are published to Amazon CloudWatch.

If your cache cluster does not have the desired hit rate, you can easily add more nodes, thereby increasing the total available memory in your cache cluster. You will need to obtain an updated endpoint list from the ElastiCache CLI, API or AWS Management Console, and configure your clients to use the additional node(s).

If your cache cluster turns out to be bound by CPU but has sufficient hit rate, then try setting up a new cluster with a different cache node type.

ElastiCache supports adding or removing cache nodes from an existing cache cluster using the AWS Management Console, the API, and the command line tools, allowing you to increase both memory and compute capacity of the cluster at any time.

Note:ElastiCache does not currently support dynamically changing the cache node type for a cache cluster after it has been created. If you wish to change the Node Type of a cache cluster, you will need to set up a new cache cluster with the desired Node Type, and migrate your application to that cache cluster.

Configure PHP to use Elasticache for Sesssions

#vim /etc/php5/conf.d/memcached.ini
extension=/root/ecache/amazon-elasticache-cluster-client.so
Editing php.ini file
#vim /etc/php5/apache2/php.ini
session.save_handler = memcached
#We mention the configuration endpoint of the elasticache
session.save_path = “test.nugnqi.cfg.use1.cache.amazonaws.com:11211″
#extension is already inside the conf.d/memcache.ini file hence this file may not be necessary
extension=/root/ecache/amazon-elasticache-cluster-client.so
#The ecache dir is the Elasticache Client software that is downloaded for appropriate PHP version to be in used in place of Memcache extension of PHP.

Install phpMemcachedAdmin

wget “http://phpmemcacheadmin.googlecode.com/files/phpMemcachedAdmin-1.2.2-r262.tar.gz&#8221;
tar xvzf phpMemcachedAdmin-1.2.2-r262.tar.gz   (inside apache documentroot)
chmod +r *
chmod 0777 Config/Memcache.php

s3cmd Elaborated…

Use –rr option (reduced redundancy) for every put and sync commands !!!. 
Use –bucket-location option to mention nearest geographical location to avoid latency.

To view contents inside a bucket
#s3cmd ls s3://bucketname

To copy/sync a directory into a bucket
#s3cmd sync Desktop/check s3://bucket_name

To view all contents of all buckets one level down (only non empty buckets)
#s3cmd la -H

To sync contents of a local dir in a buckter under an existing directory (s3 object)
#s3cmd sync Desktop/checkunni/ s3://writingz/check/

To sync remote s3 contents to a local directory
#s3cmd sync s3://writingz/check/ Desktop/checkunni/

To sync contents of a local dir in a bucket under a new directory name
#s3cmd sync Desktop/checkunni/ s3://homie/newname/
Here newname directory is created on the fly and files of checkunni are copied inside s3://homie/newname

Copy a non-empty directory (on s3) from one bucket to another bucket
#s3cmd -r cp s3://homie/newname s3://writingz/

Copy a non-empty directory (on s3) from one bucket to another bucket under a new name
#s3cmd -r cp s3://homie/newname s3://writingz/newname2/

To find the size of a bucket/directory
#s3cmd du -H s3://writingz

To download only a single file
#s3cmd get s3://homie/dirname/filename .

To download a remote directory locally.
#s3cmd get -rf s3://writingz/checkunni .
use a / (forward slash) after checkunni to download only the files in it.

To upload a single file
#s3cmd put PSY.mp3 s3://homie/newname/

To upload a local dir to bucket
#s3cmd put -rf s3test s3://homie/newname/

Delete a file
#s3cmd del s3://writingz/abc.jpg

Delete a directory
#s3cmd del -rf s3://writingz/check/

Move a file (can also be used for rename with files only)
#s3cmd mv s3://writingz/abc.png s3://haye/

Move a directory to another bucket 
#s3cmd mv -rf s3://writingz/newname2 s3://haye/

Know the s3cmd version
#s3cmd –version

Make a file public using
#s3cmd put –acl-public hangover3.jpg s3://viewzz/abc.jpg

Make a file private using
#s3cmd setacl –acl-private s3://viewzz/hangover3.jpg

Set all files in a bucket to public/private
#s3cmd setacl –acl-public -r s3://writingz/

If an md5 checksum is need to verify files integrity use
#sudo s3cmd info s3://viewzz/hangover3.jpg (an amazon s3 object)
#md5sum hangover3.jpg (locally downloaded file from s3)
and compare the checksum value.

To delete a bucket (bucket has to be empty use s3cmd del – to delete all files)
#s3cmd rb s3://logix.cz-test (use -f option if bucket is non-empty)

Get various information about Buckets or Files
#s3cmd info s3://BUCKET[/OBJECT]

Other useful options
–delete-removed Delete remote objects with no corresponding local file
[sync]

–no-delete-removed Don’t delete remote objects.
–skip-existing Skip over files that exist at the destination (only
for [get] and [sync] commands).

–continue Continue getting a partially downloaded file (only for
[get] command).

–reduced-redundancy, –rr
Store object with ‘Reduced redundancy’. Lower per-GB
price. [put, cp, mv, sync]

–acl-public Store objects with ACL allowing read for anyone.

–acl-private Store objects with default ACL allowing access for you

only.

–bucket-location=BUCKET_LOCATION Datacentre to create bucket in. Eg :  ap-northeast-1  (Tokyo)

The ACL (Access Control List) of a file can be set at the time of upload using –acl-public or –acl-private options with ‘s3cmd put’ or s3cmd sync’ commands (see below).

Alternatively the ACL can be altered for existing remote files with ‘s3cmd setacl –acl-public’ (or –acl-private) command.

Additional Links on 

AWS EC2 Internal Security Structure

Reference

An insight into internal structure of EC2.

The Hypervisor
Amazon EC2 currently utilizes a highly customized version of the Xen hypervisor, taking advantage of paravirtualization
(in the case of Linux guests). Because paravirtualized guests rely on the hypervisor to provide support for operations that
normally require privileged access, the guest OS has no elevated access to the CPU. The CPU provides four separate
privilege modes: 0-3, called rings. Ring 0 is the most privileged and 3 the least. The host OS executes in Ring 0. However,
rather than executing in Ring 0 as most operating systems do, the guest OS runs in a lesser-privileged Ring 1 and
applications in the least privileged Ring 3. This explicit virtualization of the physical resources leads to a clear separation
between guest and hypervisor, resulting in additional security separation between the two.

Paravirtualization: In computing, paravirtualization is a virtualization technique that presents a software interface to
virtual machines that is similar but not identical to that of the underlying hardware.

Instance Isolation
Different instances running on the same physical machine are isolated from each other via the Xen hypervisor. Amazon
is active in the Xen community, which provides awareness of the latest developments. In addition, the AWS firewall
resides within the hypervisor layer, between the physical network interface and the instance’s virtual interface. All
packets must pass through this layer, thus an instance’s neighbors have no more access to that instance than any other
host on the Internet and can be treated as if they are on separate physical hosts. The physical RAM is separated using
similar mechanisms.

awssecurity

Customer instances have no access to raw disk devices, but instead are presented with virtualized disks. The AWS proprietary disk virtualization layer automatically resets every block of storage used by the customer, so that one customer’s data are never unintentionally exposed to another. AWS recommends customers further protect their data using appropriate means. One common solution is to run an encrypted file system on top of the virtualized disk device.

Guest Operating System: Virtual instances are completely controlled by you, the customer. You have full root access or administrative control over accounts, services, and applications. AWS does not have any access rights to your instances or the guest OS. AWS recommends a base set of security best practices to include disabling password-only access to your guests, and utilizing some form of multi-factor authentication to gain access to your instances (or at a minimum certificate-based SSH Version 2 access). Additionally, you should employ a privilege escalation mechanism with logging on a per-user basis. For example, if the guest OS is Linux, after hardening your instance you should utilize certificatebased SSHv2 to access the virtual instance, disable remote root login, use command-line logging, and use ‘sudo’ for privilege escalation. You should generate your own key pairs in order to guarantee that they are unique, and not shared with other customers or with AWS.

You also control the updating and patching of your guest OS, including security updates. Amazon-provided Windows and Linux-based AMIs are updated regularly with the latest patches, so if you do not need to preserve data or customizations on your running Amazon AMI instances, you can simply relaunch new instances with the latest updated AMI. In addition, updates are provided for the Amazon Linux AMI via the Amazon Linux yum repositories.

Well-informed traffic management and security design are still required on a perinstance basis. AWS further encourages you to apply additional per-instance filters with host-based firewalls such as  IPtables or the Windows Firewall and VPNs. This can restrict both inbound and outbound traffic.

Why take snapshots if EBS is storing data redundantly?
Well-informed traffic management and security design are still required on a perinstance basis. AWS further encourages you to apply additional per-instance filters with host-based firewalls such as IPtables or the Windows Firewall and VPNs. This can restrict both inbound and outbound traffic.

Security Features on S3
1.Identity and Access Management (IAM) Policies.
2.Access Control Lists (ACLs).
3.Bucket Policies.
4.Encryption/Decryption
Server-side encryption (SSE): An option for S3 storage for automatically encrypting data at rest. With Amazon S3 SSE,
customers can encrypt data on upload simply by adding an additional request header when writing the object.
Decryption happens automatically when data is retrieved.

Security Features on RDS
Amazon RDS has multiple features that enhance reliability for critical production databases, including DB security
groups, permissions, SSL connections, automated backups, DB snapshots, and multi-AZ deployments. DB instances can
also be deployed in an Amazon VPC for additional network isolation.

How to Set Up Master Slave Replication in MySQL

About MySQL replication


MySQL replication is a process that allows you to easily maintain multiple copies of a MySQL data by having them copied automatically from a master to a slave database. This can helpful for many reasons including facilating a backup for the data,a way to analyze it without using the main database, or simply as a means to scale out.

This tutorial will cover a very simple example of mysql replication—one master will send information to a single slave. For the process to work you will need two IP addresses: one of the master server and and one of the slave.

This tutorial will use the following IP addresses:

12.34.56.789- Master Database

12.23.34.456- Slave Database

Setup


This article assumes that you have user with sudo privileges and have MySQL installed. If you do not have mysql, you can install it with this command:

sudo apt-get install mysql-server mysql-client

 

Step One—Configure the Master Database


Open up the mysql configuration file on the master server.

sudo nano /etc/mysql/my.cnf

Once inside that file, we need to make a few changes.

The first step is to find the section that looks like this, binding the server to the local host:

bind-address            = 127.0.0.1

Replace the standard IP address with the IP address of server.

bind-address            = 12.34.56.789

The next configuration change refers to the server-id, located in the [mysqld] section. You can choose any number for this spot (it may just be easier to start with 1), but the number must be unique and cannot match any other server-id in your replication group. I’m going to go ahead and call this one 1. Make sure this line is uncommented.

server-id               = 1

Move on to the log_bin line. This is where the real details of the replication are kept. The slave is going to copy all of the changes that are registered in the log. For this step we simply need to uncomment the line that refers to log_bin:

log_bin                 = /var/log/mysql/mysql-bin.log

Finally, we need to designate the database that will be replicated on the slave server. You can include more than one database by repeating this line for all of the databases you will need.

binlog_do_db            = newdatabase

After you make all of the changes, go ahead and save and exit out of the configuration file.

Refresh MySQL.

sudo service mysql restart

The next steps will take place in the MySQL shell, itself.

Open up the MySQL shell.

mysql -u root -p

We need to grant privileges to the slave. You can use this line to name your slave and set up their password. The command should be in this format:

GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'%' IDENTIFIED BY 'password';

Follow up with:

FLUSH PRIVILEGES;

The next part is a bit finicky. To accomplish the task you will need to open a new window or tab in addition to the one that you are already using a few steps down the line.

In your current tab switch to “newdatabase”.

USE newdatabase;

Following that, lock the database to prevent any new changes:

FLUSH TABLES WITH READ LOCK;

Then type in:

SHOW MASTER STATUS;

You will see a table that should look something like this:

mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000001 |      107 | newdatabase  |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

This is the position from which the slave database will start replicating. Record these numbers, they will come in useful later.

If you make any new changes in the same window, the database will automatically unlock. For this reason, you should open the new tab or window and continue with the next steps there.

Proceeding the with the database still locked, export your database using mysqldump in the new window (make sure you are typing this command in the bash shell, not in MySQL).

mysqldump -u root -p --opt newdatabase > newdatabase.sql

Now, returning to your your original window, unlock the databases (making them writeable again). Finish up by exiting the shell.

UNLOCK TABLES;
QUIT;

Now you are all done with the configuration of the the master database.

Step Two—Configure the Slave Database


Once you have configured the master database. You can put it aside for a while, and we will now begin to configure the slave database.

Log into your slave server, open up the MySQL shell and create the new database that you will be replicating from the master (then exit):

CREATE DATABASE newdatabase;
EXIT;

Import the database that you previously exported from the master database.

mysql -u root -p newdatabase < /path/to/newdatabase.sql

Now we need to configure the slave configuration in the same way as we did the master:

sudo nano /etc/mysql/my.cnf

We have to make sure that we have a few things set up in this configuration. The first is the server-id. This number, as mentioned before needs to be unique. Since it is set on the default (still 1), be sure to change it’s something different.

server-id               = 2

Following that, make sure that your have the following three criteria appropriately filled out:

relay-log               = /var/log/mysql/mysql-relay-bin.log
log_bin                 = /var/log/mysql/mysql-bin.log
binlog_do_db            = newdatabase

You will need to add in the relay-log line: it is not there by default.

Once you have made all of the necessary changes, save and exit out of the slave configuration file.

Restart MySQL once again:

sudo service mysql restart

The next step is to enable the replication from within the MySQL shell.

Open up the the MySQL shell once again and type in the following details, replacing the values to match your information:

CHANGE MASTER TO MASTER_HOST='12.34.56.789',MASTER_USER='slave_user', MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=  107;

This command accomplishes several things at the same time:

    1. It designates the current server as the slave of our master server.

 

    1. It provides the server the correct login credentials

 

  1. Last of all, it lets the slave server know where to start replicating from; the master log file and log position come from the numbers we wrote down previously.

With that—you have configured a master and slave server. Activate the slave server:

START SLAVE;

You be able to see the details of the slave replication by typing in this command. The \G rearranges the text to make it more readable.

SHOW SLAVE STATUS\G

If there is an issue in connecting, you can try starting slave with a command to skip over it:

SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; SLAVE START;

All done.

How to Install and Use Memcache on Ubuntu 12.04

About Memcache


Memcache is a system that works to speed up virtual private servers by caching server information. The program allows you to allocate a specific amount of the server ram toward caching recently queried data for a certain amount of time. Once the data is requested again, memcache speeds up the process of retrieving it by displaying the cached information instead of generating the result from the database.

Setup

The steps in this tutorial require the user to have root privileges. You can see how to set that up in the Basic Users Tutorial.

Before starting off, it’s a good idea to update apt-get to make sure that all of the packages we download to the VPS are up to date.

sudo apt-get update

Additionally, you should have MySQL and PHP installed on the virtual server.

sudo apt-get install mysql-server php5-mysql php5 php5-memcache

 

Install Memcache


Installing memcache takes several steps.

To start, install memcached via apt-get.

sudo apt-get install memcached

The next step is to install php-pear, the repository that stores memcache.

sudo apt-get install php-pear

If you do not have a compiler on your server, you can download build-essential in order to install memcache:

sudo apt-get install build-essential

Finally use PECL (PHP Extension Community Library) to install memcache:

sudo pecl install memcache

Say yes by pressing enter during the installation when you are asked if you would like to “Enable memcache session handler support? [yes] :”

Once you have completed the installation of memcache with PECL on the VPS, add memcached to memcache.ini:

sudo echo "extension=memcache.so" > sudo /etc/php5/conf.d/memcache.ini

Now you are ready to start using Memcache.

Confirm Memcache and See Stats


After Memcache is downloaded, you can check that it has been installed by searching for it:

ps aux | grep memcache

Additionally, you can see the memcache stats by typing:

 echo "stats settings" | nc localhost 11211

 

Step Three—How Memcache Works


Memcache works by redirecting code to first attempt to retrieve data from the cache before querying the server’s database. The cache populates by saving recently retrieved server data for a certain amount of time. By caching recently requested information, future queries do not have to go through the longer process of retrieving the information from a database and can, instead, access it through the cache.

The memcache page shows this abbreviated code on its homepage to summarize the memcache process:

function get_foo(foo_id)
    foo = memcached_get("foo:" . foo_id)
    return foo if defined foo

    foo = fetch_foo_from_database(foo_id)
    memcached_set("foo:" . foo_id, foo)
    return foo
end

A Simple Memcache Example


This section will set up a simple php script to use memcache for retrieving a single value originally found in a mysql table.

The following steps set up a mysql user who can access the appropriate database, create a table to query, and insert the one value that we will test in the new mysql table.

Log into mysql: mysql -u root -p and execute the following commands:

use test;

grant all on test.* to test@localhost identified by 'testing123';

create table example (id int, name varchar(30));

insert into example values (1, "new_data");

exit;

Once you have exited MySQL, create the memcache script file:

nano memtest.php

We are now going to build up the php script step by step (the entire script will be at the end of the section):

    • Start off by creating a new persistent connection with memcache, which runs on memcache’s default port, 11211.
      <?php
      $meminstance = new Memcache();
      $meminstance->pconnect('localhost', 11211);

 

    • The next step is to connect to the new mysql database with the user that we created earlier:
      mysql_connect("localhost", "test", "testing123") or die(mysql_error());
      mysql_select_db("test") or die(mysql_error());

 

    • After that, go ahead and create the query that we will pose to the server, as well as provide a key to identify that specific action:
      $query = "select id from example where name = 'new_data'";
      $querykey = "KEY" . md5($query);

 

  • The script first searches the cache for the answer to the query. If the result does not exist, the script reroutes the question to the original database. Once the query has been answered by the original database, the script stores the result in memcache, using the “set” command– which both saves it and allows the user to designate the number of seconds that it should remain in the cache (600 would save it in the cache for 10 minutes).

    When we run the script for the first time, it will inform us that the data was collected from the mysql database. However, as it does so, it stores the information in the cache, so that a second run of the script retrieves it from the cache and lets the user know.

    In 10 minutes the cache is emptied once more and running the script will make it access the database once again.

    $result = $meminstance->get($querykey);
    
    if (!$result) {
           $result = mysql_fetch_array(mysql_query("select id from example where name = 'new_data'")) or die('mysql error');
           $meminstance->set($querykey, $result, 0, 600);
    print "got result from mysql\n";
    return 0;
    }
    
    print "got result from memcached\n";
    return 0;
    
    ?>

Altogether the script looks like this:

<?php
$meminstance = new Memcache();
$meminstance->pconnect('localhost', 11211);

mysql_connect("localhost", "test", "testing123") or die(mysql_error());
mysql_select_db("test") or die(mysql_error());

$query = "select id from example where name = 'new_data'";
$querykey = "KEY" . md5($query);

$result = $meminstance->get($querykey);

if (!$result) {
       $result = mysql_fetch_array(mysql_query("select id from example where name = 'new_data'")) or die('mysql error');
       $meminstance->set($querykey, $result, 0, 600);
print "got result from mysql\n";
return 0;
}

print "got result from memcached\n";
return 0;

?>

Running the script on the command line produces the following results:

# php memtest.php 
got result from mysql

# php memtest.php 
got result from memcached

# php memtest.php 
got result from memcached

 

Conclusion


This tutorial covers speeding up the retrieval of data from a database by connecting it to memcache. However, do keep in mind that memcache’s strengh originates from the fact that is a cache—it is not a datastore. When using memcache, do not expect it to replace a database. Because memcache only holds values for a set length of time for a given key, you may not always find the information you need cached, and in cases like these, having the original server database is imperative.

Nevertheless, memcache is a very useful program and can do a lot to increase the server efficiency.

If you have any other questions about Memcache, feel free to ask