ThinkingCog

Key Takeaway:

Redis has the facility to setup replication with the help of master-slave configurations. A master can have multiple slaves and a slave can further have slaves. In this article we will focus on a simple setup having a single master and two slaves and will discuss a general usage pattern which would allow for a robust setup.

Read on:

Redis allows for configuration in two flavors:

1. With the help of a configuration file,

2. At runtime through commands.

In this article we will setup both the master and slaves with the help of configuration files, as that is something more understandable and how instances are configured in a production environment.

Download the Redis for Windows from MSOpenTech’s GitHub Release page for 64 bit architecture, or if you are having a 32 bit computer, please refer to my previous article on how to compile Redis for a 32 bit Windows environment. Put the folder having all the files needed to run Redis in C drive.

Next we are going to discuss some of the settings required to be implemented in the configuration files, one for each instance of Redis. The general architecture that I am trying to produce here looks something like the following:

Figure 1 Redis replication basic architecture

Explanation of the architecture:

In Redis, master instance allows for both reads and writes, and in addition to that allows for disk persistence. Slaves, by default, are read only and allow for disk persistence. Over here, since this is just an introductory article, we are going to learn how to setup the simplest master-slave configuration. A more prudent setup would allow master to engage only with memory writes, and offload disk persistence to one of the slaves, and one or more slaves will dedicatedly handle the read queries. But we will discuss this in some later article.

In order to implement the aforementioned architecture we need to create three configuration files, one for master and one for each of the two slaves.

1. Nomenclature of configuration file:

It is important to name a Redis configuration file in such a way that the purpose and some vital information contained can be gleaned off from the name itself.

We will follow the pattern: redis.hosting environment.type of instance.purpose of instance.port number.conf

So a configuration meant for a master instance would bear the name like redis.windows.master.writes.5555.conf.

2. Creation of configuration files:

Redis master: Copy the configuration file that comes pre-packaged with Redis and rename it to redis.windows.master.writes.5555.conf, where 5555 is the port that will be dedicated to master instance. You can name it differently according to the port availability on your machine. Open the configuration in a text editor and change the default port from 6379 to the one that is available in your machine.

Figure 2 Configure master instance to run on port 5555

Redis slave 1: Make a copy of the master’s configuration file and name it like redis.windows.slaveof5555.reads.5556.conf. Change the port in the file to 5556 or something that is available on your machine. Now search for the “Replication” section and un-comment the setting of “slaveof” and provide the IP address on which the master instance will be hosted and the port number. Since we will be just running all the three instances locally, the IP address should be 127.0.0.1 and the port number used in the master’s configuration file. The slave instance that we will run will take it’s configuration from this file.

Figure 3 Configure slave instance to receive synchronization from master

Redis slave 2: Repeat the aforementioned steps, with the exception of changing the port number to 5557 or something that is available on your machine and accordingly use the same port in the name of the file. I have named mine to redis.windows.slaveof5555.reads.5557.conf.

3. Running instances

Redis master: Open a command prompt and navigate to the folder where you are having Redis executable files and execute redis-server.exe in conjunction with the name of the configuration where fro it is supposed to pick it’s configuration from.

Figure 4 Master instance receives requests for data sync from slaves

Redis slave 1: Open another command prompt and again run the redis-server.exe file, this time specifying the slave configuration file. This will enable running a slave instance connected to the master. As soon as the slave instance will come up, the master will receive a request from slave for synchronization of data. This is shown in the screenshot.

Figure 5 Slave 1 receives and syncs data with master

Redis slave 2: Repeat the aforementioned step for slave 1, but with the other configuration file meant for slave 2.

Figure 6 Slave 2

Now run another command prompt and run redis-cli.exe and connect to the master instance. Insert some keys in the master and query them, just to make sure, they have gotten stored. Now disconnect from the master instance and connect to the first slave hosted on port 5556 (or where you hosted it.) and query for the same keys, that you inserted in the master. You will find them. Similarly you will find the same information synchronized in slave 2.

Figure 7 Redis client shows that master and slaves are at parity

Conclusion:

Running replication in Redis is very simple and minimal configuration. The pattern shown here, is elementary, just to give an idea about Redis replication. There are more robust architectures that should be used in production settings.

Visit DALHelper.Codeplex.com for source code, ready to use assembly and related documentation.

Brief description

The DALHelper is a project that was born out of my needs to connect to SQL Server database for my projects. I generally work on projects and design them using domain model and to keep them flexible, understandable and fast, I prefer to connect to the database using ADO.NET. Writing ADO.NET commands to connect to database can become monotonous and repetitive. Using Microsoft's Enterprise Library's Data Access Application Block is one solution, but I generally find it requiring a bit more effort than should be invested. Of course it gives you the freedom to change your supporting back end database. Generally back end databases in applications do not change, especially in enterprise environments and small applications for which the express version of SQL Server can be sufficient, and if you work with such applications then, I believe DALHelper can make your life a bit easier.

NOTE: This library cannot be used with any other database other than SQL Server, as it is built upon SQL Server specific ADO.NET commands.

The library contains various helper methods that can simplify the task of doing CRUD (Create, Read, Update and Delete) on a SQL Server database in a .NET application. The library can be used with any .NET compliant language, is open sourced and will remain that way.

DALHelper touches upon the following technologies:
1. Visual C# 4.0
2. .NET Framework 4.0
3. ADO.NET technology, part of .NET framework, used to connect to databases.
4. Code Contracts, Microsoft’s implementation of Design by Contact paradigm.

The DALHelper library has been designed to be provided with the connection string to the database once via property injection and then perform various operations as desired by the user with the help of the methods provided in the library. Some of the best practices that have been followed while the methods were designed are the following:

1. The methods follow a pattern in their signature and are easy to use.

2. The methods provided in the library are overloaded, thus providing you just the signature that you need in order to accomplish what you desire, without cluttering up your code.

3. The names of the methods are very much self-explanatory in context what they service they are supposed to provide.

The variety of methods available should fit majority of the requirements found in any small to medium scale enterprise application. Every method available in the library allows you to execute a SQL query or a stored procedure.

Every method available in the library has two overloads:

1. Without parameters, and

2. With parameters

Every method available in the library has the signature of the format:

1. SQL text: Is the SQL query or stored procedure that needs to be executed.

2. SQLTextType: An enum which signifies whether the SQL text parameter is a SQL query or a stored procedure. By default, the choice is that of a stored procedure.

3. List<SqlParameter>: An optional generic list of SQL parameters type that might be required by the SQL query or stored procedure. The parameters can be of both input and output type.

Brief description of methods available in the library:

1. ExecSQL

The ExecSQL method can be used to execute a SQL statement or a stored procedure without returning any value. Available in two overloads:
public void ExecSQL(string sqlText, SQLTextType sqlTextType = SQLTextType.Stored_Proc)
public void ExecSQL(string sqlText, SQLTextType sqlTextType = SQLTextType.Stored_Proc, List<SqlParameter> parameterCollection = null)

2. GetRowsAffected

The GetRowsAffected method can be used to execute a SQL query or a stored procedure and return the total number of rows affected by execution. Available in two overloads:
public int GetRowsAffected(string sqlText, SQLTextType sqlTextType = SQLTextType.Stored_Proc)
public int GetRowsAffected(string sqlText, SQLTextType sqlTextType = SQLTextType.Stored_Proc, List<SqlParameter> parameterCollection = null)

3. GetDataTable

The GetDataTable method can used to execute a SQL query or a stored procedure and return the result set in a data table. Available in two overloads:
public DataTable GetDataTable(string sqlText, SQLTextType sqlTextType = SQLTextType.Stored_Proc)
public DataTable GetDataTable(string sqlText, SQLTextType sqlTextType = SQLTextType.Stored_Proc, List<SqlParameter> parameterCollection = null)

4. GetDataView

The GetDataView method can be used to execute a SQL query or a stored procedure and return the result set as a data view. Available in two overloads:
public DataView GetDataView(string sqlText, SQLTextType sqlTextType = SQLTextType.Stored_Proc)
public DataView GetDataView(string sqlText, SQLTextType sqlTextType = SQLTextType.Stored_Proc, List<SqlParameter> parameterCollection = null)

5. GetDataSet

The GetDataSet method can be used to execute a SQL query or a stored procedure and return the result set(s) in a dataset. Available in two overloads:
public DataSet GetDataSet(string sqlText, SQLTextType sqlTextType = SQLTextType.Stored_Proc)
public DataSet GetDataSet(string sqlText, SQLTextType sqlTextType = SQLTextType.Stored_Proc, List<SqlParameter> parameterCollection = null)

DALHelper is an excellent way to implement repository architecture style and simple CRUD implementations.

Future Roadmap

In future I plan, to extend this library into a framework, supporting various data sources such as XML, Excel, CSV files etc.

Helpful Links

Some of the links that might be helpful in understanding the programming concepts and technologies that have gone in making of this library:
Visual C#:

1. http://msdn.microsoft.com/en-us/vstudio/hh341490.aspx

.NET Framework:

1. http://www.microsoft.com/net

2. http://msdn.microsoft.com/en-US/vstudio/aa496123

ADO.NET:

1. http://msdn.microsoft.com/en-us/library/e80y5yhx.aspx

Code Contracts:

1. http://research.microsoft.com/en-us/projects/contracts

2. http://msdn.microsoft.com/en-us/devlabs/dd491992.aspx

Dependency Injection:

1. http://en.wikipedia.org/wiki/Dependency_injection

2. http://msdn.microsoft.com/en-us/magazine/cc163739.aspx

One of the essential skillset in understanding Object Oriented concepts is getting the entities and the relationships among them correct, that make up the domain model. The domain model is the real world representation of the problem area that is required to be solved with software development. The modeling of such a domain model can be done using a variety of modeling techniques, dominant of which these days is the Unified Modeling Language or UML

The traditional approach to inheritance has been to define a superclass and a sub class approach. An example of that would be that of Car being a general class embodying all the common attributes and behavior of a car, and a specific car segment such as sedan, sports car, wagon etc. being a subclass inheriting from the Car class. But in real world relationships among entities are much more complicated than that. As object oriented paradigm matured, object oriented languages also advanced and gave us a lot more fundamental structures that supported abstraction and realization of relationships that did not fit well into the classical inheritance model.

OK, so time to get to the gist of the matter. The following are the principle relationships that can be used to cast a domain model and do programming against:

OK, so time to get to the gist of the matter. The following are the principle relationships that can be used to cast a domain model and do programming against:

1. Generalization relationship

2. Dependency relationship

3. Realization relationship

4. Association relationship

a) Aggregation relationship

b) Composition relationship

A complementary concept that goes hand in hand with the relationships is the concept of multiplicity, explained later in this post.

1. Generalization Relationship

A generalization relationship is a parent class relationship, embodying the classical OOP concept of traditional inheritance. The relationship emphasizes a “is-a” relationship, and the shown with the help of a solid line with a hollow arrow going from the from the derived class and pointing to the parent class.

Example: A manager and a worker both are person on a higher abstract level, hence both Manager and Worker classes can be thought to derive from the Employee class.

Figure 1 Generalization relationship

2. Dependency Relationship:

As the name suggests, dependency relationship suggests a dependency of an entity over another. Dependency relationship is also called a client-supplier relationship, in which the client depends upon the supplier for accomplishing something. One important defining characteristic of this kind of relationship is that if the supplier changes in a way, then it affects client in some way. In UML, dependency relationship is shown by a dashed line with an arrow on one end, going from client and pointing to the supplier.

Figure 2 Dependency Relationship

Let’s take an example: I am a developer and I use a laptop to accomplish the objective of writing code. So it is a dependency relationship. If you change the laptop to a tablet, then it certainly affects me as it will have an impact on productivity.

Figure 3 Dependency relationship between me and my computer

In terms if code, it will be the Me class using an object of computer in DevelopCode method.

public class Computer

{

    …

}

public class Me

{

    public object DevelopCode(Computer computer)

    {

        …

    }

}

3. Realization Relationship:

This form of relationship is primarily used to model classes that implement any interface. In a way, it says that the objects of such classes realize the implemented interface(s). The relationship is depicted by joining the class and interface with a dashed line with a hollow arrow head pointing to the interface realized.

Figure 4 Realization relationship

A practical example in the .Net world is the implementation of the IDisposable interface which needs the public method Dispose() implemented.

public class MyClass:IDisposable

{

    public Dispose();

}

4. Association Relationship:

An association relationship signifies a stronger form of relationship between two entities. While generalization relationship signifies a “is-a” relationship, association relationship signifies a “has-a” relationship. This type of relationship comes into play when the relationship between two entities is more of containment than a superclass-subclass relationship. A car “has-a” engine (that “a” should have been “an”, but I just want to make my example clear without getting into English grammar rules). Similarly a library “has” books. A student “has-a” schedule.

Association relationship is shown by a solid line between the two entities sharing the relationship.

Figure 5 Association relationship

Figure 6 A Car "has-a" engine

In code this strong relationship manifests in the form of a one entity using the object of the other entity as a class variable or property.

In real life, entities rarely stand all by themselves. They are complex, and they are complex because of their interaction with other entities. If you notice the examples mentioned for association relationship, then you might be thinking, that there are some scenarios in which an entity can have a scope which is dependent/independent of the other entity. An engine can be taken out of a car and fixed into another car, a book can be sold by a library if it does not find any patron for a long time and a schedule has no meaning without a student. In order to take this scenario into account, association relationship can be broken down into aggregation relationship and composition relationship, depending upon the fact whether the lifetime of one entity depends upon the lifetime of the other entity, which in turn is determined by the strength of relationship between the two entities. For example: A car engine can have a life outside of a car, but a schedule cannot have any meaning without a student.

4.1 Aggregation Relationship:

Aggregate relationship is a special form of association relationship. In this relationship, the lifetime of the individual entity is independent of the aggregate entity (complex object). In code, the individual entity’s class object is used as a variable or property in aggregate entity’s class code. Aggregation relationship, in UML is denoted by a solid line between the two entities, with a hollow diamond at one end attached to the aggregate entity.

Figure 7 Aggregation relationship

Figure 8 Car has an engine that can exists independently outside of the car

public class Engine

{

    public int EngineCapacity {get; set;}

    …

}

public class Car

{

  public Engine Engine {get;set;}

}

4.2 Composition Relationship:

Composition relationship is a special form of association relationship in which the lifetime of the individual entity depends on the lifetime of the composited entity (complex object). In code, the individual entity is defined within the composited entity. Composition relationship, in UML is denoted by a solid line between two entities, with a solid diamond at one end attached to the composited entity.

Figure 9 Composition relationship

Figure 10 Book is made up of chapter(s)

public class Book

{

    public class Chapter

    {

        public int NoOfPages {get;set;}

        …

    }

        …

}

I hope this post will help clarify some of the concepts related to inheritance relationships in classes.

References:

1. http://www.uml-diagrams.org/

2. http://en.wikipedia.org

Diagrams made with Visual Paradigm for UML Community Edition

The title and subject of this blog post is a tribute to the pioneering course that I am attending at Coursera: An Introduction to Mathematical Thinking. The course is basically a transition course helping students coming from a high school mathematics background transition to university mathematics. So far I have attended two lectures and I must say the course is what I needed when I was still in school and planning my career. Such transition courses, the motive of which is not teaching the hardcore pure subject, or even the applied form of the subject, are the need of the hour, be it any field of science or academia for that matter. These not only help the prospective candidates prime up to what lies ahead, but also helps them evaluate if they are genuinely interested in the subject matter. The reason accurately put up by Prof. Keith Devlin is “For all the time schools devote to the teaching on mathematics, very little (is any) is spent trying to convey just what the subject is about. Instead, the focus is on learning and applying various procedures to solve math problems. That’s a bit like explaining soccer by saying it is executing a series of maneuvers to get the ball into the goal. Both accurately describe various key features, but they miss the “what” and the “why” of the big picture.”

The same reasoning can be extending to computer programming using the Object Oriented paradigm. Of all the learning, students and developers starting in the field of computer programming using modern object oriented languages do, almost all of it is comprised of learning the semantics of the language, and hardly, if any related to thinking in terms of objects. They then wonder about the various constructs available to them in a programming language, and how to use them but are unable to get a clear answer. The situation worsens by the fact that they are supposed to learn about the various supporting frameworks like .NET Framework for doing general programming, Windows Communication Framework (WCF) for creating services, Windows Presentation Framework (WPF) for making presentation layer, ASP.NET etc. apart from the technologies like SQL for data access etc. On top of that developers are always under the constant pressure to develop and deliver the applications in the shortest time possible, leading to a diluted learning experience all along the way. Some of the effects of this diluted experience are:

1. Having a theoretical understanding of advanced constructs offered by an object oriented language like C# or Java, but not a practical understanding as to where they can be used.

2. Trouble understanding the source code of well-constructed, well documented software.

3. The developer ends up creating code, that is hard to maintain.

When these developers come on to more responsible positions, they are not able to effectively lead and mentor the incumbent developers under them.

I believe that technical writers should also take upon the responsibility of including a short section on object oriented thinking before touching upon the technical semantics of object oriented language.

Majority of the developers get their first experience of computer programming in an object oriented language while they are in school. Due to questionable quality of teacher(s) available, they end up learning to use the language in a procedural manner, thus not learning the true nature of the object oriented paradigm. Recently, I was having a discussion with a colleague about a domain model and the kinds of design patterns that we could use to structure things, and he made a thought provoking remark, about how things differ in the Java and .NET world. He made the remark that a Java developer thinks in terms of objects, because of the way Java is taught and maturity of the developers in the Java world, while in .NET arena most of the developers think in terms of screens and the resulting code that can then support those screens. Most of the .NET developers transitioned from the VB6 (a non-object oriented language) and continued software development in more or less the same way in object oriented languages supported by the .NET Framework. Microsoft worked to make the transition easier by emphasizing on VB.NET (an easier language for someone coming from VB6 background) and developing technologies like ASP.NET Web forms which were event based and could be used without a strong understanding of object oriented concepts. Please bear in mind that I am not saying that Web forms are bad, it is just that they are easily misused (I myself am a Web forms developer). It is only recently that Microsoft has started releasing documentation centering around practices which emphasize organized object oriented development, rather than the traditional selling point of Rapid Application Development. Still I feel there is a dearth of simple examples in MSDN that educates the developers on how to develop applications in an object oriented manner. Still a lot of documentation only tangentially touches upon the foundational object oriented concepts. A lot of examples assume a very simple domain model work with the assumption of having a very direct correlation between the data and the object model, resulting in usage of Entity Framework in virtually every example. This further makes the learning from examples a bit harder than it ought to be.

Anyways, I hope for the best and believe that all .NET developers (me included) will learn about the basics of object oriented paradigm in the manner intended. I myself learnt about the languages and frameworks first and later on learnt about the true nature of Object Oriented paradigm and how to use the constructs of an object oriented language (C# in my case) effectively. I list some resources below that have helped me to learn about the object oriented paradigm and how to use the language constructs effectively.

1. Series on how to build layered web applications using Microsoft ASP.NET 2.0 by Imaar Spaanjars

2. Series on how to build n-layered web applications using Microsoft ASP.NET 3.5 by Imaar Spaanjars

3. Beginning C# 2008 Objects: From Concept to Code

4. The Object-Oriented Thought Process

In my next post I hope to touch upon a practical example of object oriented programming using Visual C# and using .NET Framework.

Background

When software development started out, the software made was relatively simple and the industry was still climbing the learning curve. Even with procedural languages, layering of source code was done so as to allow a logical separation of responsibilities and concerns. This got to the next level with the introduction of object oriented languages and the entities forming the problem domain in the source code acquired a layer of their own. As the complexity of softwares developed increased aka the functionality offered by the softwares in general increased, so increased the number of layers. With consolidation of IT assets, came the next wave of computing in the form of client – server computing with softwares living life at a single place - server rather than living individually on client computers. As the number of consumers increased, measures were taken to tackle the workload and here is where the concept of layers was taken further to introduce the concept of tiers. The tiers were simply layers bound together in one physical unit, primarily made from the perspective of scaling (installing on more than one machine). The unit of layers could depend upon other such units to provide the required functionality and have some kind of communication ability depending upon the environment and usage scenario.

So after formal background information as to why layers and tiers exist, we can now focus on what exactly is the core difference between the two. An accepted difference exists as follows:

Difference between Layer and Tier

Layers emphasize logical grouping of related functionality while tiers signify the physical separation of such logical groups. An easy way to visualize this is by imagining that all your code resides in a single project and is running on a single server in a single process, while in case of a tiered approach the tiers are located on a single machine and being handled by different processes and communicating among themselves via some mechanism to achieve the desired outcome.

A typical software project consists of 3 layers – Presentation, Business Logic Layer (BLL) and Data Access Layer (DAL).

It is when there are more than 3 layers, that your project can be termed as an N-Layer project. Similarly when your project contains more than 3 tiers, it is accepted to term it is an N-Tier project.

Different cases can exist when we talk about tiers:

1. Single machine, single process: the simplest case, tiers can exist on a single machine and can be handled by a single process.

2. Single machine, n processes: A scenario where each tier is being handled by a dedicated process.

3. True N Tier: Where each tier is hosted on a separate machine and is being handled by a dedicated process.

So one might wonder why we would convert a layered project to a tiered one. Mainly three reasons come to my mind:

1. Performance: Separating out the layers into tiers can give performance benefits as it gives us an opportunity to harness the processing power of more than one computer.

Example: One server can be made responsible to process traffic related to business logic, one to handle the data access, while if the UI load cannot be handled on one server, it can be made to be served via two servers. Thus we only increase the hardware where it is needed, giving us a flexible and performance centric solution.

2. Scalability: Scalability means the ability to handle increased workload. That in IT world means the ability of software to handle increased workload via expansion into additional hardware. Translated, it means that tiers can be moved to additional hardware according to the needs and made to handle the additional growth.

Examples:

a) If users of an application experience a slowdown, it might be an issue of the Data Access Tier requiring more horsepower. Thus we can expand only that tier with the help of additional hardware.

b) If an application experiences peak load only during certain times of the day, then it is a prime candidate for being made in a tiered fashion. Say, in the morning time the application needs to be served by a set of two servers on a per tier basis, but by noon the load decreases to an extent that it can be handled by only one set of servers on a per tier basis.

3. Fault Tolerance: Critical applications are served by redundant servers i.e. more than one server serving the same version of an application. This is done so that in case when one of the servers goes down, the other server can take over and continue to serve the application. This can be extended to tiers, when more than one server serves the same version of a tier, thus providing fault tolerance. In most of the cases these servers are load balanced using some kind of a load balancing scheme.

Now that we have learnt about the benefits of tiers, let’s learn to apply this pragmatically. Whenever we work with tiers, we have to deal with inter physical boundary communication in order to make tiers work with each other. Physical boundary is defined by either a process within which the tier is running when all the tiers are available on the same machine or by a network boundary when they are available on different machines and have to communicate across network in order to work together. This inter boundary communication introduces a lot of overhead not only in terms of communication latency, but also programming complexity. Thus an application should only be tiered when it meets a certain level of requirements – load conditions, performance requirements, fault tolerance and of course budget.

So all in all, play with tiers but only after doing a through cost-benefit analysis of the application and considering the life span of the application.