ThinkingCog

Articles written by Parakh Singhal

Installing Plex Media Server on Raspberry Pi

 

Key Takeaway

In my last post I wrote about how to install and operate Raspbian operating system in a headless mode using a static IP address with the intention of starting and using services that require a static IP address. Plex Media Server is one such service which can run fruitfully on Raspberry Pi 3 Model B and use the hardware to the fullest. In this post I will demonstrate how to install Plex Media Server on your Raspberry Pi and consume media from it.

Read On

Plex Media Server as the name suggests is a media server and can serve various types of media on a variety of devices. The basic premise behind the server is to provide media to the consuming point per the capabilities of the client. For example, if you have a video file in mkv (Matroska) format and you want to see the video on an iPad, you cannot do until you transcode the file into a format that is compatible with iPad. In this case it will be mp4. Plex provides on the fly transcoding capabilities, so you do not have to wait for the entire video file to be transcoded and then see it. It instead, transcodes the file on the fly and streams it to your device.

To get started, make sure that you have the latest version of Raspbian operating system installed and Pi is configured to communicate to your router using a static IP address. Before we start, let’s make sure that the system is having the latest packages installed using the following commands:

sudo apt-get update sudo apt-get upgrade

Update command brings information about the newer versions of the packages installed in the operating system and are available in the repository, while upgrade actually downloads the packages and installs them. Please note that upgrading the packages can download a significant amount of data, so if you are on metered bandwidth, beware.

After this we need to ensure that the traffic between Pi and clients travel on an encrypted channel. For that we need to make sure that the “https” package is installed and running. Run the following command:

sudo apt-get install apt-transport-https

 

Plex is not officially supported on Pi, so we will download the port created by the good folks at dev2day. The first step to do that is to add the public key corresponding to the package. Run the following command to get it done:

 

wget -O - https://dev2day.de/pms/dev2day-pms.gpg.key | sudo apt-key add -

 

01 Installation of Key - Cropped

 

I tried searching for the Plex package made for Stretch, but was not available at the time of writing this post/. So we will install the one available for Jessie. Add the package to the list of packages by running the following command:

echo "deb https://dev2day.de/pms/ jessie main" | sudo tee /etc/apt/sources.list.d/pms.list

 

02 Correct - Inclusion in the package List - Cropped


Now update the package list so we are able to pull in the latest Plex Media Server package:

sudo apt-get updateNow get and install the package: sudo apt-get install -t jessie plexmediaserver

 

 

03 Updating the packages - Cropped

 


04 Installation of Plex - Cropped

We need to change the default user under which the PMS is supposed to run from “plex” to “pi”:

sudo nano /etc/default/plexmediaserver.prev

 

05 Change of user - Cropped


Now restart the plex service and reboot Pi:

sudo service plexmediaserver restart sudo reboot

 Once Pi comes up, go to the static IP address at which Pi operates appended by “:32400/web” and it will take you to the welcome screen. From there on you can register for an account and add media to your library. Note that creating and using an account is not necessary to use Plex.

06 Plex Installed - Cropped

Welcome to Plex Media Server on Raspberry Pi.

How to Setup Raspberry Pi Without a Monitor and Keyboard and Ethernet Cable (Headless Mode)

Key Takeaway:

Installation of the Raspbian operating system on a Raspberry Pi requires an external keyboard and mouse for providing input and an external HDMI monitor to see what’s happening. In this post, we will learn how to configure the boot media, so that we are able to leverage our existing Windows hardware for the installation and subsequent operations without requiring an external keyboard, mouse and a monitor, also known as the “headless mode”. We will also be connecting our Pi to our Wi-Fi network without the any intermediate use of an Ethernet cable and will allocate it a static IP address.

Read On:

Raspberry Pi is a single board computer that offers a fun way to learn about Linux and is capable of handling various projects that require light to medium computing power. It is one of the most favored platforms for prototyping an IoT project.

Installation of most of the Raspberry Pi compatible operating systems at the time of writing this post requires the presence of an external USB keyboard and mouse to provide input and an HDMI capable monitor to receive video feed of what’s happening. These requirements prove to be a bit of a hurdle in getting Raspberry Pi up and running. Fortunately, things are improving and creators of Linux distros are making sure that there are ways to configure Raspberry Pi to facilitate a headless install, configuration and subsequent operation with the use of existing integrated hardware like a pc or a laptop.

In this post I will demonstrate how to install and operate Raspbian operating system in a truly headless fashion using your existing Windows PC hardware. We will accomplish the following:

1. Enable SSH for logging into Pi after the first boot.

2. Connect to existing Wi-Fi with the help of supplied credentials.

3. Have a static IP address, so in future we will have the ability to reliably connect to Pi at a known address and enable us to operate services like a NAS or a media server.

4. Enable VNC server on Pi so we can remotely login into Pi and use the convenience of GUI to get things done.

 

To accomplish all that we need the following hardware and software:

1. Raspberry Pi 3 Model B.

2. A Windows or Linux enabled laptop or a desktop computer to download the Raspbian image to.

3. A micro-SD card to install Raspbian image onto.

4. A micro SD card reader-writer.

5. Image of the latest version of Raspbian operating system (with desktop).

6. Etcher to transfer the Raspbian operating system onto the micro-SD card.

7. Notepad++ to create configuration files. Please note that Windows Notepad won’t cut it.

8. Advanced IP Scanner to scan the network to note the subnet and ip addresses of various devices.

9. Putty to SSH into Pi after the installation of the operating system.

10. VNC viewer to remotely login into Raspbian operating system.

I am writing this post assuming that you are working with a Windows enabled computer.

Head over to RaspberryPi.org and download the latest version of Raspbian operating system with desktop. Now we need to transfer this operating system to the micro-SD card.

To transfer the operating system onto the micro-SD card we will be using Etcher from resin.io. Brilliantly simple to use and just works with a wide variety of image formats. Etcher can directly work with zipped images, eliminating the need to unzip the downloaded images. You can also download the portable version and use it without installing it.

clip_image002

 

 

 

 

Figure 1 Etcher in action

Once you have flashed and validated the micro-SD card with the operating system, prepare to perform some steps that are going to make the entire installation and subsequent operation a headless one. If you open the card in Windows Explorer, then you will be able to access the boot folder containing a few files. Since Windows does not recognize EXT4 file system, you will not be able to see or access any other of the partition on the card or any folder or file contained therein.

Enabling SSH

To enable headless configuration and operation of Raspberry Pi, it is essential that we have some mechanism to login into our Pi. SSH enables that. By default, SSH now comes disabled in Raspbian Stretch operating system. But it can be easily enabled by introducing a file named “ssh” with no extension. You do not have to bother putting anything in the file as just the presence of the file will indicate your intention to enable SSH in Raspbian at the first and subsequent boots.

clip_image004

 

 

 

 

Figure 2 Insert a blank text file SSH excluding any extension

Configuring Wi-Fi

One of the changes introduced in Raspbian Stretch was the ability to put “wpa_supplicant.conf” file into the boot folder which at first boot could be used to configure network settings. So open up your Notepad++ and open a new text file. Make sure to change the “End of Line” setting set to “UNIX” (Edit->EOL Conversion). If you do not change this setting, Notepad++ will use Windows end of line settings and network settings will not take effect.

clip_image006

 

 

 

 

 

 

 

Figure 3 Change the EOL setting in Notepad++ to UNIX when handling any file for Raspbian

Insert the following settings into the file:

   1: country=in
   2: update_config=1
   3: ctrl_interface=/var/run/wpa_supplicant
   4:  
   5: network={
   6: scan_ssid=1
   7: ssid="MyNetworkSSID"
   8: psk="MyNetworkPassword"
   9: }

Change the country entry to the applicable one. Similarly use the SSID of your Wi-Fi network and corresponding password.

Save the text file in the boot folder and name it “wpa_supplicant.conf”. The significance this file is that it serves the configuration to the supplicant (basically hardware or software that connects to a network. More info is available here), and after the first boot gets copied into the “/etc/wpa_supplicant” directory for operational purposes. You can read more about this here (in context with Raspbian Stretch).

Now that we have taken care of SSH and Wi-Fi settings, let’s get ourselves a static IP address where we can SSH to. Boot up your Pi.

Configuring for a static IP address

Now to assign a static IP address to our Pi, we need to login into Pi at the IP address that gets allocated to our Pi at its first boot. We will use that IP address in Putty and start our SSH session and carry out further configuration to work with a static address.

We can find the IP addresses allocated to various devices with the help of the Advanced IP Scanner tool available from Famatech. The best thing about this tool is that you can run this tool without installing it, in portable mode. Boot up your Pi and then run this tool to see the IP address allocated to the device.

clip_image008

 

 

 

Figure 4 Raspberry Pi connected to Wi-Fi with a random IP address allocated

Once you have noted down the IP address allocated to Pi, use Putty or your favorite SSH tool to login into Pi.

clip_image009

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 5 SSH into Pi using the allocated IP address

The default username and password to be used for logging into Pi are “pi” and “raspberry” respectively. After logging into Pi via SSH, use the following steps to configure Pi to have a static address.

clip_image011

Figure 6 Successful SSH login

Now give the following command to know about the gateway (your router in this case), just to be sure that your Pi is communicating at the IP address shown by the IP Scanner. This IP address is the one that will always be used by Pi to communicate to the router.

   1: route -ne

 

clip_image013

Figure 7 Gateway IP address

Once we have noted down the gateway’s IP address, it is time to figure out the name server. This setting is stored in resolv.conf file. Use the following command to pull it up and note down the IP address of the name server.

   1: cat /etc/resolv.conf]

 

clip_image015

Figure 8 Name server IP address

Now we need to modify the file “dhcpcd.conf” file which contains the network settings that go into effect once Pi boots up. Use the following command to pull up the file in editable mode in nano text editor:

   1: sudo nano /etc/dhcpcd.conf]

You should be able to see some pre-existing but, commented out entries showing you the way to configure the settings in this file. We will create a new entry block at the bottom of the file. Use the following entries to configure your Pi to use a static IP address at boot time and communicate to your gateway and use the designated name server:

   1: interface wlan0
   2: static ip_address=”your desired IP address”
   3: static routers=”your router’s IP address” 
   4: static domain_name_servers=”your name server’s IP adderss”

 clip_image017

Figure 9 Configure the entries with the desired IP addresses

Once the aforementioned steps are complete, reboot your Pi for the network settings to take into effect. Use the command to reboot:

   1: sudo reboot –p

Once your Pi boots up, it will acquire the configured static IP address, and you should be able to login into it, using Putty.

clip_image019

Figure 10 Verification of newly allocated static IP address with the help of IP Scanner

clip_image020

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 11 Using the new static IP address to SSH into Pi

Once you have gotten into Pi, pull up the raspberry pi configuration utility to configure the Pi for the following:

1. Change the password from the default “raspberry” to something that only you know. This is an essential security measure.

2. Change the setting in “Advanced Options” to allow the Pi to see and use the entire file space. By default, that is not the case.

3. Change the setting in “Advanced Options” to change the resolution of the screen to what is native to your Windows machine.

4. Enable graphical desktop at boot from “Boot Options”.

5. Enable VNC from the “Interfacing Options” setting so we can use the VNC viewer to login into Pi using the GUI capabilities of Raspbian OS.

clip_image022

Figure 12 Raspberry configuration utility with all the options available

clip_image024

Figure 13 Configuring to boot into desktop mode

clip_image026

Figure 14 Changing the resolution to that of my Windows machine

Once all the changes have been done, reboot Pi. Now we will be able to login into Pi using VNC viewer.

clip_image028

Figure 15 First boot into GUI via VNC viewer

We have accomplished all that we had set out to achieve. Now every time you will access Pi, you will be able to access it over a static IP address and login into GUI. From here-on you can go ahead and configure Pi for services that require a static IP address. Depending on your expertise level and requirements, you can completely skip the VNC part and just configure Pi to have a static IP address and operate it over SSH.

Running Redis in a Master-Slave Replication Setup

Key Takeaway:

Redis has the facility to setup replication with the help of master-slave configurations. A master can have multiple slaves and a slave can further have slaves. In this article we will focus on a simple setup having a single master and two slaves and will discuss a general usage pattern which would allow for a robust setup.

Read on:

Redis allows for configuration in two flavors:

1. With the help of a configuration file,

2. At runtime through commands.

In this article we will setup both the master and slaves with the help of configuration files, as that is something more understandable and how instances are configured in a production environment.

Download the Redis for Windows from MSOpenTech’s GitHub Release page for 64 bit architecture, or if you are having a 32 bit computer, please refer to my previous article on how to compile Redis for a 32 bit Windows environment. Put the folder having all the files needed to run Redis in C drive.

Next we are going to discuss some of the settings required to be implemented in the configuration files, one for each instance of Redis. The general architecture that I am trying to produce here looks something like the following:

Master-Slave-Client Diagram






























Figure 1 Redis replication basic architecture

Explanation of the architecture:

In Redis, master instance allows for both reads and writes, and in addition to that allows for disk persistence. Slaves, by default, are read only and allow for disk persistence. Over here, since this is just an introductory article, we are going to learn how to setup the simplest master-slave configuration. A more prudent setup would allow master to engage only with memory writes, and offload disk persistence to one of the slaves, and one or more slaves will dedicatedly handle the read queries. But we will discuss this in some later article.

In order to implement the aforementioned architecture we need to create three configuration files, one for master and one for each of the two slaves.

1. Nomenclature of configuration file:

It is important to name a Redis configuration file in such a way that the purpose and some vital information contained can be gleaned off from the name itself.

We will follow the pattern: redis.hosting environment.type of instance.purpose of instance.port number.conf

So a configuration meant for a master instance would bear the name like redis.windows.master.writes.5555.conf.

2. Creation of configuration files:

Redis master: Copy the configuration file that comes pre-packaged with Redis and rename it to redis.windows.master.writes.5555.conf, where 5555 is the port that will be dedicated to master instance. You can name it differently according to the port availability on your machine. Open the configuration in a text editor and change the default port from 6379 to the one that is available in your machine.

00 Master Configuration

Figure 2 Configure master instance to run on port 5555

Redis slave 1: Make a copy of the master’s configuration file and name it like redis.windows.slaveof5555.reads.5556.conf. Change the port in the file to 5556 or something that is available on your machine. Now search for the “Replication” section and un-comment the setting of “slaveof” and provide the IP address on which the master instance will be hosted and the port number. Since we will be just running all the three instances locally, the IP address should be 127.0.0.1 and the port number used in the master’s configuration file. The slave instance that we will run will take it’s configuration from this file.

00 Slave Configuration

Figure 3 Configure slave instance to receive synchronization from master

Redis slave 2: Repeat the aforementioned steps, with the exception of changing the port number to 5557 or something that is available on your machine and accordingly use the same port in the name of the file. I have named mine to redis.windows.slaveof5555.reads.5557.conf.

3. Running instances

Redis master: Open a command prompt and navigate to the folder where you are having Redis executable files and execute redis-server.exe in conjunction with the name of the configuration where fro it is supposed to pick it’s configuration from.

01 Redis Master

Figure 4 Master instance receives requests for data sync from slaves

Redis slave 1: Open another command prompt and again run the redis-server.exe file, this time specifying the slave configuration file. This will enable running a slave instance connected to the master. As soon as the slave instance will come up, the master will receive a request from slave for synchronization of data. This is shown in the screenshot.

02 Slave 1

Figure 5 Slave 1 receives and syncs data with master

Redis slave 2: Repeat the aforementioned step for slave 1, but with the other configuration file meant for slave 2.

03 Slave 2

Figure 6 Slave 2

Now run another command prompt and run redis-cli.exe and connect to the master instance. Insert some keys in the master and query them, just to make sure, they have gotten stored. Now disconnect from the master instance and connect to the first slave hosted on port 5556 (or where you hosted it.) and query for the same keys, that you inserted in the master. You will find them. Similarly you will find the same information synchronized in slave 2.

04 Redis Client

Figure 7 Redis client shows that master and slaves are at parity

Conclusion:

Running replication in Redis is very simple and minimal configuration. The pattern shown here, is elementary, just to give an idea about Redis replication. There are more robust architectures that should be used in production settings.

Running Redis as a Service in 32 bit Windows

Key Takeaway:

Learning a lot of NoSQL data-stores is easy in part because all you have to do in order to run the server, is to click on an executable file and it starts listening on a local IP address and port. Redis works the same way. But in order to run Redis in production environment, one cannot rely on a console application listening on a port. After all, there’s always a risk of someone closing the console application and shutting down the entire cache.

To remediate this, we run such products as services. Redis is no exception and in this article we will learn how to run Redis as a Windows service.

Read on:

In my last article I showed how to compile and run Redis in a 32 bit Windows environment. In this article, I am going to use the same build to run it as a Windows service.

NOTE: If you have a 64 bit processor computer, you can directly go to this link and download the binaries, in ready to use condition.

NOTE: It is always preferable to run Redis on non-default port, as that gives us the option of running an ad-hoc instance of Redis server for quick experimentation and learning. In this exercise we are going to run Redis as a service in 32 bit Windows environment and provide the runtime configuration via a configuration file.

1. Navigate to the folder with a sample configuration file. In the source code that folder will be

redis-3.0->msvs->setups->documentation->redis.windows.conf.

2. Locate the line that signifies the port on which the Redis will listen for connections. This is found in “General” category. To search, use the word “port” and change the port number to something that is available on your machine. For this exercise, change it to 6377.

Redis configuration

 

 

 

 

 

 

 

 

 

 

 

3. Copy the file in the same folder as redis-server.exe. If you compiled the code that folder will be located at the following location: redis-3.0->msvs->Win32(*mine is 32 bit environment)->Release(* compile configuration). For more information on how compile Redis source code for 32 bit Windows environment, refer to my last post.

4. Open a command prompt and navigate to the location where you are storing the Redis executable named redis-server.exe and execute the following commands:

a. C:/Redis-3.0>Redis-server - -service-install redis.windows.conf  - -service-name “Redis Server”

b. C:/Redis-3.0>net start “Redis Server”

c. C:/Redis-3.0>Redis-cli –h 127.0.0.1 –p 6377

Test Redis installation as a service

 

 

 

 

 

 

 

 

 

 

 

 

 

 

5. If the service installation was successful and it started successfully, then you will be able to ping the server and receive a response back.

6. Just in case you decide to uninstall the service, issue the following commands:

a. C:/Redis-3.0\msvs\Win32\Release>net stop “Redis Server”

b. C:/Redis-3.0\msvs\Win32\Release>sc delete “Redis Server”

Stop and uninstall Redis service

 

 

 

 

 

 

 

 

 

 

Now you can go ahead and practice with Redis command line interface by connecting to this service.

Running Redis in 32 bit Windows

Key Takeaway:

Redis started its life in Linux environment and to this date, it is officially supported only in Linux environment. However, the good folks at Microsoft had started a new division with the sole purpose of porting such useful products to Windows environment and let developers use them with confidence and with the support of Microsoft behind them. Even though that division has now merged back into Microsoft, engineers at Microsoft continue to make contributions to ported projects.

With the 64 bit architecture becoming the de-facto standard in commodity computers, open source contributors and organizations around the world have started to focus on releasing binaries for 64 bit architectures. That leaves people like me who are living perfectly fine life with their old trusty computers having 32 bit processors powering them. The silver lining in this scenario, is that if you have access to the source code of a product, then you can compile it for 32 bit architecture and then use it.

This article shows how to download Redis source code, compile it using Visual Studio for a 32 bit architecture and then run it in a Windows environment.

Read on:

If you are a .Net developer, chances are that you have for majority of your professional life, written code on Windows platform, for Windows platform. On top of that if you have an old computer powered by a 32 bit processor, then you are striving to find ways to learn the new and up-coming technologies, such as NoSQL databases – MongoDB, Redis etc. which primarily have been released for 64 bit architectures.

I have been working with Redis since the past couple of months now and started learning Redis on my personal laptop which at the time of writing this article is 8 years young. This sometimes leaves me in a bit of a lurch as the newer projects and products are mainly focusing on releasing for 64 bit architectures. Well, all is not lost if you are willing to put up a little fight and compile the source code yourself, if it is available.

Thankfully Microsoft recognized that .Net developers should not get left behind when it comes to awesome products like Redis, just because they are not available to be run in the Windows environment, and hence they created a dedicated entity, Microsoft Open Tech Group devoted to porting these technologies in the Windows environment, and further the collaboration with the open source community. Redis is one of the projects that the group is handling at the moment.

Alright, so let’s run Redis on Windows in 32 bit architecture. Pre-requisite for accomplishing this is the newest version of Visual Studio 2013 with update 5, as without update 5, the process might not work successfully. If you do not have a paid version of Visual Studio, Community Version which is available for free here will also work.

1. Head over to Microsoft Open Tech Group’s Redis GitHub page for version 3.0 and download the source code available as a zip file.

2. Unzip the code in a folder in C drive. Make sure that the folder name in which the source code is housed bears a name with no space or special character.

3. Open the solution located in the msvs folder in Visual Studio.

4. Open the solution’s properties and go to the configuration manager and change the “Active solution configuration” to Release and “Active solution platform” to x86.

clip_image002

 

 

 

 

 

 

 

 

 

 

 

5. Now build the solution. Note there will be a few warnings that’ll come up, but ignore them.

6. Once the build completes, you will notice that there will be a new folder named “Win32” that would have gotten created. Inside this folder will be another folder “Release”. Release folder contains the final build and the executable files that we can use to run and learn Redis.

7. Locate the file “redis-server.exe” and execute it. It should come up looking something like shown here. This is the Redis server which by default listens on IP address 127.0.0.1 and port 6379.

clip_image004

 

 

 

 

 

 

 

 

 

 

8. Locate the file “redis-cli” and execute it. It should come up looking something like shown here. This is Redis command line interface and by default it sends commands on the address and port of 127.0.0.1 and 6379 respectively.

clip_image006

 

 

 

 

 

 

 

 

 

 

Lo and behold, we have Redis running in 32 bit Windows environment. Thanks to the brilliant folks at Microsoft for porting this valuable piece of technology to Windows environment.

Hello Redis

2015 is coming to an end, and I have not written as much as I should have. It has been a busy year for me on the learning end and I am glad about it. Slowly, but surely, I am moving my career in the direction that I always wanted it to go. This year has been the year of NoSQL databases for me.

There’s a storm going on in the NoSQL database world, each one vying to grab mindshare and occupy a place in your development stack. Now that developers have started to understand the segment better than before, these data-stores can be safely classified into broad categories per their use case fit, such as:

1. Want to write everything to file in a schema-less environment – go for document oriented databases like MongoDB, RavenDB etc.

2. Want to deal with deep hierarchical data and process it real fast – go for graph databases such as Neo4J, OrientDB etc.

3. Want to store everything in-memory for fast retrieval – go for in-memory data-stores like Memcached, Redis, Hazelcast etc.

and so on.

It is the first and the third categories in the aforementioned list, that can be leveraged in general purpose applications and either are already in enterprise-ready state or will be in a release or two.

I recently started learning about distributed caching solutions for one of my open source projects and thus began my journey to learn one.

A distributed cache is built upon the fundamental idea of separating out the caching component from the ones provided by programming frameworks like .Net and be hosted independently. When this idea spans several servers, it becomes distributed in nature, hence the name distributed cache.

Some of the major distributed cache products available can be found over at Wikipedia. Of particular interest are Memcached, Riak, Redis and Hazelcast, as they are free for any use and have got a wide community support. At the core of all products is the central idea of storing data in RAM as a key-value pair. Of course, different products differentiate from one another on the basis of features.

It has been a short while since I have been working with Redis, and I have decided to include it in all my web projects, if I have my way.

While this article is not having anything usable technical stuff or practical utility for that matter, I just wanted to write something, and at this time, Redis is all I have in my mind. I hope the departing year was a fruitful one for you, and I wish my best for the upcoming year.

Maintaining deep hierarchical data in SQL Server

Key Takeaway:

Every development project in some way has to deal with hierarchical data. That comes in the form of catalogs, cascading choices etc. In my previous article I demonstrated a way to maintain simple 2 level hierarchical data. In this article I will demonstrated the most optimal way of storing deep hierarchical data, that cannot be efficiently maintained using key-value pair setup. The way to do this is by using a dedicated data type in SQL Server – HierarchyId.

The article will not go deep into explaining about hierarchyid or the methods that accompany it, but rather will show how to use the data type in real world scenarios.

Read On:

When you have to store a complex hierarchy, traversing and manipulating that hierarchy becomes a tedious task. Nested hierarchy that goes beyond a flat 2 tier level is better managed with the help of a dedicated data type in SQL Server – HierarchyId.

HierarchyId is an in-built data-type in SQL Server and is based on CLR representation of binary data. It essentially stores an ordered path in an efficient manner. Because it is a CLR based data-types, there are methods that are available that can be leveraged in SQL queries to work with the data type. Although it is based on CLR, you do not have to take any additional steps in SQL Server to enable the use of the data type. It was first introduced in SQL Server 2008.

I will use some of the methods in this article to show you how to maintain a simple hierarchy. I will be re-using the example from my previous post and will try to store a hierarchy of different types of vehicles depending upon the environment in which they can be used.

Hierarchy of vehicles

 

 

 

 

 

 

 

 

Figure 1 Hierarchy of vehicles

In order to accommodate such a hierarchy, let’s create a data model. The data model in our case will only consists of a single table. This table can then be connected to the rest of the tables in the main data model with the help of primary-foreign key relationship.

image

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 2 Table that will capture the hierarchy. Bold entries are required.

HierarchyData Table:

Column Name

Data type

Description

Id

Int

The auto incrementing integer type primary key.

Hierarchy

Hierarchyid

Binary representation of ordered path using data type hierarchyid.

HierarchyPath

Computed

A computed column to show path of a node in the hierarchy as a string for our easy comprehension.

HierarchyLevel

Computed

A computed column to show level of a node in the hierarchy as am integer value for our easy comprehension.

Entry

Navarchar(100)

Entry denoting either a parent or child.

Description

Nvarchar(1000)

Description about entry.

ShortName

Nvarchar(20)

Short string about node which will be leveraged in where clause in SQL queries.

IsActive

Bit

Bit field denoting if an entry is active or not.

CreatedBy

Nvarchar(100)

Entry to denote who actually created the entry.

CreatedDate

Datetime

Entry denoting the date and time when the entry was created.

ModifiedBy

Nvarchar(100)

Entry to denote who modified the entry.

ModifiedDate

Datetime

Entry denoting the date and time when the entry was modified.

Contrast this with the data model as presented in the previous article. You will notice that in this case we are only using a single table to store both the parent and the child entries.

As is done in the previous article, for our ease of querying, we will maintain a column called “ShortName” that will be used in where clause in all the queries. This makes the queries independent of the primary key column, in which the value may change as we move the data from one environment to another.

Now here’s the SQL that you can use to create the table and fill it up with data.

-- Creation of table
Create Table dbo.HierarchyData
(
    Id int identity(1,1) primary key,
    Hierarchy hierarchyid not null,
    HierarchyPath as Hierarchy.ToString(),
    HierarychyLevel as Hierarchy.GetLevel(),
    Entry nvarchar(100) not null,
    Description nvarchar(1000),
    ShortName nvarchar(20) not null,
    IsActive bit not null,
    CreatedBy nvarchar(100) not null,
    CreatedDate datetime not null,
    ModifiedBy nvarchar(100) null,
    ModifiedDate datetime null
);
 
-- Insertion of data
Declare 
@Root hierarchyid,
@Parent hierarchyid,
@SiblingOnRight hierarchyid,
@SiblingOnLeft hierarchyid,
@HierarchyValue hierarchyid;
 
Set @Root = hierarchyid::GetRoot();
Insert into dbo.HierarchyData Values
(@Root,'Root Level','Root Level','ROOT', 1, 'Parakh Singhal',GetUtcDate(),null,null);
 
Select @HierarchyValue = @Root.GetDescendant(null,null);
Insert into dbo.HierarchyData values
(@HierarchyValue,'Ground Vehicles','Ground Vehicles','GRND_VEH',1,'Parakh Singhal', GetUtcDate(),null,null);
 
Select @SiblingOnLeft = Hierarchy from dbo.HierarchyData where ShortName = 'GRND_VEH';
Set @HierarchyValue = @Root.GetDescendant(@SiblingOnLeft,null);
Insert into dbo.HierarchyData values
(@HierarchyValue,'Air Vehicles','Air Vehicles','AIR_VEH',1,'Parakh Singhal', GetUtcDate(),null,null);
 
Select @SiblingOnLeft = Hierarchy from dbo.HierarchyData where ShortName = 'AIR_VEH';
Set @HierarchyValue = @Root.GetDescendant(@SiblingOnLeft,null);
Insert into dbo.HierarchyData values
(@HierarchyValue,'Water Vehicles','Water Vehicles','WATER_VEH',1,'Parakh Singhal',GetUtcDate(),null,null);
 
Select @HierarchyValue = Hierarchy.GetDescendant(null,null) from dbo.HierarchyData where ShortName = 'GRND_VEH';
Insert into dbo.HierarchyData values
(@HierarchyValue,'Trucks','Trucks','GRND_VEH_TRUCKS',1,'Parakh Singhal',GetUtcDate(),null,null);
 
Select @Parent = Hierarchy from dbo.HierarchyData where ShortName = 'GRND_VEH';
Select @SiblingOnLeft = Hierarchy from dbo.HierarchyData where ShortName = 'GRND_VEH_TRUCKS';
Set @HierarchyValue = @Parent.GetDescendant(@SiblingOnLeft,null);
Insert into dbo.HierarchyData values
(@HierarchyValue,'Cars','Cars','GRND_VEH_CARS',1,'Parakh Singhal',GetUtcDate(),null,null);
 
Select @Parent = Hierarchy.GetDescendant(null,null) from dbo.HierarchyData where ShortName = 'AIR_VEH';
Set @HierarchyValue = @Parent;
Insert into dbo.HierarchyData values
(@HierarchyValue,'Hot Air Baloon','Hot Air Baloon','AIR_VEH_BALOON',1,'Parakh Singhal',GetUtcDate(),null,null);
 
Select @Parent = Hierarchy from dbo.HierarchyData where ShortName = 'AIR_VEH';
Select @SiblingOnLeft = Hierarchy from dbo.HierarchyData where ShortName = 'AIR_VEH_BALOON';
Set @HierarchyValue = @Parent.GetDescendant(@SiblingOnLeft,null);
Insert into dbo.HierarchyData values
(@HierarchyValue,'Airplane','Airplane','AIR_VEH_AIRPLANE',1,'Parakh Singhal',GetUtcDate(),null,null);
 
Select @Parent = Hierarchy from dbo.HierarchyData where ShortName = 'WATER_VEH';
Set @HierarchyValue = @Parent.GetDescendant(null,null);
Insert into dbo.HierarchyData values
(@HierarchyValue,'Ship','Ship','WATER_VEH_SHIP',1,'Parakh Singhal',GetUtcDate(),null,null);
 
Select @Parent = Hierarchy from dbo.HierarchyData where ShortName = 'WATER_VEH';
Select @SiblingOnLeft = Hierarchy from dbo.HierarchyData where ShortName = 'WATER_VEH_SHIP';
Set @HierarchyValue = @Parent.GetDescendant(@SiblingOnLeft,null);
Insert into dbo.HierarchyData values
(@HierarchyValue,'Submarine','Submarine','WATER_VEH_SUBMARINE',1,'Parakh Singhal',GetUtcDate(),null,null);

And here’s the SQL that you can use to query some of the most common scenarios like:

1. Traversing down the hierarchy from a parent node,

2. Traversing up the hierarchy from a child node,

3. Finding the level of a node in the hierarchy,

4. Determining is a node is the child of a node,

5. Finding the ancestor of a child node.

Declare
@Root hierarchyid,
@Parent hierarchyid,
@Child hierarchyid,
@Node hierarchyid;
 
-- Selecting the entire table contents.
Select * from dbo.HierarchyData;
 
-- Finding the root element.
Set @Root = hierarchyid::GetRoot();
Select * from dbo.HierarchyData where Hierarchy = @Root;
 
-- Climbing down a hierarchy
-- Selecting a hierarhcy of nodes belonging to a parent.
Select @Parent = Hierarchy from dbo.HierarchyData where ShortName = 'GRND_VEH';
 
Select * from dbo.HierarchyData 
where Hierarchy.IsDescendantOf(@Parent) = 1;
 
-- Climbing up the hierarchy
-- Selecting all the parents of a node
Select @Node = Hierarchy from dbo.HierarchyData where ShortName = 'WATER_VEH_SUBMARINE';
 
Select * from dbo.HierarchyData 
where @Node.IsDescendantOf(Hierarchy) = 1;
 
-- Getting level of a node in the hierarchy.
Select Hierarchy.GetLevel() as HierarchyLevel from dbo.HierarchyData where ShortName = 'AIR_VEH_AIRPLANE';
 
-- Determining if a node is a child of a node.
Select @Parent = Hierarchy from dbo.HierarchyData where ShortName = 'GRND_VEH';
Select @Child = Hierarchy from dbo.HierarchyData where ShortName = 'GRND_VEH_TRUCKS'
 
Select @Child.IsDescendantOf(@Parent) as Result;
 
-- Determining an ancestor
-- The query selects the immediate ancestor.
Select @Node = Hierarchy from dbo.HierarchyData where ShortName = 'GRND_VEH_CARS';
Select * from dbo.HierarchyData where Hierarchy = @Node.GetAncestor(1);

Brief overview of some of the methods used in aforementioned sql queries:

1. GetRoot(): GetRoot methods when used on a hierarchyid type value will return the hierarchyid type value of the absolute root of the hierarchy.

2. GetDescendant(): Methods when used on a hierarchyid type value will return a hierarchyid type value. You can use this method to generate a new hierarchyid value when inserting data, the way I have demonstrated.

Notice that in order to create nodes that are siblings, you need to specify a pre-existing sibling (See the creation of water and air vehicles that are siblings to ground vehicles). Depending upon where you want to place the sibling in the hierarchy, you might need to provide both the sibling on the left and right in the hierarchy. Since I want to treat both air and water vehicles, I only mentioned sibling on the left in the hierarchy. If you want to place another category, say, amphibious vehicles between water and ground vehicles, then you will have to mention ground vehicles as the sibling on the left and water vehicles as the sibling on the right.

3. IsDescendantOf(): Method takes in a hierarchy id as input and returns a Boolean, if that is true.

4. GetAncestor(): Method takes in an integer value and gives the parent of the child node, depending on the integer value provided. The integer depicts the level in the hierarchy.

As you can see in the code above, maintaining a deep hierarchy with the help of hierarchyid is easy. It does not require cumbersome recursive Common Table Expressions and multiple tables to maintain, and yet offers more flexibility.

I have not gone deep into explaining about hierarchyid and the methods that accompany it. The main motive of this article is to demonstrate with the help of working code, how to maintain hierarchical data with the help of in-built mechanism in SQL Server.

References:

1. Model Your Data Hierarchies With SQL Server 2008

2. SQL Server 2008 - HierarchyID - Part I

3. Hierarchies with HierarchyID in SQL 2008

Simple way to maintain 2 level hierarchical data in SQL Server

Key takeaway:

Often we are required to maintain simple 2 level hierarchical data in databases. Such data is generally used to power dropdowns, radio button groups, checkboxes etc. where user input is required. This data is generally persisted in relational databases in separate tables which connect to the tables of interest via one to many relationships.

This method suggests a key-value like system to maintain such data, thereby reducing the total number of tables and at the same time maintaining flexibility to extend your data-model.

Read on:

I have been working on projects and have implemented a simple way to maintain simple 2 level hierarchical data. A lot of them were focused on replacing a Content Management System that has been in use since the last decade with small dedicated systems that are flexible and correspond to the changing business landscape. One concept that we borrowed from the legacy system was the system to maintain data that drive business logic in an application using key-value like 2 level hierarchical data. Since a CMS can be used to manage a variety of data that the company making the CMS cannot know about in advance, the pattern is used in such systems.

At the heart of the concept are two tables – one to store parent data and the other table that houses child data. Hence key-value like one to many hierarchy.

Let’s take an example:

We will be modeling the kinds of transports that exist and the corresponding vehicles.

Consider the following data-model for such a system:

Data model

Description: The data-model of the setup of key-value tables. Bold entries are required fields.

Key Table:

Column Name

Data type

Description

KeyID

Int (Primary key)

The auto incrementing int type primary key.

Key

Nvarchar(100)

The entry that denotes the parent.

Description

Nvarchar(1000)

Description about the parent.

ShortName

Nvarchar(50) (Unique)

A short abbreviation for the parent entry. This will be used in where clause in sql queries instead of primary key, as there’s a possibility of primary key changing when data is moved from one environment to another (e.g. dev to test). The short name is going to remain constant and unique

IsActive

Bit

Denotes whether the entry is active or not.

CreatedBy

Nvarchar(100)

Entry to denote who actually created the entry.

CreatedDate

DateTime

Entry denoting the date and time when the entry was created.

ModifiedBy

Nvarchar(100)

Entry to denote who modified the entry.

ModifiedDate

DateTime

Entry denoting the date and time when the entry was modified.

Value Table:

Column Name

Data type

Description

ValueID

Int (Primary key)

The auto incrementing int type primary key.

KeyID

Int (Foreign key)

Reference to the primary key ValueListID in ValueList table.

Sequence

Int

The sequence denotes the order in which the child entries should appear in a select statement. This is helpful in defining an order in case of dropdowns.

Value

Nvarchar(100)

The entry that denotes the child of the parent entry in ValueList table.

Description

Nvarchar(1000)

Description about the child entry.

ShortName

Nvarchar(50) (Unique)

A short abbreviation for the child entry. This will be used in where clause in sql queries instead of primary key, as there’s a possibility of primary key changing when data is moved from one environment to another (e.g. dev to test). The short name is going to remain constant and unique

IsActive

Bit

Denotes whether the entry is active or not.

CreatedBy

Nvarchar(100)

Entry to denote who actually created the entry.

CreatedDate

DateTime

Entry denoting the date and time when the entry was created.

ModifiedBy

Nvarchar(100)

Entry to denote who modified the entry.

ModifiedDate

DateTime

Entry denoting the date and time when the entry was modified.

As you can see from the data model, the parent entries are kept in the Key table and the child entries are kept in the Value table. The two tables are related via one to many relationship. The role of the ShortName field in both the tables is to help in the creation of sql queries that include the where clause based on which selection can be done. Often data is moved from one environment to other, and in the process it may happen that the primary key may change. This is especially true of databases where uniqueidentifier data type keys are used as surrogate primary keys, like data models of content management systems.

Now let’s create these tables and fill them up with some data.

Create Table dbo.[Key]
(
    KeyID Int identity(1,1) Primary key,
    [Key]    Nvarchar(100),
    Description    Nvarchar(1000),
    ShortName    Nvarchar(50) Unique,
    IsActive    Bit,
    CreatedBy    Nvarchar(100),
    CreatedDate    DateTime,
    ModifiedBy    Nvarchar(100),
    ModifiedDate    DateTime
);
Go
 
Create Table dbo.Value
(
    ValueID    Int Identity(1,1) Primary key,
    KeyID    Int,
    Sequence    Int,
    Value    Nvarchar(100),
    Description    Nvarchar(1000),
    ShortName    Nvarchar(50) Unique,
    IsActive    Bit,
    CreatedBy    Nvarchar(100),
    CreatedDate    DateTime,
    ModifiedBy    Nvarchar(100),
    ModifiedDate    DateTime
);
Go
 
Alter Table dbo.Value
Add Constraint FK_Value_Key Foreign Key (KeyId)
References dbo.[Key](KeyId);
Go
 
Create NonClustered Index NCI_Value_KeyId on dbo.Value(KeyId);
Go
 
Create NonClustered Index NCI_Value_ShortName on dbo.Value(ShortName);
Go
 
Insert into dbo.[Key] Values
('Ground Vehicles','Vehicles available for use on ground','GRND_VEH',1,'Parakh Singhal',GetDate(),null,null),
('Sea Vehicles','Vehicles available for use in sea','SEA_VEH',1,'Parakh Singhal',GetDate(),null,null),
('Air Vehicles','Vehicles available for use in air','AIR_VEH',1,'Parakh Singhal',GetDate(),null,null);
 
Insert into dbo.Value Values
(1,0,'Car','Car','GRND_VEH_CAR',1,'Parakh Singhal',GetDate(),null,null),
(1,0,'Truck','Truck','GRND_VEH_TRK',1,'Parakh Singhal',GetDate(),null,null),
(2,0,'Submarine','Submarine','SEA_VEH_SBM',1,'Parakh Singhal',GetDate(),null,null),
(2,0,'Ship','Ship','SEA_VEH_SHP',1,'Parakh Singhal',GetDate(),null,null),
(3,0,'Air Plane','Air Plane','AIR_VEH_PLN',1,'Parakh Singhal',GetDate(),null,null),
(3,0,'Hot Air Baloon','Hot Air Baloon','AIR_VEH_BLN',1,'Parakh Singhal',GetDate(),null,null);

 

Now suppose we have to query the different types of vehicles that available for use on ground, then we can run the following query:

SELECT dbo.Value.ValueID
      ,dbo.Value.KeyID
      ,dbo.Value.Sequence
      ,dbo.Value.Value
      ,dbo.Value.Description
      ,dbo.Value.ShortName
      ,dbo.Value.IsActive
      ,dbo.Value.CreatedBy
      ,dbo.Value.CreatedDate
      ,dbo.Value.ModifiedBy
      ,dbo.Value.ModifiedDate
  FROM dbo.Value Inner Join dbo.[Key]
on dbo.Value.KeyID = dbo.[Key].KeyID
where dbo.[Key].ShortName = 'GRND_VEH'

Summary:

Key-Value tables form a very concise way to store 2 level hierarchical data. The idea is primarily applied in case of content management systems where the product company cannot know in advance as what all categories will be created in the system. Using this concept you can get rid of a lot of tables that are just used to store ‘choice’ or ‘category’ data.

Adios 2014!!

Boy it is good to be back!!

If someone, somewhere has already invented a memory wipe, I would like to borrow it for a single use and would love to eradicate all the memories of the year 2014. Crazy year, which taught me a lot of things both on professional and personal fronts.

2014 certainly made me more mature person in terms of how to handle people both who I am working for and who I am working with. It ascertained the principles that have been imbibed in me from the very childhood to be a good human being first before being anything else, to be empathetic, and have patience in people.

2014 gave me experiences on a personal front that I did not wanted in the first place. But here I am..still sane, still moving on. What doesn't kill you, only makes you stronger. I hope to enjoy 2015 by writing quality posts and bring some contribution to my beloved community of developers. Have completed two open source projects, and will be writing about them shortly.

Thanks,

Parakh

Inserting Multiple Rows in SQL Server via Table Valued Parameters

Key take away:

In my last posts, I have described ways to insert multiple rows worth of data in a SQL Server database via a Comma Separated List and XML. In this post I will describe how to insert multiple rows from a .Net application using the most preferred way, a native feature in SQL Server, Table Valued Parameters.

Read on:

A short introduction to Table Valued Parameters:

Parameters are place holder for values that are passed from various constructs outside of a programming artifact. We declare parameters all the time, in various technologies, and in various forms - command line arguments, query string values in ASP.NET, sql parameters that carry information to database in a type safe way, etc. In SQL Server parameters are the way to pass and receive information that needs to be dealt with, and information can come from either within SQL Server or outside of SQL Server like from a .Net application. Parameters are generally made of a primal data type like int, char, varchar etc. When such parameters are modeled on some pre-defined table type, such parameters are called Table Valued Parameters. The important words to note here are “table type”. Not table, but table type, which means a user defined type based on some table structure.

Table Valued Parameters have been supported by SQL Server since version 2008.

To demonstrate the concept of TVPs, I will be using a very simple data model consisting of three tables: Student, Course and an association table supporting many-to-many relationship between Student and Course, StudentCourse.

The Entity-Relationship diagram will clarify the relationship between the tables:

ERD diagram:

ERD diagram

 

 

 

 

 

 

 

According the ER diagram, a student can take many courses and a course can be taken by many students. The insertion of courses that a student is interested in the association table is an ideal application of this technique.

Now consider the following code to make a table type. this type will be used to model a table valued parameter on the table type:

   1:  CREATE TYPE StudentCoursesTableType AS Table
   2:  ([StudentID] [int] NULL,
   3:   [CourseID] [int] NULL);

 

Making a table type makes the type available in the User-Defined Table Types sub-section of Types section.

clip_image002

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Execution of the following code should give you some result depending on what you have available in the StudentCourse table.

   1:  Declare @StudentCoursesTVP as StudentCoursesTableType;
   2:   
   3:  Insert into @StudentCoursesTVP(StudentID, CourseID)
   4:  Select [StudentID], [CourseID] from [dbo].[StudentCourse];
   5:   
   6:  Select * from @StudentCoursesTVP;
 
 

clip_image002[5]

 

 

 

 

 

 

 

 

 

 

 

So now that we are through with the basics of TVP, let’s see how we can use them in our main purpose of processing multiple rows of data. To demonstrate this, I will be using a simple web form application powered by the same data model discussed above. The application helps a student get enroll in the course(s) of his/her choice.

clip_image004

 

 

 

 

 

 

 

 

 

 

 

 

The coding for the webpage is very simple. The pseudo-code is as follows:

1. Select a student from the drop down.

2. Select from the available courses that the student needs enrollment for.

3. Click on the submit button.

I am going to leave it up to the reader to understand the programming in the web application. It is straight forward and the domain model powering the application is a reflection of the data model depicted above.

The main work is being done at two places:

1. Web application’s repository method which parses the incoming object data into a datatable.

2. The stored procedure responsible for receiving the data in the form of a table valued parameter and using it to feed data into the desired table.

Consider the following repository method in the StudentSQLRepository class:

   1:  public int GetEnrolled(List<Course> courses, int studentID)
   2:          {
   3:              DataTable table = new DataTable("data");
   4:              table.Columns.Add("StudentID");
   5:              table.Columns.Add("CourseID");
   6:              foreach (Course course in courses)
   7:              {
   8:                  table.Rows.Add(new object[] { studentID, course.CourseID});
   9:              }
  10:              string sql = @"dbo.EnrollStudentInCourses";
  11:              int result = 0;
  12:              SqlParameter studentCoursesTVP = new SqlParameter("StudentCoursesTVP", table);
  13:              studentCoursesTVP.SqlDbType = SqlDbType.Structured;
  14:              using (SqlConnection connection = new SqlConnection(connectionString))
  15:              {
  16:                  using (SqlCommand command = new SqlCommand(sql, connection))
  17:                  {
  18:                      command.CommandType = CommandType.StoredProcedure;
  19:                      command.Parameters.Add(studentCoursesTVP);
  20:   
  21:                      connection.Open();
  22:                      result = command.ExecuteNonQuery();
  23:                      connection.Close();
  24:                  }
  25:              }
  26:              return result;
  27:          }

There are two noteworthy points in the code above:

1. We are passing the datatable to the SQL Server database to be processed. The datatable is modeled on the table type present in the SQL Server database, used by the table valued parameter.

2. The SqlDbtype of the SqlParameter is set to Structured type. This is strictly done to improve the readability of our code and convey meaning in an explicit way. ADO.NET will perfectly parse the datatable correctly had we not declared the SqlDbType to Structured explicitly.

Following is the SQL script of the dbo.EnrollStudentInCourses responsible for parsing the data in the @StudentCoursesTVP table valued parameter correctly.

   1:  CREATE PROCEDURE [dbo].[EnrollStudentInCourses]
   2:      
   3:      @StudentCoursesTVP StudentCoursesTableType readonly
   4:   
   5:  AS
   6:  BEGIN
   7:   
   8:      Insert into StudentCourse (StudentID, CourseID)
   9:      Select StudentID, CourseID from @StudentCoursesTVP    
  10:      
  11:  END
  12:   
  13:  GO

 

If you go and read my two other posts on inserting multiple rows using a comma separated values list and XML, you surely will come to the conclusion that using table valued parameters is the cleanest approach, that doesn’t include any code acrobatics either in .Net code or in SQL code. Not only that, table valued parameters code is highly performant as long as there’s reasonable data involved.

The sample code for this post can be downloaded from the following location:

References:

1. Table Valued Parameters at TechNet