ThinkingCog

Key take away:

In my last posts, I have described ways to insert multiple rows worth of data in a SQL Server database via a Comma Separated List and XML. In this post I will describe how to insert multiple rows from a .Net application using the most preferred way, a native feature in SQL Server, Table Valued Parameters.

Read on:

A short introduction to Table Valued Parameters:

Parameters are place holder for values that are passed from various constructs outside of a programming artifact. We declare parameters all the time, in various technologies, and in various forms - command line arguments, query string values in ASP.NET, sql parameters that carry information to database in a type safe way, etc. In SQL Server parameters are the way to pass and receive information that needs to be dealt with, and information can come from either within SQL Server or outside of SQL Server like from a .Net application. Parameters are generally made of a primal data type like int, char, varchar etc. When such parameters are modeled on some pre-defined table type, such parameters are called Table Valued Parameters. The important words to note here are “table type”. Not table, but table type, which means a user defined type based on some table structure.

Table Valued Parameters have been supported by SQL Server since version 2008.

To demonstrate the concept of TVPs, I will be using a very simple data model consisting of three tables: Student, Course and an association table supporting many-to-many relationship between Student and Course, StudentCourse.

The Entity-Relationship diagram will clarify the relationship between the tables:

ERD diagram:

According the ER diagram, a student can take many courses and a course can be taken by many students. The insertion of courses that a student is interested in the association table is an ideal application of this technique.

Now consider the following code to make a table type. this type will be used to model a table valued parameter on the table type:

   1:  CREATE TYPE StudentCoursesTableType AS Table

   2:  ([StudentID] [int] NULL,

   3:   [CourseID] [int] NULL);

Making a table type makes the type available in the User-Defined Table Types sub-section of Types section.

Execution of the following code should give you some result depending on what you have available in the StudentCourse table.

   1:  Declare @StudentCoursesTVP as StudentCoursesTableType;

   2:   

   3:  Insert into @StudentCoursesTVP(StudentID, CourseID)

   4:  Select [StudentID], [CourseID] from [dbo].[StudentCourse];

   5:   

   6:  Select * from @StudentCoursesTVP;

So now that we are through with the basics of TVP, let’s see how we can use them in our main purpose of processing multiple rows of data. To demonstrate this, I will be using a simple web form application powered by the same data model discussed above. The application helps a student get enroll in the course(s) of his/her choice.

The coding for the webpage is very simple. The pseudo-code is as follows:

1. Select a student from the drop down.

2. Select from the available courses that the student needs enrollment for.

3. Click on the submit button.

I am going to leave it up to the reader to understand the programming in the web application. It is straight forward and the domain model powering the application is a reflection of the data model depicted above.

The main work is being done at two places:

1. Web application’s repository method which parses the incoming object data into a datatable.

2. The stored procedure responsible for receiving the data in the form of a table valued parameter and using it to feed data into the desired table.

Consider the following repository method in the StudentSQLRepository class:

   1:  public int GetEnrolled(List<Course> courses, int studentID)

   2:          {

   3:              DataTable table = new DataTable("data");

   4:              table.Columns.Add("StudentID");

   5:              table.Columns.Add("CourseID");

   6:              foreach (Course course in courses)

   7:              {

   8:                  table.Rows.Add(new object[] { studentID, course.CourseID});

   9:              }

  10:              string sql = @"dbo.EnrollStudentInCourses";

  11:              int result = 0;

  12:              SqlParameter studentCoursesTVP = new SqlParameter("StudentCoursesTVP", table);

  13:              studentCoursesTVP.SqlDbType = SqlDbType.Structured;

  14:              using (SqlConnection connection = new SqlConnection(connectionString))

  15:              {

  16:                  using (SqlCommand command = new SqlCommand(sql, connection))

  17:                  {

  18:                      command.CommandType = CommandType.StoredProcedure;

  19:                      command.Parameters.Add(studentCoursesTVP);

  20:   

  21:                      connection.Open();

  22:                      result = command.ExecuteNonQuery();

  23:                      connection.Close();

  24:                  }

  25:              }

  26:              return result;

  27:          }

There are two noteworthy points in the code above:

1. We are passing the datatable to the SQL Server database to be processed. The datatable is modeled on the table type present in the SQL Server database, used by the table valued parameter.

2. The SqlDbtype of the SqlParameter is set to Structured type. This is strictly done to improve the readability of our code and convey meaning in an explicit way. ADO.NET will perfectly parse the datatable correctly had we not declared the SqlDbType to Structured explicitly.

Following is the SQL script of the dbo.EnrollStudentInCourses responsible for parsing the data in the @StudentCoursesTVP table valued parameter correctly.

   1:  CREATE PROCEDURE [dbo].[EnrollStudentInCourses]

   2:      

   3:      @StudentCoursesTVP StudentCoursesTableType readonly

   4:   

   5:  AS

   6:  BEGIN

   7:   

   8:      Insert into StudentCourse (StudentID, CourseID)

   9:      Select StudentID, CourseID from @StudentCoursesTVP    

  10:      

  11:  END

  12:   

  13:  GO

If you go and read my two other posts on inserting multiple rows using a comma separated values list and XML, you surely will come to the conclusion that using table valued parameters is the cleanest approach, that doesn’t include any code acrobatics either in .Net code or in SQL code. Not only that, table valued parameters code is highly performant as long as there’s reasonable data involved.

The sample code for this post can be downloaded from the following location:

References:

1. Table Valued Parameters at TechNet

Key takeaway:

Frequently there are scenarios here you might be required to key in multiple rows into a SQL Server database table. In order to do that with minimum number of calls and a low memory footprint, there are several approaches available. I know of three, of which I am presenting the comma separated values (CSV) list approach.

Read on:

A few days ago, I ran into a requirement that required me to insert multiple rows of data into a table in a SQL Server database. There are a couple of approaches to do this and I am planning to do a series on this. Some of the approaches to do this are the following:

1. Passing a comma separated value (CSV) list to the database and parsing it at the database level with the help of a user defined function to split the list into rows and inserting those rows.

2. Passing the data as XML hierarchy,

3. Using Table Valued Functions, a feature that SQL Server supports version 2008 onwards.

All the three approaches are valid approaches and usage depends upon the kind of constraints that you are operating with. I was dealing with a legacy codebase, that did not afforded me the luxury of table valued parameters, which is the most straight forward and efficient way of accomplishing this, and had to settle for the CSV list and eventually the XML way.

In this post I will be covering the CSV list method. The comma separated values list method requires the following main ingredient: a user defined function that can be used to split text on the defined delimiter. Since this post is all about demonstrating the comma separated values list approach, I will not delve into the internal mechanics of such function. I just grabbed the first function that I could get off of the internet, and made it work.

I got the split function from the following url: SQL Server Forums - Best split function courtesy of the member Corey alias Seventhnight.

The technique works on the following strategy:

1. Parse the information that needs to be stored in the database in the form of a CSV list.

2. Pass it to the database packed in an appropriate type of parameter.

3. De-serialize the information into a table variable with the help of the aforementioned splitter function.

4. Depending upon the total number of columns in the target table, run a combination of insert and update commands, fetching the information from the table variable. A combination is made this way in order to minimize the disk activity, and complete the operation in as short duration as possible.

To demonstrate I will be using a very simple data model consisting of three tables: Student, Course and an association table supporting many-to-many relationship between Student and Course, StudentCourse. The Entity-Relationship diagram will clarify the relationship between the tables:

According the ER diagram, a student can take many courses and a course can be taken by many students. The insertion of courses that a student is interested in the association table is an ideal application of this technique.

The following ASP.NET webform that is the web front end that we will use to form a complete example to demonstrate this approach:

The coding for the webpage is very simple. The pseudo-code is as follows:

1. Select a student from the drop down.

2. Select from the available courses that the student needs enrollment for.

3. Click on the submit button.

I am going to leave it up to the reader to understand the programming in the web application. It is straight forward and the domain model powering the application is a reflection of the data model depicted above.

All the magic happens in the stored procedure powering the enrollment of a student in one of the listed courses. The stored procedure is as follows:

   1:  CREATE PROCEDURE [dbo].[EnrollStudentInCourses]

   2:   

   3:  @StudentID int,

   4:   

   5:  @Courses nvarchar(max)

   6:   

   7:  AS

   8:   

   9:  BEGIN

  10:   

  11:  Declare @TempStudentCourseTable Table

  12:   

  13:  (

  14:   

  15:  StudentID int null,

  16:   

  17:  CourseID int null

  18:   

  19:  );

  20:   

  21:  Insert into @TempStudentCourseTable (CourseID)

  22:   

  23:  Select CONVERT(int,Data) from dbo.Split(@Courses,',');

  24:   

  25:  Update @TempStudentCourseTable Set StudentID = @StudentID;

  26:   

  27:  Insert into dbo.StudentCourse (StudentID,CourseID)

  28:   

  29:  Select StudentID,CourseID from @TempStudentCourseTable;

  30:   

  31:  END

  32:   

  33:  GO

  34:   

The pseudo-code for the stored procedure is as follows:

1. There are two parameters in the stored procedure – The student ID of the student, who needs to be enrolled, and the courses passed as an nvarchar(max) parameter representing a CSV list of the courses in which the enrollment needs to be done.

2. The procedure uses a table variable that mimics the table structure of the StudentCourse table, sans the primary key.

3. Insert command is carried out on the table variable, with the CSV list comprised of the courses passed to the splitter function. Since out splitter function returns a table, with two columns – ID (int type identity column), and Data (nvarchar type), we can right away leverage the Data field, and insert it into the table variable by carrying out required conversion on the fly. Here I am converting the Data to integer, since CourseID is of type int.

4. Once the insertion has been done, I update the StudentID field in the table variable with the passed parameter value.

5. Once this is complete, I ran the Insert…Select… command to key in data into the final StudentCourse table. Depending upon the settings in the database, this will be a minimally logged operation, thereby giving you speed benefits.

Note that this is an overly simple example of the stored procedure, not employing transaction, error handling etc. and is meant to convey the concept of usage of comma separated values list in inserting multiple rows.

Emerging pattern:

If you look at this and carry it forward to more complicated use cases, then you’ll see a pattern emerging out. The pattern is:

1. Model the table variable after the final table that is supposed to receive the data.

2. Insert the most variable information in the table variable first.

3. Follow up the insertion with updation of table variable with information that is not changing much, like the StudentID in this example. It remains constant for all the courses inserted into the table variable.

4. If you need selective action within the table variable, make use of an integer type identity column. In this example, we did not require that.

5. The complexity of the procedure will depend upon the total columns in the final table. The more the columns, the more the number of update commands. There will also be a loop that you will have to manage in order to update the table variable with the right row form the table emitted by the splitter function.

In the next post I will cover how to insert multiple rows in SQL Server database using XML.

The sample code for this post consists of the web application and the powering database. Download it from:

Key take away:

A Microsoft Reporting Services report needs a strongly typed dataset to work with, but it is generally overridden with an in-code dataset that is used to power the report. The report is sensitive to the name of the in-code dataset, and only works if provided with same name as what gets recorded in the XML innards of the report.

Read on:

Recently I got an opportunity to help a team shift their reports from Microsoft Reporting Services (MRS) to SQL Server reporting Services. Now I had left working with Microsoft Reporting Services back in 2010, when my team migrated all the reports to hosted SSRS environment, so I spent some time creating a simple yet effective proof of concept ASP.NET application working with a Microsoft Reporting Services report that I can show to the team in transition, on how import MRS reports in hosted SSRS environment. Nearly two-third of my time got spent in dealing with a quirk in Microsoft Reporting Services, which is providing the correct name of the dataset in the code, as the report wants it. I remembered this quirk from my days of working with Microsoft Reporting Services, but had forgotten the specifics of remedying the problem.

Microsoft Reporting Services is a port of SSRS that produces report(s) you can store in an ASP.NET application, along with the app contents and use the computing resources of the web application server to process data and serve the rendered view to the client. SSRS reports are bit more capable, easier to work with and can be offloaded to a dedicated report server, and thus computing resources (compute cycles, memory, bandwidth etc.) be saved on your primary web application server. Microsoft Reporting Services reports are a good choice when your reporting needs are simple, report generation is less frequent, and when on a shoe string budget. Otherwise, try to always opt for hosted SSRS environment.

Figure 1. Setup in Microsoft Reporting Services compared with hosted SQL Server Reporting Services (SSRS)

Microsoft Reporting Services requires a strongly typed dataset to work with, and process data from it. For SSRS the requirements are the same, but the process of making and dealing with a strongly typed dataset is easy and somewhat hidden. Both the types of reports use Report Definition Language (RDL), a format used to convey the report contents and related artifacts. RDL in turn is based on XML, which means that RDL is case sensitive and any change in the schema can potentially render your report unusable. SSRS reports inherit the extension of “.rdl” while Microsoft Reporting Services reports end with “.rdlc” extension, with “c” in “.rdlc” denoting client, i.e. processing at application server.

Now that we have understood a little background about the Microsoft Reporting Services, let’s understand the quirk.

Almost always, when developers deal with Microsoft Reporting Services reports, they make a report with a very simple strongly typed dataset; connect artifacts on the report with this simple dataset (tablix, matrix, charts etc.), but then override this dataset in their application code, containing data table(s) filled with data from the actual query that essentially produces the same columns, but different data qualitatively. I usually do this because strongly typed datasets cannot be de-bugged, and they are generally more time consuming to deal with than actual verbose code. Once the functionality to override the dataset is in place the strongly typed dataset can be removed entirely. But till the overriding facility is not in place, you have to retain the strongly typed dataset in place, otherwise the report won’t build.

Here I present a simple example consisting of three tables in a data model. The tables are self-explanatory in nature. I have filled the tables up with some junk data. I will use these to make a report in an ASP.NET Webforms website, and will show where one needs to be careful in producing code that over-rides the strongly typed dataset powering the report.

Figure 2 Data model

Use case: I might bind a tablix on an MRS report with a simple query of “Select FirstName, MiddleName, LastName, DateOfBirth from Person”. While the actual query that I want to run is “Select FirstName, MiddleName, LastName, DateOfBirth fromPerson inner join Address on Person.PersonID = Address.PersonID where Address.State = @State” to get me details of people living in a certain state. So the columns that need to go on report are the same, but the data that will now appear is different qualitatively.

Figure 3 Dataset name appears as “DataSet1” in MRS report powering my tablix.

The moment you insert a tablix, MRS is going to prompt to create a data source containing a dataset powering the tablix. Once you finish the operation, you will find a strongly typed dataset in the App_Code folder of your ASP.NET web application.

Figure 4 Dataset name as it appears in a strongly typed dataset.

You can double click the typed dataset and get the name of the strongly typed dataset by clicking in the blue section and bringing up the properties. The name came “DatabaseDataSet” came up by default. You can change that, but what matters is what is getting registered in the report.

Figure 5 The name of the dataset as it appears in XML that makes up the report is of most importance.

Now here’s the code that you have to produce very carefully.

Make sure that when you are passing the report data source object to the report, it bears the same name as that of the dataset bound to the report, and more appropriately as found in the XML of the report. It doesn’t matter what is the name of the strongly bound dataset.

To see the name as it appears in the report’s innards, open with the inbuilt XML editor or just open it in Notepad and look for the entry with the name.

Figure 6 The name of the dataset that you provide in a report data source should be the same as in report's innards.

This is quirk is only applicable on the name of the dataset that you need to provide. The name of the data source can be anything, and so is the name of the data table that carries the actual data in the dataset.

The sample code consists of two ASP.NET websites. One consists of a strongly typed dataset, while the other one does not consists of it. Both the websites have the same report being powered by an in-code dataset. The code was made in Visual Studio 2012 and targets .NET Framework 4.5. The code can be downloaded from:

Key Takeaway:

SignalR is the newest member in the .NET ecosystem fully supported by Microsoft, offering an abstraction over the transport layer, that enables developers to make scalable applications that can have an real time asynchronous communication between server and client(s) and vice versa.

Read On

In the client server model, there are two modes of communication at play – push or pull. In pull paradigm, it is the client that pulls the information from the server. In push, it is the server that explicitly sends the information to the client. The web model is based on the pull technology, affectionately known as the request-response pattern. The client (browser) requests information from the server, and gets the response (web page, data etc.) Once this transaction is done the server merrily goes back into the state of amnesia. This pattern is the reason, why the web is stateless.

Whenever there is a requirement of data being explicitly pushed from the server, it poses a different set of problems than what the stateless nature of web is designed to solve. You then have to keep track of connections going to various clients all the while maintaining scalability and performance.

To achieve push from server this there are various kinds of hacks that are put in place, but all based on pull paradigm. The various sleights include opening a never ending connection to the server to polling the server again and again. There is even a standard term for such kinds of client-server interactions – COMET.

COMET operations fall under two kinds of categories –polling and streaming. Streaming is when the client opens up a connection to the server and keeps it open. Such kind of connection is called as a persistent connection and a characteristic of such a connection is the “Connection: keep-alive” command at the transport layer level. Persistent connection is predominantly used in web applications targeting real time data which needs to be server to a variety of versions of various browsers. One fine example is that of Google Finance. Notice that it uses persistent connection to bring in data continuously about the live condition of the financial market.

Google Finance uses persistent connection to update the webpage continuously.

Polling is the technique when the client keeps on sending requests to the server, demanding any new data that the server might have. A good example of polling is the Dropbox client application.

Dropbox client application in Windows 7 does polling.

Yet other trick involves using an iframe, which is fundamentally an HTML document embedded within a parent HTML document. This technique is primarily used to show ads dynamically within a web page and periodically pull a new one within the iframe.

Yahoo Mail’s new interface uses iframe to show advertisements.

The newest technology on the block is WebSocket protocol. WebSocket is part of HTML5 draft and is truly duplex and persistent form of connection to the server side processes. But sadly, not all the browsers and more importantly web servers support websocket protocol as of today. So if you are developing a public facing web site, you cannot truly rely on websocket yet.

So, all in all, right now we have a conundrum of technical tricks and an upcoming technology that is not uniformly supported by all client browsers and web servers, to solve the problem of serving real time data from server(s) to various client(s). Adding to the complexity is the fact that there are various kinds of clients available ranging from web browsers to sensors (that can use .NET Micro Framework) to native applications in iOS and Android ecosystems to desktop applications that require real time data pushed from a central server.

This is where SignalR comes into play and provides a unified approach to handle the requirement of asynchronous real time duplex communication between client(s) and server.

The way SignalR works is that it auto negotiates the protocol between the client and the server based on the capabilities of the pair. So if they support websocket, then that is used. If not websocket, then SignalR falls back to server sent events, if not that then forever frames and so on. So the developer is not required to worry about the protocol detection and usage and eventual graceful degradation. SignalR handles this automatically, all this while providing a uniform API to program against. SignalR is being actively developed by Damian Edwards’s team at Microsoft and they have already released the first version in the fall update (2012) of ASP.NET. SignalR scales well and already supports SQL Server, Service Bus and key-value system like Redis. Client side API is available for .NET (4.0 and 4.5), HTML in the form of a jQuery client, WPF, Silverlight 5 and Windows Phone 8.

In the next post I will make a simple application using SignalR. In the meantime please learn more about this awesome technology from the following videos:

1. Damian Edwards and David Fowler presenting basics of SignalR (Sorry video is too big to fit in here.)

2. Keeping It Realtime Conference - SignalR - Realtime on ASP.NET (Scott Hanselman & Paul Batum)

Please head out to the following links to get more information:

1. SignalR.net

2. www.asp.net/signalr

3. Source code of SignalR at GitHub