Code Library - C# / VB.NET

Simple DataViewReader class that implements IDataReader for a DataView

Jeff Smith — Thu, 28 Feb 2008 15:34:22 GMT

I previously wrote that the .NET 2.0 DataTableReader class is really handy, but unfortunately there is no DataViewReader class. Thus, the only way to use the IDataReader interface with a sorted/filtered DataView was to first use the ToRows() method of the view to create a brand new DataTable, and then call CreateDataReader() on that new table. This is not the most efficient process when all you want to do is enumerate a DataView.

So, here's a solution: a simple, efficient, mostly-complete DataViewReader class. Just create a new instance of it passing in the source DataView and off you go. This will be much more efficient than creating a brand new DataTable from the DataView, since we just use the default enumerator of the DataView and do not need to copy/create any more things in memory.

Most methods are implemented but a few aren't, feel free to complete the class if you need the missing functionality. It's pretty simple and straightforward.

If you find any bugs or have any suggestions for improvement, please let me know.

public class DataViewReader : IDataReader
    {
        DataView _source;
        IEnumerator _enum;
        bool _closed = true;

        public DataViewReader(DataView source)
        {
            _source = source;
        }

        DataRowView current
        {
            get
            {
                return (DataRowView)_enum.Current;
            }
        }

        #region IDataReader Members

        public void Close()
        {
            _closed = true;
            return;
        }

        public int Depth
        {
            get { throw new Exception("The method or operation is not implemented."); }
        }

        public DataTable GetSchemaTable()
        {
            return _source.Table;
        }

        public bool IsClosed
        {
            get { return _closed; }
        }

        public bool NextResult()
        {
            return false;
        }

        public bool Read()
        {
            if (_closed)
                _enum = _source.GetEnumerator();

            _closed = false;
            return _enum.MoveNext();
        }

        public int RecordsAffected
        {
            get { throw new Exception("The method or operation is not implemented."); }
        }

        #endregion

        #region IDisposable Members

        public void Dispose()
        {
            _source.Dispose();
        }

        #endregion

        #region IDataRecord Members

        public int FieldCount
        {
            get { return _source.Table.Columns.Count; }
        }

        public bool GetBoolean(int i)
        {
            return (bool)current[i];
        }

        public byte GetByte(int i)
        {
            return (byte)current[i];
        }

        public long GetBytes(int i, long fieldOffset, byte[] buffer, int bufferoffset, int length)
        {
            throw new Exception("The method or operation is not implemented.");
        }

        public char GetChar(int i)
        {
            return (char)current[i];
        }

        public long GetChars(int i, long fieldoffset, char[] buffer, int bufferoffset, int length)
        {
            throw new Exception("The method or operation is not implemented.");
        }

        public IDataReader GetData(int i)
        {
            throw new Exception("The method or operation is not implemented.");
        }

        public string GetDataTypeName(int i)
        {
            return current[i].GetType().Name;
        }

        public DateTime GetDateTime(int i)
        {
            return (DateTime)current[i];
        }

        public decimal GetDecimal(int i)
        {
            return (Decimal)current[i];
        }

        public double GetDouble(int i)
        {
            return (Double)current[i];
        }

        public Type GetFieldType(int i)
        {
            return current[i].GetType();
        }

        public float GetFloat(int i)
        {
            return (float)current[i];
        }

        public Guid GetGuid(int i)
        {
            return (Guid)current[i];
        }

        public short GetInt16(int i)
        {
            return (short)current[i];
        }

        public int GetInt32(int i)
        {
            return (int)current[i];
        }

        public long GetInt64(int i)
        {
            return (long)current[i];
        }

        public string GetName(int i)
        {
            return _source.Table.Columns[i].ColumnName;
        }

        public int GetOrdinal(string name)
        {
            return _source.Table.Columns[name].Ordinal;
        }

        public string GetString(int i)
        {
            return (string)current[i];
        }

        public object GetValue(int i)
        {
            return current[i];
        }

        public int GetValues(object[] values)
        {
            int count = Math.Min(values.Length, FieldCount);

            for (int i = 0; i < count; i++)
                values[i] = current[i];

            return count;
        }

        public bool IsDBNull(int i)
        {
            return current[i] == DBNull.Value;
        }

        public object this[string name]
        {
            get { return current[name]; }
        }

        public object this[int i]
        {
            get { return current[i]; }
        }

        #endregion
    }

Exporting data to a remote server with SQL Express

Jeff Smith — Wed, 27 Feb 2008 20:04:45 GMT

I recently helped a friend out who only had access to SQL Server Management Studio Express, and he needed to copy a database locally from his PC to his remote web hosting company. Normally, the process is a simple backup/restore, but his hosting company does not allow restoring databases. Luckily, however, the company does allow direct access to his hosted database via client tools such as SSMS. Unluckily, SSMS Express does not include any tools that allow you to export data to a remote server.

So, here's what we did: First, we scripted out the entire database, including all tables, indexes, constraints, views, functions, procs, etc, using SSMS Express. This took some time to execute, but it was easy to do.

Then, we ran that script remotely on the hosted database to create the environment. Easy enough.

The final step was potentially the hardest part: Now we had to copy all of the data in all of the tables from the local database to the remote database. I thought about generating INSERT scripts, or even using a tool like MS Access (via linked tables) to get this done, but I decided against them because the INSERT script would be huge (there is a lot of data we are inserting) and any of these methods would require loading data in the exact order needed to satisfy FK constraints. In addition, there was the issue of IDENTITY values and handling that. All could be done, of course, but we were pressed for time and we needed something simple and efficient and that we could easily run again and again as needed.

Then, I remembered my little friend SQLBulkCopy, the really handy .NET 2.0 class that facilitates quickly copying data to a SQL Server database. I had forgotten it includes options to allow for identity values to be specified when bulk inserting, and for FK constraints to be be ignored as well. So, the order that the tables are uploaded in doesn't matter, and identity values are handled fine as well.

A few minutes later, a simple C# console app was written and we were ready to roll:

using System;
using System.Data;
using System.Collections.Generic;
using System.Data.SqlClient;
using System.Text;

namespace CopyData
{
    class Program
    {
        static void Main(string[] args)
        {
            string Src="-- source connection string here--";
            string Dest = "-- dest connection string here --";
            int BatchSize = 500;
            int NotifyAfter = 500;

            SqlBulkCopy c = new SqlBulkCopy(Dest, SqlBulkCopyOptions.KeepIdentity | SqlBulkCopyOptions.TableLock);
            SqlConnection srcConn = new SqlConnection(Src);
            srcConn.Open();

            SqlCommand cm = new SqlCommand("select * from sysobjects where type='u'");
            cm.Connection = srcConn;
            cm.CommandType = CommandType.Text;

            DataTable AllTables = new DataTable();
            SqlDataAdapter a = new SqlDataAdapter(cm);
            a.Fill(AllTables);
            SqlDataReader dr;

            int n = 0;
            int tot = AllTables.Rows.Count;

            Console.WriteLine("Found {0} Tables to copy.", tot);
            Console.WriteLine("");

            c.SqlRowsCopied += new SqlRowsCopiedEventHandler(c_SqlRowsCopied);
            foreach (DataRow r in AllTables.Rows)
            {
                Console.WriteLine("Coping table {0} of {1}: {2}", ++n, tot, r["name"]);
                cm.CommandText = String.Format("select * from [{0}]", r["name"]);
                dr = cm.ExecuteReader();
                c.BatchSize = BatchSize;
                c.DestinationTableName = r["name"].ToString();
                c.NotifyAfter = NotifyAfter;
                c.WriteToServer(dr);
                dr.Close();
            }

            srcConn.Close();
            c.Close();

            Console.WriteLine("");
            Console.WriteLine("Done .. press any key.");
            Console.ReadLine();
        }

        static void c_SqlRowsCopied(object sender, SqlRowsCopiedEventArgs e)
        {
            Console.WriteLine(" -- copied {0} rows ...", e.RowsCopied);
        }
    }
}

That's pretty much it... The app runs quicky, gives a nice progress report as it goes, and can easily be modified to use command line arguments or whatever else you need to make it simple and re-usable.

So, if you are using SQL Express to develop an application, and you need to export that database to a hosted web server and a simple backup/restore is not an option, here's all you need to do:

1) Create your destination database
2) Script out all objects in your source database
3) Run the script in your destination database
4) Create the C# console application shown above, enter in your connection strings in the first few lines where indicated, and run it.

That's it! Quick, simple and easy, and really not much harder than a backup/restore would be at all ... in fact, in some ways it is even easier than FTP'ing a backup to your hosted server and dealing with all that.

If you are you using Visual Web Developer to write your application, and therefore don't have the ability to write console applications like the one shown to do your export, you can do this by downloading Visual C# Express.

Splitting a single DataTable into Parent/Child DataTables for Hierarchical Processing (e.g., nested ASP.NET Repeater controls)

Jeff Smith — Fri, 02 Nov 2007 16:16:21 GMT

In ASP.NET, we often would like to output "grouped" data on our web pages, like this:

Customer Product     Sales
-------- ---------- -----
ABC Foundation
         Product 1   $200
         Product 2   $437
         Product 3   $523

The XLZ Company
        Product 1   $240
        Product 2   $892
        Product 3   $395

The easiest way to do this is with nested Repeater controls; one for the outer group (Customers, this case), and within that Repeater's ItemTemplate we'd have another Repeater control for the details (Products).

To use nested repeaters, you would return two separate result sets from SQL:

A "Customers" result set, listing one row per Customer to display on the report
A "Products" result set, listing one row per Product to display.

You would put both of those results into a single DataSet as DataTables, and then create a relation between the two (on the CustomerID column). Now, in our DataSet, we have a parent-child relation and each row in the parent Customers DataTable will have related child rows in the Products DataTable.

Then, in ASP.NET, we create nested repeaters: the outer repeater is bound to the Customers DataTable, and the inner repeater gets bound for each Customer row to the child rows in the Products DataTable. There are lots of good examples detailing this technique here and here, so I won't get into specifics on that in this space.

However -- what if you already have a stored procedure that returns this data in a single result set? The nested Repeater controls won't do you much good, since we need two separate (but related) DataTables. Or, what if it is inefficient to return these results separately in SQL? For example, suppose we want to output only the products for customers that have a total sales of greater than $200 for a specified time period. We'd need to process the sales transactions table twice in SQL to generate the list of Customers separate from the list of Products. Thus, our SQL becomes twice as inefficient as it needs to be to use this technique.

To accommodate those possibilities, I recently wrote a simple function in C# that takes a single DataTable and creates a related parent DataTable in the same DataSet based on specified parameters. This is useful if you only have a single result set from SQL (or any other data source) and you'd like to break that data up into two separate parent-child DataTables in order to use nested repeaters or any other type of parent-child processing on the data.

As an example, if you start with a single DataTable like this:

Products:

CustomerID CustomerName    Product     Sales
---------- --------------- ---------- -----
1           ABC Foundation Product 1   $200
1           ABC Foundation Product 2   $437
1           ABC Foundation Product 3   $523
2         The XLZ Company Product 1   $240
2         The XLZ Company Product 2   $892
2         The XLZ Company Product 3   $395

We can pass that DataTable to the CreateParentTable() function and you would end up with two related DataTables in the DataSet, like this:

Customers:

CustomerID CustomerName
---------- ---------------
1           ABC Foundation
2         The XLZ Company

Products:

CustomerID Product     Sales
---------- ---------- -----
1           Product 1   $200
1           Product 2   $437
1           Product 3   $523
2         Product 1   $240
2         Product 2   $892
2         Product 3   $395

Then we can simply bind the outer Repeater to the Customers DataTable, and then bind the inner Repeater to a call to each DataRow's GetChildRows() method to get the related products. All from a single SQL result set.

Also, notice that the function removes the CustomerName column from the Products DataTable, but leaves the CustomerID column for the relation.

Here's the function:

/// <summary>
///   Creates a parent DataTable within the DataSet using distinct rows from
///   an existing "source" DataTable, based on the column(s) specified.
///   The source table then becomes the "child" table of the newly created
///   parent. A DataRelation is also created between the parent table and the
///   child table, using the column(s) specified.
/// </summary>
/// <param name="sourceTable">
///    The source DataTable, which must be within a DataSet.
///    This source table will become the "child" table.
/// </param>
/// <param name="parentTableName">
///    This name will be assigned to the parent table once it is created.
/// </param>
/// <param name="relationColumns">
///    Specify the columns used to relate the parent table to the child table.
/// </param>
/// <param name="additionalColumns">
///    Any additional column(s) in the source table that will be extracted to the
///    parent table. These columns will be removed from the source table.
/// </param>
/// <param name="relationName">
///    The name of the relation that will be created between the parent and
///    the child table.
/// </param>
///
public static void AddParentTable(DataTable sourceTable, string parentTableName,
     string[] relationColumns, string[] additionalColumns, string relationName)
{
    DataSet dataSet = sourceTable.DataSet;

    if (dataSet == null)
    {
        throw new Exception("The source DataTable must be contained in a DataSet");
    }

    // generate the set of columns to use to create the Parent table:
    string[] cols = new string[relationColumns.Length+additionalColumns.Length];
    relationColumns.CopyTo(cols,0);
    additionalColumns.CopyTo(cols,relationColumns.Length);

    // create the parent table, copying unique rows from the Child table:
    DataTable parent = sourceTable.DefaultView.ToTable(parentTableName, true, cols);

    // add the parent table to the DataSet:
    dataSet.Tables.Add(parent);

    // remove the additional columns from the child table that were
    // copied to the parent table:
    foreach (string s in additionalColumns)
        sourceTable.Columns.Remove(s);

    // create the relation between the new parent table and the child table:
    DataColumn[] parentColumns = new DataColumn[relationColumns.Length];
    DataColumn[] childColumns = new DataColumn[relationColumns.Length];

    for (int i=0; i< relationColumns.Length;i++)
    {
        parentColumns[i] = parent.Columns[relationColumns[i]];
        childColumns[i] = sourceTable.Columns[relationColumns[i]];
    }

    // And, finally, add the relation to the parent table:
    parent.ChildRelations.Add(relationName, parentColumns, childColumns);
}

The parameters are all explained in the XML documentation. Feel free to make modifications or overloads of this function. Some good overloads might be one that accepts a single relation column name, or that accepts a parameter that determines whether or not columns are removed from the child table.

Using our example, we'd call it like this:

SqlConnection cn = new SqlConnection(<... your connection string ...>)
SqlCommand cm = new SqlCommand(<... the name of the stored proc ...>, cn);
cm.CommandType= CommandType.StoredProcedure;
cn.Open();

SqlDataAdapter da = new SqlDataAdapter(cm);
DataSet ds = new DataSet();
da.Fill(ds, "Products");

CreateParentTable(ds.Tables["Products"], "Customers", new string[] {"CustomerID"},
new string[] {"CustomerName"}, "CustomerProducts")

After that is executed, the DataSet now contains an additional "Customers" table, and a relation called "CustomerProducts" to the "Products" table. The relation is on "CustomerID", and the "Customers" table also includes the "CustomerName" column.

Thus, if we had nested repeaters in our ASP.NET code like this:

<table>
    <tr>
        <th>Customer</th>
       <th>Product</th>
       <th>Sales</th>
    </tr>
<asp:Repeater runat="server" ID="rpt1">
    <ItemTemplate>
       <tr>
            <td colspan="3"><%# ((System.Data.DataRowView) Container.DataItem)["CustomerName"] %></td>
       </tr>
       <asp:Repeater runat="server" ID="rpt2"
        DataSource='<%# ((System.Data.DataRowView) Container.DataItem).Row.GetChildRows("CustomerProducts") %>'>
            <ItemTemplate>
            <tr>
                <td></td>
                <td> <%# ((System.Data.DataRow) Container.DataItem)["Product"] %></td>
                <td> <%# ((System.Data.DataRow) Container.DataItem)["Sales"] %></td>
            </tr>
        </ItemTemplate>
       </asp:Repeater>
    </ItemTemplate>
</asp:Repeater>
</table>

we can simply assign rpt1's DataSource to our DataSet, set its DataMember as "Customers", call DataBind(), and we are good to go:

rpt1.DataSource = ds;
rpt1.DataMember = "Customers";
rpt1.DataBind();

A couple of more notes:

The outer Repeater's DataItems are DataRowView objects, but the inner Repeater's DataItems are DataRows. This is because we called the DataRowView's DataRow's GetChildRows() method, and not the DataRowView's CreateChildView() method. I have found that this provides much better performance. For small results, you may wish to call CreateChildView().
Instead of databinding in the HTML markup, you may wish to bind the inner Repeaters during the outer Repeater's DataBinding() event. This keeps your HTML markup a little cleaner and shorter.
This function only works in .NET 2.0 or above, since it uses ToTable() method of the DataRowView class.

For more on nested repeaters, see:

Note that all of those example use two separate SQL result sets. Using the function provided, we can do the same with a single result.

see also:

Regular Expression Replace in SQL 2005 (via the CLR)

Jeff Smith — Fri, 27 Apr 2007 20:20:43 GMT

I had to do some data clean up the other day, and really needed some regular expression replacements to do the job. Since .NET has a great RegularExpressions namespace, and since SQL 2005 allows you to integrate .NET CLR functions in your T-SQL code, I thought I'd go ahead and experiment with creating a RegExReplace() function.

read more...

Sorting Columns with the C# Pivot Function

Jeff Smith — Fri, 20 Apr 2007 16:05:42 GMT

Time for another exciting edition of the mailbag!

Maxime writes:

Hi,

First of all, your class is really nice and it handles my problem really well.

I would like to ask you something about the class. I would like to order the pivot columns by column name but I'm unable to do that yet. Do you think you could help me with that ?

Thank you in advance for your time and sorry to bother you with this question but I've tried to post in forums but didn't get any results.

Maxime

(Note: Maxime is referring to my simple Pivot function discussed here. The idea is, as always, to let SQL focus on returning raw, summarized and filtered data in a standard row/column result, and to let your very capable and efficient presentation layer handle all formatting -- which also happens to include pivoting data from rows to columns.)

Sure thing, Maxime, that's a reasonable request and a great idea for improving the code.

If you are using .NET 2.0, you can easily re-arrange columns in a DataTable, but this is not so easy in 1.1. By using the setOrdinal method of a DataColumn, we can ensure that we are creating those new columns in the proper place.

All you need to do is take the existing code for the Pivot function shown here and change just this line:

if (!tmp.Columns.Contains(s))

tmp.Columns.Add(s, dataValues.GetFieldType(pValIndex));

To:

if (!tmp.Columns.Contains(s))
{
DataColumn c = tmp.Columns.Add(s, dataValues.GetFieldType(pValIndex));
// set the index so that it is sorted properly:
int newOrdinal = c.Ordinal;
for (i = newOrdinal - 1; i >= dataValues.FieldCount - 2; i--)
if (c.ColumnName.CompareTo(tmp.Columns[i].ColumnName) < 0)
newOrdinal = i;
c.SetOrdinal(newOrdinal);
}

That should result in "sorted" pivot columns. Use similar tricks if necessary to move other columns around in any DataTable as needed. To sort the columns in descending order, change the CompareTo() method to compare with > 0. In addition, keep in mind that this is a regular text comparison; if you need to sort numeric or date column headers, you should first convert those strings to proper datatypes before doing the compare, or at the very least to a string format that compares and sorts you way you'd like.

If you are using .NET 1.1, your only options are to a) move the data from this DataTable to another one, with columns that are properly sorted as the new table is created, or b) gather the column names in advance and pass those to the function and create the DataTable with those columns already added and ready to be filled in the proper order.

I hope this helps, and congratulations on writing simple, clear, short and efficient SQL code while letting your presentation layer do its job!

see also:

.NET CrossTabs versus SQL Server CrossTabs

Jeff Smith — Thu, 12 May 2005 14:01:00 GMT

As promised in my last post, here is some performance testing to help you determine the performance benefits (if any) of performing your crosstabs at the presentation or code layer, as opposed to forcing SQL Server to do this.

read more...

.NET CrossTabs: Transforming a Normalized DataReader into a Pivoted DataTable

Jeff Smith — Wed, 11 May 2005 19:49:00 GMT

In my last post, I spoke briefly about how I felt that in general crosstabbing data is something that a presentation layer should do and not the database. Consider the result of a crosstab operation -- the columns returned will vary depending on the data. There is pretty much no further manipulation you can do with that result in T-SQL; in the relational database world, the column names of our database objects should be constants and not continually changing as the data changes. Also, in T-SQL there is no easy way to dynamically pivot data, and even doing it with hard-coded pivot values results in long, complicated SELECT statements. So, in my opinion there is really is no benefit of doing this transformation within SQL Server. (Unless you are in the mood for some fun dynamic sql and cursors ...)

Almost all modern report designers allow you to quickly and easily crosstab a standard recordset from SQL Server, so be sure to learn how to use this feature and take advantage of it. Some are more flexible than others, but for the most part they work as you'd expect. Either way, for report-writing crosstabing data at the presentation layer should not be a concern.

The main issue is programmers who are writing their own presentation layer. Many programmers have difficulty writing code to do the transformation; I believe this may be why it is common to be tempted to force SQL Server to do all of the work.

Here's a quick example of an easy way to implement a crosstab at the presentation layer using C#. To keep things short and simple, I have just written a small static function that accepts any object that implements IDataReader and returns a crosstabbed DataTable object. The code for the function is here.

All you need to do is open up an IDataReader object that returns the data you wish to Pivot. The data should be fully summarized by SQL Server already -- i.e., you should GROUP BY your pivot column plus any other columns you wish to return, and you should aggregate your pivot value accordingly. The DataReader should also be sorted so that all rows which will be pivoted into one are sorted together.

For example, consider the Orders table in the Northwind database. Suppose you want to display CompanyNames as rows, ProductNames as columns, and the total quantity of orders per CompanyName per ProductName in your crosstab. The DataReader's recordset should be grouped by CompanyName and ProductName, it should include a SUM(Qty) calculation, and it should be sorted by CompanyName. (see the example for this exact scenario using Northwind).

What's nice is, this is all standard SQL and requires very little effort in terms of writing the SELECT and for the server to process and return the results. We are letting the presentation layer handle the formatting. Another benefit is that we can use this technique with any datasource that ADO.NET can connect to, not just SQL Server. (In fact, we can do this with anything that implements IDataReader)

The arguments for the Pivot() function are as follows:

public static DataTable Pivot (IDataReader dataValues, string keyColumn, string pivotNameColumn, string pivotValueColumn)

dataValues -- this is any open DataReader object, ready to be transformed and pivoted into a DataTable. As mentioned, it should be fully grouped, aggregated, sorted and ready to go.
keyColumn -- This is the column in the DataReader which serves to identify each row. In the previous example, this would be CustomerID. Your DataReader's recordset should be grouped and sorted by this column as well.
pivotNameColumn -- This is the column in the DataReader that contains the values you'd like to transform from rows into columns. In the example, this would be ProductName.
pivotValueColumn -- This is the column that in the DataReader that contains the values to pivot into the appropriate columns. For our example, it would be Qty, which has been defined in the SELECT statement as SUM(Qty).

And, that's it. You just call the Pivot() function on any object that implements IDataReader, specify the pivot information, and a DataTable is returned that you can use however you like. The example uses this to fill up a DataGrid so you can see the results on a form.

Stay tuned for a performance comparison between this method and doing the crosstab transformation on the SQL Server .... You might be surprised by the results.