.NET (C# / VB)

Minimize a DropDownList's ViewState

Jeff Smith — Thu, 17 Apr 2008 16:21:16 GMT

Let's say you have a very large DropDownList with lots of values and text. We need to maintain ViewState in this DropDownList so that we can retrieve the selected value on a post back. Of course, this means that now the ViewState contains the data for every single value in the list, both values and text included. Even though the page itself may be fairly simple and lightweight, the result of having this simple DropDownList on the page is that the page size is quite large and the amount of data passed back and forth on a postback is very large as well.

If you have an efficient database, and/or if you are caching the data anyway, you might not mind re-loading the list items each time the page posts back to eliminate the need for the page itself to hold all of this ViewState data.

However, if you turn off ViewState on the DropDownList, you will notice that it now does not remember the selected value on post backs. The solution is very simple -- just manually set the DropDownList after re-loading it to the value from the HTML form post.

To do this, first we disable ViewState on the DropDownList (EnableViewState="False"). Then, instead of a typical PageLoad() method like this:

protected void Page_Load(object sender, EventArgs e)

{

if (!IsPostBack)

{

loadList();

// other stuff here

}

we would load the list every time the page posts back:

protected void Page_Load(object sender, EventArgs e)

{

loadList();

if (!IsPostBack)

{

// other stuff here

}

And, in that loadList() method, instead of simply doing this:

void loadList()

{

dlTest.DataSource = // get data from DB

dlTest.DataBind();

}

you also would check to see if it has a posted value:

void loadList()

{

dlTest.DataSource = // get data from DB

dlTest.DataBind();

string r = Request[dlTest.UniqueID];

if (r != null)

dlTest.SelectedValue = r;

}

That loads the DropDownList each time and ensures that the value set is what was posted back. Thus, the control will now "remember" the selected value on each postback, but without requiring any data stored in the ViewState at all.

It would also be easy to implement this logic in a control that inherits from DropDownList; say, a LightDropDownList. Techniques like this will of course work on other controls as well, such as a ListBox or a CheckedList.

With fast database connections and performance, as well as data caching, sometimes persisting data on the page is not the optimal way to go. Like always, there are different ways to skin a cat and sometimes simple little tweaks like this can have huge benefits on performance.

Update: See the comments for other options are that even easier/better. Also in the comments, Richard has provided a link to this excellent post on ViewState that I highly recommend checking out. It's long, but very informative and quite entertaining as well. Thanks for the great feedback.

DataTable, DataView and CreateDataReader

Jeff Smith — Fri, 22 Feb 2008 17:30:19 GMT

Here's something I was not aware of in .NET 2.0+ that I recently discovered.

There is a new DataTableReader class that implements IDataReader so that you can loop through an in-memory DataTable using the same interface that you would to loop through a SqlDataReader. You can create this object by using the CreateDataReader() method of a DataTable. This is great because now you can create a generic ReadFromDatabase() method for your classes that accepts an IDataReader parameter and you can create those objects from pretty much any data source, including DataTables, which wasn't as easy to do previously.

There is one catch: There is no CreateDataReader() method for the DataView class, meaning if you want to use a DataTableReader to loop through a DataTable that is sorted or filtered, you are out of luck. This was pretty disappointing -- in fact, I would think that CreateDataReader(), if only implemented in one place, makes more sense to implement in the DataView class rather than the DataTable class, since you would then get the best of both worlds.

The way to solve this is that you unfortunately need create another DataTable that contains the data loaded from your sorted and/or filtered DataView. Luckily, there is a ToTable() method in the DataView class that allows you to do this.

Here's an example of creating a DataTableReader that enumerates a DataView:

DataTable t = ... some data table here ...

t.DefaultView.Sort = ... your sort here ...

t.DefaultView.RowFilter = ... your filter here ...

DataTableReader r = t.DefaultView.ToTable().CreateDataReader();

Not ideal, but it works. The benefit of being able to use clean, generic code that accepts IDataReader objects and therefore allows you to use very efficient SqlDataReaders whenever possible makes it worth it.

Update: I have created a simple DataViewReader class here if you need this functionality. Using this will be much more efficient than creating a brand new DataTable if you would like to create an IDataReader on a DataView.

Splitting a single DataTable into Parent/Child DataTables for Hierarchical Processing (e.g., nested ASP.NET Repeater controls)

Jeff Smith — Fri, 02 Nov 2007 16:16:21 GMT

In ASP.NET, we often would like to output "grouped" data on our web pages, like this:

Customer Product     Sales
-------- ---------- -----
ABC Foundation
         Product 1   $200
         Product 2   $437
         Product 3   $523

The XLZ Company
        Product 1   $240
        Product 2   $892
        Product 3   $395

The easiest way to do this is with nested Repeater controls; one for the outer group (Customers, this case), and within that Repeater's ItemTemplate we'd have another Repeater control for the details (Products).

To use nested repeaters, you would return two separate result sets from SQL:

A "Customers" result set, listing one row per Customer to display on the report
A "Products" result set, listing one row per Product to display.

You would put both of those results into a single DataSet as DataTables, and then create a relation between the two (on the CustomerID column). Now, in our DataSet, we have a parent-child relation and each row in the parent Customers DataTable will have related child rows in the Products DataTable.

Then, in ASP.NET, we create nested repeaters: the outer repeater is bound to the Customers DataTable, and the inner repeater gets bound for each Customer row to the child rows in the Products DataTable. There are lots of good examples detailing this technique here and here, so I won't get into specifics on that in this space.

However -- what if you already have a stored procedure that returns this data in a single result set? The nested Repeater controls won't do you much good, since we need two separate (but related) DataTables. Or, what if it is inefficient to return these results separately in SQL? For example, suppose we want to output only the products for customers that have a total sales of greater than $200 for a specified time period. We'd need to process the sales transactions table twice in SQL to generate the list of Customers separate from the list of Products. Thus, our SQL becomes twice as inefficient as it needs to be to use this technique.

To accommodate those possibilities, I recently wrote a simple function in C# that takes a single DataTable and creates a related parent DataTable in the same DataSet based on specified parameters. This is useful if you only have a single result set from SQL (or any other data source) and you'd like to break that data up into two separate parent-child DataTables in order to use nested repeaters or any other type of parent-child processing on the data.

As an example, if you start with a single DataTable like this:

Products:

CustomerID CustomerName    Product     Sales
---------- --------------- ---------- -----
1           ABC Foundation Product 1   $200
1           ABC Foundation Product 2   $437
1           ABC Foundation Product 3   $523
2         The XLZ Company Product 1   $240
2         The XLZ Company Product 2   $892
2         The XLZ Company Product 3   $395

We can pass that DataTable to the CreateParentTable() function and you would end up with two related DataTables in the DataSet, like this:

Customers:

CustomerID CustomerName
---------- ---------------
1           ABC Foundation
2         The XLZ Company

Products:

CustomerID Product     Sales
---------- ---------- -----
1           Product 1   $200
1           Product 2   $437
1           Product 3   $523
2         Product 1   $240
2         Product 2   $892
2         Product 3   $395

Then we can simply bind the outer Repeater to the Customers DataTable, and then bind the inner Repeater to a call to each DataRow's GetChildRows() method to get the related products. All from a single SQL result set.

Also, notice that the function removes the CustomerName column from the Products DataTable, but leaves the CustomerID column for the relation.

Here's the function:

/// <summary>
///   Creates a parent DataTable within the DataSet using distinct rows from
///   an existing "source" DataTable, based on the column(s) specified.
///   The source table then becomes the "child" table of the newly created
///   parent. A DataRelation is also created between the parent table and the
///   child table, using the column(s) specified.
/// </summary>
/// <param name="sourceTable">
///    The source DataTable, which must be within a DataSet.
///    This source table will become the "child" table.
/// </param>
/// <param name="parentTableName">
///    This name will be assigned to the parent table once it is created.
/// </param>
/// <param name="relationColumns">
///    Specify the columns used to relate the parent table to the child table.
/// </param>
/// <param name="additionalColumns">
///    Any additional column(s) in the source table that will be extracted to the
///    parent table. These columns will be removed from the source table.
/// </param>
/// <param name="relationName">
///    The name of the relation that will be created between the parent and
///    the child table.
/// </param>
///
public static void AddParentTable(DataTable sourceTable, string parentTableName,
     string[] relationColumns, string[] additionalColumns, string relationName)
{
    DataSet dataSet = sourceTable.DataSet;

    if (dataSet == null)
    {
        throw new Exception("The source DataTable must be contained in a DataSet");
    }

    // generate the set of columns to use to create the Parent table:
    string[] cols = new string[relationColumns.Length+additionalColumns.Length];
    relationColumns.CopyTo(cols,0);
    additionalColumns.CopyTo(cols,relationColumns.Length);

    // create the parent table, copying unique rows from the Child table:
    DataTable parent = sourceTable.DefaultView.ToTable(parentTableName, true, cols);

    // add the parent table to the DataSet:
    dataSet.Tables.Add(parent);

    // remove the additional columns from the child table that were
    // copied to the parent table:
    foreach (string s in additionalColumns)
        sourceTable.Columns.Remove(s);

    // create the relation between the new parent table and the child table:
    DataColumn[] parentColumns = new DataColumn[relationColumns.Length];
    DataColumn[] childColumns = new DataColumn[relationColumns.Length];

    for (int i=0; i< relationColumns.Length;i++)
    {
        parentColumns[i] = parent.Columns[relationColumns[i]];
        childColumns[i] = sourceTable.Columns[relationColumns[i]];
    }

    // And, finally, add the relation to the parent table:
    parent.ChildRelations.Add(relationName, parentColumns, childColumns);
}

The parameters are all explained in the XML documentation. Feel free to make modifications or overloads of this function. Some good overloads might be one that accepts a single relation column name, or that accepts a parameter that determines whether or not columns are removed from the child table.

Using our example, we'd call it like this:

SqlConnection cn = new SqlConnection(<... your connection string ...>)
SqlCommand cm = new SqlCommand(<... the name of the stored proc ...>, cn);
cm.CommandType= CommandType.StoredProcedure;
cn.Open();

SqlDataAdapter da = new SqlDataAdapter(cm);
DataSet ds = new DataSet();
da.Fill(ds, "Products");

CreateParentTable(ds.Tables["Products"], "Customers", new string[] {"CustomerID"},
new string[] {"CustomerName"}, "CustomerProducts")

After that is executed, the DataSet now contains an additional "Customers" table, and a relation called "CustomerProducts" to the "Products" table. The relation is on "CustomerID", and the "Customers" table also includes the "CustomerName" column.

Thus, if we had nested repeaters in our ASP.NET code like this:

<table>
    <tr>
        <th>Customer</th>
       <th>Product</th>
       <th>Sales</th>
    </tr>
<asp:Repeater runat="server" ID="rpt1">
    <ItemTemplate>
       <tr>
            <td colspan="3"><%# ((System.Data.DataRowView) Container.DataItem)["CustomerName"] %></td>
       </tr>
       <asp:Repeater runat="server" ID="rpt2"
        DataSource='<%# ((System.Data.DataRowView) Container.DataItem).Row.GetChildRows("CustomerProducts") %>'>
            <ItemTemplate>
            <tr>
                <td></td>
                <td> <%# ((System.Data.DataRow) Container.DataItem)["Product"] %></td>
                <td> <%# ((System.Data.DataRow) Container.DataItem)["Sales"] %></td>
            </tr>
        </ItemTemplate>
       </asp:Repeater>
    </ItemTemplate>
</asp:Repeater>
</table>

we can simply assign rpt1's DataSource to our DataSet, set its DataMember as "Customers", call DataBind(), and we are good to go:

rpt1.DataSource = ds;
rpt1.DataMember = "Customers";
rpt1.DataBind();

A couple of more notes:

The outer Repeater's DataItems are DataRowView objects, but the inner Repeater's DataItems are DataRows. This is because we called the DataRowView's DataRow's GetChildRows() method, and not the DataRowView's CreateChildView() method. I have found that this provides much better performance. For small results, you may wish to call CreateChildView().
Instead of databinding in the HTML markup, you may wish to bind the inner Repeaters during the outer Repeater's DataBinding() event. This keeps your HTML markup a little cleaner and shorter.
This function only works in .NET 2.0 or above, since it uses ToTable() method of the DataRowView class.

For more on nested repeaters, see:

Note that all of those example use two separate SQL result sets. Using the function provided, we can do the same with a single result.

see also:

Creating CSV strings in SQL: Should Concatenation and Formatting Be Done at the Database Layer?

Jeff Smith — Tue, 09 Oct 2007 19:02:57 GMT

A question I see very often in the SQLTeam forums is how to return data in a summarized form by concatenating a list of values into a single CSV column. This can be done fairly easily in T-SQL, but as the formatting and concatenation requirements becomes more elaborate, be sure to ask yourself: Am I forcing presentation code into the database layer? read more...

Using SQLBulkCopy to copy any .NET Object to a SQL Server Table

Jeff Smith — Tue, 22 May 2007 18:40:10 GMT

In my SQLBulkCopy article, I mentioned that you can quickly copy data from anything that implements IDataReader to a SQL Server table using SQLBulkCopy (new in .NET 2.0). In this SQLTeam forum post, Jesse Hersch (jezemine) tells us that SQLBulkCopy only actually uses 3 methods of the interface. This means that it is very easy to quickly create your own custom class which implements this interface to bulk copy pretty much any type of object collection or array to a SQL Server table. He even provides code for a simple abstract SqlBulkCopyReader class that you can inherit from to implement only the necessary parts of the interface.

So, if you need to persist large amounts of data from your .net applications directly to SQL Server, and that data is not stored in DataTables, be sure to consider this option. Thanks, Jesse !

Using SQLBulkCopy to quickly transfer data from .NET to SQL Server

Jeff Smith — Mon, 07 May 2007 13:29:46 GMT

My latest article has just been posted over at SQLTeam.com:

Use SqlBulkCopy to Quickly Load Data from your Client to SQL Server

.NET Framework 2.0 introduces a very handy new class in the System.Data.SqlClient namespace called SqlBulkCopy that makes it very easy and efficient to copy large amounts of data from your .NET applications to a SQL Server database. You can even use this class to write a short .NET application that can serve as a "middleman" to move data between database servers.

If you ever need to move large amounts of data to SQL Server from a .NET application, SQLBulkCopy is the way to go. What's great about it is that you can use a populated DataTable as the source, or anything that implements IDataReader. This means that you can use SQLBulkCopy to quickly move data from any OLEDB/ODBC datasource to SQL Server.

Informal testing shows that it is about 4 times as fast as executing INSERT statements over and over when copying large amounts of data from a DataTable, and it certainly is much shorter and easier to write and maintain.

I hope you find the article and the examples given useful.

Article link

Regular Expression Replace in SQL 2005 (via the CLR)

Jeff Smith — Fri, 27 Apr 2007 20:20:43 GMT

I had to do some data clean up the other day, and really needed some regular expression replacements to do the job. Since .NET has a great RegularExpressions namespace, and since SQL 2005 allows you to integrate .NET CLR functions in your T-SQL code, I thought I'd go ahead and experiment with creating a RegExReplace() function.

read more...

Sorting Columns with the C# Pivot Function

Jeff Smith — Fri, 20 Apr 2007 16:05:42 GMT

Time for another exciting edition of the mailbag!

Maxime writes:

Hi,

First of all, your class is really nice and it handles my problem really well.

I would like to ask you something about the class. I would like to order the pivot columns by column name but I'm unable to do that yet. Do you think you could help me with that ?

Thank you in advance for your time and sorry to bother you with this question but I've tried to post in forums but didn't get any results.

Maxime

(Note: Maxime is referring to my simple Pivot function discussed here. The idea is, as always, to let SQL focus on returning raw, summarized and filtered data in a standard row/column result, and to let your very capable and efficient presentation layer handle all formatting -- which also happens to include pivoting data from rows to columns.)

Sure thing, Maxime, that's a reasonable request and a great idea for improving the code.

If you are using .NET 2.0, you can easily re-arrange columns in a DataTable, but this is not so easy in 1.1. By using the setOrdinal method of a DataColumn, we can ensure that we are creating those new columns in the proper place.

All you need to do is take the existing code for the Pivot function shown here and change just this line:

if (!tmp.Columns.Contains(s))

tmp.Columns.Add(s, dataValues.GetFieldType(pValIndex));

To:

if (!tmp.Columns.Contains(s))
{
DataColumn c = tmp.Columns.Add(s, dataValues.GetFieldType(pValIndex));
// set the index so that it is sorted properly:
int newOrdinal = c.Ordinal;
for (i = newOrdinal - 1; i >= dataValues.FieldCount - 2; i--)
if (c.ColumnName.CompareTo(tmp.Columns[i].ColumnName) < 0)
newOrdinal = i;
c.SetOrdinal(newOrdinal);
}

That should result in "sorted" pivot columns. Use similar tricks if necessary to move other columns around in any DataTable as needed. To sort the columns in descending order, change the CompareTo() method to compare with > 0. In addition, keep in mind that this is a regular text comparison; if you need to sort numeric or date column headers, you should first convert those strings to proper datatypes before doing the compare, or at the very least to a string format that compares and sorts you way you'd like.

If you are using .NET 1.1, your only options are to a) move the data from this DataTable to another one, with columns that are properly sorted as the new table is created, or b) gather the column names in advance and pass those to the function and create the DataTable with those columns already added and ready to be filled in the proper order.

I hope this helps, and congratulations on writing simple, clear, short and efficient SQL code while letting your presentation layer do its job!

see also:

ASP.NET 1.1 -- Appsettings in Web.config

Jeff Smith — Wed, 28 Mar 2007 16:09:16 GMT

It's great to be able to put settings in the Web.Config file for my ASP.NET projects. The problem for me, though, is that when I use System.Configuration.ConfigurationSettings.AppSettings(name) to return a setting that doesn't exist in the file ...

read more

Always Use Parameters. Even if you don't use Stored Procedures.

Jeff Smith — Fri, 21 Jul 2006 14:43:00 GMT

Let's assume that for some reason you are not using Stored Procedures. While I can respect your choice in that regard, that doesn't mean that you cannot still use parameters when constructing your SQL statements at the client.

read more...