Back to Basics

Back to Basics: Count, Count, Count, Sum or how to Count

Mladen Prajdić — Mon, 04 Feb 2008 16:24:04 GMT

Probably everyone is familiar with the Count(*) function in SQL Server.

But there seems to be a great deal of confusion amongst youngsters (SQL wise) about how all its possible options work.

Let us banish the confusion back to the dark realms where in belongs to:

DECLARE @t TABLE (val INT)
INSERT INTO @t
SELECT 1 UNION ALL
SELECT 1 UNION ALL
SELECT NULL UNION ALL
SELECT NULL UNION ALL
SELECT 4 UNION ALL
SELECT 4 UNION ALL
SELECT 5

SELECT COUNT(*) AS CountAll, -- counts all rows
       COUNT(val) AS CountAllNoNull, -- counts rows that don't contain NULL
       COUNT(DISTINCT val) AS CountDistinctNoNulls, -- counts the number of distinct values
       COUNT(*) - COUNT(val) AS CountOfNullValues -- count of NULL values in the column 
FROM   @t

There are 3 ways in which Count() works:

COUNT(*) - Counts all rows in the table

COUNT(ColumnName) - Counts all rows in the table that don't contain NULL in the specified column

COUNT(DISTINCT ColumnName) - Counts all DISTINCT rows in the table that don't contain NULL in the specified column

We can see that with the use of these 3 options we can get easily get the number of NULL values in the column.

So where does the Sum come in here? Sum can be used to get a count of practically anything with just one table scan.

Let's illustrate:

SELECT SUM(CASE WHEN val < 3 THEN 1 ELSE 0 END) AS LessThanTree,
       SUM(CASE WHEN val > 4 THEN 1 ELSE 0 END) AS MoreThanFour
       -- SUM(CASE WHEN <Any condition you can think of> THEN 1 ELSE 0 END) AS Col1
FROM   @t

To follow the primary school logic here:

If something fits the condition return 1 else return 0. The sum of all 1's returns the number of items that satisfy the desired condition.

And the condition is limited only by your business requirement.

You see there's no need to write nested select statements similar to this:

SELECT (SELECT COUNT(*) FROM MyTable WHERE <someCondition1>) AS cnt1, 
       (SELECT COUNT(*) FROM MyTable WHERE <someCondition1>) AS cnt2

Jeff has also has a good post about translating these kind of queries to case statements here.

And this might even provide a nice interview question...

How to solve problems and why is that perceived as hard?

Mladen Prajdić — Tue, 18 Dec 2007 08:48:06 GMT

it's my opinion that today we unfortunately live in a world that requires some kind of harder problem solving skills from maybe 20% of the population. And I'm being very very generous.

So putting that percentage with the question in the title would give an answer similar to: "Because not every one can do it." DOH, right? I'd say you're wrong since everyone had a problem to solve at least once in a lifetime. So what's the problem? Why is it perceived as hard? Let's take a little trip through your past.

When you were just an infant, what was your biggest problem? Getting fed, changed and burped, right? You cried for a bit and your problem was solved by your caretakers. Then you got a little older and you added playtime to the list (hopefully you could do burping by yourself, and potty training was fully successful). You played with your toys in your little world and could do wonders there. Life was simple. Then you started going to school and all hell broke loose. you started puberty and thought everyone else is an idiot. School ended more or less painless and since this is not a blog about gardening or ballet but about technology and your reading it means you probably went to a science oriented university. Unfortunately 95% of primary and secondary schools and universities aren't driven towards problem solving but towards cramming information. After you finish that and come to work to realize you know nothing about anything in the real world. You get thrown into the harsh reality with stuff coming at you from all over the place all the time. Things just get more complex with more and more parts mixing in unbelievable ways.

And herein lies the beauty of engineering. Since the dawn of time engineering has been about solving problems and not everyone thinks like an engineer. If you remember what you've learned when playing with your Lego's :) or at school at a class called Systems Theory or something similar sounding (if you paid attention and wasn't too tired from previous nights activities :)) you can see the whole concept of problem solving in basically all areas of life comes down to one simple methodology:

Modular Design also know as Black Box Design.

Every single good technical principle today in every single piece of technology that has to work well relies on that simple methodology. Remember this sentence for the rest of your life:

Solutions to big problems are built from solutions to smaller problems.

To do this well you have to have two very important skill sets:

- An ability of Visualization

- A good understanding of the tools you work with

An ability of Visualization

Visualization is necessary because without it you simply can't split your one big problem into smaller parts. Good sense of visualization leads to better sense of abstraction. Abstraction simply means lessening the problem scope. So visualization and abstraction are tied into a continuous circular reference. You visualize a solution and split it into pieces/modules/black boxes. You essentially abstract it into smaller parts.

Like this Renault Clio V6 engine on the left. Smaller parts or modules only have to perform a single task. A single unit of work. If they don't, abstract them further. This gives the ability of module reuse which lessens costs and increases robustness which in turn increases fault tolerance and the lifetime of the final product. To achieve all this a single black box/module has to be deterministic. The output must always be the same for the same input. In other words the inner state of the module is time independent.

With visualization and abstraction comes a sense of beauty. Beauty is one of the the best ways to evaluate how good a solution is. A good solution is pleasing to the senses. It's symmetrical. It simply looks good. And of course it also works good. It's a simple law of nature. Good and beautiful things work better and thus prevail.

A good ability of Visualization means you can design things that will work well, look good and generally be optimal.

A redesign process when you abstract more, keeping the functionality is called refactoring. Consequently refactored solutions are generally better since they are more beautiful which is one of the best indicators of quality.

A good understanding of the tools you work with

Understanding your tools at disposal gives you the ability to develop a solution better, faster and more elegant. You have to know the abilities and limits of your tools. You must also be open to possibilities of new tools doing a better job than the old ones. Don't be afraid of the change. Change is good. But also don't try to solve everything just with a hammer or just with a needle. Combine them.

Pretty picture for easier visualization of abstraction of visualization of abstraction of ....

Let's not kid ourselves. Solving problems isn't a piece of cake. It takes time, patience, care for details, etc... So don't be overly bothered if you don't succeed on the first try. Practice makes perfect and we always have to learn new things.

But the first thing you have to do is:

KNOW AND DEFINE THE REAL PROBLEM!

Introduction to locking in SQL Server 2005

Mladen Prajdić — Wed, 12 Dec 2007 19:36:13 GMT

I've written an article here on SQL Team about locking in SQL Server.

It's an introductory article that shows lock modes, lock granularity and lock compatibility matrix

and it will be followed by a few more advanced ones on the topic of locking.

Part 1: Introduction to locking in SQL Server 2005

Back To Basics: What is a Clustered and a Non-Clustered index (plus an appealing visual prop)

Mladen Prajdić — Tue, 18 Sep 2007 18:15:42 GMT

Indexes are a constant problem in understanding for beginners (and the "not so beginners") in the database world. And don't you just love the

hardcore mathematical explanation of B-Trees and their traversal. Personaly I much rather have visual props and a story to support an explanation.

Appealing visual props are even better. That's how this post originated. So let us begin!

Imagine you live in a pre-computer-in-every-nook-and-crane world (around 1960's :)).

You wake up one morning with a huge craving to read Agatha Christie's books. So you go to the library which has a few million books.

You walk up to the cute librarian (look left for the appealing visual prop :)) and ask her if she can show you to the Agatha Christie

books. She goes to the huge drawer (drawer #1) of little cards which hold just basic info about each book like ISBN number, author and

title and are sorted by ISBN number. She pulls out the first card, looks at its bookshelf id and takes you to the correct bookshelf.

You get your book and are on your merry way home. When you read your book you go back to the library, return

the books and ask the cute librarian (again look left for the appealing visual prop :)) to get you all the books by Agatha Christie written

in the 1951. This time she goes again to the huge drawer of little cards (drawer #2) but these are different than the previous ones. These hold the year of

publishing and the ISBN number of the book. She writes down the ISBN numbers of all books and goes back to the first huge drawer and

looks up cards that match ISBN numbers she found earlier. Again you get your books and are on your merry way home.

You read them return them, get some more books until you muster up the courage to ask the cute librarian out and you live happily ever after.

Nice story isn't it?

"So what the hell am i talking about?" you might ask. Well a library is a very good analogy for indexes. How? Let's review:

library = table

drawer with ISBN Number = Clustered index

drawer with year and ISBN Number = Non-Clustered index

cute librarian = a perk :)

So what you acctually did in this story is this:

You executed a query (i want some books with a condition) and a query processor (cute librarian) performed a Clustered index

seek on the first book hunt and a Non-Clustered index seek on the second book hunt.

It produced a shortest plan for the desired action (go to the bookshelf and return with the books) and it retured the desired data back (your books).

On both times she had to return a full row (a book) of data by locating it in the Clustered index (looking in the Drawer #1).

This operation is also known as a Bookmark Lookup and is the most expensive operation you can do with indexes short of index scan.

And how do those huge card drawers translate to clustered and non clustered index?

ISBN Number is the column on which the clustered index is defined. The books on the shelves are sorted accordingly. The bookshelf numbers

are pages of data in the table. The second drawer contains cards with year of publisihing and an ISBN Number. Year of publisihing is a Non-Clustrered index.

Each non-clustered index is connected to a Clustered index, thus the ISBN Number on the second drawer's cards.

You can see that as long as the ISBN Number is contained in the Clustered index the non clustered won't be changed.

This means that reindexing (not rebuilding) the Clustered Index won't rebuild the non clustered index as long as ISBN Number stays in

the Clustered Index.

But admit it! You don't care about indexes, you just care about the cute librarian :)