Counting in MySQL When Joins are Involved

There’s a MySQL pattern that I use fairly frequently that I want to share, both to help anyone else who might find it useful and also to find out if there’s a beter way.

Here’s an example: you have one table for users, another for posts, another for pages:

And we want to know how many posts and pages each user has:

Attempt 1: COUNT with JOIN

For the moment, lets focus on just getting the post count by user.

We might be tempted to try JOINing the two tables and using COUNT:

The problem is that when we JOIN the two tables, the results will only include users who have posts. In this case, there’s no result for Jen (user id 3) because she doesn’t have any records in the posts table.

Attempt 2: COUNT with LEFT JOIN

Ok, so if JOIN doesn’t work, lets try LEFT JOIN:

Now Jen has a count, but it’s reflecting 1 post, not 0 like we expect.

We can see why it’s broken by looking at the LEFT JOIN results without aggregating them:

Jen doesn’t have any posts, but because we’re LEFT JOINing, her users record is still included. When we then aggregate the results with GROUP BY and COUNT, MySQL sees that the results have one record so returns a count of 1.

A better way to do this (as suggested by Tom Davies) is instead of counting all records, only count post ids:

Attempt 3: SUM/IF, and LEFT JOIN

Another way we can achieve what we want (for a single table join) is to use SUM/IF:

Here we’re saying “When the post id is null, give it a 0, otherwise a 1, then sum the results” which gives us the correct count. This is actually I trick I learned from the real Jen while at Automattic :).

The problem arises when we need to join multiple tables:

Now instead of Simon (user id 2) having 2 posts, he has 4 – what happened? Again, we can look at the un-aggregated results:

The problem is that each post gets joined with each page result. Simon has 2 posts, each of which gets joined with the 2 pages, so when we use COUNT it sees 4 results and returns that amount.

The solution: Subqueries and COALESCE

Here’s how I would solve it:

To understand how it works, lets focus on counting posts. The first subquery counts how many posts each user has if they have any posts:

We can then LEFT JOIN users on this derived table:

For the users with posts, the result has the post count. For the users without posts, the result is NULL. This is where COALESCE comes into play. COALESCE takes any number of arguments and returns the first non-NULL result:

So we’re saying “If the user has a posts count, use that, otherwise use 0”.

We can then LEFT JOIN again on the pages table and do the same thing with the pages count to get the posts and pages count per user. Because each subquery only returns a max of one result per user, we don’t run into the issue we did earlier where posts get joined with pages to return the incorrect count.

Somewhat complicated, but the only way I know how to do it. If you know of a better way, please drop a comment below. Thanks!


This simpler method also works:

By counting distinct post ids and page ids, we avoid counting NULLs and also avoid counting duplicates due to joining on posts and pages.

12 thoughts on “Counting in MySQL When Joins are Involved

  1. My solution involves the use of dependent subqueries.

    select user_id,
    (select count(*) from posts where posts.user_id=users.user_id) as post_count,
    (select count(*) from pages where pages.user_id=users.user_id) as page_count
    from users;

    To test performance differences, I loaded the tables with 16,000 posts and nearly 25,000 pages. Limited testing showed nearly identical performance with this query to your query using left join to select subqueries. Your updated simpler method took over 2000 times as long (nearly 3 minutes compared to .02 seconds) to process the same data.

    Using EXPLAIN with each of the queries shows that both of your approaches involves a filesort which is avoided with my query. Adding a key to the user_id on the posts and pages tables avoids the file sort and sped up the slow query to only take 18 seconds. That is still significantly slower then the other two queries.

    I do believe my approach is a bit easier to follow. I ran across this while trying to perform a similar task with a query containing about a dozen columns. More columns also required adding to the GROUP BY portion of the query.

  2. Hey, I stumbled upon this problem today. Here is what I found.

    I came across 3 ways to solve this “puzzle”, all of which have been mentioned:
    1. left join (select count group by)
    2. select (select count where)
    3. select count(distinct) group by

    Then I benchmarked the solutions against each other (over a 50k dataset), and there is a clear winner: left join (select count group by)
    (0.1s for 1, 0.5s for 2 and 5min for 3)

    It also has a small benefit: you may add other computations, such as sum and avg, and it’s cleaner than having multiple subqueries (select count), (select sum), etc.

    TLDR: choose subqueries and COALESCE

  3. Hi, how about, for example the post have dates corresponding to every post, and for example i wanted to filter date range how can i do that? thanks.

  4. Thank you very much for this. This is exactly what i was looking for. In my case the simple version of the query left out NULL results from the other table so i wasn’t getting the full data I needed. The option with COALESCE with the sub queries worked perfectly for me.

  5. Excellent article thanks. Turns out I was grouping by wrong column (the related one instead of the main table’s). Also didn’t know about GROUP BY 1, cool feature.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s