Git is a distributed version control system that allows data engineering teams to track changes in code and collaborate effectively. It provides a full history of modifications, enabling easy rollbacks and parallel development. Key benefits include:

• Version Tracking: Every change is saved as a commit with metadata, making it easy to audit and revert changes when needed.

• Collaboration: Multiple engineers can work on different features or bug fixes simultaneously via branches, later merging their work without overwriting each other’s code.

• Reproducibility: Git ensures that code for data pipelines, ETL scripts, or infrastructure is versioned, so you can reproduce past configurations exactly – critical for debugging and regulatory compliance.

Git’s advantages in version control and collaboration make it an essential tool for data engineers, helping streamline code management and sharing (Git for Data Engineers: Essential Commands | by Vijay Gadhave | Medium). By mastering Git’s command-line usage – from basic commits to advanced history editing – data engineers of all levels can efficiently manage project repositories.

Getting Started with Git: Repository Setup and Basics

Installing Git: First, install Git on your system (e.g., via package manager or the official installer). On Linux, you might use sudo apt install git, or on macOS brew install git. After installation, configure your identity:

• Configure Identity: Set your name and email for commits:

    git config --global user.name "Your Name"  
    git config --global user.email "you@example.com"
  

  This ensures your commits are labeled properly. The --global flag applies this setting for all repositories (you can omit it to set per-project).

Initializing a Repository: To start version controlling a new or existing project, navigate into the project directory and initialize Git:

• Run git init – this creates a new .git subdirectory, containing all Git metadata for the repo ( git init | Atlassian Git Tutorial ). After git init, your directory becomes a Git repository; most Git commands can then be used inside it. (If the project already had a .git folder, re-initializing with git init is safe – it won’t overwrite existing version history ( git init | Atlassian Git Tutorial ).)

Cloning a Repository: To obtain a working copy of an existing repository (for example, from GitHub or a shared network path), use git clone:

• Run git clone <repo-url> [<directory>]. This command copies an existing repository to your local machine. Internally, git clone first initializes a new repo then downloads all the data from the source repository and checks out the latest snapshot into your working directory ( git init | Atlassian Git Tutorial ). After cloning, you have the full history and the working files ready to use.

Basic Workflow – Stage and Commit Changes: Once a repo is initialized or cloned, the typical cycle of work is: edit files, stage changes, commit, and (when collaborating) push to a remote.

1. Check Repository Status: Use git status to review which files are modified, added, or removed. This helps you see what will be included in the next commit.

2. Stage Changes: Use git add <file> to add a file’s changes to the staging area (index). For example:

   • git add script.py stages modifications in script.py.

   • Use git add . to stage all changes (new, modified, deleted files) in the current directory.

3. Commit Changes: Once staged, create a commit with git commit -m "Message describing the change". A commit is a snapshot of the repository state. It’s good practice to write clear, concise commit messages.

4. View Commit History: Use git log to view the history of commits. By default, it shows commits in reverse chronological order, with their hash, author, date, and message. For a one-line summary per commit, try git log --oneline.

5. Difference (Optional): To see what changes have been made (before or after staging), use git diff. For example, git diff HEAD shows differences between the working directory and the latest commit.

Each commit in Git is identified by a SHA-1 hash. You can refer to commits by these hashes (full or abbreviated) or by pointers like branch names and tags (more on these later).

Using .gitignore: In any project, you’ll have files that shouldn’t be tracked (e.g., data outputs, environment files, credentials). Create a .gitignore file listing patterns of files to exclude (for example, *.csv to ignore all CSV files). Git will ignore those files when staging changes.

Branching and Merging

(File:Basic git branching workflow (GitLab).png - Wikimedia Commons) Basic Git branching workflow: feature branches diverge from the main branch and are merged back after development. This allows parallel development without disrupting the main codebase.

What is a Branch? A branch in Git is effectively a movable pointer to a series of commits (a timeline of changes). Branches allow you to isolate work. For example, you might create a branch for a new feature or a bug fix. This way, unstable or experimental code is kept separate from the main branch (sometimes called master in older repos) until it’s ready. In Git, creating and switching branches is quick and cheap – it doesn’t copy the entire project, just points to a commit snapshot ( Git Branch | Atlassian Git Tutorial ).

• To create a new branch: git branch <branch-name>. This makes a new branch pointer at the current commit, but keeps you on the current branch.

• To switch to a branch: git checkout <branch-name>. This updates your working directory to the state of that branch’s latest commit. (In newer Git versions, you can use git switch <branch-name> for clarity when just changing branches.)

• To create and switch in one step: git checkout -b <branch-name> (or git switch -c <branch-name>). This is commonly used to start a new branch and begin working on it immediately.

For example, git checkout -b feature/data-cleanup will create a branch called “feature/data-cleanup” and check it out. Now commits you make will advance this branch, while the main branch remains untouched until you merge the feature branch back.

Why Branches? Branches enable parallel development. You can have multiple feature branches, bug-fix branches, or experiment branches all evolving independently. This isolation prevents incomplete or broken code on one branch from affecting others. When a branch’s work is complete and tested, it can be merged back into a main line (such as main or develop branch), integrating the changes.

Common Branching Strategies

Choosing a branching strategy helps define how your team collaborates and delivers code. Here are a few strategies/data engineers may encounter:

• Trunk-Based Development: All developers commit to a single long-lived branch (often main or trunk). Feature branches are short-lived; developers merge small, frequent updates back into main. This approach emphasizes continuous integration and is often considered a best practice for fast-paced DevOps environments ( Gitflow Workflow | Atlassian Git Tutorial ). It minimizes merge conflicts by integrating changes often.

• Feature Branch Workflow: Each feature or issue is developed in its own branch off of main (or off a develop branch). When the work is done, it’s merged via a pull request (PR) into the main line. This is very common – it keeps work isolated and allows code review before merging. Feature branches are usually deleted after merge to keep the repository tidy.

• Gitflow Workflow: A more complex, legacy model with multiple long-lived branches (e.g. develop, master, plus supporting branches for features, releases, and hotfixes). In Gitflow, developers integrate feature branches into a central develop branch, releases are prepared on separate release branches, and finished releases are merged into master (and back into develop) ( Gitflow Workflow | Atlassian Git Tutorial ). Gitflow was popularized by Vincent Driessen and suits projects with scheduled release cycles. However, it has fallen out of favor in modern continuous delivery practices in favor of simpler trunk-based workflows ( Gitflow Workflow | Atlassian Git Tutorial ), as Gitflow can be challenging to use with fast CI/CD pipelines.

• Forking Workflow: Common in open-source, this is where each contributor works on their personal fork (copy) of the repository, then submits changes back to the main repository via pull requests. Internally, they use branches for features, but collaboration happens through the fork rather than a shared repo. In corporate data engineering, forking may be less common than in open source, except when collaborating across organizations.

In practice, many teams adopt a hybrid of these. For example, a data engineering team might use a simple feature branch model on a trunk-based approach: everyone branches off main for each task, then integrates back into main frequently (perhaps deploying changes to an analytics pipeline continuously). Choose a strategy that fits your release cadence and collaboration style. For stable data pipeline releases, you can also use release branches or tags (discussed below) to mark production versions.

Merging Branches

Once work in a branch is complete, you’ll merge it back into a target branch (often main or a integration branch like develop). Merging takes the changes from one branch and integrates them into another. The result is that the target branch now contains all the work that was in the source branch.

Basic merge: Check out the branch you want to merge into (e.g., main), then run git merge <source-branch>. This will bring the commits from <source-branch> into the current branch. There are a couple of outcomes:

• If the current branch’s HEAD is an ancestor of the source branch’s HEAD (in other words, the target branch has no new commits since the source branched off), Git will do a fast-forward merge. In a fast-forward, no new commit is created; the HEAD pointer of the target just moves forward to the source’s latest commit. The history stays linear (no merge commit). Fast-forwards often happen when one person was working on a feature branch and nothing changed in main in the meantime.

• If the branches have diverged (both have new commits since they split), Git will perform a three-way merge and create a merge commit by default. This special commit has two parent commits (the tips of the branches being merged) and a combined set of changes. The merge commit message is auto-generated (you can edit it) to note the two branches that were merged. This merge commit keeps history non-linear but records the branch structure, which some teams prefer for traceability.

You can control merge behavior:

• To force a merge commit even if a fast-forward is possible, use git merge --no-ff <branch>. This is sometimes done to ensure a merge commit exists (for example, to keep a history of all merges or to group a feature’s commits under one merge commit).

• To squash merge (combine all of a branch’s commits into a single commit on the target), you can use git merge --squash <branch> and then commit. This doesn’t produce a merge commit with two parents; instead, it produces one new commit with the cumulative changes. Squash merging is used to keep history linear and concise (e.g., squashing “fixup” commits on feature branch into one).

After merging, it’s good practice to delete the feature branch if it’s no longer needed (git branch -d <branch>), especially if using short-lived feature branches, to reduce clutter. (The commits remain in history even if the branch label is removed.)

Merge Conflicts and Resolution

Sometimes Git cannot automatically merge changes because the same part of the same file was edited differently in the two branches – this is a merge conflict. When a conflict occurs, Git will merge what it can and mark the conflicted files for manual resolution. You’ll see in the terminal output which files have conflicts, and git status will list those files as “unmerged” or with conflicts.

What a conflict looks like: Git inserts conflict markers in the affected files to indicate the differing sections. For example, a conflicted file will contain lines like:

<<<<<<< HEAD
your code on the current branch
=======
someone else’s code on the branch being merged
>>>>>>> feature-branch

Everything between <<<<<<< HEAD and ======= is the content from the current branch, and between ======= and >>>>>>> feature-branch is the content from the other branch. Your job is to edit this file to reconcile the differences – decide what the final content should be.

Steps to resolve a conflict:

1. Identify conflicts: After a merge attempt, Git stops at conflicts. Run git status to see which files are in conflict. Open those files in a text editor or IDE.

2. Edit the files: Find the conflict markers (<<<<<<<, =======, >>>>>>>). Decide how to combine the changes. You might keep one side, the other, or a mix of both. Remove the conflict marker lines and make the file look exactly as it should after the merge.

3. Mark as resolved: After editing, stage the file with git add <file>. (Git now knows you’ve resolved that file’s conflict.)

4. Finalize the merge: Once all conflicts in all files are resolved and staged, complete the merge by running git commit (if Git didn’t auto-create a merge commit earlier) or git merge --continue (if you were in the middle of a rebase or cherry-pick conflict scenario – see rebasing below). The merge commit will be created with the changes you just made.

If you want to abort the merge during a conflict (to return to the pre-merge state), you can use git merge --abort. This will abandon the merge and leave the branch as it was.

Tip: To simplify conflict resolution, you can use graphical mergetools (git mergetool) or IDE features that present side-by-side comparisons. Tools like VSCode, GitKraken, or P4Merge can make resolving conflicts more visual.

Rebasing and History Rewriting

Rebasing is another method to integrate changes from one branch into another, by moving the base of a branch. In contrast to merging (which adds a new merge commit), rebasing reapplies your branch commits on top of another base commit, producing a linear history.

Basic Rebase: Suppose you have a feature branch that forked from main, and main has progressed (new commits) while you worked. To update your feature branch with the latest main changes, you can rebase:

git checkout feature
git rebase main

This takes the commits on feature and replays them as if you started feature at the tip of main. Technically, it finds the common ancestor of feature and main, takes the diff introduced by each commit on feature since that ancestor, and re-applies those diffs on top of the current main HEAD, one by one. The feature branch pointer is then moved to the last of these new commits ( Merging vs. Rebasing | Atlassian Git Tutorial ).

Effectively, git rebase main “moves the entire feature branch to begin on the tip of the main branch, incorporating all new commits in main. But instead of using a merge commit, rebasing re-writes the project history by creating brand new commits for each commit in the original branch” ( Merging vs. Rebasing | Atlassian Git Tutorial ).

After the rebase, your feature branch will have a new set of commits (new hashes) that include the latest from main in its history. The history becomes one straight line, as if you had developed the feature on top of the updated main all along. This yields a cleaner, linear history with no merge commits ( Merging vs. Rebasing | Atlassian Git Tutorial ).

When to rebase vs. merge: Rebasing achieves the same end result as merging (your feature has main’s updates), but the history looks different:

• A merge preserves the context of parallel development (with a merge commit joining two lines of history), whereas

• a rebase flattens it into a single line, as if work was serial.

Many teams rebase feature branches before merging to main (often via a pull request) to maintain a linear history in main. This can make tools like git log or git bisect simpler to use ( Merging vs. Rebasing | Atlassian Git Tutorial ). However, rebase is effectively changing history, which comes with caveats.

Caution – The Golden Rule of Rebasing: Do not rebase commits that exist outside your local repository. In other words, never rebase a public/shared branch. If you rebase commits that others have already pulled (for example, rebasing the main branch or any published branch), you’ll force everyone else to reconcile the rewritten history, which is error-prone and confusing ( Merging vs. Rebasing | Atlassian Git Tutorial ). Only rebase local feature branches or branches that only you work on. The golden rule: never use git rebase on public branches ( Merging vs. Rebasing | Atlassian Git Tutorial ), because rebasing replaces old commits with new ones, and anyone who has the old ones will get conflicts pushing or pulling.

Rebase Conflicts: Rebasing can also result in conflicts (similar to a merge) if your commits and the new base have touched the same lines. Git will stop at a conflict during the rebase. You resolve it by editing the file and using git add just like a merge, then continue the rebase with git rebase --continue. If things go wrong, git rebase --abort will return you to the state before the rebase started.

Interactive Rebase – cleaning up commits: A powerful feature is git rebase -i (interactive). For example, git rebase -i main (while on feature) allows you to edit the commit history of your feature branch before finishing the rebase. Git opens an editor with a list of your commits and allows actions like reordering commits, squashing multiple commits into one, editing commit messages, or dropping commits. For instance, you can squash a “fix typo” commit into an earlier commit instead of having two separate commits for clarity. Interactive rebase is great for crafting a clean history (especially before merging to main) ( Merging vs. Rebasing | Atlassian Git Tutorial ) ( Merging vs. Rebasing | Atlassian Git Tutorial ). After you finalize the changes in the editor and save, Git will apply the commits as specified. You’ll end up with a rewritten branch history (make sure this is only done on your own branch, not something others depend on).

Reset (Rewriting History): While discussing history, another related tool is git reset. This command moves the HEAD of the current branch to a specified commit, optionally updating the working directory/index:

• git reset --soft <commit>: move HEAD to <commit> but leave all changes after that commit staged (in index). Useful if you want to “uncommit” some changes but keep them ready to recommit.

• git reset --mixed <commit> (default): move HEAD to <commit>, keep changes after that commit in the working directory (unstaged). This essentially “unstages” and “uncommits” those changes.

• git reset --hard <commit>: move HEAD to <commit> and wipe out any changes in the index and working directory (all changes after that commit are lost, unless they exist somewhere else like another branch or the reflog). Use with caution, as --hard can discard work permanently if not saved elsewhere.

Reset is useful for rewriting history locally (e.g., undoing a bad commit or several commits) before pushing. But like rebase, never reset public history (especially --hard on a shared branch) – others who pulled the old history will have issues. For public fixes, prefer git revert (discussed next) which preserves history.

Reverting Commits: If you need to undo a commit that has already been pushed/shared, use git revert <commit>. This creates a new commit that inverses the changes of the specified commit (without altering history behind it). It’s the “safe” way to undo a commit because history remains chronological; you don’t remove the commit, you add a new one that says “undo that change.” This is especially relevant for production code or database migration scripts in data engineering – if a bad change was committed and pushed, revert will apply an opposite change as a new commit (which can itself be pushed). You can revert a single commit or a range of commits. Each revert will prompt for a commit message (by default, noting the commit hash being reverted).

Stashing Changes (Shelving Work in Progress)

In data engineering, you might often find yourself in the middle of developing a pipeline when something urgent comes up (e.g., a bug on another branch needs fixing). You have uncommitted changes that aren’t ready to commit or push. Git stash is a handy feature to temporarily save this work and get back to a clean working state.

What is stashing? Stashing takes your working directory changes (and by default, the staging area as well) and saves them on a stack of “stashes,” then reverts your working copy to match the HEAD commit. It’s like putting your work in a locker temporarily.

• Run git stash to stash current changes. Git will save all modified tracked files, staged changes, and delete markers, then revert your working directory to a clean state (the last commit). Your changes aren’t lost – they’re stored in the stash stack. By default, stash entries are named like “WIP on branch-name…​” with the timestamp, but you can provide a description: git stash push -m "message".

Stash example: If you have edited 3 files but need to switch to main to hotfix something, do git stash. Now your repo is clean (no edits). After fixing on main and switching back to your feature, you can apply the stash to continue where you left off.

Stash list and apply: You can have multiple stashes:

• git stash list – shows all stashed entries in a stack (indexed as stash@{0}, stash@{1}, ... with 0 being the most recent). You’ll see the message and branch info for each.

• git stash show [stash@{n}] – show what changes are in a stash (a summary diff). Add -p for a full diff.

• git stash apply <stash> – apply a stash’s changes back onto the working directory without removing it from the stash list (so it can be applied again if needed). If you omit the stash reference, it applies the latest stash.

• git stash pop <stash> – applies the stash and drops it from the stash list (applying and popping the most recent stash is the default if none specified). Use this when you’re done with that saved state.

After applying a stash, your previously saved changes are restored in your working copy (you might need to resolve conflicts if the code changed in the meantime). If a stash is no longer needed, you can drop it manually: git stash drop stash@{n} or clear all with git stash clear.

Stashing is great for keeping work in progress aside. For instance, stashing allows you to switch branches (git switch <other-branch>) without committing half-done work, and then come back later and unstash to resume. Keep in mind that stash is local to your repository (stashes aren’t pushed to remotes). It’s essentially a convenience—some workflows prefer committing to a “WIP” branch instead—but stash is quick and tidy for short-term use.

In summary: git stash "saves your local changes to a separate place and reverts your working directory to the last commit" (From Changes to Safe Keeping: Git Stash - DEV Community), letting you safely check out a different branch or pull updates. Later, git stash apply or git stash pop restores those saved changes so you can continue where you left off.

Recovering Lost Work with Reflog

Git has a safety net called the reflog (reference log) that records when heads and branch tips are updated in your local repository. Even if you lose track of a commit (say you reset a branch or a commit isn’t on any branch), the reflog allows you to find its hash and recover it.

Every time HEAD moves, Git records an entry in the reflog. This includes commits, checkouts, resets, rebases, stash operations, etc. Reflog entries are local (they’re not transferred to remotes) and expire typically after 90 days by default.

• Run git reflog to see the log of HEAD changes (it’s essentially shorthand for git reflog show HEAD) ( Git Reflog Configuration | Atlassian Git Tutorial ). The output will look like:

    abc1234 HEAD@{0}: commit: added data validation step
    98f00b2 HEAD@{1}: checkout: moving from main to feature/cleanup
    0123fed HEAD@{2}: commit: WIP commit
    ...
  

  Each line shows a reference (here HEAD) with an index {n} (0 is the latest action, then 1, 2, etc. going back in time), the commit hash at that moment, and a description of the action. You can also view reflog for specific branches: git reflog show branchName (or git reflog branchName shorthand) ( Git Reflog Configuration | Atlassian Git Tutorial ), and even the stash has a reflog (git reflog show stash).

Using the reflog, you can recover from mistakes. For example, if you accidentally reset main to an earlier commit and “lost” some commits, you can look at git reflog to find the hash of where main was before the reset (it will show a HEAD entry for the reset). You might see main@{1}: reset: moving to <old-sha>. The commit before that in reflog (main@{2} maybe) is the commit that got “lost”. You can then do git checkout <lost-commit-hash> or create a branch at that hash to recover it.

In short, reflog tracks updates to branch heads (and HEAD itself) in your repository ( Git Reflog Configuration | Atlassian Git Tutorial ). It’s extremely useful for undoing operations that seem irreversible (like an unintended hard reset or an overwritten branch). Keep in mind reflog entries expire over time (and some aggressive Git GC operations can clean them), so don’t rely on reflog for extremely long-term recovery – but for recent oopsies, it’s a lifesaver.

Tagging: Marking Specific Commits

Tags are references to specific commits, typically used to mark release points (v1.0, v2.0, etc.) or other important milestones in history. Unlike branches, tags don’t move – they always point to the same commit. There are two types of tags in Git:

• Annotated Tags: These are full Git objects stored in the repo. An annotated tag has a tagger name, email, date, and a message (like a commit message) (Tagger, Date, and Message fields missing in Webhook Payloads and the Rest API | Bitbucket Cloud | Atlassian Support). Annotated tags are created with git tag -a <tagname> -m "message" (or -s to sign with GPG). They are recommended for releases because they can contain metadata and can be signed for security.

• Lightweight Tags: These are basically just named pointers (like a branch that never moves). A lightweight tag has no extra data – it’s just a name for a commit (Tagger, Date, and Message fields missing in Webhook Payloads and the Rest API | Bitbucket Cloud | Atlassian Support). You create one with git tag <tagname> (no -a). It’s quick and simple but lacks the description and metadata.

Example:

git tag -a v1.0 -m "Release version 1.0"

This tags the current commit with an annotated tag “v1.0” with the given message. Alternatively, git tag v1.0 (no -m) would make a lightweight tag on the current commit.

Use git tag (with no arguments) to list all tags. You can also filter tags: git tag -l "v1.*" lists tags matching the pattern (e.g., all tags that start with v1.).

To see details of a tag (especially annotated tag message), use git show <tagname> which will display the tag message and the commit it points to.

Pushing Tags: By default, git push does not send tags to the remote. You must explicitly push tags:

• git push origin <tagname> to push a specific tag.

• git push origin --tags to push all tags at once.

It’s important to push tags if others need to see them (e.g., teammates or CI systems that deploy specific tags). Tag names, once pushed, are global in the repository like branch names.

Using Tags in Data Engineering: You might tag a commit that corresponds to a deployed version of a data pipeline. For example, if you release “Batch ETL pipeline v2”, you could tag that commit as pipeline-v2-prod. This makes it easy to check out exactly what code was running in production at that version, or to compare changes between versions.

Annotated vs Lightweight: “Annotated tags are full objects stored within Git’s database and contain all information (checksum, tagger, date, message). Lightweight tags are simply pointers to a commit and contain no additional data.” (Tagger, Date, and Message fields missing in Webhook Payloads and the Rest API | Bitbucket Cloud | Atlassian Support) (Tagger, Date, and Message fields missing in Webhook Payloads and the Rest API | Bitbucket Cloud | Atlassian Support) For important milestones, annotated tags are preferred due to the extra context.

If you ever need to delete a tag (for instance, a mistyped one): git tag -d <tagname> removes it locally. To remove it from the remote, you’d push a deletion: git push origin :refs/tags/<tagname>.

Working with Remote Repositories (GitHub, GitLab, Bitbucket, etc.)

In professional settings, your Git repositories are usually hosted on a remote server or service (such as GitHub, GitLab, or Bitbucket). These platforms provide a central location for your team’s Git repo and add features like pull requests, issue tracking, and CI integration.

After doing local commits, you’ll push them to the remote so others can pull them. Similarly, you pull others’ changes to update your local repository.

Adding a Remote: If you cloned a repository, the remote (often named origin) is already configured. If you started from scratch with git init, you can add a remote:

git remote add origin <repo-url>

This links the name “origin” to the repository URL (could be an HTTPS URL or SSH like git@github.com:account/repo.git). You can check remotes with git remote -v.

Pushing Changes: To share your commits, push them to a remote branch:

git push origin <branch-name>

This will send the commits on your local branch to the remote origin. If the remote branch doesn’t exist yet, Git will create it. The first time you push a new branch, you might use git push -u origin <branch> to set the “upstream tracking” – meaning your local branch will track that remote branch, so future git pull or git push commands know which branch to synchronize with by default.

For example, on a new repo:

git push -u origin main

pushes your local main branch to the remote’s main and sets it as the upstream. Next time, just git push is enough.

Fetching and Pulling: To get commits from others, there are two commands:

• git fetch origin will fetch all new commits and update remote-tracking branches (like origin/main) in your local repo, but it won’t alter your working copy or local branches. After fetching, you can inspect what’s new (with git log origin/main, for example) and then merge or rebase manually.

• git pull is essentially a shortcut that does a git fetch followed by a git merge (or rebase) of the fetched branch into your current branch. Typically, running git pull in your local main branch will fetch updates from origin and then merge origin/main into your local main (bringing you up to date).

Because git pull by default merges, some prefer to avoid potential merge commits by configuring pull to rebase (git config pull.rebase true or using git pull --rebase). This will replay your local commits on top of the fetched commits instead of a merge commit, similar to doing fetch then rebase. Use whichever strategy your team prefers.

Collaboration via Pull Requests: On platforms like GitHub/GitLab/Bitbucket, a common workflow is:

• Developer pushes a feature branch to the remote.

• They open a Pull Request (GitHub/Bitbucket term) or Merge Request (GitLab term) to merge their branch into another (e.g., into main).

• Teammates review the code via the platform, discuss, and eventually the PR is merged (often the platform will handle the merge by creating a merge commit or squashing, depending on settings).

As a data engineer, you’ll use these features to collaborate on pipelines and analytics code. Pull requests enable code review which is valuable to maintain code quality in ETL scripts or SQL transformations.

Remote Branch Tracking: When you clone, your local main is automatically set to track origin/main. You’ll see this with git branch -vv (which shows what each local branch is tracking and if it’s behind/ahead). If your local branch is tracking a remote, you can use git pull and git push without specifying the remote/branch every time.

If you have multiple remotes (say origin and another called upstream), you specify which remote in commands (e.g., git fetch upstream, git push origin feature-branch).

Tip: It’s a good habit to pull the latest changes on main before starting new work, and also before pushing, ensure your local main is up to date (fast-forward merged) with remote to avoid conflicts.

Advanced Topics and Tooling Integration

This section touches on how Git interacts with other tools and workflows common in data engineering.

Git Integration with CI/CD and Automation Tools

Jenkins (Continuous Integration server): Jenkins is a popular CI tool often used to automate builds, tests, and deployments in data pipelines. Jenkins has robust Git support – the Git plugin in Jenkins can poll repositories, fetch branches, and even push tags or code as part of a pipeline (Git - Jenkins Plugins). In a typical Jenkins pipeline (defined by a Jenkinsfile stored in the repo), you might see a step like checkout scm which clones the repository at the correct commit of a build. For data engineers, this means whenever code is pushed to a branch, Jenkins can automatically trigger jobs: run data pipeline tests, lint SQL scripts, build container images for data apps, etc. The integration is seamless – you point Jenkins to the repo and branch, and it handles the rest (it uses your Git credentials or webhooks to know when to pull new commits).

Jenkins can be configured to trigger on certain branch pushes or PR merges. For example, you might set up Jenkins to run a nightly ETL job; Jenkins will git pull the latest scripts from main before execution, ensuring the job uses the most up-to-date code. In summary, Jenkins + Git allows continuous integration: every change in Git can automatically go through a pipeline of tests/deployments. The Git plugin in Jenkins supports all fundamental operations (clone, fetch, branch, merge) needed to integrate your repository into CI workflows (Git - Jenkins Plugins).

GitHub Actions, GitLab CI, Bitbucket Pipelines: These are CI/CD systems integrated into Git hosting platforms. They all revolve around Git events. For instance, GitHub Actions can be triggered on a push or PR; the workflow will automatically checkout the repository (uses: actions/checkout@v3 in config) which grabs the code at that commit. Similarly, GitLab CI/CD is configured via a .gitlab-ci.yml in the repo – on each push, runners fetch the code and execute jobs. As a data engineer, you can leverage these to automate testing of data transformations or even deploy infrastructure as code from your repo.

Apache Airflow (Workflow Orchestration): Airflow is used to schedule and manage data pipelines (DAGs). While Airflow itself isn’t a version control system, it’s a best practice to treat your Airflow DAG files as code and manage them in Git (Version Control For Ai Airflow Dag Api | Restackio). Typically, you develop DAGs (Python scripts defining tasks and dependencies) in a repository. Benefits of using Git here include tracking changes to workflows, code reviews for DAG changes, and the ability to roll back to prior versions of pipelines if a new DAG run fails.

Many Airflow deployments integrate with Git:

• The Astronomer/Airflow Kubernetes setup can use a sidecar container to periodically git pull a repo of DAGs (using tools like git-sync). This way, updating the DAG code in Git automatically updates what Airflow runs.

• In a simpler scenario, you might manually deploy DAGs by pulling from Git, or use a CI job to deploy: e.g., a GitHub Actions workflow detects changes in the dags/ folder and pushes the updated files to your Airflow instance.

Best practices: Use branches and pull requests for DAG changes, just like any code. For instance, implement a new data pipeline on a feature branch, get it reviewed, merge to main, then your deployment mechanism pulls main to update Airflow. You can also use tags to mark stable DAG versions that correspond to production deployments (Version Control For Ai Airflow Dag Api | Restackio).

In fact, guidelines for Airflow version control mirror normal software:

• “Store your DAG files in a Git repository. This allows you to track changes, collaborate with team members, and revert to previous versions if necessary.” (Version Control For Ai Airflow Dag Api | Restackio)

• “Implement a branching strategy (feature branches for development, main branch for stable releases) to isolate changes and test them before merging into the main workflow.” (Version Control For Ai Airflow Dag Api | Restackio)

• “Use Git tags to mark stable releases of your DAGs, making it simpler to reference specific versions or roll back if issues arise.” (Version Control For Ai Airflow Dag Api | Restackio).

By following these, your data pipeline code benefits from the same rigor as application code.

GitOps and Infrastructure as Code: GitOps is an approach where Git is the single source of truth for infrastructure and deployments. In GitOps workflows, all changes to infrastructure or configuration (Kubernetes manifests, Terraform files, etc.) are made via Git commits, and automated processes apply those changes to the environment.

For example, in a Kubernetes-based data platform, you might have all your cluster configs and pipeline manifests in a Git repo. A tool like Argo CD or Flux monitors the repo and applies changes to the cluster whenever a commit is made. This means deployment is triggered by a git push, not a manual kubectl or CLI call. GitOps heavily relies on Infrastructure as Code (IaC): “GitOps uses a Git repository as the single source of truth for infrastructure definitions” (What is GitOps?) and uses pull/merge requests as the change mechanism. When changes are merged to the main config repo, a CI/CD pipeline or GitOps operator applies those changes (e.g., deploys new services, updates database config, etc.) ensuring the live environment always matches what’s in Git (What is GitOps?).

In data engineering, you can apply GitOps to things like: managing Kubernetes-based ETL services, configuration for data processing jobs, or even scheduling configurations. It brings benefits of auditability (every infra change is logged in Git history) and reliability (reducing ad-hoc manual changes). For instance, if you manage an Apache Kafka cluster config via code, any topic creation or ACL change could be done by editing a file in Git and letting an automated process enact it.

In summary, Git isn’t just for application code – it’s central to modern DataOps practices too. With Git-managed workflows, you achieve consistent, repeatable deployments for data pipelines and infrastructure, often via GitOps tooling.

Submodules (Including External Repos in Your Repo)

Sometimes a project may want to include another repository within itself – for example, a data engineering team might maintain a separate repo for common SQL scripts or a shared library that they want to use in multiple pipeline projects. Git submodules allow you to embed one repository as a sub-directory of another repository. The submodule still retains its own history and can be developed somewhat independently.

A submodule is essentially a reference to another repo at a specific commit (a snapshot) ( Git submodule | Atlassian ). Your main repo will store a pointer (the commit SHA) of the external repo that you’ve locked in. This enables you to version external code along with your project.

Adding a submodule: In the parent repository, run:

git submodule add <repo-url> <path>

This will: clone the external repo into a sub-folder (the path you specified), and record that path and the current commit of the external repo in the parent repo’s index. After adding, you’ll see a new file .gitmodules that maps submodule paths to their URLs.

For example:

git submodule add https://github.com/org/data-utils.git libs/data-utils

This creates a folder libs/data-utils in your repo containing that repository. By default, it will checkout the default branch’s HEAD of that repo at that time. Commit the new submodule (it appears like a new file with a special mode in git status). The commit in your repo doesn’t include all of the submodule’s files, just a reference to the commit ID of the sub-repo.

Working with submodules: If the submodule repository changes (you want to update to a newer commit), you go into the submodule directory (cd libs/data-utils) which is itself a Git repo. You can pull or checkout a different commit/tag there. Then go back to the main repo and you’ll see that Git recognizes the submodule’s commit changed (like it sees that “libs/data-utils” now points to a new commit). You then commit that change in the parent to update the submodule reference.

Cloning a repo with submodules: By default, git clone won’t fetch submodule contents. After cloning a repo that has submodules, you’ll see empty directories for them until you initialize them. To get them:

• Run git submodule update --init --recursive to fetch all submodules (and any nested submodules, hence --recursive) and check them out at the recorded commits.

• Alternatively, clone with --recurse-submodules flag: git clone --recurse-submodules <repo-url>, which clones and automatically initializes and checks out submodules.

Common submodule commands:

• git submodule status – lists submodules with their current commit and whether they are up-to-date or modified.

• git submodule update --remote --merge – can update submodules to the latest commit from a branch (if configured), but generally submodules stick to specific commits until manually changed.

• Each submodule is an independent git repo, so you can treat it like one: go into its folder, create branches, commit, push, etc., but remember to update the parent repo’s pointer if you want the parent to use those new commits.

When (not) to use submodules: Submodules are useful for pulling in external code without copying it, and keeping it versioned. However, they add complexity: developers need to remember to init/update them, and two repos’ workflows now intersect. For data engineers, a common use might be including a shared utils library across multiple projects, or maybe embedding a specific version of a third-party tool’s source for auditability. If the coupling isn’t strict, an alternative is to use package managers (pip, etc.) or service endpoints for shared resources. But if you need to vendor code with full history, submodules can work.

Submodules are considered an advanced feature – they can confuse new Git users. Weigh the pros/cons (ease of single-repo versus multiple) before using them ( Git submodule | Atlassian ). A quick recap: “Git submodules allow you to keep a Git repository as a subdirectory of another. They are simply a reference to another repository at a particular snapshot in time, enabling you to incorporate and track version history of external code within your project.” ( Git submodule | Atlassian )

Git Large File Storage (LFS) for Big Data

Data engineers often deal with large data files, binaries, or machine learning models. Git LFS (Large File Storage) is an extension to Git that replaces large files in your repository with lightweight pointers, while storing the actual file contents on a remote server optimized for big files. If you find yourself needing to version datasets or other large artifacts, consider using Git LFS instead of normal Git.

For example, without LFS, adding a 100MB CSV to a repo will bloat the repo and every clone must download it. With LFS, that CSV would be replaced by a small text pointer in Git, and the actual 100MB content is stored on the LFS server (which may be provided by your Git host). When you clone or pull, LFS will download the big file separately (only if needed).

Git LFS was co-developed by GitHub and Atlassian, among others, to mitigate large file issues ( Git LFS - large file storage | Atlassian Git Tutorial ). To use it:

• Install the git lfs client and run git lfs install (one-time per repo or machine).

• Track specific extensions: e.g., git lfs track "*.csv" – this adds patterns to an .gitattributes file indicating those files should use LFS.

• From then on, any new CSV you add will be stored via LFS (you’ll see a pointer file in Git instead of actual content). Pushing will push through LFS.

This keeps your repository lean and fast while still versioning large files. It’s worth noting that Git LFS requires server support (most major hosts support it, though there might be storage limits). If working with truly massive data (GBs), often it’s better to store data in proper data storage (cloud buckets, HDFS, etc.) and just keep references or scripts in Git, but LFS is a good middle-ground for moderately large files that need version tracking (like model binaries under 500MB, etc.).

These study notes have covered Git from fundamentals to advanced topics, focusing on command-line usage which is the backbone of many data engineering workflows. We saw how to set up repos, manage branching and merging strategies, handle conflicts, rewrite history carefully, stash work for flexibility, use reflog to recover, tag releases, and even bring in external code or large files. We also touched on how Git underpins collaboration platforms (GitHub/GitLab/Bitbucket) and modern workflows with CI/CD (Jenkins, Airflow) and GitOps for infrastructure.

By mastering these Git commands and concepts, a data engineer can confidently version-control pipelines, collaborate with team members, and integrate with automation tools – ensuring that data projects are reproducible, auditable, and maintainable for the long run.

Sources: Git documentation and tutorials ( git init | Atlassian Git Tutorial ) ( git init | Atlassian Git Tutorial ) ( Git Branch | Atlassian Git Tutorial ) ( Gitflow Workflow | Atlassian Git Tutorial ) ( Merging vs. Rebasing | Atlassian Git Tutorial ) (From Changes to Safe Keeping: Git Stash - DEV Community) ( Git Reflog Configuration | Atlassian Git Tutorial ) (Tagger, Date, and Message fields missing in Webhook Payloads and the Rest API | Bitbucket Cloud | Atlassian Support) (Tagger, Date, and Message fields missing in Webhook Payloads and the Rest API | Bitbucket Cloud | Atlassian Support) (Git - Jenkins Plugins) (Version Control For Ai Airflow Dag Api | Restackio) (Version Control For Ai Airflow Dag Api | Restackio) (What is GitOps?)

Data Analysis Expressions (DAX) is the formula language for the data modeling layer in Microsoft Power BI, Excel Power Pivot, and SQL Server Analysis Services (Tabular mode). DAX allows analysts to create custom calculations over their data in a relational model. It is similar in syntax to Excel formulas, but operates on tables and columns rather than individual cells. DAX formulas are used to create calculated columns, measures, and even entire tables within the data model. This enables dynamic analyses where results respond to user selections and filters (Context in DAX Formulas - Microsoft Support). In Power BI and Power Pivot, DAX is essential for building interactive reports and dashboards.

Key characteristics of DAX:

• Relational & Contextual: DAX works with tables that can be related. Calculations respect the relationships between tables (similar to a database join). The result of a DAX formula can change depending on the current context (such as filters applied in a report) (Context in DAX Formulas - Microsoft Support).

• Calculated at Query Time: Many DAX calculations (especially measures) are not materialized until needed in a visual or query. This means they can adapt to user slicers or pivot selections in real-time.

• Use Cases: Commonly used for financial aggregations, time intelligence (year-to-date totals, year-over-year comparisons), filtering data for specific conditions, creating new classifications or groupings, and performing lookups between tables in your model.

• Power BI vs Excel: The DAX language is the same in Power BI Desktop and Excel’s Power Pivot (and Analysis Services). In Excel 2013, measures were called Calculated Fields, but since Excel 2016 they are also called Measures ( Calculated Columns and Measures in DAX - SQLBI) ( Calculated Columns and Measures in DAX - SQLBI). In Power BI Desktop, you create DAX calculations using the New Measure, New Column, or New Table buttons in the modeling interface.

Tip: If you're familiar with Excel formulas, think of DAX as serving a similar purpose but for tables of data. Unlike Excel’s cell-by-cell computation, DAX computes over entire columns and tables at once, leveraging an in-memory engine for fast aggregation. Always ensure you have a well-designed data model (with lookup tables and relationships) to get the most out of DAX.

Calculated Columns

A Calculated Column is a new column added to a table in the data model whose values are defined by a DAX formula (expression). When you create a calculated column, the formula is evaluated for each row of the table, and the results are stored in the model. In other words, a calculated column uses a row context: it knows the “current row” of the table and computes a value based on that row’s data ( Row Context and Filter Context in DAX - SQLBI). Once computed, the values exist in memory just like any other column in the table.

• How to create: In Power BI Desktop, use New Column and write an expression. For example, to add a profit margin column in a Sales table:

    Profit = Sales[SalesAmount] - Sales[TotalCost]
  

  This formula runs for every row of the Sales table, subtracting cost from sales amount for that row. In Power Pivot (Excel), you would insert a new column and provide a similar formula (without the “Sales[Column] =” part).

• Context: Calculated columns inherently evaluate row by row. You can reference other columns from the same row easily by name. If you reference a column from a related table, you typically need to use functions like RELATED() (more on that in the Relationships section) because the row context doesn’t automatically traverse to lookup tables without help.

• When they calculate: Calculated column values are computed at data refresh (or when the column is first created). They do not automatically respond to slicers or filters in a report. In other words, their values are static for each row until the data is refreshed. This is a key difference from measures.

• Storage and performance: Once computed, a calculated column’s values are stored in the model, increasing the model size. Every intermediate or extra calculated column consumes memory (Measures vs Calculated Columns in DAX and Power BI). Too many calculated columns can bloat your data model. Since they are pre-calculated, they don’t add query-time overhead (no recalculation per interaction), but they do use RAM and can slow down data refresh if complex.

Example – Creating a Calculated Column
Suppose we want a category for each product indicating if it’s a high-value item (price > $100). We have a Products table with a [Price] column. We create a new column:

HighValueFlag = IF(Products[Price] > 100, "High Value", "Standard")

This will mark each product as “High Value” or “Standard” based on its own price. The result is stored in the Products table as a new field. We can use this field in our visuals like any other column (for example, count of High Value products).

Note: The row context means each row’s calculation is independent. You cannot “peek” at other rows from a calculated column without an explicit aggregation or function. For instance, in a calculated column, you can’t directly ask for the “max price in the table” unless you wrap it in a function like MAX(Products[Price]), which would ignore row context and give the same result for every row (the overall max). Generally, if you find yourself needing an aggregate in a calculated column, it might be a sign that you should use a measure instead.

When to use Calculated Columns:

• You need a fixed value per row that will be used for slicing or grouping in reports (e.g., classification, category, or an intermediate calculation that won’t change with filter context).

• The calculation is simple and necessary for the data model (for example, a concatenation of two text fields for a unique ID, or extracting a year from a date if you don’t have a separate Date table).

• Avoid using calculated columns for values that should dynamically respond to filters (those should be measures). Also avoid creating many calculated columns as a substitute for proper data transformation in Power Query or the source; if it can be done in ETL, often that’s preferable for model size.

Quick tip: It’s generally better to perform row-by-row calculations in the data source or Power Query (during data load) if possible. Use calculated columns sparingly for model-specific needs or conditional flags needed in the model. Overuse of calculated columns can lead to large models and memory pressure.

Measures

A Measure (sometimes called a calculated measure or calculation) is a DAX formula that is evaluated on the fly, in the context of the filter selections in your report (the filter context). Measures are not stored as a new column; instead, a measure is a formula saved in the model that computes a result at query time. Because of this, measures always reflect the current slicers, filters, and rows/columns of your report visualization (Measures vs Calculated Columns in DAX and Power BI). They are effectively re-calculated for each cell of a pivot table or visual.

• How to create: In Power BI Desktop, clicking New Measure lets you write a formula that usually aggregates or computes over data. For example:

    Total Sales = SUM( Sales[SalesAmount] )
  

  This measure adds up the SalesAmount for whatever filters are applied. In an Excel pivot table or Power BI visual, if you put Total Sales and segment by Year, it will show the sum for each Year (each cell’s value differs based on the year filter).

• No inherent row context: Measures don’t evaluate per row of a table by default. If you try to reference a column directly in a measure (e.g. Sales[SalesAmount] alone), DAX will throw an error because it doesn’t know which row’s value to take ( Row Context and Filter Context in DAX - SQLBI). Measures require aggregation or an expression that can work on potentially many rows. That’s why most measures use aggregator functions like SUM, AVERAGE, COUNT, etc., or more complex expressions with SUMX, CALCULATE, etc. Measures always produce a single value (scalar) for the context in which they are evaluated.

• Filter context: A critical aspect of measures is that they are evaluated in a filter context. Filter context is essentially the set of filters (from slicers, visuals, or page/report filters) that define the current subset of data for evaluation ( Row Context and Filter Context in DAX - SQLBI). Measures automatically consider any filters on their underlying tables. For example, if a report page is filtered to Region = "Europe", the Total Sales measure will yield the sum of SalesAmount only for European sales without you having to explicitly specify that in the formula. Similarly, placing fields on rows/columns of a pivot creates a filter context for each cell (the intersection of that row and column’s values) ( Row Context and Filter Context in DAX - SQLBI).

• Calculated at query time: Unlike calculated columns, measures are calculated on the fly. They don’t increase the stored data size since only the formula is stored, not the results (Measures vs Calculated Columns in DAX and Power BI). The computation happens whenever the measure is used in a visual or PivotTable. If no visual is using a measure, it doesn’t calculate at all. This makes measures very powerful and flexible – they respond instantly to user interaction, enabling truly dynamic analysis.

• Example measure calculations:

  • Basic sum: Total Quantity = SUM(Sales[Quantity])

• Calculated ratio: Avg Price = DIVIDE( [Total Sales], [Total Quantity] ) – Here we reused two measures to get an average unit price. Measures can reference other measures.

• Conditional measure: Sales (High Value Customers) = CALCULATE( [Total Sales], Customers[HighValueFlag] = "High Value" ) – (Assuming there's a HighValueFlag column in Customers). This measure uses CALCULATE to modify filter context (more on CALCULATE later), so it only sums sales for customers flagged as High Value.

Calculated Columns vs. Measures – Key Differences:

• Row-by-row vs. aggregate: Calculated columns compute a result per each row of a table (row context), whereas measures compute a result per filter context (which could be an aggregate of many rows). A measure like [Total Sales] aggregates many rows’ values into one number for whatever filters apply, but a calculated column has one result per row of the original table.

• Storage vs. on-the-fly: Calculated column results are stored in the model (using memory) and do not change until data is refreshed (Measures vs Calculated Columns in DAX and Power BI). Measures are formulas evaluated on the fly, so they don’t take up space per se and can react to user selections. A measure is stored only as an expression (similar to how a query or view works).

• Filter context availability: Measures are aware of and evaluated within the report’s filter context (slicers, rows, etc.) (Measures vs Calculated Columns in DAX and Power BI). Calculated columns are evaluated without any interactive filter context – they cannot know what the user selects on a report; they only know the data in that single row (though they can lookup related data in the model). This means measures can respond to user interactions (e.g., “sales for whatever year the user selected”), while calculated columns cannot (they are fixed once computed).

• Usage: Use calculated columns for categories or values you need as part of the data model (especially if you need to slice/filter on them). Use measures for dynamic calculations, especially anything aggregated (sum, counts, ratios, etc.) or that needs to respect user filters. In many cases, you can choose either approach for a given result, but the general guidance is to prefer measures for aggregation and business metrics, and use columns only when necessary for slicing or as intermediate building blocks.

(Measures vs Calculated Columns in DAX and Power BI) (Measures vs Calculated Columns in DAX and Power BI) summarizes these differences: a calculated column is computed for each row at data refresh (increasing model size), whereas a measure is computed at query time and is evaluated in the context of report filters and slicers (so it’s more dynamic).

Real-world example: If you wanted “Sales per Region”: You could add a calculated column on each sales transaction row for Region (via relationship) and then sum it, but that’s inefficient. Instead, create a measure that sums Sales and use Region from a related table to slice the data. The measure will automatically give the correct total per region in a chart or pivot, without needing a new stored column.

Calculated Tables

A Calculated Table is a table added to your model by writing a DAX formula that returns a table. Unlike calculated columns or measures, which live inside an existing table, a calculated table is a new table in the field list. You create them using the New Table feature in Power BI (or in the model diagram of Power Pivot) (Using calculated tables in Power BI Desktop - Power BI | Microsoft Learn). Calculated tables are useful for intermediate results, staging tables, or supporting tables like a date/calendar table generated via DAX.

• How to create: Write a DAX expression that returns a table (enclosed in curly braces for literal rows, or using functions that produce tables). For example, to combine two tables with identical structure (like appending rows):

    CombinedTable = UNION( 'Northwest Employees', 'Southwest Employees' )
  

  This creates a new table named CombinedTable with rows from both regional employee tables (Using calculated tables in Power BI Desktop - Power BI | Microsoft Learn). The calculated table will appear in the Fields list, and you can create relationships to it or use its fields just like an imported table.

• When they calculate: A calculated table is computed during the data refresh process (or when first created). The results are stored in the model. Essentially, it’s as if the data was loaded from an external source – except the source is a DAX expression based on other data already in the model. If the underlying data changes and the model is refreshed, the calculated table will recalc. Note that for DirectQuery sources, calculated tables do not update until the model itself is refreshed (they don’t dynamically reflect query changes) (Using calculated tables in Power BI Desktop - Power BI | Microsoft Learn).

• Usage examples:

  • Date Table: Many models create a Date table using DAX. For instance:

      Calendar = CALENDAR( DATE(2020,1,1), DATE(2025,12,31) )
    

    This produces a table with one date per day for the given range. You can then add calculated columns to this Calendar table for Year, Month, etc., or use built-in functions like YEAR() to create those columns. Mark this table as the official Date table in the model so time intelligence functions work properly.

• Union or filtered tables: As in the previous example, combining two or more tables (UNION), or taking a subset of a table. For instance, you could create a table of Top 100 Customers by revenue:

    TopCustomers = TOPN( 100, SUMMARIZE( Sales, Sales[CustomerID], "TotalSales", SUM(Sales[Amount]) ), [TotalSales], DESC )
  

  This DAX snippet (advanced) creates a table of 100 customer IDs with highest sales. This can be used for special reporting or further analysis.

• Snapshot or summarized data: Sometimes you might create a calculated table to store a summary (e.g., a distinct list of values with some pre-calculated measures). However, remember these are static once computed at refresh – if you want truly dynamic summaries, a measure or visual-level calculation might be more appropriate.

• Relation to data model: Calculated tables can participate in relationships just like imported ones (Using calculated tables in Power BI Desktop - Power BI | Microsoft Learn). For example, a Date table created via DAX can be related to a Sales table on the Date field. Calculated tables can also have calculated columns and measures of their own. Essentially, once created, they are normal tables in the model.

(Using calculated tables in Power BI Desktop - Power BI | Microsoft Learn) (Using calculated tables in Power BI Desktop - Power BI | Microsoft Learn) — Calculated tables let you add new tables based on data already in your model, using DAX formulas to define their content. They are best used for intermediate calculations or data that you want as part of the model (as opposed to measures which are calculated on the fly).

Best practice: Use calculated tables for supporting structures like a calendar (date) table or a table of specific reference values that you cannot easily get from the data source. If your goal is purely to display a summary in a report, often a measure or visual-level aggregation is sufficient. Calculated tables shine when you need the table for further model logic (relationships, reusable subsets, etc.). Keep in mind they increase model size and refresh time, so ensure they’re truly needed (for instance, a small static table of dates or categories is fine, but creating a huge calculated table of millions of rows might be better done in the query or source).

Relationships in DAX

Data models in Power BI/Power Pivot are built with relationships between tables (often primary key to foreign key, like a Dimensional model with dimension tables and fact tables). Understanding how DAX works with relationships is critical:

• Filter Propagation: Relationships allow filter context to flow from one table to another. For example, if you have a one-to-many relationship between a Product table (one side) and a Sales table (many side), placing a field like Product[Category] in a visual along with a measure summing Sales[Amount] will filter the Sales table to only those rows for the current category. The report’s filter context on Product filters down to Sales automatically via the relationship. This is why you typically create measures on fact tables (like Sales) and slice by dimensions (like Product or Date) – the relationships make the measure respond to dimension filters without additional coding.

• Directional context: In a one-to-many, the default filter propagation is from the one side to the many side (one -> many). Many Power BI relationships are single-direction by default. If you have a bidirectional relationship (or set cross-filter to both), filter context can propagate both ways. Use bidirectional filtering carefully, as it can introduce ambiguity or performance issues. However, it’s useful for scenarios like many-to-many relationships via a bridge table.

• Using RELATED() in calculated columns: In a calculated column, you often have a row context on a fact table and want to fetch a value from a related lookup table. DAX won’t automatically pull that value just by referencing it (unlike a VLOOKUP in Excel, you must explicitly tell DAX to traverse the relationship). The function RELATED(Table[column]) is used in a row context to get the corresponding value from the one-side (lookup) table for the current row’s relationship. For example, in a Sales table (many side) that has a relationship to Product (one side), if we want the product category name on each sales row, we could create:

    CategoryName = RELATED( Product[Category] )
  

  This will go to the Product table, find the row related to the current Sales row, and return the Category. RELATED() works when there is a single related row (which is true in a one-to-many where you’re on the many side and pulling from the one side) ( Using RELATED and RELATEDTABLE in DAX - SQLBI) ( Using RELATED and RELATEDTABLE in DAX - SQLBI).

• RELATEDTABLE() for the inverse: If you are on the one side (say Product table) and want to bring in information from the many side (Sales), you can use RELATEDTABLE(Sales) which returns a table of all sales rows related to the current product ( Using RELATED and RELATEDTABLE in DAX - SQLBI). In a calculated column scenario, RELATEDTABLE can be used inside an aggregator. For instance, in the Product table you might define:

    TotalProductSales = SUMX( RELATEDTABLE(Sales), Sales[Amount] )
  

  This will iterate over all Sales rows related to the current product (via the relationship) and sum the Amount. Essentially, RELATEDTABLE gives you the table of related rows, and then you can aggregate or count them. This is less common in measures (measures have an easier way – they naturally sum by product if you slice by product), but it’s good for calculated columns that need facts.

• Active vs Inactive relationships: Power BI allows multiple relationships between the same two tables, but only one can be active at a time (the one that propagates filters by default). The others are inactive and do nothing unless explicitly invoked. For example, you might have an Order Date and a Ship Date both linking Sales to the Date table. One (say Order Date) is active. If you want to create a measure that uses the Ship Date, you use the function USERELATIONSHIP inside a measure to activate the inactive relationship for that calculation ( Using USERELATIONSHIP in DAX - SQLBI) ( Using USERELATIONSHIP in DAX - SQLBI). For example:

    Shipped Sales = 
      CALCULATE( [Total Sales],
         USERELATIONSHIP( Sales[ShipDate], 'Date'[Date] )
      )
  

  Here, within this measure, the relationship on Sales[ShipDate] to Date is used instead of the default Sales[OrderDate] -> Date. USERELATIONSHIP is only valid inside CALCULATE (or functions that accept filter modifiers) ( Using USERELATIONSHIP in DAX - SQLBI). Once the CALCULATE is done, the active relationship reverts back. This technique is crucial for multi-date-table models (role-playing dimensions like Date).

• Cross-filtering direction: DAX also provides CROSSFILTER(table1[col], table2[col], direction) as a CALCULATE modifier to override the relationship’s filtering direction (useful for certain calculations where you temporarily need a different filter behavior). Another function, TREATAS, can apply the values of a table or column as filters on another table’s column – effectively creating a relationship on the fly for a specific measure.

Real-world perspective on relationships: If your model is well-designed with correct relationships, writing DAX measures becomes much easier, since you typically do not have to explicitly join tables. The filter context does the heavy lifting. For example, if you want “Sales for Contoso brand in 2021”, and your model has Sales -> Product -> Brand and Sales -> Date relationships, simply filtering Brand = Contoso and Year = 2021 in the report or within a CALCULATE will yield the correct sales. No manual joining in the DAX formula is needed. Understanding this allows you to trust the model and focus on what you want to calculate (sum, average, etc.) rather than how to join tables.

Tip: Always ensure you have necessary relationships in place (and the correct active one for default behaviors). Use USERELATIONSHIP in measures to utilize any secondary relationships (like multiple dates), and consider marking your Date table as a “Date Table” in Power BI which helps time intelligence and makes sure relationships on date behave as expected. Avoid modeling pitfalls like many-to-many relationships unless necessary; if you do have them, be mindful that filter context might flow in less obvious ways.

Row Context vs Filter Context

Understanding context is foundational to mastering DAX. There are two primary types of context in DAX: Row context and Filter context. The combination of these (at any point of execution) is sometimes called the evaluation context. Context determines what data a DAX expression is currently operating over.

• Row Context: This is essentially “the current row” in a table that an expression is iterating over ( Row Context and Filter Context in DAX - SQLBI). You have a row context whenever a formula is being evaluated for each row of a table. As discussed, calculated columns inherently have a row context (each row is calculated separately). Also, any iterator function (like SUMX, FILTER, AVERAGEX, etc.) creates a row context as it loops through a table ( Row Context and Filter Context in DAX - SQLBI). Within a row context, you can directly reference columns of that table, and it will understand you mean “the value of this column in the current row.” For example, in a SUMX(Sales, Sales[Quantity] * Sales[Price]), the expression Sales[Quantity] * Sales[Price] is evaluated in a row context for each row of Sales, so it multiplies quantity and price of that specific row.

  If a row context is present, you must use functions like SUMX/AVERAGEX etc. to aggregate, or use CALCULATE to introduce a filter context (more on that soon) if you want to do operations across rows. Without a row context, a naked column reference is ambiguous and errors out (e.g., a measure cannot just say Sales[Amount] by itself) ( Row Context and Filter Context in DAX - SQLBI).

  Another key point: Row context by itself does not do filtering on other tables automatically. In a calculated column, if you need a value from a related table, you still need RELATED() as discussed. Row context can propagate through relationships when using certain functions (like RELATEDTABLE which uses the current row’s context to fetch related rows ( Row Context and Filter Context in DAX - SQLBI)), but generally just having a row context on one table doesn’t filter other tables unless you explicitly tell DAX to do so.

• Filter Context: This is the set of filters applied to the data model before evaluating a DAX expression ( Row Context and Filter Context in DAX - SQLBI). Filter context comes from a few places: the user’s selections in the report (slicers, filters, the current row/column of a pivot or visual), and filters applied within DAX formulas (like filter arguments in CALCULATE, or filters from measures in filter visuals). Filter context can be thought of as “a subspace of the data” that’s currently active. For instance, if you put Year=2022 and Region=Europe as page filters in Power BI, those constitute a filter context that will affect all measures (only 2022, Europe data is visible to them). If you add a visual with Product Category on rows, each category value in a row has an additional filter (Category = that value) for that specific evaluation.

  In practical terms, when a measure is evaluated, first any relevant filters from the report are applied to the tables (this is like a WHERE clause limiting rows). Then the measure’s formula runs against that filtered data. If the measure is [Total Sales] and the filter context is Year=2022 and Region=Europe, it will sum only the sales that meet those criteria.

  Filter context can also be modified within DAX (the primary function to do this is CALCULATE, which we’ll cover next). The filter context is additive by default – multiple filters combine (they function like an AND of conditions). It’s possible to have no filter context (meaning all data is considered), such as a measure in a card with no filters applied, or you can explicitly clear filters using functions like ALL.

In summary: Row context = current row (for iterative calculations), Filter context = current filters (for aggregation). A single DAX calculation can have both contexts at play: e.g., in a SUMX, you have a row context iterating the table, and also a filter context coming from outside (like slicers on other fields). The row context applies to that inner expression, while the outer filter context restricts which rows of the table are iterated.

( Row Context and Filter Context in DAX - SQLBI) defines filter context as “the set of filters applied to the evaluation of a DAX expression.” Any pivot table cell or visual corresponds to a filter context (even if it's "all data", that's essentially an empty filter context meaning no filters). And ( Row Context and Filter Context in DAX - SQLBI) explains that a row context exists when iterating over each row in a calculated column or with an X iterator function.

Example to illustrate contexts:
Imagine a PivotTable showing Total Sales by Year and Product Category. When computing the measure for Year=2021 and Category=“Electronics”, the filter context is Year=2021 AND Category=Electronics (and any other global filters applied). The measure [Total Sales] = SUM(Sales[Amount]) will then sum only those sales that match those filters. There is no row-by-row loop in the measure; the Vertipaq engine (Power BI’s data engine) efficiently scans the pre-filtered data. Now, if we had a measure like Average Sale per Order = AVERAGEX( Sales, Sales[Amount] ), here AVERAGEX creates a row context iterating each Sale (each row in Sales table) within the current filter context. If Year=2021 and Category=Electronics filters are on, AVERAGEX will iterate only Sales rows for 2021 Electronics (filter context first), then within that, row context for each sale to pick Sales[Amount], and then average those. So filter context and row context worked together in that calculation.

Context Transition and Evaluation Context

Evaluation context is a term for the combined context that applies when an expression is evaluated – it includes any existing filter context and any row context(s) that might be in play. One of the more advanced concepts is context transition, which is how DAX converts a row context into a filter context (so that a measure or aggregation can be evaluated per row).

• Context Transition: This typically occurs when you use the CALCULATE (or CALCULATETABLE) function. When CALCULATE is called from within a row context, it will take the values of the current row’s columns and treat them as filter context for the expression inside CALCULATE (Evaluation Contexts in DAX - Context Transition). In effect, it transitions the existing row context into an equivalent set of filters. This is what allows measures to work when called in a row context. For instance, if you write a calculated column like = CALCULATE( [Total Sales] ) in a Sales table, here [Total Sales] is a measure (which normally works on filter context). By wrapping it in CALCULATE, DAX will transition the row context (the current Sales row’s values for all relevant columns, such as Product, Date, etc.) into filters, then evaluate [Total Sales] under those filters – effectively giving you the sales of just that one row (which is actually just SalesAmount itself in this trivial example). This might seem pointless in a simple case, but it’s how a measure can be evaluated per row of another table or in an iterator. In fact, calling a measure within an iterator like SUMX automatically triggers context transition even without explicitly writing CALCULATE (CALCULATE function (DAX) - DAX | Microsoft Learn) – DAX sees a measure needing filter context and converts the current row context to filters for that measure.

  Another example: Suppose you have a calculated table of customers and you want a column "Total Sales to Customer" in that table. You might do: Total Sales to Customer = CALCULATE( [Total Sales], Sales[CustomerID] = Customers[CustomerID] ). Here for each customer row, CALCULATE takes that customer’s ID and filters the Sales table to that ID, then [Total Sales] sums only those. This works because of context transition – effectively CALCULATE says "current row’s CustomerID -> apply it as a filter on Sales[CustomerID]". If you omitted CALCULATE and just tried [Total Sales] in that row context, it wouldn’t know the row context of Customers unless transitioned.

  In summary, context transition allows row context to be converted to filter context, usually via CALCULATE (Evaluation Contexts in DAX - Context Transition). This is essential for many advanced calculations.

• CALCULATE specifics: When you use CALCULATE(Expression, filters...) in general, it creates a new filter context for evaluating Expression: starting from the existing one, then applying the filters you give. If CALCULATE is invoked where there is a current row context (like inside an iterator or a calculated column), that row context becomes part of the new filter context (context transition) (CALCULATE function (DAX) - DAX | Microsoft Learn). We will discuss CALCULATE more in the next section, but it’s important to highlight its role in context transition here. Also, a measure placed in a visual cell inherently has no row context, only filter context. If that measure was defined using row-based logic, it's likely using CALCULATE internally to simulate that.

• Evaluation context: At any point, DAX evaluates an expression with possibly multiple row contexts and one filter context in effect. For example, if you have nested iterators (like a SUMX inside an AVERAGEX), you could have two row contexts (one for the inner, one for the outer) plus the global filter context. The evaluation context is the union of all active contexts. If a column reference matches a current row context, it takes that; if it’s used where no row context exists, it tries the filter context. Understanding which context applies to each part of your formula is key to debugging DAX.

One way to think about evaluation context: Filter context restricts which rows are visible to the calculation, while row context dictates the current row from those rows when needed. When you call an aggregation like SUM, it looks at all rows visible in the filter context (ignoring row context unless context transition happened). When you are in an iterator, the iterator moves row by row (that’s the row context), but inside it you can still call CALCULATE or measures to leverage filter context logic.

To reinforce context transition with a straightforward statement: “When used inside a filter context, CALCULATE performs context transition, transforming the current row context into a filter context.” (Evaluation Contexts in DAX - Context Transition) This is often on exam tests and DAX theory — it means if you are in a row context and want to do something like a measure or any calculation that needs a filter context, wrapping it in CALCULATE will turn your row’s values into filters so the calc can happen.

Advanced note: Most DAX measures you write will implicitly handle context correctly without you needing to think about context transition. It mainly comes into play when writing calculated columns that use measures, or when writing measures that iterate (like using X functions or manual filters). A common pitfall for newcomers is thinking a measure can be used directly in a calculated column and yield row-specific results – it won’t, unless you wrap it in CALCULATE. If you ever see the same value repeated on every row of a calc column when using a measure, it’s likely because context transition was not applied and the measure was evaluated in the overall filter context (probably empty) instead of per row.

CALCULATE and FILTER Functions

CALCULATE is the most powerful (and arguably most important) function in DAX. It allows you to modify the filter context under which an expression is evaluated. In other words, CALCULATE enables you to add or override filters for a calculation. This is how you perform tasks like “Sales for 2021 only” or “Number of customers in US, regardless of any country filter on the report” inside your DAX measures.

• Syntax: CALCULATE( <Expression>, <Filter1>, <Filter2>, ... ).
  The <Expression> is typically a measure or an aggregation you want to compute. The filters can be given as Boolean filter expressions (like Table[Column] = value), table expressions (often using the FILTER() function or all/values functions), or special filter modifier functions (like ALL(), REMOVEFILTERS(), USERELATIONSHIP() etc.). Each filter argument modifies the context separately: they either add a new filter or override an existing one on the specified column/table.

• Adding vs overriding filters: By default, if you specify a filter on a column that isn’t already filtered in the current context, CALCULATE will add that filter. If that column is already being filtered (say by a slicer or visual), CALCULATE will override it with the one you specified (CALCULATE function (DAX) - DAX | Microsoft Learn). For example, if a report is filtering Region = "Europe", but your measure does CALCULATE([Total Sales], Region[Country] = "USA"), the result inside that CALCULATE will act as if Region = USA (the filter on Europe is replaced for that measure). This replacement behavior is often what you want for explicit overrides. If instead you want to add a filter without removing existing ones on the same column, there’s a function KEEPFILTERS we can use (more on that in advanced tips).

• Basic usage examples:

  • Explicit filter: Sales 2021 = CALCULATE( [Total Sales], 'Date'[CalendarYear] = 2021 ). This measure will give the total sales for the year 2021, no matter what year is filtered in the report (it overrides year filter to 2021). You can list multiple filters: e.g. CALCULATE( [Total Sales], 'Date'[CalendarYear]=2021, Product[Category]="Electronics" ) to apply both Year and Category filters at once (CALCULATE function (DAX) - DAX | Microsoft Learn).

• Removing filters: All Sales = CALCULATE( [Total Sales], REMOVEFILTERS( Product[Category] ) ). This would calculate total sales ignoring any filter on Product Category (i.e., returning total across all categories even if a specific category is in context). Similarly, ALL(Table) can be used to ignore filters on a whole table or column. (ALL used inside CALCULATE is considered a filter modifier, telling CALCULATE to clear those filters rather than add a new one).

• Using CALCULATE for context transition: As mentioned earlier, if you use CALCULATE in a calculated column or within an X iterator, it will take the current row’s context and turn it into filters. For instance, a calculated column: SalesAmount_Check = CALCULATE( [Total Sales] ) in the Sales table would for each row filter Sales to that row and compute [Total Sales], effectively just returning that row’s SalesAmount (this is a trivial example of context transition).

• FILTER function: FILTER is a DAX function that returns a table which is a subset of another table, filtering rows by a condition. Its syntax: FILTER( <Table>, <Condition> ). The condition is evaluated for each row of the table, and any row that returns TRUE is kept. FILTER is an iterator (it creates a row context over the table you provide and evaluates the condition for each row) (Explanation of Context in Calculating Cumulative Values). This means if the condition itself involves aggregates or references to other tables, it will still evaluate row by row (often you might see conditions like FILTER( Sales, Sales[Amount] > 1000 && Sales[Amount] < [Some Threshold] ) etc., where it filters on Sales rows meeting certain criteria).

  By itself, FILTER is most often used as an argument to other functions that take a table. Common use cases:

  • Inside CALCULATE: e.g. CALCULATE( [Total Sales], FILTER( Sales, Sales[Amount] > 1000 ) ). Here, instead of a simple Sales[Amount] > 1000 filter (which is actually not allowed directly in CALCULATE because it’s not a simple equality), we use FILTER to apply a more complex condition. This will filter the Sales table to only rows where amount > 1000, then CALCULATE will evaluate [Total Sales] (which sums SalesAmount) under that modified context (so effectively “Sales of transactions > 1000”).

• In iterator combinations: e.g. COUNTROWS( FILTER( Customers, Customers[HasEmail] = TRUE ) ) which would count how many customers have an email. (This could also be done with a CALCULATE: CALCULATE( COUNTROWS(Customers), Customers[HasEmail] = TRUE ) – in fact, CALCULATE with simple conditions is often simpler.)

Remember that FILTER(Table, condition) returns the filtered table, but doesn’t on its own do anything to existing filter context unless wrapped in CALCULATE or used in an iterator. So you often see it paired with functions like SUMX, COUNTX, CALCULATE, etc.

• Efficiency note: If a simple filter (like Column = Value) suffices in CALCULATE, use that rather than FILTER, because CALCULATE can set simple filters efficiently. FILTER should be used for more complex logic (such as multiple conditions, ranges, or involving measures/aggregates). For example, to filter on a measure or an aggregate, you must use FILTER (e.g., “filter products to those with sales > X” – you can’t put [Total Sales] > X directly as a CALCULATE filter without FILTER). But if you just need Product[Color] = "Red", you can pass that directly to CALCULATE. Under the hood, direct CALCULATE filters and FILTER(...) achieve the same outcome (modifying filter context), but direct filters are simpler and often faster.

Example – Using CALCULATE and FILTER:
Let’s say we want a measure for High Value Sales: total sales where the transaction amount was over 1000. We have a measure [Total Sales]. We can do:

High Value Sales = 
CALCULATE( [Total Sales],
    FILTER( Sales, Sales[SalesAmount] > 1000 )
)

Here, FILTER(Sales, Sales[SalesAmount] > 1000) produces a table of only the sales transactions over 1000. CALCULATE then applies that as the filter context for [Total Sales]. The result is the sum of SalesAmount for those high-value transactions. We could also achieve this with an iterator: High Value Sales = SUMX( FILTER(Sales, Sales[SalesAmount] > 1000), Sales[SalesAmount] ) – which in effect does the same thing (iterates and sums). Using CALCULATE with an existing [Total Sales] measure is a bit cleaner.

Another example: Percentage of total often uses CALCULATE. Imagine [Total Sales] as current context sales, and you want a measure for “Percentage of All Sales that this context represents”. You could write:

% of All Sales = 
DIVIDE( [Total Sales],
        CALCULATE( [Total Sales], REMOVEFILTERS() )
      )

The CALCULATE( [Total Sales], REMOVEFILTERS() ) part gives the total sales with all filters removed (i.e., the denominator is total sales for all data). This uses the filter modifier REMOVEFILTERS() to clear filters (CALCULATE function (DAX) - DAX | Microsoft Learn). The DIVIDE then gives the fraction. This pattern is common in DAX for percent of total, percent of parent, etc.

Think of CALCULATE as: "Evaluate this expression, but under these filter conditions (possibly in place of whatever external filters there were)." It’s how you explicitly control context inside a measure. Meanwhile, think of FILTER (function) as: "From this table, give me only the rows that meet this test." They often work together but serve different purposes.

Advanced filter modifiers: Within CALCULATE, you can use special functions:

• ALL(Table/Column): ignores filters on that table/column (CALCULATE function (DAX) - DAX | Microsoft Learn). (Use ALL or REMOVEFILTERS to get overall totals or to override specific filters.)

• ALLEXCEPT(Table, Column1, Column2, ...): clears all filters on a table except the ones explicitly listed.

• KEEPFILTERS(FilterExpression): changes how filter application works by intersecting with existing filters instead of overwriting (DAX Best Practices | MAQ Software Insights). For example, CALCULATE( [Measure], KEEPFILTERS( Table[Column] = "Value" ) ) will keep any existing filters on Table[Column] and require that "Value" also be true (i.e., an intersection), whereas normally CALCULATE would replace any filter on that column with "Value". Use case: maybe a measure that refines an existing report filter rather than replacing it.

• USERELATIONSHIP(col1, col2): as mentioned, activates an inactive relationship for the calculation ( Using USERELATIONSHIP in DAX - SQLBI).

• CROSSFILTER(col1, col2, direction): temporarily change relationship filter direction (or disable it) for the calc.

• There are others like ALLSELECTED, ALLNOBLANKROW, but those are more specialized.

(CALCULATE function (DAX) - DAX | Microsoft Learn) (Microsoft docs) underscores how CALCULATE modifies filter context by adding or overriding filters provided in its arguments. And (Explanation of Context in Calculating Cumulative Values) reminds us that FILTER returns a table of rows that meet a condition (and acts as an iterator).

Important: CALCULATE only works in measures or calculated columns, not in row-level security or DirectQuery mode in certain cases (with exceptions). But in general use, you’ll be writing CALCULATE a lot in measures. Also note, you cannot nest CALCULATE inside another CALCULATE’s filter (there’s a restriction on that), and you cannot use a measure as a direct filter argument (you must use a measure in a filter through something like FILTER or a comparison as part of a boolean expression).

Aggregation and Iterator Functions

DAX has a variety of aggregation functions (like SUM, MIN, MAX, COUNT, AVERAGE) and their iterator counterparts (SUMX, MINX, MAXX, COUNTX, AVERAGEX, etc.). Understanding the difference between these is key to writing correct calculations:

• Simple aggregators (non-X): Functions such as SUM(column), AVERAGE(column), MIN(column), MAX(column), COUNT(column) operate over a column in the current filter context. They take a column reference and compute the aggregate of all values currently visible in that column. For example, SUM(Sales[Amount]) adds all Sales Amount values in whatever context (filters) is active. These functions do not have an inherent row context; they implicitly consider all rows allowed by the filter context. They are highly optimized to work on the column data.

• Iterator functions (X functions): These include SUMX(table, expression), AVERAGEX(table, expression), MINX, MAXX, COUNTX, RANKX, etc. These functions iterate over a specified table, evaluating the given expression for each row, then aggregate the results. During their execution, a row context is created for the table they iterate ( Row Context and Filter Context in DAX - SQLBI). For instance, SUMX(Sales, Sales[Quantity] * Sales[Price]) will go row by row in the Sales table, compute Quantity * Price for each row, and then sum up those computed values. If you tried to achieve the same with simple SUM, you couldn’t directly, because you’d need to multiply per row first (which SUM can’t do by itself).

  Another example: AVERAGEX( Dates, [Daily Sales] ) (imagine [Daily Sales] is a measure that gives sales on a single date context). This would iterate each date in the Dates table (perhaps filtered to a year), retrieve [Daily Sales] for that date (context transition happens for the measure inside, effectively), and then average those values.

• Why use X iterators? They let you perform calculations per row that are more complex than just the raw column value, and then aggregate. Common scenarios:

  • Calculated weighted averages or ratios per row then summed.

• Concatenate values (there’s even CONCATENATEX which joins strings from rows).

• Applying a filter with a condition within an aggregation (though often there are alternatives, like CALCULATE with filters).

• Ranking or sorting by an expression (RANKX lets you rank rows of a table by an expression, e.g., rank products by sales within a category).

• Performance considerations: If a simple aggregator can do the job, use it instead of an X iterator because it’s usually faster (it leverages the optimized column storage). For example, SUM(Sales[Amount]) is preferred to SUMX(Sales, Sales[Amount]) – they return the same result, but the latter introduces unnecessary row-by-row iteration. In fact, SUM(column) is essentially a shorthand for SUMX(table, column) where the engine can optimize it directly (SUM, SUMX – DAX Guide - SQLBI). As a rule: use the X version only when you need to do something per row that a simple function can’t handle.

• Examples:

  • Basic sum vs. sumx:
    SUM – Total Sales = SUM(Sales[Amount]) adds the Amount column directly (fast).
    SUMX – Total Sales (SUMX) = SUMX( Sales, Sales[Amount] ) would theoretically do the same thing but less efficiently.
    A more meaningful SUMX example: Total Revenue = SUMX( Sales, Sales[Quantity] * Sales[UnitPrice] ). Here we multiply quantity and unit price per row (to get revenue per sale), then sum it up. There is no single column that directly has revenue, so SUMX is needed to evaluate the expression for each row.

• AVERAGE vs AVERAGEX: If you want the average of a calculated value per row, use AVERAGEX. For instance, “average gross margin per product” might be AVERAGEX( Products, Products[Margin] ) if [Margin] itself is a calculated column or expression. But if you wanted the overall gross margin (total profit / total sales), you’d better do it as a measure ratio (which might be different from averaging margins of each product). Distinguishing when to average an aggregate vs aggregate an average is important. AVERAGEX is row-by-row, then average; whereas AVERAGE just takes a column of already stored values.

• COUNTX: You might not use COUNTX often; usually COUNT or COUNTROWS suffice. COUNTX(table, expression) will count the number of rows where the expression is not blank. A typical use might be something like counting non-empty results of an expression.

• RANKX: This is an iterator used for ranking. E.g., Rank Sales by Product = RANKX( ALL(Product), [Total Sales], , DESC ) will rank each product’s total sales against all products. RANKX iterates over a table (here ALL(Product) which is the table of all products ignoring filters) and compares the [Total Sales] for each product to determine rank.

(SUM, SUMX – DAX Guide - SQLBI) succinctly says: SUM adds all numbers in a column; SUMX returns the sum of an expression evaluated for each row of a table. That applies to other aggregates too (e.g. AVERAGE vs AVERAGEX, etc.).

• Iterators and filter context: An iterator will respect the outside filter context on the table it’s iterating. For instance, if you do SUMX(Sales, ...) and the report or outer CALCULATE has Year=2020 filter active, then that Sales table passed into SUMX is already filtered to 2020 sales. The iterator then goes through just those. You can also embed CALCULATE inside an iterator’s expression if needed to override context per row (advanced use case).

• Avoiding common mistakes: A frequent error is writing a measure that tries to sum an expression without SUMX. For example, writing Total Revenue = Sales[Quantity] * Sales[UnitPrice] as a measure will error out – because there’s no row context in a measure by default. The correct approach is SUMX(Sales, Sales[Quantity] * Sales[UnitPrice]). Conversely, sometimes newcomers use SUMX when a SUM would do; e.g., SUMX(Sales, Sales[Amount]) – this works but is unnecessarily verbose. Overuse of iterators can hurt performance if the table is large, because it forces a row-by-row evaluation in the formula engine rather than a swift columnar aggregation.

Table functions: In addition to X iterators, DAX has other functions that return tables (e.g., FILTER, ALL, VALUES, SUMMARIZE, ADDCOLUMNS, etc.). These aren't aggregators themselves but are used to shape data for either iterators or for producing calculated tables. For instance, VALUES(Column) returns a one-column table of distinct values (and in a measure context is often used to get a single value of a filter if present). ADDCOLUMNS(Table, "NewCol", Expression) can add a calculated column on the fly to a table (useful within measures to create a temporary table with extra data).

For most everyday use, focus on choosing between non-X and X correctly:

• If you need a calculation per row -> use an X iterator.

• If you just need to aggregate a single existing column -> use the simple aggregator.

Quick Reference – Common Aggregators vs Iterators:

• SUM vs SUMX: SUM(Column) adds up column values; SUMX(Table, Expr) evaluates Expr for each row and sums. (SUM, SUMX – DAX Guide - SQLBI)

• AVERAGE vs AVERAGEX: AVERAGE(Column) vs AVERAGEX(Table, Expr) – similar pattern. Use AVERAGEX if the thing you want to average isn’t a base column.

• COUNT/COUNTROWS vs COUNTX: COUNTROWS(Table) counts rows (optionally filtered via CALCULATE). COUNTX(Table, Expr) counts non-blank results of Expr per row. Often COUNTROWS combined with FILTER is sufficient for conditional counts (e.g., CALCULATE(COUNTROWS(Sales), Sales[Amount] > 1000) to count transactions >1000).

• MIN/MAX vs MINX/MAXX: min or max of a column vs of an expression per row. For example, MAXX( Dates, [Daily Sales] ) might give the maximum daily sales value in the current context.

• DISTINCTCOUNT (counts distinct values in a column) doesn’t have an X version because the operation is inherently on a column set, but if you needed distinct count of an expression, you might do something like COUNTROWS( DISTINCT( GENERATE( table, ... ) ) ) which is more advanced. Typically stick to provided functions.

Time Intelligence Functions

One of DAX’s strengths is built-in time intelligence – functions that make calculating time-based metrics easier. Time intelligence functions allow you to manipulate dates to get calculations like year-to-date, quarter-to-date, same period last year, year-over-year growth, moving averages, etc., without manually writing complex filter logic every time.

Prerequisite: Date Table. In order to use most time intelligence functions properly, you should have a dedicated Date table in your model (a table that contains all dates over the period of interest, with contiguous date ranges, with no missing dates). Mark this table as the "Date Table" in Power BI. The Date table should have a relationship to your fact table (e.g., Sales) on the date field. The functions assume this setup. If you don’t have a proper date table, functions like TOTALYTD or SAMEPERIODLASTYEAR might not work correctly. (Time intelligence functions (DAX) - DAX | Microsoft Learn) emphasizes marking a date table before using these functions.

• Year-to-Date (YTD), Quarter-to-Date, Month-to-Date:
  DAX provides functions like TOTALYTD( <Measure>, <DatesColumn> [, <FiscalYearEndDate>] ), TOTALQTD, TOTALMTD to accumulate a measure from the start of the year/quarter/month up to the current context date. For example:

    Sales YTD = TOTALYTD( [Total Sales], 'Date'[Date] )
  

  If you put this measure in a visual with Month, it will show running total of sales from Jan 1 to that month’s end for each month (assuming 'Date'[Date] is a continuous date column in your Date table). There are equivalent TOTALQTD, TOTALMTD for quarter and month. These are convenience wrappers around a combination of CALCULATE and filter logic on the date. They automatically detect year boundaries (or you can specify a fiscal year end).

• Same Period Last Year / Previous Periods:

  • SAMEPERIODLASTYEAR(<dates>): returns the set of dates exactly one year before the dates in the current filter context. Often used like:

      Sales LY = CALCULATE( [Total Sales], SAMEPERIODLASTYEAR( 'Date'[Date] ) )
    

    This gives the sales for the same period last year corresponding to the current filter context period. For example, if the current context is March 2025, it filters to March 2024’s dates and returns that sales.

• There are also PREVIOUSYEAR, PREVIOUSQUARTER, PREVIOUSMONTH which give the entire previous period (regardless of alignment with current). For instance, if you’re filtering a specific month, PREVIOUSMONTH would give the entire immediately preceding month.

• DATEADD(<dates>, -1, YEAR): an alternative to SAMEPERIODLASTYEAR; DATEADD is more flexible since you can shift by any number of intervals (e.g., -1 year, -3 months, etc.). It returns a table of dates shifted by the interval. E.g., CALCULATE( [Total Sales], DATEADD('Date'[Date], -1, YEAR) ) is effectively same as above.

• Period-to-date vs period-over-period: YTD/QTD/MTD are cumulative from period start. For year-over-year or quarter-over-quarter comparisons, you typically use SAMEPERIODLASTYEAR or DATEADD to get a comparable period in the past, then maybe compute differences or percentages. Example:

    Sales YoY % =
      DIVIDE( [Total Sales] - CALCULATE( [Total Sales], SAMEPERIODLASTYEAR('Date'[Date]) ),
              CALCULATE( [Total Sales], SAMEPERIODLASTYEAR('Date'[Date]) )
            )
  

  This measure calculates year-over-year growth percentage. It takes current sales minus last year’s sales, divided by last year’s sales.

• First/Last/Opening/Closing balances: There are functions like FIRSTDATE, LASTDATE, which give the first or last date in the current filter context. And specialized ones like CLOSINGBALANCEMONTH(<expr>, <dates>), OPENINGBALANCEYEAR(<expr>, <dates>) which evaluate an expression at the end or start of a period. These are used for things like inventory or account balances where you need the value at a period boundary.

• Working with months, quarters, etc.: DAX’s time functions often expect a contiguous date range. They often internally figure out things like “all dates in the same year up to this date” etc. For example, TOTALYTD essentially does CALCULATE( <measure>, DATESYTD(<dates>) ) where DATESYTD returns all dates from the start of the year to the max date in the current context. Similarly SAMEPERIODLASTYEAR is essentially shifting that set by -1 year. If your Date table has gaps or multiple entries per day, results may be off.

(Time intelligence functions (DAX) - DAX | Microsoft Learn) (Microsoft documentation) states: DAX includes time-intelligence functions that enable you to manipulate data using time periods (days, months, quarters, years) and build and compare calculations over those periods. It also reminds to mark a table as Date Table.

Example – Year-over-Year Sales:
Imagine you want a column chart showing sales this year vs last year by month. You’d have [Total Sales] and [Sales LY] as measures. We defined Sales LY above. You place Month on X-axis, and both measures as values. Thanks to the DAX time functions, for each month context, [Sales LY] automatically grabs the equivalent month last year’s data. This simplifies writing such comparisons.

Example – Year-to-Date Total:
If you show a cumulative line chart of sales across months, [Sales YTD] will give a running total. The measure we gave Sales YTD = TOTALYTD([Total Sales], 'Date'[Date]) will accumulate from the start of each year. The function knows to reset at year boundaries (if using calendar year or the fiscal year you provide). Under the hood, TOTALYTD is doing something akin to:

Sales YTD =
CALCULATE( [Total Sales],
    DATESYTD( 'Date'[Date] )
)

where DATESYTD returns all dates from Jan 1 of the current year up to the current date context.

• Rolling averages or moving sums: There isn’t a single built-in for moving average (like 30-day rolling), but you can combine functions. For example, to get last 30 days sales:

    Last30DaysSales =
    CALCULATE( [Total Sales],
        DATESINPERIOD( 'Date'[Date], MAX('Date'[Date]), -30, DAY )
    )
  

  Here DATESINPERIOD takes the max date in current context (say you’re at March 31, 2025 in a visual, max date is that) and goes back 30 days to produce that set of dates, and CALCULATE sums over that set. This would let you then do a moving average by dividing by 30 or adjusting context in a measure.

Time intelligence functions rely on having that full range of dates. They can handle fiscal years (with year_end parameter in TOTALYTD, etc.), and can work with quarters, months, weeks (though week-based calculations might require more custom logic or a week number in the date table).

Tip: Always use a dedicated Date table and relate it to fact tables. Use the built-in functions like TOTALYTD, SAMEPERIODLASTYEAR, etc., to save time – they implement common patterns. If you need something custom (like “sales in the same month of the previous year but align by fiscal week” or “trailing 12 months”), you might have to use combinations of DATEADD or DATESINPERIOD. The DAX Patterns website and Microsoft docs have recipes for many of these scenarios (Week-Based Time Intelligence in DAX - SQLBI). Start with simple ones and be sure to test the results to ensure they match expected values.

Variables and Debugging Techniques

DAX introduced the VAR and RETURN syntax to allow variables within expressions. Using variables can greatly improve the clarity, performance, and debuggability of your DAX formulas (Variables in DAX - SQLBI). A variable lets you compute a sub-expression once, name it, and reuse it multiple times within the same measure (or calculated column). Variables do not persist beyond the single evaluation of that formula; they are not like Excel named ranges that stick around globally – they’re local to the measure or query.

• Syntax:

    VAR <varName> = <expression>  
    VAR <varName2> = <expression2>  
    ...  
    RETURN <final_expression_using_vars>
  

  You can define one or multiple VARs, and then after the RETURN keyword, put the expression that produces the final result (usually involving those variables). In the final expression, you refer to the variable by name (without any special quoting).

• Benefits of variables:

  1. Clarity: You can break a complex calculation into understandable parts. For example, instead of a nested monstrosity, you can do VAR Intermediate = ... RETURN ... to make it clear what each piece represents.

1. Avoid repetition: If the same sub-calculation is used multiple times, putting it in a variable means it’s calculated once and reused. For example, VAR totalRows = COUNTROWS(Sales) then use totalRows twice is better than writing COUNTROWS(Sales) in two places (DAX Best Practices | MAQ Software Insights). This not only avoids writing it twice, but also ensures the value is identical and computed once (which can improve performance).

1. Debugging: You can use variables to inspect intermediate results. During development, you might temporarily set the RETURN expression to just a variable to see what it evaluates to in a visual. For instance, if a measure isn’t working, you can break it into parts with VAR and then output one part at a time to verify each. This is a common debugging trick – and since variables are not output normally, you make them output by returning them or including them in a simple way for testing (Power BI DAX Debugging Tricks! - Medium).

• Example of using VAR for performance:

    BigRatio =
    VAR totalOrders = COUNTROWS( Sales )
    VAR totalSales = [Total Sales]   -- assume [Total Sales] is defined elsewhere
    RETURN 
      IF( totalOrders = 0, 
          BLANK(), 
          totalSales / totalOrders 
      )
  

  In this measure, we calculate totalOrders (number of sales) once. We also capture [Total Sales] into totalSales variable (technically if [Total Sales] is a measure, calling it multiple times would give the same result in the same filter context – the engine might already optimize it, but using a VAR guarantees it’s only evaluated once here). Then we return an IF that uses those variables. If we did not use variables, we might write IF(COUNTROWS(Sales)=0, BLANK(), [Total Sales] / COUNTROWS(Sales)). In that original form, COUNTROWS(Sales) would be evaluated twice (once for the IF check, once for the division). Using VAR, it’s evaluated once (DAX Best Practices | MAQ Software Insights). This is a trivial example, but with heavier expressions this matters a lot.

• Scope of variables: A variable is computed before the filter context is modified by row context in an iterator, if used inside one. That’s an advanced detail: basically, in a measure or query, all VARs are evaluated in the context they are defined, then the RETURN uses those values. In an iterator, if you define a VAR outside the iterator and use it inside, it stays constant; if defined inside, it might be computed per row. Typically, you define at the top of a measure and it’s one per overall evaluation. You cannot have variables that change per row in a simple way (they’re not like looping counters; they’re constants during that evaluation). So consider them like let-bindings in math.

• Debugging with variables:
  Let’s say you have a complex measure and you suspect the problem is in one part of it. You can refactor:

    ComplexMeasure =
    VAR part1 = ... some expression ...
    VAR part2 = ... another expression possibly using part1 ...
    RETURN part2  -- temporarily return part2 to see if it's right
  

  If the result of the measure in a visual seems off, you might set RETURN part1 just to see what part1 is producing in that context. This way, you isolate which part is incorrect. Another trick: sometimes output multiple variables by combining them in a table or concatenated string just for debugging purposes, though in a measure you can only return one scalar value – but you could make a temporary calculated table measure or use DAX Studio to run a query to see multiple variables. There is also a feature in tools like Tabular Editor or DAX Studio to evaluate intermediate steps.

(DAX Debugging Tricks ( Basic to Advanced ) | Power BI! - Medium) indicates: Variables in DAX help split complex formulas into manageable pieces and allow intermediate results to be reused. This captures why they are good for both debugging and performance.

• Example of a complex measure using multiple variables:
  Suppose we want a measure “% of category sales that are current product’s sales”. This could be done in steps:

    Product Sales % of Category =
    VAR currProductSales = [Total Sales]  -- sales for the current product (filter context)
    VAR categorySales = CALCULATE( [Total Sales], ALL( Product[ProductName] ) )
        -- remove product filter, so it's total for the category (assuming Category is still filtered)
    RETURN
      DIVIDE( currProductSales, categorySales )
  

  Here we used two variables to clarify our intent: one for the current product’s sales, one for the whole category’s sales (we removed only the product filter, so category context remains). This makes the measure easier to read and double-check. We could even debug by returning categorySales alone to ensure it’s doing what we expect.

• DAX debugging tools: Outside of writing variables, it’s worth mentioning DAX Studio (an external tool) which allows you to run DAX queries and see results or performance metrics, and Performance Analyzer in Power BI which shows how long measures take. In a purely formula sense, variables and the RETURN trick are your best friend to understand complex logic. SQLBI’s article on debugging DAX measures suggests techniques like returning table constructs or using the ERROR() function cleverly, but those are advanced. Usually, incrementally building the measure and testing as you go is the best approach.

Tip: Use descriptive names for variables (and measures). This self-documents your code. For example, VAR avgSales = DIVIDE([Total Sales], [Order Count]) is clearer than VAR x = .... While DAX variables can’t be viewed outside, the measure definition itself becomes easier to understand when you or someone else revisits it. Also, keep in mind a variable is calculated once per filter context (not once per row, unless used inside an iterator per row). If you need a per-row calculation, you still need an iterator or a calc column.

Performance Tuning and Best Practices

As you get into more complex DAX scenarios, it’s important to follow best practices to ensure your measures run efficiently and your results are correct. DAX is powerful, but performance can suffer if formulas are not optimized or if the model is not designed well. Here are some key practices and tips:

1. Model First, DAX Second: A good data model (star schema with proper relationships, necessary columns pre-computed in Power Query or source, no overly high cardinality columns for no reason) lays the foundation for simpler DAX. Whenever possible, shape your data so that DAX measures don’t have to do heavy lifting like string parsing or complex lookups at query time.

1. Prefer Measures over Calculated Columns for Dynamic Calculations: If a value can be computed on the fly and especially if it needs to respond to filters, use a measure. Measures are evaluated lazily (only when needed) and don’t bloat your model. Calculated columns are best for static categorizations or when you explicitly need to slice data by that result. Measures keep your model lean. (As noted, calculated columns increase RAM usage and file size, and too many can slow refresh).

1. Avoid Repeating Calculations – Use Variables or Separate Measures: If you notice the same expression used multiple times in a measure, factor it out. For example, instead of [Result] = IF(X > 0, X/total, X) where X is some complex sum, do VAR X = complex sum RETURN IF(X > 0, X/total, X). This way complex sum runs once. Similarly, you can create helper measures for reusability. E.g., define [Total Sales] once and reuse it in many other measures rather than writing SUM(Sales[Amount]) in every measure (and possibly adding filters each time incorrectly). This modular approach also makes maintenance easier. (DAX Best Practices | MAQ Software Insights) illustrated how using a variable prevented double calculation of a measure.

1. Use Built-in Functions Optimally:

   • Use DIVIDE(numerator, denominator, alternateResult) instead of the / operator when dividing measures, to gracefully handle division by zero without extra IF logic (DAX Best Practices | MAQ Software Insights). It’s both clearer and potentially a tiny bit more efficient than doing an IF yourself.

• Use SELECTEDVALUE(column, alternate) instead of the pattern IF(HASONEVALUE(column), VALUES(column), alternate) (DAX Best Practices | MAQ Software Insights) (DAX Best Practices | MAQ Software Insights). SELECTEDVALUE does exactly that check internally (returns the single value if one exists, otherwise the alternate or BLANK). This makes code cleaner and potentially avoids performance issues of accidentally getting an error from VALUES() when multiple values exist.

• Use CONCATENATEX for string aggregations instead of trying to stitch strings via hacks (like using PATH functions or so).

• Leverage COALESCE() (as of newer updates) to handle blanks in a cleaner way than nested IFs for default values.

1. Context Modification Functions: Be mindful with FILTER inside CALCULATE. Recall that giving a filter like Table[Column] = value directly is usually faster than using FILTER(Table, Table[Column]=value) because the latter iterates the whole table. If you have multiple conditions on the same table, you can often combine them in one FILTER. If you want to preserve an existing filter and add a new one, consider KEEPFILTERS instead of FILTER (DAX Best Practices | MAQ Software Insights). For example, CALCULATE([Measure], KEEPFILTERS(Table[Col] = "X")) will respect existing filters on Col and require "X" too, whereas FILTER would disregard the existing filter on Col. This can both affect correctness and performance (because of how queries get optimized).

1. Avoid Iterating Over Huge Tables Unnecessarily: If your dataset is large (millions of rows), an iterator like FILTER(Sales, ...) or SUMX(Sales, ...) will iterate all those rows in the formula engine, which can be slow. If you can push a filter to the storage engine (VertiPaq) it’s faster. E.g., CALCULATE([Measure], Table[Col] = "Value") will let the engine use its internal indexes to filter, which is usually very fast. But FILTER(Table, Table[Col]="Value") would enumerate rows one by one. So use iterator functions judiciously. When possible, structure your measure using CALCULATE with filter arguments or use COUNTROWS( FILTER(...) ) patterns that at least limit the rows early.

1. Limit the Scope of Calculations: If you need a calculation at a certain granularity, consider doing it in a summarized or calculated table instead of on the fly for every detail row. For example, if you frequently need to calculate something per customer and then sum it up, you could create a calculated table or use SUMMARIZE to pre-calc per customer. Alternatively, a measure with SUMX( VALUES(Customer[ID]), ... ) can ensure you only iterate unique customers not every transaction.

1. Use the RIGHT grain of context in measures: This is a bit conceptual: Sometimes you can write a measure in different ways – one might do a heavy calc on each row of a fact, another might leverage an aggregated table. For example, to count customers who purchased something, rather than doing COUNTROWS( DISTINCT( Sales[CustomerID] ) ) for a huge sales table, you could have a Customers table and do COUNTROWS( FILTER( Customers, [Total Sales for Customer] > 0 ) ) where [Total Sales for Customer] is maybe a calc column or measure. The latter might iterate fewer rows (just customers). This depends on model and need, but think about what you’re iterating over.

1. Formatting and Readability: Use a consistent formatting style. Tools like DAX Formatter (by SQLBI) can format your DAX code for readability (indenting, line breaks). Clean, well-formatted DAX is easier to debug and often helps you spot logical issues. Also, as a best practice, name your measures descriptively (and include units or time frame if relevant, e.g., "Total Sales LY" for last year). You can also add descriptions to measures in the model for documentation.

1. Measure Dependencies and Ordering: Be cautious of measures that depend on others that depend on others... While modularizing is good, a very long chain might be harder to troubleshoot. Try to keep measure logic coherent. But referencing a measure within another is fine (the engine calculates them in the right order as needed).

1. Testing Performance: Use the Performance Analyzer in Power BI to see which measures are slow. If a particular measure is slow, consider if it’s doing something complex like nested iterators or scanning a huge table with FILTER. Try simplifying it or see if the heavy part can be precomputed in a column or table. Also, check your model size; large models might benefit from aggregations or reducing cardinality of columns (like splitting high cardinality columns if possible, or removing unneeded detail).

1. Common Mistakes to Avoid:

   • Avoid using VALUES() in a context where it can return multiple values without wrapping in an aggregator or SELECTEDVALUE; it will error if multiple values appear. If you expect one value but sometimes there are many, use SELECTEDVALUE or an iterator to handle it.

• Be careful with ALLSELECTED vs ALL – ALLSELECTED is context-sensitive (it respects outer filters but removes filters from the current visual context), can be confusing for percent of total calcs. Use ALL when you mean truly all, ALLSELECTED when doing total of the selection (for subtotals etc.).

• Don’t assume a calculated column will update when you use a slicer. This misconception can lead to wrong approaches. If you need dynamic behavior, measure is the way.

• In measures, don’t ignore the filter context. For example, instead of trying to store an intermediate result globally, think in terms of context. Each evaluation has to consider “given the filters, what’s the output”.

1. Iterate only as needed: For example, instead of SUMX(Filter(Sales,...), Sales[Amount]), consider if CALCULATE(SUM(Sales[Amount]), ... ) could do the job with less explicit iteration. The DAX optimizer might do similar things under the hood, but writing it in a set-based way often aligns with engine optimizations.

1. Use of USERELATIONSHIP and relationships: Only activate alternate relationships in measures that require them, and isolate their use to those measures. That keeps your model’s default behavior predictable and uses heavier calc (like USERELATIONSHIP in CALCULATE) only when necessary (like a special measure). Similarly, try to avoid bi-directional cross-filtering on whole model unless needed; it can usually be handled with specific CALCULATE modifiers or measure logic for those few cases (like a many-to-many through a bridge using TREATAS or CROSSFILTER).

In short, write DAX clearly and efficiently. Favor the powerful filter context over manual row iteration when possible. Test your measures with small filters to ensure they compute expected results before applying to big data. And remember, DAX is a mix of a query language and formula language – sometimes there are multiple ways to achieve something (set-based vs procedural). Aim for the approach that leverages the strengths of the VertiPaq engine (set-based filtering, compression) rather than brute-force calculations.

Finally, stay curious and keep learning: DAX has many nuances and functions (over 200 functions). Refer to official documentation and community blogs for best practices. Over time, you’ll develop an intuition for writing DAX that is not only correct but also performant and easy to maintain. Happy DAX-ing!

I have just finished a study guide on BigQuery @ GCP. Below you can find it on Heptabase Whiteboard

In Summary, you can find it in here:

BigQuery SQL Statement Guide (Standard SQL & Legacy SQL)

Google BigQuery’s SQL dialect (Google Standard SQL) extends traditional SQL with unique features. Below, we explore BigQuery-specific SQL statements in the classic categories – DDL, DQL, DML, DCL, TCL – with examples and notes on legacy SQL where applicable, followed by additional topics like aggregations, window functions, and performance tips.

1. DDL – Data Definition Language

Definition: DDL statements are used to define or alter the structure of the database objects. In BigQuery, DDL can create, modify, and delete datasets (schemas), tables, views, materialized views, table snapshots, table clones, functions (UDFs), procedures, and row-level access policies (Learn All About DDL Commands). DDL operations affect the schema/metadata but not the actual data contents (Create and Delete Tables in BigQuery: A 2025 DDL Guide).

BigQuery’s legacy SQL did not support DDL in queries (table management had to be done via the UI or CLI). BigQuery Standard SQL (also called GoogleSQL) introduced in-query DDL support. Key BigQuery DDL statements include CREATE, ALTER, and DROP, each extended with BigQuery-specific options:

• CREATE – Used to create new datasets, tables, views, routines, etc. BigQuery’s CREATE TABLE supports additional clauses for partitioning and clustering data for performance. You can also use CREATE TABLE ... AS SELECT (CTAS) to create a table from a query result in one statement (How to Create a Table in BigQuery: A Step-by-Step Guide). For example, the following creates a partitioned, clustered table:

    -- Create a partitioned table for sales data, partitioned by date and clustered by product
    CREATE TABLE `project_id.my_dataset.sales` (
      sale_date DATE,
      product STRING,
      quantity INT64,
      price NUMERIC
    )
    PARTITION BY sale_date
    CLUSTER BY product;
  

  In the above, PARTITION BY and CLUSTER BY are BigQuery-specific extensions to define date or integer range partitions and clustered indexing on a column, which help optimize query performance (BigQuery Best Practices to Optimize Cost and Performance) (BigQuery Best Practices to Optimize Cost and Performance). BigQuery also allows adding a table description or labels via the OPTIONS clause in DDL. For example, you could append OPTIONS(description="Sales data", labels=[("team","finance")]) to the CREATE TABLE to document the table.

  BigQuery datasets are analogous to schemas: use CREATE SCHEMA dataset_name (Standard SQL) to create a new dataset (Create and Delete Tables in BigQuery: A 2025 DDL Guide) (Create and Delete Tables in BigQuery: A 2025 DDL Guide). Similarly, CREATE VIEW creates a logical view from a query, and CREATE MATERIALIZED VIEW creates a view that caches results for faster reuse.

• ALTER – Used to modify existing objects. BigQuery allows altering table schemas flexibly. For example, you can add a column to a table without rewriting it:

    ALTER TABLE `my_dataset.sales` 
    ADD COLUMN IF NOT EXISTS comments STRING;
  

  This adds a new nullable field comments to the sales table. Other BigQuery ALTER operations include ALTER TABLE SET OPTIONS (to change table metadata like description or default expiration), ALTER COLUMN (to change column options like descriptions), or renaming tables/columns. You can also alter datasets (schemas) – e.g. ALTER SCHEMA my_dataset SET OPTIONS(default_table_expiration_days=90) to set a default expiration for tables in a dataset.

• DROP – Removes BigQuery objects. For example, DROP TABLE my_dataset.sales; will delete the sales table. BigQuery also supports DROP SCHEMA my_dataset CASCADE; to delete a dataset along with all its tables. (Legacy SQL required using the API or console for such operations, since it lacked DDL.)

Example – Creating and Dropping a Table:

-- Create a new table from a query (CTAS example):
CREATE TABLE `project_id.my_dataset.top_customers` AS
SELECT customer_id, SUM(total_spend) AS total_spend
FROM `project_id.my_dataset.orders`
GROUP BY customer_id
HAVING SUM(total_spend) > 1000;

-- Later, drop the table if no longer needed:
DROP TABLE `project_id.my_dataset.top_customers`;

The above CREATE TABLE ... AS SELECT will create top_customers with the results of the query (How to Create a Table in BigQuery: A Step-by-Step Guide). The DROP TABLE statement then deletes it. BigQuery DDL statements can include IF EXISTS/IF NOT EXISTS to guard against errors when objects may or may not exist (Create and Delete Tables in BigQuery: A 2025 DDL Guide).

2. DQL – Data Query Language

Definition: DQL refers to data query statements, primarily the SELECT statement and its clauses used to retrieve data. In BigQuery, SELECT queries support standard SQL semantics and many BigQuery-only extensions. (Legacy SQL mode also allowed SELECT queries but with different functions and handling of nested data.)

A basic BigQuery SELECT query supports the usual clauses: SELECT ... FROM ... JOIN ... WHERE ... GROUP BY ... HAVING ... ORDER BY ... LIMIT. BigQuery’s Standard SQL has full support for SQL joins, subqueries, and set operations. The WHERE clause filters rows before aggregation, and HAVING filters groups after aggregation (more on HAVING in Section 6). BigQuery’s implementation follows standard evaluation order: WHERE filters, then GROUP BY aggregates, then HAVING filters aggregated groups (A Complete Guide to WHERE, HAVING, and QUALIFY in SQL) (A Complete Guide to WHERE, HAVING, and QUALIFY in SQL).

BigQuery-specific Query Features:

• SELECT * EXCEPT / REPLACE: BigQuery extends SELECT * with the ability to exclude or replace specific columns. For example, SELECT * EXCEPT(password) will select all columns except password, and SELECT * REPLACE(customer_id * 100 AS customer_id) would select all columns but use a calculated value in place of customer_id. This is unique to BigQuery’s SQL (and now adopted by some other databases). Usage is straightforward: SELECT * EXCEPT(col1, col2) FROM ... to drop columns (Select All Columns Except Some in Google BigQuery? - Stack Overflow). You can even combine them, e.g. SELECT * EXCEPT(id) REPLACE("widget" AS product_name) FROM products; (Select All Columns Except Some in Google BigQuery? - Stack Overflow). This saves time when you need most columns but not all.

• Qualify (Filtering on Window Functions): BigQuery supports the QUALIFY clause to filter the results of window functions (analytic functions) in the same query, rather than using a subquery. QUALIFY is analogous to HAVING but for window function results (A Complete Guide to WHERE, HAVING, and QUALIFY in SQL). For example, after using a ranking function in the SELECT, you can add QUALIFY ROW_NUMBER() OVER(PARTITION BY category ORDER BY sales DESC) = 1 to return only the top-selling row per category. This clause is evaluated after window functions are computed, letting you filter on their outcomes (QUALIFY clause is now available in BigQuery - gnarlyware) (QUALIFY clause is now available in BigQuery - gnarlyware). (Without QUALIFY, you would need a subquery or CTE to filter by a window function result.)

• Window Functions (OVER ... PARTITION BY): BigQuery fully supports SQL window functions for running totals, ranking, lag/leads, etc. You can include expressions like SUM(amount) OVER(PARTITION BY region ORDER BY date) in the SELECT clause to compute a running total, or RANK() OVER(PARTITION BY region ORDER BY sales DESC) to rank rows within partitions. (We cover window functions in detail in Section 8.) These use the OVER() syntax with optional PARTITION BY and ORDER BY. BigQuery even allows filtering by these in the same query via QUALIFY as noted. Legacy SQL did not support window functions; this is a Standard SQL feature.

• Arrays and Structs (Nested Data): BigQuery can store nested and repeated fields (arrays and structs) in tables, and its SQL can directly query such structures. For example, a table can have a column which is an ARRAY of STRUCTs. In Standard SQL, you use the UNNEST() function in the FROM clause to flatten arrays for analysis. BigQuery’s ability to handle nested data is a distinguishing feature – it allows denormalized schemas (e.g., an order with an array of line-items) that you can still query with SQL. (Legacy SQL had a different approach using FLATTEN() for repeated fields.) For instance:

    SELECT order_id, item.product_name, item.quantity
    FROM `project.dataset.orders`, 
         UNNEST(items) AS item
    WHERE item.product_category = 'Electronics';
  

  Here, items is an ARRAY field in orders; UNNEST(items) produces a table of item structs, which we alias as item. This query will output one row per array element. BigQuery’s SQL treats each element as a row but still associates it with its parent order. This nested data capability allows BigQuery to avoid expensive joins by keeping related data together (BigQuery Best Practices to Optimize Cost and Performance) (the data is physically nested). It’s a powerful feature not found in many SQL dialects.

• Wildcard Table Queries: BigQuery can query multiple tables in one go using wildcards or table decorators. For example, you might have monthly partitioned tables like sales_2019Jan, sales_2019Feb, ... and run a query across all of them with FROM \project.dataset.sales_2019*`. BigQuery provides a pseudo-column _TABLE_SUFFIXto filter which underlying tables to include. (In legacy SQL, functions likeTABLE_DATE_RANGE()` were used for similar effect.) This helps when data is sharded into many tables by date.

• Set Operators: BigQuery supports standard SQL set operations: UNION [ALL], EXCEPT, and INTERSECT. These allow combining result sets from multiple SELECT queries. BigQuery follows the standard behavior (e.g., UNION deduplicates unless ALL is specified).

• Advanced Grouping: BigQuery Standard SQL (recent versions) supports grouping sets, rollups, and cubes for aggregation. For example, you can do GROUP BY ROLLUP(region, product) to get subtotals by region and product combinations, or use GROUP BY CUBE or explicit GROUPING SETS. These were not available in legacy SQL. (Support for GROUP BY ROLLUP, CUBE, GROUPING SETS was added to BigQuery’s SQL (Google BigQuery now supports Cubes | by Christianlauer - Medium).)

Example – BigQuery Query with Unique Features:

SELECT 
  category,
  product,
  SUM(sales) AS total_sales,
  RANK() OVER(PARTITION BY category ORDER BY SUM(sales) DESC) AS product_rank_in_cat
FROM `my_dataset.product_sales`
GROUP BY category, product
HAVING total_sales > 1000
QUALIFY product_rank_in_cat = 1

In this (contrived) example, we use GROUP BY to aggregate sales by category and product, HAVING to keep only products with >1000 sales in a category, and a window function (RANK() OVER(PARTITION BY category ORDER BY SUM(sales) DESC)) to rank products by sales within each category. Finally, QUALIFY product_rank_in_cat = 1 filters the results to return only the top product per category. This single query finds the top-selling product in each category with over 1000 sales, demonstrating BigQuery’s ability to combine aggregation, window functions, and qualify filtering. (Note: The SUM(sales) inside the RANK() would actually be computed as an analytic function – BigQuery allows using aggregations in analytic functions by computing them over the partition – here it effectively ranks by the same SUM per partition.)

3. DML – Data Manipulation Language

Definition: DML statements modify table data (insert, update, delete rows). BigQuery’s DML enables adding or changing data in BigQuery tables via SQL, which was a major enhancement over the early append-only model. Standard DML in BigQuery includes INSERT, UPDATE, DELETE, and the combined MERGE statement (Bigquery SQL how to update rows AND Insert new data - Stack Overflow). These statements operate in BigQuery with some constraints (for example, BigQuery executes DML on a snapshot of the table to avoid conflicts, and prior to 2020 there were quotas on the number of DML operations per day, which have since been removed (DML without limits, now in BigQuery | Google Cloud Blog)).

INSERT: Adds new rows to a table. BigQuery supports two forms: inserting explicit values, or inserting the results of a query. For example:

-- Insert a single row with explicit values:
INSERT INTO `my_dataset.users` (user_id, name, signup_date)
VALUES (123, 'Alice', CURRENT_DATE);

-- Insert multiple rows using a subquery:
INSERT INTO `my_dataset.gold_customers` (customer_id, total_spend)
SELECT customer_id, SUM(amount)
FROM `my_dataset.sales`
GROUP BY customer_id
HAVING SUM(amount) > 10000;

The first INSERT adds one row to users. The second inserts the results of a query (all customers with >10000 total spend) into the gold_customers table. BigQuery’s Standard SQL does not require a VALUES list for each row if using a SELECT; it will append all rows from the query. (Legacy SQL did not support DML; data was usually loaded in bulk or via streaming inserts.)

UPDATE: Modifies existing rows, setting new values for some columns where a condition is met. BigQuery’s UPDATE syntax allows a FROM clause for more complex updates (e.g. updating a table based on a join with another table). Example:

UPDATE `my_dataset.users`
SET last_login = CURRENT_TIMESTAMP
WHERE user_id = 123;

This updates the last_login timestamp for the user with ID 123. If we needed to update based on another table (say we have a staging table of latest logins), we could do something like:

UPDATE `my_dataset.users` AS u
SET last_login = s.new_login_time
FROM `my_dataset.login_updates` AS s
WHERE u.user_id = s.user_id;

BigQuery executes the update in a single pass (it's atomic). Under the hood, BigQuery may rewrite the entire table or the modified partitions, but the user sees it as an in-place update.

DELETE: Removes rows that meet a condition. For example:

DELETE FROM `my_dataset.users`
WHERE is_active = FALSE AND last_login < DATE_SUB(CURRENT_DATE(), INTERVAL 1 YEAR);

This would delete users who are marked inactive and have not logged in for over a year. As with updates, deletes in BigQuery are atomic. (There is no need for a separate “TRUNCATE” in BigQuery; you can DELETE all rows or use DROP TABLE to quickly remove a table’s data.)

MERGE: BigQuery’s MERGE statement combines insert, update, and delete logic in one operation – essentially an “upsert” capability. MERGE allows you to compare a target table with a source (such as a stage or delta table) and specify actions for when the rows match or don’t match. BigQuery performs the specified inserts/updates/deletes atomically as one transaction (Bigquery SQL how to update rows AND Insert new data - Stack Overflow). This is particularly useful for incremental data pipelines where you apply changes (new records and updates) to a master table.

A typical MERGE example is merging daily new data into a master table:

MERGE `my_dataset.Inventory` AS T
USING `my_dataset.NewArrivals` AS S
ON T.product_id = S.product_id
WHEN MATCHED THEN 
  UPDATE SET T.quantity = T.quantity + S.quantity
WHEN NOT MATCHED THEN 
  INSERT(product_id, product_name, quantity) 
  VALUES(S.product_id, S.product_name, S.quantity);

In this example, the Inventory table is updated by adding quantities for products that already exist (matched on product_id), and inserting new rows for products that are not yet in Inventory (Transaction Management in Google Bigquery - Stack Overflow). The MERGE can have multiple WHEN clauses, including WHEN MATCHED [AND condition] THEN DELETE to delete rows that meet some condition. All these changes (updates to some rows, inserts of others) happen in one combined operation. This MERGE effectively keeps the Inventory table in sync with the NewArrivals table.

BigQuery’s MERGE syntax and behavior are similar to ANSI SQL MERGE (as in Oracle or SQL Server). Note that when using MERGE, the source and target can each be a table or subquery, giving flexibility to, for example, merge the result of an aggregation into a table. Because MERGE is atomic, it can be used within a multi-statement transaction (see Section 5) or as a standalone way to ensure data consistency.

Legacy SQL vs Standard SQL for DML: Legacy SQL did not support these DML statements in queries. Data manipulation had to be done outside of query jobs (e.g., via the BigQuery API, or by writing query results to a new table). With Standard SQL, BigQuery can handle transactional modifications. Initially BigQuery imposed a limit of 1,000 DML statements per table per day (to maintain performance) (Google Big Query: "Exceed rate limits" - How to upload large tables), but as of 2020 this limit has been removed, allowing unlimited DML on tables (DML without limits, now in BigQuery | Google Cloud Blog). Still, very large or complex DML operations may be slower than using BigQuery’s fast load jobs or using MERGE to batch changes, so usage should be planned accordingly.

4. DCL – Data Control Language

Definition: DCL statements control access permissions on database objects (tables, views, etc.) – typically GRANT and REVOKE. In BigQuery, access control is usually managed by Google Cloud IAM (Identity and Access Management), where you assign roles like BigQuery Data Viewer or Data Editor at the project or dataset level. However, BigQuery has introduced SQL DCL syntax to allow granting and revoking fine-grained permissions at the SQL level for convenience. These statements effectively map to IAM under the hood and let you manage table or dataset access with SQL commands (How to Create a Table in BigQuery: A Step-by-Step Guide).

Using BigQuery’s DCL requires appropriate permissions (you generally need to be a BigQuery admin or the owner of the dataset/table to change its IAM policy). You can grant permissions to users, groups, or service accounts.

• GRANT: Gives a user or role certain privileges on a BigQuery resource (such as SELECT rights on a table). BigQuery’s syntax is similar to other SQL dialects. For example, to grant read access on a table to a specific user:

    GRANT SELECT ON TABLE `my_project.my_dataset.sales` 
    TO 'user:john.doe@example.com';
  

  This statement grants SELECT (read/query) privileges on the sales table to the user john.doe@example.com. In effect, this user would be able to query the table (equivalent to the BigQuery IAM role BigQuery Data Viewer on that table). Similarly, you could grant at the dataset (schema) level. For instance, to allow a group to read all tables in a dataset:

    GRANT SELECT ON SCHEMA `my_project.finance_data` 
    TO 'group:analysts@example.com';
  

  This would grant read access on all current and future tables in the finance_data dataset to all members of the analysts@example.com group. (Behind the scenes, BigQuery creates an entry in the dataset’s access list for that group.)

  BigQuery supports privileges like SELECT, INSERT, UPDATE at the table level. It also allows granting roles; for example, you can grant the predefined IAM role roles/bigquery.dataEditor at a dataset level via a GRANT statement if needed. (In the GRANT syntax, BigQuery identifies tables vs. schemas using the ON TABLE or ON SCHEMA keywords.)

• REVOKE: Removes previously granted permissions. For example, if we want to revoke the SELECT permission we gave above:

    REVOKE SELECT ON TABLE `my_project.my_dataset.sales` 
    FROM 'user:john.doe@example.com';
  

  This will strip that user’s access to the table (assuming they didn’t have access via some other route). Similarly, REVOKE ... ON SCHEMA ... FROM ... would remove dataset-level access. Revokes take effect immediately – the next query the user tries on that table will fail with a permission error.

Managing Permissions Example: Suppose we have a dataset analytics and we want to grant a data scientist read-only access to a specific table analytics.marketing_data without giving access to the whole project. We can run:

GRANT SELECT ON TABLE `my_project.analytics.marketing_data` 
TO 'user:data.scientist@company.com';

Now that user can query marketing_data but no other tables (unless separately granted). If later we decide they shouldn’t see it, we do:

REVOKE SELECT ON TABLE `my_project.analytics.marketing_data` 
FROM 'user:data.scientist@company.com';

Under the hood, these DCL statements are updating BigQuery’s IAM policy for that table (or dataset). Note that some privileges (like dataset-level roles) might map to broader IAM roles. Also, BigQuery has the concept of authorized views (a view that exposes certain data and can be shared instead of the raw table) – those are managed by granting a view access to a dataset, which can also be done via DCL by granting to a special identifier representing the view.

Legacy SQL: Legacy BigQuery had no SQL GRANT/REVOKE; all permission management was via Cloud IAM in the console or command-line. The introduction of DCL in Standard SQL makes it easier to script and automate access control. It’s important to adhere to the principle of least privilege (only grant the minimum required access) (How to use grant role in BigQuery?), especially since BigQuery often holds sensitive data.

5. TCL – Transaction Control Language

Definition: TCL statements manage transactions – units of work that can be committed or rolled back together. In traditional SQL, this includes commands like BEGIN (start transaction), COMMIT (save changes), and ROLLBACK (undo changes). BigQuery historically operated on an append-only model without transactions (each query was its own atomic operation). However, BigQuery now supports multi-statement transactions inside BigQuery scripting (and within stored procedures or via API) for executing multiple DML statements as a single atomic unit (Transaction Management in Google Bigquery - Stack Overflow).

BigQuery Multi-statement Transactions: In BigQuery scripting, you can start a transaction with BEGIN TRANSACTION;, execute a series of statements (DML or even certain DDL on temporary tables), then end with COMMIT TRANSACTION; to apply changes atomically, or ROLLBACK TRANSACTION; to cancel if something went wrong (Now Available: Multi Statements Transactions in BigQuery - Medium) (BigQuery transactions over multiple queries, with sessions). All the statements in between will either all succeed (on commit) or all be undone (on rollback). This is crucial when you need to ensure consistency across multiple operations – for example, updating two tables in sync, or doing a “delete then insert” (replace) safely.

Example Transaction: Suppose we want to transfer inventory from one warehouse to another. We need to deduct from one table and add to another, and ensure both succeed or neither does. In a BigQuery script, we could do:

BEGIN TRANSACTION;

-- 1. Insert new stock into Warehouse B's table
INSERT INTO `my_dataset.warehouseB_inventory` (product_id, qty)
SELECT product_id, qty 
FROM `my_dataset.new_stock_delivery`
WHERE warehouse = 'B';

-- 2. Remove that stock from Warehouse A's table
DELETE FROM `my_dataset.warehouseA_inventory`
WHERE product_id IN (
    SELECT product_id FROM `my_dataset.new_stock_delivery` WHERE warehouse = 'B'
);

COMMIT TRANSACTION;

In this example, we start a transaction. Then we INSERT rows for products delivered to Warehouse B and DELETE those products from Warehouse A’s inventory. Only when both statements execute successfully do we call COMMIT to finalize the changes. If any error occurred in between (or if a condition we check indicates a problem), we could issue ROLLBACK TRANSACTION; instead, and BigQuery would undo any partial changes from the insert/delete (in practice, BigQuery would not have made them permanent yet until commit).

BigQuery’s multi-statement transactions can span multiple tables (even across datasets or projects) and multiple DML operations (Transaction Management in Google Bigquery - Stack Overflow) (Transaction Management in Google Bigquery - Stack Overflow). All locks are managed behind the scenes by BigQuery. Note that BigQuery transactions are scoped within a single script execution or session – you cannot yet have an interactive multi-step transaction across separate query jobs; it must be done in one script or procedure call.

Inside a transaction, you can also create temporary tables or use SELECT queries to assist your logic. BigQuery currently does not allow arbitrary DDL on permanent tables inside a transaction (you can only create temp tables or do certain DDL like creating a temp function).

If you don’t explicitly use BEGIN/COMMIT, each DML statement in BigQuery is by default atomic on its own. For many use cases, a single MERGE is enough to apply multiple changes atomically (obviating the need for an explicit transaction). But if you do need to break a complex operation into multiple steps, BigQuery transactions ensure all-or-nothing execution (Transaction Management in Google Bigquery - Stack Overflow).

Legacy SQL: There was no concept of user-controlled transactions in legacy BigQuery. Every query was standalone. The introduction of scripting and transactions in Standard SQL (around 2020) was a significant improvement for ETL workflows that require multi-step operations with rollback on failure.

6. Aggregations and GROUP BY in BigQuery

Aggregations (using functions like SUM, AVG, COUNT, etc. along with GROUP BY clauses) work as in standard SQL, with some BigQuery enhancements.

When you use an aggregate function in a SELECT, you normally need a GROUP BY clause to define how rows are grouped (except when aggregating the entire table). BigQuery Standard SQL follows the standard rule: every non-aggregated select expression must be either in the GROUP BY or be an aggregation of a group. (Legacy SQL was more permissive: it would implicitly treat non-grouped fields as ANY_VALUE, but this could lead to indeterminate results. Standard SQL is stricter, enforcing proper grouping or aggregation.)

Basic Example:

SELECT department, AVG(salary) AS avg_salary, MAX(salary) AS max_salary
FROM `company.employees`
GROUP BY department;

This returns one row per department with the average and maximum salary in each. BigQuery handles large-scale aggregations efficiently in its distributed engine.

BigQuery-Specific Behavior and Functions:

• COUNT(*) returns 0 rows vs 0: If no rows match the WHERE clause, aggregate functions like COUNT return 0 (not null). This is standard SQL behavior. BigQuery adheres to that.

• Approximate Aggregation: For very large datasets, exact distinct counts or quantiles can be expensive. BigQuery provides approximate aggregate functions that trade a tiny error for big speed gains. For example, APPROX_COUNT_DISTINCT(column) uses HyperLogLog++ to estimate the number of distinct values in a column (How to reduce BigQuery costs: 10 effective life hacks for optimization | Lead Panda Media). The result is usually within about 1% of the true value, but the computation is faster and uses less memory, which can be crucial at petabyte scale. Similarly, BigQuery has APPROX_TOP_COUNT and APPROX_TOP_SUM for approximate heavy-hitter analysis. Use these when exact precision isn’t required – it can dramatically reduce query cost on huge tables (How to reduce BigQuery costs: 10 effective life hacks for optimization | Lead Panda Media). (Legacy SQL had COUNT(DISTINCT x) but would error if the number of distinct elements was too high; approximate functions solve that problem in Standard SQL.)

• COUNTIF and SUMIF: BigQuery includes convenient conditional aggregation functions. COUNTIF(condition) counts rows where the condition is true (How to reduce BigQuery costs: 10 effective life hacks for optimization | Lead Panda Media), and SUMIF(expr, condition) sums expr over rows where the condition is true. For example, COUNTIF(status = "ERROR") counts only error status rows. These are shorthand; you can always use COUNT(CASE WHEN ... END) or SUM(IF(..., value, 0)) in standard SQL, but BigQuery provides these for simplicity.

• ANY_VALUE(): BigQuery supports ANY_VALUE(field) which returns an arbitrary value of field from within each group. This is useful when you know all values in the group are the same (or you don’t care which one is picked) and you want to avoid grouping by it. (MySQL has a similar function.) For example, if you group by user_id and want to pick any single email for that user (assuming email doesn’t vary per row), you could do: SELECT user_id, ANY_VALUE(email) FROM ... GROUP BY user_id;. This function was introduced to help with migrating queries from legacy SQL’s behavior of non-grouped columns (Select All Columns Except Some in Google BigQuery? - Stack Overflow).

• Grouping Sets / Rollup / Cube: As mentioned, BigQuery allows advanced grouping. For instance:

    SELECT region, product, SUM(sales) as total_sales
    FROM sales
    GROUP BY ROLLUP(region, product);
  

  This query would produce aggregated totals at multiple levels: by (region, product), by region (overall product total per region), and a grand total (all regions/products) (Google BigQuery now supports Cubes | by Christianlauer - Medium). GROUP BY CUBE(a, b) would produce all combinations (by a, by b, by both, by neither). And GROUPING SETS allows explicit list of group combinations. BigQuery also provides the GROUPING() function to identify subtotal rows (returning 1 for subtotal vs 0 for detail row). These features let you produce pivot-table style summaries in a single query. (These capabilities did not exist in legacy SQL; they are part of Standard SQL improvements.)

• Distinct Aggregates: BigQuery can do COUNT(DISTINCT col) like other SQLs. One nuance: BigQuery in some cases allows multiple distinct aggregates in the same query (e.g., COUNT(DISTINCT col1), COUNT(DISTINCT col2)), which some SQL databases do not unless you use more complex workarounds. BigQuery can handle multiple distinct counts by internally using approximation or additional query stages. Just be mindful of performance: each distinct aggregate may add overhead.

Example – Using APPROX_COUNT_DISTINCT:

SELECT 
  country, 
  APPROX_COUNT_DISTINCT(user_id) AS approx_unique_users
FROM `my_dataset.web_logs`
GROUP BY country;

This query quickly estimates the number of unique users per country in the web_logs table using HyperLogLog++ (How to reduce BigQuery costs: 10 effective life hacks for optimization | Lead Panda Media). The approximation significantly reduces computational load for extremely large log tables with minimal loss of accuracy (usually ~1% error). If exact counts are needed, you could use COUNT(DISTINCT user_id) at higher cost.

Example – Grouping Sets (Rollup):

SELECT 
  region, product, 
  SUM(sales) AS total_sales,
  GROUPING(region) AS region_total_flag,
  GROUPING(product) AS product_total_flag
FROM `my_dataset.sales`
GROUP BY ROLLUP(region, product);

This will produce rows for each (region, product) combination, plus subtotal rows where product is NULL (total per region) and a grand total row where both region and product are NULL (Google BigQuery now supports Cubes | by Christianlauer - Medium). The GROUPING() function returns 1 when the column is NULL because of a subtotal. For example, the grand total row will have region_total_flag=1 and product_total_flag=1. Such SQL constructs can eliminate the need for manual UNION of multiple grouping queries.

7. Using HAVING with Aggregations

The HAVING clause is used to filter aggregated results. It is applied after the GROUP BY step, unlike WHERE which filters before grouping. In BigQuery (as in standard SQL), you use HAVING to impose conditions on aggregate values (SUM, COUNT, AVG, etc.) for each group (A Complete Guide to WHERE, HAVING, and QUALIFY in SQL).

For example, if we want to find departments with more than 3 employees, we would use HAVING on a COUNT:

SELECT department, COUNT(employee_id) AS num_employees
FROM `owox-analytics.myDataset.employee_data`
WHERE status = 'active'
GROUP BY department
HAVING COUNT(employee_id) > 3;

(A Complete Guide to WHERE, HAVING, and QUALIFY in SQL)

This query groups employees by department (considering only active employees due to the WHERE clause), then the HAVING clause filters out any groups that have 3 or fewer employees (A Complete Guide to WHERE, HAVING, and QUALIFY in SQL). The result will only include departments with count > 3.

Let’s break down the logic:

• The WHERE filter (status = 'active') runs first, limiting rows to active employees before grouping.

• GROUP BY department then aggregates the data so we have one row per department.

• COUNT(employee_id) computes the number of employees in each department.

• The HAVING COUNT(employee_id) > 3 condition is evaluated on each aggregated group, and discards any department that doesn’t satisfy it (A Complete Guide to WHERE, HAVING, and QUALIFY in SQL).

• The SELECT outputs the department name and the count (alias num_employees).

So, if “HR” had 5 active employees and “Sales” had 2, the result would include “HR, 5” but not “Sales, 2”.

Important notes about HAVING in BigQuery Standard SQL:

• You can reference aggregate expressions by alias in HAVING. In our example, we could have written HAVING num_employees > 3 because we alias COUNT(employee_id) as num_employees. BigQuery (like many SQLs) allows this alias usage in HAVING.

• If there is no GROUP BY, a HAVING still can be used – it would treat the entire result as one group. E.g., SELECT SUM(x) as total FROM table HAVING total > 100 would return nothing if the total is not >100. But usually, HAVING is paired with GROUP BY.

• Legacy SQL in BigQuery also had HAVING, with similar usage, but in legacy SQL you might see HAVING used without GROUP BY as a workaround to filter on an aggregate. In Standard SQL, you can often use a window function + QUALIFY or a subquery instead, but HAVING remains the direct way to filter grouped results.

Example – HAVING vs WHERE:

Suppose you want all products that have total sales > 1000 in a sales table:

-- Incorrect: This will fail or return wrong results
SELECT product, SUM(amount) 
FROM sales
WHERE SUM(amount) > 1000   -- not allowed, aggregate in WHERE
GROUP BY product;

This is invalid because you cannot use an aggregate (SUM(amount)) in a WHERE clause. The correct approach is to use HAVING:

SELECT product, SUM(amount) as total_sales
FROM sales
GROUP BY product
HAVING total_sales > 1000;

Now, HAVING total_sales > 1000 will filter the grouped products, and only those with sum > 1000 remain. In BigQuery, this works as expected (you could also write HAVING SUM(amount) > 1000 directly) (A Complete Guide to WHERE, HAVING, and QUALIFY in SQL). If you had an initial filter on individual rows (say, WHERE region = 'US' to consider only US sales), that would be applied before the grouping.

To summarize, use WHERE for conditions on raw rows (especially non-aggregated columns) and HAVING for conditions on aggregated results. BigQuery enforces this order of execution, just like standard SQL, ensuring that HAVING only sees grouped results (A Complete Guide to WHERE, HAVING, and QUALIFY in SQL) (A Complete Guide to WHERE, HAVING, and QUALIFY in SQL).

8. Window Functions (OVER, PARTITION BY) – Advanced SQL Analytics

Window functions (also known as analytic functions) are powerful in BigQuery for performing calculations across sets of rows related to the current row, without collapsing those rows into a single result. They use the OVER(...) clause to define a “window” of rows for the calculation. BigQuery’s implementation is fully compliant with standard SQL window functions and adds a few functions of its own.

What Are Window Functions?
Unlike GROUP BY aggregations which reduce rows (one result per group), window functions produce a value for each row while looking at a window of multiple rows (which you define using PARTITION BY and ORDER BY in the OVER clause). This lets you compute running totals, ranks, moving averages, percentiles, lead/lag comparisons, etc., all while retaining the detail of each row (Using Window Functions in BigQuery: A 2025 Guide) (Using Window Functions in BigQuery: A 2025 Guide). They are called “window” functions because each calculation considers a frame of rows (the window) relative to the current row’s position.

Basic Syntax:

function_name(expression) 
OVER (
  [PARTITION BY partition_columns...] 
  [ORDER BY sort_columns [ASC|DESC]] 
  [window_frame_clause]
)

• PARTITION BY divides the data into partitions (sub-groups) for the function, similar to GROUP BY but without collapsing – the function will reset at partition boundaries (Using Window Functions in BigQuery: A 2025 Guide) (Using Window Functions in BigQuery: A 2025 Guide).

• ORDER BY defines the ordering within each partition that the window function will use (e.g., for running totals or ranking).

• The optional frame clause (like ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) further refines which rows in the partition are considered for that row’s calculation.

BigQuery supports three main categories of window functions (Using Window Functions in BigQuery: A 2025 Guide):

1. Aggregate functions as window functions: e.g. SUM(), AVG(), MIN(), MAX(), COUNT() can be used as window functions by adding OVER(). This produces a running or total aggregate value per row rather than one per group. By default, without a frame clause, SUM() OVER (PARTITION BY X) gives a total sum per partition (like a group total attached to each row of the group), and SUM() OVER (ORDER BY Y) gives a cumulative sum from the start up to the current row (the default frame is UNBOUNDED PRECEDING ... CURRENT ROW if ORDER BY is present).

1. Ranking functions: ROW_NUMBER(), RANK(), DENSE_RANK(), NTILE(n). These assign rank numbers based on ordering within partitions. For example, ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary DESC) will give 1 to the highest-paid person in each department, 2 to the next, and so on.

1. Analytic functions for navigation: e.g. LAG(value, N), LEAD(value, N) to pull data from previous or next rows, FIRST_VALUE(value)/LAST_VALUE(value) to get first/last in the window, etc. These allow comparisons across row boundaries (e.g., compare this row’s value to last week’s value in a time series).

Example 1 – Ranking: Let’s rank employees by salary within each department:

SELECT 
  employee_id,
  department,
  salary,
  RANK() OVER (PARTITION BY department ORDER BY salary DESC) AS salary_rank
FROM `company.employees`;

This will output each employee and their rank within their department by salary (1 = highest salary in that dept, etc.). The PARTITION BY department means ranking restarts for each department (Using Window Functions in BigQuery: A 2025 Guide). The ORDER BY salary DESC means highest salary gets rank 1. Employees with equal salaries get the same rank number (and rank numbers will have gaps if there's a tie, since RANK is being used; use DENSE_RANK() if you want no gaps). If we wanted a global rank ignoring departments, we’d omit the partition clause. If we wanted row number instead (no gaps, strict ordering), we’d use ROW_NUMBER(). This kind of query could help, for example, to find the top 3 earners in each department (you would then add QUALIFY salary_rank <= 3 in BigQuery to filter the top 3 per dept).

Example 2 – Running Total: Suppose we have a table of daily sales and we want a running cumulative sales amount by date:

SELECT 
  sales_date,
  amount,
  SUM(amount) OVER (ORDER BY sales_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total
FROM `my_dataset.daily_sales`
ORDER BY sales_date;

This uses SUM(amount) OVER (ORDER BY sales_date ... CURRENT ROW) to calculate a cumulative sum up to the current row (assuming one row per date, this is a running total over time). The frame ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW is actually the default for an ORDER BY if not specified, so we could omit it and BigQuery assumes cumulative sum from the start. The result will list each date, the sales for that date, and the total sales from the beginning up through that date. Unlike a GROUP BY, we still see one row per date (not one final total). This is very useful for time-series analysis. In BigQuery, window aggregate functions like this allow things like moving averages by using a frame of a certain width (e.g., last 7 days).

Example 3 – Partitioned Window: Using the same sales example, if we partition by year and compute running totals within each year:

SELECT 
  YEAR(sales_date) as year,
  sales_date,
  amount,
  SUM(amount) OVER (
       PARTITION BY YEAR(sales_date) 
       ORDER BY sales_date 
       ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
  ) AS year_running_total
FROM `my_dataset.daily_sales`
ORDER BY sales_date;

Here PARTITION BY YEAR(sales_date) means the running total resets at the start of each year (each year treated separately) (Using Window Functions in BigQuery: A 2025 Guide). So Jan 1 of each year starts a new cumulative sum. Partitioning is helpful when you want window calculations within subgroups (like per department, per year, per region, etc.).

Window vs. GROUP BY: It’s worth noting the difference: GROUP BY collapses rows, but window functions produce values that are attached to each row. For instance, a GROUP BY year to get total sales per year would return 1 row per year. The window SUM...PARTITION BY year returns the total per year on each row of that year’s data (Using Window Functions in BigQuery: A 2025 Guide) (Using Window Functions in BigQuery: A 2025 Guide). You might use that to calculate each day’s share of the annual total, for example. Window functions let you mix detail and summary in one result.

BigQuery Specifics: BigQuery supports all standard window functions. In terms of performance, window functions are efficient in BigQuery, but do consider the data volume – partitioning by a high-cardinality field or not partitioning at all (window over the entire table) means a lot of data to process in each function. If you only need a grouped result, use GROUP BY; use window functions when you need the detail with the analytic result. BigQuery also now supports the QUALIFY clause (as discussed) to filter the output of window functions easily, which is particularly useful with ranking and row_number queries (e.g., QUALIFY RANK() ... = 1 to get top N per group).

Practical Example – Combining Window and Aggregation:

A common use-case is to find the contribution of each row to a group total. We can use a window sum (total per group on each row) along with the row’s value:

SELECT 
  department,
  employee,
  salary,
  SUM(salary) OVER (PARTITION BY department) AS total_dept_salary,
  salary * 100.0 / SUM(salary) OVER (PARTITION BY department) AS percent_of_department
FROM `company.employees`;

This produces each employee’s salary, the total salary of their department (repeated for all in dept), and the percentage of the department’s payroll that the employee’s salary represents. Here SUM(salary) OVER (PARTITION BY department) gives the department total attached to each row (Using Window Functions in BigQuery: A 2025 Guide), and we use it in a calculation. This kind of query would be difficult to do with pure GROUP BY (you’d have to join the aggregated results back to the detail), but window functions make it straightforward.

Legacy SQL: Legacy BigQuery did not have window functions like this. Users often had to do self-joins or subqueries to achieve similar results, which is more cumbersome and sometimes less efficient. The introduction of Standard SQL with window functions greatly simplified analytical queries in BigQuery.

9. BigQuery Performance Optimization Tips

BigQuery is a columnar, massively parallel query engine. Query cost and performance are largely determined by how much data you scan and process. Here are some BigQuery-specific optimization tips to make queries efficient:

• Select Only Needed Columns: Avoid SELECT * unless you truly need all columns. BigQuery charges by data scanned, which is per byte of column data read. Selecting unnecessary columns causes BigQuery to read more data and slows down the query (How to reduce BigQuery costs: 10 effective life hacks for optimization | Lead Panda Media). Always project only the columns you need. For example, if you only need two columns from a 100-column table, write SELECT col1, col2 – this could be the difference between scanning 1 GB versus 100 GB. This is one of the simplest and most effective cost optimizations in BigQuery (Maximizing Efficiency: Best Practices for BigQuery | Galaxy.ai).

• Filter Early and Specifically: Use WHERE clauses to restrict data as much as possible. BigQuery will prune partitions (if partitioned) and skip irrelevant data. Also, filtering on clustered columns leverages sorted storage to read less. For example, if your table is partitioned by date, always include a date range filter in the WHERE clause so BigQuery can scan only the needed partitions.

• Partition Your Tables: When creating tables, use BigQuery’s partitioning feature for large fact tables. Partition by a date or integer range that you commonly filter on (e.g., PARTITION BY date for a daily log table). Queries with a filter on the partition column will then scan only the relevant partition rather than the entire table, greatly reducing I/O (BigQuery Best Practices to Optimize Cost and Performance). Time-unit partitioning (DAY, MONTH, etc. on a TIMESTAMP/DATE) or integer range partitioning can be chosen based on data. For example, a web logs table partitioned by date might cut a year’s data (365 days) into 365 partitions; a query on one month would only read ~30 partitions (1/12 of data) (BigQuery Best Practices to Optimize Cost and Performance). Partitioning also helps manage data lifecycle (you can set expiration for partitions).

• Cluster Your Tables: Clustering sorts data on specified columns, which can dramatically speed up filtering and aggregating on those columns (BigQuery Best Practices to Optimize Cost and Performance). For example, if you cluster a table by user_id, all rows with the same user_id are stored close together. A query like WHERE user_id = X will only read a small portion of each partition (only the blocks for that user). Clustering is often used in combination with partitioning: e.g., partition by date and cluster by user_id or product category. This way, BigQuery first prunes partitions by date, then within each partition it can binary search on the clustered column to find relevant data (BigQuery Best Practices to Optimize Cost and Performance). Clustering also helps GROUP BY performance on the clustered columns, because data comes pre-sorted (reducing shuffle).

• Denormalize Data (Use Nested Fields): BigQuery performs best with fewer large tables rather than many joins. Where appropriate, denormalize your schema – e.g., instead of storing user info in one table and events in another (and joining), you might store user info as repeated/nested fields within the events table. BigQuery’s storage can handle repeated (array) fields efficiently. This avoids join overhead; querying nested data is essentially a local unpacking operation, not a distributed join (BigQuery Best Practices to Optimize Cost and Performance) (Transaction Management in Google Bigquery - Stack Overflow). Joins in BigQuery are still fine for reasonably sized dimensions, but for very large datasets, reducing the number of joins (through nesting or pre-joining data) can improve performance. As a rule: small lookup tables (dimensions) are fine to join, but consider denormalizing big fact tables or using nested structures for one-to-many relations (like an order with an array of items) (BigQuery Best Practices to Optimize Cost and Performance). This trades some storage space for speed, which is usually worthwhile in BigQuery’s cost model (storage is cheap; computation is comparatively expensive) (Transaction Management in Google Bigquery - Stack Overflow).

• Avoid Excessive JOINs on Huge Tables: If you must join large tables, ensure the join keys are well-partitioned or clustered if possible. Use JOIN each (the default in Standard SQL) which is a shuffle join. In legacy SQL there was a concept of “JOIN EACH” required for big joins; in Standard SQL, BigQuery automatically shuffles data for large joins. However, joining two enormous tables will multiply the data that needs to be processed. Wherever possible, filter both sides of the join heavily and only select necessary columns before joining (possibly using subqueries or CTEs for pre-filtering).

• Use WITH (CTE) or Temp Tables to Break Queries: BigQuery will happily execute extremely complex queries in one go, but sometimes breaking a query into stages can help the optimizer or reduce duplicate work. For example, if you have a subquery that is used multiple times, consider materializing it as a CTE (Common Table Expression) or temp table so it’s computed once and reused. BigQuery will not automatically reuse results from identical subqueries unless you explicitly CTE them. Using CTEs can also improve readability, but note BigQuery does not materialize CTEs by default – they are inlined. If you want to materialize intermediate results (to reduce data scanned in subsequent steps), you might need to write to a temporary table in one step, then query it in the next.

• Leverage BigQuery’s Caching: By default, BigQuery caches the results of query for ~24 hours for the same user if the underlying data hasn’t changed. If you rerun an identical query, you won’t be charged and it will return faster (because it hits the cache) (How to reduce BigQuery costs: 10 effective life hacks for optimization | Lead Panda Media). This is automatic. However, note that the cache is per user — if your colleague runs the same query, they won’t get your cached results (How to reduce BigQuery costs: 10 effective life hacks for optimization | Lead Panda Media). For dashboards or repeated analyses by the same person, result caching is beneficial. For multi-user scenarios, consider caching at the application level or using scheduled queries/materialized views.

• Use Materialized Views for Frequent Aggregations: BigQuery supports materialized views which automatically cache the results of a query (typically an aggregation on a table) and incrementally update as the base table changes. If you have a very expensive aggregation that many queries run (like daily totals per category), a materialized view can serve those results quickly and save cost. Queries that can use the materialized view (e.g., they request data that the view pre-computed) will automatically do so. Design the materialized view on the heavy computation part of your data.

• Consider BI Engine for Dashboards: If you are doing a lot of repetitive queries (especially dashboard queries that hit the same tables with filter variations), Google’s BI Engine can cache data in-memory for super-fast responses. It’s not a query tip per se, but a feature to be aware of for performance tuning in BigQuery when used with tools like Data Studio/Looker.

• Use EXPLAIN to Understand Query Plans: BigQuery now has an EXPLAIN statement that shows the query execution plan and stages. This can help identify if a query is doing a large shuffle or scan that you didn't expect. For instance, it can reveal if a partition filter is being applied or if a join is causing a huge data shuffle. Using EXPLAIN ANALYZE will actually run the query and provide timing and resource usage statistics for each step. This is useful for performance debugging – if a certain stage takes 90% of time, focus optimization efforts there.

• Use TABLESAMPLE for Testing: When developing queries on huge tables, use the TABLESAMPLE SYSTEM() clause to run on a fraction of the data (How to reduce BigQuery costs: 10 effective life hacks for optimization | Lead Panda Media). For example, FROM big_table TABLESAMPLE SYSTEM (1 PERCENT) will read a 1% random sample of the table. This lets you test query logic quickly and cheaply on large data (with the caveat that results are approximate because it’s a sample). Once you are satisfied with the query on 1%, you can run it on 100%. This can save a lot of time and cost during development iterations.

• Approximate Results for Big Data: As mentioned, if you only need an estimate (like approx distinct count, top frequencies, etc.), use BigQuery’s approximate functions. For example, APPROX_QUANTILES can get percentile estimates without sorting all data, and HLL_COUNT.INIT / HLL_COUNT.MERGE functions allow creating your own HyperLogLog distinct counts over streams of data (advanced usage). Using these can cut down CPU time for large-scale analytics (How to reduce BigQuery costs: 10 effective life hacks for optimization | Lead Panda Media).

• Monitoring and Slot Tuning: BigQuery automatically manages resources, but if you have a critical query or consistent workload, consider reserving slots (with BigQuery Reservations) and using concurrency tuning. For one-off performance, ensure you’re using the default parallelism effectively by not having overly complex single-threaded UDFs or external calls that serialize things.

In summary, BigQuery performance is about scanning less data and distributing work efficiently. Techniques like partitioning, clustering, filtering, and pre-aggregating data (or using materialized views) directly reduce the amount of data scanned. Others, like using denormalized schemas and caching, reduce the amount of work per query. Following these best practices can lead to massive improvements in query speed and cost – for example, a query that originally scanned 1 TB daily could scan only 100 GB with proper partitioning and clustering, and further drop to 10 GB if you only select needed columns and use a WHERE clause, resulting in faster execution and 1/100th the cost.

Lastly, always test your optimizations. BigQuery’s web UI or CLI will show you how many bytes a query will process before you run it (when you click “Query Validator” or use dry-run flag). Use that as a guide: small changes in the query can sometimes accidentally increase bytes scanned. The goal is to minimize that while still getting correct results. BigQuery’s design (columnar + parallel) will handle large data if you follow these patterns to help it avoid unnecessary scans and shuffles.