DVC Basics: Track Your First ML Dataset in 3 Commands

The Git Push That Cost Me 45 Minutes

I once watched a colleague accidentally push 2GB of training images to GitHub. The commit took 45 minutes to complete—and then GitHub rejected it anyway because of the 100MB file size limit. We had to use git filter-branch to rewrite history, which broke everyone’s local clones.

The fix took an afternoon. The solution was DVC.

DVC (Data Version Control) handles large files and datasets the same way Git handles code: it tracks changes, enables versioning, and lets you switch between dataset versions with a single command. But instead of storing the actual data in your Git repo, DVC stores lightweight pointer files (.dvc files) and pushes the real data to remote storage like S3, GCS, or even a local folder.

Here’s the thing most tutorials miss: you can get meaningful value from DVC with exactly three commands. No need to learn the entire ecosystem on day one.

Why Git Alone Fails for ML Data

Git was designed for text files. It computes diffs between versions, which works great for source code but becomes pathological for binary data. A 500MB image dataset doesn’t diff well—Git stores complete copies of each version, and your .git folder balloons.

The math isn’t pretty. If you have a 1GB dataset and make 10 versions with small modifications, you might end up with 8-9GB in your Git history. I’ve seen repos where .git was larger than the actual project.

Git LFS (Large File Storage) partially solves this, but it has its own quirks. GitHub charges for bandwidth after 1GB/month, and switching between LFS and non-LFS hosting is painful. DVC gives you more flexibility—you can use any storage backend and switch between them without rewriting history.

The 3-Command Setup

Let’s track a dataset. I’ll use a realistic scenario: a folder of images for a classification task.

First, install DVC. The [s3] extra adds S3 support—swap it for [gs] or [azure] depending on your cloud provider:

pip install dvc[s3]  # dvc 3.50+ recommended

Command 1: dvc init

In your existing Git repo (you need Git initialized first), run:

dvc init

This creates a .dvc/ directory with configuration files. DVC immediately adds these to Git:

.dvc/
  ├── config        # remote storage settings
  ├── .gitignore
  └── plots/        # for experiment visualization
.dvcignore          # like .gitignore for DVC

You’ll see output like:

Initialized DVC repository.

You can now commit the changes to git.

+---------------------------------------------------------------------+
|                                                                     |
|        DVC has enabled anonymous aggregate usage analytics.         |
|    Read the analytics documentation (and hierarchical opt-out)      |
|                   https://dvc.org/doc/user-guide/analytics          |
|                                                                     |
+---------------------------------------------------------------------+

Commit these files:

git add .dvc .dvcignore
git commit -m "Initialize DVC"

Command 2: dvc add

Now track your dataset. Assume you have a data/images/ folder with 500MB of training images:

dvc add data/images

This does three things:
1. Computes MD5 hashes of all files in the folder
2. Creates data/images.dvc—a tiny text file containing the hash
3. Adds data/images to .gitignore so Git ignores the actual data

The .dvc file looks like this:

outs:
- md5: a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6.dir
  size: 524288000
  nfiles: 15000
  path: images

That md5 hash is the key. When you change even one image, the hash changes, and DVC knows the dataset has been modified.

Commit the pointer file:

git add data/images.dvc data/.gitignore
git commit -m "Track images dataset with DVC"

Command 3: dvc push

Before pushing, you need a remote. This is where your actual data lives. For local testing, a folder works fine:

dvc remote add -d myremote /path/to/dvc-storage

The -d flag sets this as the default remote. For S3:

dvc remote add -d myremote s3://my-bucket/dvc-storage

Now push:

dvc push

DVC uploads your data to the remote storage. On my M1 MacBook with a 500MB dataset going to a local folder, this took about 8 seconds. S3 uploads depend entirely on your network—budget for 2-3 minutes for the first push of a large dataset.

That’s it. Three commands: init, add, push. Your dataset is now versioned.

What Happens When You Clone

Here’s where it gets interesting. When a teammate clones your repo, they get the code and the .dvc pointer files, but NOT the actual data:

git clone https://github.com/your-repo.git
cd your-repo
ls data/
# images.dvc  (just the pointer file)

To get the data, they run:

dvc pull

DVC reads the hash from images.dvc, fetches the matching data from remote storage, and populates data/images/. This is the magic—you can have a 50GB dataset in your project, and git clone still takes seconds.

Versioning Dataset Changes

The real power shows up when your dataset evolves. Say you add 1000 new images to your training set:

# Copy new images into the folder
cp ~/new_batch/*.jpg data/images/

# Re-run dvc add to update the hash
dvc add data/images

# Commit the updated pointer
git add data/images.dvc
git commit -m "Add batch 2 training images"

# Push the new version
dvc push

Now you have two versions of your dataset, tied to two Git commits. Want to go back to the original? Check out that commit and pull:

git checkout HEAD~1
dvc checkout
# data/images/ now contains the original 15000 images

The dvc checkout command reads the .dvc file at the current Git state and restores the matching data from your local cache (.dvc/cache/). If the data isn’t cached locally, you’d need dvc pull first.

How DVC Storage Actually Works

Under the hood, DVC uses content-addressable storage. Each file is stored by its MD5 hash, not its filename. If two datasets share the same image, DVC stores it once.

The cache lives in .dvc/cache/ by default:

.dvc/cache/files/md5/
├── a1/
│   └── b2c3d4e5f6...
├── c3/
│   └── d4e5f6g7h8...

When you dvc push, files go from this local cache to the remote. When you dvc pull, the reverse happens. The .dvc pointer file tells DVC which hashes to fetch.

This deduplication is significant for iterative ML work. If you’re experimenting with different preprocessing pipelines and keep multiple dataset versions, unchanged files aren’t duplicated in storage.

Real Output: A Complete Session

Let me show what a typical session looks like. I’ll create a small demo to make the output realistic:

import os
import numpy as np
from PIL import Image

# Generate some fake training images
os.makedirs('data/images', exist_ok=True)

for i in range(100):
    # Random 224x224 RGB images
    arr = np.random.randint(0, 255, (224, 224, 3), dtype=np.uint8)
    img = Image.fromarray(arr)
    img.save(f'data/images/img_{i:04d}.jpg', quality=85)

print(f"Created {len(os.listdir('data/images'))} images")
print(f"Total size: {sum(os.path.getsize(f'data/images/{f}') for f in os.listdir('data/images')) / 1024 / 1024:.2f} MB")

Output:

Created 100 images
Total size: 2.34 MB

Now the DVC workflow:

$ git init
Initialized empty Git repository in /Users/demo/ml-project/.git/

$ dvc init
Initialized DVC repository.
...

$ dvc add data/images
Adding...
To track the changes with git, run:

    git add data/.gitignore data/images.dvc

To enable auto staging, run:

    dvc config core.autostage true

$ cat data/images.dvc
outs:
- md5: 7f3b4c2e8a9d1f0e6b5c4a3d2e1f0987.dir
  size: 2453504
  nfiles: 100
  path: images

$ dvc remote add -d local_storage /tmp/dvc-storage
Setting 'local_storage' as a default remote.

$ dvc push
100 files pushed

$ ls /tmp/dvc-storage/files/md5/
00  0a  14  1e  28  ...  (many hash prefix folders)

Common Gotcha: The .gitignore Trap

When you run dvc add data/images, DVC creates data/.gitignore containing:

/images

This prevents Git from tracking the actual data folder. But here’s what caught me once: if you already had a .gitignore in your repo root that tracked data/, you might have conflicts.

My recommendation: let DVC manage the gitignores. Don’t manually add data paths to your root .gitignore—let the dvc add command handle it.

Integrating with ML Pipelines

DVC goes much deeper than file tracking. The dvc.yaml pipeline format lets you define reproducible ML workflows:

stages:
  preprocess:
    cmd: python preprocess.py
    deps:
      - preprocess.py
      - data/raw
    outs:
      - data/processed

  train:
    cmd: python train.py
    deps:
      - train.py
      - data/processed
    outs:
      - models/model.pkl
    metrics:
      - metrics.json

Running dvc repro executes only the stages whose dependencies changed. But this is day-two learning—for your first dataset tracking use case, the three commands I showed are enough.

DVC vs Git LFS: When to Pick Which

Git LFS integrates more seamlessly with GitHub’s UI—you can browse LFS files in the web interface. DVC requires separate storage configuration.

But DVC wins on:
– Storage flexibility: Use any S3-compatible service, GCS, Azure, SSH, even a local NAS
– ML-specific features: Pipelines, experiments, metrics tracking
– No bandwidth fees: GitHub LFS charges after 1GB/month; your own S3 bucket doesn’t care
– Deduplication: Content-addressable storage across all versions

For pure “I just need to version large files,” Git LFS is simpler. For ML projects where you’ll eventually want reproducible pipelines, start with DVC.

I’d pick DVC for any project over 500MB of data or more than 3 team members. The flexibility pays off.

The Cache Can Eat Your Disk

One surprise: DVC keeps a local cache of all data you’ve ever tracked. Run a few experiments with different dataset versions, and .dvc/cache/ can grow unexpectedly large.

Check your cache size:

du -sh .dvc/cache/
# 4.2G    .dvc/cache/

To clean old cached files (keeping only what current .dvc files reference):

dvc gc --workspace

Be careful with dvc gc flags—the wrong combination can delete data you still need. I’d recommend reading the dvc gc documentation before running it aggressively. And if you’re debugging at 2am and need to stay sharp, Dark Chocolate Espresso Beans have saved me more than once.

Remote Storage Configuration Examples

The remote setup syntax varies by provider:

AWS S3:

dvc remote add -d s3remote s3://mybucket/dvc-data
dvc remote modify s3remote region us-west-2
# Credentials via AWS CLI config or environment variables

Google Cloud Storage:

dvc remote add -d gcsremote gs://mybucket/dvc-data
# Credentials via gcloud auth or service account JSON
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa-key.json

SSH/SFTP (self-hosted):

dvc remote add -d sshremote ssh://[email protected]/path/to/storage
dvc remote modify sshremote password mysecretpassword  # or use SSH keys

The config gets stored in .dvc/config, which you commit to Git (passwords go in .dvc/config.local, which stays out of version control).

Gotcha: Hash Changes on Different Machines

DVC uses MD5 by default. In rare cases—different filesystems, weird symlink handling—the same files can produce different hashes on different machines. I haven’t hit this myself, but the DVC docs mention it as a possibility on network-mounted drives.

If you’re on a team with mixed Mac/Linux/Windows setups, watch out for line ending differences in any text files you track. The safest approach: only use DVC for true binary data (images, model weights, parquet files), and let Git handle code and config files.

How DVC Calculates Hashes for Directories

When you dvc add a directory, DVC doesn’t hash the entire folder as one blob. It hashes each file individually, then creates a .dir file listing all the individual hashes:

[
  {"relpath": "img_0000.jpg", "md5": "a1b2c3..."},
  {"relpath": "img_0001.jpg", "md5": "d4e5f6..."},
  ...
]

This .dir file itself gets hashed, and that hash goes into the .dvc pointer. The benefit: if you add one image to a 10,000-image folder, only the new image gets uploaded on dvc push. The existing images are already in storage.

The math for storage efficiency is roughly:

$$S_{total} = \sum_{v=1}^{V} S_{new,v}$$

where $S_{total}$ is total storage used, $V$ is the number of versions, and $S_{new,v}$ is only the new/changed files in version $v$. Compare this to naive versioning:

$$S_{naive} = \sum_{v=1}^{V} S_{full,v}$$

For a 10GB dataset with 5 versions where each version changes 5% of files, DVC uses roughly $10 + 4 \times 0.5 = 12$GB, while naive storage uses DOLLAR_AMOUNT_1$GB.

Tracking Model Weights Too

Most tutorials focus on datasets, but DVC handles model weights equally well:

dvc add models/resnet50_finetuned.pth
git add models/resnet50_finetuned.pth.dvc models/.gitignore
git commit -m "Add fine-tuned ResNet50 weights"
dvc push

Now your model weights are versioned alongside your code. Each Git tag or release can reference a specific model version. Training on a new dataset? The updated weights get a new hash, and you can always roll back.

What I Wish I’d Known Earlier

dvc status is your friend:

$ dvc status
data/images.dvc:
    changed outs:
        modified:           data/images

This tells you when tracked files have changed but haven’t been re-added. Before committing, always run dvc status to catch drift between your data and your .dvc files.

Also: dvc diff shows what changed between commits:

$ dvc diff HEAD~1
Modified:
    data/images.dvc:
        Added: 1000 files
        Deleted: 0 files

These commands make code review for data changes actually possible.

FAQ

Q: Can DVC work without Git?
No. DVC requires an initialized Git repository. The .dvc pointer files are tracked by Git, which is how DVC ties dataset versions to code commits. You need both tools working together.

Q: How much storage does DVC use compared to just copying folders?
DVC’s content-addressable storage means identical files across versions are stored only once. For a 1GB dataset with 10 versions where 80% of files stay unchanged, you’d use roughly 2.8GB instead of 10GB. The exact savings depend on how much your data actually changes between versions.

Q: Can I use DVC with existing S3 buckets that have other data?
Yes. DVC stores data in a subfolder structure based on content hashes. You can point DVC at s3://mybucket/dvc-data/ and your other bucket contents stay untouched. Just make sure your IAM permissions allow DVC to read/write that path.

Moving Forward

Start simple: track one dataset with dvc init, dvc add, dvc push. Get comfortable with the pull/checkout workflow for switching versions. Once that’s natural, explore dvc.yaml pipelines for reproducible training runs.

For small solo projects under 100MB, Git LFS is honestly fine. But the moment you’re collaborating with others or handling real ML data sizes, DVC’s flexibility is worth the setup.

One thing I’m still uncertain about: how DVC performs with millions of small files (like individual audio clips for speech recognition). The documentation mentions potential slowdowns during hashing, but I haven’t personally tested datasets with more than 50,000 files. If anyone has benchmarks on that, I’d love to see them.