AdaGrad - TorchCode

Problem Description

Implement AdaGrad from scratch. AdaGrad gives each weight its own learning rate by accumulating the sum of squared gradients. Frequently & strongly updated weights get progressively smaller steps — great for sparse features, but because the cache only grows, the effective learning rate keeps shrinking and can stall.

Signature

class MyAdaGrad:
    def __init__(self, params, lr=1e-2, eps=1e-10): ...
    def step(self): ...
    def zero_grad(self): ...

Algorithm (per parameter)

cache += grad² p -= lr * grad / (sqrt(cache) + eps)

Template

Implement the class below. Use only basic PyTorch operations.

# ✏️ YOUR IMPLEMENTATION HERE

class MyAdaGrad:
    def __init__(self, params, lr=1e-2, eps=1e-10):
        pass  # store params; init cache buffers to zeros

    def step(self):
        pass  # cache += grad**2 ; p -= lr*grad/(sqrt(cache)+eps)

    def zero_grad(self):
        pass  # zero all gradients

Test Your Implementation

Use this code to debug before submitting.

# 🧪 Debug
torch.manual_seed(0)
w = torch.randn(4, 3, requires_grad=True)
opt = MyAdaGrad([w], lr=0.1)
for i in range(5):
    loss = (w ** 2).sum()
    loss.backward()
    opt.step()
    opt.zero_grad()
    print(f'Step {i}: loss={loss.item():.4f}')

Reference Solution

Try solving it yourself first! Click below to reveal the solution.

# ✅ SOLUTION

class MyAdaGrad:
    def __init__(self, params, lr=1e-2, eps=1e-10):
        self.params = list(params)
        self.lr = lr
        self.eps = eps
        self.cache = [torch.zeros_like(p) for p in self.params]

    def step(self):
        with torch.no_grad():
            for i, p in enumerate(self.params):
                if p.grad is None:
                    continue
                self.cache[i] = self.cache[i] + p.grad ** 2
                p -= self.lr * p.grad / (torch.sqrt(self.cache[i]) + self.eps)

    def zero_grad(self):
        for p in self.params:
            if p.grad is not None:
                p.grad.zero_()

Tips

Run Locally

For interactive practice with auto-grading, run TorchCode locally:
pip install torch-judge then use check("adagrad")

Key Concepts

Monotonically growing squared-gradient cache → the per-parameter step size only ever decreases. Strong for sparse gradients; the shrinking step is exactly what RMSProp fixes with a decaying average. Covered in DL Module 7 §5.