PyTorch教程-12.11。學(xué)習(xí)率調(diào)度

到目前為止，我們主要關(guān)注如何更新權(quán)重向量的優(yōu)化算法，而不是更新權(quán)重向量的速率。盡管如此，調(diào)整學(xué)習(xí)率通常與實(shí)際算法一樣重要。有幾個(gè)方面需要考慮：

最明顯的是學(xué)習(xí)率的大小很重要。如果它太大，優(yōu)化就會(huì)發(fā)散，如果它太小，訓(xùn)練時(shí)間太長(zhǎng)，或者我們最終會(huì)得到一個(gè)次優(yōu)的結(jié)果。我們之前看到問(wèn)題的條件編號(hào)很重要（例如，參見(jiàn)第 12.6 節(jié)了解詳細(xì)信息）。直觀地說(shuō)，它是最不敏感方向的變化量與最敏感方向的變化量之比。

其次，衰減率同樣重要。如果學(xué)習(xí)率仍然很大，我們可能最終會(huì)在最小值附近跳來(lái)跳去，因此無(wú)法達(dá)到最優(yōu)。12.5 節(jié) 詳細(xì)討論了這一點(diǎn)，我們?cè)?2.4 節(jié)中分析了性能保證。簡(jiǎn)而言之，我們希望速率下降，但可能比O(t?12)這將是凸問(wèn)題的不錯(cuò)選擇。

另一個(gè)同樣重要的方面是初始化。這既涉及參數(shù)的初始設(shè)置方式（詳見(jiàn) 第 5.4 節(jié)），也涉及它們最初的演變方式。這在熱身的綽號(hào)下進(jìn)行，即我們最初開(kāi)始朝著解決方案前進(jìn)的速度。一開(kāi)始的大步驟可能沒(méi)有好處，特別是因?yàn)槌跏紖?shù)集是隨機(jī)的。最初的更新方向也可能毫無(wú)意義。

最后，還有許多執(zhí)行循環(huán)學(xué)習(xí)率調(diào)整的優(yōu)化變體。這超出了本章的范圍。我們建議讀者查看 Izmailov等人的詳細(xì)信息。( 2018 )，例如，如何通過(guò)對(duì)整個(gè)參數(shù)路徑進(jìn)行平均來(lái)獲得更好的解決方案。

鑒于管理學(xué)習(xí)率需要很多細(xì)節(jié)，大多數(shù)深度學(xué)習(xí)框架都有自動(dòng)處理這個(gè)問(wèn)題的工具。在本章中，我們將回顧不同的調(diào)度對(duì)準(zhǔn)確性的影響，并展示如何通過(guò)學(xué)習(xí)率調(diào)度器有效地管理它。

12.11.1。玩具問(wèn)題

我們從一個(gè)玩具問(wèn)題開(kāi)始，這個(gè)問(wèn)題足夠簡(jiǎn)單，可以輕松計(jì)算，但又足夠不平凡，可以說(shuō)明一些關(guān)鍵方面。為此，我們選擇了一個(gè)稍微現(xiàn)代化的 LeNet 版本（relu而不是 sigmoid激活，MaxPooling 而不是 AveragePooling）應(yīng)用于 Fashion-MNIST。此外，我們混合網(wǎng)絡(luò)以提高性能。由于大部分代碼都是標(biāo)準(zhǔn)的，我們只介紹基礎(chǔ)知識(shí)而不進(jìn)行進(jìn)一步的詳細(xì)討論。如有需要，請(qǐng)參閱第 7 節(jié)進(jìn)行復(fù)習(xí)。

%matplotlib inline
import math
import torch
from torch import nn
from torch.optim import lr_scheduler
from d2l import torch as d2l


def net_fn():
  model = nn.Sequential(
    nn.Conv2d(1, 6, kernel_size=5, padding=2), nn.ReLU(),
    nn.MaxPool2d(kernel_size=2, stride=2),
    nn.Conv2d(6, 16, kernel_size=5), nn.ReLU(),
    nn.MaxPool2d(kernel_size=2, stride=2),
    nn.Flatten(),
    nn.Linear(16 * 5 * 5, 120), nn.ReLU(),
    nn.Linear(120, 84), nn.ReLU(),
    nn.Linear(84, 10))

  return model

loss = nn.CrossEntropyLoss()
device = d2l.try_gpu()

batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)

# The code is almost identical to `d2l.train_ch6` defined in the
# lenet section of chapter convolutional neural networks
def train(net, train_iter, test_iter, num_epochs, loss, trainer, device,
     scheduler=None):
  net.to(device)
  animator = d2l.Animator(xlabel='epoch', xlim=[0, num_epochs],
              legend=['train loss', 'train acc', 'test acc'])

  for epoch in range(num_epochs):
    metric = d2l.Accumulator(3) # train_loss, train_acc, num_examples
    for i, (X, y) in enumerate(train_iter):
      net.train()
      trainer.zero_grad()
      X, y = X.to(device), y.to(device)
      y_hat = net(X)
      l = loss(y_hat, y)
      l.backward()
      trainer.step()
      with torch.no_grad():
        metric.add(l * X.shape[0], d2l.accuracy(y_hat, y), X.shape[0])
      train_loss = metric[0] / metric[2]
      train_acc = metric[1] / metric[2]
      if (i + 1) % 50 == 0:
        animator.add(epoch + i / len(train_iter),
               (train_loss, train_acc, None))

    test_acc = d2l.evaluate_accuracy_gpu(net, test_iter)
    animator.add(epoch+1, (None, None, test_acc))

    if scheduler:
      if scheduler.__module__ == lr_scheduler.__name__:
        # Using PyTorch In-Built scheduler
        scheduler.step()
      else:
        # Using custom defined scheduler
        for param_group in trainer.param_groups:
          param_group['lr'] = scheduler(epoch)

  print(f'train loss {train_loss:.3f}, train acc {train_acc:.3f}, '
     f'test acc {test_acc:.3f}')

%matplotlib inline
from mxnet import autograd, gluon, init, lr_scheduler, np, npx
from mxnet.gluon import nn
from d2l import mxnet as d2l

npx.set_np()

net = nn.HybridSequential()
net.add(nn.Conv2D(channels=6, kernel_size=5, padding=2, activation='relu'),
    nn.MaxPool2D(pool_size=2, strides=2),
    nn.Conv2D(channels=16, kernel_size=5, activation='relu'),
    nn.MaxPool2D(pool_size=2, strides=2),
    nn.Dense(120, activation='relu'),
    nn.Dense(84, activation='relu'),
    nn.Dense(10))
net.hybridize()
loss = gluon.loss.SoftmaxCrossEntropyLoss()
device = d2l.try_gpu()

batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)

# The code is almost identical to `d2l.train_ch6` defined in the
# lenet section of chapter convolutional neural networks
def train(net, train_iter, test_iter, num_epochs, loss, trainer, device):
  net.initialize(force_reinit=True, ctx=device, init=init.Xavier())
  animator = d2l.Animator(xlabel='epoch', xlim=[0, num_epochs],
              legend=['train loss', 'train acc', 'test acc'])
  for epoch in range(num_epochs):
    metric = d2l.Accumulator(3) # train_loss, train_acc, num_examples
    for i, (X, y) in enumerate(train_iter):
      X, y = X.as_in_ctx(device), y.as_in_ctx(device)
      with autograd.record():
        y_hat = net(X)
        l = loss(y_hat, y)
      l.backward()
      trainer.step(X.shape[0])
      metric.add(l.sum(), d2l.accuracy(y_hat, y), X.shape[0])
      train_loss = metric[0] / metric[2]
      train_acc = metric[1] / metric[2]
      if (i + 1) % 50 == 0:
        animator.add(epoch + i / len(train_iter),
               (train_loss, train_acc, None))
    test_acc = d2l.evaluate_accuracy_gpu(net, test_iter)
    animator.add(epoch + 1, (None, None, test_acc))
  print(f'train loss {train_loss:.3f}, train acc {train_acc:.3f}, '
     f'test acc {test_acc:.3f}')

%matplotlib inline
import math
import tensorflow as tf
from tensorflow.keras.callbacks import LearningRateScheduler
from d2l import tensorflow as d2l


def net():
  return tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(filters=6, kernel_size=5, activation='relu',
                padding='same'),
    tf.keras.layers.AvgPool2D(pool_size=2, strides=2),
    tf.keras.layers.Conv2D(filters=16, kernel_size=5,
                activation='relu'),
    tf.keras.layers.AvgPool2D(pool_size=2, strides=2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(120, activation='relu'),
    tf.keras.layers.Dense(84, activation='sigmoid'),
    tf.keras.layers.Dense(10)])


batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)

# The code is almost identical to `d2l.train_ch6` defined in the
# lenet section of chapter convolutional neural networks
def train(net_fn, train_iter, test_iter, num_epochs, lr,
       device=d2l.try_gpu(), custom_callback = False):
  device_name = device._device_name
  strategy = tf.distribute.OneDeviceStrategy(device_name)
  with strategy.scope():
    optimizer = tf.keras.optimizers.SGD(learning_rate=lr)
    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
    net = net_fn()
    net.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])
  callback = d2l.TrainCallback(net, train_iter, test_iter, num_epochs,
               device_name)
  if custom_callback is False:
    net.fit(train_iter, epochs=num_epochs, verbose=0,
        callbacks=[callback])
  else:
     net.fit(train_iter, epochs=num_epochs, verbose=0,
         callbacks=[callback, custom_callback])
  return net

WARNING:tensorflow:From /home/d2l-worker/miniconda3/envs/d2l-en-release-1/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.
Instructions for updating:
Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089

讓我們看看如果我們使用默認(rèn)設(shè)置調(diào)用此算法會(huì)發(fā)生什么，例如學(xué)習(xí)率為0.3并訓(xùn)練 30迭代。請(qǐng)注意訓(xùn)練準(zhǔn)確性如何不斷提高，而測(cè)試準(zhǔn)確性方面的進(jìn)展卻停滯不前。兩條曲線之間的差距表明過(guò)度擬合。

lr, num_epochs = 0.3, 30
net = net_fn()
trainer = torch.optim.SGD(net.parameters(), lr=lr)
train(net, train_iter, test_iter, num_epochs, loss, trainer, device)

train loss 0.159, train acc 0.939, test acc 0.882

lr, num_epochs = 0.3, 30
net.initialize(force_reinit=True, ctx=device, init=init.Xavier())
trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': lr})
train(net, train_iter, test_iter, num_epochs, loss, trainer, device)

train loss 0.135, train acc 0.948, test acc 0.885

lr, num_epochs = 0.3, 30
train(net, train_iter, test_iter, num_epochs, lr)

loss 0.218, train acc 0.918, test acc 0.889
46772.7 examples/sec on /GPU:0

12.11.2。調(diào)度程序

調(diào)整學(xué)習(xí)率的一種方法是在每一步明確設(shè)置它。這通過(guò)該set_learning_rate方法方便地實(shí)現(xiàn)。我們可以在每個(gè) epoch 之后（甚至在每個(gè) minibatch 之后）向下調(diào)整它，例如，以動(dòng)態(tài)方式響應(yīng)優(yōu)化的進(jìn)展情況。

lr = 0.1
trainer.param_groups[0]["lr"] = lr
print(f'learning rate is now {trainer.param_groups[0]["lr"]:.2f}')

learning rate is now 0.10

trainer.set_learning_rate(0.1)
print(f'learning rate is now {trainer.learning_rate:.2f}')

learning rate is now 0.10

lr = 0.1
dummy_model = tf.keras.models.Sequential([tf.keras.layers.Dense(10)])
dummy_model.compile(tf.keras.optimizers.SGD(learning_rate=lr), loss='mse')
print(f'learning rate is now ,', dummy_model.optimizer.lr.numpy())

learning rate is now , 0.1

更一般地說(shuō)，我們想要定義一個(gè)調(diào)度程序。當(dāng)使用更新次數(shù)調(diào)用時(shí)，它會(huì)返回適當(dāng)?shù)膶W(xué)習(xí)率值。讓我們定義一個(gè)簡(jiǎn)單的學(xué)習(xí)率設(shè)置為 η=η0(t+1)?12.

class SquareRootScheduler:
  def __init__(self, lr=0.1):
    self.lr = lr

  def __call__(self, num_update):
    return self.lr * pow(num_update + 1.0, -0.5)

class SquareRootScheduler:
  def __init__(self, lr=0.1):
    self.lr = lr

  def __call__(self, num_update):
    return self.lr * pow(num_update + 1.0, -0.5)

class SquareRootScheduler:
  def __init__(self, lr=0.1):
    self.lr = lr

  def __call__(self, num_update):
    return self.lr * pow(num_update + 1.0, -0.5)

讓我們繪制它在一系列值上的行為。

scheduler = SquareRootScheduler(lr=0.1)
d2l.plot(torch.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])

scheduler = SquareRootScheduler(lr=0.1)
d2l.plot(np.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])

scheduler = SquareRootScheduler(lr=0.1)
d2l.plot(tf.range(num_epochs), [scheduler(t) for t in range(num_epochs)])

現(xiàn)在讓我們看看這對(duì) Fashion-MNIST 的訓(xùn)練有何影響。我們只是將調(diào)度程序作為訓(xùn)練算法的附加參數(shù)提供。

net = net_fn()
trainer = torch.optim.SGD(net.parameters(), lr)
train(net, train_iter, test_iter, num_epochs, loss, trainer, device,
   scheduler)

train loss 0.284, train acc 0.896, test acc 0.874

trainer = gluon.Trainer(net.collect_params(), 'sgd',
            {'lr_scheduler': scheduler})
train(net, train_iter, test_iter, num_epochs, loss, trainer, device)

train loss 0.524, train acc 0.810, test acc 0.812

train(net, train_iter, test_iter, num_epochs, lr,
   custom_callback=LearningRateScheduler(scheduler))

loss 0.381, train acc 0.860, test acc 0.848
49349.5 examples/sec on /GPU:0

這比以前好很多。有兩點(diǎn)很突出：曲線比以前更平滑。其次，過(guò)度擬合較少。不幸的是，關(guān)于為什么某些策略在理論上會(huì)導(dǎo)致較少的過(guò)度擬合，這并不是一個(gè)很好解決的問(wèn)題。有人認(rèn)為較小的步長(zhǎng)會(huì)導(dǎo)致參數(shù)更接近于零，從而更簡(jiǎn)單。然而，這并不能完全解釋這種現(xiàn)象，因?yàn)槲覀儾](méi)有真正提前停止，而只是輕輕地降低學(xué)習(xí)率。

12.11.3。政策

雖然我們不可能涵蓋所有種類(lèi)的學(xué)習(xí)率調(diào)度器，但我們嘗試在下面簡(jiǎn)要概述流行的策略。常見(jiàn)的選擇是多項(xiàng)式衰減和分段常數(shù)計(jì)劃。除此之外，已發(fā)現(xiàn)余弦學(xué)習(xí)率計(jì)劃在某些問(wèn)題上憑經(jīng)驗(yàn)表現(xiàn)良好。最后，在某些問(wèn)題上，在使用大學(xué)習(xí)率之前預(yù)熱優(yōu)化器是有益的。

12.11.3.1。因子調(diào)度器

多項(xiàng)式衰減的一種替代方法是乘法衰減，即ηt+1←ηt?α為了 α∈(0,1). 為了防止學(xué)習(xí)率衰減超過(guò)合理的下限，更新方程通常被修改為 ηt+1←max?(ηmin,ηt?α).

class FactorScheduler:
  def __init__(self, factor=1, stop_factor_lr=1e-7, base_lr=0.1):
    self.factor = factor
    self.stop_factor_lr = stop_factor_lr
    self.base_lr = base_lr

  def __call__(self, num_update):
    self.base_lr = max(self.stop_factor_lr, self.base_lr * self.factor)
    return self.base_lr

scheduler = FactorScheduler(factor=0.9, stop_factor_lr=1e-2, base_lr=2.0)
d2l.plot(torch.arange(50), [scheduler(t) for t in range(50)])

class FactorScheduler:
  def __init__(self, factor=1, stop_factor_lr=1e-7, base_lr=0.1):
    self.factor = factor
    self.stop_factor_lr = stop_factor_lr
    self.base_lr = base_lr

  def __call__(self, num_update):
    self.base_lr = max(self.stop_factor_lr, self.base_lr * self.factor)
    return self.base_lr

scheduler = FactorScheduler(factor=0.9, stop_factor_lr=1e-2, base_lr=2.0)
d2l.plot(np.arange(50), [scheduler(t) for t in range(50)])

class FactorScheduler:
  def __init__(self, factor=1, stop_factor_lr=1e-7, base_lr=0.1):
    self.factor = factor
    self.stop_factor_lr = stop_factor_lr
    self.base_lr = base_lr

  def __call__(self, num_update):
    self.base_lr = max(self.stop_factor_lr, self.base_lr * self.factor)
    return self.base_lr

scheduler = FactorScheduler(factor=0.9, stop_factor_lr=1e-2, base_lr=2.0)
d2l.plot(tf.range(50), [scheduler(t) for t in range(50)])

這也可以通過(guò) MXNet 中的內(nèi)置調(diào)度程序通過(guò) lr_scheduler.FactorScheduler對(duì)象來(lái)完成。它需要更多的參數(shù)，例如預(yù)熱周期、預(yù)熱模式（線性或恒定）、所需更新的最大數(shù)量等；展望未來(lái)，我們將酌情使用內(nèi)置調(diào)度程序，并僅在此處解釋它們的功能。如圖所示，如果需要，構(gòu)建您自己的調(diào)度程序相當(dāng)簡(jiǎn)單。

12.11.3.2。多因素調(diào)度程序

訓(xùn)練深度網(wǎng)絡(luò)的一個(gè)常見(jiàn)策略是保持學(xué)習(xí)率分段不變，并每隔一段時(shí)間將其降低一個(gè)給定的數(shù)量。也就是說(shuō)，給定一組降低速率的時(shí)間，例如 s={5,10,20}減少 ηt+1←ηt?α每當(dāng) t∈s. 假設(shè)值在每一步都減半，我們可以按如下方式實(shí)現(xiàn)。

net = net_fn()
trainer = torch.optim.SGD(net.parameters(), lr=0.5)
scheduler = lr_scheduler.MultiStepLR(trainer, milestones=[15, 30], gamma=0.5)

def get_lr(trainer, scheduler):
  lr = scheduler.get_last_lr()[0]
  trainer.step()
  scheduler.step()
  return lr

d2l.plot(torch.arange(num_epochs), [get_lr(trainer, scheduler)
                 for t in range(num_epochs)])

scheduler = lr_scheduler.MultiFactorScheduler(step=[15, 30], factor=0.5,
                       base_lr=0.5)
d2l.plot(np.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])

class MultiFactorScheduler:
  def __init__(self, step, factor, base_lr):
    self.step = step
    self.factor = factor
    self.base_lr = base_lr

  def __call__(self, epoch):
    if epoch in self.step:
      self.base_lr = self.base_lr * self.factor
      return self.base_lr
    else:
      return self.base_lr

scheduler = MultiFactorScheduler(step=[15, 30], factor=0.5, base_lr=0.5)
d2l.plot(tf.range(num_epochs), [scheduler(t) for t in range(num_epochs)])

這種分段恒定學(xué)習(xí)率計(jì)劃背后的直覺(jué)是，讓優(yōu)化繼續(xù)進(jìn)行，直到根據(jù)權(quán)重向量的分布達(dá)到穩(wěn)定點(diǎn)。然后（也只有那時(shí)）我們會(huì)降低速率以獲得更高質(zhì)量的代理到良好的局部最小值。下面的例子展示了這如何產(chǎn)生更好的解決方案。

train(net, train_iter, test_iter, num_epochs, loss, trainer, device,
   scheduler)

train loss 0.186, train acc 0.931, test acc 0.897

trainer = gluon.Trainer(net.collect_params(), 'sgd',
            {'lr_scheduler': scheduler})
train(net, train_iter, test_iter, num_epochs, loss, trainer, device)

train loss 0.195, train acc 0.926, test acc 0.893

train(net, train_iter, test_iter, num_epochs, lr,
   custom_callback=LearningRateScheduler(scheduler))

loss 0.237, train acc 0.913, test acc 0.882
49476.3 examples/sec on /GPU:0

12.11.3.3。余弦調(diào)度程序

Loshchilov 和 Hutter ( 2016 )提出了一種相當(dāng)令人費(fèi)解的啟發(fā)式方法 。它依賴(lài)于這樣的觀察，即我們可能不想在開(kāi)始時(shí)過(guò)分降低學(xué)習(xí)率，而且我們可能希望最終使用非常小的學(xué)習(xí)率來(lái)“完善”解決方案。這導(dǎo)致類(lèi)似余弦的時(shí)間表具有以下范圍內(nèi)學(xué)習(xí)率的函數(shù)形式t∈[0,T].

(12.11.1)ηt=ηT+η0?ηT2(1+cos?(πt/T))

這里η0是初始學(xué)習(xí)率，ηT是當(dāng)時(shí)的目標(biāo)利率T. 此外，對(duì)于t>T我們只需將值固定到ηT無(wú)需再次增加。在下面的例子中，我們?cè)O(shè)置最大更新步長(zhǎng)T=20.

class CosineScheduler:
  def __init__(self, max_update, base_lr=0.01, final_lr=0,
        warmup_steps=0, warmup_begin_lr=0):
    self.base_lr_orig = base_lr
    self.max_update = max_update
    self.final_lr = final_lr
    self.warmup_steps = warmup_steps
    self.warmup_begin_lr = warmup_begin_lr
    self.max_steps = self.max_update - self.warmup_steps

  def get_warmup_lr(self, epoch):
    increase = (self.base_lr_orig - self.warmup_begin_lr) 
            * float(epoch) / float(self.warmup_steps)
    return self.warmup_begin_lr + increase

  def __call__(self, epoch):
    if epoch < self.warmup_steps:
      return self.get_warmup_lr(epoch)
    if epoch <= self.max_update:
      self.base_lr = self.final_lr + (
        self.base_lr_orig - self.final_lr) * (1 + math.cos(
        math.pi * (epoch - self.warmup_steps) / self.max_steps)) / 2
    return self.base_lr

scheduler = CosineScheduler(max_update=20, base_lr=0.3, final_lr=0.01)
d2l.plot(torch.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])

scheduler = lr_scheduler.CosineScheduler(max_update=20, base_lr=0.3,
                     final_lr=0.01)
d2l.plot(np.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])

class CosineScheduler:
  def __init__(self, max_update, base_lr=0.01, final_lr=0,
        warmup_steps=0, warmup_begin_lr=0):
    self.base_lr_orig = base_lr
    self.max_update = max_update
    self.final_lr = final_lr
    self.warmup_steps = warmup_steps
    self.warmup_begin_lr = warmup_begin_lr
    self.max_steps = self.max_update - self.warmup_steps

  def get_warmup_lr(self, epoch):
    increase = (self.base_lr_orig - self.warmup_begin_lr) 
            * float(epoch) / float(self.warmup_steps)
    return self.warmup_begin_lr + increase

  def __call__(self, epoch):
    if epoch < self.warmup_steps:
      return self.get_warmup_lr(epoch)
    if epoch <= self.max_update:
      self.base_lr = self.final_lr + (
        self.base_lr_orig - self.final_lr) * (1 + math.cos(
        math.pi * (epoch - self.warmup_steps) / self.max_steps)) / 2
    return self.base_lr

scheduler = CosineScheduler(max_update=20, base_lr=0.3, final_lr=0.01)
d2l.plot(tf.range(num_epochs), [scheduler(t) for t in range(num_epochs)])

在計(jì)算機(jī)視覺(jué)的背景下，這個(gè)時(shí)間表可以帶來(lái)更好的結(jié)果。但請(qǐng)注意，不能保證此類(lèi)改進(jìn)（如下所示）。

net = net_fn()
trainer = torch.optim.SGD(net.parameters(), lr=0.3)
train(net, train_iter, test_iter, num_epochs, loss, trainer, device,
   scheduler)

train loss 0.229, train acc 0.916, test acc 0.886

trainer = gluon.Trainer(net.collect_params(), 'sgd',
            {'lr_scheduler': scheduler})
train(net, train_iter, test_iter, num_epochs, loss, trainer, device)

train loss 0.345, train acc 0.876, test acc 0.866

train(net, train_iter, test_iter, num_epochs, lr,
   custom_callback=LearningRateScheduler(scheduler))

loss 0.261, train acc 0.905, test acc 0.881
49572.1 examples/sec on /GPU:0

12.11.3.4。暖身

在某些情況下，初始化參數(shù)不足以保證獲得良好的解決方案。對(duì)于一些可能導(dǎo)致不穩(wěn)定的優(yōu)化問(wèn)題的高級(jí)網(wǎng)絡(luò)設(shè)計(jì)來(lái)說(shuō)，這尤其是一個(gè)問(wèn)題。我們可以通過(guò)選擇足夠小的學(xué)習(xí)率來(lái)解決這個(gè)問(wèn)題，以防止在開(kāi)始時(shí)出現(xiàn)分歧。不幸的是，這意味著進(jìn)展緩慢。相反，較大的學(xué)習(xí)率最初會(huì)導(dǎo)致發(fā)散。

解決這個(gè)難題的一個(gè)相當(dāng)簡(jiǎn)單的方法是使用一個(gè)預(yù)熱期，在此期間學(xué)習(xí)率增加到其初始最大值，并降低學(xué)習(xí)率直到優(yōu)化過(guò)程結(jié)束。為簡(jiǎn)單起見(jiàn)，通常為此目的使用線性增加。這導(dǎo)致了如下所示表格的時(shí)間表。

scheduler = CosineScheduler(20, warmup_steps=5, base_lr=0.3, final_lr=0.01)
d2l.plot(torch.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])

scheduler = lr_scheduler.CosineScheduler(20, warmup_steps=5, base_lr=0.3,
                     final_lr=0.01)
d2l.plot(np.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])

scheduler = CosineScheduler(20, warmup_steps=5, base_lr=0.3, final_lr=0.01)
d2l.plot(tf.range(num_epochs), [scheduler(t) for t in range(num_epochs)])

請(qǐng)注意，網(wǎng)絡(luò)最初收斂得更好（特別是觀察前 5 個(gè)時(shí)期的表現(xiàn)）。

net = net_fn()
trainer = torch.optim.SGD(net.parameters(), lr=0.3)
train(net, train_iter, test_iter, num_epochs, loss, trainer, device,
   scheduler)

train loss 0.173, train acc 0.936, test acc 0.902

trainer = gluon.Trainer(net.collect_params(), 'sgd',
            {'lr_scheduler': scheduler})
train(net, train_iter, test_iter, num_epochs, loss, trainer, device)

train loss 0.347, train acc 0.875, test acc 0.871

train(net, train_iter, test_iter, num_epochs, lr,
   custom_callback=LearningRateScheduler(scheduler))

loss 0.270, train acc 0.901, test acc 0.880
49891.2 examples/sec on /GPU:0

預(yù)熱可以應(yīng)用于任何調(diào)度程序（不僅僅是余弦）。有關(guān)學(xué)習(xí)率計(jì)劃和更多實(shí)驗(yàn)的更詳細(xì)討論，另請(qǐng)參閱（Gotmare等人，2018 年）。特別是，他們發(fā)現(xiàn)預(yù)熱階段限制了非常深的網(wǎng)絡(luò)中參數(shù)的發(fā)散量。這在直覺(jué)上是有道理的，因?yàn)槲覀冾A(yù)計(jì)由于網(wǎng)絡(luò)中那些在開(kāi)始時(shí)花費(fèi)最多時(shí)間取得進(jìn)展的部分的隨機(jī)初始化會(huì)出現(xiàn)顯著差異。

12.11.4。概括

在訓(xùn)練期間降低學(xué)習(xí)率可以提高準(zhǔn)確性并（最令人費(fèi)解的是）減少模型的過(guò)度擬合。

每當(dāng)進(jìn)展趨于平穩(wěn)時(shí)，學(xué)習(xí)率的分段降低在實(shí)踐中是有效的。從本質(zhì)上講，這可以確保我們有效地收斂到一個(gè)合適的解決方案，然后才通過(guò)降低學(xué)習(xí)率來(lái)減少參數(shù)的固有方差。

余弦調(diào)度器在某些計(jì)算機(jī)視覺(jué)問(wèn)題上很受歡迎。有關(guān)此類(lèi)調(diào)度程序的詳細(xì)信息，請(qǐng)參見(jiàn)例如GluonCV 。

優(yōu)化前的預(yù)熱期可以防止發(fā)散。

優(yōu)化在深度學(xué)習(xí)中有多種用途。除了最小化訓(xùn)練目標(biāo)外，優(yōu)化算法和學(xué)習(xí)率調(diào)度的不同選擇可能導(dǎo)致測(cè)試集上的泛化和過(guò)度擬合量大不相同（對(duì)于相同數(shù)量的訓(xùn)練誤差）。

12.11.5。練習(xí)

試驗(yàn)給定固定學(xué)習(xí)率的優(yōu)化行為。您可以通過(guò)這種方式獲得的最佳模型是什么？

如果改變學(xué)習(xí)率下降的指數(shù)，收斂性會(huì)如何變化？PolyScheduler為了方便您在實(shí)驗(yàn)中使用。

將余弦調(diào)度程序應(yīng)用于大型計(jì)算機(jī)視覺(jué)問(wèn)題，例如訓(xùn)練 ImageNet。相對(duì)于其他調(diào)度程序，它如何影響性能？

熱身應(yīng)該持續(xù)多長(zhǎng)時(shí)間？

你能把優(yōu)化和抽樣聯(lián)系起來(lái)嗎？首先使用Welling 和 Teh ( 2011 )關(guān)于隨機(jī)梯度朗之萬(wàn)動(dòng)力學(xué)的結(jié)果。