譯者 | 崔皓
審校? | 孫淑娟
開篇
Transformer模型通過在語言翻譯、文本分類和序列建模中提供卓越的性能，徹底改變了自然語言處理（NLP）任務(wù)。?
Transformer的架構(gòu)是基于一種自我關(guān)注機(jī)制，它允許序列中的每個元素關(guān)注其他元素并處理輸入序列的堆疊編碼器。?
本文將演示如何建立一個Transformer模型來生成新的雞尾酒配方。文中將使用Cocktail DB數(shù)據(jù)集，該數(shù)據(jù)集包含了成千上萬種雞尾酒的信息，包括它們的成分以及配方。?
下載Cocktail DB數(shù)據(jù)集?
首先，我們需要下載并預(yù)處理Cocktail DB數(shù)據(jù)集。我們將使用Pandas庫來完成這一工作。?
import pandas as pd
url = 'https://www.thecocktaildb.com/api/json/v1/1/search.php?s=' cocktail_df = pd.DataFrame() for i in range(1, 26): response = pd.read_json(tr(i)) cocktail_df = pd.concat([cocktail_df, response['drinks']], ignore_index=True)
預(yù)處理數(shù)據(jù)集?
cocktail_df = cocktail_df.dropna(subset=['strInstructions']) cocktail_df = cocktail_df[['strDrink', 'strInstructions', 'strIngredient1', 'strIngredient2', 'strIngredient3', 'strIngredient4', 'strIngredient5', 'strIngredient6']] cocktail_df = cocktail_df.fillna('')接下來，我們需要使用標(biāo)記器對雞尾酒菜譜進(jìn)行標(biāo)記和編碼。?
將TensorFlow數(shù)據(jù)集導(dǎo)入為tfds。?
定義標(biāo)記器和詞匯量大小?
tokenizer = tfds.features.text.SubwordTextEncoder.build_from_corpus( (text for text in cocktail_df['strInstructions']), target_vocab_size=2**13)
定義編碼函數(shù)?
def encode(text): encoded_text = tokenizer.encode(text) return encoded_text
將編碼函數(shù)應(yīng)用到數(shù)據(jù)集上?
cocktail_df['encoded_recipe'] = cocktail_df['strInstructions'].apply(encode)
定義菜譜的最大長度?
MAX_LEN = max([len(recipe) for recipe in cocktail_df['encoded_recipe']] )?
有了標(biāo)記化的雞尾酒菜譜，就可以定義轉(zhuǎn)化器解碼層了。Transformer解碼器層由兩個子層組成：masked multi-head self-attention 層和point-wise feed-forward 層。?
import tensorflow as tf from tensorflow.keras.layers import LayerNormalization, MultiHeadAttention, Dense
class TransformerDecoderLayer(tf.keras.layers.Layer): def init(self, num_heads, d_model, dff, rate=0.1): super(TransformerDecoderLayer, self).init()
self.mha1 = MultiHeadAttention(num_heads, d_model)
self.mha2 = MultiHeadAttention(num_heads, d_model)
self.ffn = tf.keras.Sequential([
Dense(dff, activation='relu'),
Dense(d_model)
])
self.layernorm1 = LayerNormalization(epsilon=1e-6)
self.layernorm2 = LayerNormalization(epsilon=1e-6)
self.layernorm3 = LayerNormalization(epsilon=1e-6)
self.dropout1 = tf.keras.layers.Dropout(rate)
self.dropout2 = tf.keras.layers.Dropout(rate)
self.dropout3 = tf.keras.layers.Dropout(rate)
def call(self, x, enc_output, training, look_ahead_mask):
attn1 = self.mha1(x, x, x, look_ahead_mask)
attn1 = self.dropout1(attn1, training=training)
out1 = self.layernorm1(x + attn1)
attn2 = self.mha2(enc_output, enc_output, out1, out1, out1)
attn2 = self.dropout2(attn2, training=training)
out2 = self.layernorm2(out1 + attn2)
ffn_output = self.ffn(out2)
ffn_output = self.dropout3(ffn_output, training=training)
out3 = self.layernorm3(out2 + ffn_output)
return out3
在上面的代碼中，TransformerDecoderLayer類需要四個參數(shù)：masked multi-head self-attention層的個數(shù)、模型的維度、point-wise feed-forward層的單元數(shù)和dropout率。?
調(diào)用方法定義了解碼器層的前向傳遞，其中x是輸入序列；enc_output是編碼器的輸出；training是一個布爾標(biāo)志，表示模型是處于訓(xùn)練還是推理模式；look_ahead_mask是一個掩碼，防止解碼器關(guān)注未來的令牌。?
我們現(xiàn)在可以定義轉(zhuǎn)化器模型，它由多個堆疊的轉(zhuǎn)化器解碼器層和一個密集層組成，密集層將解碼器的輸出映射到詞匯量上。?
從tensorflow.keras.layer導(dǎo)入Input?
input_layer = Input(shape=(MAX_LEN,))
定義Transformer解碼器層
NUM_LAYERS = 4 NUM_HEADS = 8 D_MODEL = 256 DFF = 1024 DROPOUT_RATE = 0.1
decoder_layers = [TransformerDecoderLayer(NUM_HEADS, D_MODEL, DFF, DROPOUT_RATE) for _ in range(NUM_LAYERS)]
定義輸出層
output_layer = Dense(tokenizer.vocab_size)
連接層?
x = input_layer look_ahead_mask = tf.linalg.band_part(tf.ones((MAX_LEN, MAX_LEN)), -1, 0) for decoder_layer in decoder_layers: x = decoder_layer(x, x, True, look_ahead_mask) output = output_layer(x)
定義模型?
model = tf.keras.models.Model(inputs=input_layer, outputs=output)
在上面的代碼中，我們定義了輸入層，接受長度為MAX_LEN的填充序列。然后，通過創(chuàng)建一個堆疊在一起的TransformerDecoderLayer對象的列表來定義轉(zhuǎn)化器解碼層，從而處理輸入序列。?
最后一個Transformer解碼層的輸出通過一個密集層，其詞匯量與標(biāo)記器中的子詞數(shù)量相對應(yīng)。我們可以使用Adam優(yōu)化器來訓(xùn)練模型，并在一定數(shù)量的epochs后評估其性能。?
定義損失函數(shù)?
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction='none')
def loss_function(real, pred): mask = tf.math.logical_not(tf.math.equal(real, 0)) loss_ = loss_object(real, pred)
mask = tf.cast(mask, dtype=loss_.dtype)
loss_ *= mask
return tf.reduce_mean(loss_)
定義學(xué)習(xí)率時間表?
class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule): def init(self, d_model, warmup_steps=4000): super(CustomSchedule, self).init()
self.d_model = tf.cast(d_model, tf.float32)
self.warmup_steps = warmup_steps
def __call__(self, step):
arg1 = tf.math.rsqrt(step)
arg2 = step * (self.warmup_steps**-1)
return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)
定義優(yōu)化器?
LR = CustomSchedule(D_MODEL) optimizer = tf.keras.optimizers.Adam(LR, beta_1=0.9, beta_2=0.98, epsilon=1e-9)
定義準(zhǔn)確度指標(biāo)?
train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')
定義訓(xùn)練步驟函數(shù)?
@tf.function def train_step(inp, tar): tar_inp = tar[:, :-1] tar_real = tar[:, 1:]
look_ahead_mask = tf.linalg.band_part(tf.ones((tar.shape[1], tar.shape[1])), -1, 0) look_ahead_mask = 1 - look_ahead_mask
with tf.GradientTape() as tape: predictions = model(inp, True, look_ahead_mask) loss = loss_function(tar_real, predictions)
gradients = tape.gradient(loss, model.trainable_variables) optimizer.apply_gradients(zip(gradients, model.trainable_variables))
train_accuracy.update_state(tar_real, predictions)
return loss Train the model EPOCHS = 50 BATCH_SIZE = 64 NUM_EXAMPLES = len(cocktail_df)
for epoch in range(EPOCHS): print('Epoch', epoch + 1) total_loss = 0
for i in range(0, NUM_EXAMPLES, BATCH_SIZE): batch = cocktail_df[i:i+BATCH_SIZE] input_batch = tf.keras.preprocessing.sequence.pad_sequences(batch['encoded_recipe'], padding='post', maxlen=MAX_LEN) target_batch = input_batch
loss = train_step(input_batch, target_batch)
total_loss += loss
print('Loss:', total_loss) print('Accuracy:', train_accuracy.result().numpy()) train_accuracy.reset_states
一旦模型被訓(xùn)練出來，我們就可以通過給模型提供一個種子序列，并通過反復(fù)預(yù)測來生成新的雞尾酒配方。Shapetokentar：產(chǎn)生Shapee序列末尾的標(biāo)記。?
def generate_recipe(seed, max_len):
encoded_seed = encode(seed) for i in range(max_len):
input_sequence = tf.keras.preprocessing.sequence.pad_sequences([encoded_seed],
padding='post', maxlen=MAX_LEN) predictions = model(input_sequence, False, None)
predicted_id = tf.random.categorical(predictions[:, -1, :], num_samples=1)
if predicted_id == tokenizer.vocab_size:
break encoded_seed = tf.squeeze(predicted_id).numpy().tolist()
recipe = tokenizer.decode(encoded_seed)
return recipe
總結(jié)?
總之，Transformer是一個強(qiáng)大的序列建模工具，可以用于NLP以外的廣泛應(yīng)用。?
通過遵循本文所述的步驟，可以建立一個Transformer模型來生成新的雞尾酒配方，展示了Transformer架構(gòu)的靈活性和通用性。?
譯者介紹?
崔皓，51CTO社區(qū)編輯，資深架構(gòu)師，擁有18年的軟件開發(fā)和架構(gòu)經(jīng)驗，10年分布式架構(gòu)經(jīng)驗。?
原文標(biāo)題：??Cocktail Alchemy: Creating New Recipes With Transformers??，作者：Pavan madduru
