{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Deep Learning with Torch - Tutorial 5" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Learning language models with recurrent networks\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, we need to handle our data. We will use a text file and load it at character level.\n", "\n", "We also need to encode the strings into indexed symbols (our vocabulary)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "\n", "function loadTextFileChars(filename, vocab)\n", " local file = torch.DiskFile(filename, 'r')\n", " file:seekEnd()\n", " local length = file:position() - 1\n", " file:seek(1)\n", " local byteVec = torch.ByteTensor(length)\n", " file:readByte(byteVec:storage())\n", "\n", " local vocab = vocab or {}\n", " local currentNum = 1\n", " local data = byteVec:data()\n", " for i=0, length-1 do\n", " local encodedNum = vocab[data[i]]\n", " if not encodedNum then\n", " vocab[data[i]] = currentNum\n", " encodedNum = currentNum\n", " currentNum = currentNum + 1\n", " end\n", " data[i] = encodedNum\n", " end\n", " local decoder = {}\n", " for val, num in pairs(vocab) do\n", " decoder[num] = string.char(val)\n", " end\n", " return byteVec, vocab, decoder\n", "end\n", "\n", "\n", "function reshapeData(wordVec, seqLength, batchSize)\n", " local offset = offset or 0\n", " local length = wordVec:nElement()\n", " local numBatches = torch.floor(length / (batchSize * seqLength))\n", "\n", " local batchWordVec = wordVec.new():resize(numBatches, batchSize, seqLength)\n", " \n", " local endIdxs = torch.LongTensor()\n", " for i=1, batchSize do\n", " local startPos = torch.round((i - 1) * length / batchSize ) + 1\n", " local sliceLength = seqLength * numBatches\n", " batchWordVec:select(2,i):copy(wordVec:narrow(1, startPos, sliceLength))\n", " end\n", "return batchWordVec\n", "end\n", "\n", "\n", "function decodeFunc(decoder)\n", " local space = ''\n", " local func = function(vec)\n", " local output = ''\n", " for i=1, vec:size(1) do\n", " output = output .. space .. decoder[vec[i]]\n", " end\n", " return output\n", " end\n", " return func\n", "end\n", "\n", "\n", " function encodeFunc(vocab)\n", " local func = function(str)\n", " local length = #str\n", " local encoded = torch.ByteTensor(length):zero()\n", "\n", " for i=1, length do\n", " local encodedNum = vocab[str[i]]\n", " if not encodedNum then\n", " encodedNum = -1\n", " end\n", " encoded[i] = encodedNum\n", " end\n", " return encoded\n", " end\n", " \n", " return func\n", "end" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "collapsed": false }, "outputs": [], "source": [ "byteVec, vocab, decoder = loadTextFileChars('hebrewBible.txt')\n", "--byteVec, vocab, decoder = loadTextFileChars('tinyshakespeare.txt')\n", "vocabSize = #decoder\n", "encodeTensor = encodeFunc(vocab)\n", "decodeString = decodeFunc(decoder)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "39\t\n", " 3215738\n", "[torch.LongStorage of size 1]\n", "\n", "א,א בראשית, ברא אלוהים, את השמיים, ואת הארץ. א,ב והארץ, הייתה תוהו ובוהו, וחושך, על-פני תהום; ורוח אלוהים, מרחפת על-פני המים. א,ג ויאמר אלוהים, יהי אור; ויהי-אור. א,ד וירא אלוהים את-האור, כי-טוב; ויבדל אלוהים, בין האור ובין החושך. א,ה ויקרא אלוהים לאור יום, ולחושך קרא לילה; ויהי-ערב ויהי-בוקר, יום אחד. {פ}\n", "\n", "א,ו ויאמר אלוהים, יהי רקיע בתוך המים, ויהי מבדיל, בין מים למים. א,ז ויעש אלוהים, את-הרקיע, ויבדל בין המים אשר מתחת לרקיע, ובין המים אשר מעל לרקיע; ויהי-כן. א,ח ויקרא אלוהים לרקיע, שמיים; ויהי-ערב ויהי-בוקר, יום שני. {פ}\n", "\n", "א,ט ויאמר אלוהים, ייקוו המים מתחת השמיים אל-מ\t\n" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print(vocabSize)\n", "print(byteVec:size())\n", "print(decodeString(byteVec:narrow(1,1,1000)))\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ " 3062\n", " 50\n", " 21\n", "[torch.LongStorage of size 3]\n", "\n" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "batchSize = 50\n", "seqLength = 20\n", "trainData = reshapeData(byteVec, seqLength + 1, batchSize)\n", "print(trainData:size())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will use the ```recurrent``` package for torch.\n", "https://github.com/eladhoffer/recurrent.torch\n", "\n", "\n", "A popular alternative is the ```rnn``` package\n", "https://github.com/Element-Research/rnn" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "----RNN - Single----\t\n", "Output size: \t\n", " 16\n", " 7\n", "[torch.LongStorage of size 2]\n", "\n", "State size: \t\n", " 16\n", " 7\n", "[torch.LongStorage of size 2]\n", "\n", "----LSTM - Sequence----\t\n" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/plain": [ "Output size: \t\n", " 16\n", " 20\n", " 7\n", "[torch.LongStorage of size 3]\n", "\n", "State size: \t\n", " 16\n", " 14\n", "[torch.LongStorage of size 2]\n", "\n" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "require 'recurrent'\n", "local inputSize = 5\n", "local outputSize = 7\n", "\n", "local rnn = nn.RNN(inputSize, outputSize)\n", "local lstm = nn.LSTM(inputSize, outputSize)\n", "local gru = nn.GRU(inputSize, outputSize)\n", "\n", "\n", "print('----RNN - Single----')\n", "-- To feed single time step, we use 'single' mode\n", "local batch = 16\n", "local x = torch.rand(batch, inputSize)\n", "rnn:single()\n", "\n", "-- And now we can feed the input to get output. We can also peek at state\n", "local y = rnn:forward(x)\n", "print('Output size: '); print(y:size())\n", "print('State size: '); print(rnn:getState():size())\n", "\n", "\n", "print('----LSTM - Sequence----')\n", "--We mostly feed sequences through rnns\n", "local timeLength = 20\n", "x = torch.rand(batch, timeLength, inputSize)\n", "\n", "-- We will enable this by putting the recurrent model to 'sequence' mode\n", "lstm:sequence()\n", "\n", "-- And now we can feed the sequence to get output. We can also peek at state\n", "local y = lstm:forward(x)\n", "print('Output size: '); print(y:size())\n", "print('State size: '); print(lstm:getState():size())\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Non recurrent layers and criterions are wrapped in ```TemporalModule``` and ```TemporalCriterion``` containers" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "#Parameters = \t535335\t\n" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "require 'recurrent'\n", "require 'cunn'\n", "\n", "rnnSize = 256\n", "embedder = nn.LookupTable(vocabSize, rnnSize)\n", "classifier = nn.Linear(rnnSize, vocabSize)\n", "rnn = nn.LSTM(rnnSize, rnnSize)\n", "\n", "model = nn.Sequential()\n", "model:add(embedder)\n", "model:add(rnn)\n", "model:add(nn.TemporalModule(classifier))\n", "model:add(nn.TemporalModule(nn.LogSoftMax()))\n", "\n", "\n", "model:cuda()\n", "model:sequence()\n", "embedder:share(classifier, 'weight', 'gradWeight')\n", "w, dE_dw = model:getParameters()\n", "print('#Parameters = ', w:nElement())\n", "\n", "criterion = nn.TemporalCriterion(nn.ClassNLLCriterion()):cuda()\n", "\n", "w:uniform(-0.08, 0.08)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Training the network" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [], "source": [ "require 'optim'\n", "\n", "Vmax = 5\n", "optimState = {learningRate = 1e-3}\n", "\n", "\n", "function forwardNet(data, train)\n", " local lossAcc = 0\n", " local numBatches = data:size(1)\n", " model:sequence()\n", " model:forget()\n", " model:zeroState()\n", " if train then\n", " --set network into training mode\n", " model:training()\n", " else\n", " model:evaluate()\n", " end\n", " local x = torch.CudaTensor(batchSize, seqLength)\n", " local yt = torch.CudaTensor(batchSize, seqLength)\n", " for i = 1, numBatches do\n", " -- our input and target are the same sequence shifted\n", " x:copy(data[i]:narrow(2, 1, seqLength))\n", " yt:copy(data[i]:narrow(2, 2, seqLength))\n", " \n", " \n", " local y = model:forward(x)\n", " local err = criterion:forward(y, yt)\n", "\n", " lossAcc = lossAcc + err / seqLength\n", " \n", " \n", " if train then\n", " function feval()\n", " model:zeroGradParameters() --zero grads\n", " local dE_dy = criterion:backward(y,yt)\n", " model:backward(x, dE_dy) -- backpropagation\n", " local norm = dE_dw:norm()\n", " if norm > Vmax then -- gradient renormalization to avoid explosion\n", " local shrink = Vmax / norm\n", " dE_dw:mul(shrink)\n", " end\n", " return err, dE_dw\n", " end\n", " \n", " optim.adam(feval, w, optimState)\n", " end\n", " end\n", " \n", "\n", " local avgLoss = lossAcc / numBatches\n", " \n", " \n", " return avgLoss\n", "end\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll also introduce a sampling function - to try and generate sequences from our network" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": true }, "outputs": [], "source": [ "\n", "function sample(num, space, temperature)\n", " local num = num or 50\n", " local temperature = temperature or 1\n", " local function smp(preds)\n", " if temperature == 0 then\n", " local _, num = preds:max(2)\n", " return num\n", " else\n", " preds:div(temperature) -- scale by temperature\n", " local probs = preds:squeeze()\n", " probs:div(probs:sum()) -- renormalize so probs sum to one\n", " local num = torch.multinomial(probs:float(), 1):typeAs(preds)\n", " return num\n", " end\n", " end\n", "\n", " local sampleModel = nn.Sequential():add(embedder):add(rnn):add(classifier):add(nn.SoftMax():cuda())\n", "\n", " sampleModel:evaluate()\n", " sampleModel:single()\n", "\n", " \n", " local pred, predText, embedded\n", " \n", " wordNum = torch.Tensor(1):random(vocabSize):cuda()\n", "\n", " local predText = {}\n", " \n", " for i=1, num do\n", " pred = sampleModel:forward(wordNum)\n", " wordNum = smp(pred)\n", " predText[i] = wordNum:squeeze()\n", " end\n", " return torch.Tensor(predText)\n", "end" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "Sampled text:\t\n" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/plain": [ "Training Loss: 1.2775420697977\t\n", "Sampled text:\t\n" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/plain": [ "ר-הארץ; גיקר; והשם בין-גוליים תצחבה-אל-תשלחנו בארץ-אשר: המזבקת חמר מבצר, התאקרה אשר באלמים ואיש, מיה; לשמח-גמתיהם, לך הסהב וכדור, ומלך-הזאקרוח.\n", "עב,א גשלו, מפני-יהוד תעללים כי-עוזה, מפני יהובני המלכישלגא, והיה רבחנה לירחנה: כי-ראשי, נתבורו ידון אל-רגבל יעקול, והעיר ליהוה. ב,יא חזק\t\n", "Epoch 2:\t\n" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/plain": [ "Training Loss: 1.1135815120382\t\n", "Sampled text:\t\n" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/plain": [ "Training Loss: 1.0722586075166\t\n", "Sampled text:\t\n" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/plain": [ " דיבר-יהוה--נדים, מקומו--טמא, על-בני שלכים, ישראל ישראל, מאהב בבל; ויעזבם, מי לאלה לאמור: את כל-הוא וחושם מצוהרי, סביבות את-כל-גיד המלך משילות, אשר שבע צסון--מלך-פדר; והנפש את-כל-דרכו, מזמור--בחוץ הנשמר, במלך: בתית לו שקר הנביא הראשון, להם בנו-החצר הדבר, אשר יקרא, עוונם הגונינים. ב ד,טו \t\n", "Epoch 4:\t\n" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/plain": [ "Training Loss: 1.0446834487124\t\n", "Sampled text:\t\n" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/plain": [ "Training Loss: 1.0277193483771\t\n", "Sampled text:\t\n" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/plain": [ "קני יהוה. נא,כב אז אל-הקשת כי-תוכיח על-הפקודים--את-שלושה נמוצדים הניחמם איש-אחאב, מלץ; וישאו בת-אמור, אבשמו ארץ גדול. מז,יז לנשא, עליו, וייפול ודויד. ס}\n", "\n", "כב,ח טופג על-חואני--ברכו שפרה כתף, חילו מלך אביו; ותהיו מכים זהב, {ס} בעימתי, וצבא שאול, מיניו לבבל, כחרב מלחם עלי כל-הכריס; ומבנות נ\t\n" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "epochs = 5\n", "trainLoss = torch.Tensor(epochs)\n", "\n", "print('Sampled text:')\n", "print(decodeString(sample(500)))\n", "for e = 1, epochs do\n", " print('Epoch ' .. e .. ':')\n", " trainLoss[e] = forwardNet(trainData, true)\n", " \n", " print('Training Loss: ' .. trainLoss[e])\n", " \n", " print('Sampled text:')\n", " print(decodeString(sample(500)))\n", "end" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "Sampled text:\t\n" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" }, { "data": { "text/plain": [ " נולוה. {ס}\n", "\n", "כג,א בשמן בדי-שדה, עליי אל-מימינה מותנו.\n", "קב,ג כי-עמד ראפו שמונה דמיקים תבוכו; ולבנך טובה, בושני יעשו נלשכם, תבקר היית; ולך מחטאו במושו; - אם-אמרים שם חמדו נשינו, חושש תרביבני; אנוכי יתאול, יחדיתי שתיה, פתח יהוה; וארון ישר, כי-הוא, הקהלתו.\n", "קמא,ב קין מצדקה תיקח יהוה-\t\n" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ " print('Sampled text:')\n", " print(decodeString(sample(500)))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "iTorch", "language": "lua", "name": "itorch" }, "language_info": { "name": "lua", "version": "5.1" } }, "nbformat": 4, "nbformat_minor": 0 }