{
    "componentChunkName": "component---src-templates-blog-post-js",
    "path": "/playing-with-gpt-2-text-generation-in-python/",
    "result": {"data":{"site":{"siteMetadata":{"title":"catch { snail }","author":"Andrew"}},"markdownRemark":{"id":"ab301b50-efd0-5fb0-9eba-2da813138b10","excerpt":"OpenAI finally released the full 1.5B parameter version of GPT-2 back in November, after spending most of 2019 saying it was “too dangerous” to release…","html":"<p>OpenAI finally released the full 1.5B parameter version of GPT-2 back in November, after spending most of 2019 saying it was “too dangerous” to release. Meanwhile someone built <a href=\"https://aidungeon.io/\">AI Dungeon</a> on top of it and half the internet is playing a text adventure game powered by a neural network. So I figured it was time to poke at it myself.</p>\n<p>The easiest way to run GPT-2 today is the <code class=\"language-text\">transformers</code> library from Hugging Face. No TensorFlow checkpoints, no cloning the OpenAI repo:</p>\n<div class=\"gatsby-highlight\" data-language=\"bash\"><pre class=\"language-bash\"><code class=\"language-bash\">pip <span class=\"token function\">install</span> transformers torch</code></pre></div>\n<p>Then generating text is a few lines:</p>\n<div class=\"gatsby-highlight\" data-language=\"python\"><pre class=\"language-python\"><code class=\"language-python\"><span class=\"token keyword\">from</span> transformers <span class=\"token keyword\">import</span> GPT2LMHeadModel<span class=\"token punctuation\">,</span> GPT2Tokenizer\n<span class=\"token keyword\">import</span> torch\n\ntokenizer <span class=\"token operator\">=</span> GPT2Tokenizer<span class=\"token punctuation\">.</span>from_pretrained<span class=\"token punctuation\">(</span><span class=\"token string\">\"gpt2-medium\"</span><span class=\"token punctuation\">)</span>\nmodel <span class=\"token operator\">=</span> GPT2LMHeadModel<span class=\"token punctuation\">.</span>from_pretrained<span class=\"token punctuation\">(</span><span class=\"token string\">\"gpt2-medium\"</span><span class=\"token punctuation\">)</span>\n\nprompt <span class=\"token operator\">=</span> <span class=\"token string\">\"Docker Swarm is dead because\"</span>\ninput_ids <span class=\"token operator\">=</span> tokenizer<span class=\"token punctuation\">.</span>encode<span class=\"token punctuation\">(</span>prompt<span class=\"token punctuation\">,</span> return_tensors<span class=\"token operator\">=</span><span class=\"token string\">\"pt\"</span><span class=\"token punctuation\">)</span>\n\noutput <span class=\"token operator\">=</span> model<span class=\"token punctuation\">.</span>generate<span class=\"token punctuation\">(</span>\n    input_ids<span class=\"token punctuation\">,</span>\n    max_length<span class=\"token operator\">=</span><span class=\"token number\">100</span><span class=\"token punctuation\">,</span>\n    do_sample<span class=\"token operator\">=</span><span class=\"token boolean\">True</span><span class=\"token punctuation\">,</span>\n    top_k<span class=\"token operator\">=</span><span class=\"token number\">50</span><span class=\"token punctuation\">,</span>\n    top_p<span class=\"token operator\">=</span><span class=\"token number\">0.95</span><span class=\"token punctuation\">,</span>\n<span class=\"token punctuation\">)</span>\n\n<span class=\"token keyword\">print</span><span class=\"token punctuation\">(</span>tokenizer<span class=\"token punctuation\">.</span>decode<span class=\"token punctuation\">(</span>output<span class=\"token punctuation\">[</span><span class=\"token number\">0</span><span class=\"token punctuation\">]</span><span class=\"token punctuation\">,</span> skip_special_tokens<span class=\"token operator\">=</span><span class=\"token boolean\">True</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span></code></pre></div>\n<p>A few notes from a couple of evenings with it:</p>\n<p>The <code class=\"language-text\">gpt2-medium</code> (355M) model runs fine on CPU on my laptop, just slowly — a few seconds per generation. The full <code class=\"language-text\">gpt2-xl</code> (1.5B) wants about 6GB of RAM and is painful without a GPU.</p>\n<p>Sampling parameters matter a lot. Greedy decoding produces repetitive garbage (“the the the”). <code class=\"language-text\">top_k</code> + <code class=\"language-text\">top_p</code> (nucleus sampling) is what makes the output feel coherent.</p>\n<p>And “coherent” is doing a lot of work in that sentence. The text is grammatical and stays on topic for a paragraph or two, then confidently drifts into nonsense. It will invent APIs, quotes, and facts with total confidence. Fun toy, but I wouldn’t put it anywhere near production.</p>\n<p>Still, between this, BERT now running inside Google Search, and Google’s new Meena chatbot paper from a few weeks ago, it feels like NLP is where the interesting stuff is happening right now. The Hugging Face library making it a <code class=\"language-text\">pip install</code> away is a big part of that.</p>\n<p>Next I want to try fine-tuning the small model on my own commit messages and see what horror it produces.</p>","frontmatter":{"title":"Playing with GPT-2 text generation in Python","date":"February 21, 2020","description":"Running OpenAI's GPT-2 locally with the Hugging Face transformers library"}}},"pageContext":{"slug":"/playing-with-gpt-2-text-generation-in-python/","previous":{"fields":{"slug":"/configer-docker-swarm/"},"frontmatter":{"title":"Configure Docker Swarm"}},"next":{"fields":{"slug":"/i-fine-tuned-gpt-2-on-my-commit-messages-im-sorry/"},"frontmatter":{"title":"I fine-tuned GPT-2 on my commit messages (I'm sorry)"}}}},
    "staticQueryHashes": ["3435731857","426816048","63159454"]}