We started this project by wanting to compare the in-game lore of weapons, armor, and other items in the video game Dark Souls with results that could be generated from the openai chatbot, ChatGPT.
Our goal was to simply run prompts through chatGPT and get a fictional set of items that are generated by the chat bot and would have the ability to function as actual in-game lore. With that in mind, we were planning on evaluating the similarities and differences between the existing set of items from the google sheet we discovered and our fictional set that would be generated. The google sheet gave us a vast list of information on each item that we could use, but we decided to pick only the item type, name, and english description, as it made the most sense and was allowing us to reformat and remove things we didn't see necessary.
To start this process, we had to be able to scrape the information that we needed from the google sheet. We started looking at resources for Python that could allow us to do this, but soon realized there was a much simpler way to achieve this. We copied the columns we needed into a new excel sheet, and then exported it as a text file. This came with a few more bits of work, but ultimately it was what we wanted. The last steps to output only the item type, name, and description were fairly simple. We performed simple regEx to clean up the “\n” characters and other spaces that appeared when we output it to a text file.
From here, we discovered that we could simply tell ChatGPT to generate new items and put them in the format we desire. However, this was too simple. We decided we could work on using API Keys and OpenAI to train our own chatbot to fulfill a purpose of making a set of generated items that are entirely fictional, but based on what information we feed it. From this, we began work on using the Open AI documentation to begin writing Python that performs this task. With some help from other sources giving us overviews of the system, we began to learn more about training an AI to make our idea work. Our python is visible here. It includes a lot of commented out portions, but the important parts are the ones that highlight our use of ai models. This involves importing the required parts, adding the model section which allows for a process called few-shot learning, and finally printing our output. This is the basis for all of our python, and in the file, you'll find the remnants of us using it to process our data in different ways. Feel free to look around our python project directory to gain an understanding of what we were trying to accomplish.
Soon after getting our API Key working and being able to communicate with the ada model in GPT-Turbo 3.5, we realized we still have much work to do in making our AI work for us. The method for achieving this lies in utilizing fine-tuning. This involves creating a ruleset that our model follows so that instead of manually telling the model what we want to see for examples, it has background information that it applies so that it can do more with less in the prompt given. For fine tuning, we looked at the Open AI Cookbook and the Fine-Tuning documentation provided by OpenAI
From here, we found that fine-tuning is not as easy as it seems. We discovered that we needed to use a JSON file in order to train an ai model. This was a big part of the process, as we learned that we needed to implement a prompt and completion system. This involves giving an example of a prompt, and then an example of a completion. We decided the prompts should simply be asking the ai to generate a new item, and the completions should be each item type, name, and description from our source document. By using a TSV file, we were able to regex out any error-causing symbols in the format of our source information and send wrap the correct tags and prompt/completion tags around the correct information. Our JSON file is located here, and is useful in showing what the model looks like, and how our ai reads through the information that we gave it.
After making the JSON file, we needed to train our model with it so that it can be a unique kind of ai model. We formatted this to work with the base ada model by OpenAI, and used our command line to process this, since on our local computers, we were having differing issues in associating the model with the JSON file in Python. We followed the official OpenAI documentation for fine-tuning and intiated our model in command line, or CLI. This was succesful, but we still had questions on how to use it.
With our issues going on in the Python environment, we realized we had to use the CLI to send a prompt to the ai. We sent out a few, and the responses came back somewhat unfinished. Words would be cut off, and we were a little lost on what to do with it. The prompts and answers looked a bit like this:
| Prompts | Completions |
|---|---|
| "Could you generate a new item in the dark souls universe?" | Could you generate a new item in the dark souls universe? This act disturbs the sanctity of the Gods, and thousands of ordinary humans |
On top of being short and unfinished, they are still stuck in a command line. We got the idea that there was another way to view thiss, but mostly we wanted to implement it in python and switch the model. From here, we encountered another issue. We were finding many errors about chat completion lines and connecting to the OpenAI service to use our model. After some more research, we found out a big game-changer. Our model cannot be trained as a chat-completion model! As it turns out, it is currently only possible to train completion models on specific data. This explains why we got somewhat confusing data back, along with completion lines that were shorter and less cohesive and original compared to our tests with other models and Chat GPT. We then discovered that we were actually able to use the OpenAI Playground in order to alter, control, and use the fine-tuned model. You can see some examples of these outputs and visit the playground here.
That's about everything we did along the way. Everything after (and somewhat mixed in throughout this whole process) is the story of us building our site and preparing our data in a way that helps explain what we have been working on. Overall, we learned a lot and discovered the inner workings of ai. We feel this was a very adventurous project due to the growing implementation of ai models everywhere.
Eventually, we will have different parts of the page that will be viewable and will show the results of our tinkerings with OpenAI. We will be looking at different aspects of the source files, and comparing them with our eventual outputs from our trained model. This will allow us to study the creativity and intelligence of the current AI system, along with learning how to work with AI models. Overall, some major goals of this project are to gain a better understanding of AI models, compare generated results with ideas by humans, and analyze the errors made by the models.
For more information, visit our GitHub repository here. There, you can find what we're working with and see a preview of our code and more.
