Our newest AI models: Pico 1 and Tavik-1
Two small language models of our own, trained from scratch on a single RTX 3060 Ti, small enough for a Raspberry Pi.
Pico 1 · Tavik-1
We have built two small language models of our own. We trained both entirely ourselves and from scratch, on a single ordinary graphics card (RTX 3060 Ti). They are so small that they even run on a Raspberry Pi.
Pico 1: the mini model
- Only 254,000 parameters; the file is about 0.5 MB.
- Can hold simple small talk in English, knows that it is a tiny AI, and honestly says that it cannot do maths or write code.
- An experiment in how small a chat model can be at all. Its answers often don't fit the question.
Tavik-1: the big brother
- 20.4 million parameters, about 80 times larger than Pico 1. The file is 41 MB.
- Trained overnight in three stages: first it learned English from instructional texts (almost 800 million tokens), then chatting from about 586,000 conversations, and finally who it is.
- Answers in complete, fluent sentences, can write short lists, knows for example that Paris is the capital of France, and introduces itself as "Tavik, made by Convergent".
- It is still a very small model, though. Facts are often wrong, some tips are nonsense, and it cannot do maths or code.