Google Coral Integration

[OishiiCha]

Possible integration of google coral acceleration card.

[d13g4]

Does that even work with llms?
I did some research a couple of weeks ago and my results were:

1. LLMs need to calculate on huge chunks of data for every iteration
2. CoralAI has no RAM
3. The speed of the fastest interface I could find (PCI-E Gen 3) is about 1GB/s for every lane
4. Every iteration of the network needs multiple runs though the coral-AI-card
5. A 13b net is about 8GB
   --
   Suppose we have PCI-E x16 (we don't) and we need to push the data though the card only once per token (we need way more to my knowledge) we could have at maximum:
   16GB/s divided by 8GB: 2 tokens per second.

More realistic would be about 0.3 tokens per second (my estimate) or even less.

So: Probably no benefit to have coral-acceleration support... :(
(I was quite sad when I came to that conclusion)

--
If someone spots an error please say so!