Tulip is a lightweight gRPC inference server written in Rust by Perplexity, positioned between the Ivy and ROSE engines. It collects incoming requests, batches them, and then sends them to the GPU.
Tulip is a lightweight gRPC inference server written in Rust by Perplexity, positioned between the Ivy and ROSE engines. It collects incoming requests, batches them, and then sends them to the GPU.
For small embedding models, runtime depends on the number of tokens rather than the number of queries, so approximately 512 tokens are enough to saturate the GPU. https://t.co/2hTVDKIndU