We use cookies to ensure you get the best experience on our website.
Accept
Deny
Glossary
/
Batch inference

Batch inference

Batch inference is the process of running models with batched inputs to increase throughput.

Related Articles

Real-Time vs Batch Inference for LLMs: Use Cases, Costs, Workflow
Costco of Inference: Introducing Doubleword Batched, the Inference Provider Built for Batched Workloads
Learn More
Previous Term
Next Term

No Previous Term!

Check out our other Terms

No Next Term!

Check out our other Terms

AGI
API
API-based large language models
Activation aware quantization (AWQ)
Agent
Attention
Auto regressive model
Autoscaling
BERT
Backpropagation
Bandwidth
Batch
Batch accumulation
Batch inference
Batch size
Big data
CI/CD Pipelines
CPU
Chain of thought prompting
Classification
Classification model
Cloud computing
Compression
Containerized
Context Length
Continuous batching
Data parallelism
Deep learning
Deep neural network (DNN)
Distillation
Docker
Dynamic batching
Encoder
Epoch
F-score
Falcon
Few shot learning
Few shot prompting
Fine tuning
Foundation model
GPT
GPU
Generative AI (GenAI)
HIPPA
Hallucination
Human in the loop
Inference
Inference Server
Inference optimization
Instruction tuning
Kernel
Kubernetes
LLaMA
Language model
Large language model
Latency
Llava
Machine learning (ML)
Machine learning inference
Mistral
Mixture of Expert Models (MoE)
Model
Model compilation
Model monitoring
Model parallelism
Model serving
Multi-GPU inference
Natural language processing (NLP)
Natural language understanding (NLU)
Neural networks
Ngram
Node
On-prem
Paged Attention
Perplexity
Pretrained model
Prompt engineering
Pruning
Public Cloud
Quantization
Quantization aware training
RAG (Retrieval Augmented Generation)
Rate Limits
Recurrent neural network (RNN)
Repetition penalty
Rust
Sampling temperature
Self-hosted models
Sentiment analysis
Serving
Speculative decoding
Supervised learning
Synthetic data
TPU
Tensor Parallelelism
Throughput
Titan Takeoff Inference Server
Token
Tokenization
Top K