This collaborative space allows users to contribute additional information, tips, and insights to enhance the original deal post. Feel free to share your knowledge and help fellow shoppers make informed decisions.
noob question- what can this do in terms of locally hosted llm? In terms of tokens for large models (wondering if I can use this as llm host for work where we don't want to send data to cloud).
Primary usage will be internal integration with automated jobs that handle sensitive data, coding assistant usage, potential internal application development
noob question- what can this do in terms of locally hosted llm? In terms of tokens for large models (wondering if I can use this as llm host for work where we don't want to send data to cloud).
Primary usage will be internal integration with automated jobs that handle sensitive data, coding assistant usage, potential internal application development
noob question- what can this do in terms of locally hosted llm? In terms of tokens for large models (wondering if I can use this as llm host for work where we don't want to send data to cloud).
Primary usage will be internal integration with automated jobs that handle sensitive data, coding assistant usage, potential internal application development
Max 4-5 parallel users
Too slow with 300GB memory bandwidth, you looking to share maybe 20 tokens per second max for Qwen 3.8 27B?
google: Token Generation: On the dense 27B variant of Qwen 3.8, inference speeds typically range around 13.4 to 27+ tokens per second, depending on the quantization level (e.g., Q4_K_M), inference engine (llama.cpp, SGLang, or vLLM), and whether speculative decoding like DFlash is enabled.
Divide the 20 tgs amongst the users and that's gonna be your max speed simultaneously.
This is for larger LLM, but then they get slower and slower based on foot-print.
If you are just running Qwen 3.8 27B, you're better off with AMD R9700 32GB on linux/vllm which can reach 150+ tps and better throughput for sure. But that's limited to 32GB VRAM on a single card but it's just $1700 for the card.
If you are running larger models and have a real need, you may not be asking and looking for this device.
Like
Helpful
Funny
Not helpful
Join The Conversation
Share your experience with the Slickdeals community
Share information with the community. Please follow our Community Guidelines and be kind!
Join The Conversation
Share your experience with the Slickdeals community
Share information with the community. Please follow our Community Guidelines and be kind!
7 Comments
Sign up for a Slickdeals account to remove this ad.
Primary usage will be internal integration with automated jobs that handle sensitive data, coding assistant usage, potential internal application development
Max 4-5 parallel users
Primary usage will be internal integration with automated jobs that handle sensitive data, coding assistant usage, potential internal application development
Max 4-5 parallel users
tl;dr: Unless privacy is an absolute must, you will be far better off with a $20 ChatGPT subscription.
Primary usage will be internal integration with automated jobs that handle sensitive data, coding assistant usage, potential internal application development
Max 4-5 parallel users
google: Token Generation: On the dense 27B variant of Qwen 3.8, inference speeds typically range around 13.4 to 27+ tokens per second, depending on the quantization level (e.g., Q4_K_M), inference engine (llama.cpp, SGLang, or vLLM), and whether speculative decoding like DFlash is enabled.
Divide the 20 tgs amongst the users and that's gonna be your max speed simultaneously.
This is for larger LLM, but then they get slower and slower based on foot-print.
If you are just running Qwen 3.8 27B, you're better off with AMD R9700 32GB on linux/vllm which can reach 150+ tps and better throughput for sure. But that's limited to 32GB VRAM on a single card but it's just $1700 for the card.
If you are running larger models and have a real need, you may not be asking and looking for this device.
Join The Conversation
Share your experience with the Slickdeals community
Share information with the community. Please follow our Community Guidelines and be kind!