Просмотр новости
Найдите то, что Вас интересует
Спутники
РН
SpaceX
Роскосмос
NASA
СпутникиРНSpaceXРоскосмосNASA
I built a CLI, tokcost, that counts tokens and estimates cost for
LLM text — files, stdin, or a live-wrapped command. Cargo.toml has an
empty [dependencies] section: the BPE encoder, JSON output, arg parsing,
and the live ticker's ANSI are all hand-rolled.
Why zero deps
Mostly because the alternative for exact token counts today is
pip install tiktoken or shelling out to Python, and I wanted something
that's a single static binary with no runtime to bring along. The two
real tiktoken vocabularies (cl100k_base, o200k_base, ~300k merge rules
combined) are embedded at build time from the canonical rank files and
checked in CI against real tiktoken output.
The one design choice worth discussing
The pretokenizer doesn't reproduce tiktoken's split regexes boundary-for-
boundary — it's deliberately coarser in one place. Real o200k_base cuts
a word at every case transition (myVariable → my | Variable); mine
keeps the whole alphabetic run as one piece. That's safe rather than
approximate: BPE vocabularies are trained on text that's pretokenized by
this same regex first, so no merge rule in the shipped rank tables can
ever bridge a boundary the real splitter always draws. Coalescing across
it can't change the merge sequence — there's nothing there to merge.
Splitting finer than the real regex would be unsafe (it can suppress a
real merge); I only ever go coarser. Golden tests check the final token
IDs against real tiktoken output on ~40 cases including camelCase and
case-folded contractions.
Repo (MIT): GitHub - abhiprd2000/tokcost: Exact BPE token counts and cost estimates for LLM text. Zero dependencies, single Rust binary · GitHub , crate : crates.io: Rust Package Registry
cargo install tokcost
Genuinely interested in pushback on the pretokenizer approach above, and
in clippy/idiom nits — this is a smaller codebase than most things I post
about, so it's a good one to get right.
2 posts - 2 participants
Read full topic
| # | Наименование новости | Тональность | Информативность | Дата публикации |
|---|---|---|---|---|
| 1 | Cargo-Rail v0.20: one workspace engine instead of dependency, CI, cache, release, and split-repo glue/plugins | 0 | 10.4 | 30-07-2026 |
| 2 | Multicalc: Scientific computing for real-time embedded systems in no_std Rust | 0 | 14.36 | 10-08-2026 |
| 3 | My state lib PurrGress | 0 | 12.62 | 13-08-2026 |
| 4 | [Show] taetype: Pure Rust font engine with zero unsafe code | 0 | 10.19 | 27-07-2026 |
| 5 | Dllb — the NoSQL DBMS Designed for AST Manipulation | 0 | 14.47 | 30-07-2026 |
| 6 | Multicalc: Scientific computing for real time embedded systems in no_std rust | 0 | 9.4 | 28-07-2026 |
| 7 | Сепаратор для логов. Сжимаем логи для контекста LLM без потери читаемости | 0 | 11.92 | 06-05-2026 |
| 8 | Daily Hacker News for 2026-07-23 | 0 | 10.96 | 24-07-2026 |
| 9 | graphlens: превращаем репозиторий в типизированный граф — Python, TypeScript, Go и Rust в одной модели | 0 | 7.26 | 26-06-2026 |
| 10 | Maplike - Crate with traits for abstract containers and operations on them | 2 | 7 | 06-07-2026 |