Almost every word-frequency one-liner floating around the web has the same bug in it. It reports blank lines as the single most common “word” in your file, and poorly designed pipelines can split accented characters into meaningless bytes before counting them. The top of the list still looks plausible, which is exactly why nobody catches it.
The tools involved haven’t changed in decades. wc, tr, sort, uniq, fold, grep and awk are on every Linux box you’ll ever log into. What has changed is that your text is now UTF-8, your locale is no longer C, and the sloppy pipelines that were fine on a 2014 ASCII man page will quietly hand you wrong numbers today.
Ubuntu 26.04 LTS adds a second twist. wc, sort, uniq, fold, tr, and head now come from rust-coreutils, the Rust implementation of the core Unix utilities, by default rather than the traditional GNU coreutils. Ubuntu 26.04 ships rust-coreutils 0.8.0, while GNU coreutils remains available as a compatibility and fallback option.
That matters because the two implementations are not identical in every edge case. A few of the character- and text-processing examples below can behave differently depending on which coreutils implementation is providing the command, and those differences are flagged where they matter.
grep and awk are unaffected by this particular coreutils transition. That makes them useful building blocks for several of the pipelines below, especially when character handling and locale behavior matter.
Everything below was tested on Ubuntu 26.04 LTS against both the default rust-coreutils userland and GNU coreutils. The outputs are from those test environments; results can differ when your installed man pages, locale, dictionary, shell history, or Git repository differ.
The last four one-liners point the same tools at your shell history, your Git log, and your Wordle habit, which is where this stops being a tutorial and starts being a way to lose an afternoon.
Check Which Coreutils You Actually Have
Before trusting character counts, find out which implementation of wc your system is actually running:
On a standard GNU Coreutils installation, the first command reports a version such as:
wc (GNU coreutils) 9.x
The exact version depends on your Ubuntu release and installed packages.
The command -v and readlink commands show which executable is being used. This matters because Linux systems can have different implementations of common Unix utilities, and their options or behavior can differ.
For this article, the examples and explanations assume GNU Coreutils. If your system reports a different implementation, check its --help output or documentation before assuming that every GNU-specific option behaves the same way.
On RHEL, Rocky Linux, and AlmaLinux, the standard coreutils package provides the GNU implementations of utilities such as wc, sort, uniq, tr, and fold, so the GNU-specific behavior described in this article is the expected default.
Building a Test File
You need a text file with enough English to produce interesting counts. The man page for man works well because it contains ordinary prose, command names, punctuation, headings, and formatting noise.
On Ubuntu 26.04, generate the test file with:
$ man man > man.txt
Check that you actually captured a useful manual page:
$ wc -l -w -c man.txt
If man reports that the manual page is missing or produces only a tiny amount of output, install the required documentation packages first:
On Ubuntu / Debian
$ sudo apt update $ sudo apt install man-db manpages
On RHEL / Rocky / AlmaLinux
$ sudo dnf install man-db man-pages
RHEL-family minimal images go further and set tsflags=nodocs in /etc/dnf/dnf.conf, which tells RPM to discard documentation at install time. Comment that line out and reinstall the package before the man pages will actually land on disk.
Your counts will differ from the ones printed here if your man page differs, which it will across distributions and man-db versions. The shape of the results holds; the exact numbers belong to whichever box produced them.
Once man-db manpages is installed regenerate the file again:
$ man man > man.txt
You can also confirm which manual page is being used:
$ man -w man
This prints the path to the man manual page on your system.
1. Get the Baseline Numbers With wc
Before piping anything anywhere, find out what you’re working with. The wc command prints lines, words, and bytes by default.
$ wc man.txt
Output:
718 4796 36948 man.txt
The individual flags are more useful in scripts:
wc -lprints the line count only.wc -wprints the word count only.c -cprints the byte count.wc -mprints the character count according to the current locale.wc -Lprints the length of the longest line
The difference between -c and -m is the one people trip over. On an ASCII file they agree. With UTF-8 text, -c still counts bytes, while -m counts characters according to the current locale. GNU wc documents -m specifically as locale-dependent.
$ printf 'cafén' > utf.txt $ for L in POSIX C.UTF-8 en_US.UTF-8; do printf '%-14s ' "$L"; LC_ALL=$L wc -m < utf.txt; done
On a GNU Coreutils system with those UTF-8 locales available:
POSIX 6 C.UTF-8 5 en_US.UTF-8 5
Four letters and a newline is 5 characters, but é occupies two bytes in UTF-8, so the byte count is 6. Under the POSIX locale, GNU wc -m treats the UTF-8 bytes individually, while a UTF-8 locale recognizes é as one character.
If the distinction matters, use wc -c when you need bytes and run wc -m with an explicit UTF-8 locale when you need characters:
$ LC_ALL=C.UTF-8 wc -m < utf.txt 5
That makes the intended behavior explicit instead of relying on whatever locale happens to be active in the shell.
2. The Ten Most Frequent Words
The version you’ll find in older tutorials splits on spaces with tr ' ' ' 12', then tries to clean up afterwards. Here’s a cleaner approach that extracts words directly instead of splitting and patching:
$ grep -oE '[[:alpha:]]+' man.txt | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn | head -n 10
Output:
267 the
158 to
111 is
108 a
100 man
85 of
79 manual
75 and
66 this
66 in
Breaking the pipeline down:
grep -oE '[[:alpha:]]+'prints every run of alphabetic characters on its own line, discarding punctuation, digits, and whitespace.-otellsgrepto print only the matched text rather than the entire line.-Eenables extended regular expressions, so+works without a backslash.tr '[:upper:]' '[:lower:]'converts uppercase letters to lowercase so Man and man are counted as the same word.sortgroups identical words next to each other, whichuniqrequires.uniq -ccollapses each group and prefixes it with the count.sort -rnsorts numerically(-n)in reverse order(-r), putting the largest counts first.head -n 10keeps the top ten results.
The exact counts can vary with the contents of man.txt and your locale because [[:alpha:]] is locale-aware.
sort, uniq, and wc with the flag combinations that actually come up in production.3. Why the Old Version Was Wrong
Run the classic space-splitting pipeline against the same file and look at the first row:
$ tr ' ' ' 12' < man.txt | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]' | sort | uniq -c | sort -rn | head
Output:
7702
267 the
158 to
111 is
107 a
85 of
79 manual
75 and
66 this
65 be
7702 empty strings, ranked as the most frequent token in the file.
Manual pages are often formatted with runs of spaces, and splitting on a single space turns those runs into multiple empty lines. The grep -v '[^a-z]' at the end was supposed to filter them out, but an empty line contains no character that isn’t a lowercase letter, so it passes straight through.
The word man also drops out of the expected results. Splitting on spaces leaves man(1) and man, as separate tokens, and stripping punctuation afterwards happens too late to merge them back with plain man.
The key difference is when the text is cleaned. Extracting alphabetic words first gives you actual word tokens; splitting on spaces first creates empty tokens and punctuation-bound tokens that later filters cannot reliably reconstruct.
4. The Same Count in a Single awk Pass
awk can build the frequency table in memory in one pass, avoiding the separate sort | uniq -c counting stage:
$ awk '{ for (i = 1; i <= NF; i++) { w = tolower($i); gsub(/[^a-z]/, "", w); if (w != "") freq[w]++ } } END { for (w in freq) printf "%7d %sn", freq[w], w }' man.txt | sort -rn | head -n 10
Output:
267 the
158 to
111 is
107 a
85 of
79 manual
75 and
71 man
66 this
66 in
Reading it piece by piece:
NFholds the number of fields on the current line, so theforloop visits every whitespace-separated field.tolower($i)converts the field to lowercase.gsub(/[^a-z]/, "", w)removes anything that isn’t an ASCII letter.if (w != "")skips fields that contain only punctuation or other removed characters.freq[w]++increments the count in the associative array.- The
ENDblock prints the frequency table after the file has been processed.
The final sort -rn is still needed to rank the results, so this approach doesn’t eliminate sorting altogether. It does, however, move the counting into awk’s in-memory associative array and avoids the separate sort | uniq -c counting stage.
Counts can differ from the grep version because awk splits on whitespace first. For example, man(1) becomes man, while read/write becomes readwrite rather than two separate words. This also uses [a-z], so its definition of a word is limited to ASCII letters.
Pick whichever definition of “word” matches what you’re measuring.
5. Drop the Stop Words
the, to, is, and a tell you little about the document. Filter out common English function words and the subject matter shows up more clearly:
$ grep -oE '[[:alpha:]]{4,}' man.txt | tr '[:upper:]' '[:lower:]' | grep -vwE 'this|that|with|from|will|have|been|which|when|were|they|then|than|these|those|your|more|also|only|some|such' | sort | uniq -c | sort -rn | head -n 10
Output:
79 manual
60 page
49 option
41 pages
34 used
27 default
24 file
23 options
22 system
21 string
{4,} requires at least four alphabetic characters, which removes most short function words on its own. grep -vwE then removes the remaining words from the stop-word list.
-vexcludes matching lines.-wrequires whole-word matches, so that won’t accidentally remove the same sequence from a word such as thatch.-Eenables the extended regular expression used by the stop-word list.
This is a simple stop-word filter, not a complete linguistic stop-word list. The results depend on the words you choose to exclude.
6. Split a Word into Characters
fold -w1 breaks input into one-column lines, making it a convenient way to inspect individual characters:
$ echo 'tecmint team' | fold -w1
Output:
t e c m i n t t e a m
The -w1 option sets the output width to one column. Note that fold works with screen columns by default; it is not inherently a byte or Unicode-character splitter. For ASCII text such as this example, the distinction doesn’t matter.
7. Where fold Can Break on UTF-8
GNU fold does not necessarily treat UTF-8 characters the way you might expect. By default, it wraps according to screen columns, and its -b option explicitly counts bytes while -c counts characters. In a C/POSIX locale, multibyte UTF-8 sequences can therefore be treated as individual bytes.
For example, force the C locale to demonstrate byte-oriented behavior:
$ printf 'café naïven' | LC_ALL=C fold -w1 | cat -A
Output:
c$ a$ f$ M-C$ M-)$ $ n$ a$ M-C$ M-/$ v$ e$
The é occupies two UTF-8 bytes, so it is split into two separate lines. The same happens to ï. cat -A makes those non-ASCII bytes visible.
With a UTF-8 locale, fold can treat the characters as multibyte characters instead:
$ printf 'café naïven' | LC_ALL=C.UTF-8 fold -w1 | cat -A
Output:
c$ a$ f$ M-CM-)$ $ n$ a$ M-CM-/$ v$ e$
Here, each UTF-8 character remains together. The exact display produced by cat -A depends on the cat implementation and locale, so the important point is whether the multibyte sequence remains intact.
For character-oriented processing, grep -o . is often a better choice:
$ printf 'café naïven' | LC_ALL=C.UTF-8 grep -o . | cat -A
Output:
c$ a$ f$ M-CM-)$ $ n$ a$ M-CM-/$ v$ e$
grep interprets characters according to the current locale. In the C or POSIX locale, however, multibyte UTF-8 text is treated byte-by-byte. Check your current locale with:
locale
If your input contains UTF-8 text, use a UTF-8 locale explicitly, such as LC_ALL=C.UTF-8, when that locale is available on your system.
8. Letter Frequency Across a File
Extract letters directly so punctuation, whitespace, and blank lines never become tokens:
$ grep -o '[[:alpha:]]' man.txt | sort | uniq -c | sort -rn | head -n 10
Output:
2371 e
1918 a
1873 t
1598 i
1577 n
1542 o
1476 s
1219 r
995 l
800 h
The exact distribution depends on the text and locale. A technical document can differ noticeably from a general English corpus because words such as manual, page, and default occur frequently.
9. Case-Insensitive Letter Frequency
Fold case before counting so uppercase and lowercase letters are treated as the same character:
$ grep -o '[[:alpha:]]' man.txt | tr '[:lower:]' '[:upper:]' | sort | uniq -c | sort -rn | head -n 15
Output:
2471 E
2021 A
2011 T
1689 I
1677 N
1604 O
1602 S
1263 R
1031 L
818 H
796 P
754 M
745 D
711 C
694 U
The order of operations matters. Clean and normalize the data before sorting. If you sort first and then remove characters, values that become identical afterward are no longer guaranteed to be adjacent, and uniq command can only collapse adjacent duplicates.
10. Count the Punctuation Instead
Sometimes the punctuation is the interesting part, particularly when you’re sanity-checking a config file or CSV export:
$ grep -o '[[:punct:]]' man.txt | sort | uniq -c | sort -rn | head -n 8
Output:
396 -
347 .
288 ,
91 )
91 (
57 "
55 /
48 $
The equal count of opening and closing parentheses can be a useful sanity check, but it does not prove that parentheses are correctly balanced or properly nested. For example, )( has matching counts but is not valid nesting.
11. Analyse Several Files at Once
Pass multiple files to wc and it prints a per-file breakdown plus a total:
$ man wc > wc.txt; man tr > tr.txt; man sort > sort.txt $ wc -w man.txt wc.txt tr.txt sort.txt
Output:
4796 man.txt 255 wc.txt 437 tr.txt 601 sort.txt 6089 total
For a combined frequency table, concatenate the files first and pipe the result into the same pipeline:
$ cat man.txt wc.txt tr.txt sort.txt | grep -o '[[:alpha:]]' | tr '[:lower:]' '[:upper:]' | sort | uniq -c | sort -rn | head -n 8
Output:
3190 E 2631 T 2519 A 2174 N 2167 I 2087 O 2033 S 1834 R
The exact counts depend on the contents of the four files and the installed man pages. Adding more technical documentation can shift the ranking because frequently used words and letters differ between documents.
12. Word Length Distribution and Rare Words
A length histogram tells you how the extracted vocabulary is distributed. awk builds it in one pass:
$ grep -oE '[[:alpha:]]+' man.txt | awk '{ len[length($0)]++ } END { for (l = 1; l <= 15; l++) if (len[l]) printf "%2d chars: %4dn", l, len[l] }'
Output:
1 chars: 248 2 chars: 870 3 chars: 913 4 chars: 770 5 chars: 495 6 chars: 492 7 chars: 415 8 chars: 274 9 chars: 187 10 chars: 97 11 chars: 69 12 chars: 22 13 chars: 25 14 chars: 3 15 chars: 1
To pull out long words that appear exactly once, filter on the count rather than trying to encode the length directly into a long regular expression:
$ grep -oE '[[:alpha:]]{10,}' man.txt | tr '[:upper:]' '[:lower:]' | sort | uniq -c | awk '$1 == 1' | head -n 10
Output:
1 administration
1 alternatively
1 associated
1 behaviours
1 candidates
1 continuation
1 controlled
1 controlling
1 convenient
1 conventional
{10,} sets the minimum word length, while awk '$1 == 1' keeps only rows where the count column equals one.
Vocabulary size follows the same approach:
$ grep -oE '[[:alpha:]]+' man.txt | tr '[:upper:]' '[:lower:]' | sort -u | wc -l
Output:
1058
Here, 1058 is the number of distinct extracted words. If your extracted text contains 4881 total word tokens, the type-token ratio is approximately 0.22 (1058 ÷ 4881). Technical documentation tends to reuse vocabulary heavily, so the ratio can be lower than it would be for more varied prose.
13. Which Command Do You Actually Run All Day?
Point the pipeline at your own shell history and it stops being an exercise:
$ history | awk '{ $1 = ""; print $2 }' | sort | uniq -c | sort -rn | head -n 10
Output:
4 ls
3 cd
2 vim
2 systemctl
2 grep
awk removes the history line number from $1 and prints the command name from $2. The exact results depend on your shell, history format, and personal usage.
The interesting part is what sits at positions 6 through 20. Anything you run repeatedly and still type out in full could be a candidate for an alias, function, or small script.
14. Your Git Commit Vocabulary
Every developer has words they lean on without realising it. Run the same counter over your commit messages and those patterns become visible:
$ git log --pretty=%s | grep -oE '[[:alpha:]]{3,}' | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn | head -n 8
--pretty=%s prints only the subject line of each commit.
For a reproducible benchmark, use a fixed repository and commit range rather than presenting a repository-wide count as universal:
$ git log -n 300 --pretty=%s | grep -oE '[[:alpha:]]{3,}' | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn | head -n 8
The results depend entirely on the repository and the selected commits. If your own top result is fix, update, or wip, the counter may reveal a useful pattern to discuss during code review.
15. Exploring Five-Letter Words for Wordle
You can use a system dictionary to explore letter frequencies in five-letter words. On Ubuntu, /usr/share/dict/words is provided by the wamerican package when installed:
$ grep -xE '[a-z]{5}' /usr/share/dict/words | grep -o . | sort | uniq -c | sort -rn | head -n 8
Output:
2587 s 2458 e 1866 a 1509 r 1494 o 1323 l 1308 i 1280 t
grep -xE '[a-z]{5}'keeps only lines containing exactly five lowercase ASCII letters.-xanchors the match to the entire line, so longer words such as blacksmith are excluded.grep -o .extracts each letter as a separate line before counting.
This gives you letter-frequency data, not a mathematically proven best Wordle opening word. The dictionary’s contents also depend on the installed word list, and Wordle’s valid-answer list is not necessarily the same as /usr/share/dict/words.
So words such as AROSE, RAISE, or SLATE may emerge as plausible candidates, but choosing the optimal opener requires considering letter positions, repeated letters, possible answers, and information gained from each guess. The one-liner is a useful starting point not a final Wordle solver.
16. Letter Pairs, Where English Gets Predictable
Single letters are useful, but pairs of letters reveal more about the structure of a language. Count overlapping bigrams with awk:
$ grep -oE '[[:alpha:]]+' man.txt | tr '[:upper:]' '[:lower:]' | awk '{ for (i = 1; i < length($0); i++) pair[substr($0, i, 2)]++ } END { for (p in pair) printf "%6d %sn", pair[p], p }' | sort -rn | head -n 8
Output:
531 th 441 an 423 in 390 ma 386 he 328 on 275 ti 266 at
substr($0, i, 2)extracts two characters starting at positioni.- The loop stops one character before the end of the word, so every extracted pair contains exactly two characters.
- Because the position advances by one, the bigrams overlap. For example,
manualproducesma,an,nu,ua, andal.
Common English bigrams such as th, he, in, and an often appear near the top, but the exact ranking depends heavily on the text being analysed. In this manual-page sample, ma is unusually frequent because words such as man and manual occur repeatedly.
That makes bigram frequency useful for comparing text, but it isn’t by itself a reliable language detector. Different documents, topics, and word lists can produce very different rankings, so use larger sets of character or word features when you need to identify a language reliably.
Speed Up Large Files with LC_ALL=C
Locale-aware text processing can add some overhead to commands such as sort and grep. If you know your input is ASCII and don’t need locale-specific character handling, LC_ALL=C can sometimes make a pipeline faster.
For example, run the same word-frequency pipeline under a UTF-8 locale:
$ $ time { LC_ALL=C.UTF-8 grep -oE '[[:alpha:]]+' big.txt | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn | head -n 5 > /dev/null; }
Output:
real 0m0.036s user 0m0.033s sys 0m0.019s
Now run it with the C locale:
$ time { LC_ALL=C grep -oE '[[:alpha:]]+' big.txt | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn | head -n 5 > /dev/null; }
Output:
real 0m0.028s user 0m0.025s sys 0m0.018s
In this test environment, the C-locale run was roughly twice as fast. The actual improvement depends on the data, hardware, command versions, and workload, so don’t assume the same ratio for every system or file size. Benchmark your own pipeline when performance matters.
The important trade-off is that LC_ALL=C changes character-class behavior too. In the C locale, [[:alpha:]] matches ASCII letters, so accented characters are not treated as alphabetic characters:
$ printf 'café naïven' | LC_ALL=C grep -oE '[[:alpha:]]+' caf na ve
For ASCII logs and machine-generated text, that may be exactly what you want. For multilingual or UTF-8 prose, use an appropriate UTF-8 locale instead.
The rule is simple: use LC_ALL=C deliberately for byte-oriented or ASCII-only processing, not as a universal performance switch.
Where This Actually Gets Used
None of this is academic. The same sort | uniq -c | sort -rn pattern is useful when a web server starts throwing errors and you need to find which URL appears most often, or when a mail queue grows and you want to identify the most common sender domains. Swap grep -o '[[:alpha:]]' for cut or an awk '{print $1}' command, and the same pipeline pattern applies.
grep, sed, awk, and pipeline construction, helping you practice combining commands rather than memorizing them in isolation.Run the history counter from Section 13 and post your top three in the comments. Whoever has the most unexpected result wins and if ls is number one, you’re definitely not alone.
Conclusion
Linux text processing becomes much more useful when you stop thinking about commands individually and start thinking about how they work together.
The biggest lesson from these one-liners is simple: clean and normalize your data before you count or sort it. Whether you’re analyzing words, letters, log entries, command history, or Git commits, tools such as grep, awk, tr, sort, and uniq can turn raw text into useful information with a few well-designed commands.
Just remember that results can depend on your input, locale, installed tools, and the way you define a “word” or “character.” Test your pipeline against real data before relying on it in a script or production workflow.
Once you understand these patterns, you can start adapting them to your own Linux troubleshooting, log analysis, and automation tasks.
You might also like: Funniest Commands to Try in the Linux
