Strings, Bytes, and Runes in Go

A username has four visible symbols, but len reports ten. Taking its first two positions produces invalid UTF-8. Converting it to []rune fixes that particular boundary problem, but still does not define what a user considers one character.

Text processing starts with choosing the right unit: bytes for encoded storage, runes for Unicode code points, and grapheme clusters for user-perceived characters. Confusing those units causes both correctness bugs and unnecessary conversions.

This is Part 4 of Go Under the Hood, following Arrays and Slices Under the Hood. You should recognise loops, functions, and slices. The examples and benchmarks use Go 1.27.1, continuing the series’ Go 1.27 baseline.

A string is immutable bytes, not necessarily valid text

A Go string holds a sequence of bytes, which need not be valid UTF-8. Its contents cannot be changed through ordinary Go operations. A string variable can still be assigned a different value.

byte is an alias for uint8; rune is an alias for int32, conventionally used for Unicode code points. UTF-8 encodes a valid Unicode scalar value in one to four bytes. Indexing a string retrieves a byte; ranging over it decodes runes and reports their starting byte offsets. Go blog: strings, bytes, runes and characters

Compare the representations before choosing one

A single byte or rune is a numeric value; []byte and []rune are slices of those values. Neither slice type is automatically a better version of string. Choose according to the operation and the interface consuming the result.

For read-only Unicode processing, you often need a string plus iteration, not a rune slice. For example, counting or examining each rune once does not require storing all the decoded runes. Conversely, repeated edits at known code-point positions can justify materialising []rune once and converting back at the end.

Why the choice matters in a backend service

Consider a proposed profile API. It receives a display name, validates it, stores it, and later produces a short preview. Each step raises a different question:

• Transport: how many encoded bytes may the request contain?
• Validation: must the field be valid UTF-8, and which characters are allowed?
• Product behaviour: does a length limit count bytes, code points, or visible characters?
• Storage and lookup: should canonically equivalent spellings compare as the same value?
• Presentation: where can the preview stop without damaging the text?

One len check cannot answer all five. If the rule allows ten code points, ten ASCII letters occupy ten bytes while ten copies of 界 occupy thirty. A ten-byte check rejects one while accepting the other, despite the stated code-point rule.

The reverse mismatch also matters: limiting rune count does not directly enforce a byte budget. If both constraints exist, check both. Define the units in the API contract so the frontend, backend, and storage layer enforce the intended rule.

There is also a data-preservation concern. If input is an opaque payload rather than text, decoding and re-encoding can change malformed byte sequences. The second experiment demonstrates that failure directly.

Experiment 1: follow the byte offsets

Create a module using a Go 1.27 toolchain:

go version
mkdir text-demo
cd text-demo
go mod init example.com/text-demo
go mod edit -go=1.27.0

Save this as main.go:

package main

import (
	"fmt"
	"unicode/utf8"
)

func main() {
	s := "Aé界🙂"
	fmt.Println("counts:", len(s), utf8.RuneCountInString(s))
	fmt.Printf("bytes: % x\n", s)
	for offset, r := range s {
		fmt.Printf("offset=%d rune=%U\n", offset, r)
	}
	fmt.Printf("index 1: %02x\n", s[1])
	fmt.Printf("prefix: % x valid=%v\n", s[:2], utf8.ValidString(s[:2]))
	fmt.Println("two runes:", string([]rune(s)[:2]))
}

Run:

go run .
go vet ./...

Expected output:

counts: 10 4
bytes: 41 c3 a9 e7 95 8c f0 9f 99 82
offset=0 rune=U+0041
offset=1 rune=U+00E9
offset=3 rune=U+754C
offset=6 rune=U+1F642
index 1: c3
prefix: 41 c3 valid=false
two runes: Aé

The range offsets are 0, 1, 3, and 6, not 0, 1, 2, and 3. Each offset identifies the first byte of the next decoded rune.

The first two bytes contain A and only the beginning of é. This slice is within bounds, so it does not panic; it simply produces an invalid encoding. Bounds safety is not text-boundary safety.

Converting to runes makes the two-code-point prefix easy to express here. For a single prefix operation on a large string, you can instead walk rune boundaries and slice at the chosen byte offset. Avoid decoding the whole string into a slice when you only need an early boundary.

A rune is not always a displayed character

The strings "\u00e9" and "e\u0301" can display as the same accented letter. The first contains one code point; the second contains a base letter and combining accent. They have different bytes and compare unequal in Go. Normalisation is a separate operation, not an automatic property of string comparison. Go blog: text normalisation

Unicode grapheme segmentation groups code points into boundaries useful for user-facing text operations. Combining sequences and some emoji sequences contain multiple code points within one extended grapheme cluster. A rune count is therefore not a general character-count rule for an editor or display limit. Unicode text segmentation

Define the product requirement before implementing truncation. A transport byte limit, a database field constraint, and a cursor movement need not use the same unit. Splitting only at rune boundaries preserves valid UTF-8 for valid input, but can still separate a combining mark from its base.

Experiment 2: distinguish invalid bytes from a replacement character

Replace main.go with:

package main

import (
	"fmt"
	"unicode/utf8"
)

func main() {
	s := string([]byte{'A', 0xff, 'B'})
	fmt.Println("valid:", utf8.ValidString(s))
	for offset, r := range s {
		fmt.Printf("offset=%d rune=%U\n", offset, r)
	}
	for _, input := range []string{"\xff", "\uFFFD"} {
		r, width := utf8.DecodeRuneInString(input)
		fmt.Printf("decode: %U width=%d\n", r, width)
	}
	fmt.Printf("byte round trip: % x\n", string([]byte(s)))
	fmt.Printf("rune round trip: % x\n", string([]rune(s)))
}

Expected output:

valid: false
offset=0 rune=U+0041
offset=1 rune=U+FFFD
offset=2 rune=U+0042
decode: U+FFFD width=1
decode: U+FFFD width=3
byte round trip: 41 ff 42
rune round trip: 41 ef bf bd 42

utf8.DecodeRuneInString returns RuneError with width one for an invalid encoding, but a correctly encoded U+FFFD consumes three bytes. Empty input returns width zero. Checking the rune value alone cannot distinguish malformed input from an intentional replacement character. utf8.ValidString provides whole-string validation. unicode/utf8 documentation

The rune conversion loses the original invalid byte. For opaque binary input, preserving the bytes may be essential. For an API requiring UTF-8 text, validate at the boundary and choose an explicit reject-or-replace policy. Conversion alone is not validation.

Experiment 3: mutation after conversion

Replace main.go with:

package main

import "fmt"

func main() {
	s := "cat"
	b := []byte(s)
	b[0] = 'b'
	t := string(b)
	b[1] = 'i'
	fmt.Println(s, t, string(b))
	fmt.Println(string(rune(65)), string([]byte{65}))
}

It prints:

cat bat bit
A A

Changing b does not change either string. The conversions must preserve those observable semantics, even when the compiler can avoid a physical copy in a particular context. A byte-slice conversion preserves string bytes; a rune-slice conversion decodes them. Converting an integer to a string encodes a code point, so string(65) produces A, not the decimal digits 65. Go specification: string conversions

Try inserting s[0] = 'b': compilation fails because string elements are not assignable. Restore the program afterwards. For decimal formatting, use an operation such as strconv.Itoa(65).

Choose an approach for the actual use case

Validate a text field at its boundary

For an endpoint requiring UTF-8 text, first enforce the request’s size limit, then validate the decoded field with utf8.ValidString. If the contract specifies a code-point limit, count with utf8.RuneCountInString; do not build []rune solely to take its length. If the limit concerns user-perceived characters, use a grapheme-segmentation implementation instead.

Validation order is useful here: malformed bytes should not quietly become replacement runes before you decide whether the input is acceptable. Valid UTF-8 still permits control characters and does not implement your field’s character policy. unicode/utf8 validation and counting

Treat normalisation as a separate contract decision. For example, an application may normalise a lookup key while preserving the original display spelling. Do not silently normalise arbitrary tokens or byte-sensitive identifiers. Normalisation handles canonical representation; it does not make every visually similar string identical. Go text normalisation

Parse or transform data in its existing representation

If input already arrives as bytes and the consumer expects bytes, keep the work in []byte where practical. The bytes package offers operations such as Contains and Cut, avoiding a string conversion merely to search or split. If the input is already a string, use the corresponding strings operations. bytes documentation, strings documentation

This is a design starting point, not a blanket optimisation rule. A parser may sensibly turn a completed field into a string for a domain API. What deserves scrutiny is a pipeline that repeatedly converts the same value back and forth without needing either mutation or a different interface.

For byte buffers, also remember the sharing rules from Part 3. A subslice can still refer to reusable input storage. Decide whether downstream code consumes it immediately or retains an independent copy.

Build output in the form the next step needs

strings.Builder is intended for constructing strings and must not be copied after it has been used. bytes.Buffer provides a growable byte buffer with reading and writing operations. strconv.AppendInt appends formatted digits and returns the resulting slice, which the caller must retain. These are API capabilities, not benchmark results comparing those tools. strings.Builder, bytes.Buffer, strconv.AppendInt

Edit text without assuming one rune is one character

Use []rune when the algorithm explicitly works on code points and benefits from indexed edits. For a one-pass transformation, iteration or a string helper may express the operation more directly.

For a display preview, cursor movement, or deleting the last visible character, code-point indexing can still split a combining sequence or emoji cluster. Choose grapheme boundaries for those requirements. If a preview must also fit an encoded byte limit, select complete clusters within that budget and include any ellipsis in the calculation.

Measure conversions with a defined lifetime

“String conversion allocates” is too broad. Cost depends on the input, conversion, and how the result is used. This benchmark deliberately retains each converted result in a package variable so the result outlives the expression. It compares that workload with counting runes directly.

Alongside main.go, create text_test.go:

package main

import (
	"strings"
	"testing"
	"unicode/utf8"
)

var (
	textInput  = strings.Repeat("Aé界🙂", 64)
	byteInput  = []byte(textInput)
	bytesSink  []byte
	runesSink  []rune
	stringSink string
	countSink  int
)

func BenchmarkToBytes(b *testing.B) {
	for b.Loop() {
		bytesSink = []byte(textInput)
	}
}

func BenchmarkToString(b *testing.B) {
	for b.Loop() {
		stringSink = string(byteInput)
	}
}

func BenchmarkToRunes(b *testing.B) {
	for b.Loop() {
		runesSink = []rune(textInput)
	}
}

func BenchmarkRuneCount(b *testing.B) {
	for b.Loop() {
		countSink = utf8.RuneCountInString(textInput)
	}
}

The workload is 640 UTF-8 bytes containing 256 runes. Input construction happens outside the benchmark loops. B.Loop controls iteration and timing; -benchmem reports heap-allocation statistics. Go testing benchmarks

Run five samples:

go test -run '^$' -bench . -benchmem -benchtime=200ms -count=5 -cpu=1

Measured on 24 September 2026 with Go 1.27.1, macOS 15.3 (build 24D2059), Apple M4, darwin/arm64, and -cpu=1. There are no external dependencies. Each row summarises five samples of the exact workload above.

The byte-conversion samples show substantial timing variation; these measurements illustrate this workload, not a stable ranking across machines or applications. B/op reports allocated heap bytes per operation, not the input size or total memory footprint.

The sinks make these measurements relevant to retained conversion results. They do not establish the cost of temporary conversions that the compiler can optimise differently. Zero heap allocations also does not mean zero memory use or zero CPU work. The rune-count operation answers a different question from materialising a rune slice.

Use direct byte operations for byte-oriented work, range or UTF-8 helpers for sequential decoding, and conversion when the resulting representation serves the task. Measure your actual workload before introducing a conversion cache or a lower-level optimisation. A Measurement-First Performance Investigation

Exercise: change the unit before changing the code

In Experiment 1, replace the input with "e\u0301". Predict the counts and offsets: three bytes, two runes, offsets zero and one. A prefix of one rune contains only the unaccented base letter. Compare it with "\u00e9", which has two bytes and one rune; change the final rune slice to [:1] for that input to avoid an out-of-range panic.

In Experiment 2, replace 0xff with 0x7f. The input becomes valid UTF-8, and the byte and rune round trips now agree. The character is a control code, demonstrating that valid UTF-8 is not the same as acceptable display text.

Finally, rerun the benchmark using an ASCII-only input of the same byte length. Explain why the number of runes changes before comparing the results. Keep the original and modified measurements separate: changing the text changes the workload.

The next planned part, Maps: Semantics Before Internals, examines keys, zero values, iteration, and shared access. For text keys, today’s distinction already matters: byte equality and visual similarity are different requirements.

Sources

• Go blog: strings, bytes, runes and characters
• Go specification: string conversions
• unicode/utf8: validation, decoding, and rune counting
• Go blog: text normalisation
• Unicode Standard Annex #29: grapheme cluster boundaries
• strings: search, joining, and Builder
• bytes: byte-oriented operations and Buffer
• strconv: appending decimal representations
• Go testing package: benchmarks