Read between the strokes with PencilKit

Highlight

Apple opens the on-device handwriting recognition engine PKStrokeRecognizer to third-party developers. It supports 29 languages and adds Bezier path conversion, stroke splitting, and stable Stroke IDs, turning PencilKit from a canvas component into a lower-level tool that can be embedded in any rendering pipeline.

Core Ideas

Handwriting recognition: from system-only to available to everyone

Before iOS 27, only Freeform and Notes could use Apple’s best on-device handwriting recognition. Third-party apps either integrated expensive third-party SDKs or tried to piece something together with the Vision framework, with results far below the native system experience.

Now PKStrokeRecognizer is available to all developers. It is a Swift Actor, and all of its methods are asynchronous, so recognition does not block the main thread. The model runs fully offline, is built into the system, requires no network connection, and does not consume extra storage.

(01:46)

PKStrokeRecognizer provides three core capabilities:

• Recognized text: returns the most likely recognition result, suitable for showing what the user wrote in real time
• Indexable content: returns a string that combines all candidate words, suitable for Spotlight indexing
• Search: searches for keywords inside the canvas and returns bounding boxes for matching regions

(03:41)

Breaking out of the PKCanvasView walls

PencilKit stores stroke paths internally as cubic uniform B-splines. This representation is friendly for drawing, but it is not compatible with the Bezier paths more commonly used across the industry.

iOS 27 adds path conversion APIs that support lossless conversion between PKStrokePath and Bezier paths. This means you can convert Bezier paths from your own rendering pipeline into PKStrokePath, feed them to PKStrokeRecognizer for recognition, and never use PKCanvasView at all.

(08:38)

A more open drawing model

iOS 27 adds Identifiable conformance to PKStroke and PKStrokePath, returning stable UUIDs. After undo, redo, and transforms, the same stroke keeps the same ID.

PKCanvasView adds the canvasViewSelectionDidChange delegate method, which fires automatically when the selection changes. renderGroupID also moves from internal implementation detail to controllable property, so you can manually control which strokes are treated as “wet ink” for blended rendering.

(10:32)

Stroke splitting

iOS 27 introduces two categories of splitting APIs:

• Programmatic erasing: use a PKStrokePath as the eraser path to cut strokes programmatically, with the same effect as when a user erases on the canvas
• Substroke extraction: use parameterized subscripts to extract arbitrary segments from a stroke while preserving texture-particle consistency

Splitting operations are expensive on complex drawings with thousands of strokes, so Apple recommends running them on a background thread.

(11:36)

Details

Basic handwriting recognition

(03:53)

import PencilKit

let recognizer = PKStrokeRecognizer()
await recognizer.updateDrawing(drawing)
myLabel.text = await recognizer.recognizedText()

Key points:

• PKStrokeRecognizer is an Actor and is thread-safe.
• updateDrawing and recognizedText are both asynchronous methods.
• It uses the device language by default, and can also be configured explicitly through preferredLanguages.
• 29 languages are supported and can be queried through supportedLanguages.
• Simulator only supports Latin-script languages. Chinese, Japanese, Korean, and similar languages require testing on a real device.

Generating indexable content

(05:22)

import PencilKit

let recognizer = PKStrokeRecognizer()
await recognizer.updateDrawing(drawing)
if let indexedContent = await recognizer.indexableContent {
    index(text: indexedContent)
}

Key points:

• indexableContent returns a string that concatenates all candidate words, making it suitable for Spotlight indexing.
• When handwriting is ambiguous, such as “1” versus “l”, recognizedText returns only the best answer, while indexableContent returns all candidates.
• The recognizerVersion property identifies the current recognition model version. Persist it with indexed content, and compare the version when loading to decide whether to rebuild the index.
• Calling the recognizer after every stroke update consumes power, so throttle calls.

In-canvas search

(06:58)

import PencilKit

let recognizer = PKStrokeRecognizer()
await recognizer.updateDrawing(drawing)
let results = await recognizer.search("apple")
for result in results {
    highlight(bounds: result.bounds)
}

Key points:

• search returns an array of SearchResult, and each result contains the matching region’s bounds.
• Combine it with UIFindInteraction to implement a system-level find and replace experience.
• Search also provides a foundation for accessibility: VoiceOver can read handwritten content, and search lets screen readers locate specific words.

Key Takeaways

• Build a children’s literacy app: use PKStrokeRecognizer to check in real time whether a child has written the correct Chinese character. Combine search to judge stroke order, and use Substroke extraction to replay the writing process. Entry APIs: PKStrokeRecognizer.recognizedText(), PKStroke.subscript(_:).

• Build a handwriting-capable notes app: sync the user’s handwritten content to Core Spotlight through indexableContent, so full-text search covers handwritten notes. Entry APIs: PKStrokeRecognizer.indexableContent, recognizerVersion.

• Build a custom drawing canvas: use Metal or SwiftUI Canvas for your own rendering layer, then feed stroke data into PKStrokeRecognizer through Bezier conversion. Entry API: Bezier conversion methods on PKStrokePath.

• Build an intelligent annotation tool: use the Programmatic erasing API to implement a “strike through to delete” gesture. When the user draws a line through text, automatically recognize and delete the corresponding content. Entry API: splitting methods that use PKStrokePath as the eraser path.

• Build a handwritten formula calculator: recognize mathematical formulas written by the user, convert them to LaTeX, or use PKStrokeRecognizer to recognize numbers and operators directly, then calculate the result. Entry APIs: PKStrokeRecognizer.preferredLanguages, recognizedText().

Related Sessions

• Meet PaperKit — PaperKit is built on top of PencilKit, and all the new APIs in this session are also available in PaperKit
• SwiftUI — Learn how to integrate a PencilKit canvas in SwiftUI
• Swift Testing — Write tests for handwriting recognition logic
• UIKit modernization — Integration with the system find experience through UIFindInteraction