Meet Scribble for iPad

Highlight

iPadOS 14 introduces system-level Apple Pencil handwriting recognition function Scribble, users can directlyUITextField、UITextVieworUISearchBarWritten by hand and converted into text by the system in real time.

---

Core Content

In the past, when using Apple Pencil to enter text on the iPad, I often encountered a split process: first click into the text box, and then switch to the keyboard or a separate handwriting area. For interfaces such as forms, search boxes, and reminder lists, this step will interrupt the continuous operation of Pencil. With Pencil in hand, users frequently have to return to touch and keyboard logic.

Scribble makes handwriting recognition a system input capability in iPadOS 14. The user writes directly in the editable text area, and the system recognizes the handwriting on the device and passes the results to the existing text input system. Standard UIKit text controls, standard WebKit editables, and web forms are available by default, except for password fields, which are recommended to be entered via AutoFill.

Where developers really need to intervene is in customizing the text experience. Custom text editor needs to be fully implementedUITextInput, as Scribble reads content, selects ranges, and modifies text. Customized canvas, clickable area that becomes editable, and search box that moves with focus, you need to useUIScribbleInteractionorUIIndirectScribbleInteractionTell the system where it can be written, when it can be written, and which responder it should focus on after writing.

The core of this session is not complicated: try to use standard text controls; when customization is necessary, design the Pencil as an input device that requires a stable layout and sufficient writing space.

Detailed Content

The system supports standard text controls by default

(06:12) Scribble relies on existing iOS Text Input APIs.UITextField、UITextView、UISearchBarSuch UIKit standard controls, as well as standard editable content and web forms in WebKit, are automatically plugged into Scribble. Key points to check in custom text editorsUITextInput, because the system needs to use it to obtain the text content, select the range, and write the recognized text back to the view.

There are no new access codes here. The first step for developers is to write less code: use standard text controls if you can. Only when the view is not a standard text control, or it originally requires a tap to enter the editing state, will the subsequent interaction API be entered.

Key points:

• UITextInputIt is the core protocol for custom text editors to connect to Scribble. -UITextInteractionAllows custom editors to obtain the system standard cursor and selection UI.
• Scribble is not supported for password fields, and session explicitly recommends using AutoFill.

useUIScribbleInteractionAdjust the UI during writing

(09:15) Inline completion is common in keyboard input and may block handwriting when writing by hand. session is given by adding the method to the text fieldUIScribbleInteractionDetermine whether handwriting is currently being processed, and then hide the completed text.

func updateSearchCompletion() {
    customSearchField.hideCompletionText = interaction.isHandlingWriting
}

Key points:

• interaction.isHandlingWritingOnly describes whether the current interaction is processing Scribble handwriting.
• There is no change to the text input logic here, just to avoid visual elements that would cover the handwriting.
• Applicable scenarios are search suggestions, inline completion, placeholder prompts and other UIs that will appear in the writing area.

(09:35) If the interface can determine from the beginning that the user may use Pencil,UIScribbleInteraction.isPencilInputExpectedCan be used to leave space for the input area in advance.

override func viewDidAppear(_ animated: Bool) {
    if UIScribbleInteraction.isPencilInputExpected {
        let lineHeight = textField.font?.lineHeight ?? 17.0
        let heightForScribble = lineHeight * 4.0
        heightConstraint.constant = heightForScribble
    }
}

Key points:

• isPencilInputExpectedIs a class attribute used to determine whether Pencil input is possible in the current environment.
• The code uses the font line height to calculate the height of four lines of text, making the input area more suitable for handwriting.
• Session emphasizes that UI changes should occur when the user is not writing to avoid layout movement during writing.

(09:51) Another approach is to adjust the layout after the user has finished writing. The entry is the delegate methodscribbleInteractionDidFinishWriting。

func scribbleInteractionDidFinishWriting(_ interaction: UIScribbleInteraction) {
    let lineHeight = textField.font?.lineHeight ?? 17.0
    let heightForScribble = lineHeight * 4.0
    heightConstraint.constant = heightForScribble
}

Key points:

• scribbleInteractionDidFinishWritingCalled after Scribble is finished writing.
• It is suitable for expanding the input box to a more comfortable height, or restoring the layout that has been temporarily affected by handwriting.
• This code uses the same set of height calculations as the previous example, the difference lies in the triggering time.

Turn off Scribble in drawing mode

(10:08) Some apps have both drawing and text editing. When the user writes on the canvas, the system cannot automatically determine whether to draw a line or enter text this time.UIScribbleInteractionDelegateofshouldBeginAtScribble can be rejected by the current mode.

func scribbleInteraction(_ interaction: UIScribbleInteraction,
                         shouldBeginAt location: CGPoint) -> Bool {
    return !appIsInDrawingMode()
}

Key points:

• locationIt is the position where Pencil starts writing. The delegate can determine it based on the current position and app status.
• returnfalsewill prevent Scribble from starting processing on this view.
• For drawing, annotation, and whiteboard apps, this judgment can prevent the user’s strokes from being mistaken for text input.

useUIIndirectScribbleInteractionMake non-text areas writable

（10:27）UIIndirectScribbleInteractionFacing another type of problem: a certain area is not usually a text input control, but users naturally write there. The blank list area of ​​Reminders and the engraving area on the back of the notebook in the session example all fall into this scenario.

The first step is to install the interaction on the hosting view.

override init(frame: CGRect) {
    super.init(frame: frame)
    indirectScribbleInteraction = UIIndirectScribbleInteraction(delegate: self)
    addInteraction(indirectScribbleInteraction)
    ...
}

Key points:

• UIIndirectScribbleInteractionAlso added to the view.
• The delegate is responsible for describing the writable area to the system.
• The ellipsis in the official snippet represents other initialization code for the sample view, which is not part of the Scribble access step.

(11:48) The system will first ask what writable elements are in the current rect. The example only has one engraving area, so it returns an identifier directly.

func indirectScribbleInteraction(_ interaction: UIInteraction,
                                 requestElementsIn rect: CGRect,
                                 completion: @escaping ([ElementIdentifier]) -> Void) {
    completion(["EngravingIdentifier"])
}

Key points:

-The element identifier is a stable identifier defined by the app itself.

• Multiple writable areas can return multiple identifiers.
• completion is an asynchronous entry. Complex views can first calculate the hit elements and then return.

(12:14) Next, you need to tell the system the geometric position of the element in the view. The example allows the user to write over the entire lettering area, so returnbounds。

func indirectScribbleInteraction(_ interaction: UIInteraction,
                                 frameForElement elementIdentifier: String) -> CGRect {
    return bounds
}

Key points:

• frameForElementDetermines the area in which Scribble can accept handwriting.
• You can return a smaller rect so that the writable area only covers part of the control.
• The frame must be consistent with the writable area seen by the user, otherwise Pencil hits will appear unreliable.

(12:28) When the user actually writes, the system needs a responder that supports text input. The example creates a text field when needed, sets it as the first responder, and returns it via completion.

func indirectScribbleInteraction(_ interaction: UIInteraction,
                                 focusElementIfNeeded elementIdentifier: String,
                                 referencePoint focusReferencePoint: CGPoint,
                                 completion: @escaping ((UIResponder & UITextInput)?) -> Void)
{
    if editingTextField == nil {
        createTextField()
    }
    editingTextField?.becomeFirstResponder()

    completion(editingTextField)
}

Key points:

• The return value must be bothUIResponderandUITextInput, so that the system can write the recognized text into it. -createTextField()is an implementation detail in the sample app that installs the actual text field in the view hierarchy. -becomeFirstResponder()Route subsequent input to this text field.

(12:57) Finally, the delegate also tells the system whether this element is currently focused.

func indirectScribbleInteraction(_ interaction: UIInteraction,
                                 isElementFocused elementIdentifier: String) -> Bool {
    // Indicate if our only element is currently installed and focused
    return editingTextField?.isFirstResponder ?? false
}

Key points:

• This method lets the system know if the focus process needs to be executed again.
• Example only has one element, so only checkseditingTextField.
• Multi-element scenes need to be pressedelementIdentifierJudge separately.

Core Takeaways

1. Pencil-friendly search box

What to do: Add a layer of Scribble adaptation to the search box, hide completion prompts when handwriting, and expand the input area when necessary.

Why it’s worth doing: The Spotlight example of session shows that inline completion will interfere with handwriting; the Messages example shows that the small input box will limit the writing of long sentences.

How ​​to get started: Add search fieldsUIScribbleInteraction,useisHandlingWritingTo control text completion, useisPencilInputExpectedorscribbleInteractionDidFinishWritingAdjust height.

2. Text annotation mode of drawing app

What to do: Support line drawing and text annotation on the same canvas, disable Scribble in drawing mode, and allow handwriting input in text mode.

Why it’s worth doing: session explicitly mentions that when drawing and editable text are mixed, it should be passedshouldBeginAtDetermines whether Scribble starts.

How ​​to start: Add on the canvas viewUIScribbleInteraction, read the current tool status in the delegate, and return when the drawing tool is activated.false。

3. Add list items directly in the blank area

What it does: Let users handwrite in the blank space at the bottom of a list to automatically create a new task, reminder, or note entry.

Why it’s worth doing: The Reminders example demonstrates this interaction: the empty space is not originally a text control, but the user will naturally write a new content there.

How ​​to start: Add on list containerUIIndirectScribbleInteraction, return the bottom blank area as element, infocusElementIfNeededCreate and focus the new entry’s text input control.

4. Engraving input in product customization

What to do: Handwrite engraving content in the designated area on the product preview, such as a cup, notebook, or device case.

Why it’s worth doing: The session engraving example shows a complete link from a non-editing area to a hidden text field.

How ​​to start: UserequestElementsInTo expose the engraving area, useframeForElementReturn the writable range, and then use the actual text field asUIResponder & UITextInputReturn to the system.

5. Pencil input of special QA list

What to do: Add a set of Pencil input tests to the iPad app: standard text control, search box, form, blank list area, custom editor.

Why it’s worth doing: Scribble is available by default for standard controls, but session reminds developers to check that they need to tap first, the layout will move, the space is too narrow, and customizationUITextInputIncomplete scene.

How ​​to start: First use Apple Pencil to clear all text entries, and then patch the abnormal entries separately.UITextInput、UIScribbleInteractionorUIIndirectScribbleInteraction。

Related Sessions

• What’s new in PencilKit — PencilKit handles drawing, writing, and animation in iPad apps, introducing updates to PKToolPicker, PKCanvasView, PKStroke, and more.
• Inspect, modify, and construct PencilKit drawings — In-depth PKDrawing, PKStroke, ink, path, point, suitable for recognition and editing after Pencil input.
• Support hardware keyboards in your app — Explains hardware keyboard navigation, shortcut keys, responder chain, and raw keyboard events for iPadOS and Mac Catalyst apps.
• Build for the iPadOS pointer — Use the pointer interaction API to customize buttons, custom views, and area animations for the iPad pointer.
• Build for iPad — Explains the multi-column layout, lists, and low-interruption navigation of iPad apps, which are the basis for designing the Pencil input space.