Make apps smarter with Natural Language

Highlight

Natural Language at WWDC 2020 forNLTaggerIncreasetagHypothesesThe confidence level isNLEmbeddingAdd sentence-level vectors and extend Create ML’s transfer learning to custom word sequence taggers.

Core Content

A news app wants to extract places, people and organizations from articles. In the past, it couldNLTaggerprediction result, but I don’t know how reliable this result is. The Buzz example in Session encounters this kind of problem:Do Not Disturb while drivingIt was regarded as the organization name, and a useless entity appeared on the right side of the interface.

WWDC 2020’s Natural Language adds confidence to such predictions.tagHypothesesCandidate labels and their confidence scores are returned. Developers can calibrate thresholds with data representing their own business and set different thresholds by category to reduce false positives in named entity recognition.

Another pain point is search. The FAQ of Nosh, a food delivery app, can only be found by users scrolling to find the answer. Word vectors can express the distance between words, but averaging the vectors per word will lose the sentence structure. The new sentence embedding directly encodes a sentence into a 512-dimensional vector, allowing queries such as “Where do you deliver?” to match semantically similar FAQs.

As the text size continues to grow, linear traversal becomes slower. The Verse example compresses the sentence vectors of each line of poetry into custom embedding, and then uses nearest neighbor search to find the semantically closest verse. Finally, Nosh also uses a custom word tagger to extract fields such as food, restaurant, from_city, and to_city from user input, turning free text into executable business parameters.

Detailed Content

Use confidence to filter named entity misidentifications

(03:46) Natural Language’s basic capabilities include language identification, word segmentation, part-of-speech tagging, lemmatization, and named entity recognition. In the past these APIs returned predicted labels. New in 2020tagHypothesesReturns the candidate label and corresponding confidence, which the Buzz example uses to filter out low-confidence misidentifications of organization names.

import NaturalLanguage

let tagger = NLTagger(tagSchemes: [.nameType])
let str = "Tim Cook is very popular in Spain."
let strRange = Range(uncheckedBounds: (str.startIndex, str.endIndex))
tagger.string = str
tagger.setLanguage(.english, range: strRange)

tagger.enumerateTags(in: str.startIndex..<str.endIndex, unit: .word, scheme: .nameType, options: .omitWhitespace) { (tag, tokenRange) -> Bool in
    let (hypotheses, _) = tagger.tagHypotheses(at: tokenRange.lowerBound, unit: .word,
                                               scheme: .nameType, maximumCount: 1)
    print(hypotheses)
    return true
}

Key points:

• NLTagger(tagSchemes: [.nameType])Use named entity recognition. -tagger.stringandsetLanguageHand over text and language information to the tagger. -enumerateTagsWord-by-word traversal prediction results. -tagHypotheses(at:unit:scheme:maximumCount:)Returns the candidate label and confidence at the token position.
• Session recommends calibrating thresholds with the application’s own representative data and setting thresholds separately by category.

Use word vectors and sentence vectors to express similarity

（12:16）NLEmbedding.wordEmbedding(for:)Provides static word embeddings. Given a word, you can get a vector, calculate the distance between two words, or enumerate nearest neighbor words. It is suitable for processing word-level similarity.

import NaturalLanguage

if let embedding = NLEmbedding.wordEmbedding(for: .english) {
    let word = "bicycle"

    if let vector = embedding.vector(for: word) {
        print(vector)
    }

    let dist = embedding.distance(between:word, and: "motorcycle")
    print(dist)

    embedding.enumerateNeighbors(for: word, maximumCount: 5) { neighbor, distance in
        print("\(neighbor): \(distance.description)")
        return true
    }
}

Key points:

• wordEmbedding(for:)Get the system’s built-in word embeddings by language. -vector(for:)Returns the representation of the word in vector space. -distance(between:and:)Calculate the distance between two words. -enumerateNeighborsEnumerate neighboring words in vector space.

(16:30) FAQ search requires sentence-level semantics. Session description sentence embedding will encode a sentence into a 512-dimensional vector and calculate the distance between two sentences, which is suitable for processing matching between user queries and FAQ questions.

import NaturalLanguage

if let embedding = NLEmbedding.sentenceEmbedding(for: .english) {
    let sentence = "This is a sentence."

    if let vector = sentenceEmbedding.vector(for: sentence) {
        print(vector)
    }

    let dist = sentenceEmbedding.distance(between: sentence, and: "That is a sentence.")
    print(dist)
}

Key points:

• sentenceEmbedding(for:)It is a sentence-level entry. -vector(for:)Encode complete sentences into fixed-dimensional vectors. -distance(between:and:)Compare the distance between two sentences in semantic space.
• Transcript clearly states that this ability is suitable for single sentences, several sentences or short paragraphs, and long texts should be broken into sentences for processing.

Use sentence vectors to perform nearest neighbor search

(18:31) Nosh’s FAQ search prepares two or three example queries for each answer in advance, and saves the sentence vectors of these queries. When running, the user input is also converted into a sentence vector, and then the distance is calculated with the vector in the table. The key with the smallest distance corresponds to the answer to be displayed.

func answerKey(for string: String) -> String? {
        guard let embedding = NLEmbedding.sentenceEmbedding(for: .english) else { return nil }
        guard let queryVector = embedding.vector(for: string) else { return nil }

        var answerKey: String? = nil
        var answerDistance = 2.0

        for (key, vectors) in self.faqEmbeddings {
            for (vector) in vectors {
                let distance = self.cosineDistance(vector, queryVector)
                if (distance < answerDistance) {
                    answerDistance = distance
                    answerKey = key
                }
            }
        }

        return answerKey
    }

Key points:

• The method first obtains the English sentence embedding, and then converts the user input intoqueryVector。
• faqEmbeddingsWhat is stored is a precomputed example query vector. -cosineDistanceUsed to compare the distance between the user query and each example query. -answerDistanceRecord the current minimum distance,answerKeyRecord the corresponding answers.

(22:16) When the amount of data becomes thousands of lines of poetry, linear search will slow down. The Verse example puts the sentence vector of each line of poetry into the dictionary, and then uses Create ML to generate a Core ML custom embedding. This model both compresses the vector table and preserves geometric information for nearest neighbor searches.

import NaturalLanguage
import CreateML


let embedding = try MLWordEmbedding(dictionary: sentenceVectors)

try embedding.write(to: URL(fileURLWithPath: "/tmp/Verse.mlmodel"))

Key points:

• sentenceVectorsThe key can be a custom string. The example uses a key that can reverse the poem and line number. -MLWordEmbedding(dictionary:)Convert a vector dictionary into an embedding that can be written by Create ML. -write(to:)The output is a Core ML model file.
• Session Description This custom embedding allows nearest neighbor queries to avoid traversing the entire table.

(22:43) When the App is running, sentence embedding is still used to calculate the user query vector, but the nearest neighbor retrieval is completed by custom embedding.

func answerKeyCustom(for string: String) -> String? {
        guard let embedding = NLEmbedding.sentenceEmbedding(for: .english) else { return nil }
        guard let queryVector = embedding.vector(for: string) else { return nil }

        guard let (nearestLineKey, _) = self.customEmbedding.neighbors(for: queryVector, maximumCount: 1).first else { return nil }

        return self.poemKeyFromLineKey(nearestLineKey)
    }

Key points:

• queryVectorstill fromNLEmbedding.sentenceEmbedding(for:)。
• customEmbedding.neighbors(for:maximumCount:)Returns the nearest neighbor directly. -nearestLineKeyIs the key reserved when writing custom embedding. -poemKeyFromLineKeyConvert line-level keys back to poem-level results.

Training and using a custom word sequence tagger

(29:03) The task of Custom word tagger is to tag each token in a sentence. The Nosh example wants to extract fields such as food, from_city, to_city, restaurant, etc. from “Do you deliver pizza from Cupertino”. Transcript mentioned that a simple word list can be usedNLGazetteer, but the vocabulary cannot cover all values ​​and does not understand the context; the word tagger can use context to distinguish the departure city and the delivery city.

(33:21) In the training phase, labels are first defined, and sample sentences are collected and labeled. Session shows the token/label parallel sequence in JSON form, and explains that transfer learning uses word tagging in 2020. Create ML will hand over the training data to Natural Language and finally produce the Core ML model.

(36:45) Used in the deployment phaseNLModelLoad the compiled model and hook it toNLTaggerOn the custom tag scheme.NLTaggerResponsible for word segmentation and word-by-word application models.

func findTags(for string: String) {
        let model = try! NLModel(contentsOf: Bundle.main.url(forResource: "Nosh", withExtension: "mlmodelc")!)
        let tagger = NLTagger(tagSchemes: [NoshTags])

        tagger.setModels([model], forTagScheme: NoshTags)
        tagger.string = string

        tagger.enumerateTags(in: string.startIndex..<string.endIndex, unit: .word, scheme: NoshTags, options: .omitWhitespace) { (tag, tokenRange) -> Bool in

            let name = String(string[tokenRange])

            switch tag {
                case NoshTagRestaurant:
                    self.noteRestaurant(name)
                case NoshTagFood:
                    self.noteFood(name)
                case NoshTagFromCity:
                    self.noteFromCity(name)
                case NoshTagToCity:
                    self.noteToCity(name)
                default:
                    break
            }
            return true
        }
    }

Key points:

• NLModel(contentsOf:)Load already compiled into bundlemlmodelc。
• NLTagger(tagSchemes: [NoshTags])Create a custom label system. -setModels([model], forTagScheme:)Bind the model to this tag system. -enumerateTagsReturns the labels predicted by the model word by word. -switch tagHand over restaurant, food, from_city, and to_city to the business processing functions respectively.

Core Takeaways

• Do a local FAQ semantic search. What it does: Let users enter a natural language question and jump directly to the closest FAQ answer. Why it’s worth doing: Session shows the process of using sentence embedding to compare user queries and precomputed FAQ queries. How to get started: Prepare a few example questions for each answer, precompute vectors, and useNLEmbedding.sentenceEmbedding(for:)Take the query vector and calculate the distance.

• Confidence panel for news entity extraction. What to do: Display people, places, and organization names on the right side of the article page, and hide low-confidence entities. Why it’s worth doing: Buzz example shows that named entity recognition can misidentify phrases as organization names.tagHypothesesCan provide filtering basis. How to start: Record the confidence score of each entity, and use real article samples to calibrate the thresholds for person, place, and organization respectively.

• Do natural language retrieval of pictures or documents. What it does: Find images and documents that most closely match a caption, summary, or paragraph to a user-entered sentence. Why it’s worth doing: Transcript mentions that FindMyShot can search using sentence embedding of image captions, and also mentions that documents, messages, comments and problem reports can be clustered. How to start: Calculate sentence vectors for each caption or abstract, linear search for small data sets, and then convert to custom embedding for large data sets.

• Make takeaway or travel input parser. What to do: Extract food, restaurant, departure and arrival cities from a sentence of free text. Why it’s worth doing: Nosh example shows how custom word tagger can be contextually differentiatedfrom_cityandto_city. How to start: Define tags, collect sample sentences that users may enter, use Create ML to train word tagger, and then useNLModelandNLTaggerRun within the app.

Related Sessions

• Control training in Create ML with Swift — Continue learning how to control Create ML training from code, save checkpoints, and monitor training progress.
• Get models on device using Core ML Converters — Learn how to convert external machine learning models into Core ML and connect them to the device-side inference process.
• Use model deployment and security with Core ML — After the text model training is completed, you can continue to look at the model deployment, encryption, and remote update strategies.
• Build Image and Video Style Transfer models in Create ML — Understand the complete path to training data, parameters, and on-device model delivery with another class of Create ML custom models.