Class SimpleNaiveBayesClassifier

    • Field Detail

      • textFieldNames

        protected final java.lang.String[] textFieldNames
        names of the fields to be used as input text
      • classFieldName

        protected final java.lang.String classFieldName
        name of the field to be used as a class / category output
      • analyzer

        protected final Analyzer analyzer
        Analyzer to be used for tokenizing unseen input text
      • indexSearcher

        protected final IndexSearcher indexSearcher
        IndexSearcher to run searches on the index for retrieving frequencies
      • query

        protected final Query query
        Query used to eventually filter the document set to be used to classify
    • Constructor Detail

      • SimpleNaiveBayesClassifier

        public SimpleNaiveBayesClassifier​(IndexReader indexReader,
                                          Analyzer analyzer,
                                          Query query,
                                          java.lang.String classFieldName,
                                          java.lang.String... textFieldNames)
        Creates a new NaiveBayes classifier.
        Parameters:
        indexReader - the reader on the index to be used for classification
        analyzer - an Analyzer used to analyze unseen text
        query - a Query to eventually filter the docs used for training the classifier, or null if all the indexed docs should be used
        classFieldName - the name of the field used as the output for the classifier NOTE: must not be havely analyzed as the returned class will be a token indexed for this field
        textFieldNames - the name of the fields used as the inputs for the classifier, NO boosting supported per field
    • Method Detail

      • assignClass

        public ClassificationResult<BytesRef> assignClass​(java.lang.String inputDocument)
                                                   throws java.io.IOException
        Description copied from interface: Classifier
        Assign a class (with score) to the given text String
        Specified by:
        assignClass in interface Classifier<BytesRef>
        Parameters:
        inputDocument - a String containing text to be classified
        Returns:
        a ClassificationResult holding assigned class of type T and score
        Throws:
        java.io.IOException - If there is a low-level I/O error.
      • getClasses

        public java.util.List<ClassificationResult<BytesRef>> getClasses​(java.lang.String text)
                                                                  throws java.io.IOException
        Description copied from interface: Classifier
        Get all the classes (sorted by score, descending) assigned to the given text String.
        Specified by:
        getClasses in interface Classifier<BytesRef>
        Parameters:
        text - a String containing text to be classified
        Returns:
        the whole list of ClassificationResult, the classes and scores. Returns null if the classifier can't make lists.
        Throws:
        java.io.IOException - If there is a low-level I/O error.
      • getClasses

        public java.util.List<ClassificationResult<BytesRef>> getClasses​(java.lang.String text,
                                                                         int max)
                                                                  throws java.io.IOException
        Description copied from interface: Classifier
        Get the first max classes (sorted by score, descending) assigned to the given text String.
        Specified by:
        getClasses in interface Classifier<BytesRef>
        Parameters:
        text - a String containing text to be classified
        max - the number of return list elements
        Returns:
        the whole list of ClassificationResult, the classes and scores. Cut for "max" number of elements. Returns null if the classifier can't make lists.
        Throws:
        java.io.IOException - If there is a low-level I/O error.
      • assignClassNormalizedList

        protected java.util.List<ClassificationResult<BytesRef>> assignClassNormalizedList​(java.lang.String inputDocument)
                                                                                    throws java.io.IOException
        Calculate probabilities for all classes for a given input text
        Parameters:
        inputDocument - the input text as a String
        Returns:
        a List of ClassificationResult, one for each existing class
        Throws:
        java.io.IOException - if assigning probabilities fails
      • countDocsWithClass

        protected int countDocsWithClass()
                                  throws java.io.IOException
        count the number of documents in the index having at least a value for the 'class' field
        Returns:
        the no. of documents having a value for the 'class' field
        Throws:
        java.io.IOException - if accessing to term vectors or search fails
      • tokenize

        protected java.lang.String[] tokenize​(java.lang.String text)
                                       throws java.io.IOException
        tokenize a String on this classifier's text fields and analyzer
        Parameters:
        text - the String representing an input text (to be classified)
        Returns:
        a String array of the resulting tokens
        Throws:
        java.io.IOException - if tokenization fails
      • calculateLogLikelihood

        private double calculateLogLikelihood​(java.lang.String[] tokenizedText,
                                              Term term,
                                              int docsWithClass)
                                       throws java.io.IOException
        Throws:
        java.io.IOException
      • getTextTermFreqForClass

        private double getTextTermFreqForClass​(Term term)
                                        throws java.io.IOException
        Returns the average number of unique terms times the number of docs belonging to the input class
        Parameters:
        term - the term representing the class
        Returns:
        the average number of unique terms
        Throws:
        java.io.IOException - if a low level I/O problem happens
      • getWordFreqForClass

        private int getWordFreqForClass​(java.lang.String word,
                                        Term term)
                                 throws java.io.IOException
        Returns the number of documents of the input class ( from the whole index or from a subset) that contains the word ( in a specific field or in all the fields if no one selected)
        Parameters:
        word - the token produced by the analyzer
        term - the term representing the class
        Returns:
        the number of documents of the input class
        Throws:
        java.io.IOException - if a low level I/O problem happens
      • calculateLogPrior

        private double calculateLogPrior​(Term term,
                                         int docsWithClassSize)
                                  throws java.io.IOException
        Throws:
        java.io.IOException
      • docCount

        private int docCount​(Term term)
                      throws java.io.IOException
        Throws:
        java.io.IOException
      • normClassificationResults

        protected java.util.ArrayList<ClassificationResult<BytesRef>> normClassificationResults​(java.util.List<ClassificationResult<BytesRef>> assignedClasses)
        Normalize the classification results based on the max score available
        Parameters:
        assignedClasses - the list of assigned classes
        Returns:
        the normalized results