Class LimitTokenCountAnalyzer

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable

    public final class LimitTokenCountAnalyzer
    extends AnalyzerWrapper
    This Analyzer limits the number of tokens while indexing. It is a replacement for the maximum field length setting inside IndexWriter.
    Since:
    3.1
    See Also:
    LimitTokenCountFilter
    • Field Detail

      • delegate

        private final Analyzer delegate
      • maxTokenCount

        private final int maxTokenCount
      • consumeAllTokens

        private final boolean consumeAllTokens
    • Constructor Detail

      • LimitTokenCountAnalyzer

        public LimitTokenCountAnalyzer​(Analyzer delegate,
                                       int maxTokenCount)
        Build an analyzer that limits the maximum number of tokens per field. This analyzer will not consume any tokens beyond the maxTokenCount limit
        See Also:
        LimitTokenCountAnalyzer(Analyzer,int,boolean)
      • LimitTokenCountAnalyzer

        public LimitTokenCountAnalyzer​(Analyzer delegate,
                                       int maxTokenCount,
                                       boolean consumeAllTokens)
        Build an analyzer that limits the maximum number of tokens per field.
        Parameters:
        delegate - the analyzer to wrap
        maxTokenCount - max number of tokens to produce
        consumeAllTokens - whether all tokens from the delegate should be consumed even if maxTokenCount is reached.
    • Method Detail

      • getWrappedAnalyzer

        protected Analyzer getWrappedAnalyzer​(java.lang.String fieldName)
        Description copied from class: AnalyzerWrapper
        Retrieves the wrapped Analyzer appropriate for analyzing the field with the given name
        Specified by:
        getWrappedAnalyzer in class AnalyzerWrapper
        Parameters:
        fieldName - Name of the field which is to be analyzed
        Returns:
        Analyzer for the field with the given name. Assumed to be non-null
      • wrapComponents

        protected Analyzer.TokenStreamComponents wrapComponents​(java.lang.String fieldName,
                                                                Analyzer.TokenStreamComponents components)
        Description copied from class: AnalyzerWrapper
        Wraps / alters the given TokenStreamComponents, taken from the wrapped Analyzer, to form new components. It is through this method that new TokenFilters can be added by AnalyzerWrappers. By default, the given components are returned.
        Overrides:
        wrapComponents in class AnalyzerWrapper
        Parameters:
        fieldName - Name of the field which is to be analyzed
        components - TokenStreamComponents taken from the wrapped Analyzer
        Returns:
        Wrapped / altered TokenStreamComponents.
      • toString

        public java.lang.String toString()
        Overrides:
        toString in class java.lang.Object