Class IndexStore


  • public final class IndexStore
    extends java.lang.Object
    A class that manages the storage of a set of transactional index lists in a way that is fast to modify. This class has a number of objectives;

    • To prevent corruption as best as possible.
    • To be able to modify lists of integers very fast and persistantly.
    • To have size optimization features such as defragmentation.
    • To provide very fast searches on sorted lists (caching features).
    • To be able to map a list to an IntegerListInterface interface.

    None intuitively, this object also handles unique ids.

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      private class  IndexStore.IndexIntegerList
      The IntegerListInterface implementation that is used to represent a mutable snapshop of the indices at a given point in time.
      private class  IndexStore.MappedListBlock
      An IntegerListBlockInterface implementation that maps a block of a list to an underlying file system representation.
      private class  IndexStore.SnapshotIndexSet
      The implementation of IndexSet which represents a mutable snapshot of the indices stored in this set.
    • Field Summary

      Fields 
      Modifier and Type Field Description
      private int allocation_length
      The current of the allocation information.
      private int allocation_sector
      The start sector where the block allocation information is currently stored.
      private int block_size
      The size of a 'block' element of index information in a list.
      private DebugLogger debug
      A DebugLogger object used to log debug messages to.
      private static IndexStore.IndexIntegerList[] EMPTY_INTEGER_LISTS
      A convenience static empty integer list array.
      private java.io.File file
      The name of the index store file.
      private byte[] flush_buffer  
      private java.util.ArrayList index_set_garbage
      The list of SnapshotIndexSet objects that have been deleted and are ready for garbage collection.
      private FixedSizeDataStore index_store
      The FixedSizeDataStore that contains all the data of the index store.
      private ByteBuffer index_table_list
      The list of table sector entries that are currently committed.
      private byte[] index_table_list_buf  
      private java.util.ArrayList memory_index_set_list
      The list of SnapshotIndexSet objects returned via the 'getSnapshotIndexSet' method.
      private Cache sector_cache
      A cache of int[] array blocks that are accessed by this store.
      private long SET_ID_KEY
      A unique key that is incremented each time a new IndexSet object is created.
      private long unique_id
      Unique id field that contains a unique number that can be incremented atomically.
    • Constructor Summary

      Constructors 
      Constructor Description
      IndexStore​(java.io.File file_name, DebugLogger logger)
      Constructs the IndexStore.
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void addIndexLists​(int count, byte type)
      Adds a number of blank index tables to the index store.
      private void addIndexSetToList​(IndexSet index_set)
      Adds a SnapshotIndexSet to the list of sets that this store has dispatched.
      void close()
      Cleanly closes the index store.
      void commitIndexSet​(IndexSet index_set)
      Commits changes made to a snapshop of an IndexSet as being permanent changes to the state of the index store.
      void copyTo​(java.io.File path)
      Copies the persistant part of this to another store.
      void create​(int block_size)
      Creates a new black index store and returns leaving the newly created store in an open state.
      (package private) long currentUniqueID()
      The current unique id.
      void delete()
      Deletes the store.
      boolean exists()
      Returns true if the index store file exists.
      boolean fix​(UserTerminal terminal)
      Performs checks to determine that the index store is stable.
      void flush()
      Flushes all information in this index store to the file representing this store in the file system.
      (package private) int getBlockSize()
      Returns the block size of this store.
      IndexSet getSnapshotIndexSet()
      Returns a current snapshot of the current indexes that are committed in this store.
      void hardSynch()
      Performs a hard synchronization of this index store.
      void init()
      Initializes the IndexStore.
      private void initBlank()
      Initializes the index store to a blank state.
      boolean isReadOnly()
      Returns true if this store is read only.
      (package private) long nextUniqueID()
      Atomically returns the next 'unique_id' value from this file.
      boolean open​(boolean read_only)
      Opens this index store.
      private void readIndexTableList()
      Reads the index table allocation list in to the ByteBuffer object.
      private void removeIndexSetFromList​(IndexSet index_set)
      Removes a SnapshotIndexSet from the list of sets that this store is managing.
      (package private) void setUniqueID​(long value)
      Sets the unique id for this store.
      java.lang.String statusString()
      Returns a string that contains diagnostic information.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Field Detail

      • debug

        private DebugLogger debug
        A DebugLogger object used to log debug messages to.
      • file

        private java.io.File file
        The name of the index store file.
      • block_size

        private int block_size
        The size of a 'block' element of index information in a list. This has a direct relation to the size of the sectors in the store. This value can be tuned for specific tables. For example, a table that will only ever contain a few items can save disk space by having a smaller block size.
      • index_store

        private FixedSizeDataStore index_store
        The FixedSizeDataStore that contains all the data of the index store.
      • index_table_list

        private ByteBuffer index_table_list
        The list of table sector entries that are currently committed. Each entry of this list points to a table index list. The list is formatted as follows;

           0 (byte)         - the type of block.
           1 (int)          - the number of blocks in this list.
           5 (int)          - the sector of column status information or -1 if
                              no stats available.
           9 to (n * (4 + 4 + 4 + 2))
                            - the sector (int), the first and last entry of the
                              block and the number of indices in the block
                              (short) of each block in this list.
           9 + (n * (4 + 4 + 4 + 2)) .... [ next block ] ....
         
      • index_table_list_buf

        private byte[] index_table_list_buf
      • allocation_sector

        private int allocation_sector
        The start sector where the block allocation information is currently stored.
      • allocation_length

        private int allocation_length
        The current of the allocation information.
      • memory_index_set_list

        private java.util.ArrayList memory_index_set_list
        The list of SnapshotIndexSet objects returned via the 'getSnapshotIndexSet' method. This can be inspected to find all sectors currently being used to store index information.
      • index_set_garbage

        private java.util.ArrayList index_set_garbage
        The list of SnapshotIndexSet objects that have been deleted and are ready for garbage collection.
      • unique_id

        private long unique_id
        Unique id field that contains a unique number that can be incremented atomically.
      • sector_cache

        private Cache sector_cache
        A cache of int[] array blocks that are accessed by this store.
      • flush_buffer

        private byte[] flush_buffer
      • SET_ID_KEY

        private long SET_ID_KEY
        A unique key that is incremented each time a new IndexSet object is created.
    • Constructor Detail

      • IndexStore

        public IndexStore​(java.io.File file_name,
                          DebugLogger logger)
        Constructs the IndexStore.
        Parameters:
        file_name - the path to the file of the index store in the file system.
    • Method Detail

      • readIndexTableList

        private void readIndexTableList()
                                 throws java.io.IOException
        Reads the index table allocation list in to the ByteBuffer object. The position of the table allocation list can be determined by looking in the reserved area of the index file.
        Throws:
        java.io.IOException
      • initBlank

        private void initBlank()
                        throws java.io.IOException
        Initializes the index store to a blank state.
        Throws:
        java.io.IOException
      • exists

        public boolean exists()
                       throws java.io.IOException
        Returns true if the index store file exists.
        Throws:
        java.io.IOException
      • create

        public void create​(int block_size)
                    throws java.io.IOException
        Creates a new black index store and returns leaving the newly created store in an open state. This method initializes the various data in the index store for a blank set of index tables. Must call the 'init' method after this is called.

        Parameters:
        block_size - the number of ints stored in each block. This can be optimized for specific use. Must be between 0 and 32768.
        Throws:
        java.io.IOException
      • open

        public boolean open​(boolean read_only)
                     throws java.io.IOException
        Opens this index store. If 'read_only' is set to true then the store is opened in read only mode.

        Returns true if opening the store was dirty (was not closed properly) and may need repair.

        If the index store does not exist before this method is called then it is created.

        Throws:
        java.io.IOException
      • init

        public void init()
                  throws java.io.IOException
        Initializes the IndexStore. Must be called after it is opened for normal use, however it should not be called if we are fixing or repairing the store.
        Throws:
        java.io.IOException
      • fix

        public boolean fix​(UserTerminal terminal)
                    throws java.io.IOException
        Performs checks to determine that the index store is stable. If an IndexStore is not stable and can not be fixed cleanly then it deletes all information in the store and returns false indicating the index information must be rebuilt.

        Assumes the index store has been opened previous to calling this.

        Returns true if the IndexStore is stable.

        Throws:
        java.io.IOException
      • isReadOnly

        public boolean isReadOnly()
        Returns true if this store is read only.
      • delete

        public void delete()
        Deletes the store. Must have been closed before this is called.
      • copyTo

        public void copyTo​(java.io.File path)
                    throws java.io.IOException
        Copies the persistant part of this to another store. Must be open when this is called.
        Throws:
        java.io.IOException
      • close

        public void close()
                   throws java.io.IOException
        Cleanly closes the index store.
        Throws:
        java.io.IOException
      • flush

        public void flush()
                   throws java.io.IOException
        Flushes all information in this index store to the file representing this store in the file system. This is called to persistantly update the state of the index store.
        Throws:
        java.io.IOException
      • hardSynch

        public void hardSynch()
                       throws java.io.IOException
        Performs a hard synchronization of this index store. This will force the OS to synchronize the contents of the data store.

        For this to be useful, 'flush' should be called before a hardSynch.

        Throws:
        java.io.IOException
      • currentUniqueID

        long currentUniqueID()
        The current unique id.
      • nextUniqueID

        long nextUniqueID()
        Atomically returns the next 'unique_id' value from this file.
      • setUniqueID

        void setUniqueID​(long value)
        Sets the unique id for this store. This must only be used under extraordinary circumstances, such as restoring from a backup, or converting from one file to another.
      • getBlockSize

        int getBlockSize()
        Returns the block size of this store.
      • addIndexLists

        public void addIndexLists​(int count,
                                  byte type)
        Adds a number of blank index tables to the index store. For example, we may want this store to contain 16 index lists.

        NOTE: This doesn't write the updated information to the file. You must call 'flush' to write the information to the store.

      • addIndexSetToList

        private void addIndexSetToList​(IndexSet index_set)
        Adds a SnapshotIndexSet to the list of sets that this store has dispatched.
      • removeIndexSetFromList

        private void removeIndexSetFromList​(IndexSet index_set)
        Removes a SnapshotIndexSet from the list of sets that this store is managing.

        NOTE: This may be called by the finalizer of the IndexSet object if the index_set is not disposed.

      • getSnapshotIndexSet

        public IndexSet getSnapshotIndexSet()
        Returns a current snapshot of the current indexes that are committed in this store. The returned object can be used to create mutable IntegerListInterface objects. The created index lists are isolated from changes made to the rest of the indexes after this method returns.

        A transaction must grab an IndexSet object when it opens.

        NOTE: We MUST guarentee that the IndexSet is disposed when the transaction finishes.

      • commitIndexSet

        public void commitIndexSet​(IndexSet index_set)
        Commits changes made to a snapshop of an IndexSet as being permanent changes to the state of the index store. This will generate an error if the given IndexSet is not the last set returned from the 'getSnapshotIndexSet' method.

        For this to be used, during the transaction commit function a 'getSnapshopIndexSet' must be obtained, changes made to it from info in the journal, then a call to this method. There must be a guarentee that 'getSnapshotIndexSet' is not called again during this process.

        NOTE: This doesn't write the updated information to the file. You must call 'flush' to write the information to the store.

        NOTE: We must be guarenteed that when this method is called no other calls to other methods in this object can be called.

      • statusString

        public java.lang.String statusString()
                                      throws java.io.IOException
        Returns a string that contains diagnostic information.
        Throws:
        java.io.IOException