Class MasterTableDataSource
- java.lang.Object
-
- com.mckoi.database.MasterTableDataSource
-
- Direct Known Subclasses:
V1MasterTableDataSource
,V2MasterTableDataSource
abstract class MasterTableDataSource extends java.lang.Object
A master table data source provides facilities for read/writing and maintaining low level data in a table. It provides primitive table operations such as retrieving a cell from a table, accessing the table's DataTableDef, accessing indexes, and providing views of transactional versions of the data.Logically, a master table data source contains a dynamic number of rows and a fixed number of columns. Each row has an associated state - either DELETED, UNCOMMITTED, COMMITTED_ADDED or COMMITTED_REMOVED. A DELETED row is a row that can be reused by a new row added to the table.
When a new row is added to the table, it is marked as UNCOMMITTED. It is later tagged as COMMITTED_ADDED when the transaction that caused the row addition is committed. If a row commits a row removal, the row is tagged as COMMITTED_REMOVED and later the row garbage collector marks the row as DELETED when there are no remaining references to the row.
A master table also maintains a list of indexes for the table.
How the master table logical structure is translated to a form that is stored persistantly is implementation specific. This allows us flexibility with different types of storage schemes.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private class
MasterTableDataSource.MMutableTableDataSource
A MutableTableDataSource object as returned by the 'createTableDataSourceAtCommit' method.private class
MasterTableDataSource.MRawDiagnosticTable
A RawDiagnosticTable implementation that provides direct access to the root data of this table source bypassing any indexing schemes.
-
Field Summary
Fields Modifier and Type Field Description protected BlobStoreInterface
blob_store_interface
An abstracted reference to a BlobStore for managing blob references and blob data.protected DataCellCache
cache
A reference to the DataCellCache object.private TableName
cached_table_name
A cached TableName for this data source.protected int
column_count
The number of columns in this table.protected RIDList[]
column_rid_list
The list of RIDList objects for each column in this table.protected boolean
DATA_CELL_CACHING
Set to false to disable cell caching.protected java.lang.String
delete_hits_key
protected java.lang.String
file_hits_key
protected MasterTableGarbageCollector
garbage_collector
Manages scanning and deleting of rows marked as deleted within this data source.protected DataIndexSetDef
index_def
A DataIndexSetDef object that describes the indexes on the table.protected java.lang.String
insert_hits_key
protected boolean
is_closed
True if this table source is closed.private OpenTransactionList
open_transactions
The list of all open transactions managed by the parent conglomerate.private int
root_lock
The number of root locks this table data source has on it.protected java.lang.String
root_lock_key
The keys we use for Database.stats() for information for this table.private StoreSystem
store_system
The StoreSystem implementation that represents the data persistence layer.private TransactionSystem
system
The global TransactionSystem object that points to the global system that this table source belongs to.protected DataTableDef
table_def
A DataTableDef object that describes the table topology.protected int
table_id
An integer that uniquely identifies this data source within the conglomerate.protected MultiVersionTableIndices
table_indices
A multi-version representation of the table indices kept for this table including the row list and the scheme indices.protected java.lang.String
total_hits_key
-
Constructor Summary
Constructors Constructor Description MasterTableDataSource(TransactionSystem system, StoreSystem store_system, OpenTransactionList open_transactions, BlobStoreInterface blob_store_interface)
Constructs the MasterTableDataSource.
-
Method Summary
All Methods Static Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description (package private) void
addRootLock()
Grabs a root lock on this table.(package private) int
addRow(RowData data)
Adds a new row to this table and returns an index that is used to reference this row by the 'getCellContents' method.(package private) void
buildIndex(int index_number)
Builds the given index number (from the DataIndexSetDef).(package private) void
buildIndexes()
Builds a complete index set on the data in this table.(package private) TableName
cachedTableName()
Returns a cached TableName for this data source.(package private) abstract void
checkForCleanup()
Checks to determine if it is safe to clean up any resources in the table, and if it is safe to do so, the space is reclaimed.protected void
clearAllRootLocks()
Clears all root locks on the table.(package private) abstract void
commitIndexSet(IndexSet index_set)
Commits changes made to an IndexSet returned by the 'createIndexSet' method.(package private) void
commitTransactionChange(long commit_id, MasterTableJournal change, IndexSet index_set)
Adds a new transaction modification to this master table source.(package private) abstract IndexSet
createIndexSet()
Creates and returns an IndexSet object that is used to create indices for this table source.(package private) SelectableScheme
createSelectableSchemeForColumn(IndexSet index_set, TableDataSource table, int column)
Creates a SelectableScheme object for the given column in this table.(package private) SelectableScheme
createSelectableSchemeForIndex(IndexSet index_set, TableDataSource table, int index_i)
Creates a SelectableScheme object for the given index in the index set def in this table.(package private) MutableTableDataSource
createTableDataSourceAtCommit(SimpleTransaction transaction)
Returns a MutableTableDataSource object that represents this data source at the time the given transaction started.(package private) MutableTableDataSource
createTableDataSourceAtCommit(SimpleTransaction transaction, MasterTableJournal journal)
Returns a MutableTableDataSource object that represents this data source at the time the given transaction started, and also also makes any modifications that are described by the journal in the table.(package private) abstract long
currentUniqueID()
Atomically returns the current 'unique_id' value for this table.DebugLogger
Debug()
Returns the DebugLogger object that can be used to log debug messages.(package private) abstract void
dispose(boolean pending_drop)
Disposes of all in-memory resources associated with this table and invalidates this object.private void
doHardRowRemove(int row_index)
Actually deletes the row from the table.protected void
doOpeningScan()
This is called by the 'open' method.(package private) abstract boolean
drop()
Disposes and drops this table.(package private) MasterTableJournal[]
findAllJournalsSince(long commit_id)
Returns a list of all MasterTableJournal objects that have been successfully committed against this table that have an 'commit_id' that is greater or equal to the given.(package private) TObject
getCellContents(int column, int row)
Returns the cell contents of the given cell in the table.(package private) DataIndexSetDef
getDataIndexSetDef()
Returns the DataIndexSetDef object that represents the indexes on this table.(package private) DataTableDef
getDataTableDef()
Returns the DataTableDef object that represents the topology of this table data source (name, columns, etc).java.lang.String
getName()
Returns the name of this table source.(package private) RawDiagnosticTable
getRawDiagnosticTable()
Returns an implementation of RawDiagnosticTable that we can use to diagnose problems with the data in this source.java.lang.String
getSchema()
Returns the schema name of this table source.(package private) abstract java.lang.String
getSourceIdent()
Returns a string that uniquely identifies this table within the conglomerate context.TransactionSystem
getSystem()
Returns the TransactionSystem for this table.(package private) int
getTableID()
Returns table_id - the unique identifier for this data source.TableName
getTableName()
Returns the TableName of this table source.(package private) boolean
hardCheckAndReclaimRow(int record_index)
Checks the given record index, and if it's possible to reclaim it then it does so here.(package private) void
hardRemoveRow(int record_index)
Permanently removes a row from this table.(package private) boolean
hasTransactionChangesPending()
Returns true if this table has any journal modifications that have not yet been incorporated into master index.(package private) abstract int
internalAddRow(RowData data)
Adds a new row to this table and returns an index that is used to reference this row by the 'getCellContents' method.(package private) abstract void
internalDeleteRow(int row_index)
Removes the row at the given index so that any resources associated with the row may be immediately available to be recycled.(package private) abstract TObject
internalGetCellContents(int column, int row)
Returns the cell contents of the given cell in the table.(package private) boolean
isClosed()
Returns true if this table source is closed.(package private) boolean
isReadOnly()
Returns true if the source is read only.(package private) boolean
isRootLocked()
Returns true if the table is currently under a root lock (has 1 or more root locks on it).(package private) boolean
isWorthCompacting()
Returns true if a compact table is necessary.protected void
loadInternal()
Loads the internal variables.protected static java.lang.String
makeTableFileName(TransactionSystem system, int table_id, TableName table_name)
Creates a unique table name to give a file.(package private) void
mergeJournalChanges(long commit_id)
Updates the master records from the journal logs up to the given 'commit_id'.protected TableDataSource
minimalTableDataSource(IntegerListInterface master_index)
Creates a minimal TableDataSource object that represents this MasterTableDataSource.(package private) abstract long
nextUniqueID()
Atomically returns the next 'unique_id' value from this table.(package private) abstract int
rawRowCount()
Returns the raw count or rows in the table, including uncommited, committed and deleted rows.(package private) abstract int
readRecordType(int row_index)
Reads the record state for the given record in the table.(package private) abstract boolean
recordDeleted(int row_index)
Returns true if the record with the given index is deleted from the table.(package private) int
recordTypeInfo(int record_index)
Returns the record type of the given record index.(package private) void
removeRootLock()
Removes a root lock from this table.(package private) void
rollbackTransactionChange(MasterTableJournal change)
Rolls back a transaction change in this table source.(package private) abstract void
setUniqueID(long value)
Sets the unique id for this store.protected void
setupDataIndexSetDef()
Sets up the DataIndexSetDef object from the information set in this object.protected void
setupDataTableDef(DataTableDef table_def)
Sets up the DataTableDef.(package private) abstract void
shutdownHookCleanup()
Called by the 'shutdown hook' on the conglomerate.protected StoreSystem
storeSystem()
Returns the StoreSystem object used to manage stores in the persistence system.(package private) java.lang.String
transactionChangeString()
(package private) abstract int
writeRecordType(int row_index, int row_state)
Sets the record type for the given record in the table and returns the previous state of the record.
-
-
-
Field Detail
-
system
private TransactionSystem system
The global TransactionSystem object that points to the global system that this table source belongs to.
-
store_system
private StoreSystem store_system
The StoreSystem implementation that represents the data persistence layer.
-
table_id
protected int table_id
An integer that uniquely identifies this data source within the conglomerate.
-
is_closed
protected boolean is_closed
True if this table source is closed.
-
root_lock
private int root_lock
The number of root locks this table data source has on it.While a MasterTableDataSource has at least 1 root lock, it may not reclaim deleted space in the data store. A root lock means that data is still being pointed to in this file (even possibly committed deleted data).
-
table_def
protected DataTableDef table_def
A DataTableDef object that describes the table topology. This includes the name and columns of the table.
-
index_def
protected DataIndexSetDef index_def
A DataIndexSetDef object that describes the indexes on the table.
-
cached_table_name
private TableName cached_table_name
A cached TableName for this data source.
-
table_indices
protected MultiVersionTableIndices table_indices
A multi-version representation of the table indices kept for this table including the row list and the scheme indices. This contains the transaction journals.
-
column_rid_list
protected RIDList[] column_rid_list
The list of RIDList objects for each column in this table. This is a sorting optimization.
-
DATA_CELL_CACHING
protected boolean DATA_CELL_CACHING
Set to false to disable cell caching.
-
cache
protected final DataCellCache cache
A reference to the DataCellCache object.
-
column_count
protected int column_count
The number of columns in this table. This is a cached optimization.
-
open_transactions
private OpenTransactionList open_transactions
The list of all open transactions managed by the parent conglomerate. This is a thread safe object, and is updated whenever new transactions are created, or transactions are closed.
-
garbage_collector
protected MasterTableGarbageCollector garbage_collector
Manages scanning and deleting of rows marked as deleted within this data source.
-
blob_store_interface
protected BlobStoreInterface blob_store_interface
An abstracted reference to a BlobStore for managing blob references and blob data.
-
root_lock_key
protected java.lang.String root_lock_key
The keys we use for Database.stats() for information for this table.
-
total_hits_key
protected java.lang.String total_hits_key
-
file_hits_key
protected java.lang.String file_hits_key
-
delete_hits_key
protected java.lang.String delete_hits_key
-
insert_hits_key
protected java.lang.String insert_hits_key
-
-
Constructor Detail
-
MasterTableDataSource
MasterTableDataSource(TransactionSystem system, StoreSystem store_system, OpenTransactionList open_transactions, BlobStoreInterface blob_store_interface)
Constructs the MasterTableDataSource. The argument is a reference to an object that manages the list of open transactions in the conglomerate. This object uses this information to determine how journal entries are to be merged with the master indices.
-
-
Method Detail
-
getSystem
public final TransactionSystem getSystem()
Returns the TransactionSystem for this table.
-
Debug
public final DebugLogger Debug()
Returns the DebugLogger object that can be used to log debug messages.
-
getTableName
public TableName getTableName()
Returns the TableName of this table source.
-
getName
public java.lang.String getName()
Returns the name of this table source.
-
getSchema
public java.lang.String getSchema()
Returns the schema name of this table source.
-
cachedTableName
TableName cachedTableName()
Returns a cached TableName for this data source.
-
mergeJournalChanges
void mergeJournalChanges(long commit_id)
Updates the master records from the journal logs up to the given 'commit_id'. This could be a fairly expensive operation if there are a lot of modifications because each change could require a lookup of records in the data source.NOTE: It's extremely important that when this is called, there are no transaction open that are using the merged journal. If there is, then a transaction may be able to see changes in a table that were made after the transaction started.
After this method is called, it's best to update the index file with a call to 'synchronizeIndexFiles'
-
findAllJournalsSince
MasterTableJournal[] findAllJournalsSince(long commit_id)
Returns a list of all MasterTableJournal objects that have been successfully committed against this table that have an 'commit_id' that is greater or equal to the given.This is part of the conglomerate commit check phase and will be on a commit_lock.
-
getTableID
int getTableID()
Returns table_id - the unique identifier for this data source.
-
getDataTableDef
DataTableDef getDataTableDef()
Returns the DataTableDef object that represents the topology of this table data source (name, columns, etc). Note that this information can't be changed during the lifetime of a data source.
-
getDataIndexSetDef
DataIndexSetDef getDataIndexSetDef()
Returns the DataIndexSetDef object that represents the indexes on this table.
-
makeTableFileName
protected static java.lang.String makeTableFileName(TransactionSystem system, int table_id, TableName table_name)
Creates a unique table name to give a file. This could be changed to suit a particular OS's style of filesystem namespace. Or it could return some arbitarily unique number. However, for debugging purposes it's often a good idea to return a name that a user can recognise.The 'table_id' is a guarenteed unique number between all tables.
-
getSourceIdent
abstract java.lang.String getSourceIdent()
Returns a string that uniquely identifies this table within the conglomerate context. For example, the filename of the table. This string can be used to open and initialize the table also.
-
writeRecordType
abstract int writeRecordType(int row_index, int row_state) throws java.io.IOException
Sets the record type for the given record in the table and returns the previous state of the record. This is used to change the state of a row in the table.- Throws:
java.io.IOException
-
readRecordType
abstract int readRecordType(int row_index) throws java.io.IOException
Reads the record state for the given record in the table.- Throws:
java.io.IOException
-
recordDeleted
abstract boolean recordDeleted(int row_index) throws java.io.IOException
Returns true if the record with the given index is deleted from the table. A deleted row can not be read.- Throws:
java.io.IOException
-
rawRowCount
abstract int rawRowCount() throws java.io.IOException
Returns the raw count or rows in the table, including uncommited, committed and deleted rows. This is basically the maximum number of rows we can iterate through.- Throws:
java.io.IOException
-
internalDeleteRow
abstract void internalDeleteRow(int row_index) throws java.io.IOException
Removes the row at the given index so that any resources associated with the row may be immediately available to be recycled.- Throws:
java.io.IOException
-
createIndexSet
abstract IndexSet createIndexSet()
Creates and returns an IndexSet object that is used to create indices for this table source. The IndexSet represents a snapshot of the table and the given point in time.NOTE: Not synchronized because we synchronize in the IndexStore object.
-
commitIndexSet
abstract void commitIndexSet(IndexSet index_set)
Commits changes made to an IndexSet returned by the 'createIndexSet' method. This method also disposes the IndexSet so it is no longer valid.
-
internalAddRow
abstract int internalAddRow(RowData data) throws java.io.IOException
Adds a new row to this table and returns an index that is used to reference this row by the 'getCellContents' method.Note that this method will not effect the master index or column schemes. This is a low level mechanism for adding unreferenced data into a conglomerate. The data is referenced by committing the change where it eventually migrates into the master index and schemes.
- Throws:
java.io.IOException
-
internalGetCellContents
abstract TObject internalGetCellContents(int column, int row)
Returns the cell contents of the given cell in the table. It is the responsibility of the implemented method to perform caching as it deems fit. Some representations may not require such extensive caching as others.
-
currentUniqueID
abstract long currentUniqueID()
Atomically returns the current 'unique_id' value for this table.
-
nextUniqueID
abstract long nextUniqueID()
Atomically returns the next 'unique_id' value from this table.
-
setUniqueID
abstract void setUniqueID(long value)
Sets the unique id for this store. This must only be used under extraordinary circumstances, such as restoring from a backup, or converting from one file to another.
-
dispose
abstract void dispose(boolean pending_drop) throws java.io.IOException
Disposes of all in-memory resources associated with this table and invalidates this object. If 'pending_drop' is true then the table is to be disposed pending a call to 'drop'. If 'pending_drop' is true then any persistant resources that are allocated may be freed.- Throws:
java.io.IOException
-
drop
abstract boolean drop() throws java.io.IOException
Disposes and drops this table. If the dispose failed for any reason, it returns false, otherwise true. If the drop failed, it should be retried at a later time.- Throws:
java.io.IOException
-
shutdownHookCleanup
abstract void shutdownHookCleanup()
Called by the 'shutdown hook' on the conglomerate. This method should block until the table can by put into a safe mode and then prevent any further access to the object after it returns. It must operate very quickly.
-
isWorthCompacting
boolean isWorthCompacting()
Returns true if a compact table is necessary. By default, we return true however it is recommended this method is overwritten and the table tested.
-
createSelectableSchemeForColumn
SelectableScheme createSelectableSchemeForColumn(IndexSet index_set, TableDataSource table, int column)
Creates a SelectableScheme object for the given column in this table. This reads the index from the index set (if there is one) then wraps it around the selectable schema as appropriate.NOTE: This needs to be deprecated in support of composite indexes.
-
createSelectableSchemeForIndex
SelectableScheme createSelectableSchemeForIndex(IndexSet index_set, TableDataSource table, int index_i)
Creates a SelectableScheme object for the given index in the index set def in this table. This reads the index from the index set (if there is one) then wraps it around the selectable schema as appropriate.
-
minimalTableDataSource
protected TableDataSource minimalTableDataSource(IntegerListInterface master_index)
Creates a minimal TableDataSource object that represents this MasterTableDataSource. It does not implement the 'getColumnScheme' method.
-
buildIndexes
void buildIndexes() throws java.io.IOException
Builds a complete index set on the data in this table. This must only be called when either, a) we are under a commit lock, or b) there is a guarentee that no concurrect access to the indexing information can happen (such as when we are creating the table).NOTE: We assume that the index information for this table is blank before this method is called.
- Throws:
java.io.IOException
-
buildIndex
void buildIndex(int index_number) throws java.io.IOException
Builds the given index number (from the DataIndexSetDef). This must only be called when either, a) we are under a commit lock, or b) there is a guarentee that no concurrect access to the indexing information can happen (such as when we are creating the table).NOTE: We assume that the index number in this table is blank before this method is called.
- Throws:
java.io.IOException
-
commitTransactionChange
void commitTransactionChange(long commit_id, MasterTableJournal change, IndexSet index_set)
Adds a new transaction modification to this master table source. This information represents the information that was added/removed in the table in this transaction. The IndexSet object represents the changed index information to commit to this table.It's guarenteed that 'commit_id' additions will be sequential.
-
rollbackTransactionChange
void rollbackTransactionChange(MasterTableJournal change)
Rolls back a transaction change in this table source. Any rows added to the table will be uncommited rows (type_key = 0). Those rows must be marked as committed deleted.
-
createTableDataSourceAtCommit
MutableTableDataSource createTableDataSourceAtCommit(SimpleTransaction transaction)
Returns a MutableTableDataSource object that represents this data source at the time the given transaction started. Any modifications to the returned table are logged in the table journal.This is a key method in this object because it allows us to get a data source that represents the data in the table before any modifications may have been committed.
-
createTableDataSourceAtCommit
MutableTableDataSource createTableDataSourceAtCommit(SimpleTransaction transaction, MasterTableJournal journal)
Returns a MutableTableDataSource object that represents this data source at the time the given transaction started, and also also makes any modifications that are described by the journal in the table.This method is useful for merging the changes made by a transaction into a view of the table.
-
setupDataIndexSetDef
protected void setupDataIndexSetDef()
Sets up the DataIndexSetDef object from the information set in this object. This will only setup a default IndexSetDef on the information in the DataTableDef.
-
setupDataTableDef
protected void setupDataTableDef(DataTableDef table_def)
Sets up the DataTableDef. This would typically only ever be called from the 'create' method.
-
loadInternal
protected void loadInternal()
Loads the internal variables.
-
isClosed
boolean isClosed()
Returns true if this table source is closed.
-
isReadOnly
boolean isReadOnly()
Returns true if the source is read only.
-
storeSystem
protected StoreSystem storeSystem()
Returns the StoreSystem object used to manage stores in the persistence system.
-
addRow
int addRow(RowData data) throws java.io.IOException
Adds a new row to this table and returns an index that is used to reference this row by the 'getCellContents' method.Note that this method will not effect the master index or column schemes. This is a low level mechanism for adding unreferenced data into a conglomerate. The data is referenced by committing the change where it eventually migrates into the master index and schemes.
- Throws:
java.io.IOException
-
doHardRowRemove
private void doHardRowRemove(int row_index) throws java.io.IOException
Actually deletes the row from the table. This is a permanent removal of the row from the table. After this method is called, the row can not be retrieved again. This is generally only used by the row garbage collector.There is no checking in this method.
- Throws:
java.io.IOException
-
hardRemoveRow
void hardRemoveRow(int record_index) throws java.io.IOException
Permanently removes a row from this table. This must only be used when it is determined that a transaction does not reference this row, and that an open result set does not reference this row. This will remove the row permanently from the underlying file representation. Calls to 'getCellContents(col, row)' where row is deleted will be undefined after this method is called.Note that the removed row must not be contained within the master index, or be referenced by the index schemes, or be referenced in the transaction modification list.
- Throws:
java.io.IOException
-
hardCheckAndReclaimRow
boolean hardCheckAndReclaimRow(int record_index) throws java.io.IOException
Checks the given record index, and if it's possible to reclaim it then it does so here. Rows are only removed if they are marked as committed removed.- Throws:
java.io.IOException
-
recordTypeInfo
int recordTypeInfo(int record_index) throws java.io.IOException
Returns the record type of the given record index. Returns a type that is compatible with RawDiagnosticTable record type.- Throws:
java.io.IOException
-
doOpeningScan
protected void doOpeningScan() throws java.io.IOException
This is called by the 'open' method. It performs a scan of the records and marks any rows that are uncommitted as deleted. It also checks that the row is not within the master index.- Throws:
java.io.IOException
-
getRawDiagnosticTable
RawDiagnosticTable getRawDiagnosticTable()
Returns an implementation of RawDiagnosticTable that we can use to diagnose problems with the data in this source.
-
getCellContents
TObject getCellContents(int column, int row)
Returns the cell contents of the given cell in the table. This will look up the cell in the file if it can't be found in the cell cache. This method is undefined if row has been removed or was not returned by the 'addRow' method.
-
addRootLock
void addRootLock()
Grabs a root lock on this table.While a MasterTableDataSource has at least 1 root lock, it may not reclaim deleted space in the data store. A root lock means that data is still being pointed to in this file (even possibly committed deleted data).
-
removeRootLock
void removeRootLock()
Removes a root lock from this table.While a MasterTableDataSource has at least 1 root lock, it may not reclaim deleted space in the data store. A root lock means that data is still being pointed to in this file (even possibly committed deleted data).
-
isRootLocked
boolean isRootLocked()
Returns true if the table is currently under a root lock (has 1 or more root locks on it).
-
clearAllRootLocks
protected void clearAllRootLocks()
Clears all root locks on the table. Should only be used during cleanup of the table and will by definition invalidate the table.
-
checkForCleanup
abstract void checkForCleanup()
Checks to determine if it is safe to clean up any resources in the table, and if it is safe to do so, the space is reclaimed.
-
transactionChangeString
java.lang.String transactionChangeString()
-
hasTransactionChangesPending
boolean hasTransactionChangesPending()
Returns true if this table has any journal modifications that have not yet been incorporated into master index.
-
-