Package com.mckoi.store
Class AbstractStore
- java.lang.Object
-
- com.mckoi.store.AbstractStore
-
- All Implemented Interfaces:
Store
- Direct Known Subclasses:
JournalledFileStore
public abstract class AbstractStore extends java.lang.Object implements Store
Provides an abstract implementation of Store. This implements a bin based best-fit recycling algorithm. The store manages a structure that points to bins of freed space of specific sizes. When an allocation is requested the structure is searched for the first bin that contains an area that best fits the size requested.Provided the derived class supports safe atomic IO operations, this store is designed for robustness to the level that at no point is the store left in a unworkable (corrupt) state.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description (package private) static class
AbstractStore.AreaOutputStream
A simple OutputStream implementation that is on top of an AreaWriter object.private class
AbstractStore.StoreArea
private class
AbstractStore.StoreAreaInputStream
private class
AbstractStore.StoreAreaWriter
private class
AbstractStore.StoreMutableArea
-
Field Summary
Fields Modifier and Type Field Description private byte[]
bin_area
A buffered work area we work with when reading/writing bin pointers from the file header.protected static long
BIN_AREA_OFFSET
The offset into the file that the bin area starts.protected static int
BIN_ENTRIES
private static int[]
BIN_SIZES
The default bin sizes in bytes.private byte[]
buf
Reads an 8 byte long at the given position in the data area.protected static long
DATA_AREA_OFFSET
The offset into the file that the data areas start.private boolean
dirty_open
True if the store was opened dirtily (was not previously closed cleanly).protected static long
FIXED_AREA_OFFSET
The offset into the file of the 64 byte fixed area.protected long[]
free_bin_list
The free bin list contains 128 entries pointing to the first available block in the bin.protected byte[]
header_buf
private long[]
header_info
private long[]
header_info2
protected static int
MAGIC
The magic value.private static int
MAX_BIN_SIZE
protected boolean
read_only
True if this is read-only.protected long
total_allocated_space
The total amount of allocated space within this store since the store was openned.protected long
wilderness_pointer
A pointer to the wilderness area (the last deleted area in the store), or -1 if there is no wilderness area.
-
Constructor Summary
Constructors Modifier Constructor Description protected
AbstractStore(boolean read_only)
Constructs the store.
-
Method Summary
All Methods Static Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description private void
addToBinChain(long pointer, long size)
Adds the given area to the bin represented by the bin_chain_index.private long
alloc(long size)
Allocates a block of memory from the backing area of the given size and returns a pointer to that area.protected void
checkPointer(long pointer)
Checks the pointer is valid.void
close()
Closes the store.protected void
coalescArea(long pointer, long size)
Coalesc one or more areas into a larger area.AreaWriter
createArea(long size)
Allocates a block of memory in the store of the specified size and returns an AreaWriter object that can be used to initialize the contents of the area.private void
cropArea(long pointer, long allocated_size)
Crops the area to the given size.void
deleteArea(long id)
Deletes an area that was previously allocated by the 'createArea' method by the area id.protected abstract long
endOfDataAreaPointer()
Returns a pointer to the end of the current data area.protected long
expandDataArea(long minimum_size)
Expands the data area by at least the minimum size given.java.util.ArrayList
findAllocatedAreasNotIn(java.util.ArrayList list)
Scans the area list, and any areas that aren't deleted and aren't found in the given ArrayList are returned as leaked areas.private void
free(long pointer)
Frees a previously allocated area in the store.java.util.List
getAllAreas()
Returns a List of Long objects that contain a complete list of all areas in the store.Area
getArea(long id)
Returns an object that allows for the contents of an area (represented by the 'id' parameter) to be read.protected void
getAreaHeader(long pointer, long[] header)
Sets the 'header' array with information from the header of the given pointer.java.io.InputStream
getAreaInputStream(long id)
Returns an InputStream implementation that allows for the area with the given identifier to be read sequentially.private long
getAreaSize(long pointer)
Convenience for finding the size of an area.MutableArea
getMutableArea(long id)
Returns an object that allows for the contents of an area (represented by the 'id' parameter) to be read and written.protected long
getNextAreaHeader(long pointer, long[] header)
Sets the 'header' array with information from the next header to the given pointer, and returns a pointer to the next area.protected long
getPreviousAreaHeader(long pointer, long[] header)
Sets the 'header' array with information from the previous header to the given pointer, and returns a pointer to the previous area.private void
initializeToEmpty()
Initializes the store to an empty state.protected abstract void
internalClose()
Internally closes the backing area.protected abstract void
internalOpen(boolean read_only)
Internally opens the backing area.protected static boolean
isValidBoundarySize(long size)
Returns true if the given area size is valid.boolean
lastCloseClean()
Returns true if the store was closed cleanly.private int
minimumBinSizeIndex(long size)
Returns the bin index that would be the minimum size to store the given object.boolean
open()
Opens the data store.void
openScanAndFix(UserTerminal terminal)
Opens/scans the store looking for any errors with the layout.protected void
readBins()
Reads the bins from the header information in the file.protected abstract int
readByteArrayFrom(long position, byte[] buf, int off, int len)
Reads a byte array from the given position in the file.protected abstract int
readByteFrom(long position)
Reads a byte from the given position in the file.private long
readLongAt(long position)
protected void
reboundArea(long pointer, long[] header, boolean write_headers)
Rebounds the given area with the given header information.private void
removeFromBinChain(long pointer, long size)
Removes the given area from the bin chain.private boolean
repairScan(java.util.ArrayList areas_to_fix, long pointer, long end_pointer, boolean scan_forward, int max_repairs)
Performs a repair scan from the given pointer.protected abstract void
setDataAreaSize(long length)
Sets the size of the data area.protected void
splitArea(long pointer, long new_boundary)
Splits an area pointed to by 'pointer' at a new boundary point.void
statsScan(java.util.HashMap properties)
Performs an extensive lookup on all the tables in this store and sets a number of properties in the given HashMap (property name(String) -> property description(Object)).long
totalAllocatedSinceStart()
Returns the total allocated space since the file was openned.protected void
writeAllBins()
Updates all bins to the data area header area.protected void
writeBinIndex(int index)
Updates the given bin index to the data area header area.protected abstract void
writeByteArrayTo(long position, byte[] buf, int off, int len)
Writes a byte array to the given position in the file.private void
writeByteArrayToPT(long position, byte[] buf, int off, int len)
WriteByteArrayTo pass-through method.protected abstract void
writeByteTo(long position, int b)
Writes a byte to the given position in the file.private void
writeByteToPT(long position, int b)
WriteByteTo pass-through method.-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface com.mckoi.store.Store
lockForWrite, unlockForWrite
-
-
-
-
Field Detail
-
free_bin_list
protected long[] free_bin_list
The free bin list contains 128 entries pointing to the first available block in the bin. If the list item contains -1 then there are no free blocks in the bin.
-
wilderness_pointer
protected long wilderness_pointer
A pointer to the wilderness area (the last deleted area in the store), or -1 if there is no wilderness area.
-
read_only
protected boolean read_only
True if this is read-only.
-
total_allocated_space
protected long total_allocated_space
The total amount of allocated space within this store since the store was openned. Note that this could be a negative amount if more space was freed than allocated.
-
dirty_open
private boolean dirty_open
True if the store was opened dirtily (was not previously closed cleanly).
-
DATA_AREA_OFFSET
protected static final long DATA_AREA_OFFSET
The offset into the file that the data areas start.- See Also:
- Constant Field Values
-
FIXED_AREA_OFFSET
protected static final long FIXED_AREA_OFFSET
The offset into the file of the 64 byte fixed area.- See Also:
- Constant Field Values
-
BIN_AREA_OFFSET
protected static final long BIN_AREA_OFFSET
The offset into the file that the bin area starts.- See Also:
- Constant Field Values
-
MAGIC
protected static final int MAGIC
The magic value.- See Also:
- Constant Field Values
-
buf
private byte[] buf
Reads an 8 byte long at the given position in the data area.
-
bin_area
private final byte[] bin_area
A buffered work area we work with when reading/writing bin pointers from the file header.
-
header_buf
protected final byte[] header_buf
-
header_info
private long[] header_info
-
header_info2
private long[] header_info2
-
BIN_SIZES
private static final int[] BIN_SIZES
The default bin sizes in bytes. The minimum size of a bin is 32 and the maximum size is 2252832.
-
BIN_ENTRIES
protected static final int BIN_ENTRIES
-
MAX_BIN_SIZE
private static final int MAX_BIN_SIZE
-
-
Method Detail
-
initializeToEmpty
private void initializeToEmpty() throws java.io.IOException
Initializes the store to an empty state.- Throws:
java.io.IOException
-
open
public boolean open() throws java.io.IOException
Opens the data store. Returns true if the store did not close cleanly.- Throws:
java.io.IOException
-
close
public void close() throws java.io.IOException
Closes the store.- Throws:
java.io.IOException
-
isValidBoundarySize
protected static boolean isValidBoundarySize(long size)
Returns true if the given area size is valid. Currently the criteria for a valid boundary size is (size >= 24) and (size % 8 == 0) and (size < 200 gigabytes)
-
readLongAt
private long readLongAt(long position) throws java.io.IOException
- Throws:
java.io.IOException
-
repairScan
private boolean repairScan(java.util.ArrayList areas_to_fix, long pointer, long end_pointer, boolean scan_forward, int max_repairs) throws java.io.IOException
Performs a repair scan from the given pointer. This is a recursive algorithm that looks for at most 'n' number of repairs before giving up. Returns false if a repair path could not be found.- Throws:
java.io.IOException
-
openScanAndFix
public void openScanAndFix(UserTerminal terminal) throws java.io.IOException
Opens/scans the store looking for any errors with the layout. If a problem with the store is detected, it attempts to fix it.- Throws:
java.io.IOException
-
statsScan
public void statsScan(java.util.HashMap properties) throws java.io.IOException
Performs an extensive lookup on all the tables in this store and sets a number of properties in the given HashMap (property name(String) -> property description(Object)). This should be used for store diagnostics.Assume the store is open.
- Throws:
java.io.IOException
-
getAllAreas
public java.util.List getAllAreas() throws java.io.IOException
Returns a List of Long objects that contain a complete list of all areas in the store. This is useful for checking if a given pointer is valid or not. The returned list is sorted from start area to end area.- Specified by:
getAllAreas
in interfaceStore
- Throws:
java.io.IOException
-
findAllocatedAreasNotIn
public java.util.ArrayList findAllocatedAreasNotIn(java.util.ArrayList list) throws java.io.IOException
Scans the area list, and any areas that aren't deleted and aren't found in the given ArrayList are returned as leaked areas. This is a useful method for finding any leaks in the store.- Throws:
java.io.IOException
-
totalAllocatedSinceStart
public long totalAllocatedSinceStart()
Returns the total allocated space since the file was openned.
-
minimumBinSizeIndex
private int minimumBinSizeIndex(long size)
Returns the bin index that would be the minimum size to store the given object.
-
internalOpen
protected abstract void internalOpen(boolean read_only) throws java.io.IOException
Internally opens the backing area. If 'read_only' is true then the store is openned in read only mode.- Throws:
java.io.IOException
-
internalClose
protected abstract void internalClose() throws java.io.IOException
Internally closes the backing area.- Throws:
java.io.IOException
-
readByteFrom
protected abstract int readByteFrom(long position) throws java.io.IOException
Reads a byte from the given position in the file.- Throws:
java.io.IOException
-
readByteArrayFrom
protected abstract int readByteArrayFrom(long position, byte[] buf, int off, int len) throws java.io.IOException
Reads a byte array from the given position in the file. Returns the number of bytes read.- Throws:
java.io.IOException
-
writeByteTo
protected abstract void writeByteTo(long position, int b) throws java.io.IOException
Writes a byte to the given position in the file.- Throws:
java.io.IOException
-
writeByteArrayTo
protected abstract void writeByteArrayTo(long position, byte[] buf, int off, int len) throws java.io.IOException
Writes a byte array to the given position in the file.- Throws:
java.io.IOException
-
endOfDataAreaPointer
protected abstract long endOfDataAreaPointer() throws java.io.IOException
Returns a pointer to the end of the current data area.- Throws:
java.io.IOException
-
setDataAreaSize
protected abstract void setDataAreaSize(long length) throws java.io.IOException
Sets the size of the data area.- Throws:
java.io.IOException
-
writeByteToPT
private final void writeByteToPT(long position, int b) throws java.io.IOException
WriteByteTo pass-through method.- Throws:
java.io.IOException
-
writeByteArrayToPT
private final void writeByteArrayToPT(long position, byte[] buf, int off, int len) throws java.io.IOException
WriteByteArrayTo pass-through method.- Throws:
java.io.IOException
-
checkPointer
protected void checkPointer(long pointer) throws java.io.IOException
Checks the pointer is valid.- Throws:
java.io.IOException
-
readBins
protected void readBins() throws java.io.IOException
Reads the bins from the header information in the file.- Throws:
java.io.IOException
-
writeAllBins
protected void writeAllBins() throws java.io.IOException
Updates all bins to the data area header area.- Throws:
java.io.IOException
-
writeBinIndex
protected void writeBinIndex(int index) throws java.io.IOException
Updates the given bin index to the data area header area.- Throws:
java.io.IOException
-
getAreaHeader
protected void getAreaHeader(long pointer, long[] header) throws java.io.IOException
Sets the 'header' array with information from the header of the given pointer.- Throws:
java.io.IOException
-
getPreviousAreaHeader
protected long getPreviousAreaHeader(long pointer, long[] header) throws java.io.IOException
Sets the 'header' array with information from the previous header to the given pointer, and returns a pointer to the previous area.- Throws:
java.io.IOException
-
getNextAreaHeader
protected long getNextAreaHeader(long pointer, long[] header) throws java.io.IOException
Sets the 'header' array with information from the next header to the given pointer, and returns a pointer to the next area.- Throws:
java.io.IOException
-
reboundArea
protected void reboundArea(long pointer, long[] header, boolean write_headers) throws java.io.IOException
Rebounds the given area with the given header information. If 'write_headers' is true, the header (header[0]) is changed. Note that this shouldn't be used to change the size of a chunk.- Throws:
java.io.IOException
-
coalescArea
protected void coalescArea(long pointer, long size) throws java.io.IOException
Coalesc one or more areas into a larger area. This alters the boundary of the area to encompass the given size.- Throws:
java.io.IOException
-
expandDataArea
protected long expandDataArea(long minimum_size) throws java.io.IOException
Expands the data area by at least the minimum size given. Returns the actual size the data area was expanded by.- Throws:
java.io.IOException
-
splitArea
protected void splitArea(long pointer, long new_boundary) throws java.io.IOException
Splits an area pointed to by 'pointer' at a new boundary point.- Throws:
java.io.IOException
-
addToBinChain
private void addToBinChain(long pointer, long size) throws java.io.IOException
Adds the given area to the bin represented by the bin_chain_index.- Throws:
java.io.IOException
-
removeFromBinChain
private void removeFromBinChain(long pointer, long size) throws java.io.IOException
Removes the given area from the bin chain. This requires a search of the bin chain for the given size.- Throws:
java.io.IOException
-
cropArea
private void cropArea(long pointer, long allocated_size) throws java.io.IOException
Crops the area to the given size. This is used after an area is pulled from a bin. This method decides if it's worth reusing any space left over and the end of the area.- Throws:
java.io.IOException
-
alloc
private long alloc(long size) throws java.io.IOException
Allocates a block of memory from the backing area of the given size and returns a pointer to that area.- Throws:
java.io.IOException
-
free
private void free(long pointer) throws java.io.IOException
Frees a previously allocated area in the store.- Throws:
java.io.IOException
-
getAreaSize
private long getAreaSize(long pointer) throws java.io.IOException
Convenience for finding the size of an area. If the area is deleted throws an exception.- Throws:
java.io.IOException
-
createArea
public AreaWriter createArea(long size) throws java.io.IOException
Description copied from interface:Store
Allocates a block of memory in the store of the specified size and returns an AreaWriter object that can be used to initialize the contents of the area. Note that an area in the store is undefined until the 'finish' method is called in AreaWriter.- Specified by:
createArea
in interfaceStore
- Parameters:
size
- the amount of memory to allocate.- Returns:
- an AreaWriter object that allows the area to be setup.
- Throws:
java.io.IOException
- if not enough space available to create the area or the store is read-only.
-
deleteArea
public void deleteArea(long id) throws java.io.IOException
Description copied from interface:Store
Deletes an area that was previously allocated by the 'createArea' method by the area id. Once an area is deleted the resources may be reclaimed. The behaviour of this method is undefined if the id doesn't represent a valid area.- Specified by:
deleteArea
in interfaceStore
- Parameters:
id
- the identifier of the area to delete.- Throws:
java.io.IOException
- (optional) if the id is invalid or the area can not otherwise by deleted.
-
getAreaInputStream
public java.io.InputStream getAreaInputStream(long id) throws java.io.IOException
Description copied from interface:Store
Returns an InputStream implementation that allows for the area with the given identifier to be read sequentially. The behaviour of this method, and InputStream object, is undefined if the id doesn't represent a valid area.When 'id' is -1 then a fixed area (64 bytes in size) in the store is returned. The fixed area can be used to store important static static information.
- Specified by:
getAreaInputStream
in interfaceStore
- Parameters:
id
- the identifier of the area to read, or id = -1 is a 64 byte fixed area in the store.- Returns:
- an InputStream that allows the area to be read from the start.
- Throws:
java.io.IOException
- (optional) if the id is invalid or the area can not otherwise be accessed.
-
getArea
public Area getArea(long id) throws java.io.IOException
Description copied from interface:Store
Returns an object that allows for the contents of an area (represented by the 'id' parameter) to be read. The behaviour of this method, and Area object, is undefined if the id doesn't represent a valid area.When 'id' is -1 then a fixed area (64 bytes in size) in the store is returned. The fixed area can be used to store important static static information.
- Specified by:
getArea
in interfaceStore
- Parameters:
id
- the identifier of the area to read, or id = -1 is a 64 byte fixed area in the store.- Returns:
- an Area object that allows access to the part of the store.
- Throws:
java.io.IOException
- (optional) if the id is invalid or the area can not otherwise be accessed.
-
getMutableArea
public MutableArea getMutableArea(long id) throws java.io.IOException
Description copied from interface:Store
Returns an object that allows for the contents of an area (represented by the 'id' parameter) to be read and written. The behaviour of this method, and MutableArea object, is undefined if the id doesn't represent a valid area.When 'id' is -1 then a fixed area (64 bytes in size) in the store is returned. The fixed area can be used to store important static static information.
- Specified by:
getMutableArea
in interfaceStore
- Parameters:
id
- the identifier of the area to access, or id = -1 is a 64 byte fixed area in the store.- Returns:
- a MutableArea object that allows access to the part of the store.
- Throws:
java.io.IOException
- (optional) if the id is invalid or the area can not otherwise be accessed.
-
lastCloseClean
public boolean lastCloseClean()
Description copied from interface:Store
Returns true if the store was closed cleanly. This is important information that may need to be considered when reading information from the store. This is typically used to issue a scan on the data in the store when it is not closed cleanly.- Specified by:
lastCloseClean
in interfaceStore
-
-