Package org.apache.lucene.util.bkd
Class BKDWriter
- java.lang.Object
-
- org.apache.lucene.util.bkd.BKDWriter
-
- All Implemented Interfaces:
Closeable,AutoCloseable
public class BKDWriter extends Object implements Closeable
Recursively builds a block KD-tree to assign all incoming points in N-dim space to smaller and smaller N-dim rectangles (cells) until the number of points in a given rectangle is <=config.maxPointsInLeafNode. The tree is partially balanced, which means the leaf nodes will have the requestedconfig.maxPointsInLeafNodeexcept one that might have less. Leaf nodes may straddle the two bottom levels of the binary tree. Values that fall exactly on a cell boundary may be in either cell.The number of dimensions can be 1 to 8, but every byte[] value is fixed length.
This consumes heap during writing: it allocates a
Long[numLeaves], abyte[numLeaves*(1+config.bytesPerDim)]and then uses up to the specifiedmaxMBSortInHeapheap space for writing.NOTE: This can write at most Integer.MAX_VALUE *
config.maxPointsInLeafNode/ config.bytesPerDim total points.- WARNING: This API is experimental and might change in incompatible ways in the next release.
-
-
Field Summary
Fields Modifier and Type Field Description static StringCODEC_NAMEprotected BKDConfigconfigBKD tree configurationstatic floatDEFAULT_MAX_MB_SORT_IN_HEAPDefault maximum heap to use, before spilling to (slower) diskprotected FixedBitSetdocsSeenprotected byte[]maxPackedValueMaximum per-dim values, packedprotected byte[]minPackedValueMinimum per-dim values, packedprotected longpointCountstatic intVERSION_CURRENTstatic intVERSION_LEAF_STORES_BOUNDSstatic intVERSION_LOW_CARDINALITY_LEAVESstatic intVERSION_META_FILEstatic intVERSION_SELECTIVE_INDEXINGstatic intVERSION_START
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidadd(byte[] packedValue, int docID)voidclose()IORunnablefinish(IndexOutput metaOut, IndexOutput indexOut, IndexOutput dataOut)Writes the BKD tree to the providedIndexOutputs and returns aRunnablethat writes the index of the tree if at least one point has been added, ornullotherwise.IORunnablemerge(IndexOutput metaOut, IndexOutput indexOut, IndexOutput dataOut, List<MergeState.DocMap> docMaps, List<PointValues> readers)More efficient bulk-add for incomingPointValuess.protected intsplit(byte[] minPackedValue, byte[] maxPackedValue, int[] parentSplits)Pick the next dimension to split.IORunnablewriteField(IndexOutput metaOut, IndexOutput indexOut, IndexOutput dataOut, String fieldName, MutablePointTree reader)Write a field from aMutablePointTree.
-
-
-
Field Detail
-
CODEC_NAME
public static final String CODEC_NAME
- See Also:
- Constant Field Values
-
VERSION_START
public static final int VERSION_START
- See Also:
- Constant Field Values
-
VERSION_LEAF_STORES_BOUNDS
public static final int VERSION_LEAF_STORES_BOUNDS
- See Also:
- Constant Field Values
-
VERSION_SELECTIVE_INDEXING
public static final int VERSION_SELECTIVE_INDEXING
- See Also:
- Constant Field Values
-
VERSION_LOW_CARDINALITY_LEAVES
public static final int VERSION_LOW_CARDINALITY_LEAVES
- See Also:
- Constant Field Values
-
VERSION_META_FILE
public static final int VERSION_META_FILE
- See Also:
- Constant Field Values
-
VERSION_CURRENT
public static final int VERSION_CURRENT
- See Also:
- Constant Field Values
-
DEFAULT_MAX_MB_SORT_IN_HEAP
public static final float DEFAULT_MAX_MB_SORT_IN_HEAP
Default maximum heap to use, before spilling to (slower) disk- See Also:
- Constant Field Values
-
config
protected final BKDConfig config
BKD tree configuration
-
docsSeen
protected final FixedBitSet docsSeen
-
minPackedValue
protected final byte[] minPackedValue
Minimum per-dim values, packed
-
maxPackedValue
protected final byte[] maxPackedValue
Maximum per-dim values, packed
-
pointCount
protected long pointCount
-
-
Method Detail
-
add
public void add(byte[] packedValue, int docID) throws IOException- Throws:
IOException
-
writeField
public IORunnable writeField(IndexOutput metaOut, IndexOutput indexOut, IndexOutput dataOut, String fieldName, MutablePointTree reader) throws IOException
Write a field from aMutablePointTree. This way of writing points is faster than regular writes withadd(byte[], int)since there is opportunity for reordering points before writing them to disk. This method does not use transient disk in order to reorder points.- Throws:
IOException
-
merge
public IORunnable merge(IndexOutput metaOut, IndexOutput indexOut, IndexOutput dataOut, List<MergeState.DocMap> docMaps, List<PointValues> readers) throws IOException
More efficient bulk-add for incomingPointValuess. This does a merge sort of the already sorted values and currently only works when numDims==1. This returns -1 if all documents containing dimensional values were deleted.- Throws:
IOException
-
finish
public IORunnable finish(IndexOutput metaOut, IndexOutput indexOut, IndexOutput dataOut) throws IOException
Writes the BKD tree to the providedIndexOutputs and returns aRunnablethat writes the index of the tree if at least one point has been added, ornullotherwise.- Throws:
IOException
-
close
public void close() throws IOException- Specified by:
closein interfaceAutoCloseable- Specified by:
closein interfaceCloseable- Throws:
IOException
-
split
protected int split(byte[] minPackedValue, byte[] maxPackedValue, int[] parentSplits)Pick the next dimension to split.- Parameters:
minPackedValue- the min values for all dimensionsmaxPackedValue- the max values for all dimensionsparentSplits- how many times each dim has been split on the parent levels- Returns:
- the dimension to split
-
-