LLamaSharp/docs/xmldocs/llama.native.llamabatch.md

• LLamaBatch
• • Properties
  • • TokenCount
    • • Property Value
    • SequenceCapacity
    • • Property Value
  • Constructors
  • • LLamaBatch()
  • Methods
  • • ToNativeBatch(LLamaNativeBatch&)
    • • Parameters
      • Returns
    • Add(LLamaToken, LLamaPos, ReadOnlySpan<LLamaSeqId>, Boolean)
    • • Parameters
      • Returns
    • Add(LLamaToken, LLamaPos, List<LLamaSeqId>, Boolean)
    • • Parameters
      • Returns
    • Add(LLamaToken, LLamaPos, LLamaSeqId, Boolean)
    • • Parameters
      • Returns
    • AddRange(ReadOnlySpan<LLamaToken>, LLamaPos, LLamaSeqId, Boolean)
    • • Parameters
      • Returns
    • Clear()
    • GetLogitPositions(Span<ValueTuple<LLamaSeqId, Int32>>)
    • • Parameters
      • Returns

LLamaBatch

Namespace: LLama.Native

A batch allows submitting multiple tokens to multiple sequences simultaneously

public class LLamaBatch

Inheritance Object → LLamaBatch

Properties

TokenCount

The number of tokens in this batch

public int TokenCount { get; private set; }

Property Value

Int32

SequenceCapacity

Maximum number of sequences a token can be assigned to (automatically grows if exceeded)

public int SequenceCapacity { get; private set; }

Property Value

Int32

Constructors

LLamaBatch()

Create a new batch for submitting inputs to llama.cpp

public LLamaBatch()

Methods

ToNativeBatch(LLamaNativeBatch&)

internal GroupDisposable ToNativeBatch(LLamaNativeBatch& batch)

Parameters

batch LLamaNativeBatch&

Returns

GroupDisposable

Add(LLamaToken, LLamaPos, ReadOnlySpan<LLamaSeqId>, Boolean)

Add a single token to the batch at the same position in several sequences

public int Add(LLamaToken token, LLamaPos pos, ReadOnlySpan<LLamaSeqId> sequences, bool logits)

Parameters

token LLamaToken

The token to add

pos LLamaPos

The position to add it att

sequences ReadOnlySpan<LLamaSeqId>

The set of sequences to add this token to

logits Boolean

Returns

Int32

The index that the token was added at. Use this for GetLogitsIth

Remarks:

ad93962657/common/common.cpp (L829C2-L829C2)

Add(LLamaToken, LLamaPos, List<LLamaSeqId>, Boolean)

Add a single token to the batch at the same position in several sequences

public int Add(LLamaToken token, LLamaPos pos, List<LLamaSeqId> sequences, bool logits)

Parameters

token LLamaToken

The token to add

pos LLamaPos

The position to add it att

sequences List<LLamaSeqId>

The set of sequences to add this token to

logits Boolean

Returns

Int32

The index that the token was added at. Use this for GetLogitsIth

Remarks:

ad93962657/common/common.cpp (L829C2-L829C2)

Add(LLamaToken, LLamaPos, LLamaSeqId, Boolean)

Add a single token to the batch at a certain position for a single sequences

public int Add(LLamaToken token, LLamaPos pos, LLamaSeqId sequence, bool logits)

Parameters

token LLamaToken

The token to add

pos LLamaPos

The position to add it att

sequence LLamaSeqId

The sequence to add this token to

logits Boolean

Returns

Int32

The index that the token was added at. Use this for GetLogitsIth

Remarks:

ad93962657/common/common.cpp (L829C2-L829C2)

AddRange(ReadOnlySpan<LLamaToken>, LLamaPos, LLamaSeqId, Boolean)

Add a range of tokens to a single sequence, start at the given position.

public int AddRange(ReadOnlySpan<LLamaToken> tokens, LLamaPos start, LLamaSeqId sequence, bool logitsLast)

Parameters

tokens ReadOnlySpan<LLamaToken>

The tokens to add

start LLamaPos

The starting position to add tokens at

sequence LLamaSeqId

The sequence to add this token to

logitsLast Boolean

Whether the final token should generate logits

Returns

Int32

The index that the final token was added at. Use this for GetLogitsIth

Clear()

Set TokenCount to zero for this batch

public void Clear()

GetLogitPositions(Span<ValueTuple<LLamaSeqId, Int32>>)

Get the positions where logits can be sampled from

internal Span<ValueTuple<LLamaSeqId, int>> GetLogitPositions(Span<ValueTuple<LLamaSeqId, int>> dest)

Parameters

dest Span<ValueTuple<LLamaSeqId, Int32>>

Returns

Span<ValueTuple<LLamaSeqId, Int32>>