LLamaSharp/docs/xmldocs/llama.native.safellamacontexthandle.md

• SafeLLamaContextHandle
• • Properties
  • • VocabCount
    • • Property Value
    • ContextSize
    • • Property Value
    • EmbeddingSize
    • • Property Value
    • ModelHandle
    • • Property Value
    • IsInvalid
    • • Property Value
    • IsClosed
    • • Property Value
  • Constructors
  • • SafeLLamaContextHandle(IntPtr, SafeLlamaModelHandle)
    • • Parameters
  • Methods
  • • ReleaseHandle()
    • • Returns
    • Create(SafeLlamaModelHandle, LLamaContextParams)
    • • Parameters
      • Returns
      • Exceptions
    • Clone(LLamaContextParams)
    • • Parameters
      • Returns
    • Tokenize(String, Boolean, Encoding)
    • • Parameters
      • Returns
      • Exceptions
    • GetLogits()
    • • Returns
    • TokenToString(Int32, Encoding)
    • • Parameters
      • Returns
    • TokenToString(Int32, Encoding, StringBuilder)
    • • Parameters
    • TokenToSpan(Int32, Span<Byte>)
    • • Parameters
      • Returns
    • Eval(ReadOnlySpan<Int32>, Int32, Int32)
    • • Parameters
      • Returns
    • GetStateSize()
    • • Returns
    • GetState(Byte, UInt64)*
    • • Parameters
      • Returns
      • Exceptions
    • GetState(IntPtr, UInt64)
    • • Parameters
      • Returns
      • Exceptions
    • SetState(Byte)*
    • • Parameters
      • Returns
    • SetState(IntPtr)
    • • Parameters
      • Returns

SafeLLamaContextHandle

Namespace: LLama.Native

A safe wrapper around a llama_context

public sealed class SafeLLamaContextHandle : SafeLLamaHandleBase, System.IDisposable

Inheritance Object → CriticalFinalizerObject → SafeHandle → SafeLLamaHandleBase → SafeLLamaContextHandle

Implements IDisposable

Properties

VocabCount

Total number of tokens in vocabulary of this model

public int VocabCount { get; }

Property Value

Int32

ContextSize

Total number of tokens in the context

public int ContextSize { get; }

Property Value

Int32

EmbeddingSize

Dimension of embedding vectors

public int EmbeddingSize { get; }

Property Value

Int32

ModelHandle

Get the model which this context is using

public SafeLlamaModelHandle ModelHandle { get; }

Property Value

SafeLlamaModelHandle

IsInvalid

public bool IsInvalid { get; }

Property Value

Boolean

IsClosed

public bool IsClosed { get; }

Property Value

Boolean

Constructors

SafeLLamaContextHandle(IntPtr, SafeLlamaModelHandle)

Create a new SafeLLamaContextHandle

public SafeLLamaContextHandle(IntPtr handle, SafeLlamaModelHandle model)

Parameters

handle IntPtr

pointer to an allocated llama_context

model SafeLlamaModelHandle

the model which this context was created from

Methods

ReleaseHandle()

protected bool ReleaseHandle()

Returns

Boolean

Create(SafeLlamaModelHandle, LLamaContextParams)

Create a new llama_state for the given model

public static SafeLLamaContextHandle Create(SafeLlamaModelHandle model, LLamaContextParams lparams)

Parameters

model SafeLlamaModelHandle

lparams LLamaContextParams

Returns

SafeLLamaContextHandle

Exceptions

RuntimeError

Clone(LLamaContextParams)

Create a new llama context with a clone of the current llama context state

public SafeLLamaContextHandle Clone(LLamaContextParams lparams)

Parameters

lparams LLamaContextParams

Returns

SafeLLamaContextHandle

Tokenize(String, Boolean, Encoding)

Convert the given text into tokens

public Int32[] Tokenize(string text, bool add_bos, Encoding encoding)

Parameters

text String

The text to tokenize

add_bos Boolean

Whether the "BOS" token should be added

encoding Encoding

Encoding to use for the text

Returns

Int32[]

Exceptions

RuntimeError

GetLogits()

Token logits obtained from the last call to llama_eval()
The logits for the last token are stored in the last row
Can be mutated in order to change the probabilities of the next token.

Rows: n_tokens

Cols: n_vocab

public Span<float> GetLogits()

Returns

Span<Single>

TokenToString(Int32, Encoding)

Convert a token into a string

public string TokenToString(int token, Encoding encoding)

Parameters

token Int32

Token to decode into a string

encoding Encoding

Returns

String

TokenToString(Int32, Encoding, StringBuilder)

Append a single llama token to a string builder

public void TokenToString(int token, Encoding encoding, StringBuilder dest)

Parameters

token Int32

Token to decode

encoding Encoding

dest StringBuilder

string builder to append the result to

TokenToSpan(Int32, Span<Byte>)

Convert a single llama token into bytes

public int TokenToSpan(int token, Span<byte> dest)

Parameters

token Int32

Token to decode

dest Span<Byte>

A span to attempt to write into. If this is too small nothing will be written

Returns

Int32

The size of this token. nothing will be written if this is larger than dest

Eval(ReadOnlySpan<Int32>, Int32, Int32)

Run the llama inference to obtain the logits and probabilities for the next token.

public bool Eval(ReadOnlySpan<int> tokens, int n_past, int n_threads)

Parameters

tokens ReadOnlySpan<Int32>

The provided batch of new tokens to process

n_past Int32

the number of tokens to use from previous eval calls

n_threads Int32

Returns

Boolean

Returns true on success

GetStateSize()

Get the size of the state, when saved as bytes

public ulong GetStateSize()

Returns

UInt64

GetState(Byte, UInt64)*

Get the raw state of this context, encoded as bytes. Data is written into the dest pointer.

public ulong GetState(Byte* dest, ulong size)

Parameters

dest Byte*

Destination to write to

size UInt64

Number of bytes available to write to in dest (check required size with GetStateSize())

Returns

UInt64

The number of bytes written to dest

Exceptions

ArgumentOutOfRangeException

Thrown if dest is too small

GetState(IntPtr, UInt64)

Get the raw state of this context, encoded as bytes. Data is written into the dest pointer.

public ulong GetState(IntPtr dest, ulong size)

Parameters

dest IntPtr

Destination to write to

size UInt64

Number of bytes available to write to in dest (check required size with GetStateSize())

Returns

UInt64

The number of bytes written to dest

Exceptions

ArgumentOutOfRangeException

Thrown if dest is too small

SetState(Byte)*

Set the raw state of this context

public ulong SetState(Byte* src)

Parameters

src Byte*

The pointer to read the state from

Returns

UInt64

Number of bytes read from the src pointer

SetState(IntPtr)

Set the raw state of this context

public ulong SetState(IntPtr src)

Parameters

src IntPtr

The pointer to read the state from

Returns

UInt64

Number of bytes read from the src pointer