Using Ollama API for Custom Model Parameters and Batch Processing in Python

The article introduces advanced features of the Ollama API through a Python client class ollamaAdvancedClient. It explains how to customize generation parameters such as temperature (0.0‑2.0), top_p (0.0‑1.0), top_k, num_predict, repeat_penalty, presence_penalty, frequency_penalty, and seed. A default option dictionary is defined and merged with user‑provided overrides before calling self.client.generate.

An example script advanced_config_demo shows two use cases: (1) generating a poem with a higher temperature (0.9) and top‑p (0.95) for more creativity, and (2) creating a custom model named coding-assistant based on llama3 with a system prompt that positions the model as a professional code assistant. The custom model is then used to generate a Python function for computing the nth Fibonacci number.

The second part covers batch processing and performance optimization. A

@dataclass

GenerationTask

encapsulates a prompt, model name, temperature, and token limit. The ollamaBatchProcessor class creates a thread pool (default 3 workers) and defines process_batch, which submits each task to _process_single_task and collects results, preserving order and handling exceptions.

The helper _process_single_task sends a generation request, records start and end times, and returns a dictionary containing the response text, duration, token count, and model identifier. Errors are raised as exceptions.

A benchmarking method benchmark_models iterates over a list of model names, runs each model three times with a fixed prompt, measures per‑run latency, prints each duration, and finally computes average, minimum, maximum latency and success rate for each model.

The script batch_processing_demo creates a processor with two workers, defines three distinct GenerationTask instances (explaining machine learning, reversing a string in Python, summarizing the solar system), and executes them in parallel. It prints total batch time, then iterates over the results, displaying success status, per‑task duration, and a truncated snippet of the model's response, or the error message if a task failed.