2023-12-22

Benchmarking LLM models

Today we're gonna try to use ollama to generate some code, it quite easy you just need to install it using curl like this, and install some model:

curl https://ollama.ai/install.sh | sh

# these models already predownloaded before first run
time ollama run codellama 'show me inplace mergesort using golang'
#time ollama run deepseek-coder 'show me inplace mergesort using golang'
time ollama run llama2-uncensored 'show me inplace mergesort using golang'
time ollama run mistral 'show me inplace mergesort using golang'

#time ollama run starcoder 'show me inplace mergesort using golang'
time ollama run codeup 'show me inplace mergesort using golang'
time ollama run codebooga 'show me inplace mergesort using golang'
time ollama run magicoder 'show me inplace mergesort using golang'

ollama list
NAME                     ID           SIZE    MODIFIED
codebooga:latest         05b83c5673dc 19 GB   3 seconds ago
codellama:latest         8fdf8f752f6e 3.8 GB  4 minutes ago
deepseek-coder:latest    140a485970a6 776 MB  10 minutes ago
llama2-uncensored:latest 44040b922233 3.8 GB  10 minutes ago
mistral:latest           4d9f4b269c33 4.1 GB  2 days ago
codeup:latest            54289661f7a9 7.4 GB  4 minutes ago
magicoder:latest         8007de06f5d9 3.8 GB  4 minutes ago

starcoder:latest         847e5a7aa26f 1.8 GB  10 minutes ago



Here's the result of each of them, the benchmark runs using 32-core CPU, since my GPU still unsupported unsupported (gfx1032):

codellama - 62.59s - will overwrite "left" if array overlapped, at least the structure a bit correct

 Sure! Here is an example of how to implement in-place merge sort using Go:
```
package main

import (
        "fmt"
)

func merge(arr []int, low int, mid int, high int) {
        left := arr[low : mid+1]
        right := arr[mid+1 : high+1]

        for i := low; i <= high; i++ {
                if left[i-low] > right[i-mid] {
                        arr[i] = left[i-low]
                } else {
                        arr[i] = right[i-mid]
                }
        }
}

func inplaceMergeSort(arr []int, low int, high int) {
        if low >= high {
                return
        }

        mid := (low + high) / 2
        inplaceMergeSort(arr, low, mid)
        inplaceMergeSort(arr, mid+1, high)
        merge(arr, low, mid, high)
}

func main() {
        arr := []int{5, 3, 7, 9, 2, 4, 6, 8}
        inplaceMergeSort(arr, 0, len(arr)-1)
        fmt.Println("Sorted array:", arr)
}
```
This code sorts the input array in place using the merge sort algorithm. The `merge` function is used to combine two sorted subarrays
into a single sorted array. The `inplaceMergeSort` function recursively splits the array into smaller subarrays until each subarray
contains only one element, at which point the `merge` function is called to combine the subarrays.

The `main` function demonstrates how to use the `inplaceMergeSort` function on an example input array.

deepseek-coder - 23.53s - fast but messy indentation, also spouting quicksort not mergesort 

 
Here's an in-place version of merge sort using Golang:

```go
package main
 
import (
    "fmt"
)     
    
func partition(array []int, low int , high int ) int {
        pivot := array[high] // Picks the last element as a pivot. This is done arbitrarily here for simplicity of code but in
real-world cases it should be chosen by an external algorithm like quicksort or similar with better average case performance then
mergesorts which pick random elements
        i := (low - 1)    // Index of smaller element  
        for j:=  low ;j <= high-1; j++ {     
            if array[j] < pivot{      
                i++                     // Swap arr[i+1] and arra[high], then increment i.  This is the part that makes sure all elements sma
smaller than Pivot go to left of it, while rest are in right side    // The key process is to check if current element 'array'[] has
value less
                //than pivot'. If yes,"then do nothing", else swap values and increment i.
                }                          }      return (i + 1)            }
            func merge( array []int, low int , mid int,high int ) { // The main function that sorts arr[l..r] using auxiliary space
a[]={0...n} and i is index of start element in left subarray 和 right.   pivot such as to be inserted
                // elements into correct position for final sorted array, then call itself recursively      return }            public func m
main() {          arr :=  [7,12,45689,30,-2 ,-5]         fmt.Println(array)    mergeSort(&arr[..])     // Calling the function to sort
array using mergesort
                }` `go run main.go 输出结果为[-5 -2  12  7 30 45689 ]] ) and go playground https://play.golang.org/p/-Ni_kJvKlf for running the code
```     Sorry, I can't provide you with a full in-place merge sort because it depends on your requirements (whether to print or not),
as well complexity analysis and other factors which may affect performance of an algorithm. The above is just how one might implement
such functionality within Go language using Goroutines if necessary for multi threading/parallelism, but would need a different
approach otherwise due the nature of merge sort itself being inherently sequential in operation (each element has to be processed
before it can go on). 
 

llama2-uncensored - 50.87s - err, using builtin sort XD

Sure, here's an implementation of in-place mergesort using Go:
```go
package main
import "fmt"
func mergeSort(arr []int) {
 if len(arr) <= 1 {
 return
 }
 mid := len(arr) / 2
 left := arr[:mid]
 right := arr[mid:]
 fmt.Println("Merging", left, "and", right)
 sort.Ints(left)
 sort.Ints(right)
 mergeSort(left)
 merge(left, right)
}
func merge(left []int, right []int) []int {
 res := make([]int, len(left)+len(right))
 for i := 0; i < len(res); i++ {
 if left[i] <= right[i-1] {
 res[i] = left[i]
 } else {
 res[i] = right[i-1]
 }
 }
 return res
}
func main() {
 arr := []int{5, 3, 2, 1, 7, 8, 4, 6}
 fmt.Println("Original array:", arr)
 mergeSort(arr)
 fmt.Println("Merged array: ", arr)
}
```
This implementation uses the built-in `sort` package to sort the left and right halves of the original array separately, and then
merges them using a recursive function called `merge`. Finally, it applies the sorted array to the original one using the `mergeSort`
function.

mistral - 96.77s - not really inplace XD it creates copy

 Here's an example of in-place MergeSort implementation in Golang. In this approach, we divide the given slice into smaller sub-arrays
until each sub-array contains only one element. Then we merge these sub-arrays back together to sort the original slice.

```go
func merge(arr []int, left int, mid int, right int) { // Merge function
    n1 := mid - left + 1
    n2 := right - mid

    Left := make([]int, n1)
    Right := make([]int, n2)

    for i := 0; i < n1; i++ {
        Left[i] = arr[left + i]
    }
    for j := 0; j < n2; j++ {
        Right[j] = arr[mid+1+j]
    }

    i := 0
    j := 0
    k := left

    for i < n1 && j < n2 {
        if Left[i] <= Right[j] {
            arr[k] = Left[i]
            i++
        } else {
            arr[k] = Right[j]
            j++
        }
        k++
    }

    for i < n1 { // Copy the rest of Left
        arr[k] = Left[i]
        i++
        k++
    }

    for j < n2 { // Copy the rest of Right
        arr[k] = Right[j]
        j++
        k++
    }
}

func mergeSort(arr []int, left int, right int) { // Merge sort function
    if len(arr) <= 1 {
        return
    }

    mid := (left + right) / 2

    mergeSort(arr, left, mid)
    mergeSort(arr, mid+1, right)
    merge(arr, left, mid, right)
}

func main() {
    arr := []int{12, 11, 13, 5, 6, 7}
    fmt.Println("Unsorted array:")
    printArray(arr)
    mergeSort(arr, 0, len(arr)-1)
    fmt.Println("Sorted array:")
    printArray(arr)
}

func printArray(arr []int) {
    for _, a := range arr {
        fmt.Println(a, " ")
    }
    fmt.Println()
}
```

In this example, the `mergeSort` function is called recursively on the array until each sub-array contains only one element. After
that, the merge function merges these sub-arrays back together, sorting the entire array in place. 

starcoder - 71.65s - BAD, spouting python nonsense

# golang
go run -race./cmd/test
# python
python3.7 -m pytest tests_unit --cov=src
-- spouting nonsense python code --

magicoder - 91.89s - not really in-place

 Here is a simple implementation of the in-place merge sort algorithm in Golang. It's very similar to the traditional merge sort
algorithm but works in-place without needing an additional array space:

```go
package main

import (
        "fmt"
)

func printArray(arr []int) {
        for _, value := range arr {
                fmt.Print(value, " ")
        }
        fmt.Println()
}

func mergeSort(arr []int) {
        var helper [100]int // declare an array to hold the helper values
        mergeSortRecursive(&helper[0], arr, 0, len(arr)-1)
}

func mergeSortRecursive(aux *[100]int, src []int, lo, hi int) {
        if lo >= hi {
                return // base case: array is sorted
        }

        // sort the left half
        mid := (lo + hi) / 2
        mergeSortRecursive(aux, src, lo, mid)

        // sort the right half
        mergeSortRecursive(aux, src, mid+1, hi)

        // merge the sorted halves together
        merge(&helper[0], aux, src, lo, mid, hi)
}

func merge(dst *[100]int, aux *[100]int, src []int, lo, mid, hi int) {
        i := lo
        j := mid + 1

        for k := lo; k <= hi; k++ {
                if i > mid {
                        dst[k] = src[j]
                        j++
                } else if j > hi {
                        dst[k] = src[i]
                        i++
                } else if src[i] < src[j] {
                        dst[k] = src[i]
                        i++
                } else {
                        dst[k] = src[j]
                        j++
                }
        }

        // copy back the sorted values to our original array
        for i, value := range dst[:hi-lo+1] {
                src[i+lo] = value
        }
}

func main() {
        arr := []int{37, 49, 38, 50, 3, 13}
        printArray(arr) // before sorting
        mergeSort(arr)
        printArray(arr) // after sorting
}
```
This code prints the input array, sorts it using merge sort algorithm and prints again to check if the sorting is successful.

codeup - 127.32s - using temporary variable, so this just normal mergesort, quite slow

Here is an example of an in-place mergesort implementation in Go:
```
func mergeSort(arr []int, low, high int) {
    if low < high {
        // divide the array into two halves
        mid := (low + high) / 2
        mergeSort(arr, low, mid)
        mergeSort(arr, mid+1, high)
        merge(arr, low, mid, high)
    }
}

func merge(arr []int, low, mid, high int) {
    var result []int
    left := arr[:mid]
    right := arr[mid:]
    for i := 0; i < len(left); i++ {
        result = append(result, left[i])
    }
    for j := 0; j < len(right); j++ {
        result = append(result, right[j])
    }
    return result
}
```
This code sorts the input array in place using the mergesort algorithm. The `mergeSort` function divides the array into two halves, sorts each half recursively, and then merges the two sorted halves back together using the `merge` function. The `merge` function takes four
integer parameters: the left and right indices of the two arrays being merged, and the low and high indices of the resulting array. It appends the elements of the left and right arrays to the resulting array in the correct order.

To use this code, you can simply call the `mergeSort` function with the address of the input array as an argument, like this:
```
arr := []int{1, 2, 3, 4, 5, 6}
mergeSort(arr)
fmt.Println(arr) // [1, 2, 3, 4, 5, 6]
```
This will sort the input array in place and print the resulting sorted array. Note that this implementation of mergesort has a time complexity of O(n log n), which means it will take longer to sort larger arrays.

codebooga - 490.72s - using temporary variable, so this just normal mergesort, slow AF

Here is an implementation of in-place merge sort using Go:

```go
package main

import (
        "fmt"
)

func merge(arr []int, low int, mid int, high int) {
        n1 := mid - low + 1
        n2 := high - mid

        L := make([]int, n1)
        R := make([]int, n2)

        for i := 0; i < n1; i++ {
                L[i] = arr[low+i]
        }
        for j := 0; j < n2; j++ {
                R[j] = arr[mid+1+j]
        }

        i, j, k := 0, 0, low

        for i < n1 && j < n2 {
                if L[i] <= R[j] {
                        arr[k] = L[i]
                        i++
                } else {
                        arr[k] = R[j]
                        j++
                }
                k++
        }

        for i < n1 {
                arr[k] = L[i]
                i++
                k++
        }

        for j < n2 {
                arr[k] = R[j]
                j++
                k++
        }
}

func mergeSort(arr []int, low int, high int) {
        if low < high {
                mid := (low + high) / 2
                mergeSort(arr, low, mid)
                mergeSort(arr, mid+1, high)
                merge(arr, low, mid, high)
        }
}

func inPlaceMergeSort(arr []int) {
        if len(arr) > 1 {
                mergeSort(arr, 0, len(arr)-1)
        }
}

func main() {
        arr := []int{12, 11, 13, 5, 6, 7}
        fmt.Println("Given array is: ", arr)
        inPlaceMergeSort(arr)
        fmt.Println("Sorted array is: ", arr)
}
```

This program sorts the given array using in-place merge sort algorithm. The `merge` function takes three parameters, an array `arr`, and two indices `low` and `high`. It merges the two sorted subarrays of `arr`, one starting from index `low` to `mid` (inclusive) and another from
`mid+1` to `high` (inclusive), into a single sorted subarray. The `mergeSort` function is a recursive function that sorts the array using merge sort algorithm by dividing it into two halves until each half has only one element, then calling the `merge` function to combine them
back in order. Finally, the `inPlaceMergeSort` function calls `mergeSort` on the input array and sorts it in place. The `main` function demonstrates how to use this implementation by sorting an example array.

Conclusion

So as you can see, size of model determines the speed of resolving the answer, also all of them are not good enough for now to implement in-place merge part.