Refactoring Rust Code with ChatGPT o1

I wanted to use gpt-4o to refactor the code of the trie-hard crate that was featured on hacker news recently. I thought if I removed the enums from the library, I can better understand the simplified code and might be able to play with different trie implementations in rust. Then I woke up today and read the announcement for ChatGPT-o1, had the idea to compare ChatGPT-o1 with ChatGPT-4o on the task of doing this refactoring. I don’t know rust, so I quickly learnt how to use cargo and a few rust basics. Then I upgraded to ChatGPT Plus to get access to the new o1-mini and o1-preview models.

First I tried getting chatgpt-4o to do the refactoring using the prompt:

simplify this rust library by removing the different sized enums and only using the U8 size.

The result was that there was no difference except line endings may have been changed to unix.

When I updated the prompt to provide an example of the MasksByByte datatype, simplify this rust library by removing the different sized enums and only using the U8 size. For example MasksByByte is an enum, change it to be an alias for the U8 datatype. , the only change chatgpt-4o made was to change the type MarksByByte from:

enum MasksByByte {
    U8(MasksByByteSized<u8>),
    U16(MasksByByteSized<u16>),
    U32(MasksByByteSized<u32>),
    U64(MasksByByteSized<u64>),
    U128(MasksByByteSized<u128>),
    U256(MasksByByteSized<U256>),
}

to:

type MasksByByte = MasksByByteSized<u8>;

I have made this file available on github as modified_4o.rs.

When I tried gpt-o1-mini with the same prompt, see: modified-o1-mini.rs, it made extensive changes, but the result wouldn’t compile. Note that I had to paste in the file instead of using file upload functionality. The chat output stopped near the end of the tests in the code, so I copied the few lines from the original file to complete the module.

Here’s one example error message:

error[E0121]: the placeholder `_` is not allowed within types on item signatures for functions
   --> src/lib.rs:534:51
    |
534 |     fn evaluate(&self, c: u8, trie: &TrieHard<'_, _>) -> Option<usize> {
    |                                                   ^ not allowed in type signatures
    |

In the original code, there is a macro (varying by the size used) that defines this function. o1-mini has successfully removed the macro, but hasn’t been able to rewrite the parameters successfully. This is the original line within the macro for context:

macro_rules! trie_impls {
    ($($int_type:ty),+) => {
        $(
            trie_impls!(_impl $int_type);
        )+
    };

    (_impl $int_type:ty) => {

        impl SearchNode<$int_type> {
            fn evaluate<T>(&self, c: u8, trie: &TrieHardSized<'_, T, $int_type>) -> Option<usize> {

I made a second attempt at using o1-mini, with the following prompt:

simplify this rust library by removing the different sized enums and only using the U8 size. For example MasksByByte is an enum, change it to be an alias for the U8 datatype. Also the u256 datatype isn't required, we only want U8, so remove all references to U256 as well.

This resulting file is available as modified-o1-mini2.rs . The modifications were quite different for this second run, but still the same parameter error is present. This output is much shorter because o1-mini removed all the code examples this time around. The code looks much the same from the brief look I’ve had to compare, but this second attempt failed to provide a ‘new’ constructor for TrieHard.

error[E0599]: no function or associated item named `new` found for struct `TrieHard` in the current scope
   --> src/lib.rs:438:19
    |
424 | pub struct TrieHard<'a, T> {
    | -------------------------- function or associated item `new` not found for this struct
...
438 |         TrieHard::new(values)
    |                   ^^^ function or associated item not found in `TrieHard<'_, _>`

Finally I decided to see if using the larger chatgpt-o1-preview model would make any difference.

Suprisingly this finished faster than the o1-mini model, at 40s vs 70s. It has no errors when compiled, see modified-o1-preview.rs. I have saved this version to rsc/lib.rs in the github repository as well, and my brief test code worked. This is my test code:

use trie_hard::TrieHard;  // Import the TrieHard type from the trie-hard crate

fn main() {
    let words = ["and", "ant", "dad", "do", "dot"];
    let trie = words.into_iter().collect::<TrieHard<'_, _>>();
    println!("Trie created with words: {:?}", words);

    // Perform a prefix search for the prefix "do"
    let prefix = "do";
    let results = trie.prefix_search(prefix);

    // Step 3: Iterate over the results and print them
    println!("Words with prefix '{}':", prefix);
    for (word_bytes, _value) in results {
        // Convert the byte slice to a string and print it
        if let Ok(word_str) = std::str::from_utf8(word_bytes) {
            println!("{}", word_str);
        } else {
            println!("Error: Could not convert bytes to string");
        }
    }
}

When running unit tests with cargo test, the only tests that fail are test_full_text, because they are the only ones that use too many different characters to fit into a U8-sized bitmap. This U8 bitmap only allows for 8 different characters, so I’ll increase it to U32 with a simple search and replace, to allow for playing with it for all lower-case letters.

Concluding thoughts.

The o1 family of models use chain of thought reasoning to improve the ability to take on complex tasks. The thoughts are output to a hidden token memory and the conclusions and summarised thoughts are output to the user. This process takes a lot more time and computation than the 4o generation of models. I suppose the existing Transformer structure has the ability to be aware of what it has already output, it just needs to be trained in how best to use the “train of thought” reasoning. I expect some changes to the structure of the OpenAI’s models have been made, but we’ll only know when this information is leaked. o1 models are definitely useful for coding when making large-scale changes to a piece of code, but o1-mini turns out to be insufficiently smart to do the refactoring I asked for. Rust is a complex language and the refactoring I asked for was extensive. It’s impressive that o1-preview was able to code sufficiently well in rust to have no compilation errors.

Someone pointed out that having the reasoning tokens hidden from the user is done to prevent other llm companies from using the output of chatgpt-o1 to train competing models, which happened to chatgpt-4 in the past.

Chat gpt 4o is useful in helping get up to speed in new tools, such as cargo. Rust looks like a very powerful language, but there is a lot to learn. Chatgpt can help me to get up to speed on the language iself, too.