The KoLMogorov-Test: Can CodeLMs lead to the next breakthrough in data compression?

In the paper we show that stronger models perform better on KT and thatn CodeLMs can outperform other compression methods on synthetic distributions when training data is avialable. Another exciting direction for future work is to encode programs as lambda experssions, making it feasible to fit a small interpreter of only 383 in the decoder as suggested here.

To make a new submission, upload three files - a file with the encoded program, a file that decodes the programs to executable code, and a file with the expected results. Let's consider a toy example for the sequence - [5, 10, 13, 14, 16, 5, 5, 5]

The programs file is a jsonl that follows the following format. We recommend submitting a single program for the whole sequence:

{"sub_sequence_start_index": 0, "sub_sequence_end_index": 7 (the length of the sequence), "encoded_program": "H4sIAI7Te2cC/8svLSkoLVGwVYg21VEwNABiYyA2AWIzHQVTCIoFAMxnYTIlAAAA (An encoding of the program whose execution results in the input sequence. This is the Gzip encoding of the program that returns the input sequence)."}

For simplicity, we allow splitting to subsequent sub-sequences:

{"sub_sequence_start_index": 0, "sub_sequence_end_index": k, "encoded_program": "An encoding of the program whose execution results in the subsequence between indeses [0, n]."}
{"sub_sequence_start_index": k+1, "sub_sequence_end_index": k+1+j, "encoded_program": "An encoding of the program whose execution results in the subsequence between indeses [n+1, n+1+m]."}
...
{"sub_sequence_start_index": n, "sub_sequence_end_index": len(sequence), "encoded_program": "An encoding of the program whose execution results in the subsequence between indeses [n, len(sequence)]."}

For decoding, upload a python file that implements the 'decode' method, which receives as input an programs from the programs file and returns the executable python program. Executing the decoded program must result in the input sequence. For example, this is a decoder that decompresses using Gzip.

import gzip
import base64

def decode(program):
    return gzip.decompress(base64.b64decode(program)).decode('utf-8')

For the results, upload a Json file that matches the Result object returned from the official evaluation script. If you are interested in verification of your results, please send an email with the submission details. We will then execute the code and verify the execution matches the original sequece.

{
    "compressed_programs_size": 64,
    "decoder_size": 118,
    "compressed_size": 182,
    "compression_rate_without_decoder": 8.0,
    "compression_rate": 22.75,
    "accuracy": 1,
    "gold_data_size": 8,
    "first_error": null
}