Logging

We can use the following schema for the logs of the large output file. Per Cooper's idea, we can split these into smaller files later if we would like, but for now, lets first clean up the main file we currently have. These files live at: <output-directory>/log.json

[
    0: {
        'TIME'   : <time stamp>,
        'PROMPT' : <input from the user or preplanned prompt>,
        'OUTPUT' : <output displayed to the user>,
        'STATUS' : {
                'LLM' : <primary large language model being used>,
                'Databases' : {
                        'RAG' : <primary data base>,
                        <other databases>: <more databases can be added>,
                        ...
                    },
                'config' : {
                        'debug' : True,
                        'output-directory' : <path to output directory>,
                        ...
                    }
            },
        'PROCESS' : {
                'MODULE' : <name of module i.e. RAG, DATABASE, CODER, etc.>
                'STEPS' [
                        # information for a particular step involved in executing the module. Some examples
                        {
                           'LLM' : <language model>,
                           'prompt template' : <prompt template>,
                           'model input' : <input to model>,
                           'model output' : <output of model>
                        },
                        {
                           'func' : 'rag.retreival',
                           'num articles retrieved' : 10,
                           'multiquery' : True,
                           'compression' : True,
                        }
                    ]
            },
        'PLANNED' : [
                <Next prompt in queue>,
                ...
            ]
    },
1 : {
        'TIME'   : <time stamp>,
        'PROMPT' : <second prompt>,
        ...
    },
...
]