Visualising experiments

This tutorial introduces the built-in plotting utilities defined in plots.py. These functions help you inspect and compare the results of a BLADE experiment.

We assume you have already executed an experiment and saved the results using an ExperimentLogger. Replace results/simple_exp with your own experiment folder.

from iohblade.loggers import ExperimentLogger
from iohblade.plots import plot_convergence, plot_experiment_CEG, plot_code_evolution_graphs, plot_boxplot_fitness, plot_boxplot_fitness_hue, fitness_table, plot_token_usage
logger = ExperimentLogger('results/simple_exp')

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[1], line 3
      1 from iohblade.loggers import ExperimentLogger
      2 from iohblade.plots import plot_convergence, plot_experiment_CEG, plot_code_evolution_graphs, plot_boxplot_fitness, plot_boxplot_fitness_hue, fitness_table, plot_token_usage
----> 3 logger = ExperimentLogger('results/simple_exp')

File ~/work/BLADE/BLADE/iohblade/loggers/base.py:61, in ExperimentLogger.__init__(self, name, read)
     59         self._write_progress()
     60 else:
---> 61     self.dirname = self.create_log_dir(name)
     62     self.dirs.append(self.dirname)
     63     self.progress = {
     64         "start_time": datetime.now().isoformat(),
     65         "end_time": None,
     66         "current": 0,
     67         "total": 0,
     68     }

File ~/work/BLADE/BLADE/iohblade/loggers/base.py:90, in ExperimentLogger.create_log_dir(self, name)
     88     tempi += 1
     89     dirname = f"{name}-{tempi}"
---> 90 os.mkdir(dirname)
     91 return dirname

FileNotFoundError: [Errno 2] No such file or directory: 'results/simple_exp'

Convergence plot

plot_convergence shows the average best fitness over time for all methods on each problem.

plot_convergence(logger)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[2], line 1
----> 1 plot_convergence(logger)

NameError: name 'logger' is not defined

Code evolution graphs (CEG)

plot_experiment_CEG visualises how code evolves during optimisation. Each run is plotted separately.

plot_experiment_CEG(logger)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 plot_experiment_CEG(logger)

NameError: name 'logger' is not defined

plot_code_evolution_graphs is the lower level function used by plot_experiment_CEG. It plots a single run with various complexity metrics.

run_data = logger.get_data().query('method_name == "LLaMEA" and seed == 1')
plot_code_evolution_graphs(run_data, save=False)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[4], line 1
----> 1 run_data = logger.get_data().query('method_name == "LLaMEA" and seed == 1')
      2 plot_code_evolution_graphs(run_data, save=False)

NameError: name 'logger' is not defined

Boxplots

Use plot_boxplot_fitness or plot_boxplot_fitness_hue to compare final fitness across methods or problems.

plot_boxplot_fitness(logger)
plot_boxplot_fitness_hue(logger)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 plot_boxplot_fitness(logger)
      2 plot_boxplot_fitness_hue(logger)

NameError: name 'logger' is not defined

Fitness table

fitness_table creates a LaTeX table summarising mean and standard deviation per method/problem.

fitness_table(logger)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[6], line 1
----> 1 fitness_table(logger)

NameError: name 'logger' is not defined

Token usage

plot_token_usage shows how many API tokens were consumed by each method.

plot_token_usage(logger)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[7], line 1
----> 1 plot_token_usage(logger)

NameError: name 'logger' is not defined