Weights and Biases

What is Weights and Biases

_images/WB_logo.png

For probably 99% of all neural networks, a hyperparameter search is needed. Things like the number of neurons, then number of layers, the optimizer, the regularization etc. all need to be varied if you wish to get the best performing model. This can quickly become cumbersome (given how quickly permuations grow).

One way of hangling a hyperparameter search is to use a program called Weights and Biases. In order to get started with W&B, follow this document. This quickstart was written by Dr. Monique Shotande, so reach out to her if you have questions!

Quickstart for W&B on Schooner

  1. Create an account and install W&B

      1. Sign up for free: https://wandb.ai/site and then login

  2. Install the wandb into a Python 3 environment

    1. pip install wandb

    or

    1. mamba install -c conda-forge wandb

  3. From the command line, login to wandb

    1. You’ll need your API key to login via the CLI

    2. wandb login

  4. Start a run and track hyperparameters

    1. Initialize a W&B Run object with wandb.init() and pass a dictionary to the config parameter with key-value pairs of hyperparameter names and values:

    run = wandb.init(
    # Set the project where this run will be logged
    project="my-awesome-project",
    # Track hyperparameters and run metadata
    config={"learning_rate": 0.01,
            "epochs": 10,})

    1. Use the WandbMetricsLogger() callback to automatically track the loss and metrics

    # Modern lightweight callback
# Tracks the loss and metrics
# note: wandb callbacks can only be initialized after
#       wandb.init()
metrics_callback = WandbMetricsLogger()

# Train model
history = model.fit(x_train, y_train, epochs=config['epochs'],
            callbacks=[metrics_callback])

    1. Go to your wandb home to view the details of the experiment and watch run in real-time

    2. You can find the specific urls for the unique runs in the error field generated by slurm jobs

Additional Resources:

AI2ES Coding Standards: Documentation Standards by Jay Rothenberger

Using wandb with Keras Tuners: