Code

Mitchish65#388

Code

Merged

dirkgr merged 72 commits intomainallenai/OLMo:mainfrom mitchish65allenai/OLMo:mitchish65Copy head branch name to clipboardJan 5, 2024

Commits

Commits on Nov 15, 2023

• Adds a 65B config

  dirkgr

  committed
  f661bfd

  Copy full SHA for f661bfd

Commits on Nov 16, 2023

• Make the config actually 65B

  dirkgr

  committed
  28cf940

  Copy full SHA for 28cf940

• More parameters

  dirkgr

  committed
  0c122e9

  Copy full SHA for 0c122e9

Commits on Nov 20, 2023

• Set to the most memory efficient setting

  dirkgr

  committed
  5401904

  Copy full SHA for 5401904

• Change some names

  dirkgr

  committed
  a4af8af

  Copy full SHA for a4af8af

• Default to local checkpointing

  dirkgr

  committed
  cc72acd

  Copy full SHA for cc72acd

• Script for the 65b config

  dirkgr

  committed
  c609fb4

  Copy full SHA for c609fb4

Commits on Nov 21, 2023

• Put some learnings into the script

  dirkgr

  committed
  addc631

  Copy full SHA for addc631

• Decay everything

  dirkgr

  committed
  135cf11

  Copy full SHA for 135cf11

• New image version for logging into nodes

  dirkgr

  committed
  ce380b4

  Copy full SHA for ce380b4

• Merge remote-tracking branch 'origin/epwalsh/threaded-data-loading' into mitchish65

  dirkgr

  committed
  518b2e1

  Copy full SHA for 518b2e1

• Read the longrunfix data

  dirkgr

  committed
  2b59d48

  Copy full SHA for 2b59d48

• Mokey patch FSDP stream management

  dirkgr

  committed
  12fa63c

  Copy full SHA for 12fa63c

• 3 streams

  dirkgr

  committed
  74a7670

  Copy full SHA for 74a7670

• Revert "3 streams"

  Show description for ddd69c5
  dirkgr

  committed
  ddd69c5

  Copy full SHA for ddd69c5

• Revert "Mokey patch FSDP stream management"

  Show description for 36f2b17
  dirkgr

  committed
  36f2b17

  Copy full SHA for 36f2b17

• Smaller global batch for batch size warmup

  dirkgr

  committed
  f52fa18

  Copy full SHA for f52fa18

Commits on Nov 22, 2023

• Grad clipping warmup

  dirkgr

  committed
  8952655

  Copy full SHA for 8952655

• Adjust the run script for batch size warmup

  dirkgr

  committed
  573c0c0

  Copy full SHA for 573c0c0

• Typo

  dirkgr

  committed
  14ea8fa

  Copy full SHA for 14ea8fa

• Mitchich 35

  dirkgr

  committed
  750202f

  Copy full SHA for 750202f

• Improved 50

  dirkgr

  committed
  471945a

  Copy full SHA for 471945a

• 35

  dirkgr

  committed
  addac71

  Copy full SHA for addac71

Commits on Nov 28, 2023

• Load the model in turns

  dirkgr

  committed
  6bae22b

  Copy full SHA for 6bae22b

• Merge remote-tracking branch 'origin/main' into mitchish65

  Show description for 4ad3982
  dirkgr

  committed
  4ad3982

  Copy full SHA for 4ad3982

Commits on Nov 30, 2023

• Fix missing import

  dirkgr

  committed
  caafbec

  Copy full SHA for caafbec

• Double the batch size

  dirkgr

  committed
  5ee7fb6

  Copy full SHA for 5ee7fb6

• More nodes

  dirkgr

  committed
  2d5f5d5

  Copy full SHA for 2d5f5d5

• Load twice as fast

  dirkgr

  committed
  b4626ab

  Copy full SHA for b4626ab

• 2T tokens

  dirkgr

  committed
  966ca47

  Copy full SHA for 966ca47

• Load models in a separate thread, so we can keep making NCCL calls that don't time out.

  Show description for 67592d8
  dirkgr

  committed
  67592d8

  Copy full SHA for 67592d8

• Merge remote-tracking branch 'origin/main' into mitchish65

  Show description for 0b8b3dc
  dirkgr

  committed
  0b8b3dc

  Copy full SHA for 0b8b3dc

• Need a device

  dirkgr

  committed
  999a323

  Copy full SHA for 999a323

• That's not how you call reduce

  dirkgr

  committed
  21dd204

  Copy full SHA for 21dd204

• Not sure why positional args don't work

  dirkgr

  committed
  41aa520

  Copy full SHA for 41aa520

• Forgot load_path

  dirkgr

  committed
  beab360

  Copy full SHA for beab360

• Got the wrong timeout error

  dirkgr

  committed
  71ed346

  Copy full SHA for 71ed346

• Safetensors

  dirkgr

  committed
  6ab70cc

  Copy full SHA for 6ab70cc

Commits on Dec 2, 2023

• All ranks load at the same time

  dirkgr

  committed
  aa2301d

  Copy full SHA for aa2301d

• Formatting

  dirkgr

  committed
  bcd41c2

  Copy full SHA for bcd41c2

• Revert "All ranks load at the same time"

  Show description for ac7a80b
  dirkgr

  committed
  ac7a80b

  Copy full SHA for ac7a80b

• Now something else is slow

  dirkgr

  committed
  600437d

  Copy full SHA for 600437d

• I feel like this should have been False all along.

  dirkgr

  committed
  72f5666

  Copy full SHA for 72f5666

• Try restoring checkpoints with summon_full_params()

  dirkgr

  committed
  893d897

  Copy full SHA for 893d897

Commits on Dec 4, 2023

• Fixes how we load from an unsharded checkpoint

  dirkgr

  committed
  834374c

  Copy full SHA for 834374c

• Merge remote-tracking branch 'origin/mitchish65-summon' into mitchish65

  Show description for 2f69e80
  dirkgr

  committed
  2f69e80

  Copy full SHA for 2f69e80

Commits on Dec 5, 2023

• Call GC a bunch while loading

  dirkgr

  committed
  1d86c8b

  Copy full SHA for 1d86c8b

Commits on Dec 6, 2023

• Run longer, and make time limit work

  dirkgr

  committed
  8cb5e7c

  Copy full SHA for 8cb5e7c

• Merge remote-tracking branch 'origin/mitchish65-timelimit' into mitchish65

  dirkgr

  committed
  6e4cbfd

  Copy full SHA for 6e4cbfd

• Try closing and reopening the safetensors file every once in a while

  dirkgr

  committed
  e86e9ba

  Copy full SHA for e86e9ba

• Fix counts

  dirkgr

  committed
  e312b04

  Copy full SHA for e312b04

Commits on Dec 8, 2023

• We need even more time it seems

  dirkgr

  committed
  5452658

  Copy full SHA for 5452658

Commits on Dec 10, 2023

• Checkpoint even more often, because we clearly can't keep a job going for 24h

  dirkgr

  committed
  72bdc96

  Copy full SHA for 72bdc96

Commits on Dec 12, 2023

• Add support for ephemeral checkpoints

  Show description for 00305eb
  epwalsh

  committed
  00305eb

  Copy full SHA for 00305eb

• fix typo

  epwalsh

  committed
  a3e1a56

  Copy full SHA for a3e1a56

Commits on Dec 13, 2023

• fix

  epwalsh

  committed
  1ac7666

  Copy full SHA for 1ac7666

Commits on Dec 14, 2023

• Merge remote-tracking branch 'origin/mitchish65-timelimit' into mitchish65

  dirkgr

  committed
  5ca9a5d

  Copy full SHA for 5ca9a5d

• Merge remote-tracking branch 'origin/epwalsh/ephemeral-checkpoints' into mitchish65

  dirkgr

  committed
  498afaa

  Copy full SHA for 498afaa

• Use the new ephemeral checkpoint feature

  dirkgr

  committed
  45fc06d

  Copy full SHA for 45fc06d

• Restores checkpoints directly into flat params from unsharded checkpoints. But also still contains the old code.

  dirkgr

  committed
  d6dcf90

  Copy full SHA for d6dcf90

• Merge branch 'mitchish65' into mitchish65-flatparamrestore

  dirkgr

  committed
  028d978

  Copy full SHA for 028d978

• Clean up the full checkpoint restorer and make it useful

  dirkgr

  committed
  288ccc5

  Copy full SHA for 288ccc5

Commits on Dec 15, 2023

• Makes checkpoint loading finally work

  dirkgr

  committed
  4935c0f

  Copy full SHA for 4935c0f

Commits on Dec 16, 2023

• Backwards compatibility

  dirkgr

  committed
  933cce4

  Copy full SHA for 933cce4

• We are not yet ready to double the batch size.

  dirkgr

  committed
  a0d480b

  Copy full SHA for a0d480b

Commits on Dec 17, 2023

• Need to clean up the key some more

  dirkgr

  committed
  52bcd3f

  Copy full SHA for 52bcd3f

Commits on Dec 18, 2023

• Keep Pete's fixes

  dirkgr

  committed
  8e7b0fd

  Copy full SHA for 8e7b0fd

Commits on Jan 4, 2024

• Merge remote-tracking branch 'origin/main' into mitchish65

  dirkgr

  committed
  6425924

  Copy full SHA for 6425924

• Remove unused import

  dirkgr

  committed
  fd95100

  Copy full SHA for fd95100

• Productivity through sorted imports

  dirkgr

  committed
  472ea1e

  Copy full SHA for 472ea1e

Commits on Jan 5, 2024

• Adds documentation

  dirkgr

  committed
  f581951

  Copy full SHA for f581951

• Merge branch 'main' into mitchish65

  dirkgr

  authored
  8bd7452

  Copy full SHA for 8bd7452