BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Penn Engineering Events - ECPv6.16.3//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:Penn Engineering Events
X-ORIGINAL-URL:https://seasevents.nmsdev7.com
X-WR-CALDESC:Events for Penn Engineering Events
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20240310T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20241103T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20250309T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20251102T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20260308T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20261101T060000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20251110T143000
DTEND;TZID=America/New_York:20251110T153000
DTSTAMP:20260601T201811
CREATED:20251031T181243Z
LAST-MODIFIED:20251031T181243Z
UID:15058-1762785000-1762788600@seasevents.nmsdev7.com
SUMMARY:FOLDS SEMINAR: The Hidden Width of Deep ResNets
DESCRIPTION:Zoom link: https://upenn.zoom.us/j/6130182858 \n  \nWe present a mathematical framework to analyze the training dynamics of deep ResNets that rigorously captures practical architectures (including Transformers) trained from standard random initializations. Our approach combines stochastic approximation of ODEs with propagation-of-chaos arguments to obtain tight convergence rates to the “infinite size” limit of the dynamics. It yields the following insights:\n1/ Depth begets width: infinite-depth ResNets of any hidden width behave throughout training as if they were infinitely wide;\n2/ Phase diagram: we derive the phase diagram of the training dynamics\, which singles out an “ideal” scaling of hyper-parameters (initialization scale and learning-rates)\, extending “CompleteP” to more general architectures;\n3/ Optimal shape scaling: our analysis suggests how to scale depth\, hidden width and embedding dimension of a ResNet when scaling up parameter count. With the optimal shape and a parameter budget P\, we argue that the model converges to its limiting dynamics at rate P^{-1/6}.
URL:https://seasevents.nmsdev7.com/event/folds-seminar-the-hidden-width-of-deep-resnets/
LOCATION:Amy Gutmann Hall\, Room 414\, 3333 Chestnut Street\, Philadelphia\, 19104\, United States
CATEGORIES:Seminar,Colloquium
END:VEVENT
END:VCALENDAR