BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Penn Engineering Events - ECPv6.15.18//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:Penn Engineering Events
X-ORIGINAL-URL:https://seasevents.nmsdev7.com
X-WR-CALDESC:Events for Penn Engineering Events
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20220313T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20221106T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20230312T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20231105T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20240310T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20241103T060000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20230419T120000
DTEND;TZID=America/New_York:20230419T133000
DTSTAMP:20260404T124740
CREATED:20230418T190908Z
LAST-MODIFIED:20230418T190908Z
UID:8952-1681905600-1681911000@seasevents.nmsdev7.com
SUMMARY:ASSET Seminar: Thinking fast with Transformers – Algorithmic Reasoning via Shortcuts (Surbhi Goel\, University of Pennsylvania)
DESCRIPTION:PRESENTATION ABSTRACT: \nIn this new era of deep learning\, the emergent algorithmic reasoning capabilities of Transformer models have led to significant advancements in natural language processing\, program synthesis\, and theorem proving. Despite their widespread success\, the underlying reasons for their efficacy and the nature of their internal representations remain elusive. In this talk\, we take the lens of learning the dynamics of finite-state machines (automata) as the underlying algorithmic reasoning task and shed light on how shallow\, non-recurrent Transformer models emulate these recurrent dynamics. By employing tools from circuit complexity and semigroup theory\, we characterize “shortcut” solutions that allow a shallow Transformer to precisely replicate $T$ computational steps of an automaton with only $o(T)$ layers. We show that Transformers are efficiently able to represent these “shortcuts” using their parameter-efficient ability to compute sparse functions and averages. Furthermore\, through synthetic experiments\, we confirm that standard training successfully discovers these shortcuts. We conclude with highlighting the brittleness of these “shortcuts” in out-of-distribution scenarios. \nThis talk is based on joint work with Bingbin Liu\, Jordan T. Ash\, Akshay Krishnamurthy\, and Cyril Zhang.
URL:https://seasevents.nmsdev7.com/event/asset-seminar-thinking-fast-with-transformers-algorithmic-reasoning-via-shortcuts-surbhi-goel-university-of-pennsylvania/
LOCATION:Levine 307\, 3330 Walnut Street\, Philadelphia\, PA\, 19104\, United States
CATEGORIES:Seminar
ORGANIZER;CN="Computer and Information Science":MAILTO:cherylh@cis.upenn.edu
END:VEVENT
END:VCALENDAR