BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Penn Engineering Events - ECPv6.15.18//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:Penn Engineering Events
X-ORIGINAL-URL:https://seasevents.nmsdev7.com
X-WR-CALDESC:Events for Penn Engineering Events
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20230312T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20231105T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20240310T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20241103T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20250309T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20251102T060000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20240215T153000
DTEND;TZID=America/New_York:20240215T163000
DTSTAMP:20260403T191256
CREATED:20240129T171413Z
LAST-MODIFIED:20240129T171413Z
UID:10578-1708011000-1708014600@seasevents.nmsdev7.com
SUMMARY:CIS Seminar: "Accessible Foundation Models: Systems\, Algorithms\, and Science"
DESCRIPTION:The ever-increasing scale of foundation models\, such as ChatGPT and AlphaFold\, has revolutionized AI and science more generally. However\, increasing scale also steadily raises computational barriers\, blocking almost everyone from studying\, adapting\, or otherwise using these models for anything beyond static API queries. In this talk\, I will present research that significantly lowers these barriers for a wide range of use cases\, including inference algorithms that are used to make predictions after training\, finetuning approaches that adapt a trained model to new data\, and finally\, full training of foundation models from scratch.  For inference\, I will describe our LLM.int8() algorithm\, which showed how to enable high-precision 8-bit matrix multiplication that is both fast and memory efficient. LLM.int8() is based on the discovery and characterization of sparse outlier sub-networks that only emerge at large model scales but are crucial for effective Int8 quantization. For finetuning\, I will introduce the QLoRA algorithm\, which pushes such quantization much further to unlock finetuning of very large models on a single GPU by only updating a small set of the parameters while keeping most of the network in a new information-theoretically optimal 4-bit representation. For full training\, I will present SWARM parallelism\, which allows collaborative training of foundation models across continents on standard internet infrastructure while still being 80% as effective as the prohibitively expensive supercomputers that are currently used. Finally\, I will close by outlining my plans to make foundation models 100x more accessible\, which will be needed to maintain truly open AI-based scientific innovation as models continue to scale.
URL:https://seasevents.nmsdev7.com/event/cis-seminar-accessible-foundation-models-systems-algorithms-and-science/
LOCATION:Wu and Chen Auditorium (Room 101)\, Levine Hall\, 3330 Walnut Street\, Philadelphia\, PA\, 19104\, United States
ORGANIZER;CN="Computer and Information Science":MAILTO:cherylh@cis.upenn.edu
END:VEVENT
END:VCALENDAR