This summer, OpenAI announced escaped a training environment and hacked into the AI company Hugging Face. Anthropic quickly followed with news that its AI agents also .Ěý

Last week, an outgoing Anthropic employee took to X, posting that the “.” Such talk has for years, though many AI experts have argued that these Terminator-esque claims are distractions from the real risks posed by current AI systems. Nevertheless, that viral X thread is .Ěý 

To help make sense of all this, UW News talked to five AI researchers from the şÚÁĎŔĎËľ»ú: 

• , associate professor in the Information School;
• , professor in the Information School;
• , professor in the Paul G. Allen School of Computer Science & Engineering;
• , professor in the Allen School and the UW’s vice provost for AI;
• and , professor in the Information School and the School of Law.

How alarming do you find the hacks announced by OpenAI and Anthropic? 

Franziska Roesner: I do find them somewhat alarming — not due to the hypothetical risks from an anthropomorphized runaway AI, but because complex interconnected systems are being built and seemingly run without much in the way of standard safeguards and auditing. The resulting outcomes are unsurprising to security experts, but are sensationalized as AI risk.

Ryan Calo: The timing makes me a little skeptical. Is OpenAI trying to match Anthropic by arguing that its systems are just as scary? Is Hugging Face trying to look relevant in advance of its purchase by Nvidia? But yes — this sort of emergent behavior is concerning.

Noah A. Smith: We’ve been told that the beast got out of the cage, but we don’t know enough about the cage the beast was in. The demonstrations may establish an important new capability in these AI models without establishing the broader risk people are inferring. Assessing the underlying risk depends on what access, scaffolding, permissions and safeguards the system had. shows that the alarming behavior depended heavily on what tools the model was given, what it was allowed to access, and how the experiment was set up, not just on the model itself.

Chirag Shah: I’m in half-agreement with scholars like who warn that the big AI labs are creating this scare to distract us from real problems that AI is causing. I also concur with and others who have been warning us about the security threats posed by the frontier models. I don’t think these two viewpoints are mutually exclusive: Yes, there are many other potential harms being created by AI, but the hacks and other security issues are real too and could be more devastating. Worse, we may not have time or opportunity to react, fix or reverse.

Aylin Caliskan: When such a complex system is equipped with tools and capabilities that enable it to interact with other complex systems, we should expect unforeseen exploits, problems and unintended consequences by default. The safety of these systems needs to be rigorously evaluated under controlled conditions and in real time, and appropriate guardrails should be dynamically integrated while they’re running.

What do you make of former Anthropic that, “The people building AI earnestly believe that it could kill us all by the end of the decade”?

RC: I worry engineers like Mr. Coxon are playing into an industry rhetoric that would have society focus on speculative, existential threats, rather than immediate, real-world harms. I argued as much in 2023 in .

The people making these claims and announcements largely have financial stakes in these companies, which are . How are you thinking about ulterior motives here?

CS: I see this as an attempt to steer the public into believing these companies are building world-changing tech that everyone needs to invest in or they’d miss out; that this tech would be so powerful that they rise up to national security level and gain power; and that the same tech could also be so dangerous that only they have the ability to curb it and they can self-regulate.

NS: It doesn’t take a conspiracy theorist to note that there are incentives at work. The financial stakes around prospective IPOs are enormous, and there are also long-standing concerns that safety arguments can shape regulation in ways that favor incumbent firms. Rules could reduce competition and independent scrutiny, concentrating both technological power and the authority to define what counts as “safe” in the hands of a few companies. They could also bar many people from participating in what the technology is designed to do, for example, by slowing or stopping work on open-source alternatives.

What should be done about AI risk?

NS: Risks need to be defined based on independent scrutiny and high-quality evidence, not messaging from organizations and people with a stake in what the response to risk looks like. We need sensible liability and accountability for harms, and governance proportional to demonstrated risks in real-world contexts rather than speculative narratives and science fiction. We should be especially wary of rules that entrench incumbent interests or treat closed, centralized control as synonymous with safety.Ěý

Openness is part of safety: If outsiders cannot inspect, reproduce and challenge claims about dangerous behavior, we are left trusting the organizations that have the strongest incentives to frame the narrative.

RC: Some combination of common law liability and regulation needs to create adequate incentives for AI companies to address the inevitable harms of this trillion-dollar industry.Ěý

FR: To me, the bigger question for safety is less, “What can AI models do in isolation?” and more, “How and why are we building these models into increasingly complex systems?” Computer systems security, for example, has already offered us examples of how to build these systems. More generally, we should all — whether we are building, integrating or using AI — anticipate how systems might be misused by people or harm them and adjust our systems accordingly.

AC: Academic freedom, independent evaluation and development, and open science play critical roles in analyzing and mitigating AI risks, as well as in effectively disseminating findings and evidence to inform policy and the public. To better manage risks, we should be designing AI deployment contexts in collaboration with stakeholders and communities, providing evidence to demonstrate net positive deployment effects that do not disproportionately benefit specific entities or groups, and iteratively identifying, isolating, and minimizing risks.

CS: Establish and fund commissions and taskforces that audit these companies and models and make independent assessments and recommendations. Make the companies rolling out these models accountable for any harms caused by their tech. Educate and empower the public through media, policies and democratic frameworks that give them a real say in what happens to their lives and labor through these technologies.

To set up an interview with an AI expert, contact Stefan Milne at stmilne@uw.edu.

Source

The şÚÁĎŔĎËľ»ú today announced a foundational $10 million gift from philanthropists Charles and Lisa Simonyi to support groundbreaking work in artificial intelligence and emerging technologies.

The gift will establish a new initiative, AI@UW, to support the UW’s global leadership in advancing AI, machine learning and related areas of computing. Noah A. Smith, currently the Amazon Professor of Machine Learning in the Paul G. Allen School of Computer Science & Engineering, will become the vice provost for artificial intelligence and the inaugural Charles and Lisa Simonyi Endowed Chair for Artificial Intelligence and Emerging Technologies. The chair appointment is pending Board of Regents approval.

“With this generous gift from Charles and Lisa Simonyi, we will further position the UW as a model for how universities can responsibly and creatively adapt to the age of AI across education, research, administration and governance,” UW Provost Tricia Serio said. “By leading the AI@UW initiative, Vice Provost Noah Smith will guide our efforts to accelerate innovation and collaboration, illuminate achievements, propagate effective practices throughout the UW community and beyond, and ensure that our graduates are prepared for the workforce of today and tomorrow.”

UW researchers and faculty already are globally recognized for cultivating a deep understanding of the science and potential of these rapidly developing technologies. Work at the UW is creating practical and responsible applications for AI that span the academic enterprise, contribute to industry and uplift society.

Charles and Lisa Simonyi have a long history of supporting the UW. Lisa Simonyi is the chair of the UW Foundation Board, and Charles Simonyi is a technical fellow at Microsoft, where he also was a pioneer in developing software applications.

“The future of computing, research and innovations is deeply connected to the next era in artificial intelligence and machine learning,” Lisa and Charles Simonyi said. “We believe in the UW’s ability to engage students and faculty toward discoveries that will transform the university, the region and, indeed, the world. We are pleased to lend our support to advancing this exciting, interdisciplinary field.”

The Charles and Lisa Simonyi gift also will support the creation of an AI governance committee, student scholarships, community engagement and investments in computing resources and equipment.

“This extraordinary gift from the Simonyis demonstrates their vision and deep trust in the UW’s role as a global leader in innovation,” UW President Robert J. Jones said. “It is a foundational investment that will help ensure artificial intelligence is developed and applied responsibly — serving humanity and advancing knowledge in ways that reflect our shared values.”

In the near term, the vice provost for artificial intelligence will establish a SEED-AI grant program to fund projects, led by UW faculty, that elevate the use of AI in UW educational activities. SEED-AI grants will support innovative, exploratory projects aiming to discover how AI can enhance learning and teaching across disciplines, enlighten the UW community, and inspire future developments of AI in the educational context.

Thanks to the Simonyi gift, Smith said, the UW will model how universities can responsibly and creatively adapt to the age of AI across education, research, administration and governance.

“The UW’s people are already leading the way in shaping universities in the time of AI,” Smith said. “While its rapid rise has been surprising, as an AI researcher and teacher I’m energized by the chance to promote AI literacy, explore how AI can enrich learning across disciplines and help steer AI’s development in ways that are most useful to the University’s mission.”

Contact Smith at nasmith@cs.washington.edu.

Source

Language models have, somewhat surreptitiously, dominated news for the last year. Often called “artificial intelligence,” these systems underlie chatbots like ChatGPT and Google Bard.

But a team of researchers at the şÚÁĎŔĎËľ»ú noticed that, even amid a year of AI commotion, many people struggle to find accurate, comprehensible information on what language models are and how they work. News articles frequently focus on the latest advances or corporate controversies, while research papers are too technical and granular for the public. So recently, the team published “,” a paper explaining language models in lay terms.

For answers to some common questions, UW News spoke with lead author , a UW doctoral student in the Paul G. Allen School of Computer Science & Engineering; co-author , a master’s student in the Allen School; and senior author , a professor in the Allen School.

Briefly, what are language models and how do they work?

Sofia Serrano: A language model is essentially a next-word predictor. It looks at a lot of text and notices which words tend to follow after which sequences of other words. Typically, when we’re talking about a language model, we’re now talking about a large machine learning model, which contains a lot of different numbers called parameters. Those numbers are tweaked with each new bit of textual data that the model is trained on. The result is a giant mathematical function that overall is pretty good at predicting which words come next, given the words that have been supplied in a prompt, or that the model has produced so far. It turns out that these large models also pick up things about the structure of language and things that fall under the umbrella of common sense or world knowledge.

In the paper you bring up this idea of the “black box,” which refers to the difficulty in knowing what’s going on inside this giant function. What, specifically, do researchers still not understand?

Noah Smith: We understand the mechanical level very well — the equations that are being calculated when you push inputs and make a prediction. We also have some understanding at the level of behavior, because people are doing all kinds of scientific studies on language models, as if they were lab subjects.

In my view, the level we have almost no understanding of is the mechanisms above the number crunching that are kind of in the middle. Are there abstractions that are being captured by the functions? Is there a way to slice through those intermediate calculations and say, “Oh, it understands concepts, or it understands syntax”?

It’s not like looking under the hood of your car. Somebody who understands cars can explain to you what each piece does and why it’s there. But the tools we have for inspecting what’s going on inside a language model’s predictions are not great. These days they have anywhere from a billion to maybe even a trillion parameters. That’s more numbers than anybody can look at. Even in smaller models, the numbers don’t have any individual meaning. They work together to take that previous sequence of words and turn it into a prediction about the next word.

Why do you distinguish between AI and language models?

SS: “AI” is an umbrella term that can refer to a lot of different research communities that revolve around making computers “learn” in some way. But it can also refer to systems or models that are developed using these “learning” techniques. When we say “language model,” we’re being more specific about a particular concept that falls under the umbrella of AI.

NS: The term “AI” brings with it a lot of preconceived ideas. I think that’s part of why it’s used in marketing so much. The term “language model” has a precise technical definition. We can be clear about exactly what a language model is and is not, and it isn’t going to bring up all these preconceptions and feelings.

SS: Even within natural language processing research communities, people talk about language models “thinking” or “reasoning.” In some respects that language makes sense as shorthand. But when we use the term “thinking,” we mostly know how that works for humans. Yet when we apply that terminology to language models, it can create this perception that a similar process is happening.

Again, a language model is a bunch of numbers in a learned mathematical function. It’s fair game to say that those numbers are capable of recovering or surfacing information that the model has seen before, or finding connections between input text. But often there’s a tendency to go further and make assumptions about any kind of reasoning the models might possess. We haven’t really seen this level of fluency decoupled from other aspects of what we consider intelligence. So it’s really easy for us to mistake fluency for all of the other things that we typically roll into the term “intelligence.”

Could you give an example of how that fluency translates to things that would be perceived as intelligent?

Zander Brumbaugh: I think determining what a display of intelligence is can be quite difficult. For example, if someone asked a model, “I’m struggling and feeling down — what should I do?” The model may offer seemingly reasoned advice. Someone with limited experience with language models might perceive that as intelligence, instead of next-word prediction.

NS: If you tell a model, “I’m having a bad day,” and its response sounds like a therapist, it has likely read a bunch of articles online that coach people on empathy, so it can be very fluent when it’s latching on to the right context. But if it starts feeding on your sadness and telling you you’re awful, it’s probably latching on to some other source of text. It can reproduce the various qualities of human intelligence and behavior that we see online. So if a model behaves in a way that seems intelligent, you should first ask, “What did it see in the training data that looks like this conversation?”

What makes compiling a good data set to train a language model difficult in some instances?

ZB: Today’s models roughly comprise the entire public internet. It takes enormous amounts of resources to be able to gather that data. In language modeling, essentially, what you put in is what you’re going to get out. So people are researching how to best collect data, filter it and make sure that you’re not putting in something that’s toxic or harmful or just at its lowest quality. Those all present separate challenges.

Why is it vital to have testing data that’s not in the original training data set?

NS: I call this the cardinal rule of machine learning. When you’re evaluating a model, you want to make sure that you’re measuring how well it does on something it hasn’t seen before. In the paper, we compare this to a student who somehow gets a copy of the final exam answer key. It doesn’t matter whether they looked at it. Their exam is just not useful in judging whether they learned anything. It’s the same with language models. If the test examples were in the training data, then it could have just memorized what it saw. There’s a large contingent of researchers who see these models as doing a lot of memorization — maybe not perfect memorization, but fuzzy memorization. Sometimes the word “contamination” gets used. If the training data was contaminated with the test, it doesn’t mean the language model is stupid or smart or anything. It just means we can’t conclude anything.

What’s it important for the public to understand about language models right now?

ZB: We need to keep separating language models from notions of intelligence. These models are imperfect. They can sound very fluent, but they’re prone to hallucinations — which is when they generate erroneous or fictional information. I know people who are using language models for something relatively important, such as looking up information. But they give a fuzzy representation of what they’ve learned. They’re not databases or Google search.

NS: If you look at great technological achievements — the airplane or the internet — most resulted from having a clear goal. We wanted to move people through the air, or send information between computers. But just a few years ago, language models were largely research artifacts. A few were being used in some systems, such as Google Translate. But I don’t think researchers had a clear sense of solving a problem by creating a product. I think we were more saying, “Let’s see what happens if we scale this up.” Then, serendipitously, this fluency yielded these other results. But the research wasn’t done with a target in mind, and even now nobody quite knows what that target is. And that’s kind of exciting because some of us would like to see these models made more open because we think there is a lot of potential. But big tech companies have no reason to make a tool that works really well for Sofia or me or you. So the models have to be democratized.

What are some basic steps toward that democratization?

NS: Some organizations are building language models that are open, where the parameters, code and data are shared. I work part-time for one of those organizations, the , but there are others. Meta has put out models, without the data, but that’s still better than nothing. A company called EleutherAI puts out open models. These models are still often quite expensive to run. So I think we need more investment in research that makes them more efficient, that lets us take a big model and make it cheap enough to run on a laptop.

For more information, contact Serrano at sofias6@cs.washington.edu, Brumbaugh at brumbzan@cs.washington.edu and Smith at nasmith@cs.washington.edu.

Source