The shape of a knowledge worker
PREVIOUSLY, ON ED'S RAMBLINGS, I fumbled towards the description of LLMs as a cognitive exponent. But, as with the numerical concept, it turns out that the base of your bx really kinda matters—and at the time I wasn't sure how to even start calculating it.
Good news! I'm still not. But I did run across something that's interesting enough to kick around a little bit.
Jordan Carlson on Bluesky tipped me off to the existence of the OECD's Programme for International Assessment of Adult Competencies (PIAAC), and specifically as to their December 2024 report: "Do Adults Have the Skills They Need to Thrive in a Changing World?" [PDF]. In so doing, Jordan was kind enough to counteract my inherent laziness by pointing me at the definitions of levels of functional literacy that came out of their testing. Which was enough of a hook baited to get me to read more of it.
Jordan undersold it, though. PIAAC does a lot of work. Their investigations span 31 countries and over 160,000 adults. Their studies don't just look at literacy, though that's a major component: their evaluations also cover numeracy and a new-to-2024 category of adaptive problem solving, with each domain measured on a scale of 0 to 500 and grouped into levels of execution competence. And it's in the definitions they give of effective literacy and effective problem solving that I think we—maybe!—have something interesting to kick around.
PIAAC level definitions for literacy Here's a sidenote that will age poorly: I'm experimenting with this tabbed-subdocument thing and apparently my CSS has decided to rebel. I trust you'll get the idea, though.
At Level 2, adults are able to access and understand information in longer texts with some distracting information. They can navigate within simple multi-page digital texts to access and identify target information from various parts of the text. They can understand by paraphrasing or making inferences, based on single or adjacent pieces of information. Adults at Level 2 can consider more than one criterion or constraint in selecting or generating a response.
The texts at this level can include multiple paragraphs distributed over one long or a few short pages, including simple websites. Noncontinuous texts may feature a two-dimension table or a simple flow diagram. Access to target information may require the use of signaling or navigation devices typical of longer print or digital texts. The texts may include some distracting information. Tasks and texts at this level sometimes deal with specific, possibly unfamiliar situations. Tasks require respondents to perform indirect matches between the text and content information, sometimes based on lengthy instructions. Some tasks statements provide little guidance regarding how to perform the task. Task achievement often requires the test taker to either reason about one piece of information or to gather information across multiple processing cycles.
Level 2 (31.4%)
At Level 2, adults are able to access and understand information in longer texts with some distracting information. They can navigate within simple multi-page digital texts to access and identify target information from various parts of the text. They can understand by paraphrasing or making inferences, based on single or adjacent pieces of information. Adults at Level 2 can consider more than one criterion or constraint in selecting or generating a response.
The texts at this level can include multiple paragraphs distributed over one long or a few short pages, including simple websites. Noncontinuous texts may feature a two-dimension table or a simple flow diagram. Access to target information may require the use of signaling or navigation devices typical of longer print or digital texts. The texts may include some distracting information. Tasks and texts at this level sometimes deal with specific, possibly unfamiliar situations. Tasks require respondents to perform indirect matches between the text and content information, sometimes based on lengthy instructions. Some tasks statements provide little guidance regarding how to perform the task. Task achievement often requires the test taker to either reason about one piece of information or to gather information across multiple processing cycles.
Level 3 (30.9%)
Adults at Level 3 are able to construct meaning across larger chunks of text or perform multi-step operations in order to identify and formulate responses. They can identify, interpret or evaluate one or more pieces of information, often employing varying levels of inferencing. They can combine various processes (accessing, understanding and evaluating) if required by the task. Adults at this level can compare and evaluate multiple pieces of information from the text(s) based on their relevance or credibility.
Texts at this level are often dense or lengthy, including continuous, noncontinuous, mixed. Information may be distributed across multiple pages, sometimes arising from multiple sources that provide discrepant information. Understanding rhetorical structures and text signals becomes more central to successfully completing tasks, especially when dealing with complex digital texts that require navigation. The texts may include specific, possibly unfamiliar vocabulary and argumentative structures. Competing information is often present and sometimes salient, though no more than the target information. Tasks require the respondent to identify, interpret, or evaluate one or more pieces of information, and often require varying levels of inferencing. Tasks at Level 3 also often demand that the respondent disregard irrelevant or inappropriate text content to answer accurately. The most complex tasks at this level include lengthy or complex questions requiring the identification of multiple criteria, without clear guidance regarding what has to be done.
Level 4 (10.6%)
At Level 4, adults can read long and dense texts presented on multiple pages in order to complete tasks that involve access, understanding, evaluation and reflection about the text(s) contents and sources across multiple processing cycles. Adults at this level can infer what the task is asking based on complex or implicit statements. Successful task completion often requires the production of knowledge-based inferences.
Texts and tasks at Level 4 may deal with abstract and unfamiliar situations. They often feature both lengthy contents and a large amount of distracting information, which is sometimes as prominent as the information required to complete the task. At this level, adults are able to reason based on intrinsically complex questions that share only indirect matches with the text contents, and/or require taking into consideration several pieces of information dispersed throughout the materials. Tasks may require evaluating subtle evidence-claims or persuasive discourse relationships. Conditional information is frequently present in tasks at this level and must be taken into consideration by the respondent. Response modes may involve assessing or sorting complex assertions.
Level 5 (1.1%)
At Level 5, the assessment provides no direct information on what adults can do. This is mostly because feasibility concerns (especially with respect to testing time) precluded the inclusion of highly difficult tasks involving complex interrelated goal structures, very long or complex document sets, or tools containing highly complex texts (e.g. extensive catalogues, complex menu structures, or lists of unstructured results from search engines), which require advanced skills to access and process the information they contain. These tasks, however, form part of the construct of literacy in today's world, and future assessments aiming at a better coverage of the upper-end of the proficiency scale may seek to include testing units tapping on literacy skills at Level 5.
From the characteristics of the most difficult tasks at Level 4, some suggestions regarding what constitutes proficiency at Level 5 may be offered. Adults at Level 5 may be able to reason about the task itself, setting up reading goals based on complex and implicit requests. They can presumably search for and integrate information across multiple, dense texts containing distracting information in prominent positions. They are able to construct syntheses of similar and contrasting ideas or points of view; or evaluate evidence-based arguments and the reliability of unfamiliar information sources. Tasks at Level 5 may also require the application and evaluation of abstract ideas and relationships. Evaluating reliability of evidentiary sources and selecting not just topically relevant but also trustworthy information may be key to achievement.
PIAAC level definitions for adaptive problem solving
PIAAC doesn't have a Level 5 definition for adaptive problem solving.
Adults at this level can identify and apply solutions that consist of several steps in problems that require considering one target variable to judge whether the problem has been solved. In dynamic problems that exhibit change, adults at this level can identify relevant information if they are prompted to specific aspects of the change or if changes are transparent, occur only one at a time, relate to a single problem feature, and are easily accessible. Problems at this level are presented in well-structured environments and contain only a few information elements with direct relevance to the problem. Minor impasses may be introduced but these can be resolved easily by adjusting the initial problem-solving procedure.
Adults at Level 2 engage in the following cognitive processes:
- develop mental models for simple to moderately difficult problems and adapt these as needed,
- adequately react to changes that are presented in visible increments, and
- adapt resolution strategies to changes in the problem statement and the environment if these changes are of low or moderate cognitive complexity.
Adults at this level engage in the following metacognitive processes:
- monitor progress towards a solution that consists of one specific goal,
- search for optimal solutions by evaluating alternative solution paths within a given problem environment of low to moderate complexity, and
- reflect on the chosen solution strategy if an impasse occurs and when explicitly prompted to adapt.
Level 2 (38.5%)
Adults at this level can identify and apply solutions that consist of several steps in problems that require considering one target variable to judge whether the problem has been solved. In dynamic problems that exhibit change, adults at this level can identify relevant information if they are prompted to specific aspects of the change or if changes are transparent, occur only one at a time, relate to a single problem feature, and are easily accessible. Problems at this level are presented in well-structured environments and contain only a few information elements with direct relevance to the problem. Minor impasses may be introduced but these can be resolved easily by adjusting the initial problem-solving procedure.
Adults at Level 2 engage in the following cognitive processes:
- develop mental models for simple to moderately difficult problems and adapt these as needed,
- adequately react to changes that are presented in visible increments, and
- adapt resolution strategies to changes in the problem statement and the environment if these changes are of low or moderate cognitive complexity.
Adults at this level engage in the following metacognitive processes:
- monitor progress towards a solution that consists of one specific goal,
- search for optimal solutions by evaluating alternative solution paths within a given problem environment of low to moderate complexity, and
- reflect on the chosen solution strategy if an impasse occurs and when explicitly prompted to adapt.
Level 3 (27.3%)
Adults at this level understand problems that are either more complex static problems or problems that have an average to high level of dynamics. They can solve problems with multiple constraints or problems that require the attainment of several goals in parallel. In problems that change and require adaptivity, adults deal with frequent and, to some extent, continuous changes. They discriminate between changes that are relevant and those that are less relevant or unrelated to the problem.
Adults at this level can identify and apply multi-step solutions that integrate several important variables simultaneously and consider the impact of several problem elements on each other. In dynamically changing problems, they predict future developments in the problem situation based on information collected from past developments. They adapt their behaviour according to the predicted change.
Adults at Level 3 engage in the following cognitive processes:
- generate mental models for moderately to highly complex problems,
- actively search for solutions by continuously evaluating the information provided in the problem environment, and
- distinguish between relevant and irrelevant information.
Adults at this level engage in the following metacognitive processes:
- monitor comprehension of the problem and the changes in the problem,
- monitor and evaluate progress towards the goal of the problem,
- search for solutions by setting sub-goals and evaluating alternative solutions to the problem, and
- reflect on their approach to solving the problem and, if necessary, revise their strategy.
Level 4 (5.0%)
Adults at this level are able to define the nature of problems in ill-structured and information-rich contexts. They integrate multiple sources of information and their interactions, identify and disregard irrelevant information, and formulate relevant cues.
Adults identify and apply multi-step solutions towards one or more complex goals. They adapt the problem-solving process to changes even if these changes are not obvious, occur unexpectedly, or require a major reevaluation of the problem. Adults are able to distinguish between relevant and irrelevant changes, predict future developments of the problem situation, and consider multiple criteria simultaneously to judge whether the solution process is likely to lead to success.
Adults at Level 4 engage in the following cognitive processes:
- develop complex mental models of problems by integrating information from multiple sources,
- establish connections between tasks and stimuli even if these connections are difficult to detect or contain complex interactions,
- develop strategies to reach several goals in parallel and implement multi-step solutions, and continuously update their mental model, search strategies, and solutions during problem solving.
Adults at this level engage in the following metacognitive processes:
- continuously reflect and monitor the problem-solving process even if the environment is complex and changes unexpectedly,
- constantly revisit and reevaluate their mental model, the available information, and goal attainment,
- show adequate and immediate reactions to change, and
- cope with frequent and unpredictable change and adapt their solution strategy accordingly
So if you've given these a quick skim, you can probably guess why I called this an LLM-related post. The higher levels of these descriptions sound like an LLM operations manual. "Define the nature of problems in ill-structured and information-rich contexts." "Integrate multiple sources of information and their interactions." "Identify and disregard irrelevant information." "Adapt the problem-solving process to changes even if [...] changes are not obvious, occur unexpectedly, or require a major reevaluation of the problem."
Or try this one on for size: "Continuously reflect and monitor the problem-solving process," while "constantly revisit[ing] and reevaluat[ing] [our] mental model, available information, and goal attainment." I mean—this isn't something abstractly related to prompt and context engineering, that's just what this is. It's the exact skillset that most of my coworkers and 99% of Bluesky are tired of hearing me go on about.
PIAAC's report also counterintuitively mentions that the levels tested-for don't strongly correlate in individuals; someone scoring very high in literacy doesn't suggest a high score in adaptive problem solving. I think this is an interesting lens through which to regard the more successful cohorts of LLM users that I see in my misadventures I try very hard to draw the line between prodigious uses of LLMs (ones that output a lot of text that somebody has to slog through) and effective uses of LLMs (outputs that match the requirements and are designed to be validatable). The former is not success: reading and evaluating the thing, applying criticism, and feeding one's own analysis back into the machine is part of adaptive problem solving.. Let me pull from my Cognitive Exponents piece:
where I've actually found the biggest immediate jump is in staff engineers who code because they like to code, who can communicate clearly and are comfortable moving up and down the stack.
And I had this in mind when I tried to come up with a clear definition of what an LLM actually is:
an LLM is a semi-autonomous tool for working with labeled, coherent information at a continuously variable level of abstraction.
The mental picture picture of the person I run into who capital-G Gets the definition laid out above, who is Having A Great Time with LLM tools and is Getting Things Done, is a mid- to late-career staff engineer The use of "engineer" is not an endorsement here. I'm not an engineer, I'm a software developer. But the industry calls me an engineer. Sorry.. They're not necessarily an amazing writer, but they read and they read effectively, both in terms of extracting meaning from words but shortcutting around what parts matter and what parts don't Which is to say, if I give them a doc with a table of contents, I don't have to then go "and pages 6-8 are relevant here".; we're talking about what PIAAC classifies as Level 3-ish, roughly top half of the distribution. But we start talking about top-third, Level 3 (this time 3 out of 4, remember), for adaptive problem-solving. Sometimes I use the phrase "theory of mind" as shorthand, but you can expand that just fine to "generating mental models for moderately to highly complex problems" and "monitor comprehension of the problem and the changes in the problem", where "the problem" is defined as "what the hell is that giant pile of matrix math doing to my document/codebase/whatever?".
These aren't new skillsets, of course. If anything, they're new problems being stacked on top of the old one; interpreting the large language model is an and not an instead of understanding business requirements and translating them into systems, modules, and contracts. (But, of course, the payoff is accelerating the process of chewing on all of that stuff, so I think it's worth it.)
Almost invariably, too, those are the staff engineers who practice systems thinking as if by habit. The book dorks (hi) among us talk about stuff like Thinking in Systems, Normal Accidents, A Pattern Language, or at least know what we're nodding at when they come up. And I think this is a relevant commonality, too. They're about understanding systems, but in so understanding they acknowledge that you can't avoid abstraction. You might slide down that ladder Sorry, Tim Rogers, cool people slide down ladders. when the implementation matters to the system as a whole or when you need to rationalize the behavior and the "worldview" of two disparate subsystems, but most of the time? You're looking at the box with the label, eyeing the hose ports and plugs coming off of it, and figuring out what needs to connect where Software development, being a matryoshka doll of systems, means that this often holds when you're not talking about the whole, too; most folks (for better or worse) aren't exactly eyeballing every line of every dependency they pull in, either..
The wielding of abstractions and systems thinking is the straw that stirs the LLM drink, too. New context, same skill. (And, relatedly, the drive to put LLMs in every set of hands in my profession, whether or not they're exhibiting these skills, keeps me up at night.)
I see people, smart on many axes who don't succeed with these, too. When discussing cognitive exponents, I had to contrast it with "cognitive amplifiers", which are almost a failure case of LLMs: amplifiers amplify everything. Signal? It gets bigger. Noise? That gets bigger, too. And you often see this showing up a couple different ways. One is insufficient self-criticism to see whether or not those outputs are actually any good. The other is more of a quitting kind of it. "It hallucinates all the time." "Spicy autocomplete." The assumption that an LLM is an oracle or it's useless, that determinism is the only way to get something useful out the other end Remind me again how deterministic a human being is?, the rejection of the idea that a system with probabilistic elements can be guided and can have correcting functions layered on top of them to produce consistent results—this is, if you squint, a different failure mode of adaptive thinking.
As with the discussion of cognitive exponents, I'm not here to give you a One Weird Trick for finding these people. And I have no idea if it's something that can be trained in situ rather than made as the product of long experience of doing stuff in the world (whether with a computer or not). But the reason I wanted to write briefly Briefly for me, anyway. about the PIAAC stuff is I think that their definitions port pretty cleanly as a lodestar for what we want and where we want people to go in order to succeed with these tools and in the weird new working world that they're opening up. If nothing else, the PIAAC assessment provides a naming-of-parts for the prerequisite skills that I can get behind, and maybe from there there's a way to start thinking about how to make work-sample tests or other ways to find what we're looking for.
–Ed