American and Chinese AI models: different companies, surprisingly similar data pipelines

The public discussion around artificial intelligence often treats American and Chinese models as if they come from completely different technological worlds. Politically and legally, they do operate under different jurisdictions. Their companies have different ownership structures, regulatory obligations, cloud environments and terms of service. Those differences matter.

But when we look at how the models themselves are trained, the broad pattern is strikingly similar. OpenAI, Anthropic, Google, Meta, Alibaba and DeepSeek all rely on some mixture of large public datasets, licensed or partner material, human-created examples, synthetic data and extensive post-training. The details vary, but the core industrial process is recognisable across borders.

For Māori organisations, this means “American model” versus “Chinese model” is not a sufficient risk framework. The better questions are what data sources were used, how transparent the developer is, what product is being used now, where current prompts are processed, what jurisdiction applies, and whether the model can be run locally without sending new Māori information back to the developer.

OpenAI

OpenAI describes three broad sources for foundation-model development: information freely and openly accessible on the internet, information obtained through partnerships with third parties, and information supplied or generated by users, human trainers and researchers. Synthetic data is increasingly important as well.

OpenAI has historically said that large public-web collections such as Common Crawl and Wikipedia were major sources for its earlier model lineage. GPT-3’s published model card provided unusually concrete numbers, including filtered Common Crawl, WebText, books and Wikipedia. Current frontier models do not come with the same URL-level transparency.

That creates a limitation for Māori research. We can say that Māori webpages were technically available through public-web pipelines and that some are visible in Common-Crawl-derived corpora. We cannot inspect a current OpenAI training manifest and prove that a named iwi page or Tribunal report was used in a particular current model.

See OpenAI’s foundation-model development explanation and GPT-3 model card.

Anthropic

Anthropic describes Claude models as trained on a mixture of publicly available internet information, non-public third-party data, material from paid contractors and data-labelling services, internally generated material and, for some current products, data from users who have deliberately allowed model improvement.

Anthropic’s public transparency is strong on model behaviour, safety and broad source categories, but it does not publish a complete list of webpages or domains used in current model training.

For Māori organisations, the practical consequence is similar to OpenAI. Public Māori material may plausibly have been available to the training pipeline, but individual-source claims need evidence. Consumer and commercial Claude products then have different present-day data-use terms.

See Anthropic’s Transparency Hub.

Google Gemini

Google’s original Gemini technical material describes a multimodal pre-training collection containing web documents, books, code, images, audio and video. Modern Gemini models build on this approach, with increasing emphasis on multimodal capability and extremely large context windows.

Google has access to enormous amounts of public web data through its search and crawling infrastructure, but that fact alone should not be converted into an unsupported claim that Google uses every indexed page for Gemini training. The company’s model cards and technical reports describe source classes rather than complete domain manifests.

For Māori material, the important point is that text is only one channel. Modern multimodal training can include images, audio and video as well. That broadens the sovereignty conversation beyond documents.

A Māori-language speech recording, an image containing text, a scanned map and a video can all potentially become machine-readable inputs to modern AI systems if they enter the relevant training pipeline.

See the Gemini technical report.

Meta Llama

Meta has been unusually open about scale. It says Llama 3 was pre-trained on more than 15 trillion tokens from publicly available sources. More than five percent of the pre-training material was described as high-quality non-English data spanning more than 30 languages.

That sounds substantial until scale is considered. Five percent of 15 trillion tokens is still enormous, but it has to cover many languages. Smaller languages remain a tiny proportion of the whole corpus.

Meta also stated that Llama 3’s training data did not include Meta user data. That distinction is useful because it shows why “the company owns Facebook” does not automatically tell us what was used to train a particular open-weight model.

The larger issue for Māori organisations is that Llama models can be downloaded and run locally. Historical pre-training remains externally determined, but new prompts and documents do not have to be sent back to Meta if the model is operated in a genuinely local environment.

See Meta’s Llama 3 announcement.

Alibaba Qwen

Alibaba’s Qwen disclosures are particularly useful because they reveal how modern training pipelines are moving beyond ordinary webpage scraping.

Qwen3’s developers describe a corpus of roughly 36 trillion tokens across 119 languages and dialects. The pipeline included ordinary web material but also PDF-like documents. Alibaba says vision-language models were used to extract text from PDF-style sources and further models were used to improve the extracted content.

That matters in Aotearoa because much of the most detailed Māori-related public material is distributed as PDFs rather than webpages. Waitangi Tribunal reports, submissions, planning documents, iwi environmental plans, research reports and academic theses are obvious examples.

The Qwen disclosure does not prove that any one of those collections was included. It does prove that modern training pipelines are technically capable of ingesting document forms that older web corpora sometimes handled poorly.

Alibaba’s September 2026 training-data disclosure describes Qwen and Wan models as using public internet information, non-public third-party data, paid or labelled data and synthetic data. It also acknowledges that training datasets can contain copyright-protected material.

See Qwen3 and Alibaba Cloud’s Qwen and Wan training-data disclosure.

DeepSeek

DeepSeek describes pre-training using public internet information and licensed third-party data, followed by optimisation and post-training using constructed, annotated and synthetic material.

DeepSeek-V3 was trained on 14.8 trillion tokens. Its documentation also distinguishes pre-training from inference and describes pathways under which a limited proportion of user interactions may be used for improvement after de-identification, with opt-out arrangements.

Technically, that looks familiar. Large web-scale pre-training, curated or licensed sources, synthetic data and substantial post-training are not uniquely American or Chinese approaches. They are now standard methods across the industry.

What differs more significantly is jurisdiction, corporate governance, contractual protections, regulatory environment and what happens to current user information in the product being used.

See DeepSeek’s model algorithm disclosure and DeepSeek-V3 information.

Scale changes the meaning of inclusion

A model trained on 15, 20 or 36 trillion tokens can contain Māori material while still having weak Māori capability.

This is important because public debate sometimes assumes that if a model knows anything about Māori, it must have ingested an enormous Māori corpus. The opposite assumption is also made: if the model makes mistakes in te reo Māori, perhaps Māori data were absent.

Neither inference is reliable.

A relatively small corpus can still expose a model to many names, phrases and associations. But those patterns may be overwhelmed by much larger English-language and other-language distributions. Tokenisation can also handle smaller languages less efficiently. Post-training may contain few high-quality Māori examples. Evaluation may be weak. The result can be a model that has seen Māori material but does not handle it consistently.

That is why extraction and underrepresentation can coexist.

Public web data is culturally selective

The other problem is not just how much data each model gets. It is which Māori information is easiest to acquire.

Public-web training favours material already made machine-readable. Government websites, published reports, news media, academic articles, legislation and official datasets are easier to collect than locally held knowledge, oral history, restricted archives or material that has never been digitised.

This can produce a documentary bias in the resulting model. A model may know the Crown’s published account of a place better than the locally held account. It may know modern official spelling better than historical or customary naming. It may encounter nineteenth-century ethnography many times while never encountering the contemporary kōrero that would contextualise it.

That is not necessarily a deliberate political choice by a model developer. It can be an emergent consequence of the data pipeline.

For Māori GIS, this matters because official spatial data are especially machine-readable. Cadastral boundaries, place-name databases, published maps and government datasets can dominate what an AI system appears to “know” about geography.

Open-weight changes current control, not historical provenance

Chinese and US developers alike now release some models with downloadable weights. Meta Llama, Alibaba Qwen, Google Gemma, Mistral models and others can be run on local infrastructure.

That creates an important opportunity. An iwi can use a model whose historic pre-training was assembled internationally while still preventing new internal documents from being sent back to the developer.

The organisation gains control over current inference without pretending it controls the historic training corpus.

This is a more realistic concept of model sovereignty. Complete sovereignty over a foundation model would require control of the original training data, training compute, weights, post-training and deployment environment. Most organisations will not build that from scratch.

But meaningful control can still be gained over the deployment layer, document layer, access controls and derived outputs.

Jurisdiction remains real

None of this means Chinese and US services should be treated as interchangeable.

A public DeepSeek or Qwen service can be subject to Chinese law, corporate structures and data-handling arrangements. OpenAI, Anthropic, Google and Microsoft operate under US and other jurisdictions and have their own law-enforcement, contractual and subprocessor environments. New Zealand organisations may also have public-sector, privacy, records-management or security obligations affecting which jurisdictions are acceptable.

The point is narrower: jurisdiction is one layer of the decision. It should not be confused with the technical question of how a foundation model was trained.

A locally downloaded Qwen model on a disconnected iwi workstation has a different current data pathway from using Qwen’s consumer cloud service. Likewise, a local Llama model differs from using Meta’s online products.

Product architecture matters as much as model nationality.

Transparency is uneven

No major frontier developer provides perfect transparency about training data.

Some disclose source classes. Some provide token counts. Some identify language coverage. Some describe filtering and PDF extraction. Some publish model cards and evaluation results. Few provide complete source inventories, and even fewer provide a practical mechanism for Indigenous communities to determine whether a particular item influenced a model.

That limits what Māori organisations can establish retrospectively.

The best response is to be disciplined about evidence. Distinguish what the developer explicitly discloses from what can be demonstrated in public training corpora and from what is merely plausible based on the architecture.

That is more useful than treating either US or Chinese companies as uniquely transparent or uniquely opaque.

What to compare instead of flags

When choosing a model or service, ask how it was trained to the extent this is disclosed, whether it can be run locally, what licence applies, how well it performs for the actual Māori and GIS tasks, where current prompts are processed, whether customer material is used for training, what retention applies, which jurisdiction governs the service, what tools or plugins create additional data flows, and whether the organisation can verify the architecture itself.

Those questions produce a much richer comparison than “American versus Chinese AI”.

For some tasks an enterprise US service may be appropriate. For others a locally run Chinese open-weight model may provide stronger practical control. For still others the best answer may be no AI at all.

The kaupapa should determine the architecture, not the flag attached to the model developer.

The next article looks at a very New Zealand part of this story: What does Southland’s proposed AI factory actually mean?.