Call for Evidence: Generative AI third call for evidence: accuracy of training data and model outputs Brief Response
The ICO recently released a call for evidence on the topic of generative AI, and its accuracy of training data and model output. You can read more here. The following is my position statement in response to this call, and the emerging position of the ICO. This is a collective submission with Colleagues from King's College London
Response to Call for Evidence
We strongly support the focus on the accuracy of training data and the necessity for robust reporting and validation of model outputs. However, as highlighted in our discussions, there is a critical need for explicit provenance of the training data and model outputs to ensure transparency and accountability. Adherence to the ICO's fundamental data principles (e.g. data minimisation) should remain paramount, while fostering innovation. To achieve this balance, we propose a simplified regulatory framework that is easy to implement (e.g. a toolkit). Our response to the questions outlined in the Call for Evidence also focus on the importance of public education on the strengths and limitations of these models, ensuring all stakeholders are well-informed.
Consultation question: Do you agree with the analysis presented in this document?
Response: We agree somewhat with the ICO analysis statement. Broadly, the analysis highlights ethical, legal, and moral obligations that all stakeholders need to consider. At present, this isn’t necessarily happening. The ICO's focus on accuracy and statistical accuracy aligns with the wider academic community concern that high quality training datasets, which have a clear data provenance are required (but not available). But also, there is a clear urgent need for generative AI competencies across organisational structures [1], [2]. An important area not directly addressed by the consultation thus far is the need to train developers and stakeholders in understanding and interpreting generative AI outputs correctly. Not just developing the statistical framework, but also how to interpret the results (generative AI competencies).
It is important to recognise the wider community concern that data in the public domain shouldn't automatically be repurposed for generative AI, particularly when it involves manipulating or aggregating data to create similar outputs. A strong emphasis should be placed on a clear consenting pathway and a straightforward explanation on data use are key for end-user comprehension. This is important when most generative AI approaches use end-user input to refine and retrain models ‘on the fly’ without manual review on the quality of these inputs. This includes organisations who use LLM (for example), trained on user inputs, often opaquely returning outputs of likely different models, with undocumented training data. This aspect needs to be acknowledged, understood, and considered in any future ICO position statement.
We agree that it is important for developers to document, record decision and processes. The ongoing Post Office Inquiry has laid bare the disconnect between technical teams and senior management, and this needs to be considered in the context of generative AI accountability. The translatability of technical details to both the public and senior organisational stakeholders is vital to develop trust in generative AI. Furthermore, it stresses the central role of data privacy and the risks posed by insufficient understanding of data management and processes which feed into organisational risk appetite. It must be acknowledged that there is a tension between generative AI and the principle of data minimisation. This could be addressed with thorough consent practices and the careful selection of trusted, albeit potentially biased, data sources.
Consultation question: How can organisations deploying generative AI models effectively communicate to their employees, customers, or other people the extent to which their outputs are statistically accurate?
Response: Organisations (including developers, academic researchers, directors etc) should be required to articulate the extent to which their outputs are statistically accurate within a timely fashion. This includes, using straightforward language, such as through a transparency statement. Statements such as these should be available on websites for customers and translated in other languages. Where internal, it should be integrated into regular training sessions and staff briefings. This approach ensures that both internal and external stakeholders are consistently informed about how AI models function and their level of accuracy. It is also important to consider the models themselves, and provide information on the model version itself. It is also possible to establish routine testing of their AI models to monitor accuracy. The results of these tests should be communicated using clear, unambiguous language to those who 1) use the system, 2) those who have provided data for training (depending on content options). This practice will help in maintaining trust and transparency, providing stakeholders with a clear understanding of the model's performance over time.
The ICO could promote the adoption of a standardised testing framework, supported by a Toolkit, to enhance consistency in reporting AI results. This simplifies understanding AI outputs and aids in industry benchmarking. Alternatively, the industry could be encouraged to develop agreements that include a voluntary framework with peer review and validation elements. Such collaborative efforts also foster the adoption of best practices and continual improvement of AI applications. While ensuring accuracy and transparency is crucial, it is equally important that regulatory measures do not stifle innovation. A light regulatory touch will encourage the ongoing development and ethical use of generative AI, while maintaining compliance.
Consultation question: What technical measures can organisations developing generative AI models use to classify or mark audio, image, or videos outputs as being AI-generated, rather than human-generated?
Response: Generative AI has the potential drive innovation, empower individuals, and increase productivity within the UK. However, it is increasingly difficult to differentiate between human-generated content, and synthetic content generated by AI. It is getting increasingly difficult to identify ‘deep fake’ content versus real content. A recent briefing by the European Parliament identified Watermarking as one viable solution for tagging AI content [3]. We believe this approach has merit, and its adoption by Google [4], Microsoft [5] and Meta [6] indicate strong industry support. This also enables the ability to track provenance of the data.
The intent from these organisations is to develop systems and frameworks which 1) mark the generated content, 2) identify the watermarked signature within the content to identify and trace its source. Watermarking isn’t without issue, and this will be discussed later. Other technical measures have been proposed, such as using an open repository (with data provenance assured), or enabling organisations to query generative AI models to see if content has been generated by its algorithms, but these give rise to further challenges related to privacy and are not viable in a large scale.
Consultation question: What are the benefits and limitations of these methods?
Response: Developing multiple frameworks for validation will hamper any large-scale rollout of a classification framework for content arising from generative AI. Therefore, we believe that the focus should be towards watermarking as the most promising technology. This already has a significant benefit as some of the largest players in the field are working on this technology implementation. There main limitations of watermarking are presented focused on lack of standardisation across the sector, accuracy of watermarking and the algorithmic potential for false-positives and modification where watermarks can be removed or altered.
It is important to recognise that watermarking is only viable for image and video content generated by AI. It is very difficult, if not impossible, to accurately predict, trace and authenticate text written by generative AI. This limitation needs to be acknowledged in any arising guidelines, policy, regulation, or legalisation.
Consultation question: What technical and organisational measures can organisations use to test the extent to which the output of a generative AI model is statistically accurate?
Response: To assess the statistical accuracy of generative AI model outputs, model owners can implement a combination of technical and internal controls. First, employing data validation techniques ensures the training data's quality and relevance. It is important that the source of the data is known, and consent for its processing of generative AI demonstrable. Regular testing and auditing should also be undertaken. There is no viable reason not to undertake these activities. As part of the testing process, the models should be tested against established benchmarks. These benchmarks would preferable be publicly available industry standardised datasets. Statistical methods like cross-validation can further validate the generalisability of the AI model's predictions. Transparent reporting of these processes enhances accountability, while collaborative benchmarking with industry standards promotes continuous improvement and consistency across similar AI applications. Models which are publicly available, the corresponding statistical outputs should also be provided.
Consultation question: What technical and organisational measures can organisations use to improve the statistical accuracy of generative AI models?
To improve the statistical accuracy of generative AI models, model owners can adopt several technical and internal controls. Technically, enhancing data quality through rigorous pre-processing and validation ensures robust training datasets. Incorporating diverse and comprehensive datasets also helps reduce bias. Regularly updating and refining models based on new data and feedback loops can further boost accuracy. At an organisational level, establishing dedicated teams for continuous AI monitoring and implementing strict governance frameworks ensures the integrity and accuracy of AI outputs. Additionally, collaborating with external experts for audits and adherence to industry standards can provide further validation and improvement.
Consultation question: As a developer / supplier of generative AI models, how will the proposed regulatory approach affect your ability to offer services to the UK market?
As a developer or supplier of generative AI models, the proposed regulatory approach in the UK may have significant implications on your service offerings. This regulatory framework will likely require adherence to specific standards and practices, potentially increasing the overhead for compliance and operations. However, it also presents an opportunity to enhance the educational aspect of AI technologies. By integrating educational programmes and developing knowledge bases early in the development process, you can not only comply with regulatory expectations but also add value to your offerings, making your AI solutions more robust and trustworthy. This proactive approach in education and compliance can help maintain a competitive edge in the UK market.
References
[1] F. Fui-Hoon Nah, R. Zheng, J. Cai, K. Siau, and L. Chen, “Generative AI and ChatGPT: Applications, challenges, and AI-human collaboration,” Journal of Information Technology Case and Application Research, vol. 25, no. 3, pp. 277–304, Jul. 2023, doi: 10.1080/15228053.2023.2233814.
[2] E. Brynjolfsson, D. Li, and L. Raymond, “Generative AI at Work,” Cambridge, MA, Apr. 2023. doi: 10.3386/w31161.
[3] T. Madiega, “Generative AI and watermarking,” 2023. [Online]. Available: https://www.europarl.europa.eu/RegData/etudes/BRIE/2023/757583/EPRS_BRI(2023)757583_EN.pdf
[4] S. Gowal and P. Kohli, “Identifying AI-generated images with SynthID - Google Deep Mind.” [Online]. Available: https://deepmind.google/discover/blog/identifying-ai-generated-images-with-synthid/
[5] K. Wiggers, “Microsoft pledges to watermark AI-generated images and videos.” [Online]. Available: https://techcrunch.com/2023/05/23/microsoft-pledges-to-watermark-ai-generated-images-and-videos/
[6] “Meta to introduce watermarking feature for some AI products.” [Online]. Available: https://www.reuters.com/technology/meta-introduce-watermarking-feature-some-ai-products-2023-12-06/