Trustworthy Artificial Intelligence and Public Procurement

See Oluka Nagitta et al. (2022) and Sánchez Graells (2022a).

McBride et al. (2021), p. 11.

In Spain, see Estrategia Nacional de Inteligencia Artificial, according to which Public Administration may ‘act as a demander of technology solutions, thereby driving the development of technology solutions, thereby driving the development of new programmes and capabilities by the private sector’.

Available in https://​www3.​weforum.​org. See also WEF (2020).

McBride et al. (2021).

Documentation available, only partly in English at Instructional materials, Krattide veebileht (https://​www.​kratid.​ee/​) translation.

Guidelines for AI Procurement (2020), available in Guidelines for AI procurement - GOV.​UK (www.​gov.​uk).

Available in Contractual terms for algorithms - Innovation (amsterdam.​nl).

A European approach to excellence and trust. Brussels, 19.2.2020. COM(2020) 65 final.

EUROPEAN COMMISION (2023) Proposal for standard contractual clauses for the procurement of Artificial Intelligence (AI) by public organisations. High Risk version. Available in https://​living-in.​eu/​groups/​solutions/​ai-procurement, A new version has been published in September 2023, available in https://​public-buyers-community.​ec.​europa.​eu/​communities/​procurement-ai/​resources/​eu-model-contractual-ai-clauses-pilot-procurements-ai.

Brussels, 21.4.2021 COM(2021) 206 final.

Available partially in English in Instructional materials | Krattide veebileht (https://​www.​kratid.​ee/​).

Guidelines for AI Procurement (2020) available in Guidelines for AI procurement - GOV.​UK (www.​gov.​uk).

McBride et al. (2021), p. 11.

See Article 40 Directive 2014/24/EU of the European Parliament and of the Council of 26 February 2014 on public procurement and repealing Directive 2004/18/EC (OJ L 94, of 28.3.2014).

See Commission notice, Guidance on Innovation Procurement, Brussels, 15.5.2018, C(2018) 3051 final.

See British Guidelines for AI procurement, p. 26. See also WEF (2019) according to which: ‘pre-market engagement is also often essential in helping you to describe your problem and narrow down the task that AI may be able to assist with’ (p. 8).

WEF (2019), p. 9.

See a specific AI risk assessment tool: WEF (2020).

WEF (2019), p. 8.

WEF (2019), p. 8.

WEF (2019), p. 10.

UK (2020): Guidelines for AI Procurement.

UK (2020): Guidelines for AI Procurement.

See Article 72 Directive 2014/24/EU.

See Article 41 EU Charter of Fundamental Rights.

WEF (2019), pp. 13–15.

McBride et al. (2021), p. 11.

Furthermore, access to the source code may also have implications from the perspective of compliance with public procurement rules: the absence of assignment of intellectual property rights may mean that extensions and developments not included in the initial object of the contract must be carried out by the same contractor may entail a breach of the principle of free competition. On this topic, see Sánchez Graells (2022b).

At least, Spanish authors. See Boix Palop (2020), pp. 223–270; Gómez Jiménez (2021), p. 150.

Martín Delgado (2023).

Martín Delgado (2023). See also Ponce Solé (2019).

UK (2020), Guidelines for AI procurement. A summary of best practices addressing specific changes of acquiring Artificial Intelligence in the public sector, p. 29.

‘Article 5 Transparency of the Algorithmic System. (5.1) On demand of NAME OF ORGANIZATION, the Contractor will provide NAME OF ORGANIZATION with Procedural Transparency. NAME OF ORGANIZATION will be entitled to share the information provided by the Contractor in that context with third parties and to disclose it. On demand of NAME OF ORGANIZATION, the Contractor will complete a register for Algorithmic Systems to be designated by NAME OF ORGANIZATION. (5.2) On demand of NAME OF ORGANIZATION, the Contractor will provide NAME OF ORGANIZATION with Technical Transparency in order to enable NAME OF ORGANIZATION to carry out an audit as referred to in article 8. NAME OF ORGANIZATION will only request and use such information if and to the extent necessary for purposes of article 8. NAME OF ORGANIZATION will, pursuant to article 5.2, keep business-confidential information provided to it confidential and destroy it after an audit as referred to in article 8, unless a legal obligation on the part of NAME OF ORGANIZATION opposes confidentiality or destruction or NAME OF ORGANIZATION needs the information in the context of a dispute with the Contractor or a third party. (5.3) For purposes of article 5.2, the Contractor may choose not to issue the source code of the Algorithmic System to NAME OF ORGANIZATION, but to an independent third party to be designated and engaged by NAME OF ORGANIZATION who will perform the audit referred to in article 8 on behalf of NAME OF ORGANIZATION. Any additional costs incurred as a result will be payable by the Contractor. NAME OF ORGANIZATION may require that the Contractor pay an advance in connection with the costs of the independent third party. (5.4) NAME OF ORGANIZATION should at all times be able to explain the operation of the Algorithmic System (Explainability). The Contractor will be under the obligation to lend its full cooperation in making the Algorithmic System Explainable and to provide NAME OF ORGANIZATION with all such information as may be required in that respect. NAME OF ORGANIZATION will be entitled to share the information provided by the Contractor in that context with third parties and to disclose it. (5.5) During the term of the Agreement, the demands described in this article will constitute obligations to realize those demands and, unless the Parties agree otherwise, no additional fee will be due by NAME OF ORGANIZATION to the Contractor in consideration of the performance of such obligations. After expiry of the term of the Agreement, the demands described in this article will constitute an obligation to perform to the best of the Parties’ ability and an additional fee will be due by NAME OF ORGANIZATION to the Contractor in consideration of the services to be provided by the Contractor in that respect. (5.6) In all situations where the information provided on the basis of the various paragraphs of this article overlaps, NAME OF ORGANIZATION will be free to choose the regime that is most favourable to it’.

As it explained in the document, those decisions show that, if an administrative body makes an administrative decision using an Algorithmic System, that administrative body must be able to explain how an Algorithmic System has arrived at a particular result. Furthermore, the administrative body must offer the other party the opportunity to assess the operation of the Algorithmic System, so that realistic legal protection can be offered to the other party.

‘Article 6 Transparency. (6.1.) The Supplier ensures that the AI System shall be designed and developed in such a way to ensure that the operation of the AI System is sufficiently transparent to enable [NAME OF PUBLIC ORGANISATION] to interpret the system’s output and use it appropriately. (6.2.) <optional> During the term of the Agreement, the Supplier is obliged to assist [NAME OF PUBLIC ORGANISATION] at [NAME OF PUBLIC ORGANISATION]’s first request to explain on an individual level how the AI System arrived at a particular decision or outcome. In any event, this will include a clear indication of the key factors that led the AI System to arrive at a particular result and the changes to the input that must be made in order for it to arrive at a different outcome. (6.3.) <optional> The obligation as described in article 6.2 comprises in any event the provision to [NAME OF PUBLIC ORGANISATION] of all the technical and other information required in order to explain how the AI System arrived at a particular decision or outcome and to offer the other party and any other interested parties the opportunity to verify the way in which the AI System arrived at a particular decision or outcome. The Supplier hereby grants the [NAME OF PUBLIC ORGANISATION] the right to use, share and disclose this information, if and to the extent necessary to inform natural persons about the functioning of the AI System and/or in any legal proceedings. (6.4.) <optional> Without prejudice to Article 4, the obligations referred to in article 6.2 and article 6.3 include the source code of the AI System, the technical specifications used in developing the AI System, the Data Sets, technical information on how the Data Sets used in developing the AI System were obtained and edited, information on the method of development used and the development process undertaken, substantiation of the choice for a particular model and its parameters, and information on the performance of the AI System’.

According to Article 11 i) RD 203/2021, Public administration shall report the description of the design and functioning of the automated administrative action, the accountability and transparency mechanisms, as well as the data used in its configuration and learning.

According to Spanish Law 40/2015, of 1 October, on the Legal Regime of the Public Sector (Article 41), in the case of automated administrative action, the competent body or bodies, as the case may be, for the definition of the specifications, programming, maintenance, supervision and quality control and, where appropriate, auditing of the information system and its source code, shall be established beforehand.

An automated administrative action is understood to be any act or action carried out entirely by electronic means by a Public Administration within the framework of an administrative procedure and in which a public employee has not intervened directly.

Cerrillo I Martínez (2019), p. 296.

Valero Torrijos (2019), p. 91.

AI HLEG (2019), pp. 14–19.

‘Article 19 Compliance and Audit. (19.1.) At first request of the Public Organisation, the Suppliers must make available to the Public Organisation all information necessary to demonstrate compliance with these Clauses. (19.2.) The Supplier is obliged to cooperate in an audit or other type of inspection to be carried out by or on behalf of the Public Organisation to assess whether the Supplier complies with its obligations laid down in these Clauses at all times. Such cooperation will include providing all information required by the Public Organisation, providing an insight into the risk management system implemented, making Supplier staff available for interviews and providing access to the locations of the Supplier. (19.3.) The Public Organisation will prepare, or cause the preparation of, a report in which the conclusions of the audit are recorded. In the report, the Public Organisation will record the extent to which the Supplier complies with the obligations under the Agreement. If the Public Organisation establishes that the Supplier does not comply with the obligations under this article, the Supplier will be obliged to remedy the defects identified by the Public Organisation within the reasonable term set by the Public Organisation in the report. If the Supplier fails to remedy the defects identified by the Public Organisation within the term set in the report for remedying such defects, the Supplier will be in default by operation of law. (19.4.) The Public Organisation will be entitled to publish the conclusions of the report referred to in article 19.3. (19.5.) The Public Organisation will be entitled to perform, or cause the performance of, an audit once per calendar year or in relation to any Substantial Modification. (19.6.) The Public Organisation may decide to have all or part of the audit performed by an independent auditor. (19.7.) The costs of the auditor to be engaged by the Public Organisation, if any, will be paid by the Public Organisation. The Public Organisation will pay the Supplier a reasonable fee for any costs to be incurred by the Supplier in the context of the audit. In no event will a dispute about the amount of such fee give the Supplier the right to suspend its obligations under these Clauses. No such fee will be owed by the Public Organisation if the audit reveals that the Supplier has failed to perform its obligations under these Clauses’.

Available in Auditing machine learning algorithms (auditingalgorith​ms.​net).

Laws and regulations include the General Data Protection Regulation.

‘Article 2 Data quality. (2.1) If and to the extent that the Algorithmic System is developed on the basis of data provided by NAME OF ORGANIZATION to the Contractor, the Contractor will take the measures that may reasonably be expected of it to ensure that the data used in the development of the Algorithmic System will be analysed, structured and/or edited: (a.) according to a motivated approach, the purpose of which includes, without limitation, the avoidance of socially constructed distortion, inaccuracies, errors, mistakes, and undesired prejudice (“bias”) in such data to the extent possible; (b.) in a manner that is in compliance with applicable laws and regulations’.

Sánchez Graells (2022c).