This is an excerpt from the annex of this October 2013 EU study. Read the introduction to the study here.

The available evidence indicates the use of electronic surveillance practices that go beyond traditional, targeted surveillance for intelligence purposes in five EU countries: the UK, Sweden, France, Germany and the Netherlands. Each member state is examined with the following criteria in mind: the basic technical features of large-scale surveillance programmes; stated purpose of programmes, targets and types of data collected; actors involved in collection and use, including evidence of cooperation with the private sector; cooperation or exchange of data with foreign intelligence services, including the NSA; and the legal framework and oversight governing the execution of the programme(s).

UK

Of the five member states examined, the evidence suggests that the UK government is engaged in by far the most extensive large-scale surveillance activities in the EU.

Internet surveillance in the UK is primarily carried out by the agency known as the Government Communications Headquarters (GCHQ), which produces signals intelligence (SIGINT) for the UK government. GCHQ is mandated to work “in the interests of national security, with particular reference to the defence and foreign policies of Her Majesty’s government; in the interests of the economic wellbeing of the United Kingdom; and in support of the prevention and the detection of serious crime”. In budgetary terms GCHQ receives the greatest investment of all the UK’s intelligence services (approximately £1 billion annually) and its human resources are twice the size of the workforce of MI5 and MI6 combined (6,000 staff).

The disclosures by former NSA contractor Edward Snowden and revelations in the US and European press, particularly the Guardian newspaper, have provided a much broader understanding of the depth and range of GCHQ’s activities than experts previously had access to. These reports describe a range of programmes and projects linked to the large-scale access, processing and storage of data that fall within the overarching framework of a GCHQ project named by the agency ‘Mastering the Internet’ (MTI). Reports indicate a budget of over £1 billion devoted to the MTI project over a three-year period, creating capacities for the interception, storage and processing of data on a par with, and potentially even exceeding that of, the NSA with whom it works in close cooperation.

Programme(s) for large-scale surveillance

Potentially the most far-reaching of the programmes run by GCHQ within the MTI project is the so-called ‘Tempora programme’. According to disclosures by the Guardian newspaper, the UK is engaged in the routine interception of undersea cables for the purpose of capturing internet content. Reports allege that GCHQ has placed data interceptors on approximately 200 of the UK-based fibre-optic cables that transmit Internet data into and out of the British Isles carrying data to Western Europe from telephone exchanges and Internet servers in North America. The Tempora programme is estimated to be around five years old, having first been developed and piloted in 2009 and operational since at least early 2012.

The technique of directly tapping the fibre-optic cables entering and exiting the UK (known as Special Source Exploitation) appears to have given GCHQ access to unprecedented quantities of information. In terms of scale, leaked official documents claim that by 2012, GCHQ was able to process data from at least 46 fibre-optic cables at any one time, giving the agency the possibility to intercept, in principal, more than 21 petabytes of data a day. [1] This is estimated to have contributed to a 7,000% increase in the amount of personal data available to GCHQ from internet and mobile traffic in the past five years and given the UK the biggest Internet access in ‘Five Eyes’. Data are understood to be stored at underground storage centres at GCHQ headquarters in Cheltenham, and potentially other agency sites (GCHQ’s sister base in Bude, Cornwall as well as another unnamed base outside of the UK).

The data intercepted and processed consist both of ‘content’ – referring to recordings of phone calls, content of email messages, entries on Facebook, histories of an Internet user’s access to websites, etc. – as well as ‘metacontent’ – data recording the means of creation of transmitted data, the time and date of its creation, its creator and location where it was created. Content intercepted by Tempora is kept for up to three days, while metacontent is stored for up to 30 days. Around 300 GCHQ and 250 NSA operatives are charged with analysing the data intercepted by Tempora.

Both content and metacontent are filtered using a technique called Massive Volume Reduction (MVR). Approximately 30% of the data is removed early in the process, classified as ‘high-volume, low-value’ traffic (consisting for instance of peer-to-peer music, film and computer programme downloads). The remaining data are searched using so-called ‘selectors’, which can include keywords, email addresses and phone numbers of targeted individuals. There are approximately 40,000 such selectors identified by GCHQ.

The objectives underpinning this mass collection of data and the individuals targeted are ambiguous, and as yet they are not clearly delineated in the documents and reported disclosures. According to an intelligence source quoted by the Guardian, the criteria governing the use of selectors to search and filter the data relate to security, terrorism, organised crime and economic well-being. An internal GCHQ memo dated October 2011 stated: “[Our] targets boil down to diplomatic/military/commercial targets/terrorists/ organised criminals and e-crime/cyber actors.”

In principal, the UK legal framework allows Tempora only to target ‘external’ communications, in other words communications between non-UK residents, or between a UK resident and a non-UK resident. However, in practice, given that a substantial proportion of internal UK communications is routed offshore, all internet users are potential targets of the Tempora programme, both British citizens (and UK residents) as well as non-British citizens and residents. As the UK is an important landing point for the vast majority of transatlantic fibre-optic cables, the monitoring of these cables means that a large proportion of communications from around the world would be intercepted.

Details concerning the logistical operation of the Tempora programme imply some cooperation with private-sector telecommunications companies. On 2 August 2013, the Süddeutsche newspaper published the names of the commercial companies cooperating with GCHQ and providing access to their customer’s data within the Tempora programme. The newspaper cited seven companies (BT, Vodafone Cable, Verizon Business, Global Crossing, Level 3, Viatel and Interroute), referred to as ‘intercept partners’, which together operate a large proportion of the undersea fibre-optic internet cables. Allegations claim that companies are paid for logistical and technical assistance and are obliged to cooperate under the 1984 Telecommunications Act. Spokespersons of the companies concerned have stated that they are legally obliged to cooperate, and all cooperation is in accordance with European and national laws. Allegations have also been made that GCHQ has accessed cables without the consent or knowledge of the companies that own or operate them.

The Guardian’s reports on the Tempora programme have been verified and deemed credible by external experts, such as Ian Brown, member of the UK Information Commissioner’s Technology Reference Panel. According to Dr. Brown’s statement in the application to the European Court of Human Rights Big Brother Watch and others vs. the United Kingdom:

The Guardian reports appear to me to be credible. Some of the details have been confirmed by the US government, and by previous leaks (including by statements by former senior NSA officials such as William Binney.) Much of the technology used (such as optical splitter equipment) is commercially available. The budgetary resources required fit within the publicly known budgets of the UK and US intelligence agencies.

Another key dimension of GCHQ’s large-scale surveillance activity that has emerged from the Guardian's disclosures is the UK’s participation in the PRISM programme. Following press revelations concerning the US surveillance activities and programmes operated by the NSA, the Guardian reported that the US shares information it obtains via the PRISM programme with the UK authorities. According to reports, GCHQ has had access to the data gathered under the PRISM programme since June 2010 and generated 197 intelligence reports from this data in 2012. It has been subsequently presumed that GCHQ also has access to wider information obtained by NSA surveillance activities under section 1881a, including material that is directly intercepted from so-called ‘upstream collection’ – the direct interception of communications as they pass through fibre-optic cables and electronic infrastructures of telecommunication companies or online service providers in the US (and potentially around the world).

Privacy advocacy groups and experts have claimed that through its access to US programmes such as PRISM, the UK is able to obtain information about UK citizens’ or residents’ internal communications that would otherwise be out of bounds to UK intelligence agencies without first obtaining a warrant under the Regulation of Investigatory Powers Act 2000 (RIPA). The allegations that this cooperation has effectively allowed the UK authorities to circumvent the UK legal regime have been investigated by the ISC and are further discussed in section 1.3 of this Annex.

Leaked documents have also cited a decryption programme named ‘Edgehill’. On 6 September 2013, the Guardian published a report alleging that GCHQ has been cooperating with a 10-year programme by the NSA against encryption technologies. According to documents seen by the Guardian, a GCHQ pilot programme attempted to establish a system that could identify encrypted traffic from its internet cable-tapping programmes (e.g. Tempora). Reports indicate that the decryption programme, named ‘Edgehill’, was seen as critical in maintaining the strategic advantage that GCHQ has gained with its Tempora programme, as large internet providers began increasingly to encrypt their communications traffic.

GCHQ documents show that Edgehill's initial aim was to decode the encrypted traffic certified by three major (unnamed) internet companies and 30 types of Virtual Private Network (VPN), used by businesses to provide secure remote access to their systems. It is reported that by 2015, GCHQ hoped to have cracked the codes used by 15 major internet companies and 300 VPNs. The Guardian also claims that analysts on the Edgehill project were working on ways into the networks of major webmail providers as part of the decryption project.

Documents leaked by Edward Snowden have also indicated that the UK has engaged in GCHQ-coordinated offensive operations aimed at diplomatic or economic espionage. Internal GCHQ powerpoint slides published by the Guardian in June 2013 indicated that GCHQ intercepted the phones and monitored internet use of foreign politicians and diplomats taking part in two G20 summit meetings in London in 2009.

In September 2013, Der Spiegel published revelations that GCHQ coordinated a project code-named ‘Operation Socialist’ which saw a cyber-attack against the Belgian telecoms company Belgacom. During the European Parliament hearing of 3 October 2013, Belgacom Vice-President Geert Standaert stated that the ‘spyware’, discovered in June 2013, had penetrated 124 of its 26,000 IT systems. Belgacom executives indicated that the scale and sophistication of the attack implied a state actor, but neither confirmed nor denied allegations alluding to GCHQ’s involvement.

In addition to the main disclosures relating to GCHQ large-scale surveillance activities discussed above, other programmes about which less is known have come to light. These include the so-called ‘Global Telecoms Exploitation’ programme which is understood to also be conducted through tapping fibre-optic cables and which allows GCHQ to handle 600 million ‘telephone events’ each day.

Further, documents leaked to the Guardian reveal a ‘mobile’ project designed to exploit mobile devices, collecting voice, sms and geo-locations as well as the additional functionalities that come with smartphones, such as emails, internet searches and social media posts. Internal GCHQ documents underscore the importance of this project in order to keep pace with the increased use of smart phones. It is estimated that 90% of all internet traffic will come from mobile phones by 2015.

According to the Guardian, it had seen documents which make it clear that “GCHQ was now capable of ‘attacking’ hundreds of apps, and a ‘mobile capability map’ from June last year stated the agency had found ways of looking at the search patterns, emails and conversations on many commonly used phone services.”

Cooperation with foreign intelligence services

Evidence that has come to public attention over the past four months indicates a close working relationship between the NSA and GCHQ on mass cyber-surveillance activities. This concerns both data and intelligence-sharing but also in the collaborative development of pilot programmes and technologies. For example, early internal GCHQ documents describing Tempora initially referred to this programme as “a joint GCHQ/NSA research initiative”. Reports also allege close cooperation between GCHQ and NSA in the development of decryption technologies.

In terms of data and intelligence-sharing, the UK appears to conduct a substantial and routine reciprocal relationship of data exchange with the US authorities. Reflecting the details of the UK’s access to PRISM data, a UK government paper that set out the views of GCHQ in the wake of the 2010 strategic defence and security review admitted that 60% of the UK's high-value intelligence “is based on either NSA end-product or derived from NSA collection” (end product referring to official reports that are distillations of raw intelligence.)

Similarly, the UK is reported to provide access to the data collected through the Tempora and other programmes, available to the NSA, with Guardian reports implying that while the UK had the means to collect huge amounts of data through Tempora and its access to undersea internet cables, the NSA could provide the resources (850,000 operatives) and technologies to process and analyse that data. An internal report explained that “GCHQ and NSA avoid processing the same data twice and proactively seek to converge technical solutions and processing architectures.”

The degree of cooperation between the two agencies is reflected in revelations exposing the details of the NSA payments to GCHQ in the last years. The Guardian reports that the payments, which are set out in GCHQ's annual ‘investment portfolios’ seen by the newspaper, show that the US government has paid at least £100 million to the UK spy agency GCHQ over the last three years. The papers show that NSA gave GCHQ £22.9 million in 2009. The following year the NSA’s contribution increased to £39.9 million, of which £17.2 million was allocated for the agency's Mastering the Internet project. The NSA also paid £15.5 million towards redevelopments at GCHQ's sister site in Bude, Cornwall, which intercepts communications from the transatlantic cables that carry internet traffic. In 2011-12, the NSA paid another £34.7 million to GCHQ.

Legal framework and oversight

Surveillance of communications in the UK are carried out within the legal framework established by the UK’s 2000 Regulation of Investigatory Powers Act (RIPA). The warranting process under RIPA falls under two separate regimes, depending on the types of data accessed. Interception of content is authorised by a warrant signed by the Secretary of State specifying an individual or premises and is valid for 3-6 months. Access to ‘communications data’ is regulated under a separate Chapter of RIPA and permits some agencies to self-authorise access to some of this data [2].‘Communications data’ are here defined in relatively vague terms and refers to ‘traffic data’ that includes identities of individuals and equipment as well as location details, routing information and signalling information.

An interception warrant specifying an individual or premises is not needed where UK authorities intercept communications external to the UK. In this scenario, an authorising certificate from the Secretary of State is required which describes the nature/classification of material to be examined. It is under the latter legal mechanism by which data exchange with the US, including that implicated in the PRISM programme, as well as Tempora Programme activities are understood to have been authorised.

In addition, under the Telecommunications Act 1984, the Secretary of State may give providers of public electronic networks “directions of a general character... in the interests of national security or relations with the government of a country or territory outside the United Kingdom”.

Although RIPA is stated to be compatible with the ECHR and includes explicit tests of proportionality and necessity before communications content and metadata may be accessed, experts have noted that “the standards according to which these tests of proportionality are carried out are mainly secret, and applied by the government’s legal advisers and the Secretary of State, with limited oversight.”

The UK’s intelligence oversight regime is composed of the Intelligence and Security Committee, an Interception of Communications Commissioner (IoCC) and the Investigatory Powers Tribunal.

On 7 June 2013, the Intelligence and Security Committee (ISC) [3] issued a statement indicating that it had launched an investigation into allegations that the agency circumvented UK law by using the NSA’s PRISM programme to access the content of private communications within the UK without proper authorisation. On 17 July 2013, the Chairman of the Intelligence and Security Committee of Parliament, the Rt Hon Sir Malcolm Rifkind MP, issued a follow-up statement regarding the outcome of those investigations. The statement concluded that, after taking detailed evidence from GCHQ, any suggested allegations are “unfounded” and complied with the legal safeguards set out in RIPA. The ISC maintained that “in each case” that it examined, GCHQ had a warrant for interception in accordance with RIPA, although the terms of those warrants have not been published. Experts have concluded from the ISC’s public statements that it was not previously aware of the PRISM Programme. While the ISC concluded that GCHQ has not circumvented the law, it nevertheless acknowledged the need “to consider further whether the current statutory framework governing access to private communications remains adequate”.

An Investigatory Powers Tribunal, appointed from current or former senior members of the judiciary, also exists to explore complaints covering the eligibility of GCHQ activities under RIPA. Both the UK charity Privacy International and the civil rights group Liberty have submitted claims to the IPT following the revelations of GCHQ’s activities in PRISM and Tempora. However, this body has not in the past demonstrated a strong oversight function of GCHQ.

[1] A petabyte is approximately 1,000 terabytes, which in turn is 1,000 gigabytes. The comparison made by the Guardian was that this is equivalent to sending all the books in the British Library 192 times every 24 hours.

[2] According to RIPA, communications data can be accessed by a range of government agencies on a broad set of grounds, including in the interests of national security, preventing or detecting crime or disorder, economic well-being and so on, and includes any purpose specified in an order made by the Secretary of State.

[3] The Intelligence and Security Committee of Parliament (ISC) is a statutory committee of Parliament that has responsibility for oversight of the UK intelligence community. The Committee was originally established by the Intelligence Services Act 1994. The Committee oversees the intelligence and security activities of the UK, including the policies, expenditure, administration and operations of the Security Service (MI5), the Secret Intelligence Service (MI6) and the Government Communications Headquarters (GCHQ). The Committee consists of nine Members drawn from both Houses of Parliament.

Read more from our 'Joining the dots on state surveillance' series here.