[NetBehaviour] Fwd: CRYPTO-GRAM, June 15, 2021

Alan Sondheim sondheim at gmail.com
Thu Jun 17 15:44:50 CEST 2021


I read this almost religiously (as much as I can, given I'm not religious)
- for me it says more about a potential future than anything I could write.
Unfortunately -
Best, Alan

---------- Forwarded message ---------
From: Bruce Schneier <schneier at schneier.com>
Date: Tue, Jun 15, 2021 at 9:52 AM
Subject: CRYPTO-GRAM, June 15, 2021
To: <sondheim at panix.com>


A monthly newsletter about cybersecurity and related topics.
Crypto-Gram
June 15, 2021

by Bruce Schneier
Fellow and Lecturer, Harvard Kennedy School
schneier at schneier.com
https://www.schneier.com

A free monthly newsletter providing summaries, analyses, insights, and
commentaries on security: computer and otherwise.

For back issues, or to subscribe, visit Crypto-Gram's web page
<https://www.schneier.com/crypto-gram/>.

Read this issue on the web
<https://www.schneier.com/crypto-gram/archives/2021/0615.html>

These same essays and news items appear in the Schneier on Security
<https://www.schneier.com/> blog, along with a lively and intelligent
comment section. An RSS feed is available.

** *** ***** ******* *********** *************
In this issue:

*If these links don't work in your email client, try reading this issue of
Crypto-Gram on the web.
<https://www.schneier.com/crypto-gram/archives/2021/0615.html>*

   1. Is 85% of US Critical Infrastructure in Private Hands?
   <#m_-1416174297983045154_cg1>
   2. Adding a Russian Keyboard to Protect against Ransomware
   <#m_-1416174297983045154_cg2>
   3. Apple Censorship and Surveillance in China
   <#m_-1416174297983045154_cg3>
   4. Bizarro Banking Trojan <#m_-1416174297983045154_cg4>
   5. Double-Encrypting Ransomware <#m_-1416174297983045154_cg5>
   6. AIs and Fake Comments <#m_-1416174297983045154_cg6>
   7. New Disk Wiping Malware Targets Israel <#m_-1416174297983045154_cg7>
   8. The Story of the 2011 RSA Hack <#m_-1416174297983045154_cg8>
   9. The Misaligned Incentives for Cloud Security
   <#m_-1416174297983045154_cg9>
   10. Security Vulnerability in Apple's Silicon "M1" Chip
   <#m_-1416174297983045154_cg10>
   11. The DarkSide Ransomware Gang <#m_-1416174297983045154_cg11>
   12. Security and Human Behavior (SHB) 2021 <#m_-1416174297983045154_cg12>
   13. The Supreme Court Narrowed the CFAA <#m_-1416174297983045154_cg13>
   14. Vulnerabilities in Weapons Systems <#m_-1416174297983045154_cg14>
   15. Information Flows and Democracy <#m_-1416174297983045154_cg15>
   16. Detecting Deepfake Picture Editing <#m_-1416174297983045154_cg16>
   17. FBI/AFP-Run Encrypted Phone <#m_-1416174297983045154_cg17>
   18. TikTok Can Now Collect Biometric Data <#m_-1416174297983045154_cg18>
   19. Upcoming Speaking Engagements <#m_-1416174297983045154_cg19>

** *** ***** ******* *********** *************
Is 85% of US Critical Infrastructure in Private Hands?

*[2021.05.17]*
<https://www.schneier.com/blog/archives/2021/05/is-85-of-us-critical-infrastructure-in-private-hands.html>
Most US critical infrastructure is run by private corporations. This has
major security implications, because it’s putting a random power company in
-- say -- Ohio -- up against the Russian cybercommand, which isn’t a fair
fight.

When this problem is discussed, people regularly quote the statistic that
85% of US critical infrastructure is in private hands. It’s a handy number,
and matches our intuition. Still, I have never been able to find a factual
basis, or anyone who knows where the number comes from. Paul Rosenzweig
investigates <https://www.lawfareblog.com/it-really-85-percent>, and
reaches the same conclusion.

So we don’t know the percentage, but I think we can safely say that it’s a
lot.

** *** ***** ******* *********** *************
Adding a Russian Keyboard to Protect against Ransomware

*[2021.05.18]*
<https://www.schneier.com/blog/archives/2021/05/adding-a-russian-keyboard-to-protect-against-ransomware.html>
A lot of Russian malware -- the malware that targeted the Colonial
Pipeline, for example -- won’t install on computers with a Cyrillic
keyboard installed. Brian Krebs wonders
<https://krebsonsecurity.com/2021/05/try-this-one-weird-trick-russian-hackers-hate/>
if this could be a useful defense:

In Russia, for example, authorities there generally will not initiate a
cybercrime investigation against one of their own unless a company or
individual within the country’s borders files an official complaint as a
victim. Ensuring that no affiliates can produce victims in their own
countries is the easiest way for these criminals to stay off the radar of
domestic law enforcement agencies.

[...]

DarkSide, like a great many other malware strains, has a hard-coded
do-not-install list of countries which are the principal members of the
Commonwealth of Independent States (CIS) -- former Soviet satellites that
mostly have favorable relations with the Kremlin.

[...]

Simply put, countless malware strains will check for the presence of one of
these languages on the system, and if they’re detected the malware will
exit and fail to install.

[...]

Will installing one of these languages keep your Windows computer safe from
all malware? Absolutely not. There is plenty of malware that doesn’t care
where in the world you are. And there is no substitute for adopting a
defense-in-depth posture, and avoiding risky behaviors online.

But is there really a downside to taking this simple, free, prophylactic
approach? None that I can see, other than perhaps a sinking feeling of
capitulation. The worst that could happen is that you accidentally toggle
the language settings and all your menu options are in Russian.

** *** ***** ******* *********** *************
Apple Censorship and Surveillance in China

*[2021.05.19]*
<https://www.schneier.com/blog/archives/2021/05/apple-censorship-and-surveillance-in-china.html>
Good investigative reporting
<https://www.nytimes.com/2021/05/17/technology/apple-china-censorship-data.html>
on how Apple is participating in and assisting with Chinese censorship and
surveillance.

** *** ***** ******* *********** *************
Bizarro Banking Trojan

*[2021.05.20]*
<https://www.schneier.com/blog/archives/2021/05/bizarro-banking-trojan.html>
Bizarro is a new banking trojan
<https://gizmodo.com/bizarro-a-new-banking-trojan-is-sweeping-through-euro-1846927565>
that is stealing financial information and crypto wallets.

...the program can be delivered in a couple of ways -- either via malicious
links contained within spam emails, or through a trojanized app. Using
these sneaky methods, trojan operators will implant the malware onto a
target device, where it will install a sophisticated backdoor that
“contains more than 100 commands and allows the attackers to steal online
banking account credentials,” the researchers write.

The backdoor has numerous commands built in to allow manipulation of a
targeted individual, including keystroke loggers that allow for harvesting
of personal login information. In some instances, the malware can allow
criminals to commandeer a victim’s crypto wallet, too.

Research report
<https://securelist.com/bizarro-banking-trojan-expands-its-attacks-to-europe/102258/>
.

** *** ***** ******* *********** *************
Double-Encrypting Ransomware

*[2021.05.21]*
<https://www.schneier.com/blog/archives/2021/05/double-encrypting-ransomware.html>
This seems to be a new tactic
<https://www.wired.com/story/ransomware-double-encryption/>:

Emsisoft has identified two distinct tactics. In the first, hackers encrypt
data with ransomware A and then re-encrypt that data with ransomware B. The
other path involves what Emsisoft calls a “side-by-side encryption” attack,
in which attacks encrypt some of an organization’s systems with ransomware
A and others with ransomware B. In that case, data is only encrypted once,
but a victim would need both decryption keys to unlock everything. The
researchers also note that in this side-by-side scenario, attackers take
steps to make the two distinct strains of ransomware look as similar as
possible, so it’s more difficult for incident responders to sort out what’s
going on.

** *** ***** ******* *********** *************
AIs and Fake Comments

*[2021.05.24]*
<https://www.schneier.com/blog/archives/2021/05/ais-and-fake-comments.html>
This month, the New York state attorney general issued a report
<https://ag.ny.gov/sites/default/files/oag-fakecommentsreport.pdf> on a
scheme by “U.S. Companies and Partisans [to] Hack Democracy.” This wasn’t
another attempt by Republicans to make it harder for Black people and urban
residents to vote. It was a concerted attack on another core element of US
democracy -- the ability of citizens to express their voice to their
political representatives. And it was carried out by generating millions of
fake comments and fake emails purporting to come from real citizens.

This attack was detected because it was relatively crude. But artificial
intelligence technologies are making it possible to generate
genuine-seeming comments at scale, drowning out the voices of real citizens
in a tidal wave of fake ones.

As political scientists like Paul Pierson have pointed out
<http://www.henryfarrell.net/nixonland.pdf>, what happens between elections
is important to democracy. Politicians shape policies and they make laws.
And citizens can approve or condemn what politicians are doing, through
contacting their representatives or commenting on proposed rules.

That’s what should happen. But as the New York report shows, it often
doesn’t. The big telecommunications companies paid millions of dollars to
specialist “AstroTurf” companies to generate public comments. These
companies then stole people’s names and email addresses from old files and
from hacked data dumps and attached them to 8.5 million
<https://twitter.com/NewYorkStateAG/status/1390298453201362944> public
comments and half a million letters to members of Congress. All of them
said that they supported the corporations’ position on something called “net
neutrality <https://www.wired.com/story/guide-net-neutrality/>,” the idea
that telecommunications companies must treat all Internet content equally
and not prioritize any company or service. Three AstroTurf companies --
Fluent, Opt-Intelligence and React2Media -- agreed to pay nearly $4 million
in fines.

The fakes were crude. Many of them were identical, while others were
patchworks of simple textual variations: substituting “Federal
Communications Commission” and “FCC” for each other, for example.

Next time, though, we won’t be so lucky. New technologies are about to make
it far easier to generate enormous numbers of convincing personalized
comments and letters, each with its own word choices, expressive style and
pithy examples. The people who create fake grass-roots organizations have
always been enthusiastic early adopters of technology, weaponizing letters,
faxes, emails and Web comments to manufacture the appearance of public
support or public outrage.

Take Generative Pre-trained Transformer 3, or GPT-3, an AI model created by
OpenAI, a San Francisco based start-up. With minimal prompting, GPT-3 can
generate convincing seeming newspaper articles
<https://www.theguardian.com/commentisfree/2020/sep/08/robot-wrote-this-article-gpt-3>,
résumé cover letters
<https://www.linkedin.com/pulse/i-used-ai-write-cover-letter-application-google-jacob-jacquet>,
even Harry Potter fan fiction in the style of Ernest Hemingway
<https://www.gwern.net/GPT-3#fnref32>. It is trivially easy to use these
techniques to compose large numbers of public comments or letters to
lawmakers.

OpenAI restricts access to GPT-3, but in a recent experiment, researchers
used a different text-generation program to submit 1,000 comments
<https://techscience.org/a/2019121801/> in response to a government request
for public input on a Medicaid issue. They all sounded unique, like real
people advocating a specific policy position. They fooled the Medicaid.gov
administrators, who accepted them as genuine concerns from actual human
beings. The researchers subsequently identified the comments and asked for
them to be removed, so that no actual policy debate would be unfairly
biased. Others won’t be so ethical.

When the floodgates open, democratic speech is in danger of drowning
beneath a tide of fake letters and comments, tweets and Facebook posts. The
danger isn’t just that fake support can be generated for unpopular
positions, as happened with net neutrality. It is that public commentary
will be completely discredited. This would be bad news for specialist
AstroTurf companies, which would have no business model if there isn’t a
public that they can pretend to be representing. But it would empower still
further other kinds of lobbyists, who at least can prove that they are who
they say they are.

We may have a brief window to shore up the flood walls. The most effective
response would be to regulate what UCLA sociologist Edward Walker has
described as the “grassroots for hire
<https://www.amazon.com/Grassroots-Hire-Consultants-American-Democracy/dp/1107619017>”
industry. Organizations that deliberately fabricate citizen voices
shouldn’t just be subject to civil fines, but to criminal penalties.
Businesses that hire these organizations should be held liable for failures
of oversight. It’s impossible to prove or disprove whether
telecommunications companies knew their subcontractors would create bogus
citizen voices, but a liability standard would at least give such companies
an incentive to find out. This is likely to be politically difficult to put
in place, though, since so many powerful actors benefit from the status quo.

This essay was written with Henry Farrell, and previously appeared
<https://www.washingtonpost.com/outlook/2021/05/20/ai-bots-grassroots-astroturf/>
in the *Washington Post*.

EDITED TO ADD: CSET published
<https://cset.georgetown.edu/publication/truth-lies-and-automation/> an
excellent report on AI-generated partisan content. Short summary: it’s
pretty good, and will continue to get better. Renee DeRista has also written
about this
<https://www.theatlantic.com/ideas/archive/2020/09/future-propaganda-will-be-computer-generated/616400/>
.

This paper <https://jods.mitpress.mit.edu/pub/2gnso48a/release/8> is about
a lower-tech version of this threat. Also this
<https://mediamanipulation.org/case-studies/astroturfing-how-hijacked-accounts-and-dark-public-relations-faked-support-chinas>
.

EDITED TO ADD: Another essay
<https://techpolicy.press/fake-net-neutrality-comment-campaign-a-harbinger-of-automated-disinformation-to-come/>
on the same topic.

** *** ***** ******* *********** *************
New Disk Wiping Malware Targets Israel

*[2021.05.26]*
<https://www.schneier.com/blog/archives/2021/05/new-disk-wiping-malware-targets-israel.html>
Apostle seems to be a new strain
<https://arstechnica.com/gadgets/2021/05/disk-wiping-malware-with-irananian-fingerprints-is-striking-israeli-targets/>
of malware that destroys data.

In a post published Tuesday <https://s1.ai/agrius>, SentinelOne researchers
said they assessed with high confidence that based on the code and the
servers Apostle reported to, the malware was being used by a newly
discovered group with ties to the Iranian government. While a ransomware
note the researchers recovered suggested that Apostle had been used against
a critical facility in the United Arab Emirates, the primary target was
Israel.

** *** ***** ******* *********** *************
The Story of the 2011 RSA Hack

*[2021.05.27]*
<https://www.schneier.com/blog/archives/2021/05/the-story-of-the-2011-rsa-hack.html>
Really good long article
<https://www.wired.com/story/the-full-story-of-the-stunning-rsa-hack-can-finally-be-told/>
about the Chinese hacking of RSA, Inc. They were able to get copies of the
seed values to the SecurID authentication token, a harbinger of
supply-chain attacks to come.

** *** ***** ******* *********** *************
The Misaligned Incentives for Cloud Security

*[2021.05.28]*
<https://www.schneier.com/blog/archives/2021/05/the-misaligned-incentives-for-cloud-security.html>
Russia’s Sunburst cyberespionage campaign
<https://www.bbc.com/news/technology-55321643>, discovered late last year,
impacted more than 100 large companies and US federal agencies, including
the Treasury, Energy, Justice, and Homeland Security departments. A crucial
part of the Russians’ success was their ability to move through these
organizations by compromising cloud and local network identity systems
<https://www.fireeye.com/content/dam/collateral/en/wp-m-unc2452.pdf> to
then access cloud accounts and pilfer emails and files.

Hackers said by the US government to have been working for the Kremlin
targeted
<https://www.fireeye.com/content/dam/collateral/en/wp-m-unc2452.pdf> a
widely used Microsoft cloud service that synchronizes user identities. The
hackers stole security certificates
<https://www.darkreading.com/attacks-breaches/solarwinds-campaign-focuses-attention-on-golden-saml-attack-vector/d/d-id/1339794>
to create
<https://www.crowdstrike.com/blog/crowdstrike-launches-free-tool-to-identify-and-help-mitigate-risks-in-azure-active-directory/>
their own identities, which allowed them to bypass
<https://techcommunity.microsoft.com/t5/azure-sentinel/solarwinds-post-compromise-hunting-with-azure-sentinel/ba-p/1995095>
safeguards such as multifactor authentication and gain access to Office 365
accounts, impacting thousands of users at the affected companies and
government agencies.

It wasn’t the first time cloud services were the focus of a cyberattack,
and it certainly won’t be the last. Cloud weaknesses were also critical
<https://www.securonix.com/web/wp-content/uploads/2019/08/Securonix_Threat_Research_Capital_One_Cyberattack.pdf>
in a 2019 breach at Capital One
<https://www.cnn.com/2019/07/29/business/capital-one-data-breach/index.html>.
There, an Amazon Web Services cloud vulnerability, compounded
<https://blog.cloudsploit.com/a-technical-analysis-of-the-capital-one-hack-a9b43d7c8aea>
by Capital One’s own struggle
<https://www.cloudpassage.com/articles/preventing-a-capital-one-cloud-data-breach/>
to properly configure a complex cloud service, led to the disclosure of
tens of millions
<https://www.cnn.com/2019/07/29/business/capital-one-data-breach/index.html>
of customer records, including credit card applications, Social Security
numbers, and bank account information.

This trend of attacks
<https://www.atlanticcouncil.org/in-depth-research-reports/report/broken-trust-lessons-from-sunburst/>
on cloud services by criminals, hackers, and nation states is growing
<https://www.infosecurity-magazine.com/news/report-attacks-cloud-services/>
as cloud computing takes over worldwide as the default model for
information technologies. Leaked data is bad enough, but disruption to the
cloud, even an outage at a single provider, could quickly cost the global
economy billions of dollars a day
<https://www.lloyds.com/~/media/files/news-and-insight/risk-insight/2018/cloud-down/aircyberlloydspublic2018final.pdf>
.

Cloud computing is an important source of risk both because it has quickly
supplanted traditional IT and because it concentrates ownership of design
choices at a very small number of companies. First, cloud is increasingly
the default mode of computing for organizations, meaning ever more users
and critical data from national intelligence and defense agencies ride on
these technologies. Second, cloud computing services, especially those
supplied by the world’s four largest providers -- Amazon, Microsoft,
Alibaba, and Google -- concentrate key security and technology design
choices inside a small number of organizations. The consequences of bad
decisions or poorly made trade-offs can quickly scale to hundreds of
millions of users.

The cloud is everywhere. Some cloud companies provide software as a
service, support your Netflix habit, or carry your Slack chats. Others
provide computing infrastructure like business databases and storage space.
The largest cloud companies provide both.

The cloud can be deployed in several different ways, each of which shift
the balance of responsibility for the security of this technology. But the
cloud provider plays an important role in every case. Choices the provider
makes in how these technologies are designed, built, and deployed influence
the user’s security
<https://www.mcafee.com/enterprise/en-us/forms/gated-form.html?docID=59d987b2-5df5-4fa2-a6b3-f9f7c204140f>
-- yet the user has very little influence over them. Then, if Google or
Amazon has a vulnerability in their servers -- which you are unlikely to
know about and have no control over -- you suffer the consequences.

The problem is one of economics. On the surface, it might seem that
competition between cloud companies gives them an incentive to invest in
their users’ security. But several market failures get in the way
<https://www.nytimes.com/2021/02/23/opinion/solarwinds-hack.html> of that
ideal. First, security is largely an externality for these cloud companies,
because the losses due to data breaches are largely borne by their users.
As long as a cloud provider isn’t losing customers by the droves -- which
generally doesn’t happen after a security incident -- it is incentivized to
underinvest in security. Additionally, data shows
<https://www.napier.ac.uk/~/media/worktribe/output-1476796/impact-of-cyber-attacks-on-stock-performance-a-comparative-study.pdf>
that investors don’t punish the cloud service companies either: Stock price
dips after a public security breach are both small and temporary.

Second, public information about cloud security generally doesn’t share the
design trade-offs involved in building these cloud services or provide much
transparency about the resulting risks. While cloud companies have to
publicly disclose copious amounts of security design and operational
information, it can be impossible for consumers to understand which threats
the cloud services are taking into account, and how. This lack of
understanding makes it hard to assess a cloud service’s overall security.
As a result, customers and users aren’t able to differentiate between
secure and insecure services, so they don’t base their buying and use
decisions on it.

Third, cybersecurity is complex -- and even more complex when the cloud is
involved. For a customer like a company or government agency, the security
dependencies of various cloud and on-premises network systems and services
can be subtle and hard to map out. This means that users can’t adequately
assess the security of cloud services or how they will interact with their
own networks. This is a classic “lemons market” in economics, and the result
<https://www.usenix.org/system/files/login/articles/login_fall19_12_geer.pdf>
is that cloud providers provide variable levels of security, as documented
by Dan Geer, the chief information security officer for In-Q-Tel, and Wade
Baker, a professor at Virginia Tech’s College of Business, when they looked
at the prevalence of severe security findings at the top 10 largest cloud
providers. Yet most consumers are none the wiser.

The result is a market failure where cloud service providers don’t compete
to provide the best security for their customers and users at the lowest
cost. Instead, cloud companies take the chance that they won’t get hacked,
and past experience tells them they can weather the storm if they do. This
kind of decision-making and priority-setting takes place at the executive
level, of course, and doesn’t reflect the dedication and technical skill of
product engineers and security specialists. The effect of this
underinvestment is pernicious, however, by piling on risk that’s largely
hidden from users. Widespread adoption of cloud computing carries that risk
to an organization’s network, to its customers and users, and, in turn, to
the wider internet.

This aggregation of cybersecurity risk creates a national security
challenge. Policymakers can help
<https://www.atlanticcouncil.org/in-depth-research-reports/report/broken-trust-lessons-from-sunburst/>
address the challenge by setting clear expectations for the security of
cloud services -- and for making decisions and design trade-offs about that
security transparent. The Biden administration, including newly nominated
National Cyber Director Chris Inglis, should lead an interagency effort to
work with cloud providers to review their threat models and evaluate the
security architecture of their various offerings. This effort to require
greater transparency from cloud providers and exert more scrutiny of their
security engineering efforts should be accompanied by a push to modernize
cybersecurity regulations for the cloud era.

The Federal Risk and Authorization Management Program (FedRAMP), which is
the principal US government program for assessing the risk of cloud
services and authorizing them for use by government agencies, would be a
prime vehicle for these efforts. A recent executive order outlines several
steps
<https://www.lawfareblog.com/everything-you-need-know-about-new-executive-order-cybersecurity>
to make FedRAMP faster and more responsive. But the program is still
focused largely on the security of individual services rather than the
cloud vendors’ deeper architectural choices and threat models.
Congressional action should reinforce and extend the executive order by
adding new obligations for vendors to provide transparency about design
trade-offs, threat models, and resulting risks. These changes could help
transform FedRAMP into a more effective tool of security governance even as
it becomes faster and more efficient.

Cloud providers have become important national infrastructure. Not since
the heights of the mainframe era between the 1960s and early 1980s has the
world witnessed computing systems of such complexity used by so many but
designed and created by so few. The security of this infrastructure demands
greater transparency and public accountability -- if only to match the
consequences of its failure.

This essay was written with Trey Herr, and previously appeared
<https://foreignpolicy.com/2021/05/24/cybersecurity-cyberattack-russia-hackers-cloud-sunburst-microsoft-office-365-data-leak/>
in *Foreign Policy*.

** *** ***** ******* *********** *************
Security Vulnerability in Apple's Silicon "M1" Chip

*[2021.06.01]*
<https://www.schneier.com/blog/archives/2021/06/security-vulnerability-in-apples-silicon-m1-chip.html>
The website for the M1racles <https://m1racles.com/> security vulnerability
is an excellent demonstration that not all vulnerabilities are exploitable.
Be sure to read the FAQ through to the end.

EDITED TO ADD: Wired article
<https://www.wired.com/story/apples-m1-chip-has-fascinating-flaw/>.

** *** ***** ******* *********** *************
The DarkSide Ransomware Gang

*[2021.06.02]*
<https://www.schneier.com/blog/archives/2021/06/the-darkside-ransomware-gang.html>
The *New York Times* has a long story
<https://www.nytimes.com/2021/05/29/world/europe/ransomware-russia-darkside.html?searchResultPosition=1>
on the DarkSide ransomware gang.

A glimpse into DarkSide’s secret communications in the months leading up to
the Colonial Pipeline attack reveals a criminal operation on the rise,
pulling in millions of dollars in ransom payments each month.

DarkSide offers what is known as “ransomware as a service,” in which a
malware developer charges a user fee to so-called affiliates like Woris,
who may not have the technical skills to actually create ransomware but are
still capable of breaking into a victim’s computer systems.

DarkSide’s services include providing technical support for hackers,
negotiating with targets like the publishing company, processing payments,
and devising tailored pressure campaigns through blackmail and other means,
such as secondary hacks to crash websites. DarkSide’s user fees operated on
a sliding scale: 25 percent for any ransoms less than $500,000 down to 10
percent for ransoms over $5 million, according to the computer security
firm, FireEye.

** *** ***** ******* *********** *************
Security and Human Behavior (SHB) 2021

*[2021.06.04]*
<https://www.schneier.com/blog/archives/2021/06/security-and-human-behavior-shb-2021.html>
Today is the second day of the fourteenth Workshop on Security and Human
Behavior <https://www.cl.cam.ac.uk/~rja14/shb20/>. The University of
Cambridge is the host, but we’re all on Zoom.

SHB is a small, annual, invitational workshop of people studying various
aspects of the human side of security, organized each year by Alessandro
Acquisti, Ross Anderson, and myself. The forty or so attendees include
psychologists, economists, computer security researchers, sociologists,
political scientists, criminologists, neuroscientists, designers, lawyers,
philosophers, anthropologists, business school professors, and a smattering
of others. It’s not just an interdisciplinary event; most of the people
here are individually interdisciplinary.

Our goal is always to maximize discussion and interaction. We do that by
putting everyone on panels, and limiting talks to six to eight minutes,
with the rest of the time for open discussion. The format translates well
to Zoom, and we’re using random breakouts for the breaks between sessions.

I always find this workshop to be the most intellectually stimulating two
days of my professional year. It influences my thinking in different, and
sometimes surprising, ways.

This year’s schedule is here
<https://www.cl.cam.ac.uk/~rja14/shb21/schedule.html>. This page
<https://www.cl.cam.ac.uk/~rja14/shb21/> lists the participants and
includes links to some of their work. As he does every year, Ross Anderson
is liveblogging the talks
<https://www.lightbluetouchpaper.org/2021/06/03/security-and-human-behaviour-2021/>
.

Here are my posts on the first
<http://www.schneier.com/blog/archives/2008/06/security_and_hu.html>, second
<http://www.schneier.com/blog/archives/2009/06/second_shb_work.html>, third
<http://www.schneier.com/blog/archives/2010/06/third_shb_works.html>, fourth
<http://www.schneier.com/blog/archives/2011/06/fourth_shb_work.html>, fifth
<https://www.schneier.com/blog/archives/2012/06/security_and_hu_1.html>,
sixth
<https://www.schneier.com/blog/archives/2013/06/security_and_hu_2.html>,
seventh
<https://www.schneier.com/blog/archives/2014/06/security_and_hu_3.html>,
eighth
<https://www.schneier.com/blog/archives/2015/06/security_and_hu_4.html>,
ninth
<https://www.schneier.com/blog/archives/2016/06/security_and_hu_5.html>,
tenth
<https://www.schneier.com/blog/archives/2017/05/security_and_hu_6.html>,
eleventh
<https://www.schneier.com/blog/archives/2018/05/security_and_hu_7.html>,
twelfth
<https://www.schneier.com/blog/archives/2019/06/security_and_hu_8.html>,
and thirteenth
<https://www.schneier.com/blog/archives/2020/06/security_and_hu_9.html> SHB
workshops. Follow those links to find summaries, papers, and occasionally
audio recordings of the various workshops. Ross also maintains a good
webpage of psychology and security resources
<https://www.cl.cam.ac.uk/~rja14/psysec.html>.

** *** ***** ******* *********** *************
The Supreme Court Narrowed the CFAA

*[2021.06.07]*
<https://www.schneier.com/blog/archives/2021/06/the-supreme-court-narrowed-the-cfaa.html>
In a 6-3 ruling
<https://www.supremecourt.gov/opinions/20pdf/19-783_k53l.pdf>, the Supreme
Court just narrowed the scope
<https://therecord.media/supreme-court-narrows-scope-of-cfaa-computer-hacking-law/>
of the Computer Fraud and Abuse Act:

In a ruling delivered today, the court sided with Van Buren and overturned
his 18-month conviction.

In a 37-page opinion written and delivered by Justice Amy Coney Barrett,
the court explained that the “exceeds authorized access” language was,
indeed, too broad.

Justice Barrett said the clause was effectively making criminals of most US
citizens who ever used a work resource to perform unauthorized actions,
such as updating a dating profile, checking sports scores, or paying bills
at work.

What today’s ruling means is that the CFAA cannot be used to prosecute
rogue employees who have legitimate access to work-related resources, which
will need to be prosecuted under different charges.

The ruling does not apply to former employees accessing their old work
systems because their access has been revoked and they’re not “authorized”
to access those systems anymore.

More
<https://www.reuters.com/technology/us-supreme-court-limits-reach-federal-computer-fraud-law-2021-06-03/>
.

It’s a good ruling, and one that will benefit security researchers. But the
confusing part is footnote 8:

For present purposes, we need not address whether this inquiry turns only
on technological (or “code-based”) limitations on access, or instead also
looks to limits contained in contracts or policies.

It seems to me that this is exactly what the ruling does address. The court
overturned the conviction because the defendant was not limited by
technology, but only by policies. So that footnote doesn’t make any sense.

I have written about this general issue before
<https://www.schneier.com/academic/archives/2020/07/legal_risks_of_adver.html>,
in the context of adversarial machine learning research.

** *** ***** ******* *********** *************
Vulnerabilities in Weapons Systems

*[2021.06.08]*
<https://www.schneier.com/blog/archives/2021/06/vulnerabilities-in-weapons-systems.html>
“If you think any of these systems are going to work as expected in
wartime, you’re fooling yourself.”

That was Bruce’s response at a conference hosted by US Transportation
Command in 2017, after learning that their computerized logistical systems
were mostly unclassified and on the Internet. That may be necessary to keep
in touch with civilian companies like FedEx in peacetime or when fighting
terrorists or insurgents. But in a new era facing off with China or Russia,
it is dangerously complacent.

Any twenty-first century war will include cyber operations. Weapons and
support systems will be successfully attacked. Rifles
<https://www.wired.com/2015/07/hackers-can-disable-sniper-rifleor-change-target/>
and pistols
<https://money.cnn.com/2017/07/27/technology/hack-smart-gun/index.html>
won’t work properly. Drones will be hijacked
<https://www.businessinsider.com/department-13-mesmer-drones-2017-1> midair
<https://www.bellingcat.com/news/2019/10/01/has-iran-been-hacking-u-s-drones/>.
Boats won’t sail
<https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-crashed-the-world/>,
or will be misdirected
<https://www.newscientist.com/article/2143499-ships-fooled-in-gps-spoofing-attack-suggest-russian-cyberweapon/>.
Hospitals won’t function
<https://www.ft.com/content/acf4ac78-c738-48c6-8de1-077697e062d6>.
Equipment and supplies will arrive late
<https://www.overdriveonline.com/electronic-logging-devices/article/14893028/hacking-trucks-cybersecurity-and-the-eld-mandate>
or not at all.

Our military systems are vulnerable
<https://www.wired.com/story/dire-possibility-cyberattacks-weapons-systems/>.
We need to face that reality by halting the purchase of insecure weapons
and support systems and by incorporating the realities of offensive
cyberattacks into our military planning.

Over the past decade, militaries have established cyber commands
<https://www.cybercom.mil/> and developed
<https://www.washingtonpost.com/opinions/2021/03/16/next-world-war-wont-be-anything-like-last-heres-how-us-must-prepare/>
cyberwar
<https://www.government.nl/binaries/government/documents/parliamentary-documents/2019/09/26/letter-to-the-parliament-on-the-international-legal-order-in-cyberspace/International+Law+in+the+Cyberdomain+-+Netherlands.pdf>
doctrine
<https://blog.lukaszolejnik.com/french-application-of-international-rules-to-cyberwarfare/>.
However, much of the current discussion is about offense. Increasing our
offensive capabilities without being able to secure them is like having all
the best guns in the world, and then storing them in an unlocked, unguarded
armory. They just won’t be stolen; they’ll be subverted.

During that same period, we’ve seen increasingly
<https://www.nytimes.com/2021/01/02/us/politics/russian-hacking-government.html>
brazen
<https://www.technologyreview.com/2021/03/10/1020596/how-chinas-attack-on-microsoft-escalated-into-a-reckless-hacking-spree/>
cyberattacks
<https://www.theguardian.com/world/2021/apr/11/israel-appears-confirm-cyberattack-iran-nuclear-facility>
by everyone
<https://www.cfr.org/blog/global-consequences-escalating-us-russia-cyber-conflict>
from criminals
<https://www.cbsnews.com/news/colonial-pipeline-ransomware-attack-darkside-criminal-gang/>
to governments
<https://www.newyorker.com/magazine/2021/04/26/the-incredible-rise-of-north-koreas-hacking-army>.
Everything is now a computer, and those computers are vulnerable
<https://nymag.com/intelligencer/2017/01/the-internet-of-things-dangerous-future-bruce-schneier.html>.
Cars, medical devices, power plants, and fuel pipelines have all been
targets. Military computers, whether they’re embedded inside weapons
systems or on desktops managing the logistics of those weapons systems, are
similarly vulnerable. We could see effects as stodgy as making a tank
impossible to start up, or sophisticated as retargeting a missile midair.

Military software is unlikely to be any more secure than commercial
software. Although sensitive military systems rely on domestically
manufactured chips as part of the Trusted Foundry
<https://www.dmea.osd.mil/TrustedIC.aspx> program, many military systems
contain the same foreign chips and code that commercial systems do: just
like everyone around the world uses the same mobile phones, networking
equipment, and computer operating systems. For example, there has been serious
concern <https://www.nytimes.com/2020/02/06/us/politics/barr-5g.html> over
Chinese-made 5G networking equipment that might be used by China to install
“backdoors” that would allow the equipment to be controlled. This is just
one of many risks to our normal civilian computer supply chains
<https://www.atlanticcouncil.org/in-depth-research-reports/report/broken-trust-lessons-from-sunburst/>.
And since military software is vulnerable to the same cyberattacks as
commercial software, military supply chains have many of the same risks.

This is not speculative. A 2018 GAO report
<https://www.gao.gov/assets/gao-19-128.pdf> expressed concern regarding the
lack of secure and patchable US weapons systems. The report observed that
“in operational testing, the [Department of Defense] routinely found
mission-critical cyber vulnerabilities in systems that were under
development, yet program officials GAO met with believed their systems were
secure and discounted some test results as unrealistic.” It’s a similar
attitude to corporate executives who believe that they can’t be hacked --
and equally naive.

An updated GAO report <https://www.gao.gov/assets/gao-21-179.pdf> from
earlier this year found some improvements, but the basic problem remained:
“DOD is still learning how to contract for cybersecurity in weapon systems,
and selected programs we reviewed have struggled to incorporate systems’
cybersecurity requirements into contracts.” While DOD now appears aware of
the issue of lack of cybersecurity requirements, they’re still not sure yet
how to fix it, and in three of the five cases GAO reviewed, DOD simply
chose to not include the requirements at all.

Militaries around the world are now exploiting these vulnerabilities in
weapons systems to carry out operations. When Israel in 2007 bombed a
Syrian nuclear reactor, the raid was preceded by what is believed to have
been a cyber attack
<https://csis-website-prod.s3.amazonaws.com/s3fs-public/legacy_files/files/publication/101001_ieee_insert.pdf>
on Syrian air defenses that resulted in radar screens showing no threat as
bombers zoomed overhead. In 2018, a 29-country NATO exercise, Trident
Juncture <https://www.nato.int/cps/en/natohq/157833.htm>, that included
cyberweapons was disrupted
<https://www.cnn.com/2018/11/14/politics/russia-nato-jamming/index.html> by
Russian GPS jamming. NATO does try to test cyberweapons outside such
exercises, but has limited scope in doing so. In May, Jens Stoltenberg, the
NATO secretary-general, said
<https://www.thetimes.co.uk/article/enemy-hackers-target-nato-with-constant-cyberattacks-dj263x2lh>
that “NATO computer systems are facing almost daily cyberattacks.”

The war of the future will not only be about explosions, but will also be
about disabling the systems that make armies run. It’s not (solely) that
bases will get blown up; it’s that some bases will lose power, data, and
communications. It’s not that self-driving trucks will suddenly go mad and
begin rolling over friendly soldiers; it’s that they’ll casually roll off
roads or into water where they sit, rusting, and in need of repair. It’s
not that targeting systems on guns will be retargeted to 1600 Pennsylvania
Avenue; it’s that many of them could simply turn off and not turn back on
again.

So, how do we prepare for this next war? First, militaries need to
introduce a little anarchy into their planning. Let’s have wargames where
essential systems malfunction or are subvertednot all of the time, but
randomly. To help combat siloed military thinking, include some civilians
as well. Allow their ideas into the room when predicting potential enemy
action. And militaries need to have well-developed backup plans, for when
systems are subverted. In Joe Haldeman’s 1975 science-fiction novel *The
Forever War,* he postulated a “stasis field” that forced his space marines
to rely on nothing more than Roman military technologies, like javelins
<https://www.google.com/books/edition/Military_institutions_of_Vegetius_in_fiv/sQlXAAAAcAAJ?hl=en&gbpv=1&dq=inauthor:%22Flavius+VEGETIUS+RENATUS%22&printsec=frontcover>.
We should be thinking in the same direction.

NATO isn’t yet allowing
<https://foreignpolicy.com/2020/12/22/nato-we-want-to-go-to-war-with-you/>
civilians not employed by NATO or associated military contractors access to
their training cyber ranges where vulnerabilities could be discovered and
remediated before battlefield deployment. Last year, one of us (Tarah) was
listening to a NATO briefing after the end of the 2020 Cyber Coalition
<https://shape.nato.int/news-releases/exercise-cyber-coalition-2020>
exercises, and asked how she and other information security researchers
could volunteer to test cyber ranges used to train its cyber incident
response force. She was told that including civilians would be a “welcome
thought experiment in the tabletop exercises,” but including them in
reality wasn’t considered. There is a rich opportunity for improvement
here, providing transparency into where improvements could be made.

Second, it’s time to take cybersecurity seriously in military procurement,
from weapons systems to logistics and communications contracts. In the
three year span from the original 2018 GAO report to this year’s report,
cybersecurity audit compliance went from 0% to 40% (those 2 of 5 programs
mentioned earlier). We need to get much better. DOD requires
<https://www.csoonline.com/article/3535797/the-cybersecurity-maturity-model-certification-explained-what-defense-contractors-need-to-know.html>
that its contractors and suppliers follow the Cybersecurity Maturity Model
Certification <https://www.acq.osd.mil/cmmc/> process; it should abide by
the same standards. Making those standards both more rigorous and mandatory
would be an obvious second step.

Gone are the days when we can pretend that our technologies will work in
the face of a military cyberattack. Securing our systems will make
everything we buy more expensive -- maybe a lot more expensive. But the
alternative is no longer viable.

The future of war is cyberwar. If your weapons and systems aren’t secure,
don’t even bother bringing them onto the battlefield.

This essay was written with Tarah Wheeler, and previously appeared
<https://www.brookings.edu/techstream/hacked-drones-and-busted-logistics-are-the-cyber-future-of-warfare/>
in Brookings TechStream.

** *** ***** ******* *********** *************
Information Flows and Democracy

*[2021.06.09]*
<https://www.schneier.com/blog/archives/2021/06/information-flows-and-democracy.html>
Henry Farrell and I published a paper on fixing American democracy:
“Rechanneling
Beliefs: How Information Flows Hinder or Help Democracy
<https://snfagora.jhu.edu/publication/rechanneling-beliefs/>.”

It’s much easier for democratic stability to break down than most people
realize, but this doesn’t mean we must despair over the future. It’s
possible, though very difficult, to back away from our current situation
towards one of greater democratic stability. This wouldn’t entail a
restoration of a previous status quo. Instead, it would recognize that the
status quo was less stable than it seemed, and a major source of the
tensions that have started to unravel it. What we need is a dynamic
stability, one that incorporates new forces into American democracy rather
than trying to deny or quash them.

This paper is our attempt to explain what this might mean in practice. We
start by analyzing the problem and explaining more precisely why a
breakdown in public consensus harms democracy. We then look at how these
beliefs are being undermined by three feedback loops, in which
anti-democratic actions and anti-democratic beliefs feed on each other.
Finally, we explain how these feedback loops might be redirected so as to
sustain democracy rather than undermining it.

To be clear: redirecting these and other energies in more constructive ways
presents enormous challenges, and any plausible success will at best be
untidy and provisional. But, almost by definition, that’s true of any
successful democratic reforms where people of different beliefs and values
need to figure out how to coexist. Even when it’s working well, democracy
is messy. Solutions to democratic breakdowns are going to be messy as well.

This is part of our series of papers looking at democracy as an information
system. The first paper was “Common-Knowledge Attacks on Democracy
<https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3273111>.”

** *** ***** ******* *********** *************
Detecting Deepfake Picture Editing

*[2021.06.10]*
<https://www.schneier.com/blog/archives/2021/06/detecting-deepfake-picture-editing.html>
“Markpainting” is a clever technique
<https://www.lightbluetouchpaper.org/2021/06/07/a-new-way-to-detect-deepfake-picture-editing/>
to watermark photos in such a way that makes it easier to detect ML-based
manipulation:

An image owner can modify their image in subtle ways which are not
themselves very visible, but will sabotage any attempt to inpaint it by
adding visible information determined in advance by the markpainter.

One application is tamper-resistant marks. For example, a photo agency that
makes stock photos available on its website with copyright watermarks can
markpaint them in such a way that anyone using common editing software to
remove a watermark will fail; the copyright mark will be markpainted right
back. So watermarks can be made a lot more robust.

Here’s the paper: “Markpainting: Adversarial Machine Learning Meets
Inpainting <https://arxiv.org/abs/2106.00660>,” by David Khachaturov, Ilia
Shumailov, Yiren Zhao, Nicolas Papernot, and Ross Anderson.

*Abstract:* Inpainting is a learned interpolation technique that is based
on generative modeling and used to populate masked or missing pieces in an
image; it has wide applications in picture editing and retouching.
Recently, inpainting started being used for watermark removal, raising
concerns. In this paper we study how to manipulate it using our
markpainting technique. First, we show how an image owner with access to an
inpainting model can augment their image in such a way that any attempt to
edit it using that model will add arbitrary visible information. We find
that we can target multiple different models simultaneously with our
technique. This can be designed to reconstitute a watermark if the editor
had been trying to remove it. Second, we show that our markpainting
technique is transferable to models that have different architectures or
were trained on different datasets, so watermarks created using it are
difficult for adversaries to remove. Markpainting is novel and can be used
as a manipulation alarm that becomes visible in the event of inpainting.

** *** ***** ******* *********** *************
FBI/AFP-Run Encrypted Phone

*[2021.06.11]*
<https://www.schneier.com/blog/archives/2021/06/fbi-afp-run-encrypted-phone.html>
For three years, the Federal Bureau of Investigation and the Australian
Federal Police owned and operated a commercial encrypted phone app, called
AN0M, that was used by organized crime around the world. Of course, the
police were able to read everything -- I don’t even know if this qualifies
as a backdoor. This week, the world’s
<https://www.nytimes.com/2021/06/08/world/australia/operation-trojan-horse-anom.html>
police
<https://www.abc.net.au/news/2021-06-08/fbi-afp-underworld-crime-bust-an0m-cash-drugs-murder/100197246>
organizations
<https://www.theguardian.com/australia-news/2021/jun/08/anom-encrypted-app-fbi-afp-australia-federal-police-sting-operation-ironside-an0m>
announced
<https://www.smh.com.au/national/mass-raids-arrests-across-australia-after-police-sting-dismantles-encrypted-app-used-by-criminals-20210607-p57yya.html>
800
<https://www.reuters.com/world/asia-pacific/australian-police-arrest-over-200-after-cracking-underworld-messaging-app-2021-06-08/>
arrests
<https://www.washingtonpost.com/world/2021/06/08/fbi-app-arrests-australia-crime/>
based on text messages sent over the app. We’ve seen law enforcement take
over encrypted apps before: for example, EncroChat
<https://www.nytimes.com/2020/07/02/world/europe/encrypted-network-arrests-europe.html>.
This operation, code-named Trojan Shield, is the first time law enforcement
managed an app from the beginning.

If there is any moral to this, it’s one that all of my blog readers should
already know: trust is essential to security. And the number of people you
need to trust is larger than you might originally think. For an app to be
secure, you need to trust the hardware, the operating system, the software,
the update mechanism, the login mechanism, and on and on and on. If one of
those is untrustworthy, the whole system is insecure.

It’s the same reason blockchain-based currencies are so insecure
<https://www.schneier.com/blog/archives/2019/02/blockchain_and_.html>, even
if the cryptography is sound.

** *** ***** ******* *********** *************
TikTok Can Now Collect Biometric Data

*[2021.06.14]*
<https://www.schneier.com/blog/archives/2021/06/tiktok-can-now-collect-biometric-data.html>
This
<https://techcrunch.com/2021/06/03/tiktok-just-gave-itself-permission-to-collect-biometric-data-on-u-s-users-including-faceprints-and-voiceprints/>
is probably worth paying attention to:

A change to TikTok’s U.S. privacy policy
<https://www.tiktok.com/legal/privacy-policy?lang=en#privacy-us> on
Wednesday introduced a new section that says the social video app *“may
collect biometric identifiers and biometric information”* from its users’
content. This includes things like *“faceprints and voiceprints,”* the
policy explained. Reached for comment, TikTok could not confirm what
product developments necessitated the addition of biometric data to its
list of disclosures about the information it automatically collects from
users, but said it would ask for consent in the case such data collection
practices began.

** *** ***** ******* *********** *************
Upcoming Speaking Engagements

*[2021.06.14]*
<https://www.schneier.com/blog/archives/2021/06/upcoming-speaking-engagements-9.html>
This is a current list of where and when I am scheduled to speak:

   - I’ll be part of a European Internet Forum virtual debate
   <https://www.internetforum.eu/events/events/1127-decrypting-the-encryption-debate-how-to-ensure-public-safety-with-a-privacy-preserving-and-secure-internet.html>
   on June 17, 2021. The topic is “Decrypting the encryption debate: How to
   ensure public safety with a privacy-preserving and secure Internet?”
   - I’m speaking at the all-online Society for Philosophy and Technology
   Conference 2021 <https://lillethics.com/spt-2021/>, June 28-30, 2021.
   - I’m keynoting the 5th International Symposium on Cyber Security
   Cryptology and Machine Learning
   <https://www.cs.bgu.ac.il/~fradmin/cscml21/index.html> (via Zoom), July
   8-9, 2021.
   - I’m speaking (via Internet) at SHIFT Business Festival
   <https://theshift.fi/persons/bruce-schneier/> in Finland, August 25-26,
   2021.
   - I’ll be speaking at an Informa <https://www.informa.com/> event on
   September 14, 2021. Details to come.

The list is maintained on this page <https://www.schneier.com/events/>.

** *** ***** ******* *********** *************

Since 1998, CRYPTO-GRAM has been a free monthly newsletter providing
summaries, analyses, insights, and commentaries on security technology. To
subscribe, or to read back issues, see Crypto-Gram's web page
<https://www.schneier.com/crypto-gram/>.

You can also read these articles on my blog, Schneier on Security
<https://www.schneier.com>.

Please feel free to forward CRYPTO-GRAM, in whole or in part, to colleagues
and friends who will find it valuable. Permission is also granted to
reprint CRYPTO-GRAM, as long as it is reprinted in its entirety.

Bruce Schneier is an internationally renowned security technologist, called
a security guru by the Economist. He is the author of over one dozen books
-- including his latest, We Have Root <https://www.schneier.com/books/root/>
-- as well as hundreds of articles, essays, and academic papers. His
newsletter and blog are read by over 250,000 people. Schneier is a fellow
at the Berkman Klein Center for Internet & Society at Harvard University; a
Lecturer in Public Policy at the Harvard Kennedy School; a board member of
the Electronic Frontier Foundation, AccessNow, and the Tor Project; and an
Advisory Board Member of the Electronic Privacy Information Center and
VerifiedVoting.org. He is the Chief of Security Architecture at Inrupt, Inc.

Copyright © 2021 by Bruce Schneier.

** *** ***** ******* *********** *************

Mailing list hosting graciously provided by MailChimp
<https://mailchimp.com/>. Sent without web bugs or link tracking.

This email was sent to: sondheim at panix.com
*You are receiving this email because you subscribed to the Crypto-Gram
newsletter.*

unsubscribe from this list
<https://schneier.us18.list-manage.com/unsubscribe?u=f99e2b5ca82502f48675978be&id=22184111ab&e=9954231fa6&c=cac190423f>
    update subscription preferences
<https://schneier.us18.list-manage.com/profile?u=f99e2b5ca82502f48675978be&id=22184111ab&e=9954231fa6&c=cac190423f>
Bruce Schneier · Harvard Kennedy School · 1 Brattle Square · Cambridge, MA
02138 · USA


-- 
*=====================================================*

*directory http://www.alansondheim.org <http://www.alansondheim.org> tel
718-813-3285**email sondheim ut panix.com <http://panix.com>, sondheim ut
gmail.com <http://gmail.com>*
*=====================================================*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.netbehaviour.org/pipermail/netbehaviour/attachments/20210617/c2c5e580/attachment.htm>


More information about the NetBehaviour mailing list