Update: On January 29, 2019, I heard back from Facebook (27-days after I initially published this research) and they informed me they had taken action with Adobe to turn off auto-fill functionality on their forms based on the mkt_tok URL parameter.
Facebook’s about-face on this type of URL authentication is a huge win for privacy-advocates concerned with One-Token-User-Authentication URL parameters and their impact on user privacy.
Facebook also made sure their marketing subdomain @ content.fb.com was properly served over HTTPS (this is still an ongoing migration), they are evaluating adding text to not forward marketing emails, are working to potentially whitelist certain domains and other efforts to limit the “mkt_tok” URL token leaks, and Facebook even announced they were limiting the use of external resources (JS, CSS) on pages that provide auto-fill functionality and receive mkt_tok values.
Facebook also reversed their original decision to issue me an average-sized bounty for the research and sent over some other helpful tips for future submissions. All in all, Facebook seems to have taken nearly all of the suggestions in this original article still included below (and they seem to be working on a few of the more advanced strategies to protect users who leak their own tokens but wouldn’t provide details when I inquired further).
Here’s the email Facebook sent to me announcing their changes:
Facebook Adopted Fixes, Adobe Still Largely Silent
I’m still a little surprised that Facebook made the changes that I suggested in this research, particularly some of the changes that could impact their opt-in conversion rates, which is a surefire sign that the Facebook Growth team is having to take a backseat to the privacy and security team when subtle conflicts arise.
I’m not however surprised that Adobe and their subsidiary Marketo have been largely silent. I’ve seen no messages to their clients, no steps to help their other non-Facebook clients implement security precautions, and it’s likely that the steps Facebook took to protect itself from Adobe’s Marketo software vulnerabilities are not being taken by other Marketo clients. If I hear differently, I’ll update this post. I’ll also continue to monitor the use of One-Token-User-Authentication URL parameters like “mkt_tok” and will urge more folks start to look into the importance of using multiple cookies and values to authenticate a user prior to any user data sharing.
…Here’s the original research on Facebook + Adobe that I published on January 2, 2019, that led to Facebook’s changes at the end of January 2019….
Over the last 2 months, Facebook has quietly changed several ways they deploy Adobe’s Marketo email software across core Facebook marketing programs and websites in order to reduce the impact of a one-token-user-authentication user data vulnerability. But Facebook also continues to send Facebook users emails on a daily basis containing one-token-user-authentication vulnerabilities, and they continue to host websites and signup forms that are vulnerable to targeted user data scraping.
Facebook’s user data vulnerabilities have existed for over a year, and there are Adobe Marketo forms built in 2017 that are still hosted on Facebook’s domains. These web pages and the ongoing Facebook email strategy make it possible for a Facebook user to share a URL that can be used to expose their email address and other private user information.
One important caveat to this user data vulnerability is that it *does not* impact normal Facebook user accounts that are used by the ~2 billion people (newsfeed, messenger, ads manager, etc.). This user data vulnerability merely impacts the “Shadow b2b Facebook User Accounts” built by Facebook and hosted by Adobe’s Marketo email software that Facebook uses to engage industry experts and encourage them to utilize Facebook tools, and then more recently the vulnerability was extended to Facebook Workplace users due to a deep integration between Facebook Workplace and the Adobe Marketo email software.
One additional caveat, Facebook has been notified about these user data vulnerabilities via their Whitehat disclosure program in a thread with me containing over 30 messages, initially started on October 29, 2018.
Facebook acknowledged the insecurities in Marketo in plain English on December 3, 2018, by writing “at this time sharing those links is not recommended,” but have continued to send out the links to users via email on a regular basis, and have kept dozens of other web pages hosting Marketo forms active that facilitate the data scraping vulnerability. Unfortunately, both sides of this user data vulnerability equation are active and ongoing.
Facebook has since closed the whitehat thread with me, marked it as “Informative” but have kept huge portions of the vulnerability active and ongoing. Due to Facebook’s lack of remedy and ongoing actions, it seemed important to disclose this to users so they can protect themselves. I let Facebook know I was going to disclose this via a public post and sent them one more update this week.
Finally, Adobe was notified and confirmed receipt of this user data vulnerability existing in Marketo on September 25, 2018 (more on this and screenshots of the conversations below), more than a month before they completed the acquisition of Marketo on October 31st, 2018 and have taken no visible steps since then to alert clients to ongoing user data vulnerabilities or fix these foundational problems in their Marketo software.
This blog post is going to largely focus on Facebook’s deployment of Adobe’s Marketo email software, and why due to Facebook’s internal marketing team’s needs to “move fast and break things” and their need to flexibly control external email newsletters, a feature that (for some reason) doesn’t currently exist within Facebook’s tech stack, their reliance on Marketo created vulnerabilities across different Facebook divisions, external programs, and VIP user bases containing high-spending ad buyers and Facebook platform developers.
Unfortunately, by disclosing this one-token-user-authentication user data vulnerability, it also puts other Adobe Marketo clients at risk of targeted data exfiltration — but Adobe has experience fixing this problem with their own Adobe Marketing Cloud software (I disclosed a similar problem to them last Spring, more below on that too), and Adobe needs to take the same steps with their new acquisition Marketo.
Unfortunately for Adobe, it may be more complicated to deploy two-token-user-authentication within a legacy data architecture like Marketo due to Marketo’s heavy-reliance on one-token-user-authentication. Marketo uses the same user token to both track email opens and clicks, and also uses that same exact token to control updating and accessing user preferences and contact information. “One user token to rule them all,” if you will.
In short, this vulnerability exists due to a string of text, a variable called “mkt_tok” — this string of text is 23 alphanumeric characters — it’s appended to the end of website URLs by Marketo’s email software after a user clicks a link in an email sent to them by a Marketo client.
That “mkt_tok” token controls both read/write access to the user data, and it also tracks the email opens/clicks. The “mkt_tok” user token is shared from Marketo’s email software, and it seems that nearly all email newsletters sent through their platform have this user token appended to URLs. This means that anytime Client A shares Website B with User C, if that user clicks to visit the website, the user token will be appended to the URL and Website B will see this private one-token-user-authentication variable in all their server logs and analytics tools — and that user token is also shared across the data supply chain to dozens or hundreds of advertising and analytics companies on some popular websites.
On multiple occasions, Facebook sent newsletter emails to their users that linked the user to a 3rd party website, while also appending the users private “mkt_tok” to the URL after clicking the link in the email, and in milliseconds of a user clicking an email, their private user token would pass to the 3rd party website and any advertising and analytics tools that are fired on that domain that grab the final resolved URL.
This one-token-user-authentication leak also occurs when a user clicks to visit a web page, has their user token appended to their URL, and then shares that blog post or page they are visiting publicly — like by writing a public post on Facebook. This is the most common way that Facebook’s VIP users are exposing themselves, and it starts with them sharing Facebook Marketing and Developer news that they were sent via email from Facebook, back onto Facebook. This full loop of “promoting Facebook content on Facebook” exposes a users “mkt_tok” token publicly in the URL they share.
And unfortunately, Facebook also still lets any user search Facebook to look for public posts that contain the URL variable “mkt_tok” (as does Google) — so these user tokens are widely available in public locations. Facebook was notified about these issues but made only minimal changes before closing the issue, so it seems important to alert my fellow marketers and developers to this vulnerability so they stop sharing links sent via these Facebook programs. Facebook Workplace users should also be warned, due to the inclusion of these tokens in new-user Facebook Workplace onboarding email campaigns sent by Facebook.
Finally, this blog post will briefly touch upon the dangers of one-token-user-authentication, why user tokens are rightfully-protected in both Europe’s user privacy framework known as GDPR but also the more-recent California Consumer Privacy Act (CCPA), and also will briefly critique the general concept of “auto-fill forms” and the need to treat them much more like a pseudo-user-login forms than some sort of marketing growth hack, which got Facebook into this mess in the first place.
Without further adieu…
Facebook user flow prior to accidental “mkt_tok” one-token-user-authentication vulnerability
While disclosing this to Facebook, I showed them a video of me navigating from a Facebook Developer Newsletter to an Instagram + Adobe Spark signup webpage @ https://go.fb.com/sparkAR-instagram.html, and then how the form on that page was using Marketo auto-fill form technology via one-token-user-authentication. There are screenshots of that process below.
After disclosure, Facebook removed the form I was testing on and redirected that page to a new secure form @ https://www.facebook.com/arp/ig/beta, but dozens of other legacy forms with the original vulnerability exists across Facebook domains, and Facebook continues to send out the “mkt_tok” user token-appended URLs on a daily basis via multiple Facebook divisions/programs.
Globally, any legacy form with Marketo auto-fill technology that uses one-token-user-authentication via “mkt_tok” and is “properly” setup via Marketo’s CNAME DNS data mapping, can be turned into a user data scraping machine with minimal effort to pull data from a client Marketo account. By merely copying/pasting/replacing the “mkt_tok” text string parameter in any URL hosting a legacy Marketo form with “auto-fill” turned on, and by using a new cookie-free session to make each page request, Marketo will append unique user data into a form via auto-fill (and also pass that user data via HTMl/the data layer, depending on the page setup). For custom forms, any unique form fields will also leak unique data. Furthermore, the unsubscribe pages and other email preference pages built by Marketo and hosted on client subdomains are ripe for additional data leaking via mkt_tok one-token-user-authentication, but fortunately, Facebook has a custom setup that requires multiple tokens to access these pages.
Facebook’s user data exfiltration vulnerability only exists with auto-fill Marketo forms on their pages, but this is still a Facebook user data exfiltration vulnerability and far too simple for an attacker to execute. This is what a vulnerable URL looks like:
Here’s a simplistic technical map of this data flow from DNSdumpster.com showing how the insecure http://content.fb.com subdomain is CNAME-mapped to Adobe’s Marketo to share user data between the two companies via this non-SSL subdomain.
Here’s a screenshot walkthrough of a Facebook Developer newsletter email that contained the “mkt_tok” one-token-user-authentication vulnerability, and the front-end user experience as data is leaked.
Step One: Facebook Developer Gets an Email from Facebook Promoting Facebook Developer News
Step Two: After Clicking the Email, the User is Sent Through a URL Redirect on http://content.fb.com,
This insecure redirect is hosted on content.fb.com in dynamically generated (but permanent) pages and uses a javascript snippet to append a “mkt_tok” one-token-user-authentication URL variable to the final resolved URL.
Step Three: Facebook User is Redirected to the Final Resolved URL with “mkt_tok” Appended as a URL Parameter, Open and Exposed — a Growth Hack Feature and a User Data Vulnerability
Step Four: On a “Properly Setup” Page with Adobe Marketo Auto-Fill Form Technology, the User Data is Exposed by Only Appending the “mkt_tok” Parameter to the URL
Step Four Bonus: Facebook Users Easily Exposed Their “mkt_tok” URLs via Native Facebook Sharing Tools
Unfortunately, many Facebook users exposed themselves and their own “mkt_tok” user parameter, including the VIP marketer pictured at the top of this post with Facebook CEO Mark Zuckerberg.
This is how users exposed their own “mkt_tok” token by promoting Facebook news:
Facebook’s Adobe Marketo problem and the “mkt_tok” one-token-user-authentication user data vulnerability comes from http://content.fb.com
Facebook’s user data exfiltration vulnerability impacts Facebook Workplace users, Facebook Marketing IQ program users, Facebook Developers, and Facebook for Business users.
Facebook’s vulnerability exists due to their implementation of software owned by Adobe called Marketo — this Marketo email software is installed on the Facebook-controlled http://content.fb.com subdomain via CNAME DNS record mapping and the Marketo software is then used by internal Facebook marketing staff to promote Facebook programs and tools.
The insecure subdomain used in this data flow is ripe for attack, both locally and at a network level, but more importantly is an avenue for user data exfiltration.
From my perspective, I don’t totally understand all of the vulnerabilities with this subdomain being non-SSL, beyond the issues outlined in this post. But I look forward to more external developers viewing and analyzing the http://content.fb.com subdomain and the redirect javascript to provide additional context. I won’t, however, share one of the redirect URls, you’ll have to find one yourself or signup for Facebook newsletter emails.
Deployment of Adobe Marketo in the Facebook Workplace New User Onboarding Emails
Facebook uses Adobe’s Marketo email software for a wide variety of their programs, but many people would be surprised to find that every new user of Facebook Workplace will receive emails containing their private“mkt_tok” token.
These insecure emails are part of an “onboarding email series” to get new users familiar with Facebook Workplace. At least one email in this Facebook Workplace new user onboarding links users to Facebook’s TransformYourWorkplace.com, which also passes those visitors “mkt_tok” user data to Google Analytics.
One email in the onboarding flow is un-ironically titled, “Keep your data safe with Workplace” and the link in the email appends the “mkt_tok” user token:
Not-so Safety Check
Additionally, Workplace by Facebook has been onboarding “Early Access” to their “Safety Check” feature for workplace emergencies, and built a webpage and form for this using Adobe’s Marketo software, a form that has the user data exfiltration vulnerability. This form also includes a custom field that can be scraped (if a user submitted that specific form and also had their “mkt_tok” leaked) that asks users the open-ended question, “Anything else we should know?”
Want to signup for email updates via Facebook’s Newsletters or Facebook Workplace?
As of publishing, each of these Facebook newsletter signups could lead to you receiving an email with an insecure “mkt_tok” token. You shouldn’t share any link containing this URL parameter publicly.
Facebook Developer Updates (You’ll need to click around, these signups are across a bunch of programs)
Facebook Confirmed that Adobe Marketo’s Email Links are Unsafe for Users
Facebook was initially informed about this user data vulnerability on October 29th, 2018, but my initial email didn’t properly explain the form scraping vulnerability, and their initial response on November 1st, 2018 rejected any security concerns and stating, “This is intentional behavior in our product” and “While it’s true that these links, when inadvertently shared the way that you’ve shown they can be, may lead to unintended changing of a user’s email preferences, we do not encourage users to share these links or to make them publicly available. As such, we do not consider it a security vulnerability, but we do have controls in place to monitor and mitigate abuse.”
((NOTE: Facebook uses Cloudflare on http://content.fb.com to protect against malicious script injections on their insecure subdomain, but Cloudflare can’t do anything to protect against a “software feature” that facilitates user data scraping with a known “mkt_tok” variable. Cloudflare can protect against all kinds of URL injections and automated scripts, but I saw almost no Cloudflare errors during my tests because it’s considered “intended user behavior” to auto-append the data into the forms))
After Facebook’s initial rejection of security concerns, I recorded a video (the screenshots above in this post are from that video), and further explained how the one-token-user-authentication “mkt_tok” URL variables were being appended to *all* links being sent out in these newsletters, and also how the auto-fill forms they were using facilitated the scraping of user data using any exposed user tokens. After a bit of back and forth, on November 18th, 2018, Facebook asked for “concise reproduction instructions/steps for how an attacker would exploit this token after the victim unwittingly shares it?”
I responded to Facebook on November 19th with additional steps on how an attacker could append a user “mkt_tok” one-token-user-authentication to a URL on Facebook that was hosting an Adobe Marketo email form with auto-fill technology to access and scrape the user data. On November 29th, 2018, Facebook security sent a rambling response that †ruly must be read to be believed:
After several additional messages, Facebook responded on December 3rd, 2018 writing, “At this time sharing those links is not recommended. While not ideal those links are generated by Marketo, which is not a Facebook product.”
Facebook doesn’t consider one-token-user-authentication secure, and will admit as much, but hasn’t changed their practices…. so what next?
This data exfiltration vulnerability appears to only impact a small number of Facebook users — albeit mostly VIP users and core platform developers and marketers.
One of the most alarming parts of this disclosure is that even though it was mentioned to them many times, they have taken no efforts to remove the ability to search for this exposed token via Facebook’s native search. As of publishing time, you can still go on Facebook and search for “mkt_tok” (*and other insecure one-token-user-authentication URL variables from other software companies*) and view not only the Facebook users exposing themselves via Facebook’s programs but also users exposing themselves from links sent by non-Facebook Adobe Marketo clients.
Facebook made a few minor changes as I was reporting these issues to them, but the issue persists and user tokens continue to get exposed. This is also only the tip of the iceberg for one-token-user-authentication URL variables and the need for software vendors to move away from this insecure authentication protocol. Unfortunately, other insecure URL variables are also being shared on Facebook daily without being removed. The question remains, who is responsible for protecting a user from themselves when a URL parameter can be used to access their private information?
Adobe was Notified of Marketo’s “mkt_tok” User Data Vulnerability on September 25th, 2018, Over a Month Before Completing the Purchase of Marketo on October 31st, 2018
Adobe’s security team is typically very responsive via email and they obviously triage some security problems within hours of an initial report. As was typical, in the screenshot below you will see their team responding to my initial email to them on September 25th, 2018 about the Marketo “mkt_tok” user token vulnerability — I sent this email to them after seeing news reports that they were in the process of acquiring Marketo. They opened a ticket (PSIRT -8927) about the report on September 25th, but went silent on it. Adobe completed the acquisition of Marketo on October 31st and still hasn’t followed up.
Adobe *had* user data problems similar to Marketo, and were alerted to a similar one-token-user-authentication user data vulnerability via Adobe’s “pkey” URL variable on April 2, 2018
Way back on April 2, 2018, I emailed Adobe and alerted them to a vulnerability impacting their “pkey” one-token-user-authentication URL variable and client subdomains that were caching user sessions via this URL variable and putting Adobe end-user data at risk (mostly exposing user emails and certain documents that were being emailed to users). The initial organization I flagged for them which had user data exposed was their client the Boy Scouts of America. They responded to my initial email in under 90 minutes and from my limited-view of their response efforts, they moved pretty swiftly from a couple fronts to fix the problems. Since then, and prior to their completed acquisition of Marketo, it appears as though they removed all one-token-user-authentication from their software stacks.
Here’s a screenshot of my initial email and Adobe’s PSIRT case #8018 — this was from April 2, 2018:
For the last few months, even though someone could find remnants of problems across Adobe client installs, (google the phrase: “inurl:pkey”), it would appear that every one-token-user-authentication problem of Adobe’s seems to have been fixed by denying the requests (aka breaking the URLs) — this basically broke lots of links and may have created problems for clients — but it also prevented any data exposure and was probably the correct decision. This was a bold choice from Adobe that I was surprised they never publicized, but I suspect they didn’t want to publicly admit to the data exposure via URL token caching since it was probably resolved before the GDPR deadline.
But now, Adobe owns Marketo as of October 31st, 2018, and almost assuredly is planning the same process to make Marketo secure that they deployed across Adobe earlier in 2018, but Adobe has neither publicly disclosed the ongoing security problems with their new acquisition Marketo, privately reported the ongoing security problems to their Marketo clients or taken the bold-yet-necessary steps to just break all existing Marketo client URLs that attempt to authenticate and access user data via the one-token-user-authentication “mkt_tok” URL variable.
Adobe has experience fixing security problems from one-token-user-authentication via their “pkey” URL variable — when will they take steps to fix their “mkt_tok” URL variable?
The California Consumer Privacy Protection Act (CCPA) Includes Protections for User Tokens
The California Consumer Privacy Protection Act (CCPA) signed into law and going into effect in January 2020 includes this paragraph when explaining the scope of the law (and why I’d argue CCPA includes protections for “mkt_tok” from being shared/used without consent):
“(x) ‘Unique identifier’ or ‘Unique personal identifier’ means a persistent identifier that can be used to recognize a consumer, a family, or a device that is linked to a consumer or family, over time and across different services, including, but not limited to, a device identifier; an Internet Protocol address; cookies, beacons, pixel tags, mobile ad identifiers, or similar technology; customer number, unique pseudonym, or user alias; telephone numbers, or other forms of persistent or probabilistic identifiers that can be used to identify a particular consumer or device. “
It shouldn’t come as a surprise that legal opinions will eventually determine that CCPA includes protections for “mkt_tok” and the type of URL parameter/cookie with one-token-user-authentication privileges that can not only be used to track a user but also access their private information.
GDPR Already Includes Protection for “mkt_tok” and Other User Tokens
The General Data Protection Regulation framework implemented across Europe on May 25, 2018, includes this paragraph when explaining the scope of the law, a clear reference to user tokens like “mkt_tok”:
“Natural persons may be associated with online identifiers provided by their devices, applications, tools and protocols, such as internet protocol addresses, cookie identifiers or other identifiers such as radio frequency identification tags. This may leave traces which, in particular when combined with unique identifiers and other information received by the servers, may be used to create profiles of the natural persons and identify them.”
So the question remains for Adobe and Facebook, have European Authorities been notified of their “mkt_tok” user data exfiltration vulnerabilities and ongoing data leaks?
Auto-fill forms are inherently insecure — it’s very uncommon to see secure setups
During the research for this post to better understand the types of user data authentication being used by “enterprise” marketing apps, it became abundantly clear that there aren’t many ways to have an auto-fill form without creating unique user data vulnerabilities. The reality is that anytime a plain-text email address or other user PII is “pasted” into a webpage, that data still needs to remain encrypted and only partially visible (aka partially hashed emails). Otherwise, one bad development decision can turn into a massive data supply-chain leak if your marketing team starts to fire 3rd party advertising and analytics pixels for those website visitors, which passes the full URL to other 3rd-party companies (and sometimes text/fields on the page, see Adroll’s “Data Shotgun” approach). As of publishing, there are unfortunately dozens of zero-day data exfiltration vulnerabilities using one-token-user-authentication across the marketing software ecosystem.
There are also dozens of enterprise marketing companies that create user unsubscribe/preference pages that are insecure. Facebook itself argued that even though the “mkt_tok” user variable could let an attacker access and change a Facebook user’s email preferences, that wasn’t a security vulnerability — and Facebook even had the gall to cite an FTC advisory opinion about un-subscribe links and CAN-SPAM compliance. 😀
If Facebook had been paying attention, industry leaders like MailChimp have been hashing user emails on email unsubscribe/preference pages for years — aka, following the FTC opinion but also not putting user data at risk of exfiltration. MailChimp is, unfortunately, the exception and there are plenty of insecure enterprise email tools….but other companies insecure existence doesn’t mean their data architecture should be modeled by massive companies like Facebook.
Conclusion — Facebook and Adobe Need to Answer Questions and Remove User Data Exfiltration Vulnerabilities from “mkt_tok”
The information contained in this post was previously disclosed to Facebook, Adobe, and Marketo across several long email chains. Adobe appears to have taken little action to resolve or remedy their vulnerabilities since finishing the acquisition of Marketo on October 31, 2018, and took some steps that made it seem to indicate they had no intention of publicly warning their clients or reporting these vulnerabilities as data controllers under GDPR. Facebook has made a few minor changes, but continues to send out emails regularly with the “mkt_tok” user token and continues to host Marketo forms with auto-fill technology that is vulnerable to user scraping.
One of the most interesting aspects of this research has been the clear admissions from Facebook security staff that they didn’t think Adobe’s Marketo email software was secure for users. The facts are pretty clear about how these tokens are exposed daily, and how they can be used to access user data, yet it seems like either Facebook is waiting for Adobe to fix the problem, or Adobe is waiting to deny the insecure “mkt_tok” one-token-user-authentication user data requests until after they built a new data architecture.
If both Adobe and Facebook are basically partnering with a custom deployment of Marketo, and exposing user data across the global supply chain, who has exposure as the data controller? Both of them? Only the originator of the user agreement, aka Facebook?
Significant questions remain about the legal exposure of enterprise companies who deploy insecure software from other companies, and their unique data controller requirements under existing user privacy frameworks.
Questions that need answers :
When Facebook sent an email newsletter that linked Facebook Developers to the European Tech Publisher TheNextWeb (this blog post of theirs), while also appending the “mkt_tok” URL variable to the final resolved URL which was passed to TNW and dozens of their advertising partners, would Facebook be required to submit information about this to European regulators or request deletion?
Should Facebook and Google let users search for URL parameters? Should it be so easy to go on Facebook and search “mkt_tok” to find exposed users?
Should all software companies be required to organize and publish their one-token-user-authentication URL parameters so other companies can protect themselves from inadvertently ingesting or sharing those user tokens?
Should Facebook, Google, Cloudflare and other major service providers publish lists of URL parameters and cookie variables (and known details about them) to help publishers and other companies protect themselves from inadvertently ingesting or sharing those user tokens?
Should all massive companies map the URL parameters and user tokens created across their business and subsidiaries? Should companies track and create user data maps to determine data exposure and leaks along a data supply chain and along the user consent chain?
Questions? Concerns? Revisions?
Tweet me @thezedwards, @ zach@victorymedium.com, or a spam Linkedin connection request here… and while you’re at it, want to check out my proposal for a new digital Cookie Exchange using the Shapley Value formula and ETF baskets traded on a public market to automate click arbitrage? Click here to wonk out. Thanks for any feedback!
Recent Comments