[feature] Parse content warning to HTML, serialize via client API as plaintext (#3876)
* [feature] Parse content warning as HTML, serialize via API to plaintext
* tidy up some cruft
* whoops
* oops
* i'm da joker baybee
* clemency muy lorde
* rename some of the text functions for clarity
* jiggle the opts
* fiddle de deee
* hopefully the last test fix i ever have to do in my beautiful life
"content":"hello world! #welcome ! first post on the instance :rainbow: !",
"content":"<p>hello world! <a href=\"http://localhost:8080/tags/welcome\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>welcome</span></a> ! first post on the instance :rainbow: !</p>",
simpleMarkdown="# Title\n\nHere's a simple text in markdown.\n\nHere's a [link](https://example.org)."
simpleMarkdownExpected="<h1>Title</h1><p>Here's a simple text in markdown.</p><p>Here's a <a href=\"https://example.org\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">link</a>.</p>"
withCodeBlockExpected="<h1>Title</h1><p>Below is some JSON.</p><pre><code class=\"language-json\">{\n "key": "value",\n "another_key": [\n "value1",\n "value2"\n ]\n}\n</code></pre><p>that was some JSON :)</p>"
withInlineCode="`Nobody tells you about the <code><del>SECRET CODE</del></code>, do they?`"
withInlineCodeExpected="<p><code>Nobody tells you about the <code><del>SECRET CODE</del></code>, do they?</code></p>"
withInlineCode2="`Nobody tells you about the </code><del>SECRET CODE</del><code>, do they?`"
withInlineCode2Expected="<p><code>Nobody tells you about the </code><del>SECRET CODE</del><code>, do they?</code></p>"
withHashtag="# Title\n\nhere's a simple status that uses hashtag #Hashtag!"
withHashtagExpected="<h1>Title</h1><p>here's a simple status that uses hashtag <a href=\"http://localhost:8080/tags/hashtag\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>Hashtag</span></a>!</p>"
withTamilHashtag="here's a simple status that uses a hashtag in Tamil #தமிழ்"
withTamilHashtagExpected="<p>here's a simple status that uses a hashtag in Tamil <a href=\"http://localhost:8080/tags/%E0%AE%A4%E0%AE%AE%E0%AE%BF%E0%AE%B4%E0%AF%8D\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>தமிழ்</span></a></p>"
mdWithHTML="# Title\n\nHere's a simple text in markdown.\n\nHere's a <a href=\"https://example.org\">link</a>.\n\nHere's an image: <img src=\"https://gts.superseriousbusiness.org/assets/logo.png\" alt=\"The GoToSocial sloth logo.\" width=\"500\" height=\"600\">"
mdWithHTMLExpected="<h1>Title</h1><p>Here's a simple text in markdown.</p><p>Here's a <a href=\"https://example.org\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">link</a>.</p><p>Here's an image:</p>"
mdWithCheekyHTML="# Title\n\nHere's a simple text in markdown.\n\nHere's a cheeky little script: <script>alert(ahhhh)</script>"
mdWithCheekyHTMLExpected="<h1>Title</h1><p>Here's a simple text in markdown.</p><p>Here's a cheeky little script:</p>"
mdWithSmartypants="\"you have to quargle the bleepflorp\" they said with 1/2 of nominal speed and 1/3 of the usual glumping"
mdWithSmartypantsExpected="<p>\"you have to quargle the bleepflorp\" they said with 1/2 of nominal speed and 1/3 of the usual glumping</p>"
mdWithAsciiHeart="hello <3 old friend <3 i loved u </3 :(( you stole my heart"
mdWithAsciiHeartExpected="<p>hello <3 old friend <3 i loved u </3 :(( you stole my heart</p>"
mdWithStrikethrough="I have ~~mdae~~ made an error"
mdWithStrikethroughExpected="<p>I have <del>mdae</del> made an error</p>"
mdWithLink="Check out this code, i heard it was written by a sloth https://github.com/superseriousbusiness/gotosocial"
mdWithLinkExpected="<p>Check out this code, i heard it was written by a sloth <a href=\"https://github.com/superseriousbusiness/gotosocial\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">https://github.com/superseriousbusiness/gotosocial</a></p>"
mdObjectInCodeBlock="@foss_satan@fossbros-anonymous.io this is how to mention a user\n```\n@the_mighty_zork hey bud! nice #ObjectOrientedProgramming software you've been writing lately! :rainbow:\n```\nhope that helps"
mdObjectInCodeBlockExpected="<p><span class=\"h-card\"><a href=\"http://fossbros-anonymous.io/@foss_satan\" class=\"u-url mention\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">@<span>foss_satan</span></a></span> this is how to mention a user</p><pre><code>@the_mighty_zork hey bud! nice #ObjectOrientedProgramming software you've been writing lately! :rainbow:\n</code></pre><p>hope that helps</p>"
simpleMarkdown="# Title\n\nHere's a simple text in markdown.\n\nHere's a [link](https://example.org)."
simpleMarkdownExpected="<h1>Title</h1><p>Here's a simple text in markdown.</p><p>Here's a <a href=\"https://example.org\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">link</a>.</p>"
withCodeBlockExpected="<h1>Title</h1><p>Below is some JSON.</p><pre><code class=\"language-json\">{\n "key": "value",\n "another_key": [\n "value1",\n "value2"\n ]\n}\n</code></pre><p>that was some JSON :)</p>"
withInlineCode="`Nobody tells you about the <code><del>SECRET CODE</del></code>, do they?`"
withInlineCodeExpected="<p><code>Nobody tells you about the <code><del>SECRET CODE</del></code>, do they?</code></p>"
withInlineCode2="`Nobody tells you about the </code><del>SECRET CODE</del><code>, do they?`"
withInlineCode2Expected="<p><code>Nobody tells you about the </code><del>SECRET CODE</del><code>, do they?</code></p>"
withHashtag="# Title\n\nhere's a simple status that uses hashtag #Hashtag!"
withHashtagExpected="<h1>Title</h1><p>here's a simple status that uses hashtag <a href=\"http://localhost:8080/tags/hashtag\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>Hashtag</span></a>!</p>"
withTamilHashtag="here's a simple status that uses a hashtag in Tamil #தமிழ்"
withTamilHashtagExpected="<p>here's a simple status that uses a hashtag in Tamil <a href=\"http://localhost:8080/tags/%E0%AE%A4%E0%AE%AE%E0%AE%BF%E0%AE%B4%E0%AF%8D\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>தமிழ்</span></a></p>"
mdWithHTML="# Title\n\nHere's a simple text in markdown.\n\nHere's a <a href=\"https://example.org\">link</a>.\n\nHere's an image: <img src=\"https://gts.superseriousbusiness.org/assets/logo.png\" alt=\"The GoToSocial sloth logo.\" width=\"500\" height=\"600\">"
mdWithHTMLExpected="<h1>Title</h1><p>Here's a simple text in markdown.</p><p>Here's a <a href=\"https://example.org\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">link</a>.</p><p>Here's an image:</p>"
mdWithCheekyHTML="# Title\n\nHere's a simple text in markdown.\n\nHere's a cheeky little script: <script>alert(ahhhh)</script>"
mdWithCheekyHTMLExpected="<h1>Title</h1><p>Here's a simple text in markdown.</p><p>Here's a cheeky little script:</p>"
mdWithSmartypants="\"you have to quargle the bleepflorp\" they said with 1/2 of nominal speed and 1/3 of the usual glumping"
mdWithSmartypantsExpected="<p>\"you have to quargle the bleepflorp\" they said with 1/2 of nominal speed and 1/3 of the usual glumping</p>"
mdWithAsciiHeart="hello <3 old friend <3 i loved u </3 :(( you stole my heart"
mdWithAsciiHeartExpected="<p>hello <3 old friend <3 i loved u </3 :(( you stole my heart</p>"
mdWithStrikethrough="I have ~~mdae~~ made an error"
mdWithStrikethroughExpected="<p>I have <del>mdae</del> made an error</p>"
mdWithLink="Check out this code, i heard it was written by a sloth https://github.com/superseriousbusiness/gotosocial"
mdWithLinkExpected="<p>Check out this code, i heard it was written by a sloth <a href=\"https://github.com/superseriousbusiness/gotosocial\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">https://github.com/superseriousbusiness/gotosocial</a></p>"
mdWithLinkBasicExpected="Check out this code, i heard it was written by a sloth <a href=\"https://github.com/superseriousbusiness/gotosocial\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">https://github.com/superseriousbusiness/gotosocial</a>"
mdObjectInCodeBlock="@foss_satan@fossbros-anonymous.io this is how to mention a user\n```\n@the_mighty_zork hey bud! nice #ObjectOrientedProgramming software you've been writing lately! :rainbow:\n```\nhope that helps"
mdObjectInCodeBlockExpected="<p><span class=\"h-card\"><a href=\"http://fossbros-anonymous.io/@foss_satan\" class=\"u-url mention\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">@<span>foss_satan</span></a></span> this is how to mention a user</p><pre><code>@the_mighty_zork hey bud! nice #ObjectOrientedProgramming software you've been writing lately! :rainbow:\n</code></pre><p>hope that helps</p>"
// Hashtags can be italicized but only with *, not _.
html:"<p>butting into a serious discussion about programming languages*: \"elixir? I barely know 'er! honk honk!\"</p><p><small>*insofar as any discussion about programming languages can truly be considered \"serious\" since programmers are fucking clowns</small></p>",
expectedPlain:`buttingintoaseriousdiscussionaboutprogramminglanguages*:"elixir? I barely know 'er! honk honk!"
// Check <br> converted to newlines and leading / trailing space removed.
html:" <p>i'm a milf,<br>i'm a lover,<br>do your mom,<br>do your brother</p><p>i'm a sinner,<br>i'm a saint,<br>i will not be ashamed!</p><br> <br>",
expectedPlain:`i'mamilf,
i'malover,
doyourmom,
doyourbrother
i'masinner,
i'masaint,
iwillnotbeashamed!`,
},
{
// Check newlines, links, lists still more or less readable as such.
html:"<p>Hello everyone, after a week or two down the release candidate mines, we've emerged blinking into the light carrying with us <a href=\"https://gts.superseriousbusiness.org/tags/gotosocial\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>GoToSocial</span></a> <strong>v0.18.0 Scroingly Sloth</strong>!</p><p><a href=\"https://github.com/superseriousbusiness/gotosocial/releases/tag/v0.18.0\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">https://github.com/superseriousbusiness/gotosocial/releases/tag/v0.18.0</a></p><p>Please read the migration notes carefully for instructions on how to upgrade to this version. <strong>This version contains several very long migrations so you will need to be patient when upgrading, and backup your database first!!</strong></p><p><strong>Release highlights</strong></p><ul><li><strong>Status edit support</strong>: one of our most-requested features! You can now edit your own statuses, and see instance edit history from other accounts too (if your instance has them stored).</li><li><strong>Push notifications</strong>: probably the second most-requested feature! GoToSocial can now send push notifications to clients via their configured push providers.<br>You may need to uninstall / reinstall client applications, or log out and back in again, for this feature to work. (And if you're using Tusky, <a href=\"https://tusky.app/faq/#why-are-notifications-less-frequent-with-tusky\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">make sure you've got ntfy installed</a>).</li><li><strong>Global instance css customization</strong>: admins can now apply custom CSS across their entire instance via the settings panel.</li><li><strong>Domain permission subscriptions</strong>: it's now possible to configure your instance to subscribe to CSV, JSON, or plaintext lists of domain permissions.<br>Each night, your instance will fetch and automatically create domain permissions (or permission drafts) based on what it finds in a subscribed list.<br>See the <a href=\"https://docs.gotosocial.org/en/latest/admin/domain_permission_subscriptions/\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">domain permission subscription documentation</a> for more information.</li><li><strong>Trusted-proxies helper</strong>: instances with improperly configured trusted-proxies settings will now show a warning on the homepage, so admins can make sure their instance is configured correctly. Check your own instance homepage after updating to see if you need to do anything.</li><li><strong>Better outbox sorting</strong>: messages from GoToSocial are now delivered more quickly to people you mention, so conversations across instances should feel a bit snappier.</li><li><strong>Log in button</strong>: there's now a login button in the top right of the instance homepage, which leads to a helpful page about clients, with a link to the settings panel. Should make things less confusing for new users!</li><li><strong>Granular stats controls</strong>: with the <code>instance-stats-mode</code> setting, admins can now choose if and how their instance serves stats via the nodeinfo endpoints. Existing behavior from v0.17.0 is the default.</li><li><strong>Post backdating</strong>: via the API you can now backdate posts (if enabled in config.yaml). This is our first step towards making it possible to import your post history from elsewhere into your GoToSocial instance. While there's no way to do this in the settings panel yet, you can already use third-party tools like Slurp to import posts from a Mastodon export (see <a href=\"https://github.com/VyrCossont/slurp\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">Slurp</a>).</li><li><strong>Configurable sign-up limits</strong>: you can now configure your sign-up backlog length and sign-up throttling (defaults remain the same).</li><li><strong>NetBSD and FreeBSD builds</strong>: yep!</li><li><strong>Respect users <code>prefers-color-scheme</code> preference</strong>: there's now a light mode default theme to complement our trusty dark mode theme, and the theme will switch based on a visitor's <code>prefers-color-scheme</code> configuration. This applies to all page and profiles, with the exception of some custom themes. Works in the settings panel too!</li></ul><p>Thanks for reading! And seriously back up your database.</p>",
withInlineImg:="<p>Here's an inline image: <img class=\"fixed-size-img svelte-uci8eb\" aria-hidden=\"false\" alt=\"A black-and-white photo of an Oblique Strategy card. The card reads: 'Define an area as 'safe' and use it as an anchor'.\" title=\"A black-and-white photo of an Oblique Strategy card. The card reads: 'Define an area as 'safe' and use it as an anchor'.\" width=\"0\" height=\"0\" src=\"https://example.org/fileserver/01H7J83147QMCE17C0RS9P10Y9/attachment/small/01H7J8365XXRTCP6CAMGEM49ZE.jpg\" style=\"object-position: 50% 50%;\"></p>"
sanitized:=text.SanitizeToHTML(withInlineImg)
sanitized:=text.SanitizeHTML(withInlineImg)
suite.Equal(`<p>Here's an inline image: </p>`,sanitized)
"content":"hello world! #welcome ! first post on the instance :rainbow: !",
"content":"\u003cp\u003ehello world! \u003ca href=\"http://localhost:8080/tags/welcome\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\"\u003e#\u003cspan\u003ewelcome\u003c/span\u003e\u003c/a\u003e ! first post on the instance :rainbow: !\u003c/p\u003e",
"contentMap":{
"en":"hello world! #welcome ! first post on the instance :rainbow: !"
"en":"\u003cp\u003ehello world! \u003ca href=\"http://localhost:8080/tags/welcome\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\"\u003e#\u003cspan\u003ewelcome\u003c/span\u003e\u003c/a\u003e ! first post on the instance :rainbow: !\u003c/p\u003e"
"content":"hello world! #welcome ! first post on the instance :rainbow: !",
"content":"\u003cp\u003ehello world! \u003ca href=\"http://localhost:8080/tags/welcome\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\"\u003e#\u003cspan\u003ewelcome\u003c/span\u003e\u003c/a\u003e ! first post on the instance :rainbow: !\u003c/p\u003e",
"contentMap":{
"en":"hello world! #welcome ! first post on the instance :rainbow: !"
"en":"\u003cp\u003ehello world! \u003ca href=\"http://localhost:8080/tags/welcome\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\"\u003e#\u003cspan\u003ewelcome\u003c/span\u003e\u003c/a\u003e ! first post on the instance :rainbow: !\u003c/p\u003e"
"content":"hello world! #welcome ! first post on the instance :rainbow: !",
"content":"\u003cp\u003ehello world! \u003ca href=\"http://localhost:8080/tags/welcome\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\"\u003e#\u003cspan\u003ewelcome\u003c/span\u003e\u003c/a\u003e ! first post on the instance :rainbow: !\u003c/p\u003e",
testStatus.ContentWarning=`<p>First paragraph of content warning</p><h4>Here's the title!</h4><p></p><p>Big boobs<br>Tee hee!<br><br>Some more text<br>And a bunch more<br><br>Hasta la victoria siempre!</p>`
"spoiler_text":"First paragraph of content warning\n\nHere's the title!\n\nBig boobs\nTee hee!\n\nSome more text\nAnd a bunch more\n\nHasta la victoria siempre!",
"content":"\u003cp\u003ehello world! \u003ca href=\"http://localhost:8080/tags/welcome\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\"\u003e#\u003cspan\u003ewelcome\u003c/span\u003e\u003c/a\u003e ! first post on the instance :rainbow: !\u003c/p\u003e",
"content":"hello world! #welcome ! first post on the instance :rainbow: ! fnord",
"content":"\u003cp\u003ehello world! \u003ca href=\"http://localhost:8080/tags/welcome\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\"\u003e#\u003cspan\u003ewelcome\u003c/span\u003e\u003c/a\u003e ! first post on the instance :rainbow: !\u003c/p\u003e fnord",
"content":"hello world! #welcome ! first post on the instance :rainbow: ! fnord",
"content":"\u003cp\u003ehello world! \u003ca href=\"http://localhost:8080/tags/welcome\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\"\u003e#\u003cspan\u003ewelcome\u003c/span\u003e\u003c/a\u003e ! first post on the instance :rainbow: !\u003c/p\u003e fnord",
"content":"hello world! #welcome ! first post on the instance :rainbow: !",
"content":"\u003cp\u003ehello world! \u003ca href=\"http://localhost:8080/tags/welcome\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\"\u003e#\u003cspan\u003ewelcome\u003c/span\u003e\u003c/a\u003e ! first post on the instance :rainbow: !\u003c/p\u003e",
"content":"this is a very personal post that I don't want anyone to interact with at all, and i only want mutuals to see it",
"content":"\u003cp\u003ethis is a very personal post that I don't want anyone to interact with at all, and i only want mutuals to see it\u003c/p\u003e",
suite.Equal("hello world! #welcome ! first post on the instance <img src=\"http://localhost:8080/fileserver/01AY6P665V14JJR0AFVRT7311Y/emoji/original/01F8MH9H8E4VG3KDYJR9EGPXCQ.png\" title=\":rainbow:\" alt=\":rainbow:\" width=\"25\" height=\"25\" /> !",item.Content)
suite.Equal("<p>hello world! <a href=\"http://localhost:8080/tags/welcome\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>welcome</span></a> ! first post on the instance <img src=\"http://localhost:8080/fileserver/01AY6P665V14JJR0AFVRT7311Y/emoji/original/01F8MH9H8E4VG3KDYJR9EGPXCQ.png\" title=\":rainbow:\" alt=\":rainbow:\" width=\"25\" height=\"25\" /> !</p>",item.Content)
Content:"hello world! #welcome ! first post on the instance :rainbow: !",
Content:"<p>hello world! <a href=\"http://localhost:8080/tags/welcome\" class=\"mention hashtag\" rel=\"tag nofollow noreferrer noopener\" target=\"_blank\">#<span>welcome</span></a> ! first post on the instance :rainbow: !</p>",
Text:"hello world! #welcome ! first post on the instance :rainbow: !",
Content:"this is a Public local-only post that shouldn't federate, but it's still boostable, replyable, and likeable. also it has no stored content type",
Text:"this is a Public local-only post that shouldn't federate, but it's still boostable, replyable, and likeable. also it has no stored content type",
Content:"<p>this is a Public local-only post that shouldn't federate, but it's still boostable, replyable, and likeable</p>",
Text:"this is a Public local-only post that shouldn't federate, but it's still boostable, replyable, and likeable",
Content:"🐢 @the_mighty_zork hi zork, this is a direct message, shhhhhh! 🐢",
Content:"<p>🐢 <span class=\"h-card\"><a href=\"http://localhost:8080/@1happyturtle\" class=\"u-url mention\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">@<span>the_mighty_zork</span></a></span> hi zork, this is a direct message, shhhhhh! 🐢</p>",
Text:"🐢 @the_mighty_zork hi zork, this is a direct message, shhhhhh! 🐢",
Content:`<p>hi <span class="h-card"><a href="http://localhost:8080/@admin" class="u-url mention" rel="nofollow noreferrer noopener" target="_blank">@<span>admin</span></a></span> here's some media for ya</p>`,
Content:`<p>hi <span class="h-card"><a href="http://localhost:8080/@admin" class="u-url mention" rel="nofollow noreferrer noopener" target="_blank">@<span>admin</span></a></span> here's some media for ya</p>`,
tobi
1 year ago
committed by
GitHub