This is a massive PR, so apologies for the poor performance viewing it!
This merges in the content of Mastodon 4.3.0. *This PR isn't ready to
deploy to production*, and I don't recommend trying it in a production
environment. Since I started by merging 4.3.0, this will likely have
reverted at least some of the content of the post-4.3.0 security patches
in the 4.2.x series; those will be brought back as I merge in 4.4.0 and
4.5.0.
I haven't yet tested this PR, so it may not actually run without
additional tweaks. I'm putting it up now for discussion and so multiple
people can test if we want.
---
Here's a few notes on things I ran into and thoughts I had while working
on it:
* We may want to look at `server_banner.jsx,` where I ended up undoing
the v3-style rewrite due to the new react component being used. I'm not
sure what the current goal for it is.
* I may not have 100% merged in all content from `status_action_bar`
correctly.
* There's a number of places where I updated the links for remote
usernames, and they should be tested to make sure they work. They're all
`<Permalink>` elements. I have in my notes that `status_content` might
need double-checking.
* I may also not have gotten the spoiler display right in
`status_content`.
* It looks like previous versions of Hometown had customizations to the
`media_item` display, not all of which was preserved as-is due to
surrounding rewrites. I'm not sure what the vision is and could use
another set of eyes.
* The character count rendering clashed with the newly-introduced
handling of a character count variable upstream, and I'm not sure if the
version I've got here 100% works or not.
* The navigation bar was completely rewritten and no longer has the
components Hometown was customizing; did I properly catch any
replacements? Did we handle all links to accounts to make sure they go
to the remote instance?
* Polls: I reverted the single/multiple choice toggle because upstream
has an actual proper UI for this that should be integrated in one of the
versions we’re merging in. Please double-check that the current version
looks fine to you.
* The font icons will need double-checking. Hometown made some custom
use of the font-awesome icons, while Mastodon 4.3.0 replaced
font-awesome with something else. I may not have caught all places that
font-awesome icons were being used.
* We’ll want to make sure the post display is correct, including
Hometown customizations, because the previous static post page has been
replaced with the React one.
---------
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
Co-authored-by: Matt Jankowski <matt@jankowski.online>
Co-authored-by: Renaud Chaput <renchap@gmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: David Roetzel <david@roetzel.de>
Co-authored-by: Emelia Smith <ThisIsMissEm@users.noreply.github.com>
Co-authored-by: Jeong Arm <kjwonmail@gmail.com>
Co-authored-by: Christian Schmidt <github@chsc.dk>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <noreply@github.com>
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
Co-authored-by: Darius Kazemi <darius.kazemi@gmail.com>
Co-authored-by: diondiondion <mail@diondiondion.com>
Co-authored-by: Echo <ChaosExAnima@users.noreply.github.com>
Co-authored-by: Shugo Maeda <shugo.maeda@gmail.com>
Co-authored-by: emilweth <7402764+emilweth@users.noreply.github.com>
Co-authored-by: Shlee <github@shl.ee>
Co-authored-by: Joshua Rogers <MegaManSec@users.noreply.github.com>
Co-authored-by: Jessica Stokes <hello@jessicastokes.net>
Co-authored-by: PGray <77597544+PGrayCS@users.noreply.github.com>
* Move ActivityPub::FetchRemoteAccountService to ActivityPub::FetchRemoteActorService
ActivityPub::FetchRemoteAccountService is kept as a wrapper for when the actor is
specifically required to be an Account
* Refactor SignatureVerification to allow non-Account actors
* fixup! Move ActivityPub::FetchRemoteAccountService to ActivityPub::FetchRemoteActorService
* Refactor ActivityPub::FetchRemoteKeyService to potentially return non-Account actors
* Refactor inbound ActivityPub payload processing to accept non-Account actors
* Refactor inbound ActivityPub processing to accept activities relayed through non-Account
* Refactor how Account key URIs are built
* Refactor Request and drop unused key_id_format parameter
* Rename ActivityPub::Dereferencer `signature_account` to `signature_actor`
* Fix performance of account timelines
* Various fixes and improvements
* Fix duplicate results being returned
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Fix grouping for pinned statuses scope
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Fix anonymous access to outbox not being cached by the reverse proxy
Up until now, anonymous access to outbox was marked as public, but with a
0 duration for caching, which means remote proxies would only serve from cache
when the server was completely overwhelmed.
Changed that cache duration to one minute, so that repeated anonymous access
to one account's outbox can be appropriately cached.
Also added `Signature` to the `Vary` header in case a page is requested, so
that authenticated fetches are never served from cache (which only contains
public toots).
* Remove Vary: Accept header from webfinger controller
Indeed, we have stopped returning xrd, and only ever return jrd, so the
Accept request header does not matter anymore.
* Cache negative webfinger hits for 3 minutes
* Fix anonymous access to outbox not being cached by the reverse proxy
Up until now, anonymous access to outbox was marked as public, but with a
0 duration for caching, which means remote proxies would only serve from cache
when the server was completely overwhelmed.
Changed that cache duration to one minute, so that repeated anonymous access
to one account's outbox can be appropriately cached.
Also added `Signature` to the `Vary` header in case a page is requested, so
that authenticated fetches are never served from cache (which only contains
public toots).
* Remove Vary: Accept header from webfinger controller
Indeed, we have stopped returning xrd, and only ever return jrd, so the
Accept request header does not matter anymore.
* Cache negative webfinger hits for 3 minutes
* Fix issues with POSIX::Spawn, Terrapin and Ruby 3.0
Also improve the Terrapin monkey-patch for the stderr/stdout issue.
* Fix keyword argument handling throughout the codebase
* Monkey-patch Paperclip to fix keyword arguments handling in validators
* Change validation_extensions to please CodeClimate
* Bump microformats from 4.2.1 to 4.3.1
* Allow Ruby 3.0
* Add Ruby 3.0 test target to CircleCI
* Add test for admin dashboard warnings
* Fix admin dashboard warnings on Ruby 3.0
This also skips fetching the actor completely.
This will be useful if we end up distributing Update activities linked to
account suspensions more widely (they are currently only delivered to
the suspended account's followers), as currently, instances not knowing
about the suspended account would fetch it to then process the suspension.
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Add support for followers synchronization on the receiving end
Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.
* Add tests for followers synchronization on the receiving end
* Add support for follower synchronization on the sender's end
* Add tests for the sending end
* Switch from AS attributes to HTTP header
Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
receiving instance (where “receiving instance” is determined by accounts
sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor
Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.
* Add environment variable to disable followers synchronization
Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.
The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
netloc as a managed account must be managed by the same Mastodon instance
as well.
As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.
Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.
* Only set follower synchronization header on non-public statuses
This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)
* Change how ActivityPub::SynchronizeFollowersService handles follow requests
If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.
* Integrate review feeback
- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes
* Only select required fields when computing followers_hash
* Use actor URI rather than webfinger domain in synchronization endpoint
* Change hash computation to be a XOR of individual hashes
Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.
* Marginally improve followers_hash computation speed
* Further improve hash computation performances by using pluck_each
* Replace incorrect use of distinct with group
Some uses of ActiveRecord::QueryMethods#distinct pass field names but they
are incorrect for the current version of Rails.
ActiveRecord::QueryMethods#group provides the expected behavior and
benefits performance. See commit 6da24aad4cafdef8d8a2c92bac2002a5fc2fe9c8.
* Introduce ApplicationController#cache_collection_paginated_by_id
ApplicationController#cache_collection_paginated_by_id fuses
ApplicationController#cache_collection and Paginable.paginate_by_id.
An advantage of this method is that it prevents from modifying scope which
Paginable.paginate_by_id may provide.
ApplicationController#cache_collection always return an array and there
is no possibility of the scope modification. It is also clear for a
programmer, considering the implication of "cache".
This method can also emit more efficient queries by using
Cacheable.cache_ids before calling Paginable.paginate_by_id.
* Fix the replies collection returning snowflakes ids rather than URIs
Fixes#11568
* Fix min_id in replies queries once self-replies are exhausted
* Fix `next` attribute of replies collection being nil when there are no self-replies
* Rename other_accounts param to only_other_accounts
* Remove Salmon and PubSubHubbub endpoints
* Add error when trying to follow OStatus accounts
* Fix new accounts not being created in ResolveAccountService
* Return sensible HTTP status for ActivityPub inbox processing
* Return sensible HTTP status for salmon slap processing
* Return additional information to debug signature verification failures
Misty De Méo
Claire
Eugen Rochko
ThibG
Akihiko Odaki
abcang
ysksn
puckipedia
Akihiko Odaki